akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Question about multi-GPU training

Fiyero_P: <p>Hi<br>
I have a 1080ti and now thinking should I get a 2080ti<br>
If I apply DataParallel to train the model in Pytorch with 1080ti and 2080ti, will 1080ti become the bottleneck to the training process?</p>
<p>Sorry I am not familiarize with the DataParallel process.<br>
Lets say the whole dataset can be divided into 10 batches, will each GPU be assigned with <strong>fixed number of batches</strong> (1080ti and 2080ti each will train 5 batches) ?</p>
<p>or if one GPU trains faster it will take more batches?</p>
<p>Thanks</p>

ptrblck: <p>If you are using <code>nn.DataParallel</code>, each GPU will get the same batch size (if possible).<br>
If you look at the <a href="https://github.com/pytorch/pytorch/blob/74572fc985ace7f024a885218641dbfe0b0dd021/torch/nn/parallel/data_parallel.py#L168" rel="nofollow noopener">internals</a> you might adapt the code and use a custom chunking approach.</p>
<p>However, I would generally advice against a system with mixed GPUs.<br>
E.g. your 2080TI will have TensorCores, which could speedup FP16 calculations, while your 1080Ti won’t be able to do it.</p>

Fiyero_P: <p>thanks so much for your reply.<br>
Just curious for the performance’s point of view, would you replace 1080ti by 2080ti ? or<br>
train model with dual 1080ti gpu?</p>
How to load state_dict to Module in libtorch C++ for Inference

Talha_Irfan: <p>Hi, So I have use case of using our Pytorch model in C++, I have</p>
<ul>
<li>Model Architecture in Pytorch (Python code)</li>
<li>.pth file</li>
</ul>
<p>I’m really new to this, I have seen torch script and tracing. I’m also confused on what to use and also if I use any of this, how can I accommodate my ‘.pth’ file in that.</p>

tom: <p>The <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">tutorial way</a> of moving models to C++ for inference is through tracing/scripting the model. The main alternative would be to re-create the model in C++ using the nn API there and then loading the weights (e.g. torchvision does that for the conv net models for testing).</p>
<p>Best regards</p>
<p>Thomas</p>
How to use annotations in xml to train dataset

Akshat_Upadhyay: <p>Hi!, i am using SVT dataset (link - <a href="http://www.iapr-tc11.org/mediawiki/index.php/The_Street_View_Text_Dataset" rel="nofollow noopener">http://www.iapr-tc11.org/mediawiki/index.php/The_Street_View_Text_Dataset</a>) It contains street view text and i want to train a model to learn to label text  in those images. The problem is i dont know how to deal with the annotations provided in the xml file, i mean i have the input(images to the network) but how should i provide the output(labels for each image) . Please help ASAP</p>
Too many values to unpack (expected 2) error occurs when training

Stanley_C: <p>Hi, the bellow section of code is creating the error in the title.</p>
<pre><code class="lang-auto">for epoch in range(3): # 3 full passes over the data
    for data in trainset:  # `data` is a batch of data
        X, y = data  # X is the batch of features, y is the batch of targets.
        net.zero_grad()  # sets gradients to 0 before loss calc. You will do this likely every step.
        output = net(X.view(-1,11))  # pass in the reshaped batch (recall they are 28x28 atm)
        loss = F.nll_loss(output, y)  # calc and grab the loss value
        loss.backward()  # apply this loss backwards thru the network's parameters
        optimizer.step()  # attempt to optimize weights to account for loss/gradients
    print(loss)  # print loss. We hope loss (a measure of wrong-ness) declines!
</code></pre>
<p>Note: I credit my code to sentdex, I’m only trying to modify it to use for my dataset.<br>
My repository with full code: <a href="https://github.com/itisyeetimetoday/pytorch_reggression" rel="nofollow noopener">https://github.com/itisyeetimetoday/pytorch_reggression</a></p>
<p>Sorry about the messy/incorrect formating. This is my first post here. Please let me know how to fix any errors or mistakes with the formating and site usage in general.</p>

ptrblck: <p>Which line of code is throwing this error?<br>
The linked repository thrown a 404 error and there seems to be another TF implementation.</p>
<p>If the error is thrown in your <code>trainset</code>, you should check, that its <code>__getitem__</code> method returns two values, since you are unpacking them to <code>X</code> and <code>y</code>.</p>
How to add my custom layer? after load pre traind weight?

439327bcc62cd8e45e96: <p>Does anyone know how to add my custom layer to a specific location after loading pre traind weights? without breaking the pre traind weight.</p>

ptrblck: <p>You could add the extra layer, manipulate the <code>forward</code> method as you wish, and load the <code>state_dict</code> passing <code>strict=False</code>.<br>
Here is a small example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        self.fc2 = nn.Linear(1, 1)
        
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

class MyModelModified(nn.Module):
    def __init__(self):
        super(MyModelModified, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        self.fc2 = nn.Linear(1, 1)
        self.extra = nn.Linear(1, 1)

    def forwars(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.extra(x)
        return x

model = MyModel()
state_dict = model.state_dict()

model_mod= MyModelModified()
model_mod.load_state_dict(state_dict, strict=False)
&gt; _IncompatibleKeys(missing_keys=['extra.weight', 'extra.bias'], unexpected_keys=[])

# Check for equal params
print(model.fc1.weight == model_mod.fc1.weight)
&gt; True
print(model.fc1.bias == model_mod.fc1.bias)
&gt; True
print(model.fc2.weight == model_mod.fc2.weight)
&gt; True
print(model.fc2.bias == model_mod.fc2.bias)
&gt; True
</code></pre>
RecursionError calling super().__call__ in forward

theevann: <p>I want to subclass <code>Sequential</code> to create a Residual sequential class in a few lines.</p>
<pre><code class="lang-auto">class Residual(nn.Sequential):
    def forward(self, x):
        return x + super().__call__(x)
</code></pre>
<p>However, this crashes with:</p>
<blockquote>
<p><strong>RecursionError</strong>: maximum recursion depth exceeded while calling a Python object</p>
</blockquote>
<p>If I understand well, the <code>super().__call__</code> actually calls <code>nn.Module.__call__</code> which then calls the <code>forward</code> of <code>self</code> … which is the one I overrode, that calls the <code>__call__</code> again …</p>
<p>Changing the <code>__call__</code> to <code>forward</code> solves the problem :</p>
<pre><code class="lang-auto">class Residual(nn.Sequential):
    def forward(self, x):
        return x + super().forward(x)
</code></pre>
<p>But I will lose the hooks then, is that right ?<br>
Is my global understanding correct ?<br>
Could it poses any problem to modules such as <code>DataParallel</code> that the <code>forward</code> of  <code>Sequential</code> is callled directly ?</p>
<p>Thank you much in advance !</p>

prabhuteja12: <p>The simplest solution I can come up with is the following</p>
<pre><code class="lang-auto">class Residual(nn.Module):
    def __init__(self, *mods):
        super().__init__()
        self.seq = nn.Sequential(*mods)
       
    def forward(self, x):
        return x + self.seq(x)
</code></pre>
<p>This should retain all the hooks with 3 extra lines of code!</p>

albanD: <p>Hi,</p>
<p>The hooks for this module have already been called before entering the <code>forward</code> of your module.<br>
So you should use <code>super().forward(x)</code>.</p>
Feature importance / Regression trees

bsalanon: <p>Good morning all,</p>
<p>I am making some machine learning on the metro traffic volume from there: <a href="http://archive.ics.uci.edu/ml/datasets/Metro+Interstate+Traffic+Volume" rel="nofollow noopener">http://archive.ics.uci.edu/ml/datasets/Metro+Interstate+Traffic+Volume</a></p>
<p>The aim is to predict an hourly traffic volume according to various features that are:</p>
<ul>
<li>holiday : Yes/No</li>
<li>temperature in C</li>
<li>rain and snow fall in mm</li>
<li>clouds cover %</li>
<li>hours of the day : integer from 0 to 23</li>
<li>day of the week (wd_i, i=0…6).</li>
</ul>
<p>I made a <strong>random forest</strong> and get a 20% error on traffic prediction, which is OK for me. Then I have plotted the FEATURES IMPORTANCE and got:</p>
<p>Feature ranking:</p>
<ol>
<li>holiday  (0.736229)</li>
<li>temp  (0.137736)</li>
<li>rain_1h  (0.029498)</li>
<li>snow_1h  (0.028497)</li>
<li>clouds_all  (0.023778)</li>
<li>weather_main (0.019016)</li>
<li>hours  (0.004792)</li>
<li>wd_0  (0.004365)<br>
…</li>
<li>wd_6 0 (0.000018)</li>
</ol>
<p>Comments :</p>
<ul>
<li>‘holiday’ is very important because of course, on holidays, traffic is very slow the whole day !</li>
<li>‘hours’ seems to be almost insignificant.</li>
</ul>
<p>Action: I have REMOVED  ‘hours’ from the features because it was so insignificant and then I run random forest again.</p>
<p>Result: result is a disaster (error jumped from 20 to 60%) and YES, it is normal, because traffic volume depends so much on the hour of the day !</p>
<p>Question:</p>
<ul>
<li>why was ‘hours’ put with a so low ranking on the features importance analysis ?</li>
<li>it seems not being a good  idea at all to withdrawn features based on there .feature_importances_ method of RandomForestRegressor(). Is that correct ?</li>
<li>So what does .feature_importances_ precisely measure ?</li>
</ul>
<p>Thanks in advance for your comments !</p>

ptrblck: <p>Which library are you using?<br>
The lib should have some information, how the feature importance is calculated.</p>
<p>You are posting in the PyTorch forum, which might not be the best place to ask about other toolkits. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

bsalanon: <p>Yes you are correct. I was confused… I am using SKLEARN so it is not the right place !</p>
Torch.argmax return two values 0 and 2

XiaoAHeng: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/149435eaa66e5af05b9b39bb96071d043f3b98df.png" data-download-href="https://discuss.pytorch.org/uploads/default/149435eaa66e5af05b9b39bb96071d043f3b98df" title="%E5%9B%BE%E7%89%87"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/149435eaa66e5af05b9b39bb96071d043f3b98df_2_690x209.png" alt="%E5%9B%BE%E7%89%87" data-base62-sha1="2W35wO7x3mSFzlIStdQM7hlRZj9" width="690" height="209" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/149435eaa66e5af05b9b39bb96071d043f3b98df_2_690x209.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/149435eaa66e5af05b9b39bb96071d043f3b98df_2_1035x313.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/149435eaa66e5af05b9b39bb96071d043f3b98df_2_1380x418.png 2x" data-dominant-color="333032"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E5%9B%BE%E7%89%87</span><span class="informations">1547×470 88.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
my predicted segmentation outputs are 4<em>2</em>4<em>256</em>256, after torch.argmax, I want four,i.e. 0, 1,2,and 3 values in the tansor ,however, I just got two values. Anyone can help me ?</p>

ptrblck: <p><code>np.unique</code> returns all unique values in the numpy array.<br>
If there are only two unique values, you will just get these two.</p>
<p>I’m currently not sure, what your code is doing and what the dimensions refer to.<br>
Could you explain your use case a bit?</p>

XiaoAHeng: <p>my code is to segment sequence image ,the annotaiton is the ground truth, its shape is Batch<em>frames</em>Height<em>Width. There are 4 objects to segment in each image, the outputs of predicted segmentation is Batch</em>Frames<em>Channel</em>Height*Width. By rights, the outputs maxed torch.argmax have 4 values , however, why are there only two values?</p>

ptrblck: <p>The output of <code>torch.argmax</code> has 4 dimensions, not necessarily 4 unique values.</p>

XiaoAHeng: <p>Yes, I know the output of  <code>torch.argmax</code>  has 4 dimensions. I mean that the output maxed by torch.argmax  should be consist of 4 different values, why only two values?</p>

XiaoAHeng: <p>Thank you all the same</p>

albanD: <p>Hi,</p>
<p>I think it has only two values because all the max are at either index 0 or 2 for your particular tensor <code>outs</code>. If you replace that with a completely random Tensor, you will (most likely) get all 4 values.</p>
Avoiding a sqrt to 0 values is breaking the backpropagation

Dhorka: <p>Hi,</p>
<p>I am trying to apply a torch.sqrt to a vector that contains 0 values. My goal is to avoid the 0 values. I mean only a pply the torch.sqrt to values that are different of 0. To do that I have try the following:</p>
<pre><code class="lang-auto">t=torch.tensor([[0.0,1.0,2.0],[3.0,0.0,4.0],[5.0,6.0,0.0]])
aux = t[t!=0].clone()
aux = torch.sqrt(aux)
t[t!=0] = aux.clone()
</code></pre>
<p>This works, but breaks the backpropagation. I am getting the following error:</p>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [7822, 1]], which is output 0 of DivBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>It seems like this line <code>aux=t[t!=0].clone()</code> is breaking the backpropagation. How can I solve it? Is there any way to achieve my goal easily?</p>

KFrank: <p>Hi Dhorka!</p>
<aside class="quote no-group" data-username="Dhorka" data-post="1" data-topic="57373" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/b2d939/48.png" class="avatar"> Dhorka:</div>
<blockquote>
<p>I am trying to apply a torch.sqrt to a vector that contains 0 values. My goal is to avoid the 0 values. I mean only a pply the torch.sqrt to values that are different of 0.</p>
</blockquote>
</aside>
<p>I speculate that you wish to avoid <code>torch.sqrt (0)</code> because the<br>
derivative of <code>sqrt</code> is infinite at 0.</p>
<aside class="quote no-group">
<blockquote>
<p>How can I solve it? Is there any way to achieve my goal easily?</p>
</blockquote>
</aside>
<p>If your goal is, in fact, to avoid the infinite derivative, you could<br>
simply add a small “epsilon” to your value before calling <code>sqrt</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">epsilon = 1.e-8
t = torch.sqrt (t + epsilon)
</code></pre>
<p>Now the infinite derivative for (elements of) <code>t = 0</code> just becomes<br>
a large derivative (specifically <code>1 / (2 * sqrt (epsilon))</code>).</p>
<p><code>epsilon</code> should be chosen to be comfortably smaller than the<br>
typical small values that show up in this part of your computation.</p>
<p>Best.</p>
<p>K. Frank</p>
[DOUBT] Copying of Tensors

Krish: <p>I have a AutoEncoder Model in which I return both the encoding and the reconstructed output. This is the forward method that I use:</p>
<pre><code class="lang-auto">def forward(self, x):
        for module in self.encoder:
            x = module(x)
        
        encoding = x*1 # added the multiply just for testing
        
        for module in self.decoder:
            x = module(x)
        
        return x, encoding
</code></pre>
<p>In line 4, is the simple assign I am doing to copy wrong?<br>
When I check the output, they both refer to different <code>grad_fn</code>. Am I missing something here?</p>

ptrblck: <p>I’m not sure, what your use case is, but <code>encoding</code> should be assigned to the multiplication with the “old” <code>x</code> tensor, since <code>x</code> will be overwritten in the following loop.</p>

Krish: <p>Thank you for the reply. I was thinking that I would have to use <code>.clone()</code> for copying the tensor.</p>
<p>I am planning to use the <code>encoding</code> as well as <code>x</code> for optimisation. Will that work fine in this case?</p>
Creating a custom loss-function compatible with an automatic .backward()

carlstronaut: <p>Hello,</p>
<p>I’m currently looking to re-implement the paper <a href="https://arxiv.org/pdf/1603.08511.pdf" rel="nofollow noopener">Colorful Image Colorization</a>, in which they implement a custom loss function. I’m wondering on how to go about to create a custom loss function, in which I doesn’t have to specify the gradients myself (compatible with autograd?).</p>
<p>I read through <a href="https://discuss.pytorch.org/t/custom-loss-functions/29387">this</a> thread, and if I have understand it correctly, as long as I use PyTorch operations to define a loss function, autograd will be able to generate the backward-steps automatically (which later can be used by the optimizer)?</p>

albanD: <p>Hi,</p>
<p>Yes exactly, you just need to implement your function with pytorch operations !<br>
Then you can wrap it into an nn.Module if you want or just a regular python function. Whichever fits best in your code.</p>
To(device) not working for hidden variable

serenayj: <p>Hi I have this piece of codes where I want to pass the input and hidden variable to a LSTM cell, and I tried to move hidden variable h to cuda device because of error “Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu” :</p>
<p>def _run_rnn_packed(self, cell, x, x_lens, h=None):<br>
R.pack_padded_sequence(x,x_lens,batch_first=True).to(self.device)<br>
if h is not None:<br>
print(“hidden variabe device before to device : {}”.format(h.is_cuda))<br>
h.to(self.device)<br>
print(“hidden variabe device after to device : {}”.format(h.is_cuda))<br>
output, h = cell(x_packed, h)<br>
else:<br>
output, h = cell(x_packed)</p>
<p>but “h.to(device)” doesn’t seem to work. The second print statement still gave me false. Even when I did “h.cuda()”, the tensor is still not on GPU, Any idea how to fix it?</p>

scarecrow21: <p>Are you initializing the hidden state yourself? If you don’t manually initialize the hidden state, PyTorch will auto-initialize them for you (all zeros). This way, you wouldn’t have to mess around with shifting the hidden to cuda.</p>
<p>Side note: There’s an option in the editor called “Preformatted text”. Please use that to format your code to make it more readable. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Hi,</p>
<p>The <code>.to()</code> operation on Tensors is not inplace, you need to do <code>h = h.to(self.device)</code>.</p>

serenayj: <p>The hidden state is passed from another layer so I didn’t use the default initialization. But I found the reason.</p>
<p>Btw thanks for the side note I didn’t notice the  “Preformatted text” button. Will do next time!</p>

serenayj: <p>Thanks that’s also what I need.</p>
Converting the image to tensor on GPU

blackbirdbarber: <p>Can we load in PyTorch images from the disk to GPU and then possible convert them fast to tensor?</p>

albanD: <p>Hi,</p>
<p>You should load the image in a Tensor then send it to the GPU.<br>
Can you be a bit more precise on what you want to do exactly?</p>

blackbirdbarber: <p>Just loading a file into GPU memory directly and converting it to a tensor. Does this work? Or loading bunch of files to be efficient.</p>

albanD: <p>How do you load a file into GPU memory directly?</p>

blackbirdbarber: <p>This is what I was hopping to get the idea <a class="mention" href="/u/alband">@albanD</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Ho, well you can’t <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>To make sure you get enough data for your training procedure, we usually use <code>DataLoader</code> with multiple workers. That way you have multiple processes loading and preprocessing data.<br>
You should check the examples in the doc for how to do that.</p>

blackbirdbarber: <p>OK, what you said has lot of sense.<br>
I thought there is a way and that PyTorch has some routines  doing basically this:</p>
<pre><code class="lang-auto">// Declare data storage
std::vector&lt;float&gt; my_data;
float *d_my_data;
 
// Read input file
std::ifstream fin("my_data_file.txt");
while (!fin.eof()) {
	float data;
	fin &gt;&gt; data;
	if (!fin.eof()) {
		my_data.push_back(data);
	}
}
fin.close();
 
// Allocate memory on the GPU and copy data
cudaMalloc((void **)&amp;d_my_data, my_data.size()*sizeof(float));
cudaMemcpy(d_my_data, my_data.data(), my_data.size()*sizeof(float),
		   cudaMemcpyHostToDevice);
 
// Do something with the data
my_CUDA_kernel&lt;&lt;&lt;num_blocks,num_threads&gt;&gt;&gt;(d_my_data);
 
// Copy data back to the host
cudaMemcpy(my_data.data(), d_my_data, my_data.size()*sizeof(float),
		   cudaMemcpyDeviceToHost);
 
// Free the GPU memory when your done
cudaFree(d_my_data);
</code></pre>
<p>This code I just saw, but the idea to ask the question was a blitz.<br>
I don’t know if this is comparable with super duper <code>DataLoader</code>, but I guess this would not be the path you would recommend, right?</p>

albanD: <p>Ho sorry I were using python !<br>
In cpp, I would just read the file directly into a cpu tensor instead of a vector.<br>
Then send that Tensor on the GPU.</p>
<p>Note that you can always get the pointer to the gpu data from the pytorch cuda tensor to use in your own kernel.</p>
<p>Some example code for the cpp extensions include working with cuda Tensors <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-mixed-c-cuda-extension" rel="nofollow noopener">here</a>. You can ignore the part that exports it to python.</p>
Tensor index to index copy

Arun_Mallya: <p>Suppose I have 2 3-D tensors <code>A</code>, and <code>B</code> and want to copy some elements from <code>B</code> into <code>A</code>. Specifically, I have two lists of the form <code>[(x_1, y_1), (x_2, y_2), ...]</code> and <code>[(x'_1, y'_1), (x'_2, y'_2), ...]</code> and I want to perform <code>A[x_1, y_1, :] = B[x'_1, y'_1, :]</code> and so on. Is there any fast way of doing this or is a for-loop the only way?</p>
<p>Also, will such an operation support the flow of gradients from <code>A</code> to elements copied from <code>B</code>?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>Here is an example implementation.<br>
Gradients will flow as expected for the copied values from A to B (not gradient for indices of course).<br>
Note that inplace operations sometimes prevent the autograd from being able to compute gradients so there is a flag in case you encounter this error.<br>
Let me know if you have more questions.</p>
<pre><code class="lang-auto">import torch

A = torch.zeros(3, 4, 6)
B = torch.rand(3, 4, 6)

indA = torch.LongTensor([[0, 0], [0, 1], [1, 1]])
indB = torch.LongTensor([[1, 1], [2, 1], [2, 2]])

def indices_copy(A, B, indA, indB, inplace=True):
    # To make sure our views below are valid
    assert A.is_contiguous()
    assert B.is_contiguous()

    # Get the size
    size = A.size()

    # Collapse the first two dimensions, so that we index only one
    vA = A.view(size[0] * size[1], size[2])
    vB = B.view(size[0] * size[1], size[2])

    # If we need out of place, clone to get a tensor backed by new memory
    if not inplace:
        vA = vA.clone()

    # Transform the 2D indices into 1D indices in our collapsed dimension
    lin_indA = indA.select(1, 0) * size[1] + indA.select(1, 1)
    lin_indB = indB.select(1, 0) * size[1] + indB.select(1, 1)

    # Read B and write in A
    vA.index_copy_(0, lin_indA, vB.index_select(0, lin_indB))

    return vA.view(size)

print("Inputs")
print(A)
print(B)
print(indA)
print(indB)

indices_copy(A, B, indA, indB)

print("Output inplace")
print(A)


A = torch.zeros(3, 4, 6)
new_A = indices_copy(A, B, indA, indB, inplace=False)

print("Output out of place")
print(new_A)
print("Unmodified A")
print(A)
</code></pre>

Arun_Mallya: <p>Awesome, this is exactly what I needed.<br>
Thanks a lot!</p>
Maximize KL-divergence

shivangi: <p>Hello everyone,</p>
<p>I want to maximize the KL-divergence between 2 classes, The way I am doing this getting their feature vector and then optimizing for the following loss:</p>
<pre><code class="lang-auto">loss = - F.kl_div(A.log(), B, None, None, 'sum')
loss..backward()
</code></pre>
<p>There are some additional losses as well like reconstruction loss and so on. Also all of my other losses are positive and relatively small compared to this loss. Very soon my loss reaches to negative Infinity and the network doesn’t actually learn anything. Any leads on how to fix this kind of problem?<br>
Or if there is other way to maximize the difference between 2 distributions?</p>
Is there any way the optimizer can figure out he is dealing with head layers?

redknightlois: <p>Hi all,</p>
<p>I want a custom optimizer to react to the last layers of a graph in a specific way. Usually, the last group in a param_group is the last, but many networks have multiple output layers. Is there any way to know at the optimization step of the optimizer which ones are output layers?</p>
<p>Thanks in advance.</p>

paganpasta: <p>If you have named the output layers, maybe you can segregate the layers based on that while doing<br>
the back-propagation yourself.</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
    print name, param.data
</code></pre>

albanD: <p>Hi</p>
<p>All optimizers accept param_groups as argument to construct them.<br>
So you can require the user to provide parameters with two groups called <code>body</code> and <code>ouput</code> in your optimizer init.<br>
That way you know exactly which is which.</p>

redknightlois: <p>I was actually looking if there was any way to do that generically, but <a class="mention" href="/u/alband">@albanD</a> approach is a reasonable requirement.</p>

albanD: <p>Unfortunately, in pytorch, a user can define its network in any way it wants. So I don’t think there is a generic way to do this <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
How to do pixel wise regression on an image

Remote_Senser: <p>This question is conceptual as I have a model working decently well and I am looking for different things to try out.</p>
<p>Research Problem: I am trying to classify each pixel in an image with values from 0-&gt;1. This is a regression problem.</p>
<p>Dataset: I have 30 images that are 1000x1000 and ground truth for those images.</p>
<p>Current Approach: For the training, I am extracting 65x65 patches to train a CNN which gives a label to the center pixel. Then in the evaluation of the model, I am striding through the full image labelling each pixel.</p>
<p>Is there any other methods I can try out.</p>
C++ front end equivalent of python swapaxes

Source82: <p>Hi,</p>
<p>I have tried reading around to see if this question has been answered before, but haven’t found anything specific.</p>
<p>In python I could easily convert my image numpy array to cube using image.swapaxes(1, 2).swapaxes(0, 1)</p>
<p>or simply using image.transpose((2,0,1)) before converting to tensor for making inference.</p>
<p>Please is there any quick way of doing this assuming I have read in an image with opencv in the c++ front-end?</p>
<p>Any hint or example will be appreciated.</p>

tom: <p>Does permute help?</p>
<p>Best regards</p>
<p>Thomas</p>

Source82: <p>Thanks, although I have just used cv::split to split and concatenate the channels, and seems to work.</p>
Pytorch with Horovod

ammalikwaterloo: <p>I am trying to use Pytorch with Horovod. I am trying to run one of the example. I am getting the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "pytorch_imagenet_resnet50.py", line 3, in &lt;module&gt;
    import torch
  File "/ccs/home/amalik/.conda/envs/py366/lib/python3.6/site-packages/torch/__init__.py", line 84, in &lt;module&gt;
    from torch._C import *
ImportError: /ccs/home/amalik/.conda/envs/py366/lib/python3.6/site-packages/torch/lib/libtorch_python.so: undefined symbol: MPIX_Query_cuda_support
</code></pre>
<p>However, when I import torch on simple python command, I don’t get the error.</p>
<pre><code class="lang-auto">a&gt;&gt; python
&gt;&gt;&gt; import torch
&gt;&gt;&gt;
</code></pre>
<p>no error message</p>
Access to gradient through hooks on parameter's tensors

ttoosi: <p>Hello,</p>
<p>How can I compute the backward gradients for each tensor in model parameters (p for p in model.parameters())? Notice that I don’t want to use module.backward_hook(), instead, I want to be able to put the hooks on tensors of parameters using tensor.register_hook().<br>
Specifically, in the snippet below, I want to know what I should put in hookfn to have corresponding gradients for each parameter.</p>
<pre><code class="lang-auto">
model = torchvision.models.resnet18().cuda()`

class HookTensor():
    def __init__(self, tensor, backward=False):
        self.hook = tensor.register_hook(self.hook_fn)
    def hook_fn(self, tensor):
        self.tensor = tensor
        pass
    def close(self):
        self.hook.remove()

hookB = [HookTensor(p) for p in modelF.parameters() if p.requires_grad]

inputs, target = next(iter(train_loader))
inputs = inputs.cuda()
target = target.cuda()
outputs = model(inputs)
loss = criterion(outputs, target)
loss.backward()

Params = [p for p in model.parameters() if p.requires_grad]
</code></pre>
<p>Thank you,<br>
Tahereh</p>

albanD: <p>Hi,</p>
<p>The hook does not take the tensor as input but the gradient.<br>
So the right signature would be:</p>
<pre><code class="lang-auto">def hook_fn(self, grad):
    pass
</code></pre>
<p>So you directly have access to the gradient and can do whatever you want.</p>

ttoosi: <p>Thank you for your response.<br>
This <code>hook_fn</code> is equivalent to the backward hook on modules that gives the gradients. If I want to get the forward hooks what would be the correct way of defining <code>hook_fn</code>?</p>
<p>Thanks,</p>

albanD: <p>Hi,</p>
<p>There is no forward hook for tensors. Because by the time you have the tensor to be able to hook it, you just computed it’s value, so you have access to it directly, no need for hook <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ttoosi: <p>I understand, but I want to avoid outputting a lot of intermediate activities from my forward function. Just like when we use the <code>register_forward_hook</code> on Modules, I wonder if there is an equivalent way to define <code>resigter_hook</code> on the parameters of a model.</p>
<p>Thanks,</p>

ttoosi: <p>Aaa, nvm! Got it. I can directly use the values in model.parameters</p>
VAE: can't reconstruct input images using MSE loss

rim_zayani: <p>Hello,<br>
I have implemented a VAE in Pytorch with MNIST database.<br>
I noticed that when I use the F.mse_loss, I don’t get a good reconstruction of the input images.<br>
Here is an exmaple of the reconstruction of some input images:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c0d35545a72f637aa5e43c65cc5e836eedb9223.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2c0d35545a72f637aa5e43c65cc5e836eedb9223" title="mnistlr0001reg01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c0d35545a72f637aa5e43c65cc5e836eedb9223_2_689x285.jpeg" alt="mnistlr0001reg01" data-base62-sha1="6hHj3BYe98TqLjJWN7xlkkH7hib" width="689" height="285" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c0d35545a72f637aa5e43c65cc5e836eedb9223_2_689x285.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c0d35545a72f637aa5e43c65cc5e836eedb9223_2_1033x427.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c0d35545a72f637aa5e43c65cc5e836eedb9223.jpeg 2x" data-dominant-color="E3E4E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mnistlr0001reg01</span><span class="informations">1292×535 155 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But when I use the F.binary_cross_entropy loss, I get a good reconstruction:<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45ba5068826d3e63374dfcda782d50a571702758.png" alt="reconstruction_9" data-base62-sha1="9WQ7xj1KbNYdWALrcJ1XuP7ekGY" width="242" height="62"><br>
Can you please help me understand why I can’t get a good reconstruction of the input images when using the mse loss?<br>
I appreciate your help!</p>

ptrblck: <p>Based on <a href="https://www.deeplearningbook.org/contents/mlp.html" rel="nofollow noopener">Chapter 6.2.2, DeepLearningBook</a>, I would assume <code>nn.BCELoss</code> to work better, as you are dealing with a Bernoulli Output Distribution, but I’m no expert on this topic, so let’s wait for others to chime in. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

rim_zayani: <p>Thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
Can I fine fune a modified network?

439327bcc62cd8e45e96: <p>I want to do fine turning.</p>
<p>For example, there is a pre-trained weighyt traind from the layer1-&gt; layer2-&gt; layer3 structure.</p>
<p>But I want to fine-tune on layer1-&gt; my_layer1-&gt; layer2-&gt; my_layer2-&gt; layer3 structure.</p>
<p>Is there any way?</p>
<p>please. help me</p>

albanD: <p>Hi,</p>
<p>This is technically possible by setting the weights to the pretrained layers yourself with something like <code>model.layer1.load_state_dict(pretrained_layer1_state_dict)</code>.<br>
That being said, the insertion of your custom layers might change the behavior of the network completely and make the initialization useless.</p>
Save_for_backward in no_grad mode

vadimkantorov: <p>Does <code>save_for_backward</code> prevent memory reuse in <code>torch.no_grad()</code> mode? Should I skip <code>save_for_backward</code> in eval, no_grad mode?</p>

albanD: <p><code>save_for_backward</code> will have no effect when you run in <code>no_grad</code> mode. You don’t need to worry about it.</p>
Installing pytorch 1.2 to get return_counts in torch.unique

lelouedec: <p>As the title says I am trying to get a newer version of pytorch to get this option in the torch.unique function. However when I do<br>
<code>conda install pytorch torchvision cudatoolkit=10.1 -c pytorch</code></p>
<p>I do not get the latest version of pytorch (just the 1.0), which apparently does not include this option.<br>
How can I get the correct version ?</p>
<p>EDIT :<br>
I of course tried :<br>
<code>conda install pytorch=1.2 torchvision cudatoolkit=10.1 -c pytorch</code><br>
but I cannot compile C++ extension as it says the cuda installation is not found and there is no cuda_make_macros.h</p>

ptrblck: <p>Could you try to use <code>cudatoolkit=10.0</code>?<br>
If I’m not mistaken, conda will try to install PyTorch <code>1.0.0</code> with CUDA9.0 if you specify a wrong <code>cudatoolkit</code> version.</p>

lelouedec: <p>Alright that was the solution, so it is ok to use a different toolkit than the one installed on the system apparently, which is nice.<br>
Thank you very much</p>

ptrblck: <p>Yes, the binaries ship with their own CUDA and cudnn, so that you would just need to install the NVIDIA drivers locally.</p>
Normalizing/Un-normalizing image tensors

tomt: <p>From my dataset I have three numpy variables/channels which can be characterized as RGB respectively. Furthermore I converted them into tensors using torch.from_numpy(u) and concatenated them into an image tensor as:<br>
HR_data = torch.cat((u_tensor,v_tensor,w_tensor), dim=1)<br>
Then I normalized like this: HR_data_norm = (HR_data - HR_data.mean()) / HR_data.std()<br>
and used TensorDataset to put it into a dataloader.<br>
My question is, should I normalize the numpy arrays before I convert them into tensors or is my method ok?</p>

ptrblck: <p>You shouldn’t see a difference between the numpy and PyTorch approach.<br>
However, usually you would normalize the data using the statistics from the channels, i.e.:</p>
<pre><code class="lang-python">x = torch.randint(0, 256, (3, 224, 224)).float()
y = (x - x.mean([1, 2], keepdim=True)) / x.std([1, 2], keepdim=True)
print(y.mean(), y.std())
&gt; tensor(-4.6021e-08) tensor(1.0000)
</code></pre>
Run custom components on GPU

Knievel: <p>I wrote a custom network that uses custom RNN cells that I also wrote. Everything works on CPU and I now try to get it to run on a GPU for faster training.<br>
I’m registering all weights as well as a weight mask (with <code>requires_grad=False</code>) as Parameters in my RNN cells so they get passed to the GPU by just using <code>.to(device)</code> on my network. This seems to work. However in the forward pass method of my network I’m using three constructs that cause some trouble. (The code is too large to paste here, so I try to explain as good as I can.)</p>
<ol>
<li>A python list <em><strong>contexbuffer</strong></em>, where I append hidden states that I computed and need to access again in the same forward pass.</li>
<li>A torch tensor <em><strong>output</strong></em> initialized with zeros that I fill with the final outputs after all hidden states are computed.</li>
<li>Torch tensors <em><strong>state_mask</strong></em>, zero-one masks where at least one is generated per point in the sequence and passed together with input and hidden states to the RNN cell for computation of the next hidden state.</li>
</ol>
<p>I tried moving the hidden states from the context buffer as well as the state masks to the GPU right before computation in the RNN cell. This works, but I feel like this is far from ideal. The didn’t manage to get the output to be on the GPU however, I always get</p>
<blockquote>
<p>Expected object of backend CPU but got backend CUDA for argument <span class="hashtag-raw">#2</span> ‘target’</p>
</blockquote>
<p>when calculating the loss with <code>.to(device)</code> on the target.</p>
<p>Is there some way to initialize those lists/tensors so that they are moved to the GPU with the whole network?</p>

ecdrid: <p>Try this <code>torch.cuda.set_device(0)</code>; This will put everything to GPU by default.</p>
<blockquote>
<p>Expected object of backend CPU but got backend CUDA for argument <span class="hashtag-raw">#2</span> ‘target’</p>
</blockquote>
<ul>
<li>The error says that the loss was expecting something to be on CPU but rather it was found to be on GPU.</li>
<li>So share your loss function maybe and the way you are calling it (might help others to help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> )</li>
</ul>

Knievel: <p>Thanks for the answer, I’ll try when I get home.</p>
<p>To the error: I’m moving input and target to the GPU with <code>.to(device)</code>, then pass the input through my network (which has also been moved to GPU) with <code>prediction = model(input)</code>.</p>
<p>When I then call <code>loss = criterion(prediction, target)</code>, the error is thrown. So it means the prediction I get from my network is still on CPU even though the network should be on GPU. The target is on GPU as it should be.</p>
Hard disk space and data loader speed

Gkv: <p>Hi all,<br>
My 10TB hard disk has only 1TB free space now. The data loader (which was working fine before) is super slow now (around 20x slow). Will the hard disk space causes this issue?.</p>
<p>Thanks</p>

albanD: <p>Hi,</p>
<p>This is most likely the hard drive that can do some optimization for speed when it’s not too full, but cannot when it has more content. So not pytorch related.<br>
Have you tried freeing more space on the disk to see if performance come back?</p>

Gkv: <p>I will try freeing up some space and let you know the details. Thank you.</p>

Gkv: <p>Problem solved upon making some free space. Thanks</p>
How to add gradient back rather than subtract gradient?

kaka2nd: <p>Hi.<br>
I am working on one project, and the update formula is<br>
<strong>parameter = parameter + learning_rate * gradient</strong><br>
rather than the usual term which is subtracting.</p>
<p>I tried to make the learning rate negative to implement the idea but this is forbidden. So my question how to add the gradient when updating?</p>
<p>Hope for your reply. Thanks.</p>

albanD: <p>Hi,</p>
<p>I guess the simplest thing is to get <code>-gradient</code>. You can do this by flipping the sign of your loss: <code>loss = - loss</code>.</p>

paganpasta: <p>Besides flipping the loss, you can modify the gradient update rule by using a for loop over network parameters and updating the gradient accordingly. This will give you more flexibility.</p>
<pre><code class="lang-auto">learning_rate = 0.01
for f in net.parameters():
    f.data.sub_(f.grad.data * learning_rate * -1.0)
</code></pre>
<p>(Source: code snippet taken from 60-min blaze)</p>

kaka2nd: <p>Thanks so much for the reply.</p>

kaka2nd: <p>Thank you for the advice.</p>
GPU Training becomes very slow after a few iterations

Matheusih: <p>Hi, I’m training a model consisting of a nn.LSTM connected into a nn.Linear, using this for a regression problem. However training becomes real slow even inside the first epoch. I think my problem is that I’m using <code>retrain_graph=True</code> and also that the graph of my hidden state grows at each iteration, as I refeed it to my model every iteration. If I try not using <code>retrain_graph=True</code> it throws me an error “trying to backward through the graph a second time but the buffers have already been freed”. Since I’m new to RNNs I’m a bit confused, is it supposed to be like this, or am I missing something in my approach?</p>
<p>Here’s the relevant code:</p>
<blockquote>
<p>def learn(X, y, hidden):<br>
model.zero_grad()</p>
<p>output, hidden = model(X, hidden)</p>
<p>loss = criterion(output, y)</p>
<p>optimizer.zero_grad()<br>
loss.backward(retain_graph=True)<br>
optimizer.step()</p>
<p>return loss, output, hidden</p>
</blockquote>
<blockquote>
<p>hidden = None<br>
for epoch in range(epochs):<br>
loss_avg = 0<br>
for i, (X,y) in enumerate(dataloader):</p>
<pre><code>X = X.to(device)
y = y.to(device)

loss, output, hidden = learn(X,y, hidden)
 ...
</code></pre>
</blockquote>
<p>And the definition of my model follows:</p>
<blockquote>
<p>class RNN(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size, output_size = 1, num_layers = 1, batch_size = 1):<br>
super(RNN, self).<strong>init</strong>()</p>
<pre><code>    self.hidden_size = hidden_size
    self.input_size = input_size
    self.batch_size = batch_size
    
    self.lstm = nn.LSTM(input_size, hidden_size, num_layers)
    
    self.mlp = nn.Linear(hidden_size, output_size)

def forward(self, X, H=None):
  
    out, hidden = self.lstm(X,H)
    out = self.mlp(out[-1])
    return out, hidden
</code></pre>
</blockquote>
<p>EDIT (Solved):<br>
The problem was that I was non-intentionally training my hidden and saving all its graphs during training. I solved this by detaching the graph before returning, inside my <code>learn</code> function.</p>

pzg: <p>Hello I appear to be running into the same problem. Ive tried detach() on both hidden and model which I return to no effect. What exactly did you find helpful to detach?</p>
Cuda runtime error (60): peer mapping resources exhausted at

Alpha: <p>pytorch version:1.1.0<br>
cuda version:10.0.130<br>
GPU: TITAN V</p>
<p>I met a problem when I use 10 GPUs:</p>
<pre><code class="lang-auto">RuntimeError: cuda runtime error (60) : peer mapping resources exhausted at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THC/THCGeneral.cpp:164
</code></pre>
<p>And when I use 8 or 4 GPUs，I didn’t get this error.</p>

bmarami: <p>I am having the same problem with 10 GPUs, <span class="hashtag">#GPU</span> &lt;= 8 is file.<br>
pytorch version:1.2.0<br>
CUDA Version 10.1.105</p>
How to copy some of the model modules into a variable

OBouldjedri: <p>Hi , I was wondering how to store some of the model modules ('including their weight and bias ) into a global variable to copy them back after ?</p>

paganpasta: <p>You can save network parameters in a list, if that’s what you are after. Global or local that’s your design choice. Loop over the network parameters and save them. Checkout moduleList that can be of use as well.</p>

albanD: <p>You can also get all the parameters as a nice dictionnary with <code>model.state_dict()</code>. Then use <code>model.load_state_dict(state_dict)</code> to restore it.</p>

OBouldjedri: <p>I want some thing in this way<br>
store all the Linear layers (modules or weights ) into a list<br>
then loop over all model modules and if the module is linear assign the linear stored module that match the index</p>

albanD: <p>Ok,</p>
<p>In that case, I would use <code>for mod_uniq_name, mod in model.named_modules()</code> to find all the Linear layers and save their weights in some structure (like a dict) using <code>mod_uniq_name</code> as the key.<br>
Then when you want to reload it, do the same iteration and if the <code>mod_uniq_name</code> is present in your saved structure, load what is there.</p>

OBouldjedri: <p>Thank you but it is a bit different since I am suing some kind of tree so I have to save every Linear Node on Forward function so I have to use a different way to store them like:<br>
a list that store the weights or a list that stores the entire module which one do you thin is better ?</p>

albanD: <p>If you don’t plan of modifying the module itself, I think saving only the weights is better.</p>

OBouldjedri: <p>yes but iterating that list needs break you can’t index it if I try :<br>
for p55 in temppara:</p>
<pre><code>            print(p55)
            print(temppara.index(p55))
            input()
</code></pre>
<p>I get errors for the index and I need an index to know what module need to be mapped to what module or what weights should be maped to what module weights</p>

albanD: <p>If you have the same tree structure, you should encounter the same modules at the same time during the forward no? So you can simply <code>.append()</code> and <code>.pop()</code> your list.</p>
Read CudaArray for prediction/inference/image process

keithlostracco: <p>Is it possible to read directly from a CudaArray when using the C++ API?</p>
<p>I would like to build a plugin system for an existing application where I can access data directly on the GPU (that already exists on the GPU). I’m trying devise a system where I don’t have copy to from the CPU for processing or predicting using texture data.</p>
<p>Is this possible with PyTorch?</p>
<p>Thanks<br>
Keith</p>

tom: <p>You could use <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/DLConvertor.h" rel="nofollow noopener">DLManagedTensor and fromDLPack</a> as an API or dive into the <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/DLConvertor.cpp" rel="nofollow noopener">implementation</a> and use from_blob. Note that it means that you need to ensure the GPU memory blob is available for as long as the Tensor lives / until the deleter is called.</p>
<p>Best regards</p>
<p>Thomas</p>

keithlostracco: <p>Great thanks! I’ll look into those methods.</p>