akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
HTTPconnection error for trying to loard ResNet18 and ResNet34

username: liam_c
cooked: <p>How can I fix this error?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4b1a82e84c43d3468c7779cd3aefa7e9391e60c4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4b1a82e84c43d3468c7779cd3aefa7e9391e60c4" title="Screenshot_18"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4b1a82e84c43d3468c7779cd3aefa7e9391e60c4_2_690x73.jpeg" alt="Screenshot_18" data-base62-sha1="aIoBRa5TL8GRl1DEA0UINVvaoni" width="690" height="73" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4b1a82e84c43d3468c7779cd3aefa7e9391e60c4_2_690x73.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/4/4b1a82e84c43d3468c7779cd3aefa7e9391e60c4.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/4b1a82e84c43d3468c7779cd3aefa7e9391e60c4.jpeg 2x" data-dominant-color="373A40"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_18</span><span class="informations">891×95 28.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Error when trying to view image with dataloader and iter, PascalVOC 2007

username: MrPositron
cooked: <p>Hey there!<br>
I am using Pascal VOC dataset (from torchvision).<br>
And I encounter this kind of error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "mainv1.py", line 13, in &lt;module&gt;
    images, labels = next(data_iter)
  File "C:\Users\Asus\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 615, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  ...
  File "C:\Users\Asus\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 229, in &lt;listcomp&gt;
    return {key: default_collate([d[key] for d in batch]) for key in batch[0]}
TypeError: list indices must be integers or slices, not str
</code></pre>
<p>Error occurs in this code:</p>
<pre><code class="lang-auto">T = transforms.Compose([transforms.Resize((224, 224)),transforms.ToTensor()])
voc_data = VOCDetection(root, year='2007', image_set='train', 
                        download=False, transform=T)
# VOCDetection taken from here: https://pytorch.org/docs/master/_modules/torchvision/datasets/voc.html
dl = DataLoader(voc_data, batch_size=32,shuffle=False)

data_iter = iter(dl)
images, labels = next(data_iter)
</code></pre>

username: Lyu_Lin
cooked: <p>Have you solved the problem? I come across the same issue. Seek for help.</p>
Calculating Jacobian in a Differentiable Way

username: ArmenAg
cooked: <p>Is there a general way calculate the Jacobian of a module, in a way that we retain the graph and can backprop through operations on the Jacobian as well?</p>
<p>Something similar to how WGAN operates but defined on more than a single output.</p>

username: chenyuntc
cooked: <p>Do you mean backward of backward (second derivative)?<br>
try <code>loss.backward(create_graph=True)</code></p>
<p>see  <a href="https://github.com/pytorch/pytorch/pull/1016" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/1016</a></p>
<pre><code class="lang-auto">x = Variable(torch.randn(2, 2), requires_grad=True)
x.mul(2).sum().backward(create_graph=True)
y = x.grad.mul(2).sum()
# This accumulates grad of grad into x.grad, adding together results of both backward() calls
y.backward() 
</code></pre>

username: ArmenAg
cooked: <p>Yes, but in this case the output is a single number, therefore you can calculate the differential in a straightforward way.</p>
<p>I’m wondering how you do this for multiple outputs. Here is an article describing the jacobian matrix I’m referring to: <a href="https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant" rel="nofollow noopener">https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant</a></p>

username: ArmenAg
cooked: <p>I think this function is a general way to accomplish what I was saying. I borrowed this code from: <a href="https://github.com/pytorch/pytorch/blob/85a7e0f/test/test_distributions.py#L1501" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/85a7e0f/test/test_distributions.py#L1501</a></p>
<pre><code class="lang-auto">def jacobian(inputs, outputs):
    return torch.stack([grad([outputs[:, i].sum()], [inputs], retain_graph=True, create_graph=True)[0]
                        for i in range(outputs.size(1))], dim=-1)
</code></pre>

username: hzw
cooked: <p>Have you measured the running time of this method? One concern is that the for loop would make the backprop time grow linearly with the dimension of the output. I’m wondering whether there is way to directly get the Jacobian without the for loop.</p>

username: sbarratt
cooked: <p>The <a href="https://gist.github.com/sbarratt/37356c46ad1350d4c30aefbd488a4faa" rel="nofollow noopener">following code</a> will do the trick with a single call to backward, taking advantage of when the function takes batched inputs. You can modify the script by passing retain_graph=True to backward and returning .grad instead of .grad.data.</p>
Any resources for pytorch c++ api document?

username: ycliu
cooked: <p>I was writing c++ custom layer in pytorch. But the official document (<a href="https://pytorch.org/cppdocs/api/library_root.html" rel="nofollow noopener">https://pytorch.org/cppdocs/api/library_root.html</a>) is meaningless. Why is that and where can I find usable documents for the c++ api.</p>
Seq2Seq tutorial without max_length in Attention

username: keysmash97
cooked: <p>Hi everyone, i’m learning seq2seq model with pytorch tutorial. In the tutorial, it filters sentence pairs by max_length and use max_length property in attention mechanism (Bahdanau ?). Are there any way to implement attention without it ?. Because i want to train model without filter sentence pairs and i think if we set max_length is high such as 30 but actually attention need only 5 so it kinda expensive. I’m kinda new with this so correct me if I’m wrong and please give me detail explaination <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> . Thanks</p>
Run Sequence Through same autoencoder

username: Marc_Schmid
cooked: <p>Hi,<br>
I am trying to combine a autoencoder with sequential data which looks like [B,S,C,W,H]<br>
I wanna run the whole sequence through the same autoencoder and combine the latent spaces afterwards.</p>
<p>My NN structure looks like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3c10924196d619760dd6ade19c442b5c507dd78a.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c10924196d619760dd6ade19c442b5c507dd78a" title="combined_structure"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3c10924196d619760dd6ade19c442b5c507dd78a.png" alt="combined_structure" data-base62-sha1="8zm9qG9Q3fCv9rVnqL30Z68guDw" width="690" height="372" data-dominant-color="302D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">combined_structure</span><span class="informations">724×391 17.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there a possibility to iterate over the sequence of the tensor or how do I approach this problem best?</p>
<p>Kind Regards</p>

username: Marc_Schmid
cooked: <p>I found my mistake, the correct way is described in<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/HHTseng/video-classification" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/24979292?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/HHTseng/video-classification" target="_blank" rel="nofollow noopener">HHTseng/video-classification</a></h3>

<p>Tutorial for video classification/ action recognition using 3D CNN/ CNN+RNN on UCF101 - HHTseng/video-classification</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
AttributeError: 'MultiStepLR' object has no attribute 'state_dict'

username: Shisho_Sama
cooked: <p>How can I save the scheduler state ?<br>
I tried :</p>
<pre><code class="lang-auto">    temp = scheduler.state_dict()
    temp['last_epoch']=36
</code></pre>
<p>but got the error</p>
<blockquote>
<p>AttributeError: ‘MultiStepLR’ object has no attribute ‘state_dict’</p>
</blockquote>
<p>The reason I’m trying this is , I forgot to save the scheduler setting along with other needed info when saving the checkpoint! so now that I need to resume tarining, I need to choose the correct epoch. I thought the best way would be to manually create a new dictionary for it and then use <code>load_state_dict()</code> with the new information and carry on.<br>
I dont know why I’m facing this error, I saw for saving, people simply do :</p>
<pre><code class="lang-auto">save_checkpoint({
            'epoch': epoch + 1,
            'arch': args.arch,
            'state_dict': model.state_dict(),
            'best_prec1': best_prec1,
            'best_prec5': best_prec5,
            'optimizer' : optimizer.state_dict(),
            'scheduler' : scheduler.state_dict()
        }, is_best, filename, bestname)


def save_checkpoint(state, is_best, filename, bestname):
    torch.save(state, filename)
    if is_best:
        shutil.copyfile(filename, bestname)
</code></pre>
<p>so as you can see the scheduler has a state_dict(). Can anyone tell me where the problem is and how I can fix it?</p>

username: ptrblck
cooked: <p>This code works for PyTorch <code>0.4.0</code>:</p>
<pre><code class="lang-python">optimizer = optim.SGD([torch.randn(10, requires_grad=True)], lr=1e-1)
scheduler = optim.MultiStepLR(optimizer, [5, 10], 0.1)
print(scheduler.state_dict())
</code></pre>
<p>Which version are you using?</p>

username: Shisho_Sama
cooked: <p>Oh, so thats it! Thank you very much.  I checked and found out on my current system the version is 3.01!</p>

username: ttoosi
cooked: <p>my torch version is 0.4.0<br>
yet I get<br>
AttributeError: ‘ReduceLROnPlateau’ object has no attribute ‘state_dict’<br>
Here is a list of its attributes:<br>
lrs.ReduceLROnPlateau.<strong>dict</strong>.keys():<br>
dict_keys([‘<strong>module</strong>’, ‘<strong>doc</strong>’, ‘<strong>init</strong>’, ‘_reset’, ‘step’, ‘_reduce_lr’, ‘in_cooldown’, ‘_cmp’, ‘_init_is_better’, ‘<strong>dict</strong>’, ‘<strong>weakref</strong>’])</p>
Is there standard Hinge Loss in Pytorch?

username: karandwivedi42
cooked: <p>Looking through the documentation, I was not able to find the standard binary classification hinge loss function, <a href="https://en.wikipedia.org/wiki/Hinge_loss" rel="nofollow noopener">like the one defined on wikipedia page</a>:</p>
<p>l(y) = max( 0, 1 - t*y) where t E {-1, 1}</p>
<p>Is this loss implemented?</p>

username: dgriff
cooked: <p>Like for doing a MCSVM. I’m not sure was looking for that the other day myself too but didn’t see one. Let me know if you find please. Was gonna do a more thorough check later but would save me the time😁</p>

username: dgriff
cooked: <p>They have the MultiMarginLoss and MultilabelMarginLoss. But the one in particular you looking for is MarginRankingLoss and suits your needs</p>

username: Sanchit_Chiplunkar
cooked: <p>Did you find the implementation of this loss in Pytorch? Although i think it should be easier to implement this</p>

username: loicsacre
cooked: <p>May be you could do something like this</p>
<pre><code class="lang-auto">class MyHingeLoss(torch.nn.Module):

    def __init__(self):
        super(MyHingeLoss, self).__init__()

    def forward(self, output, target):

        hinge_loss = 1 - torch.mul(output, target)
        hinge_loss[hinge_loss &lt; 0] = 0
        return hinge_loss
</code></pre>
Deactivate backward

username: afshin67
cooked: <p>Hi,</p>
<p>What is the difference between these two API:</p>
<pre><code>torch::NoGradGuard guard;
torch::autograd::GradMode::set_enabled(false);
</code></pre>
<p>Can I use them interchangeably to change weights of a  network?</p>
<p>Thanks,<br>
Afshin</p>

username: yf225
cooked: <p><code>torch::NoGradGuard guard;</code> is an RAII guard while <code>torch::autograd::GradMode::set_enabled(false);</code> is not. Consider an example:</p>
<pre><code class="lang-cpp">auto t = torch::randn({2, 2});

{
  torch::NoGradGuard guard;  // This only takes effect within the scope
  t.add_(1);  // Does not accumulate gradient
}
t.add_(1);  // Accumulates gradient

{
  torch::autograd::GradMode::set_enabled(false);  // This is a global setting
  t.add_(1);  // Does not accumulate gradient
}
t.add_(1);  // Still doesn't accumulate gradient
</code></pre>

username: afshin67
cooked: <p>Thanks for the explanation. So, if I add <code>torch::autograd::GradMode::set_enabled(true);</code> after the operation, does it activate the gradient accumulation?</p>
<blockquote>
<p>{<br>
torch::autograd::GradMode::set_enabled(false);  // This is a global setting</p>
<p>t.add_(1);  // Does not accumulate gradient</p>
<p>torch::autograd::GradMode::set_enabled(true);<br>
}</p>
</blockquote>

username: yf225
cooked: <p><a class="mention" href="/u/afshin67">@afshin67</a> Yes it does <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Variable vs Tensors

username: Rayan_Nakouri
cooked: <p>Variable vs Tensors.<br>
This topic took too much time for me. And I’m still not understanding the difference between Pytorch Variables and Tensors. All I know is that Variables and Tensors are almost the same. The difference between them is that Tensors don’t have the concept of “gradients” whereas Variables do.<br>
But First, I don’t understand what does the term “gradient” mean?<br>
Could you explain it to me please ??</p>

username: moskomule
cooked: <p>Hi, AFAIK, <code>Variable</code> is deprecated and now <code>Tensor</code>has gradient. Check <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py</a> etc. or other official documents.</p>

username: Rayan_Nakouri
cooked: <p>Thanks for repling me.<br>
But could you explain what does gradient term mean please ??</p>

username: justusschock
cooked: <p>If you want to train a neural network, you have to perform backpropagation. And backprop relies on updating parameters in the direction of the derivative of a loss value w.r.t the corresponding parameter.<br>
Thus gradient in this term means this derivative.</p>
Pytorch dtype vs Numpy dtype w.r.t. cast

username: vsiles
cooked: <p>Hi !<br>
I recently looked at Numpy dtypes (<a href="https://stackoverflow.com/questions/56022497/numpy-pytorch-dtype-conversion-compatibility/56022918?noredirect=1#comment98695989_56022918" rel="nofollow noopener">https://stackoverflow.com/questions/56022497/numpy-pytorch-dtype-conversion-compatibility/56022918?noredirect=1#comment98695989_56022918</a>), more precisely how they are cast during expression evaluation.</p>
<p>When testing with Pytorch, I realize the same cast was not performed, and that I was getting a runtime error. Here is how I tested it:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import numpy
&gt;&gt;&gt; x = torch.zeros([4], dtype=torch.int32)
&gt;&gt;&gt; y = torch.zeros([4], dtype=torch.float32)
&gt;&gt;&gt; x + y
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: expected type torch.FloatTensor but got torch.IntTensor
&gt;&gt;&gt; x = numpy.zeros([4], dtype=numpy.int32)
&gt;&gt;&gt; y = numpy.zeros([4], dtype=numpy.float32)
&gt;&gt;&gt; x + y
array([0., 0., 0., 0.])
&gt;&gt;&gt; (x + y).dtype
dtype('float64')
</code></pre>
<ol>
<li>Pytorch is not doing any cast and expect both side of the operation to have the same dtype. Is this statement right ?</li>
<li>Is there any plan to follow numpy’s design or will it stay like this ?</li>
</ol>
<p>Best regards,<br>
V.</p>
Freeing memory on each step?

username: Valiox
cooked: <p>I have a model that operates on inputs of variable length. At inference time, I need to split the data into chunks of fixed size, else I will get a cuda oom. I set the size of the chunks to be as large as possible, so as to fill my GPU memory. The code goes like this:</p>
<pre><code class="lang-auto">    (...)
    model.eval()
    
    # Forward chunks
    chunks = extract_chunks(input)    # Returns numpy arrays
    for chunk in chunks:
        x, y_true = collate([chunk])  # Collate a single chunk into a batch, returns CPU tensors
        x, y_true = x.cuda(), y_true.cuda()
        y_pred = model(x)

</code></pre>
<p>And I monitor my GPU usage (with nvidia-smi) when running this code.</p>
<p>I noticed that if I add the line</p>
<pre><code class="lang-auto">        # Free the memory
        del x, y_pred, y_true
</code></pre>
<p>at the end of the loop, the memory usage is halved, and I can effectively double my chunk size. There seems to be no overhead for doing this either. What is going on here?</p>

username: Valiox
cooked: <p>I realized that I had forgotten to wrap my loop with <code>torch.no_grad()</code>. The behaviour does not occur anymore. Does that mean that gradients were not deleted until calling <code>del</code>?</p>

username: ptrblck
cooked: <p>Did you store <code>y_pred</code> or any other tensor attached to the computation graph somehow?<br>
Usually you see the memory growing if you store a tensor with the computation graph, e.g.</p>
<pre><code class="lang-python">output = model(data)
loss = criterion(output, target)
losses.append(loss)   # will store the whole computation graph -&gt; increased memory usage
</code></pre>

username: Valiox
cooked: <p>I haven’t. There isn’t much more to the code than what I wrote.</p>
Semantic segmentation loss function / shape of prediction and target

username: Knievel
cooked: <p>Hello,</p>
<p>for a research project I’m currently implementing a RNN to solve a semantic segmentation task.</p>
<p>The target segmentation maps have the shape (c, h, w) which is (11, 64, 84) in my example, so there are 64*84 (number of pixels) vectors of length 11 with a 1 in the position of the class.<br>
My network also predicts a tensor of the same shape, where the vectors for each pixel are the output of a nn.LogSoftmax layer.</p>
<p>If I now use nn.NLLLoss and try to calculate</p>
<blockquote>
<p>loss = criterion(output, target)</p>
</blockquote>
<p>I get</p>
<blockquote>
<p>Expected target size (11, 84), got torch.Size([11, 64, 84])</p>
</blockquote>
<p>So I read that NLLLoss expects labels, but I’m sure there is some way of using what I have without first applying another function, as it seems to contain all necessary information.</p>
<p>What did I understand wrongly? Is there a better way to solves this?</p>

username: ptrblck
cooked: <p>It seems your target is currently one-hot encoded.<br>
If you are using <code>nn.NLLLoss</code>, the target is expected to contain the class indices instead (without the channel dimension).<br>
In case your current target shape is <code>[batch_size, c, h, w]</code>, try to convert it using:</p>
<pre><code class="lang-python">target = torch.argmax(target, 1)
loss = criterion(output, target)
</code></pre>
Could I use that nn.Criterion module compute combined tensor?

username: isaac1644
cooked: <p>For example, I have 6 tensors with different dimension:</p>
<p>tensor11 size: (b, 64, 224, 224)<br>
tensor21 size: (b, 128, 112, 112)<br>
tensor31 size: (b, 256, 56, 56)</p>
<p>tensor12 size: (b, 64, 224, 224)<br>
tensor22 size: (b, 128, 112, 112)<br>
tensor32 size: (b, 256, 56, 56)</p>
<p>Could I get the tensors which tensor1 has the size (tensor11 size, tensor12 size, tensor13 size), tensor2 has the size(tensor12 size, tensor22 size, tensor 32 size).<br>
If possible, how could I use nn.MSELoss() to computer the loss between elements between these 2 tensor on same positions and get these 3 losses  between (tensor11, tensor12), (tensor21, tensor22) and (tensor31, tensor32) by one step while these losses will be utilised  in BP?<br>
Thanks.</p>

username: Oli
cooked: <p>Hi, maybe you could flatten the tensors and concat them? Use</p>
<p><code>flatten1 = tensor11.view(1, -1)</code> and <code>torch.cat(flatten1, flatten2, flatten3)</code></p>
<p>Would it work if you’d just calculate the 3 losses separately and then add them?</p>
<pre><code class="lang-auto">mse = nn.MSELoss()
loss1 = mse(tensor1l, tensor12)
loss2 = mse(tensor2l, tensor22)
loss3 = mse(tensor3l, tensor32)
total_loss = loss1 + loss2 + loss3
# backpropagate
</code></pre>

username: isaac1644
cooked: <p>Thanks very much. I calculate 3 losses respectively and concatenate them by () now. It’s weird that the loss can be used in back propagation in this way. I mean python always seems the element in ()  as A tuple in my computer <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>

username: Oli
cooked: <p>The wonders of Pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Slicing and assign

username: MakeDirtyCode
cooked: <p>I wanted to slice and change the values for certain conditions. Look at the toy code I wrote.</p>
<p>a = torch.randn(3, 3)<br>
a[:, :][a &gt; 0] = 0</p>
<p>This works, but if you sliced like this</p>
<p>a [:, (0, 1, 2)] [a &gt; 0] = 0</p>
<p>It does not work.</p>
<p>Why? Do they work differently?</p>

username: MariosOreo
cooked: <p>Hi,</p>
<p>I tired to explan it in storage view. I used <code>.data_ptr()</code> to check whether tensors have the same storage.<br>
In the first case, <code>a.data_ptr() == a[:,:].data_ptr()</code>, so they share the same storage, value changing will also perform on original <code>a</code>.<br>
In the second case, <code>a.data_ptr() != a[:, (0,1,2)].data_ptr()</code>, they donot use the same base.</p>
<p>Im not sure whether it is a proper way to explan it, your use case may related to indexing operations, i will google it further. If I am wrong, correct me please.</p>
<p><strong>EDIT:</strong>  <a href="https://stackoverflow.com/a/56052056/10783571" rel="nofollow noopener">here</a> is the answer.</p>
How to properly use distributed.all_gather?

username: hanbit
cooked: <p>Hi, I am trying to use torch.distributed.all_gather function and I’m confused with the parameter ‘tensor_list’.</p>
<p>here’s my snippet.</p>
<pre><code class="lang-auto">
# output: dictionary, e.g. {'key':'tensor', 'key':'tensor'}
output = self.model(input)

for key in output.keys():
    tensor_list = [torch.zeors_like(output[key]) for _ in range(dist.get_world_size())]
    output[key] = self.gather_tensor(tensor_list, output[key])


def gather_tensor(self, tensor_list, tensor):
    rt = tensor.clone()
    dist.all_gather(tensor_list, rt)
    return torch.stack(tensor_list)

</code></pre>
<p>I designed ‘gather_tensor’ following the function ‘reduce_tensor’ from <a href="https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py" rel="noopener nofollow ugc">apex example</a>.</p>
<p>However, it shows RuntimeError as below</p>
<blockquote>
<p>in gather_tensor, dist.all_gather(tensor_list, rt)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py”, line 1027, in all_gather<br>
work = _default_pg.allgather([tensor_list], [tensor])<br>
RuntimeError: Input tensor sequence should hsave the same number of tensors as the output tensor sequence</p>
</blockquote>
<p>What’s wrong with my code?</p>
<pre><code class="lang-auto">output = self.model(input)
tensor_list = [torch.zeors_like(output['key1']) for _ in range(dist.get_world_size())]
for key in output.keys():
    output[key] = self.gather_tensor(tensor_list, output[key])
</code></pre>
<p>I put the tensor_list initialization in front of ‘for loop’ and it works but it seems like ‘deadlock’ occurs.</p>
Found no NVIDIA driver on your system

username: Baron_Tsai
cooked: <p><strong>CUDA VERSION</strong><br>
cat /usr/local/cuda/version.txt<br>
CUDA Version <strong>7.5.18</strong></p>
<p><strong>PyTorch CUDA VERSION</strong><br>
__version__ = ‘0.3.0.post4’ (pytorch 1.0 does not have cuda75?)<br>
debug = False<br>
cuda = ‘7.5.18’</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cf197fd60fe98fa72c6a93435406fa131e723ebb.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf197fd60fe98fa72c6a93435406fa131e723ebb" title="MacHi%202019-05-09%2011-23-02"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cf197fd60fe98fa72c6a93435406fa131e723ebb.png" alt="MacHi%202019-05-09%2011-23-02" data-base62-sha1="ty5t2X2Dp032rXf8KalcHE8kATV" width="676" height="500" data-dominant-color="0E2A32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MacHi%202019-05-09%2011-23-02</span><span class="informations">1064×786 46.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>however, error occur when execute: “model.cuda(gpu_idx)”<br>
AssertionError:<br>
Found no NVIDIA driver on your system. Please check that you<br>
have an NVIDIA GPU and installed a driver from</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.nvidia.com/Download/index.aspx">
  <header class="source">

      <a href="https://www.nvidia.com/Download/index.aspx" target="_blank" rel="noopener nofollow ugc">nvidia.com</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/drivers/cut-graphics/nvidia-geforce-drivers-meta-image-1200x627.jpg" class="thumbnail">

<h3><a href="https://www.nvidia.com/Download/index.aspx" target="_blank" rel="noopener nofollow ugc">Download the latest official NVIDIA drivers</a></h3>

  <p>Download the latest official NVIDIA drivers</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>According to the above, the graphics card and driver should exist, how can i do with this problem? thank you!</p>
Dilated convolution is too slow!

username: lizhengwei1992
cooked: <p>I find that the dilated convolution(dilation=2,4…) is slower much than convolution without dilated (dilation=1)？my pytorch version is 0.3, CUDNN is 7.0，what should I do to accelerate it ?</p>

username: Varg_Nord
cooked: <p>THNN source code is little bit confusing because there are too many macro references, sometimes nested. But as I see, it first computes bias via GEMM, and apply im2col to the filter and input, with another GEMM following. As GEMMs are supposed to take same amount of computation, I think the bottleneck could be im2col procedure. So It might be hard to optimize.</p>

username: lizhengwei1992
cooked: <p>Thanks for your answer, what about CUDNN? it doesn’t support dilated convolution now ?</p>

username: ngimel
cooked: <p>Cudnn supports dilated convolutions and pytorch uses it for dilated convolutions, however, they are harder to optimize, so they are not expected to be as fast as non-dilated convolutions.</p>

username: MingHongL
cooked: <p>Hi, I also encountered the same problems, so, I want to know if you have solved this problems,  if it is solved, could you share the experience?</p>
CPU time vs. GPU time of ops in Pytorch

username: Shama022
cooked: <p>I am trying to profile a network with torch.autograd.profiler and I need some explanation regarding the CPU and GPU time reported. I assume that the timings are nearly equal because CPU time includes the time the kernel launch + execution. However I see inconsistency when trying to find a relationship between CPU and GPU times.</p>
<p>As shown below some ops report approximately same time , some ops report CPU time larger than GPU time and some have the CPU time smaller than GPU time. Could someone please explain the difference in time.</p>
<p>Name                CPU Time      GPU Time<br>
relu                   14.700us        15.936us<br>
sub                   112.447us      93.504us<br>
mm                   43.501us        46.912us<br>
CatBackward    84.912us        84.704us</p>
<p>Thanks</p>
Question regarding parameters for optimizer

username: Fawaz_Sammani
cooked: <p>Hello,<br>
In one of my deep learning applications, I am modifying the gradients after calling <code>loss.backward</code>. So the gradients are actually modified from their original. Now when i call <code>optimizer.step()</code>, does the optimizer consider the updated parameters? To demonstrate, here is a snippet of my code:</p>
<p><code>optimizer = torch.optim.Adam(model.parameters(), lr=0.001, amsgrad=True)</code></p>
<p>And let the function:</p>
<pre><code class="lang-auto">def change_gradients(net):
      for param in net.parameters():
                some operation on param
</code></pre>
<p>Now inside the training loop:</p>
<pre><code class="lang-auto">        model.zero_grad()
        loss.backward()
        change_gradients()
        optimizer.step()
</code></pre>
<p>So in that case, does the optimizer take the updated parameters or the original parameters when it applied the update to the weights?</p>
<p>If no, then how can I update the optimizer parameters? If i re-define the optimizer after <code>change_gradients()</code> with the model being returned from this function, as in:</p>
<pre><code class="lang-auto">def change_gradients(net):
      for param in net.parameters():
                some operation on param
      return model
</code></pre>
<p>Now inside the training loop:</p>
<pre><code class="lang-auto">        model.zero_grad()
        loss.backward()
        model = change_gradients()
        optimizer = torch.optim.Adam(model.parameters(), lr=0.001, amsgrad=True)        
        optimizer.step()
</code></pre>
<p>Would that work? How can the optimizer know the previous values of mt and vt if it is being re-defined every time and therefore everything in it is lost (except that now it receives the new model)?</p>
<p>Thanks!</p>

username: LeviViana
cooked: <p>This is related to <a href="https://discuss.pytorch.org/t/is-model-global-in-pytorch/44727">this thread</a>.</p>
<p>The optimizer have a reference to the model’s parameters, so once you change it anywhere, the optimizer will take it into account. The following snippet shows that the optimizer’s parameters point to the model’s parameters data:</p>
<pre><code class="lang-python">import torch
model = torch.nn.Linear(2, 3)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, amsgrad=True)
x = set(p.data_ptr() for p in model.parameters())
y = set(p.data_ptr() for p in optimizer.param_groups[0]['params'])
print(x == y) # prints True
</code></pre>
Created Non-leaf Tensor

username: Canurd
cooked: <p>Dear All,<br>
I am trying to add the trainable weights of my custom loss function (which is a weighted sum of 2 loss functions) to the optimizer. The weights are initialized as</p>
<pre><code class="lang-auto">w = nn.Parameter(torch.Tensor([1]).cuda())
</code></pre>
<p>And I’ve successfully registered the weights to the optimizer’s param group, but I keep getting the ‘can’t optimize a non-leaf Tensor’ error. It would be fine if I just initialize the weights as tensors</p>
<pre><code class="lang-auto">w = torch.Tensor([1]).cuda()
w.detach().requires_grad_() 
</code></pre>
<p>but in this way, even if the weights have been added to the optimizer param group, it does not update over epochs.<br>
How can I solve this problem?</p>
<p>Thanks</p>

username: JuanFMontesinos
cooked: <p>Try to do<br>
w=nn.Parameter(torch.tensor(1))<br>
w_gpu = w.cuda()</p>
<p>then use w_gpu</p>

username: Canurd
cooked: <p>Hi</p>
<p>Thanks for the reply! I’ve just tried it but didn’t work out. It’s still logging the weights as non-leaf tensor.</p>

username: JuanFMontesinos
cooked: <p>Sorry,  I’m assuming you are setting this inside a nn.module, is it the case?</p>

username: Canurd
cooked: <p>Yeah I did set this inside a nn.Module, but I also manually passed the weight variables to the optimiser</p>
<pre><code class="lang-auto">params = list(model.parameters()) + list(loss_function.w1) + list(loss_function.w2)
optimiser = optim.SGD(params)  
</code></pre>
<p>The manual passing is necessary since the weight(s) would not be added to model.parameters() even if I decleared it as nn.Parameter() within the nn.Module.</p>

username: JuanFMontesinos
cooked: <p>Let me know, what is w1 and w2?</p>
<p>Think that if you have a loss which is a nn.Module like</p>
<pre><code class="lang-auto">class MyLoss(nn.Module):
init 
w=nn.Parameter(torch.tensor(1))
w_gpu = w.cuda()
</code></pre>
<p>You only have to pass w to the optimizer, not w_gpu. w should also be a leaf tensor if defined that way.<br>
Here you don’t have to pass w_gpu as it is a copy of w (and non-leaf tensor whose backward is just passed to w)</p>
<p>Same effect should be achieved if you allocate that module</p>
<pre><code class="lang-auto">class MyLoss(nn.Module):
   init 
      w=nn.Parameter(torch.tensor(1))

instance = MyLoss().cuda()
</code></pre>
<p>In fact, you should code it this way as the previous one is a way of allocating different parts of a submodel on different gpus, thus, it’s not intended to be used like that.</p>
<p>It would be useful if you paste the essential part of your code instead</p>

username: Canurd
cooked: <p>Hi<br>
Sry for the late reply. I’ve found out that the bug is caused by me not carrying out any operations on the created weight (a mispositioned line of code) and I’ve fixed it. Thank you very much!</p>

username: Intel_Novel
cooked: <p>If you have two different loss functions, finish the forwards for both of them separately, and then finally you can do <code>(loss1 + loss2).backward()</code>.</p>
When useing inception3, testing ACC is higher than training ACC, why?

username: jack15710
cooked: <p>when useing inception3, testing ACC is  higher a lot than training ACC, why?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a407894f2ccada104a30114e3c3f9b0bf379c7e4.png" alt="image" data-base62-sha1="np4s0uftD4EnOo6dGNqAI4TWJNO" width="258" height="256"></p>

username: ptrblck
cooked: <p>The usage of dropout layers (or other regularization techniques which might be disabled for testing) might sometimes yield this observation.<br>
Due to the random dropping of neurons the model is “smaller” during training, thus have less capacity, which might in turn give a worse training accuracy.</p>
<p>On the other hand, it might also depend on the data (splitting) you are using.<br>
If you just got lucky (or even cherry-picked the validation dataset), this observation is also common.</p>

username: jack15710
cooked: <p>the model trained on cifar10 datasets. so, it is a standard split of test/train.</p>

username: Intel_Novel
cooked: <p>Dropout works only while training, while testing they are shut down by Pytorch. This affects the accuracy, is better w/o dropouts - it is better at testing time, which is what you need.</p>
FID score for MNIST pytorch

username: Kaitav_Mehta
cooked: <p>Hi all!<br>
Hope you are having a great day. Thanking in advance for your valuable time.</p>
<p>Question:<br>
Has anyone tried calculating FID score for MNIST dataset(i.e grayscale images) using DCGANs pytorch?</p>
<p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/KaitavMehta95/coegan" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/40546487?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/KaitavMehta95/coegan" target="_blank" rel="nofollow noopener">KaitavMehta95/coegan</a></h3>

<p>Code used in the experiments of the paper COEGAN: Evaluating the Coevolution Effect in Generative Adversarial Networks http://gecco-2019.sigevo.org - KaitavMehta95/coegan</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I have this project here which calculates FID for MNIST, but it is difficult to understand and debug.</p>
<p>Another question is do we need to store images and then calculate FID? or there is another way around?</p>
<p>Many Thanks</p>
Multiclass accuracy and evaluation

username: Alex_Ge
cooked: <p>Hi,</p>
<p>Let me start by saying I’ve searched for this, and apart from a single post (which doesn’t answer the question) I understand it is clear on how to train for multiclass but not on how to evaluate.<br>
I’m vectorising text which is then used as categorical input to a simple 2-hidden layer MLP.<br>
Training with CE Loss and evaluation for single class (SoftMax) works fine, but it’s not what I want: we have multiple classes, and thus I’ve ended up using <code>MultiLabelSoftMarginLoss</code> which trains the network fine.<br>
However, when trying to use the boiler template code for evaluation, it fails due to using argmax on the single class index, which doesn’t make sense in multi-class.<br>
I’ve copied a sample from another post, and this is how far I’ve gotten:</p>
<pre><code class="lang-auto">def multi_eval(model, test_data, test_loader):
    with torch.no_grad():
        correct = 0
        total = 0
        for inputs, labels in test_loader:
            # compute output
            outputs = model(inputs)
            predicted = torch.sigmoid(outputs).data &gt; 0.5

            # PROBLEM is HERE
            ok = torch.isclose(predicted, labels)
            print(ok)

    return correct, total
</code></pre>
<p>I do not understand how to make a comparison of the <code>predicted</code> and <code>labels</code> given a small margin of error <em>epsilon</em> or not, in order to sum up the correct from the incorrect outputs.<br>
I understand that in multi-class there is more complexity involved because some classes may be high and some not, I am looking for the <em>absolute</em> measure, e.g., either the vector matches or it doesn’t (always with a small degree of error) which is why I ended up using <code>sigmoid</code> on the predicted tensor.<br>
I get errors about <code>ByteTensor</code> and whatnot, and I am sure this is a very simple problem to solve.</p>
Is there any way to prepend to the computational graph?

username: cdgreenidge
cooked: <p>I’m writing a second-order optimizer that optimizes model parameters in a low-dimensional reparameterized space. Ideally, I’d like to:</p>
<ol>
<li>Take a step in the low-dimensional space</li>
<li>Pushforward the low-dimensional parameters to the high-dimensional model space</li>
<li>Compute the loss and backpropagate through both the model and the pushforward function, all the way to the low-dimensional space.</li>
</ol>
<p>Unfortunately I’m finding it difficult to prepend the pushforward function to the computational graph in the model, since in-place updates of leaf variables break autograd.</p>
<p>Is there a canonical/recommended way to approach this? Or should I just abandon <code>torch.nn</code> entirely and write everything as a pure function?</p>
<p>Thanks for your help!</p>
Using transforms.normalize() with batch norm?

username: deeplearner6
cooked: <p>Is using transforms.normalize() with a net using batch norm pointless (im using resnet).<br>
Or have i misunderstood some stuff?</p>

username: Trump
cooked: <p>You have a point.<br>
normally, transoforms.normalize is applied to images before fed into networks.<br>
batchnorm is used in the network to normalize internal representation.</p>

username: deeplearner6
cooked: <p>Okay so batch norm makes sure the internal values when propagated through the network dont get out of hands and transforms.normalization just scales the inputs?</p>

username: Trump
cooked: <p>Yup, that’s right, you heard me correctly!</p>
In tensorboard, how do I hide view() operations in a graph?

username: Trump
cooked: <p>Hi, I added a graph and …<br>
The graph shows operations of torch.Tensor.view()</p>
<pre><code class="lang-auto">def forward(self, x):
    x = self.conv1(x)
    x = self.conv2(x)
    x = self.pull1(x)
    x = self.conv3(x)
    x = self.pull2(x)

    x = x.view(x.size(0), -1)

    x = self.linear(x)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/ab1655f29fbdd2081dd78f11bc0fa678b24f9308.png" alt="Screenshot%20from%202019-05-02%2023-53-47" data-base62-sha1="opvv5TO62tGb1gSMhI23TwbtMSI" width="403" height="469"><br>
It definitely annoys me… are there any ways to make a clean graph?<br>
Thanks!</p>
InstanceNorm resulting in "Trying to backward through the graph a second time" when drop-in replacing BatchNorm?

username: mattrobin
cooked: <p>I’m working with a WGAN where the model is initially using <code>BatchNorm2D</code>s. Using the <code>BatchNorm2D</code>s, my code runs without a problem. However, when I try to drop-in replace the <code>BatchNorm2D</code>s with <code>InstanceNorm2D</code>s, then I end up with the <code>Trying to backward through the graph a second time</code> error.</p>
<p>Having worked with WGANs, I’ve seen this error before. However, I don’t believe it should occur when replacing <code>BatchNorm2D</code>s with <code>InstanceNorm2D</code>s, and applying no other changes. Attempting to debug the issue, I added <code>retain_graph=True</code> to all my <code>backward()</code>s and to the WGAN’s <code>torch.autograd.grad()</code>, but still the error occurs.</p>
<p>Does anyone happen to have an explanation as to why this should occur for <code>InstanceNorm</code> when it doesn’t for <code>BatchNorm</code>? And how I might be able to resolve this? Thank you!</p>

username: mattrobin
cooked: <p>To update, I had added <code>retain_graph=True</code> to all <code>backward</code>s except the final one (which should never need to retrain the graph. This is the <code>backward</code> used in the gradient penalty after the <code>torch.autograd.grad</code>. Notably, this is also the <code>backward</code> where the error occurs. However, adding <code>retain_graph=True</code> to this backward fixes the problem. This seems strange to me, since it means the same <code>backward</code> which is trying to use the freed graph is also the <code>backward</code> which is doing the undesired freeing. Does anyone know why this might be happening? Again, it is only occurring when using <code>InstanceNorm</code> to replace <code>BatchNorm</code>. This suggests to me that <code>InstanceNorm</code> is doing something strange when used in a second order gradient.</p>

username: AlbertZhang
cooked: <p>Hi, I also got into such a situation. By <code>retain_graph=True</code> when calling backward it fixed this problem. Thanks.</p>
Sparse.mm backward pass

username: mshuaibi
cooked: <p>My model involves a forward pass and then determines the gradients of the model output wrt to the model inputs to then calculate an additional term that is to be utilized in my loss function. The calculating of this additional term involves a sparse multiplication which seems to be giving me an error when running loss.backward(): <code>RuntimeError: Expected object of backend CPU but got backend SparseCPU for argument #2 'mat2'</code>. My model looks as follows:</p>
<pre><code class="lang-auto">    def forward(self, input_data, fprimes):
        batch_size = self.batch_size
        model_pred = torch.zeros(batch_size, 1).to(self.device)
        for index, value in enumerate(self.values):
            model_inputs = input_data[value][0]
            contribution_index = torch.tensor(input_data[value][1])
            outputs = self.valuespecific_models[index].forward(model_inputs)
            model_pred.index_add_(0, contribution_index, outputs)
            gradients = grad(
                model_pred,
                model_inputs,
                grad_outputs=torch.ones_like(model_pred),
                retain_graph=True,
                #create_graph=True
            )[0]
        #code that organizes gradients and returns a dense matrix, dO_dI
        output_2 = torch.sparse.mm(fprimes.t(), dO_dI.t())

        return model_pred, output_2

</code></pre>
<p>Where fprimes is a sparse matrix fed to the model.</p>
<p>When I change fprimes to not be a sparse matrix training seems to works fine. Is there a way to do this with sparse multiplication? Also, when computing the gradients I’m unsure whether I need to do retain_graph or create_graph = True. I need to do one of them otherwise it gives me an error saying I cant backward pass again.</p>
<p>Thanks!</p>
Need starter code for "image sequence" to "number sequence" type LSTM

username: nishitkumar89
cooked: <p>Folks, in our application we have a camera that is monitoring a scene (e.g., food cooking) and we want to output verdicts in real-time (e.g., cooking doneness). I am looking for a good starter code from where I could start modifying and experimenting.</p>
<p>More details:<br>
We have videos with N frames, where N is a variable number, say, between 1 and 1000 or so. We wish to train a model that will output monotonic sequence of length N, say for N=10, it outputs sequence 0-0-0-0-0-1-1-1-2-2. One number per video frame. WE DO NOT WANT A MODEL FOR WHICH YOU HAVE TO PROVIDE ALL N FRAMES UP FRONT TO GET THE OUTPUT i.e., I have looked at resources on video tagging, action recognition in video, etc and they do not seem relevant</p>
Pytorch Docs: Forwarding to a search result is terribly slow

username: benjamin9555
cooked: <p>Browser: Chrome 74<br>
When I use the search on <a href="https://pytorch.org/docs/stable/index.html" rel="nofollow noopener">https://pytorch.org/docs/stable/index.html</a> and then click on one of the suggestions, it takes literally seconds until the page “scrolls down” from the top to the right place and is fully loaded. This makes quick checks on the API really frustrating.</p>