akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError when gradient computation

Hung_Yu_Heng: <p>I have some problem when I implement my loss function.<br>
(In actually, I want to implement the Maximum likelihood estimation. Thus I have to maintain the forward graph and add the new forward data every time.)<br>
There is a RuntimeError about the inplace operation which may be the assign operation.<br>
Does anyone encounter the same problem and has solved it?</p>
<p>The following code can reproduce the same error with torch version 0.6.1.<br>
And there is a RuntimeError in the second optimizer.step().</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

class Network(nn.Module):
    def __init__(self, dim, hidden_size):
        super(Network, self).__init__()
        self.fc1 = nn.Linear(dim, hidden_size)
        self.activate = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, 1)
    def forward(self, x):
        return self.fc2(self.activate(self.fc1(x)))
func = Network(3, 10).cuda()
optimizer = optim.SGD(func.parameters(), lr=1e-2)

context_1 = np.array([0.1,0.1,0.1])
context_2 = np.array([0.2,0.2,0.2])
tensor_1 = torch.from_numpy(context_1).float().cuda()
tensor_2 = torch.from_numpy(context_2).float().cuda()
a = func(tensor_1)
b = func(tensor_2)
loss = a*a
func.zero_grad()
loss.backward(retain_graph=True)
optimizer.step()
loss = loss + b*b
func.zero_grad()
loss.backward(retain_graph=True)
optimizer.step()
</code></pre>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>The problem I see here is that the first <code>optimizer.step()</code> is modifying the parameters of your network inplace. But the original values of these parameters are needed to compute the second backward.<br>
Hence the error that you see.<br>
You will have to either delay the optimizer step or redo the forward I’m afraid.</p>

Hung_Yu_Heng: <p>Thanks for helping. It works after redo the forward.<br>
Is there any way not to redo the forward? Because of the maximum likelihood estimation approach, if I have to redo all the forward every time, the computation time will growth linearly.</p>

albanD: <p>If you don’t want to redo the forward, you will have to delay the optimizer step.<br>
You may be able to stash the current gradient into a temporary list (cloning them) for each backward you do. Then you can do all the steps when all the gradients are computed.</p>
When to use jit.trace, and when to use jit.script?

Flicic_Suo: <p>After I read <a href="https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html" rel="nofollow noopener">this toturial</a>, which I learnt is that I should use <code>jit.script</code> to transfer my module into TorchScript if there are conditional expressions or other uncertainties during execution in its forward propagation.</p>
<p>If my comprehension is correct, then can I say that it’s OK to use <code>jit.script</code> any time to replace <code>jit.trace</code>? Since I figure no drawback of <code>jit.script</code> against <code>jit.trace</code>.</p>
<p>Also I cannot understand why the toturial calls somthing like <code>jit.trace(jit.script(module))</code> then. Does this make any sense? Can’t I just use <code>jit.script(module)</code> alone?</p>

googlebot: <p>Yeah, jit.trace is of limited use. It can process some python code without requiring modifications (type annotations, etc.), but at the same time non-trivial code may fail to work correctly, if any non-tensor non-constants get captured.</p>
<aside class="quote no-group" data-username="Flicic_Suo" data-post="1" data-topic="95624">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flicic_suo/48/17471_2.png" class="avatar"> Flicic_Suo:</div>
<blockquote>
<p>Also I cannot understand why the toturial calls somthing like <code>jit.trace(jit.script(module))</code> then. Does this make any sense?</p>
</blockquote>
</aside>
<p>No, this doesn’t make sense. Note that jit.script(module) returns a new module, old one remains in uncompiled state.</p>
RuntimeError: non-empty 3D or 4D (batch mode) tensor expected for input

Can_Keles: <p>I get this error: <strong>“RuntimeError: non-empty 3D or 4D (batch mode) tensor expected for input”</strong> when passing a tensor of shape <strong>[1, 2, 3, 256, 256]</strong> to the layer <strong>AvgPool2d(kernel_size=3, stride=2, padding=[1, 1])</strong>.</p>
<p>I could not figure out what the problem is. From the research I have done it seems that this error is thrown when an empty tensor is passed to the pooling layer but the shape of my tensor is [1, 2, 3, 256, 256] meaning its not empty. Am I missing something? Thanks in advance.</p>

Can_Keles: <p>By the way, this error only occurs in the inference time, the model trains with no errors.</p>

ptrblck: <p>I’m a bit confused about the error message and the information in your post.<br>
While the error message suggests you are using <code>nn.AvgPool3d</code> and are passing a wrong input shape, you are mentioning <code>nn.AvgPool2d</code> for the posted input.</p>
<p>Anyway, these codes should work:</p>
<pre><code class="lang-python"># 3D
x = torch.randn([1, 2, 3, 256, 256])
pool = nn.AvgPool3d(kernel_size=3, stride=2, padding=1)
out = pool(x)

# 2D
x = torch.randn([2, 3, 256, 256])
pool = nn.AvgPool2d(kernel_size=3, stride=2, padding=1)
out = pool(x)
</code></pre>
Using NVIDIA apex for training, i cannot get same accuracy after training

Gwon: <p>Hi, I have a questions about NVIDIA apex<br>
I know NVIDIA apex package creates each process per gpu, like this</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d87451234bed9f147a02a8b7876e9e48a43bd48d.png" data-download-href="https://discuss.pytorch.org/uploads/default/d87451234bed9f147a02a8b7876e9e48a43bd48d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d87451234bed9f147a02a8b7876e9e48a43bd48d.png" alt="image" data-base62-sha1="uSQlaOfk6rntfVemxQcG9PeLDf7" width="690" height="187" data-dominant-color="351A31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">718×195 4.14 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>so, each process are referred as local_rank variable in my code<br>
I want to save best accuracy from each process and i coding like below</p>
<p>When i Using 2 gpus</p>
<pre data-code-wrap="shell"><code class="lang-shell">for epochs in range(0, args.epoch):
   train()
   test()
    ...
   save_best()
</code></pre>
<pre data-code-wrap="shell"><code class="lang-shell">def save_best():
   # 1'th gpu
   if args.local_rank == 0:
      is_best = test_acc &gt; best_acc
      best_acc = max(test_acc, best_acc)
      if is_best:
         torch.save(...)
   # 2'th gpu
   if args.local_rank == 1:
      is_best = test_acc &gt; best_acc
      best_acc = max(test_acc, best_acc)
      if is_best:
         torch.save(...)

</code></pre>
<p>After 1 epoch I can verify each accuracy<br>
<strong>0’th gpu’s accuracy is 19.906, It is saved 0’th weight file</strong><br>
<strong>1’th gpu’s accuracy is 19.269, It is saved 1’th weight file</strong></p>
<p>But, When i loading weight file and adapt to network, test accuracy is not equal to each result<br>
I got <strong>19.572(0’th file)</strong>,  <strong>19.561(1’th file)</strong></p>
<p>Surprisingly, When i using 1 gpu for training, the situation that i mentioned above is not happened(test accuracy while training is equal to accuracy which is loading from weight file)</p>
<p>I can’t understand why this situation is happened.<br>
Any body can help?</p>

ptrblck: <p>We recommend to use the native mixed-precision training utility via <a href="https://pytorch.org/docs/stable/amp.html"><code>torch.cuda.amp</code></a> instead of <code>apex</code>, as it should cover more tested use cases.<br>
More information can be found <a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">here</a>.</p>
'module' object is not iterable:

dalalaa: <p>How can I iterate nn.squential() module like a list:</p>
<pre><code class="lang-auto">RuntimeError: 
'module' object is not iterable:
at /home/dai/scripts/card_ocr_cpu/detector/model_torchscript.py:83:8
    def forward(self, x):
        out = []
        for i, m in enumerate(self.features):
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  &lt;--- HERE
            x = m(x)
            if i in [3, 6, 13, 18]:
                out.append(x)
        return out
'__torch__.extractor.forward' is being compiled since it was called from '__torch__.EAST.forward'
at /home/dai/scripts/card_ocr_cpu/detector/model_torchscript.py:209:8
    def forward(self, x, train:bool=False):
        x1,x2,x3,x4 = self.extractor(x)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        x = self.merge(x1,x2,x3,x4)
        score,geo = self.output(x)
        if not train:
            boxes = get_boxes_torch(score,geo,score_thresh=0.95,nms_thresh=0.2)
</code></pre>

ptrblck: <p>Could you try to add all modules into an <code>nn.ModuleList</code> instead of <code>nn.Sequential</code>?</p>

dalalaa: <p>Thank you , I fixed this problem by removing <code>enumerate()</code>.</p>

Enamex: <p><code>nn.ModuleList</code> can’t be enumerated or subscripted in a <code>jit.script</code> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

eellison: <p>It can be enumerated, it can’t currently be subscripted.</p>

kwanUm: <p>Are you sure? I’m trying to iterate over a ModuleList (without enumerate) and get the error “‘<strong>torch</strong>.torch.nn.modules.container.ModuleList’ object is not iterable”. Latest pytorch version</p>
Loss function shape

lycaenidae: <p>Hi, What is the meaning of this jumps in the loss function?<br>
I am decreasing the learning rate at iterations [300, 600, 900, 1200]</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e0ec375b759bde8db414b0c23ca3f2167c75ddab.png" alt="Screenshot 2020-09-09 at 07.22.00" data-base62-sha1="w5L2Krhy02kGBK7soO0nFw5pSDp" width="191" height="127"></p>

shartzog: <p>Hi, <a class="mention" href="/u/lycaenidae">@lycaenidae</a>. Some questions/comments, but fair warning… I’m not very experienced with pytorch. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<ul>
<li>What is the scale of your plot?</li>
<li>Are you calculating loss as an instantaneous value? A moving average?</li>
<li>What is your use case? Without some context, it’s tough to tell what could be going on.</li>
<li>Are you shuffling the samples in your training set? If not, your instantaneous loss values could definitely jump around like that. Datasets are sometimes “clumpy”. Imagine hitting your model with a large “clump” of a novel class after you’ve been training for a while.</li>
</ul>
Layer wise backward pass profiling

solvely99: <p>I’m trying to figure out a way to do layer wise backward pass profiling.<br>
I’ve used register_backward_hook(hook_func)<br>
and have been trying to use NVTX push and pop in the hook_func.</p>
<p>however, since the hook_func is executed after the backward pass for a certain layer,<br>
it seems that it is impossible to properly annotate each layer for backward pass.</p>
<p>is there another way to profile layer wise backward pass using NVTX?</p>
Link Regarding fusing of Alexnet model

Ganga: <ul>
<li>
<p>Hi there, Can anyone provide the useful link that explains the fusing of Alexnet model</p>
</li>
<li>
<p>and also, the  Code for fusing of layers are also appreciated</p>
</li>
</ul>

ptrblck: <p>What kind of fusion are you referring to? Could you post a reference for it?</p>

Ganga: <p>fusing Bnorm(learnable parameter) with FClinear(learnable parameter)</p>

ptrblck: <p>Thanks for the information.<br>
It seems to be a double post from <a href="https://discuss.pytorch.org/t/fusing-of-layers/93088">here</a>, where users have already tried to help.</p>

Ganga: <p>Ya,but I didn’t get What I required from  that post,as I am struck on these for longer time your help will be appreciated</p>
Profile with multi-cards running principle

yigu: <p>Hi, I am using profile to supervise the bottleneck of network.I find a weired phenomenon that, when I run a short code use cuda,I use ‘watch -n 1 nvidia-smi’ in another terminal,then I find that, four processes run on four cards one by one.<br>
I find reason in origin codes, I find profiler_cuda.cpp, in which it will warmup all cuda devices in onEachDevice interface. And in record interface, it use cudaGetDevice, which returns the current device for the calling host thread.<br>
And I am confused, if profiler use underlying thread on one single card, then why will run four processes on four cards?<br>
And how profile deals with the four results?Use the host thread result?<br>
And how to use profile in multi-cards? In which case recommend user to use multicards profile?<br>
Are there more detailed information I can find or provide with me?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a><br>
Thank you very much!<br>
May you a good day!</p>

ptrblck: <p>I don’t quite understand the issue.<br>
Are you manually calling <code>onEachDevice</code> or how are you using the profiler?<br>
Also, are you using the C++ API or are you running a Python script? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

yigu: <p>I run Python script, and I see Python code, followed by C++ code.In C++ code, I find onEachDevice is called at the beginning of profiler start.<br>
The phenomenon I saw is when I run Python code, I opened another terminal, and found four cards were running processes, but I didn’t set multi-cards and run multi-process.So I am confused.<br>
Later, I put servel kernels on different kernels with ‘torch.cuda.set_device(0/1/2/3), y = y.to(“cuda:0/1/2/3”)’, and found that profiler running on serverl cards when tensors were put to different cards , and at last deal with the result by host thread.<br>
At last, I want to summarize:profile with mulit-cuda cards will warm up all cards, and if kernels in python are put to different cards, the card will run a new process to run the kernel. Infomation such as tid will be collected in Event and return back to profile(Python) and show to user.<br>
Is there anything wrong?<br>
Thank you a lot!</p>
ResNet model did not use GPU memory size as expected

111337: <p>I’m using torchvision.models.resnet50 to do some experiment. But I found that the memory size used by by the model on GPU is less than it is expected to be.<br>
The following is my code:</p>
<pre><code class="lang-auto">    import torch
    import torchvision
    from torchsummary import summary
    import numpy as np

    model = torchvision.models.resnet50(pretrained=True)
    model = model.cuda()
    summary(model, (3, 448, 448), 24)
    input = torch.tensor(np.random.random((24, 3, 448, 448)), dtype=torch.float32)
    input = input.cuda()
    while True:
        model(input)
</code></pre>
<p>When I run this code, the summary function tell me that, total memory to be used is about 27GB, including parameters and forward/backward pass size. I think this is nearly to the expected size according to resnet50’s network.<br>
But, when the model was transformed to GPU and run training, I found the GPU memory usage was only kept about 9GB.<br>
Is this normal? I’m worried about the GPU is not doing whole computing for my training job.<br>
Could someone give me some help? Thanks!</p>

Forrest: <p>Hi, have you find the solution, I have a similar question to your’s. The expected size and actual size is not consistent, the latter is almost half of the former.</p>
Very slow for gradient penalty!

zzd1992: <p>I work on cuda8.0 and cudnn.<br>
The version of pytorch is 0.4.0.<br>
I find it is very slow when I apply gradient penalty (GP) for training cifar10 with resnet18.<br>
I test the average running time of each step with and without GP:<br>
without : 0.065s<br>
with GP: 0.330s</p>
<p>Here is my code:</p>
<pre><code class="lang-python">import torch
from torch import nn, autograd
from torch.autograd import Variable
from models import PreActResNet18
import time

net = PreActResNet18()
net.cuda()
opt = torch.optim.SGD(net.parameters(), 0.01, momentum=0.9, weight_decay=1e-4)
batchsize = 128

GP = True
start = time.time()

for i in range(100):
    x = torch.rand((batchsize, 3, 32, 32)).cuda()
    y = torch.randint(0, 10, (batchsize,)).cuda().long()
    x, y = Variable(x, requires_grad=True), Variable(y)
    opt.zero_grad()
    preds = net(x)
    loss_c = nn.CrossEntropyLoss()(preds, y)

    if GP:
        grad = autograd.grad(loss_c, x, create_graph=True, retain_graph=True,
                             only_inputs=True)[0]
        loss_g = (grad ** 2).mean() * batchsize
        (loss_c + loss_g).backward()
    else:
        loss_c.backward()
    opt.step()

end = time.time()
print("{:.3f}s for each step".format((end-start)/100))
</code></pre>
<p>Is there better implementation?</p>

Mohammad_Ashtari: <p>I am running into a very similar issue, have you heard anything back?</p>
Build_vocab function does not take the argument "specials"

JonathanSum: <p>text.build_vocab(ds_train, max_size=num_words, specials=[‘’,‘’])<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/972b2ea86344aa874b5222867c05052b423d43c9.png" data-download-href="https://discuss.pytorch.org/uploads/default/972b2ea86344aa874b5222867c05052b423d43c9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/972b2ea86344aa874b5222867c05052b423d43c9_2_690x353.png" alt="image" data-base62-sha1="lziCalSWogOn51i0h00cY09E91n" width="690" height="353" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/972b2ea86344aa874b5222867c05052b423d43c9_2_690x353.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/972b2ea86344aa874b5222867c05052b423d43c9_2_1035x529.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/972b2ea86344aa874b5222867c05052b423d43c9.png 2x" data-dominant-color="3A3B39"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1327×679 69.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Full code: <a href="https://github.com/Atcold/pytorch-Deep-Learning/blob/master/15-transformer.ipynb" class="inline-onebox" rel="noopener nofollow ugc">NYU-DLSP20/15-transformer.ipynb at master · Atcold/NYU-DLSP20 · GitHub</a><br>
Pytorch version: 1.6.0+cu101</p>
<p>If you don’t mind, I want to ask how to do a mask token in it. I see some transformer model that is masking some word and fill it in later.</p>
Datasets - aspect ratio grouping & get_get_height_and_width

HyamsG: <p>In the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">TorchVision Finetuning tutorial</a>, it was mentioned that ‘’‘get_height_and_width’’’ method should be implemented as part of the PyTorch dataset, when aspect-ratio grouping is implemented. I’ve searched the web for examples of such datasets with that method, but could not find any. Do you have an example of such datasets implementations? I wish to see how to aspect ration grouping and the get_height_and_width are commonly implemented.</p>
<p>thanks</p>

aaniin: <p>Hello <a class="mention" href="/u/hyamsg">@HyamsG</a> ,<br>
(Sorry for bumping an old thread.)<br>
I came across the same question today. I used the fine-tuning tutorial with my own custom dataset and saw a prompt asking me to implement get_height_and_width.<br>
The structure of the get_height_and_width function can be inferred from the tutorial’s group_by_aspect_ratio.py, in particular its _compute_aspect_ratios_custom_dataset() where it reads <code>height, width = dataset.get_height_and_width(i)</code>.<br>
In my case, I simply implemented like so:</p>
<pre><code class="lang-auto">    def get_height_and_width(self,idx):
        return self.img_data[idx]['height'],self.img_data[idx]['width']
</code></pre>
<p>which works for me so far.</p>
<p>Can’t answer on benefits of aspect ratio grouping and whether it’s commonly used.</p>
I can not change shape [4] to [4,1] using .view(4,1)?

ReubenZ: <p>I am very new to coding + pytorch</p>
<p>I have an input tensor torch.full((row,col), 2.0), called A<br>
and I want to have an output looks like [[2,2,2], [4,4,4], [6,6,6], …], called B<br>
The idea is to multiple A and B, pytorch will do broadcasting for me if I understand correctly.<br>
So I tried:<br>
r, c = A.size()<br>
x = torch.arange( r ) + 1.0<br>
the x has shape ‘torch.Size([4])’, it seems like not broadcastable, which means A*x reports error.</p>
<p>I tried a little experiment, y = torch.randn(4,1). Then A*y is ok. So I think this is because x has shape [4] and y has shape [4,1]. I tried to change x’s shape to [4,1] using x.view(4,1). But it didn’t work. The shape of x is still [4]. Could anyone help me on this? Thank you a lot for your time!</p>

tom: <p><code>x.view(4, 1)</code> doesn’t change x but returns a tensor (view) of shape <code>4 x 1</code> if you want you can assign it to a new variable and use that or plug it into your operation directly.</p>

Stephane_Bersier: <p>The x vector should have c rows. Try this:</p>
<pre><code class="lang-auto">&gt; &gt;&gt;&gt; import torch
&gt; &gt;&gt;&gt; row=4; col=7
&gt; &gt;&gt;&gt; A=torch.full((row,col),2.0)
&gt; &gt;&gt;&gt; r,c=A.size()
&gt; &gt;&gt;&gt; r,c
&gt; (4, 7)
&gt; &gt;&gt;&gt; x=torch.arange(c)+1.0
&gt; &gt;&gt;&gt; x
&gt; tensor([1., 2., 3., 4., 5., 6., 7.])
&gt; &gt;&gt;&gt; A*x
&gt; tensor([[ 2.,  4.,  6.,  8., 10., 12., 14.],
&gt;         [ 2.,  4.,  6.,  8., 10., 12., 14.],
&gt;         [ 2.,  4.,  6.,  8., 10., 12., 14.],
&gt;         [ 2.,  4.,  6.,  8., 10., 12., 14.]])
</code></pre>
<p>Or, in a more simple way: In A*x, the number of rows for the x vector, should be equal to the number columns of matrix A.</p>

ReubenZ: <p>Thank you very much! It works exactly the way I want</p>

ReubenZ: <p>Thank you very much!</p>
Dqn numpy only solve Pendulum, copying from pytorch

okay_All_Right: <p>dqn numpy only solve Pendulum<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/Isawyou/dqn-numpy-only-not-working" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/59315334?s=400&amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/Isawyou/dqn-numpy-only-not-working" target="_blank" rel="nofollow noopener">Isawyou/dqn-numpy-only-not-working</a></h3>

<p>the second file is pytorch implementation, solves Pendulum-v0 in one minute, I copied every piece of code, still not working, is there something wrong ? contact me if there is a bug to solve Pendul...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Library for adversarial ML

learned_fool: <p>Hello Guys!</p>
<p>I am working on a project involving Adversarial Machine Learning using PyTorch, and I wanted to know what the best package for it is. I did take a look around Github and Found <a href="https://github.com/tensorflow/cleverhans" rel="nofollow noopener">Cleverhans</a>, but that is more TensorFlow oriented. I also found <a href="https://github.com/BorealisAI/advertorch" rel="nofollow noopener">advertorch</a> and <a href="https://github.com/Trusted-AI/adversarial-robustness-toolbox" rel="nofollow noopener">ART</a>. I am not sure which is the better one/are there other packages that are better.</p>
<p>Could some one experienced help me find the best package for this purpose?</p>
Big gaps between forward and backward computations when profiling forward and backward pass of convolution

ahsabali: <p>I am profiling forward and backward pass of convolution using the following code, and visualizing the profile using chrome://tracing.</p>
<p>I notice big gaps between forward and backward computations, before and after torch::autograd::GraphRoot, shown in the following picture.</p>
<p>Any ideas why these gaps are there?</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
import torch.autograd as Ag

Device = torch.device("cuda:0")
ProblemSize = 100
NumChannels = 5
NumFilters = 96
ClassTypeName = "Single"
ClassType = torch.float32

X = torch.rand(1, NumChannels, ProblemSize, ProblemSize, dtype=ClassType, requires_grad=True).to(Device)
weights = torch.rand(NumFilters, NumChannels, 10, 10, dtype=ClassType, requires_grad=True).to(Device)
    
#warm up
Y = F.conv2d(X, weights)
Ysum = torch.sum(Y)
grad = Ag.grad(Ysum, weights)

Y = F.conv2d(X, weights)
Ysum = torch.sum(Y)
grad = Ag.grad(Ysum, weights)

#profile
with torch.autograd.profiler.profile(use_cuda=True) as prof:
    Y = F.conv2d(X, weights)
    Ysum = torch.sum(Y)
    grad = Ag.grad(Ysum, weights)
print(prof)
prof.export_chrome_trace("pytorch_profile_conv_grad.json")
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3ddf575788649ba9b6c559c9d6c06b0d5bb66c0c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3ddf575788649ba9b6c559c9d6c06b0d5bb66c0c" title="pytorch_profile_conv"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3ddf575788649ba9b6c559c9d6c06b0d5bb66c0c_2_690x385.jpeg" alt="pytorch_profile_conv" data-base62-sha1="8PlDbhwBpoBP8mfBRqU6YjXJDLC" width="690" height="385" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3ddf575788649ba9b6c559c9d6c06b0d5bb66c0c_2_690x385.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3ddf575788649ba9b6c559c9d6c06b0d5bb66c0c_2_1035x577.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3ddf575788649ba9b6c559c9d6c06b0d5bb66c0c_2_1380x770.jpeg 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_profile_conv</span><span class="informations">1670×933 33.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

yigu: <p>I also meet same question, and are finding solution, <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a></p>

albanD: <p>Hi,</p>
<p>Quite a few things happen before the start of the backward pass (discovering the graph and what needs to be computed in there). I don’t know what is the scale of the benchmark here, but for very small ops, this is possible.</p>
Pytorch molecular dynamics

solotope: <p>Wrote a working <a href="https://colab.research.google.com/drive/19IzyAS67YEhZ_WWc42KfX832CnStJs0_" rel="nofollow noopener">50-line pytorch protein molecular dynamics engine in numpy autograd</a>  which I am now porting to pytorch</p>
<p>Here’s the code:</p>
<pre><code class="lang-auto">!pip install autograd
!pip install celluloid

import matplotlib.pyplot as plt
import numpy as np
import torch

n_molecules = 50
n_atoms_per_molecule = 2
box_size = 20
temperature = 0.5

def energy(positions):
    "Compute the energy of a Lennard-Jones system."
    sigma = 1.0
    epsilon = 1.0
    rc = 3 * sigma
    e0 = 4 * epsilon * ((sigma / rc)**12 - (sigma / rc)**6)
    natoms = positions.shape[0]
    a, b = torch.triu_indices(natoms, natoms, 1)
    d = positions[a] - positions[b]
    r2 = torch.sum(d**2, dim=1)
    c6 = (sigma**2 / r2)**3
    energy = -e0 * (c6 != 0.0).sum()
    c12 = c6**2
    energy += torch.sum(4 * epsilon * (c12 - c6))
    return energy
    
def total_energy(positions):
    return energy(positions)

velocities = torch.tensor(np.random.uniform(low=-10, high=10, size = (n_molecules,  2)), requires_grad=True)
positions = torch.tensor(np.random.uniform(low=-box_size/2, high=box_size/2, size = (n_molecules, 2)), requires_grad=True)

from celluloid import Camera
camera = Camera(plt.figure())

for t in range(10000):
    energies = total_energy(positions)
    energies.backward()

    with torch.no_grad():
        forces = -positions.grad
        velocities = velocities +forces*1e-2
        #velocities = velocities * torch.sqrt(temperature/ torch.mean(velocities**2)) #thermostat
        positions= positions + velocities*1e-2
    
anim = camera.animate(blit=True, interval=50)
plt.style.use("dark_background")
anim.save('output.mp4')
</code></pre>
<p>However I get an error of</p>
<p><code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code></p>

Henry_Chibueze: <p>Sorry I don’t really have time to look up this code so can u pls indicate the line where this error is coming from I’m sure it’s printed on the console too.</p>

Henry_Chibueze: <p>Ok I think it’s in the place u typed ‘-positions.grad’<br>
Remove the ‘.grad’ because u already specified “with torch.no_grad( )” and this automatically removes all gradients from all tensors so it’s like calling the gradient that doesn’t even exist</p>

Henry_Chibueze: <p>If u want to use the gradient of that tensor what u should do is to set a variable equal to that gradient outside the with statement and then call it in the with statement</p>
Can pytorch load code from disk after running for a while?

greaber: <p>I had a surprising situation where my training was running for a while and then it crashed due to a syntax error because I had edited the script after I started the training and introduced a syntax error.  I am using distributed data parallel and <code>torch.utils.data</code>, which I know both spawn workers.  But I thought they just did this at launch.  Can someone explain how this could happen?</p>
<p>Possibly related: I noticed that after validation runs I get warning messages like <code>/home/grant/miniconda3/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 3 leaked semaphores to clean up at shutdown   len(cache))</code>, which I normally only would get when the whole script stops.  (By the way, I have no idea what these warnings mean – do they indicate that I am doing something a little wrong, or are they just unavoidable?)</p>

albanD: <p>Hey,</p>
<p>I would guess this is due to this issue: <a href="https://github.com/pytorch/pytorch/issues/15849" class="inline-onebox">DataLoader with option to re-use worker processes · Issue #15849 · pytorch/pytorch · GitHub</a><br>
The DataLoader recreates the worker processed regularly right now I’m afraid.</p>
<blockquote>
<p>By the way, I have no idea what these warnings mean – do they indicate that I am doing something a little wrong, or are they just unavoidable?</p>
</blockquote>
<p>Do you do any custom multiprocessing? Or you just use the DataLoader class?</p>

greaber: <p>Thanks!  This behavior surprised me.  I’m not doing any custom multiprocessing except for one call to <code>torch.multiprocessing.spawn</code> to use distributed data parallel.</p>

albanD: <p>I’m not sure then… If you don’t have any deadlock issues, I would ignore the warning <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Pytorch inference is slow inside Flask API

voqtuyen: <p>I encounter a problem with the inference speed between inference in main thread and inside Flask API enpoint (which is inside another thread). The inference speed is about 10 times slower.<br>
Do you have any idea about this problem and any solution?</p>

albanD: <p>Hi,</p>
<p>Do you know if flask is reducing the priority of these worker threads maybe?</p>
<p>Another possibility is that these worker threads don’t use multithreading to process your model?<br>
Or that so many of these threads use multithreading that you over-use the CPU which leads to slow down?</p>

voqtuyen: <p>I have tested it with a single API running pytorch model inference. The speed is also lower than that running in main thread, but much less worse (2 times slower). That is, multiple threads running could take up CPU usage. However, in the case of a single API, i did not figure out how to improve it</p>

tom: <p>Do you run your PyTorch model in Python or have you JITed it to TorchScript?<br>
We discuss this a bit in <a href="https://pytorch.org/deep-learning-with-pytorch" rel="nofollow noopener">Chapter 15 of our book</a>, if you run your model through the JIT - even from Python - you avoid the infamous Python GIL for all intermediates, which may give you an edge in multithreaded deployment scenarios.</p>
<p>Best regards</p>
<p>Thomas</p>

voqtuyen: <p>I run Pytorch model in Python</p>

tom: <p>I would suspect that JITing the model would already help, then.</p>

voqtuyen: <p>Thanks for your help, i have tried JIT the model and it solved the inference speed problem. I am currently verifying the performance to see if there is any degradation?</p>
WHY does batch influence the loss?

pourya_farzi: <p>HI, I’m wondering about batch size, which definitely influences the epoch-loss graph. If we suppose we have the following tensors as the predictions of the model and real values, <em>the result of loss would be different due to the shifting of elements between the batches.</em> (values are random and not  normalized)</p>
<pre><code class="lang-auto">loss = nn.MSELoss()
predicted_batch_1 = torch.tensor([40., -1,-0.7,1]).float()
ground_truth_batch_1 =  torch.tensor([1.1,180,15,1]).float() 

predicted_batch_2 =  torch.tensor([1,89.5]).float()
ground_truth_batch_2 = torch.tensor([1,14]).float()

output1 = loss(predicted_batch_1 , ground_truth_batch_1 )  # is  8630.1748
output2 = loss(predicted_batch_2 , ground_truth_batch_2 ) #  is 2850.1250
total loss= output1 +output2 # 11480.2998
</code></pre>
<p>if we concatenate the predicted tensors and also do concatenation for ground truths. then the result is 6703.4917<br>
<strong>HEREIN LIES THE PROBLEM</strong> :: if we swap one element in both predicted and ground truth from one batch to another (like the following snippet ), the result would be different and we can’t easily use the mean of the total loss for plotting EPOCH_LOSS since the mean of the above batches is different from the one below.</p>
<pre><code class="lang-auto">predicted_batch_1 = torch.tensor([40., -1,-0.7,89.5]).float()
ground_truth_batch_1 =  torch.tensor([1.1,180,15,14]).float() 

predicted_batch_2 =  torch.tensor([1,1]).float()
ground_truth_batch_2 = torch.tensor([1,1]).float()

output1 = loss(predicted_batch_1 , ground_truth_batch_1 )  # is  10055.2373
output2 = loss(predicted_batch_2 , ground_truth_batch_2 ) #  is 0.0
total loss= output1 +output2 # 10055.2373
</code></pre>
<p>I’m looking forward to your valuable comments.</p>

tom: <p>Well, here, you have a batch size of 4 and one of 2 and set up MSELoss to average over the batch items.<br>
If you concatenate, you take MSELoss to average over all 6 items and it effectively computes the weighted average <code>(4*8630+2*2850)/6 = 6703</code>.<br>
When using a typical setup with PyTorch’s DataLoader, all batches but the last (and there is the drop_last argument to just leave it out) have the same size, so the weights in the averaging would be all the same.</p>
Simple mistake when adding scalar to a tensor

youzunzhi: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/5/458f7ab0fbbd4ea44f0d7e5cac1a5489cc94804b.png" alt="image" data-base62-sha1="9VmlB6YCQvT2fGEiXtG5mPVuiAb" width="346" height="448"></p>
<p>why would such thing happen and how to avoid this?</p>

Nikronic: <p>Hi,<br>
Actually, this is not a skipped bug or something like that. Numpy also has same semantic and it is due to precision of floating point numbers.<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/14279267/float-precision-breakdown-in-python-numpy-when-adding-numbers/14279864#14279864" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/748858/mgilson" target="_blank" rel="nofollow noopener">
    <img alt="mgilson" src="https://i.stack.imgur.com/17gYz.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/14279267/float-precision-breakdown-in-python-numpy-when-adding-numbers/14279864#14279864" target="_blank" rel="nofollow noopener">Float precision breakdown in python/numpy when adding numbers</a>
</h4>

<div class="tags">
  <strong>python, numpy, precision</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/748858/mgilson" target="_blank" rel="nofollow noopener">
    mgilson
  </a>
  on <a href="https://stackoverflow.com/questions/14279267/float-precision-breakdown-in-python-numpy-when-adding-numbers/14279864#14279864" target="_blank" rel="nofollow noopener">02:10PM - 11 Jan 13 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>You can find some information about dtypes wrt to this post by using</p>
<pre><code class="lang-auto">i = torch.finfo(torch.float32)
i.eps  
</code></pre>
<p>Bests</p>

youzunzhi: <p>Yeah. Seems setting <code>a</code> to <code>torch.double</code> could solve the problem. Thanks.</p>
Spherical shell convolution?

davidlee: <p>Hi,</p>
<p>Is there any ways to employ convolution (I guess geometric convolution?) for 3D spherical shell data?<br>
Since I am not familiar with 3D data, could you please let me know any keywords that I can more dive into?</p>
<p>Also, Do you know any relevant studies that took advantage of both 3D spherical data and 2D planar data simultaneously?</p>
<p>Best,</p>
Backward fails with torch v1.6.0

cerisara: <p>I’ve a piece of code that’s running fine with torch v1.4.0, but that fails with torch v1.6.0 with error:<br>
“RuntimeError: Trying to backward through the graph a second time…”</p>
<p>Is it something known/expected within this update ?</p>
<p>The code is a basic linear layer with a custom but complex loss function, that is implemented as an nn.Module (too complex to copy here, sorry…)</p>
<p>Thank you for any hint about what happened during these updates that may relate to this error !</p>

albanD: <p>Hi,</p>
<p>We do quite a lot of changes between releases and you can check the release notes for any breaking change we make.</p>
<p>When you say “complex loss” you mean that in the sense that it is complicated or uses complex numbers? If it’s the second, we started working on autograd support for complex numbers. So you might get derivatives for things that used not to have some. You might need some extra <code>.detach()</code> to avoid any issue there.</p>
<p>Otherwise, the error comes from the fact that you’re calling backward on some part of the graph twice. Most likely because you share part of the graph between iterations. You might want to check that you don’t do any operation on Tensors that require gradients outside of the inner training loop.</p>
<p>Otherwise, we would need a code sample to give you more guidance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

cerisara: <p>Thank you for your reply.<br>
Sure, I understand that some piece of code is required, I’ll try to write a minimal version of the code and put it here if I can’t debug it myself <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I meant a complicated loss - but without parameters.<br>
I was just surprised that it works smoothly with v1.4.0… but OK, I’ll try and debug first, thanks for the great library !</p>

cerisara: <p>OK, found it: it had nothing to do with the loss, but with the inputs:<br>
the inputs xs are a list of pretrained word embeddings, and then, I used to build<br>
sentence embeddings before sending to the model training process like that:</p>
<pre><code class="lang-auto">xl = []
for xx in xs:
    # sum all words embeddings to get one sentence embedding
    x = torch.sum(xx.squeeze(),dim=0)
    xl.append(x)
# put all sentence embeddings into one tensor
x = torch.stack(xl)
</code></pre>
<p>The tensor x is then passed to the standard training process, but I forgot the</p>
<pre><code class="lang-auto">x = x.detach()
</code></pre>
<p>This used to work with 1.4.0, but not any more with 1.6.0, which makes sense.</p>
<p>Sorry for the false alarm, and thanks again !</p>
Autograd profiler total_average() calculation += exception

dkoss: <p>I had written a layer on top of the autograd profiler to collect more custom ranges of times, and with the upgrade to Pytorch 1.2 my usage is no longer working.</p>
<p>When a designated timing range begins, I have the following snippet (among a longer block in my codebase):</p>
<pre><code class="lang-auto">records = torch.autograd._disable_profiler()
events_average = EventList(parse_cpu_trace(records)).total_average()
cpu_time = events_average.cpu_time_total / 1000
cuda_time = events_average.cuda_time_total / 1000
</code></pre>
<p>When it runs, the .total_average() call in pytorch, <a href="https://github.com/pytorch/pytorch/blob/master/torch/autograd/profiler.py#L184" rel="nofollow noopener">at this line</a> (<code>total_stat += evt</code>) crashes with the Exception: <code>TypeError: unsupported operand type(s) for +=: 'FunctionEventAvg' and 'FunctionEvent'</code>. I noticed <code>FunctionEventAvg</code> doesn’t have a <code>__add__</code> override, but <a href="https://github.com/pytorch/pytorch/blob/8c3a94eaf2ee8cf5100cbb1252e9f56e871b24ee/torch/autograd/profiler.py#L413" rel="nofollow noopener">used to</a>. Seems like it changed with <a href="https://github.com/pytorch/pytorch/commit/864cfbc2162a874fd67b50414205483cb9ac6b5d" rel="nofollow noopener">this commit</a>.</p>
<p>I modified the source to change <code>+=</code> to <code>.add</code> and it seems to work now. Should <code>FunctionEventAvg</code> have an <code>__iadd__</code> method like it used to? Or should the <code>+=</code> be changed to <code>.add</code>? Am I using it wrong?</p>
<p>Thanks!</p>

ptrblck: <p>Thanks for reporting and tracking it down in the commit!<br>
I’ve never used the <code>EventList</code>, but if your use case is a “standard” one (not some kind of hack), would you mind creating a GitHub issue and liking your topic there?</p>

dkoss: <p>If I’m understanding the APIs correctly, my use case should be a “standard” one. I just filed the GitHub issue: <a href="https://github.com/pytorch/pytorch/issues/26433" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/26433</a>. Thanks for your help!</p>

yigu: <p>I had run same question, and can solve it by adding <strong>iadd</strong> , in which call self.add(other)</p>
Understanding PyTorch Profiler

ralzq01: <p>I’m currently using <code>torch.autograd.profiler.profile()</code> and <code>torch.autograd.profiler.record_function()</code> from <a href="https://pytorch.org/docs/stable/autograd.html#profiler" rel="nofollow noopener">PyTorch Profiler</a> for profiling my GPU program. I get confused with the output result by using <code>prof.key_averages().table()</code>. The output is organized as follows:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Name</th>
<th>Self CPU total %</th>
<th>Self CPU total</th>
<th>CPU total %</th>
<th>CPU total</th>
<th>CPU time avg</th>
<th>CUDA total %</th>
<th>CUDA total</th>
<th>CUDA time avg</th>
<th>Number of Calls</th>
</tr>
</thead>
<tbody>
<tr>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
</tbody>
</table>
</div><p>I don’t fully understand these, e.g., the relationship and difference between <code>Self CPU total</code> and <code>CPU total</code>. Can anyone provide an introduction to it (or provide some document links)?</p>
<p>Also I find some items like <code>to</code>, <code>index_select</code> has similar results on all <code>Self CPU total</code>, <code>CPU total</code>, <code>CUDA total</code>). I can understand <code>to</code> since it stands for data copy from host to device. But why  <code>index_select</code>?</p>

yigu: <p>I think the self cpu total is time of cost on kernel itself, for example, conv2d will call convolution.The Cpu total time cost on conv2d maybe 18.9ms, and time total cost on convolution is 17.9ms, then the self time total of conv2d and convolution is (18.9-17.9=1ms) and 17.9ms.<br>
If some kernle doesn’t call other kernel, then self cpu total is equal to cpu total.<br>
It’s my understanding.<br>
May you a goog day!</p>
Model loading from Colab

Ajinkya_Ambatwar: <p>Hi it’s a small problem I am facing with respect to model saving and loading.<br>
I am training a simple classification model in Google Colab. As per the instructions from <a href="https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_models_for_inference.html" rel="nofollow noopener">https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_models_for_inference.html</a><br>
I am saving the model as</p>
<pre><code class="lang-auto">torch.save(net.cpu().state_dict(),"/folder/model.pt")
</code></pre>
<p>After this, I am downloading the saved file from colab. While trying to load it(<strong>ONLY FOR INFERENCE</strong>)in my local machine as</p>
<pre><code class="lang-auto">model = NetClass()
model.load_state_dict(torch.load("/local_folder/model.pt")
</code></pre>
<p>It does not load the model but <strong>returns</strong> an error as</p>
<pre><code class="lang-auto">RuntimeError: ./mymodel/model.pt is a zip archive (did you mean to use torch.jit.load()?)
</code></pre>
<p>I tried unzipping the downloaded file manually and it has a directory structure which is expected. Now I don’t know where to proceed.</p>
<p>I tried the same method in the same colab file itself and <strong>IT WORKED</strong>. But in local file it is <strong>NOT WORKING</strong><br>
Any kind of help and suggestions would be highly appreciated!<br>
Thank you!</p>

ptrblck: <p>Are you using different PyTorch versions in Colab and on your local machine?<br>
If so, I guess your local installation might be older so could you update it?</p>
What does dataloader do when # data does not divide batch size

keith123: <p>I saw people using the following code:</p>
<pre><code class="lang-auto">    for i, (x_batch,) in enumerate(test_loader):
        y_pred = model(x_batch.float()).detach()      
        test_preds_fold[i * batch_size:(i+1) * batch_size] = y_pred.cpu().numpy()
</code></pre>
<p>where test_loader is a DataLoader. I know that drop_last is default to be false so this code does not look correct to me if the number of data does not divide the batch size. I would expect that it is possible for (i+1) * batch_size to exceed the capacity of test_preds_fold but when I tried running the code, there seems to be no problem. Now I am confused.</p>

pfloat: <p>Hello,<br>
maybe this code can help you see what I think is happening:</p>
<pre><code class="lang-auto">import numpy as np
a = np.arange(10)
print(a)
print(a[8: 16])
print(a[8:])
</code></pre>
<p>even if the range <code>i * batch_size:(i+1) * batch_size</code> is larger than the actual number of remaining values in <code>test_preds_fold</code> from the index <code>i * batch_size</code>, there will be no problem for assignement provided than the number of samples is equal to the length of <code>test_preds_fold</code>.</p>