Loss.backward throwing CUDA Errors

chinmay5: <p>Sorry that I am asking this again but I would need a confirmation before I try to switch to the nightly builds for the issue since they seem to be a bit unstable.</p>
<p>This is the code:-</p>
<pre><code class="lang-auto">@staticmethod
    def backward(ctx, grad_output):
         grad_label = grad_output.clone()
        num_ft = grad_output.shape[0]
        # grad_label.data.resize_(num_ft, 32, 41)
        lin_indices_3d, lin_indices_2d = ctx.saved_variables
        num_ind = lin_indices_3d.data[0]
        grad_label.data.view(num_ft, -1).index_copy_(1, lin_indices_2d.data[1:1 + num_ind],
                                                     torch.index_select(grad_output.data.contiguous().view(num_ft, -1),
                                                                        1, lin_indices_3d.data[1:1 + num_ind]))
        # raw_input('sdflkj')
        return grad_label, None, None, None

</code></pre>
<p>The error I get is:-<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c098d9846d1edfef87f966410ff69db7f1c9e91.png" data-download-href="https://discuss.pytorch.org/uploads/default/8c098d9846d1edfef87f966410ff69db7f1c9e91" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c098d9846d1edfef87f966410ff69db7f1c9e91_2_690x160.png" alt="image" data-base62-sha1="jYPjqj5TK4gUgJ0YaDboUXrAmgF" width="690" height="160" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c098d9846d1edfef87f966410ff69db7f1c9e91_2_690x160.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c098d9846d1edfef87f966410ff69db7f1c9e91_2_1035x240.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c098d9846d1edfef87f966410ff69db7f1c9e91.png 2x" data-dominant-color="282A26"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1035×240 58.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The reference issue is:-</p>
<aside class="quote quote-modified" data-post="1" data-topic="85166">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/48/12682_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/cuda-illegal-memory-access/85166">CUDA Illegal Memory Access</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    While trying to implement a backward pass, I keep getting the error of CUDA Illegal Memory Access. 
@staticmethod
    def backward(ctx, grad_output):
         grad_label = grad_output.clone()
        num_ft = grad_output.shape[0]
        # grad_label.data.resize_(num_ft, 32, 41)
        lin_indices_3d, lin_indices_2d = ctx.saved_variables
        num_ind = lin_indices_3d.data[0]
        grad_label.data.view(num_ft, -1).index_copy_(1, lin_indices_2d.data[1:1 + num_ind],
                          …
  </blockquote>
</aside>

<p>One of the solutions I saw was:-</p><aside class="quote" data-post="6" data-topic="75812">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/autograd-failed-after-add-max-pool1d-layer/75812/6">Autograd failed after add max_pool1d layer</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Thanks for the code snippet. 
Since it’s not reproducible with the last code, could you try to use: 
x= F.max_pool1d(x.transpose(1,2).contiguous(), kernel_size=x.size()[1])

and rerun the script, please? 
Also, which PyTorch version are you using? 
Make sure to update to the latest stable release, as Variables are deprecated since 0.4.
  </blockquote>
</aside>

<p>However, even after ensuring the input is contiguous I got the same error. I am still not sure about the actual reason for the error though</p>

albanD: <p>Hi,</p>
<p>You can try to enable <code>torch.autograd.set_detect_anomaly(True)</code> to see which forward function corresponds to the failing one in the backward.</p>
<p>Also You can try to run the same code on CPU to see if you get a better error message.<br>
Also, I would double check all the indices in the multiple indexing you do.<br>
Finally. You should <strong>never</strong> use <code>.data</code> anymore. This will reduce the chance of weird bugs a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

chinmay5: <p><a class="mention" href="/u/alband">@albanD</a><br>
The <code>detetct_anamaly</code> takes me to this code snippet:-</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ca712e98bead88ea68a40881db56c6ae0f87e31.png" data-download-href="https://discuss.pytorch.org/uploads/default/1ca712e98bead88ea68a40881db56c6ae0f87e31" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ca712e98bead88ea68a40881db56c6ae0f87e31_2_690x118.png" alt="image" data-base62-sha1="45tk2LL6qxu91wxjonuPjLc2gVj" width="690" height="118" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1ca712e98bead88ea68a40881db56c6ae0f87e31_2_690x118.png, https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ca712e98bead88ea68a40881db56c6ae0f87e31.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ca712e98bead88ea68a40881db56c6ae0f87e31.png 2x" data-dominant-color="2B2F25"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">957×164 57.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Which is related to MaxPool1D.</p>
<p>I think the indexing should be fine since this is a code snippet which worked on 0.4.0 version and I am trying to port it here.</p>
<p>Also, if I do not have to use <strong>.data</strong> what would be the alternative to ensure less bugs?</p>

albanD: <p>Can you share the kernel size and image size that you give to the pooling layer?<br>
Can you also add just after it:</p>
<pre><code class="lang-auto">def hook(grad):
    print("Inside hook:")
    print(grad.size())
    print(grad.stride())
imageft.register_hook(hook)
</code></pre>
<p>To know exactly the gradient properties.<br>
Thanks!</p>
<p>For the <code>.data</code>, it depends what you use it for.<br>
If you want to get a new tensor detached from the previous one wrt to the autograd, use <code>.detach()</code>.<br>
If you want to do ops that are not tracked by the autograd inplace, use the context manager <code>with torch.no_grad()</code>.</p>

chinmay5: <p><a class="mention" href="/u/alband">@albanD</a><br>
Here are the values:-</p>
<p>Input shape = torch.Size([128, 442368, 5])</p>
<p>Also output for the code snippet you asked:-</p>
<pre><code class="lang-auto">Inside hook:
torch.Size([128, 442368, 1])
(442368, 1, 1)
</code></pre>
<p>Also for the <strong>.data</strong> as you can see here, I am taking the gradients, reshaping it and doing a mask select. I am sorry but I am not sure to which of the mentioned categories would this fall to?</p>

albanD: <blockquote>
<p>Here are the values:-</p>
</blockquote>
<p>And what is the kernel size?</p>
<blockquote>
<p>Also for the <strong>.data</strong> as you can see here</p>
</blockquote>
<p>If you have no reason to use it, you can just remove it.<br>
It used to be useful when working with <code>Variable</code>s. But now that they don’t exist, you can just remove the <code>.data</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

chinmay5: <p>Kernel Size is 5. I basically remove the last dimension</p>

albanD: <p>Interesting.<br>
Running the following works fine for me, does it work for you as well?</p>
<pre><code class="lang-python">import torch

a = torch.rand([128, 442368, 5], device="cuda", requires_grad=True)

mod = torch.nn.MaxPool1d(5)

out = mod(a)

out.backward(torch.ones_like(out))
</code></pre>

chinmay5: <p><a class="mention" href="/u/alband">@albanD</a><br>
Here is what I get when trying the same thing:-</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55e7b3820c5aaf37d86b99bf85e225bee34d583d.png" data-download-href="https://discuss.pytorch.org/uploads/default/55e7b3820c5aaf37d86b99bf85e225bee34d583d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55e7b3820c5aaf37d86b99bf85e225bee34d583d_2_690x284.png" alt="image" data-base62-sha1="cfX09I7erzkQnU1hT4nYOxrm1R3" width="690" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55e7b3820c5aaf37d86b99bf85e225bee34d583d_2_690x284.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55e7b3820c5aaf37d86b99bf85e225bee34d583d_2_1035x426.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55e7b3820c5aaf37d86b99bf85e225bee34d583d_2_1380x568.png 2x" data-dominant-color="292C23"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1380×568 313 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Ah ! Making progress <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>So with the <code>Quadro GP100</code> and a source install I can’t reproduce that.</p>
<p>Do you see the same behavior on nightly build for this test script?<br>
Which version of cuda are you using and how did you installed pytorch?</p>

chinmay5: <p><a class="mention" href="/u/alband">@albanD</a><br>
I have not switched to nightly builds yet. I am using PyTorch 1.2 although I tried updating uptil 1.5 and error persists. In all the cases, I ended up using pip instllations.</p>
<p>As for the OS, it is Ubuntu 18.04 and the Cuda version is<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/2/32de47cee50185a6670d1ae32febf16b854d6626.png" alt="image" data-base62-sha1="7g06EsjYDWY4b2DhXxOJFJv2rIi" width="566" height="84"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e70c7e5425ab3c5a3d107f41fc79b3e71e768020.png" alt="image" data-base62-sha1="wXX3Ts52SR5ZI2J3wS4yPFEq2w8" width="672" height="370"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef986367d3702a78225c847a543c7b00d9414989.png" alt="image" data-base62-sha1="ybyByXXVDauyYsBPv4hSmMDREvv" width="685" height="106"></p>

albanD: <p>Thanks for all the info.</p>
<p>Can you try to create a temporary python virtual env, install nightly and try to run the simple repro script. above?</p>

chinmay5: <p><a class="mention" href="/u/alband">@albanD</a> I created a new environment.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf9505b7c3435d1df380c93ec50b3333726981cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf9505b7c3435d1df380c93ec50b3333726981cc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf9505b7c3435d1df380c93ec50b3333726981cc_2_690x167.png" alt="image" data-base62-sha1="rkOtuJkVgJBzV3XXvLKni2DPJuY" width="690" height="167" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf9505b7c3435d1df380c93ec50b3333726981cc_2_690x167.png, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf9505b7c3435d1df380c93ec50b3333726981cc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf9505b7c3435d1df380c93ec50b3333726981cc.png 2x" data-dominant-color="2A0D22"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">717×174 47.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the given case, I still get an error</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7fe2de4acd6a8fce936f0d5b14b640c8b551b64.png" data-download-href="https://discuss.pytorch.org/uploads/default/d7fe2de4acd6a8fce936f0d5b14b640c8b551b64" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7fe2de4acd6a8fce936f0d5b14b640c8b551b64_2_690x178.png" alt="image" data-base62-sha1="uOLeqAcIiQWUoFT7gwMeL20Kz3e" width="690" height="178" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7fe2de4acd6a8fce936f0d5b14b640c8b551b64_2_690x178.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7fe2de4acd6a8fce936f0d5b14b640c8b551b64_2_1035x267.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7fe2de4acd6a8fce936f0d5b14b640c8b551b64.png 2x" data-dominant-color="35192D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1177×305 268 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

chinmay5: <p>I attached the results. Do you think I am making some mistake?</p>

albanD: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> do you have an RTX 2070 somewhere to try and repro this by any chance? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

albanD: <p>I just found <a href="https://github.com/pytorch/pytorch/issues/38764">https://github.com/pytorch/pytorch/issues/38764</a> which seem to claim that this was fixed.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> will know better!</p>

chinmay5: <p>I made a mistake. I was able to run the code snippet without error on the nightly build. I will try to see if it works on the entire code next and will update you asap</p>

chinmay5: <p><a class="mention" href="/u/alband">@albanD</a> it works with the nightly version. Thank you so so much for all your support. I am pretty sure I could not have figured it out without your help</p>

ptrblck: <p>Sorry for the delayed reply, but yes it should be fixed by now. Let us know, if you run into this issue again, please.</p>

chinmay5: <p>Just a query, when would a stable version be out. I am not sure about the stability of the nightly builds <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>