CUDA Illegal Memory Access

chinmay5: <p>While trying to implement a backward pass, I keep getting the error of CUDA Illegal Memory Access.</p>
<pre><code class="lang-auto">@staticmethod
    def backward(ctx, grad_output):
         grad_label = grad_output.clone()
        num_ft = grad_output.shape[0]
        # grad_label.data.resize_(num_ft, 32, 41)
        lin_indices_3d, lin_indices_2d = ctx.saved_variables
        num_ind = lin_indices_3d.data[0]
        grad_label.data.view(num_ft, -1).index_copy_(1, lin_indices_2d.data[1:1 + num_ind],
                                                     torch.index_select(grad_output.data.contiguous().view(num_ft, -1),
                                                                        1, lin_indices_3d.data[1:1 + num_ind]))
        # raw_input('sdflkj')
        return grad_label, None, None, None
</code></pre>
<p>I tried using pdb to see what might be the possible cause</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f03aa17e2ee9fcd1dac844429c2625a1519c71a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/f03aa17e2ee9fcd1dac844429c2625a1519c71a9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f03aa17e2ee9fcd1dac844429c2625a1519c71a9_2_690x80.png" alt="image" data-base62-sha1="yhacWXcJev917o3awf9APMsYwTT" width="690" height="80" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f03aa17e2ee9fcd1dac844429c2625a1519c71a9_2_690x80.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f03aa17e2ee9fcd1dac844429c2625a1519c71a9_2_1035x120.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/0/f03aa17e2ee9fcd1dac844429c2625a1519c71a9.png 2x" data-dominant-color="272B20"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1224×142 49.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am not sure what is wrong in the implementation here. Any help would be highly appreciated.</p>
<p>I am using PyTorch 1.3 but the same error persists on 1.4 and 1.5</p>

tr_arun: <p>Hi,</p>
<p>Illegal memory access error occur when your program is trying to access an memory location for which the program does not have permission to access.</p>
<p>By setting <code>CUDA_LAUNCH_BLOCKING=1</code> , you can see where the error comes from.</p>

chinmay5: <p>When I run with the CUDA_LAUNCH_BLOCKING=1 I get the error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8ac97b5609ac07b00d4a5cc9c22de2d1e2e3a87.png" data-download-href="https://discuss.pytorch.org/uploads/default/e8ac97b5609ac07b00d4a5cc9c22de2d1e2e3a87" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8ac97b5609ac07b00d4a5cc9c22de2d1e2e3a87_2_690x160.png" alt="image" data-base62-sha1="xckyasBOgns4mLLCpjqGYBjbvV5" width="690" height="160" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8ac97b5609ac07b00d4a5cc9c22de2d1e2e3a87_2_690x160.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8ac97b5609ac07b00d4a5cc9c22de2d1e2e3a87_2_1035x240.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8ac97b5609ac07b00d4a5cc9c22de2d1e2e3a87.png 2x" data-dominant-color="282A25"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1144×266 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any idea what could be the reason?</p>

ptrblck: <p>Could you install the nightly binary (in a new virtual environment) and rerun the code?<br>
If you are still running into this error, could you post a code snippet to reproduce this issue, please?</p>

chinmay5: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I will start on that. I have another question though. Digging around it seems that the issue was not present in PyTorch 1.2 . I thought of downgrading to PyTorch 1.2 but as soon as I do that, I would get an error of<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37d7f4fe5de75a34bf149e0616f34ea0ba0320fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/37d7f4fe5de75a34bf149e0616f34ea0ba0320fd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37d7f4fe5de75a34bf149e0616f34ea0ba0320fd_2_690x85.png" alt="image" data-base62-sha1="7Y0WGPGLWEZwjF1NLHYNJENQ7I9" width="690" height="85" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37d7f4fe5de75a34bf149e0616f34ea0ba0320fd_2_690x85.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37d7f4fe5de75a34bf149e0616f34ea0ba0320fd_2_1035x127.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37d7f4fe5de75a34bf149e0616f34ea0ba0320fd_2_1380x170.png 2x" data-dominant-color="2A2D23"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1469×183 97.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For one of the PyBind modules. This error was not there when I worked with PyTorch 1.3 and above. Any ideas about this?</p>

ptrblck: <p>You might try to use <code>input_.data&lt;scalar_t&gt;()</code>, but I would recommend to stick to the latest version instead of downgrading.</p>

chinmay5: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Same error. What is the general reason for this error? I want to double check that it is not something related to input data before I open a new issue</p>

chinmay5: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I read in one of your previous posts that the error might come when the input tensor is not contiguous and I make sure that the input is contiguous by calling <code>.contiguous() </code> on the input tensor. At this point, I have no idea what is possibly wrong here.</p>

chinmay5: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> it works with the nightly version. Thank you so much for your support</p>

ptrblck: <p>I think the syntax just changed after <code>1.3</code>, as this seems to be a compilation error.<br>
Good to know, it’s working with the nightly. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>