What may cause Runtime Error in grid_sample?

username: zhezh
cooked: <p>Hi,</p>
<p>I am using pytorch 1.0.0 and 1.0.1.post2 on ubuntu 16.04</p>
<p>I use 5d tensor for grid_sample, my code is like below</p>
<pre><code class="lang-auto">for data in dataloader:
    heatmap = model(data)
    grid_flow = f(data)
    new_heatmap = F.grid_sample(heatmap, grid_flow)
</code></pre>
<p>this code run smoothly on most of the data, but at certain data(3 out of about 1500), it will cause:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "..........py", line 401, in &lt;module&gt;
    cam_R = meta['camera_R'].to(dev)
RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>
<p>I have already set <code>CUDA_LAUNCH_BLOCKING</code> to ‘1’, but it just cannot catch the right place where the error happens.</p>
<p>And I also tried comment <code>new_heatmap = F.grid_sample(heatmap, grid_flow)</code>, there will be no error.<br>
So, I am sure that the error  occurred in F.grid_sample.</p>
<hr>
<p><em><strong>Update 1</strong></em>:<br>
I change grid_sample mode to nearest, <code>new_heatmap = F.grid_sample(heatmap, grid_flow, mode='nearest')</code>  there is no runtime error.</p>
<hr>
<p>I just cannot determine what may cause this runtime error,</p>
<p>Any idea or help will be appreciated!</p>
<p>Thanks,<br>
Zhe</p>

username: ptrblck
cooked: <p>Do you get the same error if you run your code on the CPU?<br>
The error message might be a bit clearer then.</p>

username: zhezh
cooked: <p>Thanks for your advice, I run grid_flow in cpu mode with</p>
<pre><code class="lang-auto">new_heatmap = F.grid_sample(heatmap.cpu(), grid_flow.cpu() )
</code></pre>
<p>It will not cause any error on the same data which will cause CUDA Runtime error.</p>

username: zhezh
cooked: <p>This might caused by Oout of memory.<br>
I run the code in debug mode, and step on each line, the gpu memory usage start to increase and will result in</p>
<pre><code class="lang-auto">find_frame: frame not found.
    Looking for thread_id:pid_20017_id_140663683239608, frame_id:140663360355304
    Current     thread_id:pid_20017_id_140663683239608, available frames:
    94804309553960  -  94804309571304  -  140648434923752  -  140663064649256  -  94804081775976


    find_frame: frame not found.
    Looking for thread_id:pid_20017_id_140663683239608, frame_id:140663360355304
    Current     thread_id:pid_20017_id_140663683239608, available frames:
    94804309553960  -  94804309571304  -  140648434923752  -  140663064649256  -  94804081775976

RuntimeError: CUDA out of memory. Tried to allocate 600.00 MiB (GPU 0; 10.91 GiB total capacity; 9.78 GiB already allocated; 352.44 MiB free; 205.70 MiB cached)

</code></pre>
<p>But it still is weird that if I run this code in run mode, the error just happened in grid_sample().<br>
With help of <code>torch.cuda.memory_allocated(dev), torch.cuda.memory_cached(dev)</code>, I can see the allocated and cached memory are <code>2,733,509,632 and 5,054,136,320</code> just before execution of <code>grid_sample</code>, then the error <code>RuntimeError: CUDA error: an illegal memory access was encountered</code> occurs.</p>
<p>Still, if use mode=nearest in <code>grid_sample()</code>, no error occurs</p>

username: ptrblck
cooked: <p>Thanks for the debugging!<br>
Do you see this error, if you lower the memory usage, e.g. with a smaller batch size?</p>

username: zhezh
cooked: <p>Yep,<br>
the origin batch size is 2, and the error occurs at iter 422.<br>
Now, I decrease batch size to 1, and the error occurs at iter 844</p>
<p>I set a breakpoint just before <code>grid_sample</code> in iter 844, at the breakpoint, GPU memory usage is 3083MB,<br>
after execution of grid_sample(),</p>
<pre><code class="lang-auto">844it [05:01,  3.73it/s]THCudaCheck FAIL file=/pytorch/aten/src/THC/generated/../THCReduceAll.cuh line=317 error=77 : an illegal memory access was encountered
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=77 : an illegal memory access was encountered
</code></pre>
<p>and the memory usage is also 3083MB in nvidia-smi</p>

username: zhezh
cooked: <p><a href="https://1drv.ms/f/s!Ai2nc20bhVzugYI_SABdx2bYq-tp7w" class="onebox" target="_blank" rel="noopener nofollow ugc">https://1drv.ms/f/s!Ai2nc20bhVzugYI_SABdx2bYq-tp7w</a></p>
<p>I pickle save the heatmap and grid and write code to reproduce it.</p>
<p>I run it in pycharm with below option ticked<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d407c1049ebfb13f36a33266ca89ab129f99ff8c.png" data-download-href="https://discuss.pytorch.org/uploads/default/d407c1049ebfb13f36a33266ca89ab129f99ff8c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d407c1049ebfb13f36a33266ca89ab129f99ff8c_2_561x500.png" alt="image" data-base62-sha1="ufHPRIbtSneuo8tNLdX6TdhxCwc" width="561" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d407c1049ebfb13f36a33266ca89ab129f99ff8c_2_561x500.png, https://discuss.pytorch.org/uploads/default/original/2X/d/d407c1049ebfb13f36a33266ca89ab129f99ff8c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/d407c1049ebfb13f36a33266ca89ab129f99ff8c.png 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">755×672 49.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>it will throw an error:</p>
<pre><code class="lang-auto">torch.Size([4, 1, 16, 100, 4096])
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCTensorMath.cu line=238 error=77 : an illegal memory access was encountered
</code></pre>
<p>but if i run it without the option ticked, the error msg will not shown. Also not shown in debug mode or <code>python xxx.py</code><br>
I am not sure if there is no error msg or not shown.</p>

username: bnehoran
cooked: <p>I think the error you’re getting might be related to this issue<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/24823" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/bnehoran" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/9757500?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/24823" target="_blank" rel="nofollow noopener">Issue: Problematic handling of NaN and inf in grid_sample, causing segfaults, corrupted CUDA memory, and incorrect results</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/bnehoran" target="_blank" rel="nofollow noopener">bnehoran</a>
	on <a href="https://github.com/pytorch/pytorch/issues/24823" target="_blank" rel="nofollow noopener">2019-08-18</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">This issue is an expansion of the issue reported in #19826.
The discussion there diagnoses the segfault that occurs in the vectorized...</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>If so, the problem probably occurs whenever your predicted grid happens to contain an infinite or <code>NaN</code> value. Perhaps your model’s predictions blow up during training, or else you are dividing somewhere by a value that can sometimes be 0?</p>
<p>We will hopefully be able to fix this issue in <code>grid_sample</code> to handle these values without crashing, but in the mean time, my best recommendation for a workaround is to check for any <code>NaN</code> or infinite values in your grid (perhaps using <a href="https://pytorch.org/docs/stable/torch.html#torch.isfinite" rel="nofollow noopener"><code>torch.isfinite</code></a>) and replacing them with some stand-in value before passing to <code>grid_sample</code>.</p>

username: vikky2904
cooked: <p>I am getting this error when I try to access grid values that are out of range, in cuda but not in cpu.</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 64, 64)
flow = torch.nn.Tanh()(torch.randn(1, 64, 64, 2))
pred_x = F.grid_sample(x, flow)  # no problem
pred_x = F.grid_sample(x, 2*flow)  # no problem
pred_x = F.grid_sample(x.to(torch.device('cuda')), flow.to(torch.device('cuda'))) # no problem
pred_x = F.grid_sample(x.to(torch.device('cuda')), 2*flow.to(torch.device('cuda')))
# RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>
<p>I am using Pytorch 1.4.0.</p>

username: ptrblck
cooked: <p>I cannot reproduce this issue in the latest nightly.<br>
Could you update to the nightly binary (use a new conda environment, if necessary) and rerun the code, please?</p>

username: vikky2904
cooked: <p>Funnily enough, I am not able to reproduce this in stable or nightly any more.</p>
<p>I first encountered this while training a model to predict flow and warp an image as above. At one point, it gave me the “illegal memory access error”. And that error sustained in later commands with cuda tensors. Even after I restarted the python environment, data transfer from CPU to GPU (using <code>.to(torch.device('cuda'))</code>) was extremely slow. And my screen display was getting stuck repeatedly (same GPU is handling display on my laptop). I have rebooted my laptop, and now things are working fine again. (Just to let the information be recorded)</p>
<p>Still not sure why the “illegal memory access error” was encountered though. Do you think it might be because it encountered a nan or inf value in flow? Thoughts?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="vikky2904" data-post="11" data-topic="39837">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vikky2904/48/9917_2.png" class="avatar"> vikky2904:</div>
<blockquote>
<p>Do you think it might be because it encountered a nan or inf value in flow? Thoughts?</p>
</blockquote>
</aside>
<p>This might be the case, but I’m also unable to reproduce it using NaNs for <code>x</code> and <code>flow</code>.<br>
A <code>NaN</code> value should not cause an illegal memory access, but should rather yield a proper error message, so please let us know, if you are able to reproduce this behavior and we should fix it.</p>