Cublas runtime error

username: mbarnes1
cooked: <p>I get this cublas error after several epochs of training. The data is fed sequentially and the error always occurs at a different iteration, so it is difficult to reproduce in a simple example. Something is causing an assertion on line 97 of <a href="https://github.com/torch/cutorch/blob/master/lib/THC/THCTensorScatterGather.cu" rel="nofollow noopener">https://github.com/torch/cutorch/blob/master/lib/THC/THCTensorScatterGather.cu</a>. Any ideas of what could be happening, or what I should be checking for? I’ve tried just catching it and continuing onto the next iteration, but then there are just more (different) cublas errors on the next iteration, too.</p>
<p>The error always occurs during a forward call of a torch.nn.Linear object</p>
<p>RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/torch/lib/THC/THCBlas.cu:246<br>
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [29,0,0] Assertion <code>indexValue &gt;= 0 &amp;&amp; indexValue &lt; src.sizes[dim]</code> failed.</p>
<p>Thanks</p>

username: SimonW
cooked: <p>What does you architecture look like?</p>

username: mbarnes1
cooked: <p>It’s a custom RNN decoder for image captioning, described in this paper: <a href="https://arxiv.org/abs/1707.07998" rel="nofollow noopener">https://arxiv.org/abs/1707.07998</a>. Suppose I could give the full source code, though really wish I could provide a simple minimal working example.</p>
<p>Also if I do catch this exception and move onto the next iteration, I get this CUDA runtime error just trying to even move a tensor to the GPU for initialization.</p>
<p>File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 106, in init_state<br>
state_attention = (hidden_attention_lstm.cuda(), cell_attention_lstm.cuda())<br>
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/autograd/variable.py”, line 298, in cuda<br>
return CudaTransfer.apply(self, device, async)<br>
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py”, line 201, in forward<br>
return i.cuda(async=async)<br>
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/_utils.py”, line 69, in <em>cuda<br>
return new_type(self.size()).copy</em>(self, async)<br>
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generic/THCTensorCopy.c:20<br>
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [30,0,0] Assertion <code>indexValue &gt;= 0 &amp;&amp; indexValue &lt; src.sizes[dim]</code> failed.</p>

username: ngimel
cooked: <p>Cublas error is a red herring, your true problem is an assert thrown by gather kernel - is it possible that something is wrong with the indices that are sent to it? After assert us thrown, cuda context is corrupted, so don’t try catching it and continuing, the errors that you see are expected. <a href="http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#assertion" rel="nofollow noopener">http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#assertion</a></p>

username: mbarnes1
cooked: <p><a class="mention" href="/u/ngimel">@ngimel</a> How can I check for that? I’ve asserted that all the tensors being fed to this torch.nn.Linear layer are the correct dimension and don’t contain nan’s or inf values. Not sure what else could be corrupted.</p>

username: ngimel
cooked: <p>The problematic layer is not torch.nn.Linear, it’s something like index_select, or some advanced indexing that you are possibly using, hard to say without looking at your model.</p>

username: mbarnes1
cooked: <p>Okay. Just to clarify, even though the traceback (below) shows the error at torch.nn.Linear and I’ve checked the inputs to this layer, the issue could be caused by advanced indexing happening somewhere else? Doesn’t exactly make sense to me, but I can certainly go through and check the indexing in the rest of my code. Thanks</p>
<p>Traceback (most recent call last):<br>
File “example_server.py”, line 74, in <br>
validate_every=100, validater_examples=validater_examples, validate_examples_every=1000, use_cuda=True)<br>
File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/train.py”, line 61, in train<br>
_, logprobs = decoder.forward(V, y_0, state_attention, state_language, y_true=caption_true_packed)<br>
File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 78, in forward<br>
(state_language[0][0:batch_size_t], state_language[1][0:batch_size_t]))<br>
File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 158, in forward<br>
v_hat = self.attention.forward(V, new_state_attention[0])  # N x D<br>
File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 191, in forward<br>
V_hidden = [self.image_linear(v) for v in V]  # List of length N, containing 1 x k x H<br>
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 325, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/nn/modules/linear.py”, line 55, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/nn/functional.py”, line 837, in linear<br>
output = input.matmul(weight.t())<br>
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/autograd/variable.py”, line 386, in matmul<br>
return torch.matmul(self, other)<br>
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/functional.py”, line 191, in matmul<br>
output = <a href="http://torch.mm" rel="nofollow noopener">torch.mm</a>(tensor1, tensor2)<br>
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/torch/lib/THC/THCBlas.cu:246<br>
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [29,0,0] Assertion <code>indexValue &gt;= 0 &amp;&amp; indexValue &lt; src.sizes[dim]</code> failed.</p>

username: ngimel
cooked: <aside class="quote no-group" data-username="mbarnes1" data-post="7" data-topic="11243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbarnes1/48/2737_2.png" class="avatar"> mbarnes1:</div>
<blockquote>
<p>/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [29,0,0] Assertion indexValue &gt;= 0 &amp;&amp; indexValue &lt; src.sizes[dim] failed.</p>
</blockquote>
</aside>
<p>Here’s the part of the stack trace that corrupted your cuda context, all subsequent errors are stemming from this.</p>

username: mbarnes1
cooked: <p>Got it. Trying to understand where in my code there are corrupted indices, but it seems like all we know based on this traceback is its due to something that invokes the gather kernel. I’ll start with checking all the advanced indexing. Thanks for your help</p>

username: SimonW
cooked: <p>Try adding <code>torch.cuda.synchronize</code> after each cuda operation. That way the error won’t be deferred to later operations.<br>
<a href="http://pytorch.org/docs/master/cuda.html#torch.cuda.synchronize" class="onebox" target="_blank">http://pytorch.org/docs/master/cuda.html#torch.cuda.synchronize</a></p>
<p>If you can share your script, I can take a look.</p>

username: smth
cooked: <p>an easy way to get the correct stack trace with CUDA is to run:</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1 python yourscript.py
</code></pre>
<p>This puts cuda in synchronous mode (rather than it’s default of asynchronous mode)</p>

username: mbarnes1
cooked: <p>Ran a couple instances of training yesterday with CUDA_LAUNCH_BLOCKING=1, and was able to debug the issue based on the better stack trace. Long story short, it ended up being an issue with torch.gather very rarely receiving an invalid index. Thanks everyone for your help.</p>

username: Rafael_Valle
cooked: <aside class="quote no-group" data-username="SimonW" data-post="10" data-topic="11243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/48/1702_2.png" class="avatar"> SimonW:</div>
<blockquote>
<p>torch.cuda.synchronize</p>
</blockquote>
</aside>
<p>Did the trick for me!</p>

username: ezyang
cooked: <p><a class="mention" href="/u/rafael_valle">@Rafael_Valle</a> Erm, what exactly did you do?</p>

username: Rafael_Valle
cooked: <p>called torch.cuda.synchronize()</p>

username: isalirezag
cooked: <aside class="quote no-group" data-username="Rafael_Valle" data-post="15" data-topic="11243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/5f9b8f/48.png" class="avatar"> Rafael_Valle:</div>
<blockquote>
<p>torch.cuda.synchronize()</p>
</blockquote>
</aside>
<p>where did you put it?</p>

username: Rafael_Valle
cooked: <aside class="quote no-group" data-username="SimonW" data-post="10" data-topic="11243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/48/1702_2.png" class="avatar"> SimonW:</div>
<blockquote>
<p>Try adding <code>torch.cuda.synchronize</code> after each cuda operation.</p>
</blockquote>
</aside>
<p>Take a look at SimonW’s answer:<br>
“Try adding  <code>torch.cuda.synchronize</code>  after each cuda operation.”</p>

username: abdullahkhilji
cooked: <p>Yes, torch.gather is very efficient, but we need to be careful while using it. Thanks !</p>