Torch.gather throwing RuntimeError: CUDA error: device-side assert triggered

Rajan_Lagah: <pre><code class="lang-auto">reward = 3
def custom_loss(outputs,target):
        outputs  = F.softmax(outputs,dim=1)
        outputs, reservation = outputs[:,:-1], outputs[:,-1]
        gain = torch.gather(outputs, dim=1, index=target.unsqueeze(1)).squeeze()
        return loss
</code></pre>
<p>I am trying to implement custom loss but it was giving runtime cuda error so i remove lines step by step and tried to run CNN. It works fine. BUT</p>
<p>The moment i enter</p>
<p><code>        gain = torch.gather(outputs, dim=1, index=target.unsqueeze(1)).squeeze()</code></p>
<p>This line it through</p>
<p>RuntimeError: CUDA error: device-side assert triggered</p>

ptrblck: <p>The stack trace might point to an invalid index and you could rerun the code via:</p>
<pre><code class="lang-python">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>to see if this is indeed the line of code raising the error.<br>
Also, if you run the code on the CPU you might get a better error message.</p>

Rajan_Lagah: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="106505">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>CUDA_LAUNCH_BLOCKING</code></p>
</blockquote>
</aside>
<p>I am using jupyter notebook so for that i have to<br>
<code>os.environ['CUDA_LAUNCH_BLOCKING'] = '1'</code> # according to <a href="https://forums.fast.ai/t/device-side-assert-triggered-thctensorcopy/13818/2" rel="noopener nofollow ugc">this</a></p>
<p>so now the error is</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-18-43362cbf6c2c&gt; in &lt;module&gt;
      1 os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
      2 
----&gt; 3 history = train_all()
      4 history

&lt;ipython-input-17-3bc06434873b&gt; in train_all(model)
     40             outputs=model(images.float())
     41             loss = custom_loss(outputs,labels)
---&gt; 42             print(loss)
     43 
     44             loss.backward()

~/anaconda3/lib/python3.7/site-packages/torch/tensor.py in __repr__(self)
    177             return handle_torch_function(Tensor.__repr__, relevant_args, self)
    178         # All strings are unicode in Python 3.
--&gt; 179         return torch._tensor_str._str(self)
    180 
    181     def backward(self, gradient=None, retain_graph=None, create_graph=False):

~/anaconda3/lib/python3.7/site-packages/torch/_tensor_str.py in _str(self)
    370 def _str(self):
    371     with torch.no_grad():
--&gt; 372         return _str_intern(self)

~/anaconda3/lib/python3.7/site-packages/torch/_tensor_str.py in _str_intern(self)
    350                     tensor_str = _tensor_str(self.to_dense(), indent)
    351                 else:
--&gt; 352                     tensor_str = _tensor_str(self, indent)
    353 
    354     if self.layout != torch.strided:

~/anaconda3/lib/python3.7/site-packages/torch/_tensor_str.py in _tensor_str(self, indent)
    239         return _tensor_str_with_formatter(self, indent, summarize, real_formatter, imag_formatter)
    240     else:
--&gt; 241         formatter = _Formatter(get_summarized_data(self) if summarize else self)
    242         return _tensor_str_with_formatter(self, indent, summarize, formatter)
    243 

~/anaconda3/lib/python3.7/site-packages/torch/_tensor_str.py in __init__(self, tensor)
     87 
     88         else:
---&gt; 89             nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))
     90 
     91             if nonzero_finite_vals.numel() == 0:

RuntimeError: CUDA error: device-side assert triggered
</code></pre>

Rajan_Lagah: <aside class="quote no-group" data-username="Rajan_Lagah" data-post="1" data-topic="106505">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rajan_lagah/48/12519_2.png" class="avatar"> Rajan_Lagah:</div>
<blockquote>
<p><code>target.unsqueeze(1)</code></p>
</blockquote>
</aside>
<p>As you said.<br>
I log value of each variable and found that <code>target.unsqueeze(1)</code> is giving value more than valid index.<br>
I then verify output layer of my model was wrong.<br>
Thankyou</p>