Target out of bound error not raised on GPU

krishneel: <p>The following code snippet from the documentation except I changed the target to be out of bound.</p>
<pre><code class="lang-auto">loss = nn.CrossEntropyLoss().to(device)
input = torch.randn(3, 2, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(15)
output = loss(input, target)
</code></pre>
<p>On <code>device = cpu</code> it raises the correct error</p>
<pre><code class="lang-auto"># example
IndexError: Target 2 is out of bounds.
</code></pre>
<p>However, the same code snippet (with target out of bound) does not raise the same error on <code>device = cuda:0</code> but returns a <code>loss = 0</code></p>
<p>I couldn’t figure out why the behavior is as such or did I missed out something.</p>
<pre><code class="lang-auto">PyTorch version = 1.5.0
</code></pre>

futscdav: <p>Triggers the expected assertion error for me with torch 1.3.</p>
<p>Technically, you are changing the target to be random, so you have 0.2% chance of generating a valid target…</p>

krishneel: <blockquote>
<p>Technically, you are changing the target to be random, so you have 0.2% chance of generating a valid target…</p>
</blockquote>
<p>I did mention in the question that I checked that the target was out of bound.</p>

ptrblck: <p>It triggers this error on my system:</p>
<pre><code class="lang-python">device = 'cuda'
loss = nn.CrossEntropyLoss().to(device)
input = torch.randn(3, 2, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(15)
output = loss(input, target)
&gt; IndexError: Target 13 is out of bounds.
</code></pre>

krishneel: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> actually I forgot to mention I was using the <a href="https://hub.docker.com/r/pytorch/pytorch/" rel="nofollow noopener">PyTorch Docker Image</a> when I get the behavior mentioned in the question.</p>
<p>I just tested on a host machine and rightly so it does raise an error.<br>
Seems the error happens on Docker image.</p>

130: <p>Hi,</p>
<p>I got the same unexpected behaviour with PyTorch 1.5.0 using a GPU</p>
<pre><code class="lang-auto">loss = torch.nn.CrossEntropyLoss()
weights = torch.randn(10, 5)                                                                                                                         
labels = torch.arange(10)
loss(weights.cuda(), labels.cuda())
Out[13]: tensor(1.7329, device='cuda:0')
</code></pre>
<p>The error is triggered if I use the CPU though.</p>

ptrblck: <p>Could you update to the nightly binaries and recheck it, please?<br>
Recently an issue was fixed, which silenced the <code>assert</code> statements in the CUDA code.</p>

130: <p>Hi,</p>
<p>Sorry I took some time to answer !!<br>
I got this error now, which seems to be appropriate</p>
<pre><code class="lang-auto">/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [5,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [6,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [7,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [8,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [9,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/leo/Venv/test/lib/python3.7/site-packages/torch/tensor.py", line 154, in __repr__
    return torch._tensor_str._str(self)
  File "/home/leo/Venv/test/lib/python3.7/site-packages/torch/_tensor_str.py", line 333, in _str
    tensor_str = _tensor_str(self, indent)
  File "/home/leo/Venv/test/lib/python3.7/site-packages/torch/_tensor_str.py", line 229, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/home/leo/Venv/test/lib/python3.7/site-packages/torch/_tensor_str.py", line 101, in __init__
    nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))
RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered

</code></pre>
<p>Thanks!</p>