Mysterious behaviour of cross entropy loss

Pavan_Teja: <p>Hello,<br>
I  have encountered a very weird behaviour of <a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#cross_entropy" rel="noopener nofollow ugc">nn.functional.cross_entropy(pred, label, reduction=‘none’)</a> function. Following is the behaviour I observed. It would be great if someone could explain the reason for this:</p>
<p><strong>Expected behaviour</strong> - If for any i  label[i] &gt; C where pred is of shape N x C x H x W , cross_entropy should raises an IndexError.</p>
<p><strong>Showcased behaviour</strong> -</p>
<ul>
<li>If pred and target are <strong>cpu</strong> tensors. Output : <strong>IndexError</strong>: Target 20 is out of bounds.</li>
<li>If pred and target are <strong>gpu(cuda)</strong> tensors then <strong>no error</strong> is raised.</li>
</ul>
<p>Why does cross_entropy shows a discrimination in its behaviour for gpu and cpu tensors? Also the above observed behaviour for cuda tensors is unexpected and isn’t this a bug ?</p>
<p>My environment:<br>
Python : 3.7.7<br>
Pytorch : 1.6.0<br>
Cuda : 10.1</p>
<p>PS: The above behaviour almost consumed 3 hrs of my time. Hope someone would help me understand the reason for this so that I will be cautious next time I deal with gpu tensors. <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>Below is the code snippet to replicate:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
input = torch.randn(1, 19, 32, 64, requires_grad=True)
input_c = input.cuda()
target = torch.randint(22, size=(1,32,64,))
target_c = target.cuda()
F.cross_entropy(input_c, target_c, weight=None, reduction='none', ignore_index=255)
</code></pre>
<p>This code should not raise any error. Where as removing the cuda() would raise an error.</p>

InnovArul: <p>I could reproduce the issue. I think you have stumbled upon a pytorch bug in <code>nll_loss2d</code>.<br>
I had played around with your example and I could notice that <code>reduction='none'</code> could be the reason. Error is triggered for other <code>reduction</code> (<code>mean</code> / <code>sum</code>) option.</p>
<p>Further, the reason could be that, the device assertion is not checked in case of <code>reduction='none'</code>.</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu#L70-L99" target="_blank" rel="noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu#L70-L99" target="_blank" rel="noopener">pytorch/pytorch/blob/master/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu#L70-L99</a></h4>
<pre class="onebox"><code class="lang-cu"><ol class="start lines" start="70" style="counter-reset: li-counter 69 ;">
<li>if (reduction == at::Reduction::None) {</li><li>  int64_t batch_size = THCTensor_(size)(state, input, 0);</li><li>  int64_t H = THCTensor_(size)(state, input, 2);</li><li>  int64_t W = THCTensor_(size)(state, input, 3);</li><li>  int64_t count = batch_size * H * W;</li><li></li><li>  THCTensor_(resize3d)(state, output, batch_size, H, W);</li><li></li><li>  if (count == 0) {</li><li>    // This guards from unnecessary operations and launching CUDA kernel with 0 blocks.</li><li>    return;</li><li>  }</li><li>  if (weights) {</li><li>    weights = THCTensor_(newContiguous)(state, weights);</li><li>  }</li><li></li><li>  SpatialClassNLLCriterion_updateOutput_no_reduce_kernel&lt;scalar_t&gt;</li><li>    &lt;&lt;&lt;GET_BLOCKS(count), CUDA_NUM_THREADS, 0, c10::cuda::getCurrentCUDAStream()&gt;&gt;&gt;(</li><li>      count,</li><li>      toDeviceTensor&lt;scalar_t, 4&gt;(state, input),</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/THCUNN/generic/SpatialClassNLLCriterion.cu#L70-L99" target="_blank" rel="noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p><code>THCudaCheck(cudaGetLastError());</code> is missing incase of <code>reduction='none'</code>.</p>
<p>I have raised a bug on behalf of you: <a href="https://github.com/pytorch/pytorch/issues/49882" class="inline-onebox">nll_loss2d: t &gt;= 0 &amp;&amp; t &lt; n_classes assertion is not checked when using GPU tensors and reduction='none' · Issue #49882 · pytorch/pytorch · GitHub</a></p>

Pavan_Teja: <p>Thanks for raising the bug.</p>