classification problem in pytorch with loss function CrossEntropyLoss returns negative output in prediction

jzhu354: <p>I am trying to train and predict SVHN dataset (VGG architecture). I get very high validate/test accuracy by just getting the largest output class. However, the output weights are of large positive and negative numbers. Are they supposed to parsed as exp(output)/sum(exp(output)) to be converted to probability? Thank you!</p>

ptrblck: <p>If you are using <code>nn.CrossEntropyLoss</code>, you should directly pass the logits to this loss function, since internally <code>nn.NLLLoss</code> and <code>F.log_softmax</code> will be used.<br>
Your proposed softmax function should not be used for one of these loss functions, but might of course be used for debugging purposes etc. to see the probabilities.</p>

wmpauli: <p>I’m also getting negative <code>nn.CrossEntropyLoss</code> (every couple of epochs). I am evaluating the loss on the output of a Resnet model (i.e. last layer is <code>nn.Linear</code>). What other reasons are there be for negative loss?</p>

wmpauli: <p>It looks like in my case the issue was a corruption of the dataset, so that for some samples the target category was -1. Interestingly, when I try to replicate calling <code>nn.CrossEntropyLoss</code> directly on tensors on the CPU, I get an error message, if one of the targets is -1. When I run this on the GPU, I appear to get negative loss, rather than an error message.</p>

KFrank: <p>Hello Muster!</p>
<aside class="quote no-group" data-username="wmpauli" data-post="4" data-topic="43161" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/8c91f0/48.png" class="avatar"> wmpauli:</div>
<blockquote>
<p>for some samples the target category was -1. Interestingly, when I try to replicate calling <code>nn.CrossEntropyLoss</code> directly on tensors on the CPU, I get an error message, if one of the targets is -1. When I run this on the GPU, I appear to get negative loss, rather than an error message.</p>
</blockquote>
</aside>
<p>I can reproduce this on pytorch 0.3.0 (except I don’t get a negative<br>
loss).</p>
<p>If I have a target class label that is out of range, I get an error on the<br>
cpu, but not on the gpu.</p>
<p>I’m a little surprised that the gpu lets this error pass, although perhaps<br>
it forgoes checking for it in the interest of efficiency.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Do you think this is an issue or by design?</p>
<p>Here is a test script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

predc = torch.autograd.Variable (torch.FloatTensor ([[0.5, -0.1, -0.2],[-0.1, -0.1, 0.4]]))
targc = torch.autograd.Variable (torch.LongTensor ([0, 2]))

predg = predc.cuda()
targg = targc.cuda()

torch.nn.functional.cross_entropy (predg, targg)
torch.nn.functional.cross_entropy (predc, targc)

targg[0] = -1
targc[0] = -1

torch.nn.functional.cross_entropy (predg, targg)
torch.nn.functional.cross_entropy (predc, targc)
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; predc = torch.autograd.Variable (torch.FloatTensor ([[0.5, -0.1, -0.2],[-0.1, -0.1, 0.4]]))
&gt;&gt;&gt; targc = torch.autograd.Variable (torch.LongTensor ([0, 2]))
&gt;&gt;&gt;
&gt;&gt;&gt; predg = predc.cuda()
&gt;&gt;&gt; targg = targc.cuda()
&gt;&gt;&gt;
&gt;&gt;&gt; torch.nn.functional.cross_entropy (predg, targg)
Variable containing:
 0.7550
[torch.cuda.FloatTensor of size 1 (GPU 0)]

&gt;&gt;&gt; torch.nn.functional.cross_entropy (predc, targc)
Variable containing:
 0.7550
[torch.FloatTensor of size 1]

&gt;&gt;&gt;
&gt;&gt;&gt; targg[0] = -1
&gt;&gt;&gt; targc[0] = -1
&gt;&gt;&gt;
&gt;&gt;&gt; torch.nn.functional.cross_entropy (predg, targg)
Variable containing:
 0.3972
[torch.cuda.FloatTensor of size 1 (GPU 0)]

&gt;&gt;&gt; torch.nn.functional.cross_entropy (predc, targc)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\Users\LisaBrown\Documents\admin\programs\Miniconda3\lib\site-packages\torch\nn\functional.py", line 1140, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, size_average, ignore_index, reduce)
  File "C:\Users\LisaBrown\Documents\admin\programs\Miniconda3\lib\site-packages\torch\nn\functional.py", line 1049, in nll_loss
    return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
RuntimeError: Assertion `cur_target &gt;= 0 &amp;&amp; cur_target &lt; n_classes' failed.  at d:\pytorch\pytorch\torch\lib\thnn\generic/ClassNLLCriterion.c:87
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <p>This shouldn’t be by design and I get a proper error message in the latest nightly binary:</p>
<pre><code class="lang-python">ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target -1 is out of bounds.
</code></pre>
<p>using your code.</p>
<p>We had a bug in the <code>1.5.0</code> release (which should be fixed in <code>1.5.1</code>), which ignored certain assert statements in CUDA code and thus didn’t raise any proper exceptions.<br>
In other words: if you were passing a target with <code>&gt;=nb_classes</code>, you would most likely get an illegal memory access error instead of a proper error message.<br>
However, passing an index of <code>-1</code> would probably just use the “reverse indexing” and while this is technically a valid operation, the result is of course wrong and this shouldn’t work.</p>
<p><code>0.3.0</code> on the other hand might have had the same issue and I don’t know when the assert statements were added to <code>nll_loss</code>.</p>
<p><a class="mention" href="/u/wmpauli">@wmpauli</a> Could this be the case for your issue? If you are using <code>1.5.0</code>, I hightly recommend to update to <code>1.5.1</code> or the nightly binary.</p>

wmpauli: <p>Thanks for your responses. I am indeed using:</p>
<pre><code>- torch==1.5.0
- torchvision==0.6.0
</code></pre>
<p>Glad I had a sadly small batch size so that I caught these negative values. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>