About evaluating (and understanding?) the output of CrossEntropyLoss

Smth_w: <p>So while working on a Language model I was testing some individual values with nn.CrossEntropyLoss</p>
<pre><code class="lang-auto">toyloss = nn.CrossEntropyLoss(reduction='none')

toyinput = torch.zeros(1, 5)
toytarget = torch.zeros(1, dtype=torch.long)

toytarget[0] = 2
toyinput[0,2] = 1
toyoutput = toyloss(toyinput, toytarget)

print(toyoutput)
</code></pre>
<p>In short the input and targets are:</p>
<blockquote>
<p>tensor([[0., 0., 1., 0., 0.]], tensor([2])</p>
</blockquote>
<p>respectively, and the loss is:</p>
<blockquote>
<p>tensor([0.9048])</p>
</blockquote>
<p>My question is why is the loss 0.9048 if the prediction completely matches the target?<br>
My guess is that it has something to do with the fact that this Loss implements logSoftmax, but then, what should we the correct interpretation for this result?<br>
Is 0.9048 the minimum loss possible when using this criterion?</p>
<p>Edit: I decided to try using only NLLLoss and I’ve got a loss of</p>
<blockquote>
<p>tensor([-1.])</p>
</blockquote>
<p>Honestly, I was expecting it to be 0… Am I missing something about the definition of Loss? I mean aren’t we looking for the closest value to 0?</p>

KFrank: <p>Hi Daniel!</p>
<aside class="quote no-group quote-modified" data-username="Smth_w" data-post="1" data-topic="73163" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c67d28/48.png" class="avatar"> Smth_w:</div>
<blockquote>
<p>In short the input and targets are:</p>
<blockquote>
<p>tensor([[0., 0., 1., 0., 0.]], tensor([2])</p>
</blockquote>
<p>respectively, and the loss is:</p>
<blockquote>
<p>tensor([0.9048])</p>
</blockquote>
<p>My question is why is the loss 0.9048 if the prediction completely matches the target?</p>
</blockquote>
</aside>
<p>The short answer is that your prediction doesn’t completely match<br>
your target.</p>
<aside class="quote no-group">
<blockquote>
<p>My guess is that it has something to do with the fact that this Loss implements logSoftmax, but then, what should we the correct interpretation for this result?</p>
</blockquote>
</aside>
<p>Yes, this is the key to the explanation of what’s going on.</p>
<p>Your prediction – the input to <code>CrossEntropyLoss</code> needs to be<br>
understood as a set of <em>logits</em>* that get turned into probabilities<br>
by <code>CrossEntropyLoss</code>’s implicit <code>softmax()</code>.</p>
<pre><code class="lang-plaintext">softmax ([0, 0, 1, 0, 0]) = [0.1488, 0.1488, 0.4046, 0.1488, 0.1488]
</code></pre>
<p>So, although your prediction for “class 2” is the largest, favored<br>
prediction, you are, nonetheless, only predicting “class 2” with about<br>
40% probability.  So, you’re less than half right and get a non-zero loss.</p>
<p>Note that <code>-log (0.4046) = 0.9048</code>, correctly reproducing the value<br>
you report for your loss.</p>
<aside class="quote no-group">
<blockquote>
<p>Is 0.9048 the minimum loss possible when using this criterion?</p>
</blockquote>
</aside>
<p>No, the minimum loss is zero.  But to get it, you need to predict<br>
“class 2” with a <em>probability</em> of 1, not a logit of 1.  To get a probability<br>
of 1, you need a logit of <code>+inf</code> (and a logit of <code>-inf</code> to get a probability<br>
of 0).  Because of the <code>exp()</code> in <code>softmax()</code>, <code>1000</code> is effectively <code>inf</code>.</p>
<p>So, if you use <code>[-1000, -1000, 1000, -1000, -1000]</code> as your<br>
prediction, you will, in fact, be predicting “class 2” with (essentially)<br>
100% probability, and you will get 0 for your loss.</p>
<aside class="quote no-group">
<blockquote>
<p>Edit: I decided to try using only NLLLoss and I’ve got a loss of</p>
<blockquote>
<p>tensor([-1.])</p>
</blockquote>
<p>Honestly, I was expecting it to be 0…</p>
</blockquote>
</aside>
<p>This is because <code>NLLLoss</code> expects <em>log-probabilities</em> for the predictions<br>
you give it (rather than probabilities).  Pass into <code>NLLLoss</code> the prediction</p>
<pre><code class="lang-plaintext">log ([0, 0, 1, 0, 0]) = [-inf, -inf, 0, -inf, -inf]
</code></pre>
<p>and you will get your expected loss of 0.</p>
<aside class="quote no-group">
<blockquote>
<p>Am I missing something about the definition of Loss?</p>
</blockquote>
</aside>
<p>Not really.</p>
<aside class="quote no-group">
<blockquote>
<p>I mean aren’t we looking for the closest value to 0?</p>
</blockquote>
</aside>
<p>Not exactly.  The pytorch optimizers are “looking for” the algebraically<br>
smallest (least positive, most negative) loss, rather than the loss closest<br>
to zero.  (The optimizers don’t really care where zero is.)  So training on<br>
<code>new_loss = old_loss - 10,000</code> will give you the same result<br>
as training on <code>old_loss</code>.</p>
<p>*)  To be completely precise the prediction is a set of logits ± some<br>
arbitrary shift that gets washed away when you pass the shifted logits<br>
though <code>softmax()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>