LogSoftmax & NLLLoss - Why is loss not zero?

progers: <p>When using the LogSoftmax &amp; NLLLoss pair, why doesn’t a “one hot” input of the correct category produce a loss of zero? I suspect I’m missing something.</p>
<p>Variation of the example from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc">docs for NLLLoss</a>:</p>
<pre><code class="lang-auto">m = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
# input is of size N x C = 1 X 3
# Input is a perfectly matching on-hot for category 0
input = torch.tensor([[1, 0, 0]], dtype=torch.float)
# We want category 0, so we should be right on target
target = torch.tensor([0])
output = loss(m(input), target)
output
</code></pre>
<p>Result: <code>tensor(0.5514)</code></p>
<p>In <a href="https://discuss.pytorch.org/t/understanding-nllloss-function/23702/3">this post</a>, <a class="mention" href="/u/ptrblck">@ptrblck</a> noted:</p>
<blockquote>
<p><code>nn.NLLLoss</code> expects the inputs to be log probabilities</p>
</blockquote>
<p>Let’s use his trick to undo the log:</p>
<pre><code class="lang-auto">m(input).exp()
</code></pre>
<p>Result: <code>tensor([[0.5761, 0.2119, 0.2119]])</code></p>
<p>The above is exactly what we’d get if we apply <code>Softmax</code> (without <code>Log</code>) directly, which is good, but the above don’t seem to be probabilities, at least not those that give us a zero loss.</p>
<p>Let’s try log probabilities directly:</p>
<pre><code class="lang-auto">lp = torch.tensor([[1.0, 0, 0]]).log()
print(lp)
loss(lp, target)
</code></pre>
<p>Result is what we’d expect: a loss of zero:</p>
<pre><code class="lang-auto">tensor([[0., -inf, -inf]])
tensor(0.)
</code></pre>
<p>The above is a simplified version of the <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" rel="noopener nofollow ugc">MNIST example</a>.</p>
<p>The effect of this behavior is that we get a loss even for matches, which seems to cause weights to grow slowly without bound.</p>
<p>What am I doing wrong? Thanks!</p>

ptrblck: <p>As you’ve already explained, the <code>input</code> contains logits, not probabilities.<br>
The probabilities are shown in:</p>
<pre><code class="lang-python">m(input).exp()
&gt; tensor([[0.5761, 0.2119, 0.2119]])
</code></pre>
<p>which doesn’t give a <code>1</code> for the target class and zeros otherwise.</p>
<p>If you want to drive the loss towards zero, you would have to pass a large positive logit to <code>input[0]</code> as seen here:</p>
<pre><code class="lang-python">
for factor in [1, 5, 10, 100]:
    input = torch.zeros(1, 3)
    input[0, 0] = factor * 1.
    output = loss(m(input), target)
    print(F.softmax(input, dim=1))
    print(output)

&gt; tensor([[0.5761169195, 0.2119415700, 0.2119415700]])
tensor(0.5514446497)
tensor([[0.9867032766, 0.0066483542, 0.0066483542]])
tensor(0.0133859031)
tensor([[9.9990916252e-01, 4.5395805500e-05, 4.5395805500e-05]])
tensor(9.0833353170e-05)
tensor([[1.0000000000e+00, 3.7835058537e-44, 3.7835058537e-44]])
tensor(0.)
</code></pre>

progers: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the many answers you generously provide in these forums, especially for  us newbies.</p>
<p>I suspect I’m confusing the range of unit <em>outputs</em> (0-1) with the “raw” linear layer dot-products. Somewhere I read that biological neurons seldom saturate. So, I kind of expected the “raw” value to also be around 1 for a “match” (which it might be for old-school <a href="http://neuralnetworksanddeeplearning.com/chap1.html#learning_with_gradient_descent" rel="noopener nofollow ugc">squared-error loss</a> networks.)</p>
<p>Your explanation suggests that  <code>LogSoftmax</code> and <code>NLLLoss</code> networks work differently: they evidently drive the output linear layer to produce large <a href="https://en.wikipedia.org/wiki/Logit" rel="noopener nofollow ugc">logit</a>-like dot products. This would seem to require that output units learn weights which are large relative to the number of inputs. (E.g. 5 weights of 20 each, when presented with inputs of ~1, would produce a logit-like dot product of 100.) Is this roughly correct?</p>
<p>Have folks found that having extreme-value logit-like linear layers works better than more moderate (-1, 1) values? (Isn’t the one just a linear rescaling of the other?) Anyone know of literature on this?</p>
<p>And, if I did want to play with an output layer that learns weights that produce ~(-1, 1) “raw” values (before activation), which PyTorch activation &amp; cost functions are preferred? Back to sigmoid &amp; squared-error or is there something more modern? Softmax is nice because it produces probability-like values. Is there something that will produce ratios from ~(-1, 1) dot products? (I couldn’t find a “HardMax” module…)</p>

ptrblck: <aside class="quote no-group" data-username="progers" data-post="3" data-topic="112485">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e68b1a/48.png" class="avatar"> progers:</div>
<blockquote>
<p>Your explanation suggests that <code>LogSoftmax</code> and <code>NLLLoss</code> networks work differently: they evidently drive the output linear layer to produce large <a href="https://en.wikipedia.org/wiki/Logit">logit</a>-like dot products.</p>
</blockquote>
</aside>
<p>I don’t think it works differently, but removes the softmax from the model for better numerical stability.<br>
If you take a look at the applied loss function in <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss"><code>nn.CrossEntropyLoss</code></a>, you see that the <code>softmax</code> is calculated first and the <code>log</code> is applied on its result.<br>
While this works theoretically, you could run into numerical issues due to the limited floating point precision.<br>
Instead of directly using the mentioned function and adding the softmax layer to the model (which would then output “probabilities” in the range <code>[0, 1]</code>), you are instead passing the activations of the last linear layer, and apply <code>F.log_softmax</code>  in <code>nn.CrossEntropyLoss</code>.<br>
If you want to calculate the probabilities, you could of course apply <code>F.softmax</code> to the output and e.g. print it for debugging purposes. Don’t pass these <code>F.softmax</code> outputs to <code>nn.CrossEntropyLoss</code>, since internally <code>F.log_softmax</code> will be applied on them.</p>