Using log-softmax with cross entropy

J_B_28: <p>Hello, I am doing some tests using different loss function, usually we use log-softmax + nll loss or just cross-entropy loss with original output, but I found log-softmax + cross-entropy sometimes provides better results, I know this combination is not correct, because it actually has two times log scale computation, and for backward it may have some problems, but for some datasets, whatever the learning rate I used for  log-softmax + nll loss, it’s still worse than log-softmax + cross-entropy, I am wondering the reason or the potential problem for this result, and also wondering if I can use log-softmax + cross-entropy? Thanks.</p>

KFrank: <p>Hi J_B!</p>
<aside class="quote no-group" data-username="J_B_28" data-post="1" data-topic="168095" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_b_28/48/46078_2.png" class="avatar"> J_B_28:</div>
<blockquote>
<p>log-softmax + nll loss, it’s still worse than log-softmax + cross-entropy</p>
</blockquote>
</aside>
<p>This doesn’t make sense, as the two are the same.</p>
<p>Roughly speaking, <code>log()</code> undoes <code>softmax()</code>, so two <code>log_softmax()</code>s in<br>
a row are the same as just one:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0'
&gt;&gt;&gt; t = torch.arange (10.0)
&gt;&gt;&gt; t
tensor([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
&gt;&gt;&gt; t.log_softmax (0)
tensor([-9.4586, -8.4586, -7.4586, -6.4586, -5.4586, -4.4586, -3.4586, -2.4586,
        -1.4586, -0.4586])
&gt;&gt;&gt; t.log_softmax (0).log_softmax (0)
tensor([-9.4586, -8.4586, -7.4586, -6.4586, -5.4586, -4.4586, -3.4586, -2.4586,
        -1.4586, -0.4586])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

J_B_28: <p>Hi K. Frank,</p>
<p>Thanks for your reply. I also made the same test as yours, and I got the same results. But I am not sure about the backward, since log-softmax + cross-entropy actually does two times log-softmax, is the gradient still same as log-softmax + nll? Thanks again for your help.</p>

KFrank: <p>Hi J_B!</p>
<aside class="quote no-group" data-username="J_B_28" data-post="3" data-topic="168095" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_b_28/48/46078_2.png" class="avatar"> J_B_28:</div>
<blockquote>
<p>since log-softmax + cross-entropy actually does two times log-softmax, is the gradient still same as log-softmax + nll?</p>
</blockquote>
</aside>
<p>Yes, it’s the same function in the forward pass so you’ll get the same<br>
gradient from the backward pass (up to possible floating-point round-off<br>
error).</p>
<p>But it’s easy enough to test this sort of thing:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; lin = torch.nn.Linear (3, 5)
&gt;&gt;&gt; input = torch.randn (7, 3)
&gt;&gt;&gt; target = torch.randint (5, (7,))
&gt;&gt;&gt;
&gt;&gt;&gt; # log_softmax() + nll_loss()
&gt;&gt;&gt; pred = lin (input)
&gt;&gt;&gt; lossA = torch.nn.functional.nll_loss (pred.log_softmax (1), target)
&gt;&gt;&gt; lossA
tensor(1.9197, grad_fn=&lt;NllLossBackward0&gt;)
&gt;&gt;&gt; lossA.backward()
&gt;&gt;&gt; lin.weight.grad
tensor([[-0.0151,  0.2068, -0.0595],
        [-0.0683, -0.3701, -0.2444],
        [ 0.0995, -0.0853,  0.1083],
        [ 0.0341,  0.2123,  0.0165],
        [-0.0503,  0.0362,  0.1791]])
&gt;&gt;&gt; lin.bias.grad
tensor([ 0.0741, -0.1813,  0.1026,  0.0301, -0.0255])
&gt;&gt;&gt;
&gt;&gt;&gt; lin.zero_grad()
&gt;&gt;&gt;
&gt;&gt;&gt; # log_softmax() + cross_entropy()
&gt;&gt;&gt; pred = lin (input)
&gt;&gt;&gt; lossB = torch.nn.functional.cross_entropy (pred.log_softmax (1), target)
&gt;&gt;&gt; lossB
tensor(1.9197, grad_fn=&lt;NllLossBackward0&gt;)
&gt;&gt;&gt; lossB.backward()
&gt;&gt;&gt; lin.weight.grad
tensor([[-0.0151,  0.2068, -0.0595],
        [-0.0683, -0.3701, -0.2444],
        [ 0.0995, -0.0853,  0.1083],
        [ 0.0341,  0.2123,  0.0165],
        [-0.0503,  0.0362,  0.1791]])
&gt;&gt;&gt; lin.bias.grad
tensor([ 0.0741, -0.1813,  0.1026,  0.0301, -0.0255])
&gt;&gt;&gt;
&gt;&gt;&gt; lin.zero_grad()
&gt;&gt;&gt;
&gt;&gt;&gt; # plain cross_entropy()
&gt;&gt;&gt; pred = lin (input)
&gt;&gt;&gt; lossC = torch.nn.functional.cross_entropy (pred, target)
&gt;&gt;&gt; lossC
tensor(1.9197, grad_fn=&lt;NllLossBackward0&gt;)
&gt;&gt;&gt; lossC.backward()
&gt;&gt;&gt; lin.weight.grad
tensor([[-0.0151,  0.2068, -0.0595],
        [-0.0683, -0.3701, -0.2444],
        [ 0.0995, -0.0853,  0.1083],
        [ 0.0341,  0.2123,  0.0165],
        [-0.0503,  0.0362,  0.1791]])
&gt;&gt;&gt; lin.bias.grad
tensor([ 0.0741, -0.1813,  0.1026,  0.0301, -0.0255])
</code></pre>
<p>If you do seem to be getting different results (other than possible round-off<br>
error) with your two methods, then you’ve got some sort of bug hiding<br>
somewhere that you should track down.</p>
<p>Best.</p>
<p>K. Frank</p>

J_B_28: <p>Hi K. Frank,</p>
<p>Really appreciate your reply. so does this mean if the model output first go through log-softmax, then the loss function can be nll or cross entropy, it doesn’t matter, they are actually same(if the input is the log-softmax), am I right? Thanks again.</p>