CrossentropyLoss with Softmax?

username: hayridurmaz
cooked: <p>Hi there,</p>
<p>I am recently moved from keras to pytorch. I am trying to train a model for a classification problem. I am aiming to use transfer learning. I used Googlenet architecture and add custom layer below it. Also I am using CrossEntropyLoss() for criterion.</p>
<pre><code class="lang-auto">model = torchvision.models.googlenet(True)
    # Customizing fc layers of the model
model.fc = nn.Sequential(
      nn.Linear(1024, 2),
      nn.Softmax() #Question here!!
)
criterion = nn.CrossEntropyLoss()
</code></pre>
<p>As in the stated <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.CrossEntropyLoss.html?highlight=crossentropy#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">1.9.0 docs</a> CrossEntropyLoss  combines <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.LogSoftmax.html#torch.nn.LogSoftmax" rel="noopener nofollow ugc"><code>LogSoftmax</code></a> and <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc"><code>NLLLoss</code></a> in one single class. However, when it comes to the <a href="https://pytorch.org/docs/1.11/generated/torch.nn.CrossEntropyLoss.html?highlight=crossentropy#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">1.11 docs</a> this information is gone.</p>
<p>My question is that for such a 2-class classification problem, should I add softmax layer or not?</p>
<p>I tried both for sure, before writing this questions. When I add softmax explicitly, preferable results are shown, however training loss never converges to zero;</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9e08976a1313bf232413669af45b4ad1d31ca49.png" alt="loss" data-base62-sha1="xmYjLE2xircrNMbmRuOufnImi0h" width="640" height="480"></p>
<p>However, When I am not adding the softmax;</p>
<pre><code class="lang-auto">model = torchvision.models.googlenet(True)
    # Customizing fc layers of the model
model.fc = nn.Sequential(
      nn.Linear(1024, 2),
      # nn.Softmax() #Question here!!
)
criterion = nn.CrossEntropyLoss()
</code></pre>
<p>Training losses converges really quick (almost linearly), but validation loss starts to increase; (I am aware that second example includes only 10 epochs, but this information should be enough to compare.)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/265ca957cc5b80926c750912395fac41592426a9.png" alt="loss" data-base62-sha1="5tmFyjmGzXZCWrlmH3EUknQD3a9" width="640" height="480"></p>
<p>Model with the softmax Seems not much over-fitting and Not OK but at least better results.</p>
<p>There are several questions on the forum about transfer learning and no one added softmax layer, even the <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">official tutorial</a>. What is the correct way of transfer learning for classification? Should I add softmax? And When I add softmax layer, is it trained?</p>
<p>Thank you  in advance!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="hayridurmaz" data-post="1" data-topic="152986">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hayridurmaz/48/49408_2.png" class="avatar"> hayridurmaz:</div>
<blockquote>
<p>My question is that for such a 2-class classification problem, should I add softmax layer or not?</p>
</blockquote>
</aside>
<p>No, as <code>nn.CrossEntropyLoss</code> will still use <code>F.log_softmax</code> and <code>nn.NLLLoss</code> internally.</p>

username: hayridurmaz
cooked: <p>But how can you explain the results? Validation loss not even close. Or what do you suggest?</p>

username: ptrblck
cooked: <p>The additional and unneeded <code>softmax</code> will squeeze the logits to <code>[0, 1]</code> and thus also lower the gradient magnitude etc. so maybe your training could benefit from a lower learning rate?</p>

username: hayridurmaz
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="152986">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The additional and unneeded <code>softmax</code> will squeeze the logits to <code>[0, 1]</code> and thus also lower the gradient magnitude etc</p>
</blockquote>
</aside>
<p>Oh this explains my situation, thank you!</p>
<p>I am now trying to lower my learning rate. Can I ask you lasly, When I use “unneeded softmax”, why is it prevents training loss to be converged?</p>

username: ptrblck
cooked: <p>I would think for the same reason of lowering the output and thus also the gradient magnitudes, which might cause your model training to get stuck.<br>
Note that this is just pure speculation so maybe someone else has a more mathematical explanation.</p>

username: hayridurmaz
cooked: <p>Thank you! My problem is now evolved into overfitting I guess.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/309a7aa8a40921dd100584087e95b6e59d6cba74.png" alt="loss" data-base62-sha1="6VXT5N5JDwbYgvGduqSDXu6nrZq" width="640" height="480"></p>
<p>I think as you said</p>
<blockquote>
<p><code>nn.CrossEntropyLoss</code> will still use <code>F.log_softmax</code> and <code>nn.NLLLoss</code> internally</p>
</blockquote>
<p>Warning about</p>
<blockquote>
<p>CrossEntropyLoss combines <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.LogSoftmax.html#torch.nn.LogSoftmax" rel="noopener nofollow ugc"><code>LogSoftmax</code></a> and <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc"><code>NLLLoss</code></a> in one single class</p>
</blockquote>
<p>should be back to the documentation.</p>
<p>Thanks for help again!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="hayridurmaz" data-post="7" data-topic="152986">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hayridurmaz/48/49408_2.png" class="avatar"> hayridurmaz:</div>
<blockquote>
<p>should be back to the documentation.</p>
</blockquote>
</aside>
<p>It it still there, but now split into two cases.<br>
In newer PyTorch version you are able to use “soft” targets (i.e. probabilities instead of class indices) as the target tensor so the documentation was split into:</p>
<pre><code class="lang-auto">...
The target that this criterion expects should contain either:

1) Class indices in the range ...
Note that this case is equivalent to the combination of LogSoftmax and NLLLoss.

2) Probabilities for each class; 
...
</code></pre>
<p>and the first case still equals to the <code>LogSoftmax</code> and <code>NLLLoss</code> case.</p>