Need Help - Pytorch Softmax + Cross Entropy Loss function

Ramaiah_Radhakrishna: <p>Hello</p>
<p>I am taking a course on deep learning - As part of the course work I have to build a project on CNN Classification  - In my project I used SoftMax activation for the output layer as I am interested in the probabilities instead of scores. I also used Cross Entropy loss function as it worked better for my problem.  I got the expected results -   I am already aware the Cross Entropy loss function uses the combination of pytorch log_softmax &amp;  NLLLoss behind the scene.</p>
<p>But my project submission was rejected and the reviewer comment was that Softmax activation <em>should not</em> be used with Cross Entropy Loss Function per Pytorch documentation.</p>
<p>Seeking help from experts here to understand why Softmax and Cross Entropy Loss function should not used together in Pytorch.</p>
<p>Much appreciate your help</p>

ptrblck: <aside class="quote no-group" data-username="Ramaiah_Radhakrishna" data-post="1" data-topic="146243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ramaiah_radhakrishna/48/47482_2.png" class="avatar"> Ramaiah_Radhakrishna:</div>
<blockquote>
<p>the reviewer comment was that Softmax activation <em>should not</em> be used with Cross Entropy Loss Function per Pytorch documentation.</p>
</blockquote>
</aside>
<p>That’s correct and you’ve already described the reason as:</p>
<aside class="quote no-group" data-username="Ramaiah_Radhakrishna" data-post="1" data-topic="146243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ramaiah_radhakrishna/48/47482_2.png" class="avatar"> Ramaiah_Radhakrishna:</div>
<blockquote>
<p>I am already aware the Cross Entropy loss function uses the combination of pytorch log_softmax &amp; NLLLoss behind the scene.</p>
</blockquote>
</aside>
<p>If you apply a <code>softmax</code> on your output, the loss calculation would use:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = F.nll_loss(F.log_softmax(F.softmax(logits)), target)
</code></pre>
<p>which is wrong based on the formula for the cross entropy loss due to the additional <code>F.softmax</code>.</p>

Ramaiah_Radhakrishna: <p>Thank you -  If I need the probabilities as the model outcome during inference time, what should I do?  I tried softmax function outside the model architecture but  at the time of model inference but I lose the TopK function. If I need probablities as well as the functions like topk, what should I do?</p>
<p>Thanks<br>
Ramaiah</p>

ptrblck: <p>Applying <code>softmax</code> to the output “outside the model” and not passing it to the loss function would be the proper way top get the probabilities.</p>
<aside class="quote no-group" data-username="Ramaiah_Radhakrishna" data-post="3" data-topic="146243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ramaiah_radhakrishna/48/47482_2.png" class="avatar"> Ramaiah_Radhakrishna:</div>
<blockquote>
<p>but I lose the TopK function. If I need probablities as well as the functions like topk, what should I do?</p>
</blockquote>
</aside>
<p>I don’t fully understand this description, as <code>softmax</code> won’t change the order:</p>
<pre data-code-wrap="python"><code class="lang-python">logits = torch.randn(10, 10)
preds = torch.topk(logits, k=3, dim=1).indices

prob = F.softmax(logits, dim=1)
preds_prob = torch.topk(prob, k=3, dim=1).indices

print((preds==preds_prob).all())
# &gt; tensor(True)
</code></pre>

Ramaiah_Radhakrishna: <p>Thanks lot - It helps</p>