Multi-class cross entropy loss and softmax in pytorch

gary: <aside class="quote" data-post="5" data-topic="19077">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/multi-class-cross-entropy-loss-function-implementation-in-pytorch/19077/5">Multi-Class Cross Entropy Loss function implementation in PyTorch</a> 
  </div>
  <blockquote>
    You could try the following code: 
batch_size = 4
-torch.mean(torch.sum(labels.view(batch_size, -1) * torch.log(preds.view(batch_size, -1)), dim=1))
  </blockquote>
</aside>
<p>
In this topic ,ptrblck said that a  F.softmax function at dim=1 should be added before the nn.CrossEntropyLoss().<br>
In the document(<a href="https://pytorch.org/docs/stable/nn.html?highlight=crossentropy#torch.nn.CrossEntropyLoss" class="inline-onebox" rel="noopener nofollow ugc">torch.nn — PyTorch 2.1 documentation</a>) , it return nll_loss(log_softmax(input, 1) which return negative log and softmax.<br>
My question is that  should I calculated the softmax at dim=1 before the function nn.CrossEntropyLoss  which already have a softmax at dim=1 ?</p>

ptrblck: <p>No, <code>F.softmax</code> should not be added before <code>nn.CrossEntropyLoss</code>.<br>
I’ll take a look at the thread and edit the answer if possible, as this might be a careless mistake!<br>
Thanks for pointing this out.</p>
<p>EDIT: Indeed the example code had a <code>F.softmax</code> applied on the logits, although not explicitly mentioned.<br>
To sum it up: <code>nn.CrossEntropyLoss</code> applies <code>F.log_softmax</code> and <code>nn.NLLLoss</code> internally on your input, so you should pass the raw logits to it.</p>

Brando_Miranda: <p>What loss function are we supposed to use when we use the <code>F.softmax</code> layer?</p>

KFrank: <p>Hi Brando!</p>
<aside class="quote no-group" data-username="Brando_Miranda" data-post="3" data-topic="24920" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>What loss function are we supposed to use when we use the <code>F.softmax</code> layer?</p>
</blockquote>
</aside>
<p>If you want to use a cross-entropy-like loss function, you shouldn’t<br>
use a softmax layer because of the well-known problem of increased<br>
risk of overflow.</p>
<p>I gave a few words of explanation about this problem in a reply in<br>
another thread:</p>
<aside class="quote quote-modified" data-post="2" data-topic="49861">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/softmax-regression-from-scratch-nan-loss/49861/2">Softmax regression from scratch: NaN loss</a> 
  </div>
  <blockquote>
    Hi Lorenzo! 

I didn’t look at your code, but if you wrote your softmax and 
cross-entropy functions as two separate functions you are 
probably tripping over the following problem. 
Softmax contains exp() and cross-entropy contains log(), 
so this can happen: 
large number --&gt; exp() --&gt; overflow NaN --&gt; log() --&gt; still NaN 
even though, mathematically (i.e., without overflow), 
log (exp (large number)) = large number (no NaN). 
Pytorch’s CrossEntropyLoss (for example) uses standard 
techniques …
  </blockquote>
</aside>

<p>You should either use <code>nn.CrossEntropyLoss</code> (which takes<br>
pre-softmax logits, rather than post-softmax probabilities)<br>
without a softmax-like layer, or use a <code>nn.LogSoftmax</code> layer,<br>
and feed the results into <code>nn.NLLLoss</code>.  (Both of these combine<br>
an implicit softmax with the subsequent log in a way that avoids<br>
the enhanced overflow problem.)</p>
<p>If you are stuck for some reason with your softmax layer, you<br>
should run the probabilities output by softmax through <code>log()</code>,<br>
and then feed the log-probabilities to <code>nn.NLLLoss</code> (but expect<br>
increased risk of overflow).</p>
<p>(I am not aware of any single pytorch cross-entropy loss function<br>
that takes post-softmax probabilities directly.)</p>
<p>Good luck!</p>
<p>K. Frank</p>

Oormila_Ghantasala: <p>Hi, if softmax is not to be used, how do we get the output as probabilities for a multi-class classification problem? I have explained my problem <a href="https://discuss.pytorch.org/t/softmax-gives-output-vector-whose-sum-is-greater-than-1/60935">here</a>. Please take a look at my code and help me out since I am a beginner.</p>

SpeedOfSpin: <p>You can just apply it to your output as normal. So<br>
model.eval()<br>
output = net(input)<br>
sm = torch.nn.Softmax()<br>
probabilities = sm(output)<br>
print(probabilities )</p>

Lincoln10153: <p>Hello,</p>
<p>I have a question about Softmax() and CrossEntropyLoss().</p>
<p>In a multi-classification task, I set dim=1 in Softmax(). <strong>I wanna know if I need to set the similar parameter in CrossEntropyLoss().</strong> However, I did not find the similar parameter as dim in softmax().</p>
<p>Thanks!</p>

Lincoln10153: <p>Hello,</p>
<p>I am new to PyTorch, and I encountered a quesiton about Softmax() and CrossEntropyLoss().</p>
<p>In a multi-classification task, I set dim=1 in Softmax(). <strong>I wanna know if I need to set the similar parameter in CrossEntropyLoss().</strong> However, I did not find the similar parameter as dim in softmax().</p>
<p>Thanks!</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects raw logits in the shape <code>[batch_size, nb_classes, *]</code> so you should not apply a <code>softmax</code> activation on the model output. The class dimension should be in <code>dim1</code> in the model output.</p>

learner47: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,  suppose I have the output of a neural network to have shape <code>[1000, 100, 4]</code>. I have applied <code>nn.Softmax()</code> on axis <code>2</code> and then have taken <code>nn.BCELoss</code> with a target which has same shape and for each row and column index of the target I have a 1d vector (4 length) containing exactly one <code>1</code> (it is a one hot vector). Does this setup work or is there a flaw?</p>

ptrblck: <p><code>nn.BCELoss</code> can be applied with <code>torch.sigmoid</code> for a multi-label classification. Since you are using <code>softmax</code>, I assume you are working on a multi-class classification, and should probably stick to <code>nn.CrossEntropyLoss</code>. For this criterion, your shapes also seem to be wrong as described in my previous post.</p>

learner47: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for your response. My confusion roots from the fact that Tensorflow allow us to use <code>softmax</code> in conjunction with <code>BCE</code> loss. Yes, I have 4-class classification problem. I have 1000 batch size and 100 sequence length. And the last dimension corresponds to the multi-class probability. If I use <code>sigmoid</code> I need it only on the third dimension. <code>nn.CrossEntropy</code> won’t be applicable as the dimensions are not right. How should I proceed in this case?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> can be applied if you permute the output to match the expected shapes:</p><aside class="quote" data-post="9" data-topic="24920">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/multi-class-cross-entropy-loss-and-softmax-in-pytorch/24920/9">Multi-class cross entropy loss and softmax in pytorch</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    nn.CrossEntropyLoss expects raw logits in the shape [batch_size, nb_classes, *] so you should not apply a softmax activation on the model output. The class dimension should be in dim1 in the model output.
  </blockquote>
</aside>

<p>I don’t know, how TF is applying a binary cross-entropy with a softmax activation function, as I assume internally <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss">this formula</a> would be used, which involves the sigmoid.</p>