What classification loss should I choose when I have used a softmax function?

username: chunchun
cooked: <p>Just as the title, I must use the result of  softmax,then I want to use a loss.<br>
I found that NLLLoss must be after log_softmax,if I just compute a log for the result of softmax,is that right?<br>
As for nn.CrossEntropyLoss(),there can’t be a softmax.<br>
Could you please tell me which loss should I choose?</p>

username: mailcorahul
cooked: <p><code>nn.CrossEntropyLoss</code> combines <code>log_softmax</code> and <code>NLLLoss</code> which means you should not apply <code>softmax</code> at the end of your network output.<br>
So you are not required to apply softmax since the criterion takes care of it.</p>
<p>If you want to use <code>softmax</code> at the end, then you should apply log after that(as you mentioned above) and use <code>NLLLoss</code> as the criterion.</p>

username: chunchun
cooked: <p>If I do that, wouldn’t back propagation be a problem?</p>

username: chunchun
cooked: <p>If I do that, wouldn’t back propagation be a problem?</p>

username: mailcorahul
cooked: <p>Doing what will cause problem during backprop?</p>

username: chunchun
cooked: <p>I have no idea if I seperately use the softmax and log function instead of log_softmax,whether there will be a problem when BP.</p>

username: mailcorahul
cooked: <p>I don’t think it will cause any problem. It’s still same as using <code>log_softmax</code>. Maybe you can test your custom function just to make sure if it is consistent with <code>log_softmax</code>.</p>

username: KFrank
cooked: <p>Hi Raghul and Chunchun!</p>
<aside class="quote no-group" data-username="chunchun" data-post="6" data-topic="65121" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/97f17d/48.png" class="avatar"> chunchun:</div>
<blockquote>
<p>I have no idea if I seperately use the softmax and log function instead of log_softmax,whether there will be a problem when BP.</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="mailcorahul" data-post="7" data-topic="65121" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mailcorahul/48/15919_2.png" class="avatar"> mailcorahul:</div>
<blockquote>
<p>I don’t think it will cause any problem. It’s still same as using <code>log_softmax</code>.</p>
</blockquote>
</aside>
<p>Just to clarify:</p>
<p><code>log (softmax())</code> is <em>mathematically</em> the same as <code>log_softmax()</code>,<br>
but they differ numerically.  <code>softmax()</code> calculates exponentials that<br>
can “blow numbers up.”  The <code>log()</code> then undoes this, but the damage<br>
can already be done.  So <code>log (softmax())</code> can be numerically unstable,<br>
leading to reduced precision and <code>nan</code>s, and can cause problems.</p>
<p><code>log_softmax()</code> (largely) avoids this by reorganizing the calculation<br>
so that the intermediate blow-up doesn’t occur.  (That’s why pytorch<br>
(and other packages) include it as separate function.)</p>
<p>There is usually no reason to use <code>softmax()</code>.  Just feed the last linear<br>
layer of your network (that you would have fed into <code>softmax()</code>) into<br>
<code>cross_entropy()</code> as your loss function (or use <code>log_softmax()</code><br>
followed by <code>nll_loss()</code>).</p>
<p>If somebody <em>forces</em> you to use <code>softmax()</code> then you’re stuck, and<br>
have to deal with the potential numerical instability of of <code>softmax()</code><br>
followed by <code>log()</code> …</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: chunchun
cooked: <p>Thanks a lot for your answer.<br>
That’s very clear,but I must use the layer which can supply propabilities.<br>
Maybe I can use sigmoid+BCELoss?</p>

username: KFrank
cooked: <p>Hello Chunchun!</p>
<aside class="quote no-group" data-username="chunchun" data-post="1" data-topic="65121">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/97f17d/48.png" class="avatar"> chunchun:</div>
<blockquote>
<p>I must use the result of softmax</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="chunchun" data-post="9" data-topic="65121" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/97f17d/48.png" class="avatar"> chunchun:</div>
<blockquote>
<p>That’s very clear,but I must use the layer which can supply propabilities.</p>
</blockquote>
</aside>
<p>In general, there is no particular need to use probabilities <em>to feed<br>
into your loss function.</em></p>
<p>If your use case requires probabilities for some other reason,<br>
perhaps you could explain why you need them and what you<br>
need to use them for.</p>
<p>For training, you should use (based on what you’ve said so far)<br>
a linear layer that outputs numbers from <code>-inf</code> to <code>+inf</code> (that are<br>
to be understood as <em>logits</em>) fed into <code>cross_entropy()</code> as your<br>
loss function.  This will all be part of “autograd” and you will<br>
back-propagate through it.</p>
<p>Then, if you need actual probabilities for some other reason,<br>
take the outputs of your linear layer, and using<br>
<code>with torch. no_grad():</code> so you don’t affect your gradient<br>
calculation, run them through <code>softmax()</code> to convert the logits<br>
to the probabilities you want.</p>
<aside class="quote no-group">
<blockquote>
<p>Maybe I can use sigmoid+BCELoss?</p>
</blockquote>
</aside>
<p><code>BCELoss</code> (binary cross-entropy) is, <em>in essence,</em> the special two-class<br>
case of the multi-class <code>cross_entropy()</code> loss.</p>
<p><code>sigmoid()</code> → <code>BCELoss</code> has the same numerical problems as<br>
<code>softmax()</code> → <code>log()</code> → <code>nll_loss()</code>.  If you are performing a<br>
binary (two-class) classification problem, you will want to feed<br>
the (single) output of your last linear layer into<br>
<code>binary_cross_entropy_with_logits()</code> (<code>BCEWithLogitsLoss</code>).<br>
(This is the binary analog of <code>cross_entropy()</code> (<code>CrossEntropyLoss</code>).)</p>
<p>And again, if you <em>need</em> the actual probability (which you don’t for<br>
training), you would run the output of your last linear layer through<br>
<code>sigmoid()</code> (under <code>with torch. no_grad():</code>) to get the probability.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: chunchun
cooked: <p>Thank you very much!</p>