Loss criterion for classification task (argmax(softmax) vs. label or softmax vs. label)

username: L_Z
cooked: <p>So I am using cross entropy loss function in my CNN classification task. But I am not sure if that’s appropriate to compare between my labels which are integers labeling starting from 0 (e.g. 0, 1, 2) and the outputs which are as softmax in the range of 0 to 1.<br>
So if we compare theses two to find the losses, won’t that be really inaccurate? Like I should apply argmax to my outputs first in order to switch to integers before comparing with my actual labels? Thanks in advance</p>

username: KFrank
cooked: <p>Hi L!</p>
<aside class="quote no-group" data-username="L_Z" data-post="1" data-topic="207776" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l_z/48/71230_2.png" class="avatar"> L_Z:</div>
<blockquote>
<p>So I am using cross entropy loss function in my CNN classification task.</p>
</blockquote>
</aside>
<p>Pytorch’s <code>CrossEntropyLoss</code> takes the raw output of your model,<br>
that is, the output of your model’s final <code>Linear</code> layer <em>without</em> any<br>
following <code>softmax()</code> (or other “activation”).  These are to be understood<br>
as the unnormalized log-probabilities of each of the three classes.<br>
(<code>CrossEntropyLoss</code> has <code>log_softmax()</code> built into it.)</p>
<aside class="quote no-group">
<blockquote>
<p>But I am not sure if that’s appropriate to compare between my labels which are integers labeling starting from 0 (e.g. 0, 1, 2) and the outputs</p>
</blockquote>
</aside>
<p><code>CrossEntropyLoss</code> knows how to compare your integer class labels<br>
with the (unnormalized log-probabilities) outputs of your model.</p>
<aside class="quote no-group">
<blockquote>
<p>which are as softmax in the range of 0 to 1.</p>
</blockquote>
</aside>
<p>Just to emphasize, you should <em>not</em> have any sort of <code>softmax()</code><br>
between your model’s last <code>Linear</code> layer and <code>CrossEntropyLoss</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Like I should apply argmax to my outputs first in order to switch to integers before comparing with my actual labels?</p>
</blockquote>
</aside>
<p>No, independent of the previous discussion, those integers are discrete<br>
and therefore “not differentiable.”  So using <code>argmax()</code> in this way will<br>
“break the computation graph” and prevent backpropagation.  (You<br>
would certainly do this to compute a performance metric like accuracy,<br>
but not to compute a loss function that needs to be differentiable in<br>
order to backpropagate.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: L_Z
cooked: <p>Hi Frank, thank you for answering to me question. So if I need to input the output from my Linear layer directly to my CrossEntropyLoss function, then when should I apply the softmax? Or I do not actually need the softmax to find out which class I should classify into?</p>

username: KFrank
cooked: <p>Hi L!</p>
<aside class="quote no-group" data-username="L_Z" data-post="3" data-topic="207776" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l_z/48/71230_2.png" class="avatar"> L_Z:</div>
<blockquote>
<p>Or I do not actually need the softmax to find out which class I should classify into?</p>
</blockquote>
</aside>
<p>Correct, you do not need <code>softmax()</code> to predict a specific class.  This<br>
is because we (usually) predict a specific class by taking the specific<br>
class for which the predicted probability – the result of <code>softmax()</code> – is<br>
largest.</p>
<p>But <code>softmax()</code> doesn’t change the relative ordering of its inputs.<br>
That is, letting <code>pred</code> be the output of your model (and thus the<br>
unnormalized log-probabilities predicted by your model),<br>
<code>pred.softmax (-1).argmax() = pred.argmax()</code>.  So you can<br>
simply apply <code>argmax()</code> to the output of your model to get the<br>
specific predicted class without first applying <code>softmax()</code> (but you<br>
can apply <code>softmax()</code> – it doesn’t hurt anything except for taking<br>
a tiny bit of extra time).</p>
<p>Best.</p>
<p>K. Frank</p>

username: L_Z
cooked: <p>I have printed out the output right after nn.Linear and I have realized it is like a neg number and a positive number for classification of 2 classes. Will the crossEntropyLoss works fine with negative values?</p>

username: KFrank
cooked: <p>Hi L!</p>
<aside class="quote no-group" data-username="L_Z" data-post="5" data-topic="207776" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l_z/48/71230_2.png" class="avatar"> L_Z:</div>
<blockquote>
<p>it is like a neg number and a positive number for classification of 2 classes.</p>
</blockquote>
</aside>
<p>The output of your final <code>Linear</code> layer should be understood as<br>
unnormalized log-probabilities.  What would a negative number<br>
mean in this context?</p>
<p>What happens when you pass a negative number through<br>
<code>log_softmax()</code>, as <code>CrossEntropyLoss</code> does internally?  What<br>
happens when you pass a negative number through <code>softmax()</code>?<br>
How should the result of that be interpreted?</p>
<p>Let <code>pred</code> be the output of your final <code>Linear</code> layer.<br>
What is the difference between <code>pred.softmax (-1)</code><br>
and <code>(pred - 10).softmax (-1)</code>?</p>
<p>Best.</p>
<p>K. Frank</p>

username: L_Z
cooked: <p>Yeah, that’s why I am not sure what the negative values come from. Like the output before I apply the softmax is as tensor([6.0575, -5.3307]) for example, and then after I have applied softmax() it is as tensor([9.9999e-01, 1.1327e-05]). And log_softmax() provides me with the result as tensor([-1.1325e-05, -1.1389e+01]). But the result of argmax() is the same throughout. And the result for (pred - 10).softmax(-1) is the same as pred.softmax(-1). Thank you.</p>