When to use Softmax and CrossEntropyLoss together

python_Dev: <p>I am trying to understand why some code works even though when training we are not supposed to use softmax as the output and use crossentropyloss to train the model.<br>
<a href="https://github.com/eriklindernoren/Action-Recognition/blob/master/models.py" rel="noopener nofollow ugc">Here</a> it seems that the softmax is used as output and the crossentropyloss as the loss function and the model gives good results.<br>
Why is that?Are there cases where we can use the two together?</p>

thecho7: <p>I think there is no reason to combine <code>softmax</code> and <code>CrossEntropyLoss</code> in the referred repo.</p>
<p>In most of cases, <code>CrossEntropyLoss</code> forms a serial of <code>LogSoftmax</code> and <code>NLLLoss</code>.<br>
We can easily check what happens when the tensor <strong>get softmaxed</strong> twice.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; loss = nn.Softmax(dim=0)
&gt;&gt;&gt; a = torch.rand(10)
&gt;&gt;&gt; a
tensor([0.9364, 0.9396, 0.1604, 0.8733, 0.0822, 0.5598, 0.1300, 0.7800, 0.5182,
        0.9108])
&gt;&gt;&gt; b = loss(a)
&gt;&gt;&gt; b
tensor([0.1342, 0.1347, 0.0618, 0.1260, 0.0571, 0.0921, 0.0599, 0.1148, 0.0884,
        0.1309])
&gt;&gt;&gt; bb = loss(b)
&gt;&gt;&gt; bb
tensor([0.1034, 0.1035, 0.0962, 0.1026, 0.0958, 0.0992, 0.0960, 0.1014, 0.0988,
        0.1031])
</code></pre>
<p>The values inside of the given tensor is so flattened (there is no dominant class at all)<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss</a></p>

python_Dev: <p>This is what I thought and I was surprised that the model worked. Do you know why this is the case? I saw another post and they said that it is possible that the values become too similar after using softmax and cross entropy loss function</p>

thecho7: <p>Of course the model works with double softmax but,<br>
I guess the model output before the softmax might have somewhat dramatic values like</p>
<pre><code class="lang-auto">out = torch.tensor([1000., 0.02, 0.1,., 0.0015, ..., 0.00001]) # indicates class 0
</code></pre>
<p>It probably becomes the unstable learning:(<br>
This is a sample test</p>
<pre><code class="lang-auto">&gt;&gt;&gt; loss = nn.Softmax(dim=0)
&gt;&gt;&gt; a = torch.tensor([1000, 0.1, 0.1, 0.2, 0.5])
&gt;&gt;&gt; a
tensor([1.0000e+03, 1.0000e-01, 1.0000e-01, 2.0000e-01, 5.0000e-01])
&gt;&gt;&gt; b = loss(a)
&gt;&gt;&gt; b
tensor([1., 0., 0., 0., 0.])
&gt;&gt;&gt; bb = loss(b)
&gt;&gt;&gt; bb
tensor([0.4046, 0.1488, 0.1488, 0.1488, 0.1488])
</code></pre>

python_Dev: <p>I see, so in such cases which give weird results using softmax twice helps to tackle it.<br>
Also, I assume this can arise even when the values are normalised, correct?</p>

shivammehta007: <p>I think one use case could be something similar to <a href="https://stackoverflow.com/a/63471046/9384082" rel="noopener nofollow ugc">Temparature softmax</a>? i.e you don’t want your model to be very confident in predicting something, so you flatten the predicted distribution.</p>

python_Dev: <p>I see, seems to agree with <strong>thecho7</strong></p>

thecho7: <p>I agree.<br>
I think normalization cannot solve the problem.<br>
As <a class="mention" href="/u/shivammehta007">@shivammehta007</a> mentioned, if we need a smoothed label, I’d rather use Temperature softmax than double softmax lol.</p>
<p>So, is everything clear?</p>

python_Dev: <p>yes, things seems to make sense</p>