Does NLLLoss handle Log-Softmax and Softmax in the same way?

malioboro: <p>from documentation here : <a href="http://pytorch.org/docs/master/nn.html#torch.nn.LogSoftmax" rel="nofollow noopener">http://pytorch.org/docs/master/nn.html#torch.nn.LogSoftmax</a>, log-softmax is defined as:</p>
<pre><code class="lang-auto">f(x)=log(softmax(x))
</code></pre>
<p>as I know, using log in probabilities will change the high-low value, so biggest value in softmax(x) will be the smallest value in log(softmax(x)).</p>
<p>Will it change the way Negative Log Likelihood compute loss when it implements it together in <code>CrossEntropy</code>?</p>

malioboro: <p>Oh., I’ve just read from <a href="http://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss" rel="nofollow noopener">NLLLoss documentation</a>, that NLLLoss is implemented differently than what I’ve known before:</p>
<pre><code class="lang-auto">loss(x, class) = -x[class]
</code></pre>
<p>it doesn’t use <code>log</code> function. So, does it means I can’t use <code>Softmax</code> and <code>NLLLoss</code> together?</p>

ptrblck: <blockquote>
<p>so biggest value in softmax(x) will be the smallest value in log(softmax(x)).</p>
</blockquote>
<p>The <code>softmax</code> function returns probabilities between [0, 1].<br>
The log of these probabilities returns values between [-inf, 0], since <code>log(0) = -inf</code> and <code>log(1) = 0</code>.<br>
That is why the order won’t change.</p>
<p>However, you should use the <code>NLLLoss</code> with a <code>log_softmax</code> output<br>
or <code>CrossEntropyLoss</code> with logits if you prefer not to add an extra <code>log_softmax</code> layer into your model.</p>

malioboro: <p>oh… that’s right! I thought <code>log(0) = inf</code></p>
<p>Thanks,</p>

ANKITA_DAS: <p>If the order doesn’t change, why do we have to use a ‘log-softmax’ with NLLLoss?</p>

ptrblck: <p>By “order” I meant the range of the outputs will still be in the same order.<br>
I.e. if <code>p0 &lt; p1</code> created by <code>softmax</code> , then <code>log(p0) &lt; log(p1)</code>.</p>
<p><code>nn.NLLLoss</code> expects the log probabilities, as the loss will be calculated as described <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss">here</a>.<br>
Note that <code>nn.CrossEntropyLoss</code> applies internally <code>F.log_softmax</code> and <code>nn.NLLLoss</code> afterwards, which is why it expects raw logits instead.</p>

ANKITA_DAS: <p>Yes I do get that. Yet I have another question if log-softmax gives values in the range [-infinity,0] that means that the values are negative,and the NLLLoss functon is -log(y), where y=log-softmax(x) , but log of some negative value isn’t defined so how does is work?</p>

ptrblck: <p>The formula in the docs is the negative log softmax written as:</p>
<pre><code class="lang-python">- log ( exp(x[class]) / sum(exp(x[j]))
</code></pre>
<p><code>x</code> are the logits here while <code>exp()/sum(exp())</code> is the softmax function.</p>

ANKITA_DAS: <p>Sorry but, I searched for it in the documentation,I didn’t find it explicitly mentioned anywhere that negative of log_softmax goes in as input to the nllloss function.I don’t know if I’m missing something important, but please check the docs once.</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/nn.html?highlight=nllloss#torch.nn.NLLLoss">nn.NLLLoss docs</a>:</p>
<blockquote>
<p>The input given through a forward call is expected to contain log-probabilities of each class. input has to be a Tensor of size either […].<br>
Obtaining log-probabilities in a neural network is easily achieved by adding a LogSoftmax layer in the last layer of your network. You may use CrossEntropyLoss instead, if you prefer not to add an extra layer.</p>
</blockquote>
<pre><code class="lang-auto">Examples:
m = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
# input is of size N x C = 3 x 5
input = torch.randn(3, 5, requires_grad=True)
# each element in target has to have 0 &lt;= value &lt; C
target = torch.tensor([1, 0, 4])
output = loss(m(input), target)
output.backward()
</code></pre>

ANKITA_DAS: <p>But, there isn’t any mention of <strong>negative</strong> log_softmax here! as you mentioned in this :<a href="https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/8" class="inline-onebox">Does NLLLoss handle Log-Softmax and Softmax in the same way?</a></p>

ptrblck: <p>Ah, sorry for the confusion, as I can see the misunderstanding now.</p>
<p><code>nn.NLLLoss</code> expects log probabilities, so you should just apply <code>F.log_softmax</code> on your model output (not multiplying with <code>-1</code>!).</p>
<p>The formula posted in my previous post is, how the loss can be calculated, but you shouldn’t worry about the minus sign, as it will be applied internally for you.</p>
<p>The posted example shows, how to apply the criterion.</p>

ANKITA_DAS: <p>If the minus will be applied internally my doubt has been cleared, but they haven’t mentioned that in the documentation <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=9" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:"> as per my findings.</p>

ptrblck: <p>The documentation mentions that log probabilities are expected and gives a code example.</p>
<p>What is missing and what statement do you think might have been helpful to solve your misunderstanding? The docs are by far perfect, so feedback is always welcome. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ANKITA_DAS: <p>What needs to be given as input and what we get as output is fine, there is no problem in understanding that, but, if a minus is introduced internally I think there should be mention of that, as the screenshot of the documentation that you provided above it says</p>
<aside class="quote no-group" data-username="ptrblck" data-post="10" data-topic="8835">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The input given through a forward call is expected to contain log-probabilities of each class. input has to be a Tensor of size either […].</p>
</blockquote>
</aside>
<p>If it was mentioned somewhere that “nn.NLLLoss internally multiplies the log-probabilities by a minus”, it would have cleared the confusions!</p>

Ashima_Garg: <p>As you said <code>so biggest value in softmax(x) will be the smallest value in log(softmax(x)).</code><br>
Does that mean to calculate accuracy, <code>argmin</code> of predictions will be used for classification in Sentimental analysis? I am using NLLLoss to calculate the loss.</p>
<p>Thankyou.</p>

ptrblck: <p>No, I just quoted this sentence and corrected it in <a href="https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/3">this post</a>.</p>

Ashima_Garg: <p>Thanks for the reply.</p>

ADONAI_TZEVAOT: <p>Hi, pls could you clear my doubt, I have a multi-label classification problem, where each label can either be positive or negative, so each label on its own is binary in nature. The last layer in my network is LogSoftmax , so what loss function should I use ? BCEloss ? And how do i calculate the class weights? Pls help.</p>

ptrblck: <p>When each label is either positive or negative it would be a binary classification or a (2 class) multi-class classification.<br>
You can either return two logits (or log probabilities) from the model and use <code>nn.CrossEntropyLoss</code> (or <code>nn.NLLLoss</code>) or alternatively you can return a single logit and use <code>nn.BCEWithLogitsLoss</code>.</p>