Pytorch formula for NLL Loss

jp_letendre: <p>Hi,</p>
<p>I was wondering why the negative log likelihood function (<code>NLLLoss()</code>) in torch.nn expected a target. The <code>torch.nn.NLLLoss()</code> uses <code>nll_loss(input, target, weight=self.weight, ignore_index=self.ignore_index, reduction=self.reduction)</code> in his forward call. If NLL has the format : <img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a15e2c1297e5189a3ad8441065a365b36429ff28.png" alt="image" data-base62-sha1="n1wDpu0ADCjsEaDHNlbEbilo6co" width="166" height="51"> , why is the target vector needed to compute this, and not just the output of our nn.Softmax() layer?</p>
<p>Thanks,</p>
<p>JP</p>

ptrblck: <p>In your formula I assume <code>y</code> gives the target class prediction. Have a look at the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener">nn.CrossEntropyLoss docs</a> to see the applied formula.</p>

jp_letendre: <p>Thank you for your answer,</p>
<p>So the formula below describes the CrossEntropyLoss function implemented in Pytorch.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cb8d849684332310ccb4ed4472f08a7bc910bb3b.png" data-download-href="https://discuss.pytorch.org/uploads/default/cb8d849684332310ccb4ed4472f08a7bc910bb3b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cb8d849684332310ccb4ed4472f08a7bc910bb3b.png" alt="image" data-base62-sha1="t2I7T9ftMfubi7EWkZDmRMXomjp" width="690" height="87" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">827×105 10 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>From what I understand, this function never uses the label (or target) of the sample to compute the probability output for  <code>class</code>, since <code>x[class]</code> refers to the probability of our sample belonging to <code>class</code>, and sum(exp <code>(x[j])</code> is just the sum over the exponential of outputs for all classes. So where does Pytorch actually use the label of the sample in this loss function? I feel I’m missing something</p>

ptrblck: <p>By indexing <code>x</code> with <code>class</code> you are indeed using your target. Without the target you won’t know which logit to use.</p>

jp_letendre: <p>Got it, thanks!</p>
<p>Related to this, I’m actually trying to tweak this loss function. I’d like to replace the target (<code>y_o,c</code>) in the loss function below by the prediction probability output for a specific class <code>p_o,c</code>, basically turning this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e91db7a4cfd1bce5351a66dbaa7950a4ccc543ac.png" alt="image" data-base62-sha1="xgeV1yYzPBU5GtI7GUVGcrmPOPy" width="242" height="86"><br>
into this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/ce72b0d6b98c41fa0f7b48003217e76238e62dd5.png" alt="image" data-base62-sha1="tsk589SgjYkIXMeFX123WYDLUmF" width="250" height="95"></p>
<p>I tried simply replacing the <code>target</code> argument by <code>outputs</code> again when calling a <code>CrossEntropyLoss()</code> object, but Pytorch expects a datatype <code>torch.long</code>, not a <code>torch.float</code> as it is the case with <code>outputs</code>. Do you have any idea how I could obtain this new loss function?</p>
<p>Thanks,</p>
<p>JP</p>

ptrblck: <p>If you are looking for label smoothing, <a href="https://discuss.pytorch.org/t/computing-softmax-cross-entropy-for-smoothed-labels/19496">this thread</a> might have an interesting code snippet.<br>
Alternatively, you could just write out the formula.<br>
I’m not sure, how <code>p_o,c</code> is defined, but I guess it should be the probability of class <code>c</code> for output <code>o</code>?</p>
<pre><code class="lang-python">-1. * (target * F.log_softmax(x, 1)[target]).sum()
</code></pre>

jp_letendre: <p>If we’re talking mini-batch training, <code>p_o,c</code> would be an array of size (batch_size x num_classes), representing for each sample in th mini-batch it’s probability distribution over all possible classes. My concern was that I wanted a backpropagatable loss function. I’ll try that line and come back to you.</p>

rasbt: <aside class="quote no-group" data-username="jp_letendre" data-post="1" data-topic="38121">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jp_letendre/48/12853_2.png" class="avatar"> jp_letendre:</div>
<blockquote>
<p>I was wondering why the negative log likelihood function ( <code>NLLLoss()</code> ) in torch.nn expected a target. The <code>torch.nn.NLLLoss()</code> uses <code>nll_loss(input, target, weight=self.weight, ignore_index=self.ignore_index, reduction=self.reduction)</code> in his forward call</p>
</blockquote>
</aside>
<p>I think this is incorrect and nll_loss already assumes that you pass log(probabilities). You test this by giving it  just probabilities (onehot format with values in range [0, 1]) and it will give you negative values.</p>

jp_letendre: <p>I ended up modifying the function a bit for my needs, but it worked! Thanks</p>

EGanji: <p>These two losses (1, 2) are equal:</p>
<pre><code class="lang-auto">        loss1 = F.nll_loss(output, target)

        criterion = nn.CrossEntropyLoss()
        label = output.argmax(1)
        loss2 = criterion(output, label)
</code></pre>
<p>In this context</p>
<pre><code class="lang-auto">    for data, target in test_loader:
        data, target = data.to(device), target.to(device)

        data.requires_grad = True
        output = model(data)
        init_pred = output.max(1, keepdim=True)[1] 
        # get the index of the max log-probability

        # Calculate the loss
        loss1 = F.nll_loss(output, target)

        label = output.argmax(1)
        criterion = nn.CrossEntropyLoss()
        loss2 = criterion(output, label)
</code></pre>

ptrblck: <p>They are not as seen in this example:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_size = 16
nb_classes = 64

# initialize the model output as logits
output = torch.randn(batch_size, nb_classes, requires_grad=True)
target = torch.randint(0, nb_classes, (batch_size,))

# this is wrong since log-probabilities are expected
loss1 = F.nll_loss(output, target)

criterion = nn.CrossEntropyLoss()
# this is also wrong since another target is used
label = output.argmax(1)
loss2 = criterion(output, label)

# compare
print(loss1 - loss2)
# tensor(-1.9827, grad_fn=&lt;SubBackward0&gt;)


# right approach
loss1 = F.nll_loss(F.log_softmax(output, 1), target)
loss2 = criterion(output, target)

print(loss1 - loss2)
# tensor(0., grad_fn=&lt;SubBackward0&gt;)
</code></pre>
<p>since <code>F.nll_loss</code> expects log-probabilities while <code>nn.CrossEntropyLoss</code> expects raw logits. You are also not using the same target in both cases.</p>

EGanji: <p>Target and label both are 1-element tensors.<br>
Loss1 and loss2 exactly give the same values.</p>

ptrblck: <p>Using 1-element tensors for <code>target</code> and <code>label</code> (which is created via <code>output.argmax(1)</code>) indicates <code>batch_size=1</code> as well as <code>nb_classes=1</code>, which is not a valid multi-class classification use case.<br>
However, even then the losses differ and are not equal due to the reasons I’ve posted above:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_size = 1
nb_classes = 1

# initialize the model output as logits
output = torch.randn(batch_size, nb_classes, requires_grad=True)
target = torch.randint(0, nb_classes, (batch_size,))

# this is wrong since log-probabilities are expected
loss1 = F.nll_loss(output, target)

criterion = nn.CrossEntropyLoss()
# this is also wrong since another target is used
label = output.argmax(1)
loss2 = criterion(output, label)

# compare
print(loss1 - loss2)
# tensor(0.5912, grad_fn=&lt;SubBackward0&gt;)
</code></pre>

EGanji: <p>nb_classes = 10, also target = label<br>
That’s weird; this is what I got:</p>
<pre><code class="lang-auto">        loss1 = F.nll_loss(output, target)

        label = output.argmax(1)
        criterion = nn.CrossEntropyLoss()
        loss2 = criterion(output, label)
        print(loss1.detach(), loss2.detach(), (loss2 - loss1).detach())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor(0.1912) tensor(0.1912) tensor(0.)
tensor(0.0679) tensor(0.0679) tensor(0.)
tensor(0.0506) tensor(0.0506) tensor(0.)
tensor(0.2045) tensor(0.2045) tensor(0.)
tensor(0.0665) tensor(0.0665) tensor(0.)
tensor(0.1176) tensor(0.1176) tensor(0.)
tensor(0.1572) tensor(0.1572) tensor(0.)
</code></pre>
<p>I know,</p>
<ul>
<li><strong><code>loss1</code></strong> is the loss between the model’s predicted probabilities (log-softmax outputs) and the true labels.</li>
<li><strong><code>loss2</code></strong> is the loss between the logits and the model’s predicted labels.</li>
</ul>