Understanding NLLLoss function

username: pawandeep_singh
cooked: <pre><code class="lang-auto">m = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
# input is of size N x C = 1 x 3
input = torch.tensor([[0,0,0]], requires_grad=True, dtype=torch.float)
# each element in target has to have 0 &lt;= value &lt; C
target = torch.tensor([1])
output = loss(m(input), target)
print(output)

_Output: tensor(1.0986, grad_fn=&lt;NllLossBackward&gt;)_
</code></pre>
<p>I read the <a href="https://pytorch.org/docs/stable/nn.html#nllloss" rel="nofollow noopener">Documentation</a> but it’s not clear. Can someone explain the math behind this example?</p>

username: ptrblck
cooked: <p>In your example the your output has the same “probability” for all three classes, i.e. the logits have the same value.<br>
Their probability should therefore be approx <code>[0.33, 0.33, 0.33]</code>.<br>
Since you are using <code>LogSoftmax</code> we can check, if this is true by calling <code>exp</code> on it (thus getting rid of the <code>log</code>):</p>
<pre><code class="lang-auto">print(m(input))
&gt; tensor([[-1.0986, -1.0986, -1.0986]], grad_fn=&lt;LogSoftmaxBackward&gt;)
print(m(input).exp())
&gt; tensor([[0.3333, 0.3333, 0.3333]], grad_fn=&lt;ExpBackward&gt;)
</code></pre>
<p>You will get the same values every time you pass the same logits into <code>LogSoftmax</code>.<br>
Now we just have to get the right index using <code>target</code>, multiply with <code>-1</code>, and end up with a loss value of <code>1.0986</code>.</p>

username: Calin_Serban
cooked: <pre><code class="lang-auto">loss = nn.NLLLoss()
a = torch.tensor(([0.88, 0.12], [0.51, 0.49]), dtype = torch.float)
target = torch.tensor([1, 0])
output = loss(a, target)
print(output)
</code></pre>
<p>Don’t know if it’s right to post this question here, but I’m trying: Why the output of this piece of code is tensor(-0.3150)? I was expecting to be (-1/2) * ((1 * ln(0.88) + 0 * ln(0.12) + 1 * ln(0.51) + 0 * ln(0.49)), which would be equal to 0.4005, not -0.3150?</p>
<p>I found the formula for log likelihood <a href="https://stats.stackexchange.com/questions/198038/cross-entropy-or-log-likelihood-in-output-layer" rel="nofollow noopener">here</a></p>

username: ptrblck
cooked: <p><code>nn.NLLLoss</code> expects the inputs to be log probabilities, while you are passing the probabilities into the criterion.</p>
<p>Also, your manual calculation seem to mix the target indices, as the first sample will have the class1 as its target and the second one class0.</p>
<p>Here is an example showing the same result:</p>
<pre><code class="lang-python">loss = nn.NLLLoss()
a = torch.tensor(([0.88, 0.12], [0.51, 0.49]), dtype = torch.float)
target = torch.tensor([1, 0])
output = loss(torch.log(a), target)
print(output)
&gt; tensor(1.3968)
print((-torch.log(a[0, 1]) - torch.log(a[1, 0])) / 2)
&gt; tensor(1.3968)
</code></pre>

username: Calin_Serban
cooked: <p>Ahh ok, thanks for the answer! What I am trying to figure out actually is how the nn.NLLLoss works for multidimensional tensors, but I couldn’t find an example. Could you give me a simple example on how that loss is calculated for a 2D or 3D tensor?</p>

username: KFrank
cooked: <p>Hi Calin!</p>
<aside class="quote no-group" data-username="Calin_Serban" data-post="5" data-topic="23702" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/calin_serban/48/23018_2.png" class="avatar"> Calin_Serban:</div>
<blockquote>
<p>What I am trying to figure out actually is how the nn.NLLLoss works for multidimensional tensors</p>
</blockquote>
</aside>
<p>Please see (if I understand what you are asking) the description of<br>
the “K-dimensional case” in the documentation for <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss" rel="noopener nofollow ugc">NLLLoss</a>.</p>
<p>Here is an illustrative (pytorch 0.3.0) script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

torch.manual_seed (2020)

nBatch = 2
nClass = 4
width = 3
height = 5
input = torch.randn (nBatch, nClass, width, height)
target = torch.multinomial (torch.ones (nClass) / nClass, nBatch * width * height, replacement = True).resize_ (nBatch, width, height)

input.shape
target.shape
target.min()
target.max()

input = torch.autograd.Variable (input)
input = torch.nn.functional.log_softmax (input, dim = 1)
target = torch.autograd.Variable (target)

torch.nn.NLLLoss() (input, target)
</code></pre>
<p>And here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x00000170D6456630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; nBatch = 2
&gt;&gt;&gt; nClass = 4
&gt;&gt;&gt; width = 3
&gt;&gt;&gt; height = 5
&gt;&gt;&gt; input = torch.randn (nBatch, nClass, width, height)
&gt;&gt;&gt; target = torch.multinomial (torch.ones (nClass) / nClass, nBatch * width * height, replacement = True).resize_ (nBatch, width, height)
&gt;&gt;&gt;
&gt;&gt;&gt; input.shape
torch.Size([2, 4, 3, 5])
&gt;&gt;&gt; target.shape
torch.Size([2, 3, 5])
&gt;&gt;&gt; target.min()
0
&gt;&gt;&gt; target.max()
3
&gt;&gt;&gt;
&gt;&gt;&gt; input = torch.autograd.Variable (input)
&gt;&gt;&gt; input = torch.nn.functional.log_softmax (input, dim = 1)
&gt;&gt;&gt; target = torch.autograd.Variable (target)
&gt;&gt;&gt;
&gt;&gt;&gt; torch.nn.NLLLoss() (input, target)
Variable containing:
 1.9742
[torch.FloatTensor of size 1]
</code></pre>
<p>Note that <code>target</code> has one less dimension than <code>input</code>.  In particular,<br>
target does not have an <code>nClass</code> dimension, while <code>input</code> does.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Calin_Serban
cooked: <p>Hi Frank, so I took a more simple example for trying to understand:</p>
<pre><code class="lang-auto">m = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
# input is of size N x C = 2 X 2
input = torch.randn(2, 2, requires_grad=True)
# each element in target has to have 0 &lt;= value &lt; C
target = torch.tensor([1, 0])
output = loss(m(input), target)
print(m(input))
print(target)
print(output)
</code></pre>
<p>, and one of its outputs was:</p>
<pre><code class="lang-auto">tensor([[-1.1722, -0.3706],
        [-0.5150, -0.9100]], grad_fn=&lt;LogSoftmaxBackward&gt;)
tensor([1, 0])
tensor(0.4428, grad_fn=&lt;NllLossBackward&gt;)
</code></pre>
<p>So which is the formula involved in this example for getting the value 0.4428 based on the given input and target, because it’s not really clear for me how l1,…,ln from the l(x, y) formula (official NLLLoss documentation) are calculated since the loss weight is None?</p>
<p>Thanks, Calin!</p>

username: KFrank
cooked: <p>Hello Calin!</p>
<aside class="quote no-group" data-username="Calin_Serban" data-post="7" data-topic="23702" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/calin_serban/48/23018_2.png" class="avatar"> Calin_Serban:</div>
<blockquote>
<pre><code class="lang-auto">tensor([[-1.1722, -0.3706],
        [-0.5150, -0.9100]], grad_fn=&lt;LogSoftmaxBackward&gt;)
tensor([1, 0])
tensor(0.4428, grad_fn=&lt;NllLossBackward&gt;)
</code></pre>
<p>So which is the formula involved in this example for getting the value 0.4428</p>
</blockquote>
</aside>
<p><code>0.4428 = -(-0.3706 + -0.5150) /  2</code>.</p>
<p>That is, the value of your <code>output</code> is the average of the losses for each<br>
of the two samples in you batch.</p>
<aside class="quote no-group">
<blockquote>
<p>it’s not really clear for me how l1,…,ln from the l(x, y) formula (official NLLLoss documentation) are calculated since the loss weight is None?</p>
</blockquote>
</aside>
<p>Quoting from the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss" rel="noopener nofollow ugc">NLLLoss</a> documentation:</p>
<p><strong>weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"> <em>Tensor</em> </a> <em>,</em> <em>optional</em> ) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.</p>
<p><em>Optional</em> means that the argument is allowed to be <code>None</code>, i.e, absent.<br>
In such a case there is no reweighting (or, equivalently, the reweighting<br>
factors are all equal to <code>1</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

username: Calin_Serban
cooked: <p>Thank you very much, Frank!</p>
<p>It’s finally clear now. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: shatakshi_raman
cooked: <p>I don’t know, if I could post a question here. But it’d great to find a soluton because honestly , I don’t understand what wrong here</p>
<p>optimizer = optim.Adam(net.parameters(), lr=0.001 )</p>
<p>EPOCHS = 3</p>
<p>for epoch in range(EPOCHS):</p>
<p>for data in trainset:</p>
<pre><code>X, y = data 

#Make the zero_grad()

net.zero_grad()

#output of the loss created

input = net(X.view(-1, 784)



#calculating the loss

loss = F.nll_loss(input, y)  # calc and grab the loss value

loss.backward()  # apply this loss backwards thru the network's parameters

optimizer.step()  # attempt to optimize weights to account for loss/gradients
</code></pre>
<p>print(loss)  # print loss. We hope loss (a measure of wrong-ness) declines!</p>
<p>But this keeps showing error at <strong>loss = F.nll_loss(input, y)</strong> and the error is SyntaxError, how do I solve it ?<br>
I am new to deep learning, but I have experience with sklearn</p>

username: ptrblck
cooked: <p>What kind of error are you seeing?<br>
Could you post the complete error message with the stack trace here, please?</p>
<p>Often <code>F.nll_loss</code> creates a shape mismatch error, since for a multi-class classification use case the model output is expected to contain log probabilities (applied <code>F.log_softmax</code> as the last activation function on the output) and have the shape <code>[batch_size, nb_classes]</code>. The target should be a <code>LongTensor</code> in the shape <code>[batch_size]</code> and should contain the class indices in the range <code>[0, nb_classes-1]</code>.</p>

username: pchhapolika
cooked: <aside class="quote no-group" data-username="Calin_Serban" data-post="3" data-topic="23702">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/calin_serban/48/23018_2.png" class="avatar"> Calin_Serban:</div>
<blockquote>
<p>here</p>
</blockquote>
</aside>
<p>When I use <code>single label classification</code>,  My labels are either <code>0</code> or <code>1</code></p>
<p>I get output as:</p>
<p>My model has Linear layer as last layer.</p>
<p><code>SequenceClassifierOutput(loss=tensor(0.3405, grad_fn=&lt;NllLossBackward&gt;), logits=tensor([[ 0.5105, -0.3917]], grad_fn=&lt;AddmmBackward&gt;), hidden_states=None, attentions=None)</code></p>
<p>How loss is calculated here from logits?</p>

username: ptrblck
cooked: <p>For a single binary output with labels in <code>[0, 1]</code> you should use <code>nn.BCEWithLogitsLoss</code> instead of <code>nn.NLLLoss</code>.<br>
The output of the model should then be a single value containing the logit and should be passed directly to the criterion without applying <code>sigmoid</code> on it.<br>
You could also use <code>sigmoid</code> and <code>nn.BCELoss</code> but the numerical stability would be worse.</p>

username: pchhapolika
cooked: <p>But, how can I explicitly do it here. It has picked up automatically from model?</p>
<p>Also how it calculates to value 0.3405?</p>
<pre><code class="lang-auto">model = tr.XLMRobertaForSequenceClassification.from_pretrained("/home/stb/AIML/model_mlm_vocab_exp1_20epocs",problem_type="single_label_classification", num_labels=2,
                                                               ignore_mismatched_sizes=True, id2label={0: 'negative', 1: 'positive'})
</code></pre>
<pre><code class="lang-auto">training_args = tr.TrainingArguments(
    #report_to = 'wandb',
    output_dir='/home/stb/AIML/results_vocab_ext_exp1',          # output directory
    overwrite_output_dir = True,
    num_train_epochs=10,              # total number of training epochs
    per_device_train_batch_size=10,  # batch size per device during training
    per_device_eval_batch_size=10,   # batch size for evaluation
    learning_rate=2e-5,
    warmup_steps=200,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs_exp1',            # directory for storing logs
    logging_steps=6000,
    evaluation_strategy="epoch"
    ,save_strategy="epoch"
    ,load_best_model_at_end=True
    ,fp16=True
    ,run_name="run1"
    ,gradient_accumulation_steps=20
    
)
</code></pre>

username: ptrblck
cooked: <p>I’m not sure which higher-level library you are using, but I would guess that the loss is calculated internally?<br>
If so, you would need to check the internal implementation of this library.</p>