How backward() is calculated in CrossEntropyLoss?

username: Mahdi_Amrollahi
cooked: <p>I have a simple Linear model and I need to calculate the loss for it. I applied two <code>CrossEntropyLoss</code> and <code>NLLLoss</code> but I want to understand how grads are calculated on these both methods.</p>
<p>On the output layer, I have 4 neurons which mean I am going to classify on 4 classes.</p>
<pre><code class="lang-auto">L1 = nn.Linear(2,4)
</code></pre>
<p>When I use <code>CrossEntropyLoss</code> I get grads for all the parameters:</p>
<pre><code class="lang-auto">L1.weight.grad
tensor([[ 0.1212,  0.2424],
        [ 0.1480,  0.2961],
        [ 0.1207,  0.2414],
        [-0.3899, -0.7798]])
</code></pre>
<p>But when I try to use <code>NLLLoss</code>, I just get grad for params of true class:</p>
<pre><code class="lang-auto">L1.weight.grad
tensor([[ 0.,  0.],
        [ 0.,  0.],
        [ 0.,  0.],
        [-1., -2.]])
</code></pre>
<p>May I know how grads are calculated on both methods?</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue and get the expected same results:</p>
<pre><code class="lang-python">criterion1 = nn.CrossEntropyLoss()
criterion2 = nn.NLLLoss()

L1 = nn.Linear(2,4)
x = torch.randn(1, 2)
target = torch.randint(0, 4, (1,))

# CrossEntropyLoss
out1 = L1(x)
loss1 = criterion1(out1, target)
print(loss1)
# tensor(1.6417, grad_fn=&lt;NllLossBackward0&gt;)

loss1.backward()
print(L1.weight.grad)
# tensor([[-0.7159,  0.1766],
#         [-0.5968,  0.1472],
#         [-0.1265,  0.0312],
#         [ 1.4391, -0.3549]])
L1.zero_grad()

# NLLLoss
out2 = L1(x)
loss2 = criterion2(F.log_softmax(out2, dim=1), target)
print(loss2)
# tensor(1.6417, grad_fn=&lt;NllLossBackward0&gt;)

loss2.backward()
print(L1.weight.grad)
# tensor([[-0.7159,  0.1766],
#         [-0.5968,  0.1472],
#         [-0.1265,  0.0312],
#         [ 1.4391, -0.3549]])
</code></pre>
<p>I guess you might be missing the <code>F.log_softmax</code> when using <code>nn.NLLLoss</code>?</p>

username: Mahdi_Amrollahi
cooked: <p>That is right. But actually what I wanted is that, how grads are calculated based on softmax and without it?</p>
<p>Let’s suppose that we have the output like <code> [.1, .2, .3, .4]</code> and the target is <code> [3]</code>, could you tell me how the grads are really calculated based on <code>CrossEntropyLoss</code> and  pure <code>NLLLoss</code>? I cannot understand how grads are calculated?<br>
It can also be fine if tell me on your own example.<br>
Thanks</p>

username: ptrblck
cooked: <p>You can see the manual approach of these loss functions <a href="https://discuss.pytorch.org/t/cross-entropy-loss-math-under-the-hood/79749/3">here</a> which would correspond to:</p>
<pre><code class="lang-python">L1 = nn.Linear(2,4)
x = torch.randn(1, 2)
target = torch.randint(0, 4, (1,))

# PyTorch approach
criterion = nn.CrossEntropyLoss(reduction="sum")
ref = L1(x)
loss_ref = criterion(ref, target)
print(loss_ref)
# tensor(0.9717, grad_fn=&lt;NllLossBackward0&gt;)
loss_ref.backward()
print(L1.weight.grad)
# tensor([[ 0.0744, -0.0261],
#         [-0.6789,  0.2381],
#         [ 0.2678, -0.0939],
#         [ 0.3368, -0.1181]])
L1.zero_grad()

# manual
out = L1(x)
#probs = torch.softmax(out, dim=1)
probs = torch.exp(out) / torch.sum(torch.exp(out), dim=1)

# Manual approach using your formula
one_hot = F.one_hot(target, num_classes = 4)
ce = (one_hot * torch.log(probs + 1e-7))[one_hot.bool()]
ce = -1 * ce.sum()
print(ce)
# tensor(0.9717, grad_fn=&lt;MulBackward0&gt;)
ce.backward()
print(L1.weight.grad)
# tensor([[ 0.0744, -0.0261],
#         [-0.6789,  0.2381],
#         [ 0.2678, -0.0939],
#         [ 0.3368, -0.1181]])
L1.zero_grad()
</code></pre>
<p>The second approach shows the manual calls and you should be able to recompute the backward pass based on the used operations (<code>torch.exp</code>, <code>torch.sum</code>, indexing etc.).</p>

username: KFrank
cooked: <p>Hi Mahdi!</p>
<p>Before addressing a possible point of confusion, let me emphasize that you<br>
should <em>not</em> feed the output of a <code>Linear</code> directly into <code>NLLLoss</code>.</p>
<p>The output of a <code>Linear</code> will typically include positive numbers, which, when<br>
interpreted as log-probabilities, correspond to invalid “probabilities” that are<br>
greater than one.  If you permit “probabilities” greater than one, <code>NLLLoss</code><br>
will return negative values, will be unbounded below, and your training will<br>
diverge.</p>
<aside class="quote no-group" data-username="Mahdi_Amrollahi" data-post="1" data-topic="168138" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_amrollahi/48/62481_2.png" class="avatar"> Mahdi_Amrollahi:</div>
<blockquote>
<p>But when I try to use <code>NLLLoss</code>, I just get grad for params of true class:</p>
<pre><code class="lang-auto">L1.weight.grad
tensor([[ 0.,  0.],
        [ 0.,  0.],
        [ 0.,  0.],
        [-1., -2.]])
</code></pre>
</blockquote>
</aside>
<p>This is to be expected (depending on the details of your use case).</p>
<p><code>NLLLoss</code> plucks out the log-probability of the “true” class (as specified<br>
by the passed-in target), and doesn’t depend on the non-true-class<br>
probabilities.  So those gradients are zero.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; log_probs = torch.rand (1, 5, requires_grad = True)
&gt;&gt;&gt; targ = torch.tensor ([2])   # "true" class is 2
&gt;&gt;&gt; loss = torch.nn.NLLLoss() (log_probs, targ)
&gt;&gt;&gt; loss
tensor(-0.7588, grad_fn=&lt;NllLossBackward0&gt;)
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; log_probs.grad   # only non-zero for "true" class
tensor([[ 0.,  0., -1.,  0.,  0.]])
&gt;&gt;&gt; with torch.no_grad():   # change values for some "non-true" classes
...     log_probs[0, 0] = 66.6
...     log_probs[0, 4] = 99.9
...
&gt;&gt;&gt; torch.nn.NLLLoss() (log_probs, targ)   # loss doesn't change
tensor(-0.7588, grad_fn=&lt;NllLossBackward0&gt;)
</code></pre>
<p>When you pass the output of <code>Linear</code> through <code>log_softmax()</code> (or <code>softmax()</code>,<br>
for that matter), it mixes the classes together so that the “true”-class value<br>
(that <code>NLLLoss</code> plucks out) depends on all of the outputs of <code>Linear</code> and you<br>
get non-zero gradients for all elements of <code>weight</code> (and <code>bias</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

username: Mahdi_Amrollahi
cooked: <p>What my problem was, it was about derivative of softmax and now I understand how to calculate and then apply it to network.</p>