Cross Entropy Loss Math under the hood

mfcs: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you help me?<br>
Hi everyone!</p>
<p>Please, someone could explain the math under the hood of Cross Entropy Loss in PyTorch?<br>
I was performing some tests here and result of the Cross Entropy Loss in PyTorch doesn’t match with the result using the expression below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/296a217feca623ffb81de274dc94d31162656903.png" alt="image" data-base62-sha1="5UmXyyHztvcz1NWuB6PDYuAbMHN" width="231" height="45"><br>
I took some examples calculated using the expression above and executed it using the Cross Entropy Loss in PyTorch and the results were not the same.</p>
<p>I am trying this example here using Cross Entropy Loss from PyTorch:</p>
<pre><code class="lang-auto">probs1 = torch.tensor([[
    [
      [ 0.1, 0.3],
      [0.4, 0.5]
    ],
    [
      [0.2, 0.3],
      [0.4, 0.5]
    ],
    [
      [0.7, 0.4],
      [0.2, 0.0]
    ]
  ]])

target = torch.tensor([
    [
      [2, 2],
      [0, 1]
    ]
  ])

print(torch.sum(probs1, dim=0))
print(probs1.shape)

criterion = nn.CrossEntropyLoss()
loss = criterion(probs1, target)
loss

result -&gt; tensor(0.9488)
</code></pre>
<p>Each pixel along the 3 channels corresponds to a probability distribution…there is a probability distribution for each position of the tensor…and the target has the classes for each distribution.<br>
How can I know if this loss is beign computed correctly?</p>
<p>Best regards,</p>
<p>Matheus Santos.</p>

KFrank: <p>Hello Matheus!</p>
<aside class="quote no-group" data-username="mfcs" data-post="1" data-topic="79749" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mfcs/48/17149_2.png" class="avatar"> mfcs:</div>
<blockquote>
<p>I was performing some tests here and result of the Cross Entropy Loss in PyTorch doesn’t match with the result using the expression below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/296a217feca623ffb81de274dc94d31162656903.png" alt="image" data-base62-sha1="5UmXyyHztvcz1NWuB6PDYuAbMHN" width="231" height="45"></p>
</blockquote>
</aside>
<p>The issue is that pytorch’s <code>CrossEntropyLoss</code> doesn’t exactly match<br>
the conventional definition of cross-entropy that you gave above.</p>
<p>Rather, it expects raw-score <em>logits</em> as it inputs, and, in effect, applies<br>
<code>softmax()</code> to the logits internally to convert them to probabilities.<br>
(<code>CrossEntropyLoss</code> might better have been named<br>
<code>CrossEntropyWithLogitsLoss</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>I am trying this example here using Cross Entropy Loss from PyTorch:</p>
<pre><code class="lang-auto">probs1 = torch.tensor([[
    [
      [ 0.1, 0.3],
      [0.4, 0.5]
    ],
    [
      [0.2, 0.3],
      [0.4, 0.5]
    ],
    [
      [0.7, 0.4],
      [0.2, 0.0]
    ]
  ]])
</code></pre>
</blockquote>
</aside>
<p>To check this, you could apply the logit function, <code>log (p / (1 - p))</code><br>
to convert your <code>probs1</code> tensor, and then run that through<br>
<code>CrossEntropyLoss</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <p>Note that you are not using <code>nn.CrossEntropyLoss</code> correctly, as this criterion expects logits and will apply <code>F.log_softmax</code> internally, while <code>probs</code> already contains probabilities, as <a class="mention" href="/u/kfrank">@KFrank</a> explained.</p>
<p>So, let’s change the criterion to <code>nn.NLLLoss</code> and apply the <code>torch.log</code> manually.<br>
This approach is just to demonstrate the formula and shouldn’t be used, as <code>torch.log(torch.softmax())</code> is less numerically stable than <code>F.log_softmax</code>.</p>
<p>Also, the default reduction for the criteria in PyTorch will calculate the average over the observations, so lets use <code>reduction='sum'</code>.</p>
<p>Given that you’ll get:</p>
<pre><code class="lang-python">criterion = nn.NLLLoss(reduction='sum')
loss = criterion(torch.log(probs1), target)
</code></pre>
<p>The manual approach from your formula would correspond to:</p>
<pre><code class="lang-python"># Manual approach using your formula
one_hot = F.one_hot(target, num_classes = 3)
one_hot = one_hot.permute(0, 3, 1, 2)
ce = (one_hot * torch.log(probs1 + 1e-7))[one_hot.bool()]
ce = -1 * ce.sum()
</code></pre>
<p>While the manual approach from the PyTorch docs would give you:</p>
<pre><code class="lang-python"># Using the formula from the docs
loss_manual = -1 * torch.log(probs1).gather(1, target.unsqueeze(1))
loss_manual = loss_manual.sum()
</code></pre>
<p>We should get the same results:</p>
<pre><code class="lang-python">print(loss, ce, loss_manual)
&gt; tensor(2.8824) tensor(2.8824) tensor(2.8824)
</code></pre>
<p>which looks correct.</p>

mfcs: <p>Hey, thank you so much for all explanations here <a class="mention" href="/u/kfrank">@KFrank</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a> !!!</p>
<p>I was performing some tests using tensors with lower dimensions to ensure that the loss result is correct and, due to this,  expand to tensors with higher dimensions and do not worry about the possibility that the loss value is wrong.</p>
<p>I was actually using nn.CrossEntropyLoss () in the wrong way, I apologize for that.<br>
Now I understood how to use it !!</p>
<p>Thank you for your time!</p>
<p>Best regards,</p>
<p>Matheus Santos.</p>