Weight in cross entropy loss

username: Mainul
cooked: <p>I was trying to understand how <code>weight</code> is in <code>CrossEntropyLoss</code> works by a practical example. So I first run as standard PyTorch code and then manually both.  But the losses are not the same.</p>
<pre><code class="lang-auto">from torch import nn
import torch
softmax=nn.Softmax()
sc=torch.tensor([0.4,0.36])
loss = nn.CrossEntropyLoss(weight=sc)
input = torch.tensor([[3.0,4.0],[6.0,9.0]])
target = torch.tensor([1,0])
output = loss(input, target)
print(output)
&gt;&gt;1.7529
</code></pre>
<p>Now for manual Calculation, first softmax the <code>input</code>:</p>
<pre><code class="lang-auto">print(softmax(input))
&gt;&gt;
tensor([[0.2689, 0.7311],
        [0.0474, 0.9526]])
</code></pre>
<p>and then negetive log of the correct class probality and multiply with the respective weight:<br>
<code>((-math.log(0.7311)*0.36) - (math.log(0.0474)*0.4))/2</code></p>
<blockquote>
<blockquote>
<p>0.6662</p>
</blockquote>
</blockquote>
<p>What I am missing here?</p>

username: KFrank
cooked: <p>Hello Mainul!</p>
<aside class="quote no-group quote-modified" data-username="Mainul" data-post="1" data-topic="78256" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mainul/48/30342_2.png" class="avatar"> Mainul:</div>
<blockquote>
<p>But the losses are not the same.</p>
<pre><code class="lang-auto">sc=torch.tensor([0.4,0.36])
loss = nn.CrossEntropyLoss(weight=sc)
input = torch.tensor([[3.0,4.0],[6.0,9.0]])
target = torch.tensor([1,0])
output = loss(input, target)
print(output)
&gt;&gt;1.7529
</code></pre>
<p>Now for manual Calculation, first softmax the <code>input</code>:</p>
<pre><code class="lang-auto">print(softmax(input))
&gt;&gt;
tensor([[0.2689, 0.7311],
        [0.0474, 0.9526]])
</code></pre>
<p><code>((-math.log(0.7311)*0.36) - (math.log(0.0474)*0.4))/2</code></p>
<blockquote>
<blockquote>
<p>0.6662</p>
</blockquote>
</blockquote>
<p>What I am missing here?</p>
</blockquote>
</aside>
<p>When using <code>CrossEntropyLoss (weight = sc)</code> with class weights<br>
to perform the default <code>reduction = 'mean'</code>, the average loss that<br>
is calculated is the <em>weighted</em> average.  That is, you should be dividing<br>
by the sum of the weights used for the samples, rather than by the<br>
number of samples.</p>
<p>The following (pytorch version 0.3.0) script illustrates this:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

sc = torch.FloatTensor ([0.4,0.36])
loss = torch.nn.CrossEntropyLoss (weight = sc)
input = torch.autograd.Variable (torch.FloatTensor ([[3.0,4.0],[6.0,9.0]]))
target = torch.autograd.Variable (torch.LongTensor ([1,0]))
output = loss (input, target)
print (output)

probs = torch.nn.Softmax (dim = 1) (input)
output2 = -(torch.log (probs[0, 1]) * sc[1] + torch.log (probs[1, 0]) * sc[0]) / (sc[0] + sc[1])
print (output2)
print (((sc[0] + sc[1]) / 2.0) * output2)
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; sc = torch.FloatTensor ([0.4,0.36])
&gt;&gt;&gt; loss = torch.nn.CrossEntropyLoss (weight = sc)
&gt;&gt;&gt; input = torch.autograd.Variable (torch.FloatTensor ([[3.0,4.0],[6.0,9.0]]))
&gt;&gt;&gt; target = torch.autograd.Variable (torch.LongTensor ([1,0]))
&gt;&gt;&gt; output = loss (input, target)
&gt;&gt;&gt; print (output)
Variable containing:
 1.7529
[torch.FloatTensor of size 1]

&gt;&gt;&gt;
&gt;&gt;&gt; probs = torch.nn.Softmax (dim = 1) (input)
&gt;&gt;&gt; output2 = -(torch.log (probs[0, 1]) * sc[1] + torch.log (probs[1, 0]) * sc[0]) / (sc[0] + sc[1])
&gt;&gt;&gt; print (output2)
Variable containing:
 1.7529
[torch.FloatTensor of size 1]

&gt;&gt;&gt; print (((sc[0] + sc[1]) / 2.0) * output2)
Variable containing:
 0.6661
[torch.FloatTensor of size 1]
</code></pre>
<p>You can see that see the the (weighted) <code>CrossEntropyLoss</code> and<br>
“manual” results now match.  And at the end we recover your manual<br>
result by undoing the division by the sum of the weights.</p>
<p>Best.</p>
<p>K. Frank</p>