CCE loss mean reduction differs between sparse and dense labels

username: Joe_Lorentz
cooked: <p>Hello,<br>
I am currently playing around with using class weights in the cce loss and noticed that I get different results from using the reduction “mean” between using one-hot encoded targets vs a sparse label tensor.<br>
It seems to me that the class weights are not used in case of mean reduction and label targets.<br>
Please check the code snippet below where I would expect bot “mean tensor” results to be the same, which is not the case<br>
Do I misunderstood something here?</p>
<pre><code class="lang-auto">import torch

weight=torch.tensor([0.2,0.8])
softmax = torch.nn.Softmax(-1)


cce_none = torch.nn.CrossEntropyLoss(weight=weight,reduction="none")
cce_mean = torch.nn.CrossEntropyLoss(weight=weight,reduction="mean")

input = torch.tensor([[.1,.6],[.3,.4]])
target = torch.tensor([[1.0,0.0],[1.0,0.0]])
sparse_target = torch.tensor([0,0])


print(input.shape)
print(target.shape)


print("\n log ")
# print(torch.log(input))
soft = softmax(input)
print(soft)
print("softlog",torch.log(soft))
print("softlog*weight",torch.log(soft)*weight)

cce_loss = cce_none(input,target)
print("none",cce_loss)

cce_loss = cce_mean(input,target)
print("mean",cce_loss)


print("\n sparse")
print(sparse_target.shape)
</code></pre>
<p>Which gives me:</p>
<pre><code class="lang-auto">torch.Size([2, 2])
torch.Size([2, 2])

 log 
tensor([[0.3775, 0.6225],
        [0.4750, 0.5250]])
softlog tensor([[-0.9741, -0.4741],
        [-0.7444, -0.6444]])
softlog*weight tensor([[-0.1948, -0.3793],
        [-0.1489, -0.5155]])
none tensor([0.1948, 0.1489])
mean tensor(0.1718)

 sparse
torch.Size([2])
none tensor([0.1948, 0.1489])
mean tensor(0.8592)
</code></pre>

username: KFrank
cooked: <p>Hi Joe!</p>
<aside class="quote no-group" data-username="Joe_Lorentz" data-post="1" data-topic="185464" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joe_lorentz/48/62809_2.png" class="avatar"> Joe_Lorentz:</div>
<blockquote>
<p>It seems to me that the class weights are not used in case of mean reduction and label targets.</p>
</blockquote>
</aside>
<p>This is not exactly true – both cases use the class weights, but use them<br>
a little differently.  (But your basic observation that the two approaches give<br>
different results is correct.)</p>
<aside class="quote no-group">
<blockquote>
<p>Do I misunderstood something here?</p>
</blockquote>
</aside>
<p>The missing piece is that the two approaches compute the weighted mean<br>
in <code>reduction = mean</code> somewhat differently.</p>
<p>From the <a href="https://pytorch.org/docs/2.0/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss documentation,</a> the “class-index” mean reduction<br>
computes what I consider to be a proper “weighted mean” over the per-sample<br>
(per-batch-element) losses.  Namely, it sums the weights times the unweighted<br>
losses and divides by the sum of the weights.</p>
<p>However, the “probability” mean reduction sums the weights times the<br>
unweighted losses, but divides, instead, by the number of samples.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">none tensor([0.1948, 0.1489])
mean tensor(0.1718)

 sparse
torch.Size([2])
none tensor([0.1948, 0.1489])
mean tensor(0.8592)
</code></pre>
</blockquote>
</aside>
<p>Your two values differ (for the example you give) by a factor of <code>0.2</code>, which is<br>
<code>weight[0]</code>.  (I agree that this is an unexpected difference, but it is consistent<br>
with <code>CrossEntropyLoss</code>’s documentation.)</p>
<p>Here is a tweaked version of your script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

weight = torch.tensor ([0.2, 0.8])
input = torch.tensor ([[0.1, 0.6], [0.3, 0.4]])
target = torch.tensor ([[1.0, 0.0], [1.0, 0.0]])
sparse_target = torch.tensor ([0, 0])

cce_mean = torch.nn.CrossEntropyLoss (weight = weight, reduction = 'mean')

cce_loss_target = cce_mean (input, target)
cce_loss_sparse = cce_mean (input, sparse_target)
print ('cce_loss_target: ', cce_loss_target)
print ('cce_loss_sparse: ', cce_loss_sparse)

print ('cce_loss_target / weight[0]: ', cce_loss_target / weight[0])
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">2.0.1
cce_loss_target:  tensor(0.1718)
cce_loss_sparse:  tensor(0.8592)
cce_loss_target / weight[0]:  tensor(0.8592)
</code></pre>
<p>It is tempting to require that the two weighted reductions give the same<br>
results.  However, your example is a special case in that your probabilistic<br>
<code>target</code> is either exactly <code>0.0</code> or <code>1.0</code>.  There is a legitimate question of how<br>
best to define the weighted reduction for a non-trivial probabilistic target<br>
(such as <code>[0.25, 0.75]</code>).</p>
<p>This is discussed in github issues <a href="https://github.com/pytorch/pytorch/issues/61309" rel="noopener nofollow ugc">61309</a> and <a href="https://github.com/pytorch/pytorch/pull/61044" rel="noopener nofollow ugc">61044.</a>  (But I agree with you<br>
that the current choice of definition is not ideal.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Joe_Lorentz
cooked: <p>Thank you for the detailed explanation.<br>
Now I understand why the difference is happening and how to handle it.</p>