Inconsistent behaviour of cross entropy loss without weights

username: Dominika_Basaj
cooked: <p>Hi, I noticed that the output of cross-entropy loss (for semantic segmentation use case so K-dimensional one) with <code>reduction="mean"</code> is different than when I calculate it with sum and mean on unreduced output. This is most visible with a bigger batch size.</p>
<p>A minimal working example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

basic_img = torch.Tensor([arr for arr in np.random.rand(128, 2, 768, 768)])
label_basic = torch.Tensor([arr for arr in np.random.randint(2, size=(128,  768, 768))]).long()

criterion = nn.CrossEntropyLoss(reduction='none', weight=None)
loss = criterion(basic_img, label_basic)
loss.mean()

&gt;&gt;&gt; tensor(0.7137)

criterion = nn.CrossEntropyLoss(reduction='mean', weight=None)
loss = criterion(basic_img, label_basic)
loss

&gt;&gt;&gt; tensor(1.6724)
</code></pre>
<p>As far as I understand, these two versions should be equal. Am I missing something?</p>
<p>Pytorch version: 1.8.1</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: KFrank
cooked: <p>Hi Dominika!</p>
<p>The short answer is that this looks like a bug.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Maybe a bug?</p>
<aside class="quote no-group" data-username="Dominika_Basaj" data-post="1" data-topic="117540" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dominika_basaj/48/3628_2.png" class="avatar"> Dominika_Basaj:</div>
<blockquote>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss(reduction='none', weight=None)
loss = criterion(basic_img, label_basic)
loss.mean()

&gt;&gt;&gt; tensor(0.7137)

criterion = nn.CrossEntropyLoss(reduction='mean', weight=None)
loss = criterion(basic_img, label_basic)
loss

&gt;&gt;&gt; tensor(1.6724)
</code></pre>
<p>As far as I understand, these two versions should be equal. Am I missing something?</p>
</blockquote>
</aside>
<p>I can reproduce this on pytorch version 1.7.1.</p>
<p>When I use double precision, or the gpu, or smaller tensors, the issue<br>
seems to go away.  Naively, using <code>reduction = 'none'</code> and then<br>
taking the <code>mean()</code> seems to give the “right” answer.</p>
<p>(I don’t think this can be explained away as a legitimate consequence<br>
of floating-point round-off error.)</p>
<p>Here is my test script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2021)

nBatch = 128
nClass = 2
height = 768
width = 768

input = torch.randn (nBatch, nClass, height, width)
target = torch.randint (nClass, (nBatch, height, width))

print ('nBatch =', nBatch, '\nnClass =', nClass, '\nheight =', height, '\nwidth =', width)
print ('big, float:')
print ('mean:', torch.nn.CrossEntropyLoss() (input, target))
print ('none:', torch.nn.CrossEntropyLoss (reduction = 'none') (input, target).mean())
print ('sum:', torch.nn.CrossEntropyLoss (reduction = 'sum') (input, target) / target.numel())

print ('big, cuda:')
print ('mean:', torch.nn.CrossEntropyLoss() (input.cuda(), target.cuda()))
print ('none:', torch.nn.CrossEntropyLoss (reduction = 'none') (input.cuda(), target.cuda()).mean())
print ('sum:', torch.nn.CrossEntropyLoss (reduction = 'sum') (input.cuda(), target.cuda()) / target.numel())

print ('big, double:')
print ('mean:', torch.nn.CrossEntropyLoss() (input.double(), target))
print ('none:', torch.nn.CrossEntropyLoss (reduction = 'none') (input.double(), target).mean())
print ('sum:', torch.nn.CrossEntropyLoss (reduction = 'sum') (input.double(), target) / target.numel())


nBatch = 8
nClass = 2
height = 16
width = 16

input = torch.randn (nBatch, nClass, height, width)
target = torch.randint (nClass, (nBatch, height, width))

print ('nBatch =', nBatch, '\nnClass =', nClass, '\nheight =', height, '\nwidth =', width)
print ('small, float:')
print ('mean:', torch.nn.CrossEntropyLoss() (input, target))
print ('none:', torch.nn.CrossEntropyLoss (reduction = 'none') (input, target).mean())
print ('sum:', torch.nn.CrossEntropyLoss (reduction = 'sum') (input, target) / target.numel())

print ('small, cuda:')
print ('mean:', torch.nn.CrossEntropyLoss() (input.cuda(), target.cuda()))
print ('none:', torch.nn.CrossEntropyLoss (reduction = 'none') (input.cuda(), target.cuda()).mean())
print ('sum:', torch.nn.CrossEntropyLoss (reduction = 'sum') (input.cuda(), target.cuda()) / target.numel())

print ('small, double:')
print ('mean:', torch.nn.CrossEntropyLoss() (input.double(), target))
print ('none:', torch.nn.CrossEntropyLoss (reduction = 'none') (input.double(), target).mean())
print ('sum:', torch.nn.CrossEntropyLoss (reduction = 'sum') (input.double(), target) / target.numel())
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.7.1
nBatch = 128
nClass = 2
height = 768
width = 768
big, float:
mean: tensor(2.7632)
none: tensor(0.9028)
sum: tensor(0.6140)
big, cuda:
mean: tensor(0.9028, device='cuda:0')
none: tensor(0.9028, device='cuda:0')
sum: tensor(0.9028, device='cuda:0')
big, double:
mean: tensor(0.9028, dtype=torch.float64)
none: tensor(0.9028, dtype=torch.float64)
sum: tensor(0.9028, dtype=torch.float64)
nBatch = 8
nClass = 2
height = 16
width = 16
small, float:
mean: tensor(0.9148)
none: tensor(0.9148)
sum: tensor(0.9148)
small, cuda:
mean: tensor(0.9148, device='cuda:0')
none: tensor(0.9148, device='cuda:0')
sum: tensor(0.9148, device='cuda:0')
small, double:
mean: tensor(0.9148, dtype=torch.float64)
none: tensor(0.9148, dtype=torch.float64)
sum: tensor(0.9148, dtype=torch.float64)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: ptrblck
cooked: <p>Yes, this looks indeed like a bug.<br>
Thank you very much for creating this great code snippet and <a class="mention" href="/u/dominika_basaj">@Dominika_Basaj</a> thanks a lot for reporting this issue.<br>
I’ll create an issue on GitHub and will link it here.</p>
<p>EDIT: Issue created <a href="https://github.com/pytorch/pytorch/issues/55657">here</a></p>

username: ptrblck
cooked: <p>Update: <a href="https://github.com/pytorch/pytorch/pull/55674">Fix is incoming</a>. The root cause seems to be the missing accumulation <code>dtype</code> for the CPU path.<br>
CC <a class="mention" href="/u/kfrank">@KFrank</a> <a class="mention" href="/u/dominika_basaj">@Dominika_Basaj</a></p>

username: Dominika_Basaj
cooked: <p>Thank you for taking care of it <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>To not steal someone’s thunder: I’ve just forwarded the issue and <code>zsef123</code> created the proposed fix <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: KFrank
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="117540" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Update: <a href="https://github.com/pytorch/pytorch/pull/55674" rel="noopener nofollow ugc">Fix is incoming</a>. The root cause seems to be the missing accumulation <code>dtype</code> for the CPU path.<br>
CC <a class="mention" href="/u/kfrank">@KFrank</a> <a class="mention" href="/u/dominika_basaj">@Dominika_Basaj</a></p>
</blockquote>
</aside>
<p>Just to confirm, I see that the fix is in in the latest nighty.  Here is the<br>
result of my above-posted test script:</p>
<pre><code class="lang-plaintext">1.9.0.dev20210416
nBatch = 128
nClass = 2
height = 768
width = 768
big, float:
mean: tensor(0.9028)
none: tensor(0.9028)
sum: tensor(0.9028)
big, cuda:
mean: tensor(0.9028, device='cuda:0')
none: tensor(0.9028, device='cuda:0')
sum: tensor(0.9028, device='cuda:0')
big, double:
mean: tensor(0.9028, dtype=torch.float64)
none: tensor(0.9028, dtype=torch.float64)
sum: tensor(0.9028, dtype=torch.float64)
nBatch = 8
nClass = 2
height = 16
width = 16
small, float:
mean: tensor(0.9148)
none: tensor(0.9148)
sum: tensor(0.9148)
small, cuda:
mean: tensor(0.9148, device='cuda:0')
none: tensor(0.9148, device='cuda:0')
sum: tensor(0.9148, device='cuda:0')
small, double:
mean: tensor(0.9148, dtype=torch.float64)
none: tensor(0.9148, dtype=torch.float64)
sum: tensor(0.9148, dtype=torch.float64)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>