Question about weight in CrossEntropyLoss

Philipp_Singer: <p>Hi all,</p>
<p>from my understanding the <code>weight</code> parameter in <code>CrossEntropyLoss</code> is behaving different for mean reduction and other reductions. I believe in case of non-mean reductions the sample loss is just scaled by respective class weight for that sample. Yet, in the case of mean reduction, the loss is first scaled per sample, and then the sum is normalized by the sum of weights within the batch.</p>
<p>The important thing to me is the last part, namely that it is normalized by weights within the batch. Is this observation of mine correct? I wonder what is the idea behind that? If I am using weights to e.g., balance across the dataset, wouldn’t I want to normalize globally?</p>

KFrank: <p>Hi Philipp!</p>
<aside class="quote no-group" data-username="Philipp_Singer" data-post="1" data-topic="95094" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/philipp_singer/48/15250_2.png" class="avatar"> Philipp_Singer:</div>
<blockquote>
<p>in the case of mean reduction, the loss is first scaled per sample, and then the sum is normalized by the sum of weights within the batch.</p>
</blockquote>
</aside>
<p>This is correct (if I understand what you are saying).</p>
<aside class="quote no-group">
<blockquote>
<p>I wonder what is the idea behind that?</p>
</blockquote>
</aside>
<p>The “mean reduction” computes a (conventional) <em>weighted average</em>,<br>
that is, it does divide by the sum of the weights.</p>
<p>This makes sense to me because if, by happenstance, all of the<br>
samples in the batch have the same loss, <code>loss_all</code>, I would like the<br>
mean reduction over that batch also to give a batch mean of <code>loss_all</code>.<br>
The conventional weighted average does this.</p>
<p>These two threads (about <code>NLLLoss</code>, but its the same issue) give some<br>
additional words of explanation:</p>
<aside class="quote quote-modified" data-post="1" data-topic="94314">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/coolsuntraveler/48/16640_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/mean-reduction-in-nll-loss/94314">Mean Reduction in nll_loss</a> 
  </div>
  <blockquote>
    According to nll_loss documentation, for reduction parameter, " 'none' : no reduction will be applied,  'mean' : the sum of the output will be divided by the number of elements in the output,  'sum' : the output will be summed." 
However, it seems “mean” is divided by the sum of the weights of each element, not number of elements in the output. Am I misunderstanding the documentation or the documentation is inaccurate? 



input = torch.randn(3, 5, requires_grad=True) 
ls =F.log_softmax(input, 1…
  </blockquote>
</aside>

<aside class="quote quote-modified" data-post="1" data-topic="93116">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/awesomelemon/48/21583_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/weights-in-nllloss-behave-unexpectedly/93116">Weights in NllLoss behave unexpectedly</a> 
  </div>
  <blockquote>
    Hello, 
I’m having trouble understanding behaviour of class weights in CrossEntropyLoss. 
Specifically, when reduction=‘mean’. I test it like this: 

input = torch.randn(5, 2, requires_grad=True) 
m = nn.LogSoftmax(dim=1) 
mi = m(input) 
target = torch.tensor([0, 0, 1, 1, 1]) 
w = torch.tensor([1, 100]).float() 

Now, without weights everything behaves reasonably, next 2 lines give the same result 

F.nll_loss(mi, target, reduction=‘none’).mean() 
F.nll_loss(mi, target, reduction=‘mean’) 

Howev…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

Philipp_Singer: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a> -</p>
<p>thanks for the elaboration. It still does not fully make sense to me. Consider the following example where I have 4 samples and two batches with 2 samples each.</p>
<p>Now my first batch has weights [1,1] and my second batch has weights [2,2], and for all 4 samples the loss is 1, then both batches have the exact same overall loss. This does not sound intuitive to me as I want to weigh the second batch higher.</p>

KFrank: <p>Hi Philipp!</p>
<aside class="quote no-group" data-username="Philipp_Singer" data-post="3" data-topic="95094" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/philipp_singer/48/15250_2.png" class="avatar"> Philipp_Singer:</div>
<blockquote>
<p>Now my first batch has weights [1,1] and my second batch has weights [2,2], and for all 4 samples the loss is 1, then both batches have the exact same overall loss. This does not sound intuitive to me as I want to weigh the second batch higher.</p>
</blockquote>
</aside>
<p>First, note that in practice you can use <code>reduction ='sum'</code>, and then<br>
sum the the two per-batch losses together to get the relative weighting<br>
you want.  (You can divide by total number of samples, if you choose.)</p>
<p>You do, however, raise a legitimate point:  Splitting a batch in two, and<br>
averaging together the weighted averages of the two sub-batches<br>
won’t give the same result as taking the weighted average over the<br>
unsplit batch.  But it’s a trade-off, and you can’t have everything.  I’m<br>
willing to give this up in order to have the weighted average of a bunch<br>
of identical losses be equal to that shared loss value.</p>
<p>(Note, if you take an appropriately <em>weighted</em> average of the sub-batch<br>
weighted averages, you will get what you want, although, in practice,<br>
going this route likely confuses the issue.)</p>
<p>Best.</p>
<p>K. Frank</p>