Weights in NllLoss behave unexpectedly

AwesomeLemon: <p>Hello,</p>
<p>I’m having trouble understanding behaviour of class weights in CrossEntropyLoss.</p>
<p>Specifically, when reduction=‘mean’. I test it like this:</p>
<blockquote>
<p>input = torch.randn(5, 2, requires_grad=True)<br>
m = nn.LogSoftmax(dim=1)<br>
mi = m(input)</p>
<p>target = torch.tensor([0, 0, 1, 1, 1])<br>
w = torch.tensor([1, 100]).float()</p>
</blockquote>
<p>Now, without weights everything behaves reasonably, next 2 lines give the same result</p>
<blockquote>
<p>F.nll_loss(mi, target, reduction=‘none’).mean()<br>
F.nll_loss(mi, target, reduction=‘mean’)</p>
</blockquote>
<p>However, once we introduce weight, results change</p>
<blockquote>
<p>F.nll_loss(mi, target, weight=w, reduction=‘none’).mean()<br>
F.nll_loss(mi, target, weight=w, reduction=‘mean’)</p>
</blockquote>
<p>That is extremely unintuitive to me, what is the logic behind this?<br>
The formula in documentation (<a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html" class="inline-onebox" rel="noopener nofollow ugc">NLLLoss — PyTorch 2.1 documentation</a>) is no help, because it reuses variable name “n”, making it seem as if weights are the same for all samples. Are they?</p>
<p>I’ll apreciate any help</p>

KFrank: <p>Hello Awesome!</p>
<aside class="quote no-group quote-modified" data-username="AwesomeLemon" data-post="1" data-topic="93116" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/awesomelemon/48/21583_2.png" class="avatar"> AwesomeLemon:</div>
<blockquote>
<p>However, once we introduce weight, results change</p>
<blockquote>
<p>F.nll_loss(mi, target, weight=w, reduction=‘none’).mean()<br>
F.nll_loss(mi, target, weight=w, reduction=‘mean’)</p>
</blockquote>
</blockquote>
</aside>
<p>Passing weights to <code>NLLLoss</code> (and <code>CrossEntropyLoss</code>) gives, with<br>
<code>reduction = 'mean'</code>, a <em>weighted</em> average where the sum of weighted<br>
values is then divided by the sum of the weights.</p>
<p>In your `reduction = ‘none’ version:</p>
<pre data-code-wrap="python"><code class="lang-python">F.nll_loss(mi, target, weight=w, reduction=‘none’).mean()
</code></pre>
<p>by the time you get to the call to pytorch’s tensor <code>.mean()</code>, the weights<br>
are no longer available, so <code>.mean()</code> cannot divide by the sum of the<br>
weights.</p>
<p>An example script illustrating this can be found in this post:</p>
<aside class="quote quote-modified" data-post="2" data-topic="78256">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/weight-in-cross-entropy-loss/78256/2">Weight in cross entropy loss</a> 
  </div>
  <blockquote>
    Hello Mainul! 

When using CrossEntropyLoss (weight = sc) with class weights 
to perform the default reduction = 'mean', the average loss that 
is calculated is the weighted average.  That is, you should be dividing 
by the sum of the weights used for the samples, rather than by the 
number of samples. 
The following (pytorch version 0.3.0) script illustrates this: 
import torch
torch.__version__

sc = torch.FloatTensor ([0.4,0.36])
loss = torch.nn.CrossEntropyLoss (weight = sc)
input = torch.au…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

AwesomeLemon: <p>Hi KFrank,</p>
<p>Thank you, I get it now. Seems kinda weird to me, but I also understand the reasoning behind this.</p>