Weird (?) reduction behavior of CrossEntropyLoss + Weights

username: ppmt
cooked: <p>I am trying to train a binary classifier with Pytorch v1.3.1 on time series data using a LSTM with CrossEntropyLoss as loss function. Since the dataset is very unbalanced (~98% Class 0, ~2% Class 1), I want to apply weights [1., 55.] in the loss function. The training is done in batches of 24. Using a random example, where one of the 24 samples is from class 1 and the 23 remaining from class 0, I get the following losses:</p>
<p>reduction = ‘mean’, no weights: 0.7963<br>
reduction = ‘sum’, no weights: 19.11<br>
reduction = ‘none’, no weights, manually summing losses: 19.11</p>
<p>I would expect that the averages loss times the batch size equals the summed loss. Here, 24*0.7963 equals 18.96 which is not identical but close to the summed loss.</p>
<p>reduction = ‘mean’, weights in loss fct: 0.658<br>
reduction = ‘sum’, weights in loss fct: 51.48<br>
reduction = ‘none’, weights in loss fct, manually summing losses: 51.48</p>
<p>Now 0.658 * 24 (=15.79) is totally different from 51.48. Is this the intended behavior?<br>
How does Pytorch compute the average? Is there any difference in training behavior when using the sum instead of the mean reduction?</p>

username: KFrank
cooked: <p>Hello Peppermint!</p>
<aside class="quote no-group" data-username="ppmt" data-post="1" data-topic="73623" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/779978/48.png" class="avatar"> ppmt:</div>
<blockquote>
<p>reduction = ‘mean’, no weights: 0.7963<br>
reduction = ‘sum’, no weights: 19.11<br>
reduction = ‘none’, no weights, manually summing losses: 19.11</p>
<p>I would expect that the averages loss times the batch size equals the summed loss. Here, 24*0.7963 equals 18.96 which is not identical but close to the summed loss.</p>
</blockquote>
</aside>
<p>Actually, everything looks good here.  According to my calculator,<br>
<code>24* 0.7963 = 19.1112</code>.  (As an aside, <code>24 * 0.79 = 18.96</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>reduction = ‘mean’, weights in loss fct: 0.658<br>
reduction = ‘sum’, weights in loss fct: 51.48<br>
reduction = ‘none’, weights in loss fct, manually summing losses: 51.48</p>
<p>Now 0.658 * 24 (=15.79) is totally different from 51.48. Is this the intended behavior?</p>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group">
<blockquote>
<p>How does Pytorch compute the average?</p>
</blockquote>
</aside>
<p><code>CrossEntropyLoss</code> computes a <em>weighted</em> average using your weights.</p>
<p>In your example, you have 23 weights of value 1, and 1 weight of value<br>
55.  So the sum of your weights is <code>23 * 1 + 1 * 55 = 78</code>.  So the<br>
denominator or the weighted average is 78 (not 24).</p>
<p>According to my calculator <code>78 * 0.658 = 51.324</code> (which is good<br>
enough for ranch work).</p>
<p>Here is pytorch 0.3.0 script that illustrates the unweighted and weighted<br>
<code>CrossEntropyLoss</code> “mean”:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

torch.manual_seed (2020)

pred = torch.autograd.Variable (torch.randn ((24, 2)))
targ = torch.autograd.Variable (torch.bernoulli (0.11 * torch.ones(24)).long())
targ.sum()

torch.nn.CrossEntropyLoss (weight = None, size_average =  True, reduce =  True) (pred, targ) * len (targ)
torch.nn.CrossEntropyLoss (weight = None, size_average = False, reduce =  True) (pred, targ)
torch.nn.CrossEntropyLoss (weight = None, size_average = False, reduce = False) (pred, targ).sum()

wt = torch.FloatTensor ([1.0, 55.0])

torch.nn.CrossEntropyLoss (weight = wt, size_average =  True, reduce =  True) (pred, targ) * wt[targ.data].sum()
torch.nn.CrossEntropyLoss (weight = wt, size_average = False, reduce =  True) (pred, targ)
torch.nn.CrossEntropyLoss (weight = wt, size_average = False, reduce = False) (pred, targ).sum()
</code></pre>
<p>And here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x000001E4CA336630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; pred = torch.autograd.Variable (torch.randn ((24, 2)))
&gt;&gt;&gt; targ = torch.autograd.Variable (torch.bernoulli (0.11 * torch.ones(24)).long())
&gt;&gt;&gt; targ.sum()
Variable containing:
 3
[torch.LongTensor of size 1]

&gt;&gt;&gt;
&gt;&gt;&gt; torch.nn.CrossEntropyLoss (weight = None, size_average =  True, reduce =  True) (pred, targ) * len (targ)
Variable containing:
 22.5843
[torch.FloatTensor of size 1]

&gt;&gt;&gt; torch.nn.CrossEntropyLoss (weight = None, size_average = False, reduce =  True) (pred, targ)
Variable containing:
 22.5843
[torch.FloatTensor of size 1]

&gt;&gt;&gt; torch.nn.CrossEntropyLoss (weight = None, size_average = False, reduce = False) (pred, targ).sum()
Variable containing:
 22.5843
[torch.FloatTensor of size 1]

&gt;&gt;&gt;
&gt;&gt;&gt; wt = torch.FloatTensor ([1.0, 55.0])
&gt;&gt;&gt;
&gt;&gt;&gt; torch.nn.CrossEntropyLoss (weight = wt, size_average =  True, reduce =  True) (pred, targ) * wt[targ.data].sum()
Variable containing:
 302.9537
[torch.FloatTensor of size 1]

&gt;&gt;&gt; torch.nn.CrossEntropyLoss (weight = wt, size_average = False, reduce =  True) (pred, targ)
Variable containing:
 302.9537
[torch.FloatTensor of size 1]

&gt;&gt;&gt; torch.nn.CrossEntropyLoss (weight = wt, size_average = False, reduce = False) (pred, targ).sum()
Variable containing:
 302.9537
[torch.FloatTensor of size 1]
</code></pre>
<p>As you can see the results of the various “reductions” all tie out (when<br>
the average is understood as a weighted average).</p>
<aside class="quote no-group">
<blockquote>
<p>Is there any difference in training behavior when using the sum instead of the mean reduction?</p>
</blockquote>
</aside>
<p>Not a lot.  In the unweighted case, the loss using “sum” will be <code>nBatch</code><br>
times bigger (24, in your example) than with “mean.”  In effect, this<br>
means that you would be using a learning rate 24 time bigger.</p>
<p>Learning rate does matter, so you would probably want to use a<br>
proportionately smaller learning rate if you use “sum” rather than<br>
“mean.”</p>
<p>In the weighted case, the ratio between “sum” and “mean” will depend<br>
on the number of “1” target values in your batch (as illustrated by the<br>
above script).  So your effective learning rate would vary from batch<br>
to batch based on the number of "1"s.  But I think this batch-to-batch<br>
variation would largely average away over multiple batches.</p>
<p>One last note:  Because you’re training a binary classifier, you would<br>
probably be modestly better off using <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a> and its<br>
<code>pos_weight</code> argument.  (To do this you would change the last layer<br>
of you network to emit only a single prediction (per sample in the<br>
batch), rather than two.)</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: ppmt
cooked: <p>Thank you very much for your detailed explanation and further suggestions!</p>