Weighted Cross entropy is giving the same result as cross entropy when reduction='mean'

Dhorka: <p>Hi,</p>
<p>I was trying the weight parameter of the cross-entropy loss, and I observed that I’m achieving the same results when I’m using the default reduction parameter:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch

&gt;&gt;&gt; input = torch.randn(1, 5, requires_grad=True)
tensor([[ 1.2900,  0.4519, -0.2622, -0.5862, -0.5304]], requires_grad=True)

&gt;&gt;&gt; target = torch.empty(1, dtype=torch.long).random_(5)
tensor([0])

&gt;&gt;&gt; weights = torch.randn(5).abs()
tensor([1.4737, 0.8730, 0.7921, 0.2246, 1.5051])

&gt;&gt;&gt; cross_entropy = torch.nn.CrossEntropyLoss()
&gt;&gt;&gt; cross_entropy_weighted = torch.nn.CrossEntropyLoss(weight=weights)

&gt;&gt;&gt; cross_entropy(input,target)
tensor(0.6727, grad_fn=&lt;NllLossBackward&gt;)

&gt;&gt;&gt; cross_entropy_weighted(input,target)
tensor(0.6727, grad_fn=&lt;NllLossBackward&gt;)
</code></pre>
<p>Am I doing the weighting correctly? How is it supposed to be used the weight parameter of cross-entropy loss?</p>
<p>Thanks</p>

KFrank: <p>Hi Dhorka!</p>
<aside class="quote no-group" data-username="Dhorka" data-post="1" data-topic="85433" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/b2d939/48.png" class="avatar"> Dhorka:</div>
<blockquote>
<p>I observed that I’m achieving the same results</p>
</blockquote>
</aside>
<p>The short answer is that your batch size is 1, so you won’t see a<br>
difference.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">&gt;&gt;&gt; input = torch.randn(1, 5, requires_grad=True)
tensor([[ 1.2900,  0.4519, -0.2622, -0.5862, -0.5304]], requires_grad=True)

&gt;&gt;&gt; target = torch.empty(1, dtype=torch.long).random_(5)
tensor([0])
</code></pre>
<p>Am I doing the weighting correctly? How is it supposed to be used the weight parameter of cross-entropy loss?</p>
</blockquote>
</aside>
<p>Yes, you’re doing the weighting correctly and the result is correct.</p>
<p>But <code>CrossEntropyLoss</code> calculates a <em>weighted average</em> when using<br>
class weights (and the default <code>reduction = 'mean'</code>).  So when you<br>
take the weighted average over a single sample, the weight drops out.</p>
<p>Try this again with a batch with a size of more than one (and make<br>
sure that the batch contains samples with classes with different<br>
weights).  Then the values of weighted and unweighted loss functions<br>
will differ.</p>
<p>Best.</p>
<p>K. Frank</p>

Dhorka: <p>Thanks for your clarification! I suppose that should be mentioned in the documentation <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> At this moment the documentation says:</p>
<blockquote>
<p>‘mean’: the sum of the output will be divided by the number of elements in the output,</p>
</blockquote>
<p>That was the source of my misunderstanding.</p>

KFrank: <p>Hello Dhorka!</p>
<aside class="quote no-group quote-modified" data-username="Dhorka" data-post="3" data-topic="85433" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/b2d939/48.png" class="avatar"> Dhorka:</div>
<blockquote>
<p>At this moment the documentation says:</p>
<blockquote>
<p>‘mean’: the sum of the output will be divided by the number of elements in the output,</p>
</blockquote>
</blockquote>
</aside>
<p>Yes, you’re right about this.  Read literally, the current documentation<br>
for <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a> is incorrect.</p>
<p>The line you cite is wrong (or, at best, imprecise or incomplete).</p>
<p>Also, when read strictly, this passage:</p>
<blockquote>
<p>or in the case of the weight argument being specified:</p>
<p>{loss}(x, class) = weight[class] (…)</p>
<p>The losses are averaged across observations for each minibatch.</p>
</blockquote>
<p>is incorrect because it doesn’t show the final reduced loss being<br>
divided by the sum of the weights.</p>
<p>I believe I’ve seen this issue – whether the average for weighted<br>
loss is divided by the sum of the weights – come up in this forum<br>
before.  (I don’t have the references off hand.)</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>, would it make sense to have the Documentation Mavens<br>
clarify this?   (Also, it looks like <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss" rel="noopener nofollow ugc">BCELoss</a> and <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a><br>
get this wrong, while <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss" rel="noopener nofollow ugc">NLLLoss</a> gets it right.)</p>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="85433">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I believe I’ve seen this issue – whether the average for weighted<br>
loss is divided by the sum of the weights – come up in this forum<br>
before. (I don’t have the references off hand.)</p>
</blockquote>
</aside>
<p>Yes, I ran into this issue before and had to manually check, how the weights are used in the reduction settings.<br>
A fix would be very welcome. Let me know, if you would be interested in creating the PR. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>