Mean Reduction in nll_loss

username: coolsuntraveler
cooked: <p>According to nll_loss documentation, for reduction parameter, " <code>'none'</code> : no reduction will be applied,  <code>'mean'</code> : the sum of the output will be divided by the number of elements in the output,  <code>'sum'</code> : the output will be summed."</p>
<p>However, it seems “mean” is divided by the sum of the weights of each element, not number of elements in the output. Am I misunderstanding the documentation or the documentation is inaccurate?</p>
<blockquote>
<blockquote>
<blockquote>
<p>input = torch.randn(3, 5, requires_grad=True)<br>
ls =F.log_softmax(input, 1)<br>
ls<br>
tensor([[-2.5076, -2.2387, -0.4952, -2.2435, -2.3390],<br>
[-0.9897, -1.8469, -1.6642, -1.4942, -2.8676],<br>
[-0.7173, -3.8274, -2.2838, -1.0784, -3.0335]],<br>
grad_fn=)<br>
F.nll_loss(ls, torch.tensor([0,0,4]), reduction=“none”, weight=torch.tensor([1.0, 1.0, 1.0, 1.0, 3.0]))</p>
</blockquote>
</blockquote>
</blockquote>
<p>tensor([2.5076, 0.9897, 9.1005], grad_fn=)</p>
<blockquote>
<blockquote>
<blockquote>
<p>F.nll_loss(ls, torch.tensor([0,0,4]), reduction=“sum”, weight=torch.tensor([1.0, 1.0, 1.0, 1.0, 3.0]))</p>
</blockquote>
</blockquote>
</blockquote>
<p>tensor(12.5978, grad_fn=)</p>
<blockquote>
<blockquote>
<blockquote>
<p>F.nll_loss(ls, torch.tensor([0,0,4]), reduction=“mean”, weight=torch.tensor([1.0, 1.0, 1.0, 1.0, 3.0]))</p>
</blockquote>
</blockquote>
</blockquote>
<p>tensor(2.5196, grad_fn=)</p>

username: KFrank
cooked: <p>Hello Johnson!</p>
<aside class="quote no-group" data-username="coolsuntraveler" data-post="1" data-topic="94314" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/coolsuntraveler/48/16640_2.png" class="avatar"> coolsuntraveler:</div>
<blockquote>
<p>However, it seems “mean” is divided by the sum of the weights of each element, not number of elements in the output. Am I misunderstanding the documentation or the documentation is inaccurate?</p>
</blockquote>
</aside>
<p>You are correct that in the weighted case, “mean” is calculated by<br>
dividing by the sum of the weights.  (In other words, a <em>weighted<br>
average</em> is being calculated.)</p>
<p>The documentation on this is arguably inaccurate, or at least rather<br>
unclear.</p>
<p>Anyway, the way I think of it, if I have ten samples, all with the same<br>
loss, say, 1.23, I would like my “mean” loss, weighted or not, to be<br>
this same loss value, 1.23.  So dividing by the sum of the weights is,<br>
for me, the “expected behavior,” even if the documentation says<br>
otherwise.</p>
<p>(This issue has come up before on the forum, although I don’t have<br>
any references at hand.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: coolsuntraveler
cooked: <blockquote>
<p>Anyway, the way I think of it, if I have ten samples, all with the same<br>
loss, say, 1.23, I would like my “mean” loss, weighted or not, to be<br>
this same loss value, 1.23. So dividing by the sum of the weights is,<br>
for me, the “expected behavior,” even if the documentation says<br>
otherwise.</p>
</blockquote>
<p>Hello Frank, I think the example you gave is actually the expected behavior as described in the documentation. The mean should be divided by the number of samples, not by the sum of the weights of the expected class. That is what I expected from reading the documentation.</p>

username: KFrank
cooked: <p>Hello Johnson!</p>
<aside class="quote no-group" data-username="coolsuntraveler" data-post="3" data-topic="94314" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/coolsuntraveler/48/16640_2.png" class="avatar"> coolsuntraveler:</div>
<blockquote>
<p>I think the example you gave is actually the expected behavior as described in the documentation. The mean should be divided by the number of samples, not by the sum of the weights of the expected class.</p>
</blockquote>
</aside>
<p>No, I think you have my example backwards.</p>
<p>Let’s say that you have three classes, <code>0, 1, 2</code>.  Let’s say you also<br>
have three samples, one from each class, in order.  Now suppose<br>
that the (unweighted) per-sample loss happens to be the same for<br>
all three samples, <code>1.23</code>.</p>
<p>If you pass in a weight vector (class weights) of <code>[1.0, 1.0, 3.0]</code>,<br>
then after weighting, but before taking the “mean,” you will have a<br>
summed, weighted loss of<br>
<code>1.0 * 1.23 + 1.0 * 1.23 + 3.0 * 1.23  =  5.0 * 1.23  =  6.15</code></p>
<p>If you divide by the number of samples, <code>3</code>, you will get <code>2.05</code>.  If you<br>
divide by the sum of the weights, <code>5.0</code>, you will get <code>1.23</code>, the loss<br>
value, that in this example, happens to be the same for each of the<br>
three samples.</p>
<p>If I “average” together a bunch of numbers that all happen to have<br>
the same value, to me it makes sense that the average should be<br>
that value.  That’s what I meant by saying that this is my “expected<br>
behavior” (regardless of the documentation, which I do think is<br>
inaccurate).</p>
<p>Best.</p>
<p>K. Frank</p>

username: coolsuntraveler
cooked: <p>Thanks for the explanation K. Frank.</p>