Loss reduction sum vs mean: when to use each?

pumplerod: <p>I’m rather new to pytorch (and NN architecture in general).  While experimenting with my model I see that the various Loss classes for pytorch will accept a <code>reduction</code> parameter (none | sum | mean) for example.  The differences are rather obvious regarding what will be returned, but I’m curious when it would be useful to use sum as opposed to mean?  Does it have an effect on the backprop during training?  Or am I really only choosing between a large loss value or smaller (average) loss value for aesthetic reasons for human readability?</p>

ptrblck: <p>You would not only change the loss scale, but also the gradients:</p>
<pre><code class="lang-python"># setup
model = nn.Linear(10, 10)
x = torch.randn(10, 10)
y = torch.randn(10, 10)

# mean
criterion = nn.MSELoss(reduction='mean')
out = model(x)
loss = criterion(out, y)
loss.backward()
print(model.weight.grad.abs().sum())
&gt; tensor(5.6143)

# sum
model.zero_grad()
criterion = nn.MSELoss(reduction='sum')
out = model(x)
loss = criterion(out, y)
loss.backward()
print(model.weight.grad.abs().sum())
&gt; tensor(561.4255)
</code></pre>
<p>I think the disadvantage in using the <code>sum</code> reduction would also be that the loss scale (and gradients) depend on the batch size, so you would probably need to change the learning rate based on the batch size. While this is surely possibly, a <code>mean</code> reduction would not make this necessary.</p>
<p>On the other hand, the <code>none</code> reduction gives you the flexibility to add any custom operations to the unreduced loss and you would either have to reduce it manually or provide the gradients in the right shape when calling <code>backward</code> on the unreduced loss.</p>

pumplerod: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>At the risk of exposing myself as a complete novice.  Could you provide a layman’s understanding of what the impact of that large gradient has on the training process?  Do I understand that this will essentially propagate a  greater value back when adjusting the weights?  And could this be balanced out with a much smaller learning rate?  If my batch sizes are always the same will there be a noticeable impact on the ability to converge, because I am perhaps taking too large of a step when adjusting the weight values?</p>
<p>This little tidbit has really helped expose a slightly better understanding of the training process, so thank you very much for shining some light.</p>

ptrblck: <aside class="quote no-group" data-username="pumplerod" data-post="3" data-topic="115641">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pumplerod/48/36129_2.png" class="avatar"> pumplerod:</div>
<blockquote>
<p>Do I understand that this will essentially propagate a greater value back when adjusting the weights? And could this be balanced out with a much smaller learning rate?</p>
</blockquote>
</aside>
<p>Yes and yes <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> You could certainly reduce these large gradients by adjusting the learning rate, but note that the learning rate would then depend on the batch size. I.e. if you decide to use a smaller or larger batch size you would also have to change the learning rate.<br>
The first figure in e.g. <a href="https://www.jeremyjordan.me/nn-learning-rate/">this blog post</a> shows the effects of a large weight update on a simple use case.</p>
<aside class="quote no-group" data-username="pumplerod" data-post="3" data-topic="115641">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pumplerod/48/36129_2.png" class="avatar"> pumplerod:</div>
<blockquote>
<p>If my batch sizes are always the same will there be a noticeable impact on the ability to converge, because I am perhaps taking too large of a step when adjusting the weight values?</p>
</blockquote>
</aside>
<p>If you use the same script, adjust the learning rate, I would assume that you should be able to let the model converge to the same final result as with the mean reduction.</p>

pumplerod: <p>Thank you.  I really appreciate your contribution to this community.</p>

chaslie: <p>hi Ptrblck,</p>
<p>I have a similar issue, but when i use MSEloss(reduction=‘sum’) and MSEloss(reduction=‘mean’) I do not see the difference in the calculated loss function as the size of the batch after the first iteration eg:</p>
<pre><code class="lang-auto">criterion = torch.nn.MSELoss(reduction='sum')
criterion(out,real_data_I)
tensor(40190.8242, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;)

criterion = torch.nn.MSELoss()
criterion(out,real_data_I)
tensor(0.2726, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;)
</code></pre>
<p>The batchsize is 9 so i would expect the loss to be either 4,465.647 or 2.4534 depending on which is correct?</p>
<p>I am more suspicious of the 0.276 value as this looks too low for the start og the model, but I don’t understand why the error.</p>
<p>chaslie</p>

ptrblck: <p>I’m not sure I understand the issue correctly, but it seems you are concerned about the correctness of the reduction in the second approach? Could you post the shapes of <code>out</code> and <code>real_data_I</code> so that we could double check the calculation?</p>

chaslie: <p>hi ptrblk,</p>
<p>the shapes are [9,1,128,128].</p>
<p>chaslie</p>

ptrblck: <p>Thanks! Based on this shape, the loss calculation seems to be correct using the <code>reduction='mean'</code> setting:</p>
<pre><code class="lang-python">40190.8242 / (9 * 1 * 128 * 128)
&gt; 0.2725614705403646
</code></pre>

chaslie: <p>I realised that, doh. I thought the difference was based purely on batch size and not the size of the array <img src="https://discuss.pytorch.org/images/emoji/apple/exploding_head.png?v=10" title=":exploding_head:" class="emoji" alt=":exploding_head:"></p>

chaslie: <p>thanks for taking the time out to deal with an idiot…</p>

ptrblck: <p>Ah, c’mon… we are all missing these “trivial” things all the time <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

chaslie: <p>too kind my friend, too kind <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Shreeyak: <p>We could remove the dependency on batch size by dividing the loss by batch size. Would that work?</p>
<pre><code class="lang-auto">criterion = nn.MSELoss(reduction='sum')
out = model(x)
loss = criterion(out, y)
loss = loss / x.shape[0]
</code></pre>

ptrblck: <p>Yes, you could surely divide by the batch size in case you don’t want to divide by the number of elements for specific reasons.</p>

Shreeyak: <p>Thanks! I believe that for dense predictions (such as segmentation or depth estimation), the average value per pixel would not be too meaningful, since it will result in only a small change in value because of averaging so many pixels. It would be hard to capture subtle changes in the prediction. Calculating the value across an entire image should give a much better signal.</p>

ptrblck: <p>I don’t think the interesting difference is the actual range, as you could always increase or decrease the learning rate. The advantage of using the average of all elements would be to get a loss value, which would not depend on the shape (i.e. using a larger or smaller spatial size would yield approx. the same loss values assuming your model is flexible regarding the spatial shape).</p>

Raccoon_2: <p>Both are equal if the dimensions are constant because you can change the learning rate.<br>
But if the output dimensions are variable, both approaches will not be appropriate.<br>
Instead, you need to do this</p>
<pre><code class="lang-auto">loss = F.mse_loss(y_pred,y_true,reduction='none')
loss = (loss/torch.as_tensor(loss.size()).prod().sqrt()).sum()
</code></pre>

Torcione: <p>Hi, does the <code>reduction = 'mean'</code> normalization change between different loss functions?<br>
For example I read <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">here</a> that for cross entropy loss the different normalization between <code>sum</code> and <code>mean</code> <code>reduction</code> is not fixed by the input size but just by the number of element in the batch N (if we set parameter <code>weight=None</code>).<br>
When in the description says:</p>
<blockquote>
<p>and N spans the minibatch dimension</p>
</blockquote>
<p>Does it mean the number of images in a batch or the the number of images multiplied by their dimension?</p>

ptrblck: <p>You’ve copied only part of the description as it mentions:</p>
<blockquote>
<p>and N spans the minibatch dimension as well as d1,…,dk​ for the K-dimensional case</p>
</blockquote>
<p>which means that the <code>mean</code> reduction is dividing by the target shape. A quick test also shows this:</p>
<pre data-code-wrap="python"><code class="lang-python">criterion_mean = nn.CrossEntropyLoss(reduction='mean')
criterion_sum = nn.CrossEntropyLoss(reduction='sum')

output = torch.randn(2, 3, 224, 224)
target = torch.randint(0, 3, (2, 224, 224))

loss_mean = criterion_mean(output, target)
loss_sum = criterion_sum(output, target)

print(loss_mean - (loss_sum / target.nelement()))
# &gt; tensor(0.)
</code></pre>