Repository: pytorch/pytorch

Issue:
The loss computation with `size_average` should average over batch example or batch element ?
Traditionally, when we have a batched data with shape `(N, D)` where `N` is batch size and `D` is data dimension. The losses are often calculated for each training example say
```
L_i = loss(X_i), i = 1, ..., N
```
And then total loss is averaged over the batch size
```
L = (1/N)*sum(L_i)
```

However, it seems it is not what `nn.*Loss` is doing for the flag `size_average=True/False`

e.g. 

```python
import torch
import torch.nn.functional as F

input = torch.randn(3, 2)
target = torch.rand(3, 2)

print(input)
print(target)

full_loss = F.mse_loss(input, target, reduce=False)
print(full_loss)
loss_sum = F.mse_loss(input, target, size_average=False)
loss_mean = F.mse_loss(input, target, size_average=True)
print(loss_sum)
print(loss_mean)

batch_loss = full_loss.sum(dim=-1)
correct_loss_mean = batch_loss.mean()
correct_loss_sum = batch_loss.sum()
print(correct_loss_mean)
print(correct_loss_sum)

assert correct_loss_sum == loss_sum
assert correct_loss_mean == loss_mean
```
it seems the `size_average` does not properly average the loss over the batch of examples, but averaged over all dimensions. 


Discussion:
`size_average` averages over "each atomic element for which loss is computed for". For `mse_loss` `size_average` divides by _all_ elements. For something like `NLLLoss`, `size_average` divides by number of minibatches (`tensor.size(0)`) because each row in the tensor results in a loss.

We'll definitely make the `size_average` behavior clearer in the docs: this question has come up in a number of issues in the past. If you'd like different behavior, you can set `size_average=False` and divide by the number of batches.
@zou3519 Thanks a lot for the explanation. Just to make sure I don't take it wrong. By setting `size_average=True` by default to average over all elements rather than firstly summing up losses for each training example and then to average. This does not alter the gradient direction but indeed a smaller scale of the gradient. Will it have potential effects to slow down the training ? Because averaging over all elements gets smaller loss, leading to smaller gradient norm. 
We should at some point consider if we want to change this behavior. This is indeed confusing and in some cases doesn't do what you want (KL Divergence for example https://github.com/pytorch/pytorch/pull/7006).
Breaking everyones code is not a good thing, but maybe we could deprecate the `size_average` argument in favor of a new one, that consistently divides by the batch size?
@fmassa I'm a bit afraid that adding another variant of `size_average` is going to make the interface very confusing.
@apaszke I think it might be worth listing the cases where `size_average` is actually doing what most people want it to do. Maybe the current behavior of `size_average` is already good enough.

That being said, now that we have `reduce=False` for all losses (and once we get reduction over multiple dimensions merged in master), it will be super easy to let users control what they want to do.
@fmassa It seems to me that the more common behavior of `size_average` should be firstly summing up the losses across data dimension and then averaging over batch. But yes, as you pointed out, the flag `reduce=False` provides user full flexibility. 

I guess it might be good either to mention clearly to the user about the current behavior of `size_average` is to average over all elements, or change the behavior of `size_average` to the one described above which is more commonly used to train NN models and thus more user-friendly, and then the 'element-wise' averaging behavior leave to the user. 

@apaszke 
Or maybe have one more flag, say `size_average_batch` and `size_average_all` to provide both behaviors. 
Does it make more sense to average over batch always? It seems like that's the functionality that most users expect and want.

Resolving patch:
diff --git a/torch/nn/modules/loss.py b/torch/nn/modules/loss.py
--- a/torch/nn/modules/loss.py
+++ b/torch/nn/modules/loss.py
@@ -53,13 +53,14 @@ class L1Loss(_Loss):
 
     Args:
         size_average (bool, optional): By default, the losses are averaged
-           over observations for each minibatch. However, if the field
-           size_average is set to ``False``, the losses are instead summed for
-           each minibatch. Ignored when reduce is ``False``. Default: ``True``
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed
-           for each minibatch. When reduce is ``False``, the loss function returns
-           a loss per input/target element instead and ignores size_average.
-           Default: ``True``
+            for each minibatch. When reduce is ``False``, the loss function returns
+            a loss per input/target element instead and ignores size_average.
+            Default: ``True``
 
     Shape:
         - Input: :math:`(N, *)` where `*` means, any number of additional
@@ -131,13 +132,13 @@ class NLLLoss(_WeightedLoss):
 
     Args:
         weight (Tensor, optional): a manual rescaling weight given to each
-           class. If given, it has to be a Tensor of size `C`. Otherwise, it is
-           treated as if having all ones.
+            class. If given, it has to be a Tensor of size `C`. Otherwise, it is
+            treated as if having all ones.
         size_average (bool, optional): By default, the losses are averaged
-           over observations for each minibatch with weights set by
-           :attr:`weight`. However, if the field :attr:`size_average` is set to
-           ``False``, the losses are instead summed for each minibatch. Ignored
-           when :attr:`reduce` is ``False``. Default: ``True``
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         ignore_index (int, optional): Specifies a target value that is ignored
             and does not contribute to the input gradient. When
             :attr:`size_average` is ``True``, the loss is averaged over
@@ -225,9 +226,11 @@ class PoissonNLLLoss(_Loss):
 
             .. math::
                 \text{target}*\log(\text{target}) - \text{target} + 0.5 * \log(2\pi\text{target}).
-        size_average (bool, optional): By default, the losses are averaged over
-            observations for each minibatch. However, if the field `size_average`
-            is set to ``False``, the losses are instead summed for each minibatch.
+        size_average (bool, optional): By default, the losses are averaged
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         eps (float, optional): Small value to avoid evaluation of :math:`\log(0)` when
             :attr:`log_input == False`. Default: 1e-8
         reduce (bool, optional): By default, the losses are averaged
@@ -312,8 +315,10 @@ class KLDivLoss(_Loss):
 
     Args:
         size_average (bool, optional): By default, the losses are averaged
-            for each minibatch over observations **as well as** over
-            dimensions. However, if ``False`` the losses are instead summed.
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged
             over observations for each minibatch, or summed, depending on
             size_average. When reduce is ``False``, returns a loss per input/target
@@ -363,9 +368,10 @@ class MSELoss(_Loss):
 
     Args:
         size_average (bool, optional): By default, the losses are averaged
-           over observations for each minibatch. However, if the field
-           size_average is set to ``False``, the losses are instead summed for
-           each minibatch. Only applies when reduce is ``True``. Default: ``True``
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged
            over observations for each minibatch, or summed, depending on
            size_average. When reduce is ``False``, returns a loss per input/target
@@ -419,9 +425,10 @@ class BCELoss(_WeightedLoss):
             of each batch element. If given, has to be a Tensor of size
             "nbatch".
         size_average (bool, optional): By default, the losses are averaged
-            over observations for each minibatch. However, if the field
-            size_average is set to ``False``, the losses are instead summed for
-            each minibatch. Default: ``True``
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on size_average. When reduce
             is False, returns a loss per input/target element instead and ignores
@@ -483,9 +490,10 @@ class BCEWithLogitsLoss(_Loss):
             of each batch element. If given, has to be a Tensor of size
             "nbatch".
         size_average (bool, optional): By default, the losses are averaged
-            over observations for each minibatch. However, if the field
-            size_average is set to ``False``, the losses are instead summed for
-            each minibatch. Default: ``True``
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on size_average. When reduce
             is False, returns a loss per input/target element instead and ignores
@@ -547,10 +555,11 @@ class HingeEmbeddingLoss(_Loss):
 
     Args:
         margin (float, optional): Has a default value of `1`.
-        size_average (bool, optional): By default, the losses are averaged over
-            observations for each minibatch. However, if the field :attr:`size_average`
-            is set to ``False``, the losses are instead summed for each minibatch.
-            Default: ``True``
+        size_average (bool, optional): By default, the losses are averaged
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on :attr:`size_average`. When
             :attr:`reduce` is ``False``, returns a loss per batch element instead and
@@ -591,10 +600,11 @@ class MultiLabelMarginLoss(_Loss):
     This allows for different samples to have variable amounts of target classes
 
     Args:
-        size_average (bool, optional): By default, the losses are averaged over
-            observations for each minibatch. However, if the field :attr:`size_average`
-            is set to ``False``, the losses are instead summed for each minibatch.
-            Default: ``True``
+        size_average (bool, optional): By default, the losses are averaged
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on :attr:`size_average`. When
             :attr:`reduce` is ``False``, returns a loss per batch element instead and
@@ -641,12 +651,14 @@ class SmoothL1Loss(_Loss):
 
     Args:
         size_average (bool, optional): By default, the losses are averaged
-           over all elements. However, if the field size_average is set to ``False``,
-           the losses are instead summed. Ignored when reduce is ``False``. Default: ``True``
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed
-           over elements. When reduce is ``False``, the loss function returns
-           a loss per input/target element instead and ignores size_average.
-           Default: ``True``
+            over elements. When reduce is ``False``, the loss function returns
+            a loss per input/target element instead and ignores size_average.
+            Default: ``True``
 
     Shape:
         - Input: :math:`(N, *)` where `*` means, any number of additional
@@ -674,10 +686,11 @@ class SoftMarginLoss(_Loss):
         \text{loss}(x, y) = \sum_i \frac{\log(1 + \exp(-y[i]*x[i]))}{\text{x.nelement}()}
 
     Args:
-        size_average (bool, optional): By default, the losses are averaged over
-            observations for each minibatch. However, if the field :attr:`size_average`
-            is set to ``False``, the losses are instead summed for each minibatch.
-            Default: ``True``
+        size_average (bool, optional): By default, the losses are averaged
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on :attr:`size_average`. When
             :attr:`reduce` is ``False``, returns a loss per batch element instead and
@@ -737,9 +750,11 @@ class CrossEntropyLoss(_WeightedLoss):
     Args:
         weight (Tensor, optional): a manual rescaling weight given to each class.
            If given, has to be a Tensor of size `C`
-        size_average (bool, optional): By default, the losses are averaged over observations for each minibatch.
-           However, if the field `size_average` is set to ``False``, the losses are
-           instead summed for each minibatch. Ignored if reduce is ``False``.
+        size_average (bool, optional): By default, the losses are averaged
+           over each loss element in the batch. Note that for some losses, there
+           multiple elements per sample. If the field size_average is set to
+           ``False``, the losses are instead summed for each minibatch. Ignored
+           when reduce is ``False``. Default: ``True``
         ignore_index (int, optional): Specifies a target value that is ignored
             and does not contribute to the input gradient. When `size_average` is
             ``True``, the loss is averaged over non-ignored targets.
@@ -792,12 +807,13 @@ class MultiLabelSoftMarginLoss(_WeightedLoss):
 
     Args:
         weight (Tensor, optional): a manual rescaling weight given to each
-           class. If given, it has to be a Tensor of size `C`. Otherwise, it is
-           treated as if having all ones.
-        size_average (bool, optional): By default, the losses are averaged over
-            observations for each minibatch. However, if the field :attr:`size_average`
-            is set to ``False``, the losses are instead summed for each minibatch.
-            Default: ``True``
+            class. If given, it has to be a Tensor of size `C`. Otherwise, it is
+            treated as if having all ones.
+        size_average (bool, optional): By default, the losses are averaged
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on :attr:`size_average`. When
             :attr:`reduce` is ``False``, returns a loss per batch element instead and
@@ -836,10 +852,11 @@ class CosineEmbeddingLoss(_Loss):
     Args:
         margin (float, optional): Should be a number from `-1` to `1`, `0` to `0.5`
             is suggested. If `margin` is missing, the default value is `0`.
-        size_average (bool, optional): By default, the losses are averaged over
-            observations for each minibatch. However, if the field :attr:`size_average`
-            is set to ``False``, the losses are instead summed for each minibatch.
-            Default: ``True``
+        size_average (bool, optional): By default, the losses are averaged
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on :attr:`size_average`. When
             :attr:`reduce` is ``False``, returns a loss per batch element instead and
@@ -870,10 +887,11 @@ class MarginRankingLoss(_Loss):
 
     Args:
         margin (float, optional): Has a default value of `0`.
-        size_average (bool, optional): By default, the losses are averaged over
-            observations for each minibatch. However, if the field :attr:`size_average`
-            is set to ``False``, the losses are instead summed for each minibatch.
-            Default: ``True``
+        size_average (bool, optional): By default, the losses are averaged
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on :attr:`size_average`. When
             :attr:`reduce` is ``False``, returns a loss per batch element instead and
@@ -923,10 +941,11 @@ class MultiMarginLoss(_WeightedLoss):
         weight (Tensor, optional): a manual rescaling weight given to each
             class. If given, it has to be a Tensor of size `C`. Otherwise, it is
             treated as if having all ones.
-        size_average (bool, optional): By default, the losses are averaged over
-            observations for each minibatch. However, if the field :attr:`size_average`
-            is set to ``False``, the losses are instead summed for each minibatch.
-            Default: ``True``
+        size_average (bool, optional): By default, the losses are averaged
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on :attr:`size_average`. When
             :attr:`reduce` is ``False``, returns a loss per batch element instead and
@@ -972,10 +991,11 @@ class TripletMarginLoss(_Loss):
         swap (float, optional): The distance swap is described in detail in the paper
             `Learning shallow convolutional feature descriptors with triplet losses` by
             V. Balntas, E. Riba et al. Default: ``False``.
-        size_average (bool, optional): By default, the losses are averaged over
-            observations for each minibatch. However, if the field :attr:`size_average`
-            is set to ``False``, the losses are instead summed for each minibatch.
-            Default: ``True``
+        size_average (bool, optional): By default, the losses are averaged
+            over each loss element in the batch. Note that for some losses, there
+            multiple elements per sample. If the field :attr:`size_average` is set to
+            ``False``, the losses are instead summed for each minibatch. Ignored
+            when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on :attr:`size_average`. When
             :attr:`reduce` is ``False``, returns a loss per batch element instead and


Test patch: