Repository: pytorch/pytorch

Issue:
MSEloss with reduce=False returns a tensor of same size as the input and target  (rather than just one loss per batch element)
MSELoss with reduce=False doesn't behave as documented.
```
loss = nn.MSELoss(reduce=False)
input = autograd.Variable(torch.randn(3, 5)) # 3 elements in batch of size 5
target = autograd.Variable(torch.randn(3, 5))
output = loss(input, target)
print(output)
```
As documented, this should return one loss for each of the 3 batch elements. Instead, it returns:
```
Variable containing:
  5.4552   2.3369   3.3013   0.0396  13.2650
  1.4288   0.4156   0.3027   0.3006   0.2276
  7.6768   1.0778   0.1317   1.9284   3.4640
[torch.FloatTensor of size 3x5]
```

A similar setup with with CrossEntropyLoss(reduce=False) instead of MSELoss correctly returns a FloatTensor of size 3.


Discussion:

Resolving patch:
diff --git a/torch/nn/functional.py b/torch/nn/functional.py
--- a/torch/nn/functional.py
+++ b/torch/nn/functional.py
@@ -1258,7 +1258,7 @@ def poisson_nll_loss(input, target, log_input=True, full=False, size_average=Tru
         reduce (bool, optional): By default, the losses are averaged
             over observations for each minibatch, or summed, depending on
             size_average. When reduce is ``False``, returns a loss per batch
-            element instead and ignores size_average. Default: ``True``
+            instead and ignores size_average. Default: ``True``
     """
     if log_input:
         loss = torch.exp(input) - target * input
@@ -1288,7 +1288,7 @@ def poisson_nll_loss(input, target, log_input=True, full=False, size_average=Tru
         in input tensor. Default: ``True``
     reduce (bool, optional): By default, the losses are averaged
         over observations for each minibatch, or summed, depending on
-        size_average. When reduce is False, returns a loss per batch
+        size_average. When reduce is False, returns a loss per input/target
         element instead and ignores size_average. Default: ``True``
 
 """)
@@ -1315,7 +1315,7 @@ def cross_entropy(input, target, weight=None, size_average=True, ignore_index=-1
                 True, the loss is averaged over non-ignored targets. Default: -100
         reduce (bool, optional): By default, the losses are averaged or summed over
                 observations for each minibatch depending on size_average. When reduce
-                is False, returns a loss per batch element instead and ignores
+                is False, returns a loss per batch instead and ignores
                 size_average. Default: ``True``
 
     Examples::
@@ -1345,7 +1345,7 @@ def binary_cross_entropy(input, target, weight=None, size_average=True, reduce=T
                 for each minibatch. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
                 observations for each minibatch depending on size_average. When reduce
-                is False, returns a loss per batch element instead and ignores
+                is False, returns a loss per input/target element instead and ignores
                 size_average. Default: True
 
     Examples::
@@ -1388,7 +1388,7 @@ def binary_cross_entropy_with_logits(input, target, weight=None, size_average=Tr
                 for each minibatch. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
                 observations for each minibatch depending on size_average. When reduce
-                is False, returns a loss per batch element instead and ignores
+                is False, returns a loss per input/target element instead and ignores
                 size_average. Default: True
 
     Examples::
diff --git a/torch/nn/modules/loss.py b/torch/nn/modules/loss.py
--- a/torch/nn/modules/loss.py
+++ b/torch/nn/modules/loss.py
@@ -58,7 +58,7 @@ class L1Loss(_Loss):
            each minibatch. Ignored when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed
            for each minibatch. When reduce is ``False``, the loss function returns
-           a loss per batch element instead and ignores size_average.
+           a loss per input/target element instead and ignores size_average.
            Default: ``True``
 
     Shape:
@@ -145,7 +145,7 @@ class NLLLoss(_WeightedLoss):
             non-ignored targets.
         reduce (bool, optional): By default, the losses are averaged or summed
             for each minibatch. When :attr:`reduce` is ``False``, the loss
-            function returns a loss per batch element instead and
+            function returns a loss per batch instead and
             ignores :attr:`size_average`. Default: ``True``
 
     Shape:
@@ -229,7 +229,7 @@ class PoissonNLLLoss(_Loss):
             log_input==``False``. Default: 1e-8
         reduce (bool, optional): By default, the losses are averaged
             over observations for each minibatch, or summed, depending on
-            size_average. When reduce is ``False``, returns a loss per batch
+            size_average. When reduce is ``False``, returns a loss per input/target
             element instead and ignores size_average. Default: ``True``
 
     Examples::
@@ -294,7 +294,7 @@ class KLDivLoss(_Loss):
             dimensions. However, if ``False`` the losses are instead summed.
         reduce (bool, optional): By default, the losses are averaged
             over observations for each minibatch, or summed, depending on
-            size_average. When reduce is ``False``, returns a loss per batch
+            size_average. When reduce is ``False``, returns a loss per input/target
             element instead and ignores size_average. Default: ``True``
 
     Shape:
@@ -350,7 +350,7 @@ class MSELoss(_Loss):
            each minibatch. Only applies when reduce is ``True``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged
            over observations for each minibatch, or summed, depending on
-           size_average. When reduce is ``False``, returns a loss per batch
+           size_average. When reduce is ``False``, returns a loss per input/target
            element instead and ignores size_average. Default: ``True``
 
     Shape:
@@ -407,7 +407,7 @@ class BCELoss(_WeightedLoss):
             each minibatch. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on size_average. When reduce
-            is False, returns a loss per batch element instead and ignores
+            is False, returns a loss per input/target element instead and ignores
             size_average. Default: True
 
     Shape:
@@ -472,7 +472,7 @@ class BCEWithLogitsLoss(Module):
             each minibatch. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on size_average. When reduce
-            is False, returns a loss per batch element instead and ignores
+            is False, returns a loss per input/target element instead and ignores
             size_average. Default: True
 
      Shape:
@@ -616,7 +616,8 @@ class SmoothL1Loss(_Loss):
            the losses are instead summed. Ignored when reduce is ``False``. Default: ``True``
         reduce (bool, optional): By default, the losses are averaged or summed
            over elements. When reduce is ``False``, the loss function returns
-           a loss per element instead and ignores size_average. Default: ``True``
+           a loss per input/target element instead and ignores size_average.
+           Default: ``True``
 
     Shape:
         - Input: :math:`(N, *)` where `*` means, any number of additional
@@ -690,7 +691,7 @@ class CrossEntropyLoss(_WeightedLoss):
             ``True``, the loss is averaged over non-ignored targets.
         reduce (bool, optional): By default, the losses are averaged or summed over
             observations for each minibatch depending on size_average. When reduce
-            is ``False``, returns a loss per batch element instead and ignores
+            is ``False``, returns a loss per batch instead and ignores
             size_average. Default: ``True``
 
     Shape:


Test patch: