Issue 1:
Issue description

I was training an autoencoder with MSELoss, and the loss values on the training data were huge but the loss values on the validation data were small. It appeared as if the loss was not being averaged on the training pass, but it was on the validation pass. A little poking around in the debugger revealed this to be the case.

The problem is in the _pointwise_loss loss function in torch/nn/functional.py:

def _pointwise_loss(lambd, lambd_optimized, input, target, reduction='elementwise_mean'):
if target.requires_grad:
d = lambd(input, target)
if reduction == 'none':
return d
return torch.mean(d) if reduction == 'elementwise_mean' else torch.sum(d)
else:
return lambd_optimized(input, target, _Reduction.get_enum(reduction))

In the line return torch.mean(d) if reduction == 'elementwise_mean' else torch.sum(d), the variable reduction is actually an integer with value 1, not a string with value 'elementwise_mean', and so the loss is summed instead of averaged, even though the 'elementwise_mean` option was chosen.

Code example

This code reproduces the problem:

import torch
import torchvision as tv

class Net(torch.nn.Module):
def __init__(self):
super(Net, self).__init__()
self.c1 = torch.nn.Conv2d(1, 16, 3, padding=1)
self.c2 = torch.nn.Conv2d(16, 1, 3, padding=1)
self.relu = torch.nn.ReLU(inplace=True)

def forward(self, x):
x = self.c2(self.relu(self.c1(x)))
return x

net = Net()
data = torchvision.datasets.MNIST('MNIST', True, download=True, transform=tv.transforms.ToTensor())
mse = torch.nn.MSELoss(reduction='elementwise_mean')

img = data[0][0].view(1, 1, 28, 28).requires_grad_()
pred = net(img)

loss1 = mse(pred, img).item()

img.requires_grad = False
loss2 = mse(pred, img).item()

print(loss1, loss2)

And the output is:

155.7478 0.1987

System Info

PyTorch version: 0.4.1

Is debug build: No

CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.2 LTS

GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609

CMake version: version 3.10.2

Python version: 3.6

Is CUDA available: Yes

CUDA runtime version: 7.5.17

GPU models and configuration:

GPU 0: GeForce GTX 1080 Ti

GPU 1: GeForce GTX 1080 Ti

GPU 2: GeForce GTX 1080 Ti

GPU 3: GeForce GTX 1080 Ti

GPU 4: GeForce GTX 1080 Ti

GPU 5: GeForce GTX 1080 Ti

GPU 6: GeForce GTX 1080 Ti

GPU 7: GeForce GTX 1080 Ti

Nvidia driver version: 387.34

cuDNN version: Probably one of the following:

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5.0.5

/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn_static.a

/usr/local/cuda-9.0/lib64/libcudnn.so

/usr/local/cuda-9.0/lib64/libcudnn.so.7

/usr/local/cuda-9.0/lib64/libcudnn.so.7.0.3

/usr/local/cuda-9.0/lib64/libcudnn_static.a

Versions of relevant libraries:

[pip] Could not collect

[conda] Could not collect

Issue 2:
Issue description

If a tensor with requires_grad=True is passed to mse_loss, then the loss is reduced even if reduction is none.

Appeared in Pytorch 0.4.1.

Code example

import torch
x = torch.zeros((4, 5, 2))
print('Good', torch.nn.functional.mse_loss(x, x, reduction='none').shape)
x.requires_grad = True
print('Bad', torch.nn.functional.mse_loss(x, x, reduction='none').shape)

Outputs:

Good torch.Size([4, 5, 2])
Bad torch.Size([])

System Info

Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.3 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6