Issue 1:
Issue description

I just upgraded my Pytorch version from 0.3.0 to v0.4.0, and I'm running into the following issue.

I want to freeze the parameters of a layer (i.e. set requires_grad to False for that layer) and train. This worked fine of v0.3.0 but now, I get the following error at the line loss.backward():

RuntimeError: backward_input can only be called in training mode

Based on what I found online, I added model.train() to the code, but now I get the following error on loss.backward():

RuntimeError: inconsistent range for TensorList output

When I comment out the layer freezing, things work fine. It seems that model.train() assumes requires.grad=True for all the parameters. How can I make training work properly with freezing?

Giving the code example below.

Code example

The following code ran fine on v0.3.0

for name, param in model.named_parameters():
if name == 'encoder.embedding.weight':
param.requires_grad = False
loss = model.forward()
loss.backward()

Running it on v0.4.0 yields to following error:

File "/mnt/fs_default/Conversational-Agents/model.py", line 86, in train_batch
loss.backward()
File "/usr/local/lib/python3.5/dist-packages/torch/tensor.py", line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/local/lib/python3.5/dist-packages/torch/autograd/__init__.py", line 89, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: backward_input can only be called in training mode

So then I added model.train() to my code as follows:

for name, param in model.named_parameters():
if name == 'encoder.embedding.weight':
param.requires_grad = False
model.train()
loss = model.forward()
loss.backward()

but now I get the error:

File "/mnt/fs_default/kaselby/Conversational-Agents/model.py", line 88, in train_batch
loss.backward()
File "/usr/local/lib/python3.5/dist-packages/torch/tensor.py", line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/local/lib/python3.5/dist-packages/torch/autograd/__init__.py", line 89, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: inconsistent range for TensorList output

Is this a bug? If not, can someone tell me how to make layer freezing work properly with model.train()?

System Info

PyTorch version: 0.4.0

Is debug build: No

CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.3 LTS

GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609

CMake version: version 3.5.1

Python version: 3.5

Is CUDA available: Yes

CUDA runtime version: 9.0.176

GPU models and configuration:

GPU 0: Tesla P100-PCIE-16GB

GPU 1: Tesla P100-PCIE-16GB

GPU 2: Tesla P100-PCIE-16GB

GPU 3: Tesla P100-PCIE-16GB

Nvidia driver version: 384.111

cuDNN version: Could not collect

Versions of relevant libraries:

[pip3] msgpack-numpy (0.4.1)

[pip3] numpy (1.13.3)

[pip3] torch (0.4.0)

[pip3] torchtext (0.2.3)

[pip3] torchvision (0.1.9)

[conda] Could not collect

Issue 2:
Issue description

The general situation is that we have a pretrained Language Model, and during a first phase we only want to train the new embedding layer we added before fine-tuning the whole thing. This worked fine in 0.3 but now sends an error message during back propagation. Minimal reproduction is to just create a simple model with a linear layer and an LSTM, freeze this second layer (by applying require_grads=False to its parameters) and try to compute a back-propagation.

Code example

See here

System Info

PyTorch version: 0.4.0

Is debug build: No

CUDA used to build PyTorch: 9.0 OS: Microsoft Windows 10 Home

GCC version: Could not collect

CMake version: Could not collect Python version: 3.6

Is CUDA available: Yes

CUDA runtime version: 9.1.85

GPU models and configuration: Could not collect

Nvidia driver version: Could not collect

cuDNN version: Could not collect Versions of relevant libraries:

[pip] Could not collect

[conda] Could not collect

Pytorch was installed with conda, the bug also appears on my linux instances.

Thanks for your help!