Repository: pytorch/pytorch

Issue:
Error in backprop when using frozen LSTM layers (new with 0.4.0)
## Issue description

The general situation is that we have a pretrained Language Model, and during a first phase we only want to train the new embedding layer we added before fine-tuning the whole thing. This worked fine in 0.3 but now sends an error message during back propagation. Minimal reproduction is to just create a simple model with a linear layer and an LSTM, freeze this second layer (by applying require_grads=False to its parameters) and try to compute a back-propagation.

## Code example

See [here](https://github.com/sgugger/Deep-Learning/blob/master/Bug%20with%20frozen%20LSTM%20layer.ipynb)

## System Info
PyTorch version: 0.4.0                                                                                                  
Is debug build: No                                                                                                      
CUDA used to build PyTorch: 9.0                                                                                                                                                                                                                 OS: Microsoft Windows 10 Home                                                                                           
GCC version: Could not collect                                                                                         
CMake version: Could not collect                                                                                                                                                                                                                Python version: 3.6                                                                                                     
Is CUDA available: Yes                                                                                                  
CUDA runtime version: 9.1.85                                                                                            
GPU models and configuration: Could not collect                                                                         
Nvidia driver version: Could not collect                                                                                
cuDNN version: Could not collect                                                                                                                                                                                                                Versions of relevant libraries:                                                                                         
[pip] Could not collect                                                                                                 
[conda] Could not collect

Pytorch was installed with conda, the bug also appears on my linux instances.

Thanks for your help!


Discussion:
to help give local context, the notebook error is:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-52a0569421b1> in <module>()
----> 1 loss.backward()

~\Anaconda3\envs\fastai\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94 
     95     def register_hook(self, hook):

~\Anaconda3\envs\fastai\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     87     Variable._execution_engine.run_backward(
     88         tensors, grad_tensors, retain_graph, create_graph,
---> 89         allow_unreachable=True)  # allow_unreachable flag
     90 
     91 

RuntimeError: inconsistent range for TensorList output
```
The error seems to be thrown at `tools\autograd\templates\Functions.cpp` line 47. Not sure what's happening there.
I can repro this on master on a linux box as well.

Repro script: (I took the ipython notebook in the code example and copy pasted the lines from there):
```
import torch
from torch.autograd import Variable as V
import torch.nn as nn
import torch.nn.functional as F

model = nn.Sequential(nn.Linear(10,20), nn.ReLU(inplace=True),nn.LSTM(20,5, 1)).cuda()
for param in list(model.parameters())[2:]:
    param.requires_grad=False
x = torch.randn(2,4,10).cuda()
x.requires_grad = True
z = model(x)
y = torch.Tensor([0,1,2,3, 0,1,2,3]).long().cuda()
loss = F.cross_entropy(z[0].view(-1,5),y)
loss.backward()
```
I hit this to.
@ailzhang If you are busy, I can have a look, too.
My repro is
```
import torch
print (torch.__version__)
dev = torch.device('cuda')
l = torch.nn.LSTM(2, 3).to(dev)
for p in l.parameters():
    p.requires_grad = False
s = torch.randn(1, 1, 2, requires_grad=True, device=dev)
out, _ = l(s)
out.sum().backward()
```
As a workaround, you can disable the cudnn backend.

So you need to return a list of undefined tensors rather than an empty tensor list. 
So one could add an else block with `dw.resize(weight.size())` to the conditional `dw` calculation in [_cudnn_rnn_backward](https://github.com/pytorch/pytorch/blob/bb15a0830de9577b4f6bdcded5eac864f78701c2/aten/src/ATen/native/cudnn/RNN.cpp#L995).

@t-vi Cool, feel free to propose a PR for it. Thanks! 

Resolving patch:
diff --git a/tools/autograd/gen_autograd_functions.py b/tools/autograd/gen_autograd_functions.py
--- a/tools/autograd/gen_autograd_functions.py
+++ b/tools/autograd/gen_autograd_functions.py
@@ -63,6 +63,12 @@
 }
 """)
 
+DERIVATIVE_MULTI_COPY_RANGE = CodeTemplate("""\
+  if (should_compute_output({ ${name}_ix })) {
+    copy_range(grad_inputs, ${name}_ix, std::get<${i}>(grad_result));
+  }
+""")
+
 DERIVATIVE_MULTI = CodeTemplate("""\
 if (should_compute_output({ ${idx_ranges} })) {
   ${grad_input_mask}
@@ -173,7 +179,7 @@ def emit_derivative(derivative):
             idx_ranges = ', '.join("{}_ix".format(n) for n in var_names)
             copy_ranges = []
             for i, n in enumerate(var_names):
-                copy_ranges.append("copy_range(grad_inputs, {}_ix, std::get<{}>(grad_result));".format(n, i))
+                copy_ranges.append(DERIVATIVE_MULTI_COPY_RANGE.substitute(name=n, i=i))
             return DERIVATIVE_MULTI.substitute(
                 idx_ranges=idx_ranges, copy_ranges=copy_ranges,
                 derivative=formula,


Test patch:
diff --git a/test/test_autograd.py b/test/test_autograd.py
--- a/test/test_autograd.py
+++ b/test/test_autograd.py
@@ -2266,6 +2266,19 @@ def test_set_requires_grad_only_for_floats_cuda(self):
     def test_set_requires_grad_only_for_floats(self):
         self._test_set_requires_grad_only_for_floats(self, False)
 
+    @unittest.skipIf(not torch.cuda.is_available(), "CUDA unavailable")
+    def test_rnn_backward_to_input_but_not_parameters_cuda(self):
+        # this checks whether it is possible to not require
+        # weight parameters, but require inputs, see #7722
+        dev = torch.device('cuda')
+        l = torch.nn.LSTM(2, 3).to(dev)
+        for p in l.parameters():
+            p.requires_grad = False
+        s = torch.randn(1, 1, 2, requires_grad=True, device=dev)
+        out, _ = l(s)
+        out.sum().backward()
+        self.assertFalse(s.grad is None or s.grad.abs().sum().item() == 0)
+
 
 def index_variable(shape, max_indices):
     if not isinstance(shape, tuple):