Repository: pytorch/pytorch

Issue:
Cannot print sparse tensors from ATen
Fix coming.
Error in backprop when using frozen LSTM layers (new with 0.4.0)
## Issue description

The general situation is that we have a pretrained Language Model, and during a first phase we only want to train the new embedding layer we added before fine-tuning the whole thing. This worked fine in 0.3 but now sends an error message during back propagation. Minimal reproduction is to just create a simple model with a linear layer and an LSTM, freeze this second layer (by applying require_grads=False to its parameters) and try to compute a back-propagation.

## Code example

See [here](https://github.com/sgugger/Deep-Learning/blob/master/Bug%20with%20frozen%20LSTM%20layer.ipynb)

## System Info
PyTorch version: 0.4.0                                                                                                  
Is debug build: No                                                                                                      
CUDA used to build PyTorch: 9.0                                                                                                                                                                                                                 OS: Microsoft Windows 10 Home                                                                                           
GCC version: Could not collect                                                                                         
CMake version: Could not collect                                                                                                                                                                                                                Python version: 3.6                                                                                                     
Is CUDA available: Yes                                                                                                  
CUDA runtime version: 9.1.85                                                                                            
GPU models and configuration: Could not collect                                                                         
Nvidia driver version: Could not collect                                                                                
cuDNN version: Could not collect                                                                                                                                                                                                                Versions of relevant libraries:                                                                                         
[pip] Could not collect                                                                                                 
[conda] Could not collect

Pytorch was installed with conda, the bug also appears on my linux instances.

Thanks for your help!
[Docs] [Low Priority] Pytorch docs in Google search points to master/unstable
When I googled `pytorch nn functional`, this came up as the first link: `pytorch.org/docs/master/_modules/torch/nn/functional.html`

This is, as you know, docs for the master/unstable version. On clicking the link which says it'll take me to the docs for the stable release, I get sent to the home page of the docs, and now I have to search for the page again.

This can be solved by adding a canonical link to their non-stable documentation, i.e. something like this: `<link rel="canonical" href="https://pytorch.org/docs/stable/nn.html#torch-nn-functional">`.


Discussion:

to help give local context, the notebook error is:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-52a0569421b1> in <module>()
----> 1 loss.backward()

~\Anaconda3\envs\fastai\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94 
     95     def register_hook(self, hook):

~\Anaconda3\envs\fastai\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     87     Variable._execution_engine.run_backward(
     88         tensors, grad_tensors, retain_graph, create_graph,
---> 89         allow_unreachable=True)  # allow_unreachable flag
     90 
     91 

RuntimeError: inconsistent range for TensorList output
```
The error seems to be thrown at `tools\autograd\templates\Functions.cpp` line 47. Not sure what's happening there.
I can repro this on master on a linux box as well.

Repro script: (I took the ipython notebook in the code example and copy pasted the lines from there):
```
import torch
from torch.autograd import Variable as V
import torch.nn as nn
import torch.nn.functional as F

model = nn.Sequential(nn.Linear(10,20), nn.ReLU(inplace=True),nn.LSTM(20,5, 1)).cuda()
for param in list(model.parameters())[2:]:
    param.requires_grad=False
x = torch.randn(2,4,10).cuda()
x.requires_grad = True
z = model(x)
y = torch.Tensor([0,1,2,3, 0,1,2,3]).long().cuda()
loss = F.cross_entropy(z[0].view(-1,5),y)
loss.backward()
```
I hit this to.
@ailzhang If you are busy, I can have a look, too.
My repro is
```
import torch
print (torch.__version__)
dev = torch.device('cuda')
l = torch.nn.LSTM(2, 3).to(dev)
for p in l.parameters():
    p.requires_grad = False
s = torch.randn(1, 1, 2, requires_grad=True, device=dev)
out, _ = l(s)
out.sum().backward()
```
As a workaround, you can disable the cudnn backend.

So you need to return a list of undefined tensors rather than an empty tensor list. 
So one could add an else block with `dw.resize(weight.size())` to the conditional `dw` calculation in [_cudnn_rnn_backward](https://github.com/pytorch/pytorch/blob/bb15a0830de9577b4f6bdcded5eac864f78701c2/aten/src/ATen/native/cudnn/RNN.cpp#L995).

@t-vi Cool, feel free to propose a PR for it. Thanks! 
Turns out the better fix checks whether we want grad in more detail.
I've noticed this before but I didn't know there was a way to solve it. Thanks @sherjilozair for the idea; I'll try it out and see how it goes -- this would definitely be a great improvement. (You should also feel free to submit a PR if you'd like!)
This is what Python documentation uses. See, eg.: view-source:https://docs.python.org/3.8/library/functions.html 

It would take time for Google's SEO to update though.

Resolving patch:
diff --git a/torch/_torch_docs.py b/torch/_torch_docs.py
--- a/torch/_torch_docs.py
+++ b/torch/_torch_docs.py
@@ -4392,6 +4392,33 @@ def parse_kwargs(desc):
             [-0.5872,  0.6932]])
 """)
 
+add_docstr(torch.flip,
+           r"""
+flip(input, dims) -> Tensor
+
+Reverse the order of a n-D tensor along given axis in dims.
+
+Args:
+    input (Tensor): the input tensor
+    dims (a list or tuple): axis to flip on
+
+Example::
+
+    >>> x = torch.arange(8).view(2, 2, 2)
+    >>> x
+    tensor([[[ 0,  1],
+             [ 2,  3]],
+
+            [[ 4,  5],
+             [ 6,  7]]])
+    >>> torch.flip(x, [0, 1])
+    tensor([[[ 6,  7],
+             [ 4,  5]],
+
+            [[ 2,  3],
+             [ 0,  1]]])
+""")
+
 add_docstr(torch.take,
            r"""
 take(input, indices) -> Tensor


Test patch:
diff --git a/test/test_autograd.py b/test/test_autograd.py
--- a/test/test_autograd.py
+++ b/test/test_autograd.py
@@ -2509,6 +2509,10 @@ class dont_convert(tuple):
     ('reshape', (S,), (S,), '1d'),
     ('reshape', (), (dont_convert(()),), 'scalar_to_scalar'),
     ('reshape', (), (1,), 'scalar_to_1d'),
+    ('flip', (S, S, S), ([0],), 'd0'),
+    ('flip', (S, S, S), ([0, 1, 2],), 'd012'),
+    ('flip', (S, S, S), ([0, 2],), 'd02'),
+    ('flip', (S, S, S), ([2, 0],), 'd20'),
     ('view_as', (S, S, S), (non_differentiable(torch.rand(S * S, S)),)),
     ('view_as', (), (non_differentiable(torch.tensor(5.5)),), 'scalar'),
     ('view_as', (), (non_differentiable(torch.rand(1, 1)),), 'scalar_to_dims'),
diff --git a/test/test_cuda.py b/test/test_cuda.py
--- a/test/test_cuda.py
+++ b/test/test_cuda.py
@@ -409,6 +409,10 @@ def tmp(t):
     ('zero', small_3d, lambda t: [],),
     ('zeros', small_3d, lambda t: [1, 2, 3, 4],),
     ('eye', small_2d, lambda t: [3, 4],),
+    ('flip', small_3d, lambda t: [0], 'd0', types, True),
+    ('flip', small_3d, lambda t: [0, 1, 2], 'd012', types, True),
+    ('flip', small_3d, lambda t: [0, 2], 'd02', types, True),
+    ('flip', small_3d, lambda t: [2, 0], 'd20', types, True),
     ('rsqrt', lambda t: constant_tensor_add(1, small_3d(t)), lambda t: [], None, float_types),
     ('sinh', lambda t: tensor_clamp(small_3d(t), -1, 1), lambda t: [], None, float_types),
     ('tan', lambda t: tensor_clamp(small_3d(t), -1, 1), lambda t: [], None, float_types),
@@ -1372,6 +1376,9 @@ def test_gesv_batched_dims(self):
     def test_view(self):
         TestTorch._test_view(self, lambda t: t.cuda())
 
+    def test_flip(self):
+        TestTorch._test_flip(self, use_cuda=True)
+
     def test_signal_window_functions(self):
         TestTorch._test_signal_window_functions(self, device=torch.device('cuda'))
 
diff --git a/test/test_torch.py b/test/test_torch.py
--- a/test/test_torch.py
+++ b/test/test_torch.py
@@ -5953,6 +5953,51 @@ def test_permute(self):
         self.assertEqual(perm, new)
         self.assertEqual(x.size(), orig)
 
+    @staticmethod
+    def _test_flip(self, use_cuda=False):
+        if use_cuda:
+            cuda = torch.device("cuda")
+            data = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8], device=cuda).view(2, 2, 2)
+            # large data testing
+            large_data = torch.arange(0, 100000000, device=cuda).view(10000, 10000)
+            large_data.flip([0, 1])
+        else:
+            data = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8]).view(2, 2, 2)
+
+        self.assertEqual(torch.tensor([5, 6, 7, 8, 1, 2, 3, 4]).view(2, 2, 2), data.flip(0))
+        self.assertEqual(torch.tensor([3, 4, 1, 2, 7, 8, 5, 6]).view(2, 2, 2), data.flip(1))
+        self.assertEqual(torch.tensor([2, 1, 4, 3, 6, 5, 8, 7]).view(2, 2, 2), data.flip(2))
+        self.assertEqual(torch.tensor([7, 8, 5, 6, 3, 4, 1, 2]).view(2, 2, 2), data.flip(0, 1))
+        self.assertEqual(torch.tensor([8, 7, 6, 5, 4, 3, 2, 1]).view(2, 2, 2), data.flip(0, 1, 2))
+
+        # check for permute
+        self.assertEqual(torch.tensor([6, 5, 8, 7, 2, 1, 4, 3]).view(2, 2, 2), data.flip(0, 2))
+        self.assertEqual(torch.tensor([6, 5, 8, 7, 2, 1, 4, 3]).view(2, 2, 2), data.flip(2, 0))
+
+        # not allow flip on the same dim more than once
+        self.assertRaises(RuntimeError, lambda: data.flip(0, 1, 1))
+        # not allow empty list as input
+        self.assertRaises(TypeError, lambda: data.flip())
+        # not allow size of flip dim > total dims
+        self.assertRaises(RuntimeError, lambda: data.flip(0, 1, 2, 3))
+        # not allow dim < 0
+        self.assertRaises(RuntimeError, lambda: data.flip(-1))
+        # not allow dim > max dim
+        self.assertRaises(RuntimeError, lambda: data.flip(3))
+
+        # test for non-contiguous case
+        if use_cuda:
+            expanded_data = torch.arange(1, 4, device=cuda).view(3, 1).expand(3, 2)
+            tranposed_data = torch.arange(1, 9, device=cuda).view(2, 2, 2).transpose(0, 1)
+        else:
+            expanded_data = torch.arange(1, 4).view(3, 1).expand(3, 2)
+            tranposed_data = torch.arange(1, 9).view(2, 2, 2).transpose(0, 1)
+        self.assertEqual(torch.tensor([3, 3, 2, 2, 1, 1]).view(3, 2), expanded_data.flip(0))
+        self.assertEqual(torch.tensor([8, 7, 4, 3, 6, 5, 2, 1]).view(2, 2, 2), tranposed_data.flip(0, 1, 2))
+
+    def test_flip(self):
+        self._test_flip(self, use_cuda=False)
+
     def test_storage(self):
         v = torch.randn(3, 5)
         self.assertEqual(v.storage()[0], v.data[0][0])