Repository: pytorch/pytorch

Issue:
torch.cat doesn't do proper shape checking, which can lead to incorrect gradients
This is a *very* sneaky behaviour so I will call it a bug as it can easily break your code.
I paste here a self-contained example. Even though `axes1` is equivalent to `axes2`, `axes2` gives wrong gradients.
```python
from torch.autograd import Variable
from torch import autograd
import torch
import numpy as np

def norm_vec(v):
    return torch.sqrt(torch.pow(v, 2).sum())

coo = Variable(torch.from_numpy(np.array([[3, 4, 2],])).float(), requires_grad=True)
axes1 = torch.stack([coo[0] / norm_vec(coo[0]), Variable(torch.zeros(3), requires_grad=True), Variable(torch.zeros(3), requires_grad=True)], dim=1)
axes2 = torch.stack([coo[0] / norm_vec(coo[0]), Variable(torch.zeros(3, 1), requires_grad=True), Variable(torch.zeros(3, 1), requires_grad=True)], dim=1).squeeze()
new1 = torch.matmul(coo, axes1)
new2 = torch.matmul(coo, axes2)
print(autograd.grad(new1[0, 0], coo, create_graph=True), autograd.grad(new2[0, 0], coo, create_graph=True))
```


Discussion:
In case you are interested, here is a larger example that shows that indeed `axes1` is correct. I have a point in space and I project it on it's own normalized vector. Hence the x coordinate in the new coordinate system should be equal to the distance, so `x = r` meaning that `1/r = x/r**2` so the gradients of `inv` should be the same as the gradients of `test1` and `test2` (but only `test1` gives the right gradients)
```python
from torch.autograd import Variable
from torch import autograd
import torch
import numpy as np

def norm_vec(v):
    return torch.sqrt(torch.pow(v, 2).sum())

coo = Variable(torch.from_numpy(np.array([[3, 4, 2],])).float(), requires_grad=True)
r2 = (coo**2).sum(dim=1)

axes1 = torch.stack([coo[0] / norm_vec(coo[0]), Variable(torch.zeros(3), requires_grad=True), Variable(torch.zeros(3), requires_grad=True)], dim=1)
axes2 = torch.stack([coo[0] / norm_vec(coo[0]), Variable(torch.zeros(3, 1), requires_grad=True), Variable(torch.zeros(3, 1), requires_grad=True)], dim=1).squeeze()
print(axes1, axes2)
new1 = torch.matmul(coo, axes1)
new2 = torch.matmul(coo, axes2)

inv = 1/torch.sqrt(r2[0])
test1 = new1[0, 0]/r2[0]
test2 = new2[0, 0]/r2[0]

print(autograd.grad(inv, coo, create_graph=True), autograd.grad(test1, coo, create_graph=True), autograd.grad(test2, coo, create_graph=True))
```
I'm not sure if this is related, but broadcasting with torch.stack + indexing seems to cause problems.

Minimal example of broadcasting and torch.stack causing problems in a backward pass but not forward pass:
```
import torch
from torch.autograd import Variable
x = Variable(torch.Tensor([[3, 4, 2]]), requires_grad=True)
y = torch.stack([x[0], Variable(torch.zeros(3, 1))], dim=1)
out = y.sum()
out.backward() # not okay
>>>
RuntimeError: expand(torch.FloatTensor{[3, 1]}, size=[3]): the number of sizes provided (1) must be greater or equal to the number of dimensions in the tensor (2)
```

```
import torch
from torch.autograd import Variable
x = Variable(torch.Tensor([[3, 4, 2]]), requires_grad=True)
y = torch.stack([x[0], Variable(torch.zeros(3, 1))], dim=1)
out = y.sum()
out.backward() # no error
```
I think this is a torch.stack + indexing + broadcasting problem.

When I run @stefdoerr's code, making sure the tensor arguments to torch.stack all have the same size, everything is fine:
```
from torch.autograd import Variable
from torch import autograd
import torch
import numpy as np

def norm_vec(v):
    return torch.sqrt(torch.pow(v, 2).sum())

coo = Variable(torch.from_numpy(np.array([[3, 4, 2],])).float(), requires_grad=True)
axes1 = torch.stack([coo[0] / norm_vec(coo[0]), Variable(torch.zeros(3), requires_grad=True), Variable(torch.zeros(3), requires_grad=True)], dim=1)
axes2 = torch.stack([(coo[0] / norm_vec(coo[0])).unsqueeze(1), Variable(torch.zeros(3, 1), requires_grad=True), Variable(torch.zeros(3, 1), requires_grad=True)], dim=1).squeeze()
new1 = torch.matmul(coo, axes1)
new2 = torch.matmul(coo, axes2)
print(autograd.grad(new1[0, 0], coo, create_graph=True), autograd.grad(new2[0, 0], coo, create_graph=True)) # prints two identical tensors
```
`torch.cat` (and `torch.stack`) requires the arguments to have the same shape (except along the concatenating dimension). They don't implement broadcasting. We don't enforce this sufficiently, which leads to the incorrect gradients.

We should fix `THTensor_(catArray)` to do proper shape checking on its arguments. We should also clarify the documentation.

Resolving patch:
diff --git a/torch/_torch_docs.py b/torch/_torch_docs.py
--- a/torch/_torch_docs.py
+++ b/torch/_torch_docs.py
@@ -594,6 +594,8 @@
 cat(seq, dim=0, out=None) -> Tensor
 
 Concatenates the given sequence of :attr:`seq` tensors in the given dimension.
+All tensors must either have the same shape (except in the cat dimension) or be
+empty.
 
 :func:`torch.cat` can be seen as an inverse operation for :func:`torch.split`
 and :func:`torch.chunk`
@@ -601,7 +603,9 @@
 :func:`cat` can be best understood via examples.
 
 Args:
-    seq (sequence of tensors): any python sequence of tensors of the same type
+    seq (sequence of tensors): any python sequence of tensors of the same type.
+        Non-empty tensors provided must have the same shape, except in the
+        cat dimension.
     dim (int, optional): the dimension over which the tensors are concatenated
     out (Tensor, optional): the output tensor
 


Test patch:
diff --git a/test/test_cuda.py b/test/test_cuda.py
--- a/test/test_cuda.py
+++ b/test/test_cuda.py
@@ -720,6 +720,38 @@ def test_cat_autogpu(self):
         z = torch.cat([x, y], 0)
         self.assertEqual(z.get_device(), x.get_device())
 
+    def test_cat(self):
+        SIZE = 10
+        for dim in range(-3, 3):
+            pos_dim = dim if dim >= 0 else 3 + dim
+            x = torch.rand(13, SIZE, SIZE).transpose(0, pos_dim).cuda()
+            y = torch.rand(17, SIZE, SIZE).transpose(0, pos_dim).cuda()
+            z = torch.rand(19, SIZE, SIZE).transpose(0, pos_dim).cuda()
+
+            res1 = torch.cat((x, y, z), dim)
+            self.assertEqual(res1.narrow(pos_dim, 0, 13), x, 0)
+            self.assertEqual(res1.narrow(pos_dim, 13, 17), y, 0)
+            self.assertEqual(res1.narrow(pos_dim, 30, 19), z, 0)
+
+        x = torch.randn(20, SIZE, SIZE).cuda()
+        self.assertEqual(torch.cat(torch.split(x, 7)), x)
+        self.assertEqual(torch.cat(torch.chunk(x, 7)), x)
+
+        y = torch.randn(1, SIZE, SIZE).cuda()
+        z = torch.cat([x, y])
+        self.assertEqual(z.size(), (21, SIZE, SIZE))
+
+    def test_cat_bad_input_sizes(self):
+        x = torch.randn(2, 1).cuda()
+        y = torch.randn(2, 1, 1).cuda()
+        z = torch.randn(2, 1, 1).cuda()
+        self.assertRaises(RuntimeError, lambda: torch.cat([x, y, z]))
+
+        x = torch.randn(2, 1, 2).cuda()
+        y = torch.randn(2, 1, 1).cuda()
+        z = torch.randn(2, 2, 1).cuda()
+        self.assertRaises(RuntimeError, lambda: torch.cat([x, y, z], dim=1))
+
     def test_serialization(self):
         x = torch.randn(4, 4).cuda()
         with tempfile.NamedTemporaryFile() as f:
diff --git a/test/test_torch.py b/test/test_torch.py
--- a/test/test_torch.py
+++ b/test/test_torch.py
@@ -1917,6 +1917,17 @@ def test_cat(self):
 
         self.assertRaises(RuntimeError, lambda: torch.cat([]))
 
+    def test_cat_bad_input_sizes(self):
+        x = torch.randn(2, 1)
+        y = torch.randn(2, 1, 1)
+        z = torch.randn(2, 1, 1)
+        self.assertRaises(RuntimeError, lambda: torch.cat([x, y, z]))
+
+        x = torch.randn(2, 1, 2)
+        y = torch.randn(2, 1, 1)
+        z = torch.randn(2, 2, 1)
+        self.assertRaises(RuntimeError, lambda: torch.cat([x, y, z], dim=1))
+
     def test_stack(self):
         x = torch.rand(2, 3, 4)
         y = torch.rand(2, 3, 4)