Issue 1:
🐛 Bug

I think the vector strides are passed incorrectly to gemv on GPU. It works well on CPU.

To Reproduce

import torch

mat = torch.randn(2, 2).cuda()
vec = torch.randn(2).cuda().requires_grad_(True)
(mat @ vec).sum().backward()

Here I get:

RuntimeError: at::cuda::blas::gemv argument incx must be positive and less than 2147483647 but got 0

Originally I found the bug as an issue during DGER in the gradient of mat @ vec, in the generated function MvBackward::apply in torch/csrc/autograd/generated/Functions.cpp. The error message is the same as cornellius-gp/gpytorch#834 . But, attempting to write minimal code to reproduce that, I hit this other bug instead.

Expected behavior

The gradient for vec should be calculated, and the program should not crash.

Environment

I have reproduced this bug in master, and Pytorch 1.4.

Details of master version

PyTorch version: 1.6.0a0+cf82011

Is debug build: No

CUDA used to build PyTorch: 10.0

OS: Ubuntu 16.04.6 LTS

GCC version: (Ubuntu 7.1.0-10ubuntu1~16.04.york0) 7.1.0

CMake version: version 3.14.0

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: 10.0.130

GPU models and configuration:

GPU 0: GeForce GTX 1080

GPU 1: GeForce GT 710

Nvidia driver version: 418.87.01

cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:

[pip3] numpy==1.11.0

[conda] blas 1.0 mkl

[conda] gpytorch 1.0.1 dev_0

[conda] magma-cuda100 2.5.2 1 pytorch

[conda] mkl 2020.0 166

[conda] mkl-include 2020.1 217

[conda] mkl-service 2.3.0 py37he904b0f_0

[conda] mkl_fft 1.0.15 py37ha843d7b_0

[conda] mkl_random 1.1.0 py37hd6b4f25_0

[conda] numpy 1.18.1 py37h4f9e942_0

[conda] numpy-base 1.18.1 py37hde5b4d6_1

[conda] numpydoc 0.9.2 py_0

[conda] torch 1.6.0a0+cf82011 pypi_0 pypi

[conda] torchvision 0.6.0 pypi_0 pypi

Details of 1.4.0 version

PyTorch version: 1.4.0

Is debug build: No

CUDA used to build PyTorch: 10.1

OS: Ubuntu 16.04.6 LTS

GCC version: (Ubuntu 7.1.0-10ubuntu1~16.04.york0) 7.1.0

CMake version: version 3.16.0-rc3

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: 10.1.243

GPU models and configuration:

GPU 0: GeForce GTX 1080

GPU 1: GeForce GT 710

Nvidia driver version: 418.87.01

cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:

[pip3] gpytorch==1.0.1

[pip3] numpy==1.18.2

[pip3] torch==1.4.0

[pip3] torchvision==0.5.0

[conda] Could not collect

Additional context

I would fix this myself but the C++ call stack is too deep and I get lost. I also don't know where the generated/ functions come from. Thus I pass the baton to you, and will use a workaround.

Workaround

Use vec.unsqueeze(-1) to turn it into a gemm. Squeeze the result.

cc @ezyang @ssnl @albanD @zou3519 @gqchen

Issue 2:
🐛 Bug

To Reproduce

Steps to reproduce the behavior:

import torch
import torch.nn as nn

torch.set_default_tensor_type('torch.cuda.FloatTensor')
x = nn.Parameter(torch.ones(2, 2))
(x @ torch.ones(2)).sum().backward()
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-11-c3b66f275e9a> in <module>()
1 x = nn.Parameter(torch.ones(2, 2))
----> 2 (x @ torch.ones(2)).sum().backward()

1 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
97 Variable._execution_engine.run_backward(
98 tensors, grad_tensors, retain_graph, create_graph,
---> 99 allow_unreachable=True) # allow_unreachable flag
100
101

RuntimeError: cublas runtime error : an invalid numeric value was used as an argument at /pytorch/aten/src/THC/THCBlas.cu:120

Expected behavior

No exception is raised.

Environment

Collecting environment information...
PyTorch version: 1.3.1+cu100
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.12.0

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 418.67
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.3

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.3.1+cu100
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.3.1
[pip3] torchvision==0.4.2+cu100
[conda] Could not collect

This code was run on Google Colab. The bug also reproduces on my server (Ubuntu 16.04LTS, 4x GTX1080Ti GPUs).

Additional context

The bug doesn't occur if:

If I replace sum() with mean()

(x @ torch.ones(2)).mean().backward()

If I use a different shape of the tensor

(x @ torch.ones(2, 1)).sum().backward()

If I run the code on CPU

Therefore, I believe that this is not intended behavior.

cc @ezyang @gchanan @zou3519 @jerryzh168 @ssnl @albanD @gqchen