gradcheck failure with sparse matrix multiplication
Labels: module: sparse, module: autograd, triaged

### 🐛 Describe the bug

As detailed in #86963, sparse @ dense matrix mutiplication fails to pass gradcheck. A manual inspection of the gradient seems to indicate that this is a bug with gradcheck rather than the matrix multiplication itself.

Steps to reproduce:
```python
# Try with the nightly version of PyTorch
!pip3 install --pre torch torchvision torchaudio torchtext --extra-index-url https://download.pytorch.org/whl/nightly/cu117 --upgrade

import torch
print(f'Running PyTorch version: {torch.__version__}')

torchdevice = torch.device('cpu')
if torch.cuda.is_available():
  torchdevice = torch.device('cuda')
  print('Default GPU is ' + torch.cuda.get_device_name(torch.device('cuda')))
print('Running on ' + str(torchdevice))

# Dimension of the square sparse matrix
n = 5
# Number of non-zero elements (up to duplicates)
nnz = 8

rowidx = torch.randint(low=0, high=n, size=(nnz,), device=torchdevice)
colidx = torch.randint(low=0, high=n, size=(nnz,), device=torchdevice)
itemidx = torch.vstack((rowidx,colidx))
xvalues = torch.randn(nnz, dtype=torch.double, device=torchdevice)
x_coo = torch.sparse_coo_tensor(itemidx, xvalues, size=(n,n)).coalesce()
#print('x_coo:',x_coo)
#print(x_coo.to_dense())

x_csr = x_coo.to_sparse_csr()

b = torch.randn(n, dtype=torch.double, device=torchdevice).unsqueeze(1)

try:
  xx_coo = x_coo.detach().clone().requires_grad_(True)
  bb = b.detach().clone().requires_grad_(True)
  test = torch.autograd.gradcheck(torch.sparse.mm, (xx_coo, bb), check_sparse_nnz=True, eps=1e-6, atol=1e-4)
  print('\n=\ncoo torch.sparse.mm gradcheck',test)
except Exception as err:
  print('\n=\ncoo torch.sparse.mm gradcheck except')
  print(Exception, err)

try:
  xx_csr = x_csr.detach().clone().requires_grad_(True)
  bb = b.detach().clone().requires_grad_(True)
  test = torch.autograd.gradcheck(torch.sparse.mm, (xx_csr, bb), check_sparse_nnz=True, eps=1e-6, atol=1e-4)
  print('\n=\ncsr torch.sparse.mm gradcheck',test)
except Exception as err:
  print('\n=\ncsr torch.sparse.mm gradcheck except')
  print(Exception, err)

try:
  xx_coo = x_coo.detach().clone().requires_grad_(True)
  bb = b.detach().clone().requires_grad_(True)
  test = torch.autograd.gradcheck(torch.mm, (xx_coo, bb), check_sparse_nnz=True, eps=1e-6, atol=1e-4)
  print('\n=\ncoo torch.mm gradcheck',test)
except Exception as err:
  print('\n=\ncoo torch.mm gradcheck except')
  print(Exception, err)

try:
  xx_csr = x_csr.detach().clone().requires_grad_(True)
  bb = b.detach().clone().requires_grad_(True)
  test = torch.autograd.gradcheck(torch.mm, (xx_csr, bb), check_sparse_nnz=True, eps=1e-6, atol=1e-4)
  print('\n=\ncsr torch.mm gradcheck',test)
except Exception as err:
  print('\n=\ncsr torch.mm gradcheck except')
  print(Exception, err)

try:
  xx_coo = x_coo.detach().clone().requires_grad_(True)
  bb = b.detach().clone().requires_grad_(True)
  test = torch.autograd.gradcheck(torch.smm, (xx_coo, bb), check_sparse_nnz=True, eps=1e-6, atol=1e-4)
  print('\n=\ncoo torch.smm gradcheck',test)
except Exception as err:
  print('\n=\ncoo torch.smm gradcheck except')
  print(Exception, err)

try:
  xx_csr = x_csr.detach().clone().requires_grad_(True)
  bb = b.detach().clone().requires_grad_(True)
  test = torch.autograd.gradcheck(torch.smm, (xx_csr, bb), check_sparse_nnz=True, eps=1e-6, atol=1e-4)
  print('\n=\ncsr torch.smm gradcheck',test)
except Exception as err:
  print('\n=\ncsr torch.smm gradcheck except')
  print(Exception, err)

try:
  xx_coo = x_coo.detach().clone().requires_grad_(True)
  bb = b.detach().clone().requires_grad_(True)
  test = torch.autograd.gradcheck(torch.nn.functional.linear, (bb.squeeze(), xx_coo), check_sparse_nnz=True, eps=1e-6, atol=1e-4)
  print('\n=\ncoo torch.nn.functional.linear gradcheck',test)
except Exception as err:
  print('\n=\ncoo torch.nn.functional.linear gradcheck except')
  print(Exception, err)

try:
  xx_csr = x_csr.detach().clone().requires_grad_(True)
  bb = b.detach().clone().requires_grad_(True)
  test = torch.autograd.gradcheck(torch.nn.functional.linear, (bb.squeeze(), xx_csr), check_sparse_nnz=True, eps=1e-6, atol=1e-4)
  print('\n=\ncsr torch.nn.functional.linear gradcheck',test)
except Exception as err:
  print('\n=\ncsr torch.nn.functional.linear gradcheck except')
  print(Exception, err)
```

### Versions

Based on current google colab with manually updated pytorch version:
```
PyTorch version: 1.14.0.dev20221017+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.6
Libc version: glibc-2.26

Python version: 3.7.14 (default, Sep  8 2022, 00:06:44)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.14.0.dev20221017+cu117
[pip3] torchaudio==0.13.0.dev20221017+cu117
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.14.0.dev20221017
[pip3] torchvision==0.15.0.dev20221017+cu117
[conda] Could not collect
```

cc @nikitaved @pearu @cpuhrsch @amjames @bhosmer @ezyang @albanD @zou3519 @gqchen @soulitzer @Lezcano @Varal7