CUBLAS_STATUS_NOT_SUPPORTED when calling cublasDgemv
Labels: module: cuda, triaged, module: cublas

### 🐛 Describe the bug

When running the `distributed/test_data_parallel` or `test_nn` test it fails with 

```
ERROR: test_data_parallel (__main__.TestDataParallel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/shm/s3248973-EasyBuild/PyTorch/1.12.1/foss-2021b-CUDA-11.4.1/pytorch-v1.12.1/test/distributed/test_data_parallel.py", line 353, in test_data_parallel
    out = dp.data_parallel(l, i, dev_id)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 231, in data_parallel
    outputs = parallel_apply(replicas, inputs, module_kwargs, used_device_ids)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasLtMatmulAlgoGetHeuristic( ltHandle, computeDesc.descriptor(), Adesc.descriptor(), Bdesc.descriptor(), Cdesc.descriptor(), Cdesc.descriptor(), preference.descriptor(), 1, &heuristicResult, &returnedResult)`
```

and

```
ERROR: test_spectral_norm (__main__.TestNN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/shm/s3248973-EasyBuild/PyTorch/1.12.1/foss-2021b-CUDA-11.4.1/pytorch-v1.12.1/test/test_nn.py", line 4593, in test_spectral_norm
    gradcheck(fn, (input.clone().requires_grad_(),), check_batched_grad=False)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 3019, in gradcheck
    return torch.autograd.gradcheck(fn, inputs, **kwargs)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/autograd/gradcheck.py", line 1414, in gradcheck
    return _gradcheck_helper(**args)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/autograd/gradcheck.py", line 1423, in _gradcheck_helper
    func_out = func(*tupled_inputs)
  File "/dev/shm/s3248973-EasyBuild/PyTorch/1.12.1/foss-2021b-CUDA-11.4.1/pytorch-v1.12.1/test/test_nn.py", line 4590, in fn
    out1 = wrapped_m(input)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1137, in _call_impl
    result = hook(self, input)
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/utils/spectral_norm.py", line 105, in __call__
    setattr(module, self.name, self.compute_weight(module, do_power_iteration=module.training))
  File "/tmp/easybuild-tmp/eb-HhIeo8/tmpUxRyEj/lib/python3.9/site-packages/torch/nn/utils/spectral_norm.py", line 84, in compute_weight
    v = normalize(torch.mv(weight_mat.t(), u), dim=0, eps=self.eps, out=v)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasDgemv(handle, op, m, n, &alpha, a, lda, x, incx, &beta, y, incy)`
```

I found that `test_nn` uses `torch.nn.DataParallel` when multiple GPUs are present so I assume it is the same issue.
That error isn't listed in the NVIDIA docs for `cublasDgemv` so I don't know why it fails.

### Versions

- PyTorch 1.12.1
- CUDA 11.4.1
- Python 3.9

cc @ngimel @csarofeen @ptrblck @xwang233