RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)

lomahu: <p>Hi,</p>
<p>I had an error when running a script using torch:</p>
<p>…<br>
23-03-10 17:24:37.413 - INFO: Model [DDPM] is created.<br>
23-03-10 17:24:37.413 - INFO: Initial Model Finished<br>
Traceback (most recent call last):<br>
File “DDM_train.py”, line 69, in <br>
diffusion.optimize_parameters()<br>
File “/data/work/Diffusion/DDM/model/model.py”, line 53, in optimize_parameters<br>
score, loss = self.netG(self.data, self.loss_lambda)<br>
File “/home/lomahu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1194, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/data/work/Diffusion/DDM/model/ddpm_modules/diffusion.py”, line 253, in forward<br>
return self.p_losses(x, loss_lambda, *args, **kwargs)<br>
File “/data/work/Diffusion/DDM/model/ddpm_modules/diffusion.py”, line 238, in p_losses<br>
code = self.denoise_fn(torch.cat([x_in[‘S’], x_in[‘T’], x_t], dim=1), t)<br>
File “/home/lomahu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1194, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/data/work/Diffusion/DDM/model/ddpm_modules/unet.py”, line 225, in forward<br>
x = layer(x, t)<br>
File “/home/lomahu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1194, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/data/work/Diffusion/DDM/model/ddpm_modules/unet.py”, line 136, in forward<br>
x = self.attn(x)<br>
File “/home/lomahu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1194, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/data/work/Diffusion/DDM/model/ddpm_modules/unet.py”, line 117, in forward<br>
context = torch.einsum(‘bhdn,bhen-&gt;bhde’, k, v)<br>
File “/home/lomahu/.local/lib/python3.8/site-packages/torch/functional.py”, line 378, in einsum<br>
return _VF.einsum(equation, operands)  # type: ignore[attr-defined]</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling <code>cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)</code></p>
<p>It looks like the problem came from calling einsum function?</p>
<p>To test my environment setting, I went to run a test script provided earlier posted and got a similar error:</p>
<p>Python 3.8.10 (default, Nov 14 2022, 12:59:47)<br>
[GCC 9.4.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
‘1.13.1+cu117’<br>
device=“cuda:1”<br>
print(torch.cuda.get_device_properties(device))<br>
_CudaDeviceProperties(name=‘NVIDIA RTX A6000’, major=8, minor=6, total_memory=48676MB, multi_processor_count=84)<br>
import torch.nn as nn<br>
rr = torch.zeros([2,20,5000]).to(device)<br>
layer1 = nn.Conv1d(20,500,kernel_size=4,stride=4,groups=20,bias=False).to(device)<br>
layer2 = nn.Linear(500,768).to(device)<br>
l1out = layer1(rr)<br>
l2out = layer2(l1out.transpose(1,2))<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/home/lomahu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1194, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/lomahu/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py”, line 114, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling <code>cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)</code></p>
</blockquote>
</blockquote>
</blockquote>
<p>Here is my collect_env.py outputs:<br>
Collecting environment information…<br>
PyTorch version: 1.13.1+cu117<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.7<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.5 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: version 3.16.3<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)<br>
Python platform: Linux-5.4.0-144-generic-x86_64-with-glibc2.29<br>
Is CUDA available: True<br>
CUDA runtime version: 11.5.119<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration:<br>
GPU 0: NVIDIA RTX A6000<br>
GPU 1: NVIDIA RTX A6000</p>
<p>Nvidia driver version: 525.60.13<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] denoising-diffusion-pytorch==1.2.2<br>
[pip3] ema-pytorch==0.2.1<br>
[pip3] lion-pytorch==0.0.5<br>
[pip3] numpy==1.23.1<br>
[pip3] torch==1.13.1<br>
[pip3] torchvision==0.14.1<br>
[conda] Could not collect</p>
<p>Many thanks for any inputs on the error when calling “cublasSgemmStridedBatched”.</p>
<p>J. L.</p>

ptrblck: <p>Could you post a minimal and executable code snippet to reproduce the issue and wrap it into three backticks ```, please?</p>

lomahu: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your prompt response to my question. Here is the snippet I used to generate the problem although the snippet was not doing exactly the same thing I was doing in the original problem.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
device=“cuda:1”<br>
print(torch.cuda.get_device_properties(device))<br>
_CudaDeviceProperties(name=‘NVIDIA RTX A6000’, major=8, minor=6, total_memory=48676MB, multi_processor_count=84)<br>
import torch.nn as nn<br>
rr = torch.zeros([2,20,5000]).to(device)<br>
layer1 = nn.Conv1d(20,500,kernel_size=4,stride=4,groups=20,bias=False).to(device)<br>
layer2 = nn.Linear(500,768).to(device)<br>
l1out = layer1(rr)<br>
l2out = layer2(l1out.transpose(1,2))</p>
</blockquote>
</blockquote>
</blockquote>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/home/lomahu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1194, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/lomahu/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py”, line 114, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling <code>cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)</code></p>
<p>I highly suspected that the problem is related to my versions of torch, tensorflow or RTX A6000 GPU cards. Your help is much appreciated!</p>
<p>J.L.</p>

haoyum1997: <p>Any solutions so far?<br>
I also encounter the same issue, especially when I use einsum.</p>

dhkim2810: <p>Hi, I’m having same error.</p>
<p>Environments</p>
<pre><code class="lang-auto">PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-172-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.6.55
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA GeForce RTX 3090
GPU 4: NVIDIA GeForce RTX 3090
GPU 5: NVIDIA GeForce RTX 3090
GPU 6: NVIDIA GeForce RTX 3090
GPU 7: NVIDIA GeForce RTX 3090

Nvidia driver version: 550.54.14
cuDNN version: Probably one of the following:
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.9.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.5
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn.so.8.9.5
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.5
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.5
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.5
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.5
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.5
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.5
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn.so.8.9.5
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.5
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.5
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.5
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.5
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.5
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.5
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn.so.8.6.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.6.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.6.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.6.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.6.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.6.0
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.6.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==1.12.1
[pip3] torchaudio==0.12.1
[pip3] torchvision==0.13.1
[conda] blas                      1.0                         mkl    conda-forge
[conda] cudatoolkit               11.6.2              hfc3e2af_13    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2023.1.0         h213fc3f_46344
[conda] numpy                     1.26.4          py310hb13e2d6_0    conda-forge
[conda] pytorch                   1.12.1          py3.10_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.1              py310_cu116    pytorch
[conda] torchvision               0.13.1              py310_cu116    pytorch
</code></pre>
<p>NVCC</p>
<pre><code class="lang-auto">nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Fri_Dec_17_18:16:03_PST_2021
Cuda compilation tools, release 11.6, V11.6.55
Build cuda_11.6.r11.6/compiler.30794723_0
</code></pre>
<pre><code class="lang-auto">conda 24.1.1
</code></pre>
<p>Here are simple code to reproduce</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; t1 = torch.randn(4,12,1024,64).cuda()
&gt;&gt;&gt; t2 = torch.randn(4,12,1024,64).cuda()
&gt;&gt;&gt; t = torch.matmul(t1, t2.transpose(-1,-2))
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)
</code></pre>