RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

vanduong0504: <p>I am training my models from <code>Google Collab</code> with <code>batch_size = 128</code> after 1 epoch it has this problem. I don’t know have to fix it with the same batch_size (reduce batch_size to 32 can avoid this problem). Here is Colab spec: <code>driver Version: 460.32.03    CUDA Version: 11.2 </code><br>
You can find my notebook <a href="https://colab.research.google.com/drive/1Ixz0zaWUCEc_-oDN03Rib4JUR0UsKG_z?usp=sharing" rel="noopener nofollow ugc">here</a>.<br>
Thanks for your help.</p>

tom: <p>It seems that one of your operands is too large to fit in int32 (or negative, but that seems unlikely).</p>
<p>I thought that recent PyTorch will give a better error (but don’t work around it):</p>
<pre><code class="lang-python">import torch
LARGE = 2**31+1
for i, j, k in [(1, 1, LARGE), (1, LARGE, 1), (LARGE, 1, 1)]:
    inp = torch.randn(i, k, device="cuda", dtype=torch.half)
    weight = torch.randn(j, k, device="cuda", dtype=torch.half)
    try:
        torch.nn.functional.linear(inp, weight)
    except RuntimeError as e:
        print(e)
    del inp
    del weight
</code></pre>
<pre><code class="lang-auto">at::cuda::blas::gemm&lt;float&gt; argument k must be non-negative and less than 2147483647 but got 2147483649
at::cuda::blas::gemm&lt;float&gt; argument m must be non-negative and less than 2147483647 but got 2147483649
at::cuda::blas::gemm&lt;float&gt; argument n must be non-negative and less than 2147483647 but got 2147483649
</code></pre>
<p>But they don’t work around it. (It needs a lot of memory to trigger the bug…)</p>
<p>Maybe you can get a credible backtrace and record the input shapes to the operation that fails.</p>
<p>Best regards</p>
<p>Thomas</p>

vanduong0504: <p>So what can I do to solve this problem, I just know to change batch size to smaller.</p>

tom: <p>In order of difficulty:</p>
<ul>
<li>make batch size smaller,</li>
<li>make a minimal reproducing example (i.e. just two or three inputs from torch.random and the call to the torch.nn.functional.linear) and file a bug,</li>
<li>hot-patch torch.nn.functional.linear with a workaround (splitting the operation into multiple linear or matmul calls),</li>
<li>submit a PR with a fix in PyTorch and discuss whether you can add a test or whether it’d take a prohibitive large amount of GPU memory to run (or hire someone to do so).</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

vanduong0504: <p>Thank for your help.</p>

Jeremy_Cochoy: <p>For the peoples getting this error and ending up on this post, please know that it can also be caused if you have a mismatch between the dimension of your input tensor and the dimensions of your nn.Linear module. (ex. x.shape = (a, b) and nn.Linear(c, c, bias=False) with c not matching)</p>
<p>It is a bit sad that pytorch don’t give a more explicit error messages.</p>

Madhavan_Seshadri: <p><a class="mention" href="/u/jeremy_cochoy">@Jeremy_Cochoy</a>  This was really helpful. Solved my issue.</p>

Hugo: <p><a class="mention" href="/u/jeremy_cochoy">@Jeremy_Cochoy</a>  Thanks for your comments!</p>

martinoywa: <p><a class="mention" href="/u/jeremy_cochoy">@Jeremy_Cochoy</a> Thanks!</p>

SNair: <p>Hello <a class="mention" href="/u/jeremy_cochoy">@Jeremy_Cochoy</a><br>
I have added an nn.Linear(512,10) layer to my model and the shape of the input that goes into this layer is torch.Size([32,512,1,1]). I have tried reducing the batch size from 128 to 64 and now to 32, but each of these gives me the same error.<br>
Any idea what could be going wrong?</p>

Jeremy_Cochoy: <p>I think you want to transpose the dimensions of your input tensor before and after (<a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html" class="inline-onebox" rel="noopener nofollow ugc">Linear — PyTorch 1.9.0 documentation</a> say it expect a Nx*xC_in tensor and you give him a 32x…x1 tensor)</p>
<p>Something like <code>linear(x.transpose(1,3)).transpose(1,3)</code> ?</p>

NicoHambauer: <p>Thanks a lot also solved my Issue!</p>

markstein: <p>I got the same error because of a mismatch of the input dimensions in the first layer.</p>
<p>Thanks for the hint!</p>

Parshin_Shojaee: <p>helped me! thank you!</p>

black0017: <p>hello all,</p>
<p>I had this problem while I was using a smaller batch size (=4) for testing some code changes, while my initial batch size was 64. I checked the shapes for <code>nn.Linear</code> and they matched</p>
<p>After 1 hour I found that the only change was the batch size. By <strong>increasing batch size back to 64</strong> everything worked perfectly. Pytorch version 1.8.1. Not sure why this error is caused.</p>
<p>Hope it helps!</p>

Chuyang: <p>Thanks a lot! Solved my issue.</p>

wtliao: <p>Hello all,</p>
<p>I am using pytorch ‘1.13.0+cu117’, my env is  NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0</p>
<p>in the terminal of python, I tried the very simple example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x=torch.ones(2,2,1).to('cuda')
&gt;&gt;&gt; y=torch.ones(2,1,2).to('cuda')
&gt;&gt;&gt; x
tensor([[[1.],
         [1.]],

        [[1.],
         [1.]]], device='cuda:0')
&gt;&gt;&gt; y
tensor([[[1., 1.]],

        [[1., 1.]]], device='cuda:0')
&gt;&gt;&gt; y@x
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)`
&gt;&gt;&gt; torch.bmm(y,x)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)`
&gt;&gt;&gt; torch.matmul(y,x)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)`
&gt;&gt;&gt; x=torch.ones(2,1).to('cuda')
&gt;&gt;&gt; y=torch.ones(1,2).to('cuda')
&gt;&gt;&gt; y@x
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
&gt;&gt;&gt; torch.mm(y,x)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
&gt;&gt;&gt; torch.mm(x,y)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
&gt;&gt;&gt;
</code></pre>
<p>The issues are obviously not caused by the mismatch size. Anyone has any idea? thanks!</p>

ptrblck: <p>Could you post the output of <code>python -m torch.utils.collect_env</code>, please, as I cannot reproduce the error in <code>1.13.0+cu117</code> on a 3090.</p>

pt.megamozg80: <p>Hello! I had the same issue as <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-error-cublas-status-invalid-value-when-calling-cublassgemm-handle-opa-opb-m-n-k-alpha-a-lda-b-ldb-beta-c-ldc/124544/18">wtliao</a></p>
<p>This os my post of <code>python -m torch.utils.collect_env</code>:</p>
<p>PyTorch version: 1.13.0+cu117<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.7<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: CentOS Linux 7 (Core) (x86_64)<br>
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.17</p>
<p>Python version: 3.8.11 (default, Sep  1 2021, 12:33:46)  [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] (64-bit runtime)<br>
Python platform: Linux-3.10.0-1160.42.2.el7.x86_64-x86_64-with-glibc2.2.5<br>
Is CUDA available: True<br>
CUDA runtime version: 11.4.120<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration: GPU 0: Tesla T4<br>
Nvidia driver version: 470.57.02<br>
cuDNN version: Probably one of the following:<br>
/usr/lib64/libcudnn.so.8.2.4<br>
/usr/lib64/libcudnn_adv_infer.so.8.2.4<br>
/usr/lib64/libcudnn_adv_train.so.8.2.4<br>
/usr/lib64/libcudnn_cnn_infer.so.8.2.4<br>
/usr/lib64/libcudnn_cnn_train.so.8.2.4<br>
/usr/lib64/libcudnn_ops_infer.so.8.2.4<br>
/usr/lib64/libcudnn_ops_train.so.8.2.4<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.23.4<br>
[pip3] torch==1.13.0<br>
[pip3] torchaudio==0.13.0<br>
[pip3] torchcam==0.3.2<br>
[pip3] torchvision==0.14.0<br>
[conda] Could not collect</p>
<p>Also this error occurs while running</p>
<pre><code class="lang-auto">import torch.nn.functional as F
import torch
a = torch.rand((1, 2, 3)).to('cuda')
b = torch.rand((1, 3, 24, 94)).to('cuda')
grid = F.affine_grid(a, b.size())
</code></pre>
<p>File ~/.venv/default/lib64/python3.8/site-packages/torch/nn/functional.py:4332, in affine_grid(theta, size, align_corners)<br>
4329 elif min(size) &lt;= 0:<br>
4330     raise ValueError(“Expected non-zero, positive output size. Got {}”.format(size))<br>
 → 4332 return torch.affine_grid_generator(theta, size, align_corners)</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling <code>cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)</code></p>

ptrblck: <p>I cannot reproduce the issue on a T4 with <code>torch==1.13.0+cu117</code>:</p>
<pre><code class="lang-python">import torch
torch.cuda.get_device_name(0)
# 'Tesla T4'
torch.__version__
# '1.13.0+cu117'

import torch.nn.functional as F
import torch
a = torch.rand((1, 2, 3)).to('cuda')
b = torch.rand((1, 3, 24, 94)).to('cuda')
grid = F.affine_grid(a, b.size())

print(grid)
tensor([[[[0.4507, 0.2959],
          [0.4582, 0.3064],
          [0.4656, 0.3169],
          ...,
          [1.1288, 1.2493],
          [1.1363, 1.2597],
          [1.1437, 1.2702]],

         [[0.4635, 0.3081],
          [0.4710, 0.3186],
          [0.4784, 0.3291],
          ...,
          [1.1417, 1.2615],
          [1.1491, 1.2719],
          [1.1566, 1.2824]],

         [[0.4763, 0.3203],
          [0.4838, 0.3308],
          [0.4913, 0.3413],
          ...,
          [1.1545, 1.2736],
          [1.1619, 1.2841],
          [1.1694, 1.2946]],
...
</code></pre>