Linear layer throwing CUBLAS_STATUS_INVALID_VALUE error

clueless_student: <p>I recently updated my Python/PyTorch/cudatoolkit to 3.10.8, 1.13.1, and 11.7, respectively, on a computer cluster.</p>
<p>I started seeing the following error (CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code>) at a specific layer in a ML model I am running. This error was reproduced when I simplified it to 2 layers (nn.Conv1d and then nn.Linear). Wondering why this is happening when it wasn’t an issue before.</p>
<p>Code:</p>
<pre><code class="lang-auto">device = 'cuda:0'
rr = torch.zeros([2,20,5000]).to(device)
layer1 = nn.Conv1d(20,500,kernel_size=4,stride=4,groups=20,bias=False).to(device)
layer2 = nn.Linear(500,768).to(device)
l1out = layer1(rr)
l2out = layer2(l1out.transpose(1,2))
</code></pre>
<p>Dimension breakdown:</p>
<p>Input: torch.float32, shape [2,20,5000]</p>
<p>nn.Conv1d: in_channels 20, out_channels 500, kernel and stride both 4, 20 groups, no bias</p>
<p>nn.Linear: in_channels 500, out_channels 768</p>
<p>Input dimensions are transposed before being processed by the linear layer (so input to linear layer is [2,1250,500])</p>
<p>The error occurs in the linear layer:</p>
<blockquote>
<p>File “/hpc/group/collinslab/xc130/.conda/amll/lib/python3.10/site-packages/torch/nn/modules/linear.py”, line 114, in forward<br>
return F.linear(input, self.weight, self.bias)</p>
</blockquote>
<blockquote>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling <code>cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)</code></p>
</blockquote>
<p>Notes:</p>
<ul>
<li>The code runs fine on CPU, but not when I do the same operations on GPU. It’s happening on different partitions with different GPUs (including NVIDIA RTX A5000 and NVIDIA GeForce RTX 2080 Ti) with 8GB allocated.</li>
<li>No error when I directly input a tensor of zeros into the linear layer (as opposed to taking the output of the convolutional layer).</li>
<li>No error when I replace the first (convolutional) layer with a linear layer</li>
</ul>

ptrblck: <p>I cannot reproduce the issue in <code>1.13.1+cu117</code> on a 3090, but will try to use a 2080Ti next.<br>
Thanks for sharing the minimal and executable code btw.!</p>

clueless_student: <p>I had initially set up the environment with Python 3.11.1 and downgraded to 3.10.8 - could that have caused an issue with the cudatoolkit? Otherwise, do you know what issues can result in this error?</p>

ptrblck: <p>I doubt the Python version would interact with the <code>nvidia</code> pip wheels as these are shipping libraries and do not depend on any Python feature.<br>
I tried to reproduce the issue on a 2080Ti and it still works for me in a new environment:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.1+cu117'
&gt;&gt;&gt; device = "cuda:1"
&gt;&gt;&gt; print(torch.cuda.get_device_properties(device))
_CudaDeviceProperties(name='NVIDIA GeForce RTX 2080 Ti', major=7, minor=5, total_memory=12028MB, multi_processor_count=70)
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; rr = torch.zeros([2,20,5000]).to(device)
&gt;&gt;&gt; layer1 = nn.Conv1d(20,500,kernel_size=4,stride=4,groups=20,bias=False).to(device)
&gt;&gt;&gt; layer2 = nn.Linear(500,768).to(device)
&gt;&gt;&gt; l1out = layer1(rr)
&gt;&gt;&gt; l2out = layer2(l1out.transpose(1,2))
&gt;&gt;&gt; l2out
tensor([[[-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         ...,
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031]],

        [[-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         ...,
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031],
         [-0.0075,  0.0157,  0.0326,  ...,  0.0028, -0.0014, -0.0031]]],
       device='cuda:1', grad_fn=&lt;AddBackward0&gt;)
</code></pre>