CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

username: George_Guliman
cooked: <p>Hello,</p>
<p>I am trying to run a simple model using GPU acceleration. I am currently encountering 2 different issues with this.</p>
<ol>
<li>Whatever cuda-pytorch combination I use, it always takes around 15 minutes to execute the first instruction on the GPU ( no matter of the instruction executed).</li>
<li>For my model, I always and error although, it runs fine on the CPU.<br>
For 1), i am trying to install pytorch from source to see if anything works differently.<br>
For 2), I am completely stuck.</li>
</ol>
<p>The model I am using is below:</p>
<blockquote>
<p>import matplotlib.pyplot as plt<br>
import torch<br>
from torchvision import datasets, transforms<br>
import helper<br>
import numpy as np</p>
<p>data_dir = ‘…/Cat_Dog_data’</p>
<h1>TODO: Define transforms for the training data and testing data</h1>
<p>train_transforms = transforms.Compose([transforms.RandomRotation(30),<br>
transforms.RandomResizedCrop(224),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.Grayscale(),<br>
transforms.ToTensor()])</p>
<p>test_transforms = transforms.Compose([transforms.Resize(255),<br>
transforms.CenterCrop(224),<br>
transforms.Grayscale(),<br>
transforms.ToTensor()])</p>
<h1>Pass transforms in here, then run the next cell to see how the transforms look</h1>
<p>train_data = datasets.ImageFolder(data_dir + ‘/train’, transform=train_transforms)<br>
test_data = datasets.ImageFolder(data_dir + ‘/test’, transform=test_transforms)</p>
<p>trainloader = torch.utils.data.DataLoader(train_data, batch_size=32)<br>
testloader = torch.utils.data.DataLoader(test_data, batch_size=32)<br>
len(test_data)</p>
<p>from torch import nn, optim<br>
import torch.nn.functional as F</p>
<h1>if torch.cuda.is_available():</h1>
<h1>dev = “cuda:0”</h1>
<h1>else:</h1>
<h1>dev = “cpu”</h1>
<h1>print("device is: " + dev)</h1>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
<span class="hashtag-raw">#device</span> = torch.device(dev)</p>
<p>class Classifier(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.fc0 = nn.Linear(50176, 784)<br>
self.fc1 = nn.Linear(784, 256)<br>
self.fc2 = nn.Linear(256, 128)<br>
self.fc3 = nn.Linear(128, 64)<br>
self.fc4 = nn.Linear(64, 2)</p>
<pre><code>def forward(self, x):
    # make sure input tensor is flattened
    print('The shape of X is: ')
    print(x.shape)
    x = x.view(x.shape[0], -1)
    print('The shape of X flattened is: ')
    print(x.shape)
    
    x = F.relu(self.fc0(x))
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    x = F.log_softmax(self.fc4(x), dim=1)
    
    print('The shape of X is: ')
    print(x.shape)
    
    return x
</code></pre>
<p>model= Classifier()<br>
model = model.to(device)<br>
criterion = nn.NLLLoss()<br>
optimizer = optim.Adam(model.parameters(), lr=0.0003)</p>
<p>print(“Our model: \n\n”, model, ‘\n’)<br>
print(“The state dict keys: \n\n”, model.state_dict().keys())</p>
<p>epochs = 2<br>
steps = 0</p>
<p>train_losses, test_losses, test_acc = <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span>, <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
<span class="hashtag-raw">#Training</span> pass<br>
for e in range(epochs):<br>
running_loss = 0<br>
for images, labels in trainloader:<br>
images = images.to(device)<br>
print("Device for images is: ", images.get_device())<br>
labels = labels.to(device)<br>
optimizer.zero_grad()</p>
<pre><code>    log_ps = model(images)
    loss = criterion(log_ps, labels)
    loss.backward()
    optimizer.step()
   
    running_loss +=loss.item()

#Validation pass
else:
    with torch.no_grad():
        test_loss = 0
        running_accuracy = 0
        for images, labels in testloader:
            images = images.to(device)
            labels = labels.to(device)
            
            optimizer.zero_grad()

            print("Asta vrem, asta vreem: ", images.shape)
            log_psV = model(images)
            print(log_psV.shape, labels.shape)
            test_loss += criterion(log_psV, labels)

            psV = torch.exp(log_psV)
            top_p, top_class = psV.topk(1, dim=1)
            equals = top_class == labels.view(*top_class.shape)
            accuracy = torch.mean(equals.type(torch.FloatTensor))
            running_accuracy += accuracy.item()
            
    train_losses.append(running_loss/len(trainloader))
    test_losses.append(test_loss/len(testloader))
    test_acc.append(running_accuracy/len(testloader))
    
    print(f'Epoch: {e+1} epochs')
    print(f'Training Loss: {running_loss/len(trainloader)}')
    print(f'Test Loss: {test_loss/len(testloader)}')
    print(f'Test Accuracy: {(running_accuracy/len(testloader))*100}%')
</code></pre>
<p>plt.plot(range(epochs), train_losses, label=‘Training Loss’)<br>
plt.plot(range(epochs), test_losses, label=‘Test Loss’)<br>
plt.plot(range(epochs), test_acc, label=‘Accuracy’)<br>
plt.legend()</p>
</blockquote>
<p>The output and error I am getting is this:</p>
<p>Our model:</p>
<p>Classifier(<br>
(fc0): Linear(in_features=50176, out_features=784, bias=True)<br>
(fc1): Linear(in_features=784, out_features=256, bias=True)<br>
(fc2): Linear(in_features=256, out_features=128, bias=True)<br>
(fc3): Linear(in_features=128, out_features=64, bias=True)<br>
(fc4): Linear(in_features=64, out_features=2, bias=True)<br>
)</p>
<p>The state dict keys:</p>
<p>odict_keys([‘fc0.weight’, ‘fc0.bias’, ‘fc1.weight’, ‘fc1.bias’, ‘fc2.weight’, ‘fc2.bias’, ‘fc3.weight’, ‘fc3.bias’, ‘fc4.weight’, ‘fc4.bias’])<br>
Device for images is:  0<br>
The shape of X is:<br>
torch.Size([32, 1, 224, 224])<br>
The shape of X flattened is:<br>
torch.Size([32, 50176])</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
63         optimizer.zero_grad()<br>
64<br>
—&gt; 65         log_ps = model(images)<br>
66         loss = criterion(log_ps, labels)<br>
67         loss.backward()</p>
<p>~.conda\envs\pytorch18-cuda111\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
887             result = self._slow_forward(*input, **kwargs)<br>
888         else:<br>
 → 889             result = self.forward(*input, **kwargs)<br>
890         for hook in itertools.chain(<br>
891                 _global_forward_hooks.values(),</p>
<p> in forward(self, x)<br>
31         print(x.shape)<br>
32<br>
—&gt; 33         x = F.relu(self.fc0(x))<br>
34         x = F.relu(self.fc1(x))<br>
35         x = F.relu(self.fc2(x))</p>
<p>~.conda\envs\pytorch18-cuda111\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
887             result = self._slow_forward(*input, **kwargs)<br>
888         else:<br>
 → 889             result = self.forward(*input, **kwargs)<br>
890         for hook in itertools.chain(<br>
891                 _global_forward_hooks.values(),</p>
<p>~.conda\envs\pytorch18-cuda111\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)<br>
92<br>
93     def forward(self, input: Tensor) → Tensor:<br>
—&gt; 94         return F.linear(input, self.weight, self.bias)<br>
95<br>
96     def extra_repr(self) → str:</p>
<p>~.conda\envs\pytorch18-cuda111\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)<br>
1751     if has_torch_function_variadic(input, weight):<br>
1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)<br>
 → 1753     return torch._C._nn.linear(input, weight, bias)<br>
1754<br>
1755</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code></p>
<p>Does anyone have any clue on why I am getting this behaviour?</p>
<p>Thank you in advance,</p>

username: ptrblck
cooked: <ol>
<li>
<p>This sounds as if the JIT would kick in and compile the CUDA kernels for your architecture, which seem to be missing from the used binary.<br>
How did you install/build PyTorch and which GPU are you using?</p>
</li>
<li>
<p>Once 1. is solved, this might also be fixed.</p>
</li>
</ol>

username: George_Guliman
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I have an RTX2070 GPU and I tried installing torch with cuda via conda, by following the instructions from here <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a>.<br>
I tried with both cuda 10.2 and cuda 11.1. I also tried installing the conda packages or to directly install the packages via pip, in the conda env.<br>
I am now trying to install from source but I am stuck here as well with <a href="https://discuss.pytorch.org/t/pytorch-cuda-11-2-build-from-source-runtimeerror-cuda-error-no-kernel-image-is-available-for-execution-on-the-device/116392">this problem</a> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: ptrblck
cooked: <p>Which PyTorch version are you installing from conda, as I would like to reproduce this issue?<br>
Also, what kind of model are you running?</p>

username: George_Guliman
cooked: <p>I am using the following versions:<br>
python                   3.8.5           h5fd99cc_1<br>
pytorch                  1.8.1           py3.8_cuda11.1_cudnn8_0<br>
torchaudio             0.8.1           py38<br>
torchvision             0.9.1           py38_cu111</p>
<p>For the model, I am using a simple MLP network with 4 hidden layers. I posted the entire source code above.</p>
<p>The code just for the network is:</p>
<blockquote>
<p>class Classifier(nn.Module):<br>
def <strong>init</strong> (self):<br>
super().<strong>init</strong> ()<br>
self.fc0 = nn.Linear(50176, 784)<br>
self.fc1 = nn.Linear(784, 256)<br>
self.fc2 = nn.Linear(256, 128)<br>
self.fc3 = nn.Linear(128, 64)<br>
self.fc4 = nn.Linear(64, 2)<br>
def forward(self, x):<br>
# make sure input tensor is flattened<br>
print('The shape of X is: ')<br>
print(x.shape)<br>
x = x.view(x.shape[0], -1)<br>
print('The shape of X flattened is: ')<br>
print(x.shape)</p>
<pre><code>x = F.relu(self.fc0(x))
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = F.relu(self.fc3(x))
x = F.log_softmax(self.fc4(x), dim=1)

print('The shape of X is: ')
print(x.shape)

return x
</code></pre>
</blockquote>

username: George_Guliman
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>So it seems that it really isn’t any solution for me to use pytorch with cuda <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>I am completely stuck and frustration is kicking in…</p>
<p>It seems that I am reaching the same dead end, even if I install pytorch from source or if I use the provided binaries. The below output is for pytorch nightly build with cuda 10.2:</p>
<blockquote>
<blockquote>
<p>import torch<br>
x=torch.randn(1024, 1024).cuda()<br>
y = torch.matmult(x, x) &lt;------ this takes around 20 minutes to execute<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling &gt;<code>cublasCreate(handle)</code><br>
torch.<strong>version</strong><br>
‘1.9.0.dev20210409’</p>
<blockquote>
<p>torch.version.cuda<br>
‘10.2’<br>
torch.cuda.get_arch_list()<br>
[‘sm_37’, ‘sm_50’, ‘sm_60’, ‘sm_61’, ‘sm_70’, ‘sm_75’, ‘compute_37’]<br>
torch.cuda.get_device_name()<br>
‘GeForce RTX 2070’</p>
</blockquote>
</blockquote>
</blockquote>
<p>The same results I got on the installed from source variant of pytorch. Details <a href="https://here" rel="noopener nofollow ugc">https://discuss.pytorch.org/t/pytorch-cuda-11-2-build-from-source-runtimeerror-cuda-error-no-kernel-image-is-available-for-execution-on-the-device/116392/7</a></p>
<p>Any help, any tip is greatly appreciated.</p>
<p>Thank you,</p>

username: ptrblck
cooked: <p>I don’t know what might be causing this issue, as it seems no workflow (neither the binaries nor a source build) fixes it in your setup.<br>
Unfortunately I’m unable to reproduce the issue using:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc0 = nn.Linear(50176, 784)
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Linear(64, 2)

    def forward(self, x):
        # make sure input tensor is flattened
        print('The shape of X is: ')
        print(x.shape)
        x = x.view(x.shape[0], -1)
        print('The shape of X flattened is: ')
        print(x.shape)

        x = F.relu(self.fc0(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.log_softmax(self.fc4(x), dim=1)

        print('The shape of X is: ')
        print(x.shape)

        return x

device = 'cuda:0'
print(torch.cuda.get_device_name(device))

model = Classifier().to(device)
for i in [1, 8, 16, 32]:
    x = torch.randn(i, 50176, device=device)
    out = model(x)
    print(out.shape)
</code></pre>
<p>on an RTX2080Ti and the <code>1.8.1+CUDA10.2</code> binaries, so I guess your system setup might not work properly and would recommend to update the NVIDIA drivers etc.</p>

username: George_Guliman
cooked: <p>ok, i will try it on linux. hopes it works there :))</p>

username: Haris_Cheong
cooked: <p>Im having the same issue. I realized that the torch.nn.Linear layers are the problem as when I change it to a fully ocnvolutional network it runs without the cublas error.</p>
<p>System Specs:</p>
<ol>
<li>NVIDIA-SMI 465.19.01    Driver Version: 465.19.01    CUDA Version: 11.3 (installed with runfile so drivers come together)</li>
<li>Installed pytorch using this command: pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a>
</li>
<li>Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-73-generic x86_64)</li>
<li>GeForce GTX 1080 Ti</li>
</ol>
<p>To reproduce:</p>
<pre><code class="lang-auto">import os
os.environ['CUDA_VISIBLE_DEVICES']='0'
import torch

class Classifier(torch.nn.Module):
    def __init__(self):
        super().__init__()
        #classifier
        self.linear_1 = torch.nn.Linear(32*8*4*8,32*8*4)
        self.linear_2 = torch.nn.Linear(32*8*4,32*8)
        self.batch_norm_1 = torch.nn.BatchNorm1d(32*8)

        self.linear_3 = torch.nn.Linear(32*8,32)
        self.linear_4 = torch.nn.Linear(32,1)

        self.activation = torch.nn.ReLU()


    def forward(self,x):
        x = x.reshape(x.shape[0], -1)
        x = self.linear_1(x)
        x = self.activation(x)
        x = self.linear_2(x)
        x = self.activation(x)
        x = self.batch_norm_1(x)

        x = self.linear_3(x)
        x = self.activation(x)
        x = self.linear_4(x)
        return x
    
net = Classifier().cuda()
inp = torch.rand(2,32,8,4,8).cuda()
gt = torch.rand(2,1).cuda()
outp = net(inp)
loss = torch.nn.BCEWithLogitsLoss()(outp, gt)
loss.backward()
</code></pre>
<p>Error observed:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-1-a9db9cb37cd0&gt; in &lt;module&gt;
     35 outp = net(inp)
     36 loss = torch.nn.BCEWithLogitsLoss()(outp, gt)
---&gt; 37 loss.backward()

/disk4/haris/envs/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    243                 create_graph=create_graph,
    244                 inputs=inputs)
--&gt; 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    246 
    247     def register_hook(self, hook):

/disk4/haris/envs/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    143         retain_graph = create_graph
    144 
--&gt; 145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
</code></pre>

username: ptrblck
cooked: <p>You are most likely hitting <a href="https://github.com/pytorch/pytorch/issues/56747">this issue</a>, which is already fixed in the nightly CUDA11 pip wheels (and was not an issue in other binary configs), so you could either update to the nightly pip wheel, or use the CUDA10.2 or conda binaries.</p>

username: Siladittya_Manna
cooked: <p>I faced the same issue when trying to implement the SimCLR framework.</p>
<p>The files are available here<br>
(<a href="https://github.com/sadimanna/ssl_models/tree/main/src" class="inline-onebox" rel="noopener nofollow ugc">ssl_models/src at main · sadimanna/ssl_models · GitHub</a>)</p>
<p>To reproduce the error I would suggest running ‘python main.py --batch_size 8 --download True’. This will download the CIFAR-10 dataset (163MB).</p>
<p>But when I run the same code on Colab (<a href="https://colab.research.google.com/drive/1I8rMKK5Hw2DDEFscH_9XJiciT3e8Awdf?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colaboratory</a>) I don’t get the error.</p>
<p>I uploaded the files on Colab and ran the file main.py and got the error again. The notebook with the error is below<br>
(<a href="https://colab.research.google.com/drive/1w1DAWGoYF-3bRu0OuGgZVh7UNILl2295?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colaboratory</a>)</p>
<p>I can’t understand if it is an issue of CUDA or anything else??</p>

username: ptrblck
cooked: <p>Have you checked the linked issue and made sure to either update to <code>1.9.0</code> or use the conda binaries?</p>

username: Siladittya_Manna
cooked: <p>I had the issue on my Laptop too. I had 1.9.0 installed on it. And Cuda 11.1. Python version is 3.9.</p>
<p>It is confusing how the same code produces different results when ran in a different way. The code in the first Colab notebook link I gave in my last reply is just segregated into separate files, like one fike for optimizer, loss and data modules (this code is in the github link) when I run it I get the error. Which is strange because both are the same code.</p>
<p>Also on Google Colab the torch version is 1.9.0 and CUDA 10.2 and the error still occurred.</p>

username: ptrblck
cooked: <p>Thanks for the update. Could you post a minimal, executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>?</p>

username: Siladittya_Manna
cooked: <p>The code below should reproduce the error</p>
<pre><code class="lang-auto">!git clone https://github.com/sadimanna/ssl_models.git
!pip install pytorch-lightning pytorch-lightning-bolts
!python ssl_models/src/main.py --batch_size 16 --gpus 1 --download True
</code></pre>
<p>Please reply if it does not. Otherwise, the outputs can be seen in this Colab notebook (<a href="https://colab.research.google.com/drive/1w1DAWGoYF-3bRu0OuGgZVh7UNILl2295?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colaboratory</a>)</p>
<p>Output from <code>python -m torch.utils.collect_env</code> ?</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.9.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0
Libc version: glibc-2.26

Python version: 3.7 (64-bit runtime)
Python platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.9.0+cu102
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.10.0
[pip3] torchvision==0.10.0+cu102
[conda] Could not collect
</code></pre>

username: ptrblck
cooked: <p>Thanks for the repro instructions.<br>
This error is raised when executing it:</p>
<pre><code class="lang-python">Epoch 1
  0%|                                                                                                                                                                          | 0/2500 [00:01&lt;?, ?batch/s]
Traceback (most recent call last):
  File "ssl_models/src/main.py", line 96, in &lt;module&gt;
    main(args)
  File "ssl_models/src/main.py", line 53, in main
    trainer.fit()
  File "/workspace/src/ssl_models/src/trainer.py", line 51, in fit
    train_epoch_loss = self.train_epoch(self.model, self.train_loader, self.optimizer)
  File "/workspace/src/ssl_models/src/trainer.py", line 83, in train_epoch
    train_loss = model.training_step(batch, step)
  File "/workspace/src/ssl_models/src/simclr.py", line 115, in training_step
    loss = self.criterion(x1,x2)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File "/workspace/src/ssl_models/src/losses.py", line 32, in forward
    sim = self.similarity_f(z.unsqueeze(1), z.unsqueeze(0)) / self.temperature
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1056, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/distance.py", line 75, in forward
    return F.cosine_similarity(x1, x2, self.dim, self.eps)
RuntimeError: cosine_similarity requires both inputs to have the same sizes, but x1 has [32, 1, 128] and x2 has [1, 32, 128]
</code></pre>

username: Siladittya_Manna
cooked: <p>Thanks for the reply.<br>
I tried downgrading PyTorch and CUDA both on my laptop. Just a few hours ago.</p>
<p>I found out that the error was in the mismatch of dimensions between the input and the weights in the Linear layer I was using.</p>
<p>Cosine Similarity, however, is working fine on my laptop. And even in colab. I don’t understand why you are getting this error.</p>
<p>I then upgraded from torch 1.8.1 + cuda 10.2 to torch 1.9.0 + cuda 10.2 and then to cuda 11.1<br>
It is actually working without any errors for now.</p>

username: kkuchynskyi
cooked: <p>I also had the same problem. I my case the error was due incorrect input tensor into <em>nn.Linear</em> layer. The layer was initialized this way <em>nn.Linear(512 * 7 * 7, 4096)</em>,  but actual input tensor shape before resizing was [512, 6, 6].</p>
<p>My configuration: ubuntu, python 3.9, cuda 11.2, torch 1.9, rtx 3060</p>

username: my3bikaht
cooked: <p>Had same issue this morning too. Yet in my case it worked fine before, until I added new GPU via riser cable. Forced PCIe slot to switch to Gen3 mode, no issues for the last 30 min of training.<br>
So, problem can be not just software but hardware too.</p>

username: Jo-w
cooked: <p>Hi, I got the same issue and have solved it by adding a flatten layer before the linear layer. Feel free to try it out.</p>