About CUDA error: CUBLAS_STATUS_EXECUTION_FAILED

username: funny000
cooked: <h2>
<a name="when-i-use-the-torch171-and-cuda110-to-train-uperhead-transformeri-always-to-meet-problem-1" class="anchor" href="#when-i-use-the-torch171-and-cuda110-to-train-uperhead-transformeri-always-to-meet-problem-1"></a>When I use the torch1.7.1 and cuda11.0 to train uperhead-transformer，I always to meet problem.</h2>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 438, in 
    train(trainlog)
  File "train.py", line 240, in train
    tout, lout = net(img)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 159, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "train.py", line 103, in forward
    out = self.head(out0)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/train/models/head/uper_head.py", line 96, in forward
    laterals = [
  File "/train/models/head/uper_head.py", line 97, in 
    lateral_conv(inputs[i])
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/train/models/losses/utils.py", line 144, in forward
    out = self.conv(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
</code></pre>
<hr>
<h2>
<a name="and-my-torch-and-cuda-information-2" class="anchor" href="#and-my-torch-and-cuda-information-2"></a>and my torch and cuda information:</h2>
<pre><code class="lang-auto">Python 3.8.2
1.7.1
Linux pytorch-devel-171-prod-v-f5d57bb16238459eb3644237ada6279e-fqxcx 4.18.0-147.5.2.1.h579.eulerosv2r10.x86_64 #1 SMP Tue May 31 11:58:10 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
Linux version 4.18.0-147.5.2.1.h579.eulerosv2r10.x86_64 (root@dd110da1a451) (gcc version 7.3.0 (GCC)) #1 SMP Tue May 31 11:58:10 CST 2022
Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.2
[pip3] torchelastic==0.2.1
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl    defaults
[conda] cudatoolkit               11.0.221             h6bb024c_0    defaults
[conda] mkl                       2020.2                      256    defaults
[conda] mkl-service               2.3.0            py38he904b0f_0    defaults
[conda] mkl_fft                   1.2.0            py38h23d657b_0    defaults
[conda] mkl_random                1.1.1            py38h0573a6f_0    defaults
[conda] numpy                     1.22.3                   pypi_0    pypi
[conda] pytorch                   1.7.1           py3.8_cuda11.0.221_cudnn8.0.5_0    pytorch
[conda] torchaudio                0.7.2                    pypi_0    pypi
[conda] torchelastic              0.2.1                    pypi_0    pypi
[conda] torchvision               0.8.2                py38_cu110    pytorch
</code></pre>
<hr>
<p>So can you give me some advice about solve the problem?</p>

username: ptrblck
cooked: <p>Could you update PyTorch to the latest stable or nightly release and check if you are still running into this error, please?</p>

username: funny000
cooked: <p>en, i understand it, but the remote server manager is not give me the change rights of administrators, i only to provide advice how to change torch or cuda version for solve the problem.</p>
<p>and torch version is fixed in remote server, on the this torch version we have a lot of work, so only can change the cuda version, Based on your experience, which version of cuda is matching the torch version?</p>
<h2>
<a name="and-i-use-myself-local-server-to-train-the-same-network-it-not-the-problem-the-torch-and-cuda-information-in-myself-local-server-1" class="anchor" href="#and-i-use-myself-local-server-to-train-the-same-network-it-not-the-problem-the-torch-and-cuda-information-in-myself-local-server-1"></a>and i use myself local server to train the same network, it not the problem.<br>
the torch and cuda information in myself local server:</h2>
<p>Collecting environment information…<br>
PyTorch version: 1.7.1+cu110<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.0<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Amazon Linux 2 (x86_64)<br>
GCC version: (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15)<br>
Clang version: Could not collect<br>
CMake version: Could not collect</p>
<p>Python version: 3.7 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: GPU 0: Tesla T4<br>
Nvidia driver version: 460.106.00<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip] numpy==1.21.6<br>
[pip] torch==1.7.1+cu110<br>
[pip] torchaudio==0.7.2<br>
[pip] torchvision==0.8.2+cu110<br>
[conda] blas                      1.0                         mkl<br>
[conda] mkl                       2021.4.0           h06a4308_640<br>
[conda] mkl-service               2.4.0            py37h7f8727e_0<br>
[conda] mkl_fft                   1.3.1            py37hd3c417c_0<br>
[conda] mkl_random                1.2.2            py37h51133e4_0<br>
[conda] numpy                     1.21.6                   pypi_0    pypi<br>
[conda] numpy-base                1.21.5           py37hf524024_2<br>
[conda] torch                     1.7.1+cu110              pypi_0    pypi<br>
[conda] torchaudio                0.7.2                    pypi_0    pypi<br>
[conda] torchvision               0.8.2+cu110              pypi_0    pypi</p>
<hr>

username: ptrblck
cooked: <p>The issue is that you might be running into a known and already fixed error which won’t be backported to PyTorch or cublas in these older versions.</p>
<aside class="quote no-group" data-username="funny000" data-post="3" data-topic="158290">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/funny000/48/34829_2.png" class="avatar"> funny000:</div>
<blockquote>
<p>Based on your experience, which version of cuda is matching the torch version?</p>
</blockquote>
</aside>
<p>The PyTorch binaries often support multiple CUDA versions and you could select the one you want.<br>
Note that your locally installed CUDA toolkit will not be used unless you build PyTorch from source or build a custom CUDA extension, since the binaries ship with their own CUDA runtime.</p>
<p>In case you cannot directly install the latest PyTorch pip wheel on the server you might be able to use e.g. a docker container and install it there.</p>