Another RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

username: Marcelo
cooked: <p>Hi there. Having the RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code> trying to train a deep learning model in PyTorch0.</p>
<p>Here’s the environment configuration running "python -m torch.utils.collect_env’:</p>
<p>Collecting environment information…<br>
PyTorch version: 1.11.0<br>
Is debug build: False<br>
CUDA used to build PyTorch: 10.2<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.3 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)<br>
Python platform: Linux-5.13.0-40-generic-x86_64-with-glibc2.31<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: NVIDIA A100-PCIE-40GB<br>
GPU 1: NVIDIA A100-PCIE-40GB</p>
<p>Nvidia driver version: 495.29.05<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.1<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.21.5<br>
[pip3] torch==1.11.0<br>
[pip3] torchaudio==0.11.0<br>
[pip3] torchvision==0.12.0<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               10.2.89              h8f6ccaa_8    conda-forge<br>
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch<br>
[conda] mkl                       2021.4.0           h06a4308_640<br>
[conda] mkl-service               2.4.0            py39h3811e60_0    conda-forge<br>
[conda] mkl_fft                   1.3.1            py39hd3c417c_0<br>
[conda] mkl_random         1.2.2            py39hde0f152_0    conda-forge<br>
[conda] numpy                   1.21.5           py39he7a7128_1<br>
[conda] numpy-base          1.21.5           py39hf524024_1<br>
[conda] pytorch                  1.11.0          py3.9_cuda10.2_cudnn7.6.5_0    pytorch<br>
[conda] pytorch-mutex       1.0                        cuda    pytorch<br>
[conda] torchaudio             0.11.0               py39_cu102    pytorch<br>
[conda] torchvision            0.12.0               py39_cu102    pytorch</p>
<p>I’ve installed prior cudatollkit versions (like 10.2) using anaconda, but the GPU seems to don’t recognize any driver minor then version 11. Any tip to solve the problem ?</p>
<p>Cheers.</p>

username: Marcelo
cooked: <p>Trying another configuration, with cudatoolkit 11.1, pytorch 1.8.2, it returns another error:</p>
<p>/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py", line 145, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: CUDA error: device-side assert triggered<br>
/opt/conda/conda-bld/pytorch_1627336325426/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [0,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.</p>
<p>The new environment configuration after drivers upgrades is below:</p>
<p>PyTorch version: 1.8.2<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.3 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect</p>
<p>Python version: 3.8 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: NVIDIA A100-PCIE-40GB<br>
GPU 1: NVIDIA A100-PCIE-40GB</p>
<p>Nvidia driver version: 495.29.05<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.1<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.21.5<br>
[pip3] torch==1.8.2<br>
[pip3] torchvision==0.9.2<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia<br>
[conda] mkl                       2021.4.0           h06a4308_640<br>
[conda] mkl-service               2.4.0            py38h7f8727e_0<br>
[conda] mkl_fft                   1.3.1            py38hd3c417c_0<br>
[conda] mkl_random                1.2.2            py38h51133e4_0<br>
[conda] numpy                     1.21.5           py38he7a7128_1<br>
[conda] numpy-base                1.21.5           py38hf524024_1<br>
[conda] pytorch                   1.8.2           py3.8_cuda11.1_cudnn8.0.5_0    pytorch-lts<br>
[conda] torchvision               0.9.2                py38_cu111    pytorch-lts</p>

username: ptrblck
cooked: <p>The first error is raised since you are using CUDA10.2 for the A100, which needs CUDA11+.</p>
<p>The current error is raised in <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code> as you are passing invalid target values to this loss function.<br>
These loss functions expect an input tensor in the shape <code>[batch_size, nb_classes]</code> and a target in the shape <code>[batch_size]</code> containing values in the range <code>[0, nb_classes-1]</code>.</p>