CUDA error: CUBLAS_STATUS_EXECUTION_FAILED

username: Yitao_Chen
cooked: <p>I ran into a strange problem while compiling pytorch from source to support MPI backend.</p>
<p>My setup:<br>
4 rtx 2080ti GPU<br>
pytorch 1.6.0<br>
cuda version 10.1</p>
<p>The problem happens when I tried to compile pytorch 1.6.0 from source.<br>
I followed the steps from the pytorch website. <a href="https://github.com/pytorch/pytorch#from-source" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
The compilation when smoothly and I was able to see all the 4 GPUs on my machine.</p>
<p>But when I tested with a simple code that I have been using for a while, I got the following error message.</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code></p>
<p>The same code runs perfectly when I use the pytorch install directly from pip on the same machine.<br>
Can anyone give some advices to fix this problem?</p>

username: Yitao_Chen
cooked: <p>Following is the script that I used to compile pytorch from source.</p>
<pre><code class="lang-auto">mkdir $HOME/.openmpi/
wget https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.gz
gunzip -c openmpi-3.0.0.tar.gz | tar xf - \
    &amp;&amp; cd openmpi-3.0.0 \
    &amp;&amp; ./configure --prefix=$HOME/.openmpi/ --with-cuda=/usr/local/cuda-10.1 \
    &amp;&amp; make all install
export PATH=$HOME/.openmpi/bin:$PATH
export LD_LIBRARY_PATH=$HOME/.openmpi/lib:$LD_LIBRARY_PATH

######### add cuda directories 

export CUDA_NVCC_EXECUTABLE="/usr/local/cuda-10.1/bin/nvcc"
export CUDA_HOME="/usr/local/cuda-10.1/"
export CUDNN_INCLUDE_PATH="/usr/local/cuda-10.1/include/"
export CUDNN_LIBRARY_PATH="/usr/local/cuda-10.1/lib64/"
export LIBRARY_PATH="/usr/local/cuda-10.1/lib64"
export USE_CUDA=1 USE_CUDNN=1 USE_MKLDNN=1 MAX_JOBS=80


export ENV_NAME=pt16
conda create --name $ENV_NAME python=3.7.3 numpy=1.16.3 
conda activate $ENV_NAME 
conda install numpy pyyaml mkl mkl-include setuptools cmake cffi typing_extensions

# use cuda101 to match with the system 
conda install -c pytorch magma-cuda101 
export CMAKE_PREFIX_PATH="$HOME/anaconda3/envs/$ENV_NAME"
cd ~/anaconda3/envs/$ENV_NAME/compiler_compat
mv ld ld-old
sudo apt-get install libomp-dev
cd ~
git clone --recursive https://github.com/pytorch/pytorch pytorch &amp;&amp; \
cd pytorch &amp;&amp; \
git checkout v1.6.0 --recurse-submodules &amp;&amp; \
python setup.py install | tee 2&gt;&amp;1 &gt; compile.log
</code></pre>

username: ptrblck
cooked: <p>I don’t know which pip wheel you’ve installed, but you could try to rebuild PyTorch with the same cublas version and check, if you might be seeing an already fixed issue.<br>
Also, make sure that you are not running out of memory, as cublas might raise this unhelpful error message if it’s unable to allocate memory internally.<br>
In any case, I would also recommend to stick to the latest PyTorch release (<code>1.8.1</code>), as it ships with the latest bug fixes (you might also hit an internal PyTorch issue, which might have been fixed after the <code>1.6.0</code> release).</p>

username: Yitao_Chen
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thanks for your response.<br>
I tried to build PyTorch from source using different versions.<br>
I have tested 1.6, 1.7 and the latest one, 1.9. But all of them are having the same cublas issue.<br>
The memory is not running out when I tested by build.<br>
Because I can run the same code on the same machine changing to the PyTorch version that I directly install using pip.<br>
I believe I have set all the CUDA related environmental variables correctly reflect my system setting; the configure step at the start of the build was able to pick up the corresponding CUDA paths I needed.</p>

username: ptrblck
cooked: <p>Thanks for the update. Which cublas version are you using in your local installation?</p>

username: Yitao_Chen
cooked: <p>Should be cublas 10.1.<br>
Because I specifically set all the environmental variables point to my CUDA 10.1 path.<br>
Is there a way I can check?</p>

username: ptrblck
cooked: <p>Could you update to cublas10.2.2, which ships with CUDA10.2?<br>
This CUDA toolkit (with the cublas version) is used in the binaries. Since you are using an older cublas release, you might hit an already fixed issue.</p>