RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

monee_h.a: <p>Hi I have this error of :<br>
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code></p>
<p>I am just trying to follow what is in this tutorial:<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<p>It works fine on my machine:!<br>
<a href="/uploads/short-url/NKI5sqRTmYg8gngVdpVKHnuIwq.png">Screen Shot 2021-02-23 at 9.38.24 PM|621x500</a></p>
<p>but there is an error when I do the same on the server:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/354f8585c2ad19d21e8ca71423e599de7927bc75.png" data-download-href="https://discuss.pytorch.org/uploads/default/354f8585c2ad19d21e8ca71423e599de7927bc75" title="Screen Shot 2021-02-23 at 9.38.44 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/354f8585c2ad19d21e8ca71423e599de7927bc75_2_690x451.png" alt="Screen Shot 2021-02-23 at 9.38.44 PM" data-base62-sha1="7BBGamPWepyiuMDvWJb5B40DpFr" width="690" height="451" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/354f8585c2ad19d21e8ca71423e599de7927bc75_2_690x451.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/354f8585c2ad19d21e8ca71423e599de7927bc75_2_1035x676.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/354f8585c2ad19d21e8ca71423e599de7927bc75_2_1380x902.png 2x" data-dominant-color="F1DBDB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-02-23 at 9.38.44 PM</span><span class="informations">1830×1198 324 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would appreciate it if someone can help! Thanks</p>

ptrblck: <p>Could you post the setup of your server, i.e. the used GPU, CUDA and PyTorch versions, so that we could try to reproduce it, please?</p>

monee_h.a: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="112805">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>GPU</p>
</blockquote>
</aside>
<p>GPU: 8x NVIDIA Tesla Volta V100 GPUs with NVlink2<br>
CUDA Version: 11.2<br>
PyTorch Version: pytorch_19.09-py3</p>
<p>Thank you!</p>

monee_h.a: <p>GPU:<br>
NVIDIA-SMI 410.129      Driver Version: 410.129</p>

ptrblck: <p>Thanks for the information. I cannot reproduce this issue on a V100 with CUDA11.2 and am not sure what the PyTorch version 19.09 would refer to.<br>
Are you using an NGC container of this old version? If so, it wouldn’t ship with CUDA11.2, so could you post an update about your setup, please?</p>