RuntimeError: cublas runtime error : the GPU program failed to execute at at /opt/conda/conda-bld/pytorch_1549635019666/work/aten/src/THC/THCBlas.cu:258

Yiru_Shen: <p>Hi everyone, I tried to run the pointnet module on Pytorch but got the following error:<br>
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1549635019666/work/aten/src/THC/THCBlas.cu:258</p>
<p>It shows the error comes from F.linear(input, self.weight, self.bias).</p>
<p>The whole error message is:</p>
<p>==============<br>
Traceback (most recent call last):<br>
File “/home/yirus/software/pycharm-community-2018.3.4/helpers/pydev/pydevd.py”, line 1741, in <br>
main()<br>
File “/home/yirus/software/pycharm-community-2018.3.4/helpers/pydev/pydevd.py”, line 1735, in main<br>
globals = debugger.run(setup[‘file’], None, None, is_module)<br>
File “/home/yirus/software/pycharm-community-2018.3.4/helpers/pydev/pydevd.py”, line 1135, in run<br>
pydev_imports.execfile(file, globals, locals)  # execute the script<br>
File “/home/yirus/software/pycharm-community-2018.3.4/helpers/pydev/_pydev_imps/_pydev_execfile.py”, line 18, in execfile<br>
exec(compile(contents+"\n", file, ‘exec’), glob, loc)<br>
File “/home/yirus/Projects/pointnet.pytorch/train_classification.py”, line 78, in <br>
pred, _ = classifier(points)<br>
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/yirus/Projects/pointnet.pytorch/pointnet.py”, line 100, in forward<br>
x, trans = self.feat(x)<br>
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/yirus/Projects/pointnet.pytorch/pointnet.py”, line 100, in forward<br>
x, trans = self.feat(x)<br>
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/yirus/Projects/pointnet.pytorch/pointnet.py”, line 73, in forward<br>
trans = self.stn(x)<br>
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/yirus/Projects/pointnet.pytorch/pointnet.py”, line 47, in forward<br>
x = F.relu(self.bn4(self.fc1(x)))<br>
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py”, line 67, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py”, line 1352, in linear<br>
ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())</p>
<p>==============</p>
<p>I tried to search on the Internet but cannot find any posts that are helpful.</p>
<p>======================<br>
Here is the versions installed:</p>
<p>GPU info: GeForce RTX 2080<br>
$ nvidia-smi<br>
It shows “Driver Version: 415.18, CUDA version: 10.0”</p>
<p>$nvcc -V<br>
It shows “Cuda compilation tools, release 9.2, V9.2.148”</p>
<p>Pytorch version: 1.0.1.post2 (installed by anaconda3)<br>
Operation system: Fedora 28<br>
Python version: 3.7</p>
<p>I have a feeling that it’s related to the wrong version of CUDA or should I reinstall pytorch?</p>
<p>Thanks in advance!</p>

SimonW: <p>CUDA 9.2 doesn’t support turing gpus.</p>

asere: <p>I’m facing the same error:</p>
<p>RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653215914/work/aten/src/THC/THCBlas.cu:259</p>
<p>also from F.linear(input, self.weight, self.bias)</p>
<p>However, my CUDA Version is 10.0.130</p>
<p>I’m using:</p>
<ul>
<li>python 3.7.3</li>
<li>ubuntu 18.04.3</li>
<li>pytorch 1.1.0</li>
<li>GeForce RTX 2080Ti</li>
</ul>
<p>Is there a version mismatch?</p>

ptrblck: <p>Could you update PyTorch to the latest stable version, retry the code?<br>
If you are still seeing this error, could you please post a reproducible code snippet?</p>

asere: <p>Even though my CUDA version is 10.0.130 (obtained with <code>cat /usr/local/cuda/version.txt</code>), my CUDA toolkit version was 9.0. After updating this to 10.1.243, it seems to be solved now. Thank you</p>

praggya: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,I am facing the same issue, i am using a 3090 GPU cuda version 11.4,pytorch==1.0.0</p>

ptrblck: <p>PyTorch <code>1.0.0</code> doesn’t support CUDA11, which is needed for your 3090, so you would need to update to the current stable or nightly release (and select <code>CUDA11</code> as the CUDA version for the binaries).</p>

praggya: <p>Hi,inspite of upgrading torch to torch==1.9.1 and torchvision=0.10.1<br>
i am facing this issue<br>
Traceback (most recent call last):<br>
File “train.py”, line 357, in <br>
main()<br>
File “train.py”, line 112, in main<br>
model = Network(args.init_channels, num_domains, args.layers, criterion)<br>
File “/voyager-volume/code_1_test/code_1_test/code_1_test/code_1_test/code_1_test/model_search.py”, line 95, in <strong>init</strong><br>
self._initialize_alphas()<br>
File “/voyager-volume/code_1_test/code_1_test/code_1_test/code_1_test/code_1_test/model_search.py”, line 124, in _initialize_alphas<br>
self.alphas_normal = Variable(1e-3*torch.randn(k, num_ops).cuda(), requires_grad=True)<br>
RuntimeError: CUDA error: no kernel image is available for execution on the device<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>

ptrblck: <aside class="quote no-group" data-username="praggya" data-post="8" data-topic="37800">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/9fc348/48.png" class="avatar"> praggya:</div>
<blockquote>
<p>RuntimeError: CUDA error: no kernel image is available for execution on the device</p>
</blockquote>
</aside>
<p>This would indicate that the currently used PyTorch build doesn’t contain kernels for the right compute capability (8.6 in your case), which would happen if you’ve installed e.g. the CUDA10.2 binaries. Make sure to select the CUDA11.1 (or 11.3) runtime during the installation.</p>

praggya: <p>NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.</p>
<p>This is the root of the error actually,what do I need to do?</p>

ptrblck: <p>You need to install the binaries with the CUDA11.1 runtime.<br>
In the install instructions you see the CUDA selection:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8d35d8a7a38d1aee2540e1f3069c7ea432e4d85.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8d35d8a7a38d1aee2540e1f3069c7ea432e4d85" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8d35d8a7a38d1aee2540e1f3069c7ea432e4d85_2_690x338.png" alt="image" data-base62-sha1="o5v3Vl8HjOxuylWK98xAVwyTqip" width="690" height="338" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8d35d8a7a38d1aee2540e1f3069c7ea432e4d85_2_690x338.png, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8d35d8a7a38d1aee2540e1f3069c7ea432e4d85.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8d35d8a7a38d1aee2540e1f3069c7ea432e4d85.png 2x" data-dominant-color="F5E1DE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">842×413 33.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Click on “CUDA 11.1”, select other options, copy/paste the install command, and execute it in your terminal.</p>