Nvidia GTX 1080 - RuntimeError: CUDA error: no kernel image is available for execution on the device

BhandarkarPawan: <p>Hi!<br>
I am trying to run some code within the pytorch/pytorch:nightly-devel-cuda9.2-cudnn7 image container.<br>
If I try to run <code>torch.cuda.is_available()</code> it returns true</p>
<p>But when I run <code>result = model(return_loss=False, rescale=not show, **data)</code> it throws : <code>RuntimeError: CUDA error: no kernel image is available for execution on the device (launch_kernel at /opt/conda/conda-bld/pytorch_1579022071458/work/aten/src/ATen/native/cuda/Loops.cuh:103)</code></p>
<p>output of <code>nvidia-smi</code>:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75561c00713ea9ff7be4d9c42a490d1c90a38a87.png" data-download-href="https://discuss.pytorch.org/uploads/default/75561c00713ea9ff7be4d9c42a490d1c90a38a87" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75561c00713ea9ff7be4d9c42a490d1c90a38a87.png" alt="image" data-base62-sha1="gK0lRaEk9eVcQfdTaNccfWOs0MD" width="690" height="324" data-dominant-color="0D0D0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1446×680 32.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>output of <code>nvcc --version</code>:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/196dc0666b85f3fca3b60a336d686b82fe35e566.png" data-download-href="https://discuss.pytorch.org/uploads/default/196dc0666b85f3fca3b60a336d686b82fe35e566" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/196dc0666b85f3fca3b60a336d686b82fe35e566.png" alt="image" data-base62-sha1="3CX4RmVQWDIM9pn8cZkHGdKn7fw" width="690" height="119" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">842×146 11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>output of <code>conda list | grep torch</code><br>
pytorch                   1.4.0           py3.7_cuda9.2.148_cudnn7.6.3_0    pytorch<br>
torchvision               0.5.0                 py37_cu92    pytorch</p>
<p>My Ubuntu version is 16.04<br>
Please help!</p>

ptrblck: <p>Could you update to the latest PyTorch release (<code>1.8.0</code>) or the nightly binary?<br>
<code>1.4.0</code> is quite old by now and I don’t know what might have caused these issues back then.</p>

BhandarkarPawan: <p>Unfortunately, I am trying to run some code that depends on Cuda 9.2 and torch==1.4.0<br>
Is there any workaround for this?</p>

ptrblck: <p>This error is usually raised, if the application wasn’t compiled for your GPU architecture.<br>
I don’t know how PyTorch was built in your setup, but you might need to rebuild it either locally on the node with the desired GPU (which should detect the compute capability) or by specifying it manually via <code>TORCH_CUDA_ARCH_LIST=6.1</code>.</p>

BhandarkarPawan: <p>I installed pytorch using</p>
<pre><code class="lang-auto">conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=9.2 -c pytorch
</code></pre>
<p>From here <a href="https://pytorch.org/get-started/previous-versions/" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a></p>
<p>I will try reinstalling!</p>

BhandarkarPawan: <p>Unfortunately, reinstalling does not seem to help.</p>

ptrblck: <p>I don’t think that reinstalling would help and you can check the built-in GPU architectures using <a href="https://discuss.pytorch.org/t/a-ptx-jit-compilation-failed/57927/2">this method</a>.<br>
I guess that <code>sm_61</code> might not have been shipped in this binary.</p>