Unable to troubleshoot RuntimeError: CUDA error: no kernel image is available for execution on the device

username: tangolin
cooked: <p>Full error trace:</p>
<pre><code class="lang-auto">File "/clipbert/src/modeling/transformers.py", line 198, in forward
    embeddings = self.dropout(embeddings)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/dropout.py", line 54, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 936, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: CUDA error: no kernel image is available for execution on the device
</code></pre>
<p>After browsing related topics, I realised that most seems to be that the torch and gpu compute compatibility is incompatible, however running</p>
<pre><code class="lang-auto">cuobjdump build/lib.linux-x86_64-3.7/torch/lib/libtorch.so | grep arch | sort | uniq
</code></pre>
<p>from another thread just returns me</p>
<pre><code class="lang-auto">cuobjdump info    : File '/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so' does not contain device code
</code></pre>
<p>and I have no idea what to make of this error message.</p>
<p>Here is the info about my environment, appreciate it if someone can confirm whether I have a compute compatibility issue.</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.5.1+cu101
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.14.0

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: 
GPU 0: Tesla K80
GPU 1: Tesla K80
GPU 2: Tesla K80
GPU 3: Tesla K80

Nvidia driver version: 430.64
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.4

Versions of relevant libraries:
[pip] msgpack-numpy==0.4.3.2
[pip] numpy==1.17.2
[pip] torch==1.5.1+cu101
[pip] torchtext==0.4.0
[pip] torchvision==0.6.1+cu101
[conda] magma-cuda100             2.1.0                         5    local
[conda] mkl                       2019.1                      144  
[conda] mkl-include               2019.1                      144  
[conda] nomkl                     3.0                           0  
[conda] torch                     1.5.1+cu101              pypi_0    pypi
[conda] torchtext                 0.4.0                    pypi_0    pypi
[conda] torchvision               0.6.1+cu101              pypi_0    pypi
</code></pre>

username: ptrblck
cooked: <p>The K80 has a compute capability of 3.7 and should be supported in current binaries.<br>
Could you update to the latest stable release and rerun your code?</p>
<p>You are not able to find any device code in <code>libtorch.so</code> as it would ship in <code>libtorch_cuda.so</code> and you can check the supported architectures via <code>print(torch.cuda.get_atch_list())</code>.</p>

username: tangolin
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> unfortunately there are a lot of other dependency issues when I use newer versions of pytorch so I am stuck with torch1.5.1, is there any other possible causes for this(such as nvidia driver version)?</p>

username: ptrblck
cooked: <p>This mentioned CUDA error should be raised by a missing architecture in the binaries and not caused by the driver etc.<br>
However, the 1.5.1 wheels for CUDA10.2 and 10.1 contain these device codes:</p>
<pre><code class="lang-python">arch = sm_35
arch = sm_37
arch = sm_50
arch = sm_60
arch = sm_61
arch = sm_70
arch = sm_75
</code></pre>