Trouble with CUDA capability sm_86

sharifinjf: <p>Hi, all<br>
I am trying to train a network on my NVIDIA RTX A4000. I receive the following error:<br>
NVIDIA RTX A4000 with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.<br>
pytorch-lightning       1.5.0<br>
torch                   1.11.0<br>
torchaudio              0.11.0<br>
torchmetrics            0.6.0<br>
torchvision             0.12.0<br>
CUDA Version      : 11.4<br>
Python 3.8.10</p>

ptrblck: <p>You’ve most likely install the pip wheel or conda binary with the CUDA 10.2 runtime, which doesn’t support your Ampere GPU. Select CUDA 11.3 in the install setup <a href="https://pytorch.org/get-started/locally/">here</a> and it should work.</p>

sharifinjf: <p>I did it. but after I received the following error:<br>
UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  …/c10/cuda/CUDAFunctions.cpp:112.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
No CUDA runtime is found, using CUDA_HOME=’/usr’</p>

ptrblck: <p><code>CUDA unknown error</code> could point to anything broken in your setup.<br>
If you get stuck, reinstall the NVIDIA drivers, then the conda binaries again and recheck the functionality.</p>

Govardhan: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , i had the same issue and reinstalled pytorch as you suggested but now i get the following error:</p>
<p>Process finished with exit code 137 (interrupted by signal 9: SIGKILL)</p>
<p>Can you please help me out with this?</p>

ptrblck: <p>Could you try to get a stacktrace via:</p>
<pre><code class="lang-auto">gdb --args python script.py args
...
run
...
bt
</code></pre>
<p>And post it here, please?</p>

Govardhan: <p>(venv) activelearning@activelearning:~/Documents/govardhan/invokeAI/InvokeAI/scripts$ gdb --args python invoke.py args<br>
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2<br>
Copyright (C) 2020 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later <a href="http://gnu.org/licenses/gpl.html" rel="noopener nofollow ugc">http://gnu.org/licenses/gpl.html</a><br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law.<br>
Type “show copying” and “show warranty” for details.<br>
This GDB was configured as “x86_64-linux-gnu”.<br>
Type “show configuration” for configuration details.<br>
For bug reporting instructions, please see:</p>
<p>Find the GDB manual and other documentation resources online at:</p>
<p>For help, type “help”.<br>
Type “apropos word” to search for commands related to “word”…<br>
Reading symbols from python…<br>
(No debugging symbols found in python)<br>
(gdb) run<br>
Starting program: /home/activelearning/Documents/govardhan/invokeAI/venv/bin/python invoke.py args<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.<br>
path .<br>
warning: Loadable section “.note.gnu.property” outside of ELF segments<br>
[Detaching after fork from child process 2118905]<br>
[New Thread 0x7fff86eaa700 (LWP 2118911)]<br>
[New Thread 0x7fff866a9700 (LWP 2118912)]<br>
[New Thread 0x7fff81ea8700 (LWP 2118913)]<br>
[New Thread 0x7fff7f6a7700 (LWP 2118914)]<br>
[New Thread 0x7fff7cea6700 (LWP 2118915)]<br>
[New Thread 0x7fff7a6a5700 (LWP 2118916)]<br>
[New Thread 0x7fff77ea4700 (LWP 2118917)]<br>
usage: invoke.py [-h] [–laion400m LAION400M] [–weights WEIGHTS] [–conf CONF] [–model MODEL] [–sampler SAMPLER_NAME] [-F] [–free_gpu_mem]<br>
[–precision PRECISION] [–from_file INFILE] [–outdir OUTDIR] [–prompt_as_dir] [–grid] [–embedding_path EMBEDDING_PATH]<br>
[–no_restore] [–no_upscale] [–esrgan_bg_tile ESRGAN_BG_TILE] [–gfpgan_model_path GFPGAN_MODEL_PATH]<br>
[–gfpgan_dir GFPGAN_DIR] [–web] [–web_develop] [–web_verbose] [–cors [CORS [CORS …]]] [–host HOST] [–port PORT]<br>
[–gui]<br>
invoke.py: error: unrecognized arguments: args<br>
usage: invoke.py [-h] [–laion400m LAION400M] [–weights WEIGHTS] [–conf CONF] [–model MODEL] [–sampler SAMPLER_NAME] [-F] [–free_gpu_mem]<br>
[–precision PRECISION] [–from_file INFILE] [–outdir OUTDIR] [–prompt_as_dir] [–grid] [–embedding_path EMBEDDING_PATH]<br>
[–no_restore] [–no_upscale] [–esrgan_bg_tile ESRGAN_BG_TILE] [–gfpgan_model_path GFPGAN_MODEL_PATH]<br>
[–gfpgan_dir GFPGAN_DIR] [–web] [–web_develop] [–web_verbose] [–cors [CORS [CORS …]]] [–host HOST] [–port PORT]<br>
[–gui]<br>
invoke.py: error: unrecognized arguments: args<br>
[Thread 0x7fff81ea8700 (LWP 2118913) exited]<br>
[Thread 0x7fff77ea4700 (LWP 2118917) exited]<br>
[Thread 0x7fff7a6a5700 (LWP 2118916) exited]<br>
[Thread 0x7fff7cea6700 (LWP 2118915) exited]<br>
[Thread 0x7fff7f6a7700 (LWP 2118914) exited]<br>
[Thread 0x7fff866a9700 (LWP 2118912) exited]<br>
[Thread 0x7fff86eaa700 (LWP 2118911) exited]<br>
[Inferior 1 (process 2118901) exited with code 0377]<br>
(gdb) bt<br>
No stack.<br>
(gdb)</p>

ptrblck: <p>Your execution fails since <code>args</code> is undefined and you should replace it with your optional real arguments to the script. My example is only a template and shouldn’t be used directly.</p>