GPU: no CUDA-capable device is detected

izoiluzja: <p>Greetings pytorchers.</p>
<p>I am trying to run pytorch on a machine with eight Testla K10G1.8GB GPUs (confirmed by nvidia-smi).<br>
CUDA is installed correctly and the samples from nvidia run (as do my own .cu tests).<br>
A simple example gives me a runtime error, that I am finding difficult to comprehend.</p>
<p>Pointers on how to debug this problem (details are below) would be awesome.</p>
<p>Thank you. Here goes…</p>
<p>uname-ar:<br>
Linux 4398392dfc97 4.4.0-83-generic <span class="hashtag">#106-Ubuntu</span> SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux</p>
<p>torch.<strong>version</strong>:<br>
0.5.0a0+ef477b2 (compiled from source following instructions on github)</p>
<p>Command (I see the -DWITH_CUDA compiler flag flash by in gcc):<br>
CUDA_HOME=/usr/local/cuda python setup.py install</p>
<p>Additional (in the pytorch directory):<br>
grep -R “-DWITH_CUDA” ./*:<br>
./setup.py:    extra_compile_args += [’-DWITH_CUDA’]<br>
./tools/cpp_build/libtorch/CMakeLists.txt:  add_definitions(-DWITH_CUDA)<br>
./torch/lib/THD/CMakeLists.txt:    ADD_DEFINITIONS(-DWITH_CUDA=1)<br>
./torch/lib/build/THD/CMakeFiles/THD.dir/flags.make:CXX_DEFINES = -DWITH_CUDA=1 -DWITH_GLOO=1 -D_THD_CORE=1</p>
<p>nvcc --version:<br>
NVIDIA ® Cuda compiler driver<br>
Copyright © 2005-2016 NVIDIA Corporation<br>
Built on Tue_Jan_10_13:22:03_CST_2017<br>
Cuda compilation tools, release 8.0, V8.0.61</p>
<p>gcc -v:<br>
Using built-in specs.<br>
COLLECT_GCC=gcc<br>
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper<br>
Target: x86_64-linux-gnu</p>
<p>Code (test.py):<br>
import torch<br>
n_devices = torch.cuda.device_count ()<br>
a = torch.cuda.FloatTensor([1.])</p>
<p>Output:<br>
THCudaCheck FAIL file=/home/tyoung/usr/src/pytorch/aten/src/THC/THCGeneral.cpp line=71 error=38 : no CUDA-capable device is detected<br>
Traceback (most recent call last):<br>
File “cuda.py”, line 4, in <br>
a         = torch.cuda.FloatTensor([1.])<br>
File “/home/tyoung/usr/bin/anaconda3/lib/python3.6/site-packages/torch/cuda/<strong>init</strong>.py”, line 161, in _lazy_init<br>
torch._C._cuda_init()<br>
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /home/tyoung/usr/src/pytorch/aten/src/THC/THCGeneral.cpp:71</p>
<p>Note, if I do the same thing on a different machine with one GeForce GTX 680 and run<br>
CUDA_VISIBLE_DEVICES=1 python test.py<br>
ie., point it to a nonexistant device, I get the same output as above. Naturally, there,<br>
CUDA_VISIBLE_DEVICES=0 python test.py<br>
runs code correctly.</p>
<p>How (or why?) are my Testla K10s hiding from (py)torch?   <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Best,<br>
Toby</p>

ptrblck: <p>While building from source, did you get any information regarding the found CUDA version and its location etc.?</p>

izoiluzja: <p>Thank you for the response. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I will have another try tomorrow and post any configure/compile indicators I find.<br>
Best, T</p>

izoiluzja: <p>Good pointer <a class="mention" href="/u/ptrblck">@ptrblck</a> ! Thanks, that helped me to solve the issue.</p>
<p>After searching through the configuration scripts this morning, I found some dodgy looking links to cuda library. A careful, simple, fresh install from scratch and pytorch works just great and as expected.</p>
<p>Best,<br>
T</p>