RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at ..\aten\src\THC\THCGeneral.cpp:50

poojavinod100: <p>I was trying to run the extractive summarizer of the BERTSUM program(<a href="https://github.com/nlpyang/PreSumm/tree/master/src" rel="nofollow noopener">https://github.com/nlpyang/PreSumm/tree/master/src</a>) in  <strong>test</strong>  mode with the following command:</p>
<p><code>python train.py -task ext -mode test -batch_size 3000 -test_batch_size 500 -bert_data_path C:\Users\hp\Downloads\PreSumm-master\PreSumm-master\bert_data -log_file ../logs/val_abs_bert_cnndm -model_path C:\Users\hp\Downloads\bertext_cnndm_transformer -test_from C:\Users\hp\Downloads\bertext_cnndm_transformer\model_1.pt -sep_optim true -use_interval true -visible_gpus 1 -max_pos 512 -max_length 200 -alpha 0.95 -min_length 50 -result_path ../logs/abs_bert_cnndm</code></p>
<p>Here is the error log:</p>
<pre><code class="lang-auto">[2020-01-13 21:03:01,681 INFO] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at ../temp\aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
driver version : 10020
THCudaCheck FAIL file=..\aten\src\THC\THCGeneral.cpp line=50 error=100 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "train.py", line 156, in &lt;module&gt;
    test_ext(args, device_id, cp, step)
  File "C:\Users\hp\Downloads\PreSumm-master\PreSumm-master\src\train_extractive.py", line 190, in test_ext
    model = ExtSummarizer(args, device, checkpoint)
  File "C:\Users\hp\Downloads\PreSumm-master\PreSumm-master\src\models\model_builder.py", line 168, in __init__
    self.to(device)
  File "C:\Users\hp\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 426, in to
    return self._apply(convert)
  File "C:\Users\hp\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 202, in _apply
    module._apply(fn)
  File "C:\Users\hp\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 202, in _apply
    module._apply(fn)
  File "C:\Users\hp\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 202, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "C:\Users\hp\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 224, in _apply
    param_applied = fn(param)
  File "C:\Users\hp\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 424, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "C:\Users\hp\Anaconda3\lib\site-packages\torch\cuda\__init__.py", line 194, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at ..\aten\src\THC\THCGeneral.cpp:50
</code></pre>
<p>I am sure that I have a CUDA-enabled GPU, I made sure by checking the list on NVIDIA. This is an NVIDIA GeForce GTX 950M. I have also used my GPU for deep learning projects with CUDA before. I have installed CUDA and cudNN following these instructions, thinking that could be the problem:<a href="https://www.easy-tensorflow.com/tf-tutorials/install/cuda-cudnn" rel="nofollow noopener">https://www.easy-tensorflow.com/tf-tutorials/install/cuda-cudnn</a>(latest versions, CUDA 10.2). I also tried adding os.environ[‘CUDA_VISIBLE_DEVICES’]=‘0’ in train.py(as this worked for people facing the same kind of error from help posts online). But still the error persists.</p>
<p>I’d really appreciate if someone could help me figure this out.</p>

ptrblck: <p>How did you install PyTorch?<br>
If you’ve installed the binaries, note that they will ship with their own CUDA, cudnn etc., so you don’t need to install these libraries locally unless you want to build from source or CUDA extensions.</p>
<p>Could you post the install log, please?<br>
Were you able to use PyTorch with this GPU before?</p>

poojavinod100: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for the response. I remember I had installed PyTorch with conda. Around that time, I had done a pip install for a different version of torch. But ‘conda list torch’ gives me the current global version as 1.3.0.</p>
<p>Also, ‘conda list cuda’ returns this:</p>
<pre><code class="lang-auto"># packages in environment at C:\Users\hp\Anaconda3:
#
# Name                    Version                   Build  Channel
cuda100                   1.0                           0    pytorch
cudatoolkit               10.1.168                      0
</code></pre>
<p>I also recently installed CUDA 10.2, and cudnn with it globally.</p>
<p>I have used torch on my system for DL before. But I am confused now, as to whether all of the training in my DL projects was actually being done by CPU and not my GPU(I know it sounds really funny, but I seemed sure it was my GPU, because the training was quite quick). Is there any way I can check my GPU history?</p>
<p>When I went through my previous jupyter notebooks(not run from any virtual env), I found this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f99108c5c6f8d69ffdb2b0a2254260c314ea1c36.png" data-download-href="https://discuss.pytorch.org/uploads/default/f99108c5c6f8d69ffdb2b0a2254260c314ea1c36" title="pic"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f99108c5c6f8d69ffdb2b0a2254260c314ea1c36_2_690x156.png" alt="pic" data-base62-sha1="zBLCJ4eSag43O90bFtJg3rCOZ4q" width="690" height="156" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f99108c5c6f8d69ffdb2b0a2254260c314ea1c36_2_690x156.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f99108c5c6f8d69ffdb2b0a2254260c314ea1c36_2_1035x234.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/9/f99108c5c6f8d69ffdb2b0a2254260c314ea1c36.png 2x" data-dominant-color="F6F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pic</span><span class="informations">1359×309 13.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And then I ran this code on my system yesterday:</p>
<pre><code class="lang-auto">import torch 
print(torch.__version__)

print(torch.cuda.current_device())
print(torch.cuda.device(0))
print(torch.cuda.device_count())
print(torch.cuda.get_device_name(0))
print(torch.cuda.is_available())
print(torch.cuda.current_device())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">1.3.0
driver version : 10020
0
&lt;torch.cuda.device object at 0x0000028603786FD0&gt;
1
GeForce GTX 950M
True
0
</code></pre>
<p>This seems contradicting to me(how the jupyter notebook says that CUDA is not available, and how the cmd program says that CUDA is available).</p>

ptrblck: <p>I guess the Python kernel / environment in your Jupyter notebook might not be the same you are using in your terminal by running <code>python script.py</code>.<br>
Could create a (new) conda environment and execute the notebook as well as run your script in this env?</p>

wiTTyMinds_Technolog: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I am trying to run a video feature script without using CUDA.<br>
But,i am also getting the similar error. I don’t have CUDA enabled device .<br>
I tried disabling it using export CUDA_VISIBLE_DEVICES="".<br>
But, still i am getting this error:</p>
<p>File “/home/penguin/conda/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 223, in _apply<br>
param_applied = fn(param)<br>
File “/home/penguin/conda/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 304, in <br>
return self._apply(lambda t: t.cuda(device))<br>
File “/home/penguin/conda/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py”, line 197, in _lazy_init<br>
torch._C._cuda_init()<br>
RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:50</p>
<p>I have installed pytorch using pip .</p>
<p>Output of <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> import torch</p>
<p>print(torch.version.cuda)</p>
<p>print(torch.cuda.device_count())</p>
<p>print(torch.cuda.is_available())</p>
<p>10.1</p>
<p>0</p>
<p>False</p>
<p>Please Help.</p>

ptrblck: <p>It seems that somewhere in your code you are trying to push a module to the GPU.<br>
Could you check for all <code>model.cuda()</code> and <code>model.to('cuda')</code> calls and remove them?<br>
If you are using the <code>to()</code> approach, you could also write device-agnostic code by using:</p>
<pre><code class="lang-python">device = 'cuda' if torch.cuda.is_available() else 'cpu'
model.to(device)
</code></pre>

zahra: <p>Hi Ptrblck<br>
I implement my codes on colab and I received the same error. If I removed model.to(device), how is it possible to run my program on gpu?</p>

ptrblck: <p>It won’t be possible to run the code on the GPU after removing all <code>to('cuda')</code> and <code>cuda()</code> calls.<br>
<a class="mention" href="/u/wittyminds_technolog">@wiTTyMinds_Technolog</a> got the error, since he does not have a GPU and somewhere in the script the tensors or model were still being pushed to the device.</p>
<p>The original error message points to a CUDA call, which wasn’t compiled for the current GPU architecture.<br>
Are you running some custom CUDA extensions, did you build PyTorch from source, and which GPU are you using?</p>

zahra: <p>Thanks.</p>
<p>Are you running some custom CUDA extensions? No, I am using <a href="https://colab.research.google.com/" rel="nofollow noopener">https://colab.research.google.com/</a><br>
did you build PyTorch from source? No, I am using its modules and functions.<br>
which GPU are you using? I am using <a href="https://colab.research.google.com/" rel="nofollow noopener">https://colab.research.google.com/</a> and I do not know what is it.</p>
<p>I found what caused this problem:</p>
<p>In colab, I should set that I want to run my code on gpu in Runtime menu.</p>
<p>Thanks a lot</p>

Imanuel_Roz: <p>Hi I am running a model using the GPU of Google Colab and I am getting this error:</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=38 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "run.py", line 338, in &lt;module&gt;
    main()
  File "run.py", line 303, in main
    model = model.cuda()
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 260, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 193, in _apply
    param.data = fn(param.data)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 260, in &lt;lambda&gt;
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.7/dist-packages/torch/cuda/__init__.py", line 162, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:51```
I checked and my GPU is active and cuda is available, I also tried different solution proposed here but didn't solved my problem.</code></pre>

ptrblck: <p>What does <code>torch.cuda.get_device_name(0)</code> return?</p>

Imanuel_Roz: <aside class="quote no-group" data-username="ptrblck" data-post="11" data-topic="66606">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.cuda.get_device_name(0)</p>
</blockquote>
</aside>
<p>Hi it gives me ‘Tesla K80’</p>

ptrblck: <p>Thanks for the update.<br>
I cannot reproduce the issue in a new notebook on a K80 in Colab:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75fd0097d7144fdeb6b11e194ac3f99ab8d3fae2.png" data-download-href="https://discuss.pytorch.org/uploads/default/75fd0097d7144fdeb6b11e194ac3f99ab8d3fae2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75fd0097d7144fdeb6b11e194ac3f99ab8d3fae2_2_533x500.png" alt="image" data-base62-sha1="gPLUYEZIvBjP4cPSyOzkFsfCTey" width="533" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75fd0097d7144fdeb6b11e194ac3f99ab8d3fae2_2_533x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75fd0097d7144fdeb6b11e194ac3f99ab8d3fae2_2_799x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/5/75fd0097d7144fdeb6b11e194ac3f99ab8d3fae2.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">927×869 67.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
and can properly use the device.</p>

Imanuel_Roz: <p>I fixed it, I changed CUDA_VISIBLE_DEVICES = 5 to CUDA_VISIBLE_DEVICES = 0 in the python file and it works.<br>
Thank you for your rapid response.</p>