Some observations on "cuda runtime error (30)"

KFrank: <p>Hello Forum!</p>
<p>I have some information about the behavior of “cuda runtime<br>
error (30)” (probably somewhat specific to my particular<br>
configuration).</p>
<p>This is a follow-on to a number of threads about “error 30,”<br>
and, in particular, to this post:</p>
<aside class="quote" data-post="20" data-topic="1116">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/848f3c/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/resolved-cuda-runtime-error-30/1116/20">[resolved] Cuda Runtime Error(30)</a> 
  </div>
  <blockquote>
    This always works for me (Win10, Cuda 10.1, Python 3.7.2, PyTorch 1.0.1, NVIDIA GTX 1050 Ti): 
import torch
torch.cuda.current_device()

but this always fails for me: 
import torch
torch.cuda.is_available()
torch.cuda.current_device()  # fails here

<a class="mention" href="/u/mohamed_ghadban">@Mohamed_Ghadban</a>, how can I access the NVIDIA Nsight Options? Thanks in advance <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title="slight_smile" alt="slight_smile" class="emoji">
  </blockquote>
</aside>

<p>Clued in by Andrei’s observation that <code>torch.cuda.is_available()</code><br>
“breaks” cuda, I find (for me) that if <code>torch.cuda.is_available()</code><br>
is the first cuda call, subsequent cuda calls will throw “error 30”<br>
unless the first subsequent call is called promptly (&lt; ~1 sec.).</p>
<p>Specifically, start a fresh python session and <code>import torch</code>, then:</p>
<p>don’t call <code>torch.cuda.is_available()</code><br>
subsequent cuda calls work</p>
<p>call <code>torch.cuda.is_available()</code> first<br>
make another cuda in less than (about) a second<br>
it and subsequent cuda calls work</p>
<p>call <code>torch.cuda.is_available()</code> first<br>
wait about two seconds<br>
subsequent cuda calls throw “error 30” with stack trace A<br>
(except <code>torch.cuda.device_count()</code> works in this case)</p>
<p>call <code>torch.cuda.is_available()</code> first<br>
wait more than (about) ten seconds<br>
subsequent cuda calls throw “error 30” with stack trace B</p>
<p>(See example scripts, below.)</p>
<p>This seems to be repeatable for me, although, because it involves<br>
timing and the order in which calls are made, I don’t know whether<br>
it is always consistently repeatable.</p>
<p>I am using pytorch 0.3.0 for cuda 8, specifically<br>
pytorch-0.3.0-py36_0.3.0cu80.tar.bz2 from the legacy-builds link<br>
in this post of Peter’s:</p>
<aside class="quote quote-modified" data-post="2" data-topic="51263">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pre-built-pytorch-for-cuda-3-0-on-windows/51263/2">Pre-built pytorch for CUDA [compute capability] 3.0 on windows?</a> 
  </div>
  <blockquote>
    Hi Frank, 
I assume you are referring to the compute capability 3.0, which should work with CUDA6.0 - CUDA10.1. 
If I’m not mistaken, the minimal compute capability for the current binaries is &gt;=3.5, so you could build from source to support this older GPU. 
However, if you would like to play around with some legacy PyTorch version, you might get lucky finding some supported binaries <a href="https://drive.google.com/drive/folders/0B-X0-FlSGfCYdTNldW02UGl4MXM" rel="noopener nofollow ugc">here</a> (built by <a class="mention" href="/u/peterjc123">@peterjc123</a>). 
Since these binaries are quite old by now, I would recommend building from source. …
  </blockquote>
</aside>

<p>I am using a mobile (laptop) Quadro K1100M gpu with the nvidia driver<br>
version 426.00 running on windows 10.  (As a side note, I initially<br>
saw this “error 30” issue using the nvidia driver version 425.45.<br>
I would guess these observations also apply to that driver, but I don’t<br>
know for sure because I upgraded the driver before I came across this<br>
business with <code>torch.cuda.is_available()</code> “breaking” cuda and<br>
performing these tests.)</p>
<p>Anyway, I have no idea what is going on here.  I’m posting these<br>
results in the hope they might shed some light on the “error 30”<br>
issue, and be helpful to others.  I suspect that these results<br>
are specific to configurations similar to mine, and are not likely<br>
to be relevant to all instances of “error 30”.</p>
<p>Thanks.</p>
<p>K. Frank</p>
<p>Here are some illustrative scripts and their outputs.  Each was<br>
run in a fresh python session started from the command line.  (The<br>
<code>sleep (10.0)</code>in the first script is just to show that it’s waiting<br>
<em>after</em> <code>torch.cuda.is_available()</code> that is relevant, rather than,<br>
for example, waiting after <code>import torch</code>.)</p>
<p>No wait – cuda works:</p>
<pre data-code-wrap="python"><code class="lang-python">from time import sleep
import torch
print (torch.__version__)
sleep (10.0)
torch.cuda.is_available()
torch.cuda.current_device()
torch.cuda.device_count()
torch.cuda.get_device_name (0)
torch.cuda.get_device_capability (0)
quit()
</code></pre>
<p>No-wait result:</p>
<pre><code class="lang-plaintext">Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; from time import sleep
&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
0.3.0b0+591e73e
&gt;&gt;&gt; sleep (10.0)
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.cuda.current_device()
0
&gt;&gt;&gt; torch.cuda.device_count()
1
&gt;&gt;&gt; torch.cuda.get_device_name (0)
'Quadro K1100M'
&gt;&gt;&gt; torch.cuda.get_device_capability (0)
(3, 0)
&gt;&gt;&gt; quit()
</code></pre>
<p>Short wait – cuda works:</p>
<pre data-code-wrap="python"><code class="lang-python">from time import sleep
import torch
print (torch.__version__)
torch.cuda.is_available()
sleep (1.0)
torch.cuda.current_device()
torch.cuda.device_count()
torch.cuda.get_device_name (0)
torch.cuda.get_device_capability (0)
quit()
</code></pre>
<p>Short-wait result:</p>
<pre><code class="lang-plaintext">Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; from time import sleep
&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
0.3.0b0+591e73e
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; sleep (1.0)
&gt;&gt;&gt; torch.cuda.current_device()
0
&gt;&gt;&gt; torch.cuda.device_count()
1
&gt;&gt;&gt; torch.cuda.get_device_name (0)
'Quadro K1100M'
&gt;&gt;&gt; torch.cuda.get_device_capability (0)
(3, 0)
&gt;&gt;&gt; quit()
</code></pre>
<p>Medium wait – cuda fails (A):</p>
<pre data-code-wrap="python"><code class="lang-python">from time import sleep
import torch
print (torch.__version__)
torch.cuda.is_available()
sleep (2.0)
torch.cuda.current_device()
torch.cuda.device_count()
torch.cuda.get_device_name (0)
torch.cuda.get_device_capability (0)
quit()
</code></pre>
<p>Medium-wait result:</p>
<pre><code class="lang-plaintext">Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; from time import sleep
&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
0.3.0b0+591e73e
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; sleep (2.0)
&gt;&gt;&gt; torch.cuda.current_device()
THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=143 error=30 : unknown error
Traceback (most recent call last):
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 151, in _lazy_init
    queued_call()
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 103, in _check_capability
    major = get_device_capability(d)[0]
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 266, in get_device_capability
    return torch._C._cuda_getDeviceCapability(device)
RuntimeError: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:143

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 302, in current_device
    _lazy_init()
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 155, in _lazy_init
    raise_from(DeferredCudaCallError(msg), e)
  File "&lt;string&gt;", line 3, in raise_from
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:143

CUDA call was originally invoked at:

['  File "&lt;stdin&gt;", line 1, in &lt;module&gt;\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 971, in _find_and_load\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 955, in _find_and_load_unlocked\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 665, in _load_unlocked\n', '  File "&lt;frozen importlib._bootstrap_external&gt;", line 678, in exec_module\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 219, in _call_with_frames_removed\n', '  File "C:\\&lt;path_to_miniconda&gt;\\Miniconda3\\lib\\site-packages\\torch\\__init__.py", line 328, in &lt;module&gt;\n    import torch.cuda\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 971, in _find_and_load\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 955, in _find_and_load_unlocked\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 665, in _load_unlocked\n', '  File "&lt;frozen importlib._bootstrap_external&gt;", line 678, in exec_module\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 219, in _call_with_frames_removed\n', '  File "C:\\&lt;path_to_miniconda&gt;\\Miniconda3\\lib\\site-packages\\torch\\cuda\\__init__.py", line 118, in &lt;module&gt;\n    _lazy_call(_check_capability)\n', '  File "C:\\&lt;path_to_miniconda&gt;\\Miniconda3\\lib\\site-packages\\torch\\cuda\\__init__.py", line 116, in _lazy_call\n    _queued_calls.append((callable, traceback.format_stack()))\n']
&gt;&gt;&gt; torch.cuda.device_count()
1
&gt;&gt;&gt; torch.cuda.get_device_name (0)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 253, in get_device_name
    return torch._C._cuda_getDeviceName(device)
RuntimeError: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:131
&gt;&gt;&gt; torch.cuda.get_device_capability (0)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 266, in get_device_capability
    return torch._C._cuda_getDeviceCapability(device)
RuntimeError: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:143
&gt;&gt;&gt; quit()
</code></pre>
<p>Long wait – cuda fails (B):</p>
<pre data-code-wrap="python"><code class="lang-python">from time import sleep
import torch
print (torch.__version__)
torch.cuda.is_available()
sleep (10.0)
torch.cuda.current_device()
torch.cuda.device_count()
torch.cuda.get_device_name (0)
torch.cuda.get_device_capability (0)
quit()
</code></pre>
<p>Long-wait result:</p>
<pre><code class="lang-plaintext">Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; from time import sleep
&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
0.3.0b0+591e73e
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; sleep (10.0)
&gt;&gt;&gt; torch.cuda.current_device()
THCudaCheck FAIL file=D:\pytorch\pytorch\torch\lib\THC\THCGeneral.c line=120 error=30 : unknown error
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 302, in current_device
    _lazy_init()
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 140, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at D:\pytorch\pytorch\torch\lib\THC\THCGeneral.c:120
&gt;&gt;&gt; torch.cuda.device_count()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 294, in device_count
    _lazy_init()
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 140, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at D:\pytorch\pytorch\torch\lib\THC\THCGeneral.c:120
&gt;&gt;&gt; torch.cuda.get_device_name (0)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 253, in get_device_name
    return torch._C._cuda_getDeviceName(device)
RuntimeError: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:131
&gt;&gt;&gt; torch.cuda.get_device_capability (0)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\&lt;path_to_miniconda&gt;\Miniconda3\lib\site-packages\torch\cuda\__init__.py", line 266, in get_device_capability
    return torch._C._cuda_getDeviceCapability(device)
RuntimeError: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:143
&gt;&gt;&gt; quit()
</code></pre>