[resolved] Cuda Runtime Error(30)

ycszen: <p>When I run the code <code>torch.cuda.is_available()</code>, I meet the error as below:</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=109 error=30 : unknown error
Traceback (most recent call last):
  File "trainer.py", line 13, in &lt;module&gt;
    if torch.cuda.is_available():
  File "/usr/local/lib/python2.7/dist-packages/torch/cuda/__init__.py", line 30, in is_available
    return torch._C._cuda_getDeviceCount() &gt; 0
RuntimeError: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:109

</code></pre>

apaszke: <p>There must be something wrong with your driver. Maybe try rebooting?</p>

ycszen: <p>OK. I have found the problem. After I update the linux system, the driver become useless. So I will reinstall the driver. Thank you for your reply.</p>

chrisranderson: <p>I get this when I put my laptop to sleep while in the middle of training. When I put it to sleep, my script stops and I get this error:</p>
<pre><code>THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THC/generated/../THCReduceAll.cuh line=334 error=4 : unspecified launch failure
Traceback (most recent call last):
  File "trytry.py", line 111, in &lt;module&gt;
    loss = network.loss(prediction, label_batch) + 10*torch.mean(cheat_amount)
  File "trytry.py", line 73, in loss
    union = 1e-5 + prediction.sum() + label.sum()
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/variable.py", line 437, in sum
    return Sum(dim)(self)
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/_functions/reduce.py", line 16, in forward
    return input.new((fn(),))
RuntimeError: cuda runtime error (4) : unspecified launch failure at /b/wheel/pytorch-src/torch/lib/THC/generated/../THCReduceAll.cuh:334
</code></pre>
<p>And afterward I get this:</p>
<pre><code>THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THC/THCGeneral.c line=66 error=30 : unknown error
Traceback (most recent call last):
  File "trytry.py", line 77, in &lt;module&gt;
    network = Net()
  File "trytry.py", line 57, in __init__
    self.squeezenet = models.squeezenet1_1(pretrained=True).features.cuda() 
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 147, in cuda
    return self._apply(lambda t: t.cuda(device_id))
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 118, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 124, in _apply
    param.data = fn(param.data)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 147, in &lt;lambda&gt;
    return self._apply(lambda t: t.cuda(device_id))
  File "/usr/local/lib/python3.5/dist-packages/torch/_utils.py", line 65, in _cuda
    return new_type(self.size()).copy_(self, async)
  File "/usr/local/lib/python3.5/dist-packages/torch/cuda/__init__.py", line 272, in __new__
    _lazy_init()
  File "/usr/local/lib/python3.5/dist-packages/torch/cuda/__init__.py", line 85, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /b/wheel/pytorch-src/torch/lib/THC/THCGeneral.c:66
</code></pre>
<p>A reboot has fixed the problem. This is with CUDA 8.0 and an nvidia 1060.</p>

kindlychung: <p>I also have the same issue after the laptop wakes up. I think this is a bug. Tensorflow seems to work fine in such situations.</p>

arogozhnikov: <p>Just had the same failure after wake/sleep for desktop. Pytorch 0.2, ubuntu 16.04</p>

hvasbath: <p>Same here! GTX1050ti Ubuntu16.04, reboot fixes it, but one short sleep then wake breaks it!</p>

psavine42: <p>Same here. ubuntu 16.04 cuda9, pytorch 0.2</p>

Ke_Bai: <p>Have anyone solved this problem? Thanks.</p>

danakianfar: <p>This also happens on my system</p>
<ul>
<li>Ubuntu 16.04</li>
<li>Nvidia GeForce 940MX</li>
<li>PyTorch 0.3.1 running on Python 3.6</li>
<li>Cuda 8.0</li>
<li>CUDNN 7</li>
</ul>
<p>Any clues? I don’t see why this thread is marked as resolved, if the solution is to restart the laptop every time.</p>

Ste_Millington: <p>Same problem for me too</p>
<p>Ubuntu 16.04 running on Dell desktop<br>
Nvidia GeForce 1050ti<br>
PyTorch 0.3.1.post2 running on Python 3.6<br>
Cuda 9.1<br>
CUDNN 7.1</p>

Adam_Harrison: <p>Run into the same problem</p>
<p>Ubuntu 16.04<br>
TitanXp<br>
Cuda 9.1<br>
pytorch 0.3.1 running python 2.7</p>

Jimmy_Xiaoke_Shen: <p>reboot fixes the problem.</p>

Marat: <p>Same problem some strange stuff after wake up (desktop ubuntu 16.04 cuda 8 1080 gtx)</p>

happytaoxiaoli: <p>maybe use sudo can solve this problem.<br>
i reinstall my driver and cuda after my linux system updated, and  same problem happened</p>

waleeka: <p>Same problem here. After laptop goes to sleep and wake up, I get this error after calling <code>torch.cuda.current_device()</code>:</p>
<pre><code class="lang-auto">RuntimeError: cuda runtime error (30) : unknown error at 
/opt/conda/conda-bld/pytorch_1524590031827/work/aten/src/THC/THCGeneral.cpp:70
</code></pre>
<p>Ubuntu 18.04, Pytorch 0.4.1, cuda 9.2</p>

farhat_Ullah: <p>I have hp-1000 laptop without GPU. Now how it is possible to handle this error.<br>
"THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=32 error=30 : unknown error<br>
Traceback (most recent call last): File “train_nli.py”, line 62, in <br>
torch.cuda.set_device(params.gpu_id)<br>
File “/home/farhatullah/anaconda3/lib/python3.6/site-packages/torch/cuda/<strong>init</strong>.py”, line 262, in set_device    torch._C._cuda_setDevice(device)<br>
RuntimeError: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:32<br>
"<br>
actually I am training InferSent sentence embedding model. There is any availability of <a href="https://github.com/facebookresearch/InferSent/blob/master/train_nli.py" rel="nofollow noopener">https://github.com/facebookresearch/InferSent/blob/master/train_nli.py</a> this code for CPU??</p>

miladiouss: <p>The solution can be found <a href="https://github.com/tensorflow/tensorflow/issues/5777#issuecomment-301058363" rel="nofollow noopener">here</a>. Basically, run the following commands in the terminal:</p>
<pre><code class="lang-auto">sudo rmmod nvidia_uvm
sudo rmmod nvidia
sudo modprobe nvidia
sudo modprobe nvidia_uvm
</code></pre>

Mohamed_Ghadban: <p>Go to NVIDIA Nsight Options and set ‘Enable Crash Detection And Handling = True’.</p>
<p>Did the trick for me.</p>

AndreiCostinescu: <p>This always works for me (Win10, Cuda 10.1, Python 3.7.2, PyTorch 1.0.1, NVIDIA GTX 1050 Ti):</p>
<pre><code class="lang-auto">import torch
torch.cuda.current_device()
</code></pre>
<p>but this always fails for me:</p>
<pre><code class="lang-auto">import torch
torch.cuda.is_available()
torch.cuda.current_device()  # fails here
</code></pre>
<p><a class="mention" href="/u/mohamed_ghadban">@Mohamed_Ghadban</a>, how can I access the NVIDIA Nsight Options? Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>