Weird CUDA illegal memory access error

blackyang: <p>Hi all, I encountered a weird CUDA illegal memory access error. Will try to have a minimal example in a while.</p>
<p>During training, my code will run for several batches without any errors, then after a random amount of time there will be an illegal memory access error. Then error happened in this line:</p>
<pre><code class="lang-auto">conf_p = conf[pos]
</code></pre>
<p>and error messages are:</p>
<pre><code class="lang-auto">  File "/export/home/x/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py", line 74, in __getitem__
    return MaskedSelect.apply(self, key)
  File "/export/home/x/anaconda3/lib/python3.6/site-packages/torch/autograd/_functions/tensor.py", line 534, in forward
    return tensor.masked_select(mask)
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /export/home/x/code/pytorch/torch/lib/THC
/generated/../THCReduceAll.cuh:339
</code></pre>
<p>interestingly, even I replace this line of code to:</p>
<pre><code class="lang-auto">print(pos)
</code></pre>
<p>there will still be an error</p>
<pre><code class="lang-auto">  File "/export/home/x/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py", line 119, in __repr__
    return 'Variable containing:' + self.data.__repr__()
  File "/export/home/x/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 133, in __repr__
    return str(self)
  File "/export/home/x/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 140, in __str__
    return _tensor_str._str(self)
  File "/export/home/x/anaconda3/lib/python3.6/site-packages/torch/_tensor_str.py", line 294, in _str
    strt = _tensor_str(self)
  File "/export/home/x/anaconda3/lib/python3.6/site-packages/torch/_tensor_str.py", line 142, in _tensor_str
    formatter = _number_format(self, min_sz=3 if not print_full_mat else 0)
  File "/export/home/x/anaconda3/lib/python3.6/site-packages/torch/_tensor_str.py", line 74, in _number_format
    tensor = torch.DoubleTensor(tensor.size()).copy_(tensor).abs_().view(tensor.nelement())
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /export/home/x/code/pytorch/torch/lib/THC
/generic/THCTensorCopy.c:70
</code></pre>
<p>however I can run:</p>
<pre><code class="lang-auto">print(pos.size(), pos.is_contiguous())
</code></pre>
<p>I run it in cpu mode, below is the gdb back trace:</p>
<pre><code class="lang-auto">Thread 21 "python" received signal SIGBUS, Bus error.                                                                    [54/1858]
[Switching to Thread 0x7fff4ce01700 (LWP 32576)]
malloc_consolidate (av=av@entry=0x7ffe58000020) at malloc.c:4181
4181    malloc.c: No such file or directory.
(gdb) bt
#0  malloc_consolidate (av=av@entry=0x7ffe58000020) at malloc.c:4181
#1  0x00007ffff6a51678 in _int_free (av=0x7ffe58000020, p=&lt;optimized out&gt;, have_lock=0) at malloc.c:4075
#2  0x00007ffff6a5553c in __GI___libc_free (mem=&lt;optimized out&gt;) at malloc.c:2968
#3  0x00007fffe0b853fe in THLongStorage_free ()
   from /export/home/x/anaconda3/lib/python3.6/site-packages/torch/lib/libTH.so.1
#4  0x00007fffe0baf4e7 in THLongTensor_free ()
   from /export/home/x/anaconda3/lib/python3.6/site-packages/torch/lib/libTH.so.1
#5  0x00007fffe01b8839 in at::CPULongTensor::~CPULongTensor() [clone .localalias.31] ()
   from /export/home/x/anaconda3/lib/python3.6/site-packages/torch/lib/libATen.so.1
#6  0x00007fffecf88269 in torch::autograd::VariableImpl::~VariableImpl (this=0x7ffdd793fc40, __in_chrg=&lt;optimized out&gt;)
    at torch/csrc/autograd/variable.cpp:38
#7  0x00007fffecf9ad21 in at::TensorImpl::release (this=&lt;optimized out&gt;)
    at /export/home/x/code/pytorch/torch/lib/tmp_install/include/ATen/TensorImpl.h:31
---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit---
#8  at::detail::TensorBase::~TensorBase (this=&lt;optimized out&gt;, __in_chrg=&lt;optimized out&gt;)
    at /export/home/x/code/pytorch/torch/lib/tmp_install/include/ATen/TensorBase.h:27
#9  at::Tensor::~Tensor (this=&lt;optimized out&gt;, __in_chrg=&lt;optimized out&gt;)
    at /export/home/x/code/pytorch/torch/lib/tmp_install/include/ATen/Tensor.h:33
#10 at::Tensor::reset (this=0x7fffa888c288) at /export/home/x/code/pytorch/torch/lib/tmp_install/include/ATen/Tensor.h:57
#11 THPVariable_clear (self=self@entry=0x7fffa888c278) at torch/csrc/autograd/python_variable.cpp:131
#12 0x00007fffecf9ae31 in THPVariable_dealloc (self=0x7fffa888c278) at torch/csrc/autograd/python_variable.cpp:138
#13 0x00007ffff79a75f9 in subtype_dealloc (self=0x7fffa888c278) at Objects/typeobject.c:1222
#14 0x00007ffff79a5f3e in tupledealloc (op=0x7fffa8851a08) at Objects/tupleobject.c:243
#15 0x00007fffecf9072f in THPPointer&lt;_object&gt;::~THPPointer (this=0x7fff4ce00850, __in_chrg=&lt;optimized out&gt;)
    at /export/home/x/code/pytorch/torch/csrc/utils/object_ptr.h:12
#16 torch::autograd::PyFunction::apply (this=0x7fffa877e6d0, inputs=...) at torch/csrc/autograd/python_function.cpp:123
#17 0x00007fffecf7bbf4 in torch::autograd::Function::operator() (inputs=..., this=&lt;optimized out&gt;)
---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit---
    at /export/home/x/code/pytorch/torch/csrc/autograd/function.h:89
#18 torch::autograd::call_function (task=...) at torch/csrc/autograd/engine.cpp:208
#19 torch::autograd::Engine::evaluate_function (this=this@entry=0x7fffee198ca0 &lt;engine&gt;, task=...)
    at torch/csrc/autograd/engine.cpp:220
#20 0x00007fffecf7ddae in torch::autograd::Engine::thread_main (this=0x7fffee198ca0 &lt;engine&gt;, graph_task=0x0)
    at torch/csrc/autograd/engine.cpp:144
#21 0x00007fffecf7ab42 in torch::autograd::Engine::thread_init (this=this@entry=0x7fffee198ca0 &lt;engine&gt;, device=device@entry=-1)
    at torch/csrc/autograd/engine.cpp:121
#22 0x00007fffecf9da9a in torch::autograd::python::PythonEngine::thread_init (this=0x7fffee198ca0 &lt;engine&gt;, device=-1)
    at torch/csrc/autograd/python_engine.cpp:28
#23 0x00007fffcd559c5c in std::execute_native_thread_routine_compat (__p=&lt;optimized out&gt;)
    at /home/msarahan/miniconda2/conda-bld/compilers_linux-64_1507259624353/work/.build/src/gcc-7.2.0/libstdc++-v3/src/c++11/threa
d.cc:110
---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit---
#24 0x00007ffff76ba6ba in start_thread (arg=0x7fff4ce01700) at pthread_create.c:333
#25 0x00007ffff6ad83dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)     return str(self)
</code></pre>
<p>Seems that a piece of memory is wrongly freed</p>

albanD: <p>Hi,</p>
<p>You should run your code with <code>CUDA_LAUNCH_BLOCKING=1</code> to see where the error comes from.<br>
Because all cuda calls are asynchronous when you don’t specify this option, the python code will report the error on the next cuda call after the error. This is why trying to use the tensor or printing its content raise an error (you use the gpu for that) while printing the size or checking if it is contiguous does not (because these are cpu only operations).</p>

blackyang: <p>Thanks for your reply, actually I tried that method, the error message is the same as the first one</p>

albanD: <p>Weird,<br>
Could you provide us with a minimal example that reproduce the problem please?</p>

blackyang: <p>Yeah I will try that</p>
<p>I just found that even I set CUDA_LAUNCH_BLOCKING=1, there is still an error when I try to print the tensor. I was running</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1
python train.py
</code></pre>
<p>is this the right way to set this environment variable?</p>

albanD: <p>No,<br>
if you run in 2 commands, your should use <code>export CUDA_LAUNCH_BLOCKING=1</code> but that will set it for the whole terminal session.<br>
If you use <code>CUDA_LAUNCH_BLOCKING=1 python train.py</code> (in one command), that will set this env variable just for this command.</p>

blackyang: <p>Yeah I was wondering if I need to put them in one line, thanks for your reply!</p>

blackyang: <p>I put them in the same line now, here is the error message:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'thrust::system::system_error'
  what():  function_attributes(): after cudaFuncGetAttributes: an illegal memory access was encountered
./train.sh: line 14:  4111 Aborted                 (core dumped) CUDA_LAUNCH_BLOCKING=1 python train.py
</code></pre>

blackyang: <p>I finally solved this problem.</p>
<p>Although the error message is not very helpful, I guess the illegal memory access should come from an index out of range access. So I double check all my code, and finally found that in certain batch, the groundtruth target could be larger than the number of classes in softmax. I fixed it and no more errors <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p><a class="mention" href="/u/alband">@albanD</a> thanks for your time anyway</p>

albanD: <p>Good that you found the problem !</p>

ruihangdu: <p>I have a similar problem.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/d/da7709ecff97acf0651e10a46fa6d7dc26ca60cb.png" alt="image" data-base62-sha1="vaD7VQvQprjmtrV0EF3wMsgS5J1" width="527" height="47"><br>
Here weight_mask is a tensor.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0a45a9cdedf4ed92205ec2193d219f74b26d34a6.png" alt="image" data-base62-sha1="1sS1IdKFRwHUCh33bnjk6nmChDw" width="491" height="34"><br>
After the script ran for days and this line was called couple hundred of times, this error occurred. I am not sure if I am able to reproduce this error again or how long it is going to take for it to appear again. Any thoughts on this or possible explanations for this? Thank you.</p>

JCPARK: <p>Just to share my case, I had a similar error code.<br>
I commented the line <code>cudnn.benchnark=True</code> and everything works fine now.</p>
<p>Training code works fine with the commented line, but when I run my validation code, it crashes with the same 77 illegal access error.<br>
Anyways, I will share more if I find something else.</p>

zpxiang: <p>Thanks for your solution.</p>

ClaraBing: <p>I’m getting the same illegal memory access error which was caused by moving the tensors to GPU: <code>input[key] = input=[key].cuda()</code>.</p>
<p>I tried setting <code>cudnn.benmark = False</code> and <code>rm -rf ~/.nv</code> following some web search but without success. Any suggestions? Thanks a lot!</p>
<p>EDIT: I realized that <code>cudnn.benmark</code> was set to <code>True</code> on a later line ^ ^b (I was running someone else’s git repo) and after resetting it to <code>False</code> the error went away!</p>

NightRainXiaoxiang: <p>Have you solved this problem and found the reason of this?</p>

skyuuka: <p>I also met the same error when evaluating the model</p>
<p><code>RuntimeError: CUDA error: an illegal memory access was encountered</code></p>
<p>My code looks like</p>
<pre><code>correct = 0
total = 0
for i, (input, target) in tqdm.tqdm(enumerate(data_loader), total=len(dataset)//batch_size):
    target = target.to(device)
    input = input.to(device)
    output = self.model.forward_t(input)
    c = output.argmax(dim=1)
    total += len(target)
    correct += sum(target.cpu().numpy() == c.cpu().numpy())
    acc = float(correct) / total
</code></pre>
<p>It is also strange that, if I do not use <code>.cpu.numpy()</code> to convert the data first, the result will be incorrect.</p>

sbelharbi: <p>Hi,<br>
I am facing the same issue. Setting <code>cudnn.benchmark=False</code> did not help (it was set to <code>False</code> from the beginning). My code crashes after a second call to some function. (I use <code>CUDA_LAUNCH_BLOCKING=1</code> to find out where the error occured). Any pointers to the cause and how to fix it? thanks</p>
<pre><code class="lang-auto">File "../libs/bn.py", line 109, in forward
    self.training, self.momentum, self.eps, self.activation, self.slope)
  File "../libs/functions.py", line 99, in forward
    running_mean.mul_((1 - ctx.momentum)).add_(ctx.momentum * mean)
RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>
<p>When trying to print the value of the tensor <code>running_mean</code> (during the second call), it raises the following error:</p>
<pre><code class="lang-auto">
print(running_mean)
  File "..../Venvs/pytorch.1.0.1/lib/python3.7/site-packages/torch/tensor.py", line 66, in __repr__
    return torch._tensor_str._str(self)
  File "..../Venvs/pytorch.1.0.1/lib/python3.7/site-packages/torch/_tensor_str.py", line 277, in _str
    tensor_str = _tensor_str(self, indent)
  File "..../Venvs/pytorch.1.0.1/lib/python3.7/site-packages/torch/_tensor_str.py", line 195, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "..../Venvs/pytorch.1.0.1/lib/python3.7/site-packages/torch/_tensor_str.py", line 84, in __init__
    nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))
  File "..../Venvs/pytorch.1.0.1/lib/python3.7/site-packages/torch/functional.py", line 271, in isfinite
    return (tensor == tensor) &amp; (tensor.abs() != inf)
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /pytorch/aten/src/THC/generated/../THCTensorMathCompareT.cuh:69
</code></pre>
<p>—&gt; <code>running_mean</code> seems to have <code>inf</code> values!!!<br>
It seems an issue related to the machine <a href="https://discuss.pytorch.org/t/runtime-error-77/9392/9?u=sbelharbi">where</a> the code is running.  (more specifically, cuda-related. Things run fine on cpu).</p>
<p><a href="https://discuss.pytorch.org/t/runtime-error-77/9392/9?u=sbelharbi"><strong>Fix and possible explanation</strong></a>.</p>

Hady_Pranoto: <p>How i can do that? I have same problem</p>

albanD: <p>You just need to set the environment variable before launching your script. The simplest is <code>CUDA_LAUNCH_BLOCKING=1 python your_script.py</code>.</p>

NotAnyMike2: <p>Another solution is using<code> torch.cuda.set_device(1)</code>. That should also work.</p>