CUDA error: an illegal instruction was encountered

amsword: <p>a distributed training crashes with the following errors. Normally it works well, but sometimes it crashes with the following errors.</p>
<p>Any idea to resolve it?</p>
<p>pytorch 1.5, sync-bn is used, each GPU’s input has different dimensions.</p>
<pre><code class="lang-auto">  File "/opt/conda/lib/python3.6/site-packages/apex-0.1-py3.6-linux-x86_64.egg/apex/amp/_initialize.py", line 197, in new_fwd
    **applier(kwargs, input_caster))
  File "/tmp/code/quickdetection/src/FCOS/fcos_core/modeling/detector/generalized_rcnn.py", line 49, in forward
    features = self.backbone(images.tensors)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/tmp/code/quickdetection/src/qd/layers/efficient_det.py", line 1221, in forward
    _, p3, p4, p5 = self.backbone_net(inputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/tmp/code/quickdetection/src/qd/layers/efficient_det.py", line 1067, in forward
    x = self.model._bn0(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 472, in forward
    self.eps, exponential_average_factor, process_group, world_size)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/_functions.py", line 46, in forward
    count_all.view(-1).long().tolist()
RuntimeError: CUDA error: an illegal instruction was encountered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal instruction was encountered (insert_events at /opt/conda/conda-bld/pytorch_1591914742272/work/c10/cuda/CUDACachingAllocator.cpp:771)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x4e (0x7fe430441b5e in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x6d0 (0x7fe430686e30 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fe43042f6ed in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: &lt;unknown function&gt; + 0x51e58a (0x7fe45da9358a in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;
frame #31: __libc_start_main + 0xf0 (0x7fe4783c6830 in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>

ptrblck: <p>Could you update to PyTorch <code>1.5.1</code>, as <code>1.5.0</code> had a bug where internal assert statements were ignored?<br>
This should hopefully yield a better error message than the illegal memory access.</p>

gemfield: <p>With Pytorch 1.6 / CUDA 10.2 /CUDNN7, I got following error occasionally:</p>
<pre><code class="lang-bash">Traceback (most recent call last):
  File "train.py", line 212, in &lt;module&gt;
    train(None)
  File "/gemfield/hostpv/gemfield/deepvac/lib/syszux_deepvac.py", line 335, in __call__
    self.process()
  File "train.py", line 163, in process
    self.processTrain()
  File "/gemfield/hostpv/gemfield/deepvac/lib/syszux_deepvac.py", line 294, in processTrain
    self.doBackward()
  File "train.py", line 139, in doBackward
    self.loss.backward()
  File "/opt/conda/lib/python3.7/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1595629403081/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7fb3e291677d in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xb5d (0x7fb3e2b66d9d in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fb3e2902b1d in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: &lt;unknown function&gt; + 0x53f0ea (0x7fb41c1990ea in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;
frame #17: __libc_start_main + 0xe7 (0x7fb442bdfb97 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)
</code></pre>
<p>Don’t know it is a hardward issue，driver issue or pytorch issue？</p>

ptrblck: <p>Could you post a minimal code snippet to reproduce this issue as well as your currently installed NVIDIA driver and the GPU you are using?</p>

Xfan1025: <p>I got the same error with Pytorch1.6 for CUDA10.2 on Ubuntu 18.04</p>

ptrblck: <p>Could you post a minimal code snippet as given in my previous post, so that we could have a look at this issue?</p>

amsword: <p>I also encountered similar issues with pytorch 1.6 with ubuntu 18 or ubuntu 16; cuda 10.1 or cuda 10.2. it works fine with pytorch 1.5.1, but this issues occurs occasionally with pytorch 1.6</p>

ptrblck: <p>Could you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace here?</p>

amsword: <p>thanks for your reply. however, this is random. roughly, 10% of the times, it will happen. Recently, i find pytorch 1.5.1 also has this issue. Note, in the following trace, CUDA_LAUNCH_BLOCKING is not set as 1. Paste it here and hopefully it can have some information.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/0/708175a3a549d3431bd077365eb99a920bd7036d.png" data-download-href="https://discuss.pytorch.org/uploads/default/708175a3a549d3431bd077365eb99a920bd7036d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/708175a3a549d3431bd077365eb99a920bd7036d_2_690x209.png" alt="image" data-base62-sha1="g3gQenwRkHRxGJUos0WUlrUWG4Z" width="690" height="209" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/708175a3a549d3431bd077365eb99a920bd7036d_2_690x209.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/708175a3a549d3431bd077365eb99a920bd7036d_2_1035x313.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/708175a3a549d3431bd077365eb99a920bd7036d_2_1380x418.png 2x" data-dominant-color="0F3C48"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3084×938 265 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

amsword: <p>another case. It seems like the error message is also random.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dcf5a1d5e294ee564c50c35e80ed0c72c294bf86.png" data-download-href="https://discuss.pytorch.org/uploads/default/dcf5a1d5e294ee564c50c35e80ed0c72c294bf86" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcf5a1d5e294ee564c50c35e80ed0c72c294bf86_2_690x378.png" alt="image" data-base62-sha1="vwHj5VZbn2ycsOtyL8BbIFt7rpk" width="690" height="378" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcf5a1d5e294ee564c50c35e80ed0c72c294bf86_2_690x378.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcf5a1d5e294ee564c50c35e80ed0c72c294bf86_2_1035x567.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dcf5a1d5e294ee564c50c35e80ed0c72c294bf86_2_1380x756.png 2x" data-dominant-color="133F4A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3312×1816 598 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you post the stack traces by wrapping them into three backticks ``` please?<br>
If you don’t set <code>CUDA_LAUNCH_BLOCKING=1</code>, the stack trace might point to random lines of code.</p>

Bill0201: <p>Hi I got same issue. My pytorch version is 2.0.0, cuda version is 12.2, and the issue happens randomly.Could any one help me?any ideas?<br>
The massage is  blow:</p>
<blockquote>
<p>RuntimeError                              Traceback (most recent call last)<br>
Cell In[14], line 1<br>
----&gt; 1 model = AutoModelForCausalLM.from_pretrained(“Qwen/Qwen-7B-Chat”, device_map=“auto”, trust_remote_code=True).eval()</p>
<p>File ~\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py:488, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)<br>
486     else:<br>
487         cls.register(config.<strong>class</strong>, model_class, exist_ok=True)<br>
 → 488     return model_class.from_pretrained(<br>
489         pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs<br>
490     )<br>
491 elif type(config) in cls._model_mapping.keys():<br>
492     model_class = _get_model_class(config, cls._model_mapping)</p>
<p>File ~\AppData\Roaming\Python\Python310\site-packages\transformers\modeling_utils.py:2824, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)<br>
2819     logger.warn(<br>
2820         "This model has some weights that should be kept in higher precision, you need to upgrade "<br>
2821         “<code>accelerate</code> to properly deal with them (<code>pip install --upgrade accelerate</code>).”<br>
2822     )<br>
2823 if device_map != “sequential”:<br>
 → 2824     max_memory = get_balanced_memory(<br>
2825         model,<br>
2826         dtype=target_dtype,<br>
2827         low_zero=(device_map == “balanced_low_0”),<br>
2828         max_memory=max_memory,<br>
2829         **kwargs,<br>
2830     )<br>
2831 kwargs[“max_memory”] = max_memory<br>
2832 # Make sure tied weights are tied before creating the device map.</p>
<p>File ~\AppData\Roaming\Python\Python310\site-packages\accelerate\utils\modeling.py:731, in get_balanced_memory(model, max_memory, no_split_module_classes, dtype, special_dtypes, low_zero)<br>
703 “”"<br>
704 Compute a <code>max_memory</code> dictionary for [<code>infer_auto_device_map</code>] that will balance the use of each available GPU.<br>
705<br>
(…)<br>
728         Transformers generate function).<br>
729 “”"<br>
730 # Get default / clean up max_memory<br>
 → 731 max_memory = get_max_memory(max_memory)<br>
733 if not (torch.cuda.is_available() or is_xpu_available()) or is_mps_available():<br>
734     return max_memory</p>
<p>File ~\AppData\Roaming\Python\Python310\site-packages\accelerate\utils\modeling.py:624, in get_max_memory(max_memory)<br>
622 if not is_xpu_available():<br>
623     for i in range(torch.cuda.device_count()):<br>
 → 624         _ = torch.tensor([0], device=i)<br>
625     max_memory = {i: torch.cuda.mem_get_info(i)[0] for i in range(torch.cuda.device_count())}<br>
626 else:</p>
<p>RuntimeError: CUDA error: an illegal instruction was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
</blockquote>