CUDA error: an illegal memory access was encountered with reproduction

Oliver: <p>I have been getting this error.<br>
It suggested I could report an issue if the following code gets the same error, which it does:</p>
<pre><code class="lang-auto">import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([25, 128, 63, 63], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(128, 128, kernel_size=[2, 2], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()
</code></pre>
<p>What does this mean? How can I fix this?</p>
<p>Thanks!!</p>

tom: <p>What PyTorch and CUDA version and what hardware are you using?<br>
On my self-compiled recent git checkout (1.11.0a0+git0a07488) and cuda 11.3 I don’t get any error on a RTX3090.</p>
<p>Best regards</p>
<p>Thomas</p>

Oliver: <p>Hi Thomas,</p>
<p>I’m actually glad to hear, probably issue with my code then</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
</code></pre>
<p>pytorch version: 1.9.1</p>
<p>would you know what this error actually means?</p>
<p>thanks!</p>

Oliver: <p>Hmm, I just tried and I have the same error on colab (previously was on kaggle) funky…</p>

tom: <p>Try to use the latest PyTorch (1.10).<br>
The error indicates an out of bound memory access similar to a segfault on the CPU, something like an indexing error in the low level code.<br>
You do reproduce this with the exact code you posted?</p>
<p>Best regards</p>
<p>Thomas</p>

Oliver: <p>Yes, I think I at least now know what is causing this, though the error seems misleading.</p>
<p>I train a model (GAN) with a batch size larger than 4, I get this error. Unless I hard reset (factory reset the runtime), even after restarting the kernel and clearing all caches will cause anything related with the GPU to output this error.<br>
It’s not just the code above, just using .to(‘cuda’) will suffice.<br>
It’s funky, but I’m also just really learning pytorch now.<br>
Maybe it’s because I keep the image arrays in memory (about 700 MB total) instead of loading each image in in batches?</p>

tom: <p>So these CUDA errors should be fatal for the Python process they happen in but restarting the kernel should then make things work again.<br>
The perhaps most typical way to trigger it in a working PyTorch system is to have classification targets that exceed that possible through the logit tensor size passed to CrossEntropyLoss. But something seems funny with your system, then.<br>
At this point, maybe <a class="mention" href="/u/ptrblck">@ptrblck</a> knows more…</p>
<p>Best regards</p>
<p>Thomas</p>

Oliver: <p>It’s weird, though at this point it’s two seperate systems kaggle and colab.<br>
At least it works now with running on smaller batch sizes, in the next few days I may try to just load from disk.</p>

Oliver: <p>Either way, thanks for looking into it with me, it’s very much appreciated!</p>

ptrblck: <aside class="quote no-group" data-username="Oliver" data-post="6" data-topic="135532">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>I train a model (GAN) with a batch size larger than 4, I get this error.</p>
</blockquote>
</aside>
<p>Do you see the illegal memory access when running the original GAN training from a clean and working environment?<br>
If so, could you post an executable code snippet, which would reproduce the issue and post the output of <code>python -m torch.utils.collect_env</code>, please?</p>
<aside class="quote no-group" data-username="Oliver" data-post="1" data-topic="135532">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>It suggested I could report an issue if the following code gets the same error, which it does:</p>
</blockquote>
</aside>
<p>Does this minimal conv/cuDNN code snippet also yield the illegal memory access in a clean environment or only after you’ve already hit the previous error in the GAN training?</p>
<aside class="quote no-group" data-username="Oliver" data-post="6" data-topic="135532">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>Unless I hard reset (factory reset the runtime)</p>
</blockquote>
</aside>
<p>Could you explain how you are performing the “factory reset”?</p>
<aside class="quote no-group" data-username="tom" data-post="7" data-topic="135532">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>So these CUDA errors should be fatal for the Python process they happen in but restarting the kernel should then make things work again.</p>
</blockquote>
</aside>
<p>Yes, that’s correct and if you are running into <code>assert</code>s (such as in the <code>nn.CrossEntropyLoss</code> use case) the CUDA context would be corrupted and restarting the Python kernel should work.</p>

Oliver: <aside class="quote no-group" data-username="ptrblck" data-post="10" data-topic="135532">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>python -m torch.utils.collect_env</p>
</blockquote>
</aside>
<p>Hi, thanks</p>
<p>To 1.:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.9.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0
Libc version: glibc-2.26

Python version: 3.7 (64-bit runtime)
Python platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.9.0+cu111
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.10.0
[pip3] torchvision==0.10.0+cu111
[conda] Could not collect
</code></pre>
<p>to 2 and 3)<br>
This is confusing to me, restarting the kernel, etc, I believe should result in a clean environment. Maybe that is not the case on Kaggle/Colab or my knowledge is just too poor.<br>
However, after restarting the error persists on both platforms. On Kaggle there is no fix unless to completely shut off and turn on (i’m guessing this switches the gpu too)<br>
Restarting the kernel/runtime doesn’t help and henceforth anything will result in this error that has a call to GPU.</p>
<p>On Colab there is an option under Runtime called Factory Reset Runtime. It clears everything.<br>
This works.</p>
<p>So yes, I do encounter it in a clean environment but only after I hit the previous error in a previous session even after restarting.</p>
<p>If I start with batch size &lt;= 4 all is fine.</p>

ptrblck: <p>Thanks for the follow-up. I’m not familiar with how Kaggle kernels work, but based on your description it doesn’t seem to be sufficient to restart the kernel.<br>
In any case, once you’ve “completely shut off” the Kaggle runtime, what kind of error are you seeing when running the code the first time?</p>

Oliver: <p>Thanks for helping!<br>
Sorry I was unclear.<br>
If I completely shut off and start I likely get a new gpu assigned, no error. Like colab’s factory reset.</p>
<p>Could it be the issue that I’m storing the whole array in memory?<br>
That’s the only major thing that is different this time arround, I repurposed old code I made which worked fine but loaded images from disk.<br>
I’ll try that next with this dataset too.</p>
<p>Edit Update:<br>
Get the same with reading from disk, weird.<br>
I think I have issue in my code.<br>
Above it points at:<br>
<code>allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag</code></p>
<p>something similar in tensorflow woudl mean the graph is disconnected.<br>
What I can’t make out is how this relates to batch_size</p>
<p>Edit update 2:<br>
Actually can do up to batch_size 6</p>

Oliver: <p>Update 3: I still get the error<br>
Also, batch_size 6 sometimes does it (works for a few epochs I think and eventually fails or maybe it’s really just sometimes)<br>
This is so weird.</p>
<p>To circle back to my initial suspicions<br>
Tensorflow has a similar error msg and it would mean that the graph is disconnected.<br>
Do you think this could be the case?<br>
How does batch size fit into that?</p>
<p>This is where it happens:</p>
<pre><code class="lang-auto">&lt;ipython-input-21-5f2cf50439c6&gt; in train(save_model)
     34             gen_loss = get_gen_loss(gen, disc, mask, image, adv_criterion, recon_criterion, 1000)
     35             gen_loss.backward()
---&gt; 36             gen_opt.step()
     37 
     38             # Keep track of the average discriminator loss

/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py in wrapper(*args, **kwargs)
     86                 profile_name = "Optimizer.step#{}.step".format(obj.__class__.__name__)
     87                 with torch.autograd.profiler.record_function(profile_name):
---&gt; 88                     return func(*args, **kwargs)
     89             return wrapper
     90 

/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     26         def decorate_context(*args, **kwargs):
     27             with self.__class__():
---&gt; 28                 return func(*args, **kwargs)
     29         return cast(F, decorate_context)
     30 

/usr/local/lib/python3.7/dist-packages/torch/optim/adam.py in step(self, closure)
    116                    lr=group['lr'],
    117                    weight_decay=group['weight_decay'],
--&gt; 118                    eps=group['eps'])
    119         return loss

/usr/local/lib/python3.7/dist-packages/torch/optim/_functional.py in adam(params, grads, exp_avgs, exp_avg_sqs, max_exp_avg_sqs, state_steps, amsgrad, beta1, beta2, lr, weight_decay, eps)
     85         # Decay the first and second moment running average coefficient
     86         exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
---&gt; 87         exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)
     88         if amsgrad:
     89             # Maintains the maximum of all 2nd moment running avg. till no
</code></pre>
<p>w</p>
<p>Thanks a lot, guys, it works with batch_size 4 but not knowing what is causing this makes me nuts.</p>

ptrblck: <aside class="quote no-group" data-username="Oliver" data-post="14" data-topic="135532">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>Do you think this could be the case?</p>
</blockquote>
</aside>
<p>No, I don’t think so. Based on your description it rather sounds as if you are running out of memory and then getting false errors due to a sticky error. However, I’m still unsure which error message you get the first time after running your code in a clean and working environment.</p>

Oliver: <p>Oh, I’m sorry, I seem to have misunderstood what you meant (asked about) earlier.<br>
The first time I run it with a batch_size &gt; 4 gives me this error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>It stays the same for all gpu related calls unless factory reset in colab even when restarting the kernel.</p>

ptrblck: <p>OK, thanks. Could you post an executable code snippet to reproduce the issue, please?<br>
In case you are using a custom dataset, please post the input shapes which would be needed to execute the training and run into the issue.</p>

tanweer-mahdi: <p>I am getting the same error within a Kaggle kernel. I was able to train the model successfully, however, during inference I receive this error. I have noticed that substantially lowering the <em>number of epochs</em> does not throw the error but I cannot proceed with this approach because the model has way too high bias with that few number of epochs.</p>
<p>I used <code>torch.cuda.empty_cache()</code> thinking that it is possibly originating from excessive memory usage. This is the result of <code>!nvidia-smi</code> post that:</p>
<pre><code class="lang-auto">Tue Mar 29 11:33:01 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.119.04   Driver Version: 450.119.04   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   40C    P0    37W / 250W |    997MiB / 16280MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
</code></pre>
<p>Is there any solution to these error? I am participating in a competition but have not been able to submit any prediction in days due to this error. Many thanks.</p>

ptrblck: <p>An illegal memory access error won’t be raised if you are running out of memory.<br>
One recommended approach is to update PyTorch to the latest release with the latest library stack and to check if this was a known and already fixed issue. Currently it would be the nightly binaries with the cUDA 11.5 runtime.</p>