Confusing RuntimeError: CUDA error: device-side assert triggered

Matthew_Chen37: <p>I’m trying to utilize my training/testing of my model using 4 GPUs by testing with a very simple CNN:</p>
<pre><code class="lang-auto">class SimpleCNN(nn.Module):
	def __init__(self, byte_size):
		super(SimpleCNN, self).__init__()
		self.l1 = nn.Sequential(nn.Linear(byte_size, int(byte_size/2)), nn.ReLU())
		self.l2 = nn.Sequential(nn.Linear(int(byte_size/2), int(byte_size/4)), nn.ReLU())
		self.l3 = nn.Sequential(nn.Linear(int(byte_size/4), 6))
	def forward(self, x):
		out = self.l1(x)
		out = self.l2(out)
		out = self.l3(out)
		return out
</code></pre>
<p>And utilizing this tutorial on moving my model to multiple gpus <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#create-model-and-dataparallel" rel="noopener nofollow ugc">tutorial</a>:</p>
<pre><code class="lang-auto">device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = SimpleCNN(BYTE_BLOCK_SIZE)
if torch.cuda.device_count() &gt; 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    model = nn.DataParallel(model)
model.to(device)

loss_function = nn.NLLLoss() # This is a convex loss function
optimizer = torch.optim.Adam(model.parameters(), lr=.03)
</code></pre>
<p>I get the following error (full trace):</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-86-3548497e2569&gt; in &lt;module&gt;
      6     print("Let's use", torch.cuda.device_count(), "GPUs!")
      7     model = nn.DataParallel(model)
----&gt; 8 model.to(device)
      9 
     10 loss_function = nn.NLLLoss() # This is a convex loss function

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in to(self, *args, **kwargs)
    850             return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
    851 
--&gt; 852         return self._apply(convert)
    853 
    854     def register_backward_hook(

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)
    528     def _apply(self, fn):
    529         for module in self.children():
--&gt; 530             module._apply(fn)
    531 
    532         def compute_should_use_set_data(tensor, tensor_applied):

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)
    528     def _apply(self, fn):
    529         for module in self.children():
--&gt; 530             module._apply(fn)
    531 
    532         def compute_should_use_set_data(tensor, tensor_applied):

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)
    528     def _apply(self, fn):
    529         for module in self.children():
--&gt; 530             module._apply(fn)
    531 
    532         def compute_should_use_set_data(tensor, tensor_applied):

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _apply(self, fn)
    550                 # `with torch.no_grad():`
    551                 with torch.no_grad():
--&gt; 552                     param_applied = fn(param)
    553                 should_use_set_data = compute_should_use_set_data(param, param_applied)
    554                 if should_use_set_data:

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in convert(t)
    848                 return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
    849                             non_blocking, memory_format=convert_to_format)
--&gt; 850             return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
    851 
    852         return self._apply(convert)

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>Note that when using the model only on the CPU the code works fine and I’m able to easily train the model with no warnings/errors.</p>
<p>Then I checked this <a href="https://github.com/facebookresearch/maskrcnn-benchmark/issues/508" rel="noopener nofollow ugc">post</a> because they had a similar error and when I did</p>
<pre><code class="lang-auto">import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
torch.rand(1).cuda()
</code></pre>
<p>I still get the error:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-91-e97f59385399&gt; in &lt;module&gt;
      1 os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
----&gt; 2 torch.rand(1).cuda()

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>So I couldn’t get a proper stacktrace even when <code>CUDA_LAUNCH_BLOCKING=1 </code>.<br>
This was all done in Jupyter Notebook by the way. But what is even more weird is if I run the exact code above I do NOT get any error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbfa0df6972cb0f1a3a76382db9d4a5fed4b753f.png" data-download-href="https://discuss.pytorch.org/uploads/default/dbfa0df6972cb0f1a3a76382db9d4a5fed4b753f" title="weirderror"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbfa0df6972cb0f1a3a76382db9d4a5fed4b753f_2_690x332.png" alt="weirderror" data-base62-sha1="vo0iW1DpRxWrSslEiadyrSPcZHh" width="690" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbfa0df6972cb0f1a3a76382db9d4a5fed4b753f_2_690x332.png, https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbfa0df6972cb0f1a3a76382db9d4a5fed4b753f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbfa0df6972cb0f1a3a76382db9d4a5fed4b753f.png 2x" data-dominant-color="31333E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">weirderror</span><span class="informations">764×368 54.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this a problem with my CUDA version? On my system I have CUDA 11.2 but I installed cuda toolkit=10.2 with Pytorch and I thought that Pytorch would use the cuda toolkit installed with Pytorch within Conda?</p>

ptrblck: <aside class="quote no-group" data-username="Matthew_Chen37" data-post="1" data-topic="126864">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matthew_chen37/48/40322_2.png" class="avatar"> Matthew_Chen37:</div>
<blockquote>
<p>This was all done in Jupyter Notebook by the way.</p>
</blockquote>
</aside>
<p>Setting <code>CUDA_LAUNCH_BLOCKING=1</code> after the CUDA context was already created won’t have any effect, which seems to be the case here. Thus I do not recommend to set this env variable inside the script and be especially careful when running in Jupyter notebooks (restart the Kernel after each failure to avoid confusing sticky failures).</p>
<p>Use <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> (or <code>export</code> the env variable) and rerun your code.</p>

Arfeen_93: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any solution for this problem:  When I am sending my model to the Cuda device, I am getting CUDA out of memory. My GPU is also unoccupied. It was working fine some time ago. I am unable to find any bug.</p>

ptrblck: <p>This might be pointing towards a setup issue. E.g. in case you’ve recently updated the drivers and didn’t restart the machine.<br>
Do you know what has changes since the last time it was working?</p>

Arfeen_93: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> The issue was from gpu machine side. I restarted it and now it is working fine. Thanks anyway <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>