Apex | CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

username: shrutee
cooked: <p>Hi,<br>
I am getting the following error while using the  <code>apex mixed precision</code>  library</p>
<p>The model that I am training a combination of many sub-models with their corresponding optimizers.</p>
<p>I am using only 1 GPU. Tried  <code>device = torch.device('cuda:0')</code>  doesn’t help</p>
<pre><code class="lang-auto">    h = self.frontend(x_, device)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/pase/pase/models/frontend.py", line 262, in forward
    y = self.W(h)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 200, in forward
    self.padding, self.dilation, self.groups)
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/wrap.py", line 28, in wrapper
    return orig_fn(*new_args, **kwargs)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 193, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 82, in cupy.cuda.driver.check_status
cupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
</code></pre>

username: shrutee
cooked: <p>Setting  <code>torch.backends.cudnn.benchmark = False</code>  resolves the error.<br>
as mentioned <a href="https://github.com/pytorch/pytorch/issues/14019" rel="nofollow noopener">here</a></p>
<p>However, without  <code>Apex</code>  and with  <code>torch.backends.cudnn.benchmark = True</code>  my code works just fine</p>

username: ptrblck
cooked: <p>Could you post a small reproducible code snippet, please?</p>

username: shrutee
cooked: <p>Thanks for replying. Here is the <a href="https://colab.research.google.com/gist/MittalShruti/66c129fe342f153adcc8bed273bc7d11/apex-code.ipynb?authuser=3" rel="nofollow noopener">code</a>.<br>
This is not a small code, however, all the starting cells are just loading the required files and the code.</p>
<p>The last cell is where I run the train.py function</p>
<p>Please let me know if more clarification is required.</p>

username: ptrblck
cooked: <p>Could you try to create a code snippet with the model and some random inputs, which create this error message please?</p>

username: shrutee
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Sorry for late response, I didn’t get a chance to work on this last week. I am attaching the error log. Will share the code snippet in a while. Thanks</p>
<p>Code (where mo and op are both lists containing models and optimizers)<br>
<code>mo, op = amp.initialize(mo, op, opt_level='O1',loss_scale="dynamic")</code></p>
<p>The overflow only occurs when batch size is 1. For batch size equals 16, my code directly throws illegal memory error.</p>
<pre><code class="lang-auto">Epoch 0/2:   0% 0/368 [00:00&lt;?, ?it/s]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Epoch 0/2:   0% 1/368 [00:03&lt;19:30,  3.19s/it]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Epoch 0/2:   1% 2/368 [00:03&lt;14:46,  2.42s/it]
Traceback (most recent call last):
  File "train.py", line 471, in &lt;module&gt;
    train(opts)
  File "train.py", line 339, in train
    Trainer.train_(dloader, device=device, valid_dataloader=va_dloader)
  File "trainer.py", line 317, in train_
    batch = batch)
  File "worker_scheduler.py", line 24, in __call__
    return self._base_scheduler(preds, label, cls_optim, regr_optim, frontend_optim, device)
  File "worker_scheduler.py", line 75, in _base_scheduler
    scaled_loss.backward()
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 193, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 82, in cupy.cuda.driver.check_status
cupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 193, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 82, in cupy.cuda.driver.check_status
cupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
</code></pre>

username: shrutee
cooked: <p>I am unable to reproduce the error, with code snippet containing the model and some random input <img src="https://discuss.pytorch.org/images/emoji/apple/astonished.png?v=9" title=":astonished:" class="emoji" alt=":astonished:"></p>
<p>I ll keep trying and revert as and when I ll have the error reproduced.</p>
<p>Instead of simplifying various steps I ll keep the code as is</p>

username: shrutee
cooked: <p>Hey I got the error replicated</p>
<p><a href="https://colab.research.google.com/gist/MittalShruti/a028fe51cb74df9596aae240f9863990/apex-illegal-memory-and-overflow.ipynb?authuser=3" rel="nofollow noopener">Code</a><br>
It’s a ~3k lines of code. Is that okay?<br>
I tried deleting parts, can go backwards and try deleting more if this wont work</p>

username: shrutee
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Installing nightly (<a href="https://discuss.pytorch.org/t/moving-tensor-to-cuda-with-and-without-apex/74874/2">as suggested here</a>) and then installing apex [<code>import apex as apm</code>] results in the following error:<br>
<a href="https://github.com/NVIDIA/apex/issues/581" rel="nofollow noopener">RuntimeError: expected scalar type Half but found Float</a></p>
<p>On switching back to <code>pytorch 1.4.0</code> I get the same <code>overflow</code>+ <code>illegal memory error</code></p>
<p><a href="https://colab.research.google.com/gist/MittalShruti/a028fe51cb74df9596aae240f9863990/apex-illegal-memory-and-overflow.ipynb?authuser=3" rel="nofollow noopener">Code</a></p>
<pre><code class="lang-auto">  0%|          | 2/1000 [00:04&lt;45:29,  2.73s/it]  Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
  0%|          | 4/1000 [00:04&lt;23:44,  1.43s/it]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
  1%|          | 6/1000 [00:04&lt;13:07,  1.26it/s]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
  1%|          | 8/1000 [00:05&lt;07:54,  2.09it/s]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
  1%|          | 8/1000 [00:05&lt;10:57,  1.51it/s]
---------------------------------------------------------------------------
CUDADriverError                           Traceback (most recent call last)
&lt;ipython-input-4-f59e8d90be01&gt; in &lt;module&gt;()
   2439                                                          temperture=temp,
   2440                                                          alpha= alpha,
-&gt; 2441                                                          batch = batch)
   2442 
   2443             if bidx % log_freq == 0 or bidx &gt;= bpe:

4 frames
&lt;ipython-input-4-f59e8d90be01&gt; in __call__(self, preds, label, cls_optim, regr_optim, frontend_optim, device, h, dropout_rate, delta, temperture, alpha, batch)
   1995 
   1996         if self.mode == "base":
-&gt; 1997             return self._base_scheduler(preds, label, cls_optim, regr_optim, frontend_optim, device)
   1998 
   1999     def _base_scheduler(self, preds, label, cls_optim, regr_optim, frontend_optim, device):

&lt;ipython-input-4-f59e8d90be01&gt; in _base_scheduler(self, preds, label, cls_optim, regr_optim, frontend_optim, device)
   2038 
   2039         with amp.scale_loss(tot_loss, op) as scaled_loss:
-&gt; 2040             scaled_loss.backward()
   2041 
   2042 

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    193                 products. Defaults to ``False``.
    194         """
--&gt; 195         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    196 
    197     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 99         allow_unreachable=True)  # allow_unreachable flag
    100 
    101 

/usr/local/lib/python3.6/dist-packages/torchqrnn/forget_mult.py in backward(self, grad_h)
    146         grid_hidden_size = min(hidden_size, 512)
    147         grid = (math.ceil(hidden_size / grid_hidden_size), batch_size)
--&gt; 148         self.bwd_forget_mult(grid=grid, block=(grid_hidden_size, 1), args=[h.data_ptr(), f.data_ptr(), x.data_ptr(), grad_h.data_ptr(), grad_f.data_ptr(), grad_x.data_ptr(), grad_h_init.data_ptr(), seq_size, batch_size, hidden_size], stream=self.stream)
    149         ###
    150         if hidden_init is not None:

cupy/cuda/function.pyx in cupy.cuda.function.Function.__call__()

cupy/cuda/function.pyx in cupy.cuda.function._launch()

cupy/cuda/driver.pyx in cupy.cuda.driver.launchKernel()

cupy/cuda/driver.pyx in cupy.cuda.driver.check_status()

CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
</code></pre>

username: ptrblck
cooked: <p>If you install the nightly binary, you don’t need to build and install apex/amp anymore, but can use directly the core implementation as explained in <a href="https://pytorch.org/docs/master/amp.html#module-torch.cuda.amp">this doc</a>.</p>

username: shrutee
cooked: <p>installing nightly binary on cuda100, installs pytorch 1.2-dev which doesn’t have <code>torch.cuda.amp</code></p>
<p><code>!pip3 install torch_nightly -f https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html --user</code></p>
<p>do i need to install cuda101 &amp; pytorch1.5-dev?</p>

username: ptrblck
cooked: <p>Try to use the command from the website:</p>
<pre><code class="lang-python">pip install numpy
pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
</code></pre>

username: shrutee
cooked: <p>This doesn’t update anything, I have cuda100</p>
<pre><code class="lang-auto">Looking in links: https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.4.0)
Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.5.0)
Requirement already satisfied: pillow&gt;=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (7.0.0)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.12.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.18.2)
</code></pre>

username: ptrblck
cooked: <p>Uninstall all PyTorch installations in your current environment or create a new environment and try to install the latest nightly again.</p>
<p>The binaries ship with their own CUDA, cudnn etc., so your system CUDA installation won’t be used.</p>