CUDA error: device-side assert triggered after a number of epochs

username: antran96
cooked: <p>My model was run fine but after 89 epochs, I got this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 36, in &lt;module&gt;
    main()
  File "main.py", line 32, in main
    method.method(settings, job_id)
  File "/scratch/project_2002806/tranan11/AC-hybrid-transformer/processes/method.py", line 574, in method
    nb_classes=nb_classes)
  File "/scratch/project_2002806/tranan11/AC-hybrid-transformer/processes/method.py", line 328, in _do_training
    optimizer=None)
  File "/scratch/project_2002806/tranan11/AC-hybrid-transformer/tools/model.py", line 156, in module_epoch_passing
    y_hat, y, f_names_tmp = module_forward_passing(example, module, use_y)
  File "/scratch/project_2002806/tranan11/AC-hybrid-transformer/tools/model.py", line 217, in module_forward_passing
    return module(x, None), y, f_names
  File "/appl/soft/ai/miniconda3/envs/pytorch-1.3.1-1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/.../models/wavenet_rnn.py", line 94, in forward
    return self._inference(x)
  File "/.../models/wavenet_rnn.py", line 139, in _inference
    self.max_length,
  File "/.../modules/decode_utils.py", line 22, in greedy_decode
    attention_mask = None
  File "/appl/soft/ai/miniconda3/envs/pytorch-1.3.1-1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/..../AC-hybrid-transformer/modules/transformer.py", line 31, in forward
    s_mask = subsequent_mask(word_embed.size(0)).to(device)
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>This is the code where it triggers this error:</p>
<pre><code class="lang-auto">def subsequent_mask(sz):
    mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1).float()
    mask = mask.masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
    return mask
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Did you get any prints (in stderr) before the error that would hint at what is the issue?</p>
<p>Also you can rerun your code while setting the <code>CUDA_LAUNCH_BLOCKING=1</code> environment variable to make sure that the line showed by the stack trace is relevant (the API is asynchronous by default and so it most likely points to the wrong line right now).</p>

username: antran96
cooked: <p>Hi, the code was run with  <code>CUDA_LAUNCH_BLOCKING=1</code> . It worked normally until epoch 90 in my case, which is weird…</p>

username: albanD
cooked: <blockquote>
<p>It worked normally until epoch 90 in my case, which is weird…</p>
</blockquote>
<p>Actually this is quite common as these asserts often come from indexing out of bound.<br>
So maybe a label in your dataset is incorrect. Or you use a learnt value to index something and that ends up generating a value out of bound.</p>
<blockquote>
<p>the code was run with  <code>CUDA_LAUNCH_BLOCKING=1</code></p>
</blockquote>
<p>Given that the stack trace points to the l 31 in transformer.py, and not inside the <code>subsequent_mask()</code> function, I think it means that the error either comes from the <code>.size()</code> or the <code>.to()</code> function.<br>
The first one does not access the GPU (GPU Tensor metadata are saved on the cpu side).<br>
So most likely the second one. And this second one is a know sync point for the GPU and so is usually the place where the error is raised in the async case.<br>
I am surprised though that it still points here with <code>CUDA_LAUNCH_BLOCKING=1</code>. Can you double check that you do set this before doing anything else and that it is taken into account? (you can try a simple program that index out of bound on the GPU to make sure it does what you expect).</p>
<p>Also, do you use multithreading or multiprocessing? These error put the GPU in an un-recoverable state and the process needs to be restarted. So another thread doing something bad could lead another thread to through this kind of errors at random places.</p>