Division by zero with apex after 217 epoch

username: krishansubudhi
cooked: <p>Hi I am training a Bert model with amp O2 using 64 gpus . It trains successfully for 217 epochs but at 218th epoch, the loss scale goes down rapidly and model fails with division by zero error. Run does not fail when apex is disabled.</p>
<p>What can be the problem here?</p>
<p>Printed the gradients and Max param after backward . Max param is 11, gradients are Nan. Why does Nan appear ?</p>

username: ptrblck
cooked: <p>Are you seeing a decreasing loss scaler after 217 epochs or which operation triggers the division by zero error?<br>
If the loss scaling is going down rapidly, your model output or loss might be an invalid value (NaN or Inf) and thus all steps are skipped until the loss scaler dies.</p>
<p>It’s strange that this issue happens after so many epochs, so I assume your input data is clean.<br>
Could you check the max and min absolute value of the activations during the training?</p>

username: krishansubudhi
cooked: <p>It’s after few steps in 217 epochs. Loss is not NaN.</p>
<p>SystemLog: 2020-02-19 06:18:40,416:DEBUG : transformers_pretraining.trainer.apexDDP : 26 : Enabling all reduce<br>
SystemLog: 2020-02-19 06:18:40,417:DEBUG : transformers_pretraining.trainer.apexDDP : 138 : ***** Training step 1532 *****<br>
SystemLog: 2020-02-19 06:18:40,417:DEBUG : transformers_pretraining.utils : 47 : Inside &lt;function Singleprocess._forward at 0x7f7266673840&gt;<br>
SystemLog: 2020-02-19 06:18:40,417:DEBUG : transformers_pretraining.utils : 48 : torch.cuda.get_device_properties(0).total_memory = 16914055168, torch.cuda.memory_allocated() = 5356990464<br>
SystemLog: 2020-02-19 06:18:40,469:DEBUG : transformers_pretraining.trainer.apexDDP : 45 : loss scale = 64.0, loss = 0.6044921875<br>
SystemLog: 2020-02-19 06:18:40,469:DEBUG : transformers_pretraining.trainer.apexDDP : 47 : scaled loss = 38.6875<br>
model , optimizer max grad before clipping             nan, nan<br>
model , optimizer max grad after clipping             nan, nan<br>
max optimizer parameter : 11.71293830871582<br>
SystemLog: 2020-02-19 06:18:41,270:DEBUG : transformers_pretraining.trainer.apexDDP : 119 : model<br>
module.bert.embeddings.word_embeddings.weight = tensor([-0.0005, -0.0307,  0.0093,  0.0120, -0.0311], device=‘cuda:0’,<br>
dtype=torch.float16, grad_fn=),                     grad = tensor([nan, nan, nan, nan, nan], device=‘cuda:0’, dtype=torch.float16)                        , sum = nan<br>
SystemLog: 2020-02-19 06:18:41,272:DEBUG : transformers_pretraining.trainer.apexDDP : 125 : optimizer<br>
tensor([-0.0005, -0.0307,  0.0093,  0.0120, -0.0311], device=‘cuda:0’,<br>
grad_fn=) tensor([nan, nan, nan, nan, nan], device=‘cuda:0’) torch.float32 tensor(nan, device=‘cuda:0’)<br>
max model parameter : 11.7109375<br>
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32.0</p>

username: krishansubudhi
cooked: <p>Code: self._optimizer_step_flag is True every 4th step. GA = 4</p>
<pre><code class="lang-python">def _fp16_backwards(self,loss):

    self.logger.debug(f'loss scale = {_amp_state.loss_scalers[0].loss_scale()}, loss = {loss.item()}')

    with amp.scale_loss(loss,self.optimizer, delay_unscale= not self._optimizer_step_flag) as scaled_loss:

        self.logger.debug(f'scaled loss = {scaled_loss.item()}')

        scaled_loss.backward()

    # Clip gradients only when optimizer is about to step. Fixes divergence issue. 

    # TODO: patch _optimize and move code there.

    # TODO: clip grads for non FP16 training also

    if self._optimizer_step_flag:

        print(f'optimizer norm = {get_norm(amp.master_params(self.optimizer))}')

        print(f'model , optimizer max grad before clipping \

            {get_max_grad(self.model.parameters())}, {get_max_grad(amp.master_params(self.optimizer))}')

        torch.nn.utils.clip_grad_norm_(amp.master_params(self.optimizer), 1.0)

        print(f'model , optimizer max grad after clipping \

            {get_max_grad(self.model.parameters())}, {get_max_grad(amp.master_params(self.optimizer))}')

        print(f'max optimizer parameter : {max([torch.max(p).item() for p in amp.master_params(self.optimizer)])}')
</code></pre>

username: krishansubudhi
cooked: <p>Checked activation and loss. None of them are Nan or inf.</p>

username: 11118
cooked: <p>Hi, have you solved this problem? I met same error when training BERT with amp O2 after about 200k iterations and training XLM-R with amp O1 after about 10k iterations.</p>

username: ptrblck
cooked: <p>We recommend to try out <code>torch.cuda.amp</code>, which is shipped in the current master and the nightly binaries.<br>
Have a look at <a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">this post</a> for more information.</p>

username: krishansubudhi
cooked: <p>I got the same error with torch.cuda.amp too. If you are using huggungface Bert or xlm models, remove the Pooler layer from the optimizer. That solved the issue for us in O2.</p>

username: 111429
cooked: <p>Hello, I have the similar issue. I use huggingface Bart. What does it mean “remove the Pooler layer from the optimizer”?Thank you.</p>