Getting Nans from dropout layer

username: krishansubudhi
cooked: <p>My model is throwing NaNs intermittently. From debugging, i found on every occasion, dropout was the layer whose output was NaN first. Why is dropout outputing NaNs?</p>
<p>Model is being trained in mixed precission.</p>
<h2>I added this hook</h2>
<pre><code class="lang-python">def nan_hook(self, inp, output):
            if not isinstance(output, tuple):
                outputs = [output]
            else:
                outputs = output

            for i, out in enumerate(outputs):
                nan_mask = torch.isnan(out)
                if nan_mask.any():
                    print("Hook: Nan occured In", self.__class__.__name__)
for submodule in self.model.modules():
            submodule.register_forward_hook(nan_hook)
</code></pre>
<h2>Output</h2>
<p>70_driver_log_9.txt-SystemLog: 2020-02-22 17:13:45,253:DEBUG : transformers_pretraining.trainer.apexDDP : 159 : ***** Training step 172 *****<br>
70_driver_log_9.txt-SystemLog: 2020-02-22 17:13:45,253:DEBUG : transformers_pretraining.utils : 47 : Inside &lt;function Singleprocess._forward at 0x7f773402fb70&gt;<br>
70_driver_log_9.txt:Hook: Nan occured In Dropout<br>
70_driver_log_9.txt:Hook: Nan occured In BertSelfAttention<br>
70_driver_log_9.txt:Hook: Nan occured In Linear<br>
70_driver_log_9.txt:Hook: Nan occured In Dropout<br>
70_driver_log_9.txt:Hook: Nan occured In LayerNorm<br>
70_driver_log_9.txt:Hook: Nan occured In BertSelfOutput<br>
70_driver_log_9.txt:Hook: Nan occured In BertAttention</p>
<h2>Model architecture:</h2>
<p>BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=1024, out_features=1024, bias=True)<br>
(key): Linear(in_features=1024, out_features=1024, bias=True)<br>
(value): Linear(in_features=1024, out_features=1024, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=1024, out_features=1024, bias=True)<br>
(LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=1024, out_features=4096, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=4096, out_features=1024, bias=True)<br>
(LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)</p>

username: mcarilli
cooked: <p>That is strange, dropout should be a straight copy.  Also, it’s considered a neutral op for mixed precision (inputs aren’t changed).  Are you sure dropout is most recent operation that occurred?  I’d put a print statement right before and after the offending dropout function call to be sure.</p>
<p>In any case, if you’re using Apex Amp, you should be aware that will soon be deprecated.  I’m working on native Pytorch support for mixed precision, targeting the upcoming 1.5 release:<br>
<a href="https://slack-redir.net/link?url=https%3A%2F%2Fgithub.com%2Fpytorch%2Fpytorch%2Fpull%2F32140&amp;v=3" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/32140</a><br>
<a href="https://slack-redir.net/link?url=https%3A%2F%2Fgithub.com%2Fpytorch%2Fpytorch%2Fpull%2F33366&amp;v=3" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/33366</a><br>
PRs have upstream approval and are undergoing end-to-end testing locally.  It may be preferable to wait for the native integration, and see if that resolves your issue.</p>

username: krishansubudhi
cooked: <p>Hi,<br>
Yes I am sure about this. My hook always prints dropout as the first layer where Nan appears in the output. This has happening more than once.</p>
<p>It’s hard to write print statements as I am using huggungface Bert. That’s why used the hooks.</p>
<p>This Nan issue is very frequent at 217th epoch which reduces amp loss scaler to 0 resulting in division by zero error.</p>
<p>We are not sure how to solve this. What would you suggest for now ?</p>

username: Nguyen_Van_Nga_Adtec
cooked: <p>I got the same problem. Using the hook function show me that error come from dropout. Can anyone help me?</p>

username: dake
cooked: <p>Hi, I also suffered from the same problem and I found the reason.</p>
<p>Dropout layer doesn’t cause NaN values.</p>
<p>There is a softmax layer right before dropout layer and the softmax layer causes NaN.<br>
<a href="https://github.com/huggingface/transformers/blob/972fdcc77878cf7afcc8aef8979d6b4241005bb6/src/transformers/models/bert/modeling_bert.py#L355" rel="noopener nofollow ugc">https://github.com/huggingface/transformers/blob/972fdcc77878cf7afcc8aef8979d6b4241005bb6/src/transformers/models/bert/modeling_bert.py#L355</a></p>
<p>Is there any suggestion which solve numerical unstability of softmax layer?</p>
<p>I’m implementing mixed precision training and I found that softmax layer is not stable with fp16 input.</p>
<p>Thanks!</p>

username: dake
cooked: <p>This paper helped me to reduce the occurrence of NaN after softmax layer.</p>
<p><a href="https://arxiv.org/pdf/2202.03493.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2202.03493.pdf</a></p>