get NaN loss when I run the example code run_squad.py
Author: kugwzk
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I use the example run_squad.py code, and use the Readme's hyper-parameters, but I got nan loss when I trained a few batches. And I use the `autograd.detect_anomaly()` want to catch that. 

The more information is below:
> File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
self._target(*self._args, **self._kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 1211, in forward
    attention_mask=attention_mask, head_mask=head_mask)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 713, in forward
    head_mask=head_mask)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 434, in forward
    layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 414, in forward
    intermediate_output = self.intermediate(attention_output)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 386, in forward
    hidden_states = self.intermediate_act_fn(hidden_states)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 145, in gelu
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
Traceback (most recent call last):
  File "run_squad.py", line 544, in <module>
    main()
  File "run_squad.py", line 490, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_squad.py", line 165, in train
    loss.backward()
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/users4/zkwang/miniconda3/envs/MRC/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'MulBackward0' returned nan values in its 0th output.

My environment PyTorch version: torch 1.1.0, pytorch-transformers 1.2.0, and use 4 Titan X gpus for train. I don't know why the official code cause this result, could someone help me about that...

Comments:
Maybe the learning rate is too high?

I change a GPU node and this situation doesn't appear. I will change the learning rate and see results. Thanks a lot.