Bug in run_classifier.py fp16 learning rate
Author: dalek-who
Pull request: no

After the latest update, my learning rate of fp16 in run_classifier.py keeps increasing.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/2dee86319dbad575352358b8f2fb4129940e381a/examples/run_classifier.py#L857-L858

I think the right code is: lr_this_step = args.learning_rate * warmup_linear.get_lr(global_step, args.warmup_proportion). 

https://github.com/huggingface/pytorch-pretrained-BERT/blob/2dee86319dbad575352358b8f2fb4129940e381a/pytorch_pretrained_bert/optimization.py#L53-L62
In this function, the first argument should be step, and global_step/num_train_optimization_steps makes process calculated again and when input process to WarmupLinearSchedule, it would be too small to decrease
https://github.com/huggingface/pytorch-pretrained-BERT/blob/2dee86319dbad575352358b8f2fb4129940e381a/pytorch_pretrained_bert/optimization.py#L162-L171


Comments:
I'm asking the same question

I had been dealing with the issue of low and decreasing accuracy when I use fp16, as shown below,

```
Epoch 1 - Batch 1600/287417 - Training Acc. 0.106250 - Training Loss 2.295977
Epoch 1 - Batch 3200/287417 - Training Acc. 0.098125 - Training Loss 2.299707
Epoch 1 - Batch 4800/287417 - Training Acc. 0.094792 - Training Loss 2.304948
Epoch 1 - Batch 6400/287417 - Training Acc. 0.093125 - Training Loss 2.307725
Epoch 1 - Batch 8000/287417 - Training Acc. 0.092625 - Training Loss 2.305703
Epoch 1 - Batch 9600/287417 - Training Acc. 0.091667 - Training Loss 2.306758
Epoch 1 - Batch 11200/287417 - Training Acc. 0.092589 - Training Loss 2.306116
Epoch 1 - Batch 12800/287417 - Training Acc. 0.092969 - Training Loss 2.307227
Epoch 1 - Batch 14400/287417 - Training Acc. 0.091458 - Training Loss 2.310017
Epoch 1 - Batch 16000/287417 - Training Acc. 0.091000 - Training Loss 2.308750
Epoch 1 - Batch 17600/287417 - Training Acc. 0.090795 - Training Loss 2.309631
Epoch 1 - Batch 19200/287417 - Training Acc. 0.090625 - Training Loss 2.310771
Epoch 1 - Batch 20800/287417 - Training Acc. 0.090433 - Training Loss 2.310832
Epoch 1 - Batch 22400/287417 - Training Acc. 0.090625 - Training Loss 2.311030
Epoch 1 - Batch 24000/287417 - Training Acc. 0.090083 - Training Loss 2.311357
Epoch 1 - Batch 25600/287417 - Training Acc. 0.089883 - Training Loss 2.311748
Epoch 1 - Batch 27200/287417 - Training Acc. 0.089449 - Training Loss 2.312302
Epoch 1 - Batch 28800/287417 - Training Acc. 0.088993 - Training Loss 2.312582
Epoch 1 - Batch 30400/287417 - Training Acc. 0.088651 - Training Loss 2.313187
Epoch 1 - Batch 32000/287417 - Training Acc. 0.088656 - Training Loss 2.313006
Epoch 1 - Batch 33600/287417 - Training Acc. 0.088750 - Training Loss 2.313333
Epoch 1 - Batch 35200/287417 - Training Acc. 0.088665 - Training Loss 2.314015
Epoch 1 - Batch 36800/287417 - Training Acc. 0.088641 - Training Loss 2.313631
Epoch 1 - Batch 38400/287417 - Training Acc. 0.088854 - Training Loss 2.313276
Epoch 1 - Batch 40000/287417 - Training Acc. 0.089325 - Training Loss 2.312648
Epoch 1 - Batch 41600/287417 - Training Acc. 0.089183 - Training Loss 2.312943
Epoch 1 - Batch 43200/287417 - Training Acc. 0.089051 - Training Loss 2.312587
Epoch 1 - Batch 44800/287417 - Training Acc. 0.088929 - Training Loss 2.313172
Epoch 1 - Batch 46400/287417 - Training Acc. 0.088793 - Training Loss 2.312671
Epoch 1 - Batch 48000/287417 - Training Acc. 0.088479 - Training Loss 2.313255
Epoch 1 - Batch 49600/287417 - Training Acc. 0.088972 - Training Loss 2.312710
Epoch 1 - Batch 51200/287417 - Training Acc. 0.088906 - Training Loss 2.312372
```

However, after I made the change in `lr_this_step` that you indicated, I've started to get normal results, as follows,

```
Epoch 1 - Batch 1600/287417 - Training Acc. 0.156250 - Training Loss 2.224727
Epoch 1 - Batch 3200/287417 - Training Acc. 0.200937 - Training Loss 2.166289
Epoch 1 - Batch 4800/287417 - Training Acc. 0.245833 - Training Loss 2.098184
Epoch 1 - Batch 6400/287417 - Training Acc. 0.299063 - Training Loss 2.018706
Epoch 1 - Batch 8000/287417 - Training Acc. 0.351625 - Training Loss 1.937730
Epoch 1 - Batch 9600/287417 - Training Acc. 0.400833 - Training Loss 1.855378
Epoch 1 - Batch 11200/287417 - Training Acc. 0.444018 - Training Loss 1.768468
Epoch 1 - Batch 12800/287417 - Training Acc. 0.481875 - Training Loss 1.685869
Epoch 1 - Batch 14400/287417 - Training Acc. 0.513889 - Training Loss 1.606483
Epoch 1 - Batch 16000/287417 - Training Acc. 0.536937 - Training Loss 1.537816
Epoch 1 - Batch 17600/287417 - Training Acc. 0.556364 - Training Loss 1.477735
Epoch 1 - Batch 19200/287417 - Training Acc. 0.576146 - Training Loss 1.418323
Epoch 1 - Batch 20800/287417 - Training Acc. 0.592019 - Training Loss 1.367327
Epoch 1 - Batch 22400/287417 - Training Acc. 0.606429 - Training Loss 1.321059
Epoch 1 - Batch 24000/287417 - Training Acc. 0.617542 - Training Loss 1.281488
Epoch 1 - Batch 25600/287417 - Training Acc. 0.627109 - Training Loss 1.246746
Epoch 1 - Batch 27200/287417 - Training Acc. 0.637500 - Training Loss 1.211883
Epoch 1 - Batch 28800/287417 - Training Acc. 0.645938 - Training Loss 1.182604
Epoch 1 - Batch 30400/287417 - Training Acc. 0.652204 - Training Loss 1.158571
Epoch 1 - Batch 32000/287417 - Training Acc. 0.658875 - Training Loss 1.134463
Epoch 1 - Batch 33600/287417 - Training Acc. 0.665179 - Training Loss 1.111719
Epoch 1 - Batch 35200/287417 - Training Acc. 0.671023 - Training Loss 1.089363
Epoch 1 - Batch 36800/287417 - Training Acc. 0.676848 - Training Loss 1.068860
Epoch 1 - Batch 38400/287417 - Training Acc. 0.681536 - Training Loss 1.050721
Epoch 1 - Batch 40000/287417 - Training Acc. 0.685775 - Training Loss 1.034663
Epoch 1 - Batch 41600/287417 - Training Acc. 0.690361 - Training Loss 1.017672
Epoch 1 - Batch 43200/287417 - Training Acc. 0.693866 - Training Loss 1.004058
Epoch 1 - Batch 44800/287417 - Training Acc. 0.698013 - Training Loss 0.990084
Epoch 1 - Batch 46400/287417 - Training Acc. 0.701552 - Training Loss 0.977086
Epoch 1 - Batch 48000/287417 - Training Acc. 0.704854 - Training Loss 0.965735
Epoch 1 - Batch 49600/287417 - Training Acc. 0.708266 - Training Loss 0.953387
Epoch 1 - Batch 51200/287417 - Training Acc. 0.712012 - Training Loss 0.940919
Epoch 1 - Batch 52800/287417 - Training Acc. 0.714697 - Training Loss 0.929941
```

Thanks!

@burcturkoglu 
How about performance comparison with fp32?

@yeontaek 
I trained BERT for classification with my own data by _run_classifier_ script.

Here are the benchmarks for fp32 vs fp16 in both single Tesla V100 and in 4 Tesla V100 with _DataParallel_,

*_fp32:_*

- Single Tesla V100 - Training Duration 17,739 seconds

- 4 Tesla V100 - Training Duration 9,342 seconds

*_fp16:_*

- Single Tesla V100 - Training Duration 12,297 seconds

- 4 Tesla V100 - Training Duration 6,330 seconds

In both types of instances, it gives approximately 30% increase in speed without a change in accuracy. 

@burcturkoglu 
Thank you so much. It was a big help.