Fix mixed precision for TFGPT2LMHeadModel
Author: mymusise
Pull request: yes

Fixed the loss of precision when using mixed_precision.
Not sure it's the right way to do this, correct me if it's wrong.

related issue: https://github.com/huggingface/transformers/issues/8559#issuecomment-797528526


- gpt2: @patrickvonplaten, @LysandreJik
- tensorflow: @jplu

Comments:
> This is not the proper way to proceed. Here you force the dtype to float32 which is not correct, because you might encounter conflicts with other values that are in float16.

Yes, you're right, I should not force the dtype to float32, maybe I should force it to float32 when using `mixed_precision`.

> To get similar results, you have to increase the number of epochs. Usually, I get a similar model after multiplying the number of epochs by 2 or 2.5.

But in my sample: 
I always get 0.98 accuracy after 18 epochs without using the `mixed_precision`, 
but sometimes the accuracy will stop at 0.7333 when using the `mixed_precision` policy, even I train it after 100 epochs.

I think there must something wrong. I have updated my sample, have a look: https://colab.research.google.com/github/mymusise/gpt2-quickly/blob/main/examples/mixed_precision_test.ipynb

> Yes, you're right, I should not force the dtype to float32, maybe I should force it to float32 when using mixed_precision.

No! This would be even worse. Set the layer norm and the embeddings directly to `float32` is a better temporary fix.

> I always get 0.98 accuracy after 18 epochs without using the mixed_precision, but sometimes the accuracy will stop at 0.7333 when using the mixed_precision policy, even I train it after 100 epochs.

Having a lower accuracy in mixed precision is normal, but not that much. How much do you get by setting the layer norm and the embeddings directly to `float32` in mixed precision?

I cannot run any test for now as I don't have access to a computer so I cannot really use your Colab.

> No! This would be even worse. Set the layer norm and the embeddings directly to `float32` is a better temporary fix.

My apologies, I didn't get it, do you mean the first commit is better? :joy:

> Having a lower accuracy in mixed precision is normal, but not that much. How much do you get by setting the layer norm and the embeddings directly to `float32` in mixed precision?
> 
> I cannot run any test for now as I don't have access to a computer so I cannot really use your Colab.

Sorry to disturb your rest time. I can get 0.99 accuracy after setting the layer norm and the embeddings directly to `float32` in mixed precision.
Here's a partial screenshot:
![image](https://user-images.githubusercontent.com/6883957/110977074-a9b0b880-839c-11eb-98d0-d006c926ce0c.png)


> My apologies, I didn't get it, do you mean the first commit is better?

Yes.

> Sorry to disturb your rest time. I can get 0.99 accuracy after setting the layer norm and the embeddings directly to float32 in mixed precision.

Thanks for the screenshot. Can you please revert your last commit then.

> Thanks for the screenshot. Can you please revert your last commit then.

Sure!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@Rocketknight1
Hello, what should I do to help with this?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.