Convert the data type of embeddings and masks to bfloat16 for torch amp
Author: CaoE
Pull request: yes

### Motivation
Add a attribute `use_torch_bfloat16_embeddings` in `PretrainedConfig` to indicate if bfloat16 data type for embeddings and masks is used and convert the data type of embeddings and masks to bfloat16 accordingly.

This will reduce the number of data type conversion between float and bfloat16 when running models with `torch.cpu.amp.autocast(dtype=torch.bfloat16)` and improve the performance with little accuracy regression. This is because there are many residual modules in models and thus result in data type promotion by binary operations implemented by tensoriterator in PyTorch. 

For example: out = tensor1 + tensor2
If data type of tensor1 is float and tensor2 is bfloat16, pytorch will convert tensor2 to float and get float output. When running models using amp for bfloat16, the conversion will results in additional `to` operations, which will reduce performance.

### Testing


- Number of `to` operations

Model | wo/ bf16 embedding and masks| w/ bf16 embedding and masks
-- | -- | --
albert |  22 | 11
bert | 49 | 10
bart | 65 | 38
gpt2 | 56 | 29
distilbert | 40 | 19
roberta | 54 | 15


- Accuracy testing

Model | fp32 | amp bf16 | amp bf16 w/ bf16 embedding
-- | -- | -- | --
masked-language-modeling+bert-base-cased | 0.4819 | 0.4818 | 0.4819
masked-language-modeling+distilbert-base-cased | 0.3143 | 0.3158 | 0.3152
multiple-choice+distilbert-base-cased | 0.246 | 0.2461 | 0.2454
multiple-choice+google-electra-base-discriminator | 0.1193 | 0.1194 | 0.1201
text-classification+google-electra-base-generator | 0.6901 | 0.6838 | 0.6838
token-classification+google-electra-base-generator | 0.0414 | 0.0411 | 0.041
token-classification+gpt2 | 0.0379 | 0.0379 | 0.0379
albert | 0.453431373 | 0.428921569 | 0.446078431
distilbert | 0.681372549 | 0.681372549 | 0.681372549
roberta | 0.683823529 | 0.683823529 | 0.683823529
xlm-roberta | 0.637254902 | 0.637254902 | 0.639705882



## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for your PR!
We're not really interested in adding optimizations like this in each model file, as if we were to do this for all possible hardwares and dtypes, the code would be unreadable. Since each model is defined in its own file, it's easy for a user to customize the code for their specific need (like here for bfloat16).

@sgugger  Thank you for your comments!  Yes, adding optimizations like this in each model file is not general. Users can customize the code for their need based on existing models. I posted the PR for testing and discussing, and also want to see if there is any way in huggingface to avoid such additional data type conversions since for some tasks like masked-language-modeling+bert-base-cased there may **be 30% performance drop**.

@sgugger May I know if there is any way in huggingface to avoid such additional data type conversions ? Thanks for your any advice !