Added FP16 support to benchmarks.py
Author: tlkh
Pull request: yes

This PR adds in FP16 support for the inference benchmarks for TensorFlow and PyTorch, and presents the collected results. This is a "re-do" of a previous PR (#1567) taking into account changes to `benchmark.py` and also adding in the PyTorch component with additional results collected.

**TensorFlow**

Added a automatic mixed precision (AMP) option to the benchmark script. As you can see, we can get between 1.2x to up to 4.5x inference speed depending on model, batch size and sequence length. (1.0x refers to no change in speed)

| Batch Size | Speedup (XLA only) | Speedup (XLA + AMP) | Min. Seq Len* |
| -------------- | --------------------------- | ------------------------------- | ------------------ |
| 1                | 1.1 ~ 1.9                    | 1.4 ~ 2.9                       | 512                 |
| 2                | 1.1 ~ 1.9                    | 1.4 ~ 3.4                       | 256                 |
| 4                | 1.1 ~ 2.1                    | 1.2 ~ 3.8                       | 128                 |
| 8                | 1.1 ~ 3.1                    | 1.2 ~ 4.5                       | 64                   |

*Min. Seq Len refers to minimum sequence length required to not see **any** performance regression at all. For example, at batch size 1:
* Seq Len of 512 tokens see speed up of 1.4~2.1x depending on model
* Seq Len of 256 tokens see speed up of 0.8~1.2x depending on model

**PyTorch**

Added a FP16 (half precision) option to the benchmark script. As you can see, we can get between up to 4.2x inference speed depending on model, batch size and sequence length. (1.0x refers to no change in speed)

| Batch Size | Speedup (TorchScript only) | Speedup (FP16 Only) |
| -------------- | ------------------------------------- | ----------------------------- |
| 1                | 1.0 ~ 1.7                               | 1.0 ~ 3.0                      |
| 2                | 1.0 ~ 1.8                               | 1.0 ~ 3.5                      |
| 4                | 1.0 ~ 1.7                               | 1.0 ~ 4.0                      |
| 8                | 1.0 ~ 1.7                               | 1.4 ~ 4.2                      |

*FP16 and CTRL result in performance regression below 1x256, 2x128, 4x64.

**Summary of Collected Results**

Google Sheets with the TF/PyTorch results [here](https://docs.google.com/spreadsheets/d/1IW7Xbv-yfE8j-T0taqdyoSehca4mNcsyx6u0IXTzSJ4/edit#gid=1307979840). GPU used is a single V100 (16GB).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1667?src=pr&el=h1) Report
> Merging [#1667](https://codecov.io/gh/huggingface/transformers/pull/1667?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/079bfb32fba4f2b39d344ca7af88d79a3ff27c7c?src=pr&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1667/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1667?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1667      +/-   ##
==========================================
+ Coverage    85.9%   85.92%   +0.02%     
==========================================
  Files          91       91              
  Lines       13653    13653              
==========================================
+ Hits        11728    11732       +4     
+ Misses       1925     1921       -4
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1667?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1667/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2ZpbGVfdXRpbHMucHk=) | `76.37% <0%> (+2.19%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1667?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1667?src=pr&el=footer). Last update [079bfb3...2669079](https://codecov.io/gh/huggingface/transformers/pull/1667?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thank you @tlkh ! Feel free to add a link to your spreadsheet in the documentation.