Added mixed precision (AMP) to inference benchmark
Author: tlkh
Pull request: yes

I added a mixed precision option to the benchmark script and ran it on a DGX Station to get the results. As you can see, we can get between 1.2x to up to 4.5x inference speed depending on model, batch size and sequence length. 

**Summary**

| Batch Size | Speedup (XLA only) | Speedup (XLA + AMP) | Min. Seq Len* |
| -------------- | --------------------------- | ------------------------------- | ------------------ |
| 1                | 1.1 ~ 1.9                    | 1.4 ~ 2.9                       | 512                 |
| 2                | 1.1 ~ 1.9                    | 1.4 ~ 3.4                       | 256                 |
| 4                | 1.1 ~ 2.1                    | 1.2 ~ 3.8                       | 128                 |
| 8                | 1.1 ~ 3.1                    | 1.2 ~ 4.5                       | 64                   |

*Min. Seq Len refers to minimum sequence length required to not see **any** performance regression at all. For example, at batch size 1:
* Seq Len of 512 tokens see speed up of 1.4~2.1x depending on model
* Seq Len of 256 tokens see speed up of 0.8~1.2x depending on model

Google Sheets with the results [here](https://docs.google.com/spreadsheets/d/1IW7Xbv-yfE8j-T0taqdyoSehca4mNcsyx6u0IXTzSJ4/edit#gid=0). GPU used is a single V100 (16GB).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1567?src=pr&el=h1) Report
> Merging [#1567](https://codecov.io/gh/huggingface/transformers/pull/1567?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/079bfb32fba4f2b39d344ca7af88d79a3ff27c7c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1567/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1567?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1567   +/-   ##
======================================
  Coverage    85.9%   85.9%           
======================================
  Files          91      91           
  Lines       13653   13653           
======================================
  Hits        11728   11728           
  Misses       1925    1925
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1567?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1567?src=pr&el=footer). Last update [079bfb3...079bfb3](https://codecov.io/gh/huggingface/transformers/pull/1567?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Any reason why you kept the batch sizes so small? With a V100, you should be able to easily pull of a batch size of 64 for seq len 64. (Perhaps that's what the benchmark script uses as default, I don't know EDIT: yes, those are the values from the benchmark script. Not sure why, though.) I'm a bit surprised by the relatively small speed up. I've experienced **much** greater speed ups when using AMP, but that was on PyTorch with apex.

@BramVanroy this is for **inference** hence the emphasis on low batch size.

Oh, my bad. I was under the impression that the benchmark script included training profiling with PyProf.