[Benchmark] add tpu and torchscipt for benchmark
Author: patrickvonplaten
Pull request: yes

This PR adds:

- Torchscript memory and time benchmarking
- TPU memory and time benchmarking

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4850?src=pr&el=h1) Report
> Merging [#4850](https://codecov.io/gh/huggingface/transformers/pull/4850?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/42860e92a4a99a8be338644462cfc3f62d1379a3&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `78.62%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4850/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4850?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4850      +/-   ##
==========================================
+ Coverage   76.97%   77.01%   +0.03%     
==========================================
  Files         128      128              
  Lines       21533    21615      +82     
==========================================
+ Hits        16575    16646      +71     
- Misses       4958     4969      +11     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4850?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4850/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.13% <ø> (ø)` | |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4850/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.38% <46.66%> (-0.24%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4850/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.09% <50.00%> (-1.19%)` | :arrow_down: |
| [src/transformers/training\_args.py](https://codecov.io/gh/huggingface/transformers/pull/4850/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzLnB5) | `77.41% <66.66%> (-0.71%)` | :arrow_down: |
| [src/transformers/benchmark/benchmark.py](https://codecov.io/gh/huggingface/transformers/pull/4850/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrLnB5) | `68.85% <69.76%> (+0.16%)` | :arrow_up: |
| [src/transformers/benchmark/benchmark\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4850/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrX3V0aWxzLnB5) | `73.09% <96.49%> (+5.85%)` | :arrow_up: |
| [src/transformers/benchmark/benchmark\_args.py](https://codecov.io/gh/huggingface/transformers/pull/4850/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrX2FyZ3MucHk=) | `85.36% <100.00%> (-0.35%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4850?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4850?src=pr&el=footer). Last update [42860e9...0267668](https://codecov.io/gh/huggingface/transformers/pull/4850?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


**Baseline GPU results**:

======= INFERENCE - SPEED - RESULT =======
	======= MODEL CHECKPOINT: distilbert-base-uncased =======
		distilbert-base-uncased/8/8: 0.007s
		distilbert-base-uncased/8/32: 0.009s
		distilbert-base-uncased/8/128: 0.022s
		distilbert-base-uncased/8/512: 0.1s
	======= MODEL CHECKPOINT: bert-base-cased =======
		bert-base-cased/8/8: 0.015s
		bert-base-cased/8/32: 0.025s
		bert-base-cased/8/128: 0.072s
		bert-base-cased/8/512: 0.332s
======= INFERENCE - MEMORY - RESULT =======
	======= MODEL CHECKPOINT: distilbert-base-uncased =======
		distilbert-base-uncased/8/8: 274 MB
		distilbert-base-uncased/8/32: 298 MB
		distilbert-base-uncased/8/128: 324 MB
		distilbert-base-uncased/8/512: 552 MB
	======= MODEL CHECKPOINT: bert-base-cased =======
		bert-base-cased/8/8: 458 MB
		bert-base-cased/8/32: 462 MB
		bert-base-cased/8/128: 488 MB
		bert-base-cased/8/512: 728 MB


**Torchscript GPU results:**

======= INFERENCE - SPEED - RESULT =======
	======= MODEL CHECKPOINT: distilbert-base-uncased =======
		distilbert-base-uncased/8/8: 0.005s
		distilbert-base-uncased/8/32: 0.009s
		distilbert-base-uncased/8/128: 0.02s
		distilbert-base-uncased/8/512: 0.096s
	======= MODEL CHECKPOINT: bert-base-cased =======
		bert-base-cased/8/8: 0.012s
		bert-base-cased/8/32: 0.025s
		bert-base-cased/8/128: 0.073s
		bert-base-cased/8/512: 0.328s
======= INFERENCE - MEMORY - RESULT =======
	======= MODEL CHECKPOINT: distilbert-base-uncased =======
		distilbert-base-uncased/8/8: 274 MB
		distilbert-base-uncased/8/32: 296 MB
		distilbert-base-uncased/8/128: 312 MB
		distilbert-base-uncased/8/512: 552 MB
	======= MODEL CHECKPOINT: bert-base-cased =======
		bert-base-cased/8/8: 458 MB
		bert-base-cased/8/32: 460 MB
		bert-base-cased/8/128: 488 MB
		bert-base-cased/8/512: 716 MB

check colab here: https://colab.research.google.com/drive/10KSu_6X6unsKXPOiwiGP6QDC1fLtADFJ?usp=sharing

The differences seem very small to me. What do you think @LysandreJik ?

**TPU memory and time usage**

======= INFERENCE - SPEED - RESULT =======
	======= MODEL CHECKPOINT: distilbert-base-uncased =======
		distilbert-base-uncased/8/8: 0.004s
		distilbert-base-uncased/8/32: 0.005s
		distilbert-base-uncased/8/128: 0.004s
		distilbert-base-uncased/8/512: 0.005s
	======= MODEL CHECKPOINT: bert-base-cased =======
		bert-base-cased/8/8: 0.01s
		bert-base-cased/8/32: 0.008s
		bert-base-cased/8/128: 0.009s
		bert-base-cased/8/512: 0.009s
TPU was used for inference. Note that the time after compilation stabilized (after ~10 inferences model.forward(..) calls) was measured.
======= INFERENCE - MEMORY - RESULT =======
	======= MODEL CHECKPOINT: distilbert-base-uncased =======
		distilbert-base-uncased/8/8: 1027 MB
		distilbert-base-uncased/8/32: 1118 MB
		distilbert-base-uncased/8/128: 1118 MB
		distilbert-base-uncased/8/512: 1118 MB
		distilbert-base-uncased/32/512: 1028 MB
		distilbert-base-uncased/64/512: 1066 MB
	======= MODEL CHECKPOINT: bert-base-cased =======
		bert-base-cased/8/8: 1330 MB
		bert-base-cased/8/32: 1332 MB
		bert-base-cased/8/128: 1332 MB
		bert-base-cased/8/512: 1332 MB
		bert-base-cased/32/512: 1314 MB
		bert-base-cased/64/512: 1334 MB


In comparison to the GPU times - this seems reasonable to me. Kind of weird that for longer sequences it takes teh same amount of time or less...

At the moment I'm measuring CPU usage for TPU - not at all sure how to measure memory usage correctly for TPU...any ideas @LysandreJik ? UPDATE: Pretty sure that memory usage is wrong for TPU 

Google colab is here: 
https://colab.research.google.com/drive/1vp9y7R2bLYTrK8hWOIo8VFHGm6M7ft0B?usp=sharing

Requesting @julien-c's review for the move of `is_tpu_available` to the utils.

UPDATE:

I'm fine with PyTorch for CPU, GPU with and without torchscript results:
https://docs.google.com/spreadsheets/d/1vgAIG7P3AOdBp5X91rVVu8AqnZ_hAvFzKj_fTNolAlU/edit?usp=sharing

TPU running times also seem to be fine. TPU memory is not yet implemented - will probably wait here until there is a PyTorch XLA API: https://github.com/pytorch/xla/issues/2180

Good to merge for me, waiting for @julien-c to check it out

Good for me

Okey changed it to `is_torch_tpu_available()`. Think that's fine. Pinging @julien-c @LysandreJik to notice the change.

Indeed, nice change!