Benchmarks
Author: patrickvonplaten
Pull request: yes

# Benchmarks

This PR adds the functionality to measure the following functionalities for TF and PT:

**Tensorflow:** 
- Inference: CPU, GPU, GPU + XLA, GPU + eager mode, CPU + eager mode, TPU 

**PyTorch:** 
- Inference: CPU, CPU + torchscript, GPU, GPU + torchscript, GPU + mixed precision, Torch/XLA TPU 
- Training: CPU, GPU, GPU + mixed precision, Torch/XLA TPU

## How is memory measured?

**CPU**

We are always interested in the peak memory usage of the process. For CPU, the library `psutil` in combination with multiprocessing is leveraged

**GPU**

It is difficult to have exact memory measurement on GPU. Tensorflow allocates the full GPU memory by default. This is disabled with `tf.config.experimental.set_memory_growth=True`, but Tensorflow still allocates more memory than it needs for efficiency as far as I know.
=> Memory is therefore always measured to give the same maximal result as shown by `nvidia-smi`. This means that also memory for loading PyTorch / Tensorflow is taken into account which is for example not done when measuring via `torch.cuda.max_allocated_memory`.
Tensorflow also does not release GPU memory before the process is finished. Therefore, all measurement functions are wrapped into their own spawned process via Python's multiprocessing tools. 

Also note that because TF does not release memory during the same process, memory and inference is measured using a multiprocess approach in TF. Also TF does not provide an official memory monitoring function, so that the same result that `nvidia-smi` would show for TF is used. 

**TPU**

Memory measurement is currently not supported

## How is speed measured?

For all functionality that requires compilation (TPU, XLA, Torchscript), 5 warmup calls of the function are done beforehand.

Afterwards, the minimum of `self.args.repeat` x the time-averaged over 10 function calls.  

## Example Colabs:

The colabs give quick examples for each functionality with little explanation for the moment:

Pytorch TPU: https://colab.research.google.com/drive/1GJFOdcBe1pW_FKWpA0jK_AOsIQ5epcvE?usp=sharing
Tensorflow TPU:
https://colab.research.google.com/drive/1t8DW1NxA4b1BsWSZ1ehFG9oT69l0h7os?usp=sharing

GPU: https://colab.research.google.com/drive/15XTPT_GPp42Zj7_f1W9X_T3NNXE9_1Te?usp=sharing
CPU: https://colab.research.google.com/drive/1OG2rZgo18KvliS-ratybld9pHD06-v5S?usp=sharing

## Future PR:

- [ ] Make nicer examples and explanations
- [ ] Update docs and think about automatic measuring on website
- [ ] Training in TF. Because the LM Head models currently do not accept `labels` parameter as an input, adding measurement for training is left for a future PR
- [ ] GPU fp16 in TF. We currently have a bug in the lib that does not allow to run TF models in fp16 on GPU: https://github.com/huggingface/transformers/issues/3320
- [ ] PyTorch's amp package has memory leaks, so that we simply do `model.half()` to measure fp16 in Pytorch. See issue here: https://github.com/NVIDIA/apex/issues/439 . Wait until amp is supported in upstream torch 1.6
- [ ] Currently memory is not measured on TPU. Wait for more functionality for TPU
- [ ] Allow multi-gpu measurments


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4912?src=pr&el=h1) Report
> Merging [#4912](https://codecov.io/gh/huggingface/transformers/pull/4912?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/355954ffca798bb81d9db8886e30ce10f11e8a40&el=desc) will **decrease** coverage by `0.94%`.
> The diff coverage is `78.29%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4912/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4912?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4912      +/-   ##
==========================================
- Coverage   77.28%   76.34%   -0.95%     
==========================================
  Files         133      134       +1     
  Lines       22134    22369     +235     
==========================================
- Hits        17107    17078      -29     
- Misses       5027     5291     +264     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4912?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/benchmark/benchmark\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrX3V0aWxzLnB5) | `69.77% <68.51%> (-3.20%)` | :arrow_down: |
| [src/transformers/benchmark/benchmark\_args\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrX2FyZ3NfdXRpbHMucHk=) | `89.13% <71.42%> (-7.75%)` | :arrow_down: |
| [src/transformers/benchmark/benchmark.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrLnB5) | `79.13% <76.00%> (+9.70%)` | :arrow_up: |
| [src/transformers/benchmark/benchmark\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrX3RmLnB5) | `82.69% <82.69%> (ø)` | |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `76.80% <86.66%> (+1.40%)` | :arrow_up: |
| [src/transformers/benchmark/benchmark\_args\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrX2FyZ3NfdGYucHk=) | `87.50% <87.50%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.18% <100.00%> (+0.02%)` | :arrow_up: |
| [src/transformers/benchmark/benchmark\_args.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrX2FyZ3MucHk=) | `86.04% <100.00%> (+0.68%)` | :arrow_up: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.57% <100.00%> (+0.18%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl90NS5weQ==) | `50.10% <0.00%> (-43.61%)` | :arrow_down: |
| ... and [7 more](https://codecov.io/gh/huggingface/transformers/pull/4912/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4912?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4912?src=pr&el=footer). Last update [355954f...8b71041](https://codecov.io/gh/huggingface/transformers/pull/4912?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> This is great. I really like that you can import the benchmark if you want to use them during runtime, rather than the only option being to run a script.
> 
> Some remarks after playing with it:
> 
> * Maybe you should raise an error when no `model_names` are specified. Right now it crashes with `UnboundLocalError: local variable 'inference_summary' referenced before assignment` (pytorch version at least)
> * There seems to be an error in the way the runtimes are computed. PyTorch using GPU, is slower than TensorFlow on CPU (10x times slower), while PyTorch on CPU is 150x slower than TensorFlow on CPU.
> 
> Here are the results from my runs so far. The following is on CPU with TensorFlow (2ms per inference with `bert-base-cased`, seq len 8 and batch size 512 on a CPU??) I didn't test the memory usage so they're not in the results:
> 
> ```
> ====================       INFERENCE - SPEED - RESULT       ====================
> --------------------------------------------------------------------------------
>           Model Name             Batch Size     Seq Length     Time in s   
> --------------------------------------------------------------------------------
>        bert-base-cased               8               8             0.001     
>        bert-base-cased               8               32            0.001     
>        bert-base-cased               8              128            0.001     
>        bert-base-cased               8              512            0.002     
> --------------------------------------------------------------------------------
> 
> ====================        ENVIRONMENT INFORMATION         ====================
> - transformers_version: 2.11.0
> - framework: Tensorflow
> - eager_mode: False
> - use_xla: False
> - framework_version: 2.2.0
> - python_version: 3.6.10
> - system: Linux
> - cpu: 
> - architecture: 64bit
> - date: 2020-06-18
> - time: 11:57:18.595804
> - fp16: False
> - use_multiprocessing: True
> - cpu_ram_mb: 64333
> - use_gpu: False
> - use_tpu: False
> ```
> 
> Here's the test with PyTorch on GPU:
> 
> ```
> ====================       INFERENCE - SPEED - RESULT       ====================
> --------------------------------------------------------------------------------
>           Model Name             Batch Size     Seq Length     Time in s   
> --------------------------------------------------------------------------------
>        bert-base-cased               8               8             0.007     
>        bert-base-cased               8               32            0.007     
>        bert-base-cased               8              128            0.019     
>        bert-base-cased               8              512            0.074     
> --------------------------------------------------------------------------------
> 
> ====================        ENVIRONMENT INFORMATION         ====================
> - transformers_version: 2.11.0
> - framework: PyTorch
> - use_torchscript: False
> - framework_version: 1.5.0
> - python_version: 3.6.10
> - system: Linux
> - cpu: 
> - architecture: 64bit
> - date: 2020-06-18
> - time: 11:56:31.041360
> - fp16: False
> - use_multiprocessing: True
> - cpu_ram_mb: 64333
> - use_gpu: True
> - num_gpus: 1
> - gpu: N/A
> - gpu_ram_mb: N/A
> - gpu_power_watts: N/A
> - gpu_performance_state: N/A
> - use_tpu: False
> ```
> 
> I'm not sure that PyTorch on GPU is ~37x slower than TensorFlow on CPU  I tried to debug but it's not easy to debug tf functions unfortunately

Thanks a lot for checking everything! Found the error :-) One just has to return a tensor out of the tf.function context so that it is actually computed. I guess before compilation TF compilation optimizes the function so that variables that are not used outside of the @tf.function scope are not computed.

Will update the notebooks and should then getter more reasonable results :-) 

And will definitely add a better error message

The speed tests seem much more reasonable now, if you check the notebooks :-) @LysandreJik 
There seems to be a problem with GPU memory in TF now :-/ Will check tomorrow again

## GPU locally gives reasonable results of TF vs. PT.

All tests were run in this environment:

```
- transformers_version: 2.11.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-19
- time: 13:49:57.455208
- use_multiprocessing: True
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
```

for TF 2.2 and Pytorch 1.4.0

### PyTorch
`python run_benchmark.py --models gpt2 bert-base-cased --no_env_print --no_memory` gives:

--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
             gpt2                    8               8             0.006     
             gpt2                    8               32            0.007     
             gpt2                    8              128            0.026     
             gpt2                    8              512            0.104     
       bert-base-cased               8               8             0.006     
       bert-base-cased               8               32            0.006     
       bert-base-cased               8              128            0.021     
       bert-base-cased               8              512            0.094     
--------------------------------------------------------------------------------



### PyTorch FP16
`python run_benchmark.py --models gpt2 bert-base-cased --no_env_print --no_memory --fp16`


--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
             gpt2                    8               8             0.006     
             gpt2                    8               32            0.007     
             gpt2                    8              128            0.009     
             gpt2                    8              512            0.043     
       bert-base-cased               8               8             0.006     
       bert-base-cased               8               32            0.006     
       bert-base-cased               8              128            0.006     
       bert-base-cased               8              512             0.03   

### TF no eager modus

```python run_benchmark_tf.py --models gpt2 bert-base-cased --no_env_print --no_memory```

--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
             gpt2                    8               8             0.005     
             gpt2                    8               32            0.007     
             gpt2                    8              128            0.029     
             gpt2                    8              512            0.125     
       bert-base-cased               8               8             0.005     
       bert-base-cased               8               32            0.006     
       bert-base-cased               8              128            0.024     
       bert-base-cased               8              512            0.114     
--------------------------------------------------------------------------------

### TF XLA

```python run_benchmark_tf.py --models gpt2 bert-base-cased --no_env_print --no_memory --use_xla```

--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
             gpt2                    8               8             0.002     
             gpt2                    8               32            0.006     
             gpt2                    8              128            0.021     
             gpt2                    8              512            0.095     
       bert-base-cased               8               8             0.003     
       bert-base-cased               8               32            0.005     
       bert-base-cased               8              128            0.019     
       bert-base-cased               8              512            0.087     
--------------------------------------------------------------------------------


## Memory measurements 

They also seem reasonable for forward pass:.

### TF no eager mode (keeping in mind that nvidia-smi is not accurate here and TF always allocates more than it needs):

```python run_benchmark_tf.py --models gpt2 bert-base-cased --no_env_print --no_speed```

--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
             gpt2                    64              8              1704     
             gpt2                    64              32             1704     
             gpt2                    64             128             2728     
             gpt2                    64             512             8872     
       bert-base-cased               64              8              1192     
       bert-base-cased               64              32             1192     
       bert-base-cased               64             128             1704     
       bert-base-cased               64             512             4776     
--------------------------------------------------------------------------------

### PyTorch 

```python run_benchmark.py --models gpt2 bert-base-cased --no_env_print --no_speed```

--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
             gpt2                    64              8              1150     
             gpt2                    64              32             1384     
             gpt2                    64             128             2290     
             gpt2                    64             512             5890     
       bert-base-cased               64              8              1016     
       bert-base-cased               64              32             1104     
       bert-base-cased               64             128             1448     
       bert-base-cased               64             512             3224     
--------------------------------------------------------------------------------

### PyTorch FP16

```python run_benchmark.py --models gpt2 bert-base-cased --no_env_print --no_speed  --fp16```

--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
             gpt2                    64              8              1170     
             gpt2                    64              32             1164     
             gpt2                    64             128             1596     
             gpt2                    64             512             3420     
       bert-base-cased               64              8              1066     
       bert-base-cased               64              32             1060     
       bert-base-cased               64             128             1108     
       bert-base-cased               64             512             2118     
--------------------------------------------------------------------------------