[benchmark tool] trainer-benchmark.py
Author: stas00
Pull request: yes
Labels: name: Performance

name: WIP

This PR adds a benchmarking tool for HF Trainer args -  e.g. compare --fp16 vs --bf16 performance, but can do that for multiple dimensions and it prints automatic tables suitable for instant pasting in Issues, including relative performance. e.g.:


| Variation                        |   Train<br>samples<br>per<br>second |   Diff<br>% |   Train<br>loss |
|:---------------------------------|------------------------------------:|------------:|----------------:|
| --per_device_train_batch_size 1  |                                7.77 |         0 |            1.90 |
| --per_device_train_batch_size 2  |                               15.51 |         100 |            2.01 |
| --per_device_train_batch_size 4  |                               29.66 |         282 |            2.09 |
| --per_device_train_batch_size 8  |                               61.16 |         687 |            2.16 |
| --per_device_train_batch_size 16 |                              115.84 |        1392 |            2.25 |
| --per_device_train_batch_size 32 |                              224.96 |        2797 |            2.38 |

This is produced by:
```

CUDA_VISIBLE_DEVICES=0 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-base \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 5000 --dataloader_num_workers 2 --bf16' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--per_device_train_batch_size 1|--per_device_train_batch_size 2|--per_device_train_batch_size 4|--per_device_train_batch_size 8|--per_device_train_batch_size 16|--per_device_train_batch_size 32' \
--report-metric-keys train_loss --repeat-times 1 
```

To add more dimensions simply add another `--variations` arg, e.g.:

```
--variations '|--fp16|--bf16' '--tf32 0|--tf32 1' 
```

will lead to a Cartesian product of each arg with an outcome of:


| Variation       |   Train<br>samples<br>per<br>second |   Diff<br>% |   Train<br>loss |
|:----------------|------------------------------------:|------------:|----------------:|
| --tf32 0        |                              272.59 |         0 |            2.49 |
| --tf32 1        |                              581.61 |         113 |            2.49 |
| --fp16 --tf32 0 |                              643.07 |         136 |            2.49 |
| --fp16 --tf32 1 |                              635.24 |         133 |            2.49 |
| --bf16 --tf32 0 |                              616.23 |         126 |            2.50 |
| --bf16 --tf32 1 |                              612.59 |         125 |            2.50 |


See the doc at the beginning of the script for details. But it has lots of cool features like automatically reporting the hardware/software, preformatting everything for both copy-n-paste into the Issues/docs and also an easy to see console-only 2nd version at the end for when you debug things.

And more practical examples and reports for its use are here: https://github.com/huggingface/transformers/issues/15026 and https://github.com/huggingface/transformers/issues/14608

It should be relatively easy to adopt this tool to be used with `accelerate` or with any other command line tool as long as there is a defined way to get to results, so each tool will need to have its own sub-class if we decided to extend it. I think @siddk suggested he might look into extending it.

But let's see first that you like it and I placed it in a good location. The intention for it to be an internal tool, so I hope any `map` and such code will be tolerated.

It's possible that we could use it for some basic regression testing. But it'd need to be further extended to support storing results and detecting regressions.

@LysandreJik, @patrickvonplaten, @sgugger, @patil-suraj 

Comments:
_The documentation is not available anymore as the PR was closed or merged._