[Benchmark] V100/A100 benchmarks, dashboard concept
Author: tlkh
Pull request: no
Labels: name: wontfix

# 🖥 Benchmarking `transformers`

I benchmarked a couple of models (training) on V100 and A100 and wrapped everything into a Streamlit dashboard [link here](https://share.streamlit.io/tlkh/transformers-benchmarking/main/app.py).

This dashboard shows the measured performance of GPUs when training various configurations of Transformer networks, showing throughput (seq/s) and GPU memory (VRAM) usage. The idea is to allow users have an easy reference for choosing model configuration (model size/batch size/sequence length) and GPU model.

This is kind of a weekend project done out of curiosity. If there is potential, perhaps a more serious effort can be undertaken here. 

## Benchmark

Which part of `transformers` did you benchmark?

Model training: `distilroberta-base`, `roberta-base`, `roberta-large` via `AutoModelForSequenceClassification`

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

GPUs: V100 16GB, A100 40GB

Single GPU only.

More information and code: https://github.com/tlkh/transformers-benchmarking

## Results

https://share.streamlit.io/tlkh/transformers-benchmarking/main/app.py


Comments:
That's incredible @tlkh!

By the way, did you try to use the [benchmarking tool](https://huggingface.co/transformers/benchmarks.html) that we have in the library to run this evaluation?

> By the way, did you try to use the benchmarking tool that we have in the library to run this evaluation?

I did take a look at the tools, but I rolled my own in the end:

- I also wanted to write my own training script and benchmark using that, also doing a test drive of the end-to-end PyTorch Lightning + HuggingFace Datasets & Transformers library
- I wanted to do profiling and capture more metrics than just the time and memory (mostly out of curiosity)

In hindsight, I think the time and memory are the most important metrics and those are captured by the benchmarking tool in the library. 

I do think putting the benchmarks into the dashboard format would be valuable to let people visualize and see which configurations would work best for them. Can treat the one I made as a proof of concept, and we can see where to go from here if interested. 

Very cool. I was planning on running benchmarks on exactly these cards as well but now I don't need to anymore! Is it possible to update the app in such a way that you can also see a side-by-side comparison of the V100 and the A100? I imagine that some people would like to see the benefit of the A100 easily without having to change the view for each graph. A side-by-side bar chart would be cool!

> Very cool. I was planning on running benchmarks on exactly these cards as well but now I don't need to anymore! Is it possible to update the app in such a way that you can also see a side-by-side comparison of the V100 and the A100? I imagine that some people would like to see the benefit of the A100 easily without having to change the view for each graph. A side-by-side bar chart would be cool!

That's possible. In any case, the raw results are all in a CSV file [here](https://github.com/tlkh/transformers-benchmarking/blob/main/results.csv) so you can easily do comparisons for any scenarios you're interested in. 

Very cool. For a minute I thought Streamlit had shipped GPU instances in their cloud and the benchmarks were computed directly there 😉 

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.