[announcement] Community effort for storing models metrics in one place. Anyone can help to gather results
Author: knrd
Pull request: no

TL;DR help me testing and documenting models performance on datasets benchmarks provided with transformers (i.e. GLUE tasks). Anyone with Google and Weights&Biases account can help. It's super easy and costs nothing, just execute colab notebook and help communiy with gathering results! All checked tasks results are available for whole communiy under: https://app.wandb.ai/transformers


Hi,
I wanted to have one place where I would be able to check which transformer model (and with what hyperparameters) performance best on particular NLP tasks. Task for benchmarking will be taken from `./examples` dir in transformers source code. I came up with the idea of using free Google Colab (of course code can be executed on any machine) and free Weights&Biases (WandB, wandb.com) panel as a place to store results. Everyone can participate, and with usage of free resources anyone can help without any costs.

What community will gain:
1. Access to all results on https://app.wandb.ai/transformers. Every task is a separate project. Results for each task can be filtered and grouped by model or any hyperparameter. WandB is free for open source projects
2. Ability to check running time and resources needed (GPU model and memory usage is stored) to train model for a specific task
3. Ability to find best performing models with needed hyperparameters

Disclaimer: I am not connected in any way with WandB, I chose them because their functionality suits me and they are claiming to be free for open source projects: https://www.wandb.com/academic.

How to participate:
1. Create free WandB account: https://app.wandb.ai/login?signup=true
2. Open https://colab.research.google.com/drive/1wbh8hmSy_8nNbvmQ_INFIDCFSlM1ZvvN
3. Click "Open in playground", then execute notebook (Runtime -> Run all), in 4th cell you will be asked to authorize in your WandB

That's it, script will 10 times randomly choose model and task, execute it and save results directly to https://app.wandb.ai/transformers. Script is configured to submit results to "transformers" group. You don't need to join this group, as it is publicly open and anyone can submit. Feel free to modify script or any hyperparameter.

Currently, only GLUE tasks (`./examples/run_glue.py`) are available for monitoring via WandB. If the community will like the idea and want to participate I will prepare also metrics storage for `./examples/run_multiple_choice.py` and `./examples/run_squad.py`.

Unfortunately, WandB don't allow to browse all projects while not logged in, so here is the actual list:
* GLUE
    * CoLA: https://app.wandb.ai/transformers/run_glue-cola
    * SST-2: https://app.wandb.ai/transformers/run_glue-sst-2
    * MRPC: https://app.wandb.ai/transformers/run_glue-mrpc
    * STS-B: https://app.wandb.ai/transformers/run_glue-sts-b
    * QQP: https://app.wandb.ai/transformers/run_glue-qqp
    * MNLI: https://app.wandb.ai/transformers/run_glue-mnli
    * QNLI: https://app.wandb.ai/transformers/run_glue-qnli
    * RTE: https://app.wandb.ai/transformers/run_glue-rte
    * WNLI: https://app.wandb.ai/transformers/run_glue-wnli

Roadmap:
1. Extend example scripts to calculate validation metrics not only on the end
2. Add metrics monitoring for `./examples/run_multiple_choice.py` and `./examples/run_squad.py`
3. Extend colab notebook for automatic installation of Nvidia Apex for FP16 training
4. Create github repo so the community can follow it for updates in scripts and notebooks

Last but not least, if you know good hyperparameters for a particular task from `./exemples`, but don't have time for playing with my script, feel free to share it here. Me or someone else will execute training with those hyperparameters and submit results to WandB.

So how do you like the idea of gathering model metrics in one place?

Comments:
The idea is very good, and has been discussed previously.

https://github.com/huggingface/transformers/issues/2520
https://github.com/huggingface/transformers/pull/2281#issuecomment-570574944

I think HuggingFace is well aware of the challenges and intricacies that are involved, so I'm sure they'll figure it out. I don't think using a separate platform is a good idea, though. There's already the rather basic webpage of (user) models (https://huggingface.co/models) so it would be better if the functionality that you are suggesting is integrated in that webpage.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.