Trainer
Author: julien-c
Pull request: yes

This is a bottom-up refactor of the example scripts (currently `run_glue.py`, `run_language_modeling.py`, `run_ner.py` and `run_multiple_choice.py`) into a Trainer class and associated utilities, as described in [trainer-proposal](https://github.com/julien-c/trainer-proposal).

# Regression test and benchmark on `run_glue.py`

📊 All training metrics are in [**this TensorBoard**](https://tensorboard.dev/experiment/7LuNpqw3Q8y147WxfyOqdA/#scalars&_smoothingWeight=0&runSelectionState=eyJkYXRhcGFyYWxsZWxfMl9ncHUiOnRydWUsImRkcF8yX2dwdSI6dHJ1ZX0%3D):

<a href="https://tensorboard.dev/experiment/7LuNpqw3Q8y147WxfyOqdA/#scalars&_smoothingWeight=0&runSelectionState=eyJkYXRhcGFyYWxsZWxfMl9ncHUiOnRydWUsImRkcF8yX2dwdSI6dHJ1ZX0%3D"><img width="1559" alt="Screenshot 2020-04-14 at 21 23 26" src="https://user-images.githubusercontent.com/326577/79289007-3655ac80-7e96-11ea-8737-3db1839725c2.png"></a>

The Trainer class supports PyTorch's backends for parallel/distributed training so we performed the following experiment:

- Experiment: MNLI
  - Train set: 100_000 first samples
  - Dev set: Full (~9_800 samples)
- Backends: DataParallel, DistributedDataParallel, single-GPU, CPU

You can compare speed of convergence by clicking on "Relative" in the TensorBoard and comparing loss and accuracy curves:

<img width="360" alt="Screenshot 2020-04-14 at 21 40 04" src="https://user-images.githubusercontent.com/326577/79289930-baa92f00-7e98-11ea-900f-106f0ef22681.png">

## Results

### DataParallel

```
--model_name_or_path distilbert-base-cased
--task_name mnli
--data_dir ./data/glue_data/MNLI
--output_dir ./models/dataparallel_2_gpu
--overwrite_output_dir
--do_train
--do_eval
--num_train_epochs 1
--per_gpu_train_batch_size 32
--per_gpu_eval_batch_size 128
--logging_steps 100
--logging_dir ./runs/dataparallel_2_gpu
--logging_first_step
--save_steps 1000
--evaluate_during_training
```

1 Epoch = 21 mins

```
04/14/2020 23:16:34 - INFO - __main__ -   ***** Eval results mnli *****
04/14/2020 23:16:34 - INFO - __main__ -     acc = 0.7406011207335711
04/14/2020 23:16:34 - INFO - __main__ -     loss = 0.6281169515389663

04/14/2020 23:17:02 - INFO - __main__ -   ***** Eval results mnli-mm *****
04/14/2020 23:17:02 - INFO - __main__ -     acc = 0.7507119609438568
04/14/2020 23:17:02 - INFO - __main__ -     loss = 0.6062953961201203
```

### DistributedDataParallel 

```
python -m torch.distributed.launch --nproc_per_node 2 ./examples/run_glue.py
--model_name_or_path distilbert-base-cased
--task_name mnli
--data_dir ./data/glue_data/MNLI
--output_dir ./models/ddp_2_gpu
--overwrite_output_dir
--do_train
--do_eval
--num_train_epochs 1
--per_gpu_train_batch_size 32
--per_gpu_eval_batch_size 128
--logging_steps 100
--logging_dir ./runs/ddp_2_gpu
--logging_first_step
--save_steps 1000
--evaluate_during_training
```

Speed: about same speed as DataParallel on this workload and machine.

Pre-existing issue (to fix in future PR): when using DDP, the eval is not GPU-parallelized.

### single-GPU

`CUDA_VISIBLE_DEVICES=0 python ...`

<details>
<pre>
04/15/2020 00:52:24 - INFO - __main__ -   ***** Eval results mnli *****
04/15/2020 00:52:24 - INFO - __main__ -     acc = 0.7383596535914416
04/15/2020 00:52:24 - INFO - __main__ -     loss = 0.631212914144838
04/15/2020 00:53:16 - INFO - __main__ -   ***** Eval results mnli-mm *****
04/15/2020 00:53:16 - INFO - __main__ -     acc = 0.7534580960130187
04/15/2020 00:53:16 - INFO - __main__ -     loss = 0.6002480050960144
</pre>
</details>

Speed: Twice slower

### CPU

`--no_cuda`

Speed: too slow to benchmark

# Regression test on `run_ner.py`

The arguments below:

```
--model_name_or_path bert-base-multilingual-cased
--data_dir ./data/germeval
--labels ./data/germeval/labels.txt
--max_seq_length 128
--output_dir ./models/ner_dp_2_gpu
--overwrite_output_dir
--do_train
--do_eval
--do_predict
--num_train_epochs 3
--per_gpu_train_batch_size 32
--logging_dir ./runs/ner_dp_2_gpu
--logging_steps 100
--evaluate_during_training
--save_steps 750
--seed 1
```

yield the following results, consistent with the ones in the README:

```
04/17/2020 16:12:30 - INFO - __main__ -     f1 = 0.8634538152610443
04/17/2020 16:12:30 - INFO - __main__ -     loss = 0.07145964812514359
04/17/2020 16:12:30 - INFO - __main__ -     precision = 0.8434379457917262
04/17/2020 16:12:30 - INFO - __main__ -     recall = 0.8844427823485415
```

Shape of F1 and eval loss:

<img width="951" alt="Screenshot 2020-04-17 at 16 38 50" src="https://user-images.githubusercontent.com/326577/79623824-4c5caa80-80ec-11ea-866c-411600b62bb1.png">

# Regression test on `run_language_modeling.py`

Reproducing the training described in the [how to train blogpost](https://huggingface.co/blog/how-to-train):

```
--train_data_file ./data/oscar.eo.txt
--eval_data_file ./data/oscar.eo.eval.txt
--evaluate_during_training
--output_dir ./models/EsperBERTo-small-v1
--overwrite_output_dir
--mlm
--config_name ./models/EsperBERTo-small
--tokenizer_name ./models/EsperBERTo-small
--do_train
--do_eval
--line_by_line
--logging_first_step
--logging_steps 10
--logging_dir ./runs/EsperBERTo
--num_train_epochs 1
--save_total_limit 2
--save_steps 2000
--per_gpu_train_batch_size 16
--seed 42
```

# Regression test on `run_multiple_choice.py`

```
--model_name_or_path distilroberta-base
--task swag
--data_dir ./data/swag
--output_dir ./models/swag_dp_2_gpu
--overwrite_output_dir
--do_train
--do_eval
--per_gpu_train_batch_size 32
--per_gpu_eval_batch_size 512
--logging_dir ./runs/swag_dp_2_gpu
--logging_steps 100
--logging_first_step
--evaluate_during_training
```

Comments:
So, thinking about it more, I've re-run a MNLI training with the previous default max_seq_length of 128 (instead of the "new" default of tokenizer.max_len – _soon to be renamed_), and training is naturally way faster with the smaller sequence length (light blue line below in relative time):

<img width="1473" alt="Screenshot 2020-04-15 at 18 47 23" src="https://user-images.githubusercontent.com/326577/79398369-8ee88080-7f4e-11ea-843f-84b59c8f3ad5.png">

So I'm thinking of reverting the default to 128. Does it make sense? Are people familiar with GLUE mostly training models on shorter sequence length? (@VictorSanh @srush @thomwolf @LysandreJik)

Or do they debug their trainings with short seq lengths, and then train with the model's max length?

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3800?src=pr&el=h1) Report
> Merging [#3800](https://codecov.io/gh/huggingface/transformers/pull/3800?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b7cf9f43d259fbad45d899c1769110aafc9f410a&el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `63.48%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3800/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3800?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3800      +/-   ##
==========================================
- Coverage   78.26%   78.10%   -0.17%     
==========================================
  Files         106      111       +5     
  Lines       17928    18459     +531     
==========================================
+ Hits        14032    14417     +385     
- Misses       3896     4042     +146     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3800?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `43.59% <43.59%> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `71.90% <52.63%> (-3.81%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `68.86% <78.26%> (+0.86%)` | :arrow_up: |
| [src/transformers/data/processors/utils.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvdXRpbHMucHk=) | `27.15% <80.00%> (+2.46%)` | :arrow_up: |
| [src/transformers/training\_args.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzLnB5) | `87.69% <80.00%> (-12.31%)` | :arrow_down: |
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `89.23% <89.23%> (ø)` | |
| [src/transformers/data/datasets/glue.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFzZXRzL2dsdWUucHk=) | `90.19% <90.19%> (ø)` | |
| [...rc/transformers/data/datasets/language\_modeling.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFzZXRzL2xhbmd1YWdlX21vZGVsaW5nLnB5) | `91.83% <91.83%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.01% <100.00%> (+0.02%)` | :arrow_up: |
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `97.01% <100.00%> (ø)` | |
| ... and [11 more](https://codecov.io/gh/huggingface/transformers/pull/3800/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3800?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3800?src=pr&el=footer). Last update [b7cf9f4...d1db901](https://codecov.io/gh/huggingface/transformers/pull/3800?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> So, thinking about it more, I've re-run a MNLI training with the previous default max_seq_length of 128 (instead of the "new" default of tokenizer.max_len – _soon to be renamed_), and training is naturally way faster with the smaller sequence length (light blue line below in relative time):
> 
> So I'm thinking of reverting the default to 128. Does it make sense? Are people familiar with GLUE mostly training models on shorter sequence length? (@VictorSanh @srush @thomwolf @LysandreJik)
> 
> Or do they debug their trainings with short seq lengths, and then train with the model's max length?

I can answer about MNLI dataset. The sequence pairs in the dataset are quite short in MNLI both in training, dev and test with similar length distributions. The vast majority of sequences are under 128 tokens. So 128 is fine for MNLI. 
For QNLI, 256 is more suitable.

The revamps look awesome!  Really looking forward to the merge and can't wait to try out the Trainer modules (nothing against argparse :joy:)

Ok, updated the PR summary above with regression tests on `run_ner.py` and `run_language_modeling.py` that show that we reproduce the documented results.

So this should be ready to merge! 🎉