[s2s trainer examples] a tight quality regression test
Author: stas00
Pull request: no

Continuing from https://github.com/huggingface/transformers/issues/6049 and https://github.com/huggingface/transformers/issues/8154, we are trying to build a test that will be good at regression detection in the finetune training code, quoting @sshleifer:

> try to make a fast command line test/script for summarization that meets some reasonable learning requirements (using rouge score. Try to get rouge2 above 4?) 
> For example, if you set dropout=0.95 or freeze all parameters, or set the LR too low, or mess up the special tokens logic, the test should fail. 

I have some follow up questions:

1. should I make a subset dataset on s3? or should I attempt to build a tiny one and check it in git?

2. Speed-wise would xsum be the best candidate to derive the dataset from?

3. define fast? do you mean something that would not need `@slow` - but summarization is inherently a slow task so it doesn't go well with "fast"

4. if summarization it is, which model would you recommend that is light enough to fit with not `@slow` to be downloadable from CI

5. if you want a really tight test we will need to first test that the hparams in question do lead to a failure (which will be caught) when they are lower than whatever threshold we choose, which means that ideally we will need to re-run the training multiple times in the same test to make the test really good. Which again leads to the question of how fast is fast.

Of course, one of the difficulties with tight params would be that different GPUs will produce different results, but I suppose we will tune it up as we go.

Thank you!

@sshleifer 


Comments:
(FYI, this seems less important than the OOM issue you've found separately.)

1) definitely S3. Otherwise there will be lots of pressure to have small data which could hurt learning objectives. I would try to satisfy learning objectives then make data smaller. If it gets to 1MB you could check it in, but that sounds hard!

2) Yes or `gigaword` 
Note that these two commands are roughly equivalent:

```bash
s3cmd get s3://datasets.huggingface.co/summarization/gigaword.tgz
wget https://cdn-datasets.huggingface.co/summarization/gigaword.tgz
```

3) As fast as possible while learning stuff. I would expect it to need `@slow` and take 1-5min. But if you can think of a smart way (e.g. high learning rate) to make this faster, it would be cool.

4) 
```bash 
python make_student.py facebook/bart-large-xsum student_xsum_1_1 -e 1 -d 1`
```
and upload that to S3 
+ I didn't do it for you so that you can get comfortable, this should add tokenizer files for you

5) I would lean towards running it once, manually introducing regressions locally to see if they are caught.




Thank you for the awesome answers

>     5. I would lean towards running it once, manually introducing regressions locally to see if they are caught.

OK, I guess we could make it as part of the debug code where on demand it'd pre-run below-threshold settings and ensures that the test fails and then moves on to what it really tests.


As we won't have native amp working correctly until pytorch-1.8 is going to be out I'm closing this feature for now, as it'd be a pointless effort at this point.