Slow Integration Test for examples/seq2seq/finetune.py
Author: sshleifer
Pull request: no
Labels: name: Help wanted

Train `sshleifer/student-xsum-12-3` for N batches on xsum data.
Ensure that loss goes down. and is below ~2.
This test should ONLY run on GPU and takes between 30s and 3 mins.




python code to fetch xsum data:

```
import wget
import tarfile
wget.download('https://s3.amazonaws.com/datasets.huggingface.co/summarization/xsum.tar.gz')
tarball = tarfile.open('xsum.tar.gz')
tarball.extractall()
data_dir='xsum'
```
cc @williamFalcon 

Comments:
ok cool

I will work on this.

@sshleifer, is it supposed to be w/o a tokenizer?

```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sshleifer/student_xsum_12_3")
```

```
OSError: Model name 'sshleifer/student_xsum_12_3' was not found in tokenizers model name list (facebook/bart-base, facebook/bart-large, facebook/bart-large-mnli, facebook/bart-large-cnn, facebook/bart-large-xsum, yjernite/bart_eli5). We assumed 'sshleifer/student_xsum_12_3' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
```
It's broken online too (Compute button):
https://huggingface.co/sshleifer/student_xsum_12_3?text=My+name+is+Thomas+and+my+main

### Tokenizer issue
@stas00 No that's just me being lazy, fixed that model, let me know if you need others.

separately: Can you write to S3?
I just ran 
```bash
cp_bart_tok () {
	export ss=s3://models.huggingface.co/bert/sshleifer
	aw3 cp $ss/distilbart-xsum-1-1/merges.txt $1
	aw3 cp $ss/distilbart-xsum-1-1/tokenizer_config.json $1
	aw3 cp $ss/distilbart-xsum-1-1/vocab.json $1
}
cp_bart_tok $ss/student_xsum_12_1/
```
so easy for me to copy the bart tokenizer to other places.


### Status of this issue
this issue is moved forward for **translation** with `examples/seq2seq/test_bash_script.py`
It's run by  `.github/self-scheduled.yml`.


There are many possible axes for improvement: 
- testing summarization
- may not need to set `do_predict=False` [here]:
(https://github.com/huggingface/transformers/blob/c69ea5efc4eac65b183e8d07b1bf91d20bbe0c8c/examples/seq2seq/test_bash_script.py#L77). I made a PL issue where do_predict was breaking in fp16, but then I turned off fp16 here.
-  we could use a larger model like `sshleifer/student_mbart_en_ro_1_1/` and actually learn something (and wait longer).
- we could run add a new github workflow against torch 1.6 (add a new `.github/next_torch_version.yml`)
- understand current [failures](https://github.com/huggingface/transformers/runs/910644181?check_suite_focus=true)

Thanks so much for your help and let me know where/how I can support!


> @stas00 No that's just me being lazy, fixed that model, let me know if you need others.

The problem is still there - you can quickly test [here](https://huggingface.co/sshleifer/student_xsum_12_3?text=My+name+is+Thomas+and+my+main)

> separately: Can you write to S3?

I don't think I can - at least nobody gave me permissions to do so.


> ### Status of this issue
> this issue is moved forward for translation with examples/seq2seq/test_bash_script.py
> [...]
>  Thanks so much for your help and let me know where/how I can support!

Let me study it first and I will ask follow up questions once I did so.

I'm learning this library and while I have spare resources I'd be happy to help fixing problems - feel free to delegate some issue to me, so I won't need to dig through past issues looking for what I could work on.

I am not sure github issue comments is the most efficient way to collaborate - perhaps you're on skype/email/some IM? My email is stas@stason.org.

Closing this. Will make new issues that are not started.