[examples run_summarization.py] t5 worse score w/ --source_prefix "summarize: " than w/o
Author: stas00
Pull request: no

I don't think the latest incarnation of summarization examples works for t5.

I'm lost with all the proposed let's-not-do-anything special for t5, except as you will see from numbers something isn't right:

With the latest master:
```
python examples/seq2seq/run_summarization.py --model_name_or_path t5-small --do_train --do_eval \
--dataset_name xsum --output_dir /tmp/tst-summarization --per_device_train_batch_size=4 \
--per_device_eval_batch_size=4 --overwrite_output_dir --predict_with_generate --max_train_samples 50 \
--max_val_samples 50

***** eval metrics *****
  epoch                     =     3.0
  eval_gen_len              =   60.14
  eval_loss                 =  3.3003
  eval_rouge1               = 19.3055
  eval_rouge2               =  2.4192
  eval_rougeL               =  13.931
  eval_rougeLsum            = 16.3446
  eval_runtime              =  6.2317
  eval_samples              =      50
  eval_samples_per_second   =   8.023
```

Then let's add the required `--source_prefix "summarize: "`

```
python examples/seq2seq/run_summarization.py --model_name_or_path t5-small --do_train --do_eval \
--dataset_name xsum --output_dir /tmp/tst-summarization --per_device_train_batch_size=4 \
--per_device_eval_batch_size=4 --overwrite_output_dir --predict_with_generate --max_train_samples 50 \
--max_val_samples 50 --source_prefix "summarize: "

***** eval metrics *****
  epoch                     =     3.0
  eval_gen_len              =   52.94
  eval_loss                 =  3.3734
  eval_rouge1               = 18.7997
  eval_rouge2               =  2.2857
  eval_rougeL               = 13.4997
  eval_rougeLsum            = 14.7778
  eval_runtime              =  5.2697
  eval_samples              =      50
  eval_samples_per_second   =   9.488
```

As you can see the scores are worse than w/o  `--source_prefix "summarize: "` and it should be in reverse.

Where are we adding `task_specific_params`:
```
    "summarization": { "early_stopping": true, "length_penalty": 2.0, "max_length": 200,
    "min_length": 30, "no_repeat_ngram_size": 3, "num_beams": 4, "prefix": "summarize: " },
```
so that the model knows to do the right thing? 

**edit**: found where this was last discussed: https://github.com/huggingface/transformers/pull/10133#issuecomment-778071812 So should `README.md` just say that currently `run_summarization.py` cannot be used for T5 models and then find another summarization model instead.

Of course, a lot of these repetitive breakages would have been avoided if we had quality-measuring tests for examples - perhaps when the dust settles around the examples we could have some of those added.

@sgugger 


Comments:
`--max_train_samples 50` is a tiny sample, I am not surprised that the model doesn't learn anything here. Note that `t5-small` was pretrained on CNN/Dailymail and WMT, but **not** on XSum. So, it makes sense that one gets reasonable results when fine-tuning the model on just 50 translation samples of WMT16 because the model has already seen the whole training data in pretraining. However, the model has never seen XSum in pretraining, so fine-tuning on 50 samples will get us nowhere here I think. We could try to switch to `CNN/Dailymail`. I have fine-tuned the model on the whole corpus for CNN/Dailymail and have gotten good results. In the paper, it was reported that with `t5-small` a ROUGE-2 score of 19.56 can be achieved on CNN/Dailymail. So we should get something like 17 or 18 ROUGE-2 for full fine-tuning. 

Also, IMO for such low ROUGE number we cannot really say that  "no prefix" works better than "with prefix" because both cases don't work well at all. 

Let's just try it with CNN/Dailymail instead and see what we get. Maybe first with just very few samples & if this doesn't work then let's run one full fine-tuning.

Definitely a jackpot on the example using a new dataset and too short of training: might be a good idea to add some of your notes to the README as well.

New stats, this time on `--dataset_name cnn_dailymail --dataset_config "3.0.0"`

```
# w/o --source_prefix "summarize: "

python examples/seq2seq/run_summarization.py --model_name_or_path t5-small --do_train --do_eval --dataset_name cnn_dailymail --dataset_config "3.0.0" --output_dir /tmp/tst-summarization --per_device_train_batch_size=4 --per_device_eval_batch_size=4 --overwrite_output_dir --predict_with_generate --max_train_samples 50 --max_val_samples 50

***** eval metrics *****
  epoch                     =     3.0
  eval_gen_len              =   65.24
  eval_loss                 =  2.3276
  eval_rouge1               = 25.4707
  eval_rouge2               =  7.2334
  eval_rougeL               = 18.3807
  eval_rougeLsum            = 23.2505
  eval_runtime              =  6.4841
  eval_samples              =      50
  eval_samples_per_second   =   7.711

```

```
# w/ --source_prefix "summarize: "

python examples/seq2seq/run_summarization.py --model_name_or_path t5-small --do_train --do_eval --dataset_name cnn_dailymail --dataset_config "3.0.0" --output_dir /tmp/tst-summarization --per_device_train_batch_size=4 --per_device_eval_batch_size=4 --overwrite_output_dir --predict_with_generate --max_train_samples 50 --max_val_samples 50 --source_prefix "summarize: "

***** eval metrics *****
  epoch                     =     3.0
  eval_gen_len              =   62.38
  eval_loss                 =  2.3243
  eval_rouge1               = 30.0675
  eval_rouge2               = 10.2052
  eval_rougeL               =  22.154
  eval_rougeLsum            = 27.2161
  eval_runtime              =  6.0876
  eval_samples              =      50
  eval_samples_per_second   =   8.213
```

This is much better. I will try a longer train sequence next.

Pretrained with 5000 samples the score goes up nicely, this is 1/100th of the full dataset.

```
***** eval metrics *****
  epoch                     =     3.0
  eval_gen_len              =   61.66
  eval_loss                 =  2.0773
  eval_rouge1               =  30.174
  eval_rouge2               = 12.0182
  eval_rougeL               = 23.5012
  eval_rougeLsum            = 27.3718
  eval_runtime              =  5.9836
  eval_samples              =      50
  eval_samples_per_second   =   8.356
```

So I updated https://github.com/huggingface/transformers/pull/10734 with the recommendation you made @patrickvonplaten.

Closing this.