Should t5-small generate coherent text as summaries without finetuning?
Author: marton-avrios
Pull request: no
Labels: name: wontfix

I am following the summarization example and if I run `run_eval.py` for `t5-small` and `xsum` without finetuning I still get coherent, new (similar to source but not the same) and meaningful texts as summaries. The doc does not mention that it was pretrained on any kind of summarization task.

Comments:
Hi @marton-avrios can you share with me the example? Let me try them out as well and see if I found any edge cases where it is not coherent (subjectively). Can you also point out the dataset?

Will be appreciated if you can share the example reference, dataset link, or your example source code.

just go to `examples/seq2seq` follow the instructions for obtaining the XSUM  dataset and run
```
python run_eval.py t5-small xsum/val.source t5_val_generations.txt \
    --reference_path xsum/val.target \
    --score_path xsum_rouge.json \
    --task summarization \
    --n_obs 100 \
    --device cuda \
    --fp16 \
    --bs 32
```
it creates relatively coherent text in `t5_val_generations.txt` which I would not expect from a model without any finetuning.

Ahh that one. I think it's pretrained already although I'm not sure which pretraining dataset. I think your doubt is that we shoould need a bit of training iteration for different datasets to make the model good? Both are news dataset so I won't be too surprised that we don't need additional iteration. I think the XSum highlights on one sentenced and shorter summary than CNN/Daily Mail so the label is different.

Ah, so they are already finetuned versions. I thought that `t5-small` and the other `t5-*` models were only trained on denoising tasks.

I ran t5-small for the CNN/DM test dataset and the output produced are meaningful complete sentences but not close to summaries. Which is expected because they are just pre-trained on a large corpus and not fine-tuned on summarization datasets explicitly. No, they are just pretrained versions and not fine-tuned ones.

In case if you want to fine-tune them on CNN/DM or X-Sum dataset, u can run the finetune_t5.sh script maybe and use the model saved to produce outputs again. you will surely find the fine-tuned ones performing better.

I see, that explains well the observation then. When you run the t5 model, you will be warned with the following:

```
Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at t5-base and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

So I guess you still need to train them for summarization task @marton-avrios 

I also apologies for the misinformation. I thought they were pretrained on CNN/DailyMail dataset as that is the impression I get from this [doc](https://huggingface.co/transformers/task_summary.html)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.