❓ T5 pre-training dataset
Author: pietrolesci
Pull request: no

Hi there,

I am working with the T5 model and, before proceeding, I need to account for the actual datasets that it has been pre-trained on.

In the [T5v1.1 doc page](https://huggingface.co/docs/transformers/model_doc/t5v1.1) it says "Note: T5 Version 1.1 was only pre-trained on [C4](https://huggingface.co/datasets/c4) excluding any supervised training." hinting at the fact that T5 checkpoint is pre-trained on supervised tasks too. This is confirmed by its ability to work out-of-the-box really well on MNLI. However, in the [model page](https://huggingface.co/t5-small) on the hub it reports only C4 as a pre-training dataset.

- Is there an error in the T5 model description page on the hub?

- (general question) is there any source (I can cite) that specifies which data has been seen by the checkpoint available on the hub?

Thanks a lot in advance for your help.


Comments:
I recently ran into the same issue and can confirm your findings. I believe [this](https://huggingface.co/t5-small) version of T5 is fine-tuned in a multitask setup which you can learn more about from the [T5 paper](https://arxiv.org/abs/1910.10683). This is why it works super well on many downstream tasks. However if you are looking for a T5 model that is only pre-trained on C4 then I suggest this [v1.1 checkpoint](https://huggingface.co/google/t5-v1_1-base)

I also think that it would be great to add more info about how the multitask model is fine-tuned to avoid this confusion.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Sorry to answer so late here. 
Yes the following checkpoints:

- https://huggingface.co/t5-small
- https://huggingface.co/t5-base
- https://huggingface.co/t5-large
- https://huggingface.co/t5-3b
- https://huggingface.co/t5-11b

have **not** only been pretrained on C4, but if I recall correctly on all datasests of the downstream tasks as described in the section `2.3 Downstream Tasks` of the paper: https://arxiv.org/pdf/1910.10683.pdf as part of the multi-task pretraining strategy as well. Happy to update the model cards! 

@craffel - do you have a list of all datasets used for the pretraining of the first versions of T5 (the ones linked above) by any chance? 

Hey, yes, it's the datasets listed in 2.3. Note that we also train on DPR (which is not a dataset we evaluate on) and we don't train on WNLI (though we do evaluate on it).

Thanks @craffel - documenting this on the model cards :-)

Thank you very much @patrickvonplaten and @craffel for acting on this issue. Super appreciated!

Updated all of them

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.