T5 & mT5
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds T5v1.1 and MT5.

I was really unsure whether I should make a new model file `modeling_5_v1_1.py` or not. I finally decided (also after discussions in https://github.com/huggingface/transformers/issues/6285) that in this case, it is better to add the few new T5 features to the existing `modeling_t5.py` file.

The context is the following:
The T5 team released weights for a **T5v1.1** model: https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md. These model architecture of T5v1.1 is equivalent to T5 besides two changes:
1) input and output word embedding of decoder is not shared anymore
2) different feed forward layer is used
In addition, the **mT5** model checkpoints were also released and are fully based on T5v1_1: https://github.com/google-research/multilingual-t5#released-model-checkpoints .

Now the philosophy of the library is to create a new model class if the architecture slightly differs from a previously integrated model, but I'd argue that in this case it is better to add a new config param called "feed_forward_proj" which defines whether a "relu" (T5) or a "gated-gelu" (T5v1.1) should be used. The arguments for not creating a new model class are the following:

1) Both T5 and T5v1.1 are "officially" the same model and both belong to the same code base: https://github.com/google-research/text-to-text-transfer-transformer
2) T5v1.1 has no official paper and it's quite difficult to find a good name as discussed in https://github.com/huggingface/transformers/issues/6285 => `T5v11` is a very cryptic name IMO and `T5v2` is not ideal either.
3) One could argue that it makes sense to create a mixture of T5 and T5v1v1 by sharing input and output word embedding of the decoder but using the "gated-gelu" feed-forward layer. This design would allow a user to define a model whereas creating a new T5v11 model  file would make this impossible
4) `"feed-forward-proj"` is less of a model-specific architecture configuration than `"do_blenderbot_90_layernorm"` IMO

The obvious disadvantage is that I am adding new architecture code to an existing model to make a new model (or new model version) work, which is kinda what we did not want to do. 

It's a tough decision here I think, so I'd be very happy for your feedback on this @LysandreJik @sgugger @thomwolf . I'd also understand if you guys think we should make a new model file here.

I already uploaded `google/t5-v1_1-small` and `google/mt5-small`. If you guys are ok with this PR, I'll add the TF code, add the new model to README.md and we're good for merge.

Comments:
I'm in favor of this solution and happy you selected it :)

Our philosophy of simple to use and modify standalone code is made not to be a hard requirement but a guide for user-experience and in this case I also think the best is to adapt T5 as you do.

Big thanks for integrating mT5! I checked out your PR and there seems to be a problem with extra_ids. I guess, sentencepiece vocab already has <extra_id> tokens but T5Tokenizer goes on to add it's own.  E.g., real id of <external_id_0> is 250099 but T5Tokenizer translates it into 250199

> Big thanks for integrating mT5! I checked out your PR and there seems to be a problem with extra_ids. I guess, sentencepiece vocab already has <extra_id> tokens but T5Tokenizer goes on to add it's own. E.g., real id of <external_id_0> is 250099 but T5Tokenizer translates it into 250199

You're 100% correct - thanks a lot for letting me know! I corrected the behavior - should be good now: https://huggingface.co/google/mt5-small/tree/main

Ran the slow tests & added multiple new slow tests & checked the docs => good to merge.