Add script to convert T5X T5 (v1.0 and v1.1) checkpoints to PyTorch
Author: bastings
Pull request: yes

# What does this PR do?

Adds a script that can convert Google T5X (Flax) T5 and T5-v1.1 checkpoints into PyTorch checkpoints.
This allows users to convert non-standard checkpoints that have been trained with T5X and use them with the Transformers library in PyTorch.

Usage:

- In case you don't have `gsutil`, install according to https://cloud.google.com/storage/docs/gsutil_install
- Native T5X checkpoints are at https://github.com/google-research/t5x/blob/main/docs/models.md#t5-11-checkpoints. Example:
`gsutil -m cp -r gs://t5-data/pretrained_models/t5x/t5_1_1_small $HOME/`
- Create a corresponding `config.json` for the downloaded checkpoint. Often one already exists, e.g. here we can use https://huggingface.co/google/t5-v1_1-small/blob/main/config.json
- Finally `python3 convert_t5x_checkpoint_to_pytorch.py --t5x_checkpoint_path=$HOME/t5_1_1_small --config_file=config.json --pytorch_dump_path=$HOME/t5_1_1_small_pt`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Discussed with @thomwolf .
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests? The code is tested but not part of this PR, since the test requires manually downloading the T5X checkpoints from a cloud bucket.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten 
@sanchit-gandhi 
@ArthurZucker 
@younesbelkada

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I could use some clarification on the following: I'm missing a configuration option for T5 for the 1.0/original T5 checkpoints to have an `lm_head` that shares parameters with the token embeddings.

Currently there is `T5Model` (which returns hidden states) and `T5ForConditionalGeneration` (which returns logits, used for T5 v1.1 models among others). The latter assumes there is an `lm_head` layer, but for the 1.0 checkpoints there is no such thing, it reuses the embedding matrix to map to the vocab space.

Hey @bastings, when there is no `lm_head` you have to set the `tie_word_embeddings` to `True` 

I added the instructions to the top docstring. Maybe it's ready? :-)

A last nit and we can merge! Thanks a lot for bearing with me 😄 

Thanks! Committed your suggestion :)

Once the quality tests are green (requires `make fixup`) we can merge!

Oh looks like the suggestion made it fail ;)

Ah, sorry then ahha, I guess the ` make style`will correct that 😅 

> Ah, sorry then ahha, I guess the ` make style`will correct that 😅

Fixed! :)