[WIP] Adding Google T5 model
Author: thomwolf
Pull request: yes

Add Google T5 model:
- paper:  "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" https://arxiv.org/abs/1910.10683
- code: https://github.com/google-research/text-to-text-transfer-transformer

The original model makes heavy use of model and data parallelism to scale the training up to a 11B parameters model. It is based on Mesh TensorFlow (https://github.com/tensorflow/mesh).

We will use a simpler version of model parallelism by simply distributing the layers on several GPUs.

# Workflow for including a model from [README.md](https://github.com/huggingface/transformers/blob/master/templates/adding_a_new_model/README.md)

Here an overview of the general workflow: 

- [x] add model/configuration/tokenization classes
- [x] add conversion scripts
- [x] add tests
- [x] finalize

Let's details what should be done at each step

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the files to replace `XXX` (with various casing) with your model name
- [x] copy-past or create a simple configuration class for your model in the `configuration_...` file
- [x] copy-past or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0)
- [x] copy-past or create a tokenizer class for your model in the `tokenization_...` file

# Adding conversion scripts

Here is the workflow for the conversion scripts:

- [x] copy the conversion script (`convert_...`) from the present folder to the main folder.
- [x] edit this scipt to convert your original checkpoint weights to the current pytorch ones.

# Adding tests:

Here is the workflow for the adding tests:

- [x] copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the tests files to replace `XXX` (with various casing) with your model name
- [x] edit the tests code as needed

# Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] add import for all the relevant classes in `__init__.py`
- [x] add your configuration in `configuration_auto.py`
- [x] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
- [x] add your tokenizer in `tokenization_auto.py`
- [x] add your models and tokenizer to `pipeline.py`
- [x] add a link to your conversion script in the main conversion utility (currently in `__main__` but will be moved to the `commands` subfolder in the near future)
- [x] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
- [x] add a mention of your model in the doc: `README.md` and the documentation it-self at `docs/source/pretrained_models.rst`.
- [x] upload the pretrained weigths, configurations and vocabulary files.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1739?src=pr&el=h1) Report
> Merging [#1739](https://codecov.io/gh/huggingface/transformers/pull/1739?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7296f1010b6faaf3b1fb409bc5a9ebadcea51973?src=pr&el=desc) will **increase** coverage by `0.58%`.
> The diff coverage is `88.79%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1739/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1739?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1739      +/-   ##
==========================================
+ Coverage   80.35%   80.93%   +0.58%     
==========================================
  Files         114      121       +7     
  Lines       17095    18329    +1234     
==========================================
+ Hits        13736    14835    +1099     
- Misses       3359     3494     +135
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1739?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2VuY29kZXJfZGVjb2Rlci5weQ==) | `32.3% <0%> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_tf\_transfo\_xl\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX3RyYW5zZm9feGxfdGVzdC5weQ==) | `94% <100%> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_transfo\_xl\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RyYW5zZm9feGxfdGVzdC5weQ==) | `94.54% <100%> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `97.41% <100%> (+0.49%)` | :arrow_up: |
| [transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3B5dG9yY2hfdXRpbHMucHk=) | `95.2% <100%> (+0.06%)` | :arrow_up: |
| [transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3V0aWxzLnB5) | `92.3% <100%> (-0.05%)` | :arrow_down: |
| [transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2ZpbGVfdXRpbHMucHk=) | `71.29% <100%> (+0.27%)` | :arrow_up: |
| [transformers/modeling\_tf\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3hsbS5weQ==) | `90.47% <100%> (ø)` | :arrow_up: |
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `91.46% <100%> (ø)` | :arrow_up: |
| [transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fYXV0by5weQ==) | `44.18% <25%> (-0.82%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/huggingface/transformers/pull/1739/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1739?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1739?src=pr&el=footer). Last update [7296f10...cbb368c](https://codecov.io/gh/huggingface/transformers/pull/1739?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


How much work is there remaining on T5? Tried to check out this branch but it looks like the  weights and SentencePiece models aren't uploaded yet.

I would like to know if this implementation of T5 is on hold because of priority switching to different tasks or maybe there was found an fundamental obstacle, which causes impossibility to move T5 to transformers library and this pull request will not be finished?

Thanks for great work!

No fundamental obstacle, it was just stuck in the second place of my TODO list for a while.

This model will need to be fine-tuned on a downstream task before usage though because the combination of mesh-tensorflow ops and bfloat16 seems to cause a bigger discrepancy with PyTorch's float32 ops than other architectures. 

Ok merging

Good work. Will you be releasing finetuning code soon?