Different vocab_size between model and tokenizer of mT5
Author: cih9088
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: ubuntu 18.04
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1
### Who can help
@patrickvonplaten
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoModelForSeq2SeqLM
from transformers import AutoTokenizer

mt5s = ['google/mt5-base', 'google/mt5-small', 'google/mt5-large', 'google/mt5-xl', 'google/mt5-xxl']

for mt5 in mt5s:
    model = AutoModelForSeq2SeqLM.from_pretrained(mt5)
    tokenizer = AutoTokenizer.from_pretrained(mt5)

    print()
    print(mt5)
    print(f"tokenizer vocab: {tokenizer.vocab_size}, model vocab: {model.config.vocab_size}")
```
This is problematic in case when one addes some (special) tokens to tokenizer and resizes the token embedding of the model with `model.resize_token_embedding(len(tokenizer))`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
vocab_size for model and tokenizer should be the same?
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Hello! This is a duplicate of https://github.com/huggingface/transformers/issues/4875, https://github.com/huggingface/transformers/issues/10144 and https://github.com/huggingface/transformers/issues/9247

@patrickvonplaten, maybe we could do something about this in the docs? In the docs we recommend doing this:
```py
model.resize_token_embedding(len(tokenizer))
```
but this is unfortunately false for T5!

@LysandreJik @cih9088 , actually I think doing: 

```python
model.resize_token_embedding(len(tokenizer))
```

is fine -> it shouldn't throw an error & logically it should also be correct...Can you try it out?


```python
model.resize_token_embedding(len(tokenizer))
```
This works perfectly fine but here is the thing.

One might add the `model.resize_token_embedding(len(tokenizer))` in their code and use other configuration packages such as `hydra` from Facebook to train models with additional tokens or without them dynamically at runtime.
He would naturally think that `vocab_size` of tokenizer (no tokens added) and `vocab_size` of model are the same because other models are.
Eventually, he fine-tunes the `google/mt5-base` model without added tokens but because of `model.resize_token_embedding(len(tokenizer))`, model he will fine-tune is not the same with `google/mt5-base`.
After training, he wants to load the trained model to test but the model complains about inconsistent embedding size between a loaded model which is `google/mt5-base`, and the trained model which has a smaller size of token embedding.

Of course, we could resize token embedding before loading model, but what matters is inconsistency with other models I think.
I reckon that people would not very much care about how the dictionary is composed in tokenizer. Maybe we add some dummy tokens to the tokenizer to keep consistency with other huggingface models or add documentation about it (I could not find any).

What do you think?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.


> Hello! This is a duplicate of #4875, #10144 and #9247
> 
> @patrickvonplaten, maybe we could do something about this in the docs? In the docs we recommend doing this:
> 
> ```python
> model.resize_token_embedding(len(tokenizer))
> ```
> 
> but this is unfortunately false for T5!

What is the correct way to resize_token_embedding for T5/mT5?