T5v1.1 Addition of special tokens
Author: FL33TW00D
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.0.0 --rc1
- Platform: Colab
- Python version: 3.6.9
- PyTorch version (GPU?): TESLA V4
- Tensorflow version (GPU?):  2.3.0
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 T5: @patrickvonplaten
 -->

## Information

Model I am using (Bert, XLNet ...): T5-1.1

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X ] my own modified scripts: (give details below)


```python
from transformers import T5TokenizerFast, T5ForConditionalGeneration
TOKENIZER_NAME = 't5-base'
MODEL_NAME = 'google/t5-v1_1-base'
tokenizer = T5TokenizerFast.from_pretrained(MODEL_NAME)
special_tokens_dict = {'additional_special_tokens': ['<ORG>','<PERSON>']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model = T5ForConditionalGeneration.from_pretrained('google/t5-v1_1-base', return_dict=True)
model.resize_token_embeddings(len(tokenizer))
model.to("cuda")
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)
Custom dataset
## To reproduce

Steps to reproduce the behavior:

1. Attempting to add Entity tokens to T5 1.1, upon loading from pretrained the following error occurs:
`size mismatch for lm_head.weight: copying a param with shape torch.Size([32128, 768]) from checkpoint, the shape in current model is torch.Size([32102, 768]).`

I am assuming the addition of the special tokens did not get propagated to the lm head size.

I would expect the LM Head to be resized in addition to the standard layers.

Many Thanks,
Chris


Comments:
duplicate of https://github.com/huggingface/transformers/issues/8643. 

This is indeed a big problem. I'll try to get to it this week!

Hey @FL33TW00D - actually I cannot reproduce your error....Can you try to update to the `tokenizers` version as well?

I can correctly shorten T5's embedding matrix...

Hi @patrickvonplaten,
Appreciate you looking at this. 

I suspect that in this case it's user error.
I am attempting to add the special tokens like so prior to pretraining:
```python
from transformers import T5TokenizerFast, T5ForConditionalGeneration
MODEL_NAME = 'google/t5-v1_1-base'
special_tokens = ["<ORG>",
                  "<PERSON>"]
tokenizer = T5TokenizerFast.from_pretrained('t5-base')
special_tokens_dict = {'additional_special_tokens': ['<ORG>','<PERSON>']}
num_added_tokens = tokenizer.add_special_tokens(special_tokens_dict)
print(f'ADDED TOKENS: {num_added_tokens}')
model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME)
model.resize_token_embeddings(len(tokenizer))
model.to("cuda")
```
I then pretrain the model, and save like so:
`model.save_pretrained('t5_base_test')`

It is upon model loading that I receive the error:
```
T5ForConditionalGeneration.from_pretrained('./t5_base_test')
```
```
size mismatch for lm_head.weight: copying a param with shape torch.Size([32128, 768]) from checkpoint, the shape in current model is torch.Size([32102, 768]).
```

From the config.json, it looks like the rest of the layers are being scaled to the len(tokenizer) of 32102, and only the language modelling head on the final layer remaining as 32128.

Any insight into this?

Many Thanks,
Chris

I can reproduce - will fix it! Thanks for the detailed error description 

BTW, it's recommend to always use the same model identifier for model and tokenizer, even though in this case it would not have made a difference. So:

```python
tokenizer = T5TokenizerFast.from_pretrained('google/t5-v1_1-base')
```

> 
> 
> I can reproduce - will fix it! Thanks for the detailed error description

Massive thanks for fixing this. Really appreciate it.

Will try to have it merged into master by tomorrow

> Will try to have it merged into master by tomorrow

@patrickvonplaten No worries already forked and working great! :+1: 

Hi @patrickvonplaten 

I have faced the same situation with `MT5ForConditionalGeneration` when I have reproduced the [question_generation](https://github.com/patil-suraj/question_generation) with Thai language data (I have prepared the dataset from ['xquad.th'](https://huggingface.co/datasets/viewer/?dataset=xquad&config=xquad.th) ) by @patil-suraj 

This is my error messages
```bash
>>> model = MT5ForConditionalGeneration.from_pretrained('./mt5-base-qg-hl-xquad-th-6-epochs')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sakares/transformers/src/transformers/modeling_utils.py", line 1144, in from_pretrained
    raise RuntimeError(
RuntimeError: Error(s) in loading state_dict for MT5ForConditionalGeneration:
	size mismatch for lm_head.weight: copying a param with shape torch.Size([250112, 768]) from checkpoint, 
the shape in current model is torch.Size([250102, 768]).
```

I have dive into the file src/transformers/models/mt5/modeling_mt5.py and found that MT5ForConditionalGeneration 
 just overrode the T5ForConditionalGeneration, and it should not be a problem.

Sorry to bring you here @patil-suraj . I am just curious since I have modified the script run_qg.py for MT5
and, according to this [discussion](https://github.com/huggingface/transformers/pull/8880#issuecomment-737113053),  I found the script did not have things like 
`model.resize_token_embeddings(len(tokenizer))`

My question: should I run method resize_token_embeddings before start the training model.



also facing same issue as @sakares  ..Have you solved it?

Can you guys try again on master -> this should have been fixed by now: https://github.com/huggingface/transformers/issues/9055#issuecomment-745450713

@acul3 No luck yet. But I found the alternative solution with PyTorch Lightning script ["Finetune MT5 for Question Generation in Hindi"](https://www.kaggle.com/parthplc/finetune-mt5-for-question-generation-in-hindi/) and it works as expected

 i manage to solve my problem by changing tokenizer to `google/mt5-base' instead of 't5-base'(my mistake) and install transformers from source(master) as @patrickvonplaten told

will try to look that script @sakares..thank you