discrepancy between the Huggingface T5Tokenizer and the original T5tokenizer
Author: dorost1234
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `Transformers` version: 4.3.2
- Platform: Linux
- Python version: 3.7
- PyTorch version (GPU?): yes 
- Tensorflow version (GPU?): - 
- Using GPU in script?: no 
- Using distributed or parallel set-up in script?: no 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

- tokenizers: @n1t0, @LysandreJik

## Information

Model I am using T5Tokenizer, and I adapted the code of  run_mlm.py [1] to use it with T5 tokenizer, when I run the code I am getting 

```
This tokenizer does not have a mask token which is necessary for masked language modeling. "
ValueError: This tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.
```

I checked the error and this is because:  tokenizer.mask_token  is None for T5Tokenizer, checking T5 paper, they use a masked language modeling with their seq2seq objective as the pretraining objective, so they must have trained a masked token as their paper says, could you give me some insight why masked token does not exist in huggingface implementation of T5Tokenizer and how I can correct this to be able to run run_mlm codes ? thank you


[1] https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py


## To reproduce

```
tokenizer = AutoTokenizer.from_pretrained("t5-small")
print(tokenizer.mask_token) => this is None
```

## Expected behavior

The masked token as per T5 paper should exist in T5Tokenizer.

Comments:
Hi, 

T5 is an encoder-decoder Transformer. The `run_mlm.py` script can only be used for encoder-only models, such as BERT, RoBERTa, DeBERTa, etc. 

Besides this, T5 does not use the regular [MASK] token as BERT. Rather than masked language modeling, T5 is pre-trained on "unsupervised denoising training". This is explained [here](https://huggingface.co/transformers/model_doc/t5.html#training).

Hi @NielsRogge thanks, but I have checked the paper of T5 and this seems to be a unique token:

"We consider two strategies to achieve this: First, instead of replacing each corrupted token with a mask token, we replace
the entirety of each consecutive span of corrupted tokens with a unique mask token."

As for using run_mlm.py script, I do not think T5 model can be an issue as if we could add T5ForConditionalGeneration it could work to me in run_mlm.py out of the box.

Is there any place I could look see how to create datasets the way you mentioned to do T5 pretraining with huggingface codes? thanks


Hi 
@patrickvonplaten, @patil-suraj could you give me some advice how to do T5 pretraining with denoising objective? thanks 

Here is an old issue on this subject: https://github.com/huggingface/transformers/issues/5079

Also @NielsRogge is correct - T5 replaces each span of tokens with a unique mask token -> the so-called sentinel tokens.
Currently, there is sadly no script showcasing pertaining for T5. Maybe you have some luck when asking this question on the [forum](https://discuss.huggingface.co/)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> Hi @NielsRogge thanks, but I have checked the paper of T5 and this seems to be a unique token:
> 
> "We consider two strategies to achieve this: First, instead of replacing each corrupted token with a mask token, we replace
> the entirety of each consecutive span of corrupted tokens with a unique mask token."
> 
> As for using run_mlm.py script, I do not think T5 model can be an issue as if we could add T5ForConditionalGeneration it could work to me in run_mlm.py out of the box.
> 
> Is there any place I could look see how to create datasets the way you mentioned to do T5 pretraining with huggingface codes? thanks

HI, Did you successfully run the Huggingface T5 pretraining? Can you give me some advice?