DataCollatorForLanguageModeling using the wrong symbol for masking
Author: Charlie-83
Pull request: no

## Environment info

- `transformers` version: 4.17.0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.7
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

- Tokenizers: @SaulLu
- Trainer: @sgugger

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
import torch
import tempfile
import os
from transformers import BertTokenizer, DataCollatorForLanguageModeling

AMINO_ACIDS_WITH_ALL_ADDITIONAL = "RHKDESTNQCUGPAVILMFYW$.?|*"
IDX_AA = {aa: i for i, aa in enumerate(AMINO_ACIDS_WITH_ALL_ADDITIONAL)}
torch.manual_seed(146)

class myDS(torch.utils.data.Dataset):
    def __init__(self, seqs, tok):
        self.seqs = seqs
        self.tok = tok
        
    def __getitem__(self,i: int):
        return self.tok.encode(self.seqs[i])
    
    def __len__(self):
        return len(self.seqs)


data = ['C A S S L A Q G L N E Q F']
with tempfile.TemporaryDirectory() as tempdir:
    path = os.path.join(tempdir, "vocab.txt")
    with open(path, "w") as f:
        for v in IDX_AA:
            f.write(v + "\n")
                
    tok = BertTokenizer(
            path,
            do_lower_case=False,
            do_basic_tokenize=True,
            tokenize_chinese_chars=False,
            pad_token="$",
            mask_token=".",
            unk_token="?",
            sep_token="|",
            cls_token="*",
            model_max_len=16,
            padding_side="right",
        )

data_collator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=True, mlm_probability=0.15)
train_dataset = myDS(data, tok)
data_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, collate_fn=data_collator)
for i in data_loader:
    for j in tok.batch_decode(i['input_ids']):
        print(j)
```

## Expected behavior

Should mask tokens with "." but sometimes uses other symbols. In the above code, the output is "* C A S S L A Q * L N E Q F |" but should be something like "* C A S S L A Q . L N E Q F |". This issue occurs rarely and randomly (146 is the first seed where it occurs) since the masking process is random. 


Comments:
Hi @KangarooChief ,

It's indeed the expected behavior :slightly_smiling_face:  The data collator for language modeling does not only replace with a mask. You can see the logic of the `DataCollatorForLanguageModeling` here:
https://github.com/huggingface/transformers/blob/9a24b97b7f304fa1ceaaeba031241293921b69d3/src/transformers/data/data_collator.py#L768-L778

As you can see, 10% of the replacements made are with a random word (including `*`). :slightly_smiling_face: 

Thanks