deberta-v3 has 100 more vocabs than its tokenizer
Author: wenmin-wu
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.1
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.8.13
- Huggingface_hub version: 0.10.0
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): 2.8.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

Hi @LysandreJik @SaulLu , I think this issue needs both of you to help or confirm:



### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```Python
model_type = "microsoft/deberta-v3-base"
tokenizer = AutoTokenizer.from_pretrained(model_type)
print(tokenizer.vocab_size) # output: 128000
print(len(tokenizer.vocab)) # output: 128001, the extra one is padding?

config = AutoConfig.from_pretrained(model_type)
print(config.vocab_size) # output: 128100
model = AutoModel.from_pretrained(model_type, config=config)
print(print(len(model.embeddings.word_embeddings.weight)) # 128100, which is consistent with the config
```

### Expected behavior

The deberta model should have the same vocab_size as its tokenizer.

Comments:
Maybe of interest to @ArthurZucker as well

> Maybe of interest to @ArthurZucker as well

Thanks Lysandre!

Hi @wenmin-wu, 

It seems that the difference between the size of the tokenizer and the size of the `word_embeddings` matrix is voluntary on the part of the deberta-v3's authors as you can see in the issue https://github.com/microsoft/DeBERTa/issues/103.

By experience, it can happen that the size of `word_embeddings` is bigger than the total number of known tokens, for constraints on the size of the matrix (which should be a multiple of a certain number for example) or by precautions because the shape of the model was decided before having the final form of the tokenizer or to have "available" tokens which could be added later (with a little fine-tuning anyway).

Hi @SaulLu , thanks for your explanation. I'm also confused with the tokenizer:
1. The tokenizer doesn't directly return the index of that token, e.g.
```Python
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base", use_fast=True)
tokenizer("test")
# outputs: {'input_ids': [1, 1010, 2], 'token_type_ids': [0, 0, 0], 'attention_mask': [1, 1, 1]}
tokenizer.vocab["test"]
# outputs: 9982
# 9982 != 1010
```
2. How does the tokenizer handle the space before/after the special character? e.g.:
```Python
tokenizer("you (are) nice", return_attention_mask=False, return_token_type_ids=False)
# outputs:  {'input_ids': [1, 274, 287, 6614, 285, 1085, 2]}
tokenizer("you(are)nice", return_attention_mask=False, return_token_type_ids=False)
# outputs: {'input_ids': [1, 274, 555, 6614, 285, 22184, 2]}
```
seems the difference is ` (` -> `287` and `(` -> `555`, `) ` -> `1085` and `)` -> `22184`
but actually, their decoded value is the same (special character without space):
```Python
tokenizer.decode(287) == tokenizer.decode(555), tokenizer.decode(1085) == tokenizer.decode(22184)
# Outputs: True, True
```

Hi @wenmin-wu,

> The tokenizer doesn't directly return the index of that token, [...]

By default, the `"microsoft/deberta-v3-base"` model adds a prefix space. You can see it this by running (knowing that the `▁` symbol corresponds to a space):
```python
print(tokenizer.convert_ids_to_tokens(tokenizer("test").input_ids))
# outputs: ['[CLS]', '▁test', '[SEP]']
```

> How does the tokenizer handle the space before/after the special character?

To see the token to which each id corresponds I advise you to use the `convert_ids_to_tokens` method instead of `decode`. On the example you shared, you will see that spaces are well included in the following token when they exist:
```python
print(tokenizer.convert_ids_to_tokens(tokenizer("you (are) nice", return_attention_mask=False, return_token_type_ids=False).input_ids))
# outputs: ['[CLS]', '▁you', '▁(', 'are', ')', '▁nice', '[SEP]']
print(tokenizer.convert_ids_to_tokens(tokenizer("you(are)nice", return_attention_mask=False, return_token_type_ids=False).input_ids))
# outputs: ['[CLS]', '▁you', '(', 'are', ')', 'nice', '[SEP]']
```

For information, the `decode` method is a method that does its best to reconstitute a text from a sequence of token ids produced by a generative model. This task being very complicated I suggest you try to use this method only when it is really necessary (i.e. only for a sequence of token ids produced by a generative model).

I hope this helped you

@SaulLu Got it, thanks a lot for your detailed explanation

I'm so glad I could help you! I'm closing this issue as I feel like it answers all your questions. :hugs: 

Yep @SaulLu, your answers helped me a lot with the [Feedback Prize - English Language Learning](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/leaderboard) Kaggle competition. I'm in the gold zone now. Many thanks!

@SaulLu Got another improvement again. I'm a Prize Contender now! Many thanks!