Number of tokens mismatch for CodeLlama-34b-hf
Author: irenedea
Pull request: no

### System Info

- huggingface_hub version: 0.16.4
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.11.5
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /Users/irene.dea/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: osxkeychain
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.0.1
- Jinja2: 3.1.2
- Graphviz: N/A
- Pydot: N/A
- Pillow: 10.0.0
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 1.25.2
- pydantic: N/A
- aiohttp: 3.8.5
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /Users/irene.dea/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /Users/irene.dea/.cache/huggingface/assets
- HF_TOKEN_PATH: /Users/irene.dea/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False

git-commit: 05de038


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The number of tokens in the CodeLlama-34b-hf tokenizer is greater than vocab_size specified by the model config.

Additionally, when instantiating the tokenizer, the following message is output: `Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.`

To reproduce:
```
from transformers import AutoTokenizer, PretrainedConfig
model = 'codellama/CodeLlama-34b-hf'
tokenizer = AutoTokenizer.from_pretrained(model)
config = PretrainedConfig.from_pretrained(model)

assert len(tokenizer) == config.vocab_size
```
Ran with 05de038

### Expected behavior

I expect the number of tokens in my tokenizer and the vocab_size to be the same. In other words the assert in the reproduction example should pass.

Comments:
cc @ArthurZucker maybe you know what is going on here?

Hey! This model does not support `infilling`, but the correct tokenizer was not pushed to the hub. I'll do that asap.
Also your expectation that `len(tokenizer)` == `config.vocab_size`  is not necessarily correct. Some models have unused embedding (if they padded the embedding for performances for example)  while some tokenizers have holes and thus don't account for the full range of the embedding dimension. 

Ah yeah, equality is not correct but `config.vocab_size` should be greater than or equal to `len(tokenizer)`, yeah?

yep, in this specific case the 4 special tokens should not be added! 

Great, thanks!

Thanks @ArthurZucker!

Hey @ArthurZucker, I wanted to flag another issue, reproduced with the following:
```
from transformers import AutoTokenizer, CodeLlamaTokenizerFast
model = 'codellama/CodeLlama-34b-hf'
tokenizer = AutoTokenizer.from_pretrained(model, revision='81acc49ea9e2e851b1d1f33dacd763816ce7fbe4')

assert isinstance(tokenizer, CodeLlamaTokenizerFast)
print(tokenizer.encode_plus('hello')) # Errors
```
The following error occurs in CodeLlamaTokenizerFast.encode_plus (L298) because tokenizer.fill_token is None.
`TypeError: 'in <string>' requires string as left operand, not NoneType`

Yep, this needs changes in the tokenizer, will open a PR !