Number of tokens mismatch for Codellama-34b-hf
Author: irenedea
Pull request: no

### System Info

transformers 4.34.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, PretrainedConfig, CodeLlamaTokenizerFast
model = 'codellama/CodeLlama-34b-hf'
tokenizer = AutoTokenizer.from_pretrained(model)
config = PretrainedConfig.from_pretrained(model)

assert len(tokenizer) <= config.vocab_size, f'Got tokenizer size {len(tokenizer)} and vocab size {config.vocab_size}'
```

### Expected behavior

len(tokenizer) is expected to be less than or equal to config.vocab_size, but we get: `AssertionError: Got tokenizer size 32004 and vocab size 32000`. 

Comments:
Hey @ArthurZucker , would you be able to help with this? I don't think https://github.com/huggingface/transformers/issues/26156  was fully resolved. Thank you!

Hey, yes for sure! 

Hey @ArthurZucker any ETA on when this might be resolved?

@dakinggg Arthur is off for a week. If someone in the community wants to open a PR to resolve this before then feel free to ping me for a review. 

FYI for anyone else who ends up here, I've worked around this by initializing the tokenizer like so:
`t = transformers.AutoTokenizer.from_pretrained('codellama/CodeLlama-7b-hf, prefix_token=None, middle_token=None, suffix_token=None, eot_token=None, fill_token=None)`

Yep that's the fix, I'm gonna push it to the hub today! 😉 Just gotta make sure this errors out properly on the FMI 

[PR](https://huggingface.co/codellama/CodeLlama-34b-hf/discussions/23), you can use the reference head for now, I just have to test Forward Compatibility and Backward compatibility 😉 
EDIT: it's not a backward compatible change, because before 4.34.0, having non fill tokens would trigger this:
```python 
----> 1 tokenizer.tokenize("Hey
sir")

File ~/miniforge3/envs/4.29/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py:344, in PreTrainedTokenizerFast.tokenize(self, text, pair, add_special_tokens, **kwargs)
    343 def tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bool = False, **kwargs) -> List[str]:
--> 344     return self.encode_plus(text=text, text_pair=pair, add_special_tokens=add_special_tokens, **kwargs).tokens()

File ~/miniforge3/envs/4.29/lib/python3.9/site-packages/transformers/models/code_llama/tokenization_code_llama_fast.py:298, in CodeLlamaTokenizerFast.encode_plus(self, text, text_pair, suffix_first, add_special_tokens, **kwargs)
    295 def encode_plus(self, text, text_pair=None, suffix_first=False, add_special_tokens=True, **kwargs):
    296     # hack to make sure the input is pre-process but outside rust
    297     text_pair = kwargs.pop("suffix", text_pair)
--> 298     if self.fill_token in text and text_pair is None:
    299         text, text_pair = text.split(self.fill_token)
    301     if text_pair is None or len(text_pair) < 1:
```
I'd rather we keep the "wrong" tokenizer length than break for other. Hope it makes sense for you and the fix is exactly what @dakinggg proposed.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.