AUtokenizer is giving wrong result
Author: ONE-THING-9
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu121 (False)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.20
- JaxLib version: 0.4.20
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: No


### Who can help?

@ArthurZucker @younesbelkada

While using AutoTokenizer for "sarvamai/OpenHathi-7B-Hi-v0.1-Base". The tokenizer is giving the wrong output.

Tokenizer is splitting the words that are in vocab like
('▁विधायकों', 33821)

tokenizer.tokenize("विधायकों")
output
['▁', 'वि', 'धा', 'य', 'कों']

Observed this with many words : बिश्नोई , एबीवीपी......

However, it is working fine with LlamaTokenizer

https://huggingface.co/sarvamai/OpenHathi-7B-Hi-v0.1-Base

<img width="852" alt="Screenshot 2023-12-16 at 8 42 30 PM" src="https://github.com/huggingface/transformers/assets/123763769/220734ad-8ae1-4323-a1ea-a29beb2b15a2">




### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

used given code in model info page

### Expected behavior

AutoTokenizer gives the wrong output

Comments:
Hey! Thanks for opening an issue! This is not related to the actual classes but rather the way BPE works. It tries to merge individual characters based on the `merges` it has learned. But in this case `"▁", "विधायकों"` is not part of the merge pairs. While for example   `"▁","विधायक"`  is. So: 
```python 
tokenizer.tokenize("विधायक")
['▁विधायक']
```


Hey @ArthurZucker I am still getting that split thing 

tokenizer.tokenize("विधायकों'")
['▁विधा', 'य', 'कों', "'"]


![Screenshot 2023-12-20 at 8 30 08 PM](https://github.com/huggingface/transformers/assets/123763769/d9e234f6-3c1b-4e00-bf9f-418c983addc3)


Did you read my answer? 🤗 

yeah, and sorry if I am missing a very silly point, 
but how come LlamaTokenizer(which the author provided) works fine(means not splitting the words in vocab)? 

However, the author mentioned that they have trained the tokenizer using sentencePiece algorithms, why AutoTokenizer not using that algorithm or  AutoTokenizer always use BPE?

model link - https://huggingface.co/sarvamai/OpenHathi-7B-Hi-v0.1-Base

@ArthurZucker 

Okay I'll add a detail: 
```python 
tokenizer.tokenize("विधायक")
['▁विधायक']
tokenizer.tokenize("विधायकों'")
['▁विधा', 'य', 'कों', "'"]
```
and 
```python
"विधायक" == "विधायकों'"
```
we are not using the same input. 
As my answer mentions, the merges is what matters not the vocab! 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.