WhitespaceSplit not working
Author: pradeepdev-1995
Pull request: no

### System Info

torch==2.0.1
transformers==4.37.1
tokenizers==0.15.1
Python 3.8.16

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to split a sentence by subword manner and word by word manner using WhitespaceSplit
```python
  from transformers import  AutoTokenizer
  tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", trust_remote_code=True,
                                          use_fast=False)
  tokenizer.pad_token = tokenizer.eos_token
  tokenizer.padding_side = "right"
  sentence = "Transformers tokenization testing"
  tokenized_sentence = tokenizer.tokenize(sentence)
   print("without WhitespaceSplit")
   print(tokenized_sentence)
  
  from tokenizers.pre_tokenizers import WhitespaceSplit
  tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", trust_remote_code=True,
                                        pretokenizer=WhitespaceSplit(),
                                        use_fast=False)
  tokenizer.pad_token = tokenizer.eos_token
  tokenizer.padding_side = "right"
  tokenized_sentence = tokenizer.tokenize(sentence)
 print("with WhitespaceSplit ")
  print(tokenized_sentence)
```
in both cases i am getting the same splitted data as below
```
without WhitespaceSplit 
['▁Trans', 'form', 'ers', '▁token', 'ization', '▁testing']
with WhitespaceSplit 
['▁Trans', 'form', 'ers', '▁token', 'ization', '▁testing']
```

### Expected behavior

In WhitespaceSplit ,the sentence should split in word by word such as
```
["Transformers", "tokenization", "testing"]
```

Comments:
cc @ArthurZucker - possibly an issue for the tokenizers library?

I am not sure I have ever seen the `pretokenizer` argument 😅 
You should do `tokenizer._tokenizer.pre_tokenizer = Whitespace()`

@ArthurZucker 
can you show me the full code, please?

I am getting this error
```
ModuleNotFoundError: No module named 'tokenizer._tokenizer'
```

```diff
  from transformers import  AutoTokenizer
  tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", trust_remote_code=True,
                                          use_fast=False)
  tokenizer.pad_token = tokenizer.eos_token
  tokenizer.padding_side = "right"
  sentence = "Transformers tokenization testing"
  tokenized_sentence = tokenizer.tokenize(sentence)
   print("without WhitespaceSplit")
   print(tokenized_sentence)
  
  from tokenizers.pre_tokenizers import WhitespaceSplit
- tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", trust_remote_code=True,pretokenizer=WhitespaceSplit(), use_fast=False)
+ tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", trust_remote_code=True,use_fast=True)
+ tokenizer._tokenizer.pre_tokenizer = WhitespaceSplit()
  tokenizer.pad_token = tokenizer.eos_token
  tokenizer.padding_side = "right"
  tokenized_sentence = tokenizer.tokenize(sentence)
 print("with WhitespaceSplit ")
  print(tokenized_sentence)
```  
🤗 

@ArthurZucker 

```
AttributeError: 'LlamaTokenizer' object has no attribute '_tokenizer'
```
Since I am using mistral model which is Llama architecture based i think.
any way to solve this?

If you are using a slow tokenizer this cannot work. Updated the script to use fast

@ArthurZucker 
Not working
![Screenshot from 2024-01-30 16-13-39](https://github.com/huggingface/transformers/assets/41164884/e043c52d-fd46-4195-b7e6-f43dd8992b71)


That is because you still have a normalizer:
```python 
from tokenizers import normalizers
tokenizer._tokenizer.normalizer = normalizers.Sequence([])
```


Not worked as expected
@ArthurZucker 
![Screenshot from 2024-01-30 19-09-53](https://github.com/huggingface/transformers/assets/41164884/31266311-06e3-4318-a36f-779211c93375)


I am not sure I understand, if you follow the logic of the tokenization [see the doc here](https://huggingface.co/docs/tokenizers/pipeline#pretokenization) the pretokenization will split, but the final tokens are not given by the pre tokenized text:

> The role of the model is to split your “words” into tokens, using the rules it has learned. It’s also responsible for mapping those tokens to their corresponding IDs in the vocabulary of the model.

thus `Transformers` is split into it's known tokens

@ArthurZucker 

So splitting the sentence **"Transformers tokenization testing"** into **["Transformers", "tokenization", "testing"]** is not possible using **mistralai/Mistral-7B-Instruct-v0.2**  tokenizer?

You can just use `sentence.split(" ")` but no, the token `Transformers` is not part of the vocab so it's not a token 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.