V4.34 tokenizer incompatible with mistral
Author: edmondja
Pull request: no

Hello, I dont know how much my problem is related to https://github.com/huggingface/transformers/issues/26455 but when I install the last version of transformers I have the correct version of tokenizers disappearing, like they can not coexist : 

`
Requirement already satisfied: transformers in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (4.33.3)
Collecting transformers
  Using cached transformers-4.34.0-py3-none-any.whl (7.7 MB)
Requirement already satisfied: safetensors>=0.3.1 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from transformers) (0.3.1)
Requirement already satisfied: regex!=2019.12.17 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from transformers) (2022.10.31)
Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from transformers) (0.16.4)
Requirement already satisfied: pyyaml>=5.1 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from transformers) (6.0)
Requirement already satisfied: tqdm>=4.27 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from transformers) (4.64.1)
Requirement already satisfied: filelock in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from transformers) (3.9.0)
Requirement already satisfied: requests in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from transformers) (2.27.1)
Requirement already satisfied: packaging>=20.0 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from transformers) (23.1)
Requirement already satisfied: numpy>=1.17 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from transformers) (1.26.0)
Collecting tokenizers<0.15,>=0.14
  Using cached tokenizers-0.14.0-cp39-none-win_amd64.whl (2.2 MB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from huggingface-hub<1.0,>=0.16.4->transformers) (4.4.0)
Requirement already satisfied: fsspec in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from huggingface-hub<1.0,>=0.16.4->transformers) (2022.11.0)
Requirement already satisfied: colorama in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from tqdm>=4.27->transformers) (0.4.5)
Requirement already satisfied: idna<4,>=2.5 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from requests->transformers) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from requests->transformers) (1.26.12)
Requirement already satisfied: charset-normalizer~=2.0.0 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from requests->transformers) (2.0.4)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages (from requests->transformers) (2022.12.7)
Installing collected packages: tokenizers, transformers
  Attempting uninstall: tokenizers
    Found existing installation: tokenizers 0.13.4rc2
    Uninstalling tokenizers-0.13.4rc2:
      Successfully uninstalled tokenizers-0.13.4rc2
WARNING: Ignoring invalid distribution -rotobuf (c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages)
    WARNING: Ignoring invalid distribution -rotobuf (c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages)
ERROR: Could not install packages due to an OSError: [WinError 5] Accès refusé: 'C:\\Users\\Ext.Edmond_Jacoupeau\\Anaconda3\\envs\\py39\\Lib\\site-packages\\~%kenizers\\tokenizers.cp39-win_amd64.pyd'
Consider using the `--user` option or check the permissions.

WARNING: Ignoring invalid distribution -rotobuf (c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (c:\users\ext.edmond_jacoupeau\anaconda3\envs\py39\lib\site-packages)
`

### Reproduction

install tokenizers 0.13.4rc2, then install transformers 4.34.0

### Expected behavior

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1") should work

Comments:
Hey @edmondja, if you run the following command, do you have an issue?

```
pip install transformers==4.34.0 tokenizers==0.14.0
```


`transformers` version v4.34.0 brings a lot of significant improvements for tokenizers, but therefore requires tokenizers >= 0.14.0 

> Hey @edmondja, if you run the following command, do you have an issue?
> 
> ```
> pip install transformers==4.34.0 tokenizers==0.14.0
> ```


Yes this did the job thanks



Great :raised_hands: