data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 12564 column 3
Author: cyente
Pull request: no
Labels: name: Core: Tokenization

name: bug

### Describe the bug

When using transformers==4.40.0 , some unexcepted error happend, when i load codeqwen model. transformers==4.39.1 is ok, but 4.40.0 failed.

```
Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 12564 column 3
```

### Reproduction

from transformers import AutoTokenizer, AutoModelForCausalLM
device = "cuda" # the device to load the model onto
import torch
checkpoint = ""Qwen/CodeQwen1.5-7B""

TOKENIZER = AutoTokenizer.from_pretrained(checkpoint)

### Logs

```shell
Traceback (most recent call last):
  File "/xxx/test_code.py", line 8, in <module>
    TOKENIZER = AutoTokenizer.from_pretrained(checkpoint)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxx/anaconda3/envs/py311/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 862, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxx/anaconda3/envs/py311/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2089, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/xxx/anaconda3/envs/py311/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2311, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxx/anaconda3/envs/py311/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 111, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 12564 column 3
```
```


### System info

```shell
- huggingface_hub version: 0.22.2
- Platform: Linux-4.19.91-014-kangaroo.2.10.13.5c249cdaf.x86_64-x86_64-with-glibc2.31
- Python version: 3.11.9
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /root/.cache/huggingface/token
- Has saved token ?: False
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.1.2
- Jinja2: 3.1.3
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 1.26.4
- pydantic: 2.7.0
- aiohttp: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /root/.cache/huggingface/hub
- HF_ASSETS_CACHE: /root/.cache/huggingface/assets
- HF_TOKEN_PATH: /root/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10
```


Comments:
Transferring to `transformers`'s repo as it doesn't look related to `huggingface_hub` itself (cc @huggingface/transformers-core-maintainers) 

sorry for that mistakes... Do I need to provide any more information? @ArthurZucker 

cc @ArthurZucker @Narsil 

I'm gonna need a full reproducer @cyente 

we have update codeqwen config to aviod this issue.