Cannot load codeqwen Tokenizer
Author: zch-cc
Pull request: no
Labels: name: Core: Tokenization

### System Info

- `transformers` version: 4.40.0
- Platform: Linux-5.15.0-1026-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I cannot load model CodeQwen with AutoTokenizer
Code Qwen comes out this week and I would like to use it. https://huggingface.co/Qwen/CodeQwen1.5-7B
But I run into this error when loading the tokenizer 
```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("Qwen/CodeQwen1.5-7B-Chat")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chonghao/.local/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 862, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/chonghao/.local/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2089, in from_pretrained
    return cls._from_pretrained(
  File "/home/chonghao/.local/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2311, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/chonghao/.local/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 111, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 12564 column 3
```
According to https://github.com/ggerganov/llama.cpp/pull/6707 the difference between codeqwen and qwen1.5 is they use different tokenizer based on sentencepiece 

### Expected behavior

This issue maybe belong to a new model support but I think changing the tokenizer will work Successfully load the tokenizer

Comments:
Hi @zch-cc, thanks for reporting! 

I'm able to replicate. From bisect, seems to be coming from #30289 cc @Narsil @ArthurZucker. Happens even when upgrading tokenizers. 

This problem also affects other models, for example https://huggingface.co/sdadas/mmlw-retrieval-roberta-large
One thing the models have in common is the use of Metaspace pre-tokenizer, so this could be the source of the error.

The same issue for me of CodeQwen 1.5 7B

same issue!


Quick and dirty solution: downgrade tokenizers

requrements.txt:
```
tokenizers==0.15.2
transformers<4.40.0
```

```bash
$ pip install -r requrements.txt
```

It worked for me

Mmm there seems to be something wrong with the serialization / de-serialization. 
Let me check what I can do here

I think the authors fixed the format 🤗 It was probably not serialized with `0.15.2` nor `0.19.1`