microsoft/deberta-v3-base init tokenizer failed
Author: fancyerii
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.3
- Platform: Linux-5.15.0-161-generic-x86_64-with-glibc2.35
- Python version: 3.12.1
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.5.1+cu121 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "microsoft/deberta-v3-base"

tokenizer = AutoTokenizer.from_pretrained(model_name)

File ...../.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py:1857, in convert_slow_tokenizer(transformer_tokenizer, from_tiktoken)
   1855     converter_class = SLOW_TO_FAST_CONVERTERS[tokenizer_class_name]
   1856     return converter_class(transformer_tokenizer).converted()
-> 1857 elif transformer_tokenizer.vocab_file.endswith("tekken.json"):
   1858     transformer_tokenizer.original_tokenizer = transformer_tokenizer
   1859     logger.info("Converting from Mistral tekken.json")

AttributeError: 'NoneType' object has no attribute 'endswith'


### Expected behavior

no bug.

Comments:
@fancyerii The code seems to work for me, no error is coming.

Yeah, I also can't reproduce this one!

I ran into this issue. The DebertaV2Tokenizer tries to fall back to fast when sentencepiece is not installed. Those who can't reproduce might have this installed already.

Quick fix:
```
pip install sentencepiece
```

yes, after I installed sentencepiece, it works. But there is a warning message:

```
.../.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py:566: UserWarning: The sent
encepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers
. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have conve
rted these unknown tokens into a sequence of byte tokens matching the original piece of text.                                                
  warnings.warn(                                                                                                                             
The tokenizer you are loading from '././models/deberta-v3-base' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Sma
ll-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_r
egex=True` flag when loading this tokenizer to fix this issue.  
```
I don't know whether it's serious.

Not sure if it's serious, but if want to be safe you can block the convert-to-fast-tokenizer step by passing the `use_fast=False` flag.

```
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
```

Those `fix-mistral-regex` errors are erroneous, we're working on it!

Actually, I think it's fixed on the latest `v5` release!