Fix PretrainedTokenizerFast check => Fix PretrainedTokenizerFast Save
Author: CL-ModelCloud
Pull request: yes

FIX #35832 
@Qubitium 

Comments:
@ArthurZucker @itazap Pretty sure the PR fixes a class string compare bug vs correct `isinstance` check.  It fixes the issue reported here: https://github.com/huggingface/transformers/issues/35832

The existing check for class is instance of `PretrainedTokenizerFast` will never be met since it does a simple string class name check vs base class inheritance check. 

The only problem is we had to inject a code level import due to circular import. 



@ArthurZucker Please re-review. We are now using different, cleaner method to fix this bug and we do consider this a bug, and a rather severe bug as well: as the loading of re-saved tokenizer has fundamentally changed vs original. 

```py
        # Remove the Fast at the end if we can save the slow tokenizer
        if tokenizer_class.endswith("Fast") and (
            hasattr(self, "can_save_slow_tokenizer") and self.can_save_slow_tokenizer
        ):
```

Some models like recently released `deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B` does not provide `vocab.json` but has native type of a `Fast` tokenizer class: `LlamaTokenizerFast`. If this bug is not fixed, the loading of the tokenizer will break when `use_fast=False` is applied on the fast-stripped and down-classed tokenizer. The current code assumes all `Fast` models have  incuded `vocab.json` for reasons I do not yet understand. 

Full Context: We found this bug after using GPTQModel to quantize various DeepSeek models and using EvalPlus to run benchmarks on it. EvalPlus always load slow tokenizer and pass `use_fast=False`.  Native models worked fine, qunatized models failed. We traced the bug to this tokenizer save() bug that had nothing to do with quantization nor did we modify the tokenizer so this state change is unexpected. 

Please execute the following simple reproducer to re-create the tokenizer load with `use_fast=False` crash on `main`.

```py
import tempfile
from transformers import AutoTokenizer

# this model has no vocab file, use LlamaTokenizerFast
model_id = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"

with tempfile.TemporaryDirectory() as temp_dir:
    # 'tokenizer_class' in tokenizer_config.json is LlamaTokenizerFast now
    tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True) # <-- works
    # print(type(tokenizer)) => LlamaTokenizerFast
    tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False) # <-- works
    # print(type(tokenizer)) => LlamaTokenizerFast
    tokenizer.save_pretrained(temp_dir)
    # 'tokenizer_class' in tokenizer_config.json is LlamaTokenizer now, and no vocab file saved

    tokenizer = AutoTokenizer.from_pretrained(temp_dir, use_fast=False) # <-- crashes
    # can't load the tokenizer because saved tokenizer path has no vocab file
```

```
Traceback (most recent call last):
  File "/GPTQModel/sample_test.py", line 18, in <module>
    tokenizer = AutoTokenizer.from_pretrained(temp_dir, use_fast=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/mlx/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 921, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/mlx/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2032, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/mlx/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2272, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/mlx/lib/python3.11/site-packages/transformers/models/llama/tokenization_llama.py", line 169, in __init__
    self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/mlx/lib/python3.11/site-packages/transformers/models/llama/tokenization_llama.py", line 196, in get_spm_processor
    tokenizer.Load(self.vocab_file)
  File "/opt/anaconda3/envs/mlx/lib/python3.11/site-packages/sentencepiece/__init__.py", line 961, in Load
    return self.LoadFromFile(model_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lrl/opt/anaconda3/envs/mlx/lib/python3.11/site-packages/sentencepiece/__init__.py", line 316, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: not a string
```

Lastly, as we are not experts at tokenizer loading, this PR may also be fixing the correct bug at the wrong place. This possibility has not escaped me and I have a tingling feeling there is more to this bug than just this PR.

@ArthurZucker After re-looking at the latest code changes, can we just remove these check and strip altogether? Downgrading the tokenizer class type does break things as we have detailed above. It just makes more sense to remove this block of code completely rather than patch-fixing? What do you think.

@ArthurZucker  Sorry to to flood but we checked again both tokenizer `load` and `save` and here is the crust of the problem:

1. During tokenizer load, Fast tokenizer is prioritized. 
2. If `tokenizer_config.json` has `tokenizer_class` and is `Fast` tokenizer, hf code will never use non-fast even if  user passed `use_fast=False`. There appears to be a disconnect here if user explicitely ask for `use_fast=False`. At least if an warning should be issued here? Warning like "This tokenizer has no slow tokenizer, disregarding `use_fast=False`"?
3. FastTokenizer can be loaded with `use_fast=[True, False]`
4. Save tokenizer, `main`, breaks this by downcasting `Fast` tokenizer to `Slow` tokenizer where `use_fast=False` cannot be run if tokenizer is pure-fast, without `vocab.json` for slow-tokenizer compat. 
5. `use_fast=True` works on the saved but fast-stripped tokenizer because the code, below, will auto attach `Fast` when loading to always auto-detect if `fast` tokenizer can be used.
6. WIth fast-stripped toeknizer class, `use_fast=False` is never checked in same 925-936 lines of code so it loads using non-fast tokenizer class code and fails at missing `vocab.json`.

Ref:

https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/tokenization_auto.py#L925-L936

```py
    elif config_tokenizer_class is not None:
            tokenizer_class = None
            if use_fast and not config_tokenizer_class.endswith("Fast"):
                tokenizer_class_candidate = f"{config_tokenizer_class}Fast"
                tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
            if tokenizer_class is None:
                tokenizer_class_candidate = config_tokenizer_class
                tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
            if tokenizer_class is None:
                raise ValueError(
                    f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
                )
            return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
 ```           

Hey all! SUper sorry for missing the pings


Tests failing are unrelated, merging!