Add support for setting default `use_fast` via `tokenizer_config.json`
Author: ryokan0123
Pull request: no
Labels: name: Feature request

### Feature request

Currently, `AutoTokenizer.from_pretrained` only accepts the `use_fast` parameter from the keyword arguments.
However, it would be beneficial if we could set the default value for use_fast through the `tokenizer_config.json` file.

### Motivation

In our use case, we need to force users to use `use_fast=False` because `T5TokenizerFast` does not support byte fall back, while `T5Tokenizer` does. By allowing the default value to be set in `tokenizer_config.json`, we can ensure that users are using the correct tokenizer without requiring them to explicitly set the `use_fast` parameter every time.

### Your contribution

If this seems OK, I can submit a PR that adds a logic in `AutoTokenizer.from_pretrained` to read `use_fast` from `tokenizer_config.json`.

Comments:
Hi @ryokan0123, thanks for opening this feature request! 

The `use_fast` argument is specific to the `AutoTokenizer` class which controls which tokenizer it loads, rather than the tokenizers themselves. 

Adding this to the config effectively means that it's an argument which gets passed to the tokenizers. What does it mean to do `T5TokenizerFast(vocab_file=file, use_fast=False)`? 

Adding this would introduce some difficult state management and possibly unexpected behaviour. If one were to save the tokenizer and load it to the hub, why should others always load the slow tokenizer? It wouldn't be obvious from the flag name in the config why `use_fast=False` is set that way (to modify the tokenizer behaviour). The default behaviour in transformers is to load the fast class wherever possible and this would silently change this.

The main issue here seems to be the difference between the fast and slow tokenizer, rather than the selection of the tokenizers themselves. `use_fast` should be a condition of the running environment, which can be specific to each user, rather than a saved state.

WDYT @ArthurZucker @LysandreJik ? 

Hi, @amyeroberts 

Thanks for sharing the insights into the tokenizer.
Indeed, including the `use_fast` argument in the configuration will have some undesirable consequences.

While specifying `use_fast` in the config seemed like an easy fix, the real issue is the discrepancy between the implementations of slow and fast tokenizers that we are dealing with.
I think I will pursue another solution.

Hey! The T5 tokenizer fast can support bytefallback if you change the tokenizer.json / the `pre_tokenizer` of the fast tokenizer. It's not done by default in the published models, but should be "easy" to activate! 🤗 

Just to raise this issue again, I'm facing a similar problem where the slow and fast version of `LlamaTokenizer` are fundamentally incompatible due to implementation differences in the underlying C++ (of `sentencepiece`) and Rust code.

I'd like to take the responsibility to pass `use_fast=False` away from users to prevent easy-to-make mistakes.
Would a solution that allows disabling fast versions using something like `auto_map` be something you could consider for integration?

To give some motivation, it would simply be really nice to not have to supply an "untrusted" version when I won't actually add any code. :)

Hey hey! I think the `fast` version should be the source of truth. In general we run full XNLI and multilingual dataset tests, so would be surprised if there are differences when encoding as the model is BPE so no random parts (unlike unigram that can have randomness). 
I'll do my best to always make sure we have the same results! 

I can open an issue in `tokenizers` as well to go into more detail, but there are at least 2 things that are not aligned between `sentencepiece`'s and `tokenizers`'s BPE implementation, at least when using `add_dummy_prefix=True`/`add_prefix_space=True` (`sentencepiece`/`transformers` setting name, respectively):
1. [`tokenizers` does not add the prefix space when a space is already present as the first character of the string](https://github.com/huggingface/tokenizers/blob/14a07b06e4a8bd8f80d884419ae4630f5a3d8098/tokenizers/src/pre_tokenizers/metaspace.rs#L122), in contrast to `sentencepiece`, which unconditionally adds it (even for empty strings IIRC).
2. With a vocabulary of empty spaces like
```python
num_special_tokens = 4
vocab = {
    i + num_special_tokens: ' ' * (i + 1)
    for i in range(24)
}
vocab.update({i: '' for i in range(num_special_tokens)})
assert len(vocab) == 28
assert len(vocab[27]) == 24
```
  , please try to tokenize the following string after evaluating it in Python: `' ' * 63`. With my tokenizer, I get different results here between `sentencepiece` and `tokenizers`. `sentencepiece` seems to do a greedy fit with maximum-length tokens at the start and gives something like `[27, 27, 19]`. `tokenizers` does something different and returns something like `[27, 23, 23]`.

Again, I wrote this up in a hurry without checking stuff for sure. I'd be happy to clean this up for a report in `tokenizers` with correct/reproducible code examples, but just wanted to make sure we're on the same page before that. :)
Having read through other issues in here it also seemed like you were aware of Slow vs. Fast implementation differences, so I'm surprised that Fast should be considered the source of truth!

I think in generatl for 1. yes, I think initial sentencepiece would add only if it did not exists (t5). 
But for 2, I don't agree, it's a matter of merge order. We had this issue for Gemma2 and merged this: https://github.com/huggingface/transformers/pull/32191 which solved the order issue. 

I see! So the issue is with the creation of the merge table, then?

Yep! If you properly sort the merges based on what sentencepiece does (so I think it length of the merges) then you can have 1-1 match

I did try modifying the merge table, but I'll fool around with it more. Thanks already; however, what does this mean for `transformers`? Should the merge code be updated?

The merge code was already updated AFAIK this should work now

I can do something for 1. in `tokenizers` probably a simple `pre_tokenizer.prepend` 

Really awesome, thanks a lot for taking the time to discuss! I'll try to reproduce the merging problems with the latest `transformers` version and get back to you.