Fix Mistral models which contain both `tokenizer.json` and `tekken.json`
Author: hmellor
Pull request: yes
Labels: name: for patch

## Context

Some Mistral models are released with both a `tokenizers` format tokenizer (`tokenizer.json`) and a `mistral-common` format tokenizer (`tekken.json`).

If we examine `TOKENIZER_MAPPING_NAMES` in `src/transformers/models/auto/tokenization_auto.py` we see the following pattern repeated for all Mistral models:

https://github.com/huggingface/transformers/blob/08a7ef05bcf9723cb2e58855afb8dc2c799323ff/src/transformers/models/auto/tokenization_auto.py#L204-L209

From these we can reasonably assume that if `mistral-common` is:

- not installed then `AutoTokenizer` should use `TokenizersBackend` (i.e. `tokenizer.json`)
- is installed then `AutoTokenizer` should use `MistralCommonBackend` (i.e. `tekken.json`)

Therefore, for checkpoints which have **both** `tokenizer.json` and `tekken.json`, the decision of which to use comes down to whether `mistral-common` is installed.

Next, if we look at:

https://github.com/huggingface/transformers/blob/08a7ef05bcf9723cb2e58855afb8dc2c799323ff/src/transformers/models/auto/tokenization_auto.py#L718-L746

We can see that if:

- We enter the first if block because:
  - `tokenizer_config.json` exists and contains a `tokenizer_class` field
  - `TOKENIZER_MAPPING_NAMES` contains an entry for this model type
- When `mistral-common` is:
  - not installed, we **never** enter the second if block because `registered_class_name` is always `TokenizersBackend` for Mistral models
  - installed, we do enter the second if block and **always** load from `tokenizer_config_class`

This last line is a bug and leads to incorrect tokenization.

## Reproducer

Using the following file as a reproducer:

```python
# repro.py
from transformers import AutoTokenizer
from transformers.utils.import_utils import is_mistral_common_available

print(f"mistral-common is installed: {is_mistral_common_available()}")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Ministral-8B-Instruct-2410")
tokenizer_cls = type(tokenizer).__name__
print(f"AutoTokenizer resolved to {tokenizer_cls=}")
# Tokens for "Hello, how are you?"
token_ids = [22177, 1044, 2606, 1584, 1636, 1063]
print(tokenizer.decode(token_ids))
```

We see the following two outputs with and without `mistral-common`:

```console
$ python repro.py 
mistral-common is installed: True
AutoTokenizer resolved to tokenizer_cls='LlamaTokenizer'
Hello,ĠhowĠareĠyou?
```

```console
$ python repro.py 
mistral-common is installed: False
AutoTokenizer resolved to tokenizer_cls='TokenizersBackend'
Hello, how are you?
```

The case where `mistral-common` is installed resolves the incorrect class and produces incorrect detokenization.

## Solution

The solution is to treat `MistralCommonBackend` more similarly to `TokenizersBackend`. If `MistralCommonBackend` is the registered class, try to use it and fall back to `TokenizersBackend` if there wasn't actually a `tekken.json` file.

This behaviour:

- Matches the intent indicated by the snippets in `TOKENIZER_MAPPING_NAMES`
- Never allows the incorrect classes such as `LlamaTokenizer` to load; which is the same behaviour as when `mistral-common` is not installed

Running the reproducer again with and without `mistral-common` we see:

```console
$ python repro.py 
mistral-common is installed: True
AutoTokenizer resolved to tokenizer_cls='MistralCommonBackend'
Hello, how are you?
```

```console
$ python repro.py 
mistral-common is installed: False
AutoTokenizer resolved to tokenizer_cls='TokenizersBackend'
Hello, how are you?
```

which is the expected behaviour.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46622)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46622). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46622&sha=b11320

the reproducer works with / without mistral-common in https://github.com/huggingface/transformers/pull/46091

should be good to close as we merged this instead https://github.com/huggingface/transformers/pull/46667
Thanks harry!