Fix models for which we don't have a dedicated tokenizer class, and the listed one is incorrect
Author: itazap
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=45936)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45936)
<!-- ci-dashboard-badge:end -->

we don't check the tokenization of all the model paths we have in the transformers repo. Fixes https://github.com/huggingface/transformers/issues/45488, Related to https://github.com/huggingface/transformers/pull/44255

We have models that don't have their own dedicated Tokenizer class and use another model's tokenizer (ex. Granite which uses GPT2Tokenizer - related issue: https://github.com/huggingface/transformers/pull/45813) ). The different model tokenizer class would be mapped in the `tokenization_auto.py` mapping, or in the `tokenization_config.json`. Sometimes the mapped tokenizer isn't actually the one that is being used, and v5 surfaced these incorrect mappings. In order to "stay true to" the pre-v5 behavior of these models, we can map them to `TokenizersBackend` (eq. to `PreTrainedTokenizerFast` in v4) which loads the `tokenizer.json` as is. This happens because in v5 we actually try to load the mapped tokenizer class and force the same tokenizer type.

Anyway we only test tokenization of models that have their own tokenizer class but we should test tokenization for **every checkpoint we have in the repo**!


## This PR

### tools/tokenizer_compare/run_equivalence_comparison_all_checkpoints.py script 
(based on that in https://github.com/huggingface/transformers/pull/44255)

```
python tools/tokenizer_compare/run_equivalence_comparison_all_checkpoints.py
  --rerun-from-results tools/tokenizer_compare/output/tokenizer_compare_results.json
  --output output/tokenizer_compare_results_rerun.json

```

scans `tests/models/test_modeling_*.py` for `.from_pretrained(...)` and extracts all the checkpoint paths we list, and compares _tokenizers loaded via `AutoTokenizer.from_pretrained` vs `TokenizersBackend.from_pretrained`.

### tools/tokenizer_compare/compare_tokenizer_xlni_roundtrip.py script 

check if `TokenziersBackend` and `AutoTokenizer` behave the same way, producing same roundtrip results on xlni and generate the same token ids

```
python tools/tokenizer_compare/compare_tokenizer_xlni_roundtrip.py bigcode/tiny_starcoder_py v1 --backend tokenizers --compare-backend auto
```

## Report

PRE this PR / Fixes - on all the checkpoints we list: [report](https://htmlpreview.github.io/?https://raw.githubusercontent.com/huggingface/transformers/check_repo_model_tokenizers/tools/tokenizer_compare/output/tokenizer_compare_results_original.html)

POST this PR / Fixes: [report](https://htmlpreview.github.io/?https://raw.githubusercontent.com/huggingface/transformers/check_repo_model_tokenizers/tools/tokenizer_compare/output/tokenizer_compare_results_rerun.html)
### Remaining patterns: ###

#### Pattern 1 & 2: llama, jetmoe, idefics

Expected and safe to ignore 

- these behave identically as per (`tools/tokenizer_compare/compare_tokenizer_xlni_roundtrip.py`). These are legacy llama (already caught here: https://github.com/huggingface/transformers/pull/44255) 

#### dolma2

- opened a PR on the hub https://huggingface.co/allenai/dolma2-tokenizer/discussions/1 , there is no `config.json` so we cannot force `TokenizersBackend` from `tokenization_auto.py`

#### umt5

- opened a PR on the hub https://huggingface.co/google/umt5-small/discussions/2 [google/umt5-small](https://huggingface.co/google/umt5-small) 


## When we need precompiled
the charsmap is unordered and when we use a `tokenizers.normalizer` it will reorder mappings (like NFKC) 

## TODO

adding a test that will check AutoTokenizer and TokenizersBackend loads equivalent _tokenizer objects for each path we mention in the repo

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45936). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45936&sha=ec9c26

Hi @itazap, great fix! 👍 Any plans to merge this into main soon?

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: albert, auto, bert, camembert, funnel, mpnet, rembert, tipsv2, xglm, xlnet

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45936)
**Latest run:** [29111550328:1](https://github.com/huggingface/transformers/actions/runs/29111550328)
**Result:** `success` | **Jobs:** 15 | **Tests:** 163,182 | **Failures:** 0 | **Duration:** 15h 35m

<!-- ci-dashboard-recap:end -->