Revert tokenization regression, add test
Author: tomaarsen
Pull request: yes

# What does this PR do?

This PR fixes the following regression:
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
sentences = ["C'est une belle journée.", "Das ist ein schöner Tag.", "It's a beautiful day."]

tokenized = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")
decoded = tokenizer.batch_decode(tokenized["input_ids"])
print(decoded)
# `main` before #42563 or with this PR:
["
 C'est une belle journée.
", '
 Das ist ein schöner Tag.
<pad>', "
 It's a beautiful day.
"]
# `main` since #42563:
["
 c ' est une belle <unk>. 
", '
 das ist ein <unk> tag. 
 <pad>', "
 it ' s a <unk> day. 
"]
```

The issue was introduced in #42563. In essence, alongside removing a lot of sentencepiece conversion code, this PR also removed this code (https://github.com/huggingface/transformers/pull/42563/files#r2727583533) that loads the `tokenizer_file` if it exists:
```python
        # If tokenizer_file exists and tokenizer has a TokenizersBackend, replace the blank tokenizer with tokenizer.json if tokenizer_file is not None and hasattr(tokenizer, "_tokenizer"):
            from tokenizers import Tokenizer as TokenizerFast

            tokenizer._tokenizer = TokenizerFast.from_file(tokenizer_file)
            # Re-run post-initialization if the tokenizer has it
            if hasattr(tokenizer, "_post_init"):
                tokenizer._post_init()
```

Because this code was removed, the tokenizer from https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 loads with the default WordPiece model, instead of the Unigram that the model was trained for. I've added a test that fails on `main` that's resolved by reintroducing this code.

This is currently heavily affecting the performance of the https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 embedding model, causing test failures on my side.

### Note
I also simplified some
```python
        for tokenizer_class in [BertTokenizer, BertTokenizer]:
            with self.subTest(f"{tokenizer_class}"):
```
test cases. This is likely a leftover from replacing BertTokenizerFast.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@ArthurZucker @itazap

- Tom Aarsen

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43496). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This PR is causing a regression of its own with 
```
FAILED tests/models/auto/test_tokenization_auto.py::AutoTokenizerTest::test_do_lower_case - AssertionError: '[UNK]' != 'hello'
```
Perhaps this PR is a bit too naive: it likely ignores explicit kwargs when loading the `AutoTokenizer.from_pretrained` in favor of the tokenizer file. There's likely a cleaner fix. cc @ArthurZucker as you refactored this initially.

- Tom Aarsen

I've just spotted this section in the migration guide: https://github.com/huggingface/transformers/blob/main/MIGRATION_GUIDE_V5.md#tokenization
Which seems to imply that the change was intended, and that "custom" tokenizer options will no longer be supported. 

How should existing models tackle this? Introduce a custom tokenizer with `auto_map` and spring `trust_remote_code` on their users?

- Tom Aarsen

I ended up patching this by setting `"tokenizer_class": "PreTrainedTokenizerFast"` in the `tokenizer_config.json`. This prevents the `BertTokenizer` code from overriding the `tokenizer.json` Unigram configs, and works on both Transformers v4 and v5. Much better.

Relevant commit: https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/commit/e8f8c211226b894fcb81acc59f3b34ba3efd5f42

- Tom Aarsen