Repository: huggingface/transformers

Issue:
BertTokenizer with BertJapaneseTokenizer pretrained model generates unintended tokenization.
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.0.dev0
- Platform: Windows-10-10.0.19043-SP0
- Python version: 3.9.5
- PyTorch version (GPU?): 1.9.0+cpu (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

`BertTokenizer` with `BertJapaneseTokenizer` pretrained model generates unintended tokenization without any caution.



## To reproduce

Steps to reproduce the behavior:

Run
```python
EXAMPLE_BERT_JAPANESE_ID = "cl-tohoku/bert-base-japanese"
tokenizer = BertTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID)
print(tokenizer.tokenize("今日はいい天気ですね"))
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

```python
not_correct = BertTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID)
correct = BertJapaneseTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID)
print(not_correct.tokenize("今日はいい天気ですね"))
print(correct.tokenize("今日はいい天気ですね"))
```

Because the two tokenizers were made from the same pretrained model, the output should have been
```
['今日', 'は', 'いい', '天気', 'です', 'ね']
['今日', 'は', 'いい', '天気', 'です', 'ね']
```

or `BertTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID)` should have raised an error.
However, the actual result was

```
['今', '日', 'はい', '##い', '天', '気', 'です', '##ね']
['今日', 'は', 'いい', '天気', 'です', 'ね']
```
and no error or warning raised.

<!-- A clear and concise description of what you would expect to happen. -->



Discussion:
I think you're raising a good issue that tokenizers will not tell you when they're instantiated from a checkpoint that doesn't have the same tokenizer architecture.

However, I think this should be resolved for all tokenizers rather than for a single one, probably by checking the `tokenizer_class` inside the `config.json` and `tokenizer_config.json`. cc @SaulLu @sgugger 
> However, I think this should be resolved for all tokenizers rather than for a single one, probably by checking the `tokenizer_class` inside the `config.json` and `tokenizer_config.json`.

I agree. `AutoTokenizer` can choose a tokenizer automatically, so until it is solved, I think that recommending a user uses `AutoTokenizer` is a better way to prevent the silent error. 


Thank you very much for reporting this problem @europeanplaice :+1:. Indeed, I share your opinion, it would be better if a warning was logged if ever the class tokenizer used to load a pretrained tokenizer is not the same type.

I also agree with @LysandreJik, it should be possible to find this information in `config.json` and/or` tokenizer_config.json` and this would allow to have a logged warning for all types of tokenizers. 

@europeanplaice, would you like to work on this? If you want, I could of course help you. Or if you don't have the time/want to, I can take over your PR in the next days and adapt it to this new approach. What do you think? :blush: 
@SaulLu Thank you for your offer. I want to try to tackle this problem.
I plan to add something like below
https://github.com/huggingface/transformers/blob/122d7dc34fd0e397a08b8a584a632fc57d3fd5d0/src/transformers/models/auto/tokenization_auto.py#L527-L551

to `from_pretrained` in `PreTrainedTokenizerBase (tokenization_utils_base.py)` to make sure that we can check whether a user is trying to use different tokenizers between `cls` and `config.json or tokenizer_config.json`'s class before a tokenizer returns.
If this detected conflicts between them, a warning would be logged, or an error would occur.

I want my PR to be in line with your overall plan, so I hope to get your opinion about this comment.

Thank you very much for offering to take care of this issue! 

From my point of view, what you described above sounds really great! :+1: 

Resolving patch:
diff --git a/src/transformers/models/bert_japanese/tokenization_bert_japanese.py b/src/transformers/models/bert_japanese/tokenization_bert_japanese.py
--- a/src/transformers/models/bert_japanese/tokenization_bert_japanese.py
+++ b/src/transformers/models/bert_japanese/tokenization_bert_japanese.py
@@ -132,7 +132,7 @@ def __init__(
         if not os.path.isfile(vocab_file):
             raise ValueError(
                 f"Can't find a vocabulary file at path '{vocab_file}'. To load the vocabulary from a Google pretrained "
-                "model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`"
+                "model use `tokenizer = AutoTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`"
             )
         self.vocab = load_vocab(vocab_file)
         self.ids_to_tokens = collections.OrderedDict([(ids, tok) for tok, ids in self.vocab.items()])
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -1745,13 +1745,58 @@ def _from_pretrained(
         if tokenizer_config_file is not None:
             with open(tokenizer_config_file, encoding="utf-8") as tokenizer_config_handle:
                 init_kwargs = json.load(tokenizer_config_handle)
+            # First attempt. We get tokenizer_class from tokenizer_config to check mismatch between tokenizers.
+            config_tokenizer_class = init_kwargs.get("tokenizer_class")
             init_kwargs.pop("tokenizer_class", None)
             saved_init_inputs = init_kwargs.pop("init_inputs", ())
             if not init_inputs:
                 init_inputs = saved_init_inputs
         else:
+            config_tokenizer_class = None
             init_kwargs = init_configuration
 
+        if config_tokenizer_class is None:
+            from .models.auto.configuration_auto import AutoConfig
+
+            # Second attempt. If we have not yet found tokenizer_class, let's try to use the config.
+            try:
+                config = AutoConfig.from_pretrained(pretrained_model_name_or_path)
+                config_tokenizer_class = config.tokenizer_class
+            except (OSError, ValueError, KeyError):
+                # skip if an error occured.
+                config = None
+            if config_tokenizer_class is None:
+                # Third attempt. If we have not yet found the original type of the tokenizer,
+                # we are loading we see if we can infer it from the type of the configuration file
+                from .models.auto.configuration_auto import CONFIG_MAPPING
+                from .models.auto.tokenization_auto import TOKENIZER_MAPPING
+
+                if hasattr(config, "model_type"):
+                    config_class = CONFIG_MAPPING.get(config.model_type)
+                else:
+                    # Fallback: use pattern matching on the string.
+                    config_class = None
+                    for pattern, config_class_tmp in CONFIG_MAPPING.items():
+                        if pattern in str(pretrained_model_name_or_path):
+                            config_class = config_class_tmp
+                            break
+
+                if config_class in TOKENIZER_MAPPING.keys():
+                    config_tokenizer_class, config_tokenizer_class_fast = TOKENIZER_MAPPING[config_class]
+                    if config_tokenizer_class is not None:
+                        config_tokenizer_class = config_tokenizer_class.__name__
+                    else:
+                        config_tokenizer_class = config_tokenizer_class_fast.__name__
+
+        if config_tokenizer_class is not None:
+            if cls.__name__.replace("Fast", "") != config_tokenizer_class.replace("Fast", ""):
+                logger.warning(
+                    "The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. "
+                    "It may result in unexpected tokenization. \n"
+                    f"The tokenizer class you load from this checkpoint is '{config_tokenizer_class}'. \n"
+                    f"The class this function is called from is '{cls.__name__}'."
+                )
+
         # Update with newly provided kwargs
         init_kwargs.update(kwargs)
 


Test patch:
diff --git a/tests/test_tokenization_bert_japanese.py b/tests/test_tokenization_bert_japanese.py
--- a/tests/test_tokenization_bert_japanese.py
+++ b/tests/test_tokenization_bert_japanese.py
@@ -22,6 +22,7 @@
 from transformers.models.bert_japanese.tokenization_bert_japanese import (
     VOCAB_FILES_NAMES,
     BertJapaneseTokenizer,
+    BertTokenizer,
     CharacterTokenizer,
     MecabTokenizer,
     WordpieceTokenizer,
@@ -278,3 +279,23 @@ def test_tokenizer_bert_japanese(self):
         EXAMPLE_BERT_JAPANESE_ID = "cl-tohoku/bert-base-japanese"
         tokenizer = AutoTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID)
         self.assertIsInstance(tokenizer, BertJapaneseTokenizer)
+
+
+class BertTokenizerMismatchTest(unittest.TestCase):
+    def test_tokenizer_mismatch_warning(self):
+        EXAMPLE_BERT_JAPANESE_ID = "cl-tohoku/bert-base-japanese"
+        with self.assertLogs("transformers", level="WARNING") as cm:
+            BertTokenizer.from_pretrained(EXAMPLE_BERT_JAPANESE_ID)
+            self.assertTrue(
+                cm.records[0].message.startswith(
+                    "The tokenizer class you load from this checkpoint is not the same type as the class this function is called from."
+                )
+            )
+        EXAMPLE_BERT_ID = "bert-base-cased"
+        with self.assertLogs("transformers", level="WARNING") as cm:
+            BertJapaneseTokenizer.from_pretrained(EXAMPLE_BERT_ID)
+            self.assertTrue(
+                cm.records[0].message.startswith(
+                    "The tokenizer class you load from this checkpoint is not the same type as the class this function is called from."
+                )
+            )
diff --git a/tests/test_tokenization_common.py b/tests/test_tokenization_common.py
--- a/tests/test_tokenization_common.py
+++ b/tests/test_tokenization_common.py
@@ -29,7 +29,10 @@
 from huggingface_hub import HfApi
 from requests.exceptions import HTTPError
 from transformers import (
+    AlbertTokenizer,
+    AlbertTokenizerFast,
     BertTokenizer,
+    BertTokenizerFast,
     PreTrainedTokenizer,
     PreTrainedTokenizerBase,
     PreTrainedTokenizerFast,
@@ -3288,6 +3291,41 @@ def test_training_new_tokenizer_with_special_tokens_change(self):
             expected_result = tokenizer.backend_tokenizer.normalizer.normalize_str(expected_result)
         self.assertEqual(expected_result, decoded_input)
 
+    def test_tokenizer_mismatch_warning(self):
+        for tokenizer, pretrained_name, kwargs in self.tokenizers_list:
+            with self.subTest(f"{tokenizer.__class__.__name__} ({pretrained_name})"):
+                with self.assertLogs("transformers", level="WARNING") as cm:
+                    try:
+                        if self.tokenizer_class == BertTokenizer:
+                            AlbertTokenizer.from_pretrained(pretrained_name)
+                        else:
+                            BertTokenizer.from_pretrained(pretrained_name)
+                    except (TypeError, AttributeError):
+                        # Some tokenizers cannot be loaded into the target tokenizer at all and errors are returned,
+                        # here we just check that the warning has been logged before the error is raised
+                        pass
+                    finally:
+                        self.assertTrue(
+                            cm.records[0].message.startswith(
+                                "The tokenizer class you load from this checkpoint is not the same type as the class this function is called from."
+                            )
+                        )
+                    try:
+                        if self.rust_tokenizer_class == BertTokenizerFast:
+                            AlbertTokenizerFast.from_pretrained(pretrained_name)
+                        else:
+                            BertTokenizerFast.from_pretrained(pretrained_name)
+                    except (TypeError, AttributeError):
+                        # Some tokenizers cannot be loaded into the target tokenizer at all and errors are returned,
+                        # here we just check that the warning has been logged before the error is raised
+                        pass
+                    finally:
+                        self.assertTrue(
+                            cm.records[0].message.startswith(
+                                "The tokenizer class you load from this checkpoint is not the same type as the class this function is called from."
+                            )
+                        )
+
 
 @is_staging_test
 class TokenizerPushToHubTester(unittest.TestCase):
diff --git a/tests/test_tokenization_fast.py b/tests/test_tokenization_fast.py
--- a/tests/test_tokenization_fast.py
+++ b/tests/test_tokenization_fast.py
@@ -44,6 +44,11 @@ def setUp(self):
         tokenizer = PreTrainedTokenizerFast.from_pretrained(model_paths[0])
         tokenizer.save_pretrained(self.tmpdirname)
 
+    def test_tokenizer_mismatch_warning(self):
+        # We disable this test for PreTrainedTokenizerFast because it is the only tokenizer that is not linked to any
+        # model
+        pass
+
     def test_pretrained_model_lists(self):
         # We disable this test for PreTrainedTokenizerFast because it is the only tokenizer that is not linked to any
         # model