Repository: huggingface/transformers

Issue:
Cannot run  run_mlm.py on a Japanese dataset - AttributeError: module transformers.models.mbart50 has no attribute BertJapaneseTokenizerFast
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.0.dev0
- Platform: Linux-4.18.0-25-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

Models:
- albert, bert, xlm: @LysandreJik

Examples:
- maintained examples (not research project or legacy): @sgugger, @patil-suraj


## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
transformers/examples/pytorch/language-modeling/run_mlm.py

The tasks I am working on is:
* [x] my own task or dataset: (give details below)
It's a Japanese corpus in .txt format.

## To reproduce

Steps to reproduce the behavior:

1. I followed the instructions at https://huggingface.co/transformers/examples.html: git cloned the transformers repository, installed it, along with requirements in language-modeling.
2.  I tried to run it with
`python run_mlm.py     --model_name_or_path cl-tohoku/bert-base-japanese-whole-word-masking     --train_file /path/to/train/file.txt     --do_train  --output_dir output_dir/
`
Traceback (most recent call last):
  File "run_mlm.py", line 550, in <module>
    main()
  File "run_mlm.py", line 337, in main
    tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)
  File "/my/.pyenv/versions/anaconda3-5.1.0/envs/jp/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 424, in from_pretrained
    tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
  File "/my/.pyenv/versions/anaconda3-5.1.0/envs/jp/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 219, in tokenizer_class_from_name
    return getattr(module, class_name)
  File "/my/.pyenv/versions/anaconda3-5.1.0/envs/jp/lib/python3.7/site-packages/transformers/file_utils.py", line 1992, in __getattr__
    raise AttributeError(f"module {self.__name__} has no attribute {name}")
AttributeError: module transformers.models.mbart50 has no attribute BertJapaneseTokenizerFast

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
It should be done without an error. I have done this in July, and it went through without a problem.



Discussion:
Hi,

I think that you need to run the whole word masking script which can be found [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/mlm_wwm) instead of the regular `run_mlm.py` script (as you're doing whole word masking instead of just masking tokens).

I've created a Colab notebook, it seems to work fine! https://colab.research.google.com/drive/1d2yGWLYy44KgSId1WbSfusX0Jp8JhKyD?usp=sharing
It worked! Thank you so much!
I needed to run run_mlm.py, not run_mlm_wwm.py, this time, and tried to run
`python run_mlm.py --model_name_or_path cl-tohoku/bert-base-japanese --train_file /path/to/train/file.txt --do_train --output_dir output_dir/`
and got the same error message:

```
Traceback (most recent call last):
  File "run_mlm.py", line 550, in <module>
    main()
  File "run_mlm.py", line 337, in main
    tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)
  File "/home/cl/jungmin-c/.pyenv/versions/anaconda3-5.1.0/envs/bert-japanese/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 431, in from_pretrained
    tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
  File "/home/cl/jungmin-c/.pyenv/versions/anaconda3-5.1.0/envs/bert-japanese/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 226, in tokenizer_class_from_name
    return getattr(module, class_name)
  File "/home/cl/jungmin-c/.pyenv/versions/anaconda3-5.1.0/envs/bert-japanese/lib/python3.7/site-packages/transformers/file_utils.py", line 1995, in __getattr__
    raise AttributeError(f"module {self.__name__} has no attribute {name}")
AttributeError: module transformers.models.rembert has no attribute BertJapaneseTokenizerFast
```
I cannot figure out how to resolve this. I would greatly appreciate if you could look into it.
@NielsRogge 


Resolving patch:
diff --git a/src/transformers/models/auto/tokenization_auto.py b/src/transformers/models/auto/tokenization_auto.py
--- a/src/transformers/models/auto/tokenization_auto.py
+++ b/src/transformers/models/auto/tokenization_auto.py
@@ -229,12 +229,12 @@ def tokenizer_class_from_name(class_name: str):
 
     for module_name, tokenizers in TOKENIZER_MAPPING_NAMES.items():
         if class_name in tokenizers:
-            break
+            module_name = model_type_to_module_name(module_name)
 
-    module_name = model_type_to_module_name(module_name)
+            module = importlib.import_module(f".{module_name}", "transformers.models")
+            return getattr(module, class_name)
 
-    module = importlib.import_module(f".{module_name}", "transformers.models")
-    return getattr(module, class_name)
+    return None
 
 
 def get_tokenizer_config(


Test patch:
diff --git a/tests/test_tokenization_auto.py b/tests/test_tokenization_auto.py
--- a/tests/test_tokenization_auto.py
+++ b/tests/test_tokenization_auto.py
@@ -22,6 +22,7 @@
     AutoTokenizer,
     BertTokenizer,
     BertTokenizerFast,
+    CTRLTokenizer,
     GPT2Tokenizer,
     GPT2TokenizerFast,
     PreTrainedTokenizerFast,
@@ -162,6 +163,11 @@ def test_auto_tokenizer_from_local_folder(self):
         self.assertIsInstance(tokenizer2, tokenizer.__class__)
         self.assertEqual(tokenizer2.vocab_size, 12)
 
+    def test_auto_tokenizer_fast_no_slow(self):
+        tokenizer = AutoTokenizer.from_pretrained("ctrl")
+        # There is no fast CTRL so this always gives us a slow tokenizer.
+        self.assertIsInstance(tokenizer, CTRLTokenizer)
+
     def test_get_tokenizer_config(self):
         # Check we can load the tokenizer config of an online model.
         config = get_tokenizer_config("bert-base-cased")