Add support for Japanese BERT models by cl-tohoku
Author: singletongue
Pull request: yes

This PR adds new BERT models for Japanese text.
Details of the models can be found in [this repository](https://github.com/cl-tohoku/bert-japanese).

Since the way of tokenization is Japanese-specific, a new file is added for the tokenizers.
And, the locations of the files (models, configs, etc.) should be updated when the files are uploaded to S3. (How could I do this?)
 

Comments:
This looks great @singletongue. For the file upload, I will send you a way to upload to our S3, early next week, if that works for you.

That will be great. Thank you!

I could try it on my side, it seems to work well! 

I don't believe I've seen any benchmark on Japanese tasks in your repository, how did you evaluate the models? Did you use perplexity on portions of the Japanese wikipedia?

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1839?src=pr&el=h1) Report
> Merging [#1839](https://codecov.io/gh/huggingface/transformers/pull/1839?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6a73382706ce3c6905023872f63a680f0eb419a4?src=pr&el=desc) will **decrease** coverage by `0.19%`.
> The diff coverage is `61.73%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1839/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1839?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master    #1839     +/-   ##
=========================================
- Coverage   80.07%   79.87%   -0.2%     
=========================================
  Files         112      114      +2     
  Lines       16867    17058    +191     
=========================================
+ Hits        13506    13625    +119     
- Misses       3361     3433     +72
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1839?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/configuration\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1839/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fYmVydC5weQ==) | `87.09% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1839/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `87.75% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1839/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX2JlcnQucHk=) | `96.31% <100%> (ø)` | :arrow_up: |
| [transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/1839/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9hdXRvLnB5) | `44.18% <33.33%> (-0.82%)` | :arrow_down: |
| [...nsformers/tests/tokenization\_bert\_japanese\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1839/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL3Rva2VuaXphdGlvbl9iZXJ0X2phcGFuZXNlX3Rlc3QucHk=) | `53.84% <53.84%> (ø)` | |
| [transformers/tokenization\_bert\_japanese.py](https://codecov.io/gh/huggingface/transformers/pull/1839/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9iZXJ0X2phcGFuZXNlLnB5) | `67.41% <67.41%> (ø)` | |
| [transformers/tests/utils.py](https://codecov.io/gh/huggingface/transformers/pull/1839/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL3V0aWxzLnB5) | `93.54% <83.33%> (+2.24%)` | :arrow_up: |
| [transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1839/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9iZXJ0LnB5) | `96.38% <0%> (+0.45%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1839?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1839?src=pr&el=footer). Last update [6a73382...0f53007](https://codecov.io/gh/huggingface/transformers/pull/1839?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


At the moment, there isn't an official publication regarding this work yet.

As a simple experiment, I applied the models to a Japanese document classification task using [livedoor news corpus](https://www.rondhuit.com/download.html#ldcc).
It is a 9-class classification, and I've confirmed that some of our models outperformed `bert-base-multilingual-cased` model.

| Model | Macro Precision | Macro Recall | Macro F1 |
| --- | --- | --- | --- |
| `bert-base-multilingual-cased` | 94.70 | 94.86 | 94.74 |
| `bert-base-japanese` | 95.64 | 95.58 | 95.60 |
| `bert-base-japanese-whole-word-masking` | 95.18 | 95.39 | 95.27 |
| `bert-base-japanese-char` | 94.32 | 93.96 | 94.09 |
| `bert-base-japanese-char-whole-word-masking` | 94.98 | 94.88 | 94.92 |

(Note that we expect the `char` models to be effective for SQuAD-like tasks, where span-level representation is needed. More comprehensive experiments are left for future work.)

Also, I've manually confirmed that our models predict reasonable tokens given a masked Japanese text.

Shall I add an example script for the classification task?

Alright, thanks for clarifying. I don't think adding an example script is necessary.

This looks good to me but I'll wait for @thomwolf to chime in and review the PR before merging as it adds a few core functionalities (like the tokenizer).

Hi @singletongue, the first step in uploading your weights to our S3 would be for you to create an account here: https://huggingface.co/join

The models will be namespaced under the chosen username or org name.

Can you let me know if you encounter any issue?

Thank you for your guidance, @julien-c.

I've just created an account. Then I was told to use a CLI tool to log in and upload the model. However, I couldn't find the specified CLI in the repository.

Yeah I should have been clearer, it's just that step 1 for now. What was clear or not clear in that page?

I will let you know here as soon as the CLI is operational.

This looks great to me, ok for merging.

Congratulations and thanks a lot @singletongue, this is an amazing work, both on preparing the data and training the model and on the super clean integration in the library. It's a masterpiece!

I think we add a test on the new tokenizer class though.

Do you think you could copy the file `tests/tokenization_bert_test` in a file `tests/tokenization_bert_japanese_test.py` and adapt a few English tests to Japanese specific characters @singletongue?

Thank you for your review, @thomwolf.
I added a test for the tokenizers.

The test on CircleCI failed since the tokenizers need [MeCab](https://taku910.github.io/mecab/) and `mecab-python3` to be installed.
I've checked that the test run successfully on my local environment where the dependencies are installed.

@singletongue The file upload CLI has been developed in https://github.com/huggingface/transformers/pull/2044 (soon to be merged to master)

Could you please upload your weights by doing:
```bash
git checkout cli_upload
pip install -e .
transformers-cli login
transformers-cli upload
```

Thanks!

(For the tests requirements, maybe @thomwolf or @LysandreJik can chime in – I think you can either add a `pip install mecab-python3 ` to the config.yml file, or maybe create a specific test target for JP)

Thank you, @julien-c. I've uploaded all the required files.
Also, I've updated `.circleci/config.yml` to successfully run tests on the tokenizers.

Thanks!

Ok LGTM