[WIP] Add Japanese RoBERTa Model
Author: butsugiri
Pull request: yes

# Add Japanese RoBERTa Model

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

I have recently trained a Japanese version of RoBERTa-base, and would like to make our model publicly available via this wonderful library.
The files are available [here](https://www.dropbox.com/sh/tu5v06ge4hgo2c4/AADdtnrBxh73076onmpt9gUva?dl=0)

I made two major changes:

1. Added a new tokenizer file `tokenization_roberta_japanese.py`
   - If one can nicely merge this tokenizer to `tokenization_bert_japanese.py`, that would be the best. (as adding a new file is not ideal)
2. Added `do_zenkaku` option to `tokenization_bert_japanese.py`
   - This is because I normalized every hankaku character to zenkaku character in preprocessing

## Background 

I have recently trained a Japanese version of RoBERTa-base using [fairseq codebase](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md).
I believe that our Japanese RoBERTa is better than the existing Japanese pre-trained language models (i.e., BERTs) that are publicly available for the following reasons:

- **More data**: Existing BERT models (the one from [NICT](https://alaginrc.nict.go.jp/nict-bert/index.html) and [cl-tohoku](https://huggingface.co/cl-tohoku/bert-base-japanese)) use Wikipedia dump for training. We also use CC-100 corpus, which is about 17.5 times larger than Wikipedia.
- **Better model**: We trained RoBERTa that is empirically better than vanilla BERT.
- **More compute**: In order to take advantage of large training data, we trained our RoBERTa longer than vanilla BERT. The training took 1 month on DGX-2 (V100 32GB x 16).

In fact, my colleagues have conducted experiments on multiple Japanese benchmark datasets, and confirmed that our RoBERTa is indeed superior:

### [amazon_reviews dataset](https://huggingface.co/datasets/amazon_reviews_multi)
|  model  |  accuracy |
| ---- | ---- |
|  [NICT_BERT-base](https://alaginrc.nict.go.jp/nict-bert/index.html) |  0.6014 |
|  [bert_bert-base-japanese-v2](https://huggingface.co/cl-tohoku/bert-base-japanese) |  0.5980 |
| Our RoBERTa | **0.6198** |

### [paws-x dataset](https://github.com/google-research-datasets/paws/tree/master/pawsx)
|  model  |  accuracy |
| ---- | ---- |
|  [NICT_BERT-base](https://alaginrc.nict.go.jp/nict-bert/index.html) | 0.8285 |
|  [bert_bert-base-japanese-v2](https://huggingface.co/cl-tohoku/bert-base-japanese) | 0.8305 |
| Our RoBERTa | **0.8440** |

### [JRTE dataset (Macro F1)](https://github.com/megagonlabs/jrte-corpus)

| Model | BASE | ME | MLM | 
| ------------- | ------------- | ------------- | ------------- |
| [NICT_BERT-base](https://alaginrc.nict.go.jp/nict-bert/index.html) | 90.3 | **80.0** | 55.5 |
| [bert_bert-base-japanese-v2](https://huggingface.co/cl-tohoku/bert-base-japanese)| 86.1 | 75.2 | 53.8 |
| Our RoBERTa| **92.3** | 77.8 | **58.0** |



## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
    - I read the guideline and ran the `make test` command. However, I am struggling with so many FAILED tests.
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?
    - I have not written any test yet. (I am not familiar with writing a test, so it may take a while.)


## Who can review?
I would really appreciate if @LysandreJik could review the code.

Comments:
Also it seems that the CircleCI jobs were not triggered - do you mind pushing a new commit (can be empty) to trigger the jobs? Thank you!

Hi, @LysandreJik . Thank you very much for your helpful review.
I would really like to work on your suggestions (I have a lot to learn from your comments), but right now I have to deal with my paper deadline. 
So please forgive me that it will take some time to start working on it.


No worries @butsugiri, looking forward to it!

I am sorry that it took me so long to address the review comments.

Currently, the following tests are not passing on my laptop (and they should fail on CI, too):
- `test_pickle_subword_regularization_tokenizer`
- `test_subword_regularization_tokenizer`
- `test_tokenizer_slow_store_full_signature`

I am not familiar enough with the internals of the library; so I am not sure how I should deal with them.

In addition, I think I have to write document for this new tokenizer (`make quality` command gave me an error).



Hi @butsugiri - I'm currently in the process of checking with CircleCI why the runs don't trigger on your PRs. Will resolve this problem and work with you to resolve the issues above. Thanks for your patience!

I see. Thank you!

Hello @butsugiri! It seems that the CircleCI tests aren't run because they try to run with the user @lalitpagaria's credentials. Do you know why that might be so?

@lalitpagaria, if you receive a notification, from what I understand the CircleCI authentication system tries to authenticate you as a runner for this PR - but your credentials have gone stale and CircleCI cannot manage to run the tests. It was mentioned you should do a full CircleCI [re-authentication](https://support.circleci.com/hc/en-us/articles/360051228052-How-to-perform-a-full-re-authentication). 

Please let me know if this is expected or not - I don't see any commits from @lalitpagaria so I'm unaware of what might be causing this. I would expect the Hugging Face auth to be used to run these tests.

@LysandreJik oh man. I dont have any idea why this happened.
In past I added RAG related PRs (around 10 months back) that time to debug CI failure I opened Circles CI. Not sure of that caused this. But how come it automatically update credentials when I don't have permission 🤷🏽‍♂️

Not receiving any notification from circleCI. I have removed them from my approved app list long time back

That's helpful, thank you for sharing @lalitpagaria! Will report back to CircleCI's customer support.

@LysandreJik I just enabled circleCI. See if this help at least should unblock existing failing CI meanwhile you work with them to get this sorted. I really apologize if any of my actions caused this.

@butsugiri 
Thank you for making the effort to publish the Japanese Roberta model. 
If it's not a problem, I would like to know the details of how you trained tokenizer. I would like to know what kind of pre-processing you did on the pre-trained corpus and the Mecab dictionary you used.

@butsugiri, sorry this slipped through the cracks - could you close the PR and open a new one without touching your branch? Hopefully the CI should trigger, otherwise I'll take care of it. Thank you!

@kambehmw Thank you for your interest. My colleagues and I are currently preparing the release page with the details of pretraining including tokenization. I would appreciate if you could wait for it, Thanks!

@LysandreJik Absolutely no problem. Thank you for your continuous support, I really appreciate it.
Close this PR and will open new one.

@butsugiri Thanks for your reply. I understand that you are preparing a release page, and I look forward to the official release of the Japanese Roberta Model. Thanks.