Add sudachi_projection option to BertJapaneseTokenizer
Author: hiroshi-matsuda-rit
Pull request: yes

# What does this PR do?

A new feature of SudachiPy v0.6.8 allows normalization of words based on Japanese morphological analysis.

https://github.com/WorksApplications/sudachi.rs/issues/230

This morphology-based normalization functionality, named "projection" in SudachiPy, makes Japanese sub-tokenization more efficient and can improve transformer performance.

Very few changes are required to add `sudachi_projection` option to `BertJapaneseTokenizer`, and the models that do not specify `sudachi_projection` option can be used as before in environments using older versions of SudachiPy.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- text models: @ArthurZucker and @younesbelkada

Comments:
Example model for sudachi_projection option:
https://huggingface.co/hiroshi-matsuda-rit/bert-base-sudachitra-v11/blob/main/tokenizer_config.json#L16

Nice 🔥 Feel free to ping me for a review! 

The errors which were raised from the test cases for jumanpp and mecab-unidic in local environment were not reproduced in circle-ci test environment, so I removed following section from the top description of this PR.

> As a side note, some errors have been reported from the test cases for mecab-unidic and jumanpp when turning on `RUN_CUSTOM_TOKENIZERS`.
> I would also like to update these outdated test cases in this PR if required.
```console
$ RUN_CUSTOM_TOKENIZERS=1 pytest -rfs tests/models/bert_japanese/
...
======================================================================================= short test summary info =======================================================================================
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer - AssertionError: Lists differ: ['アップ[28 chars]', '\\ ', 'が', '\\ \\ ', '\\ ', '発売', 'さ', 'れた', '\\ ', '。'] != ['アップ[28 chars]', '\u3000', 'が', '\u3000', '\u3000', '\u3000'[28 chars]...
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer_ext - AssertionError: Lists differ: ['ありがとう', 'ございます', 'm', '(', '_', '\\ _', ')', 'm', '見つける', 'の[15 chars] '。'] != ['ありがとう', 'ございます', 'm(_ _)m', '見つける', 'の', 'が', '大...
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer_lower - AssertionError: Lists differ: ['アップ[28 chars]', '\\ ', 'が', '\\ \\ ', '\\ ', '発売', 'さ', 'れた', '\\ ', '。'] != ['アップ[28 chars]', '\u3000', 'が', '\u3000', '\u3000', '\u3000'[28 chars]...
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer_no_normalize - AssertionError: Lists differ: ['ｱ',[45 chars]', '\\ ', 'が', '\\ \\ ', '\\ ', '発売', 'さ', 'れた', '\u3000', '。'] != ['ｱ',[45 chars]', '\u3000', 'が', '\u3000', '\u3000', '\u3000'[28 chars] '。']
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_jumanpp_tokenizer_trim_whitespace - AssertionError: Lists differ: ['アップ[21 chars]ne', '8', '\\', 'が', '\\ \\', '\\', '発売', 'さ', 'れた', '\\', '。'] != ['アップ[21 chars]ne', '8', 'が', '発売', 'さ', 'れた', '。']
FAILED tests/models/bert_japanese/test_tokenization_bert_japanese.py::BertJapaneseTokenizationTest::test_mecab_tokenizer_unidic - RuntimeError: The unidic dictionary itself is not found. See https://github.com/polm/unidic-py for installation.
SKIPPED [2] tests/test_tokenization_common.py:2394: test is PT+TF test
SKIPPED [2] tests/test_tokenization_common.py:2534: test is slow
SKIPPED [2] tests/test_tokenization_common.py:2499: test requires TensorFlow
SKIPPED [2] tests/test_tokenization_common.py:2448: test is slow
======================================================================== 6 failed, 193 passed, 8 skipped, 7 warnings in 4.52s =========================================================================
```

@ArthurZucker All tests passed! Please review and merge this PR.

> Let's add a small test where you pass sudachi_kwargs to a BertJapaneseTokenizer to make sure this is usable

I added some test cases which pass kwargs to word tokenizers.

> I think we need a sudachi version check for this feature then!

Thanks for your suggestion. Let me explain the background.

I've been developing a Japanese Dependency parser GiNZA utilizing spacy-transformers and sudachipy, and its [ELECTRA model](https://huggingface.co/megagonlabs/transformers-ud-japanese-electra-base-ginza-520/tree/main) has become widely used for Japanese text analysis in recent years.
I'm wondering that certain users of GiNZA ELECTRA model may hope to keep sudachipy as is but they need to upgrade transformers to use newer models together with GiNZA.

This is the reason why I wrote the if else clause.

@ArthurZucker  Finally, I decided to pass projection arg to Dictionary.create() directly.
I would like to inform GiNZA users to update transformers and sudachipy at the same time.

@ArthurZucker What should I do for this type of errors? It seems I do not have a permission to rerun the failed actions.

> FAILED tests/test_pipeline_mixin.py::ImageSegmentationPipelineTests::test_small_model_pt - requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Yes if else is alright don't worry! Just wanted to make sure someone that tries to use the new feature with an older version would get an error saying he needs to update the sudachi version to use it! 

I implemented `is_sudachi_projection_available()` and `@require_sudachi_projection` to check the sudachipy version.
Does this additional implementation meet your requirements? @ArthurZucker 

Reviewing! 

@ArthurZucker Thanks! I just revised the changes as per your instructions.

@ArthurZucker Thank you. I have completed the fix.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28503). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.