Update dependency mecab to at least version 1.0
Author: BramVanroy
Pull request: yes

Require at least version 1.0 of mecab, which comes with prebuilt wheels for all major platforms (OS X, Linux, Windows). This should do away with some reported incompatibility issues for non-linux systems.

See https://github.com/SamuraiT/mecab-python3/issues/31#issuecomment-651053281

Note: code untested but nothing much to test. [Version 1.0.0 is on PyPi](https://pypi.org/project/mecab-python3/1.0.0/) so it should work as written.

Comments:
No, I pinned this on purpose because v1.0 has breaking changes. Tokenization is not done the same way (once you go through the obvious bugs) because the underlying dictionary seems different. Not sure how we can go to v1.0 until pretrained models using this tokenizer have been updated (if that's ever done).

Could you give more information so that I can post a new issue over at mecab? (Or even better, if you have the time, post the issue yourself?)

Well the breaking changes are numerous.
First, the library does not provide a dictionary again and requires to install another dependency:
```
pip install unidic-lite
```
Once the installation problem is passed,  we need to fix the tokenizer by changing [this line](https://github.com/huggingface/transformers/blob/482a5993c20ef32512a42661bfa404516763b72e/src/transformers/tokenization_bert_japanese.py#L207) to `token = line.split("\t")[0]` because the tokenizer now returns multiple tab-separated things.

Once this step is done, the [test_mecab_tokenizer](https://github.com/huggingface/transformers/blob/482a5993c20ef32512a42661bfa404516763b72e/tests/test_tokenization_bert_japanese.py#L90) still fails because the tokenization of " \tｱｯﾌﾟﾙストアでiPhone８ が  \n 発売された　。  " is not ["アップルストア", "で", "iPhone", "8", "が", "発売", "さ", "れ", "た", "。"] anymore but ['アップル', 'ストア', 'で', 'iPhone', '8', 'が', '発売', 'さ', 'れ', 'た', '。'].

And I did not look at other tests yet, since that tokenization change alone is enough to warrant a pin on older releases, given the fact it will break models pretrained with that tokenizer.

(Note that the mecab dependency is only used for these tests anyway and that a Windows user calling pytest on the tests folder will have no issue since the tokenizer tests are ignored by default.)


Gotcha. I would like to see the move to v1 in the future though, for the reasons mentioned before. Perhaps this is something to plan for transformers v4? That would give users plenty of time still to use their current models, and in having a new major version it is no surprise there can be compatibility changes.

Hello, I'm the mecab-python3 maintainer. Please feel free to tag me on any discussions about this if there's a way I can help.

I would strongly suggest switching to the new dictionary when you can - the dictionary you currently use, ipadic, hasn't been updated since at least 2007 and will never be updated again (the organization that created it no longer does that kind of work). In contrast Unidic is maintained by the NINJAL, the organization in charge of Japanese Universal Dependencies.

Hi @polm, thanks for getting involved! So from an accuracy point of view, the new dictionary is better than the previous version - presumably? The reason for updating, then, would be: better tokenisation, cross-platform pip install support. Downsides are that models that are already trained with the old tokenisation will behave differently if they use a different tokenizer. That's why I suggest to make the breaking change in v4 which is probably still some months away. If people want to use older models, they can always use an older version of mecab.

@sgugger I don't know if this is possible, but perhaps the model cards should include a section on which version of the library and/or dependencies (pip freeze) were used to train the model. This would also greatly help reproducibility issues.

ipadic was already pretty accurate, but yes, the new dictionary (Unidic) will at the very least be more accurate by virtue of having newer words, like 令和 (Reiwa), the current era name. You can read more about the different dictionaries [here](https://www.dampfkraft.com/nlp/japanese-tokenizer-dictionaries.html). 

> @sgugger I don't know if this is possible, but perhaps the model cards should include a section on which version of the library and/or dependencies (pip freeze) were used to train the model. This would also greatly help reproducibility issues.

This is also very important. One of my objectives in packaging unidic in pip was to make it easier to use the same dictionary version between projects.

AFAICT having a model on the Hub that doesn't work properly across all versions of the library would be a first, and set a dangerous precedent. IMO moving to mecab >= 1 requires to retrain the model on the hub with the new tokenizer and pulling out the old ones, as I've stated before. I don't know if @thomwolf or @julien-c feel differently.

IMO in the future the library shouldn't prescribe what tokenizer to use, i.e. the model cards from @singletongue at https://huggingface.co/cl-tohoku could let users know (or maybe even programmatically check) that they need a specific version of `mecab<1.0`, and we would remove `mecab` from the dependencies (users can install it if they need it)

Same for sacremoses and sentencepiece which shouldn't be hard dependencies of the library anymore.

In the short term, I'm ok with pinning the dependency to <1.

Thoughts?

@julien-c Yes, I agree. This joins together what I wrote before: the ability/requirement for model cards to indicate some environment variables/dependencies that were used. So for instance, this could be part of the model card in a "dependency" section:

```
mecab==1.0.0
transformers>=2.8.1
```


Then we just need to decide which version we use for the tests (since we can't have both mecab < 1 and mecab >= 1 at the same time.

Just a note, I am working on packaging ipadic the same way I packaged Unidic after several requests for backwards compatability, including from sacrebleu. That should be released in the next few weeks, see [this issue](https://github.com/SamuraiT/mecab-python3/issues/49). That way you'll get backward compatability and the better wheels in 1.0+. 

I released ipadic on PyPI. It works with the latest mecab-python3. You can use it like this:

```
import MeCab
import ipadic
tagger = MeCab.Tagger(ipadic.MECAB_ARGS)
print(tagger.parse("図書館にいた事がバレた"))
```

I hope you'll change to Unidic as soon as is feasible anyway, but this way you can use the recent version of mecab-python3 with the old models.

I believe that since #6086 has been merged this can be closed.

Yes it can.