MeCab Usage and Add User Dictionary to MeCab
By: Xu Liang
Tags: NLP, Japanese, Programming, Language, Linguistics

If you are an NLP engineer working in Japan, you mush have heard the text segmentation library, MeCab. This library is widely used in both academia and industry in Japan. And if you are a non-native Japanese speaker, it might be difficult to use this tool because the documents are mostly written by Japanese. So I write this tutorial to ease the burden.

Installation

Environment:

- macOS Mojave 10.14.6

- Xcode: 11.0

- python 3

- pip 19.1

Install MeCab and the dictionary. The mecab-python3 won’t work without these dependencies.

$ brew install mecab

$ brew install mecab-ipadic

You can run the mecab command to try the interactive interface

$ mecab

おはようございます

おはよう 感動詞,*,*,*,*,*,おはよう,オハヨウ,オハヨー

ござい 助動詞,*,*,*,五段・ラ行特殊,連用形,ござる,ゴザイ,ゴザイ

ます 助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス

EOS

Install SWIG. We have to install this library, otherwise, there would be errors when installing mecab-python3.

$ brew install swig $ swig -version SWIG Version 4.0.1 Compiled with clang++ [x86_64-apple-darwin18.6.0] Configured options: +pcre Please see http://www.swig.org for reporting bugs and further information

Finally, install the mecab-python3.

$ pip install mecab-python3

Downloading

|████████████████████████████████| 14.1MB 2.4MB/s

Installing collected packages: mecab-python3

Successfully installed mecab-python3-0.996.2 Collecting mecab-python3Downloading https://files.pythonhosted.org/packages/97/9f/3e5755e0488f608e3c2d18a0f3524434ebf36904b8fd4eec74a3e84416a9/mecab_python3-0.996.2-cp36-cp36m-macosx_10_6_intel.whl (14.1MB)|████████████████████████████████| 14.1MB 2.4MB/sInstalling collected packages: mecab-python3Successfully installed mecab-python3-0.996.2

The installable is finished. We can write a script to give a try.

import MeCab

mecab = MeCab.Tagger("-Ochasen") # Create a MeCab object

malist = mecab.parse("ＮＥＣが二位、東芝がモトローラを抜いて二年ぶりに三位になる。") # morphological analysis

print(malist) # output

ＮＥＣ エヌイーシー ＮＥＣ 名詞-固有名詞-組織

が ガ が 助詞-格助詞-一般

二 ニ 二 名詞-数

位 イ 位 名詞-接尾-助数詞

、 、 、 記号-読点

東芝 トウシバ 東芝 名詞-固有名詞-組織

が ガ が 助詞-格助詞-一般

モトローラ モトローラ モトローラ 名詞-固有名詞-組織

を ヲ を 助詞-格助詞-一般

抜い ヌイ 抜く 動詞-自立 五段・カ行イ音便 連用タ接続

て テ て 助詞-接続助詞

二 ニ 二 名詞-数

年 ネン 年 名詞-接尾-助数詞

ぶり ブリ ぶり 名詞-接尾-一般

に ニ に 助詞-格助詞-一般

三 サン 三 名詞-数

位 イ 位 名詞-接尾-助数詞

に ニ に 助詞-格助詞-一般

なる ナル なる 動詞-自立 五段・ラ行 基本形

。 。 。 記号-句点

EOS

Add User Dictionary

According to the official document, there are 2 ways to add a customized dictionary, change the “system dictionary” and add a “user dictionary”. If we change the “system dictionary”, we have to compile the dictionary and install it again. So we choose to add a new “user dictionary”.

Determine the format

First, we have to write the custom dictionary to a CSV file, the format looks like below.

表層形,左文脈ID,右文脈ID,コスト,品詞,品詞細分類1,品詞細分類2,品詞細分類3,活用型,活用形,原形,読み,発音 Surface type, left context ID, right context ID, cost, part of speech, sub POS 1, sub-POS 2, sub-POS 3, conjugation type, conjugation form, original form, ruby, pronunciation

Each line contains 13 features. For example, if we want to add noun words (活用しない語), like location, people’s name, we could write like below.

工藤,1223,1223,6058,名詞,固有名詞,人名,名,*,*,くどう,クドウ,クドウ

But if we want to add adjectives or verb (活用する語), we have to add all their conjugation type (活用型) and conjugation forms (活用形). See the below example for the adjective “いそがしい”.

いそがしい,120,120,6078,形容詞,自立,*,*,形容詞・イ段,基本形,いそがしい,イソガシイ,イソガシイ

いそがし,128,128,6080,形容詞,自立,*,*,形容詞・イ段,文語基本形,いそがしい,イソガシ,イソガシ

いそがしから,136,136,6079,形容詞,自立,*,*,形容詞・イ段,未然ヌ接続,いそがしい,イソガシカラ,イソガシカラ

いそがしかろ,132,132,6079,形容詞,自立,*,*,形容詞・イ段,未然ウ接続,いそがしい,イソガシカロ,イソガシカロ

いそがしかっ,148,148,6078,形容詞,自立,*,*,形容詞・イ段,連用タ接続,いそがしい,イソガシカッ,イソガシカッ

いそがしく,152,152,6078,形容詞,自立,*,*,形容詞・イ段,連用テ接続,いそがしい,イソガシク,イソガシク

いそがしくっ,152,152,6079,形容詞,自立,*,*,形容詞・イ段,連用テ接続,いそがしい,イソガシクッ,イソガシクッ

いそがしゅう,144,144,6079,形容詞,自立,*,*,形容詞・イ段,連用ゴザイ接続,いそがしい,イソガシュウ,イソガシュウ

いそがしゅぅ,144,144,6079,形容詞,自立,*,*,形容詞・イ段,連用ゴザイ接続,いそがしい,イソガシュゥ,イソガシュゥ

いそがしき,124,124,6079,形容詞,自立,*,*,形容詞・イ段,体言接続,いそがしい,イソガシキ,イソガシキ

いそがしけれ,108,108,6079,形容詞,自立,*,*,形容詞・イ段,仮定形,いそがしい,イソガシケレ,イソガシケレ

いそがしかれ,140,140,6079,形容詞,自立,*,*,形容詞・イ段,命令ｅ,いそがしい,イソガシカレ,イソガシカレ

いそがしけりゃ,112,112,6079,形容詞,自立,*,*,形容詞・イ段,仮定縮約１,いそがしい,イソガシケリャ,イソガシケリャ

いそがしきゃ,116,116,6079,形容詞,自立,*,*,形容詞・イ段,仮定縮約２,いそがしい,イソガシキャ,イソガシキャ

いそがし,104,104,6080,形容詞,自立,*,*,形容詞・イ段,ガル接続,いそがしい,イソガシ,イソガシ

Long vowel

Below is a demonstration of the long vowel phenomenon.

今日,,,10,名詞,副詞可能,*,*,*,*,今日,キョウ,キョー

労働,,,10,名詞,サ変接続,*,*,*,*,労働,ロウドウ,ロードー

“今日” and “労働” are the surface form. “10” is the cost. Lower cost means this word has a high frequency in the corpus. “名詞” is the POS tag. “一般” is the sub-POS tag, which can be ignored. The final three columns represent “original form”, “ruby”, and “pronunciation”. The ruby character means how a word should be pronounced. But in real conversation, people usually ignore the vowel letters (a e i o u, アイウエオ in Japanse) to pronounce a word easier. This phenomenon usually occurs with long vowels.

In the below example, “ウ” will be ignored.

キョウ -> キョー

ロウドウ -> ロードー

The Romanji of “今日” is “kyou”. The “o” and “u” are both vowels, so this is a long vowel. When we pronounce “今日”, “u” will be ignored. So the pronunciation becomes “kyo” and “o” produce as a long vowel, /uː/.

Use CSV file to augment the dictionary

For the following demonstration, I build a foo.csv file and input some nouns.

二位,,,10,名詞,一般,*,*,*,*,*,二位,ニイ,ニイ

三位,,,10,名詞,一般,*,*,*,*,*,三位,サンイ,サンイ

Next we need to compile this CSV file.

Mac(installed with Homebrew)

$ cd /home/foo/bar # the path where the foo.csv exists

$ /usr/local/Cellar/mecab/0.996/libexec/mecab/mecab-dict-index \

-d /usr/local/lib/mecab/dic/ipadic \

-u foo.dic \

-f utf-8 \

-t utf-8 foo.csv

reading foo.csv ... 2

emitting double-array: 100% |###########################################|

options:

d: directory of system dictionary

u: the name of the output user dictionary

f: encoding of CSV file

t: encoding of the output user dictionary file

Make sure the user dictionary is created successfully.

$ ls

foo.csv foo.dic

Move the foo.dic to a new directory.

$ mkdir /usr/local/lib/mecab/dic/user_dict

$ mv foo.dic /usr/local/lib/mecab/dic/user_dict

Next, we need to add the path to mecabrc file.

vim /usr/local/etc/mecabrc

Change as below

; userdic = /home/foo/bar/user.dic

->

userdic = /usr/local/lib/mecab/dic/user_dict/foo.dic

If you have multiple user dictionary file, you could write them in one line.

userdic = /usr/local/lib/mecab/dic/user_dict/foo.dic, /usr/local/lib/mecab/dic/user_dict/foo2.dic, /usr/local/lib/mecab/dic/user_dict/foo3.dic

Ok, that’s all.

Try the user dictionary

$ echo "ＮＥＣが二位、東芝がモトローラを抜いて二年ぶりに三位になる。" | mecab ＮＥＣ 名詞,固有名詞,組織,*,*,*,ＮＥＣ,エヌイーシー,エヌイーシー

が 助詞,格助詞,一般,*,*,*,が,ガ,ガ

二位 名詞,一般,*,*,*,*,*,二位,ニイ,ニイ

、 記号,読点,*,*,*,*,、,、,、

東芝 名詞,固有名詞,組織,*,*,*,東芝,トウシバ,トーシバ

が 助詞,格助詞,一般,*,*,*,が,ガ,ガ

モトローラ 名詞,固有名詞,組織,*,*,*,モトローラ,モトローラ,モトローラ

を 助詞,格助詞,一般,*,*,*,を,ヲ,ヲ

抜い 動詞,自立,*,*,五段・カ行イ音便,連用タ接続,抜く,ヌイ,ヌイ

て 助詞,接続助詞,*,*,*,*,て,テ,テ

二 名詞,数,*,*,*,*,二,ニ,ニ

年 名詞,接尾,助数詞,*,*,*,年,ネン,ネン

ぶり 名詞,接尾,一般,*,*,*,ぶり,ブリ,ブリ

に 助詞,格助詞,一般,*,*,*,に,ニ,ニ

三位 名詞,一般,*,*,*,*,*,三位,サンイ,サンイ

に 助詞,格助詞,一般,*,*,*,に,ニ,ニ

なる 動詞,自立,*,*,五段・ラ行,基本形,なる,ナル,ナル

。 記号,句点,*,*,*,*,。,。,。

EOS

We can see the “二位” and “三位” are recognized as a word.

If I want to know the cost, we could change the output format. %m is the surface form, %c is the cost, and %H is a comma-delimited list of POS, conjugation, reading, etc. You can find more detail in these documents, EN version, and JP version.

$ echo "NECが二位、東芝がモトローラを抜いて二年ぶりに三位になる。" | mecab -F '%m %c %H

' NEC 13835 名詞,固有名詞,組織,*,*,*,*

が 3866 助詞,格助詞,一般,*,*,*,が,ガ,ガ

二位 10 名詞,一般,*,*,*,*,*,二位,ニイ,ニイ

、 -2435 記号,読点,*,*,*,*,、,、,、

東芝 1426 名詞,固有名詞,組織,*,*,*,東芝,トウシバ,トーシバ

が 3866 助詞,格助詞,一般,*,*,*,が,ガ,ガ

モトローラ 4897 名詞,固有名詞,組織,*,*,*,モトローラ,モトローラ,モトローラ

を 4183 助詞,格助詞,一般,*,*,*,を,ヲ,ヲ

抜い 7346 動詞,自立,*,*,五段・カ行イ音便,連用タ接続,抜く,ヌイ,ヌイ

て 5170 助詞,接続助詞,*,*,*,*,て,テ,テ

二 2914 名詞,数,*,*,*,*,二,ニ,ニ

年 8465 名詞,接尾,助数詞,*,*,*,年,ネン,ネン

ぶり 7451 名詞,接尾,一般,*,*,*,ぶり,ブリ,ブリ

に 4304 助詞,格助詞,一般,*,*,*,に,ニ,ニ

三位 10 名詞,一般,*,*,*,*,*,三位,サンイ,サンイ

に 4304 助詞,格助詞,一般,*,*,*,に,ニ,ニ

なる 5063 動詞,自立,*,*,五段・ラ行,基本形,なる,ナル,ナル

。 215 記号,句点,*,*,*,*,。,。,。

EOS

Reference