SudachiPy: A Japanese Morphological Analyzer in Python | by Ng Wai Foong | Towards Data Science

Python version of Sudachi that focuses on Natural Language Processing (NLP) for Japanese text
This article covers the basic functionality of SudachiPy which can be used to perform simple natural language processing tasks such as tokenization for Japanese language. Based on the official Github page:
Sudachi & SudachiPy are developed in WAP Tokushima Laboratory of AI and NLP, an institute under Works Applications that focuses on Natural Language Processing (NLP).
The original version is in Java while SudachiPy is the python implementation of it. Having said that, some functions are still incompatible with Java Sudachi. This article consists of 4 main sections:
SetupCommand LineBasic UsageConclusion
Setup
Command Line
Basic Usage
Conclusion
Let’s get started by creating a virtual environment and activate it. Once you are done, run the following command in the terminal:
pip install SudachiPy
This will install the latest version of SudachiPy which is 0.3.11 at the time of this writing. SudachiPy‘s version that is higher that 0.3.0 refers to system.dic of SudachiDict_core package by default. This package is not included in SudachiPy and have to be installed manually.
You can install the package by running the following command:
pip install https://object-storage.tyo2.conoha.io/v1/nc_2520839e1f9641b08211a5c85243124a/sudachi/SudachiDict_core-20190718.tar.gz
It will automatically install the default dictionary. Let’s jump to the next section once you are ready. The paragraph below are meant for setting your own custom dictionary.
Sudachi provides us with 3 different kind of dictionaries
Small: includes only the vocabulary of UniDic
Core: includes basic vocabulary (default)
Full: includes miscellaneous proper nouns
You should be able to download a system.dic file. You can modify the dictionary to any built-in or custom dictionary.
Path modification: Create a sudachi.json and place it anywhere as you like. By default, the package contains this file at the following directory.
<virtual_env>/Lib/site-packages/sudachipy/resources/
You can choose to edit the file or create a new one and paste the following code:
Modify the systemDict to the relative path of systemDict to point to the system.dic file.
Direct Replacement: Go to the following directory:
<virtual_env>/Lib/site-packages/sudachidict_core/resources
You should be able to see just one file called system.dic
Replace the file with your custom dictionary or the one that you downloaded from official site.
If you are using Windows OS and experience issue with OSError: symbolic link privilege not held. Kindly run your terminal as administrator. Then, type the following and run it:
sudachipy link -t core
SudachiPy also provides us with a few commands line. You can straightaway use it once you have installed the SudachiPy and SudachiDict_core modules. I will just use it to test whether we have installed the modules correctly. There are 4 main usable commands:
tokenize
link
build
ubuild (contains bug when using version 0.3.*)
Open up a terminal and run the following command:
sudachipy tokenize -h
You should be able to see the following result.
If you encounter any issue, make sure that you have correctly install both the SudachiPy and SudachiDict_core modules.
First, let’s import the tokenizer and dictionary module.
from sudachipy import tokenizerfrom sudachipy import dictionary
Both of the modules are required to create a tokenizer object. Continue to add the following code:
tokenizer_obj = dictionary.Dictionary().create()
Next up, we will need to define the mode for the tokenizer. Mode is used to determine how the tokenizer should split the text.
A: texts are divided into the shortest units equivalent to the UniDic short unit
B: split the text into middle units between A and C
C: extracts named entities
Let’s test it out by setting it to mode A:
mode = tokenizer.Tokenizer.SplitMode.A
Create a new variable and assign any Japanese text. Then, call the tokenize function in a list comprehension as follow:
txt = "医薬品安全管理責任者"print([m.surface() for m in tokenizer_obj.tokenize(txt, mode)])
You should be able to obtain the following results:
['医薬', '品', '安全', '管理', '責任', '者']
You can test it out to see the results for all three output as follow:
from sudachipy import tokenizerfrom sudachipy import dictionarytokenizer_obj = dictionary.Dictionary().create()txt = "医薬品安全管理責任者"mode = tokenizer.Tokenizer.SplitMode.Aprint([m.surface() for m in tokenizer_obj.tokenize(txt, mode)])mode = tokenizer.Tokenizer.SplitMode.Bprint([m.surface() for m in tokenizer_obj.tokenize(txt, mode)])mode = tokenizer.Tokenizer.SplitMode.Cprint([m.surface() for m in tokenizer_obj.tokenize(txt, mode)])
Japanese language is known for the complicated verb conjugation. SudachiPy also provides a way for us to transform them freely via built-in functions. We will just going to explore three main functions that we can access after splitting the text via the tokenizer obj.
The first one is the original text which can be called using the surface() function
print([m.surface() for m in tokenizer_obj.tokenize(txt, mode)])
Apart from that, we also have the dictionary_form() function
print([m.dictionary_form() for m in tokenizer_obj.tokenize(txt, mode)])
The last one is the normalized_form() function
print([m.normalized_form() for m in tokenizer_obj.tokenize(txt, mode)])
Let’s combine them together to see the difference:
from sudachipy import tokenizerfrom sudachipy import dictionarytokenizer_obj = dictionary.Dictionary().create()txt = "ゲームデータを消失してしまいました。"mode = tokenizer.Tokenizer.SplitMode.Aprint([m.surface() for m in tokenizer_obj.tokenize(txt, mode)])print([m.normalized_form() for m in tokenizer_obj.tokenize(txt, mode)])print([m.dictionary_form() for m in tokenizer_obj.tokenize(txt, mode)])
You should be able to get the following results:
['ゲーム', 'データ', 'を', '消失', 'し', 'て', 'しまい', 'まし', 'た', '。']['ゲーム', 'データ', 'を', '消失', '為る', 'て', '仕舞う', 'ます', 'た', '。']['ゲーム', 'データ', 'を', '消失', 'する', 'て', 'しまう', 'ます', 'た', '。']
One of the most important feature in any Natural Language Processing is the part of speech tag. We can easily get it via the part_of_speech() function:
print([m.part_of_speech() for m in tokenizer_obj.tokenize(txt, mode)])
The output result is as follow:
[['名詞', '普通名詞', '一般', '*', '*', '*'], ['名詞', '普通名詞', '一般', '*', '*', '*'], ['助詞', '格助詞', '*', '*', '*', '*'], ['名詞', '普通名詞', 'サ変可能', '*', '*', '*'], ['動詞', '非自立可能', '*', '*', 'サ行変格', '連用形-一般'], ['助詞', '接続助詞', '*', '*', '*', '*'], ['動詞', '非自立可能', '*', '*', '五段-ワア行', '連用形-一般'], ['助動詞', '*', '*', '*', '助動詞-マス', '連用形-一般'], ['助動詞', '*', '*', '*', '助動詞-タ', '終止形-一般'], ['補助記号', '句点', '*', '*', '*', '*']]
Let’s recap on what we have learned today. We started out by installing the necessary python modules, SudachiPy and SudachiDict_core. There is also an option to configure it to load your own custom dictionary.
Then, we explored some basic commands that can be called in the terminal to test out the installation.
Next, we dive deep into the basic python codes to learn to tokenize the Japanese text. Besides, we can get the output tokens in dictionary or normalized form. Not to mention the part_of_speech() function that can help use to identify the POS tag for each of the tokens.
Thanks for reading and have a good day. See you in the next article.