How Japanese Tokenizers Work. A deep dive into Japanese tokenization... | by Wanasit Tanakitrungruang | Towards Data Science

I recently have been working on Kotori and doing research on Japanese NLP tools. I’m writing this article to share what I have learned so far about Japanese tokenization and its components.
If you haven’t heard of it, Kotori is a Japanese tokenizer and morphological analysis engine written in Kotlin. Please check Kotori on Github and give it some stars if you like this article.
Tokenization, or breaking a text into a list of words, is an important step before other NLP tasks (e.g. text classification). In English, words are often separated by whitespace or other symbols, which make it easier to tokenize. However, in Japanese, words are normally written without any space between. Japanese tokenization requires reading/analyzing the whole sentence, recognizing words, and determining word boundaries without any explicit delimiters.
Most Japanese tokenizers use Lattice-based tokenization. As the name suggested, a lattice-based tokenizer builds a lattice (or a graph-like data structure) consisting of all possible tokens (terms or substrings) that surface on the input text. It uses the Viterbi algorithm to find the best connected-path through the lattice.
The most popular lattice-based tokenizer is MeCab (written in C++). Most open-source Japanese tokenizer libraries are either simply MeCab’s wrapper or re-implementation of Lattice-based tokenization in different platforms.
For examples:
Kotori in Kotlin
Sudachi and Kuromoji in Java
Janome and SudachiPy in Python
Kagome in Go.
MeCab’s IPA dictionary (IPADIC) is also the most popular dictionary. It is used as a baseline or primary dictionary for most tokenizers.
In this article, I’ll explain in detail how the Lattice-based dictionary and tokenization works through a simplified version of Kotori’s code and MeCab’s references.
For Lattice-based tokenizers, a dictionary is an object or a data structure that provides information about available terms as well as how those terms should appear next to each other according to Japanese grammar or the probability.
Although different tokenizer libraries use different in-memory data structures and read different dictionary file formats, all Lattice-based tokenizer dictionaries share mostly the same interface or logical structure.
In this article, we will look into MeCab’s dictionary format. Specifically, MeCab’s IPADIC, which you can download here (to see the content and compare with the note for yourself).
A tokenizer dictionary consists of two important parts:
Term-Dictionary (単語辞書 or Lexicon) is a list of known terms (or words). In MeCab’s dictionary directory, all CSV files together make up this term dictionary part.
For example, in Noun.place.csv, you can find rows such as:
東京,1293,1293,3003,名詞,固有名詞,地域,一般,*,*,東京,トウキョウ,トーキョー京都,1293,1293,2135,名詞,固有名詞,地域,一般,*,*,京都,キョウト,キョート札幌,1293,1293,3440,名詞,固有名詞,地域,一般,*,*,札幌,サッポロ,サッポロ渋谷,1293,1293,4310,名詞,固有名詞,地域,一般,*,*,渋谷,シブヤ,シブヤ
Each term (or row) consists of multiple columns (or fields), but only the first four columns are important to the tokenization.
Surface Form (表層形): The string or term that should appear in the text.
Left/Right Context ID (左/右-文脈ID): This is used for determining connection cost or how likely it is for a term to follow other terms.
Cost: The cost of having this term. The cost depends on how likely it is for the term to be used. The higher the cost, the less frequently used or rarer is the term.
The remaining columns are additional features about the terms (e.g. part-of-speech, normalized form, pronunciation, ..., etc)
Notes:
A term-dictionary in practice is large and comprehensive. It often contains hundreds of thousands of terms (e.g. around 290,000 terms in IPADIC)
Please don’t confuse Dictionary (辞書) with Term-Dictionary (単語辞書). Throughout this article, I will use the term-dictionary to mean the table of terms part, while the dictionary would refer to the whole (term-dictionary + connection cost + others).
Connection Cost (連接コース) is a cost structure to measure how likely it is for a term to be before or after one another.
The connection likelihood is explained as a cost of connecting the right context ID of the previous word with the left context ID of the following word.
(Prev term's rightID, Next term's leftID) -> Cost
Similar to the term’s cost, the higher the connection cost, the less likely the given terms should be connected.
This mapping is matrix.def file in MeCab’s dictionary. The file structure is pretty straight-forward. The first line is the number of possible rigthIDs and leftIDs (both are 1316 in the example) follow by the list of (rightID, leftID, cost) tuples.
1316 13160 0 -4340 1 10 2 -16300 3 -1671...1315 1313 -43691315 1314 -17121315 1315 -129
In addition to Term-Dictionary and Connection Cost, another dictionary component that is crucial to tokenization in practice is an unknown term (or out-of-vocabulary) handling strategy. It is for recognizing some words, substrings, or symbols outside of term-dictionary (e.g. people names) and allows tokenizers to skip or tokenize them.
In MeCab, unknown term handling is done by heuristic rules based-on character types (e.g. consecutive Katakana characters is probably a noun). The rules and character types are defined in unk.def and char.def files.
In this article, I am leaving the details of unknown word handling and assume that term-dictionary includes every term in the text.
As mentioned, Lattice-based tokenization is building a lattice of terms and find the best path through the lattice.
Given the dictionary, the tokenizer identifies all terms that “surface” on the text as sub-string, add all of them into the lattice together with their substring starting/ending location, then runs the algorithm to find the best path of connected terms.
The simplified code would look like this:
val lattice = Lattice.create()for i in 0..text.length {    val terms = findAllTermStartingAt(text, i)    for term in terms {        lattice.add(term, startIndex=i, endIndex=i+term.length)    }    ...}...return lattice.findTheBestPath()
In the example code, we use findAllTermStartingAt(..) to find all terms in the dictionary having surface form starting at the location.
For example, if text="東京都に住む":
findAllTermStartingAt(text, 0) should return 東 (East), 東京 (Tokyo)
findAllTermStartingAt(text, 1) should return 京 (Capital), 京都 (Kyoto)
...
Note:
Having a fast findAllTermStartingAt(..) is very important to the tokenizer performance. Thus, the function is often implemented by efficient string matching/lookup data structure, e.g. Trie or Finite-State Transducer (FST). I’m leaving the implementation details of those data structures for future articles.
In the shown version, we identify possible terms and save all of them in memory before running the algorithm to finding the best path once at the end. There are tokenizers that use a greedy or window-based approach to actively prune out unnecessary terms to keep memory usage low. Kotori uses the shown approach because I found it is more efficient (aka. faster) and memory usage has never been a problem in practice.
The Lattice is a graph structure containing all surfaced terms as nodes. The edges of the graph connect nodes (or terms) starting at each location with nodes ending at the same location.
For example, 東 (ending at index=1) connected to 京 and 京都 (starting at index=1).
Each node and connection will have a cost associate with it:
The cost of a node (orange-colored) is its term’s cost (as defined in term-dictionary)
The cost of a connection (red-colored) is its connection cost between the ending term’s right context ID and the starting term’s left context ID (as defined in connection-cost).
Note from the example that, in addition to the nodes from surfaced terms, we need to include two special nodes into the lattice.
A special Start node index=0, rightID=0, cost=0
A special End node index=text.length, leftID=0, cost=0
Both term and connection costs represent how commonly used they are. The lower the cost the more likely the terms and connections. Thus, the tokenizer's goal is to find the minimal cost path (or the ‘best’ path) from start to end. Doing so will result in the token sequence that is the most likely in Japanese.
Viterbi Algorithm is an algorithm to find the optimal path (or most likely path, or minimal cost path, etc) through the graph. Most Viterbi algorithm examples come from its application with Hidden Markov Model (e.g. POS tagging). However, I found the Viterbi algorithm usage in tokenization is very different.
My explanation of the Viterbi Algorithm for Lattice-based tokenization application is as follows:
To find the minimal path from start to tth node, we need to consider paths through each (t-1)th node leading to tth node by calculating each path’s total cost and pick the minimal path among them.
For each (t-1)th node leading to tth node:
(Recursively) Find the minimal cost/path from start to that (t-1)th node.
Find the cost of connecting that (t-1)th node to tth node
Find the cost of tth node itself
Adding them all together. This is the total cost from start to tth node through the (t-1)th node
Then, among the paths through different (t-1)th nodes to tth node, pick the path with the minimal total cost.
An efficient way to implement this algorithm is dynamic programming. We can start finding the minimal path from the nodes closet to the start and reuse their minimal path to the nodes following them. Keep repeating until we calculate the minimal path for each node.
for i in 0..text.length {  for node in nodes.startingAt(i) {    for prevNode in nodes.endingAt(i) {      val prevTotalCost = prevNode.totalCost      val termCost = node.cost      val connectionCost = getConnectionCost(prevNode.rightId, node.leftId)      val totalCost = prevTotalCost + connectionCost + termCost            if (totalCost < node.totalCost) {                 node.totalCost = totalCost        node.prevNode = prevNode      }    }  }}
After executing the algorithm, the minimal path from Start to End can be found that the End node. We simply need to follow the prevNode pointer back to the Start node.
This article summarizes the knowledge I learned from building a Japanese tokenizer from scratch. Again, if you like this article or what to learn more about Japanese tokenization, please checkout Kotori on Github.
Note: Kotori has been heavily influenced by other open-source libraries. Special thanks to Atilika (Kuromoji), WorkApplicaitons (Sudachi), and Tomoko Uchida (Janome).
Other resources (in Japanese):
Manabu Okumura, 自然言語処理の基礎 (book)
Cookpad Developers’ Blog, 日本語形態素解析の裏側を覗く!MeCab はどのように形態素解析しているか