How Japanese Tokenizers Work
By: Wanasit Tanakitrungruang
Tags: NLP, Algorithms, Programming, Tokenization, Japanese

Japanese Tokenizers

Tokenization, or breaking a text into a list of words, is an important step before other NLP tasks (e.g. text classification). In English, words are often separated by whitespace or other symbols, which make it easier to tokenize. However, in Japanese, words are normally written without any space between. Japanese tokenization requires reading/analyzing the whole sentence, recognizing words, and determining word boundaries without any explicit delimiters.

Most Japanese tokenizers use Lattice-based tokenization. As the name suggested, a lattice-based tokenizer builds a lattice (or a graph-like data structure) consisting of all possible tokens (terms or substrings) that surface on the input text. It uses the Viterbi algorithm to find the best connected-path through the lattice.

The most popular lattice-based tokenizer is MeCab (written in C++). Most open-source Japanese tokenizer libraries are either simply MeCab’s wrapper or re-implementation of Lattice-based tokenization in different platforms.

For examples:

Kotori in Kotlin

Sudachi and Kuromoji in Java

Janome and SudachiPy in Python

Kagome in Go.

MeCab’s IPA dictionary (IPADIC) is also the most popular dictionary. It is used as a baseline or primary dictionary for most tokenizers.

In this article, I’ll explain in detail how the Lattice-based dictionary and tokenization works through a simplified version of Kotori’s code and MeCab’s references.

Dictionary

For Lattice-based tokenizers, a dictionary is an object or a data structure that provides information about available terms as well as how those terms should appear next to each other according to Japanese grammar or the probability.

Although different tokenizer libraries use different in-memory data structures and read different dictionary file formats, all Lattice-based tokenizer dictionaries share mostly the same interface or logical structure.

In this article, we will look into MeCab’s dictionary format. Specifically, MeCab’s IPADIC, which you can download here (to see the content and compare with the note for yourself).

A tokenizer dictionary consists of two important parts:

Term-Dictionary

Term-Dictionary (単語辞書 or Lexicon) is a list of known terms (or words). In MeCab’s dictionary directory, all CSV files together make up this term dictionary part.

For example, in Noun.place.csv, you can find rows such as:

東京,1293,1293,3003,名詞,固有名詞,地域,一般,*,*,東京,トウキョウ,トーキョー



京都,1293,1293,2135,名詞,固有名詞,地域,一般,*,*,京都,キョウト,キョート



札幌,1293,1293,3440,名詞,固有名詞,地域,一般,*,*,札幌,サッポロ,サッポロ



渋谷,1293,1293,4310,名詞,固有名詞,地域,一般,*,*,渋谷,シブヤ,シブヤ

Each term (or row) consists of multiple columns (or fields), but only the first four columns are important to the tokenization.

Surface Form (表層形): The string or term that should appear in the text.

(表層形): The string or term that should appear in the text. Left/Right Context ID (左/右-文脈ID): This is used for determining connection cost or how likely it is for a term to follow other terms.

(左/右-文脈ID): This is used for determining connection cost or how likely it is for a term to follow other terms. Cost: The cost of having this term. The cost depends on how likely it is for the term to be used. The higher the cost, the less frequently used or rarer is the term.

The remaining columns are additional features about the terms (e.g. part-of-speech, normalized form, pronunciation, …, etc)

Notes:

A term-dictionary in practice is large and comprehensive. It often contains hundreds of thousands of terms (e.g. around 290,000 terms in IPADIC)

Please don’t confuse Dictionary (辞書) with Term-Dictionary (単語辞書). Throughout this article, I will use the term-dictionary to mean the table of terms part, while the dictionary would refer to the whole (term-dictionary + connection cost + others).

Connection Cost

Connection Cost (連接コース) is a cost structure to measure how likely it is for a term to be before or after one another.

The connection likelihood is explained as a cost of connecting the right context ID of the previous word with the left context ID of the following word.

(Prev term's rightID, Next term's leftID) -> Cost

Similar to the term’s cost, the higher the connection cost, the less likely the given terms should be connected.

This mapping is matrix.def file in MeCab’s dictionary. The file structure is pretty straight-forward. The first line is the number of possible rigthIDs and leftIDs (both are 1316 in the example) follow by the list of (rightID, leftID, cost) tuples.

1316 1316

0 0 -434

0 1 1

0 2 -1630

0 3 -1671 ... 1315 1313 -4369

1315 1314 -1712

1315 1315 -129

Unknown Terms Handling

In addition to Term-Dictionary and Connection Cost, another dictionary component that is crucial to tokenization in practice is an unknown term (or out-of-vocabulary) handling strategy. It is for recognizing some words, substrings, or symbols outside of term-dictionary (e.g. people names) and allows tokenizers to skip or tokenize them.

In MeCab, unknown term handling is done by heuristic rules based-on character types (e.g. consecutive Katakana characters is probably a noun). The rules and character types are defined in unk.def and char.def files.

In this article, I am leaving the details of unknown word handling and assume that term-dictionary includes every term in the text.

Lattice-based Tokenization

As mentioned, Lattice-based tokenization is building a lattice of terms and find the best path through the lattice.

Given the dictionary, the tokenizer identifies all terms that “surface” on the text as sub-string, add all of them into the lattice together with their substring starting/ending location, then runs the algorithm to find the best path of connected terms.

The simplified code would look like this:

val lattice = Lattice.create() for i in 0..text.length { val terms = findAllTermStartingAt(text, i) for term in terms { lattice.add(term, startIndex=i, endIndex=i+term.length) }

...

}

... return lattice.findTheBestPath()

In the example code, we use findAllTermStartingAt(..) to find all terms in the dictionary having surface form starting at the location.

For example, if text="東京都に住む" :

findAllTermStartingAt(text, 0) should return 東 (East), 東京 (Tokyo)

should return 東 (East), 東京 (Tokyo) findAllTermStartingAt(text, 1) should return 京 (Capital), 京都 (Kyoto)

should return 京 (Capital), 京都 (Kyoto) …

Image by author

Note:

Having a fast findAllTermStartingAt(..) is very important to the tokenizer performance. Thus, the function is often implemented by efficient string matching/lookup data structure, e.g. Trie or Finite-State Transducer (FST). I’m leaving the implementation details of those data structures for future articles.

is very important to the tokenizer performance. Thus, the function is often implemented by efficient string matching/lookup data structure, e.g. Trie or Finite-State Transducer (FST). I’m leaving the implementation details of those data structures for future articles. In the shown version, we identify possible terms and save all of them in memory before running the algorithm to finding the best path once at the end. There are tokenizers that use a greedy or window-based approach to actively prune out unnecessary terms to keep memory usage low. Kotori uses the shown approach because I found it is more efficient (aka. faster) and memory usage has never been a problem in practice.

The Lattice is a graph structure containing all surfaced terms as nodes. The edges of the graph connect nodes (or terms) starting at each location with nodes ending at the same location.

For example, 東 (ending at index=1) connected to 京 and 京都 (starting at index=1).

Image by author

Each node and connection will have a cost associate with it:

The cost of a node (orange-colored) is its term’s cost (as defined in term-dictionary)

(orange-colored) is its (as defined in term-dictionary) The cost of a connection (red-colored) is its connection cost between the ending term’s right context ID and the starting term’s left context ID (as defined in connection-cost).

Note from the example that, in addition to the nodes from surfaced terms, we need to include two special nodes into the lattice.

A special Start node index=0, rightID=0, cost=0

A special End node index=text.length, leftID=0, cost=0

Both term and connection costs represent how commonly used they are. The lower the cost the more likely the terms and connections. Thus, the tokenizer's goal is to find the minimal cost path (or the ‘best’ path) from start to end. Doing so will result in the token sequence that is the most likely in Japanese.

Viterbi Algorithm

Viterbi Algorithm is an algorithm to find the optimal path (or most likely path, or minimal cost path, etc) through the graph. Most Viterbi algorithm examples come from its application with Hidden Markov Model (e.g. POS tagging). However, I found the Viterbi algorithm usage in tokenization is very different.

My explanation of the Viterbi Algorithm for Lattice-based tokenization application is as follows:

To find the minimal path from start to tᵗʰ node, we need to consider paths through each (t-1)ᵗʰ node leading to tᵗʰ node by calculating each path’s total cost and pick the minimal path among them.

For each (t-1)ᵗʰ node leading to tᵗʰ node:

(Recursively) Find the minimal cost/path from start to that (t-1)ᵗʰ node.

Find the cost of connecting that (t-1)ᵗʰ node to tᵗʰ node

Find the cost of tᵗʰ node itself

Adding them all together. This is the total cost from start to tᵗʰ node through the (t-1)ᵗʰ node

Then, among the paths through different (t-1)ᵗʰ nodes to tᵗʰ node, pick the path with the minimal total cost.

An efficient way to implement this algorithm is dynamic programming. We can start finding the minimal path from the nodes closet to the start and reuse their minimal path to the nodes following them. Keep repeating until we calculate the minimal path for each node.

for i in 0..text.length { for node in nodes.startingAt(i) { for prevNode in nodes.endingAt(i) { val prevTotalCost = prevNode.totalCost

val termCost = node.cost

val connectionCost = getConnectionCost(prevNode.rightId, node.leftId) val totalCost = prevTotalCost + connectionCost + termCost



if (totalCost < node.totalCost) {



node.totalCost = totalCost

node.prevNode = prevNode }

}

}

}