Long Unit Word Tokenization and Bunsetsu Segmentation of Historical Japanese (2024)

Abstract: In Japanese, “bunsetsu” is the natural minimal phrase of a sentence; it serves as a natural boundary of a sentence for native speakers rather than words, and thus grammatical analysis in Japanese linguistics commonly operates on the basis of bunsetsu units. By contrast, because Japanese does not have delimiters between words, there are two major categories of word definitions: Short Unit Words (SUWs) and Long Unit Words (LUWs). SUW dictionaries are available, whereas LUW dictionaries are not. Hence, this study focuses on providing deep learning-based (or LLM-based) bunsetsu and LUWs parser for the Heian period (AD 794-1185) and evaluating its performances. We model the parser as a transformerbased joint sequential labels model that combines the bunsetsu BI tag, LUW BI tag, and LUW Part-of-Speech (POS) tag for each SUW token. We trained our models on the corpora of each period including contemporary and historical Japanese. The results ranged from 0.976 to 0.996 in the f1 value for both bunsetsu and LUW reconstruction indicating that our models achieved comparable performance with models for a contemporary Japanese corpus. Through statistical analysis and a diachronic case study, it was found that the estimation of bunsetsu could be influenced by the grammaticalization of morphemes.

heading: 1 Introduction
text: In Japanese, “bunsetsu” (base-phrase) is the natural minimal phrase of a sentence. It serves as a natural boundary of a sentence for native speakers rather than words; thus grammatical analysis in Japanese linguistics commonly operates on the basis of bunsetsu units. For example, in Universal Dependencies (UD; Nivre et al., 2020), a framework for the consistent annotation of lexical dependency grammar across different human languages, some Japanese corpora have been con-
verted from dependency relations between bunsetsu (Asahara et al., 2018; Omura and Asahara, 2018).
In contrast, because Japanese does not have delimiters between words, there are many definitions of “words” in Japanese. The National Institute for Japanese Language and Linguistics defines two hierarchical word tokenization categories: Short Unit Words (SUWs) and Long Unit Words (LUWs). SUW is a minimal word token in Japanese, and is defined by a bottom-up method that consists of at most two morphological units. In contrast, LUW is defined by a top-down method that divides a bunsetsu into two parts, and it may contain several SUWs. For example, the LUW “
48
主な 項⽬ に 関する 最終 案 の 内容 は 以下 の とおり (The contents of the final draft regarding the main items is as follows)
main items regarding final draft of contents follows as
SUW
as a joint sequential label that combines the bunsetsu BI tag, LUW BI tag, and LUW Part-ofSpeech (POS) tags for each SUW token. We used a Transformer-based Language Model (TLM) to output an SUW token representation by taking the appropriate pooling of subword representations for the last layer of the transformer. We preserved the SUW boundaries when tokenizing a given sentence into subwords. We trained our models on the corpora of each period including contemporary and historical Japanese.
The results indicate that the models trained on historical Japanese achieve comparable performance (0.976-0.996 f1 values) to a model for a contemporary Japanese corpus. To trace grammatical changes in Japanese, we evaluated the zero-shot transfer performance of the Heian, Kamakura, and Muromachi periods for each other. The models trained with a corpus of the Heian and Kamakura periods performed well on each other, whereas the model trained with a corpus of the Muromachi period did not. These results support the consensus among Japanese liguists that the large grammatical changes occurred during the Muromachi period. Furthermore, the analysis focusing on sentence-ending particles revealed that new sentence-ending particle usage has emerged in the Muromachi, and they are difficult to predict by the models of the prior periods.2
2Our code is publicly available at https://github.com/ komiya-lab/monaka

heading: 2 Related Work
text: Parser for Historical Japanese Comainu, a Japanese bunsetsu and LUW parser, was originally provided for contemporary Japanese (Kozawa et al., 2014), although it can also be applied to historical Japanese. Comainu takes SUW tokens as input, which are tokenized by a CRF-based morphological analyzer MeCab3, and then outputs the bunsetsu and LUW tokens. As mentioned above, Comainu is a CRF-based parser; thus, we focused on deep-learning-based methods.
Parser for Contemporary Japanese Recent Japanese corpora of UD contain bunsetsu and LUW annotations (Omura et al., 2023); thus, some parsers trained on these corpora support bunsetsu segmentation and LUW tokenization. For example, the spaCy-based 4 Japanese UD parser5 supports LUW tokenization (Matsuda et al., 2022). The parser was trained with a Transformerbased language model (TLM) through the spaCy pipeline, and it achieved better performance than Comainu by adding some rules. GiNZA (Matsuda, 2020), which is also a spaCy-based parser, supports bunsetsu output.

heading: 3 Bunsetsu and Long Unit Word

heading: 3.1 Short Unit Word
text: Short Unit Word (SUW) is a token close to the granularity of typical Japanese word tokens. A
3https://taku910.github.io/mecab/ 4https://spacy.io/ 5https://github.com/megagonlabs/UD_
Japanese-GSD/releases/tag/r2.9-NE/
dictionary (UniDic) was established for SUWs, enabling high-performance morphological analysis based on UniDic (Den et al., 2008). As shown in the overview Figure 1, bunsetsu and LUWs are also composed of SUWs.

heading: 3.2 Bunsetsu (Base-pharase)
text: A bunsetsu is a (natural) minimal phrase that consists of a Japanese sentence. Generally, a bunsetsu boundary occurs after a particle or a sequence of particles. This is because Japanese functional words typically follow their content words, on which they depend. In Figure 1, all LUW noun (NOUN) and adposition (ADP) pairs are composed into bunsetsu segments.

heading: 3.3 Long Unit Word
text: The Long Unit Word (LUW) is a word unit based on a bunsetsu. Identification of LUW involves identifying bunsetsu and then dividing each bunsetsu into independent and attached LUWs. For example, in Figure 1, bunsetsu “
sequential labels of LUW BI, LUW POS, and Bunsetsu BI. For example, the target label of the adposition “

heading: 5.3 Hyperparameters
text: Table 2 lists the hyperparameters. We did not perform an intense hyperparameter search, thus there is a possibility for further performance improvements. Since the number of sentences in CHJ corpora is more than ten times compared to that of in UD-Japanese-GSD, we decreased the total number of epochs and the learning rate when we trained on the CHJ. We used “cl-tohoku/bert-basejapanese-whole-word-masking”8 for the TLM.

heading: 6 Results and Discussions

heading: 6.1 Contemporary Japanese
text: We first compared the model variants using UDJapanese-GSD, as shown in Table 3. The variant with the Embedding and sum pooling layers generally performed well. The head pooling layer performed well for boundary predictions. This suggests that sum pooling provides a better representation of the entire SUW content, while head pooling adequately preserves the boundary information.
The variant with incontext and the sum pooling achieved the highest precision, but a lower recall value. This is because the incontext method increases the number of subword tokens and often exceeds the maximum subword token limit (512) to represent an entire sentence. Table 4 presents a span-based LUW score comparison with that in a previous study (Matsuda et al., 2022). Our model and that of Comainu used MeCab(Kudo et al.,
8https://huggingface.co/tohoku-nlp/ bert-base-japanese-whole-word-masking
2004) for the SUW tokenization using a UniDic dictionary. The spaCy model uses SudachiPy 9 for SUW tokenizer instead of MeCab. Our model showed an improvement compared to Comainu, while spaCy outperformed the other models. This is because of the difference in the SUW tokenizers.
Because SudachiPy only supports contemporary Japanese, we are supposed to use MeCab for the SUW tokenizer and decided to use Embedding + sum pooling model for historical Japanese models.

heading: 6.2 Historical Japanese
text: Table 5 lists the overall results for the CHJ. The results evaluated on samples from the same period as
9https://github.com/WorksApplications/ SudachiPy
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
0 5
10 15 20 25 30 35 40 45 50
No un
Pa rti
cl e
Sy m
bo l Ve rb
Au xil
ia ry
V er b Ad ve rb
Ad je
ct ive Su ffi x Pr on ou n Pr ef ix
Ad je
ct iva
l N ou n Co nj un ct io n In te rje ct io n Pr eno un A dj ec tiv al
Er ro
r F re
qu en
cy
Er ro
r R at
e
Heian Kamakura Muromachi Heian Kamakura Muromachi
(a) Trained on the Heian period
0.00% 5.00% 10.00%
15.00% 20.00%
25.00% 30.00% 35.00%
40.00%
0
5
10
15
20
25
30
35
Ve rb
Pa rti
cl e
Sy m
bo l No un
Au xil
ia ry
V er b Pr ef ix
Ad je
ct ive Su ffi x Ad ve rb
Pr on
ou n
Ad je
ct iva
l N ou n Co nj un ct io n In te rje ct io n Pr eno un A dj ec tiv al
Er ro
r F re
qu en
cy
Er ro
r R at
e
Heian Kamakura Muromachi Heian Kamakura Muromachi
(b) Trained on the Muromachi period
Figure 2: POS tags contained in bunsetsu versus error rate and normalized frequency.
during training ranged from 0.976 to 0.996. Thus, our historical models have comparable or even superior results to those of contemporary Japanese (UD-Japanese-GSD), as shown in Table 3. This was because the data size of the CHJ was significantly larger than that of UD-Japanese-GSD. The LUW performances degrades with time, while the bunsetsu segmentation performances increase. As time progresses and vocabulary becomes more complex, it is suggested that styles that are more conscious of syntactic structures such as bunsetsu, increase.
Focusing on the transferability between the CHJ corpora, the model trained on samples of newer periods and applied to older periods yielded higher performance than the reverse case. This is because the vocabulary coverage of the newer samples is larger than that of the older samples. The Heian and Kamakura models work well on samples from each other, however, they do not perform well on samples from the Muromachi period, particularly for labeled LUW and bunsetsu evaluations. This implies drastic grammatical changes occured in Japanese during the Muromachi period.

heading: 6.3 Transferability between Contemporary and Historical Japanese
text: Table 6 shows the transferability performances of contemporary and historical Japanese. In this evaluation, the POS embeddings may not work, because there is a large difference in fine-grained POS categories between contemporary and historical Japanese. Thus, we used the highest level of POS tags for the labeled LUW evaluations. The model trained on UD-Japanese-GSD performed
similarly in each period (Table 6a). However, the performances of the models on samples from each period increased with time, specifically for the bunsetsu segmentation. This indicates that the syntactic structure of sentences gradually approaches modern syntactic structures over time, while the morphology of LUW is not as high.

heading: 6.4 Grammatical Changes during the Muromachi Period
text: Figure 2 plots the error rates of bunsetsu containing the SUW of a particular POS tag. Figure 2 presents the results of the models trained on samples from the Heian and Muromachi periods. We also plotted the normalized error frequency corresponding to each POS tag for all errors in the same period in Figure 2.
The model trained on Heian period data exhibited a particularly higher error rate when it predicted bunsetsu containing auxiliary verbs or verbs when evaluated on samples from the Muromachi period. This tendency was also observed when samples from the Heian period were evaluated using the Muromachi model. This indicates that there may have been significant changes in sentence endings that usually contained both verbs and auxiliary verbs.
When evaluating samples from the Muromachi period using the Heian model, the error frequency relatively increased in bunsetsu-containing nouns compared with the reverse scenario. This is because the newer model partially contains old vocabularies.
In both cases, the bunsetsu-containing particles resulted in a high error rates and frequencies.
Gold (Heian) and the Kamakura model prediction: En One doesn’t do such things, there will surely be regrets Ja
ately pooling of the subword representations for the last layer of the transformer. We trained our models on the corpora of each period, including contemporary and historical Japanese.
The results ranged from 0.976 to 0.996 in the f1 value for both bunsetsu and LUW reconstructions indicating that our models achieved comparable performance to models trained on a contemporary Japanese corpus.
Through the statistical analysis and case studies comparing each period, the bunsetsu estimation can be influenced by the grammaticalization of morphemes.
In the future, we will expand the applicable periods. We will build a syntactic parser by annotating the dependencies between bunsetsu segments.

heading: Acknowledgments
text: This research was supported by JSPS KAKENHI grant JP22K12145, as well as the National Institute for Japanese Language and Linguistics Joint Research Projects "Empirical Computational Psycholinguistics Using Annotation Data." and "Extending the Diachronic Corpus through an Open Co-construction Environment."

heading: Future Work
text: In the future, we will expand the applicable periods. We will build a syntactic parser by annotating the dependencies between bunsetsu segments