Japanese Lexical Complexity for Non-Native Readers: A New Dataset
Year: 2023

Abstract:
Lexical complexity prediction (LCP) is the task of predicting the complexity of words in a text on a continuous scale. It plays a vital role in simplifying or annotating complex words to assist readers. To study lexical complexity in Japanese, we construct the first Japanese LCP dataset. Our dataset provides separate complexity scores for Chinese/Korean annotators and others to address the readers’ L1-specific needs. In the baseline experiment, we demonstrate the effectiveness of a BERT-based system for Japanese LCP.

heading: 1 Introduction
text: Reading comprehension requires a certain level of vocabulary knowledge. The results reported by Hu and Nation (2000) suggest that most English learners need to understand 98% of tokens in a text to comprehend it. A follow-up study by Komori et al. (2004) estimates the percentage to be 96% for Japanese learners to comprehend text. Acquiring vocabulary to reach such levels, in turn, is a lengthy and challenging task for learners. This opens up opportunities for assistive applications, such as simplification or annotation of complex words. The first step necessary for such applications is to predict the complexity of the words. The task of lexical complexity prediction (LCP) is defined as predicting how difficult to comprehend words or phrases in a text are on a continuous scale (Shardlow et al., 2020). This differentiates LCP from complex word identification (CWI), i.e., binary classification of complex words (Yimam et al., 2018). As complexity is naturally perceived as continuous, a continuous scale used in LCP allows to represent it without loss of information.
The LCP research so far has been limited to English, for which two LCP datasets have been constructed (Shardlow et al., 2020, 2022), and no such dataset has been created for Japanese. Meanwhile, there are a number of features specific to the
Japanese language that could affect lexical complexity, and their effects have yet to be studied. For example, the Chinese characters, which are used extensively in Japanese, lower text readability (Tateisi et al., 1988).
Previous studies on Japanese lexical complexity used pedagogical word lists to estimate complexity level. Nishihara and Kajiwara (2020) modeled lexical complexity of words based on the Japanese Educational Vocabulary List (Sunakawa et al., 2012). The word list assigns a degree of difficulty to each item, based on the subjective judgment of Japanese language teachers, not learners themselves, and does not consider the learners’ L1 background.
In light of this, we present JaLeCoN1, Dataset of Japanese Lexical Complexity for Non-Native Readers. Our dataset has the following key features:
(1) Complexity scores for single words as well as multi-word expressions (MWEs);
(2) Separate complexity scores from Chinese/Korean annotators and others, addressing the considerable advantage of the former in Japanese reading comprehension.
Our analysis reveals that the non-Chinese/Korean annotators perceive words of Chinese origin or containing Chinese characters as especially complex. In the baseline experiment, we investigate the effectiveness of a BERT-based system in the Japanese LCP task, and how it varies according to the word complexity and L1 background.

heading: 2 Task Setting
text: Since Japanese has no explicit word boundaries, word segmentation is the first prerequisite for LCP. We use short unit words (SUWs) as the basic word unit, combining them into longer word units in the case of multi-word expressions (MWEs):
1JaLeCoN is available at https://github.com/ naist-nlp/jalecon.
477
SUWs 右肩 上がり に 増え て いる right.shoulder rise ADV increase GER be-PRS
Words 右肩上がり MWE steady.rise
にSUW ADV 増え SUW increase ている MWE PRG-PRS
“is steadily increasing”
Figure 1: Example of text segmented as SUWs and as words (either SUW or MWE ). Semantically opaque sequences are chunked into MWEs. Abbreviations in glosses: ADVerbializer, GERund, PReSent, PRoGressive.
SUW: SUWs consist of one or two smallest lexical units (Ogura et al., 2011), and are commonly used for segmentation of Japanese.
MWE: We understand MWEs as multi-SUW expressions that are fixed or semantically opaque (see Appendix C) and consequently may have higher complexity than their components. We identify MWEs either using long unit word (LUW)2 segmentation, or manually (see Section 3). Consequently, a word, can be either an SUW or an MWE (see Figure 1 for examples). A complexity score represents perceived complexity based on the annotators’ judgment on a scale from 0 (least complex) to 1 (most complex). We exclude proper nouns from our target because their complexity is influenced by factors unrelated to reading proficiency or vocabulary knowledge.3
We annotate the words in an in-context dense setting. In-context here means including both intrasentence and extra-sentence context of each word. Context is important for lexical complexity for two reasons (Gooding and Kochmar, 2019; Shardlow et al., 2021): (1) As polysemous words can have different complexity levels for each sense, context is necessary to differentiate between possible meanings of these words. (2) Presenting a word without context could increase its complexity. In particular, the recognition of abstract words relies on context (Schwanenflugel et al., 1988). Dense means annotating each word of the text with a complexity label, instead of annotating one specific word in each sentence (Shardlow et al., 2022). We adopt the dense setting to avoid any bias that could arise from targeting specific words.

heading: 3 Construction of JaLeCoN
text: In order to include both written and spoken language and a variety of vocabulary, we sourced texts
2The LUW is defined as a syntactic word by Omura et al. (2021).
3Sequences containing segmentation errors are also excluded (see Appendix D).
from two different genres: News comes from the Japanese-English data of the WMT22 General Machine Translation Task (Kocmi et al., 2022). It contains a variety of news texts written for the general Japanese reader.
Government is composed of press conference transcripts from Japanese ministries or agencies.4
The whole dataset is composed of sequences of sentences constituting either the beginning of an article (News) or a question-answer pair (Government). We restricted the length of the sequences to at least 6 and at most 11 sentences to obtain similar amounts of text, and presented each sequence as a whole for annotation.

heading: 3.1 Word Segmentation
text: We used Comainu 0.805 (Kozawa et al., 2014) to perform two-level segmentation. The low-level SUW segmentation was done using MeCab (Kudo et al., 2004), a Japanese morphological analyzer, and the UniDic 2.3.0 (Den et al., 2007) dictionary. At the second level, Comainu chunked the SUWs into LUWs. Based on the two segmentations, we segmented the text into words as follows:
(1) If an LUW is a noun, we use the constituting SUWs as words. Transparent noun compounds are ubiquitous in Japanese (e.g., 次期 |気象 |衛 星6 “next meteorological satellite”), and we do not consider them MWEs.
(2) If an LUW is not a noun, we use the LUW as a word. Such an LUW may be a single SUW, or a sequence of SUWs, which we consider an MWE. Such MWEs most importantly include functional words, such as compound particles (e.g., に |比 べ |て “compared to”) and auxiliary verbs (e.g.,な けれ |ば |なら |ない “have to”).
We also identified other MWEs manually, as explained in Section 3.3.
4The transcripts were retrieved from the websites of five organizations: JMA, JTA, MOJ, MOFA, and MLHW.
5https://github.com/skozawa/Comainu 6The vertical bars denote boundaries between SUWs.

heading: 3.2 Complexity Annotation
text: To capture the lexical complexity for a non-native Japanese reader with intermediate or advanced reading ability, we recruited 15 annotators per sentence with Japanese reading proficiency ranging from CEFR (Common European Framework of Reference for Languages) level B1 to C2. We required at least intermediate proficiency, as it has been shown that complexity judgments made by intermediate or advanced learners can be used to adequately predict the needs of beginners but not vice versa (Gooding et al., 2021). The proficiency levels were self-reported (see Appendix A for details). We used the annotations made by 14 of them, after removing one outlier, whose annotations had over 70% higher mean than those of any other annotator, clearly not corresponding to the reported reading proficiency.
Approximately half of the annotators we recruited have a Chinese/Korean L1 background (CK).7 CK learners have a considerable advantage in comprehension of words of Chinese origin, which also form a large part of Chinese and Korean vocabulary (Koda, 1989).
The annotators were asked to assign one of the following labels to each span if they find it complex: 3 (Very Difficult), 2 (Difficult), or 1 (Not Easy); otherwise the annotators were to leave the span unlabeled and we interpreted it as 0 (Easy).8 Annotators could label a span of any length if it was complex as a whole, but were asked to create as short a span as possible. To calculate the average, the labels were converted to numerical values as follows: 3 → 1, 2 → 0.67, 1 → 0.33, 0 → 0. The averaging hinges on the assumption that the labels have an equal distance between them. We always presented the labels together with the values 0 to 3 to reinforce the perception of equal distance.
7On average, the CK annotators reported higher Japanese reading proficiency than the non-CK (see Appendix A).
8See Appendix B for detailed definitions of each label.

heading: 3.3 MWE Annotation
text: In parallel with the complexity annotation, we annotated MWEs not identified by LUW segmentation (see Section 3.1). Given the absence of an MWE detector for Japanese of sufficient quality, the annotation was performed manually by a native Japanese speaker and a non-native speaker with a degree in the Japanese language. The expression categories we consider MWEs are described in Appendix C.

heading: 3.4 Complexity Scoring
text: Using annotations from the previous steps, we assigned complexity scores to words according to the following rules:
(1) If a span contains one or more words, each word receives the complexity value of the span.
(2) If an MWE (manually annotated according to Section 3.3) overlaps with or contains multiple spans, the MWE receives the maximum of the complexity values of the spans.
Finally, for each word, we calculated the complexity score for each L1 group as the average of the individual values from the annotators in that group.

heading: 4 Statistics and Analysis
text: Overall statistics for both genres and L1 groups are shown in Table 1.9 MWEs have higher mean complexity than single words for both L1 groups and are more frequent in the Government genre. There is a tendency towards perceiving higher complexity in the non-CK group, which corresponds to slightly lower average Japanese proficiency of the non-CK annotators (see Appendix A).
We measured inter-annotator agreement (IAA) using Krippendorf’s α for interval values (Krippendorff, 1970). The IAA is 0.32 in the CK group, and 0.31 in the non-CK group, while it would be 0.19 if we merged the groups. As lexical complexity is
9See Appendix E for the complexity scores and annotation distributions of several words in the non-CK group.
highly subjective (Gooding et al., 2021), the low agreement does not imply low reliability, but it indicates that perception of complexity is more alike within the L1 groups than across all annotators.
The complexity score distribution in each L1 group is shown in Figure 2. No words achieved a score greater than 0.81 and 0.86 in the CK and nonCK groups, respectively, which reflects that words are rarely labeled as Difficult or Very Difficult by all annotators in a group.
In addition to the aforementioned difference in proficiency, there is also a clear difference in how the two L1 groups perceive complexity of words based on their origin and whether they contain Chinese characters10, as analyzed in Table 2. For the CK group, the mean complexity of words of Japanese and Chinese origin was similar. For the non-CK group, however, words of Chinese origin
10Japanese vocabulary consists of words of Japanese origin, Chinese (Sino-Japanese) origin, and foreign words from other languages (gairaigo). The first two categories can be written using Chinese characters (kanji), Japanese syllabary (kana), or a combination thereof, while other foreign words are usually written in syllabary only. (See Appendix F for examples.)
were markedly more complex (0.062) than words of Japanese origin (0.010), and both categories of words were more complex when they contained Chinese characters.11

heading: 5 Experiments
text: The newly created dataset can be used to evaluate performance of LCP for non-native Japanese readers of different L1 backgrounds (CK and nonCK). We developed a baseline system based on a fine-tuned BERT (Devlin et al., 2019) model, and evaluated it using cross-validation. We fine-tuned a Japanese pre-trained BERT model released by Tohoku University, namely the base model for UniDic Lite segmentation12.
For each word w in our dataset and the sentence s that contains it at token indices i to j−1, we construct an input sequence ([CLS], si−10 , <Unused1>, w, <Unused2>, s
|s|−1 j , [SEP], w, [SEP]). The tar-
get word occurs first delimited by unused tokens (<Unusedn>) in the sentence context, and then on its own following the first [SEP] token.13 To predict the complexity score, we feed the final hidden representation of the [CLS] token into a linear layer with a single output. A similar fine-tuning approach, but without the special tokens, was used for English LCP by Taya et al. (2021), achieving one of the highest R2 values in the single-word subtask of SemEval-2021 Task 1 (Shardlow et al., 2021).
We fine-tune and evaluate models for CK and non-CK complexity separately. See Appendix G for the hyperparameters and crossvalidation scheme.
The results are reported in Table 3. In addition to R2 (coefficient of determination)14, we report mean average error (MAE) by complexity score tiers to draw the full picture of the models’ performance at different complexity levels. The score ranges of the
11The opposite tendency for gairaigo (foreign words mostly from English) to be perceived as more complex in the CK group coincides with lower English proficiency among annotators in this group (see Appendix A), and therefore should not be explained by their L1 background.
12Available from https://huggingface.co/cl-tohoku/ bert-base-japanese-v2.
13Due to a different segmentation (version of UniDic) used by Tohoku BERT and our dataset, we have to enforce segmentation at the word’s boundaries using spaces.
14Compared to correlation coefficients, R2 is more appropriate for LCP, since it also captures deviations in mean and variance. Compared to MAE or MSE, it is easier to interpret, as R2 = 0 corresponds to the mean regressor, while R2 = 1 corresponds to a perfect model.
tiers are centered at annotation values as illustrated in Figure 3. We handle zero as a special tier, and merge Very Difficult with Difficult due to a low number of words.
The fine-tuned BERT model for CK and NonCK achieves R2 of 0.4351 and 0.6142, respectively. For both L1 groups, the MAE value increases markedly in each successive complexity tier, as the number of training examples (shown in Table 4) diminishes. Similarly, the CK model achieves lower error than non-CK only in tier Zero, where it has more examples available than the nonCK model. This suggests that the scarcity of words with complexity above zero is a factor contributing to worse performance on CK data, as measured by R2.

heading: 6 Conclusion
text: In this paper, we presented the first dataset for Japanese LCP. It provides separate complexity scores based on the CK/non-CK distinction of annotators’ L1 background. Our analysis corroborates our conjecture that special consideration of L1 background is useful for the Japanese LCP task in particular. We believe it could benefit LCP in other languages as well.
In the baseline experiment, we demonstrated the efficacy of our BERT-based system for both CK and non-CK readers. Even after separating CK and non-
CK annotators, however, notable inter-annotator disagreement remains within these groups. Therefore personalized systems analogous to Gooding and Tragut (2022) could improve on our system. Future research should study this possibility, analyzing both its costs and benefits.
Models trained on JaLeCoN can be used as part of a lexical simplification pipeline for Japanese, both to identify complex words and to rank candidate simplifications. JaLeCoN itself can be further used as a basis for a lexical simplification dataset targeting words actually perceived as complex, similar to TSAR-ST datasets for English and Spanish (Štajner et al., 2022).
Limitations
Our task setting and baseline system requires that the input is already segmented into words including MWEs. The MWE identification step in the construction process of our dataset involved timeconsuming manual annotation. Building a highquality system that fully automates the process is an issue for future work. Our dataset can be used to evaluate such a Japanese MWE identification system.
Additionally, as shown in Section 5, our baseline model performed relatively poorly in the higher complexity tiers. This is an effect of the dense annotation setting; it results in uneven distributions of
complexity as shown in Figure 2, where easy words greatly outnumber difficult words. One possible solution would be creating another LCP dataset using sparse annotation, where target words are selected using frequency bands so that the words are distributed across a wide range of frequency (Shardlow et al., 2022). Our data could provide insights as to what kind of words should be targeted by sparse annotation for such a dataset.

heading: Acknowledgments
text: We would like to express our gratitude to Justin Vasselli and the anonymous reviewers for their insightful feedback. This work was supported by JSPS KAKENHI grant number JP19K20351 and NAIST Foundation.

heading: B Complexity labels

heading: D Excluded categories

heading: C MWE Categories

heading: F Examples of Words by Origin

heading: E Distributions of Annotation

heading: G Experimental Setting

heading: Future Work
text: Future research should study this possibility, analyzing both its costs and benefits. Models trained on JaLeCoN can be used as part of a lexical simplification pipeline for Japanese, both to identify complex words and to rank candidate simplifications. JaLeCoN itself can be further used as a basis for a lexical simplification dataset targeting words actually perceived as complex, similar to TSAR-ST datasets for English and Spanish (Štajner et al. , 2022). Limitations
Our task setting and baseline system requires that the input is already segmented into words including MWEs. The MWE identification step in the construction process of our dataset involved timeconsuming manual annotation. Building a highquality system that fully automates the process is an issue for future work. Our dataset can be used to evaluate such a Japanese MWE identification system. Additionally, as shown in Section 5, our baseline model performed relatively poorly in the higher complexity tiers. This is an effect of the dense annotation setting; it results in uneven distributions of
complexity as shown in Figure 2, where easy words greatly outnumber difficult words. One possible solution would be creating another LCP dataset using sparse annotation, where target words are selected using frequency bands so that the words are distributed across a wide range of frequency (Shardlow et al. , 2022). Our data could provide insights as to what kind of words should be targeted by sparse annotation for such a dataset