Venue: NAACL 2021

# User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization

 Shohei Higashiyama\({}^{1,2}\), Masao Utiyama\({}^{1}\), Taro Watanabe\({}^{2}\), Eiichiro Sumita\({}^{1}\)

###### Abstract

Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sentences annotated with morphological and normalization information, along with category information we classified for frequent UGT-specific phenomena. Experiments on the corpus demonstrated the low performance of existing MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT.

+
Footnote †: [https://pj.ninjal.ac.jp/corpus_center/bccwjj/en/subscription.html](https://pj.ninjal.ac.jp/corpus_center/bccwjj/en/subscription.html)

\({}^{1}\)National Institute of Information and Communications Technology, Kyoto, Japan

\({}^{2}\)Nara Institute of Science and Technology, Nara, Japan

{shohei.higashiyama, mutiyama, eiichiro.sumita}@nict.go.jp, taro@is.naist.jp

## 1 Introduction

Japanese morphological analysis (MA) is a fundamental and important task that involves word segmentation, part-of-speech (POS) tagging and lemmatization because the Japanese language has no explicit word delimiters. Although MA methods for well-formed text Kudo et al. (2004); Neubig et al. (2011) have been actively developed taking advantage of the existing annotated corpora of newswire domains, they perform poorly on user-generated text (UGT), such as social media posts and blogs. Additionally, because of the frequent occurrence of informal words, lexical normalization (LN), which identifies standard word forms, is another important task in UGT. Several studies have been devoted to both tasks in Japanese UGT Sasano et al. (2013); Kaji and Kitsuregawa (2014); Saito et al. (2014, 2017) to achieve the robust performance for noisy text. Previous researchers have evaluated their own systems using in-house data created by individual researchers, and thus it is difficult to compare the performance of different systems and discuss what issues remain in these two tasks. Therefore, publicly available data is necessary for a fair evaluation of MA and LN performance on Japanese UGT.

In this paper, we present the blog and Q&A site normalization corpus (BQNC),1 which is a public Japanese UGT corpus annotated with morphological and normalization information. We have constructed the corpus under the following policies: (1) available and restorable; (2) compatible with the segmentation standard and POS tags used in the existing representative corpora; and (3) enabling a detailed evaluation of UGT-specific problems.

Footnote 1: Our corpus will be available at [https://github.com/shigashiyama/jkexnorm](https://github.com/shigashiyama/jkexnorm).

For the first requirement, we extracted and used the raw sentences in the blog and Q&A site registers compiled by (the non-core data of) the Balanced Corpus of Contemporary Written Japanese (BCCWJ) Maekawa et al. (2014), in which the original sentences are preserved.2 For the second requirement, we followed the short unit word (SUW) criterion of the National Institute for Japanese Language and Linguistics (NINJAL), which is used in various NINAL's corpora, including manually annotated sentences in the BCCWJ. For the third requirement, we organized linguistic phenomena frequently observed in the two registers as word categories, and annotated each word with a category. We expect that this will contribute to future research to develop systems that manage UGT-specific problems.

Footnote 2: Twitter could be a candidate for a data source. However, redistributing original tweets collected via the Twitter Streaming APIs is not permitted by Twitter, Inc., and an alternative approach to distributing tweet URLs has the disadvantage that the original tweets can be removed in the future.

The BQNC comprises sentence IDs and annotation information, including word boundaries, POS, lemmas, standard forms of non-standard word tokens, and word categories. We will release the annotation information that enables BCCWJ applicants to replicate the full BQNC data from the original BCCWJ data.3

Footnote 3: [https://pj.ninjal.ac.jp/corpus_center/bccwjj/en/subscription.html](https://pj.ninjal.ac.jp/corpus_center/bccwjj/en/subscription.html)

Using the BQNC, we evaluated two existing

methods: a popular Japanese MA toolkit called MeCab (Kudo et al., 2004) and a joint MA and LN method (Sasano et al., 2013). Our experiments and error analysis showed that these systems did not achieve satisfactory performance for non-general words. This indicates that our corpus would be a challenging benchmark for further research on UGT.

## 2 Overview of Word Categories

Based on our observations and the existing studies (Ikeda et al., 2010; Kaji et al., 2015), we organized word tokens that may often cause segmentation errors into two major types with several categories, as shown in Table 1. We classified each word token from two perspectives: the type of vocabulary to which it belongs and the type of variant form to which it corresponds. For example, \(\mbox{$\Sigma$}\!\times\!^{\prime}\)_nihon_ 'Japan' written in _katakana_ corresponds to a _proper name_ and a _character type variant_ of its standard form \(\raisebox{-0.86pt}{\hbox{\rule{7.499886pt}{0.49986pt}\rule[0.0pt]{0.499886 pt}{0.49986pt}\rule[0.0pt]{0.499886pt}{0.49986pt}\rule[0.0pt]{0.499886pt}{0.4998 86pt}\rule[0.0pt]{0.499886pt}{0.49986pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.499886pt}\rule[0.0pt]{0.499886pt}{0.499886pt} \rule[0.0pt]{0.499886pt}{0.49886pt}\rule[0.0pt]{0.499886pt}{0.49886pt} \rule[0.0pt]{0.4998 

(1) Sentence SelectionWe manually selected sentences to include in our corpus from the blog and Q&A site registers in the BCCWJ non-core data. We preferentially extracted sentences that contained candidates of UGT-specific words, that is, word tokens that may belong to non-general vocabularies or correspond to non-standard forms. As a result, we collected more than 900 sentences.

(2) First AnnotationSentences in the non-core data have been automatically annotated with word boundaries and word attributes, such as POS and lemma. Following the BCCWJ annotation guidelines Ogura et al. (2011a, b) and UniDic Den et al. (2007), which is an electronic dictionary database designed for the construction of NINJAL's corpora, we refined the original annotations of the selected sentences by manually checking them. The refined attributes were token, POS, conjugation type, conjugation form, pronunciation, lemma, and lemma ID. Additionally, we annotated each token with a word category shown in Table 1 and a standard form ID if the token corresponded to a non-standard form.

Table 2 shows two examples of annotated sentences. We annotated each non-standard token with a standard form ID denoted as "[lemma ID]:[lemma(_[pronunciation])", which is associated with the set of acceptable standard forms shown in Table 3.

(3) Second AnnotationWe rechecked all tokens in the sentences that we finished the first annotation and fixed the annotation criteria, that is, the definitions of vocabulary types and variant form types, and standard forms for each word. Through these steps, we obtained 929 annotated sentences.

## 4 Detailed Definition of Word Categories

### Type of Vocabulary

Through the annotation process, we defined the criteria for vocabulary types as follows.

**Neologisms/Slang:** a newly invented or imported word that has come to be used collectively. Specifically, we used a corpus reference application called Chunagon7 and regarded a word as a _neologism/slang_ if its frequency in the BCCWJ was less than five before the year 2000 and increased to more than ten in 2000 or later.8

Footnote 7: [https://chunagon.ninjal.ac.jp](https://chunagon.ninjal.ac.jp)

Proper namesfollowing the BCCWJ guidelines, we regarded a single word that corresponded to a proper name, such as person name, organization name, location name, and product name, as a _proper name_. In contrast to the BCCWJ guidelines, we also regarded an abbreviation of a proper name as a _proper name_, for example, "\(\lnot\)" in Table 1.

Onomatopoeia:a word corresponds to onomatopoeia. We referred to a Japanese onomatopoeia dictionary Yamaguchi (2002) to assess whether a word is onomatopoeic. We followed the criteria in the BCCWJ guidelines on what forms of words are onomatopoeic and what words are associated with the same or different lemmas.

Interjections:a word whose POS corresponds to an interjection. Although we defined standard forms for idiomatic greeting expressions registered as single words in UniDiC,9 we did not define standard and non-standard forms for other interjections that express feelings or reactions, for example, \(\tilde{\chi}\), \(\tilde{e}\) 'uh-huh' and \(\tilde{\gamma}\)\(\lambda\)\(\lambda\)\(\lambda\)_uwd_ 'wow'.

Footnote 8: The original sentences were from posts published between 2004 and 2009.

Foreign words:a word from non-Japanese languages. We regarded a word written in scripts in the original language as a _foreign word_, for example, English words written in the Latin alphabet such as "plastic". Conversely, we regarded loanwords written in Japanese scripts (hiragana, katakana, or kanji) as general words, for example, \(\mathcal{T}\)\(\mathcal{T}\)\(\mathcal{T}\)\(\mathcal{T}\)\(\mathcal{T}\)

\begin{table}
\begin{tabular}{c l c} \hline \hline Token & Translation & Standard form ID \\ \hline \(\uparrow\!\!\!\downarrow\) & good & 38988:\(\mathbb{R}\!\!\!\downarrow\)\textbackslash{} \\ \(\Downarrow\!\!\!\downarrow^{*}\) & song & \\ \(\backsim\!\!\!\downarrow^{*}\) & (polite copula) & \\ \(\check{\mu}\)\(x\) & (emphasis marker) & 28754:\(\check{\mu}\) \\ \hline \(\exists\)\(\mathcal{H}\)\(\mathcal{V}\) & good & 38988:\(\mathbb{R}\!\!\!\downarrow\)\textbackslash{} \(\exists\)\(\mathcal{H}\)\(\mathcal{V}\) \\ \(\not\) & (past tense marker) & 21642:\(\check{\tau}\) \\ \hline \end{tabular}
\end{table}
Table 2: Examples of annotated text “\(\dashv\)\(\uparrow\)\(\uparrow\)\(\Downarrow\)\(\times\)\(\rightarrow\)\(\downarrow\)\(\to\)\(\pm\)” (It’s a good song, isn’t it?) and ”\(\exists\)\(\mathcal{H}\)\(\mathcal{V}\)\(\neq\)” (It was good.). Attributes except for token and standard form ID are abbreviated.

 

\({\cal Y}\) 'plastic'. Moreover, we did not regard English acronyms and abbreviations written in uppercase letters as foreign words because such words are typically also written in the Latin alphabet in Japanese sentences, for example, \(SN\S\).

Dialect words:a word from a Japanese dialect. We referred to a Japanese dialect dictionary Sato (2009) and regarded a word as a _dialect word_ if it corresponded to an entry or occurred in an example sentence. We did not consider normalization from a dialect word to a corresponding word in the standard Japanese dialect.

Emoticons/AA:nonverbal expressions that comprise characters to express feelings or attitudes. Because the BCCWJ guidelines does not explicitly describe criteria on how to segment emoticon/AA expressions as words, we defined criteria to follow emoticon/AA entries in UniDiC.10

Footnote 10: For example, if characters expressing body parts were outside of punctuation expressing the outline of a face, the face and body parts were segmented, but both were annotated with _emoticons/AA_, for example, “m (.___.’) m” \(\rightarrow\) “m| (. —.. . ’) \(|\m ”\).

### Type of Variant Form

There are no trivial criteria to determine which variant forms of a word are standard forms because most Japanese words can be written in multiple ways. Therefore, we defined standard forms of a word as all forms whose occurrence rates were approximately equal to 10% or more in the BCCWJ among forms that were associated with the same lemma. For example, among variant forms of the lemma \(\Ryz\Pi\Voin\! 

typographical errors, each of the other category has more than 100 instances. Our corpus contains a similar number of non-standard tokens to Kaji and Kitsuregawa (2014)'s Twitter corpus (1,831 sentences, 14.3K tokens, and 793 non-standard tokens) and Osaki et al. (2017)'s Twitter corpus (1,405 sentences, 19.2K tokens, and 768 non-standard tokens). The former follows the POS tags for the Japanese MA toolkit JUMAN and the latter follows the authors own POS tags that extend NINJAL's SUW.

In the following subsections, we evaluate the existing methods for MA and LN on the BQNC and discuss correctly or incorrectly analyzed results.

### Systems

We evaluated two existing methods. First, we used MeCab 0.996 [1],12 which is a popular Japanese MA toolkit based on conditional random fields. We used UniDicMA (unidic-cwj-2.3.0)13 as the analysis dictionary, which contains attribute information of 873K words and MeCab's parameters (word occurrence costs and transition costs) learned from annotated corpora, including the BCCWJ [4].

Footnote 12: [https://taku910.github.io/mecab/](https://taku910.github.io/mecab/)

Footnote 13: [https://unidict.ninjal.ac.jp/](https://unidict.ninjal.ac.jp/)

Second, we used our implementation of Sasano et al. (2013)'s joint MA and LN method. They defined derivation rules to add new nodes in the word lattice of an input sentence built by their baseline system, JUMAN. Specifically, they used the following rules: (i) sequential voicing (_renodaku_), (ii) substitution with long sound symbols and lowercase kana, (iii) insertion of long sound symbols and lowercase kana, (iv) repetitive onomatopoeia (XYXY-form14) and (v) non-repetitive onomatopoeia (XQYri-form and XXOto-form). For example, rule (iii) adds a node of \(\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma} \hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{\gamma}\hat{ \gamma}\hat{\gamma 

and 70.4% for POS tagging ("non-gen/std total" column). McCab+ER outperformed MeCab particularly for onomatopoeia, character type variants, alternative representations, and sound change variants. The high scores for dialect words were probably because UniDicMA contains a large portion of (19 out of 23) dialect word tokens. Interjection was a particularly difficult vocabulary type, for which both methods recognized only approximately 50% of the gold POS tags. We guess that this is because the lexical variations of interjections are diverse; for example, there are many user-generated expressions that imitate various human voices, such as laughing, crying, and screaming.

Table 7 shows the recall of MeCab+ER's normalization for each category. The method correctly normalized tokens of alternative representations and sound change variants with 30-40% recall. However, it completely failed to normalize character type variants not covered by the derivation rules and more irregular typographical errors.

### Analysis of the Segmentation Results

We performed error analysis of the segmentation results for the two methods. Table 8 shows a matrix of the number of correct or incorrect segmentations for the methods for gold words. There existed 32 tokens that only MeCab correctly segmented (T-F), 200 tokens that only MeCab+ER correctly segmented (F-T), and 413 tokens that both methods incorrectly segmented (F-F).

In Table 9, we show the actual segmentation/normalization examples using the methods for the three cases; the first, second, and third blocks show examples of T-F, F-T, and F-F cases, respectively. First, out of 32 T-F cases, MeCab+ER incorrectly segmented tokens as onomatopoeia in 18 cases. For example, (a) and (b) correspond to new nodes added by the rules for the XQY_ri_-form and XYXY-form onomatopoeia, respectively, even though (a) is a verb phrase and (b) is a repetition of interjections.

Second, out of 200 F-T cases that only MeCab+ER correctly segmented, the method correctly normalized 119 cases, such as (c), (d), and the first word in (g), and incorrectly normalized 42 cases, such as (e) and the second word in (f). The remaining 39 cases were tokens that required no normalization, such as the first word in (f), the second word in (g), and (h). The method correctly normalized simple examples of sound change variants (c: \(\vdash\neg\nmid{}^{*}\!\leftarrow\!\!\downarrow\) for \(\lnot\nmid{}^{*}\!\Downarrow\!\!\downarrow\)) and alternative representations (d: \(\uplus\vdot\upharpoonright\) for \(\uplus\vdot\upharpoonright\)) because of the substitution and insertion rules, but failed to normalize character type variants (f: \(\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\) and complicated sound change variants (e: \(\lnot\nmid{}^{*}\!\cup\!\!\uparrow\!\!\!\downarrow\) for \(\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\uplus\)).

Third, out of 413 F-F cases, 148 tokens were complicated variant forms, including a combination of historical kana orthography and the insertion of the long sound symbol (i), a combination of the character type variant and sound change variant (j), a variant written in _romaji_ (k). The remaining 265 tokens were other unknown words, including emoticons (l), neologisms/slang (m), and proper names (n).18

Footnote 18: §E. [#]_shawari_ is an abbreviation of §E. [#]_shain waribiki_ ‘employee discount’. \(\nexists\mu\nmid{}^{*}\!\neq\!\!\nmid{}^{*}\!\neq\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{ *}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\! \nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{} ^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\! \!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{} ^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\! \!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{} ^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\! \!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{} ^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\! \!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{} ^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\! \!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\! \nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{} ^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\! \!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\! \nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{} ^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\! \!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\! \nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{} ^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\! \nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\!\nmid{}^{*}\!\! \nmid{}^{*}\! 

tives (FN). Similarly, among 354 predictions (Pred), the methods incorrectly normalized 156 (99+57) false positives (FP). We further divided FN and FP according to whether they were correctly segmented (T-SEG) or not (F-SEG).

We do not show TP and FN examples here because we already introduced some examples in SS5.4. Among the FP examples, some of them were not necessarily inappropriate results; normalization between similar interjections and onomatopoeia was intuitively acceptable (e.g., \(\nexists\exists\nonscript>\) was normalized to \(\nexists\exists\nonscript>\)\(\bar{\alpha}\) 'oh' and \(\nexists\not\exists\nonscript>\)\(\overline{\gamma}\) 'was normalized to \(\uplus\neg\nonscript>\)\(\overline{\gamma}\)_sarasara_ 'smoothly'). However, we assessed these as errors based on our criterion that interjections have no (non-)standard forms and the BCCWJ guidelines that regards onomatopoeia with and without long sound insertion as different lemmas.

### Discussion

The derivation rules used in McCab+ER improved segmentation and POS tagging performance and contributed to the correct normalization of parts of variant forms, but the overall normalization performance was limited to F\({}_{1}\) of 35.3%.

We classified the main segmentation and normalization errors into two types: complicated variant forms and unknown words of specific vocabulary types such as emoticons and neologisms/slang. The effective use of linguistic resources may be required to build more accurate systems, for example, discovering variant form candidates from large raw text similar to [20], and constructing/using term dictionaries of specific vocabulary types.

## 6 Related Work

**UGT Corpus for MA and LN**Hashimoto et al. (2011) developed a Japanese blog corpus with morphological, grammatical, and sentiment information, but it contains only 38 non-standard forms and 102 misspellings as UGT-specific examples. Osaki et al. (2017) constructed a Japanese Twitter corpus annotated with morphological information and standard word forms. Although they published tweet URLs along with annotation information, we could only restore parts of sentences because of the deletion of the original tweets. Sasano et al. (2013); Kaji and Klisuregawa (2014); Saito et al. (2014, 2017) developed Japanese MA and LN methods for UGT, but most of their in-house data are not publicly available.

For English LN, Han and Baldwin (2011) constructed an English Twitter corpus and Yang and Eisenstein (2013) revised it as LexNorm 1.2. Baldwin et al. (2015) constructed an English Twitter corpus (LexNorm2015) for the W-NUT 2015 text normalization shared task. Both LexNorm 1.2 and LexNorm2015 have been used as benchmark

\begin{table}
\begin{tabular}{c|c c|c c} \hline \hline Total & \multicolumn{2}{c|}{T-SEG} & \multicolumn{2}{c}{F-SEG} \\ \hline Gold & 767 & TP & 198 & FN & 58 : & 511 \\ \hline Pred & 354 & TP & 198 & FP & 99 : & 57 \\ \hline \end{tabular}
\end{table}
Table 10: Detailed normalization results for MeCab+ER

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & VT & Gold SE\&SForms & Reading & Translation & McCab result & MeCab+ER result \\ \hline (a) & & \(\downarrow\nonscript>\)\(\ket{\nearrow}\) & \(h\text{dQ}\)\(\ket{\vec{x}}\) & paste and & \(\downarrow\nonscript>\)\(\ket{\nearrow}\) & \(\uparrow\nonscript>\)\(\top\)\(\Downarrow\) \\ (b) & & \(\subseteq\nonscript>\)\(\ket{\leq\nonscript>}\) & \(kora\)\(\ket{\ket{\ket{\ket{\ket{\ket{\ket{\ket{\ket{\ket{\ket{\ket{\ket{\ket{\ket{ \ket{\ketket{\ketket{\ketket{\ketket{\ketket{   \ketket{    \ketket{     \ketket{     \ket{ 

datasets for LN systems (Jin, 2015; van der Goot, 2019; Dekker and van der Goot, 2020).

For Chinese, Li and Yarowsky (2008) published a dataset of formal- informal word pairs collected from Chinese webpages. Wang et al. (2013) released a crowdsourced corpus constructed from microblog posts on Sina Weibo.

**Classification of Linguistic Phenomena in UGT** To construct an MA dictionary, Nakamoto et al. (2000) classified unknown words occurring in Japanese chat text into contraction (e.g., \(\exists^{*}|\vec{Y}--\) for \(\neg>\!\!\succ\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! 

Conclusion

We presented a publicly available Japanese UGT corpus annotated with morphological and normalization information. Our corpus enables the performance comparison of existing and future systems and identifies the main remaining issues of MA and LN of UGT. Experiments on our corpus demonstrated the limited performance of the existing systems for non-general words and non-standard forms mainly caused by two types of difficult examples: complicated variant forms and unknown words of non-general vocabulary types.

In the future, we plan to (1) expand the corpus by further annotating of 5-10 times more sentences for a more precise evaluation and (2) develop a joint MA and LN method with high coverage.

## Acknowledgments

We would like to thank the anonymous reviewers and area chairs for their constructive comments. We used the Balanced Corpus of Contemporary Written Japanese to construct our corpus.

## References

* Baldwin et al. [2015] Timothy Baldwin, Marie Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. In _Proceedings of the Workshop on Noisy User-generated Text_, pages 126-135, Beijing, China. Association for Computational Linguistics.
* Dekker and van der Goot [2020] Kelly Dekker and Rob van der Goot. 2020. Synthetic data for English lexical normalization: How close can we get to manually annotated data? In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 6300-6309, Marseille, France. European Language Resources Association.
* Den [2009] Yasuharu Den. 2009. A multi-purpose electronic dictionary for morphological analyzers [in Japanese]. _Journal of the Japanese Society for Artificial Intelligence_, 34(5):640-646.
* Den et al. [2007] Yasuharu Den, Toshinobu Ogiso, Hideki Ogura, Atusshi Yamada, Nobuaki Minematsu, Kiyotaka Uchiimoto, and Hanae Koiso. 2007. The development of an electronic dictionary for morphological analysis and its application to Japanese corpus linguistics [in Japanese]. _Japanese Linguistics_, 22:101-123.
* Han and Baldwin [2011] Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a #twitter. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 368-378, Portland, Oregon, USA. Association for Computational Linguistics.
* Hashimoto et al. [2011] Chikara Hashimoto, Sadao Kurohashi, Daisuke Kawahara, Keiji Shinzato, and Masaaki Nagata. 2011. Construction of a blog corpus with syntactic, anaphoric, and sentiment annotations [in Japanese]. _Journal of Natural Language Processing_, 18(2):175-201.
* Ikeda et al. [2010] Kazashi Ikeda, Tadashi Yanagihara, Kazunori Matsumoto, and Yasuhiro Takishima. 2010. Automatic rule generation approach for morphological analysis of peculiar expressions on blog documents [in Japanese]. _IPSJ Transactions on Databases_, 3(3):68-77.
* Jin [2015] Ning Jin. 2015. NCSU-SAS-ning: Candidate generation and feature engineering for supervised lexical normalization. In _Proceedings of the Workshop on Noisy User-generated Text_, pages 87-92, Beijing, China. Association for Computational Linguistics.
* Kaji and Kitsuregawa [2014] Nobuhiro Kaji and Masaru Kitsuregawa. 2014. Accurate word segmentation and pos tagging for Japanese microblogs: Corpus annotation and joint modeling with lexical normalization. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 99-109, Doha, Qatar. Association for Computational Linguistics.
* Kaji et al. [2015] Nobuhiro Kaji, Shinsuke Mori, Furihiko Takahashi, Testuro Sasada, Itsumi Saito, Keigo Hattori, Yugo Murawaki, and Kei Utsumi. 2015. Ketasio kiosicki no error bunseki (Error analysis of morphological analysis). In _Proceedings of the 21th Annual Meeting of the Association for Natural Language Processing Workshop "Error Analysis on Natural Language Processing"_, Kyoto, Japan.
* Katdo et al. [2004] Taku Katdo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying conditional random fields to Japanese morphological analysis. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_, pages 230-237, Barcelona, Spain. Association for Computational Linguistics.
* Li and Yarowsky [2008] Zhifei Li and David Yarowsky. 2008. Mining and modeling relations between formal and informal Chinese phrases from web corpora. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_, pages 1031-1040, Honolulu, Hawaii. Association for Computational Linguistics.
* Maekawa et al. [2014] Kikuo Maekawa, Makoto Yamazaki, Toshinobu Ogiso, Takehiko Maruyama, Hideki Ogura, Wakako Kashino, Hanae Koiso, Masaya Yamaguchi, Makiro Tanaka, and Yasuharu Den. 2014. Balanced corpus of contemporary written Japanese. _Language Resources and Evaluation_, 48(2):345-371 

Yasunari Nakamoto, Kazuya Mera, and Aizawa terutaki. 2000. Using sequence alignment to improve the morphological analysis [in Japanese]. _IPSJ SIG Technical Report_, 2000(11):87-93.
* Neubig et al. (2011a) Graham Neubig, Yosuke Nakata, and Shinsuke Mori. 2011a. Pointwise prediction for robust, adaptable Japanese morphological analysis. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 529-533, Portland, Oregon, USA. Association for Computational Linguistics.
* Ogura et al. (2011a) Hideki Ogura, Hanae Koiso, Yumi Fujiike, Sayaka Miyauchi, Hikari Konishi, and Yutaka Hara. 2011a. Gendai inihong kakikitotoba kink\({}^{\text{c}}\) corpus ketairon kiteshu dai 4 ban ge (Regulations of morphologically information for balanced corpus of contemporary written Japanese 4th edition volume 2). _NINIAL Internal Reports_.
* Ogura et al. (2011b) Hideki Ogura, Hanae Koiso, Yumi Fujiike, Sayaka Miyauchi, Hikari Konishi, and Yutaka Hara. 2011b. Gendai inihong kakkitotoba kink\({}^{\text{c}}\) corpus ketairon kiteshu dai 4 ban jo (Regulations of morphological information for balanced corpus of contemporary written Japanese 4th edition volume 1). _NINIAL Internal Reports_.
* Osaki et al. (2017) Ayaha Osaki, Yoshiaki Kitagawa, and Mamoru Komachi. 2017. Nihong Twitter bunsho wo taisho to shitta keretsu labeling ni yoru hyoki sekika (Text normalization by sequence labeling for Japanese Twitter documents). _IPSJ SIG Technical Report_, 2017-NL-231(12):1-6.
* Qian et al. (2015) Tao Qian, Yue Zhang, Meishan Zhang, Yafeng Ren, and Donghong Ji. 2015. A transition-based model for joint segmentation, POS-tagging and normalization. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1837-1846, Lisbon, Portugal. Association for Computational Linguistics.
* Saito et al. (2017) Itsumi Saito, Kyousuke Nishida, Kagatsu Sadamitsu, Kuniko Saito, and Junji Tomita. 2017. Automatically extracting variant-normalization pairs for Japanese text normalization. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 937-946, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Saito et al. (2014) Itsumi Saito, Kugatsu Sadamitsu, Hisako Asano, and Yoshihiro Matsuo. 2014. Morphological analysis for Japanese noisy text based on character-level and word-level normalization. In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_, pages 1773-1782, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
* Sasano et al. (2013) Ryohei Sasano, Sadao Kurohashi, and Manabu Okumura. 2013. A simple approach to unknown word processing in japanese morphological analysis. In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_, pages 162-170, Nagoya, Japan. Asian Federation of Natural Language Processing.
* Sato (2009) Ryoichi Sato. 2009. _Todgfuken bestu zenkkou hogen jiiten (Dialect Dictionary of Japanese Prefectures)_. Sanseido.
* Tolmachev et al. (2020) Arseny Tolmachev, Daisuke Kawahara, and Sadao Kurohashi. 2020. Design and structure of the Juan++ morphological analyzer toolkit. _Journal of Natural Language Processing_, 27(1):89-132.
* van der Goot (2019) Rob van der Goot. 2019. MoNoise: A multi-lingual and easy-to-use lexical normalization tool. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 201-206, Florence, Italy. Association for Computational Linguistics.
* van der Goot et al. (2018) Rob van der Goot, Rick van Noord, and Gertjan van Noord. 2018. A taxonomy for in-depth evaluation of normalization for user generated content. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation_, Miyazaki, Japan. European Language Resources Association.
* Wang et al. (2013) Aobo Wang, Min-Yen Kan, Daniel Andrade, Takashi Onishi, and Kai Ishikawa. 2013. Chinese informal word normalization: an experimental study. In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_, pages 127-135, Nagoya, Japan. Asian Federation of Natural Language Processing.
* Yamaguchi (2002) Nakami Yamaguchi. 2002. _Giongo gitiago jiten (Phonomime and Phenomime Dictionary)_. Kodansha.
* Yang and Eisenstein (2013) Yi Yang and Jacob Eisenstein. 2013. A log-linear model for unsupervised text normalization. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 61-72, Seattle, Washington, USA. Association for Computational Linguistics.