SlowGuess/NAACL_2021_OCR huggingface dataset:
Second, out of 200 F-T cases that only MeCab+ER correctly segmented, the method correctly normalized 119 cases, such as (c), (d), and the first word in (g), and incorrectly normalized 42 cases, such as (e) and the second word in (f). The remaining 39 cases were tokens that required no normalization, such as the first word in (f), the second word in (g), and (h). The method correctly normalized simple examples of sound change variants (c: 一か一も for かも) and alternative representations (d: いら for いら) because of the substitution and insertion rules, but failed to normalize character type variants (f: サ一for 野球) and complicated sound change variants (e: サは for まむ).
Third, out of 413 F-F cases, 148 tokens were complicated variant forms, including a combination of historical kana orthography and the insertion of the long sound symbol (i), a combination of the character type variant and sound change variant (j), a variant written in romaji (k). The remaining 265 tokens were other unknown words, including emoticons (l), neologisms/slang (m), and proper names (n).<sup>18</sup>
5.5 Analysis of the Normalization Results
Table 10 shows the detailed normalization results for MeCab+ER. Among 767 non-standard words (Gold), the method correctly normalized 198 true positives (TP) and missed 569  $(58 + 511)$  false nega
18社割shawariisanabbreviationof社員劃引shain waribiki‘employee discount’.《Gulbadia'is an imaginary location name in the video game Final Fantasy.
VT	Gold SEG&SForms	Reading	Translation	MeCab result	MeCab+ER result
(a)	はつ|たり	haQ|tari	paste and	はつ|たり	はつたり
(b)	こら|こら	kora|kora	hey hey	こら|こら	こらこら
(c)	S	し一か一も [しかも]	shikāmo	besides	し一|か一|も	し一か一も [しかも]
(d)	A	おいら [おいら,オイヤ]	oira	I	おいら	おいら [おいら]
(e)	S	んまい [美味い,旨い,うまい]	mmai	yummy	ん|ま|い	んまい [んまい]
(f)	C,A	も|やきゆ一 [野球]	mo|yakyū	also, baseball	もや|きゆ一	も|やきゆ一 [やきゆう]
(g)	S	たしぃ一 [確か,たしぃ] |に	tashīka|ni	surely	た|し一|かに	たしぃ一[たしぃ] |に
(h)	ふう～～ん	fūn	hmm	ふう～|~|ん	ふう～～ん [ふん]
(i)	S	ませら～ [ましぃょう]	mashō	let's	ませ|う|~	ませ|う～ [う]
(j)	C,S	けこ一人 [結婚]	kekon	marriage	けこ一人	けこ一人 [う]
(k)	A	ください|ne [ね]	kudasai|ne	Won't you...?	ください|n|e	ください|n|e
(l)	(^[^^-])	(|^[^^-|^-])	(|^[^^-|^-])
(m)	社割	shawari	employee discount	社割	社割
(n)	ガルバデア	garubadhia	Galbadia	ガルバ|デア	ガルバ|デア
Table 9: Segmentation and normalization results (shown in “[”) by MeCab and MeCab+ER. Incorrect results are written in gray. VT represents variant type. C, A, and S represent character type variant, alternative representation, and sound change variants, respectively. Gold SEG&SForms represent the gold segmentation and gold standard forms (shown in “[]).
Total	T-SEG	F-SEG
Gold	767	TP	198	FN	58	511
Pred	354	TP	198	FP	99	57
Table 10: Detailed normalization results for MeCab+ER
tives (FN). Similarly, among 354 predictions (Pred), the methods incorrectly normalized 156  $(99 + 57)$  false positives (FP). We further divided FN and FP according to whether they were correctly segmented (T-SEG) or not (F-SEG).
We do not show TP and FN examples here because we already introduced some examples in §5.4. Among the FP examples, some of them were not necessarily inappropriate results; normalization between similar interjections and onomatopoeia was intuitively acceptable (e.g., おお～was normalized toおおō‘oh’and薩ラ薩～was normalized to薩拉薩 sarasara ‘smoothly'). However, we assessed these as errors based on our criterion that interjections have no (non-)standard forms and the BCCWJ guidelines that regards onomatopoeia with and without long sound insertion as different lemmas.
The derivation rules used in MeCab+ER improved segmentation and POS tagging performance and contributed to the correct normalization of parts of variant forms, but the overall normalization performance was limited to  $\mathrm{F}_1$  of 35.3%.
We classified the main segmentation and nor
malization errors into two types: complicated variant forms and unknown words of specific vocabulary types such as emoticons and neologisms/slang. The effective use of linguistic resources may be required to build more accurate systems, for example, discovering variant form candidates from large raw text similar to (Saito et al., 2017), and constructing/using term dictionaries of specific vocabulary types.
UGT Corpus for MA and LN Hashimoto et al. (2011) developed a Japanese blog corpus with morphological, grammatical, and sentiment information, but it contains only 38 non-standard forms and 102 misspellings as UGT-specific examples. Osaki et al. (2017) constructed a Japanese Twitter corpus annotated with morphological information and standard word forms. Although they published tweet URLs along with annotation information, we could only restore parts of sentences because of the deletion of the original tweets. Sasano et al. (2013); Kaji and Kitsuregawa (2014); Saito et al. (2014, 2017) developed Japanese MA and LN methods for UGT, but most of their in-house data are not publicly available.
For English LN, Han and Baldwin (2011) constructed an English Twitter corpus and Yang and Eisenstein (2013) revised it as LexNorm 1.2. Baldwin et al. (2015) constructed an English Twitter corpus (LexNorm2015) for the W-NUT 2015 text normalization shared task. Both LexNorm 1.2 and LexNorm2015 have been used as benchmark
datasets for LN systems (Jin, 2015; van der Goot, 2019; Dekker and van der Goot, 2020).
For Chinese, Li and Yarowsky (2008) published a dataset of formal-informal word pairs collected from Chinese webpages. Wang et al. (2013) released a crowdsourced corpus constructed from microblog posts on Sina Weibo.
Classification of Linguistic Phenomena in UGT To construct an MA dictionary, Nakamoto et al. (2000) classified unknown words occurring in Japanese chat text into contraction (e.g.,rooms forroomsugoi 'awesome'), exceptional kana variant (e.g.,room for room computer'), abbreviation, typographical errors, filler, phonomime and phenomenime, proper nouns, and other types. Ikeda et al. (2010) classified "peculiar expressions" in Japanese blogs into visual substitution (e.g.,room for room 'me'), sound change (e.g.,room for room 'dekai big'), kana substitution (e.g.,room for room 'vitamin'), and other unknown words into similar categories to Nakamoto et al. (2000). Kaji et al. (2015) performed error analysis of Japanese MA methods on Twitter text. They classified missegmented words into a dozen categories, including spoken or dialect words, onomatopoeia, interjections, emoticons/AA, proper nouns, foreign words, misspelled words, and other non-standard word variants. Ikeda et al. (2010)'s classification of peculiar expressions is most similar to our types of variant forms and Kaji et al. (2015)'s classification is most similar to our types of vocabulary (shown in Table 2), whereas we provide more detailed definitions of categories and criteria for standard and non-standard forms. Other work on Japanese MA and LN did not consider diverse phenomena in UGT (Sasano et al., 2013; Saito et al., 2014).
For English, Han and Baldwin (2011) classified ill-formed English words on Twitter into extra/missing letters and/or number substitution (e.g., "b4" for "before"), slang (e.g., "lol" for "laugh out loud"), and "others". van der Goot et al. (2018) defined a more comprehensive taxonomy with 14 categories for a detailed evaluation of English LN systems. It includes phrasal abbreviation (e.g., "idk" for "I don't know"), repetition (e.g., "soooo" for "so"), and phonetic transformation (e.g., "hackd" for "hacked").
For Chinese, Li and Yarowsky (2008) classified informal words in Chinese webpages into four types: homophone (informal words with similar
pronunciation to formal words, e.g., 稀饭 ⟨xīfàn⟩ $^{19}$  “rice gruel” for 喜欢 ⟨xīhuan⟩ “like”, abbreviation and acronym (e.g., GG for 哥哥 ⟨gēge⟩ “elder brother”), transliteration (informal words are transliteration of English translation of formal words, e.g., 3Q ⟨sānqiu⟩ for 谢谢 ⟨xièxie⟩ “thank you”), and “others”. Wang et al. (2013) also classified informal words in Chinese microblog posts similar to Li and Yarowsky (2008).
Methods for MA and LN In the last two decades, previous work has explored various rules and extraction methods for formal-informal word pairs to enhance Japanese MA and LN models for UGT. Nakamoto et al. (2000) proposed an alignment method based on string similarity between original and variant forms. Ikeda et al. (2010) automatically constructed normalization rules of peculiar expressions in blogs, based on frequency, edit distance, and estimated accuracy improvements. Sasano et al. (2013) defined derivation rules to recognize unknown onomatopoeia and variant forms of known words that frequently occur in webpages. Their rules were also implemented in a recent MA toolkit Juman++ (Tolmachev et al., 2020) to handle unknown words. Saito et al. (2014) estimated character-level alignment from manually annotated pairs of formal and informal words on Twitter. Saito et al. (2017) extracted formal-informal word pairs from unlabeled Twitter data based on semantic and phonetic similarity.
For English and Chinese, various classification methods for normalization of informal words (Li and Yarowsky, 2008; Wang et al., 2013; Han and Baldwin, 2011; Jin, 2015; van der Goot, 2019) have been developed based on, for example, string, phonetic, semantic similarity, or co-occurrence frequency. Qian et al. (2015) proposed a transition-based method with append(x), separate(x), and separate_and_substitute(x,y) operations for the joint word segmentation, POS tagging, and normalization of Chinese microblog text. Dekker and van der Goot (2020) automatically generated pseudo training data from English raw tweets using noise insertion operations to achieve comparable performance without manually annotated data to an existing LN system.
19Pinyin pronunciation is shown in  $" \langle "\rangle "$
We presented a publicly available Japanese UGT corpus annotated with morphological and normalization information. Our corpus enables the performance comparison of existing and future systems and identifies the main remaining issues of MA and LN of UGT. Experiments on our corpus demonstrated the limited performance of the existing systems for non-general words and non-standard forms mainly caused by two types of difficult examples: complicated variant forms and unknown words of non-general vocabulary types.
In the future, we plan to (1) expand the corpus by further annotating of 5-10 times more sentences for a more precise evaluation and (2) develop a joint MA and LN method with high coverage.
We would like to thank the anonymous reviewers and area chairs for their constructive comments. We used the Balanced Corpus of Contemporary Written Japanese to construct our corpus.
Timothy Baldwin, Marie Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. In Proceedings of the Workshop on Noisy User-generated Text, pages 126-135, Beijing, China. Association for Computational Linguistics.
Kelly Dekker and Rob van der Goot. 2020. Synthetic data for English lexical normalization: How close can we get to manually annotated data? In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6300-6309, Marseille, France. European Language Resources Association.
Yasuharu Den. 2009. A multi-purpose electronic dictionary for morphological analyzers [in Japanese]. Journal of the Japanese Society for Artificial Intelligence, 34(5):640-646.
Yasuharu Den, Toshinobu Ogiso, Hideki Ogura, Atsushi Yamada, Nobuaki Minematsu, Kiyotaka Uchimoto, and Hanae Koiso. 2007. The development of an electronic dictionary for morphological analysis and its application to Japanese corpus linguistics [in Japanese]. *Japanese Linguistics*, 22:101-123.
Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a #twitter. In Proceedings of the 49th Annual Meeting of
the Association for Computational Linguistics: Human Language Technologies, pages 368-378, Portland, Oregon, USA. Association for Computational Linguistics.
Chikara Hashimoto, Sadao Kurohashi, Daisuke Kawahara, Keiji Shinzato, and Masaaki Nagata. 2011. Construction of a blog corpus with syntactic, anaphoric, and sentiment annotations [in Japanese]. Journal of Natural Language Processing, 18(2):175-201.
Kazushi Ikeda, Tadashi Yanagihara, Kazunori Matsumoto, and Yasuhiro Takishima. 2010. Automatic rule generation approach for morphological analysis of peculiar expressions on blog documents [in Japanese]. IPSJ Transactions on Databases, 3(3):68-77.
Ning Jin. 2015. NCSU-SAS-ning: Candidate generation and feature engineering for supervised lexical normalization. In Proceedings of the Workshop on Noisy User-generated Text, pages 87–92, Beijing, China. Association for Computational Linguistics.
Nobuhiro Kaji and Masaru Kitsuregawa. 2014. Accurate word segmentation and pos tagging for Japanese microblogs: Corpus annotation and joint modeling with lexical normalization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 99-109, Doha, Qatar. Association for Computational Linguistics.
Nobuhiro Kaji, Shinsuke Mori, Fumihiko Takahashi, Tetsuro Sasada, Itsumi Saito, Keigo Hattori, Yugo Murawaki, and Kei Utsumi. 2015. Ketaiso kaiseki no error bunseki (Error analysis of morphological analysis). In Proceedings of the 21th Annual Meeting of the Association for Natural Language Processing Workshop "Error Analysis on Natural Language Processing", Kyoto, Japan.
Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying conditional random fields to Japanese morphological analysis. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 230-237, Barcelona, Spain. Association for Computational Linguistics.
Zhifei Li and David Yarowsky. 2008. Mining and modeling relations between formal and informal Chinese phrases from web corpora. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1031-1040, Honolulu, Hawaii. Association for Computational Linguistics.
Kikuo Maekawa, Makoto Yamazaki, Toshinobu Ogiso, Takehiko Maruyama, Hideki Ogura, Wakako Kashino, Hanae Koiso, Masaya Yamaguchi, Makiro Tanaka, and Yasuharu Den. 2014. Balanced corpus of contemporary written Japanese. Language Resources and Evaluation, 48(2):345-371.
Yasunari Nakamoto, Kazuya Mera, and Aizawa teru-aki. 2000. Using sequence alignment to improve the morphological analysis [in Japanese]. IPSJ SIG Technical Report, 2000(11):87-93.
Graham Neubig, Yosuke Nakata, and Shinsuke Mori. 2011. Pointwise prediction for robust, adaptable Japanese morphological analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 529-533, Portland, Oregon, USA. Association for Computational Linguistics.
Hideki Ogura, Hanae Koiso, Yumi Fujiike, Sayaka Miyauchi, Hikari Konishi, and Yutaka Hara. 2011a. Gendai.nihongo kakikotoba kinkō corpus kētairon kitēshū dai 4 ban ge (Regulations of morphological information for balanced corpus of contemporary written Japanese 4th edition volume 2). NINJAL Internal Reports.
Hideki Ogura, Hanae Koiso, Yumi Fujiike, Sayaka Miyauchi, Hikari Konishi, and Yutaka Hara. 2011b. Gendai.nihongo kakikotoba kinkō corpus kētairon kitēshū dai 4 ban jō (Regulations of morphological information for balanced corpus of contemporary written Japanese 4th edition volume 1). NINJAL Internal Reports.
Ayaha Osaki, Yoshiaki Kitagawa, and Mamoru Komachi. 2017. Nihongo Twitter bunsho wo taishō to shita kēretsu labeling ni yoru hyōki sekika (Text normalization by sequence labeling for Japanese Twitter documents). IPSJ SIG Technical Report, 2017-NL-231(12):1-6.
Tao Qian, Yue Zhang, Meishan Zhang, Yafeng Ren, and Donghong Ji. 2015. A transition-based model for joint segmentation, POS-tagging and normalization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1837-1846, Lisbon, Portugal. Association for Computational Linguistics.
Itsumi Saito, Kyosuke Nishida, Kugatsu Sadamitsu, Kuniko Saito, and Junji Tomita. 2017. Automatically extracting variant-normalization pairs for Japanese text normalization. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 937-946, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Itsumi Saito, Kugatsu Sadamitsu, Hisako Asano, and Yoshihiro Matsuo. 2014. Morphological analysis for Japanese noisy text based on character-level and word-level normalization. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1773-1782, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Ryohei Sasano, Sadao Kurohashi, and Manabu Okumura. 2013. A simple approach to unknown word processing in Japanese morphological analysis. In
Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 162-170, Nagoya, Japan. Asian Federation of Natural Language Processing.
Ryoichi Sato. 2009. *Todofuken betsu zenkoku hogen jiten (Dialect Dictionary of Japanese Prefectures). Sanseido.*
Arseny Tolmachev, Daisuke Kawahara, and Sadao Kurohashi. 2020. Design and structure of the Juman++ morphological analyzer toolkit. Journal of Natural Language Processing, 27(1):89-132.
Rob van der Goot. 2019. MoNoise: A multi-lingual and easy-to-use lexical normalization tool. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 201-206, Florence, Italy. Association for Computational Linguistics.
Rob van der Goot, Rik van Noord, and Gertjan van Noord. 2018. A taxonomy for in-depth evaluation of normalization for user generated content. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, Miyazaki, Japan. European Language Resources Association.
Aobo Wang, Min-Yen Kan, Daniel Andrade, Takashi Onishi, and Kai Ishikawa. 2013. Chinese informal word normalization: an experimental study. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 127-135, Nagoya, Japan. Asian Federation of Natural Language Processing.
Nakami Yamaguchi. 2002. Giongo gitaigo jiten (Phonomime and Phenomime Dictionary). Kodansha.
Yi Yang and Jacob Eisenstein. 2013. A log-linear model for unsupervised text normalization. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 61-72, Seattle, Washington, USA. Association for Computational Linguistics.