SlowGuess/NAACL_2021_OCR huggingface dataset:
Shohei Higashiyama $^{1,2}$ , Masao Utiyama $^{1}$ , Taro Watanabe $^{2}$ , Eiichiro Sumita $^{1}$
$^{1}$ National Institute of Information and Communications Technology, Kyoto, Japan
$^{2}$ Nara Institute of Science and Technology, Nara, Japan
{shohei.higashiyama, mutiyama, eiichiro-sumita}@nict.go.jp, taro@is.naist.jp
Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sentences annotated with morphological and normalization information, along with category information we classified for frequent UGT-specific phenomena. Experiments on the corpus demonstrated the low performance of existing MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT.
Japanese morphological analysis (MA) is a fundamental and important task that involves word segmentation, part-of-speech (POS) tagging and lemmatization because the Japanese language has no explicit word delimiters. Although MA methods for well-formed text (Kudo et al., 2004; Neubig et al., 2011) have been actively developed taking advantage of the existing annotated corpora of newswire domains, they perform poorly on user-generated text (UGT), such as social media posts and blogs. Additionally, because of the frequent occurrence of informal words, lexical normalization (LN), which identifies standard word forms, is another important task in UGT. Several studies have been devoted to both tasks in Japanese UGT (Sasano et al., 2013; Kaji and Kitsuregawa, 2014; Saito et al., 2014, 2017) to achieve the robust performance for noisy text. Previous researchers have evaluated their own systems using in-house data created by individual researchers, and thus it is difficult to compare the performance of different systems and discuss what issues remain in these two tasks. Therefore, publicly available data is necessary for a fair evaluation of MA and LN performance on Japanese UGT.
In this paper, we present the blog and Q&A site normalization corpus (BQNC),<sup>1</sup> which is a public Japanese UGT corpus annotated with morphological and normalization information. We have constructed the corpus under the following policies: (1) available and restorable; (2) compatible with the segmentation standard and POS tags used in the existing representative corpora; and (3) enabling a detailed evaluation of UGT-specific problems.
For the first requirement, we extracted and used the raw sentences in the blog and Q&A site registers compiled by (the non-core data of) the Balanced Corpus of Contemporary Written Japanese (BCCWJ) (Maekawa et al., 2014), in which the original sentences are preserved. For the second requirement, we followed the short unit word (Suw) criterion of the National Institute for Japanese Language and Linguistics (NINJAL), which is used in various NINJAL's corpora, including manually annotated sentences in the BCCWJ. For the third requirement, we organized linguistic phenomena frequently observed in the two registers as word categories, and annotated each word with a category. We expect that this will contribute to future research to develop systems that manage UGT-specific problems.
The BQNC comprises sentence IDs and annotation information, including word boundaries, POS, lemmas, standard forms of non-standard word tokens, and word categories. We will release the annotation information that enables BCCWJ applicants to replicate the full BQNC data from the original BCCWJ data.<sup>3</sup>
Using the BQNC, we evaluated two existing
1Our corpus will be available at https://github. com/shigashiyama/jlexnorm.
${}^{2}$  Twitter could be a candidate for a data source. However, redistributing original tweets collected via the Twitter Streaming APIs is not permitted by Twitter, Inc., and an alternative approach to distributing tweet URLs has the disadvantage that the original tweets can be removed in the future.
3https://pj.ninja1.ac.jp/corpus_center/bccwj/en/subscription.html
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5532-5541 June 6-11, 2021. ©2021 Association for Computational Linguistics
Category	Example	Reading	Translation	Standard forms
Type of vocabulary	(General words)
Neologisms/Slang	コロpace	copipe	copy and paste
Proper names	ドラクエ	dorakue	Dragon Quest
Onomatopoeia	キラキラ	kirakira	glitter
Interjections	おお	ō	oops
Dialect words	ほんま	homma	truly
Foreign words	E A S Y	easy
Emoticons/AA	(^-^)
Type of variant form	(Standard forms)
Character type variants	カロイイ	kawāi	cute	かわい,可愛い
Alternative representations	大小き	ōki	big	大小き
Sound change variants	おしぃしée	oishū	tasty	おいし,美味い
Typographical errors	つたい	tsutai	tough	つらい,幸い
Table 1: Word categories in the BQNC
methods: a popular Japanese MA toolkit called MeCab (Kudo et al., 2004) and a joint MA and LN method (Sasano et al., 2013). Our experiments and error analysis showed that these systems did not achieve satisfactory performance for non-general words. This indicates that our corpus would be a challenging benchmark for further research on UGT.
2 Overview of Word Categories
Based on our observations and the existing studies (Ikeda et al., 2010; Kaji et al., 2015), we organized word tokens that may often cause segmentation errors into two major types with several categories, as shown in Table 1. We classified each word token from two perspectives: the type of vocabulary to which it belongs and the type of variant form to which it corresponds. For example, ニhon nihon 'Japan' written in katakana corresponds to a proper name and a character type variant of its standard form 日本 written in kanji.
Specifically, we classified vocabulary types into neologisms/slang, proper names, onomatopoeia, $^{4}$  interjections, (Japanese) dialect words, foreign words, and emoticons/ASCII art (AA), in addition to general words. $^{5}$  A common characteristic of these vocabularies, except for general words, is that a new word can be indefinitely invented or imported. We annotated word tokens with vocabulary type information, except for general words.
From another perspective, any word can have multiple variant forms. Because the Japanese writ
ing system comprises multiple script types including kanji and two types of kana, that is, hiragana and katakana, $^{6}$  words have orthographic variants written in different scripts. Among them, non-standard character type variants that rarely occur in well-formed text but occur in UGT can be problematic, for example, a non-standard form カイイ for a standard form カわいī kawāi ‘cute’. Additionally, ill-spelled words are frequently produced in UGT. We further divided them into two categories. The first is sound change variants that have a phonetic difference from the original form and are typically derived by deletions, insertions, or substitutions of vowels, long sound symbols (chōon “一”), long consonants (sokuon “つ”), and moraic nasals (hatsuon “ん”), for example, しおし一い oishū for しおしい oishi ‘tasty’. The second category is alternative representations that do not have a phonetic difference and are typically achieved by substitution among uppercase or lowercase kana characters, or among vowel characters and long sound symbols, for example, 大きい for 大きいōki ‘big’. Moreover, typographical errors can be seen as another type of variant form. We targeted these four types of non-standard forms for normalization to standard forms.
3 Corpus Construction Process
The BQNC was constructed using the following steps. The annotation process was performed by the first author.
${}^{6}$  Morphographic kanji and syllabographic hiragana are primarily used for Japanese native words (wago) and Japanese words of Chinese origin (Sino-Japanese words or kango), whereas syllabographic katakana is primarily used, for example, for loanwords, onomatopoeia, and scientific names. Additionally, Arabic numerals, Latin letters (rōmaji), and other auxiliary symbols are used in Japanese sentences.
4"Onomatopoeia" typically refers to both the phonomime and phenomenime in Japanese linguistics literature, similar to ideophones. We follow this convention in this paper.
5We observed a few examples of other vocabulary types, such as Japanese archaic words and special sentence-final particles in our corpus, but we treated them as general words.
Token	Translation	Standard form ID
イイ	good	38988:良い
歌	song
歌词	(polite copula)
ね工	(emphasis marker)	28754:ね
ヨカック	good	38988:良い_ヨカック
夕	(past tense marker)	21642:た
Table 2: Examples of annotated text “イイ歌です” (It's a good song, isn't it?) and “ヨカツダ” (It was good.). Attributes except for token and standard form ID are abbreviated.
Standard form ID	Standard forms
21642:た	た
28754:ね	ね
38988:良い	良い,よい,い
38988:良い_ヨカツ	良かっ,よかつ
Table 3: Examples of standard form IDs
(1) Sentence Selection We manually selected sentences to include in our corpus from the blog and Q&A site registers in the BCCWJ non-core data. We preferentially extracted sentences that contained candidates of UGT-specific words, that is, word tokens that may belong to non-general vocabularies or correspond to non-standard forms. As a result, we collected more than 900 sentences.
(2) First Annotation Sentences in the non-core data have been automatically annotated with word boundaries and word attributes, such as POS and lemma. Following the BCCWJ annotation guidelines (Ogura et al., 2011a,b) and UniDic (Den et al., 2007), which is an electronic dictionary database designed for the construction of NINJAL's corpora, we refined the original annotations of the selected sentences by manually checking them. The refined attributes were token, POS, conjugation type, conjugation form, pronunciation, lemma, and lemma ID. Additionally, we annotated each token with a word category shown in Table 1 and a standard form ID if the token corresponded to a non-standard form.
Table 2 shows two examples of annotated sentences. We annotated each non-standard token with a standard form ID denoted as "[lemma ID]:[lemma](_,[pronunciation])", which is associated with the set of acceptable standard forms shown in Table 3.
(3) Second Annotation We rechecked all tokens in the sentences that we finished the first annotation and fixed the annotation criteria, that is, the
definitions of vocabulary types and variant form types, and standard forms for each word. Through these steps, we obtained 929 annotated sentences.
4 Detailed Definition of Word Categories
4.1 Type of Vocabulary
Through the annotation process, we defined the criteria for vocabulary types as follows.
Neologisms/Slang: a newly invented or imported word that has come to be used collectively. Specifically, we used a corpus reference application called Chunagon<sup>7</sup> and regarded a word as a neologism/slang if its frequency in the BCCWJ was less than five before the year 2000 and increased to more than ten in 2000 or later.<sup>8</sup>
Proper names: following the BCCWJ guidelines, we regarded a single word that corresponded to a proper name, such as person name, organization name, location name, and product name, as a proper name. In contrast to the BCCWJ guidelines, we also regarded an abbreviation of a proper name as a proper name, for example, "ドラクエ" in Table 1.
Onomatopoeia: a word corresponds to onomatopoeia. We referred to a Japanese onomatopoeia dictionary (Yamaguchi, 2002) to assess whether a word is onomatopoeic. We followed the criteria in the BCCWJ guidelines on what forms of words are onomatopoeic and what words are associated with the same or different lemmas.
Interjections: a word whose POS corresponds to an interjection. Although we defined standard forms for idiomatic greeting expressions registered as single words in UniDic, we did not define standard and non-standard forms for other interjections that express feelings or reactions, for example, ネー e'uh-huh' and.uwa wow'.
Foreign words: a word from non-Japanese languages. We regarded a word written in scripts in the original language as a foreign word, for example, English words written in the Latin alphabet such as "plastic". Conversely, we regarded loanwords written in Japanese scripts (hiragana, katakana, or kanji) as general words, for example, フラスチツ
<sup>7</sup>https://chunagon.ninja1.ac.jp
The original sentences were from posts published between 2004 and 2009.
Eight greeting words exist, for example, ありがう arigatō ‘thank you’ and さょうなら sayonara ‘see you’.
ク 'plastic'. Moreover, we did not regard English acronyms and abbreviations written in uppercase letters as foreign words because such words are typically also written in the Latin alphabet in Japanese sentences, for example, S N S.
Dialect words: a word from a Japanese dialect. We referred to a Japanese dialect dictionary (Sato, 2009) and regarded a word as a dialect word if it corresponded to an entry or occurred in an example sentence. We did not consider normalization from a dialect word to a corresponding word in the standard Japanese dialect.
Emoticons/AA: nonverbal expressions that comprise characters to express feelings or attitudes. Because the BCCWJ guidelines does not explicitly describe criteria on how to segment emoticon/AA expressions as words, we defined criteria to follow emoticon/AA entries in UniDic. $^{10}$
4.2 Type of Variant Form
There are no trivial criteria to determine which variant forms of a word are standard forms because most Japanese words can be written in multiple ways. Therefore, we defined standard forms of a word as all forms whose occurrence rates were approximately equal to  $10\%$  or more in the BCCWJ among forms that were associated with the same lemma. For example, among variant forms of the lemma 'omoshiroi' or 'funny' that occurred 7.9K times, major forms '面白' and 'おもしろ' accounted for  $72\%$  and  $27\%$ , respectively, and other forms, such as '才モshiロイ' and '才モシロイ', were very rare. In this case, the standard forms of this word are the two former variants. We annotated tokens corresponding to the two latter non-standard forms with the standard form IDs and the types of variant forms. We defined criteria for types of variant forms as follows.
Character type variants: among the variants written in different scripts, we regarded variants whose occurrence rates were approximately equal to  $5\%$  or less in the BCCWJ as non-standard forms of character type variants. Specifically, variants written in kanji, hiragana, or katakana for native words and Sino-Japanese words, variants written in katakana or hiragana for loanwords, variants
written in uppercase or lowercase Latin letters for English abbreviations are candidates for character type variants. We assessed whether these candidates were non-standard forms based on the occurrence rates.
Alternative representations: a form whose internal characters are (partially) replaced by special characters without phonetic differences. Specifically, non-standard forms of alternative representations include native words and Sino-Japanese words written in historical kana orthography (e.g., 思か for 思う omō/omou ‘think’), and loanwords written as an unusual[11] katakana sequence (e.g., 才才ケストラ for 才一ケストラ ‘orchestra’). Additionally, alternative representations include substitution with respect to kana: substitution of the long vowel kana by the long sound symbol (e.g., しぃし～ for しぃしぃ oishi ‘tasty’), substitution of upper/ lowercase kana by the other case (e.g., しぃし for しぃし watashi ‘me’), and phonetic or visual substitution of kana characters by Latin letters and symbols (e.g., かわ E for かわい kawai ‘cute’ and ごωにちは for ごんにちは konnichiwa ‘hello’).
Sound change variants: a form whose pronunciation is changed from the original form. Specifically, sound change variants include the insertion of special moras (e.g., 強一い tsuyōi for 強い tusyoi 'strong'), deletion of moras (e.g., くさ kusa for くさい kusai 'stinking'), and substitution of characters/moras (e.g., つす ssu for てす desu polite copula and すばさ sugē for すごい sugoi 'awesome').
Typographical errors: a form with typographical errors derived from character input errors, kanakani conversion errors, or the user's incorrect understanding. For example, つたい tsutai for つら turai 'tough' and て r for て那儿 sore 'it'.
We present the statistics of the BQNC in Table 4. It comprises 929 sentences, 12.6K word tokens, and 767 non-standard word tokens. As shown in Table 6, the corpus contains tokens of seven types of vocabulary and four types of variant form. Whereas there exist fewer than 40 instances of neologisms/slang, dialect words, foreign words, and
11We assessed whether a form is unusual if its occurrence rate was approximately equal to  $5\%$  or less in the BCCWJ similar to the case of character type variants.
10 For example, if characters expressing body parts were outside of punctuation expressing the outline of a face, the face and body parts were segmented, but both were annotated with emoticons/AA, for example, "m (. . .) m"  $\rightarrow$  "m| (. . .) |m".
Regi-
ster	# sent	# word
token	# word
type	# NSW
token	# NSW
type
Q&A	379	5,649	1,699	320	221
Blog	550	6,951	2,231	447	257
Total	929	12,600	3,419	767	420
typographical errors, each of the other category has more than 100 instances. Our corpus contains a similar number of non-standard tokens to Kaji and Kitsuregawa (2014)'s Twitter corpus (1,831 sentences, 14.3K tokens, and 793 non-standard tokens) and Osaki et al. (2017)'s Twitter corpus (1,405 sentences, 19.2K tokens, and 768 non-standard tokens). The former follows the POS tags for the Japanese MA toolkit JUMAN and the latter follows the authors own POS tags that extend NINJAL's SUW.
In the following subsections, we evaluate the existing methods for MA and LN on the BQNC and discuss correctly or incorrectly analyzed results.
5.1 Systems
We evaluated two existing methods. First, we used MeCab 0.996 (Kudo et al., 2004),<sup>12</sup> which is a popular Japanese MA toolkit based on conditional random fields. We used UniDicMA (unidic-cwj-2.3.0)<sup>13</sup> as the analysis dictionary, which contains attribute information of 873K words and MeCab's parameters (word occurrence costs and transition costs) learned from annotated corpora, including the BCCWJ (Den, 2009).
Second, we used our implementation of Sasano et al. (2013)'s joint MA and LN method. They defined derivation rules to add new nodes in the word lattice of an input sentence built by their baseline system, JUMAN. Specifically, they used the following rules: (i) sequential voicing (rendaku), (ii) substitution with long sound symbols and lowercase kana, (iii) insertion of long sound symbols and lowercase kana, (iv) repetitive onomatopoeia (XYXY-form $^{14}$ ) and (v) non-repetitive onomatopoeia (XQYri-form and XXQto-form). For example, rule (iii) adds a node of 冷たあい tsumetai as a variant form of 冷た tsumetai 'cold'
Table 4: Statistics of the BQNC. NSW represents non-standard word.
Task	MeCab	MeCab+ER
P	R	F	P	R	F
SEG	89.2	95.1	92.1	93.5	96.5	95.0
POS	87.5	93.3	90.3	91.4	94.3	92.8
NOR	-	-	-	55.9	25.8	35.3
Table 5: Overall performance
and rule (iv) adds a node of ははは uhauha 'ex-hilarated' as an onomatopoeic adverb. if the input sentences contain such character sequences.
The original implementation by Sasano et al. (2013) was an extension of JUMAN and followed JUMAN's POS tags. To adapt their approach to the SUW, we implemented their rules and used them to extend the first method of MeCab using UniDicMA. We set the costs of the new nodes by copying the costs of their standard forms or the most frequent costs of the same-form onomatopoeia, whereas Sasano et al. (2013) manually defined the costs of each type of new word. We denote this method by MeCab+ER (Extension Rules). Notably, we did not conduct any additional training to update the models' parameters for either methods.
Table 5 shows the overall performance, that is, Precision, Recall, and  $\mathbf{F}_1$  score, of both methods for SEGmentation, POS tagging $^{15}$  and NORmalization. $^{16}$  Compared with well-formed text domains, $^{17}$  the relatively lower performance  $(F_1$  of  $90 - 95\%)$  of both methods for segmentation and POS tagging indicates the difficulty of accurate segmentation and tagging in UGT. However, MeCab+ER outperformed MeCab by  $2.5 - 2.9 F_1$  points because of the derivation rules. Regarding the normalization performance of MeCab+ER, the method achieved moderate precision but low recall, which indicates its limited coverage for various variant forms in the dataset.
5.3 Results for Each Category
Table 6 shows the segmentation and POS tagging recall for both methods for each category. In contrast to the sufficiently high performance for general words, both methods performed worse for words of characteristic categories in UGT; micro average recall was at most  $79.6\%$  for segmentation
$^{12}$ https://taku910.github.io/MEcab/
13https://unidic.ninja.ac.jp/
$^{14}$ “X” and “Y” represent the same kana character(s) corresponding to one mora, “Q” represents a long consonant character “つ/ツ”, “ri” represents a character “り/リ”, and “to” represents a character “と/ト”.
15We only evaluated top-level POS.
16We regarded a predicted standard form as correct if the prediction was equal to one of the gold standard forms.
For example, Kudo et al. (2004) achieved  $\mathrm{F}_1$  of 98–99% for segmentation and POS tagging in news domains.
Category	#	MeCab	MeCab+ER
SEG	POS	SEG	POS
Dialect words	23	91.3	78.3	95.7	82.6
Proper names	103	87.4	84.5	88.4	85.4
Onomatopoeia	218	79.8	73.4	87.2	77.1
Foreign words	14	78.6	78.6	78.6	78.6
Emoticons/AA	270	73.7	64.1	73.3	63.3
Interjections	174	64.9	53.5	72.4	48.9
Neologisms/Slang	37	67.6	67.6	67.6	67.6
Sound change var.	419	50.6	47.5	82.6	76.4
Char type var.	248	71.0	62.9	78.2	69.4
Alternative rep.	132	65.2	54.6	76.5	69.0
Typos	23	47.8	30.4	47.8	30.4
Non-gen/std total	1565	68.9	61.9	79.6	70.4
Standard forms of general words	11K	98.9	97.7	98.9	97.7
Table 6: Recall for each category (SEG and POS)
Category	#	MeCab+ER
Sound change variants	419	37.0
Character type variants	248	0.0
Alternative representations	132	32.6
Typographical errors	23	0.0
and  $70.4\%$  for POS tagging ("non-gen/std total" column). MeCab+ER outperformed MeCab particularly for onomatopoeia, character type variants, alternative representations, and sound change variants. The high scores for dialect words were probably because UniDicMA contains a large portion of (19 out of 23) dialect word tokens. Interjection was a particularly difficult vocabulary type, for which both methods recognized only approximately  $50\%$  of the gold POS tags. We guess that this is because the lexical variations of interjections are diverse; for example, there are many user-generated expressions that imitate various human voices, such as laughing, crying, and screaming.
Table 7 shows the recall of MeCab+ER's normalization for each category. The method correctly normalized tokens of alternative representations and sound change variants with  $30 - 40\%$  recall. However, it completely failed to normalize character type variants not covered by the derivation rules and more irregular typographical errors.
5.4 Analysis of the Segmentation Results
We performed error analysis of the segmentation results for the two methods. Table 8 shows a matrix of the number of correct or incorrect segmentations for the methods for gold words. There existed 32 tokens that only MeCab correctly segmented (T-F), 200 tokens that only MeCab+ER correctly segmented (F-T), and 413 tokens that both methods
Table 7: Recall for each category (normalization)
MeCab\MeCab+ER	T	F
T	11955	32
F	200	413
Table 8: Number of correct (T) or incorrect (F) segmentation for two methods
incorrectly segmented (F-F).
In Table 9, we show the actual segmentation/normalization examples using the methods for the three cases; the first, second, and third blocks show examples of T-F, F-T, and F-F cases, respectively. First, out of 32 T-F cases, MeCab+ER incorrectly segmented tokens as onomatopoeia in 18 cases. For example, (a) and (b) correspond to new nodes added by the rules for the XQYri-form and XYXY-form onomatopoeia, respectively, even though (a) is a verb phrase and (b) is a repetition of interjections.