SlowGuess/NAACL_2022_OCR huggingface dataset:
Jaehyung Seo $^{1*}$ , Seounghoon Lee $^{2*†}$ , Chanjun Park $^{1,3}$ , Yoonna Jang $^{1}$ , Hyeonseok Moon $^{1}$ , Sugyeong Eo $^{1}$ , Seonmin Koo $^{1}$ , Heuiseok Lim $^{1,2‡}$
$^{1}$ Department of Computer Science and Engineering, Korea University  $^{2}$ Human Inspired Artificial Intelligence Research (HIAI), Korea University  $^{3}$ Upstage
Recent natural language understanding (NLU) research on the Korean language has been vigorously maturing with the advancements of pretrained language models and datasets. However, Korean pretrained language models still struggle to generate a short sentence with a given condition based on compositionality and commonsense reasoning (i.e., generative commonsense reasoning). The two major challenges are inadequate data resources to develop generative commonsense reasoning regarding Korean linguistic features and to evaluate language models which are necessary for natural language generation (NLG). To solve these problems, we propose a text-generation dataset for Korean generative commonsense reasoning and language model evaluation. In this work, a semi-automatic dataset construction approach filters out contents inexplicable to commonsense, ascertains quality, and reduces the cost of building the dataset. We also present an in-depth analysis of the generation results of language models with various evaluation metrics along with human-annotated scores. The whole dataset is publicly available at (https://aihub.or.kr/opendata/korea-university).
With the advent of Transformer (Vaswani et al., 2017) model, the importance of language resources and language modeling in natural language processing (NLP) has been heightened. Indeed, various studies on Korean language resources, such as Korean morpheme analysis (Matteson et al., 2018; Kim and Colineau, 2020; Moon and Okazaki, 2020), natural language understanding (NU) tasks including KorNLI and KorSTS (Ham et al., 2020), KMRE (Lee et al., 2020), and KLUE (Park et al.,
Figure 1: Example results of the KoGPT2, KoBART, mBART-50, mT5, and humans on Korean generative commonsense reasoning task. The results are sentences that combine the given content morphemes (in red bold-face).
2021) are being conducted alongside the studies on off-the-shelf pretrained language models in Korean (Devlin et al., 2019; Radford et al., 2019; Lewis et al., 2020).
Despite the prosperity of Korean NLP research, two critical problems remain: (i) an absence of a research base for natural language generation (NLG) and (ii) a deficient ability for models to generate commonsense knowledge. In other words, (i) there exists neither a dataset (Gehrmann et al., 2021) nor standards for evaluating (Celikyilmaz et al., 2020) the results generated by language models in Korean because most Korean language resources are focused on NLU tasks thereby making it difficult to accelerate the development of NLG research.
(ii) Korean language models encounter difficulties even in generating sentences using simple commonsense knowledge. Commonsense knowledge is a sociocultural knowledge shared by humans (Liu and Singh, 2004). It is not visible but it is melted in their words (Tandon et al., 2018). To make natural sentences using commonsense knowledge like those made by humans, comprehensive abilities
* Equally contributed
† Present affiliation: Institute for Infocomm Research, A*STAR
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2233 - 2249 July 10-15, 2022 ©2022 Association for Computational Linguistics
of generative commonsense reasoning (Lin et al., 2020) are needed. This requires holistic understanding of commonsense reasoning (Lake and Baroni, 2018; Keysers et al., 2020) and sentence compositionality (Hahm et al., 2020).
Though people acquire commonsense knowledge and effortlessly use it in their daily lives, it is challenging for language models to imitate this ability. As shown in Figure 1, off-the-shelf Korean and multilingual language models seem to lack the competence for generative commonsense reasoning. Some model-generated sentences do not make sense (e.g., "개가 채트기 퉈을 기다고 갔다. A dog is passing over the jet.", "개가 기다고 갔다고 갔다. Sneeze is wiping the nose.", use inappropriate prepositions (e.g., "과일에어를에서 갔을 기다고 갔다. I cut a knife from a pineapple and eat it.", or misplace parts of speech (e.g., "류머신이 sounds on the dogs and let's go"). The treadmill is listening to music and running.).
To address these issues and inspired by CommonGen (Lin et al., 2020), we develop a Korean CommonGen dataset for generative commonsense reasoning. The dataset is composed of commonly used daily-life concepts and sentences made by combining those concepts. Our dataset differs from the CommonGen dataset as follows: (i) We collect Korean corpus and label it to cover Korean sociocultural commonsense knowledge. For example, the sentence of the corpus contains the unique Korean sociocultural terms "vere" (return to the farm)" and "fam" (dried persimmons)". (ii) Although we adopt the image caption data from CommonGen, we add the summary data of daily conversations into our dataset thereby obtaining diverse sentences. (iii) Because Korean language models use segmented morphemes as vocabulary (Lee et al., 2020; Kim and Colineau, 2020), we construct the concept set with content morphemes that have linguistic features and lexical meanings. (iv) We analyze the evaluation metrics including the human-annotated score to demonstrate the validity of the evaluation criteria.
We reduce the cost of data construction significantly through an automated method and inspect the quality and unethical issue of Korean Common-Gen by crowd-sourcing<sup>1</sup>. In addition, we conduct
an in-depth study of our proposed dataset with various ablation experiments on morpheme segmentation and training methods. Furthermore, the model-generated sentences are compared and analyzed by quantitative, qualitative, and human scores. We disclose the dataset used in this paper to contribute to the development of Korean NLG research.
Commonsense Knowledge Commonsense knowledge is knowledge about everyday life that all people possess, and it is arguably the most general and widely applicable knowledge (Liu and Singh, 2004). Compared to encyclopedic knowledge, which returns specific details about named entities on a modern search engine, commonsense knowledge includes elusiveness and context dependence (Tandon et al., 2018).
Compositionality Compositionality is an essential element that AI systems have to solve for given conditions. For example, the MS-COCO (Lake and Baroni, 2018) dataset is utilized for image caption task generating the description from an image data as an input data. The task demands the compositionality of the model as the model composes the natural description. Moreover, SCAN (Lake and Baroni, 2018) demonstrates mapping instructions to sequence an RNN model's ability to generate continuous behavior. However, these studies have shown that AI systems still struggle to generate complete results.
Commonsense Reasoning Commonsense reasoning is the ability to infer unrecognized commonsense knowledge or relations among given concepts. In a recent NLP research, various commonsense reasoning datasets have been disclosed. CommonsenseQA (Talmor et al., 2019) organizes the dataset with closed questions to commonsense based on the concept of ConceptNet and analyzes the skills of commonsense reasoning required for each question by categorizing them. Cosmos QA dataset (Huang et al., 2019) introduces a question answering dataset based on the fact that is not externally revealed in the context. CoS-E dataset (Rajani et al., 2019) attempts to strengthen the commonsense training of the model by adding a person's description of the commonsense QA. The XCOPA dataset (Ponti et al., 2020) mitigates the gaps in commonsense that could arise from linguistic and cultural differences, while building a dataset.
<sup>1</sup>We employ AI & Human Resources Platform CrowdWorks
Basic Statistics	Train	Validation	Test
# Content morpheme-sets	43,188	1,000	2,040
- Set size less than 3	5,089	115	334
- Set size 4	10,810	241	604
- Set size 5	13,397	332	577
- Set size 6	12,811	292	513
- Set size more than 7	1,081	20	12
# Unique content morphemes	40,874	2,000	3,272
- # Unseen single	-	332	748
- # Unseen pair	-	5,305	10,461
- # Unseen triple	-	9,728	17,648
# Additional morphemes	-	-	4,682
# Sentences	43,188	1,000	6,120
- Average length	26.06	24.74	23.54
- Caption-based rate	45.58	50	48.99
- Dialogue-based rate	54.42	50	51.01
Table 1: Statistics for Korean CommonGen dataset. We construct a test set to generate sentences by reasoning unseen content morphemes in training. # Additional morphemes is the number of unseen single morphemes counted through extra 4,080 human references. #Caption-based rate and #Dialogue-based rates mean the ratio of each data among the total dataset.
Generative Commonsense Reasoning Based on compositionality and commonsense reasoning, we concentrate on generative commonsense reasoning, the ability required to generate sentences that conform to commonsense knowledge for given conditions. In the case of English-based language models, various studies have improved the ability of generative commonsense reasoning based on the CommonGen (Lin et al., 2020) dataset. KGBART (Liu et al., 2021b) enhances the model allowing to capture the relationship between nodes in the graph, while including the ConceptNet knowledge graph in the attention calculation process for text input. RE-T5 (Wang et al., 2021) reinforces the input value by using a retriever to import sentences related to the concepts from external knowledge. KFCNet (Li et al., 2021) presents the state-of-the-art performance in CommonGen by removing low-quality sentences in external knowledge and applying contrastive learning, respectively.
However, there does not exist Korean dataset for generative commonsense reasoning and advanced research as well. Therefore, in this paper, we aim to create a new text-generation dataset inspired by CommonGen (Lin et al., 2020) and grant a direction for the future Korean NLG research.
3 Korean CommonGen
Korean CommonGen is a text-generation dataset for Korean commonsense reasoning and evalua
tion. As depicted in Figure 2, Korean CommonGen consists of concept sets typically used in daily life and sentences depicting those concepts. Language models are trained to generate a sentence by reasoning and combining the concepts based on human-generated sentences. The model-generated result should include all the given concepts, be grammatically correct and make sense.
Korean CommonGen includes 43,188 training examples and 2,040 testing examples as presented in Table 1. Corresponding to one concept set of content morphemes, the training example comprises one sentence, and the testing example contains three sentences. In the case of the training set,  $45.58\%$  of examples are English-Korean translated  $\mathrm{MS - COCO}^2$  (Lin et al., 2014) image caption data, and  $52.42\%$  are Korean dialogue summary data of AI-HUB<sup>3</sup>. To make the models learn the semantic role and relation of unseen content morphemes through commonsense reasoning, the test set has 748 single content morphemes unseen during training. Additionally, more than one pair of unseen content morphemes is included in each testing example. The test set has additional two reference sentences for every 2,040 content morpheme set annotated by crowd-sourcing. These additional references enable the consideration of the diverse possibilities in model-generated sentences (Chomsky, 1965). We allow additional references to include 4,682 newly annotated content morphemes that do not appear in the concept set. As a result, the model can obtain a higher score when the generated sentence has additional morpheme fitted on commonsense. The dataset is constructed through semi-automatic approach, and the details are described in §3.1 Automatic Dataset Construction and §3.2 Annotate and Refine by Crowd-sourcing.
3.1 Automatic Dataset Construction
We implement the automatic construction approach with a part-of-speech (POS) tagger, a named entity recognition (NER) tagger, a sentence level filtering, and Korean Hate Speech classifier. They extract the content morphemes, and screen out the sentences that include unethical expressions, violate commonsense or have unnatural sentence structures. With these automated modules, we reduce the cost for human annotation by $7,766.
$^{2}$ https://aihub.or.kr/keti_data_board/visual_intelligence
<sup>3</sup>https://aihub.or.kr/aidata/30714
Figure 2: Overview of Korean CommonGen and pipeline of semi-automatic dataset construction.
Data Sources In this study, we organize the dataset with the image caption and dialogue summary sentences describing daily life based on commonsense knowledge that does not correspond to the specific or professional domains.
For data construction, we utilize the Korean image caption dataset released by AI-HUB visual intelligence, which is translated from the original English MS-COCO dataset. The caption sentences in this dataset describe the scenes that occur in everyday life. It implies that universal commonsense knowledge implicitly lies in those sentences. As they are the combinations of objects, and relations exist in the corresponding (ground-truth) image, it is appropriate for models to learn how to compose sentences by considering all the given concepts.
The Korean dialogue summary dataset of AI-HUB is also adopted for data construction. These data contain conversations on everyday topics and include commonsense knowledge that has implicit Korean sociocultural content. We secure the diversity of sentences and commonsense knowledge by adding non-visual conversation contents not solely using image caption sentences based on visual information, such as conventional CommonGen.
We delete the sentences involving the unexpected foreign words and special symbol tokens in the phase of preprocessing. Both datasets consist of sentences that take the form of declarative statements. In this process, we unify the structure of sentences to be ended with 'da.' (i.e., an ending word in Korean declarative statement) for minimizing the variances in performance evaluation according to the decoding hyperparameters or strategies.
Content Morpheme Extraction POS tagging is the task that assigns the grammatical group tag
to the text based on the language's perspective and definition (Kanakaraddi and Nandyal, 2018). We utilize POS tagging to extract the content morphemes essential for making concept sets. We apply the morpheme segmentation of ko-mecab (KUDO, 2005) using the Korean morpheme analysis package KoNLPy (Park and Cho, 2014).
Korean is a highly inflected language with many inflectional morphemes and has multiple POS tag patterns. In addition, Korean is an agglutinative language. If the Korean language is tokenized with eojeol segmentation by white spaces, the number of vocabulary units increases exponentially, and the accuracy of correct tagging decreases. Korean embedded models also operate morpheme segmentation to avoid expensive computational costs caused by the exponential increase in new vocabularies with eojeol segmentation (Lee et al., 2020; Kim and Colineau, 2020). Therefore, we adopt the morpheme segmentation method to improve efficiency and suitability.
When creating the concept set, we employ the content morphemes. The content morphemes include the definite actions, states, or semantic information of the sentence. They involve the verb and adjective stems, and some adjectives are grammatically similar to the verbs in Korean. Therefore, it is possible to create a sentence with the content morphemes, considering both the semantic relations between them and their grammatical usage.
Based on the criteria of the content morpheme tagging, we classify the nouns (NNG, NNP, NNB, NNBC, NR, NP), determiners (MM), adverbs (MAG, MAJ), verbs (VV, VA, VX, VCP, VCN), radixes (XR), and interjections (IC)
as the content morpheme<sup>4</sup>. However, the proper noun (NNP), numeral (NR), pronoun (NP) are only understandable in certain situations and perspectives among the content morpheme. Thus, we delete the sentences including NNP, NR, and NP in the process of morpheme segmentation. The detailed differences according to the segmentation are described in §6.
Named Entity Filtering NER is a subtask of Information Extraction, which attempts to recognize the named entities such as a person, location, and quantity from the unstructured text (Nadeau and Sekine, 2007). We utilize the NER to remove the sentences including the non-commonsense knowledge from the data sources.
As mentioned above, the Korean dialogue summary dataset of AI-HUB carries the daily conversations, which means it contains commonsense knowledge in abundance. However, there is a possibility that the sentences consist of specific domain knowledge that is shared only by the particular group or time. Named entities, such as specific names of persons, organizations, and locations, are not commonsense entities because most people do not comprehend them. For instance, "운ho 是哪吒，观音菩萨” (Yoonho looks kind, but he is not talented for the entertainment.)", name of person '운ho(Yoonho)' is vague to be categorized as commonsense knowledge.
We vacate the sentences containing the non-commonsense knowledge with NER tagging in the phase of dataset construction. In the NER tagging, we adopt the neural network model in Pororo library and remove the 119,355 sentences containing the non-commonsense named entities.
Sentence Level Filtering Through NER and POS tagging, a substantial number of sentences with non-commonsense knowledge can be filtered out. However, there remain several sentences that contain awkward translations or do not properly reflect commonsense knowledge. To filter out these residues, we selectively extract sentences that follow common Korean sentence structure and contain rich commonsense knowledge at the sentence level.
These are proceeded by the comparison between sentences within the same morpheme. For the se
lection process, we apply inverted index to every  $s \in S$  that contains unique content morpheme  $x \in X$ . A set of inverted indexed sentences for each context morpheme  $x$  is denoted as  $s^x = \{s_i^x\}_{i=1}^{N_s}$  where the maximum size of  $N_s$  is set to 100.
Then, based on the contextual representation embedding of the language model, we estimate the similarity score between every two diverse sentences in  $s^x$  using the KoBERT fine-tuned with the KorSTS (Ham et al., 2020) dataset. More specifically, the similarity score  $\hat{y}_{i,j}$  between two sequences  $s_i^x, s_j^x$  is estimated as shown in Equation 1.
$$
\hat {y} _ {i, j} = \sigma \left(W \left(h _ {i j}\right) + b\right) \tag {1}
$$
$h_{ij}$  indicates the KoBERT encoded representation of the concatenated sequence  $s_i^x, s_j^x$ .  $\sigma$  denotes activation unit and  $W, b$  are trainable parameters of a linear pooling layer. To figure out descent sentences that fluently follow the common Korean sentence form and contain rich commonsense knowledge within the same content morpheme, we set up score $_i^x$  for each sentence  $s_i^x$ . score $_i^x$  is estimated by summing up all the similarity score  $\hat{y}_{i,j}$  between  $s_i^x$  and other sentences, as shown in Equation 2.
$$
\operatorname {s c o r e} _ {i} ^ {x} = \left(\sum_ {j = 1} ^ {N _ {s}} \hat {y} _ {i, j}\right) - \hat {y} _ {i, i} \tag {2}
$$
We evaluate  $score_i^x$  for each sentence  $s_i^x$  and sort all the sentences in a descending order. According to their scores, the top-2 sentences are selected for each unique content morpheme.
Ethical Consideration Machine-translated MS-COCO can deviate from the intended purpose or have aggressive terms because of cultural differences. Also, a dialogue summary of daily conversations can include socially inappropriate or discriminatory content. Thus, we filter out these expressions so that models cannot unintentionally return inappropriate responses to some triggers.
To detect unethical expressions, we use the Korean Hate Speech Detection (Moon et al., 2020) dataset. As a model, KoELECTRA $^{7}$  is pretrained with the ELECTRA (Clark et al., 2019) structure and Korean corpora. KoELECTRA is trained to classify whether input sequences contain gender biased or aggressive representation. The classifier's predicted 1,083 results are potentially unethical
$^{6}$ https://github.com/SKTBrain/KoBERT
4https://www.korean.go.kr/front/onlineQna/onlineQnaView.do?mn_id=216& qna_seq=209597
5https://github.com/kakaobrain/pororo 7https://github.com/monologg/KoELECTRA
risk statements, including either gender bias or offensive. Among them, 172 sentences contain both gender biased and offensive problems.
However, we note that the classifier is not complete (Roller et al., 2021); thus, we reschedule those results with a secondary inspection through two expert annotators. The classifier tends to be overly sensitive to words that refer to a particular gender (e.g., male, female) or words that can be used aggressively (e.g., cut, bound, etc.). In addition, the published source data: MSCOCO and AI-HUB Dialogue Summary, have completed pre-validation and data curation. We find the problem that most of the predicted sentences are false-positive. Therefore, we conduct a second round evaluation to consider whether the sentences conform to the definition of commonsense knowledge and do not deviate from social norms. In the second round, two human annotators majoring in linguistics and computer science remove 124 sentences with criminal, drug, and excessive biased among predicted 1,083 results.
3.2 Annotate and Refine by Crowd-sourcing
We generate additional reference sentences using the extracted content morphemes via crowdsourcing. Employing the expert human annotators, we also check the quality of the references and implement secondary inspection on the dataset to filter out sentences that contain ethical issues.
Reference Generation To evaluate the diversity of model-generated sentences, we produce additional 4,080 references based on the 2,040 concept set designated as test data. We employ 22 human annotators with bachelor's degrees through crowdsourcing<sup>8</sup>. The working guidelines are as follows: First, we ensure the additional references are not similar or do not merely change the position of the subject/verb/object. Second, the given content morphemes are preserved, and the annotators can append extra modifiers conforming to commonsense. Third, references do not incorporate overly specific named entities or numerical expressions.
Quality Control 17 expert annotators holding bachelor's degrees in Korean language or linguistic secure the quality of the automatic constructed data via secondary inspection and assessment<sup>9</sup>. Since the references determine the model's performance,
human correction is performed on the references to maintain high quality leading to 303 inappropriate sentences being removed.
4 Experiments Settings
The evaluation metrics consist of n-gram overlapping, semantic similarity, content morpheme coverage, and human score. We use automatic evaluation metrics based on n-gram overlapping such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005). These metrics compute the token level similarity between model-generated candidate and reference sentences. Next, we utilize BERTScore (Zhang et al., 2019) as semantic similarity and evaluate the outputs using mBERT and KoBERT to identify differences between multilingual and monolingual models. We also indicate the concept coverage which is the average percentage of given concepts that exist in model-generated sentences.