SlowGuess/NAACL_2022_OCR huggingface dataset:
To estimate human evaluation, we employ 17 expert annotators as per the conditions specified in §3.2 to evaluate four criteria as follows: (i) Grammar Correction: Is it a valid sentence for Korean grammar?; (ii) Factuality: Does it contain the given content morphemes as much as possible? (iii) Commonsense: Is it following commonsense knowledge?; (iv) Fluency: Is it a natural sentence for a mother tongue speaker? The human annotators score each measure with 2 points for excellent, 1 point for regular, and 0 points for insufficient. Moreover, we estimate human annotator performance by considering their reference sentences in the test set. We develop a system of prediction by comparing each annotator's references to calculate inter-annotator agreement. We measure the inter-annotator agreement of 3 evaluators with Fleiss' Kapa (Fleiss, 1971). Overall Fleiss' Kapa coefficient correlations for each human evaluation are Commonsense 0.426, Factuality 0.478, Fluency 0.401, and Grammar Correction 0.344; therefore have moderate reliability among evaluators<sup>10</sup>.
The baselines include GPT2 (Radford et al., 2019) using only the decoder structure of Transformer (Vaswani et al., 2017) and BART (Lewis et al., 2020) using the encoder-decoder. Among
10Each model-generated sentence is evaluated by randomly selected 3 among 17 human annotators.
The construction cost for one sentence is  $0.13$ , and the working period is three weeks.
The inspection cost for one sentence is  $0.04$ , and the working period is two weeks.
Model	Size	BLEU 3	BLEU 4	ROUGE-2	ROUGE-L	METEOR	mBERTScore	KoBERTScore	Coverage
KoGPT2 (Radford et al., 2019)	125M	29.24	18.91	43.36	60.41	39.89	84.08	90.92	79.43
KoBART (Lewis et al., 2020)	124M	39.54	29.16	53.60	68.55	51.17	87.41	92.59	93.65
mBART (Liu et al., 2020)	610M	41.83	31.63	54.21	68.36	52.08	87.25	92.26	91.39
mBART-50 (Tang et al., 2020)	610M	40.51	30.20	53.50	68.18	50.90	87.31	92.26	91.71
mT5-small (Xue et al., 2021)	300M	34.18	23.29	49.48	66.46	46.10	87.39	92.28	92.02
mT5-base (Xue et al., 2021)	580M	40.87	30.22	54.87	70.21	51.76	88.15	92.77	94.83
mT5-large (Xue et al., 2021)	1280M	46.33	35.90	58.91	72.78	56.52	88.54	92.92	95.07
Human Performance	49.12	41.64	61.02	73.29	58.60	91.13	95.26	98.30
Table 2: Experimental results of various baselines on the Korean CommonGen test set.
Figure 3: Correlation of automatic evaluation metrics among Korean and multilingual language models. The box's color is deep with reddish, and the score is the more extensive the white boldface, the higher correlation.
the models with the mentioned structure, we use KoGPT $^{11}$  and KoBART $^{12}$  pretrained with Korean corpora. We also conduct experiments on mBART (Liu et al., 2020), mBART-50 (Tang et al., 2020), and mT5 (Xue et al., 2021) based on encoder-decoder as multilingual language models.
We conduct a quantitative analysis of our dataset as shown in Table 2. First, encoder-decoder models exhibit higher performance than the decoder-only model. The decoder-only model is limited in reconstructing input content morphemes into acceptable quality sentences based on generative commonsense reasoning only with uni-directional prompt engineering (Liu et al., 2021a). Moreover, the encoder-decoder models with an encoder can formulate sentences based on bi-directive embedding information for given content morphemes.
Second, KoBART has a model parameter of 124M which is smaller than that of the mBART and mBART-50 models (whose model parameters
are 610M) but exhibits partially competent performance. This result shows that if the models are of identical structure, the multilingual model with a high proportion of machine-translated data in the pre-training process may encounter difficulties in generating high-quality sentences based on generative commonsense reasoning.
Third, mBART, and mBART-50 show similar performance. In the case of mBART-50, low-resource languages have the effect of improving, but high-resource languages present partial degradation because of the curse of dimension. Korean is a medium-resource language, but its performance decreases like other high-resource languages.
Forth, mT5-large model has the most model parameters of 1.3B, and the pre-training method using sequence-to-sequence task form with prompt engineering appears to impact it positively. Additionally, the mT5-large shows the most comparable execution to human performance and higher coverage and semantic scores than BARTs. However, most of small size baselines still have tribulation generating sentences containing all of the given content morphemes, and overall performance is lower than humans.
$^{11}$ https://github.com/SKT-AI/KoGPT2
$^{12}$ https://github.com/SKT-AI/KoBART
We attempt to demonstrate the validity of concept extraction and data sources. Thus, we perform ablation studies with respect to the concept set and data source configuration method.
6.1 Other Than The Content Morphemes
The first ablation study is conducted by tokenizing the set of concepts into noun and verb as suggested in CommonGen and free morphemes that can be used alone depending on the presence or absence of independence. According to the configuration method of the concept set, Figure 3 shows the correlation between automatic evaluation metrics and Table 3 exhibits the performance of the baselines.
The p-value for the correlation of all evaluation metrics is less than 0.05 (statistical significance). As depicted in Figure 3, content morpheme concepts have the highest overall correlation with other evaluation metrics. Table 3 also shows that this approach has the highest performance in most evaluation indicators. Through these results, we find that constructing concepts based on content morpheme has minimized the loss of information required for complete Korean sentences.
When it comes to free morpheme concepts, the concept set does not include the vocabulary of verbs and adjectives. As described in Figure 3, difficulty in constituting sentences increases dramatically, and correlations with the evaluation metrics are lowered by omitting information on verbs and adjectives, which are core components of a sentence. Consequently, free morpheme concepts overlook most of the valuable information, making it challenging to infer essential semantic components to compose sentences and exhibit inferior performance, as shown in Table 3.
Next, in the case of noun and verb concepts, Figure 3 shows the lowest overall correlation with other evaluation metrics. Table 3 also presents the performance gap between evaluation metrics which is considerable on account of omitting information on adjectives and adverbs according to the combination of concepts. In addition, it is tough to reproduce the relationship between the word root and ending according to the conjugation of Korean in forming sentences. Therefore, all baselines have difficulty with sentence composition considering inflectional units and indicate the lowest coverage on average.
Among the configuration methods of the concept
Free Morph	BLEU3/4	ROUGE2/L	METEOR	m/koBERTScore	Coverage
KoGPT2	21.22/12.39	35.60/55.15	32.48	81.02/89.92	75.74
KoBART	24.02/14.78	39.65/57.91	39.01	82.71/90.98	84.27
mBART-50	23.31/13.78	40.49/58.63	39.36	83.79/90.73	84.63
mT5-large	29.54/19.39	44.76/62.72	42.59	84.07/91.02	85.79
Noun & Verb	BLEU3/4	ROUGE2/L	METEOR	m/koBERTScore	Coverage
KoGPT2	27.27/17.21	37.60/53.43	30.07	81.35/86.02	67.48
KoBART	42.27/31.53	52.83/66.32	42.83	85.10/88.22	80.22
mBART-50	44.70/33.82	54.18/67.16	43.73	85.39/88.11	80.08
mT5-large	52.37/41.15	59.73/71.67	48.27	86.61/88.50	83.35
Content Morph	BLEU3/4	ROUGE2/L	METEOR	m/koBERTScore	Coverage
KoGPT2	29.24/18.91	43.36/60.41	39.89	84.08/90.92	79.43
KoBART	39.54/29.16	53.60/68.55	51.17	87.41/92.59	93.65
mBART-50	40.51/30.20	53.50/68.18	50.90	87.31/92.26	91.71
mT5-large	46.33/35.90	58.91/72.78	56.52	88.54/92.92	95.07
Table 3: Ablation study for concept tokenization method. Baselines train with Free Morph: Free morpheme concepts, Noun & Verb: Noun and verb concepts, and Content Morph: Content morpheme concepts.
IC	BLEU3/4	ROUGE2/L	METEOR	m/koBERTScore	Coverage
KoGPT2	20.66/12.63	34.87/53.46	31.76	79.49/89.09	68.85
KoBART	33.19/23.62	48.31/65.15	44.84	86.32/92.19	90.38
mBART-50	33.61/24.21	47.36/64.41	43.83	86.13/92.13	87.20
mT5-large	40.19/29.75	53.77/69.66	50.62	88.00/92.74	94.44
DS	BLEU3/4	ROUGE2/L	METEOR	m/koBERTScore	Coverage
KoGPT2	17.13/9.22	30.45/49.03	27.70	80.26/88.70	71.84
KoBART	23.62/14.44	39.48/58.28	37.23	85.14/91.08	91.32
mBART-50	23.78/14.71	38.48/57.80	35.63	84.86/90.93	89.78
mT5-large	33.96/23.12	48.11/65.95	45.72	86.91/92.14	94.67
IC&DS	BLEU3/4	ROUGE2/L	METEOR	m/koBERTScore	Coverage
KoGPT2	29.24/18.91	43.36/60.41	39.89	84.08/90.92	79.43
KoBART	39.54/29.16	53.60/68.55	51.17	87.41/92.59	93.65
mBART-50	40.51/30.20	53.50/68.18	50.90	87.31/92.26	91.71
mT5-large	46.33/35.90	58.91/72.78	56.52	88.54/92.92	95.07
Table 4: Ablation study for the data source. Baselines train with IC: Image caption data, DS: Dialogue summary data, and IC & DS: Image caption and dialogue summary data.
set, content morpheme concepts have the highest overall correlation with other evaluation metrics. This method also shows that the loss of information required for complete Korean sentences is minimized and that sentence composition considering the Korean linguistic features is also possible.
6.2 Image Caption or Dialogue Summary
Table 4 shows the results of baselines by dividing data source into image caption and Korean dialogue summary. The evaluation data includes both image captions and dialogue summarized sentences.
The image caption is straightforward but still contains awkward expressions owing to machine-translated results. Moreover, the Korean dialogue summary is relatively elaborated and incorporates a natural expression of mother-tongues to a wide range of everyday conversation topics.
As described in Table 4, the entire data combining the attributes of the two sources demonstrate the highest performance. These results show that our dataset requires more diverse sentence com
position and commonsense reasoning processes than solely using image caption sources offered by CommonGen (Lin et al., 2020). Additionally, the dialogue summary includes various sociocultural commonsense knowledge of mother-tongues. It is challenging to feed all contexts with only a small amount of data. Thus, models trained only with dialogue summary show lower performance than those trained only with image caption. Considering experimental results of Table 4, we organize the training such that the model first learns image captions and then highly complicated dialogue summarized sentences.
We further conduct a human evaluation for model-generated sentences, as shown in Table 5. The experimental results show that the mT5 model achieves the highest score and tendency of the score distribution parallel to the automatic evaluations. This point indicates that the estimation results with automatic metrics obtained from earlier experiments are comparable to human agreements. Closely examining the four evaluation criteria of human scores reveals that the baselines show the lowest score on average in fluency compared to other evaluation criteria and the highest score on average in factuality. This means that the models are relatively well-trained to use content morphemes in complete sentences, but their ability to generate spontaneous sentences is insufficient.
The correlation score between each evaluation criteria of human evaluation and automatic evaluation metrics is described in Figure 4. The heatmap shows that recall-based metrics have high correlations with human scores. This result corresponds to the demonstrations in (Lin, 2004) and (Banerjee and Lavie, 2005). In the case of coverage, a different trend showing low correlations with other human scores except for factuality is observed. It can be predicted that the criteria we have suggested and the basis for human judgment are somewhat consistent. The lower correlation in factuality with automatic evaluation metrics shows that containing all the given concepts does not necessarily constitute a well-crafted sentence.
Moreover, human-annotated scores have statistical significance in correlation with automatic evaluation metrics (p.value  $< 0.05$ ), and show considerably similar values to Figure 3 in BERT-based semantic scores. However, there are weak posi
Human evaluation	GC	CS	FC	FL	TT
KoGPT2	0.85	0.74	1.27	0.64	3.50
KoBART	1.32	1.25	1.74	1.15	5.49
mBART-50	1.31	1.24	1.71	1.15	5.40
mT5-large	1.44	1.36	1.80	1.28	5.89
Table 5: Human evaluation for model-generated outputs including GC: Grammar Correction, CS: Commonsense, FC: Factuality, FL: Fluency, and TT: Total
Figure 4: Correlation heatmap between automatic evaluation metrics (X axis) and human scores (Y axis). We experiment dividing Y axis into human-annotated GC: Grammar Correction, CS: Commonsense, FC: Factuality, FL: Fluency, and total score.
tive correlations with other automatic evaluation metrics. These results show that it is difficult to evaluate the model's generative commonsense reasoning in the traditional estimation approach, and advanced research is needed to improve it.
In this paper, we propose a Korean CommonGen dataset including Korean sociocultural commonsense knowledge and morpheme-based linguistic features. The dataset heeds the semi-automatic dataset construction method based on automatic construction and crowd-sourcing annotation with quality assessment. We perform a comparative analysis and an ablation study to demonstrate the validity of the dataset and evaluation metrics for generative commonsense reasoning. Moreover, we conduct a Korean and multilingual language models' standard performance experiment to investigate the dataset's problems and competencies. In future work, the dataset will enhance evaluation metrics regarding syntax and diversity of sentences to improve the interpretation of the model-generated results. We believe that our proposed dataset will serve as a fundamental resource to Korean NLG and commonsense reasoning research.
This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program(IITP-2018-0-01405) supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation). This work was supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques). This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(NRF-2021R1A6A1A03045425).
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799.
Noam Chomsky. 1965. Aspects of the theory of syntax.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. *Psychological bulletin*, 76(5):378.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto,
Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, Joao Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96-120, Online. Association for Computational Linguistics.
Younggyun Hahm, Youngbin Noh, Ji Yoon Han, Tae Hwan Oh, Hyonsu Choe, Hansaem Kim, and Key-Sun Choi. 2020. Crowdsourcing in the development of a multilingual framenet: A case study of korean framenet. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 236-244.
Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020. Kornli and korsts: New benchmark datasets for korean natural language understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 422-430.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401.
Suvarna G Kanakaraddi and Suvarna S Nandyal. 2018. Survey on parts of speech tagger techniques. In 2018 International Conference on Current Trends towards Converging Technologies (ICCTCT), pages 1-6. IEEE.
Daniel Keysers, Nathanael Scharli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations.
Myung Hee Kim and Nathalie Colineau. 2020. An enhanced mapping scheme of the universal part-of-speech for korean. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3826-3833.
T KUDO. 2005. Mecab: Yet another part-of-speech and morphological analyzer. http://mecab.sourceforge.net/.
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In 35th International Conference on Machine Learning, ICML 2018, pages 4487-4499. International Machine Learning Society (IMLS).
Young-Jun Lee, Chae-Gyun Lim, and Ho-Jin Choi. 2020. Korean-specific emotion annotation procedure using n-gram-based distant supervision and koreanspecific-feature-based distant supervision. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1603-1610.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, and Nan Duan. 2021. KFCNet: Knowledge filtering and contrastive learning for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2918-2928, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1823-1840.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211-226.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
Ye Liu, Yao Wan, Lifang He, Hao Peng, and S Yu Philip. 2021b. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6418-6425.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Andrew Matteson, Chanhee Lee, Youngbum Kim, and Heui-Seok Lim. 2018. Rich character-level information for korean morphological analysis and part-of-speech tagging. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2482-2492.
Jihyung Moon, Won Ik Cho, and Junbum Lee. 2020. BEEP! Korean corpus of online news comments for toxic speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 25-31, Online. Association for Computational Linguistics.
Sangwhan Moon and Naoaki Okazaki. 2020. Jamo pair encoding: Subcharacter representation-based extreme korean vocabulary compression for efficient subword tokenization. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3490-3497.
David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Linguisticae Investigationes, 30(1):3-26.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Eunjeong L Park and Sungzoon Cho. 2014. Konlpy: Korean natural language processing in python. In Annual Conference on Human and Language Technology, pages 133-136. Human and Language Technology.
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Junseong Kim, Youngsook Song, Taehwan Oh, et al. 2021. Klue: Korean language understanding evaluation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.
Xcopa: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2376.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4932-4942.