SlowGuess/NAACL_2022_OCR huggingface dataset:
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149-4158.
Niket Tandon, Aparna S Varde, and Gerard de Melo. 2018. Commonsense knowledge in machine intelligence. ACM SIGMOD Record, 46(4):49-52.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 30:5998-6008.
Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, and Michael Zeng. 2021. Retrieval enhanced model for commonsense generation. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 3056-3062.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
A Qualitative Evaluation
We conduct a qualitative analysis on the machine-generated sentences as illustrated in Figure 5. Qualitative evaluation is executed by composing the content morphemes of the test set to generate sentences based on commonsense. In the analysis, we separate the model-generated results embracing the Korean commonsense knowledge into the good case and the bad case.
Good Case In the case that requires low complicated compositionality and simple commonsense reasoning, the models tend to generate identical sentences. When KoGPT2 generates the sentences, the result shows prominent differences with the outputs of other models, and it has an inconsistent grammatical correction. On the contrary, KoBART, mBART-50, and mT5 yield brief sentences that adopt relatively stable grammar rules and postpositional expressions.
Bad Case In the case that requires complicated compositionality and struggling commonsense reasoning, the gap of qualities between the model-generated sentences is more significant than the opposite case. KoGPT2 generates incoherent sentences that exploit the given content morphemes repetitively or contain many errors in commonsense reasoning. KoBART induces more natural Korean sentences than the other models, but the results are still harsh to understand owing to their ambiguous expressions. From the multilingual language models, mBART-50 yields unnatural sentences like machine-translated sentences. In addition, the models show semantically incorrect results as if the models misunderstand the meaning of some words like '개내(discharge)'. The mT5 employs more manageable words and brief sentences, including all given content morphemes as much as possible.
Although the ability of the Korean monolingual language model's compositionality still seems insufficient, the results show that the models partially understand the content morpheme with a cultural specialty such as the conscription system. In contrast, the multi-language model shows substantial compositionality and abnormal usage of the content morpheme with cultural specialty as its limitation.
Joye#i# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
Joye#i# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
Joye#i # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
Joye#i # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
Joye# i# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
Joye#i# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
Joye# I# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
Joye# I# # # # # # # # # # # # # # # # # # # # # # # # # # # #
Joye# I# # # # # # # # # # # # # # # # #
Joye# I# # # # # # # # # # #
Joye# I# # # # # # # #
Joye# I# # # # # #
Joye# I# # # # #
Joye# I# # # #
Joye# I# # #
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye#I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joy e#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joyc
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joyc
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# l#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joyce# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Joye# I#
Figure 5: A case study comparing commonsense generation in the test set. We categorize the good and bad cases based on the model-generated sentences' quantitative evaluation. The two cases include Korean commonsense knowledge, 'Yut' is related to traditional Korean games, and 'military' and 'discharge' are related to the conscription, which expresses Korean sociocultural characteristics.
B Further Experiments
We conduct experiments to verify the training method of commonsense reasoning, assessment on the data dependency, and expandability within the commonsense domain.
B.1 High-Level Commonsense Reasoning
Table 6 shows the evaluation results on the models trained by randomly deleting one concept to evaluate the high-level commonsense reasoning. The seeds for the random deletion are set to  $\{42,52,62,72,82\}$ . Because of the random deletion, the training data get more challenging as if the model composed the sentence reasoning with at least one content morphemes not given. The performance of the models decreases compared with the model trained with the entire concept set. Deleted content morphemes affect the disappearance of the commonsense knowledge rather than the enhancement of the commonsense reasoning during the training phase. In the case of mBARTs, the extent of the performance decline is the most substantial, and mBARTs show inferior performance than Ko
BART on average. In addition, compared with the other models, mT5-base is sensitive to a given situation of the source input. This result means that a BART model pretrained with monolingual corpora is significantly robust, regardless of its small model size. The models trained with the identical architecture and corpora show a considerably different extent of optimization according to the model size in text generation.
B.2 Reformulated CommonGen Test set
In the following process, we introduce a new evaluation dataset that utilizes the raw data sources that are of little relevance to the training dataset.
First, based on the knowledge graph, we extract the concepts at the one-hop links ConceptNet (Liu and Singh, 2004). The top 20 ranked concepts with high weights are selected among the 25 according to the categories suggested in CommonGen. Second, the concepts are broadened based on the ConceptNet knowledge graph tagged with the grammatical role and translated as vocabulary units through Naver dictionary[13] crawling. Third, we employ two professional annotators who possess high comprehension of the task and are proficient in Korean as a native language[14]. The annotators create a new Korean evaluation dataset combining the broadened concepts based on the CommonGen test set and ConceptNet knowledge graph. Fourth, we construct the dataset with the created sentences using the reverse extraction method used in §3.1. Therefore, the new evaluation dataset consists of 1,083 sentences embracing Korean commonsense knowledge with flawless syntax. Furthermore, utilizing the concepts of the knowledge graph ConceptNet representative in the commonsense domain, we evaluate the dataset's dependency on the raw data source of the training dataset.
The performance of the models evaluate with the reformulated test set, in which two annotators create the sentences using the ConceptNet and CommonGen test set, are shown in Table 7. We train the models involving the same data and hyperparameters used in the experiment of Table 2. The models show the maintained performance with the new dataset constructed by a human using the other data source within the commonsense domain. Therefore, this result demonstrates that the models are not substantially dependent on the specific
13https://dict.naver.com/
14The construction cost for one sentence is 0.3$, and the working period is two weeks.
Model	Seed	BLEU 3	BLEU 4	ROUGE-2	ROUGE-L	METEOR	mBERTScore	KoBERTScore	Coverage
KoGPT2 (Radford et al., 2019)	42	24.23	15.24	39.73	56.56	38.09	82.29	89.09	78.41
KoGPT2	52	22.39	13.75	37.54	54.44	35.87	81.49	88.80	75.84
KoGPT2	62	24.39	15.19	40.02	56.70	38.77	83.10	90.16	79.85
KoGPT2	72	23.53	14.74	38.82	55.44	37.74	82.40	89.31	79.00
KoGPT2	82	26.14	16.67	40.73	57.78	38.48	82.53	89.71	76.93
KoGPT2 Mean (± Stdev)	24.14(±1.37)	15.12(±1.05)	39.37(±1.23)	56.18(±1.28)	37.79(±1.14)	82.36(±0.58)	89.41(±0.53)	78.01(±1.61)
KoBART (Lewis et al., 2020)	42	30.11	19.53	47.97	64.29	47.63	85.22	91.78	92.95
KoBART	52	30.31	20.05	47.55	63.76	47.42	85.37	91.65	92.67
KoBART	62	30.34	20.10	47.71	63.61	47.81	87.23	91.73	92.58
KoBART	72	30.32	19.82	47.70	63.92	47.69	85.20	91.69	92.68
KoBART	82	30.63	20.19	47.97	64.19	47.75	85.16	91.70	92.86
KoBART Mean (± Stdev)	30.34(±0.19)	19.94(±0.27)	47.78(±0.18)	63.95(±0.29)	47.66(±0.15)	85.24(±0.08)	91.71(±0.05)	92.75(±0.15)
mBART (Liu et al., 2020)	42	28.61	17.95	45.53	62.49	45.42	84.64	91.11	90.47
mBART	52	28.97	18.22	45.97	62.87	45.29	84.73	91.18	90.23
mBART	62	29.05	18.48	46.07	62.83	46.21	84.90	91.32	90.95
mBART	72	28.69	18.28	45.97	62.70	45.89	84.53	91.35	90.75
mBART	82	29.24	18.56	46.06	62.77	45.72	84.65	91.24	90.51
mBART Mean (± Stdev)	28.91(±0.26)	18.30(±0.24)	45.92(±0.22)	62.73(±0.15)	45.71(±0.37)	84.69(±0.14)	91.24(±0.10)	90.58(±0.28)
mBART-50 (Tang et al., 2020)	42	29.16	18.58	45.84	62.61	45.80	84.90	91.25	90.72
mBART-50	52	29.20	18.64	46.02	62.91	46.02	84.78	91.28	90.59
mBART-50	62	28.12	17.76	45.52	62.59	45.24	84.73	91.29	90.76
mBART-50	72	29.60	19.05	46.22	62.82	45.90	84.85	91.21	90.64
mBART-50	82	30.17	19.64	46.37	62.78	45.62	84.71	91.21	90.21
mBART-50 Mean (± Stdev)	29.25(±0.75)	18.73(±0.69)	45.99(±0.33)	62.74(±0.14)	45.72(±0.30)	84.79(±0.08)	91.25(±0.04)	90.58(±0.22)
mT5-small (Xue et al., 2021)	42	31.58	21.01	45.80	62.87	43.20	85.58	91.88	87.93
mT5-small	52	31.33	20.81	45.85	62.87	43.26	85.69	91.98	88.44
mT5-small	62	30.71	20.34	45.42	62.80	42.93	85.59	91.96	88.42
mT5-small	72	31.01	20.66	45.63	62.82	43.04	85.62	91.91	87.93
mT5-small	82	31.32	20.78	45.75	63.11	43.11	85.70	91.94	88.53
mT5-small Mean (± Stdev)	31.19(±0.34)	20.72(±0.25)	45.69(±0.17)	62.89(±0.12)	43.11(±0.13)	85.64(±0.06)	91.93(±0.04)	88.25(±0.30)
mT5-base (Xue et al., 2021)	42	30.86	20.31	47.75	64.51	46.42	86.29	91.98	93.71
mT5-base	52	33.26	23.46	48.90	64.93	47.39	85.99	91.82	93.08
mT5-base	62	30.42	20.70	47.16	63.95	46.10	85.15	91.68	93.06
mT5-base	72	31.29	21.56	47.50	64.05	46.30	85.36	91.81	93.10
mT5-base	82	29.32	19.76	45.67	62.45	44.92	85.10	91.54	93.87
mT5-base Mean (± Stdev)	31.03(±1.45)	21.16(±1.44)	47.40(±1.17)	63.92(±0.95)	46.23(±0.88)	85.58(±0.53)	91.77(±0.17)	93.36(±0.39)
mT5-large (Xue et al., 2021)	42	36.94	25.73	53.28	68.97	52.08	86.89	92.19	94.80
mT5-large	52	36.67	25.80	52.89	68.81	52.89	86.86	92.17	94.90
mT5-large	62	37.07	25.85	53.26	68.86	52.40	86.96	92.22	94.70
mT5-large	72	36.86	25.67	53.05	68.74	52.31	86.79	92.24	94.68
mT5-large	82	37.48	26.45	53.51	68.96	52.54	86.85	92.16	94.64
mT5-large Mean (± Stdev)	37.00(±0.30)	25.90(±0.31)	53.20(±0.24)	68.87(±0.10)	52.44(±0.30)	86.87(±0.06)	92.20(±0.03)	94.74(±0.11)
Table 6: Performance of generative language models in high-level commonsense reasoning test. Mean refers to an average value from sampled score using designated seeds, and Stdev is a standard deviation from sampled score using designated seeds.
data source. Moreover, KoBART is robust on the domain transfer, and the performances of mBARTs are decreased to a great extent in the case of experiment setting transition, revealing similar consequences with the experiment in Table 6. On the contrary, the performance of KoBART and mT5 models is enhanced indicating the expandability of the downstream task within the commonsense domain utilizing our dataset.
B.3 Human Evaluation
We perform human evaluation on the sentences generated via the model using the new test dataset irrelevant to the training dataset, as exhibited in Table 8. Compared to the results of the human evaluation using the test dataset relevant to the training dataset depicted in Table 5, each model has proximate results in the expected distribution and tendency, in which mT5-large achieves the best score. Despite the results in Table 8 evaluated with
the low relevance in test dataset, models achieve an only marginal lower score than Table 5. This implies that the models not only learn with the semantic and syntactic role of the content morpheme in the concept set but also their ability to acquire unseen commonsense. Moreover, factuality shows the slightest gap between Table 8 and Table 5. This result indicates that factuality is more independent of the training dataset than the other evaluation metrics.
C.1 Training
Generative language models are trained to generate a reference sentence containing  $m$  tokens  $s_{ref} = \{r_1, r_2, \dots, r_m\}$  by referring a content morphemeset containing  $l$  morphemes  $x_{set} = \{x_1, x_2, \dots, x_l\}$ . Training is implemented by optimizing the objective conditional probability of given tokens in an
Model	BLEU 3	BLEU 4	ROUGE-2	ROUGE-L	METEOR	mBERTScore	KoBERTScore	Coverage
KoGPT2	25.08	16.14	38.59	56.67	31.06	86.40	93.84	77.10
KoBART	36.12	26.55	50.63	66.64	44.81	91.23	95.61	95.14
mBART	34.58	25.21	48.14	65.15	42.28	90.78	95.29	94.55
mBART-50	33.67	24.21	47.14	63.92	40.56	90.66	95.44	93.09
mT5-small	28.80	19.08	44.54	62.26	37.34	90.45	95.34	94.18
mT5-base	34.50	24.10	50.25	66.30	43.55	91.67	96.05	96.45
mT5-large	41.86	31.72	55.51	70.06	49.55	92.27	96.23	96.80
Table 7: Performance of generative language models in translated and reformulated CommonGen test.
Human evaluation	GC	CS	FC	FL	TT
KoGPT2	0.74	0.57	1.31	0.47	3.08
KoBART	1.21	1.13	1.85	0.99	5.18
mBART-50	1.17	1.08	1.78	0.96	4.99
mT5-large	1.34	1.26	1.89	1.15	5.65
Table 8: Human evaluation for model-generated outputs in reformulated CommonGen test set including GC: Grammar Correction; CS: Commonsense; FC: Factually; FL: Fluency; TT: Total
auto-regressive generation. As formulated in Equation 3, the conditional probability is configured with model parameters for maximizing the likelihood. The pretrained model parameter is initialized to  $\theta$  by training with dataset  $D$ .
$$
\max  _ {\theta} \frac {1}{| D |} \sum_ {\left(s _ {r e f}, x _ {s e t}\right) \in D} \log \left[ \prod_ {t = 1} ^ {m} p _ {\theta} \left(r _ {t} \mid r _ {<   t}; x _ {s e t}\right) \right] \tag {3}
$$
We implement Huggingface $^{15}$  framework for language modeling in a single NVIDIA Quadro RTX 8000 GPU with 48GB and 18-core Intel Xeon Gold 6230 CPU. For KoGPT2 training, parameters are trained by batch size 4 with gradient accumulation, seed 42, learning rate  $5 \times 10^{-5}$ , warmup steps 400, AdamW optimizer (Loshchilov and Hutter, 2019)  $(\beta_{1} = 0.9, \beta_{2} = 0.999, \epsilon = 1e - 8)$ , and block size 128. In the case of encoder-decoder models, key hyper-parameters are also initialized by default settings suitable for the model architectures. We set the hyper-parameters in training stage as batch size 16 with gradient accumulation, seed 42, initial learning rate  $5 \times 10^{-5}$ , warmup steps 400, AdamW optimizer  $(\beta_{1} = 0.9, \beta_{2} = 0.999, \epsilon = 1e - 8)$ , max source length 64, max target length 256, and source prefix "summarize" (only for mT5-large).
C.3 Decoding Strategy
The decoding strategy is restricted to identical conditions in text generation. We set the beam size to 10, max sequence length to 30, min sequence length to 10, and no-repeat n-gram size to 3 for the imposition of a penalty on duplicate token generation. We re-rank the generated sentences sorted in descending order based on five candidate sentences that cover the number of the corresponding morphemes as the given content morpheme-set and select the highest rank as a concluding outcome.
D Error Analysis
This section investigates several errors found in our semi-automatic construction method.
The NER system has difficulty filtering out non-commonsense knowledge considering every person's perspectives. In this paper, we define commonsense knowledge as most people in the same society understand implicitly. However, it is challenging for commonsense knowledge to be defined with concrete scope and explanation. Therefore, some named entities can be viewed as commonsense knowledge to certain people. In our dataset construction method, the sentences holding the possibility of non-commonsense knowledge are safely deleted to maintain the rate of commonsense knowledge. Nevertheless, as the raw data sources pass through the automatic method, a proportion of sentences among the deleted sentences have the potential of reusability owing to the flexible definition of commonsense knowledge.
In the content morpheme extraction process, we solely use one POS tagger, ko-mecab, as a pilot study. Rather than discussing the performance of the various POS tagger, we mention the issues on the adoption process of the tagger. The process should include the sub-process of verification on the candidates of taggers or comparative study on the multiple datasets constructed with multiple taggers, respectively. The latter sub-process can cause
15 https://github.com/huggingface/ transformers
costly issues and require a lot of time to implement because new datasets for taggers should be reconstructed.
The usage of content morphemes can bring out uncertainty about whether the concept in the concept set should play a role as a verb or a noun. This feature embraces both pros and cons. The advantage is that the concept is granted unrestrained parts to endow the generated sentences diversity. The disadvantage is that the generation model may struggle to choose the semantic role of the concept. For example, the concept '이어지(story)’, which plays a role as a noun, holds the probability of a role as a verb by combining with the verb derivative suffix. As a consequence, the generated sentence mismatches with the answer, conversely, increasing the diversity of the outputs.
To evaluate high-level commonsense reasoning, we set the seed and randomly delete one concept of the concept set. However, this deletion method is not the best way to assess commonsense reasoning. In selecting a concept to be removed, it is necessary to delete the concept deeply related to the remaining concepts so that the model can infer the meaning of deleted concept using the remaining concepts. For example, suppose the knowledge graph with more numerous data based on Korean commonsense knowledge such as ConceptNet is established. In that case, we would delete the more proper concept from the commonsense knowledge perspective.
E Crowd-sourcing Template
Figure 6: The interfaces employed for human annotators experiments on CrowdWorks AI & Human Resources Platform. Upper box displays content morpheme concepts and a model-generated sentence to request tagging human evaluation score. 17 expert annotators assess the second-level congruency (Grammar Correction), the factuality, the commonsense, and the fluency as described in §4.1. Lower box also exhibits content morpheme concepts and an example answer to ask generating two additional human references. 22 human annotators combine given concepts and produce two references following guidelines in §3.2.