mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans (2024)

Abstract: It is very challenging to curate a dataset for language-specific knowledge and common sense in order to evaluate natural language understanding capabilities of language models. Due to the limitation in the availability of annotators, most current multilingual datasets are created through translation, which cannot evaluate such language-specific aspects. Therefore, we propose Multilingual CommonsenseQA (mCSQA) based on the construction process of CSQA but leveraging language models for a more efficient construction, e.g., by asking LM to generate questions/answers, refine answers and verify QAs followed by reduced human efforts for verification. Constructed dataset is a benchmark for cross-lingual languagetransfer capabilities of multilingual LMs, and experimental results showed high languagetransfer capabilities for questions that LMs could easily solve, but lower transfer capabilities for questions requiring deep knowledge or commonsense. This highlights the necessity of language-specific datasets for evaluation and training. Finally, our method demonstrated that multilingual LMs could create QA including language-specific knowledge, significantly reducing the dataset creation cost compared to manual creation. The datasets are available at https://huggingface.co/ datasets/yusuke1997/mCSQA.

heading: 1 Introduction
text: Can you choose the correct answer in Table 1? Each choice is semantically very close, making it difficult for non-native speakers to distinguish them. However, native speakers who have languagespecific commonsense and knowledge can choose the most plausible choice considering subtle nuances. Despite the need to consider different backgrounds for each language, the datasets to evaluate the natural language understanding (NLU) capabilities of language models (LMs) are mostly for
Japanese Question (Translated to English)
Q:お年寄りは？ (Who is the elderly person?)
(a)わし (me) (b)わたし (me) (c)ぼく (me) (d)おれ (me) (e)うち (me)
English Question (Translated to Japanese)
Q: How do we make a cake? (ケーキを作るにはどうする？)
(a) roast (焼く) (burn) (b) broil (焼く) (burn) (c) grill (焼く) (burn) (d) toast (焼く) (burn) (e) bake (焼く) (burn)
Table 1: Examples require language-specific knowledge. They cannot be solved without such knowledge, as the translations consolidate the nuances into a single word.
a few major languages such as English, and thus, many languages lack such datasets. When focusing on the cross-lingual capability of LMs, datasets created from scratch in multiple languages are lim-
1 14182
ited, and currently, evaluations mostly use datasets created through translation. However, as can be seen from the example in Table 1, datasets created through translation cannot accurately evaluate language-specific commonsense or knowledge. Therefore, it is necessary to create datasets for each language from scratch, but the manual creation of such datasets is limited by the availability of annotators and financial costs.
To tackle this problem, as shown in Figure 1, we propose a method to efficiently create multilingual NLU datasets from multilingual resources by replacing some of the manual annotation processes with generative multilingual LMs. In this study, we focus on CommonsenseQA (CSQA) (Talmor et al., 2019), a dataset for evaluating commonsense reasoning capabilities within NLU evaluations. CSQA is a major commonsense reasoning Question-Answering dataset manually created from the multilingual knowledge base ConceptNet (Speer et al., 2017). However, due to such limitations, CSQA has been created from scratch only in English and Japanese, JCommonsenseQA (JCSQA) (Kurihara et al., 2022). Therefore, we create a Multilingual CommonsenseQA (mCSQA) that extends CSQA to eight languages1 using our proposed method.
Furthermore, we evaluated the cross-lingual language-transfer capabilities of multilingual LMs focusing on language-specific common sense and knowledge using mCSQA. The results showed high language-transfer capabilities for questions that LMs could easily solve, but lower transfer capabilities for questions requiring deep knowledge or commonsense. The total cost per question in mCSQA was reduced to one-hundredth of that for CSQA.
To summarize, our contributions are as follows:
• We propose an efficient and low-cost method for creating NLU datasets by generative multilingual LMs.
• We demonstrate the potential effectiveness of using multilingual LMs for creating datasets from multilingual resources.
• mCSQA makes it possible to analyze the cross-linguistic commonsense understanding capabilities and transfer performance from each language beyond English.
1English (en), Japanese (ja), Chinese (zh), German (de), Portuguese (pt), Dutch (nl), French (fr), Russian (ru)
• The analysis revealed that, when focusing on language transfer capabilities using mCSQA, we identified cases where language-specific knowledge is required and cases where it is not, thereby confirming the need for nontranslated language-specific datasets.

heading: 2 Background and Related Work
text: Commonsense reasoning task This task evaluates how an LM can understand and infer object recognition, visual information, and cultural or societal common sense, which are not typically described in textual information. CSQA is a multiplechoice question task that asks for the most plausible choice as an answer with some variants: JCSQA is in Japanese, CommonsenseQA 2.0 (Talmor et al., 2021) is a more challenging dataset, ECQA (Aggarwal et al., 2021) requires explaining the process of deriving an answer, etc. There exist other types of commonsense tasks: COPA (Roemmele et al., 2011) and BalancedCOPA (Kavumba et al., 2019) ask about causal relationships between everyday events; SocialIQA (Sap et al., 2019b) asks about social common sense; PIQA (Bisk et al., 2020) evaluates procedural knowledge; HotpotQA (Yang et al., 2018) requires multi-hop inference; DROP (Dua et al., 2019) captures arithmetic operation capabilities; and tasks like understanding language information (Liu et al., 2022b; Kocijan et al., 2023; Sakaguchi et al., 2021; Wang et al., 2019), understanding causal relationships within documents (Mostafazadeh et al., 2020, 2016; Zhang et al., 2018; Huang et al., 2019; Ostermann et al., 2018; Smirnov, 2019), and CommonGen (Lin et al., 2020), which asks to generate common sentences from given keywords. The above datasets primarily focus on English, but there exist datasets in Japanese (Omura et al., 2020; Takahashi et al., 2019; Hayashibe, 2020), Chinese (Xu et al., 2021, 2020; Wang et al., 2022), Russian (Shavrina et al., 2020; Taktasheva et al., 2022), and Indonesian (Koto et al., 2022). For multilingual datasets, most are extended versions of existing ones through translation, such as X-COPA (Ponti et al., 2020) from COPA, X-CSQA (Lin et al., 2021) from CSQA, and X-CODAH (Lin et al., 2021) from CODAH (Chen et al., 2019). A few datasets, such as TyDiQA (Clark et al., 2020), are created for each language from scratch.
Multilingual datasets When focusing on the evaluation of multilingual performance of LMs,
2
the evaluation datasets are almost exclusively created through three methods, as shown in Table 2: (1) Translation from existing datasets in a major language, e.g., English (Lin et al., 2021; Ponti et al., 2020; Conneau et al., 2018; Artetxe et al., 2020; Yang et al., 2019); (2) Compilation of similar tasks across multiple languages (Zhang et al., 2023c; Hu et al., 2023; Adelani et al., 2022; Roy et al., 2020; Malmasi and Dras, 2015); (3) Creation from multilingual resources following the same dataset creation process (Keung et al., 2020; Huang et al., 2020; Buchholz and Marsi, 2006; Clark et al., 2020; Schwenk and Li, 2018; Kabra et al., 2023). However, (1) translated datasets often do not account for language-specific culture, knowledge, common sense, or linguistic phenomena, leading to a bias towards the background of the source language (Hu et al., 2021; Lin et al., 2021; Acharya et al., 2020; Clark et al., 2020; Park et al., 2021; Kurihara et al., 2022). (2) Simply compiling datasets curated for each individual language could allow the evaluation of language-specific knowledge and common sense. However, it is difficult to align tasks across languages since most tasks differ in their creation methods data sources or philosophies. Thus, it just leads to evaluating the transfer capability among comparable tasks, and not evaluating the true transfer capabilities across languages. Therefore, (3) only the datasets created from multilingual resources can enable the evaluation of language transfer capability, considering the differences in language-specific knowledge and common sense. Nevertheless, the manual creation of such datasets is limited by the availability of annotators and financial costs.
Dataset creation with LMs The superior performance of generative language models allows to create datasets automatically. SWAG (Zellers et al., 2018) and HellaSwag (Zellers et al., 2019) have created answer choice options through the output of LMs. Such efforts have also been extended to use LMs for data augmentation (Staliūnaitė
et al., 2021; Kumar et al., 2019, 2020; Lee et al., 2021). WANLI (Liu et al., 2022a), created from MNLI (Williams et al., 2018), employs GPT3 (Brown et al., 2020) for adversarial data augmentation with manual checks to create challenging datasets. Some studies propose methods to manually check quality of LM generation results (Tekiroğlu et al., 2020; Yuan et al., 2021; Wiegreffe et al., 2022; Wang et al., 2021a; Li et al., 2023). Additionally, there are attempts to create datasets from scratch with emergent abilities of LMs, without using any examples (He et al., 2022; Wang et al., 2021b; Schick and Schütze, 2021; Meng et al., 2022; Ye et al., 2022). However, these studies have primarily focused on a single language, e.g., English. Recently, the outputs of language models themselves have been used to create datasets (Honovich et al., 2023; Shao et al., 2023; Sun et al., 2023; Peng et al., 2023) for instruction-tuning (Wei et al., 2022a). TarGEN (Gupta et al., 2023) employs a single language model and splits the data generation process into multiple steps, inputting the suitable prompt for each step to ensure data diversity and reliability. Putri et al. (2024) focus on middle-resource (Indonesian) and low-resource (Sundanese) languages, and investigate whether LLMs can create culturally aware commonsense questions by comparing translation datasets and those generated by LLMs from scratch.

heading: 3 Datasets Creation
text: Our mCSQA construction involves three main steps (see Figure 2): extraction of sub-graphs from ConceptNet, creation of question and choice pairs with LMs, and verification of their quality by both LMs and humans. We basically follow the creation processes of CSQA and JCSQA, but modified to allow for unified processing to support multiple languages.

heading: 3.1 Extract Sub-Graphs from ConceptNet
text: ConceptNet is a graph knowledge base defined as a tuple, G = (C,R, T ), where C denotes a set of concept entities, R denotes a set of relations and T denotes a set of triples. Each triple is represented as (s, r, t) ∈ T , where s and t ∈ C are the source and target concept entities, respectively, and r ∈ R is the relation, and carry commonsense knowledge such as “(student, CapableOf, forget to do homework)”.
3
We extract subgraphs from ConceptNet, as per Figure 2-(a), that have three distinct concept entities derived from queries of concept entities and relations for each language. CSQA uses only forward queries (s, r, ?), but, similar to JCSQA, we also utilize backward queries (?, r, t). We name this subgraph as Question Sets (QSs). After extraction, we filter the QSs like CSQA and JCSQA, and applies unified filtering in mCSQA as follows:
1. Similar to CSQA, we retain only QSs that contain any types of the 22 relations2.
2. We filter out QSs where any of the concept entities consist of more than four words or only a single character3.
3. We remove QSs where any pair of concept entities is connected by a ‘Synonym’ relation in ConceptNet, or where entities are substrings of each other.
After filtering with the above settings, we randomly selected 6,000 QSs for each language4.

heading: 3.2 Create Questions with LMs
text: We employ the generative multilingual language model GPT-3.55 (Ouyang et al., 2022) to generate questions automatically to eliminate the human labor as done in CSQA and JCSQA.
2Antonym, AtLocation, CapableOf, Causes, CausesDesire, DefinedAs, DerivedFrom, Desires, DistinctFrom, EtymologicallyDerivedFrom, HasA, HasFirstSubevent, HasLastSubevent, HasPrerequisite, HasProperty, InstanceOf, MadeOf, MotivatedByGoal, NotDesires, PartOf, SymbolOf, UsedFor
3Unsegmented languages, like Japanese, are segmented by morphology in ConceptNet, so we can apply similar filtering.
4For French and Russian, the number of QSs did not reach 6,000, so we used all available QSs, totaling 4,125 and 3,901, respectively.
5We used gpt-3.5-turbo-1106.
Our construction process comprises three steps of ‘question generation’, ‘question refinement’ and ‘distractor augmentation’ as shown in Figure 2-(b). Our step differs from CSQA in the refinement step since we need to improve the question generation from LM.
We designed prompts and tuned optimized hyperparameters for each step for LMs. The details of the prompts are described in Appendix D, and the hyper-parameters are shown in Table 3.
Creating question sentences For each QS, we generated question sentences by LMs where, for each of the three target concept entities, only one serves as the answer. The prompt for LMs was inspired by the JCSQA filtering process for question creation in which systematic filtering uses textual information. The key instructions are as follows:
• Avoid including words of the target entities in the question sentence.
• Avoid using superficial information such as character count.
• End the sentence with a question mark (?).
• Be an objective question sentence.
• Consists of only one sentence.
After generating questions with LMs, we removed any question sentences that do not follow
4
these instructions or contain inappropriate expressions through pattern matching6.
Refining question sentences LMs do not always generate appropriate outputs resulting in unnatural expressions or degeneration (Liu et al., 2022c; Honovich et al., 2023; Raunak et al., 2023; Lin et al., 2020; Madaan et al., 2023). Hence, inspired by the idea of output refinement (Liu et al., 2022c; Raunak et al., 2023; Madaan et al., 2023), we refine unnatural generated question sentences into natural ones using the LM again and remove inappropriate questions as done in the previous step. Table 4 shows the percentage of sentence refinement.
Adding additional distractors We added additional incorrect choices to make the task more difficult as done in CSQA and JCSQA, but we leverage LM, not crowd workers, to formulate distractors that seemed plausible or related to the questions. Here, we asked LM to generate two plausible distractors given the three choices of a question without question itself in order to separate the question generation and answering capabilities of LMs. There is a risk of generating duplicated choices or adding correct choices since question sentence itself is not fed in this process. Hence, we remove such questions through manual verification in Section 3.3.

heading: 3.3 Question Quality Verification by LMs and Humans
text: In CSQA and JCSQA, every question is manually verified to remove low-quality questions, such as those with multiple correct answers or without correct answers in the choices. However, due to the large number of questions, manually verifying every question is not practical. Thus, we leverage simple active learning methodologies for annotation (Liu et al., 2022a; Bartolo et al., 2022; Li et al., 2023; Kratzwald et al., 2020). As shown in Figure 2-(c), initially, the LM verifies whether the questions can be answered or not, and only those
6We detected inappropriate expressions using https:// platform.openai.com/docs/guides/moderation.
questions that the LM cannot answer are manually verified.
Verification by LMs The original questions can be categorized into three types: those questions 1) which are correctly answerable by LMs, 2) which are wrongly answered by LMs, but humans can choose the correct one, 3) which are not answerable either by LMs or humans due to flaws in the question. Therefore, first, we identify the set of questions LMs can answer, and then manually verify the questions that LMs could not answer correctly to remove flawed questions.
Verification by Humans We hired two crowd workers per language via Amazon Mechanical Turk (MTurk)7. The crowd workers were presented with the question sentence, choices, and answer, and they were asked to verify if the answer could be concluded from the question and choices. We retained only those questions on which all crowd workers agreed.

heading: 3.4 Data Splitting and Statistics
text: Similar to CSQA and JCSQA, we randomly split the data for each language into training, development, and test sets with an 80/10/10 split. The mCSQA is evaluated by accuracy following the standard practice in CSQA and JCSQA. Additionally, in Section 3.3, questions that LMs could answer correctly are categorized as Easy, and those answerable by human judgment are categorized as Hard for development and test sets.
Table 5 shows the number of questions per language and split, and Figure 3 shows the percentage filtered at each step. The total cost per question is 0.002 dollars for mCSQA compared to 0.33 dollars for CSQA, reducing the cost to less than one
7There are workers for each language on MTurk (Pavlick et al., 2014). We hired workers who have an approval rate greater than 90% with at least 50 approved HITs.
5

heading: 1 .03% 12. 3% 4.0 % .05% .1 % 11.0 %
text: .4%
15.1%
. %
4.41% 4.1 %
T $36.30455 P Q $0.00266
5 .2%
20.3%
12.2%
5. % 5.35%
T $48.26667 P Q $0.00330
0. %
13. %
.2 % 3. 4% 3.5 % T $64.00842
P Q $0.00419
53. %
20. %
13. %
.35% 5.25%
T $41.93523
P Q $0.00268 0. %
1 .5%
11. %
5.33% 4. 1%
T $42.73688 P Q $0.00270
53%
21%
13. %
. 2% 5.55%
T $38.95217 P Q $0.00255
0.4%
1 .5%
11. %
5. % 4. %
T $29.00446 P Q $0.00288
1.4%
12. %
.12%
4.22% 3.3 %
T $48.57447 P Q $0.00587
Ve b H a Add D ac Ge e a e Q e Ref e Q e Ve f b LM
E gli h Ja a e e Chi e e
Ge ma P g e e D ch
F e ch R ia
Figure 4: The cost details for each language and step.
hundredth. Figure 4 shows the detailed costs. Appendix B discusses more detailed statistics,
and Figure 11 shows examples of mCSQA.

heading: 4 Evaluation for mCSQA
text: We verify that the mCSQA dataset is meaningful for evaluating the common sense reasoning capability of LMs by using various multilingual LMs.

heading: 4.1 Experimental Setup
text: Settings for LMs We used mBERT (Devlin et al., 2019), XLM-100 (Conneau and Lample, 2019), XLM-R (Conneau et al., 2020), and mDeBERTav3 (He et al., 2023) as encoder-based multilingual LMs, Llama2-70B (Touvron et al., 2023), GPT3.5 (Ouyang et al., 2022), and GPT-4 (OpenAI et al., 2024) as decoder-based multilingual LMs for the experiments. Decoder-based LMs inferred with 0-shot and 3-shot settings. For detailed experimental settings, please refer to the Appendix A.
Settings for human baseline We followed the CSQA setting and randomly selected 100 questions each from the validation and test data for every
language to measure the human baseline. We hired five new crowd-workers per language on MTurk. The answers were decided by a majority vote for each question.

heading: 4.2 Evaluation Results
text: Table 6 shows the main results. Focusing on the performance of zero-shot setting of GPT-3.5, which was used for dataset creation, we find that its performance is equivalent to or worse than that of Encoder models like XLM-RLARGE and mDeBERTav3 except for German and Russian. When comparing the results of GPT-3.5 with GPT-4, the performance of GPT-3.5 is inferior for most languages to that of GPT-4. This indicates that the questions GPT-3.5 failed to answer correctly are those that cannot be answered by the knowledge of GPT-3.5, and it implies that the root cause is a lack of knowledge or reasoning capability of GPT-3.5. Furthermore, focusing on Decoder-based models, the results are better in the 3-shot setting than in the 0-shot in most cases. This trend was observed even with the GPT-3.5 used for question creation.
The results show that the prompting technique is effective for mCSQA in exploiting the reasoning capabilities of decoder-based LMs. The trend is similar to other commonsense reasoning tasks like CSQA (Qin et al., 2023; Chowdhery et al., 2023; Wei et al., 2022c; Brown et al., 2020; Dou and Peng, 2022), indicating that mCSQA can be equally effective as a dataset for commonsense reasoning tasks. Finally, when compared to the human baseline, there is a significant gap in the results of all LMs. Thus, it can be said that even when using LMs for question creation, it is possible to create a dataset with sufficient quality and difficulty for the LMs themselves.

heading: 5 Discussion

heading: 5.1 Comparison of Easy vs. Hard
text: We compare the accuracy of Easy and Hard sets for more fine-grained analysis. Figure 5 shows the results in the test split. GPT-3.5 and GPT-4 could choose the answer correctly in most cases for the Easy sets, but the accuracy is lower in the Hard sets with a significant gap when compared with human results; note that GPT-3.5 cannot answer there sets during the dataset creation. The other LMs also show a gap in evaluation accuracy with results for Hard sets being lower than those for Easy ones.
These results, specifically the trend observed
6
English Japanese Chinese German Portuguese Dutch French Russian
dev test dev test dev test dev test dev test dev test dev test dev test
Human (Rand. 100 sent.) 87.0 93.0 89.0 95.0 91.0 87.0 96.0 96.0 93.0 93.0 98.0 97.0 96.0 92.0 87.0 94.0
mBERT-cased 60.6 61.3 66.0 63.5 65.9 63.5 58.6 57.9 65.2 61.5 54.8 57.8 46.3 47.3 32.2 31.3 mBERT-uncased 63.4 65.2 61.3 58.9 64.0 62.0 59.3 60.3 67.6 63.9 57.3 56.9 51.1 52.4 32.5 34.0 XLM-100 57.2 59.0 60.2 58.8 60.0 61.5 54.4 54.7 62.7 59.5 52.2 52.0 35.3 35.0 23.2 26.0 XLM-RBASE 68.0 69.1 68.5 66.2 69.8 68.3 63.9 62.8 69.5 67.3 62.0 64.0 47.6 45.5 36.9 37.0 XLM-RLARGE 77.2 77.5 75.7 72.6 75.0 74.1 76.2 75.4 79.0 76.4 73.0 74.7 62.0 62.3 48.9 49.5 mDeBERTa-v3 76.6 79.2 77.2 74.1 74.6 72.0 75.7 77.5 78.3 78.2 72.7 74.9 62.1 62.4 51.3 49.9
Llama2-70B (0-shot) 48.1 47.7 25.6 24.8 26.5 25.9 32.5 32.7 38.7 37.6 40.9 39.4 42.3 44.1 23.5 22.9 Llama2-70B (3-shot) 57.1 55.5 47.4 46.6 33.3 30.2 63.1 62.9 65.0 63.7 60.8 62.3 57.8 56.7 30.8 32.3 GPT-3.5 (0-shot) 76.7 77.0 76.3 76.7 64.0 63.6 81.3 81.4 77.9 77.7 82.1 81.5 78.6 77.1 53.3 53.0 GPT-3.5 (3-shot) 77.2 78.4 77.5 77.0 65.3 64.3 83.2 81.4 78.5 78.0 81.8 80.5 78.4 76.5 54.1 50.1 GPT-4 (0-shot) 80.9 80.9 78.4 77.2 66.0 65.6 81.0 81.0 78.6 77.6 83.4 81.5 78.8 77.0 49.9 47.8 GPT-4 (3-shot) 80.5 81.0 78.5 77.5 67.2 66.9 82.6 81.6 80.5 78.8 83.3 81.6 79.0 77.4 50.1 48.9
Total Easy Hard 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
Total Easy Hard 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
with GPT-3.5, show that even if LMs can create questions, it does not necessarily mean that they can answer them, and it entails that the question creation and answering are totally different capabilities. Therefore, we conclude that LMs can substitute for humans in parts of dataset creation processes from structured data and common sense reasoning task creations.

heading: 5.2 Evaluation of Multilingual LMs’ Cross-Lingual Transfer Capabilities
text: The cross-lingual transfer performance of multilingual LMs is often evaluated from English to other language directions due to linguistic resource reasons. The X-CSQA dataset (Lin et al., 2021), which consists solely of machine-translated questions from CSQA’s development and test splits, captures only the one-way cross-lingual transfer
performance of LMs that were trained in English to evaluate their performance in other languages. In contrast, mCSQA supports the evaluation of crosslingual language transfer performance in any directions among multilingual LMs that were trained in each of the eight languages.
Figure 6 shows the results of the multilingual LM, XLM-RLARGE, which was fine-tuned in each of the eight languages separately and then evaluated across all eight languages on mCSQA, using the same settings as in Table 10. The results from Figure 6 show that, regardless of the language in which they were trained, cross-lingual transfer abilities are observed in most cases for any languages given the relative lower drop of performance when compared with the monolingual performance. Moreover, in the Easy sets, the drop is within 10% for most language pairs, while in the Hard sets, it ex-
7
ceeds 20%. This indicates that questions that are relatively easy to judge (Easy sets) facilitate the language transfer capability, but questions requiring deep background knowledge (Hard sets) necessitate language-specific training and the development of LMs.

heading: 5.3 Which is Better: Monolingual Fine-tuning or Multilingual Fine-tuning?
text: Some studies (Tran and Bisazza, 2019; Dhamecha et al., 2021; Trotta et al., 2021; Barbieri et al., 2022; Portelli et al., 2023) reported that multilingual finetuning could improve a part of NLU task performance more than monolingual tuning alone. On the other hand, several studies (Tsai et al., 2019; Kondratyuk, 2019; Rønningstad, 2023; Kondratyuk and Straka, 2019) reported that it did not always improve performance in some tasks. We analyzed whether multilingual fine-tuning is effective for commonsense reasoning tasks through mCSQA. We used the whole shuffled training split data in all languages and fine-tuned XLM-RLARGE with the same setting as in Table 10. Table 7 compares the accuracy between monolingual fine-tuning, where tuning and evaluation are in the same language, and multilingual fine-tuning, where tuning is performed for all languages, evaluated for each language’s accuracy score. These results show that most languages observed improvements, especially in all cases in Easy sets. However, in Hard sets, some cases observed a decline in performance compared to the monolingual setting. Therefore, while training in a multilingual setting generally promotes
accuracy improvement, multilingual training might lead to the loss of language-specific commonsense information for questions requiring more human commonsense. This analysis complements the previous reports (Dhamecha et al., 2021; Zhang et al., 2023a; Hu et al., 2021; Mueller et al., 2020) on the successes and failures of multilingual training.
Furthermore, Table 7 shows the evaluation results of cross-lingual performance in the unseen setting, where the model was not trained on the language for evaluation data. While some languages outperform the monolingual setting, overall results indicate that training with target language data consistently yields better outcomes. This suggests that target language data acts as the secret sauce for enhancing NLU performance. Therefore, it suggests that for language-specific deep knowledge and cultural understanding, language-transfer capability alone is insufficient, and training with datasets focused on language-specific knowledge is necessary.

heading: 5.4 Case Study for Improvement through Few-Shot Learning
text: As shown in Figure 5, GPT-3.5 correctly answers most questions in the Easy setting of mCSQA, but in the Hard setting, it fails to answer most questions in the 0-shot setting. This is because GPT-3.5 is used for quality filtering of mCSQA in Section 3.3, making it inherently unable to answer the questions in the Hard setting in the 0-shot setting. However,
8
in the 3-shot setting, it shows improvement for some questions. Table 8 shows examples of questions correctly answered in the 3-shot setting. Both examples in Table 8 are mainly due to the granularity of the answers. The 3-shot setting promotes answers at an appropriate granularity for questions that are difficult to judge due to inclusive relationships.
In the top example in Table 8, careful reading of the questions narrows down the answer choices. On the other hand, in the bottom example, considering various common knowledge in daily life helps to choose the most appropriate answer. Similar characteristics were observed for other languages as well. For more details, qualitative analyses of the mCSQA dataset are described in Appendix B.

heading: 6 Conclusion and Future Directions
text: We proposed an efficient and low-cost method for creating NLU datasets from structured data by utilizing generative LMs as an alternative to traditional human annotation, often crowdsourced. Inspired by CSQA and JCSQA, we created the multilingual commonsense reasoning task dataset, mCSQA, using GPT-3.5 from the structured multilingual knowledge base ConceptNet. We demonstrated that mCSQA is useful for evaluating the commonsense reasoning capabilities of LMs. We also analyzed the language-transfer capability beyond English with mCSQA and examined the language-specific learning from two aspects: question difficulty and language information. Moreover, our study has shown that the use of multilingual LMs enables the construction of multilingual datasets. Therefore, our method can significantly reduce human labor and financial costs.
In this study, we used a single multilingual LM, but since each step is independent, it is possible to
replace the LM used in each step with another one. Furthermore, each step can be applied modularly to other methods, making it possible to use this method for creating multilingual datasets, such as those expanded through translation and manual refinement (Yanaka and Mineshima, 2022; Seo et al., 2022). We aim to extend this method to other types of commonsense reasoning tasks and NLU tasks, to efficiently create multilingual data and conduct a more comprehensive analysis of transfer capabilities across a broader range of tasks and languages.
We focused on language-specific commonsense, but languages are shared across various regions. For example, English is spoken in the United States, the United Kingdom, India, Australia, and many other regions each of which is geographically distant and diverse in terms of climate, food, and culture. Therefore, it will be necessary to create more detailed commonsense tasks that consider cultural differences rather than just language such as Kabra et al. (2023); Khanuja et al. (2024); Kim et al. (2024); Cao et al. (2024); Fung et al. (2024); Lee et al. (2024); Shwartz (2022); Hovy and Yang (2021); Yin et al. (2022); Shi et al. (2024). Our dataset construction method can be useful in creating various commonsense reasoning datasets that outgrow language limits.

heading: 7 Limitations
text: Data Resources The number of multilingual resources is significantly smaller than that of monolingual resources. Additionally, quality is not consistent, and there are imbalances in data volume across languages in these multilingual resources. In this study, we used ConceptNET, a multilingual knowledge base, and encountered these issues as well. For example, despite Spanish having a significantly higher entity count, it obtained fewer QSs
9
due to its inability to meet the required conditions because of ConceptNet’s sparsity issue, and thus it was excluded from the language selection for mCSQA. We believe these problems can be addressed through the automatic generation of knowledge bases (Zhang et al., 2020b,a; West et al., 2022; Ide et al., 2023; Nguyen et al., 2023) and data augmentation techniques for knowledge bases (Malaviya et al., 2020; Ju et al., 2022; Wu et al., 2023; Shen et al., 2023), supported by their pre-trained knowledge (Sakai et al., 2023).
Dataset Quality In this study, we used GPT-3.5 and simple prompts for data creation. Therefore, there is room for improvement in the selection of LMs and the refinement of prompts. In a pilot study, we tried using GPT-4 and recognized that it is more capable of creating datasets. However, due to budgetary constraints, we have used GPT3.5 in this study. Thus, it may become possible to create higher quality datasets at a lower cost when the API prices decrease or by switching to other strong LMs such as Gemini (Team et al., 2023), Mixtral (Jiang et al., 2024), Llama (Touvron et al., 2023), phi (Abdin et al., 2024) or Qwen (Bai et al., 2023). Additionally, employing prompt strategies that leverage the capabilities of LMs, such as Chain of Thought (CoT) (Wei et al., 2022c), Tree of Thought (ToT) (Yao et al., 2023a) and ReAct (Yao et al., 2023b), could potentially lead to the production of higher quality datasets.
Verification of dataset quality by humans The human baselines decreased the evaluation result under the Hard sets compared to Easy sets in Figure 5. Therefore, there exists a risk that the Hard sets include flawed questions, even after manual quality verification. The JCSQA has pointed out that such low-quality questions are included in CSQA, and we have confirmed that they are similarly present in JCSQA. Thus, it is extremely difficult to completely eliminate such low-quality questions. Comparing the percentage of data removed in quality verification, CSQA is 25% (3995/16242), and JCSQA is 19% (2643/13906), whereas for mCSQA, it is 27% (604/2226) and 23% (599/2510) respectively, according to Figure 3 and referenced in their respective papers. This indicates that the filtering ratios are almost comparable when compared to those, showing that this is not a problem unique to mCSQA. Therefore, reinforcing quality verification to filter out low-quality questions is a challenge in our future studies. However, as Figure 4 shows,
since more than half of the costs are already spent on manual quality verification, simply hiring more crowd workers would not be a better choice. Hence, exploring more efficient methods of quality verification as an alternative or to assist crowd workers in the future is necessary.
Human baseline The experimental results include human baselines using small sets of samples. However, Tedeschi et al. (2023) argue that human baselines may lack reliability due to factors such as the payment issues for crowd workers and the impact of random samples. Therefore, it should be noted that the human baselines in this study are merely reference values.

heading: 8 Ethical Considerations
text: License The mCSQA dataset was created entirely from the outputs of GPT-3.5 and is therefore subject to OpenAI’s license terms8. OpenAI assigns to us all rights, title, and interest in and to the output. As a result, we are retaining the ownership rights. There are no restrictions on distributing the datasets, but using OpenAI’s model output to develop models that compete with OpenAI is prohibited. However, it’s possible that these terms may change, and there may be a need to impose distribution restrictions depending on the terms.
Moderation We eliminated potentially harmful questions such as violence, sexual content, and hate speech by screening through OpenAI moderation APIs9. However, in the commonsense reasoning dataset, it cannot be guaranteed that it does not include questions that contain societal biases as collective knowledge. This issue has also been pointed out in existing datasets such as CSQA, JCSQA, and other commonsense reasoning datasets, and it is challenging to determine what is considered commonsense constitutes bias (Rajani et al., 2019; Sap et al., 2020; Bauer et al., 2023; An et al., 2023). If you encounter any harmful questions that contain such biases, please report them.
Translation Tool We used DeepL Pro10 to translate the example sentence, especially Table 1, to avoid arbitrary translation. The copyright of the translation sentences belongs to us11.
8https://openai.com/policies/terms-of-use 9https://platform.openai.com/docs/guides/
moderation 10https://www.deepl.com/translator 11https://www.deepl.com/pro-license
10

heading: A Details of the Experimental Settings
text: We used mBERT (Devlin et al., 2019), XLM100 (Conneau and Lample, 2019), XLM-R (Conneau et al., 2020), and mDeBERTa-v3 (He et al., 2023) as encoder-based multilingual LMs, Llama270B (Touvron et al., 2023), GPT-3.5 (Ouyang et al., 2022), and GPT-4 (OpenAI et al., 2024) as decoderbased multilingual LMs for the experiments. Table 9 shows the details of the LMs. Encoder-based LMs were fine-tuned following the settings in Table 10. Decoder-based LMs inferred with 0-shot and 3-shot settings12 with a fixed seed value. For GPT-3.5 and GPT-4, top_p and temperature were set to 0 to achieve as deterministic outputs as possible. For Llama2-70B, output was generated greedy, and outlines (Willard and Louf, 2023) were used to fix the output format.

heading: B Qualitative Analysis of mCSQA
text: Table 11 shows examples of mCSQAs for each language. The examples in Table 11 are accompanied by English translations using DeepL13 to avoid arbitrary translation.
B.1 Can Multilingual LMs Take into Account Language-specific Knowledge?
Case study When we examine some cases in Table 11, such as the examples from the Dutch Hard set and the Russian Hard set, we find that the English translations contain duplications among the question choices. However, these duplications arise not from differences in tense or conjugation, but from semantic differences unique to each language, which a native speaker, equipped with languagespecific knowledge and common sense, could easily distinguish. Furthermore, in the case of the German Easy sets, knowledge of Germany’s unique education system is required, which might be challenging for those unfamiliar with it. Yet, for German speakers, it is common knowledge that such
12In the 3-shot setting, the examples were selected randomly from the training data and included both Easy and Hard sets.
13We are using DeepL Pro (https://www.deepl.com/ translator), therefore, the copyright of the translations belongs to us. (https://www.deepl.com/pro-license)
education systems, such as the Abitur14 related to the Gymnasium15, making it answerable for those knowledgeable in German. This demonstrates that multilingual LMs are capable of generating questions that include the kind of language-specific knowledge and common sense that a native speaker would possess.
The effectiveness of the CSQA style QA When examining the Japanese Hard set in Table 10, all the choices translate into the names of seafood in English, which does not match the context of a female singer mentioned in the question. Japanese native speakers would normally recognize them as seafood names too, making it seem at first glance that there is no correct answer. However, the correct choice, ‘あゆ’ (ayu), when pronounced in Japanese, is read as ’ayu’. This pronunciation is widely known across Japan as the nickname for the famous singer ‘浜崎あゆみ’ (Ayumi Hamasaki)16, making it a plausible choice even though it’s not strictly correct. It allows for a satisfactory selection by Japanese native speakers with language-specific knowledge, common sense, and cultural awareness, and is not answerable by English translation only. In Japan, nicknames are often derived from abbre-
14https://en.wikipedia.org/wiki/Abitur 15https://en.wikipedia.org/wiki/Gymnasium_
(school) 16https://en.wikipedia.org/wiki/Ayumi_Hamasaki
25
viations of their names or can suggest the names of objects. The distractor ‘Wakame’ is known as the name of a character from the long-running, famous anime ‘Sazae-san’17 but not as a singer, thus serving its purpose as a distractor in this question effectively. Similarly, if there were a choice like ‘い くら’ (common meaning: red caviar; pronounced: ikura), the plausibility of choice in this question might have been divided. Recently, ‘ikura’18 has become a popular name, associated with a member of ‘Yoasobi’19, a popular artist group among young people. Adding such a choice would confuse the choice of the correct answer because both choices are plausible, so it would not serve effectively as a distractor. This case shows that the choices can define the scope of common sense, thus making the question effective in evaluating common sense accurately.
B.2 The Relationship between Knowledge, Culture, Commonsense, and Social Bias
B.2.1 What is the Commonsense? As can be seen from Table 11 and the discussions in section B.1, language-specific common sense is closely related to knowledge and culture. The ConceptNet used in this study does not limit the scope of common sense and deals with a wide range of common sense, enabling the inclusion of questions from various backgrounds into mCSQA, following the same trend as CSQA and JCSQA.
Generally, commonsense not based on the specific culture or knowledge of a language is likely to be a common understanding across all languages, making such problems potentially answerable through the language-transfer ability of multilingual LMs. However, as shown in Table 1, the granularity of actions, events, and behaviors differs by language, which can be considered to be influenced by the cultural background of the language area.
This study focuses on language-specific common sense that cannot be addressed by translations of datasets from other languages, and the culture and knowledge included in them are shared among native speakers. Therefore, answering questions that require language-specific backgrounds necessitates a certain level of knowledge and culture specific to each language. However, content that
17https://en.wikipedia.org/wiki/Sazae-san 18https://en.wikipedia.org/wiki/Lilas_Ikuta 19https://en.wikipedia.org/wiki/Yoasobi
is too specialized falls outside the scope of common sense, and common sense and backgrounds vary among individuals. Therefore, we emphasize the precision of coverage in the manual question quality verification steps and employ a majority vote baseline to avoid overly relying on specific knowledge or culture.
In this way, questions were created that have language-specific common sense which is general for native speakers but not too specialized. If there was a need to create questions asking for knowledge specialized in specific fields, other knowledge bases such as ATOMIC (Sap et al., 2019a), and CCSK (Nguyen et al., 2023) could be used. However, this study focused on multilingual performance, deeming ConceptNet appropriate for mCSQA.
B.2.2 Is Commonsense Social Bias?
Since commonsense includes implicit cognition, it may contain social and cultural biases, and some methods for the removal of explicit and implicit social biases have been proposed (Sap et al., 2020; Field and Tsvetkov, 2020; Huang et al., 2021; Lent and Søgaard, 2021; Emelin et al., 2021; Bauer et al., 2023).
Social Chemistry 101 (Forbes et al., 2020), BBQ (Parrish et al., 2022), and SODAPOP (An et al., 2023) have been proposed for identifying biases within models or for bias detection using LMs. However, it remains challenging to address situations where biased thinking may only emerge when considering multiple-choice QA, where bias does not occur in isolation.
The definition of bias and common sense changes over time and varies from society to society, and what is considered common sense can shift to bias (Lee et al., 2023). Therefore, regular updates to the commonsense reasoning datasets are necessary. Our method for generating commonsense reasoning task datasets using LMs allows for low-cost update operations, making it possible to adapt to the changing boundaries between common sense and bias over time. However, this does not fundamentally address the inclusion of bias in datasets. Moreover, such issues require a deep chain of semantic thinking for resolution, making filtering based on textual information inappropriate. Therefore, it is necessary to develop methods to remove potential biases in commonsense reasoning task datasets in future work.
26

heading: C Discoveries about the LMs Capabilities
text: C.1 Can LMs Create Questions including Commonsense?
Generation capability CommonGen (Lin et al., 2020) is one of the commonsense reasoning datasets that evaluates whether it is possible to create commonsense sentences from a given set of keywords. According to the leaderboard of CommonGen20, the performance of GPT-3.5 used in our dataset creation demonstrates a capability for generating commonsense sentences comparable to those written by humans. However, there is still room for improvement in aspects such as word order. Therefore, we introduced refinement steps to encourage corrections in word order and other errors. Since language models have high performance in Grammar Error Correction (GEC) (Loem et al., 2023; Sottana et al., 2023; Fang et al., 2023; Coyne et al., 2023; Kaneko and Okazaki, 2023; Kwon et al., 2023), combining sentence generation from keywords with GEC capabilities in a pipeline helps to compensate for the weaknesses of language models. We believe that the quality of mCSQA questions is at least not inferior to those created by crowd-workers. The capability of multilingual LMs to create commonsense sentences from given keywords has also been demonstrated in the Korean CommonGen (Seo et al., 2022), indicating that it is possible to generate commonsense sentences multilingually.
Ensuring the quality of questions In this study, we have created commonsense reasoning dataset questions using keywords extracted from ConceptNet. Therefore, the language-specific knowledge and commonsense for each language are guaranteed by ConceptNet. Moreover, the LM creates questions following the given instructions through its emergent capabilities from each keyword. To enhance the language-specific performance of the multilingual LM for each language, we have created prompts for each language in this study. As can be seen from the discussion in section B.1 and Table 11, it has become possible to generate questions that possess language-specific knowledge. One of the reasons for the capability to create questions with language-specific knowledge may be attributed to the training data of the LM. For example, Wikipedia, one of the common training data for LMs, has each language which contains descrip-
20https://github.com/allenai/CommonGen-Eval
tions of knowledge unique to that language, so by posing questions in each language, it is thought that knowledge specific to each language is invoked, enabling the generation of questions based on the knowledge of each language. However, this is a hypothesis, and further analysis will be necessary for verification in future work. Moreover, we have added distractors in addition to the keywords used for generating the question, which means that even if a question can be generated, it may not necessarily be answerable. Furthermore, questions that cannot be answered have been removed, thus ensuring the difficulty and answerability of the QA.
C.2 Multilingual Capabilities
Is polyglot template effective? We translated the prompt to use question generation for each language and tuned it to convey the same meaning in each language in Section D aimed to emergence the language-specific knowledge. However, it is known that current generative LMs have mainly trained on English, which is better performance for queries made in English. However, several studies (Ahn et al., 2022; Shi et al., 2023; Wei et al., 2022b; Awasthi et al., 2023; Kasai et al., 2023; Jin et al., 2023) show enough performance even if multilingual queries. Note that the reported performance focuses on the ability to answer specific tasks on benchmarks and does not evaluate the emergent multilingual ability, especially question generation. Nevertheless, Whitehouse et al. (2023) shows that the text generation capability beyond English. As shown in Table 10, we were able to generate questions containing language-specific knowledge from the given keywords as intended by using prompts translated into each language. We were able to generate questions that require deep reasoning, including cultural backgrounds and language-specific pronunciation information as shown in Section B.1. Therefore, we conclude that using prompts tailored for each language is effective.
Is GPT-3.5 Multilingual LM? Yes, some studies (Lai et al., 2023; Armengol-Estapé et al., 2022; Zhang et al., 2023b) have indeed examined multilingual performance, and the training data also includes multilingually21. Therefore, the multilingual capabilities of GPT-3.5, GPT-4, and Llama used in our experiment have also been evalu-
21https://github.com/openai/gpt-3/tree/master/ dataset_statistics
27
ated (Ahuja et al., 2023; Schott et al., 2023; Chen et al., 2024), leading us to consider these as multilingual LMs. However, they still rely predominantly on information from Western norms (Cao et al., 2023; Arora et al., 2023; Havaldar et al., 2023), making this issue an ongoing challenge to be addressed in the future.
Exhortation to multilingual instruction-tuning dataset. Instruction-tuning (Wei et al., 2022a; Longpre et al., 2023; Chung et al., 2022; Wang et al., 2023) can enhance the quality of LMs, e.g. ability to follow instructions and NLU performance. However, in Section 2, the current multilingual datasets include those created through translation, which means that instruction-tuning using such data may not lead to the acquisition of data bias or language-specific knowledge. Given these considerations, the multilingual instruction-tuning data (Kew et al., 2023; Singh et al., 2024) proposed recently often utilize datasets created through translation, leading to the occurrence of the aforementioned issues to a considerable extent. Consequently, the effectiveness of such instruction-tuning may be diminished. For commonsense reasoning tasks in multilingual instruction-tuning datasets, they sometimes use X-CSQA (Lin et al., 2021). However, since it cannot handle language-specific knowledge or commonsense effectively, it is preferable to use data created from scratch, like mCSQA. Currently, due to data resource issues, reliance on translated data is inevitable, but we hope that in the future, it will be replaced by language-specific data.
C.3 Hard Sets are Truly Hard?
The Hard sets consist of questions that the LM used for question creation could not answer, thus reflecting the characteristics of that LM. However, despite the influence of specific LM’s character, a performance decline in the Hard sets compared to the Easy sets was observed across all models. Therefore, while the strict division of sets depends on the model, it has become clear that there is a similar trend across LMs as a whole. For this reason, scoring is conducted without distinguishing between Easy and Hard, using a total score for the entire set, which allows for the absorption of differences due to the models.
C.4 Generation Bias and Annotation Artifacts It has been pointed out that datasets created by LMs contain generation bias (Omura et al., 2020; Zellers et al., 2019; Tamborrino et al., 2020), and those created by crowd-workers include specific patterns (Annotation Artifacts) (Gururangan et al., 2018; Chen et al., 2019; Omura et al., 2020). Annotation artifacts, in particular, have been noted in natural language inference tasks such as MNLI (Williams et al., 2018) and SNLI (Bowman et al., 2015), where choices can be easily distinguished by superficial words like “not”.
However, Tamborrino et al. (2020) show that the impact of Annotation Artifacts is not present in the CSQA task. Similarly, in this study, we have separated question generation ability and answering ability during the question generation process and shuffled the options, so there are no clues included in the dataset. Moreover, we create Hard sets, even if such biased questions existed, the evaluation is conducted without these biases, allowing for an evaluation that removes these biases.

heading: D Prompts for Creating mCSQA
text: The prompts used for creating mCSQA are presented as follows: English in Table 12, Japanese in Table 13, Chinese in Table 14, German in Table 15, Portuguese in Table 16, Dutch in Table 17, French in Table 18 and Russian in Table 19.
In each prompt template, the words within the curly brackets are replaced with data-specific terms22 before input to the LM.
Furthermore, as discussed in Section C.2, each template was translated exactly to elicit languagespecific knowledge of each language. The translations were carried out using both GPT-3.5 and DeepL to ensure there were no semantic differences, with manual fixing applied as needed. We use the OpenAI API’s JSON mode23 has facilitated the retrieval of generation results.
Our findings as a tip, when inputting structured data such as keywords, doing so in a format similar to a programming code like list type, allows us to obtain results that more following the prompt instructions. This improvement can be attributed to the LM’s learning to enhance coding abilities, which is believed to have improved its recognition capabilities.
22https://peps.python.org/pep-0498/ 23https://platform.openai.com/docs/guides/
text-generation/json-mode
28
29
30
31
32
33

heading: Future Work
text: However, this is a hypothesis, and further analysis will be necessary for verification in future work. Moreover, we have added distractors in addition to the keywords used for generating the question, which means that even if a question can be generated, it may not necessarily be answerable. Furthermore, questions that cannot be answered have been removed, thus ensuring the difficulty and answerability of the QA. C. 2 Multilingual Capabilities
Is polyglot template effective? We translated the prompt to use question generation for each language and tuned it to convey the same meaning in each language in Section D aimed to emergence the language-specific knowledge. However, it is known that current generative LMs have mainly trained on English, which is better performance for queries made in English. However, several studies (Ahn et al. , 2022; Shi et al. , 2023; Wei et al. , 2022b; Awasthi et al. , 2023; Kasai et al. , 2023; Jin et al. , 2023) show enough performance even if multilingual queries. Note that the reported performance focuses on the ability to answer specific tasks on benchmarks and does not evaluate the emergent multilingual ability, especially question generation. Nevertheless, Whitehouse et al. (2023) shows that the text generation capability beyond English. As shown in Table 10, we were able to generate questions containing language-specific knowledge from the given keywords as intended by using prompts translated into each language. We were able to generate questions that require deep reasoning, including cultural backgrounds and language-specific pronunciation information as shown in Section B. 1. Therefore, we conclude that using prompts tailored for each language is effective. Is GPT-3. 5 Multilingual LM? Yes, some studies (Lai et al. , 2023; Armengol-Estapé et al. , 2022; Zhang et al. , 2023b) have indeed examined multilingual performance, and the training data also includes multilingually21. Therefore, the multilingual capabilities of GPT-3. 5, GPT-4, and Llama used in our experiment have also been evalu-
21https://github. com/openai/gpt-3/tree/master/ dataset_statistics
27
ated (Ahuja et al. , 2023; Schott et al. , 2023; Chen et al. , 2024), leading us to consider these as multilingual LMs. However, they still rely predominantly on information from Western norms (Cao et al. , 2023; Arora et al. , 2023; Havaldar et al. , 2023), making this issue an ongoing challenge to be addressed in the future. Exhortation to multilingual instruction-tuning dataset. Instruction-tuning (Wei et al. , 2022a; Longpre et al. , 2023; Chung et al. , 2022; Wang et al. , 2023) can enhance the quality of LMs, e. g. ability to follow instructions and NLU performance. However, in Section 2, the current multilingual datasets include those created through translation, which means that instruction-tuning using such data may not lead to the acquisition of data bias or language-specific knowledge. Given these considerations, the multilingual instruction-tuning data (Kew et al. , 2023; Singh et al. , 2024) proposed recently often utilize datasets created through translation, leading to the occurrence of the aforementioned issues to a considerable extent. Consequently, the effectiveness of such instruction-tuning may be diminished. For commonsense reasoning tasks in multilingual instruction-tuning datasets, they sometimes use X-CSQA (Lin et al. , 2021). However, since it cannot handle language-specific knowledge or commonsense effectively, it is preferable to use data created from scratch, like mCSQA. Currently, due to data resource issues, reliance on translated data is inevitable, but we hope that in the future, it will be replaced by language-specific data. C. 3 Hard Sets are Truly Hard?
The Hard sets consist of questions that the LM used for question creation could not answer, thus reflecting the characteristics of that LM. However, despite the influence of specific LM’s character, a performance decline in the Hard sets compared to the Easy sets was observed across all models. Therefore, while the strict division of sets depends on the model, it has become clear that there is a similar trend across LMs as a whole. For this reason, scoring is conducted without distinguishing between Easy and Hard, using a total score for the entire set, which allows for the absorption of differences due to the models. C. 4 Generation Bias and Annotation Artifacts It has been pointed out that datasets created by LMs contain generation bias (Omura et al. , 2020; Zellers et al. , 2019; Tamborrino et al. , 2020), and those created by crowd-workers include specific patterns (Annotation Artifacts) (Gururangan et al. , 2018; Chen et al. , 2019; Omura et al. , 2020). Annotation artifacts, in particular, have been noted in natural language inference tasks such as MNLI (Williams et al. , 2018) and SNLI (Bowman et al. , 2015), where choices can be easily distinguished by superficial words like “not”. However, Tamborrino et al. (2020) show that the impact of Annotation Artifacts is not present in the CSQA task. Similarly, in this study, we have separated question generation ability and answering ability during the question generation process and shuffled the options, so there are no clues included in the dataset. Moreover, we create Hard sets, even if such biased questions existed, the evaluation is conducted without these biases, allowing for an evaluation that removes these biases