Venue: NAACL 2025

# KMMLU:

Measuring Massive Multitask Language Understanding in Korean

Guijin Son1,2,6 Hanwool Lee2,3 Sungdong Kim4,5 Seungone Kim5, Niklas Muennighoff7,

Taekyoon Choi4, Cheonbok Park4, Kang Min Yoo4, and Stella Biderman2

###### Abstract

We propose KMMLU, a Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. While prior Korean evaluation tools heavily rely on translated versions of existing English benchmarks, KMMLU is collected from original Korean exams, thereby capturing linguistic and cultural aspects of the Korean language. Recent models struggle to show performance over 60%, significantly below the pass mark of the source exams (80%), highlighting the room for improvement. Notably, one-fifth of the questions in KMMLU require knowledge of Korean culture for accurate resolution. KMMLU thus provides a more accurate reflection of human preferences compared to translated versions of MMLU and offers deeper insights into LLMs' shortcomings in Korean knowledge. The dataset and codes are made publicly available for future research.

1Yonsei University, 2EleutherAI, 3NCSOFT AI, 4NAVER Cloud, 5KAIST

AI, 6OnelineAI, 7Contextual AI

## 1 Introduction

Recent works often leverage translated versions of MMLU Hendrycks et al. (2020) to evaluate the multilingual capabilities of large language models (LLMs) OpenAI (2023; Qwen, 2024; Chen et al., 2023; Zhao et al., 2024). However, naively translating English benchmarks into a language of interest faces critical limitations. First, machine translation can lead to a compromised dataset with issues like unnatural language, typos, and grammatical mistakes Xia et al. (2019); Riley et al. (2023); Yao et al. (2023). Second, MMLU, designed primarily for English speakers, includes content that assumes knowledge of the American legal system or requires familiarity with English slang and culture Lee et al. (2023); Jin et al. (2023); Son et al. (2023); Li et al. (2023a); ZaloAI-JAIST (2023). Thus, while translated versions hint at multilingual proficiency, they fail to capture the cultural aspects that native speakers might consider to be crucial.

To address this issue for the Korean NLP community, we introduce KMMLU, a comprehensive benchmark consisting of 35,030 questions spanning 45 subjects. Unique to KMMLU is its sourcing: _all_ questions are derived from Korean exams, ensuring authentic Korean language without any translated material. Additionally, our questions are _localized_ to Korea: they reflect the cultural attitudes of Koreans, rather than Westerners (see Figure 1). Our comparative analysis, depicted in Figure 3, shows that KMMLU surpasses previously translated benchmarks by offering questions that are linguistically natural and steeped in the Korean cultural context, providing a deeper insight into LLM performance in Korean (Section 6). To address the shortcomings of Korean benchmarking, which relies heavily on translated Park et al. (2024); Ham et al. (2020); Jin et al. (2023) or private datasets Park et al. (2024, 2021); Lee et al. (2024), we make KMMLU and its corresponding evaluation codes publicly available 1.

Footnote 1: [https://huggingface.co/datasets/HAERAE-HUB/KMMLU](https://huggingface.co/datasets/HAERAE-HUB/KMMLU)

We evaluate 24 different LLMs across 4 categories: (1) Multilingual Pretrained Models Touvron et al. (2023); Young et al. (2024); Bai et al. (2023); (2) Multilingual Chat Models; (3) Korean Pretrained Models Ko et al. (2023); and (4) Proprietary Models including those serviced in Korea OpenAI (2023; Team et al., 2023; Yoo et al., 2024). Our results show significant room for improvement, with GPT-4 scoring the highest at 59.95%. Surprisingly, we see little evidence of a "curse of multilinguality" Conneau et al. (2019); Pfeiffer et al. (2022) discussed in previous work comparing BLOOM Workshop et al. (2022) to monolingual English models Biderman et al. (2023); Peng et al. (2023).

We conduct a detailed analysis to better understand how LLMs utilize Korean knowledge in

question-answering. Initially, we observe that, despite GPT-4's overall excellence, it displays gaps in areas demanding _localized knowledge_, demonstrating the importance of localizing benchmarks. For example, in Korean History, GPT-4 (OpenAI, 2023) achieves a 35% success rate compared to HyperCLOVA X (Yoo et al., 2024), a Korean-specific LLM, which scores 44%. Notably, HyperCLOVA X is unique in its consistent improvement with the use of Chain-of-Thought (CoT) prompting, indicating the challenge non-Korean LLMs face in producing accurate Korean explanations.

## 2 Related Work

### Benchmarks for Large Language Models

Benchmarks are essential for accurately understanding and tracking the evolving capabilities of LLMs. Traditionally, benchmarks focused on linguistic tasks Rajpurkar et al. (2016); Wang et al. (2019, 2019), but with the recent surge of more capable LLMs, such approaches have become obsolete. To address this gap, new benchmarks have emerged, focusing on higher-level abilities such as commonsense reasoning Clark et al. (2018); Sakaguchi et al. (2021); Zellers et al. (2019), mathematical reasoning Hendrycks et al. (2021); Cobbe et al. (2021), code generation Chen et al. (2021); Li et al. (2023), and multi-turn conversations Zheng et al. (2023). Notably, some efforts have concentrated on evaluating the capabilities via expansive datasets covering a wide range of knowledge-based topics Hendrycks et al. (2020); Srivastava et al. (2022); Sawada et al. (2023). Most famously, MMLU (Massive Multitask Language Understanding) Hendrycks et al. (2020) spans 57 subjects, evaluating LLMs across various disciplines. While many of these efforts have primarily focused on the English language, there has been progress in adapting and creating similar benchmarks for other languages Li et al. (2023); Huang et al. (2023); Zeng (2023); Ghahroodi et al. (2024); Koto et al. (2024, 2023); Tam et al. (2024). Although KMMLU may be similar to the mentioned works in its motivation, it is the first of its kind in Korea, providing

Figure 1: Examples of questions from KMMLU categorized by the type of Korean knowledge required. English translations are added for broader accessibility.

 

a valuable foundation for Korean NLP research.

### Korean Benchmarks

Prior benchmarks for the Korean language specialized on natural language understanding tasks, including natural language inference (Ham et al., 2020), machine reading comprehension (Lim et al., 2019), and hate speech detection (Moon et al., 2020). The Korean Language Understanding Evaluation (KLUE) benchmark (Park et al., 2021), analogous to the General Language Understanding Evaluation (GLUE) (Wang et al., 2019b), compiled eight downstream tasks aimed at gauging the comprehension of the Korean language. Nonetheless, these benchmarks exhibited limitations in evaluating reasoning abilities, rendering them insufficient for assessing LLMs. The Ko-H5 (Park et al., 2024) aims to overcome these shortcomings by offering an extensive array of reasoning benchmarks, such as HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), ARC (Clark et al., 2018), and TruthfulQA (Lin et al., 2021). Nevertheless, they also rely on machine/human translations that are susceptible to errors. Moreover, their datasets are private and they do not provide evaluation for models larger than 14 billion parameters, thus limiting its transparency and usefulness.

Recent benchmarks developed for Korean have shifted their focus toward preserving the linguistic and cultural nuances, going beyond mere translation of original English benchmarks. In this context, Jin et al. (2023) developed the Korean Bias Benchmark for Question Answering, which is derived from the original BBQ dataset (Parrish et al., 2022). This benchmark is specifically adjusted to align with Korean cultural contexts. Additionally, efforts have been made to create authentic Korean datasets from scratch to capture subtle cultural nuances. For instance, the HAE-RAE Benchmark (Son et al., 2023) and CLlcK (Kim et al., 2024) assess cultural and linguistic understanding specific to Korea. Furthermore, KorNAT (Lee et al., 2024) was introduced to specifically evaluate alignment with national characteristics of South Korea, focusing on two key aspects: social values and common knowledge.

The KMMLU benchmark extends beyond previous efforts as the first extensive knowledge benchmark covering a broad and deep array of fields, including Biology, Chemistry, Criminal Law, Taxation, Electrical Engineering, Aviation Engineering, and others. Additionally, it is unique in that the dataset, prompts, and evaluation codes are publicly available, ensuring transparent and reproducible assessment, Importantly, it is built entirely from Korean exams, providing a representation of Korean culture and linguistic nuances.

## 3 Kmmlu

### Task Overview

KMMLU is a collection of 35,030 multiple-choice questions spanning 45 categories, including HUMSS (Humanities and Social Science), STEM (science, technology, engineering, and mathematics), Applied Science, and other professional-level knowledge. Within STEM, the focus is on topics emphasizing scientific principles, from the natural and physical sciences to technological and engineering disciplines. Meanwhile, Applied Science encompasses industry-specific subjects such as Aviation Engineering and Maintenance, Gas Technology and Engineering, and Nondestructive Testing. HUMSS covers an extensive range of subjects, including history and psychology, offering in-depth insights into the diverse facets of human society and culture. The remaining subjects that do not fit into any of the three categories are put into Other.

We predominantly source the questions from Ko

\begin{table}
\begin{tabular}{l c c c r r l} \hline \hline
**Korean Benchmarks** & **Translated** & **Public?** & **\# Category** & **\# Instances** & **Type** \\ \hline KonNII \& KorSTS & O & O & 2 & 6,389 & Understanding \\ KLUE & X & X & 8 & 38,500 & Understanding \\ Ko-H5 & O & X & 5 & 25,700 & Reasoning \\ KobBQ & \(\triangle\) & O & 268 & 76,048 & Bias \\ HAE-RAE Bench. & X & O & 6 & 1,538 & Cultural Knowledge \\ CLiCK & X & O & 11 & 1,995 & Cultural Knowledge \\ KornAT & X & \(\triangle\) & 16 & 10,000 & Cultural Alignment \\ \hline KMMLU (Ours) & X & O & 45 & 35,030 & Expert Know. \& Reasoning \\ \hline \hline \end{tabular}
\end{table}
Table 1: Overview of Korean benchmarks. The “# Instances” denotes the size of the test set. The KLUE dataset has not released its test set, thus we do not consider it public. The complete release of the KöNAT dataset is scheduled for the future and is currently unavailable as indicated by \(\triangle\). Parts of KoBBQ are translated.

 

rean License Tests, notably, some of the license tests KMMLU draws from exams that requires at least 9 years of industry experience. In addition, KMMLU includes questions that require an understanding of cultural, regional, and legal knowledge to solve, as shown in Figure 1. For further details, see Table 2.

Footnote 3: [https://www.tensorflow-project.org/](https://www.tensorflow-project.org/)

### Dataset Creation

Our dataset is a compilation of questions from 533 diverse sources, spanning the Public Service Agtitude Test (PSAT), Korean License Tests, and the College Scholastic Ability Test (CSAT). This collection includes a broad academic spectrum, from high school to professional levels.

Initially, we collected 371,002 questions using automatic crawling. We then implement heuristic filters to erase duplicated samples or parsing errors, including stopwords, regex patterns, and model-based classifiers. Additionally, the format is standardized by excluding questions with fewer than four options and adjusting those with more than four. This filtering reduces the dataset by 34% to 243,777 questions. The significant reduction in dataset size stems from two main factors: firstly, we prioritize quality over quantity, employing broad filters to eliminate any questionable content, even at the cost of removing some valid samples. Secondly, we observe a high degree of similarity among questions, especially those that are seasonally repeated, resulting in extensive deduplication.

We gather human accuracy data from actual test-takers where available. Approximately 90% of our dataset's exams include human performance data, with an average accuracy of 62.6%. Most of the license exams in the dataset require an 80% score to pass. For the PSAT, the average passing score of the last 5 years has been 83.7%. Thus, achieving over 80% on KMMLU can be considered the equivalent of the minimum performance of a human expert, while the best experts are likely to score close to 100%. The dataset is structured into three components: a training set, a few-shot development set, and a test set. The few-shot development set features five questions per subject to support context learning Brown et al. (2020). The training set includes 208,522 questions, suitable for both hyperparameter tuning and model training. The test set is a collection of questions with the lowest human accuracies, each subject consisting of 100 instances at a minimum and 35,050 questions in total. However, it is important to note that comparing human accuracy directly may not be desirable due to variations in test origins and test-taker populations, with some groups being more professional than others. While collecting the human data ourselves could potentially address this issue, we could not do so due to budget constraints.

Before finalization, we released the dataset to the public for six months. During this period, we received five issues reported by the community, and 741 instances were modified accordingly. Additionally, the 35,030 questions in the test set underwent manual review to remove copyrighted materials. We replaced 147 instances, including copyrighted materials. No additional errors were identified during this process. Finally, we conduct an analysis based on Xu et al. (2024)'s method to look for potential data leakages. We observe both open and proprietary LLMs fail to recall the KMMLU benchmark implying low likelihood of benchmark contamination. For details, see Section C.

### CoT Exemplar Creation

Chung et al. (2022) devise 5-shot of exemplars to test CoT reasoning over MMLU Hendrycks et al. (2020)2. Similarly, we create 5-shot of CoT exemplars for each subject to test models' reasoning capabilities on our benchmark. However, writing an accurate rationale for expert-level tests with various ranges is a difficult problem. Although the ideal solution might be to invite experts for each test, we decide to leverage assistance from various LLMs,

\begin{table}
\begin{tabular}{l c} \hline \hline
**Category** & **\# Questions** \\ \hline \multicolumn{3}{c}{_Prerequisites_} \\ None & 59,909 \\
1 Prerequisite Test & 12,316 \\
2 Prerequisite Tests & 776 \\
2+ Years of Experience & 65,135 \\
4+ Years of Experience & 98,678 \\
9+ Years of Experience & 6,963 \\ \hline \multicolumn{3}{c}{_Question Type_} \\ Positive & 207,030 \\ Negation & 36,747 \\ \hline \multicolumn{3}{c}{_Split_} \\ Train & 208,522 \\ Validation & 225 \\ Test & 35,030 \\ \hline Total & 243,777 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Overview of questions in KMMLU: This table summarizes questions by number of prerequisites for human examinees, whether the question contains negation, and train/validation/test splits.

 

considering resource constraints. Specifically, we employ two LLMs, GPT-4 and HyperCLOVA X, with diverse prompt techniques, zero-shot CoT Kojima et al. (2022) and browsing-augmented CoT 3.

Footnote 3: It is similar to ReAct prompting Yao et al. (2022).

First, we elicit rationale and corresponding answers from the LLMs using both prompt techniques. Besides, we utilize a majority voting method, self-consistency Wang et al. (2022), over ten reasoning paths obtained by oversampling. As a result, this step produces \(4\times 10\) rationales for each input, i.e., \(4=2\) LLMs and \(2\) prompt types. Then, we choose the top-\(4\) rationales ordering by longer and less repetitive output. Finally, authors manually select the most appropriate rationale among the top-\(4\) and revise it with thorough inspections if necessary. For quality control, we ensure two workers for each question. We find about 87% of agreement between two workers at the first iteration. We iteratively validate the remaining conflicted examples. In total, we create \(45\times 5=225\) exemplars for the CoT inference within our benchmark. Please see Appendix H for more details.

### Kmmlu-Hard

KMMLU comprises 35,030 questions, outnumbering its predecessors, MMLU Hendrycks et al. (2020) and CMMLU Li et al. (2023a). Thus, in addition to KMMLU, we create KMMLU-HARD for more targeted and efficient evaluation. The KMMLU-HARD subset includes 4,104 questions that at least one of the following models--GPT-3.5 Turbo, Gemini Pro, HyperCLOVA X, and GPT-4--fails to answer correctly. These questions are equally distributed across all categories, each containing 23 to 100 questions.

## 4 Experimental Setup

### Evaluation Methodology

In our evaluations of LLMs on KMMLU, we employ two distinct settings for a comprehensive comparison. First, the Direct method prompts the model to generate the most plausible option via greedy decoding. In this process, each model generates a response from its entire vocabulary, which makes \(\frac{1}{vocab_size}\) its random baseline. Second, CoT allows the model to generate text freely and leverages RegEx to parse the results. By generating a sequence of reasoning before the final answer, CoT has succeeded in aiding LLMs to solve reasoning-heavy tasks. Models are set to use greedy decoding for the CoT generation. All evaluations in this paper, regardless of the method, are done in a few-shot setting with five exemplars. Due to hardware constraints, we run our experiments with open models using 8-bit quantization.

### Models

In our study, to provide a comprehensive overview of existing LLMs in answering expert-level Korean questions, we evaluate 24 models varying in size, language, and training phase.

The 24 models include:

1. Multilingual Pretrained Models: Llama-2 (7B, 13B, 70B) Touvron et al. (2023), Qwen (7B, 14B, 72B) Bai et al. (2023), and Yi (6B, 34B) Young et al. (2024);
2. Multilingual Chat Models: Chat versions of Llama-2, Qwen, and Yi;
3. Korean Pretrained Models: Polyglot-Ko (1.3B, 3.8B, 5.8B, 12.8B) Ko et al. (2023);
4. Proprietary Models: GPT-3.5/4 OpenAI, 2023)4, Gemini Pro Team et al. (2023) and HyperCLOVA X Yoo et al. (2024)5. Footnote 4: We use the 0613 version for both GPT models.

The inclusion of English & Chinese bilingual models aims to explore potential spillover effects, given the historical influence of Chinese Hanja on the Korean language. Further details on the models are provided in Appendix E and Table 18. Evaluation results for newer models are updated in Tables 20 and 21. It should be noted that the results in no way change the conclusions of our paper.

## 5 Evaluation Results

Pretraining ComputeWe compare the performance of 27 LLMs using the Direct method in Table 3. We observe a clear trend across pretrained and finetuned models, where those with a larger computing budget exhibit superior performance.6 This scaling behavior indicates that increased computing resources - reflected in the number of parameters and the size of the training corpus - enhance a model's capacity to handle complex language tasks more accurately. Notably, despite being trained exclusively in Korean, Polyglot-Ko-12.8B' 

performance only marginally exceeds the random baseline of 25%, is on par with that of the English-centric Llama-2-13B, and lags behind Yi and Qwen models of similar size. This emphasizes the importance of long training runs in achieving high performance: while Polyglot-Ko-12.8B is approximately compute-optimally trained (Hoffmann et al., 2022), the order of magnitude increase in the training data size brings substantial increases in the performance of these non-optimally trained models. This disparity in training resources is further illustrated in Figure 2, where Polyglot-Ko's significantly lower training budget compared to its counterparts is evident.

Fine-Tuning In Table 3, we also observe that fine-tuning Pretrained Models do not necessarily lead to better performance. In our experiments, models often exhibit minor performance differences between their base and chat versions. This aligns with past studies that suggested fine-tuning methods such as supervised fine-tuning, direct preference optimization, or reinforcement learning to have minor improvements in the knowledge of language models (Bi et al., 2024). Interestingly, Qwen-72B and Llama-2-70B experience -3.55% and -5.81% of performance drop respectively. We suspect that the ability to solve Korean questions in pretrained models of different languages originally stems from their failure to filter out Korean text from their pretraining corpora perfectly. However, datasets used during the post-training process are often curated with greater precision, possibly excluding all non-target languages. Therefore, such might harm the Korean language proficiency of such models.

Multilinguality at ScaleThe "curse of multilinguality" (Conneau et al., 2019; Pfeiffer et al., 2022) refers to the apparent decrease in model capabilities

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline \multicolumn{1}{c}{**Model**} & \multicolumn{1}{c}{**STEM**} & \multicolumn{1}{c}{**Applied Science**} & \multicolumn{1}{c}{**HUMSS**} & \multicolumn{1}{c}{**Other**} & \multicolumn{1}{c}{**Average**} \\ \hline  & \multicolumn{6}{c}{_Mutillingual Pretrained Models_} \\ \hline LLama-2–7B & 24.68 & 25.90 & 25.06 & 24.30 & 25.00 \\ LLama-2-13B & 33.81 & 33.86 & 26.26 & 30.86 & 31.26 \\ LLAMA-2-70B & 41.16 & 38.82 & 41.20 & 40.06 & 40.28 \\ Yi-6B & 35.47 & 34.23 & 33.46 & 35.70 & 34.70 \\ Yi-34B & 44.31 & 40.59 & 47.03 & 43.96 & 43.90 \\ Qwen-7B & 22.74 & 23.83 & 9.44 & 17.59 & 18.52 \\ Qwen-14B & 36.68 & 35.85 & 21.44 & 29.26 & 30.92 \\ Qwen-72B & 50.69 & 47.75 & 54.39 & 50.77 & 50.83 \\ \hline \multicolumn{6}{c}{_Multilingual Chat Models_} \\ \hline LLama-2-7B-Chat & 28.60 & 29.03 & 26.01 & 27.10 & 27.71 \\ LLama-2-13B-Chat & 30.36 & 29.09 & 26.40 & 29.05 & 28.73 \\ LLama-2-70B-Chat & 35.98 & 34.36 & 32.19 & 35.35 & 34.47 \\ Yi-6B-Chat & 35.58 & 34.55 & 34.39 & 35.95 & 35.11 \\ Yi-34B-Chat & 41.83 & 38.05 & 46.94 & 42.05 & 42.13 \\ Qwen-7B-Chat & 20.26 & 22.16 & 8.67 & 15.70 & 16.82 \\ Qwen-14B-Chat & 32.78 & 33.94 & 19.31 & 26.75 & 28.33 \\ Qwen-72B-Chat & 47.57 & 46.26 & 49.05 & 46.33 & 47.28 \\ \hline \multicolumn{6}{c}{_Korean Pretrained Models_} \\ \hline Polyglot-Ko-1.3B & 28.77 & 28.02 & 26.99 & 28.11 & 27.97 \\ Polyglot-Ko-3.8B & 29.68 & 31.07 & 26.59 & 29.54 & 29.26 \\ Polyglot-Ko-5.8B & 29.18 & 30.17 & 26.73 & 29.12 & 28.83 \\ Polyglot-Ko-12.8B & 29.27 & 30.08 & 27.08 & 30.55 & 29.26 \\ \hline \multicolumn{6}{c}{_Proprietary Models_} \\ \hline GPT-3.5-Turbo & 44.64 & 42.11 & 40.54 & 42.61 & 42.47 \\ Gemini-Pro & 51.30 & 49.06 & 49.87 & 50.61 & 50.18 \\ HyperCLOVA X & 50.82 & 48.71 & 59.71 & 54.39 & 53.40 \\ GPT-4 & **59.95** & **57.69** & **63.69** & **58.65** & **59.95** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Average accuracy(%) calculated using the Direct method in a 5-shot setting across the entire test set. The highest-scoring model across the entire table is highlighted in **bold**, and the best model within each category is underlined. Random guessing has an accuracy of 25% on all subjects. Please see Tables 22-26 for detailed results.

 

when models are trained on multilingual corpora. While the curse can be severe for small models, it has been observed for masked language models that it weakens with scale (Goyal et al., 2021; Pfeiffer et al., 2022). Empirically, this seems not to be the case for the decoder-only BLOOM model (Workshop et al., 2022) as several papers have found that monolingual English models substantially outperform BLOOM on English tasks (Biderman et al., 2023; Peng et al., 2023). In contrast, we find evidence of positive transfer between languages, with large multilingual models like Llama-2, Yi, and Qwen substantially outperforming the monolingual Polyglot-Ko. Though the multilingual models are trained on an order of magnitude more tokens than Polyglot-Ko, they encounter much less Korean text during their pretraining phases. For instance, Llama-2 is trained on 2 trillion tokens, with only 0.06% in Korean, amounting to 1.2 billion tokens. Yi employs a language filter to exclude languages other than Chinese and English, and Qwen mentions that a significant portion of its data is in English and Chinese. In comparison, Polyglot-Ko models are trained on 167 billion to 219 billion tokens depending on the model size. Our results show that scaled decoder-only models acquire capabilities in languages they are severely undertrained in, a finding that aligns with prior work (Muennighoff et al., 2023).

Chain-of-Though PromptingWe employ a few-shot CoT prompting (Wei et al., 2022), leveraging 5-shot exemplars (Section 3.3) to examine whether advanced prompting method could improve performance. Since the CoT prompting requires much longer sequence generation than the Direct method, we compare four LLMs based on the KMMLUHard subset, considering resource constraints 7. In Table 4, we find that only HyperCLOVA X reliably improves the performances across categories with the CoT prompting, while other LLMs often show degradation with the CoT. In particular, GPT-3.5-Turbo and GPT-4-Turbo show better performances with CoT on STEM and Applied Science, but drastic performance drops on HUMSS. We presume the Korean-specific context in the HUMSS category is relatively hard to generalize by learning other languages, resulting in unfaithful explanations (Turpin et al., 2023).

Footnote 7: We utilize GPT-4-Turbo (gpt-4-0125-preview) instead of GPT-4 for the same reason.

## 6 Are Translated Benchmarks Enough?

The simplest approach to scale multilingual benchmarks involves translation. Efforts to translate MMLU have been implemented repeatedly (Park et al., 2023; Lai et al., 2023), some including professional human translators (OpenAI, 2024; Institute, 2024), offering clear benefits by enabling parallel performance comparisons and insights. However, KMMLU, sourced directly from texts originally written in Korean, provides irreplaceable advantages and evaluations. This section compares KMMLU with translated benchmarks (Section 6.1) and further analyzes performances related to Korean contexts. Further details are in Appendix B.5.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**STEM**} & \multicolumn{2}{c}{**Applied Science**} & \multicolumn{2}{c}{**HUMSS**} & \multicolumn{2}{c}{**Other**} & \multicolumn{2}{c}{**Total**} \\  & Direct & CoT & Direct & CoT & Direct & CoT & Direct & CoT & Direct & CoT \\ \hline Qwen-72b-Chat & 24.36 & 19.00 & 24.25 & 18.67 & 18.52 & 16.50 & 23.09 & 18.38 & 22.59 & 18.18 \\ HyperCLOVA X & 14.36 & 28.00 & 14.58 & 24.83 & 20.62 & **30.21** & 18.90 & **25.59** & 17.06 & **27.11** \\ GPT-3.5-Turbo & 22.36 & 23.27 & 21.00 & 23.67 & 19.74 & 15.35 & 21.30 & 20.25 & 21.10 & 20.70 \\ GPT-4-Turbo & **28.64** & **30.91** & **28.25** & **34.84** & **33.37** & 19.68 & **30.55** & 20.10 & **30.52** & 25.28 \\ \hline \hline \end{tabular}
\end{table}
Table 4: 5-shot accuracy on KMMUL-Hard subset (Section 3.4) according to prompting method, Direct and CoT (Wei et al., 2022). Please see Table 27 for detailed results.

Figure 2: Average performance of Polyglot-Ko, Llama-2, Yi, and Qwen models. ExaFLOP on the axis represents the scale of computational operations, where each unit corresponds to \(10^{18}\) floatingpoint operations (FLOPs). The total FLOPs are estimated as \(6\times\#param\times\#tokens\)(Kaplan et al., 2020).

 



### Analysis of Korea-Specific Instances

To provide a deeper insight into how KMMLU differs from past efforts that translate MMLU (Park et al., 2023; Chen et al., 2023), we compare the two on two fronts: the naturalness of phrasing and the necessity for specialized Korean knowledge. For the analysis, we randomly selected ten questions from each category within both datasets, resulting in 570 questions from a Korean-translated MMLU and 450 questions from KMMLU--two authors evaluated each question.

Figure 3 reveals a difference in how the two subsets appear to native Korean speakers. KMMLU questions are significantly more natural and culturally relevant, highlighting the limitations of MMLU in reflecting the nuances of the Korean language and cultural specifics. MMLU, derived from American tests, is inherently biased towards western culture. We provide examples in Table 11. Conversely, 20.4% of KMMLU requires understanding Korean cultural practices, societal norms, and legal frameworks. Leveraging the data from instructkr (2024), a Korean equivalent to Chiang et al. (2024), we measure the impact of such characteristics. Analyzing the ranking correlations of KMMLU and a translated MMLU against Elo rankings from 3253 human votes, Table 6 confirms that KMMLU more accurately mirrors human preferences, as shown by closer Levenshtein distances and stronger ranking correlations.

### kmmlu-Kor

To specifically assess models on questions that necessitate Korean knowledge, we introduce the KMMLU-KOR subset, a collection of 1,305 hand-picked questions (Further details in Appendix I). Table 6 presents a performance analysis, focusing on the discrepancy between general performance and Korea-specific questions. HyperCLOVA X, shows the smallest performance decline at 3.4%, whereas GPT-3.5-Turbo and GPT-4 experience drops of 7.11% and 8.74%, respectively. This underscores the significance of incorporating Korean linguistic and cultural contexts in model training to enhance performance on KMMLU.

Error AnalysisWe observe HyperCLOVA X to correctly answer 47 questions that GPT-4 misses. Of these, 18 pertain to history, 11 to accounting and taxation, and 7 to geology. We pinpoint 73 Koreae-specific questions that all models fail, including 31 concerning the Security Services Industry Act, a law first implemented in 2017 and frequently amended since. This highlights gaps in models' understanding of recent legislative changes. Additionally, we also find 23 questions that use the phrase "our country," presuming knowledge that this refers to Korea, further complicating comprehension for non-specialized models.

## 7 Conclusion

In this work, we introduce the **KMMLU** Benchmark--a comprehensive compilation of 35,030 expert-level multiple-choice questions spanning 45 subjects, all sourced from original Korean exams without any translated content. Our findings highlight significant room for improvement in the Korean proficiency of state-of-the-art LLMs. We discover that the improvements in the performance of non-Korean LLMs stem from capabilities unrelated to Korean, underscoring the importance of Korean pre-training for better performance in Korea-specific contexts. We expect the KMMLU benchmark to aid researchers in identifying the shortcomings of current models, enabling them to assess and develop better Korean LLMs effectively.

\begin{table}
\begin{tabular}{l c c r} \hline \hline
**Model** & **Original** & **KOR Sub.** & \(\Delta\) \\ \hline Qwen-72B & 50.83 & 42.56 & \(-8.27\) \\ Qwen-72B-CHAT & 47.28 & 35.13 & \(-12.15\) \\ GPT-3.5-Turbo & 42.47 & 35.36 & \(-7.11\) \\ Gemini-Pro & 50.18 & 40.83 & \(-9.35\) \\ HyperCLOVA X & 53.40 & 50.00 & \(-3.40\) \\ GPT-4 & 59.95 & 51.21 & \(-8.74\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance comparison on the original KMMLU and the KMMLU-KOR subset.

Figure 3: Comparison of MMLUL (translated via GPT-4) and KMMLU (ours).





## 8 Limitations

While we put our greatest effort into creating a benchmark with extensive coverage, there are some limitations that future research will need to address. First, due to concerns over copyright issues, we removed a substantial number of questions from the Korean language, medical, and financial domains, thereby creating coverage gaps. Secondly, the recent surge of chat-aligned LLMs has cast doubt on the effectiveness of traditional benchmarks for assessing generative abilities and instruction-following skills. While MMLU continues to be a de facto standard for evaluating a broad range of knowledge, there is a shifting trend towards using dedicated LLM Judges and crowdsourced human preferences, such as the LMSys Chatbot Arena Zheng et al. (2023), for assessing generative capabilities. Future efforts should aim to expand Korean benchmarking tools to include assessments of generative abilities. Moreover, the potential misuse of benchmarks may pose societal risks. Optimizing solely for benchmarks may create models that perform poorly in real-world applications and should be avoided.

## References

* Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. _arXiv preprint arXiv:2309.16609_.
* Bi et al. (2024) Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepsee llvm: Scaling open-source language models with longtermism. _arXiv preprint arXiv:2401.02954_.
* Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In _International Conference on Machine Learning_, pages 2397-2430. PMLR.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. _arXiv preprint arXiv:2107.03374_.
* Chen et al. (2023) Zhihong Chen, Shuo Yan, Juhao Liang, Feng Jiang, Xiangbo Wu, Fei Yu, Guuming Hardy Chen, Junying Chen, Hongbo Zhang, Li Jianquan, Wan Xiang, and Benyou Wang. 2023. MultilingualSIFT: Multilingual Supervised Instruction Fine-tuning.
* Chiang et al. (2024) Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Ducheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating lllms by human preference. _arXiv preprint arXiv:2403.04132_.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. _arXiv preprint arXiv:1803.05457_.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, et al. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. _arXiv preprint arXiv:1911.02116_.
* Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Goldling, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023. A framework for few-shot language model evaluation.
* Ghahroodi et al. (2024) Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgneib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. Khayyam challenge (persianmmluj): Is your llvm truly wise to the persianlanguage? _arXiv preprint arXiv:2404.06644_.
* Goyal et al. (2021) Naman Goyal, Jingfei Du, Myle Ott, Giri Annantharan, and Alexis Conneau. 2021. Larger-scale transformers for multilingual masked language modeling. _arXiv preprint arXiv:2105.00572_.
* Ham et al. (2020) Jiyeon Ham, Yo Jeong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Shi. 2020. KorNLI and KorSTS:

New benchmark datasets for Korean natural language understanding. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 422-430, Online. Association for Computational Linguistics.
* Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. _arXiv preprint arXiv:2009.03300_.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. _arXiv preprint arXiv:2103.03874_.
* Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. _arXiv preprint arXiv:2203.15556_.
* Hosseini et al. (2021) Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. 2021. Understanding by understanding not: Modeling negation in language models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1301-1312, Online. Association for Computational Linguistics.
* Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. _arXiv preprint arXiv:2305.08322_.
* Institute (2024) Alexandra Institute. 2024. Multilingual mmlu. [https://huggingface.co/datasets/alexandrainst/mmlu](https://huggingface.co/datasets/alexandrainst/mmlu).
* instructkr (2024) instructkr. 2024. ko-chatbot-arena-leaderboard. [https://huggingface.co/spaces/instructr/ko-chatbot-arena-leaderboard](https://huggingface.co/spaces/instructr/ko-chatbot-arena-leaderboard).
* Jin et al. (2023) Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Huaran Lee. 2023. Kobbq: Korean bias benchmark for question answering. _arXiv preprint arXiv:2307.16778_.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _arXiv preprint arXiv:2001.08361_.
* Kim et al. (2024) Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. 2024. CLIck: A benchmark dataset of cultural and linguistic intelligence in Korean. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 3335-3346, Torino, Italia. ELRA and ICCL.
* Ko et al. (2023) Hyunwoong Ko, Richang Yang, Minho Ryu, Taekyoon Choi, Seungmu Yang, Sungho Park, et al. 2023. A technical report for polyglot-ko: Open-source large-scale korean language models. _arXiv preprint arXiv:2306.02254_.
* Kojima et al. (2022) Takesh Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. _Advances in neural information processing systems_, 35:22199-22213.
* Koto et al. (2023) Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in indonesia: A comprehensive test on indommlu. _arXiv preprint arXiv:2310.04928_.
* Koto et al. (2024) Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Asiba Almaeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, et al. 2024. Arabicmnmulu: Assessing massive multitask language understanding in arabic. _arXiv preprint arXiv:2402.12840_.
* Loh (2023) Taekyoon Choi L. Junbu. 2023. llama-2-koen-13b.
* Lai et al. (2023) Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dermoncourt, Ryan A Rossi, and Thien Huh Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. _arXiv preprint arXiv:2307.16039_.
* Lee et al. (2023) Hwaran Lee, Seokhee Hong, Joonsuk Park, Takeyoung Kim, Meeyoung Cha, Yejin Choi, Byoung Pi Kim, Gunhee Kim, Eun-Ju Lee, Yong Lim, et al. 2023. Square: A large-scale dataset of sensitive questions and acceptable responses created through human-machine collaboration. _arXiv preprint arXiv:2305.17696_.
* Lee et al. (2024) Jiyoung Lee, Minwoo Kim, Seungho Kim, Junghwan Kim, Seunghyun Won, Huaran Lee, and Edward Choi. 2024. Kornat: Llm alignment benchmark for korean social values and common knowledge. _arXiv preprint arXiv:2402.13605_.
* Li et al. (2023a) Haonan Li, Yixuan Zhang, Faji Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023a. Cmmlu: Measuring massive multitask language understanding in chinese. _Preprint_, arXiv:2306.09212.
* Li et al. (2023b) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. 2023b. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval).
* Lim et al. (2019) Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. Korquad1. 0: Korean qa dataset for machine reading comprehension. _arXiv preprint arXiv:1909.07005_.
* Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. _arXiv preprint arXiv:2109.07958_ 

Jihyung Moon, Won Ik Cho, and Junbun Lee. 2020. Beep! korean corpus of online news comments for toxic speech detection. _arXiv preprint arXiv:2005.125303_.
* Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubark, Samuel Albanie, Zaidi Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. _Preprint_, arXiv:2211.01786.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* OpenAI (2024) OpenAI. 2024. Multilingual massive multitask language understanding (mmmlu). [https://huggingface.co/datasets/openai/MMMLU](https://huggingface.co/datasets/openai/MMMLU).
* Park et al. (2024) Chanjun Park, Hyeonwoo Kim, Dahhyun Kim, Seonghwan Cho, Sanghoon Kim, Sukyoung Lee, Yungi Kim, and Huwalsuk Lee. 2024. Open k-llm leaderboard: Evaluating large language models in Korean with kohreshold searchmark. _arXiv preprint arXiv:2405.20574_.
* Park et al. (2023) Chanjun Park, Hwasuke Lee, Hyunbyung Park, Hyeonwoo Kim, Senghoon Kim, Seonghwan Cho, Sunghun Kim, and Sukyang Lee. 2023. Open ko-llm leaderboard. [https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard).
* Park et al. (2021) Sungjoon Park, Jhiyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwen Park, Chisung Song, Jungseong Kim, Yongsook Song, Taehwan Oh, et al. 2021. Klue: Korean language understanding evaluation. _arXiv preprint arXiv:2105.09680_.
* Parrish et al. (2022) Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Hutt, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 2086-2105, Dublin, Ireland. Association for Computational Linguistics.
* Peng et al. (2023) Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranth Kiran GV, et al. 2023. Rwkv: Reinventing rnns for the transformer era. _arXiv preprint arXiv:2305.13048_.
* Pfeiffer et al. (2022) Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. _arXiv preprint arXiv:2205.06266_.
* Qwen (2024) Qwen. 2024. Owen 1.5.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. _arXiv preprint arXiv:1606.05250_.
* Riley et al. (2023) Parker Riley, Timothy Dozat, Jan A Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, and Noah Constant. 2023. Frmt: A benchmark for few-shot region-aware machine translation. _Transactions of the Association for Computational Linguistics_, 11:671-685.
* Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. _Communications of the ACM_, 64(9):99-106.
* Sawada et al. (2023) Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nag, Kshitij Gupta, and Aran Komatsuzaki. 2023. Arb: Advanced reasoning benchmark for large language models. _arXiv preprint arXiv:2307.13692_.
* Son et al. (2023) Guijin Son, Hanwood Lee, Suwan Kim, Huigeo Kim, Jaecheol Lee, Je Won Yeom, Jiyhu Jung, Jung Woo Kim, and Songseong Kim. 2023. Hae-rae bench: Evaluation of Korean knowledge in language models. _Preprint_, arXiv:2309.02706.
* Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abuwal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _arXiv preprint arXiv:2206.04615_.
* Tam et al. (2024) Zhi-Rui Tam, Ya-Ting Pai, Yen-Wei Lee, Sega Cheng, and Hong-Han Shuai. 2024. An improved traditional chinese evaluation suite for foundation model. _arXiv preprint arXiv:2403.01858_.
* Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. _arXiv preprint arXiv:2312.11805_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Turpin et al. (2023) Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. _arXiv preprint arXiv:2305.04388_.
* Wang et al. (2019) Alex Wang, Yada Prusachatchau, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019_ 

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. Glue: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019_.
* Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Akakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. _arXiv preprint arXiv:2203.11171_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Workshop et al. (2022) BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. _arXiv preprint arXiv:2211.05100_.
* Xia et al. (2019) Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig. 2019. Generalized data augmentation for low-resource translation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5786-5796.
* Xu et al. (2024) Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. Benchmarking benchmark leakage in large language models. _arXiv preprint arXiv:2404.18824_.
* Yao et al. (2023) Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. 2023. Empowering llm-based machine translation with cultural awareness. _arXiv preprint arXiv:2305.14328_.
* Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. _arXiv preprint arXiv:2210.03629_.
* Yoo et al. (2024) Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, et al. 2024. Hyperclova x technical report. _arXiv preprint arXiv:2404.01954_.
* Young et al. (2024) Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. _arXiv preprint arXiv:2403.04652_.
* ZaloAI-JAIST (2023) ZaloAI-JAIST. 2023. Vmlu.
* Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? _arXiv preprint arXiv:1905.07830_.
* Zeng (2023) Hui Zeng. 2023. Measuring massive multitask chinese understanding. _arXiv preprint arXiv:2304.12986_.
* Zhao et al. (2024) Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. _arXiv preprint arXiv:2401.01055_.
* Zheng et al. (2023) Linamin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhughao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-jaduge with mt-bench and chatbot arena. _arXiv preprint arXiv:2306.05685_.

## Appendix A Dataset Details

In this section, we provide additional details on the KMMLU benchmark.

### Category Distribution

The KMMLU dataset consists of three subsets Train, Validation and Test. In Figures 4 and 5 we visualize the distribution of each sub-category for the train and test set respectively. The validation set contains 5 instances for all sub-categories.

### Source Distribution

KMMLU sources questions from 533 diverse exams, including the PSAT, Korean License Tests, and the CSAT. Table 7 presents the distribution of the sources, with License Tests comprising the majority at 235,976 samples.

Figure 6 provides an overview of the years each question was sourced from. The questions spans 25 years starting from 1999 to 2023. The average number of instances for each year is 9030.52, with the most at 2015 by 13583 samples and the least at 1999 with 170 samples.

### Copyrights and License

To prevent copyright issues, we conduct a manual review of the _Test_ and _Validation_ sets of the KMMLU benchmark. We remove instances from exams with restrictive licenses, primarily in the medical and financial domains. We also filter questions that include segments of Korean literature where the respective authors hold copyright and are challenging to manage individually. However, the _Train_ set consists of 208,522 instances, making

\begin{table}
\begin{tabular}{p{56.9pt} p{284.5pt}} \hline \hline
**Source** & **\# Instances** \\ \hline License Tests & 235,976 \\ PSAT & 7373 \\ CSMET & 428 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Source distribution 

manual review impractical. Instead, we use insights from our manual review of the test and validation subsets to identify and remove questions in the train set that originate from sources identified as private. Additionally, we collect headers and footers from each webpage during the crawling process to remove those containing copyright information. The final dataset is published under a CC-BY-ND license and is freely available at HuggingFace8. Additionally during our manual review we do not identify any personally identifiable information or offensive content.

Footnote 7: [https://huggingface.co/datasets/HAERAE-HUB/KMPLU](https://huggingface.co/datasets/HAERAE-HUB/KMPLU)

Footnote 8: [https://github.com/EleutherAI/](https://github.com/EleutherAI/)

For better reproducibility, our evaluation codebase9 is built using Eleu

Figure 4: Sub-Category distribution of the train set in log-scale.

Figure 5: Sub-Category distribution of the test set in log-scale.

 

(MIT License) Gao et al. (2023). This includes the evaluation code, generation configuration for all settings (open and proprietary models, direct and CoT settings), and our prompts and CoT exemplars. All components are available under the MIT license.

## Appendix B Additional Analysis

### How Does Continual Pretraining Affect Korean Proficiency?

Models from the Yi series, initially trained for bilingual (English and Chinese) usage, significantly outperform the Polyglot-Ko models. This performance gap further expands with the continual pretraining of these models. In Table 8, we assess the Yi-Ko 6B and 34B models, each continually trained for an additional 60 billion and 40 billion tokens, respectively, after expanding their vocabulary to include Korean. Additionally, we analyze the available checkpoints of the Yi-Ko-34B model in Figure 7, noting a consistent performance increase following an initial decline at the first checkpoint. This early drop is likely due to the expanded vocabulary, which can disrupt training initially Zhao et al. (2024).

Additionally, in Figure 8, we evaluate the performance of three models--Llama-2-Ko-7B (both with and without vocabulary expansion) and Yi-Ko-34B--across 40 billion tokens of continual pretraining.10 Our analysis reveals two key insights: First, while vocabulary expansion initially leads to a drop in performance, the models subsequently recover and stabilize after surpassing 10 billion tokens of training. This challenges previous findings that suggested vocabulary expansion is unfavorable at such training scales. Interestingly, Table 12 shows that the perplexity of Yi-Ko-34B is higher than its original model Yi-34B, likely due to the undertraining of newly added tokens, indicating room for improvement. Second, learning outcomes vary despite being trained on the same dataset, suggesting that differences in inherent model capabilities significantly influence performance.

Footnote 10: The Yi–Ko-6B model is excluded as intermediary checkpoints are unavailable.

### Do Machines Handle Problems with Negation Effectively?

Table 9 demonstrates a notable trend in language model performance on the KMMLU test set: models perform better on questions that include negations. This finding contrasts with previous studies Hosseini et al. (2021); Li et al. (2023) that identified LLMs to suffer when dealing with negated questions. However, this does not suggest that negation in Korean presents a lower difficulty level than in other languages. Instead, the improved performance may be attributed to the nature of the questions in KMMLU, where negation is more common in declarative knowledge questions, which are generally easier for models to handle compared to procedural knowledge questions Hendrycks et al. (2015).

Figure 8: Performance of Llama-2-Ko-7B (both with and without vocabulary expansion) and Yi-Ko-34B on the KMMLU benchmark for each checkpoint.

Figure 6: Overview of the years each question was sourced from.

Figure 7: Performance of Yi-Ko-34B on the KMMLU benchmark on each checkpoint.

 

2020). For example, the math subset, which is the most challenging subset for most LLMs, does not include any negated questions. Furthermore, Table 10 illustrates that only 20% of STEM and 19% of Applied Science questions include negation, in contrast to 45% in the HUMSS subset.

### When Do Korean Proprietary Models Outperform GPT-4?

Figure 9 provides a comparative performance analysis between the top-performing Korean model, HyperCLOVA X, and GPT-4 across various disciplines, with detailed numerical results available in Appendix 9. The comparison shows that GPT-4 generally outperforms HyperCLOVA X in most subjects, with performance differentials ranging from a significant 22.0% in Accounting to a marginal 0.5% in Taxation. Although specific details on the pretraining of both models are kept private HyperCLOVA X is designed for bilingual (English and Korean) use, while GPT-4 supports many more languages. This observation corroborates our earlier findings from Section 5 that the curse of multilinguality diminish as models scale. Notably, HyperCLOVA X demonstrates superior performance over GPT-4 in Korean History and Criminal Law. This is likely attributable to HyperCLOVA X's specialized focus on the Korean language, which presumably enhances its proficiency in topics requiring regional-specific knowledge and understanding.

These characteristics are evident in Table 19, which compares the performance of models on the KMMLU-KOR subset. HyperCLOVA X and GPT-4 scores 50.00 and 51.21, respectively. The performance gap narrows as GPT-4 experiences a decrease of 8.74 points, while HyperCLOVA X sees a smaller decline of 3.4 points. This indicates that HyperCLOVA X is more resilient to questions on Korean knowledge, maintaining closer to its original performance.

### Do Machines Also Err Where Humans Often Do?

In Figure 10, we compare the performance of LLMs against human accuracy. The findings indicate that LLMs do not exhibit a performance trend that correlates with human performance. Instead, the models display similar performance levels irrespective of the variance in human accuracy. This observation aligns with insights from the Hendrycks et al. (2020), which reported that GPT-3 achieved a higher score in College Mathematics at 35.0%, compared to 29.9% in Elementary Mathematics, suggesting that the model's performance does not necessarily scale with the complexity of the task as judged by human standards. Interestingly, the models demonstrate a strong correlation with each other, implying that despite being trained on distinct datasets, they possess similar capabilities. This phenomenon indicates that there may be underlying commonalities in how these models process and generate responses, leading to a similar performance trend.

\begin{table}
\begin{tabular}{l c} \hline \hline  & **Category** & **\% of Negated Q.** \\ \hline STEM & 20.54\% \\ Applied Science & 19.16\% \\ HUMSS & 45.76\% \\ Other & 34.83\% \\ \hline Math & 0.00\% \\ Electrical Eng. & 9.70\% \\ Aviation Eng. \& Maint. & 14.40\% \\ \hline \hline \end{tabular}
\end{table}
Table 10: Ratio of Negated Questions in each category.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **STEM** & **Applied Science** & **HUMSS** & **Other** & **Average** \\ \hline llama-2-Ko-7B & 31.60 & 32.50 & 26.33 & 30.00 & 30.10 \\ Yi-Ko-6B & 40.69 & 39.52 & 40.50 & 41.60 & 40.55 \\ Yi-Ko-34B & 50.44 & 46.95 & 53.63 & 51.13 & 50.46 \\ \hline \hline \end{tabular}
\end{table}
Table 8: 5-shot accuracy on KMMLU with models Continual Pretrained on Korean.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Models** & **W Negation** & **W/O Negation** \\ \hline llama-2-70B & 40.2 & 40.08 \\ Yi-34B & 47.26 & 42.43 \\ dyen-72B & 53.57 & 48.82 \\ \hline Gemini-Pro & 55.05 & 48.63 \\ GPT-3.5-Turbo & 45.61 & 40.39 \\ GPT-4 & 65.53 & 57.88 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Comparison of accuracy between questions with and without negation. Evaluation is done in 5-shot setting using the Direct Method.

 



### Why are native benchmarks important?

Continuing from Section 6, we highlight the importance of original benchmarks over translated versions. Figure 3 illustrates that MMLU questions often reflect U.S.-centric content, such as categories like _"highschoolgovernmentand__politics,"_ which require understanding of the American governmental system, and _"miscellaneous,"_ which assumes knowledge of American slang. These examples, shown in Table 11, reveal the cultural biases inherent in the dataset. Regardless of translation quality, using such content as a measure of Korean language competency is fundamentally flawed. For instance, proficiency in questions about the U.S. Sixth Amendment does not translate into effectiveness of a legal LLM for Korean contexts.

## Appendix C Contamination Check

The questions for the KMMLU benchmark are sourced from publicly available Korean tests. Accordingly, there may be concerns about whether the sources have been included in the pretraining corpora of the tested language models. Unfortunately, none of the examined models disclose their training data, which precludes a direct assessment of potential knowledge leakage. As an alternative, we refer to the methods of Xu et al. (2024) to estimate potential leakage using perplexity and n-gram accuracy. We sample 2,000 instances from the KMMLU test set and paraphrase them using GPT-4-Turbo. The paraphrasing is conducted twice to reduce potential bias, with temperatures of 0.7 and 0.8. See Figure 14 for the prompts used.

In Table 12, we observe that all models demonstrate lower perplexity on the paraphrased versions of the benchmark. While this does not entirely rule out the possibility of whether the models have encountered the benchmark during pretraining, the preference for a rewritten version--which does not exist online and thus could not have been included in the pretraining--implies that the models are not familiar with the original material.

For proprietary models where perplexity examinations are unavailable, we use n-gram accuracy. First, we combine the question and answer parts with a single space for each sample, creating a combined text \(X\). Second, we uniformly sample \(K\) (i.e., 5) starting points within the interval from 2 to \(|X|\). The text from the beginning to each starting point

Figure 9: Comparison of GPT-4Blue and HyperCLOVA XRed using the Direct method in a 5-shot setting.

Figure 10: Comparison of model performance and human accuracy. Model performance is calculated using the Direct method in a 5-shot setting.



serves as the prompt, with the subsequent n-gram used as the prediction target. The prediction results are shown in Figure 11.

GPT-4 and HyperCLOVA X demonstrate low average scores for predicting target n-grams, with 0.004 and 0.001 for N=3 and N=5 in GPT-4, and 0.014 and 0.005 for N=3 and N=5 in HyperCLOVA X, respectively. However, GPT-4 scores above 0.4 in 21 instances for N=3 and 9 instances for N=5. Examining the samples in Tables 13 and 14 reveals that the target is a repetition of the prompt, suggesting the model's guesswork rather than data contamination. HyperCLOVA X exhibits a marginally higher average score than GPT-4, scoring above 0.4 in 69 instances for N=3 and 37 instances for N=5. Tables 15, 16, and 17 present similar findings, with the model occasionally repeating the target verbatim despite its absence from the prompt, potentially indicating leakage. Nonetheless, given the low average scores, we hypothesize that the model might have encountered internet sources reproducing exam segments, rather than contamination of the entire source.

Figure 11: Comparison of model performance and human accuracy. Model performance is calculated using the Direct method in a 5-shot setting.

\begin{table}
\begin{tabular}{c|l l} \hline \hline Subset & High school government and politics \\ Question & Which of the following statements does NOT accurately describe voting behavior in the United States? \\ \hline Subset & high school us history \\  & This question refers to the following information. “I was once a tool of oppression And as green as a sucker could be And monopolies handed together To beat a poor hayesed like me. ’The railroads and old party boxes Together did sweedy agree; Andy they thought there would be little trouble In working a hayesed like me...”—’–The Hayseed’ The song, and the movement that it was connected to, highlight which of the following developments in the broader society in the late 1800s? \\ \hline Subset & professional accounting \\ Question & The FASB makes changes to the Accounting Standards Codification by issuing? \\ \hline Subset & professional law \\  & In which of the following stages does an indigent person not have the Sixth Amendment right to counsel? \\ \hline Subset & us foreign policy \\ Question & Who was the first American president to visit communist China? \\ \hline Subset & miscellaneous \\  & What US president is mentioned by name in the opening theme song of TV’s ‘All in the Family’? \\ \hline \hline \end{tabular}
\end{table}
Table 11: Example questions in MMLU targetted towards American history or culture.

 



## Appendix D License

The KMMLU benchmark is released under a CC-BY-ND license. This license prohibits the distribution of the remixed or transformed version of the dataset. The code and prompts for evaluation are released by an MIT license via LM-Eval-Harness.

## Appendix E Evaluated Models

**Polyglot-Ko**[14]. Introduced by the Polyglot Team of EleutherAI Polyglot-Ko is a comprehensive suite of Korean-centric autoregressive language models featuring models with 1.3, 3.8, 5.8, and 12.8 billion parameters. The models

\begin{table}
\begin{tabular}{l l} \hline \hline Model & HyperCLOVA X \\ \hline Prompt & \(\mathcal{Y|A|Z|X|Y|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|Y|O|O|Y|O|Y|O|Y|O|Y|O|Y|O|O|Y|O|Y|O|Y|O|Y|O|O|Y|O|Y|O|Y|O|O|Y|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|Y|O|O|Y|O|O|Y|O|Y|O|O|Y|O|O|Y|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|Y|O|O|O|Y|O|O|Y|O|O|Y|O|O|O|Y|O|O|Y|O|O|O|Y|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|Y|O|O|O|O|Y|O|O|O|Y|O|O|O|O|Y|O|O|O|O|Y|O|O|O|O|Y|O|O|O|O|Y|O|O|O|O

are pretrained on Korean corpus ranging from 167 to 219 billion tokens.

**Llama-2**Touvron et al. (2023). Llama-2 is a suite of large language models ranging from 7 to 70 billion parameters developed by Meta. The models are pretrained on 2 trillion tokens, and whether Korean is included is not reported. The suite also provides Llama-2-Chat an aligned version for instruction-following and ssafety.

**Yi**Young et al. (2024). The Yi model, developed by 01.AI, is a series of bilingual language models available in two variants: 6B and 34B. It employs an architecture similar to Llama-2 and is pretrained on a multilingual corpus of 3 trillion tokens. Additionally, the model features chat versions tailored for instruction-following.

**Qwen**Bai et al. (2023). Qwen is a suite of bilingual language models developed by Alibaba Cloud, with variants spanning from 1.8 billion to 72 billion parameters. Each model within the series is pretrained on a dataset of 3 trillion tokens. The Qwen also includes specialized chat models designed for following instructions.

**GPT-3.5 & GPT-4**OpenAI (2023). Developed by OpenAI, the GPT series is renowned for exhibiting state-of-the-art performance across various benchmarks and tasks, including exceptional instruction-following capabilities. Specific details regarding the parameter count and the scope of the training data are not open to the public.

**Gemini**Team et al. (2023). Gemini is a series of models developed by Google, encompassing four variants: Nano-1, Nano-2, Pro, and Ultra. In our experiments, we utilize Gemini-Pro. Details regarding the parameter count and the dataset used for training are not disclosed.

**HyperCLOVA X**Yoo et al. (2024). HyperCLOVA X, developed by NAVER, is a bilingual language model proficient in both English and Korean.

## Appendix F Compute Resources

Models with openly available weights were evaluated on an internal cluster comprising 4 NVIDIA A100 HBM2e 80GB PCIe GPUs. We performed 44 direct evaluations on the original KMMLU set and 2 CoT evaluations on the KMMLU-HARD set, totaling approximately 300 A100 GPU hours.

## Appendix G Prompting Format

For evaluation, we use the following promptting formats. For n-shot evaluation, the identical format is repeated.

For the paraphrasing in Section C, we use the foll

Figure 12: Prompt used in our Direct Evaluation.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt}} \hline \hline Model & HyperCLOVA X \\ \hline Prompt & \multicolumn{1}{c}{} \\ Target & \multicolumn{1}{c}{} \\  & \multicolumn{1}{c}{} \\ Generation & \multicolumn{1}{c}{} \\  & \multicolumn{1}{c}{} \\  & \multicolumn{1}{c}{} \\  & \multicolumn{1}{c}{} \\  & \multicolumn{1}{c}{} \\ Accuracy (N=3, N=5) & \multicolumn{1}{c}{} \\ \hline \hline \end{tabular}
\end{table}
Table 17: An example where HyperCLOVA X scores high n-gram accuracy

Figure 13: Prompt used in our CoT Evaluation.

 



## Appendix H More details for CoT Exemplar Creation

We use the zero-shot CoT prompt of Figure 15 to collect the exemplar CoTs for our dataset. We request to use browsing for more accurate explanations if it is available. For GPT-4, we manually input the prompt to the ChatGPT Web interface (chat.openai.com). For HyperCLOVA X, we devise 3-shot demonstrations to generate relevant queries to the NAVER search engine (www.naver.com). Then, we concatenate top-3 search results to generate explanations.

## Appendix I KMMLU-KOR Subset

For more targeted evaluation, alongside KMMLU-HARD, we introduce a second subset, KMMLU-KR. KMMLU-KR is a collection of questions that specifically require Korean knowledge to solve so that users can assess a language model's proficiency in Korea-specific contexts. We initially adopted a keyword filter to collect 2,149 candidate questions. Then, two authors independently reviewed each question, eliminating irrelevant entries and categorizing the pertinent ones into four distinct categories: _Cultural_, which delves into Korean history and societal norms; _Regional_, focusing on the geographical details of Korea; _Legal_, concerning Korea's legal and governmental structures; _Others_, comprising questions that demand knowledge of the Korean context but do not fit into the previous categories. Following the filtering, 1,305 questions remained, constituting the KMMLU-KOR subset. Please refer to Figure 1 for detailed examples of each category. For evaluation results, see Table 19.

## Appendix J Evaluation Results

In this section, we present the results of our evaluation, broken down by category for each model assessed. Tables 22-26 include results using the Direct method. Table 27 presents the results evaluated using the CoT method. Figure 9 presents a comparative performance analysis between the most capable Korean model, HyperCLOVA X, and GPT-4.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
* \\ GPT-4 OpenAI [2023] & undisclosed & API &
- \\ Gemini-Pro Team et al. [2023] & undisclosed & API &
- \\ HyperCLOVA X Yoo et al. [2024] & undisclosed & API &
- \\ \hline \hline \end{tabular}
\end{table}
Table 18: Overview of the 31 LLMs evaluated in this paper.




\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Model** & **Cultural** & **Legal** & **Regional** & **Other** & **Total** \\ \hline  & \multicolumn{4}{c}{_Multilingual Pretrained Models_} \\ \hline Llama-2-7B & 21.51 & 22.77 & 38.46 & 25.53 & 23.26 \\ Llama-2-13B & 29.34 & 27.16 & 50.00 & 28.87 & 28.13 \\ Llama-2-70B & 35.33 & 34.05 & 30.77 & 40.14 & 34.88 \\ Yi-6B & 31.33 & 33.01 & 50.00 & 33.58 & 33.13 \\ Yi-34B & 37.35 & 41.02 & 38.46 & 47.01 & 41.18 \\ Qwen-7B & 10.47 & 10.08 & 26.92 & 21.28 & 11.63 \\ Qwen-14B & 22.16 & 21.00 & 34.62 & 26.76 & 22.04 \\ Qwen-72B & 37.95 & 43.20 & 34.62 & 46.27 & 42.56 \\ \hline  & \multicolumn{4}{c}{_Multilingual Chat Models_} \\ \hline Llama-2-7B-Chat & 23.84 & 26.72 & 42.31 & 28.37 & 26.78 \\ Llama-2-13B-Chat & 27.11 & 24.76 & 46.15 & 29.10 & 25.95 \\ Liama-2-70B-Chat & 27.71 & 30.83 & 53.85 & 29.85 & 30.62 \\ Yi-6B-Chiat & 25.58 & 32.85 & 42.31 & 32.62 & 31.98 \\ Yi-34B-Chiat & 37.95 & 41.02 & 42.31 & 45.52 & 41.09 \\ Qwen-7B-Chiat & 6.36 & 8.97 & 26.92 & 16.78 & 9.81 \\ Qwen-14B-Chiat & 20.36 & 17.56 & 34.62 & 26.76 & 19.34 \\ Qwen-72B-Chiat & 34.10 & 35.27 & 30.77 & 37.58 & 35.13 \\ \hline  & \multicolumn{4}{c}{_Korean Pretrained Models_} \\ \hline Polyglot-Ko-1.3B & 30.81 & 29.00 & 34.62 & 26.24 & 29.00 \\ Polyglot-Ko-3.8B & 25.58 & 25.88 & 26.92 & 21.99 & 25.48 \\ Polyglot-Ko-3.8B & 24.10 & 27.79 & 19.23 & 26.12 & 26.90 \\ Polyglot-Ko-12.8B & 30.12 & 28.03 & 26.92 & 24.63 & 27.85 \\ \hline  & \multicolumn{4}{c}{_Korean Continual Pretrained Models_} \\ \hline Yi-Ko-6B & 34.94 & 36.65 & 30.77 & 44.78 & 37.11 \\ Yi-Ko-34B & 44.19 & 44.70 & 42.31 & 51.77 & 45.14 \\ \hline  & \multicolumn{4}{c}{_Proprietary Models_} \\ \hline GPT-3.5-Turbo & 38.73 & 34.15 & 26.92 & 41.61 & 35.36 \\ Gemini-Pro & 45.18 & 39.68 & 42.31 & 43.28 & 40.83 \\ HyperCLOVA X & **47.90** & 50.65 & 42.31 & 51.41 & 50.00 \\ GPT-4 & 41.57 & **52.31** & **46.15** & **58.21** & **51.21** \\ \hline \hline \end{tabular}
\end{table}
Table 19: Average accuracy(%) calculated using the Direct method in a 5-shot setting on the KMMLU-KOR subset. We report the micro-average accuracy due to the imbalance on each category. The highest-scoring model across the entire table is highlighted in **bold**, and the best model within each category is underlined.





\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Models** & **HUMSS** & **Other** & **Applied Science** & **STEM** \\ \hline Mistral-7B-nv0.3 & 35.48 & 36.24 & 35.88 & 39.40 \\ Mistral-8x7B-v0.1 & 42.01 & 39.14 & 37.48 & 40.62 \\ Mistral-8x22B-vv0.1 & 23.35 & 22.14 & 21.15 & 23.14 \\ Gemma-2-2B & 29.16 & 30.67 & 31.39 & 32.76 \\ Gemma-2-pgb & 43.85 & 47.67 & 46.99 & 48.96 \\ Gemma-2-27B & 49.25 & 49.74 & 49.29 & 51.97 \\ Qwen1.5-0.5B & 23.75 & 17.55 & 15.59 & 17.38 \\ Qwen1.5-1.8B & 26.45 & 23.78 & 20.88 & 22.82 \\ Qwen1.5-4B & 32.32 & 34.66 & 35.12 & 35.36 \\ Qwen1.5-7B & 37.97 & 39.30 & 36.51 & 39.38 \\ Qwen1.5-14B & 45.08 & 44.85 & 42.96 & 46.29 \\ Qwen1.5-32B & 47.98 & 48.90 & 45.34 & 48.73 \\ Qwen1.5-72B & 54.38 & 52.85 & 50.13 & 51.65 \\ Qwen1.5-110B & 60.24 & 58.23 & 54.55 & 57.16 \\ Qwen2-0.5B & 28.51 & 29.63 & 28.70 & 28.77 \\ Qwen2-1.5B & 38.04 & 37.98 & 35.55 & 37.62 \\ Qwen2-7B & 51.44 & 49.29 & 46.46 & 49.08 \\ Qwen2-72B & 67.68 & 63.84 & 63.72 & 65.88 \\ Meta-Llama-3.1-8B & 40.53 & 42.18 & 39.38 & 42.85 \\ Meta-Llama-3.1-70B & 53.81 & 52.43 & 49.31 & 53.84 \\ \hline \hline \end{tabular}
\end{table}
Table 20: 5-shot accuracy using the Direct method for 20 recent LLMs broken down by category.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Models** & **HUMSS** & **Other** & **Applied Science** & **STEM** \\ \hline Mistral-7B-nstruct-v0.3 & 33.52 & 30.74 & 28.85 & 33.52 \\ Mistral-8x7B-Instruct-v0.1 & 41.90 & 39.10 & 38.49 & 42.10 \\ Mistral-8x22B-Instruct-v0.1 & 47.58 & 47.63 & 45.19 & 48.67 \\ Mistral-Large-instruct-2407 & 63.08 & 59.79 & 60.32 & 61.68 \\ Gemma-2-2B-it & 33.52 & 34.77 & 31.58 & 34.56 \\ Gemma-2-9B-it & 47.12 & 48.00 & 43.50 & 47.08 \\ Gemma-2-27B-it & 53.07 & 50.89 & 48.43 & 52.45 \\ Qwen1.5-0.5B-Chat & 25.86 & 24.07 & 26.18 & 25.00 \\ Qwen1.5-1.8B-Chat & 27.59 & 24.52 & 21.67 & 23.26 \\ Qwen1.5-4B-Chat & 32.28 & 33.00 & 32.77 & 33.36 \\ Qwen1.5-7B-Chat & 37.37 & 39.65 & 37.60 & 40.48 \\ Qwen1.5-14B-Chat & 43.90 & 44.05 & 40.84 & 43.92 \\ Qwen1.5-32B-Chat & 47.28 & 48.20 & 43.66 & 47.00 \\ Qwen1.5-72B-Chat & 52.76 & 51.16 & 49.51 & 50.84 \\ Qwen1.5-110B-Chat & 53.25 & 50.49 & 49.74 & 52.55 \\ Qwen2-0.5B-Instruct & 29.37 & 30.98 & 31.03 & 30.78 \\ Qwen2-1.5B-Instruct & 37.74 & 38.55 & 34.51 & 37.46 \\ Qwen2-7B-Instruct & 48.52 & 45.79 & 43.49 & 46.94 \\ Qwen2-72B-Instruct & 65.40 & 62.88 & 62.21 & 64.25 \\ Meta-Llama-3.1-8B-Instrct & 42.02 & 41.86 & 39.57 & 42.98 \\ Meta-Llama-3.1-70B-Instruct & 53.60 & 49.50 & 50.47 & 54.01 \\ Command-R-v01 & 43.20 & 41.27 & 36.47 & 40.08 \\ Command-R-Plus & 48.99 & 48.61 & 45.81 & 47.84 \\ \hline \hline \end{tabular}
\end{table}
Table 21: 5-shot accuracy using the Direct method for 23 recent LLMs, broken down by category.





\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Category** & \multicolumn{4}{c}{Polyglot-Ko} & \multicolumn{2}{c}{Yi-Ko} \\  & 1.3B & 3.8B & 5.8B & 12.8B & 6B & 34B \\ \hline accounting & 30.0 & 32.0 & 32.0 & 30.0 & 38.0 & 46.0 \\ agricultural\_sciences & 27.0 & 30.3 & 30.1 & 32.0 & 32.7 & 39.6 \\ aviation\_engineering\_and\_maintenance & 30.2 & 29.7 & 29.9 & 30.7 & 36.1 & 45.2 \\ biology & 24.0 & 26.7 & 28.6 & 25.3 & 32.1 & 41.7 \\ chemical\_engineering & 25.3 & 27.9 & 24.7 & 24.7 & 36.2 & 49.7 \\ chemistry & 30.3 & 25.2 & 26.0 & 29.2 & 40.8 & 52.33 \\ civil\_engineering & 27.4 & 31.8 & 31.9 & 34.3 & 38.0 & 45.3 \\ computer\_science & 32.1 & 35.9 & 34.8 & 33.9 & 61.5 & 73.9 \\ construction & 33.6 & 31.0 & 31.7 & 32.0 & 34.7 & 38.3 \\ criminal\_law & 26.0 & 29.0 & 29.5 & 28.5 & 31.5 & 37.0 \\ ecology & 28.7 & 29.4 & 31.8 & 32.7 & 45.2 & 57.4 \\ economics & 23.8 & 26.2 & 24.6 & 24.6 & 41.5 & 58.5 \\ education & 23.0 & 20.0 & 24.0 & 25.0 & 53.0 & 71.0 \\ electrical\_engineering & 29.3 & 32.5 & 32.0 & 32.6 & 34.9 & 36.4 \\ electronics\_engineering & 30.5 & 30.0 & 35.2 & 33.3 & 47.1 & 56.6 \\ energy\_management & 28.8 & 26.5 & 24.5 & 26.9 & 30.0 & 38.4 \\ environmental\_science & 26.1 & 32.9 & 27.3 & 30.9 & 33.9 & 40.8 \\ fashion & 27.0 & 29.5 & 29.2 & 29.8 & 46.1 & 50.9 \\ food\_processing & 27.3 & 31.8 & 33.5 & 29.4 & 36.1 & 45.8 \\ gas\_technology\_and\_engineering & 31.9 & 30.9 & 30.2 & 30.9 & 32.5 & 38.5 \\ geomatics & 29.2 & 30.0 & 31.1 & 31.0 & 41.6 & 46.9 \\ health & 26.0 & 32.0 & 27.0 & 25.0 & 52.0 & 73.0 \\ industrial\_engineer & 27.4 & 32.3 & 33.1 & 31.2 & 43.0 & 48.7 \\ information\_technology & 34.2 & 34.1 & 34.0 & 30.8 & 57.1 & 70.9 \\ interior\_architecture\_and\_design & 32.4 & 29.6 & 29.7 & 31.8 & 47.3 & 61.1 \\ korean\_history & 34.0 & 26.0 & 25.0 & 31.0 & 33.0 & 42.0 \\ law & 26.0 & 24.2 & 24.4 & 23.9 & 41.8 & 53.6 \\ machine\_design\_and\_manufacturing & 28.7 & 34.0 & 26.9 & 30.3 & 39.9 & 45.9 \\ management & 27.6 & 27.7 & 27.7 & 28.0 & 43.7 & 61.9 \\ maritime\_engineering & 24.8 & 31.5 & 26.7 & 26.5 & 44.0 & 51.8 \\ marketing & 24.4 & 30.6 & 26.4 & 33.5 & 69.6 & 81.2 \\ materials\_engineering & 30.9 & 30.2 & 30.1 & 26.9 & 39.8 & 51.8 \\ math & 30.0 & 21.3 & 20.0 & 24.7 & 24.0 & 31.3 \\ mechanical\_engineering & 24.2 & 31.5 & 27.1 & 26.9 & 38.0 & 44.1 \\ nondestructive\_testing & 26.4 & 32.1 & 34.2 & 30.3 & 39.0 & 50.6 \\ patent & 29.0 & 23.0 & 22.0 & 31.0 & 32.0 & 34.0 \\ political\_science\_and\_ sociology & 25.7 & 25.7 & 25.7 & 41.7 & 64.7 \\ psychology & 26.5 & 25.9 & 27.7 & 25.9 & 40.1 & 53.8 \\ public\_safety & 28.5 & 30.7 & 31.5 & 31.3 & 32.1 & 40.6 \\ railway\_and\_automotive\_engineering & 23.6 & 29.0 & 28.9 & 26.8 & 34.7 & 39.6 \\ real\_estate & 27.0 & 27.5 & 29.5 & 32.0 & 45.0 & 60.0 \\ refrigerating\_machinery & 27.0 & 28.9 & 29.7 & 28.3 & 30.0 & 37.9 \\ social\_welfare & 25.3 & 28.9 & 30.0 & 28.8 & 44.7 & 58.5 \\ taxation & 29.0 & 27.0 & 23.5 & 26.5 & 36.5 & 43.0 \\ telecommunications\_and\_ wireless\_technology & 28.6 & 33.9 & 34.1 & 32.2 & 52.4 & 60.4 \\ \hline \hline \end{tabular}
\end{table}
Table 22: 5-shot accuracy using the Direct method for Polyglot-Ko, and Yi-Ko broken down by category.





\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Category** & \multicolumn{2}{c}{LLama-2-7B} & LLAMA-2-13B & ILAMA-2-70B \\  & Org. & Chat & Org. & Chat & Org. & Chat \\ \hline accounting & 25.0 & 22.0 & 20.0 & 16.0 & 34.0 & 26.0 \\ agricultural\_sciences & 23.7 & 31.0 & 29.6 & 27.4 & 33.6 & 32.7 \\ aviation\_engineering\_and\_maintenance & 23.7 & 26.8 & 30.3 & 26.8 & 35.9 & 33.0 \\ biology & 23.6 & 26.4 & 28.8 & 25.2 & 33.0 & 28.1 \\ chemical\_engineering & 27.0 & 28.5 & 32.7 & 31.3 & 38.5 & 33.1 \\ chemistry & 26.8 & 26.7 & 30.3 & 27.7 & 41.8 & 32.3 \\ civil\_engineering & 26.9 & 32.1 & 33.8 & 31.1 & 36.4 & 35.4 \\ computer\_science & 24.1 & 28.0 & 47.4 & 41.5 & 67.3 & 58.9 \\ construction & 22.9 & 31.3 & 30.1 & 28.2 & 31.8 & 33.6 \\ criminal\_law & 26.5 & 26.5 & 30.0 & 22.0 & 30.0 & 25.0 \\ ecology & 16.8 & 28.0 & 32.5 & 31.0 & 43.7 & 38.7 \\ economics & 27.7 & 30.8 & 27.7 & 38.5 & 45.4 & 40.0 \\ education & 24.0 & 29.0 & 26.0 & 28.0 & 56.0 & 38.0 \\ electrical\_engineering & 27.4 & 29.4 & 34.0 & 28.0 & 30.8 & 32.3 \\ electronics\_engineering & 33.0 & 32.2 & 38.8 & 31.5 & 47.1 & 39.9 \\ energy\_management & 23.5 & 25.4 & 26.6 & 24.8 & 30.8 & 28.9 \\ environmental\_science & 27.5 & 30.4 & 32.9 & 29.0 & 28.3 & 29.6 \\ fashion & 27.8 & 30.0 & 32.2 & 32.4 & 41.8 & 36.2 \\ food\_processing & 17.4 & 24.3 & 31.1 & 26.6 & 33.9 & 29.9 \\ gas\_technology\_and\_engineering & 22.3 & 28.0 & 29.1 & 26.4 & 31.4 & 29.6 \\ geomatics & 26.9 & 31.0 & 35.4 & 30.5 & 40.2 & 36.9 \\ health & 22.0 & 21.0 & 30.0 & 25.0 & 53.0 & 42.0 \\ industrial\_engineer & 24.5 & 28.9 & 36.5 & 34.3 & 41.9 & 38.6 \\ information\_technology & 27.3 & 29.3 & 44.4 & 37.3 & 62.8 & 52.0 \\ interior\_architecture\_and\_design & 28.3 & 30.2 & 36.0 & 33.0 & 47.8 & 40.8 \\ korean\_history & 26.0 & 21.0 & 25.0 & 25.0 & 32.0 & 23.0 \\ law & 24.4 & 25.5 & 26.5 & 27.6 & 40.8 & 34.9 \\ machine\_design\_and\_manufacturing & 24.0 & 29.2 & 34.1 & 27.9 & 41.8 & 35.0 \\ management & 24.0 & 25.5 & 29.7 & 27.1 & 47.8 & 37.2 \\ maritime\_engineering & 30.0 & 30.3 & 32.8 & 29.7 & 40.3 & 34.7 \\ marketing & 24.0 & 25.1 & 38.7 & 37.2 & 70.7 & 57.4 \\ materials\_engineering & 21.2 & 28.5 & 29.0 & 26.2 & 40.4 & 30.8 \\ math & 25.0 & 28.3 & 24.3 & 26.7 & 27.0 & 23.7 \\ mechanical\_engineering & 25.3 & 29.4 & 34.6 & 28.0 & 31.0 & 30.5 \\ nondestructive\_testing & 24.8 & 29.9 & 34.2 & 25.8 & 41.5 & 32.1 \\ patent & 25.0 & 24.0 & 26.0 & 26.0 & 33.0 & 25.0 \\ political\_science\_and\_ sociology & 23.7 & 27.7 & 25.3 & 30.7 & 47.3 & 36.0 \\ psychology & 24.7 & 24.9 & 25.2 & 23.5 & 39.1 & 28.0 \\ public\_safety & 28.5 & 30.4 & 32.6 & 31.0 & 33.0 & 34.0 \\ railway\_and\_automotive\_engineering & 22.7 & 26.7 & 31.2 & 27.3 & 32.4 & 30.0 \\ real\_state & 23.5 & 24.5 & 24.5 & 25.0 & 32.0 & 26.5 \\ refrigerating\_machinery & 24.2 & 26.3 & 28.7 & 27.8 & 30.1 & 30.8 \\ social\_weflare & 26.7 & 28.8 & 31.9 & 27.6 & 47.8 & 35.0 \\ taxation & 23.0 & 24.5 & 21.5 & 24.5 & 33.0 & 31.0 \\ telecommunications\_and\_wireless\_technology & 27.9 & 29.5 & 44.4 & 35.1 & 54.2 & 44.0 \\ \hline \hline \end{tabular}
\end{table}
Table 23: 5-shot accuracy using the Direct method for Llama-2 (original and chat versions) broken down by category.





\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Category** & \multicolumn{2}{c}{Y1-6B} & \multicolumn{2}{c}{Y1-34B} \\  & Org. & Chat & Org. & Chat \\ \hline accounting & 29.0 & 30.0 & 46.0 & 45.0 \\ agricultural\_sciences & 32.7 & 29.5 & 36.0 & 34.7 \\ aviation\_engineering\_and\_maintenance & 31.9 & 30.9 & 36.9 & 34.8 \\ biology & 28.9 & 29.4 & 32.5 & 30.9 \\ chemical\_engineering & 31.8 & 31.5 & 40.8 & 40.7 \\ chemistry & 36.7 & 35.0 & 47.5 & 42.3 \\ civil\_engineering & 32.8 & 33.1 & 40.9 & 36.9 \\ computer\_science & 54.0 & 56.8 & 72.1 & 72.0 \\ construction & 30.9 & 30.9 & 34.7 & 30.4 \\ criminal\_law & 34.5 & 36.5 & 39.0 & 37.5 \\ ecology & 34.3 & 35.1 & 46.7 & 44.5 \\ economics & 36.9 & 36.9 & 43.1 & 48.5 \\ education & 40.0 & 44.0 & 58.0 & 62.0 \\ electrical\_engineering & 33.0 & 31.5 & 33.3 & 28.4 \\ electronics\_engineering & 41.9 & 43.2 & 50.4 & 50.1 \\ energy\_management & 28.8 & 30.5 & 33.8 & 32.7 \\ environmental\_science & 31.5 & 29.5 & 34.1 & 29.5 \\ fashion & 33.8 & 35.0 & 43.3 & 40.9 \\ food\_processing & 29.6 & 31.6 & 38.1 & 36.6 \\ gas\_technology\_and\_engineering & 27.7 & 27.5 & 30.8 & 28.5 \\ geomatics & 34.9 & 36.6 & 41.6 & 38.9 \\ health & 40.0 & 44.0 & 59.0 & 52.0 \\ industrial\_engineer & 36.3 & 35.9 & 43.1 & 41.1 \\ information\_technology & 51.9 & 51.3 & 69.0 & 66.5 \\ interior\_architecture\_and\_design & 38.5 & 39.5 & 48.3 & 49.0 \\ korean\_history & 30.0 & 24.0 & 34.0 & 36.0 \\ law & 30.3 & 31.3 & 42.9 & 42.0 \\ machine\_design\_and\_manufacturing & 33.2 & 33.6 & 40.6 & 37.9 \\ management & 35.5 & 38.0 & 57.7 & 54.4 \\ maritime\_engineering & 36.7 & 39.2 & 44.2 & 43.0 \\ marketing & 57.4 & 57.7 & 74.6 & 74.9 \\ materials\_engineering & 30.2 & 30.1 & 39.4 & 36.9 \\ math & 26.7 & 29.0 & 29.7 & 31.0 \\ mechanical\_engineering & 29.9 & 28.6 & 35.5 & 30.0 \\ nondestructive\_testing & 33.0 & 34.2 & 42.6 & 39.0 \\ patent & 33.0 & 31.0 & 38.0 & 40.0 \\ political\_science\_and\_ sociology & 36.0 & 37.0 & 55.0 & 51.7 \\ psychology & 28.3 & 29.9 & 44.1 & 41.4 \\ public\_saferty & 30.8 & 29.4 & 34.1 & 30.2 \\ railway\_and\_automotive\_engineering & 33.0 & 32.0 & 33.7 & 29.4 \\ real\_estate & 37.0 & 37.5 & 44.5 & 44.0 \\ refrigerating\_ machinery & 29.0 & 29.4 & 33.0 & 29.9 \\ social\_weflare & 37.0 & 37.2 & 55.1 & 53.9 \\ taxation & 30.5 & 33.5 & 42.5 & 44.0 \\ telecommunications\_and\_wireless\_technology & 41.9 & 41.5 & 55.3 & 51.7 \\ \hline \hline \end{tabular}
\end{table}
Table 24: 5-shot accuracy using the Direct method for Y1 (original and chat versions) broken down by category.





\begin{table}
\begin{tabular}{l c r l l l l} \hline \hline
**Category** & \multicolumn{2}{c}{Qwen-7B} & \multicolumn{2}{c}{Qwen-14B} & \multicolumn{2}{c}{Qwen-72B} \\  & Org. & Chat & Org. & Chat & Org. & Chat \\ \hline accounting & 9.0 & 9.0 & 25.0 & 15.0 & 15.0 & 46.0 \\ agricultural\_sciences & 28.8 & 34.3 & 38.5 & 24.7 & 34.1 & 40.4 \\ aviation\_engineering\_and\_maintenance & 23.3 & 33.6 & 49.2 & 19.9 & 31.9 & 48.7 \\ biology & 15.2 & 29.0 & 40.5 & 15.4 & 26.5 & 39.7 \\ chemical\_engineering & 19.0 & 32.7 & 50.8 & 17.9 & 28.3 & 45.2 \\ chemistry & 24.3 & 44.2 & 54.3 & 21.2 & 37.7 & 50.7 \\ civil\_engineering & 17.6 & 31.3 & 46.5 & 17.5 & 31.7 & 46.7 \\ computer\_science & 32.0 & 54.2 & 75.7 & 30.6 & 52.8 & 76.4 \\ construction & 21.7 & 32.1 & 38.0 & 12.4 & 20.0 & 26.0 \\ criminal\_law & 4.5 & 12.5 & 40.0 & 4.0 & 9.0 & 36.5 \\ ecology & 34.2 & 46.2 & 52.4 & 35.3 & 45.7 & 53.1 \\ economics & 5.4 & 10.8 & 60.0 & 3.8 & 9.2 & 54.6 \\ education & 10.0 & 29.0 & 71.0 & 7.0 & 29.0 & 74.0 \\ electrical\_engineering & 22.3 & 27.7 & 34.8 & 18.8 & 26.9 & 35.0 \\ electronics\_engineering & 14.2 & 30.3 & 59.3 & 16.1 & 32.1 & 62.9 \\ energy\_management & 26.4 & 32.3 & 40.3 & 22.1 & 29.8 & 38.2 \\ environmental\_science & 26.4 & 32.6 & 38.0 & 28.0 & 34.4 & 41.4 \\ fashion & 32.6 & 42.8 & 49.6 & 29.6 & 41.6 & 48.7 \\ food\_processing & 8.3 & 19.0 & 45.8 & 5.7 & 12.9 & 36.8 \\ gas\_technology\_and\_engineering & 16.7 & 26.0 & 39.9 & 12.0 & 21.4 & 31.2 \\ geomatics & 22.8 & 31.8 & 43.8 & 19.5 & 29.4 & 41.8 \\ health & 9.0 & 30.0 & 71.0 & 13.0 & 28.0 & 61.0 \\ industrial\_engineer & 23.6 & 42.3 & 49.0 & 22.1 & 41.7 & 47.1 \\ information\_technology & 38.9 & 56.5 & 74.2 & 24.2 & 42.1 & 63.5 \\ interior\_architecture\_and\_design & 19.5 & 37.3 & 58.8 & 17.2 & 34.3 & 58.6 \\ korean\_history & 2.0 & 9.0 & 37.0 & 2.0 & 10.0 & 30.0 \\ law & 6.0 & 15.6 & 50.2 & 6.9 & 14.0 & 45.6 \\ machine\_design\_and\_manufacturing & 24.4 & 37.9 & 51.0 & 23.0 & 33.8 & 48.4 \\ management & 8.9 & 23.7 & 64.4 & 8.1 & 23.1 & 58.7 \\ maritime\_engineering & 21.3 & 40.8 & 49.8 & 18.0 & 31.8 & 43.3 \\ marketing & 37.8 & 59.7 & 85.1 & 37.1 & 60.2 & 81.9 \\ materials\_engineering & 15.1 & 29.0 & 50.2 & 7.2 & 20.9 & 37.9 \\ math & 18.7 & 26.7 & 36.7 & 20.3 & 22.7 & 28.7 \\ mechanical\_engineering & 12.8 & 26.1 & 41.5 & 14.5 & 25.4 & 46.4 \\ nondestructive\_testing & 27.2 & 40.9 & 48.4 & 26.4 & 38.7 & 48.5 \\ patent & 7.0 & 16.0 & 39.0 & 4.0 & 11.0 & 33.0 \\ political\_science\_and\_ sociology & 11.7 & 30.7 & 62.0 & 13.3 & 27.3 & 56.7 \\ psychology & 18.4 & 31.1 & 51.5 & 15.2 & 30.1 & 45.4 \\ public\_saafety & 7.9 & 14.1 & 40.3 & 7.5 & 16.0 & 41.0 \\ railway\_and\_automotive\_engineering & 20.5 & 31.7 & 40.1 & 22.2 & 31.6 & 39.2 \\ real\_estate & 2.0 & 7.5 & 53.0 & 3.5 & 8.5 & 45.0 \\ refrigerating\_machinery & 18.9 & 29.1 & 39.4 & 18.0 & 27.6 & 37.2 \\ social\_welfare & 25.0 & 41.0 & 64.7 & 22.0 & 38.2 & 60.1 \\ taxation & 3.0 & 7.5 & 42.5 & 4.0 & 7.5 & 32.0 \\ telecommunications\_and\_wiireless\_technology & 39.1 & 50.0 & 64.2 & 36.6 & 50.7 & 64.4 \\ \hline \hline \end{tabular}
\end{table}
Table 25: 5-shot accuracy using the Direct method for Qwen (original and chat versions) broken down by category.





\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Category** & Gemini-Pro & HCX. & GPT-3.5-Turbo & GPT-4 \\ \hline accounting & 44.0 & 46.0 & 42.0 & 71.0 \\ agricultural\_sciences & 42.5 & 42.7 & 34.1 & 50.2 \\ aviation\_engineering\_and\_ maintenance & 53.0 & 49.0 & 43.5 & 63.9 \\ biology & 46.5 & 47.9 & 35.0 & 51.3 \\ chemical\_engineering & 51.9 & 47.9 & 42.9 & 61.3 \\ chemistry & 50.2 & 48.8 & 45.0 & 64.8 \\ civil\_engineering & 47.6 & 45.1 & 41.2 & 53.9 \\ computer\_science & 75.0 & 78.5 & 66.1 & 87.7 \\ construction & 37.6 & 41.9 & 34.9 & 46.7 \\ criminal\_law & 39.0 & 48.5 & 32.5 & 50.5 \\ ecology & 52.6 & 57.3 & 47.0 & 59.2 \\ economics & 53.1 & 65.4 & 40.8 & 67.7 \\ education & 58.0 & 72.0 & 40.0 & 84.0 \\ electrical\_engineering & 39.1 & 35.3 & 34.8 & 43.2 \\ electronics\_engineering & 60.2 & 59.8 & 52.1 & 69.9 \\ energy\_management & 38.1 & 37.6 & 33.9 & 43.9 \\ environmental\_science & 38.0 & 36.3 & 34.8 & 44.4 \\ fashion & 53.0 & 57.2 & 46.6 & 61.7 \\ food\_processing & 50.1 & 50.3 & 39.6 & 57.4 \\ gas\_technology\_and\_engineering & 42.0 & 42.3 & 34.5 & 49.0 \\ geomatics & 41.7 & 49.4 & 41.8 & 50.9 \\ health & 65.0 & 72.0 & 50.0 & 71.0 \\ industrial\_engineer & 50.7 & 50.2 & 43.3 & 58.1 \\ information\_technology & 72.3 & 73.1 & 66.3 & 83.7 \\ interior\_architecture\_and\_design & 63.5 & 69.1 & 51.0 & 69.8 \\ korean\_history & 41.0 & 42.0 & 32.0 & 35.0 \\ law & 48.5 & 58.7 & 40.2 & 58.6 \\ machine\_design\_and\_manufacturing & 54.4 & 50.8 & 43.9 & 64.9 \\ management & 59.7 & 64.3 & 51.2 & 74.1 \\ maritime\_engineering & 51.2 & 54.3 & 45.2 & 60.8 \\ marketing & 81.0 & 83.1 & 71.1 & 89.3 \\ materials\_engineering & 53.8 & 52.1 & 43.5 & 66.0 \\ math & 26.7 & 26.7 & 30.3 & 31.0 \\ mechanical\_engineering & 48.7 & 46.3 & 38.9 & 57.3 \\ nondestructive\_testing & 52.9 & 50.6 & 42.8 & 59.9 \\ patent & 37.0 & 52.0 & 34.0 & 43.0 \\ political\_science\_and\_ sociology & 57.7 & 66.7 & 47.7 & 74.0 \\ psychology & 47.0 & 58.7 & 37.0 & 61.3 \\ public\_saferty & 41.3 & 41.0 & 36.5 & 51.5 \\ railway\_and\_automotive\_engineering & 42.8 & 41.2 & 34.7 & 51.7 \\ real\_estate & 45.0 & 53.0 & 37.0 & 56.5 \\ refrigerating\_machinery & 40.7 & 40.0 & 33.9 & 48.1 \\ social\_welfare & 60.6 & 61.6 & 49.6 & 76.4 \\ taxation & 40.0 & 48.0 & 33.0 & 48.0 \\ telecommunications\_and\_wireless\_technology & 63.7 & 63.0 & 54.8 & 74.9 \\ \hline \hline \end{tabular}
\end{table}
Table 26: 5-shot accuracy using the Direct method for Gemini-Pro, GPT-3.5-Turbo, GPT-4 and HyperCLOVA X broken down by category.





\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Category** & Qwen-72B-Chat & HCX. & GPT-3.5-Turbo & GPT-4 \\ \hline accounting & 21.7 & 17.4 & 19.6 & 26.1 \\ agricultural\_sciences & 13.0 & 14.0 & 15.0 & 13.0 \\ aviation\_engineering\_and\_maintenance & 21.0 & 24.0 & 26.0 & 38.0 \\ biology & 21.0 & 24.0 & 15.0 & 14.0 \\ chemical\_engineering & 17.0 & 31.0 & 26.0 & 43.0 \\ chemistry & 22.0 & 30.0 & 29.0 & 44.0 \\ civil\_engineering & 17.0 & 25.0 & 20.0 & 16.0 \\ computer\_science & 25.0 & 36.0 & 18.0 & 25.0 \\ construction & 26.0 & 28.0 & 18.0 & 24.0 \\ criminal\_law & 9.0 & 24.0 & 9.0 & 8.0 \\ ecology & 12.0 & 24.0 & 16.0 & 11.0 \\ economics & 23.8 & 33.3 & 26.2 & 28.6 \\ education & 17.4 & 26.1 & 0.0 & 26.1 \\ electrical\_engineering & 11.0 & 24.0 & 20.0 & 30.0 \\ electronics\_engineering & 23.0 & 20.0 & 34.0 & 48.0 \\ energy\_management & 18.0 & 15.0 & 25.0 & 26.0 \\ environmental\_science & 16.0 & 22.0 & 17.0 & 27.0 \\ fashion & 20.0 & 29.0 & 24.0 & 16.0 \\ food\_processing & 17.0 & 24.0 & 21.0 & 28.0 \\ gas\_technology\_and\_engineering & 19.0 & 29.0 & 25.0 & 31.0 \\ geomatics & 18.0 & 24.0 & 20.0 & 24.0 \\ health & 8.7 & 26.1 & 26.1 & 21.7 \\ industrial\_engineer & 13.0 & 27.0 & 19.0 & 22.0 \\ information\_technology & 28.0 & 33.0 & 41.0 & 46.0 \\ interior\_architecture\_and\_design & 21.0 & 37.0 & 29.0 & 24.0 \\ korean\_history & 11.4 & 47.7 & 18.2 & 9.1 \\ law & 13.0 & 35.0 & 11.0 & 17.0 \\ machine\_design\_and\_manufacturing & 19.0 & 32.0 & 23.0 & 32.0 \\ management & 26.0 & 24.0 & 20.0 & 23.0 \\ maritime\_engineering & 21.0 & 27.0 & 19.0 & 21.0 \\ marketing & 29.0 & 18.0 & 17.0 & 18.0 \\ materials\_engineering & 21.0 & 24.0 & 20.0 & 24.0 \\ math & 18.0 & 32.0 & 31.0 & 51.0 \\ mechanical\_engineering & 17.0 & 25.0 & 20.0 & 36.0 \\ nondestructive\_testing & 19.0 & 23.0 & 27.0 & 24.0 \\ patent & 18.0 & 23.5 & 23.5 & 11.8 \\ political\_science\_and\_ sociology & 24.4 & 27.8 & 4.4 & 14.4 \\ psychology & 16.0 & 36.0 & 14.0 & 9.0 \\ public\_safety & 21.0 & 30.0 & 13.0 & 12.0 \\ railway\_and\_automotive\_engineering & 12.0 & 25.0 & 19.0 & 29.0 \\ real\_estate & 10.1 & 25.8 & 10.1 & 14.6 \\ refrigerating\_ machinery & 18.0 & 26.0 & 26.0 & 38.0 \\ social\_welfare & 13.0 & 35.0 & 36.0 & 51.0 \\ taxation & 5.2 & 26.0 & 10.4 & 4.2 \\ telecommunications\_and\_ wireless\_technology & 25.0 & 30.0 & 30.0 & 38.0 \\ \hline \hline \end{tabular}
\end{table}
Table 27: 5-shot accuracy using the CoT method for Qwen-72B-Chat, GPT-3.5-Turbo, GPT-4 and HyperCLOVA X broken down by category.



You are a problem rewriter. Your task is to paraphrase the problem and the answer presented below.

Please follow the instructions below:

1. Please paraphrase the problem by rewording it with new expressions and sentence structures.
2. Please do not change the essence of the problem and the answer.
3. Please make sure not to deviate too much from the original content, and maintain the style as much as possible.
4. Please write in coherent Korean. It should sound natural to a native Korean speaker.

Please write "Rewritten Question: <question>" to output your rewritten question without any additional information, and write "Rewritten Answer: <answer>" to output your rewritten answer without any additional information.

There is an example for your reference:

Original Question: (?)-?+-? 0x [1] [