Venue: NAACL 2025

# Open Ko-LLM Leaderboard2:

Bridging Foundational and Practical Evaluation for Korean LLMs

\({}^{1}\)Hyeonwoo Kim, \({}^{2}\)Dahyun Kim , \({}^{1}\)Jihoo Kim

\({}^{1}\)Sukyung Lee, \({}^{3}\)Yungi Kim, \({}^{4}\)Chanjun Park\({}^{\dagger}\)

###### Abstract

The Open Ko-LLM Leaderboard has been instrumental in benchmarking Korean Large Language Models (LLMs), yet it has certain limitations. Notably, the disconnect between quantitative improvements on the overly academic leaderboard benchmarks and the qualitative impact of the models should be addressed. Furthermore, the benchmark suite is largely composed of translated versions of their English counterparts, which may not fully capture the intricacies of the Korean language. To address these issues, we propose Open Ko-LLM Leaderboard2, an improved version of the earlier Open Ko-LLM Leaderboard. The original benchmarks are entirely replaced with new tasks that are more closely aligned with real-world capabilities. Additionally, four new native Korean benchmarks are introduced to better reflect the distinct characteristics of the Korean language. Through these refinements, Open Ko-LLM Leaderboard2 seeks to provide a more meaningful evaluation for advancing Korean LLMs.

+
Footnote †: dagger}\) Corresponding Author

## 1 Introduction

The Open Ko-LLM Leaderboard was originally established as a critical evaluation platform to benchmark Korean-specific Large Language Models (LLMs) Park et al. (2024); Park and Kim (2024). Its motivation stemmed from the growing need to adapt existing English-centric benchmarks to Korean, thereby fostering the development of language models that can effectively handle the complexities of Korean syntax and semantics. However, the leaderboard has faced significant limitations over time.

For instance, as improvements in benchmark scores no longer translated to real-world advancements due to the overly academic nature of the benchmark suite, submission rates decreased as the leaderboard results were not as meaningful as before. The benchmark suite need tasks that correlate more with real-world performance. Further, the leaderboard's tasks, primarily configured by translating English counterparts, do not sufficiently capture the nuances of the Korean language. In fact, although the leaderboard was designed for Korean LLMs, only one of the five benchmarks, Ko-CommonGen v2, was specifically tailored for Korean, highlighting a gap in its linguistic specificity.

To address these challenges, we propose the Open Ko-LLM Leaderboard2. This next-generation framework replaces the previous benchmarks with a suite of tasks focusing on Korean linguistic nuances and real-world applications. Notably, the introduction of KorNAT benchmarks Lee et al. (2024) and practical, real-world evaluations like Ko-IFEval Zhou et al. (2023) and Ko-GPQA Rein et al. (2023) ensures the leaderboard's continued relevance. Furthermore, the shift toward fine-tuned models aligns with industry trends, enabling a more meaningful assessment of task-specific performance in Korean LLMs Peng et al. (2024); Guo et al. (2023).

## 2 Open Ko-LLM Leaderboard Season 1

The Open Ko-LLM Leaderboard (Season 1) Park et al. (2024); Park and Kim (2024) was established to provide a comprehensive evaluation framework for Korean-specific Large Language Models (LLMs). Its development was driven by two primary motivations: (i) ensuring alignment with the English Open LLM Leaderboard to facilitate consistent and comparable evaluations across global and Korean LLMs, and (ii) utilizing private test sets to prevent data contamination and ensure rigorous evaluation across a variety of models.

The evaluation relied on the Ko-H5 benchmark, which consisted of five tasks: Ko-ARC Clark

et al., 2018), Ko-HellaSwag Zellers et al. (2019), Ko-MMLU Hendrycks et al. (2020), Ko-TruthfulQA Lin et al. (2021), and Ko-CommonGen v2 Seo et al. (2024). While these tasks provided a foundational assessment of Korean LLMs, four of the five benchmarks were direct translations from English datasets, limiting their linguistic specificity. Only Ko-CommonGen v2 was developed with a focus on Korean, underscoring the need for more Korean-centric benchmarks in future iterations.

## 3 Open Ko-LLM Leaderboard2

### Task Overview

The Open Ko-LLM Leaderboard2 introduces a comprehensive overhaul of its evaluation framework by replacing all previous benchmarks with nine newly designed tasks. These tasks assess a wide range of linguistic and practical capabilities essential for testing Korean LLMs in both academic and real-world settings.

The newly added benchmarks are as follows. _Ko-GPQA (Diamond)_Rein et al. (2023), a general-purpose question-answering task that evaluates deep reasoning in the Korean context. _Ko-WinoGrande_Sakaguchi et al. (2021) focuses on commonsense reasoning by challenging models to resolve ambiguities in everyday Korean scenarios. _Ko-GGSM8K_Cobbe et al. (2021) assesses mathematical reasoning, requiring models to solve complex arithmetic and word problems. _Ko-EQBench_Paech (2023) tests emotional intelligence by evaluating the model's ability to generate contextually appropriate responses in emotionally charged conversations. _Ko-IFEval_Zhou et al. (2023) examines instruction-following skills, gauging how well models can interpret and execute complex Korean instructions. _KorNAT-Knowledge_Lee et al. (2024), a newly introduced benchmark, tests factual recall and application in Korean-specific contexts. _ForNAT-Social-Value_Lee et al. (2024) evaluates models on their understanding of social norms and values that are unique to Korean culture. _Ko-Harmlessness_Lee et al. (2024) measures the model's capacity to produce safe and non-toxic responses in sensitive scenarios, while _Ko-Helpfulness_Lee et al. (2024) focuses on the model's ability to provide relevant and practical information across a variety of real-world situations.

### Task Motivation

The selection of the newly added benchmarks was guided by considerations of cost-efficiency, task diversity, and practical applicability, resulting in a comprehensive yet scalable evaluation framework for Korean LLMs.

First, cost-efficiency was prioritized by adopting GPT-free automated evaluation methods, which significantly reduced costs. The dataset sizes were optimized to balance evaluation depth and computational efficiency, minimizing time and resource requirements while maintaining reliability. This approach ensures a practical and accessible evaluation process.

Second, task diversity was central to the benchmark design, covering both general LLM capabilities, such as reasoning (_Ko-WinoGrande_, _Ko-GPQA_, _Ko-GSM8K_), instruction-following (_Koe-IFEval_), and emotional intelligence (_Ko-EQ-Bench_), and Korea-specific elements like cultural knowledge (_KorNAT-Knowledge_) and social values (_KorNAT-Social-Value_). Furthermore, tasks on harmlessness (_Ko-Harmlessness_) and helpfulness (_Ko-Helpfulness_) ensure safe and practical results in real-world scenarios.

Lastly, practical considerations shaped the selection of the benchmark. The evaluation framework was inspired by the Open LLM Leaderboard, ensuring consistency with established evaluation standards. The task configurations were calibrated to match the submission volumes, guaranteeing scalability and feasibility.

Overall, the chosen benchmarks achieve a thoughtful balance of evaluation rigor, efficiency, and relevance, providing a reliable platform to assess the diverse capabilities of Korean LLMs.

### Dataset Sizes

Each of the nine benchmarks in the Open Ko-LLM Leaderboard2 features datasets of varying sizes to reflect the complexity and scope of the tasks. Table 1 provides a summary of the dataset sizes for each benchmark.

### Curation Process

The nine benchmarks were curated using two distinct approaches. Five of the tasks--_Ko-GPQA (Diamond)_, _Ko-WinoGrande_, _Ko-GSM8K_, _Ko-EqBench_, and _Ko-IFEval_--were adapted from existing English benchmarks Park et al. (2024). These datasets were professionally translated and then 

rigorously reviewed and modified to align with Korean language and cultural nuances. This process involved a thorough human correction phase to ensure that the benchmarks accurately reflected the Korean context.

The remaining four tasks--_KorNAT-Knowledge_, _KorNAT-Social-Value_, _Ko-Harmlessness_, and _Ko-Helpfulness_--were developed entirely from scratch using _native_ Korean corpora. These benchmarks were designed by domain experts to address specific challenges in Korean LLM evaluation, focusing on areas such as factual knowledge, social norms, safety, and utility in real-world situations. The creation of these benchmarks ensures that the leaderboard not only reflects the technical capabilities of models but also their cultural and contextual understanding of Korean language and society.

All datasets in the Open Ko-LLM Leaderboard2 are kept fully private, following the precedent set by the Open Ko-LLM Leaderboard Season 1. This ensures the integrity of the evaluation process by preventing data leakage and guaranteeing a fair and unbiased assessment of model performance.

### Task Evaluation Methodology

The evaluation methodology for each of the nine tasks in the Open Ko-LLM Leaderboard2 is tailored to the nature of the benchmark and the specific capabilities being tested.

For _Ko-GPQA (Diamond)_, _Ko-WinoGrande_, _KorNAT-Knowledge_, _KorNAT-Social-Value_, _Ko-Harmlessness_, and _Ko-Helpfulness_, the evaluation is based on a multiple-choice format. These tasks are evaluated using accuracy metrics, with

\begin{table}
\begin{tabular}{l l} \hline \hline
**Task** & **Dataset Size** \\ \hline Ko-GPQA (Diamond) & 198 samples \\ Ko-WinoGrande & 1,267 samples \\ Ko-GSM8K & 1,319 samples \\ Ko-EQ-Bench & 171 samples \\ Ko-IFEval & 494 samples \\ KorNAT-Knowledge & 6,008 samples \\ KorNAT-Social-Value & 4,000 samples \\ Ko-Harmlessness & 10,000 samples \\ Ko-Helpfulness & 2,000 samples \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dataset sizes for each task in the Open Ko-LLM Leaderboard2. The ”Diamond” in Ko-GPQA (Diamond) represents the subset of the most challenging questions.

Figure 1: Screenshot of the Open Ko-LLM Leaderboard interface showing the current rankings of models evaluated in Season 2. The interface displays model names, overall performance scores, and task-specific results. Users can view detailed evaluation metrics for each model, enabling comparisons based on both quantitative and qualitative performance. This transparent interface encourages healthy competition, fosters continuous improvement, and provides a real-time overview of Korean LLM development progress.

 

_Ko-GPQA_, _KorNAT-Knowledge_, _Ko-Harmlessness_, and _Ko-Helefulness_ assessed using normalized accuracy (acc\({}_{\text{norm}}\)), while _KorNAT-Social-Value_ employs the A-SVA metric specific to social value assessments.

In contrast, _Ko-GSM8K_, _Ko-EQ-Bench_, and _Ko-IFEval_ use generation-based evaluation. _Ko-GSM8K_ focuses on strict exact-match for mathematical reasoning, and _Ko-EQ-Bench_ uses a task-specific emotional intelligence scoring system (eqbench). _Ko-IFEval_ evaluates the model's ability to follow instructions using prompt-level and instruction-level strict accuracy metrics. These tasks explicitly evaluate the generated output of the model, which is more aligned with actual usage scenarios.

The number of few-shot examples varies by task, with tasks such as _Ko-WinoGrande_ and _Ko-GSM8K_ using 5-shot setups, while others like _Ko-GPQA_ and _Ko-IFEval_ use a 0-shot configuration.

The number of few-shot examples varies by task and is determined based on the configurations proposed by the original benchmark authors and widely adopted settings. These configurations were chosen deliberately by the authors for specific reasons, making them meaningful for evaluating the model's capabilities. For instance, tasks like _Ko-WinoGrande_ and _Ko-GSM8K_ use a 5-shot setup to provide the model with minimal but sufficient context for complex reasoning, while others, such as _Ko-GPQA_ and _Ko-IFEval_, employ a 0-shot configuration to directly test the model's ability to generalize without prior examples. Notably, for _Ko-EQ-Bench_, the original paper explicitly states that zero-shot was used to minimize the biasing effect, ensuring a fair and unbiased assessment of emotional intelligence. By adhering to these few-shot configurations, the evaluation remains aligned with the intentions of the benchmark designers and facilitates meaningful comparisons across models.

### Infrastructure and Platform

The infrastructure for the Open Ko-LLM Leaderboard2 has been significantly upgraded to accommodate the increased complexity and scale of the new benchmarks. The system now utilizes both H100 and A100 GPUs, ensuring faster and more efficient evaluations to meet the demands of larger and more complex tasks. The leaderboard operates on the Hugging Face platform Jain (2022), just like in Season 1, providing a user-friendly and familiar environment for participants. By maintaining the same interface and submission process as the original leaderboard, users can seamlessly transition to the new version without additional learning curves, while benefiting from the enhanced infrastructure. This consistency ensures broad accessibility and fosters greater community participation, supporting ongoing innovation in Korean LLM development.

Figure 4: Correlation matrices for pre-trained models (left) and fine-tuned models (right) between Season 1 and Season 2 scores.

\begin{table}
\begin{tabular}{l c c c} \hline  & Seasonal & Season2 (Legg) & Season2 (Generation) \\ \hline Session1 & 1.00 & 0.78 & 0.36 \\ Session2 (Logit) & 0.78 & 1.00 & 0.33 \\ Session2 (Generated) & 0.36 & 0.33 & 1.00 \\ \end{tabular}
\end{table}
Table 2: Correlation between Season 1 tasks and logit-based or generation-based Season 2 tasks.

Figure 3: Example model answers to the same questions from one of top-ranking AI models from Season 1 (left) and Season 2 (right).

 



### Leaderboard Interface Overview

The Open Ko-LLM Leaderboard interface, as shown in Figure 1, provides users with a clear and transparent way to track model rankings and their performance across multiple benchmarks. Season 2's updated evaluation system offers real-time results for both pre-trained and fine-tuned models, encouraging developers to continuously monitor and improve their models. By providing detailed task-specific metrics, the leaderboard fosters competition while ensuring that the evaluations remain practical and aligned with real-world applications.

## 4 Empirical Analysis

### Initial Peak and Slow Decline in Submission Trends

The submission trends from Season 1 highlight the evolving interest in Korean language model evaluations, providing crucial motivation for Season 2. Figure 2 shows a strong initial response, peaking in November and December 2023, with a steady decline starting in January 2024, dropping to 123 submissions by July 2024. This decline is linked to dissatisfaction with the gap between leaderboard scores and real-world performance, as well as limitations in evaluation metrics. The community's engagement waned as models optimized for benchmarks failed to demonstrate practical utility.

These trends emphasize the necessity of implementing more relevant benchmarks and qualitative metrics in Season 2, focusing on real-world applications and broader model capabilities.

### Correlation with Real-World Usage

The logit-based academic evaluation methods in Season 1 are not well-suited to reflect the real-world usability of the models. In contrast, Season 2 aims to better capture the usability of the models by making sure that high-ranking models in Season 2 also work well in practice.

In Figure 3, models answers to questions are illustrated for high-ranking models in the Season 1 and 2 leaderboards. The answers on the left show awkward phrases with mixed symbols and inconsistent language, despite being generated from a high-ranking model in the Season 1 leaderboard. Comparatively, the responses on the right, which is from a top-ranking model in Season 2, feature coherent and natural phrases.

### Correlation Between Season 1 and Season 2 Evaluations

Season 2 is different from Season 1.In Figure 4, we show the correlation between the model scores between Season 1 and 2. The correlation are calculated among pre-trained and fine-tuned models separately.

For pre-trained models, a relatively low correlation coefficient of 0.48 was observed between the two seasons. This suggests that the newly configured benchmarks that aim to align more closely to real-world scenarios are different from the mostly academic evaluation methods used in Season 1. Furthermore, fine-tuned models exhibited a slightly higher but still low correlation of 0.65 between the two seasons. This also reinforces the notion that Season 2 benchmarks are indeed different from

Figure 5: Correlation between the nine new tasks in the Season 2 Open Ko-LLM Leaderboard.



Season 1, hopefully by being able to better reflect realistic use cases.

Generation tasks are different from logit-based tasks.A key difference in Season 2 is the addition of three generation-based tasks - Ko-GSM8K, Ko-EQ-Bench, Ko-IFEval - in contrast to _zero_ in Season 1. Evaluating generated outputs of models are much more likely to align with real-world usages than logit-based evaluation. Note that pre-trained models are more likely to fail on such generation tasks than fine-tuned models, which is why _fine-tuned_ models are used in real-world scenarios.

In Table 2, we show the correlation between Season 1 tasks, which are all logit-based, and the logit-based (Ko-GPQA, Ko-WinoGrande, KorNAT-Knowledge, KorNAT-Social-Value, Ko-Harmlessness, Ko-Helpfulness) and generation-based (Ko-GSM8K, Ko-EQ-Bench, Ko-IFEval) tasks of Season 2. The correlation coefficient between Season 1 and Season 2 (Generation) is 0.36, which is notably low. This indicates that the generation-based evaluation measures model capabilities that are quite different from the benchmarks of Season 1. Not only that, even within Season 2, the correlation between the logit-based and generation-based tasks is 0.33. This reinforces the notion that generation tasks in Season 2 capture different aspects of model capabilities than logit-based tasks from Season 1 or 2.

### Correlation Within the Open Ko-LLM Leaderboard2

We perform a correlation study between the Open Ko-LLM Leaderboard2 benchmark datasets. The high correlation of 0.81 between the Ko-Harmlessness and Ko-Helpfulness metrics suggests that models performing well in terms of safety also tend to provide more useful outputs. This indicates that both safety and usefulness can be evaluated simultaneously in a reliable manner. Additionally, the Ko-GSM8k and Ko-EQ Bench metrics exhibit a significant correlation of 0.64, implying that a model's mathematical problem-solving abilities are related to its general performance on EQ tasks.

Conversely, we observe lower or negative correlations in certain pairs of metrics. For example, the KorNAT-SVA metric shows little to weak negative correlations with other metrics, which suggests that its performance, particularly related to Social Value Alignment (SVA), operates independently of other tasks.

### Evaluation Times for Open Ko-LLM Leaderboard: Season 1 and Season 2

As shown in Table 3, the benchmark evaluation time for Open Ko-LLM Leaderboard2 was significantly reduced in comparison to Season 1, requiring only about 13% of the time. This allows for faster evaluation of more complex tasks and ensures more convenient access for users. Benchmarks in Season 1, such as Ko-ARC-Challenge, Ko-HellaSwag, and Ko-MMLU, took a total of 20,544 seconds, whereas evaluations in Season 2, including Ko-GPQA, Ko-WinoGrande, and Ko-GSM8k, were completed in just 2,628 seconds. As a result, this signifies smoother user accessibility and faster, more efficient evaluations.

## 5 Conclusion

In this paper, we introduced Open Ko-LLM Leaderboard2, addressing critical limitations from Season 1 by incorporating nine benchmarks that better reflect the real-world capabilities of Korean LLMs. Our analysis of submission trends and performance correlations highlights the importance of aligning evaluations with real-world usage, especially through generation-based tasks. With these enhancements, Open Ko-LLM Leaderboard2 establishes a stronger framework for Korean LLM evaluation.

\begin{table}
\begin{tabular}{l l r} \hline \hline
**Season** & **Benchmark** & **Evaluation Times (s)** \\ \hline \multirow{7}{*}{Season 1} & Ko-ARC-Challenge & 789 \\  & Ko-HellaSwag & 6,409 \\  & Ko-MMLU & 12,692 \\  & Ko-TruthfulQa-mc2 & 380 \\  & Ko-CommonGen-v2 & 274 \\  & **Total** & **20,544** \\ \hline \multirow{7}{*}{Season 2} & Ko-GPQA (Diamond) & 89 \\  & Ko-WinoGrande & 87 \\  & Ko-GSMSk & 887 \\  & Ko-IFeval & 615 \\  & Ko-EQ-Bench & 153 \\  & KorNAT-Knowledge & 137 \\  & KorNAT-Social-Value & 188 \\  & Ko-Hermlessness & 395 \\  & Ko-Helpfulness & 77 \\  & **Total** & **2,628** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Benchmark Evaluation Times for Open Ko-LLM Leaderboard Season 1 and Season 2, measured using the upstage/solar-10.7b-instruct-v1.0 model.

 



## Acknowledgments

We would like to express our sincere gratitude to the National Information Society Agency (NIA), Korea Telecom (KT), Artificial Intelligence Industry Cluster Agency (AICA), SELECTSTAR, Graduate School of AI at KAIST and Flitto. Additionally, we would like to acknowledge the Hugging Face teams, particularly Clementine Fourier, Lewis Tunstall, Omar Sanseviero, and Philipp Schmid. Moreover, we would like to express our gratitude to Professor Heuiseok Lim from Korea University, Professor Harksoo Kim from Konkuk University, Professor Hwangjo Yu from Pohang University of Science and Technology, Professor Sangkeun Jung from Chungnam National University, and Professor Alice Oh from KAIST for their valuable advice provided for the Open Ko-LLM Leaderboard. Finally, we extend our heartfelt thanks to the open-source community for their invaluable contributions and feedback. This work was supported by Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. RS-2024-00338140, Development of learning and utilization technology to reflect sustainability of generative language models and up-to-dateness over time).

### Limitations

While the Open Ko-LLM Leaderboard2 represents a significant improvement over its predecessor, there are several limitations to consider. First, despite efforts to introduce a diverse set of benchmarks, certain tasks may still not fully capture the breadth of real-world applications, especially in highly specialized domains. Additionally, the leaderboard focuses primarily on evaluating Korean language models, which limits the generalizability of the results to other languages. Another limitation is the reliance on private datasets, which, while ensuring fairness, may hinder transparency and reproducibility for the broader research community. Finally, computational resources, despite the infrastructure upgrade, remain a challenge for small teams or independent researchers, potentially limiting participation.

### Ethics Statement

This work adheres to the highest ethical standards in the development and evaluation of language models. All datasets used in the Open Ko-LLM Leaderboard2 were carefully curated to avoid biases related to sensitive topics, and efforts were made to ensure that models are evaluated for harmful or toxic outputs through specific benchmarks like Ko-Harmlessness. Additionally, the leaderboard promotes fair competition by using private datasets to prevent data contamination and ensure equal opportunities for all participants. No personal data was used in the creation of the datasets, and all experiments were conducted with respect to privacy and ethical considerations.

## References

* T. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018)Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: SS1.
* K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, L. Jauz Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021)Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: SS1.
* Z. Guo, R. Jin, C. Liu, Y. Huang, D. Shi, L. Yu, Y. Liu, J. Li, B. Xiong, D. Xiong, et al. (2023)Evaluating large language models: a comprehensive survey. arXiv preprint arXiv:2310.19736. Cited by: SS1.
* D. Hendrycks, G. Burns, S. Basart, A. Zou, N. Mazeika, D. Song, and J. Steinhardt (2020)Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: SS1.
* S. Mohan Jain (2022)Hugging face. In Introduction to transformers for NLP: With the hugging face library and models to solve problems, pp. 51-67. Cited by: SS1.
* J. Lee, M. Kim, S. Kim, J. Kim, S. Won, H. Lee, and E. Choi (2024)Kornat: llm alignment benchmark for korean social values and common knowledge. arXiv preprint arXiv:2402.13605. Cited by: SS1.
* S. Lin, J. Hilton, and O. Evans (2021)Truthfulqa: measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958. Cited by: SS1.
* S. J. Paech (2023)Eq-bench: an emotional intelligence benchmark for large language models. arXiv preprint arXiv:2312.06281. Cited by: SS1.
* C. Park and H. Kim (2024)Understanding llm development through longitudinal study: insights from the open ko-llm leaderboard. arXiv preprint arXiv:2409.03257.



Cited by: SS1, SS2.
* C. Park, H. Kim, D. Kim, S. Cho, S. Kim, S. Lee, Y. Kim, and H. Lee (2024)Open to k-kim: a deep learning approach for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 3798-3806. Cited by: SS5.
* J. Peng, S. Cheng, E. Diau, Y. Shih, P. Chen, Y. Lin, and Y. Chen (2024)A survey of useful llm evaluation. arXiv preprint arXiv:2406.00936. Cited by: SS1, SS2.
* D. Rein, B. Li Hou, A. Cooper Stickland, J. Petty, R. Yuanzhe Pang, J. Dirani, J. Michael, and S. R Bowman (2023)Gpqa: a graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022. Cited by: SS1, SS2.
* K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi (2021)Winogrande: an adversarial winograd schema challenge at scale. Communications of the ACM64 (9), pp. 99-106. External Links: Document, ISSN 0021-909X Cited by: SS1, SS2.
* J. Seo, J. Lee, C. Park, S. Hong, S. Lee, and H. Lim (2024)Kocommongen v2: a benchmark for navigating Korean commonsense reasoning challenges in large language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 2390-2415. Cited by: SS1, SS2.
* R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi (2019)Hellaswag: can a machine really finish your sentence?. arXiv preprint arXiv:1905.07830. Cited by: SS1, SS2.
* J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou (2023)Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. Cited by: SS1, SS2.