Venue: NAACL 2025

# Evaluating Multimodal Generative AI with Korean Educational Standards

Sanghee Park\({}^{*}\)

NAVER Cloud AI

parksangheeeee@gmail.com &Geewook Kim\({}^{*,\dagger}\)

NAVER Cloud AI

KAIST AI

gwkim.rsch@gmail.com

 Sanghee Park and Geewook Kim contributed equally to this work and share first authorship.Corresponding author.

###### Abstract

This paper presents the Korean National Educational Test Benchmark (KoNET), a new benchmark designed to evaluate Multimodal Generative AI Systems using Korean national educational tests. KoNET comprises four exams: the Korean Elementary General Educational Development Test (KoEGED), Middle (KoMGED), High (KoHGDed), and College Scholastic Ability Test (KoCSAT). These exams are renowned for their rigorous standards and diverse questions, facilitating a comprehensive analysis of AI performance across different educational levels. By focusing on Korean, KoNET provides insights into model performance in less-explored languages. We assess a range of models--open-source, open-access, and closed APIs--by examining difficulties, subject diversity, and human error rates. The code and dataset builder will be made fully open-sourced at [https://github.com/naver-ai/KoNET](https://github.com/naver-ai/KoNET).

## 1 Introduction

The advancement of Large Language Models (LLMs) has spurred the integration of sophisticated generative AI systems into various applications [4]. Recent developments combining LLMs with computer vision have resulted in powerful Multimodal LLMs (MLLMs) [23, 20, 29, 5]. However, questions remain about the true intelligence of these systems, especially their ability to generalize across novel tasks similar to human cognition.

Current benchmarks predominantly focus on English, overlooking the linguistic diversity worldwide and offering limited insights into low-resource languages like Korean. Moreover, many benchmarks do not compare AI performance to that of humans, making it difficult to precisely measure AI proficiency. Some benchmarks are also less connected to real-world application scenarios, hindering the applicability of MLLMs.

To address these challenges, we introduce KoNET, a benchmark dataset leveraging four key Korean educational tests (refer to Figure 1). Each

Figure 1: **Examples and Performance Overview of KoNET. (a) Illustration of mathematics problem examples, highlighting the increased complexity and difficulty as the educational level progresses. (b) Demonstration of how the accuracy of contemporary AI models decreases with more advanced curricula. A detailed analysis is provided in Section 4.



exam--KoEGED, KoMGED, KoHGED, and KoSCAT--provides detailed analyses of question difficulty, enabling nuanced evaluation of AI capabilities. Notably, KoCSAT includes data on the percentage of incorrect responses per item among examinees (human error rate), facilitating thorough comparisons of model behaviors with human performance. This benchmark allows for direct comparisons to human performance and underscores essential competencies crucial for AI-driven educational technologies, offering potential real-world applicability in the AI tutoring market.

Our key contributions include:

1. The introduction of KoNET, a comprehensive benchmark for evaluating Multimodal Generative AI Systems via Korean educational tests.
2. A thorough evaluation of various open-source, open-access, and closed API models.
3. Insights through multiple analytical frameworks, examining the relationship between human and model error rates.

## 2 Related Work

Text Benchmarks.MMLU Hendrycks et al. (2021) assesses general language proficiency, while GSM8K Cobbe et al. (2021), CS-Bench Song et al. (2024), and SciBench Wang et al. (2024b) focus on math, computer science, and science skills. These offer a focused evaluation of AI capabilities within educational contexts.

Multimodal Benchmarks.SEEDBench Li et al. (2024) and MMStar Chen et al. (2024a) provide general multimodal evaluations. Notably, there are educationally focused benchmarks such as ScienceQA Lu et al. (2022) and MathVista Lu et al. (2024), which assess AI's ability with scientific and mathematical content. Further, MMMU Yue et al. (2024a) provides diverse subject evaluations, including Art and Medicine, while AI2D Kembhavi et al. (2016) examines diagram interpretation in grade school science.

Korean Benchmarks.Korean benchmarks are limited, but efforts like K-MMLU Son et al. (2024) and Ko-H5 Park et al. (2024) have emerged. In multimodal contexts, KVQA Kim et al. (2019) and CVQA Romero et al. (2024) focus on VQA and cultural understanding. Despite the advances, there is a notable absence of Korean educational benchmarks, particularly in the multimodal domain. No existing frameworks comprehensively evaluate AI's educational performance across various school subjects within a Korean context.

## 3 Proposed Benchmark: KoNET

To offer a robust evaluation framework that facilitates comprehensive comparisons with human educational levels, we converts questions from Korea's national educational tests into a multimodal VQA format. Table 1 presents key statistics of KoNET, while Table 2 shows its main contributions.

### Education System and Qualification

Exams in Korea

Education is core to societal progress in Korea, with a structured system consisting of 6 years in elementary, 3 in middle, 3 in high school, and 4 in university or 2-3 in junior college Centre (2020).

The **General Educational Development (GED)** exams assess basic academic knowledge for individuals who have not completed formal schooling, granting qualifications equivalent to traditional graduation upon passing. The **College Scholastic Ability Test (CSAT)**, also known as "Suneung," is instrumental for college admissions and is recognized for its difficulty and ability to distinguish academic excellence.

 



### Construction of KoNET

KoNET is constructed by parsing publicly available official PDFs from the Korea Institute of Curriculum and Evaluation1. The GED tests include all questions from the first and second sessions of 2023, with each exam comprising 20 or 25 multiple-choice questions per subject, with four options provided for each question. The CSAT incorporates questions from various subjects conducted in 2023, with a range of 20 to 45 questions each. While most are multiple-choice, some subjects have subjective questions. For the CSAT, human error rates are available for a selective subset of 327 questions. This subset reflects the challenges and complexities of these questions, as human error rate data is disclosed primarily for items with higher difficulty levels. Each data sample in KoNET is represented by a single image. More details are in Appendix A.

Footnote 1: [https://www.kice.re.kr](https://www.kice.re.kr)

## 4 Experiment and Analysis

### Setup

To thoroughly test contemporary models, we use 18 open-source LLMs, 20 open-source MLLMs, 4 closed-source LLMs, and 4 closed-source MLLMs, covering a range of sizes and complexities.

Response Generation.We employ the Chain-of-Thought (CoT) Wei et al. (2022) as some KoNET problems requires complex reasoning. We use the OCR API2, specialized for Korean, to translate image content for LLM models lacking vision capabilities. MLLMs use OCR as supplementary information. The ablations on CoT prompting and OCR are in Section 4. The CoT prompts used in this study are in Appendix B. In this study, we ensured a consistent evaluation environment for LLMs and MLLMs across multiple benchmarks, including KoNET, MMMU, and MathVista, using a unified prompt structure and input format. Recent multimodal benchmarks like MMMU-Pro Yue et al. (2024b) and EXAMS-V Das et al. (2024) embed all necessary information within images, requiring MLLMs to extract and interpret content directly. KoNET follows this approach, incorporating both questions and answer choices into images, eliminating the need for explicit question and option placeholders (Figure 4). LLMs do not receive direct textual inputs but can infer information via OCR-extracted text. Furthermore, KoNET includes problems where answer choices are images rather than text, requiring MLLMs to rely on visual reasoning. This design enables a more realistic assessment of multimodal comprehension and reasoning abilities.

Footnote 2: [https://www.ncloud.com/product/aiService/ocr](https://www.ncloud.com/product/aiService/ocr)

Evaluation.We utilize the LLM-as-a-Judge approach Zheng et al. (2023) with GPT-4o OpenAI (2023) to verify correctness. This method eliminates the need for manually parsing each model output, thereby minimizing potential errors.

### Main Results

Table 3 outlines the main results, comparing KoNET performance with benchmarks like MathVista and ScienceQA. It also details subset performances for KoNET's components--elementary, middle, high school, and college exams.

Key insights include a general performance improvement with larger model sizes. Notably, there's a significant gap between closed-source APIs and open-source models, especially for KoNET, indicating open-source models lack tuning for Korean domains. Closed-source APIs likely excel due to Korea-targeted business strategies.

Models experience increased difficulty with advancing levels in the Korean curriculum, evident in subset performances. Complexity rises significantly at each educational stage, particularly in KoCSAT, highlighting the rigorous nature of these questions aligned with real-world standards.

The EXAONE-3.0-7.8B-Instruct model, a sovereign AI model specifically designed for the Korean language (bilingual in English and Korean), achieved a K-NET score of 45.5, significantly outperforming other models of similar size (7-8B). This suggests that benchmarks centered solely on English may not accurately assess AI performance in non-English or East Asian language environments. For instance, in the KoHGED (high school education exam), a question was based on the classic literary work Yongbiecheonga Songs of the Dragons Flying to Heaven), a historical text from Korea's Joseon Dynasty published in 1445. This work is part of the standard curriculum in Korean education. Models lacking an understanding of the cultural context struggled to interpret the question and failed to provide the correct answer. In contrast, the EXAONE-3.0-7.8B-Instruct model successfully derived the correct response, demonstrating how linguistic and cultural specificity significantly impacts AI performance. No

tably, open-source models such as EXAONE and Qwen2 have shown strong performance in Korean and East Asian contexts, highlighting the need for greater focus on non-English languages in future research and open-source AI development.

### Further Analyses

**Q1: Do MLLMs perform better on KoNET due to their support for multimodal inputs?**

Table 3 indicates unexpected results, with MLLMs sometimes lagging behind LLMs on KoNET, contrary to other benchmarks. We analyze model pairs sharing LLM backbones in Table 4. Without the off-the-shelf OCR assistance, closed-source MLLMs demonstrate competitive performance, comparable to LLMs with OCR support. However, many open-source MLLMs do not perform as effectively, revealing a specific challenge with text recognition in the Korean context.

**Q2: Can CoT prompting improve performance on KoNET?**

As shown in Table 4, CoT generally enhances performance across all models. Notably, this improvement is more pronounced in high-performing closed-source models compared to open-source models. This suggests that while CoT is beneficial, some open-source models are not yet fully optimized for reasoning in the Korean context, making CoT less effective.

\begin{table}
\begin{tabular}{l c|c c c c c c c c|c} \hline \multirow{2}{*}{**Model**} & \multicolumn{6}{c|}{_Previous Benchmarks_} & \multicolumn{6}{c|}{_Proposed ReLU Benchmarks_} & \multirow{2}{*}{**KoNET**} \\ \cline{2-9}  & \multicolumn{2}{c}{**Malibrata**} & \multicolumn{2}{c}{**ScienceQA**} & \multicolumn{2}{c}{**ALED**} & \multicolumn{2}{c}{**MMMU**} & \multicolumn{2}{c}{**KoGCD**} & \multicolumn{2}{c|}{**KoMED**} & \multicolumn{2}{c|}{**KoMIGD**} & \multicolumn{2}{c|}{**KoST**} \\ \hline Open Source LLM & \multicolumn{5}{c}{} \\ \hline Qwen20.5 SB-Insort (Yang et al., 2024) & 0.5 & 4.9 & 29.8 & 20.2 & 4.5 & 17.8 & 19.6 & 16.7 & 12.8 & 16.0 \\ Qwzn2+5B-Insort (Yang et al., 2024) & 1.5 & 2.8 & 32.6 & 19.6 & 6.1 & 25.8 & 20.6 & 22.0 & 14.3 & 19.2 \\ gamma2-2+6b (Team et al., 2024) & 2.0 & 1.0 & 30.0 & 24.7 & 9.8 & 30.0 & 30.7 & 32.4 & 16.5 & 25.3 \\ Pii-3mini-kit-Insort (Abdin et al., 2024) & 3.8 & 5.1 & 31.4 & 26.1 & 14.1 & 37.0 & 37.0 & 37.4 & 18.1 & 29.5 \\ Pii-3.5mini-structor (Abdin et al., 2024) & 3.8 & 5.5 & 34.9 & 26.8 & 10.9 & 29.0 & 28.0 & 23.5 & 14.6 & 21.8 \\ Yi-1.5 6B-Chut (Young et al., 2024) & 6.0 & 5.2 & 33.8 & 25.6 & 14.2 & 39.2 & 36.7 & 36.1 & 19.7 & 30.2 \\ Mixtar-Hstmart-v3.0Jiang et al., 2023) & 7.0 & 7.6 & 36.7 & 34.2 & 20.5 & 36.5 & 29.4 & 34.4 & 16.5 & 26.5 \\ Owen2-FB-Insort (Yang et al., 2024) & 7.0 & 6.4 & 35.4 & 33.2 & 23.3 & 54.0 & 53.1 & 50.7 & 20.3 & 39.6 \\ EXAONE-j-0.7 SB-Instruct (Research, 2024) & 7.8 & 7.1 & 39.3 & 34.1 & 21.9 & 64.5 & 59.1 & 56.9 & 24.2 & 45.5 \\ Meta-LDA-3.88Bi-Interrupt(Dubby et al., 2024) & 8.0 & 6.0 & 37.3 & 39.2 & 22.3 & 46.5 & 46.9 & 43.3 & 20.5 & 38.5 \\ Meta-LDA-3.13Bi-Interrupt(Dubby et al., 2024) & 8.0 & 5.3 & 38.2 & 36.7 & 19.7 & 42.5 & 41.9 & 40.6 & 18.4 & 32.3 \\ Vi-1.5-gngn-Chut (Young et al., 2024) & 9.0 & 8.2 & 37.5 & 38.6 & 20.7 & 47.0 & 43.7 & 45.0 & 22.5 & 36.0 \\ ggamma2-2+6b (Team et al., 2024) & 9.0 & 6.7 & 41.7 & 41.8 & 20.0 & 63.0 & 61.3 & 59.3 & 29.8 & 48.5 \\ Prii-3-medium-4k-invert (Abhin et al., 2024) & 14.0 & 12.6 & 48.7 & 41.6 & 17.3 & 34.8 & 34.8 & 32.0 & 17.7 & 27.4 \\ gamma2-2+7b (Team et al., 2024) & 27.0 & 18.8 & 49.6 & 47.3 & 24.6 & 74.5 & 69.6 & 68.5 & 33.9 & 55.9 \\ Yi-1.5-SAh Chat (Young et al., 2024) & 34.0 & 18.9 & 61.5 & 44.2 & 25.1 & 64.0 & 57.4 & 55.4 & 25.8 & 45.4 \\ Meta-LDA-3.17OB-Instruct(Meta, 2024) & 70.0 & 20.3 & 67.5 & 49.5 & 31.5 & 63.2 & 65.6 & 62.6 & 31.2 & 50.8 \\ Owei-ZP-2RB-Instruct (Yang et al., 2024) & 72.0 & 21.7 & 69.1 & 49.4 & 32.3 & 76.0 & 74.1 & 71.9 & 36.0 & 58.7 \\ Open Source VLM & \multicolumn{5}{c}{} \\ InterN/Y1-2.1B Chen et al., 2024) & 1.0 & 33.5 & 59.6 & 65.2 & 35.0 & 0.8 & 0.4 & 0.9 & 0.4 & 0.6 \\ InternVL2-2.2B Chen et al., 2024) & 2.0 & 35.4 & 62.0 & 74.0 & 35.7 & 2.2 & 2.0 & 3.3 & 1.7 & 2.2 \\ OuraVL-2.2B-Instruct (Wang et al., 2024) & 2.0 & 42.9 & 65.4 & 76.5 & 40.2 & 13.2 & 13.0 & 12.2 & 8.4 & 11.0 \\ palipform-3-via-Attos (Beyer et al., 2024) & 3.0 & 29.1 & 65.3 & 69.8 & 33.4 & 8.2 & 8.7 & 8.7 & 4.9 & 7.1 \\ IntevmL2-4B (Chen et al., 2024) & 4.0 & 57.0 & 71.5 & 78.7 & 46.5 & 1.5 & 2.0 & 1.7 & 0.9 & 1.4 \\ Pin-3.5 vision-insortor (Abhin et al., 2024) & 4.2 & 44.8 & 68.6 & 77.8 & 39.3 & 15.0 & 17.0 & 13.1 & 4.6 & 10.9 \\ OveraVL-7.7-TB-Instant (Wang et al., 2024) & 7.0 & 53.2 & 66.7 & 71.5 & 59.1 & 49.5 & 46.9 & 42.0 & 16.9 & 34.3 \\ Itava-1.5-7th (Li et al., 2024) & 7.0 & 30.9 & 67.3 & 53.0 & 30.8 & 3.2 & 4.6 & 4.8 & 3.2 & 3.9 \\ Java+7.1-6v-team-7b-for (Liu et al., 2024) & 7.0 & 35.2 & 71.7 & 53.9 & 34.0 & 3.0 & 2.8 & 1.9 & 1.6 & 2.1 \\ InternVL2-BB (Cheng et al., 2024) & 8.0 & 58.2 & 61.9 & 65.9 & 53.3 & 12.2 & 11.7 & 8.0 & 4.0 & 7.9 \\ JamaX-java-asu-esd-BM (Liu et al., 2024) & 8.0 & 37.1 & 70.5 & 58.3 & 35.1 & 10.2 & 7.8 & 7.2 & 2.6 & 6.0 \\ lavax-1.5-13kb-ful (Liu et al., 2024) & 13.0 & 26.6 & 49.3 & 57.6 & 37.5 & 11.8 & 8.1 & 7.4 & 4.6 & 7.1 \\ Itava-r1.5-strain-13bd (Lin et al., 2024) & 13.0 & 37.0 & 71.5 & 60.3 & 34.9 & 5.0 & 5.0 & 7.2 & 6.9 & 6.3 \\ copyright- 



### Q3: Do AI models have similar error patterns to students?

We compare human error rates on 327 questions with AI error rates. The human error rates in KoCSAT are derived from the Korean College Scholastic Ability Test (KoCSAT), which plays a crucial role in university admissions in South Korea. This exam is a large-scale standardized assessment taken by hundreds of thousands of students each year, who systematically prepare and sit for the test under controlled conditions. In this study, human error rates are calculated based on data from approximately 505K students, using official statistics published by the Korea Institute for Curriculum and Evaluation (KICE3). KICE is the official national institution responsible for the development and evaluation of all exams included in KoNET.

Footnote 3: [https://www.kiec.re.kr](https://www.kiec.re.kr)

To analyze error rates, we explore variability in model responses by assigning different personas [14] and adjusting parameters like temperature. Using gpt-4o-2024-05-13, the strongest of our test models, we create 10 persons,4 generating 10 responses per persona for a total of 120 responses. For gpt-4o-2024-05-13, gemini-1.5-pro, HyperCLOVA-X, and claude-3-5-sonnet-20240620, we use three personas ('student,' 'teacher,' and 'professor'),5 also generating 10 responses per persona for a total of 120 responses. This setup addresses the challenge of limited high-performing AI models by using personas to expand the response pool, thus enabling comprehensive trend comparisons between AI models and student groups.

Footnote 4: Personas include ‘student,’ teacher,’ professor,’ engineer, ’scientist,’ mathematician,’ doctor, ’lawyer,’ master student,’ and ’PhD student.’

Figure 2 indicates a weaker than expected positive correlation. Detailed analysis shows AI models excel in comprehension tasks, likely due to human attention lapses, while humans perform better in memorization tasks, especially in long-tail questions for exams like the CSAT. These outcomes align with expectations and underscore the benchmark's value by integrating human error data, providing a rich resource for future studies.

## 5 Conclusion

We present KoNET as a benchmark for evaluating multimodal generative AI models using Korean educational tests. Our findings reveal varying performance with multimodal inputs and highlight specific challenges. The disparity between open and closed-source models points to the need for advancements in open-source models within non-English contexts. Our analysis of human error rates offers valuable insights into AI and human performance comparisons. Through KoNET, we aim to encourage research in multimodal and multilingual AI, thereby promoting inclusivity and diversity.



processes. Although a small proportion of the questions are subjective (see Table 2), these generally involve short-response formats. To address this, future work could focus on evaluating models' reasoning abilities by incorporating rationales behind their answers. This advancement necessitates the development of comprehensive reference answers and a consideration of the increased computational costs involved.

Moreover, as is common with all benchmarks, periodic updates to the test set are necessary to mitigate potential biases and data contamination upon public release. Given that KoNET is based on annually updated national tests, it is inherently suited for regular renewal. We anticipate that our dataset construction methodology, along with the open-source dataset builder, will empower the research community to continuously update KoNET, ensuring its ongoing relevance and utility in advancing AI systems to better meet diverse needs.

## References

* A. Abdin, S. J. Jacobs, A. Ahmad Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl, et al. (2024)Phi-3 technical report: a highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. Cited by: SS5.
* L. Beyer*, A. Steiner*, A. Susano Pinto*, A. Kolesnikov*, X. Wang*, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, T. Unterthiner, D. Keysers, S. Koppula, F. Liu, A. Grycer, A. Gritsenko, N. Houlsby, M. Kumar, K. Rong, J. Eisenschlos, R. Kabra, M. Bauer, M. Bosnjak, X. Chen, M. Minderer, P. Voigtlaender, I. Bica, I. Balazevic, J. Puigcerver, P. Papalampidi, O. Henaff, X. Xiong, R. Soricut, J. Harmsen, and X. Zhai* (2024)PaliGemma: a versatile 3B vLM for transfer. arXiv preprint arXiv:2407.07726. Cited by: SS5.
* K. Centre (2020)Education in Korea -- koreaneducentreinuk.org. Note: [http://koreaneducentreinuk.org/en/education-in-korea](http://koreaneducentreinuk.org/en/education-in-korea) External Links: Link Cited by: SS5.
* Lin Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, et al. (2024)Are we on the right way for evaluating large vision-language models?. arXiv preprint arXiv:2403.20330. Cited by: SS5.
* Zhe Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. (2024)How far are we to gpt+4y? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821. Cited by: SS5.
* Karl Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021)Training verifics to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: SS5.
* Rocktim Jyoti Das, S. Emilov Hristov, H. Li, D. Iliyanov Dimitrov, I. Kovchev, and P. Nakov (2024)Exams-v: a multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. arXiv preprint arXiv:2403.10378. Cited by: SS5.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Lettman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Harshorn, A. Yong, A. Mitra, A. Szwankumar, A. Korenev, A. Hinnsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marras, C. McConnell, C. Keller, C. Tonuret, C. Wu, C. Wong, C. Canton Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Esioub, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Lupkes, E. Lakomkin, E. Albaday, E. Lobanova, E. Dinan, E. Michael Smith, F. Radenovic, F. Zhang, G. Synnaeye, G. Lee, G. Lewis Anderson, G. Van Eng, G. Cucurell, H. Nguyen, H. Korvaar, H. Xu, H. Touvron, I. Zarov, I. Arrieta Ibarra, I. Klumann, I. Misra, I. Evtimov, J. Copet, J. Lee, J. Geffert, J. Varnes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spiask, J. Park, J. Rocca, J. Johnston, J. Saxe, J. Jia, K. VasadeniAlwa, K. Upasani, K. Plawiak, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Maleik, K. Chiu, K. Bhalla, L. Rantala-Yaery, L. van de Maaten, L. Tan, L. Jenkins, L. Martin, L. Masdaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Karadas, M. Oldham, M. Rita, M. Pavkova, M. Kambadur, M. Lewis, M. Si, M. Kumer Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, O. Duchenne, Onur Celebi, P. Alrassy, P.

 

Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prijwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Pavia Xu, Qing He, Qingsziao Dong, Ragavin Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Siliveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhasale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheaasha, Thomas Georgiou, Thomas Sciadom, Tobias Speckbechacher, Todor Mihalyo, Tong Xiao, Ujjwal Karm, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Voggei, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyn Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaqiang Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yamine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierte Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafori, Abha Jain, Adam Kelsey, Adam Shahinfeld, Adithya Gangidi, Adolfo Victoria, Ahiva Golsdstand, Ajay Menou, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Beavski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharamebe, Assif Eisenman, Azadeh Yadadan, Beau James, Ben Maurer, Benjamin Leonhardri, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changshan Wang, Changkyyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damoin Civin, Dana Beatty, Daniel Keymer, Daniel Li, Danny Wyat, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkmann, Esteban Areute, Evvan Dunbar, Evan Snothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzman, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Gunna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zhao, Harouw Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molyboog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhiet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Rezienstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingji Yang, Joe Cummings, Jon Carroll, Jon Sherpard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Chandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kwash Lakhotia, Kyle Huang, Lailin Chen, Labshyag Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Livron Moshkovich, Luca Wehrstedt, Madiano Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoulkiti, Martynas Markus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Kenelly, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vytsakov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquet Jubert Hermoso, Mo Metanet, Mohammad Rastegari, Munish Bansal, Nandhnii Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nagyan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavloviol Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chermoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafiyu, Raghotham Murthy, Raghum Najani, Rahul Mitra, Raymond Li, Rebekkan Hoban, Robin Battey, Rocky Wang, Rohan Mahseswari, Russ Howes, Ruuty Rinot, Said Jayesh Bondu, Samyak Datta, Sara Chaph, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shawn Linghe's, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shivan Shankar, Shuquiang Zhang, Sinno Wang, Pasha Agarwal, Soil Sajuyigibe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Surai Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianne Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitt, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Visual Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouazz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojiang Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xiubo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zel Rosenbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3 herd of models. _Preprint_, arXiv:2407.21783.
* Google (2024) Google. 2024. Gemini 1.5 pro.
* Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazieika, Dawn Song, and Jacob Steinhardt.

 

2021. Measuring massive multitask language understanding. In _International Conference on Learning Representations_.
* Hong et al. (2024) Wenyi Hong, Weihan Wang, Ming Ding, Wenneng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. 2024. Cogvlm2: Visual language models for image and video understanding. _Preprint_, arXiv:2408.16500.
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. _arXiv preprint arXiv:2310.06825_.
* Kembhavi et al. (2016) Aniruddha Kembhavi, Michael Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. _ArXiv_, abs/1603.07396.
* Kim et al. (2019) Jin-hwa Kim, Soohyun Lim, Jaesun Park, and Hansu Cho. 2019. Korean Localization of Visual Question Answering for Blind People. In _AI for Social Good workshop at NeurIPS_.
* Laurencon et al. (2024a) Hugo Laurencon, Andres Marafioti, Victor Sanh, and Leo Tronchon. 2024a. Building and better understanding vision-language models: insights and future directions. _Preprint_, arXiv:2408.12637.
* Laurencon et al. (2024b) Hugo Laurencon, Leo Tronchon, Matthieu Cord, and Victor Sanh. 2024b. What matters when building vision-language models? _Preprint_, arXiv:2405.02246.
* Li et al. (2024) Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimo Zhang, and Ying Shan. 2024. SEEDBench: Benchmarking Multimodal Large Language Models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 13299-13308.
* Liu et al. (2024a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 26296-26306.
* Liu et al. (2022b) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llavanext: Improved reasoning, ocr, and world knowledge.
* Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. In _Advances in Neural Information Processing Systems_, volume 36, pages 34892-34916. Curran Associates, Inc.
* Lu et al. (2024) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chupyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In _International Conference on Learning Representations (ICLR)_.
* Lu et al. (2022) Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tarfjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_.
* Meta (2024)Meta. 2024. LLaMA 3.1. Accessed: 23 Jul. 2024.
* Cloud (2024) NAVER Cloud. 2024. HyperCLOVA X Vision. Accessed: 19 Aug. 2024.
* OpenAI (2023) OpenAI. 2023. GPT-4 Technical Report. _Preprint_, arXiv:2303.08774.
* OpenAI (2024) OpenAI. 2024. Gemini 1.5 pro.
* Park et al. (2024) Chanjun Park, Heyouwoo Kim, Dahyun Kim, SeongHwan Cho, Sanghoon Kim, Sukyung Lee, Yunqi Kim, and Hwalsuk Lee. 2024. Open Ko-LLM leaderboard: Evaluating large language models in Korean with Koh- h5 benchmark. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3220-3234, Bangkok, Thailand. Association for Computational Linguistics.
* LG AI Research (2024)LG AI Research. 2024. Exaone 3.0 7.8b instruction tuned language model. _arXiv preprint arXiv:2408.03541_.
* Romero et al. (2021) David Romero, Chenyang Lyu, Haryo Akbarianto Wisobowe, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragometti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifelouwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajiri Koto, Fauzana Facoolai, Frederico Belcavelto, Ganoziro Batnasan, Gisela Vallejo, Granine Caulfield, Guido Ivetta, Haiyue Song, Henok Biadgligna Ademtew, Hernan Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajos, Jinheon Baek, Joyen Dunstan, Laura Alonso Alemany, Kumarangaev Ravindu Yass Nagasingne, Luciana Benotti, Luis Fernando D'Haro, Marcelo Virdiiano, Marcos Estecha-Garitagoitia, Maria Camila Buittrago Cabrera, Mario Rodriguez-Cantellar, Melanie Joutiteau, Mihail Mihalyov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adiliazavada, Munkhjjagal Gochoo, Muhh-Erdeme Otgondolo, Naome Eloroi, Olivier Niyomugisha, Paula Monica Silva, Pranjal Chitale, Raj Dabre, Rendi Chewi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyahvijaya, Santiago Gongora, Soyeong Jeong, Sukannya Purkyastha, Tatsuki Kuribayashi, Thanmay Jayakumar, Tiago Timponi Torrent, Toqeer Ehsan, Vladimir Araujo, Yova Kenentchedipleiva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Osana Ignat, Joan Wyatu, Rada Mihalcea, Thamar Solorio, and Albham Fiki Ariji. 2024. Cvqa: Culturally-diverse multilingual visual question answering benchmark. _Preprint_, arXiv:2406.05967.

 

* Saffari et al. (2023) Mustafa Safdiar, Greg Serapio-Garcia, Clement Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Mataric. 2023. Personality traits in large language models. _arXiv preprint arXiv:2307.00184_.
* Son et al. (2024) Guijin Son, Hanwoul Lee, Sungdong Kim, Seungongo Kim, Niklas Muennighoff, Taekyoon Choi, Cheombok Park, Kang Min Yoo, and Stella Biderman. 2024. Krmmlu: Measuring massive multitask language understanding in korean. _Preprint_, arXiv:2402.11548.
* Song et al. (2024) Xiaoshuai Song, Muxi Diao, Guanting Dong, Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, et al. 2024. Cs-bench: A comprehensive benchmark for large language models towards computer science mastery. _arXiv preprint arXiv:2406.08587_.
* Team et al. (2021) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Suraya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, Pouya Tatfi, Leonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amelie Heliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clement Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Nolan, Gang Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Michel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahlinze, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Shlotto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowice, Yu hui Chen, Zafrali Ahmed, Zhitao Gong, Tris Watkerentin, Ludovic Peran, Minh Giang, Clement Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Keneasy. 2024. Gemma: Open models based on gemini research and technology. _Preprint_, arXiv:2403.08295.
* Wang et al. (2024a) Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Hui Men, Dayingue Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. _arXiv preprint arXiv:2409.12191_.
* Wang et al. (2024b) Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jiyu Zhang, Satyen Subramanian, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2024b. SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. In _Proceedings of the Forty-First International Conference on Machine Learning_.
* Wei et al. (2022a) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022a. Chain-of-thought prompting elicits reasoning in large language models. In _Advances in Neural Information Processing Systems_, volume 35, pages 24824-24837. Curran Associates, Inc.
* Yang et al. (2024) An Yang, Baosong Yang, Bingyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengping Li, Chengyuan Li, Daviheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingreen Zhou, Jinze Bai, Jinzheg He, Juanying Lin, Kai Dang, Keming Lu, Kapin Chen, Kevin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru peng, Rui Men, Ruiz Gao, Runji Lin, Shijing Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yougino Liu, Zeeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report. _Preprint_, arXiv:2407.10671.
* Yoo et al. (2024) Kang Min Yoo, Jaegeu Han, Sooko In, Hewon Jeon, Jisen Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungiu Kim, et al. 2024. Hyperclova x technical report. _arXiv preprint arXiv:2404.01954_.
* Young et al. (2024) Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. _arXiv preprint arXiv:2403.04652_.
* Yue et al. (2024a) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024a. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In _Proceedings of CVPR_.
* Yue et al. (2024b) Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. 2024b. Mmnu-pro: A more robust multi-discipline multimodal understanding benchmark. _arXiv preprint arXiv:2409.02813_.
* Zheng et al. (2024)Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, 

Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.

## Appendix A Details on the KoNET Construction

KoNET encompasses a wide range of subjects across each exam, as detailed in Table 5. For K-GED (comprising KoEGED, KoMGED, KoHGED), core subjects are included as common components, while each exam features additional unique subjects. The KoCSAT comprises core subjects and optional subjects, with each optional subject further divided into specialized areas. Although students typically select specific subjects for their exams, this study includes questions from all subjects to ensure comprehensive coverage. All images within KoNET are presented in gray-scale, encapsulating the question, answer choices, and comprehension elements within a single image--a format that varies across problems. We adopt the simplest input method to evaluate both LLMs and MLLMs models. Each provided image is structured to contain both the question and all the information necessary to solve it. For text input, no additional text is provided beyond instruction-following prompts and OCR tokens (See Figure 4). This input format also allows us to indirectly assess the MLLMs models' overall understanding of the image and their ability to recognize Korean characters.

KoNET is constructed by parsing publicly available official PDFs from the Korea Institute of Curriculum and Evaluation6. We remain mindful of licensing issues, acknowledging the inherent copyright of these questions. However, details regarding specific licensing terms remain elusive; the only guidance available from the Korea Institute of Curriculum and Evaluation indicates permission for non-commercial use. We uphold the copyrights of the original owners with utmost respect. Rather than distributing the data directly, we provide dataset builder code that allows users to convert downloaded official PDFs into benchmark-ready formats. In this paper, we include images that mimic various question types rather than actual problem images. The rendered images in the form of test sheets, based on these mimicked images, are shown in Figure 3. Actual problem images can be generated and reviewed using the provided dataset builder.

Footnote 6: [https://www.kice.re.kr](https://www.kice.re.kr)

## Appendix B Details of the Used Prompts

In this study, we use Korean prompts to generate and assess the response generation capabilities of the models. Two types of prompts are employed: the Direct prompt and the Chain of Thought (CoT) prompt. The Direct prompt involves extracting an

\begin{table}
\begin{tabular}{l|p{179.8pt}} \hline \hline
**Test** & **Subjects** \\ \hline
**KoEGED** & Korean, English, Mathematics, Social Studies, Science, Music, Physical Education, Ethics, Art, Practical \\
**KoMGED** & Korean, English, Mathematics, Social Studies, Science, Music, Physical Education, Ethics, Art, Information, Technology \\
**KoHGED** & Korean, English, Mathematics, Social Studies, Science, Music, Physical Education, Ethics, Art, Technology, Korean History \\
**KoCSAT** & Korean (Common), Korean (Speech Writing), Korean (Language and Media), Mathematics (Common), Mathematics (Statistics), Mathematics (Calculus), Mathematics (Geometry), English, Korean History, Social Studies (Every Ethics), Social Studies (Ethical Ideology), Social Studies (Korean Geography), Social Studies (International Geography), Social Studies (East Asia History), Social Studies (International History), Social Studies (Economics), Social Studies(Politics and Law), Social Studies(Social Culture), Science (Physics I), Science (Chenistry I), Science (Bio Science II), Science (Earth Science II), Job Studies (Successful Career Life), Job Studies (Agricultural Technology), Job Studies (General Industry), Job Studies (Commercial Economy), Job Studies (Fisheries Shipping Industry), Job Studies (Human Development), Second Language (German), Second Language (French), Second Language (Spanish), Second Language (Chinese), Second Language (Russian), Second Language (Arabic), Second Language (Vietnamese), Second Language (Chinese characters) \\ \hline \hline \end{tabular}
\end{table}
Table 5: **List of subjects categorized under various Korean educational tests.** KoEGED represents subjects for elementary-level general education (10 subjects), KoMFGD covers middle-level general education (11 subjects), and KoHFGD encompasses high school-level general education (11 subjects). KoCSAT includes the 41 subjects evaluated in the Korean College Scholastic Ability Test, spanning multiple disciplines, including languages, mathematics, sciences, social studies, and job studies.

 

swer's directly from the provided options for each question. Conversely, the CoT prompt allows the model to reason through the problem to infer the answer. Additionally, a Judge prompt is used within

Figure 3: **Illustrative Representation of the KoNET. The test includes various types of questions, such as those requiring comprehension of images and queries, reading and understanding of lengthy texts, and simple knowledge-based queries.**



the CoT framework to evaluate the responses generated by comparing them with the correct answers. While the original prompts are in Korean, English translations are also provided for reference. The format of these prompts is exemplified in Figure 4.

## Appendix C Additional Analysis

### On the Performance Gap Between LLMs and MLLMs

Figure 5 illustrates the score distribution of LLMs and MLLMs on both conventional benchmarks and KoNET. As shown in our work, the KoNET reveals a distinct distribution pattern compared to traditional benchmarks. Notably, MLLMs under-perform relative to LLMs. As analyzed in the paper, we suggest that public LLMs may actually achieve better performance when supported by Korean OCR and many commercially available MLLMs are less effective in processing non-English contexts. This finding provides a novel perspective for model analysis that diverges from traditional benchmarks.

### Comparison of LLM-as-a-Judge with Manual Grading

To see whether LLM-as-a-Judge provide similar user experience or performance to manual grading, we conduct an additional analysis on this. Given the multiple-choice nature of the tests and the potential for varying text responses, we adopt the LLM-as-a-Judge strategy to ensure grading accuracy. Table 6 indicates that this approach closely mirrors manual grading results, demonstrating its reliability and potential as an efficient evaluation method.

### Analysis of Human Error Rates

We employ the error rates from the KoCSAT to assess and compare the performance of models against human performance. Human error rates range from 10.6% to 98.2%, as illustrated in Figure 6.

In the first analysis, we calculate model error rates using four closed-source MLLM APIs. For each model, we configure ten personas (i.e., different system messages), set the temperature to 1.0, and generate outputs three times.

In the second analysis, we utilize the GPT-4o model across ten personas, generating twelve distinct responses per persona. We then compute the model error rates and compare them with the human error rates. Figure 7 illustrates the distribution of error rates across subjects, while Figure 8 provides a point-by-point comparison of human and model error rates.

This rigorous analysis enhances our understanding of model performance relative to human benchmarks, offering valuable insights into the strengths and limitations of current MLLMs in processing complex educational content.

### Multilingual Ability Assessment

We assess multilingual capabilities using specific subjects from KoSET. The KoCSAT includes subjects for nine different languages. Traditionally, multilingual capabilities are evaluated by translating English-based benchmarks into other languages or by making indirect comparisons using benchmarks crafted in different linguistic regions. However, the multilingual subjects in KoCSAT consist of independent questions with comparable difficulty levels, enabling a more equitable and valid comparison of multilingual abilities. Figure 9 illustrates the multilingual capabilities across different model types.

\begin{table}
\begin{tabular}{l l l l l l} \hline  & Test 1 & Test 2 & Test 3 & Test 4 & Test 5 \\ \hline
**Accuracy** & 96.9\% & 98.3\% & 98.2\% & 97.4\% & 98.2\% \\ \end{tabular}
\end{table}
Table 6: **Agreement Rate Between Human Evaluation and Judge Model. When using the LLM-as-a-Judge approach, results may vary slightly with each evaluation. To ensure consistency, we conduct evaluations five times to assess whether the LLM-as-a-Judge method aligns closely with answers annotated manually by the authors. When considering the authors’ evaluation results as the ground truth, we find that the accuracy is consistently high. This suggests that LLMs can reliably substitute human evaluators with a high degree of confidence.

 

* [25] M. C. Bader, R. D. Kiefer, and A. J. Brox, "A new approach to the detection of a single person using deep neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-8, 2009.
* [26] S. G. Hinton, P. Hariharan, and I. Sutskever, "Deep learning with deep convolutional neural networks," in _Advances in Neural Information Processing Systems (NIPS'04)_, volume 37 of _Lecture Notes in Artificial Intelligence_, pages 1-10, 2004.
* [27] L. Bottou, Y. Bengio, and V. LeCun, "Generative adversarial nets for large scale image recognition," _arXiv preprint arXiv:1412.6090_, 2014.
* [28] T. Darrell, E. Farhadi, and F. Fergus, "Image classification by visual domain adaptation," _IEEE Transactions on Image Processing_, vol. 12, no. 1, pp. 1-10, Jan. 2014.
* [29] N. DeVito, M. Girshick, and Z. Donahue, "Very diverse objects from tiny images," in _Computer Vision and Pattern Recognition Workshops (CVPR)_, pp. 1-10, Jan. 2014.
* [30] W. Dong, X. Li, and Q. Weinberger, "Visualizing object recognition via deep neural networks," _CoRR_, abs/1412.0338v1, 2014.
* [31] W. Dong, X. Li, and Q. Weinberger, "Learning object recognition via deep neural networks," _Journal of Machine Learning Research_, vol. 12, no. 1, p. 100001, 2014.

[MISSING_PAGE_POST]

 

Figure 5: **Performance of LLMs and MLLMs across Previous benchmarks and KoNET. These present a performance comparison between LLMs and MLLMs across various benchmarks, including KoNET. These illustrate the accuracy distribution for each model type, but KoNET shows a different distribution trend between LLMs and MLLMs compared to other benchmarks.**



Figure 6: **Examples of human error rate. These illustrates human error rates across three types of comprehension tasks: sentence selection (left), sentence ordering (middle), and sentence insertion (right). The percentages at the top represent the error rates calculated based on responses from students. Higher error rates indicate more challenging tasks requiring deeper comprehension. Notably, as the complexity of the comprehension text increases, the error rate also rises, suggesting a greater cognitive load in understanding and structuring the given information.**





## 6 Conclusion

Figure 7: **Distribution of human and models error rate by subjects. These compares the error rate distributions between humans (blue) and models (pink) across various academic subjects. The x-axis represents the error rate, while the y-axis lists different subjects, covering social sciences, natural sciences, Korean language, history, and mathematics. The varying distributions highlight the differences in performance between humans and models, with some subjects showing a greater disparity.



Figure 8: **Distribution of human and models error rate by points. These presents the error rate distribution of humans (green) and models (brown) based on different point values assigned to questions. The x-axis represents the percentage of incorrect answers, while the y-axis categorizes questions by their point values. Higher-point questions generally require deeper reasoning and comprehension, which is reflected in the increasing error rates for both humans and models.**





## Appendix A

Figure 9: **Performance of multilingual ability. These illustrations depict the accuracy distribution of various models across multiple languages, highlighting their multilingual capabilities. The x-axis represents accuracy percentages, while the y-axis lists different languages. In general, Open Source models tend to support a narrower range of languages fluently compared to Closed Source models. However, even among Closed Source LLMs, performance tends to decline for certain languages; for instance, Arabic differs from English in writing direction, which can impact model performance.**