Title: Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark
Benchmark: Ko-H5
Summary: This paper introduces the Open Ko-LLM Leaderboard and the Ko-H5 Benchmark to evaluate large language models (LLMs) in Korean. It includes human and machine translations of the English Open LLM Leaderboard benchmarks and adds a benchmark curated from scratch. They keep private test sets to prevent data leakage, offering empirical insights through correlation and temporal analyses.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: natural language generation in Korean (commonsense reasoning and generation, truthfulness, common and domain-specific knowledge)
Phenomenon defined by authors: No
Phenomenon definition: "Ko-MMLU and KoHellaSwag (...) broadly assess various natural language understanding abilities and commonsense capabilities. Conversely, Ko-ARC, Ko-TruthfulQA, and KoCommonGen v2 focus on more specialized abilities such as domain-specific knowledge, truthfulness, and common sense generation, respectively."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Purpose notes: The authors argue for a dynamic expansion of benchmarks when performance saturation occurs.
Task definition: Multi-task benchmark: The LLM has to answer a questions about general knowledge (Ko-MMLU), domain-specific knowledge (Ko-ARC), commonsense reasoning (Ko-HellaSwag), truthfulness (Ko-TruthfulQA), and generate common knowledge (Ko-CommonGen v2).
Task item definition: Ko-ARC: question and list of choices;
Ko-HellaSwag: activity label, context a, context b, and list of possible endings; 
Ko-MMLU: question and list of choices;
Ko-TruthfulQA: question, best answer, list of correct answers, and list of incorrect answers;
Ko-CommonGen v2: list of words to generate a sentence (not described in the paper)
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task source detail: Ko-MMLU, Ko-ARC, Ko-HellaSwag, Ko-TruthfulQA are translated. The only information about the new benchmark is "The Korean CommonGen v2 is curated from scratch, inspired by CommonGen (Lin et al., 2019)."
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 26700 total samples (Ko-ARC: 1.1k, Ko-HellaSwag: 10.0k, Ko-MMLU: 14.0k, Ko-TruthfulQA: 0.8k, Ko-CommonGen v2: 0.8k)
Dataset size detail: "Ko-MMLU and KoHellaSwag necessitate a larger samples to broadly assess various natural language understanding abilities and commonsense reasoning capabilities. Conversely, Ko-ARC, Ko-TruthfulQA, and KoCommonGen v2 focus on more specialized abilities such as domain-specific knowledge, truthfulness, and common sense generation, respectively, where a smaller number of high-quality samples may be more appropriate for evaluation."
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Response format detail: Format inferred from base English benchmarks, not specified in the paper
Metric definition: Exact Match (accuracy, F1, precision, recall), Ko-H5 score
Metric definition detail: The paper only refers to the average score of Ko-H5 benchmark (Ko-H5 score, in short). The above information is again inferred based on the English base benchmarks.
Metric aggregation: Simple Mean
Metric statistics: simple mean, correlation
Metric subscores: No
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: No, no link is provided
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes, implicitly
Author validity detail: - Face validity is addressed through alignment with the established English Open LLM Leaderboard.
- Predictive/concurrent validity is supported via temporal analyses showing that model improvements (e.g., pretrained → instruction-tuned) are reflected in Ko-H5 scores.
- Content validity is demonstrated through the benchmark's design—using expert-reviewed translations, domain adaptation, and the introduction of a novel task (Ko-CommonGen v2).
- Convergent/discriminant validity is directly addressed in a correlation study across tasks in the Ko-H5 suite.
Taxonomy: NLP / 
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding
Procedural notes: Leaderboard on Hugging Face: https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard