Benchmark: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Summary: Extends MMLU (hard, diverse multiple choice llm reasoning dataset) to be harder, more diverse, and have more multiple choice options,.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: "language comprehension and reasoning across diverse domains" and literally just "measuring future (stronger) LLMs"
Phenomenon defined by authors: No
Phenomenon definition: "expert-level intelligence, characterized by performance that meets or surpasses the top 10% of skilled adults in a diverse range of tasks"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: You are given a text question from across math, physics, chemistry, etc etc. You must chose one of 10 multiple choice answers. 
Task item definition: Question: A refracting telescope consists of two converging lenses separated by 100 cm. The eye-piece lens has a focal length of 20 cm. The angular magnification of the telescope is...
Options: A. 10, B. 40, C. 6, D. 25, E. 15, F. 50, G. 30, H. 4, I. 5, J. 20
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 12000
Dataset size notes: validation: 70
Dataset metadata available: Yes
Dataset metadata detail: category (ie maths, physics etc) and source (the original dataset the question was taken from)
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Reported by category, ie maths, physics etc
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: It measures the ability to solve STEM multiple choice questions, but not as the authors claim "expert level intelligence across a diverse range of tasks". 
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: Yes
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The MMLU-Pro dataset, while enhancing the complexity of MMLU by incorporating more challenging, reasoning-focused questions, remains constrained by the limitations of the multiple-choice format. This format may not capture the depth of comprehension and creative response generation as effectively as open-ended answers, which better reflect real-world scenarios. Additionally, MMLUPro exclusively focuses on language models and does not include assessments for multi-modal models, limiting its applicability in scenarios requiring synthesis of visual, auditory, and textual data.
Taxonomy: Knowledge / General
New taxonomy: General Application / General Knowledge