Please summarise: MMLU-Pro: A More Robust and Challenging
Multi-Task Language Understanding Benchmark
1Yubo Wang∗
, 1Xueguang Ma∗
, 1Ge Zhang, 1Yuansheng Ni, 1Abhranil Chandra,
1Shiguang Guo, 1Weiming Ren, 1Aaran Arulraj, 1Xuan He, 1Ziyan Jiang, 1Tianle Li,
1Max Ku, 2Kai Wang, 1Alex Zhuang, 1Rongqi Fan, 3Xiang Yue, 1Wenhu Chen∗
1University of Waterloo, 2University of Toronto, 3Carnegie Mellon University
Abstract
In the age of large-scale language models, benchmarks like the Massive Multitask
Language Understanding (MMLU) have been pivotal in pushing the boundaries
of what AI can achieve in language comprehension and reasoning across diverse
domains. However, as models continue to improve, their performance on these
benchmarks has begun to plateau, making it increasingly difficult to discern dif-
ferences in model capabilities. This paper introduces MMLU-Pro, an enhanced
dataset designed to extend the mostly knowledge-driven MMLU benchmark by in-
tegrating more challenging, reasoning-focused questions and expanding the choice
set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and
noisy questions in MMLU. Our experimental results show that MMLU-Pro not
only raises the challenge, causing a significant drop in accuracy by 16% to 33%
compared to MMLU but also demonstrates greater stability under varying prompts.
With 24 different prompt styles tested, the sensitivity of model scores to prompt
variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Additionally,
we found that models utilizing Chain of Thought (CoT) reasoning achieved better
performance on MMLU-Pro compared to direct answering, which is in stark con-
trast to the findings on the original MMLU, indicating that MMLU-Pro includes
more complex reasoning questions. Our assessments confirm that MMLU-Pro is a
more discriminative benchmark to better track progress in the field.
Figure 1: Comparing between MMLU and MMLU-Pro: (Left) Performance gap; (Center) Accuracy
distributions affected by 24 prompts, with taller and thinner profiles indicating more stability and
shorter and wider profiles indicating greater fluctuations; (Right) Performance using CoT vs. Direct.
∗Core Contributors. B: {y726wang, x93ma, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
arXiv:2406.01574v4 [cs.CL] 23 Jun 2024
1 Introduction
In recent years, advancements in large language models (LLMs) have significantly transformed
the field of natural language processing (NLP). These models, including state-of-the-art examples
like GPT-4, Gemini, and Claude [ 2, 30 , 6 ], are pushing the envelope both in general applicability
across various tasks and specialized performance in specific areas. A key objective in this ongoing
development is achieving expert-level intelligence, characterized by performance that meets or
surpasses the top 10% of skilled adults in a diverse range of tasks [26].
To effectively track the progress towards the goal of this expert-level intelligence, it is essential to
evaluate these models on a broad range of tasks. There are multiple popular benchmarks used to
measure such general intelligence. For example, AGIEval [48 ] focuses on general-exam questions
from SAT, Gaokao, GRE, etc. ARC [ 12 ] focuses on science-based questions. BBH [33 ] focuses on
solving hard synthetic tasks. MMLU [18] includes a broad range of exam questions from 57 subjects
across STEM, the humanities, the social sciences, etc. Among these benchmarks, MMLU emerged
as the de facto standard for evaluating LLMs due to its broad coverage and high quality. However,
the rapid progress of current LLMs has quickly led to performance saturation on MMLU. Since
GPT-4 achieved 86.4% in March 2023, there has not been any significant progress on the benchmark.
Most recent frontier models like GPT-4-Turbo, Gemini-1.5-Pro, Claude, and LLaMA-3-400B (all
published in early-mid 2024) all settle at an accuracy between 86% - 87%. The recent published
GPT-4o has achieved remarkable performance boost (10+% improvement) on MATH [ 19 ], Chatbot
Arena [ 11 ]. However, it only obtains 1% improvement on MMLU to obtain 87.4%. This leads us to
re-examine the effectiveness of MMLU in measuring future (stronger) LLMs. Besides the saturation
issue, the performance on MMLU is also known to be highly sensitive to the prompt and scoring
function [ 47, 3 ], which causes significant order changes in the leaderboard. Here, we conjecture that
these issues are due to the following causes:
1. The questions in MMLU only have three distractor options. LLMs could potentially exploit
shortcuts to derive the answer without truly understanding the rationale. This could lead to an
overestimate of LLMs’ true performance, also leading to a degree of instability.
2. The questions in MMLU are mostly knowledge-driven without requiring too much reasoning,
especially in the STEM subjects, which reduces its difficulty. In fact, most models achieve better
performance with ‘direct’ answer prediction without chain-of-thought [41].
3. There is a portion of questions that are either unanswerable or mistakenly annotated. This dataset
noise leads to a lower ceiling, which the frontier models hit.
These issues have highlighted the need for more challenging, discriminative, and reliable datasets to
track the progress of LLMs. In this paper, we introduce MMLU-Pro: a comprehensive benchmark
designed for proficient-level multi-discipline language understanding and reasoning. MMLU-Pro
spans 14 diverse domains including mathematics, physics, chemistry, law, engineering, psychology,
and health, encompassing over 12,000 questions and thus meeting the breadth requirement. MMLU-
Pro is distinctive from MMLU in the following aspects:
1. MMLU-Pro has ten options, which contain 3x more distractors than MMLU. By increasing the
distractor numbers, we significantly reduce the probability of correct guess by chance to boost the
benchmark’s difficulty and robustness.
2. MMLU-Pro increases the portion of challenging college-level exam problems. These questions
require LLM to perform deliberate reasoning in different domains to derive the final answer.
3. We integrate two rounds of expert reviews to reduce the noise of the dataset. The first round is
based on expert verification. In the second round, we utilize the SoTA LLMs to identify potential
errors and employ annotators to perform more targeted verification.
We evaluated more than 50 LLMs including open-source and closed-source models, such as GPT-
4o [ 17 ], Claude-3-Opus [ 6], and Gemini [ 30 ], LLaMA-3 [ 35], Phi-3 [ 1] on MMLU-Pro. Our key
findings are summarized as follows:
1. MMLU-Pro presents significant challenges; notably, the leading model, GPT-4o, only achieves an
accuracy of 72.6% and GPT-4-Turbo reaches 63.7%, indicating substantial room for improvement.
2
2. MMLU-Pro is more discriminative than MMLU in distinguishing the nuances between models.
For example, the gap between GPT-4o and GPT-4-Turbo is 1% on MMLU, while it becomes 9%
on MMLU-Pro. This discriminative nature makes MMLU-Pro a more suitable benchmark.
3. Advanced open-source models like Llama-3-70B-Instruct [ 24] and DeepSeek-V2-Chat [ 15 ], while
not yet performing at the level of leading closed-source models such as GPT-4o and Claude-3-Opus,
have shown performance that is close to Claude-3-Sonnet.
4. MMLU-Pro necessitates chain-of-thought (CoT) [ 41 ] to achieve promising results. For instance,
CoT can boost the performance of GPT-4o by 19%. In contrast, CoT will actually hurt the
performance of models on MMLU. This reflects the necessity to perform deliberate reasoning on
MMLU-Pro, which is not required in the knowledge-driven MMLU questions.
5. Our error analysis on 120 erroneous cases of GPT-4o, the current top-performing model, reveals
that 39% of errors are due to flaws in the reasoning process, 35% stem from a lack of specific
domain expertise, and another 12% from computational errors. These results highlight the MMLU-
Pro benchmark’s difficulties and indicate areas needing further research and model enhancement.
2 Related Work
2.1 Large Language Models
Recent advancements in LLMs have significantly propelled the field of natural language processing.
GPT-3 [7 ] demonstrated robust few-shot prediction capabilities, interpreting tasks and examples from
natural language inputs. Subsequent models like InstructGPT [28 ], which employ human-feedback
reinforcement learning, have achieved strong user instruction-following capability. More recent
models including GPT-4o, GPT-4, Claude-3, Gemini, and Llama-3, have shown notable improvements
in complex reasoning across various domains. To rigorously assess and push the capabilities of these
LLMs, we introduce MMLU-Pro, a new benchmark designed to test the upper limits of reasoning
and knowledge in advanced language models.
2.2 LLMs Evaluation Benchmarks
In recent years, the development of various benchmarks has significantly enhanced the evaluation of
Large Language Models (LLMs). For instance, GLUE [ 37] and its successor SuperGLUE [38 ], have
played a pivotal role in advancing language understanding tasks, setting the stage for more specialized
evaluations. Other recent benchmarks, including MMLU [ 18], HELM [22 ], BigBench [32 ], Hel-
laSwag [45 ], and the AI2 Reasoning Challenge (ARC) [12], have broadened the scope by assessing
capabilities across language generation, knowledge understanding, and complex reasoning [9].
To facilitate performance comparison across diverse LLMs, several leaderboards have been es-
tablished, such as the OpenLLM Leaderboard [ 27] and OpenCompass [ 14 ]. However, as LLM
capabilities have rapidly advanced, leaderboard scores have become increasingly concentrated at the
top, with models like GPT-4 achieving near-perfect scores on multiple benchmarks. This trend high-
lights the urgent need for more challenging benchmarks to fully test the limits of LLM capabilities.
Recent studies have revealed that the performance of Large Language Models (LLMs) on current
benchmarks is not robust to minor perturbations [25, 31 ]. Specifically, slight variations in the style or
phrasing of prompts can lead to significant shifts in model scores. Beyond the inherent non-robustness
of the models themselves, the typical four-option format of multiple-choice questions (MCQs) also
contributes to this instability in model scoring. This format may not sufficiently challenge the models
or differentiate between closely performing systems, leading to potential overestimations of their
capabilities. Our new benchmark, MMLU-Pro, aims to address these issues by introducing more
complex questions and increasing the number of answer choices from four to ten, thus enhancing
performance differentiation and reducing the likelihood of correct guesses by chance.
3 The MMLU-Pro Benchmark
3.1 Overview
Our dataset comprises 14 discipline subsets, totaling 12,032 questions, with their distribution and
origins detailed in Figure 3. It integrates questions from several sources: (1) Original MMLU
3
Figure 2: MMLU-Pro Dataset Construction Pipeline11.2%10.8%
9.41%
9.15%
8.05%
7.68%
7.01% 6.8%
6.63%
6.56%
5.96%
4.15%
3.41%
3.17%
Math
Physics
Chemistry
Law
Engineering
Other
Economics
Health
Psychology
Business
Biology
Philosophy
Computer Science
Distribution of Disciplines in MMLU-Pro
(a) Distribution of Disciplines in MMLU-Pro56.6%
33.9%
4.97%
4.5%
Original MMLU Questions
STEM Website
TheoremQA
Scibench
Data Source Distribution in MMLU-Pro (b) Data Source Distribution in MMLU-Pro
Figure 3: Distribution of disciplines and question sources in the MMLU-Pro dataset
Questions, which form the core of our dataset and include questions adapted from the original MMLU
dataset with trivial and erroneous questions removed; (2) the STEM Website, providing selected
high-quality STEM problems from online platforms; (3) TheoremQA, featuring high-quality, human-
annotated questions that necessitate the application of theorems for resolution; and (4) SciBench,
which includes advanced science questions derived from college exams, ensuring the inclusion of
curriculum-aligned questions. For more details about detailed data statistics, prompt instructions, and
incorrect answer cases we eliminated, please refer to Appendix A.1.
3.2 Dataset Construction Pipeline
Initial Filtering As shown in Figure 2, our dataset construction begins with a comprehensive review
of the original MMLU dataset, which we streamline by merging the previous 57 subject categories
into 14 broader categories. This restructuring aims to better focus the evaluation on key areas of
knowledge and reduce redundancy. We also aim to eliminate overly simple questions that fail to
challenge the models effectively. We employ eight models: Llama-2-7B, Llama-2-7B-Chat, Llama-2-
13B, Llama-2-13B-Chat [35 ], Mistral-7B [20 ], Gemma-7B [34 ], Yi-6B, and Yi-6B-Chat [ 43]. Each
model is evaluated on the MMLU, and questions answered correctly by more than four models are
considered as “too easy” and subsequently excluded from consideration. Using this criterion, a total
of 5,886 questions are filtered out across the various subjects.
Question Collection and Integration We then expand our dataset by incorporating questions from
the STEM Website 2, TheoremQA [10], and SciBench [ 40]. It is important to note that the questions
from the STEM Website are in the form of problem statements with solutions, while those from
TheoremQA are in the format of questions accompanied by brief answers. To adapt these for our
dataset, we utilize GPT-4-Turbo3 to extract short answers from the solutions, serving as the correct
answer options. We also generate three additional distractors for each question. We then compare
the solutions with the extracted answers manually, to remove questions where the extracted answers
are incomplete or incorrect. This step is essential for aligning the STEM Website and TheoremQA
questions with those from other sources, preparing them for future option augmentation.
Option Augmentation We enhance the question options from four to ten using GPT-4-Turbo,
introducing six additional choices. These are not merely quantitative additions; rather, they serve
as plausible distractors that necessitate discerning reasoning for correct selection. This approach
2https://stemez.com/subjects
3In this paper, GPT-4-Turbo refers to GPT-4-turbo-2024-04-09. For more details, see https://platform.
openai.com/docs/models/gpt-4-turbo-and-gpt-4.
4
Table 1: Distribution of Issues Identified during the Expert Review Process
MMLU TheoremQA SciBench STEM Website
Incorrect Answer 350 0 11 483
False Negative Options 1953 5 15 293
Bad Question 385 1 15 862
significantly lowers the likelihood of correctly guessing an answer, thereby increasing both the
difficulty and the robustness of the benchmark. In experiments, we found that GPT-4-Turbo does not
gain additional advantage from such an augmentation procedure.
Expert Review The expert review process for our dataset construction comprises two main phases
to ensure its quality and reliability. Phase 1: Verification of Correctness and Appropriateness
involves experts verifying the accuracy of each answer, removing questions unsuitable for a multiple-
choice format, and discarding questions that lack necessary information or require non-textual
elements like images or tables. Phase 2: Ensuring Distractor Validity involves the Gemini-1.5-Pro
model re-evaluating all answer options to identify potential false negatives. In this context, a ‘false
negative’ refers to a correct option initially misclassified as incorrect. Subsequently, human experts
rigorously review these identified options to ensure that actual distractors are indeed incorrect and
distinctly different from the correct answer. In Table 1, we present a distribution of issues identified
during the expert review process. For better illustration, we categorize them into three types:
• Incorrect Answers: refer to instances where the provided answer is incorrect. There are two main
sources of these errors: the pre-existing errors within the original MMLU dataset and errors on the
STEM Website arising from flawed or incomplete answer extraction.
• False Negative Options: primarily arise from distractors generated in two key stages: converting
single answers into four options from sources like the STEM Website and TheoremQA, and further
expanding these four options to ten in the option augmentation phase. Human experts will remove
each False Negative Option, keeping the correct answer and suitable distractors. In our dataset,
83% have ten options, 17% have fewer, and the average options count per question is 9.47.
• Bad Questions: include several problematic aspects: (1) Questions that require non-text infor-
mation such as images or tables. (2) Questions that lack sufficient textual information to derive a
conclusive answer. (3) Questions that are unsuitable for a multiple-choice format, such as proof
problems, true or false questions, and open-ended questions.
4 Experimental Setup
Few-Shot Chain-of-Thought Prompting
We utilize a 5-shot Chain-of-Thought (CoT) [41 ] approach to measure model performance on
challenging tasks presented by MMLU-Pro. This CoT reasoning, adapted from Chain-of-Thought
Hub [ 16], incorporates essential reasoning steps from the original MMLU dataset. Our approach
introduces two enhancements: firstly, extending the original options available from the Chain-of-
Thought Hub, and secondly, selecting five representative demonstration examples for each discipline.
Unlike traditional performance measures such as Perplexity (PPL) which primarily focus on linguistic
probabilities, our method emphasizes reasoning capabilities, crucial for handling the complexities of
MMLU-Pro. A comprehensive comparison of performances using direct answering and CoT methods
will be presented in Section 6.2, demonstrating the effectiveness of the CoT approach.
Answer Extraction
To extract answers from the model-generated reasoning content in response to the MMLU-Pro
dataset, we initially use the regular expression ‘answer is \(?\([A-J]\)?\)’ to match the
format specified in the prompt instructions and few-shot examples. If this regex fails to retrieve a
valid response, possibly due to formatting deviations by the model, we employ a secondary regex
‘\.*\[aA\]nswer:\s*\([A-J]\)’ for a second attempt to extract the answer. If both of them fail
to retrieve a valid response, a fallback mechanism is implemented where a random option from the
answer choices is selected. This ensures consistent answer provision across all evaluations.
5
Table 2: Models Performance on MMLU-Pro, CoT. Values are accuracies in percentages. (All the
models use 5 shots except Gemini-1.5-pro and Gemini-1.5-Flash, which use 0 shots.)
Models Overall Math Physics Engineering History Law Psychology
Closed-source Models
GPT-4o [17] 72.6 76.1 74.7 55.0 70.1 51.0 79.2
Gemini-1.5-Pro [30] 69.0 72.8 70.4 48.7 65.6 50.8 77.2
Claude-3-Opus [13] 68.5 69.6 69.7 48.4 61.4 53.5 76.3
GPT-4-Turbo [2] 63.7 62.8 61.0 35.9 67.7 51.2 78.3
Gemini-1.5-Flash [30] 59.1 59.6 61.2 44.2 53.8 37.3 70.1
Yi-large [23] 58.1 64.8 57.0 45.4 49.6 36.2 50.6
Claude-3-Sonnet [13] 56.8 49.0 53.1 40.5 57.2 42.7 72.2
Open-source Models
Llama-3-70B-Instruct [24] 56.2 54.0 49.6 43.6 56.9 39.9 70.2
Phi-3-medium-4k-instruct [1] 55.7 52.2 49.4 37.9 57.2 38.3 73.4
DeepSeek-V2-Chat[15] 54.8 53.7 54.0 31.9 45.3 40.6 66.2
Llama-3-70B [24] 52.8 49.7 49.8 35.0 57.7 35.0 71.4
Qwen1.5-72B-Chat [5] 52.6 52.3 44.2 36.6 55.9 38.5 67.7
Yi-1.5-34B-Chat [43] 52.3 56.2 49.4 34.4 52.8 34.8 64.3
MAmmoTH2-8x7B-Plus[44] 50.4 50.3 45.7 34.0 50.9 35.5 63.8
Qwen1.5-110B [5] 49.9 50.4 41.4 35.3 54.1 35.1 66.3
Phi-3-mini-4k-instruct [1] 45.7 41.8 41.0 28.7 41.5 28.5 65.2
Mixtral-8x7B-Instruct-v0.1 [21] 43.3 36.3 39.9 29.2 44.6 32.1 63.4
Yi-34B [43] 43.0 31.8 35.0 32.6 52.0 32.7 62.5
Mixtral-8x7B-v0.1 [21] 41.0 34.1 37.2 27.9 47.5 27.1 61.0
Llama-3-8B-Instruct [24] 41.0 36.1 34.4 31.3 42.3 26.5 59.4
Starling-7B [49] 37.9 34