Summarise: Experimental Setup
Few-Shot Chain-of-Thought Prompting
We utilize a 5-shot Chain-of-Thought (CoT) [41 ] approach to measure model performance on
challenging tasks presented by MMLU-Pro. This CoT reasoning, adapted from Chain-of-Thought
Hub [ 16], incorporates essential reasoning steps from the original MMLU dataset. Our approach
introduces two enhancements: firstly, extending the original options available from the Chain-of-
Thought Hub, and secondly, selecting five representative demonstration examples for each discipline.
Unlike traditional performance measures such as Perplexity (PPL) which primarily focus on linguistic
probabilities, our method emphasizes reasoning capabilities, crucial for handling the complexities of
MMLU-Pro. A comprehensive comparison of performances using direct answering and CoT methods
will be presented in Section 6.2, demonstrating the effectiveness of the CoT approach.
Answer Extraction
To extract answers from the model-generated reasoning content in response to the MMLU-Pro
dataset, we initially use the regular expression ‘answer is \(?\([A-J]\)?\)’ to match the
format specified in the prompt instructions and few-shot examples. If this regex fails to retrieve a
valid response, possibly due to formatting deviations by the model, we employ a secondary regex
‘\.*\[aA\]nswer:\s*\([A-J]\)’ for a second attempt to extract the answer. If both of them fail
to retrieve a valid response, a fallback mechanism is implemented where a random option from the
answer choices is selected. This ensures consistent answer provision across all evaluations.
5
Table 2: Models Performance on MMLU-Pro, CoT. Values are accuracies in percentages. (All the
models use 5 shots except Gemini-1.5-pro and Gemini-1.5-Flash, which use 0 shots.)
Models Overall Math Physics Engineering History Law Psychology
Closed-source Models
GPT-4o [17] 72.6 76.1 74.7 55.0 70.1 51.0 79.2
Gemini-1.5-Pro [30] 69.0 72.8 70.4 48.7 65.6 50.8 77.2
Claude-3-Opus [13] 68.5 69.6 69.7 48.4 61.4 53.5 76.3
GPT-4-Turbo [2] 63.7 62.8 61.0 35.9 67.7 51.2 78.3
Gemini-1.5-Flash [30] 59.1 59.6 61.2 44.2 53.8 37.3 70.1
Yi-large [23] 58.1 64.8 57.0 45.4 49.6 36.2 50.6
Claude-3-Sonnet [13] 56.8 49.0 53.1 40.5 57.2 42.7 72.2
Open-source Models
Llama-3-70B-Instruct [24] 56.2 54.0 49.6 43.6 56.9 39.9 70.2
Phi-3-medium-4k-instruct [1] 55.7 52.2 49.4 37.9 57.2 38.3 73.4
DeepSeek-V2-Chat[15] 54.8 53.7 54.0 31.9 45.3 40.6 66.2
Llama-3-70B [24] 52.8 49.7 49.8 35.0 57.7 35.0 71.4
Qwen1.5-72B-Chat [5] 52.6 52.3 44.2 36.6 55.9 38.5 67.7
Yi-1.5-34B-Chat [43] 52.3 56.2 49.4 34.4 52.8 34.8 64.3
MAmmoTH2-8x7B-Plus[44] 50.4 50.3 45.7 34.0 50.9 35.5 63.8
Qwen1.5-110B [5] 49.9 50.4 41.4 35.3 54.1 35.1 66.3
Phi-3-mini-4k-instruct [1] 45.7 41.8 41.0 28.7 41.5 28.5 65.2
Mixtral-8x7B-Instruct-v0.1 [21] 43.3 36.3 39.9 29.2 44.6 32.1 63.4
Yi-34B [43] 43.0 31.8 35.0 32.6 52.0 32.7 62.5
Mixtral-8x7B-v0.1 [21] 41.0 34.1 37.2 27.9 47.5 27.1 61.0
Llama-3-8B-Instruct [24] 41.0 36.1 34.4 31.3 42.3 26.5 59.4
Starling-7B [49] 37.9 34.9 38.5 27.0 43.6 24.7 32.5
c4ai-command-r-v01 [8] 37.9 26.3 28.3 24.8 47.5 34.0 58.5
Llama-2-70B [35] 37.5 26.8 28.2 23.5 45.9 28.6 59.0
OpenChat-3.5-8B [39] 37.2 36.2 30.5 26.9 39.9 24.6 54.5
InternMath-20B-Plus [42] 37.1 56.1 24.0 30.4 20.5 15.2 42.3
Llama3-Smaug-8B [29] 36.9 33.2 37.3 19.8 42.0 26.5 28.6
Llama-3-8B [24] 35.4 30.4 31.4 25.5 36.2 19.6 53.3
Gemma-7B [34] 33.7 25.1 27.6 22.7 36.8 21.7 51.8
InternMath-7B-Plus [42] 33.5 48.3 22.8 28.2 19.2 14.4 38.3
Zephyr-7B-Beta [36] 33.0 23.6 35.7 23.9 32.0 22.0 28.2
Mistral-7B-v0.1 [20] 30.9 23.5 24.8 22.4 32.6 20.7 48.9
Neo-7B-Instruct [46] 28.7 35.5 23.7 19.1 28.2 18.0 36.2
Llemma-7B [4] 23.5 21.6 25.7 23.8 15.2 14.8 29.6
Gemma-2B [34] 15.9 16.3 15.6 12.7 15.4 12.3 16.1
5 Results and Analysis
Table 2 showcases the performance of frontier models on the MMLU-Pro benchmark. Due to space
constraints, we selected a subset of models and representative domains, including three reasoning-
focused subjects (Mathematics, Physics, Engineering) and three knowledge-heavy subjects (History,
Law, Psychology). Full results are available on our leaderboard 4.
5.1 Overall Performance
GPT-4o [17 ] emerges as the strongest model with an overall accuracy of 72.6%, showing superior
performance across all subjects. Additionally, Phi-3-medium-4k-instruct (14B parameters) and Phi-3-
mini-4k-instruct (3.8B parameters) perform exceptionally well, possibly due to their pre-training on
high-quality educational data and code.
Additionally, Results from Table 2 indicate that top-tier closed-source models outperform the open-
source ones. Among the leading open-source models, Llama-3-70B-Instruct performs the best,
achieving an accuracy of 56.2%, close to that of Yi-Large and Claude-3-Sonnet. However, it still
significantly lags behind GPT-4o and Claude-3-Opus in all subjects.
4Please visit our leaderboard at https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro.
6
5.2 Subject-Specific Insights
Math and Physics: In computation and reasoning-intensive subjects like Math and Physics, we
observe significant performance disparities among models. The gap stretches from over 70% accuracy
for GPT-4o to just over 20% for Mistral-7B-v0.1, illustrating a wide range in capabilities and
underscoring the value of our benchmark in distinguishing these differences.
History and Psychology: In knowledge-intensive subjects such as History and Psychology, models
generally show a higher performance floor compared to reasoning-intensive disciplines. Interestingly,
the DeepSeek-V2-Chat model underperforms relative to its peers in these subjects, indicating its
comparatively stronger reasoning abilities over its knowledge retrieval capabilities.
Engineering and Law: Among the 14 subjects evaluated, Engineering and Law consistently scored
lower. Upon reviewing model outputs, we found that the lower scores in Engineering are largely due
to the addition of new questions sourced from the STEM Website, which require complex formula
derivations and multi-step calculations. This aspect leaves substantial room for improvement in
future, more advanced models. Law scores suffer as questions become more intricate and detailed
with additional options, necessitating a deeper understanding of legal reasoning.
5.3 Error Analysis
In this section, we explore an error analysis of GPT-4o, currently the best-performing model on the
MMLU-Pro benchmark, to examine its performance strengths and weaknesses. This examination not
only highlights areas where the model falls short but also provides insights that could inform future
improvements in both its architecture and training processes. We conducted a detailed review of 120
randomly selected erroneous predictions made by GPT-4o. These errors were analyzed by expert
annotators who determined the underlying causes of each misprediction using their expert judgment.
Specific cases and further detailed discussions are provided in Appendix A.6.
Reasoning Errors (39%) The model frequently encounters difficulties with logical reasoning,
even when it recalls the correct information and knowledge. These issues often arise from logical
inconsistencies in its responses, likely due to its dependence on recognizing patterns in training data
rather than engaging in a true understanding of the problem.
Lack of Specific Knowledge (35%) A fundamental root cause of domain-specific errors in the
GPT-4o model is the lack of specialized knowledge. Errors such as incorrect financial calculations
and misapplications of optical principles highlight this issue.
Calculation Errors (12%) We distinguish calculation errors from reasoning errors to aid model
developers, as many AI systems can effectively utilize calculators or Python for complex, multi-step
calculations. In our review of error cases, it is common to find instances where the model has the
correct formula but makes errors in computing values.
Other Errors The remaining errors include No Selection Made (5%), Question Understanding
Errors (4%), Generation Issues (2%), Annotation Errors (2%), and Answer Extraction Errors (1%).
These errors are attributed to various factors, such as limitations in final response selection, complex
text interpretation challenges, limitations in response generation, inaccuracies in data annotation, and
issues in extracting precise answers from model outputs.
6 Comparison with MMLU
In this section, we will compare the MMLU and MMLU-Pro benchmarks from three perspectives:
difficulty level, reasoning strength, and robustness degree.
6.1 Difficulty Level
In Figure 4, we present scores of different models on both MMLU and MMLU-Pro benchmarks. It is
evident that as language model capabilities enhance, the scores on MMLU are not only increasing
but also clustering closely together, making it difficult to distinguish between models. For instance,
7
GPT-4o
Claude-3-Opus
GPT-4-Turbo
Gemini-1.5-Flash
Llama-3-70B-Instruct
Phi-3-medium-4k-
instruct
Qwen1.5-110B
Yi-34B
Llama-2-70B
Gemma-7B0
20
40
60
80
Accuracy (%)
88.7 86.8 86.5
78.9 82.0
78.0 80.2
76.3
69.7 66.0
72.5
68.5
63.7
59.1 56.2 53.5 49.9
43.0
37.5
33.7
MMLU
MMLU-ProFigure 4: Performance Comparison: MMLU vs.
MMLU-ProLlama-2-7b-hf
Mistral-7B-v0.1
Mistral-7B-
Instruct-v0.2
gemma-7b
Meta-Llama-3-8B
Meta-Llama-3-8B-
Instruct
10
20
30
40
50
60
70
80
Performance Range (%)
34.2
45.2
14.8
18.6
57.3
61.5
27.5
28.7
54.5
60.3
22.4
25.5
58.7
62.2
25.2
27.0
59.9
64.8
28.7
31.5
61.4
66.0
33.1
35.2
MMLU
MMLU-Pro Figure 5: Performance Variability under Differ-
ent Prompts on MMLU and MMLU-Pro
Table 3: Accuracy Differences of Models Between CoT and Direct Answering (%)
Model Name MMLU MMLU-Pro
CoT Direct Answer CoT - DA CoT Direct Answer CoT - DA
GPT-4o 88.7 87.2 1.5 72.6 53.5 19.1
GPT-4-Turbo 86.5 86.7 -0.2 63.7 48.4 15.3
Phi3-medium-4k-instruct 79.4 78.0 1.4 55.7 47.5 8.2
Llama-3-8B 62.7 66.6 -3.9 35.4 31.5 3.9
Gemma-7B 62.4 66.0 -3.6 33.7 27.0 6.7
models like Gemini-1.5-Flash, Llama-3-70B-Instruct, Phi-3-medium-4k-instruct, and Qwen1.5-110B
all score between 78% and 82%, a narrow 4% range that encompasses four models, challenging the
differentiation of their performance. MMLU-Pro expands this range to approximately 10%. Similarly,
the score difference between models like GPT-4o, Claude-3-Opus, and GPT-4-Turbo has widened
from about 2% on MMLU to around 9% on MMLU-Pro. Additionally, the increased difficulty in
MMLU-Pro ensures ample room for future model improvement. Currently, the best-performing
model, GPT-4o, scores 72.6% on MMLU-Pro, leaving a substantial margin of 27.4% for potential
improvement, whereas MMLU offers only about 11.3% space for further enhancement.
6.2 Reasoning Level
According to Table 3, we can observe differences in performance between the Chain of Thought
(CoT) method and Direct Answering (DA) across various models on MMLU and MMLU-Pro.
The comparison shows that the CoT method generally results in more significant performance
improvements on MMLU-Pro compared to MMLU.
Specifically, GPT-4o improves by 1.5% using the Chain of Thought (CoT) method compared to
direct answering on MMLU, while on MMLU-Pro, its improvement reaches 19.1%. Similarly,
GPT-4-Turbo shows a 15.3% increase in performance using CoT over direct answering on MMLU-
Pro, although its performance slightly decreases by 0.2% on MMLU. Other models such as Phi3-
medium-4k-instruct, Llama-3-8B, and Gemma-7B also display similar trends, exhibiting greater
performance improvements using CoT on MMLU-Pro compared to direct answering. These findings
indicate that the MMLU-Pro benchmark is specifically designed to assess deeper and more complex
reasoning skills, as evidenced by the enhanced performance of models using chain-of-thought (CoT),
highlighting its focus on professional-level problem-solving.
6.3 Robustness Degree
It is widely recognized that even minor variations in prompts can significantly impact model outputs,
leading to substantial fluctuations when evaluating models. This poses challenges for accurately
ranking models and maintaining consistent leaderboards [ 3]. This sensitivity is generally attributed to
8