MMLU-Pro: A More Robust and Challenging Multi-Task Language
  Understanding Benchmark
Authors: name: Yubo Wang; name: Xueguang Ma; name: Ge Zhang; name: Yuansheng Ni; name: Abhranil Chandra; name: Shiguang Guo; name: Weiming Ren; name: Aaran Arulraj; name: Xuan He; name: Ziyan Jiang; name: Tianle Li; name: Max Ku; name: Kai Wang; name: Alex Zhuang; name: Rongqi Fan; name: Xiang Yue; name: Wenhu Chen
Published: 2024-06-03 17:53:00

Abstract: In the age of large-scale language models, benchmarks like the Massive
Multitask Language Understanding (MMLU) have been pivotal in pushing the
boundaries of what AI can achieve in language comprehension and reasoning
across diverse domains. However, as models continue to improve, their
performance on these benchmarks has begun to plateau, making it increasingly
difficult to discern differences in model capabilities. This paper introduces
MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven
MMLU benchmark by integrating more challenging, reasoning-focused questions and
expanding the choice set from four to ten options. Additionally, MMLU-Pro
eliminates the trivial and noisy questions in MMLU. Our experimental results
show that MMLU-Pro not only raises the challenge, causing a significant drop in
accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability
under varying prompts. With 24 different prompt styles tested, the sensitivity
of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in
MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT)
reasoning achieved better performance on MMLU-Pro compared to direct answering,
which is in stark contrast to the findings on the original MMLU, indicating
that MMLU-Pro includes more complex reasoning questions. Our assessments
confirm that MMLU-Pro is a more discriminative benchmark to better track
progress in the field.