Are We Done with MMLU?
Authors: name: Aryo Pradipta Gema; name: Joshua Ong Jun Leang; name: Giwon Hong; name: Alessio Devoto; name: Alberto Carlo Maria Mancino; name: Rohit Saxena; name: Xuanli He; name: Yu Zhao; name: Xiaotang Du; name: Mohammad Reza Ghasemi Madani; name: Claire Barale; name: Robert McHardy; name: Joshua Harris; name: Jean Kaddour; name: Emile van Krieken; name: Pasquale Minervini
Published: 2024-06-06 14:49:06

Abstract: Maybe not. We identify and analyse errors in the popular Massive Multitask
Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted,
our analysis demonstrates numerous ground truth errors that obscure the true
capabilities of LLMs. For example, we find that 57% of the analysed questions
in the Virology subset contain errors. To address this issue, we introduce a
comprehensive framework for identifying dataset errors using a novel error
taxonomy. Then, we create MMLU-Redux, which is a subset of 3,000 manually
re-annotated questions across 30 MMLU subjects. Using MMLU-Redux, we
demonstrate significant discrepancies with the model performance metrics that
were originally reported. Our results strongly advocate for revising MMLU's
error-ridden questions to enhance its future utility and reliability as a
benchmark. Therefore, we open up MMLU-Redux for additional annotation
https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux.