Summarize the given content.




MMLU-Pro Leaderboard
Introduction

We introduce MMLU-Pro, an enhanced benchmark designed to evaluate language understanding models across broader and more challenging tasks. Building on the Massive Multitask Language Understanding (MMLU) dataset, MMLU-Pro integrates more challenging, reasoning-focused questions and increases the answer choices per question from four to ten, significantly raising the difficulty and reducing the chance of success through random guessing. MMLU-Pro comprises over 12,000 rigorously curated questions from academic exams and textbooks, spanning 14 diverse domains including Biology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Math, Philosophy, Physics, Psychology, and Others.
What's new about MMLU-Pro

Compared to the original MMLU, there are three major differences:

    The original MMLU dataset only contains 4 options, MMLU-Pro increases it to 10 options. The increase in options will make the evaluation more realistic and challenging. The random guessing will lead to a much lower score.
    The original MMLU dataset contains mostly knowledge-driven questions without requiring much reasoning. Therefore, PPL results are normally better than CoT. In our dataset, we increase the problem difficulty and integrate more reasoning-focused problems. In MMLU-Pro, CoT can be 20% higher than PPL.
    By increasing the distractor numbers, we significantly reduce the probability of correct guess by chance to boost the benchmark’s robustness. Specifically, with 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro.

For detailed information about the dataset, visit our page on Hugging Face: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro.

If you are interested in replicating these results or wish to evaluate your models using our dataset, access our evaluation scripts available on GitHub: https://github.com/TIGER-AI-Lab/MMLU-Pro.

If you would like to learn more details about our dataset, please check out our paper: https://arxiv.org/abs/2406.01574.

Below you can find the accuracies of different models tested on this dataset.
Models
Data Source
Overall
Biology
Business
Chemistry
Computer Science
Economics
Engineering
Health
History
Law
Math
Philosophy
Physics
Psychology
Other

DeepSeek-Coder-V2-Lite-Instruct
	

Self-Reported
	
0.7824
	
0.8912
	
0.8162
	
0.8012
	
0.8073
	
0.8365
	
0.6471
	
0.7726
	
0.7664
	
0.6431
	
0.8201
	
0.7711
	
0.8014
	
0.8183
	
0.8019
ModelsData SourceOverallBiologyBusinessChemistryComputer ScienceEconomicsEngineeringHealthHistoryLawMathPhilosophyPhysicsPsychologyOther

Arx-0.3

TIGER-Lab
0.7824 0.8912 0.8162 0.8012 0.8073 0.8365 0.6471 0.7726 0.7664 0.6431 0.8201 0.7711 0.8014 0.8183 0.8019

Claude-3.5-Sonnet

TIGER-Lab
0.7612 0.8856 0.8023 0.773 0.7976 0.8246 0.6153 0.7531 0.7585 0.6385 0.7683 0.7475 0.7667 0.8221 0.7846

Grok-2

Self-Reported
0.7546 0.8842 0.7896 0.7703 0.7585 0.8187 0.6078 0.7592 0.6982 0.6167 0.7927 0.7234 0.7729 0.8133 0.7662

GPT-4o (2024-08-06)

TIGER-Lab
0.7468 0.8926 0.801 0.727 0.7829 0.8164 0.5531 0.7604 0.7323 0.5895 0.7942 0.7034 0.7506 0.8271 0.7955

GPT-4o (2024-05-13)

TIGER-Lab
0.7255 0.8675 0.7858 0.7393 0.7829 0.808 0.55 0.7212 0.7007 0.5104 0.7609 0.7014 0.7467 0.7919 0.7748

Grok-2-mini

Self-Reported
0.7185 0.8465 0.7566 0.7429 0.7317 0.8092 0.5624 0.7518 0.6719 0.5367 0.7609 0.6553 0.7328 0.7994 0.7197

Qwen2.5-72B

Self-Reported
0.7159 0.8326 0.7693 0.7314 0.7488 0.8104 0.5645 0.6956 0.6745 0.4914 0.812 0.6473 0.7498 0.7857 0.71

Gemini-1.5-Pro-002

TIGER-Lab
0.7025 0.8645 0.8094 0.6221 0.7122 0.8171 0.5899 0.7479 0.7008 0.5522 0.5174 0.7234 0.8072 0.8294 0.7359

Qwen2.5-32B

Self-Reported
0.6923 0.8396 0.7567 0.7032 0.739 0.7725 0.548 0.6932 0.5932 0.4541 0.8053 0.6152 0.7259 0.7569 0.6645

Gemini-1.5-Pro

Self-Reported
0.6903 0.8466 0.7288 0.7032 0.7293 0.7844 0.4871 0.7274 0.6562 0.5077 0.7276 0.6172 0.7036 0.772 0.7251

Claude-3-Opus

TIGER-Lab
0.6845 0.8507 0.7338 0.693 0.6902 0.798 0.484 0.6845 0.6141 0.5349 0.6957 0.6352 0.6966 0.7631 0.6991

DeepSeek-Chat-V2_5

TIGER-Lab
0.6583 0.8271 0.7364 0.6979 0.7098 0.7678 0.517 0.6247 0.5564 0.3715 0.7535 0.5631 0.7052 0.7268 0.6385

Qwen2-72B-Chat

TIGER-Lab
0.6438 0.8107 0.6996 0.5989 0.6488 0.7589 0.6724 0.4603 0.6781 0.4587 0.7098 0.5892 0.6089 0.7669 0.6652

Gemini-1.5-Flash-002

TIGER-Lab
0.6409 0.8368 0.7145 0.6708 0.6341 0.7628 0.407 0.6284 0.5932 0.4286 0.6255 0.6052 0.7141 0.7623 0.6453

magnum-72b-v1

TIGER-Lab
0.6393 0.8219 0.6339 0.5967 0.7116 0.7497 0.4847 0.6626 0.6706 0.4378 0.6737 0.6017 0.602 0.7657 0.6692

GPT-4-Turbo

TIGER-Lab
0.6371 0.8243 0.673 0.5592 0.6854 0.7476 0.3591 0.7078 0.6772 0.5123 0.6277 0.6433 0.6097 0.7832 0.7186

Qwen2.5-14B

Self-Reported
0.6369 0.7978 0.7085 0.6873 0.6707 0.731 0.4954 0.6222 0.5774 0.366 0.6788 0.5711 0.6844 0.7243 0.6288

DeepSeek-Coder-V2-Instruct

TIGER-Lab
0.6363 0.7657 0.7326 0.6686 0.6878 0.7464 0.5175 0.6112 0.5184 0.3506 0.6342 0.5621 0.6813 0.7206 0.6537

Higgs-Llama-3-70B

Self-Reported
0.6316 0.8354 0.6743 0.6034 0.6902 0.7512 0.4737 0.6687 0.6404 0.4432 0.6321 0.5591 0.5989 0.7619 0.6613

GPT-4o-mini

TIGER-Lab
0.6309 0.802 0.706 0.6299 0.6707 0.7334 0.3942 0.676 0.5879 0.3724 0.7232 0.5591 0.6366 0.7381 0.6613

Llama-3.1-70B-Instruct

TIGER-Lab
0.6284 0.8117 0.6641 0.591 0.6634 0.7524 0.4582 0.6846 0.6614 0.4696 0.6047 0.6172 0.5912 0.7556 0.6602

RRD2.5-9B

Self-Reported
0.6184 0.7824 0.682 0.6078 0.6561 0.7204 0.4857 0.6198 0.5381 0.3733 0.7365 0.5471 0.602 0.7118 0.6082

Reflection-Llama-3.1-70B

TIGER-Lab
0.6035 0.795 0.6324 0.5433 0.6268 0.737 0.4396 0.6907 0.6194 0.4242 0.6136 0.5772 0.5327 0.7444 0.6504

Gemini-1.5-Flash

TIGER-Lab
0.5912 0.8131 0.667 0.613 0.5951 0.6943 0.4416 0.6039 0.538 0.3732 0.5958 0.4949 0.612 0.7005 0.58

Yi-large

TIGER-Lab
0.5809 0.6987 0.6413 0.6166 0.6341 0.6813 0.4541 0.6443 0.4961 0.3624 0.6481 0.5531 0.5704 0.5063 0.6472
·
Built with Gradio logo