Title: MPP: MODEL POWER PARITY FOR REASONING LARGE LANGUAGE MODEL COMPARISON ON CROSS-DOMAIN BENCHMARKS
Primary area: datasets and benchmarks
Keywords: Large Language Models, Reasoning, Purchasing Power Parity, Evaluation

Abstract: The ability of Large Language Models is usually evaluated by their average performance scores on various benchmarks. However, this way of comparison does not consider the difficulty difference between benchmarks, and the average score also lacks interpretability. Inspired by the International Comparison Program (ICP), we introduce model power parity (MPP), a principled framework that adapts the idea of purchasing power parity (PPP) to LLM evaluation. MPP performs multilateral, direct comparisons among models and benchmarks without assuming a common difficulty scale. Extensive experiments show that the MPP framework provides reasonable evaluation results and brings new insight into the relation between models and benchmarks. We evaluate over 30 open-source reasoning large language models on 8 prevalent reasoning benchmarks. Our results reveal three key insights: (1) The idea of PPP suits well for building a ranking framework evaluating models' ability and benchmark difficulty. (2) MPP manages to infer missing performance scores matching actual inference results within about 0.5 to 3 points. (3) MPP helps distinguish whether a benchmark is suitable for evaluating a given model family. We believe that MPP offers a fresh, economically grounded perspective on equitable LLM capability assessment and will facilitate more reliable model selection and benchmark building.