Title: The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks
Primary area: datasets and benchmarks
Keywords: Benchmark reliability, meta-evaluation of benchmarks, evaluation reliability, diagnostic evaluation

Abstract: Benchmarks shape scientific conclusions about model capabilities and steer model development. This creates a feedback loop: stronger benchmarks drive better models, and better models demand more discriminative benchmarks. Ensuring benchmark reliability is therefore essential for trustworthy evaluation and meaningful progress. In this work, we study benchmark reliability from a \emph{distributional} perspective and introduce benchmark harmony, which measures \textit{how uniformly a model's performance is distributed across the subdomains of a benchmark}. We posit that high harmony is a desirable benchmark property, indicating that the aggregate metric reflects uniform competence across subdomains. Across 19 multiple-choice benchmarks and five model families, we map each benchmark onto a mean-variance plane of harmony computed across models, where high mean and low variance signal more reliable evaluation. Our analysis shows that less harmonious benchmarks can give misleading results, since overall accuracy may be disproportionately influenced by specific subdomains. For instance, \emph{ARC-Easy} is overwhelmed by questions on \emph{Biological Concepts}, overshadowing other critical subdomains such as Geography, Physics, Chemistry, and Environmental Science. By recommending that harmony should be reported alongside accuracy, we reframe evaluation from 
simple performance averages to a more robust, distributionally reliable measurement of performance.