Title: Model Utility Law: Evaluating LLMs beyond Performance via Mechanistically Interpretable Metric
Primary area: interpretability and explainable AI
Keywords: LLMs, evaluation

Abstract: Conventional performance scores capture only a fraction of a large language model's (LLM) true ability, leaving real-world capability under-evaluated. To close this gap, we advocate a new generalizable evaluation paradigm that estimates a model's unseen capacities—those not covered by testing samples.
We introduce the Model Utilization Index (MUI) to realize this paradigm. Inspired by mechanistic interpretability, MUI measures the proportion of neurons or features activated during inference, quantifying the effort a model expends on a task. Our central hypothesis is that a stronger model should achieve higher performance while exhibiting lower utilization, reflecting efficient use of its internal capacity.
Experiments across diverse LLMs reveal an inverse relationship between MUI and accuracy, establishing the Utility Law and several corollaries that guide pre-/post-training strategies, enable fairer ranking, and highlight correlations with data diversity. MUI thus provides a scalable, mechanism-aware foundation for evaluating LLMs beyond traditional benchmarks. Our code can be found at https://anonymous.4open.science/r/MUI

TL;DR: a new metric and law for generalizable evaluation