Title: Metritocracy: Representative Metrics for Lite Benchmarks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Evaluation

benchmarks

social choice
Abstract: A common problem in LLM evaluation is how to choose a subset of metrics from a full suite of possible metrics. Subset selection is usually done for efficiency or interpretability reasons, and the goal is often to select a "representative" subset of metrics. However, "representative" is rarely clearly defined. In this work, we use ideas from social choice theory to formalize two notions of representation for the selection of a subset of evaluation metrics. We first introduce *positional representation*, which guarantees every alternative is sufficiently represented at every position cutoff. We then introduce *positional proportionality*, which guarantees no alternative is proportionally over- or under-represented by more than a small error at any position. We prove upper and lower bounds on the smallest number of metrics needed to guarantee either of these properties in the worst case. We also study a generalized form of each property that allows for additional input on groups of metrics that must be represented. Finally, we tie theory to practice through real-world case studies on both LLM evaluation and hospital quality evaluation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of selecting a \"representative\" subset of metrics from a full suite for efficient evaluation, particularly in contexts like LLM assessment and hospital quality evaluation, where using all metrics is computationally expensive or hard to interpret. Key contributions include:  \n1. Formalizing two notions of representation: Positional representation ensures every alternative is sufficiently represented at each rank cutoff, parameterized by a group size $ g $; positional proportionality guarantees no alternative is over- or under-represented by more than an additive error $ \\epsilon $ at any position.  \n2. Theoretical bounds: For positional representation, tight (up to logarithmic factors) upper and lower bounds on the minimum number of metrics needed are established, with a polynomial-time greedy algorithm achieving the upper bound. For positional proportionality, tight (up to constants) bounds are provided, and it is shown to approximate social choice scoring rules.  \n3. Generalizations: Extending both properties to handle pre-defined groups of metrics, with proofs that finding the smallest satisfying subset is NP-hard, while bounds from the base cases extend to these generalizations.  \n4. Empirical validation: Case studies on BIG-bench, HELM, and hospital quality evaluation demonstrate that the proposed methods outperform existing \"lite\" subsets in efficiency while maintaining representativeness, and can augment existing subsets to improve performance.\n\nSTRENGTHS AND WEAKNESSES:\nRefer to Questions\n\nQUESTIONS:\n1. The paper highlights the lack of a formal definition for \"representativeness\" in metric subset selection, while existing lite benchmarks (e.g., BIG-bench Lite, HELM Lite) rely on manual curation. What specific flaws in representativeness do these existing subsets exhibit? For example, are there systematic under-representations of certain alternatives? Does the paper quantify these flaws using empirical data?  \n2. The paper draws parallels between LLM evaluation and hospital quality assessment as scenarios requiring representative metric subsets. However, LLM metrics and hospital metrics differ significantly in semantics, distribution, and evaluation goals. How does the paper justify generalizing the proposed framework across such diverse domains? Are adjustments to parameters (e.g., $g$ or $\\epsilon$) needed for domain-specific contexts?  \n3. Positional Representation requires that if an alternative $a$ is ranked in the top $r^2$ positions by $\\ell \\cdot g$ metrics in $N$, it must be ranked in the top $r$ positions by at least $\\ell$ metrics in $K$. Why is $r^2$ chosen as the positional mapping? Is this choice theoretically validated or empirically tested? How would switching to a linear relationship $r$ affect the theoretical bounds (upper and lower)?  \n4. The paper claims that Positional Proportionality approximates any social choice scoring rule with an error bounded by $\\epsilon$. However, scoring rules weight ranks differently, Borda assigns higher weights to top positions. How does Positional Proportionality adapt to such weight variations? Are there scoring rules that cannot be effectively approximated?  \n5. The greedy algorithm for Positional Representation Alg 1 uses a \"coloring\" step where every $g$ uncolored instances of an alternative are labeled with the same color. What is the rationale for this coloring strategy? Could alternative strategies, e.g., grouping by rank positions, yield better subsets?  \n6. Positional Proportionality uses an additive error $\\epsilon$, with bounds scaling with $1/\\epsilon^2 \\log(m)$. Why is additive error preferred over multiplicative error? In domains like hospital quality assessment, relative errors in metric proportions may be more critical—does the framework need extension to support multiplicative error?  \n7. The paper notes that Positional Representation and Positional Proportionality are \"mutually non-implied,\" with examples showing Positional Proportionality can be more efficient, e.g., $|K|=1$ when all metrics agree. However, in scenarios with highly conflicting rankings, which property better ensures representativeness? Can experimental data compare their applicability?  \n8. Figure 1 shows \"minimum $g$ for existing subsets\", but the calculation method is unclear. How is the smallest $g$ for which an existing subset satisfies Positional Representation determined? Does this account for biases in manual curation?  \n9. In the hospital quality case study, adding <5 metrics to the existing 12 significantly reduces $\\epsilon$. What criteria guided the selection of these additional metrics? Was domain knowledge incorporated? Would results remain robust if selection were fully algorithmic?\n\nLIMITATIONS:\nRefer to Questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces and studies the problem of finding a small representative set of metrics $K$ from a larger set of metrics $N$ specified by the user. The goal is for $K$ to be a ``good approximation’’ of $N$. Drawing on ideas from computational social choice, the authors formalize two notions of “representativeness” for $K\\subseteq N$. The first notion – positional representation – requires that for every rank cutoff $r$ and alternative $a$, if at least $g\\ell$ of the $n$ metrics rank $a$ in the top $r$, then at least $\\ell$ metrics in $K$ must do so too. The second notion, positional proportionality, extends this to a two-sided requirement. The authors also extend both notions to. The authors also extend both notions to The setting where one has to be representative of different subsets of metrics $G_1, G_2,\\dots\\subseteqq N$.\n\nFor the first notion, the paper shows that a lower bound on the size of the selected set K and an efficient algorithm that matches this size up to a logarithmic factor. For the second notion, the paper provides lower and upper bounds on the size of the select set of metrics required, and matches it where the lower and upper bounds match up to constant factors. \n\nThere seems to be an interesting fundamental problem in social choice theory that has not received much attention, to the best of my knowledge. The authors claim that this also has applications in evaluating LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper introduces an interesting problem in social choice theory and provides clean theoretical results for it.   \n2. The paper is well-written and easy to follow.   \n3. The paper evaluates its proposed solutions using carefully designed empirical studies.\n\nWeaknesses:\n\n1. While the authors chiefly motivate their problem through its applications in LLM evaluation, I think its application is weak. And instead of focusing on the fundamental nature of the problem might make the paper more grounded and interesting.\n\nQUESTIONS:\nIn the paper, different notions of positional representation typically look at item-wise guarantees. Is it possible to extend the techniques in this work when we require, for instance, pairs of items to be treated similarly by the sub-small subset of metrics selected and the large input subset of metrics? \n\nSince the notions of representation studied in this work assume that the underlying metrics output a ranking, I think they cannot directly be applied to metrics which output scores for individuals, such as Borda. Can the authors comment on extensions to such score-based voting rules?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of selecting a representative subset of evaluation metrics from a larger set, particularly in settings like LLM or hospital quality evaluations. The authors formalize two new concepts -- positional representation and positional proportionality-- inspired by social choice theory. They derive tight upper/lower bounds, prove NP-hardness, and propose practical algorithms for subset selection. Their approach is validated via three real-world case studies: BIG-bench, HELM, and Cal Hospital Compare.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Quality: Strong theoretical analysis with tight bounds and clear proofs; algorithms are sound and practically applicable.\n2. Originality: Introduces novel formalizations of metric representativeness based on social choice theory—fresh perspective in benchmarking.\n3. Significance: Addresses a real need in scalable, interpretable benchmark design; applicable to various domains.\n4. Clarity: Generally well-written and organized, with intuitive examples and thoughtful discussion.\n\n## Weaknesses\n1. Bias inheritance from full metric set: The method assumes the full set of metrics is well-balanced. If the original set is biased or overrepresents certain aspects, the selected subset will likely reflect those biases as well.\n2. Limited robustness to distribution shift: While the subset can be efficiently recomputed, it may still fail to generalize well when evaluating substantially different or unseen alternatives. In such cases, user intervention (e.g., choosing new values for g or \\epsilon) may be needed repeatedly.\n3. Scalability of IP: Integer programming guarantees optimal subsets but may become infeasible in large-scale settings with hundreds or thousands of metrics.\n4. Assumption of complete rankings: The framework assumes that all metrics provide complete rankings over alternatives, which is not always true in real-world LLM evaluations where missing data is common.\n\nQUESTIONS:\n1. Incorporating metric importance or cost.\n\nHow would the framework extend to settings where metrics have differing levels of importance, quality, or computational cost?\n\n2. Integrating other constraints.\n\nCan fairness, diversity, or cost-sensitive constraints be integrated into the subset selection procedure (e.g., ensuring representation across language groups or domains)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}