metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models

Authors: Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, Eric Schulz
Primary area: datasets and benchmarks
Keywords: llm, benchmarking, item response theory, factor analysis, information

Abstract:
Large Language Models (LLMs) vary in their abilities on a range of tasks. Initiatives such as the Open LLM Leaderboard aim to quantify these differences with several large benchmarks (sets of test items to which an LLM can respond either correctly or incorrectly).
   However, high correlations within and between benchmark scores suggest that (1) there exists a small set of common underlying abilities that these benchmarks measure, and (2) items tap into redundant information and the benchmarks may thus be considerably compressed.
   We use data from n > 5000 LLMs to identify the most informative items of six benchmarks, ARC, GSM8K, HellaSwag, MMLU, TruthfulQA and WinoGrande (with d = 28,632 items in total). From them we distill a sparse benchmark, metabench, that has less than 3% of the original size of all six benchmarks combined. This new sparse benchmark goes beyond point scores by yielding estimators of the underlying benchmark-specific abilities.
   We show that these estimators (1) can be used to reconstruct each original individual benchmark score with, on average, 1.24% root mean square error (RMSE), (2) reconstruct the original total score with 0.58% RMSE, and (3) have a single underlying common factor whose Spearman correlation with the total score is r = 0.94.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730686571419

thread_type:
official_review

<thread_object>
creator: reviewer_fxth

<content>
summary:
This paper proposes metabench, a compressed version of six popular LLM benchmarks (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, and WinoGrande) that achieves comparable evaluation capability while using less than 3% of the original items. The authors leverage psychometric techniques, particularly Item Response Theory (IRT), to identify the most informative test items and estimate latent abilities that can reconstruct original benchmark scores with high accuracy.

strengths:
1. Novel application of psychometric methods to LLM evaluation
2. Impressive compression ratio (<3% of original size) while maintaining accuracy. Low reconstruction error (1.24% RMSE for individual benchmarks, 0.58% for total score).
3. Comprehensive ablation studies and baseline comparisons. Thorough investigation of factor structure across benchmarks.

weaknesses:
1. Memorization risks: 
    (1) Smaller benchmark size increases memorization vulnerability
    (2) Proposed mitigation strategies need further validation
2. Theoretical Assumptions:
    (1) IRT assumptions about LLMs need more justification
    (2) Independence assumptions between models may be violated due to shared architectures/training data

questions:
1. Could alternative item selection methods (beyond Fisher information) yield better results?
2. How stable are the results across different random seeds and model subsets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713076142

thread_type:
official_review

<thread_object>
creator: reviewer_cph2

<content>
summary:
The paper introduces Metabench, a sparse benchmarking method designed to evaluate large language models (LLMs) with minimal redundancy and resource demands. By analyzing data from over 5000 LLMs across six benchmarks (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, and WinoGrande), Metabench distills these into a much smaller subset, reducing the combined item count by over 97%. Using psychometric techniques such as Item Response Theory (IRT), Metabench selects the most informative items, facilitating efficient and accurate evaluation while maintaining the integrity of the original benchmarks. The sparse benchmark achieves impressive fidelity, reconstructing original scores with less than 1.24% RMSE on average, and identifying a single common latent factor strongly correlating with general model ability.

strengths:
1. The paper’s technical approach is methodologically sound, with robust use of IRT and statistical modeling to identify informative items.
2. It is well-organized, with a clear explanation of Metabench’s goals and psychometric techniques.
3. It makes a substantial contribution to LLM evaluation, providing a novel, efficient, and scalable benchmarking solution.

weaknesses:
1. The framework currently focuses on six benchmarks; additional work could explore its applicability across a broader range of LLM tasks or domains.
2. Metabench’s dependence on psychometric models, especially IRT, could be limiting if these models do not fully capture the complexities of LLM behavior, as they were traditionally designed for human subjects.

questions:
1. Could authors elaborate on potential limitations when applying Metabench to other domains?
2. How might Metabench handle scenarios where specific benchmarks assess unique skills not captured by a general latent factor?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723936437

thread_type:
official_review

<thread_object>
creator: reviewer_iuxe

<content>
summary:
This paper introduces metabench, a sparse benchmark distilled from six prominent benchmarks (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, and WinoGrande). Simple criteria, cross-validated subsampling, and information-based filtering are used to reduce the size of the benchmark. Original scores are reconstructed in a cross-validated manner.

strengths:
1. This paper distills six prominent LLM benchmarks into a much smaller one with less than 3% of the size, which enables more streamlined and cost-effective evaluation methods;
2. The new sparse benchmark yields estimators able to reconstruct the original benchmark score.

weaknesses:
As mentioned in the limitations section, a smaller benchmark has the risk of being memorized.

questions:
Will a small benchmark lead to a large variance in evaluation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730775321567

thread_type:
official_review

<thread_object>
creator: reviewer_rtab

<content>
summary:
This paper considers the six LLM benchmarks included in the Open LLM Leaderboard (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, and WinoGrande) and seeks to create a much smaller benchmark that is predictive of the original suite by subselecting items. This is done using data from more than 5000 LLMs included in the leaderboard and a pyschometric method called item response theory (IRT) which in essence fits a model that estimates the item's difficulty and how well the item discrimates between models whose "abilities" are close to the item's difficulty. (Note this model ability is also fit by the method in an alternating fashion.) The presented method results in a benchmark that is only 3% the size of the original benchmark but is able to effectively reconstruct both the original individual benchmark scores and the joint score. Finally, using factor analysis, the authors demonstate that a single latent is predictive of all 6 benchmarks.

strengths:
The paper is well-written and clearly communicates its ideas and methods. In the choice of the IRT, multiple models and methods for estimating the ability are explored. The method proposed produces a much smaller benchmark which the authors demonstrate has better predictive power than randomly subsampling items (Figure 1B). Careful consideration is given to potential limitations of the method, including assumptions about the conditional independence of the LLMs used for the study. The work also considers the interesting idea of a benchmark that performs adaptive testing in which items are selected sequentially based on a current estimate of the model's ability.

Overall I think the paper makes meaningful contributions to studying LLM benchmarks and making model evaluation more efficient, and I thus lean towards acceptance. However, I do think the benchmarks considered are missing some of the abilities that people seek to measure in LLMs (e.g. coding), somewhat limiting the work's impact. I seek to provide concrete suggestions regarding this in the next section.

weaknesses:
My comments in this section are not intended to be required changes to the paper but rather a discussion of what I think the authors could add to have more significant impact.

Currently the main output of the paper is a much smaller benchmark that can be used to efficiently rank models on the six benchmarks as well as evidence from factor analysis that all six benchmarks are measuring a single latent ability. However, across the broader field of LLM benchmarks, it is generally assumed that there are multiple latent dimensions to the abilities of LLMs. For example, if a code benchmark was added into the set, I would assume this would require another latent dimension to fit model performance, and it would be intriguing if this was not true! Also I would be curious if a larger fraction of the test items is required to reconstruct the scores when the set of included benchmarks require multiple latent ability dimensions to represent.

In essence, the most interesting direction I see for this work is to apply the methods to a more comprehensive set of benchmarks to try to discover latent ability dimensions that might be interpretable as what we think of as LLM capabilities. This should then also provide a characterization of which of these abilities each benchmark measures.

questions:
For quite a few models on the leaderboard, the MMLU score will be random chance (~25%, which you can see in Figure 1). Would it be a useful preprocessing step to subtract out random chance from the score and renormalize? E.g. take (score - 0.25) / (1 - 0.25).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732220173526

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their constructive and helpful feedback. Their input was immensely valuable to further improve our submission. The reviewers’ assessment was overall positive with only one reviewer giving a score of 5:
- Reviewer iUxe deemed our methods as “streamlined and cost-effective”.
- Reviewer Fxth found our analyses “comprehensive [and] thorough” and deemed our results “impressive”.
- Reviewer rtaB praised our paper as “well-written [and] clear". They further assessed that our paper and benchmark are "a meaningful contribution[...] to studying LLM benchmarks and making model evaluation more efficient”.
- Reviewer cph2 added that our paper is “well-organized, with a clear explanation of goals and [...] techniques”. They also acknowledged our contribution as “substantial [for the field of] LLM evaluation”.

In response to the reviewers' feedback, we have made the following major additions:
1. We re-ran the entire pipeline for benchmark distillation with 4 additional seeds and checked the stability of our reported results with regard to test error, rank preservation and benchmark sizes.
2. We implemented a clustering-based item selection technique whose performance we compared with our information filtering approach, while keeping all other variables constant.
3. We fit 2-dimensional IRT models and tested if adding a second latent ability significantly improved score reconstruction.
4. We validated our strategies against test memorization by fine-tuning 3 LLMs on the main set of metabench and testing its performance on the disjoint alternative set and on both sets with permuted answer labels.
5. We ran a simulation study that reveals, under which conditions synergy effects across benchmarks arise for score reconstruction.

These analyses, its results, and answers to further reviewer questions are contained in our responses to the individual reviews below. Corresponding code and figures are contained in the Supplementary Material. We again want to thank the reviewers for their valuable time, attention and for actively taking part in the review process.

Update:
We have worked in the corresponding changes into our current version of the manuscript and updated the pdf available on openreview.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors were able to address all issues raised by the reviewers. All reviewers except one were positive about the work. That reviewer was not able to respond, but the authors addressed the issues by them by adding more experiments to evaluate the potential for data contamination, clarify the applicability of assumptions, and discuss alternative data selection strategies.