SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization
Authors: Taolin Zhang
Hang Guo
Wang Lu
Tao Dai
Shu-Tao Xia
Jindong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient Evaluation
LLM Evaluation
Primary area: foundation or frontier models, including LLMs

Abstract: As large language models (LLMs) continue to scale up, their performance on various downstream tasks has significantly improved. However, evaluating their capabilities has become increasingly expensive, as performing inference on a large number of benchmark samples incurs high computational costs.
In this paper, we revisit the model-item performance matrix and show that it exhibits sparsity, that representative items can be selected as anchors, and that the task of efficient benchmarking can be formulated as a sparse optimization problem. 
Based on these insights, we propose SparseEval, a method that, for the first time, adopts gradient descent to optimize anchor weights and employs an iterative refinement strategy for anchor selection. We utilize the representation capacity of MLP to handle sparse optimization and propose the Anchor Importance Score and Candidate Importance Score to evaluate the value of each item for task-aware refinement. Extensive experiments demonstrate the low estimation error and high Kendall’s $\tau$ of our method across a variety of benchmarks, showcasing its superior robustness and practicality in real-world scenarios. Code is available at https://github.com/taolinzhang/SparseEval.

Published as a conference paper at ICLR 2026
SPARSEEVAL: EFFICIENT EVALUATION OF LARGE LAN- GUAGE MODELS BY SPARSE OPTIMIZATION
Taolin Zhang! Hang Guo?, Wang Lu Tao Dai Shu-TToo Xia² Jindong Wang 1 !College of Computer Science and Software Engineering. Shenzhen University
"Tsinghua Shenzhen International Graduate School, Tsinghua University IIndependent, * "Microsoft Research Asia [newlw230630, daitao.edu @ gmail.com
ABSTRACT
As large language models (LLMs) continue to scale up, their performance on various downstream tasks has significantly improved. However, evaluating their capabilities has become increasingly expensive, performing inference on a large number benchmark samples incurs high computational costs. In this paper, revisit the mrodel-item performance matrix and show that it exhibits sparsity, that representative items can be selected as anchors, and that the task of efficient benchmarking can be formulated
sparse optimization problem. Based on these insights, we propose SparseEval, a method that, for the first time, adopts gradient descent optimize anchor weights and employs iterative refinement strategy for anchor selection. We utilize the representation capacity of MLP handle sparse optimization and propose the Anchor Importance Score and Candidate Importance Score to evaluate the value of each item for task- refinement. Extensive experiments demonstrate the low estimation error and high Kendall's r of our method across variety of benchmarks, showcasing its superior robustness and practicality real-world scenarios. Code is available https: / / thub. com ttol inzhang/ Spar
INTRODUCTION
In recent years, the capabilities of large language models (LLMs) have improved dramatically as model scales have grown (Brown, 2020; Achiam al., 2023; Bai et al., 2023; Yang et al., 2024; Team, 2025; Guo al., 2025). From early small- and medium-sized models to today's 's hundred-billins parameter giants, LLMs have achieved remarkable performance in natural language understanding, reasoning, and generation tasks. However, this performance gain comes along with the increased inference cost and evaluation overheads. Larger models require significantly more computational resources and time for inference, particularly during evaluation where the cost of running on large- scale datasets may become prohibitively high. This raises a crucial question in both research and practical deployment: how can we reduce inference costs while maintaining evaluation quality?
Previous studies have noted that evaluation datasets often contain significant amount of redundancy, as demonstrated by visualization techniques such as UMAP (MeInnes et al., 2018) and t-SNE (Maaten and Hinton, 2008). These approaches analyze similarities between samples based on input prompts or probability distributions, revealing underlying patterns in the data. However, such methods typically rely on prompts or predicted probabilities to detect redundancy, which requires additional resources that may be costly or difficult to acquire. Building on these observations, recent work employs Item Response Theory to reduce the number of evaluation samples (Polo et al., 2024). while others utilize adaptive clustering strategies for data selection (Yuan et al., 2025; Wang et al., 2025).
Building such efficient evaluation benchmark necessitates addressing three challenges. First is sparsity existence: how can we intuitively and quantitatively demonstrate the presence sparsity in evaluation tasks justify efficient evaluation? Second is anchor weighting optimization: given pre-selected anchors, how can we optimize their weights so that they effectively represent the
"Corresponding author.

---

Published as a conference paper at ICLR 2026
hntreClaster: 2.51 Inde--llater 0.48
Intre-uuder 0.79 IntenCruee:: 0.47
Intra-Clater 0.08 Inoerllater: 0.33
280
4001
183
680
12:0
22
12
Ben 15m
bar
(a) ARC
(b) GSM8K
(c) Hellaswag
Intra-Clutee 0,72 IndenCluster: 0.50
Incro-Cluseen 0.8 InteriCusaar 0.48
Intra Cluster 0.62 Incer Chatter 0.56
7051
Lod Late
11:03
1783
1485
(d) MMLU
(c) TruthfulQA
(f) Winogrande
Figure 1: Evidence of Evaluation Sparsity in LLM Benchmarks. We construct an item-item simi- larity matrix by computing the cosine similarity between item vectors. The presence of pronounced diagonal blocks along with both high intra- and inter-cluster similarity, suggests the existence of evaluation sparsity and redundancy the benchmark.
characteristics of the entire dataset? Third is anchor selection: how should the choice anchors be guided by the optimized weights and downstream tasks to ensure higher task relevance in evaluation results? These challenges form the backbone designing an effective evaluation framework.
In this work, revisit the model-item matrix and spectral clustering to uncover its highly structured and sparse nature. Our analysis reveals that even clustering data with few anchor points leads to high degree of intra-cluster similarity and strong inter-cluster predictability. This suggests that the model predictions across samples encode large amount transferable information, providing support for efficient evaluation. Therefore, propose SparseEval, task-aware efficient evaluation for large foundation models by sparse optimization. For anchor weight optimization, we propose training strategy based on gradient descent optimization and dynamically adjusts weights by optimizing reconstruction loss. For anchor selection, we design task-aware strategy that leverages error-corelation anchor refinement identify more representative anchor subsets. SparseEval not only reduces evaluation cost but also enhances alignment between the anchors and downstream task.
Unlike existing methods, our framework capture evaluating sparsity directly from the model-item matrix without relying on additional embeddings. By integrating spectral clustering with gradient- based optimization, our approach accurately predicts model performance using a minimal number of anchor points. This results in practical and scalable solution that strikes strong balance between computational efficiency and evaluation accuracy. Our experiments show that compared to traditional full-dataset evaluation, our method can reduce inference costs to only 100 instances while maintaining low estimate errors and highly consistent performance estimates. Moreover, our framework highly generalizable and can be easily adapted to various evaluation settings and task types, offering new direction for designing efficient evaluation benchmarks the era large-scale language models.
RELATED WORKS
Efficient Data Selection Various studies have explored efficient data selection across different settings. LIMA (Zhou al., 2023) demonstrates that large language model can be effectively
2

---

Published as a conference paper at ICLR 2026
aligned using only 1,000 high-quality examples, most of the model's knowledge is obtained during pretraining. LIMR (Li et al., 2025) argues that in the context of reinforcement learning for LLM training, small but strategically curated dataset can outperform significantly larger ones. LIMO (Ye et al., 2025) further shows that strong mathematical reasoning capabilities can be elicited from a knowledge-rich LLM by fine-tuning on just 800 high-quality examples, surpassing models trained on datasets more than 100 times larger. These approaches primarily aim to extract high-quality samples from the original dataset, rather than aligning performance on the original dataset.
Efficient LLM Evaluation Evaluating LLMs is often resource intensive due to the large number of models and test items involved. Several methods have been proposed to reduce evaluation costs by selecting subset of models or items, while closely aligning with the original benchmark in terms of both accuracy and ranking- field often referred to as Efficient L.M Evaluation. Anchor Points (Vivek et al., 2023) analyzes language models using small and representative subsets of evaluation data, while Flash-HELLM (Perlitz al., 2023) adopts a coarse-to-fing strategy that adaptively adjusts the number of evaluation samples for higher-ranking models. In addition, Pacchiardi et al. (2024) proposes using performance on a small set reference instances as input features to a general assessor model predict
LLM's performance unseen instances. Tiny TinyBenchmark (Polo et al., 2024) employs Item Response Theory (IRT) to guide data selection and TailoredBench (Yuan al., 2025) presents another approach by creating customized, adaptive benchmark tailored each model. Recently, EffiEval (Wang et al., 2025) aims to improve evaluation efficiency by selecting a small subset of data that maximizes the coverage of model's internal capabilities.
LLM EVALUATION CAN BE SPARSE
In this section, we first discuss the evalution sparsity in current benchmarks by revisiting the model- item performance matrix. Next, we formally define the problem formulation of efficient evaluation as sparse optimization.
3.1 EVALUATING SPARSITY IN LLM BENCHMARKS
Given a model-item score matrix S E (-1, 1 where m and n denote the number of models and items respectively, and where St,, = 1 indicates a correct prediction and St, -1 indicates an incorrect one. We observe an inherent sparsity structure in this evaluation matrix, which enables more efficient evaluation. In particular, we analyze inter-item relationships by examining the column vectors of 5, where each column represents the response patterns of all models to a specific item. By computing the cosine similarity between these item vectors, construct item-item similarity matrix Sitem E R2x", where each entry reflects the similarity between a pair of test items.
To further explore the structure of the dataset, we apply spectral clustering Snem and partition the test items into five clusters, as shown in Figure 1. The resulting clusters reveal strong intra-cluster similarity, as evidenced by the pronounced diagonal blocks the clustered similarity matrix. This observation indicates that many test items are highly similar to each other in terms of model response patterns, suggesting the presence of redundancy and potential for sparsity in the benehmark. Such a clustering structure implies that it may be feasible to select small subset of representative items (referred to as anchors) that can effectively capture the diversity of the entire dataset. Moreover, we observe that there also exists considerable inter-cluster similarity among the items after clustering, indicating that all the item vectors share substantial common information, making mutual prediction possible. This reveals the widespread existence of Evaluation Sparsity in LLM benchmarks. By focusing evaluation on these anchors, we can potentially reduce the overall evaluation cost while preserving the discriminative power of the benchmark, which motivates the subsequent development of anchor-based sparse evaluation.
3.2 PROBLEM FORMULATION
The goal is to reduce the evaluation cost of LLM benchmarks by selecting small subset of informative items, such that performance evaluated this subset closely approximate the result the full dataset. We achieve this by learning a item-level weighting function, and cast this as an sparse optimization problem.
3

---

Published as a conference paper at ICLR 2026
Anchor Selection
Importance (AIS)
Model
AIB
True Accuracy
Refinement
Grad Norm
Item
6.9
0.5
6.2
Swap
0,5
Model
Lyyez
Initalization
CIS
Accuracy
True Accuracy
True Accuracy
Candidate Importance (CIS)
Underestimate
Proxy
Product
0.5
Predicted
0,9
Predicted Accuracy
Overestimate
Figure 2: Anchor Refinement in SparseEval. We leverage proxy model perform task aware anchor refinement. By iteratively replacing items with Anchor Importance Scores with those having high Candidate Importance Scores, we are able to obtain more representative anchors for efficient evaluation.
General Formulation Let S [$1, $ sns E Rm denote the per-item evaluation scores. With given sparsity k. construct the sparse input S' = S @ (1m W by introducing weighting factor W E R (|]W & k), where @ denotes the element-wise product. Let f TR" - RR be an aggregation function that maps the sparse input to overall benchmark score. We now formulate the evaluation task as the following optimization problem:
minimize W.J
[ E (1mWT)) SWa/,
subject to
|| 1
where W, = 1, is a uniform averaging vector over the full set of items and SWa denotes the real performance of the model. This formulation encourages the learned function S(S @ (1m applied to small subset of items approximate the full evaluation WT S.
Linear Formulation When f is a linear function, it can be absorbed by W and we can derive the objective of all previous methods (Polo et al., 2024; Yuan al., 2025), which directly selects a sparse subset of items whose weighted average matches the full benchmark score.
minimize W
SSW SWall
subject to
[ loo S k
METHODOLOGY
In this section, we first describe the anchor weight predictor for pre-selected anchors in an end-to-end optimization. We then discuss the initialization and refinement strategy of anchors in SparseEval.
4.1 ANCHOR WEIGHT PREDICTOR
We begin by focusing on a simplified version of the problem: given a fixed anchor points, we aim to find an appropriate method for assigning weights to these anchors for perfomance estimation. Traditional methods, such BinyBenchmark (Polo et al., 2024), typically train Item Response Theory (IRT) model and then apply h-means clustering to the resulting IRT representations in order to identify anchor points. The weights these anchors are subsequently assigned based on the distribution of the clusters. Other approaches (Yuan et al., 2025) leverage scaling factors to calibrate the performance estimation. contrast, by formulating the task as a sparse optimization problem, observe that the aggregation function can be approximated using MLP model for end-to-end optimization. Therefore, becomes possible to directly optimize the anchor weights via gradient descent by minimizing reconstruction loss.
4

---

Published as a conference paper at ICLR 2026
Algorithm SparseEval
Require:
Prediction matrix S E { {-1, j number of anchors k, refinement steps R. learning rate proxy training epochs E, final training epochs F
Anchor Selection and Refinement:
2: Initialize anchors A using k-means or random sampling based on validation set
3: for ror==-- do
4: Train proxy MLP fr on current anchors A with E' epochs predict overall model perfor-
mance
5:
Compute prediction residuals: in Eq.
6: for each anchor E A do
Compute Anchor Importance Score AIS; in Eq. for anchor 8: end for
7:
9:
for each candidate j $ A do
10: Compute Candidate Importance Score CIS, in Eq. 5 for candidate j
end for
12: Remove weakest anchor i* = arg min, AIS; from A
Add strongest candidate j arg max, CIS,
14:
end for
Final MIP Training:
16: Train final MLP model selected anchors A with F epochs to predict overall model perfor- mance
17: Return final trained MLP model f and anchor set A
Formally, let W denote the given sparse weight factor (1 S k), Wa the ground-trui truth uniform averaging weights, and Strain RMM the training score matrix over M cxamples and n items. The square loss function for anchor weight optimization is defined as:
1 C= = - [[((rrann O (1MWT)) SSrain Wall2
4.2 ANCHOR INITIALIZATION
Based on our earlier observations, we find that most datasets exhibit strong intra-cluster similarity. which motivates us to initialize anchor points directly by k-means clustering. Applying k-means directly captures the cluster structure of the original dataset, leading to the selection of representative anchors that work well with subsequent gradient-bant based optimization. However, for datasets with weaker intra-cluster similarity such as MMLU, we observe that random initialization can also perform surprisingly well, and some cases, even outperforms k-means initialization.
Therefore, in practice, we adopt an adaptive strategy for anchor initialization. For each dataset, evaluate both k-means and random initialization on a small set models as the validation set and choose the better method for the final initialization.
4.3 ANCHOR REFINEMENT
Directly optimizing the weights of the anchors initialized described above presents key limitation that these initialization strategies are tailored the downstream task. The random initialization selects arbitrary points from the dataset, while the k-means approach captures only the clustering structure of the data. However, neither method participates in the end-to-end optimization process based on reconstruction loss, and the anchors remain fixed after initialization. As result, the selected subsets may be suboptimal for perfomance estimation tasks.
To address this, we explore the relationship between anchors and candidates within the MLP model for anchor refinement. In our setup, the MLP takes the predictions the anchors as input and uses the predictions of the candidates targets to learn from. Two essential components naturally emerge from end-to-ent training with reconstruction loss and backpropagation: errors and gradients. The emor be attributed each non-anchor individually, reflecting how well the current of anchors

---

Published as a conference paper at ICLR 2026
predicts each non-anchor. Conversely, the gradients can be obtained by propagating the enrors back through the network and can be assigned to each anchor, indicating its contribution and importance in the perfomance estimation.
Formally, given m models, n items, and prediction matrix E {-1, {-1,1)mX3 xm the model-level calibration residuals computed as:
e f (1n W) SW..
Intuitively, e, O, it means that the proxy model overestimates model j. In contrast, if ej underestimates it. Now, consider the prediction pattern candidate item as feature vector, representing the model performance over this item. If this feature tends to be positive where the residual vector is positive and negative where it is negative, then the absolute value of their dot product will be high. That is, features that aligned with the structure of the residuals can consistently indicate whether a model is being overestimated or underestimated. For example, if a feature value of +1 typically corresponds to overestimated models and - to underestimated ones, the dot product with the residuals will have large absolute magnitude, suggesting that it is an informative feature. Based on this intuition, we define the Candidate Importance Score (CIS) for candidate item as the absolute value of the dot product between its prediction pattern and the residual vector:
CIS: ((..,) (S..)² (f(S C 1m - SW
(5)
Simultaneously, we can leverage the magnitude of the gradient during backpropagation to assess influence of a given anchor on optimizing the prediction error. This is especially useful in the context of fast-traine fast-training proxy models, where the absolute gradient is often a more direct indicator impact on the error than the weight activation from the first layer. Therefore, can compute the Anchor Importance Score (AIS) for anchor as follows:
0C 1 a as.. N as ((S0L.. SW.
AIS,
(6)
At each step of the iterative refinement process, apply gradient descent with proxy MLP model on the current set of anchors to calculate AIS of each anchor and CIS of each candidate. We then replace the anchor with the lowest AIS with the candidate that has the highest CIS. After R refinement steps, we train a final MIP model which sharing the same architecture but with a different number of input features as the final model.
4.4 THEORETICAL ANALYSIS
Proposition (More anchors yield no larger reconstruction error). In linear weight setting, let mpxx be the model-sample score matrix and define the true overall average as
SW, --s1, E R
where W. 1,1 n is the uniform weight vector. For any anchor set S (L...., n., define the feasible linear weight class
W(A,k) = {w E IR",, supp(w) A, 1/ S k}
and the optimal reconstruction error
E(A): = min
||Sw pll1.
WEW(A.))
Whenever A B, holds that
E(B) S E(A).
Proposition 2 (Anchor refinement decreases L reconstruction error). In the linear weight setting.
consider the linear reconstruction model ğ Sw with L2 loss L(w) = ||Sw -u -1122 Let be the current anchor set and w the current linear weights. Let the residual be r Sw p.. In linear weight setting, we have
CTS; = 18] rl.
j&

---

Published as a conference paper at ICLR 2026
Table Main results on LLM benchmarks. SparseEval consistently outperform baselines by up to 2% lower estimate errors and 0.1 improvement in Kendall's 7 than baselines.
Dataset
Method
Anchor 20
Anchor 40
Anchor 60
Anchor = -00
Anchor = 100
MAE
MAR (%)
MAE (%)
MAE (%)
MAR (%)
Anchor Points
4.004
0.769
2.375
0.866
2,890
0.867
2.289
0.863
10,620
0.578
ARC
HP-RRT
TailoredBench
34266
5.332
0.641
0.821
2,616
3.642
0.854
0.696
2.445
2,959
0.852
0.758
2.816
2.612
0.862
0.761
2.274
2.413
0.787
0.573
SpanseEval
Anchor Points
4.433
0.844
3,756
0,878
3.631
0.916
2,778
0.906
5.295
0.842
GSMSK
gp-
BailrreBBench
IRT
5.275
0.802
3,984
0.832
3.161
0.871
2.774
0.880
2.424
0.887
0.912
SpanseEval
3,305
0.872
2.321
0.908
1.960
0.925
1.754
0.931
1.619
0.936
Anchox Points
3.272
0.796
2,619
0.875
2.416
0.856
1,962
0.847
2,012
0,889
HellsSwag
gp-IRT
BaloredBench
2.352
0.811
2.257
0.847
1.868
0.861
1.957
0.857
L.750
1,968
0.876
SpanseEval
1.477
0.857
1,210
0.590
0.993
0.906
0.942
0.910
0.827
0.918
Anchor Points
0.764
MMLU
gP-RRT
TäilareBBench
4.046
5.910
0.755
0.569
3,802
2.677
0.845
0.692
3.190
2.421
0.857
0.710
2.537
2.216
0.876
0.798
2.019
2.202
0.862
0.829
SpanseEvll
1.718
0.832
1,282
0.571
0.997
0.890
0.962
0.896
0,842
0,908
Anchor Poines
3.215
0.803
0,838
1.958
1.758
0.
1.733
0.891
TruthfuQQ
SP-RRT
TailrreBBench
2.554
4.452
0.847
0.712
2.105
3.032
0.855
0,771
2:250
1.928
0.823
0.863
1.973
1,718
0.874
0.836
1.577
LS00
0.847
0.895
SpanseEval
Anchor Points
5.577
0.691
4.037
0.737
4,038
0.710
3.621
0.752
3.019
0,810
Winogrande
TeiloredBench
gp-RRT
4.903
0.533
3/117
0.547
2.617
0.630
2.284
0.659
1.957
0.725
SparseFaull
2,088
0.794
1.500
0.883
1.361
0.867
1.182
0.882
1.027
0.897
AIS; = |s] T.:
i A.
Suppose the refinement step removes it = arg minge J = arg A4A CIS,, and forms A' = (iX After re-optimizing w over A', the new optimal linear reconstruction error
satisfies
E(A') S R(A).
Moreover, if |s]rr] [sTTr| and 83- is not in the linear span of (8:: E A}, then
< < E(A).
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets We collect the LLM evaluation results from Open-LLM Leaderboard (Fourrier et al., 2024), and we obtain model-item accuracy matrix six LLM benchmarks including ARC (Clark et al., 2018), GSM8K (Cobbe et al., 2021), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), and Winogrande (Sakaguchi et al., 2021). For LLM benchmarks, it is noteworthy that we expand the number of models from 300 in TinyBenchman to 5,000, allowing us more thoroughly evaluate generalization performance across wider range of models.
Implementation Details We randomly select 200 models as the validation and test sets for the LLM benchmarks, with equal sizes. The remaining data are used the training set. We utilize 4-layer MLP in LLM Benchmarks and 6e-4 as the learning rate. The refinement step is set to be 10. We utilize MAE and Kendall's two metrics for evaluation.
5.2 MAIN RESULTS
In Table 1, we compare our method with several baselines, including Anchor Points (Vivek et al., 2023), gp-IRT (Polo et al., 2024), and TailoredBench (Yuan et al., 2025). The results demonstrate that SparseLvval significantly outperforms all baseline methods, consistently achieving lower estimation error and higher correlation coefficients across varying numbers of anchors. As the number of models scales up 5000, traditional cluster-based and IRT-based methods struggle to fully utilize the large volume of data. In particular, when using only 100 items for prediction, these methods often exhibit estimation errors exceeding 2%. In contrast, SparseEval leverages anchor refinement
7

---

Published as a conference paper at ICLR 2026
-0.90
0.92
-0.85
Ancior Poinss
3,0
3.0
-0.80
MAE
IT
0.90
MAE
2.5
Sdoeeteren that
Narseta 0902
SparseEval
0,70
0.88
155
2,0
10
0.05
0.86
Training Data Proportion(%)
80
Number of Anchors
20
40
60
80
100
Number Anchors
Figure 3: Error Trend on ARC. EvarseEval consistently outper- forms baselines.
Figure 5: Ablation over train- Figure 4: Ablation over net- ing data proportion on Hel- work architecture on GSM8K. laSwag.
Table 2: Ablation results on anchor selection. SparseEval benefits from anchor refinement for downsteam estimates tasks.
Dataset
Method
Anchor
Anchor 40
Anchor
Anchor = 80
Anchor = 100
MAE (%6
MAE
[
MAE((%)
MAR (%)
Random
3,131
0.741
2,121
0.819
1.770
0.856
1,433
0.881
1.339
0.890
ARC
k-maans
1,945
0.847
1,656
0.878
1.427
0.897
1.314
0.903
1.218
0.913
SpurseLival
1.227
Random
3.831
0.86
2.694
0.896
2.467
0.908
2,086
0.918
1.857
0.928
GSMBK
k-means
3,544
0,860
2.491
0.898
2.042
0.925
1.767
0,933
0,938
SparseEval
3,305
0.872
2.321
0.908
1.960
0.925
1.754
0.931
1.619
0.936
Random
2,465
0,728
1. 1804
0.788
1.389
0.839
1 1.207
0.858
1,152
0.867
HellaSwag
Sparsefival
1.477
0.857
1,210
0.890
0.993
0.906
0.942
0.910
0.827
0.918
Random
2,152
0.774
1,393
0.854
1.117
0.879
0.996
0,894
0.850
0.903
MMLU
k-means
2.152
1,393
0.854
1.117
0.996
0.894
0.903
SparseEval
1,718
0.832
1,282
0,871
0.997
0,890
0.962
0.896
0.842
0,908
Random
0.854
1,356
0.902
TruthfulQA
m-mann
2.107
0.849
1.417
0.892
1.296
0.902
1.181
0.914
1.058
0.926
SparseFical
1.756
0.878
1,589
0.886
1.243
0.911
1,083
0.922
1.027
0.931
1.277
Winogrande
k-mxans
2.617
0.726
1,761
0.821
1.450
0.851
1 1,277
0.870
1.104
0.886
SparseEval
2,088
0.794
1,500
0.853
1.361
0.867
1.182
0.882
1.027
0.897
Bofore
Densit
Dercky
After
0.
10.0
0.0
0.5
7.5
10.0
(a) GSMSK AIS
(b) ARC AIS
(c) GSM8K CIS
(d) ARC CIS
Figure 6: AIS and CIS distribution w.r.t anchor refinement. The importance scores shift noticeably to the right after refinement, indicating an improvement in the quality of the anchor set.
and weight optimization fully exploit the training data and enhance generalization via gradient descent, achieving estimation errors below 1%, outperforming prior state-of-the-art approaches by a substantial margin. In addition to low MAE, SparseEval also achieves Kendall's scores above 0.90, indicating a strong correlation with the original model rankings and further validating the effectiveness and practicality of our method.
5.3 ABLATION STUDY
Over 5x reduction in anchor points than baselines. As shown in Fig. 3, SparseEval achieves comparable performance to other baselines that use 500 anchors, with only 100 anchors, representing more than 5x reduction in anchor points. Interestingly, the number of anchors increases, the performance of Anchor Points drops significantly, indicating that clustering alone without adaptive anchor weight adjustment is far from sufficient.
8

---

Published as a conference paper at ICLR 2026
Table 3: MAE on the Hold-out Model Families. SparseEval generalize better on the unseen models.
Method
TailoredBench
gp-IRT
SparseEval
deepseek-coderr- -.b-instrutt
6,464
1.873
1.392
(eepseee--ooder-6.77--e 6b-bbase
3.751
2.083
2.132
deepseck-coder-6 6.-iinsstuct
1.096
2.304
0.264
deepseckk-ooleer--bntttuuuttt
4.182
1.681
3.700
deepseck-llm-660bbase
0.020
1.882
0.730
1,448
1,595
0.156
deepseek-Ilm--7b-huu
2.816
2.624
0.738
deepseek-mathh-7-bbuss
2.459
2.019
1.777
deepseek wate----ttttttttttteeeeeee
3.712
1,950
3,810
deep-eek-matt--7b-11
3.407
2,374
2.542
deepseek-moe- 16b-base
0.926
2,244
1,535
deepseek-mue- 6b-chal
12.865
6.195
3,740
deepscek-IIm- 1alllmm----beee
11.662
5.710
4.616
Max MAE Average MAE
12.865
6.195
4,616
4.216
2.657
2.090
Sparse optimization benefits from deeper architecture. We compare the estimation performance between traditional linear weight and MLP-based estimator in Fig. 4. Due to its limited representational capacity, the linear model performs worse, whereas the deeper MLP architecture is better suited for the challenging task of anchor weight optimization.
Comparable performance with limited training data. We adjust the proportion of training data and present the results in Fig. 5. As expected, model performance degrades as the amount of training data decreases. However, SvarseEval consistently outperforms the baseline, regardless of whether limited or full data used. Even with only 20% of the data, SparseEval is able maintain MAE under 1% and T higher than 0.90, demonstrating the robustness of our method.
Anchor Selection. We compare different anchor selection methods investigate the effectiveness of our anchor refinement strategy. As shown in Table 2, anchor refinement clearly leads to superior anchor set. This indicates that both AIS and CIS, used as metrics for selecting anchors and candidates respectively, are highly aligned with the objective of the downstream prediction task. Notably, the improvement brought by anchor refinement becomes more significant when the number of anchors is limited. This suggests that, in such cases, strategies like k-means and random selection fail to correctly represent characteristics of the original dataset, and multiple rounds refinement are necessary to discover better anchor set.
5.4 ANALYSIS AND DISCUSSION
AIS and CIS improvement through refinement. We further investigate the mechanism behind anchor refinement process by analyzing the distributions of both anchors and candidates. As can be observed from 6ig.6, both AIS and CIS improve significantly after refinement. AIS reflects the sensitivity of the selected anchors to prediction error, while CIS measures how strongly candidate responds to the predictive behavior of the current proxy model. The consistent improvement of both metrics through refinement indicates a continuous enhancement the quality of the anchor set.
Hold-out Families. To further validate the robustness of our methods unseen models, we select the DeepSeek family of models as hold-out families. These models span wide range of scales (from 1.3B to 64B), architectures (both dense and MoE),, and application situations (including coder, math, and general models). We train the models on evaluation data from other models and test on these hold-out models. As shown in Table 3, SparseEval outperforms other baselines in terms of the average and maximum prediction error, demonstrating strong generalization across different models.
Error Analysis. We sampled subset of test examples from both SparseEval and gp-IRT, and visualized their prediction errors, as demonstrated Fig. 7. We observed that some models eval- uated by gR-IRT exhibit errors close 10%. making it unsuitable for precise prediction tasks. In contrast, SparseEval shows consistently lower average and maximum errors compared to gp-IRT, demonstrating its generality and robustness.

---

Published as a conference paper at ICLR 2026
10
20-17
Error (N)
Frequency
Training Trme (s)
600
400
200
Accuracy
3.000
6.010 0.15
0.000
0.25
5033
27.0
Weight
SparseEval GP
60C
Hed 65A3g
MMU
Figure 8: Weight values on Figure 9: Efficiency Analysis. MMLU. Negative value expand SparseEvall more efficient than the optimization space.
Figure 7: Error Analysis on MMLU. SparseEval shows smaller max error than gp-IRT.
IRT-based method.
10
1253
100
2000
2009
12:0
3003
100
4230
5037
60
(a) Are
(b) GSM8K
(c) HellaSwag
Figure 10: Sparsity the model-model similarity matrix. Efficient benchmark can benefit from the similarity and sparsity among models.
Weight Analysis. We visualize the distribution of weights obtained from linear weight optimization, shown in Fig.8. The weight values relatively concentrated and mostly close Interest- ingly, we observe that some weights take negative values that is impossible in traditional cluster- methods, thereby significantly expanding the optimization space.
Training efficiency. We further investigate the efficiency between SparseEval and baselines. The inference cost is based the number of anchors. After both methods generate anchors and weights, the inference time cost is roughly the same, so we focus mainly on the training time cost. In gp-IRT, the time is mostly spent on training the IRT model. This training process is very time-consuming, taking as long as 16 minutes for training on MMLU. For SparseEval, its time cost is much lower than that gp-IRT, demonstrating the superior efficiency of our method.
Model-model similarity. Surprisingly, we also observe sparsity in the model-model similarity matrix, as shown in Fig. 10. This suggests that the performance of unknown models can potentially be predicted from that of known models, which is consistent with the findings in TailorBench (Yuan al., 2025). We leave incorporating this sparsity structure into anchor weight optimization and anchor selection in the future work.
CONCLUSIONS
In this work, we introduce SparseEval, novel framework that significantly reduces evaluation costs by selecting and weighting a small set of representative test items. We formalize sparse evaluation as sparse optimization problem and propose the anchor refinement strategy for better anchor selection representativenesss. Empirically, SvarseEval outperforms baselines with more stable predictions and substantially lower training overhead across various LLM benchmarks. However, certain limitations since gradient descent requires sufficient samples to guarantee performance. We hope work could bring inspiration for future work in the field of efficient evaluations.
approach has
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is supported in part by the National Natural Science Foundation of China, under Grant (62302309, 62571298)
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint wwXiv:2303.0877: 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Tom B Brown, Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803::8803.0555 05455, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 2021.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard v2. https: /ttps://hhuue 2024.
co/ httes/paccs//ee ooen-- 11m leaderboard/
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint TrXii:2501.1294: 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.000 000000000003300,, 2020.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502. 11886. 2025.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3214-3252, 2022.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2600) 2008
Leland McInnes. John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1802.03260, 2018.
Lorenzo Pacchiardi, Lucy G Cheke, and José Hernández-Orallo 100 instances is all you need: predicting the success of a new 11m on unseen data by testing on a few instances. arXiv preprint arXiv:2409.03563, 2024.
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheued and Leshem Choshen. Efficient benchmarking of language models. arXiv preprint arXiv:2308. 1/696, 2023.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating Imms with fewer examples. In International Conference on Machine Learning, pages 34303-34326. PMLR, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
11

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwen3: Think deeper, act faster. http::/wwtppo...e.ee.....//aa//a 2025.
Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples. arXiv preprint urXiv:2309008638, 2023.
Yaoning Wang, Jiahao Ying. Yixin Cao, Yubo Ma, and Yugang Jiang. Effieval: Efficient and general- izable model evaluation via capability coverage maximization. arXiv preprint arXiv:2508.0090 2025.
An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 technical report. arXiv preprint arXiv:2407.0677, 2024.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, arXiv preprint arXiv:2502.03387 2025.
Peiwen Yuan, Yueqi Zhang, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, and Kan Li. Beyond one-size-fits-al: Tailored benchmarks for efficient evaluation. arXiv preprint ar0iv:2502.1 /3576, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07880 2019.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 00.550006-5022) 2023.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A DATASET AND LICENSE
We collect the LLM evaluation results of more than 5000 models from Open LLM Leaderboard Fourrier et al. (2024). Below are the datasets used in this paper that have known license information.
The following datasets used in this paper are under the MIT licenses: GSM8K Cobbe et al. (2021) and MMLU Hendrycks et al. (2020).
The following datasets used in this paper are under the CC BY 4.0 licenses: HellaSwag Zellers et al. (2019).
The following datasets used in this paper are under the CC BY-SA 4.0 licenses: ARC Clark al. (2018).
The following datasets used in this paper are under the Apache-2.0 license: TruthfulQA Lin al. (2022), Winogrande Sakaguchi et al. (2021).
B
LLM USAGE
In the preparation of this manuscript, we made limited use of LLMs as general-purpose writing assistant. Specifically, the LLMs are employed to polish wording, improve sentence fluency, and adjust grammatical structure for clarity and readability. At no point did the LLMs contribute to research ideation, the formulation of hypotheses, methodological design, execution of experiments, or interpretation of results. Their role was strictly confined to surface-level language refinement, comparable to the functions of a seammar-chheking or style-editing tool. All intellectual contributions, including the conception of ideas, development of approaches, and analysis of findings, entirely the work of the authors.
C PROPOSITIONS PROOFS
PROOF OF PROPOSITION
Proof. We begin with the notation defined above. Let S [s1, 18,, 8n] RRmx and let
1 51,,
1 = SW
For an anchor set A C define
W(A, k)) = (w E RR supp(w) S A, |wllo $ k),
and the optimal reconstruction error
E(A) min wEW(A.K)
||S pl/11-
If A C B, then every feasible weight vector supported on A is also feasible on B. Thus
W(A,k) S W(B
Since minimization performed over larger feasible set, we have
E(B) min
Su wll1 S min
||w μll E(A).
W
wEW(A.))
Therefore increasing the number of anchors never increases the optimal error.
PROOF OF PROPOSITION
Proof. Since the weights are not at the Op-ommiaal solution on A, the residual Sw p is not orthogonal span (8; i E A), so sf contains meaningful variation.
13

---

Published as a conference paper at ICLR 2026
7.4%
Top 11.9%
0.8
Top 17.4%
Average Similarity
0.6
0.4
0.2
0.0
Items (Sorted by Similarity)
Figure 11: Representative items analysis on MMLU. Representative items tends to share high similarity with other items.
For feature s,, the one-step decrease direction of the linear least-squares loss satisfies
ac Ow:
287 r,
so the magnitude |s]rr measures how much loss can be reduced by adjusting weight wy. Thus CIS identifies the most beneficial candidate add, and AIS identifies the least beneficial anchor to keep.
Replacing i* by j' with ss].r| > |sTTrl strictly enlarges the set of descent directions for linear least squares. Hence the attainable minimum over the new support A' cannot be larger:
E(A') S E(A).
additionally 85. & span : E A), then the refinement introduces a new independent descent direction aligned with the residual, yielding a strictly smaller optimal loss:
E(A') < E(A).
D ANALYSIS OF REPRESENTATIVE ITEMS
In Figure 8, we observed that certain items have weights with notably high magnitudes, suggesting that they representative among all items. To better understand these anchors, compute the average similarity of each item with all other items and highlight those assigned high-magnitude weights. As shown in Fig. 11, these items consistently exhibit high average similarity with other items. This suggests that using them as anchors allows for better representation of other items, which explains why they assigned larger weights.
Additionally, we also provide the details of these representative items.
14

---

Published as a conference paper at ICLR 2026
Representative Items Samples on MMLU
The following multiple choice questions (with answers) about professional law.
A man on trial for rape. The alleged victim testified that she went out to dinner with the man. Afterward. he invited her to his apartment for coffee. Upon entering the apartment, he violently assaulted her. Although she tried to resist, he overpowered and raped her. The man testified that during dinner, he and the victim drank two bottles of Champagne. When they returned his apartment, he was so intoxicated that he honestly believed that she consented the intercourse. The jury determined that the victim did not consent to the intercourse. The jury also found that the man, as result of his intoxication, honestly but unreasonably believed that she was consenting. As consequence, the defendant should be found
A. not guilty, because honestly helieved that victim consented. B. not guilty, because his intoxication negated his criminal intent.
C. guilty, because rape is general intent crime.
D. guilty, because she did not consent, and his belief that she consenting unreasonable. Answer:
The following are multiple choice questions (with answers) about clinical knowledge.
Which of following is true about involuntary movements in the arm?
A. Alcohol makes the tremor of benign essential tremor worse
B. Hemiballismus is due to stroke causing paralysis of the distal half of the arm
C. A 'milkmaid' grip is sometimes found in dystonia
D. Writer's cramp is example of focal dystonia
Answer:
The following multiple choice questions (with answers) about high school physics.
During an isothermal expansion, a confined ideal gas does 150 work against surroundings. Which of the following describes the heat transfer during this process?
A. 150 of heat was added to the gas.
B. 150 of heat
removed from the gas.
C. 300 of heat added the gas.
D. 300 of heat was removed from the gas.
Answer:
15


Reviews:
[{"review_id": "Ch4Heygvn0", "reviewer": "Reviewer_cdnC", "summary": "The paper proposes SparseEval, a sparse-optimization framework for efficient evaluation of LLMs. It formalizes benchmark sparsity by modeling the model–item score matrix and selecting a small set of representative “anchors.” The method optimizes anchor weights via gradient descent with an MLP predictor and refines anchors using AIS and CIS. Experiments on six LLM benchmarks show that SparseEval achieves <2% MAE and τ > 0.9 with only 100 samples, outperforming prior IRT- and clustering-based approaches.", "questions": "none", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Casting efficient LLM evaluation as a sparse optimization problem is conceptually clean and original.\n\nThe gradient-based anchor refinement (AIS/CIS) is simple, intuitive, and empirically strong.\n\nResults across multiple datasets and ablations convincingly show superior efficiency and robustness.\n\nThe approach can substantially reduce evaluation cost in large-scale LLM benchmarking.", "weaknesses": "The proposed method relies on access to a large model–item performance matrix for training and anchor refinement. Its effectiveness when only a small number of model evaluations are available, or when evaluating a completely new task with limited historical data, has not been examined. This restricts its practical usability in real-world cold-start settings.\n\nWhile the paper frames efficient evaluation as a sparse optimization problem, the theoretical analysis remains shallow. The notion of sparsity is mainly supported by empirical observations rather than formal proofs or quantitative measures, and there is no theoretical guarantee relating the number of anchors to estimation error.\n\nTypos:\nLine 410: demostrating → demonstrating \nLine 485: filed → field", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QvmhZTf9xP", "reviewer": "Reviewer_rZJG", "summary": "This paper introduces SparseEval, a novel method to efficiently evaluate large language models by framing the problem as sparse optimization. The method uses a proxy MLP model to learn optimized weights for a small subset of \"anchor\" test items via gradient descent. A task-aware refinement strategy iteratively improves the anchor set by swapping items based on new \"Anchor Importance\" and \"Candidate Importance\" scores. Experiments demonstrate that SparseEval accurately estimates model performance with significantly reduced computational cost, outperforming previous methods in estimation error and ranking correlation.", "questions": "* Could you provide generalization tests of the proposed method, including evaluations on stronger models and models with different architectures?\n* Could you provide an analysis of the method’s adaptability on Long-CoT data?\n* Could you further explain the sparsity assumption? From Figure 3.1, it appears that the similarity between different clusters of data is also quite high.\n* When testing different training checkpoints of the same model, the model’s performance tends to fluctuate — can this method accurately predict the precise performance in such cases?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper demonstrates the sparsity in the dataset, which enables the prediction of model performance using a small amount of anchor data.\n\n* The paper proposes SparseEval, which trains an MLP to make predictions based on the performance of existing models on both anchor data and the full dataset.\n\n* Experimental results show that this method can accurately select models that are representative of the entire dataset.", "weaknesses": "* Given a new dataset, this method may require testing many models and performing training, which can be costly to train in practice.\n* The approach lacks generalization to stronger models and other architectures. Under the current training setup, it remains unclear whether the architecture can generalize to more powerful models or to new architectures such as linear attention or MoE models.\n* The method lacks evaluation of its adaptability under long-chain-of-thought (long-CoT) conditions, such as on benchmarks like AIME or GPQA. Under long-CoT settings, model outputs tend to fluctuate significantly, so it is uncertain whether this method would still be effective.\n* Training the MLP requires a relatively large amount of data. When only a few existing models are available, the performance may be suboptimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FFwb5Xe6sx", "reviewer": "Reviewer_qyxY", "summary": "The paper frames large-model benchmarking as a sparse optimization problem: select a small set of anchor items from the full benchmark and learn anchor weights via gradient descent to approximate full-benchmark scores. To improve representativeness, the authors propose iterative anchor refinement using Anchor Importance Score (AIS) and Candidate Importance Score (CIS) to quantify item value; they also use an MLP as a nonlinear aggregator to realize sparse reconstruction. Experiments reportedly achieve low estimation error and high Kendall’s τ ranking consistency across multiple benchmarks, demonstrating substantial cost savings without changing conclusions.", "questions": "Same as weakness", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper offers a clear modeling perspective by framing evaluation sparsity through the model by item score matrix and performing sparse selection with learned weights directly aligned to overall scores and rankings. It introduces task aware anchor optimization via AIS and CIS to iteratively refine anchors beyond one shot selection, and employs a nonlinear MLP aggregator to capture inter item structure and model family differences. Empirically, it foregrounds practical gains, reporting low error and high Kendall's tau across benchmarks, consistent with the aim of reducing cost without altering rankings. Methodologically, it departs from IRT and heuristic clustering and external assessor approaches by explicitly learning anchor weights to reconstruct aggregate benchmark scores.", "weaknesses": "1. Do your learned anchors (and the associated aggregation/weights) directly generalize to unseen models? In other words, can we train anchors once and apply them to new model releases without retraining?\n2. What is the end-to-end cost of discovering/learning the anchor set (including any proxy-model runs, scoring, and training)? If this cost is substantial, and the resulting anchors/weights only work well for a single model, then the approach may not be worthwhile. Please quantify the cost and clarify the cross-model reusability.\n3. I’d like to see visualizations of the item/question space that justify the claimed sparsity (e.g., clustering/affinity structures). Which questions are similar, and why? Please provide empirical evidence (e.g., similarity heatmaps, spectral clustering plots, cluster exemplars) to explain why certain items are considered redundant or representative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GBAs7OfemY", "reviewer": "Reviewer_ky5X", "summary": "This paper proposes a framework for faster LLM evaluations that formalizes anchor selection and weighting as an optimization problem. The anchor weights can be learned via an MLP using gradient descent, and the anchors are iteratively updated via candidate importance score and anchor importance score. Across six benchmarks, the SparseEval achieves very low error in terms of MAE and high rank fidelity in terms of Kendall's $\\tau$ with very few items (roughly 100) and a large set of models (roughly 5000) from the Open-LLM leaderboard.", "questions": "Typos:\n\n* Line 180, $S'=S\\odot ({\\bf 1}_m^W\\top)$ -> $S'=S\\odot ({\\bf 1}_mW^\\top)$.\n\n* Line 185, standard notation for $\\ell_1$ norm is $\\\\|\\cdot\\\\|_1$ rather than $\\|\\cdot\\|_1$.\n\n* Inconsistency on the use of $k$ and k for $k$-means.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Experiments are comprehensive, results are strong. The study aggregates per-item accuracy matrices over ~5,000 models and evaluates on six widely used datasets, expanding the model coverage far beyond prior efficient-evaluation work (e.g., TinyBenchmarks’ ~300 models). The scaling is nice to see. On this large setting, SparseEval consistently beats baselines (Anchor Points, gp-IRT, TailoredBench) across anchor budgets (20–100). For example, at 100 anchors the method reaches MAE below 1% with $\\tau>0.90$ on multiple datasets; the paper highlights that competing methods frequently exceed 2% MAE in the same regime. The error-trend plots (e.g., ARC) show a clear and stable gap over baselines as anchors increase, while the ablation contrasting linear weighting vs MLP underscores that deeper architectures materially improve sparse estimation—precisely the regime SparseEval targets. The data-efficiency ablation is notable: even with 10% training data, the method maintains ≈1% MAE on HellaSwag/MMLU, supporting claims of robustness under scarce supervision.\n\n2. The CIS/AIS refinement loop is intuitive and simple: at its core, the algorithm is greedy by swapping out the anchor with the smallest AIS and in the candidate with the largest CIS. CIS selects candidates whose response patterns align with residual structure; AIS quantifies an anchor’s gradient contribution to error reduction.", "weaknesses": "1. Experiments only report MAE and Kendall's $\\tau$. Many modern evals use calibration metrics, long-form judgments, or pairwise win-rates. A brief experiment or discussion on extending the loss/aggregator to non-decomposable metrics (e.g., Brier/F1, win-rates) would broaden applicability.\n\n2. Robustness of anchors transfer to newer set of models. The study is comprehensive in-distribution over a very large model set. It would be useful to show that anchors selected on one model cohort transfer to a different cohort or to newer models (e.g., hold-out families), even if with modest degradation.\n\n3. Interpretability of learned weights: The work nicely shows negative weights expand the search space relative to cluster averages; a short qualitative analysis of which items receive high magnitude weights (e.g., skill categories in MMLU) would provide insight for benchmark curation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)