Fewer Battles, More Gain: An Information-Efficient Framework for Arena-based LLM Evaluation
Authors: Zirui Liu
Xianquan Wang
Yan Zhuang
Jiatong Li
Qi Liu
Shuanghong Shen
Mingyue Cheng
Shijin Wang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Evaluation; Chatbot Arena; Efficient Evaluation
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Arena-based evaluation has become a key method for assessing large language models (LLMs) through head-to-head model comparisons, closely reflecting human preferences. However, current arena rating systems (e.g., ELO rating system) often suffer from inefficiencies due to exhaustive or random model pair annotations, leading to redundant evaluations, longer evaluation times, and lower overall efficiency. To address these challenges, we propose a novel adaptive model-pair selection algorithm. By leveraging the asymptotic normality of LLM ability estimation under sparse conditions, our approach strategically selects high-value model pairs, focusing on confrontations with the lowest variance. Specifically, we introduce Fisher information as a metric to guide model pair selection, optimizing the evaluation process through A-optimality and D-optimality. A-optimality minimizes estimation variance, ensuring balanced reliability across models, while D-optimality reduces uncertainty by maximizing the determinant of the Fisher Information Matrix. Extensive experiments on both simulated and real-world datasets demonstrate that our method outperforms existing approaches in terms of information efficiency and result reliability. Notably, our method offers a flexible, general toolkit that can be easily integrated into existing arena-based platforms, greatly improving scalability and efficiency for large-scale LLM evaluations.

Published as a conference paper at ICLR 2026
FEWER BATTLES, MORE GAIN: AN INFORMATION- EFFICIENT FRAMEWORK FOR ARENA-BASED LLM EVALUATION
Zirui Liu', Xianquan Wang', Yan Zhuang', Jiatong Li', Qi Liu!?;
Shuanghong Shen', Mingyue Cheng'. Shijin Wang'
1: State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
3: College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics
4: iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd
{liuzirui, wxgen, zykb, cslijt c @@mall www.ustcc.u... edu. [qiliuql,
edu. cn shshen @ili ust edu, cn,
ABSTRACT
trena-based evaluation has become a key method for assessing large language models (LLMs) through head-to-heat model comparisons, closely reflecting hu- man preferences. However, current arena rating systems (e.g.. ELO rating system) often suffer from inefficiencies due exhaustive random model pair annota- tions, leading to redundant evaluations, longer evaluation times, and lower over- all efficiency. To address these challenges, we propose a novel adaptive model- pair selection algorithm. By leveraging the asymptotic normality of LLM abil- ity estimation under sparse conditions, our approach strategically selects high- value model pairs, focusing on confrontations with the lowest variance. Specif- ically, we introduce Fisher information as a metrie to guide model pair selec- tion, optimizing the evaluation process through A-optimality and D-optimality. A-optimality minimizes estimation variance, ensuring balanced reliability across models, while D-optimality reduces uncertainty by maximizing the determinant the Fisher Information Matrix. Extensive experiments on both simulated and real-world datasets demonstrate that method outperforms existing ap- proaches in terms of information efficiency and result reliability. Notably, our method offers a flexible, general toolkit that be easily integrated into exist- ing arena-based platforms, greatly improving scalability and efficiency for large- scale LLM evaluations. Our code is publicly available to promote reproducibility at mattps://githube com/ Liuz rui /tt///dpptt... inaapiiveArenn
INTRODUCTION
The rapid advancements in large language models (LLMs) have led to their widespread use in var- ious fields Li et al. (2025); ; Jiang et al. (2025), including content generation (Wang al.] 2024a customer service, and intelligent tutoring Kasneci et al. 2023). As LLM architectures (Liu et al.
2024a), fine-tuned variants (Hu al. 2023 Cheng al. 2025b), and domain-specifee models (Patil & Gudivada 22244] Cheng et al. 2026; Pan et al. 2026 Cheng et al. (202aa) continue to evolve, thus the demand for efficient and scalable evaluation frameworks is growing (Mizrahi al. 2024 Zhou
ct al. 2023 Li et al. 2024b 2024b/Zhuang et al. 2025a). Among existing evaluation methods, arena-based
evaluation is notable (Chiang et al. 2024): unlike traditional benchmark- based approaches (Wang
et al. 2024b] Mirzadeh et al. (024) that rely on predefined prompts and fixed answers, arena-based
frameworks assess models through head-to-head confrontations. This approach captures real-world
performance dynamics and produces rankings that closely align with human judgments (Zheng et al.)
2023 Luo et al. 2024), making it valuable tool for assessing LLMs' practical utility.
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
Hundreds of Models (Rapid Iteration) Scalability!
Human Annotation
ELO Score
Low Efficiencyl
1287
1253
Exhoustive Battles
Dol
New models
1006
The diagram of Arena-based Evaluation. Models engage in pairwise battles, annotators label the outcomes, and the ELO ratings are then estimated based on these annotations.
Figure 1:
However, current arena-based evaluation methods face significant issues. First, is the efficiency (Luo at 2024) Spangher et al., 2025). As shown Figure these frameworks often rely on exhaustive or random pairwise annotations to generate reliable rankings, which becomes increasingly resource- intensive. Second, is the scalability. As the number of newly proposed models grows, platforms like Chatbot Arena (Chiang al.] 2024), hosting over 190 models, require tens of thousands of confrontations to produce reliable rankings (Patil Gudivadal (024). This not only consumes considerable resources but also results in substantial evaluation cycles, making it difficult keep up with the rapid development of LLMs (Min et al.) 2025).
To address these challenges, propose novel adaptive arena-based evaluation method. Lever- aging the asymptotic normality of ability estimation under sparse conditions, we can strategically select high-value model pairs for evaluation and improve efficiency by minimizing the variance of the asymptotic distribution. Specifically, we introduce two key methods: A-optimality, which mini- mizes the variance in model estimates, and D-optimality, which reduces uncertainty by maximizing the determinant of the Fisher Information Matrix (Rissanen 1996 Ly et al. (201)). Through these techniques, we achieve more gain in each annotation, leading to fewer battles.
Our contributions are as follows:
We first introduce the concept of efficiency in arena-based LLM evaluation, using statisti- cal uncertainty to minimize redundant evaluations, significantly improving both speed and resource usage in ranking.
We apply A-optimality and D-optimality to model-pair selection, backed by rigorous theo- retical analysis, to reduce annotation costs while preserving ranking reliability.
Through extensive experiments, show that method outperforms existing approaches in efficiency and reliability, and can seamlessly integrate with current arena- based plat- forms. This makes our method not just a theoretical innovation, but practical and scalable solution for large-scale LLM evaluations.
RELATED WORK
Arena-based Evaluation Arena-based evaluation is a key approach in LLM evaluation, distin- guished from traditional benchmark methods by its focus direct head-to-head model competi- tions (Wang et al., 2025). This design provides a more realistic measure of models' relative abili- ties (Zellers et al. 2019 Hendrycks al., 2020 Cobbe et al.) 2021). Unlike traditional approaches that use predefined datasets (Liang et al. 2022 Liu et al.) 2024b). arena settings emphasize dynamic model confrontations that better reflect real-world usage scenarios. Ranking systems are central to Arena- evaluation (Busa-Fekete et al., 2014 Szzöényí et al.) 2015 [Cerrnoff (192). with most methods relying on the ELO Rating System to estimate model proficiency (Coulom) 2007] Pelánek 2016). However, ELOO's sensitivity to sample order can lead to instability in performance estimates (Boubdir et al.) 2024). To address this, recent studies have proposed adaptations, such as statistical refinements (Ameli et al. 2024 Yin et al., 2024 Liu et al. 2025) and bias-mitigation methods like UDA (Zhang et al. 22025). ) Notably, all existing Arena-based methods aim to improve ranking accuracy but overlook the critical issue of annotation efficiency. Whether using standard ELO, adapted variants, or frameworks like UDA, these approaches often involve unnecessary con- frontations, leading low annotation efficiency and high evaluation overhead.
2

---

Published as a conference paper at ICLR 2026
Efficiency of LLM Evaluation Efficiency has become key focus in LLM evaluation due to the
high costs of annotating large datasets and conducting extensive tests (Marion et al. 2023) Exist-
ing methods (Xie et al. 2023] Chung et al. 2023 Saranathan et al.] 2024: Li et al. 2024a Zhuang
et al., 2025b) share the goal of reducing the number of evaluation examples while maintaining ac-
euracy, but they are designed primarily for traditional static question-answering (QA) tasks, where
models are assessed against predefined prompts and answers. For example, [Pllo et al. (2024a) intro- duced BinyBenccmarks, which evaluates LLMs with fewer examples than traditional benchmarks by prioritizing information-dense samples, thus reducing annotation and computational costs while maintaining reliable performance estimation. Similarly, Kipnis et al. (2024) proposed MetaBench, which selects diverse, compact prompts based on statistical analysis, avoiding redundant evalua- tions while capturing core model abilities. Based on this, Polo et al. (2024b) developed an efficient evaluation method, which optimizes prompt selection across rounds to reduce redun- dant data collection and computation without sacrificing accuracy. However, these cfficiency-driven methods are not applicable to dynamic Arena-based evaluation for two key reasons: First, Arena scenarios involve pairwise head-to-head model confrontations rather than static QA tasks; second, rankings in Arena settings rely on iterative human judgments (Kruglanski & Ajzen 1983), breaking the assumption of fixed criteria inherent in traditional QA methods. Therefore, the dynamic and interactive nature of Arena evaluation makes current efficiency methods unsuitable.
Given these gaps-existing efficiency methods being incompatible with dynamic Arena scenarios, and Arena-based methods neglecting annotation efficiency-our study aims to address these issues. We propose optimizing Arena-based evaluation through strategic sample selection techniques to both accelerate system convergence and improve overall evaluation efficiency.
ADAPTIVE ARENA METHOD
3.1 MODELING ELO RATINGS
Arena-based evaluation has emerged as a key approach for LLM evaluation, where models com- pete head-to-head on benchmarks or datasets, with outcomes annotated by evaluators. Let S = [ii,j, Wi < j E [N]] represent the comparative records, where N is the total number of mod- els being evaluated. Each element (i, why) E S contains important information: indicates that model i and model j have engaged in a competition, and way is the annotation result. Specifically, Way = 1, model i wins; if Wiy = 0, model j wins; and if waj = 0.5, it indicates a tic. The goal of arena-based evaluation is to estimate the ranking scores (ri,...,,Ny rN). ). Since the ranking is relative, we assume that the ability of the last model is O, i.e., rN = 0, and in the following, we focus on the ability estimation of the first N models, denoted as (rI,...,TN-I)
To model the win probabilities, we use the Bradley-Terry model (Rao & Kupper, 1967). which forms the statistical foundation for modern ELOO-ttle rating systems. It posits that the probability of model winning against model j. denoted P.y. is logistic function of their ability difference:
Pi 1+ (h:-))
where C' is a scaling constant.
With the obtained winning probability, many methods be adopted to get the ranking scores. Among these, Maximum Likelihood Estimation (MLE))-based ELO finds the ratings that best ex- plain all observed match results simultaneously. estimate the abilities r by MLE. It is achieved by minimizing the negative log-likelihood function:
more statistically robust batch approach to
arg min Cs(r) where Ls(r)= =
Σ
[wis In Ptj + wij In Pxl
Adopting the MLE framework is crucial as it allows to rigorously analyze the statistical proper- ties of the estimated ratings, which is essential for developing our adaptive selection strategy. The detailed properties are as follows.
Statistical Properties of ELO. Consider modeling the arena as an Erdós-Rényi (E-R) Graph (N) Kumar et al. 2000 Guimera et al. 2004), where N is variable and the sample size [8]
3

---

Published as a conference paper at ICLR 2026
increases with N. Here, 9N represents the probability that confrontation occurs between models i and j. Under this framework, the estimation results ELO exhibit both uniqueness and asymptotic normality (Simons & Yao 1999 Han et al. 2020):
Lemma 1. Let G(N, qN) of the first models, respectively. We have:
(N), be the Crdos-Rényi Graph, and Pas vn be the estimated and true abilities
(1) (Uniqueness) qN =
[oggN) then ELO almost surely has a unique solution NN-1-
(2) (Asymptotic Normality) If qN (log NyUs 7 00 as N + oc, then for any < N,
N-/10
(f -r,) _ NN0, ((rr))(ra))))) where Is(r") is the Fisher information matrix. In the ELO method, Fisher information (Frieden 2004) is calculated by solving the Hessian matrix:
Is(r) = ([(H([((())) C2
Σ
- - e,,
(ij.m4))
where e, is the i-th standard basis vector, and ev 0. To explain the Fisher information matrix intuitively, consider a toy example with three models, X, Y, and Z, and two battle records S = {(X (X 7)}- Assuming all models have initial ability of 0 and that model Z is fixed during the estimation process, the calculation of the information quantity is as follows:
C2
0 X>Y 02
0.25
0.25
X= C2
0.5
0..5
(4)
0.25
0.25
0.25
0.25
and also provide case study of the information matrix based real dataset in Appendix E
Considering Lemma [ with Fisher information matrix like Equation 4 it can be confirmed that ELO has unique estimated solution and this estimation asymptotically unbiased and normally distributed, even in sparse scenarios. Therefore, from a statistical perspective, the efficiency problem in arena evaluation essentially revolves around how to reduce variance more quickly during the evaluation process. Specifically, the key lies in reducing the inverse of the Fisher information matrix (r*) under EL.O context.
3.2 SELECTION STRATEGY FOR ARENA-BASED EVALUATION
As mentioned earlier, our direct goal is to reduce the inverse of the information matrix. However, the true ability values of candidate models, r", are unknown, making impossible directly se- lect the model pairs. Therefore, we dynamically select each subsequent model pair based on the accumulated records S,-1- Specifically, by approximating the true abilitics T with the estimated abilitics pt derived from S,-1, we compute approximate information matrix. Using the Fisher information of the current abilities, we propose the following A-optimality method (Chan Wanyonyi al. 202I for Arena-based Evaluation:
1982
The A-optimality method. This method aims to select the model pair that minimizes the sum of the sampling variances of the ability estimators, which is equivalent to minimizing the trace of the inverse of the information matrix:
(it = arg min tr (I't.. + ((0m)( IgisjSN
(5)
For this method be feasible, the Fisher information matrix must be invertible. We have the fol- lowing theorem:
Theorem 1. Consider the Frdös-Rény' Graph G(N, qN) for ELO. If gN w< N
then the
information matrix Is(r) almost surely positive definite.
From Theorem I follows that as long as there are sufficient battle samples, the information matrix is guaranteed to be positive definite (Johnson] 1970). This only ensures that the information matrix is invertible but also guarantees that the determinant positive.
However, A-optimality method still requires calculating the inverse of matrix, which increases computational complexity. To address this, we consider maximizing the determinant of information matrix and propose the D-optimality method (Jonn & Draper 1975) for Arena-based Evaluation:
4

---

Published as a conference paper at ICLR 2026
Table 1: This table summarizes the characteristics of each strategy. Using r indicates whether the strategy utilizes currently estimated abilities: Global Info indicates whether global information is used, and min qN represents the minimum edge probability that ensures effective selection.
Method
Random
Nearest
Ada-Pair
A-optimality
D-optimality
Using F
Global Info
X loR
X
min gN Complexity
w los O(N
wl loc N O(N
O(1)
O(N)
O(N2)
The D-optimality method. The D-optimality method aims to select the model pair that maximizes the determinant of the information matrix in the current state, expressed
(ee: = arg max ISKJSN
| (f*-1) ))= + I ((C)))) (1-
(6)
3.3 THFORETICAL ANALYSTS
In this subsection, we analyze our proposed method from three perspectives: algorithmic prerequi- sites, numerical computation, and time complexity.
Algorithmic Prerequisites. Different selection strategies depend on distinct prerequisites. Table summarizes the conditions for each method.
For example, the Random selection method requires no additional variables, so it is often used to initialize other strategies. In contrast, methods that consider model abilities & must meet at least the conditions outlined in Lemma I],, which represents the theoretical lower bound of qN for these methods. More stringent conditions apply to methods like Ada-Pair, which considers the confidence interval of each model pair (i,j). To ensure the existence of confidence intervals, Ada-Pair requires that each pair has competed at least once, implying that density of the E-R graph must satisfy qN 1. This requirement becomes restrictive the number of models N increases. This highlights key advantage of our proposed method: it only requires the minimal condition, w log log ), to ensure the positive definiteness of the information matrix (Theorem T without requiring additional samples for initialization. This feature demonstrates the broad applicability of approach.
Numerical Computation. To avoid numerical instability and ensure computational validity, we need consider the relationship between Fisher information and the number of samples. We present the following theorem to address this:
Theorem 2. Let v be the ability parameter vector, and let (s.(r) denote the Fisher information matrix based the subset S, with size Define:
A(S.)) = (r) (average information efficiency).
. +D(S.) = II, (r)) (normalized geometric information density).
Assume that each element in St contributes bounded positive information and that Is, (r)) is positive definite for sufficiently large t. Then as t -> oo:
1. A(S,) O(t) (there exist constants C₁ and t1 such that A(S,) I C for all t1).
2. D(S.) O(1) (there exist constunts C2 and l2 such that D(S.) < C2 for all l2).
Theorem 2 provides the upper bound for the variation of Fisher information with respect to the num- ber of competitions. Specifically, the inverse of the A-optimality measure grows linearly with the increase in the number of samples, while the (N- - 1)-th root of the D-optimality measure grows linearly. Therefore, when practically implementing the D-optimality method, is advisable to trans- form the calculation into the form defined in Theorem to prevent numerical explosion.
Time Complexity. Algorithm I
presents the pseudo-code for model selection in the Arena. The main computational cost in the model pair selection phase comes from calculating the determinant

---

Published as a conference paper at ICLR 2026
Algorithm 1:
The Adaptive Arena Framework
Initialize: Step t + 1, Model's ability 10, Battle records So t O.
while True do
Select model pair: je) < arg angaaigiijss [s + [((a)))) [(())))(())) (f--1)
Get Annotation for model pair: War, Annolu (1).. Add record in S: St < S
Je. w.
Update model's ability A: ff + arg max, Ls, (r), f- 1+1.
Output: The models' estimated abilities $*.
the inverse of the matrix. Both operations have complexity of O(N)) (Kaltofen & Villard 2005), where N is the matrix size or the number models in the arena. Therefore, for the D-optimality method, which traverses all model pairs and computes the determinant of the information matrix at each step, the complexity is O(N++2). For the A-optimality method, whether calculating the trace of the inverse matrix computing N submatrices, it requires N additional calculations per iteration, resulting in a complexity of O(N N x N = ((N++3). The complexity for calculating the information matrix involves summing over iterations, making the overall time complexity for the A-optimality method 0([5) + |S| No++), and for the D-optimality method, O([S)² + |S] N2+2).
Although method has higher time complexity than previous approaches, its primary goal is to reduce human-annotated samples by leveraging computational time, ultimately minimizing the overall annotation effort. Therefore, we only require our method meet the operational needs of the Arena system providing model pairs for evaluation within a limited time for each user request. To achieve this, we have implemented the following improvements: (I) The information matrix is updated directly by adding the current model pair's information to the matrix from the previous step. (2) To address system concurrency, the top- model pairs with the highest information gain are selected for each request.
We discuss the time cost and the efficiency these improvements in section Although our method does not directly reduce the complexity related to the annotation volume, its value becomes evident within the iterative, ongoing Arena framework. For example, our method improves effi ciency by 50%, it means that the annotation results from approach achieve the same out- comes as twice the original annotation volume, given the Arena's continuous operation. This offers a significant long-term advantage, enabling comparable or even superior results with fewer resources over extended periods.
4 EXPERIMENTS
In this section, we evaluate the performance of the proposed A-optimality and D-optimality meth- ods through experiments on both simulated and real-world datasets, analyzing the abilities of each method and discussing the time cost for proposed method.
4.1 EXPERIMENTAL SETUP
Evaluation Metric. Since the ELO score is relative, we use the pairwise index (Hastie & Tibshi-
rani 1997) measure the consistency between the estimated abilities and the true abilities:
Σ Pairwise = N(N ISKJEN
2
I[( - rr)) -f) > 0j.
where 1[:] is the indicator function, r" is true abilities of the models, and Å is the estimated abilities. Since the true abilities are not available in real-world scenarios, we use the results estimated from the full dataset by ELO (Equation 2 as the ground truth (Zhuang et al. 2023 Liu et al.) 2024c). Lemma I ensures that r* derived from this process is unique and unbiased, making this ground truth feasible. In other words, our task can be viewed as selecting a subset of matchups that achieves
the same evaluation quality using the full set, thereby significantly improving efficiency.

---

Published as a conference paper at ICLR 2026
Table 2: Performance of Pairwise Metrics for Different Strategies. Results averaged over five experiments and four steps. Bold values indicate significant improvements over the baseline.
Datasct
Simu
ELO Method
Method
Total
Chatbot
PPR
Real
Simu
ELO
m-FI.O
Random
0.8797
0.8021
0.8501
0.8316
0.8379
0.8439
0.8409
Nearest
0,8825
0.7995
0.8422
0.8398
0.8412
0.8407
0.8410
Original
Ada-Pair
0.8835
0.8040
0.8452
0.8423
0.8379
0.8496
0.8437
A-optim
0,8880
0.8033
0.8494
0.8418
0.8406
0.8506
0.8456
D-optim
0.8941
0.8122
0.8596
0.8466
0.8486
0.8577
0.8531
Nearest
0.8897
0.8020
0,8525
0.8392
0.8351
0,8566
0.8459
Ada-Pair
0.8849
0.7993
0,8509
0.8332
0.8376
0,8465
0.8421
Improved
A-optim
0,8807
0.8061
0,8488
0.8380
0.8372
0,8495
0.8434
D-optim
0.8918
0.8243
0.8581
0.8581
0.8546
0.8616
0.8581
Dataset. We conducted experiments on two real-world datasets: Chatbot (Chiang et al. 2024 Zheng et al. 2023) and PPE (Frick et al., 2024). The Chatbot dataser" was collected from 13,000 distinct IP addresses on the Chatbot Arena platform, while the PPE datase containss human prefer- ence evaluations for Preference Proxy Evaluations. Detailed statistics and experimental settings are provided in Appendix C.I
Compared Approaches. This paper compares two model ability evaluation methods: ELO (Al- bers & Vries 2001) and m-ELO (Liu et al. 2025). ELO refers to the traditional ELO, it updates each model's rating iteratively by adding the difference between the annotation Wij and the predic- tion Paj. while m- m-ELO uses a MLE process like Equation 2 For model pair selection, we compare several common strategies used in both traditional competitions and modern LLM arenas. These strategies include: Random, which randomly selects model pair for annotation at each step and serves as a baseline; Nearest, which selects the two models with the closest abilities at each step, a typical strategy in competitive games; and Ada-Pair (Chiang et al.) (024), an official selection strategy from Chatbot Arena, which takes into account the confidence interval of each model pair and selects the one that maximizes the reduction of the interval. Additionally, for each method, we consider both the original version and the time- impevimpoveed version discussed in Section 3.3 referred to as Original and Improved, respectively.
4.2 EXPERIMENT RESULTS AND ANALYSIS
To evaluate the effectiveness of our proposed selection method, we conducted experiments on real- world datasets. The following conclusions were drawn:
Performance on Pairwise Metrics. Figure 2(a) shows the performance of different selection strategies each step, and Table 2 presents the average pairwise for each strategy. Our proposed method, D-optim, outperforms all others across all scenarios and is the only method that significantly surpasses the Random baseline. Additionally, variance in most cases, indicating that it lowers evaluation uncertainty and enhances efficiency.
shown in Figure 2(a) D-optim effectively reduces
The Nearest method, on the other hand, performs suboptimally in the LLM arena. This is due to the nature of the arena: rather than actual battles, models answer single question, making ties more likely than in traditional competitions. This diminishes the effectiveness of pairing models with similar abilities. We also observe that while m-ELO incurs additional time costs, its impact on selection strategies that rely on current model abilities is notable. In practical applications, engineers need to balance computational efficiency and accuracy, such as using m-ELO for high-precision tasks and ELO in resource-constraingg settings.
Performance of Time-Improved Method on Pairwise Metries. To assess the impact of our improvements proposed methods, compared the Nearest, Ada-Pair, A-optimality, and D-

---

Published as a conference paper at ICLR 2026
Chatbot Dataset
Simulation
Simulation
0.90
0.90
61.
0.8
0.45
0.7
0.7
0.95
0.95
0.9
B-F1-0
0.90
0.8
0.88
0.85
0.7
0.50
0.80
200
500
1000
2000
1000
200
1000
200
1000
Test St
Test Step
Test Step
Test Step
(a) Performance Original Methods.
Chatbot
Simulation
Simulation
0.38
0.9
E1O
0.90
0.90
0.35
0.88
0.95
0.95
0.9
IB-E1.
0.90
0.90
0.85
0.8
0.88
0.80
0.7
Test Step
Test Step
Test Step
Test Step
(b) Performance Improved Methods.
Figure 2: This Error bar illustrates pairwise metrics for each dataset and ELO method. "Dataset" indicates randomly sampled competition results, while "Simulation" denotes results simulated based on real model abilities. Error bars represent standard deviation.
ELO (Chatbot)
ELO (PPE)
m-ELO (Chatbor)
m-ELO (PPE)
Nearut
A-Inío
Deepila
0.6
0.6
X-epoles
8.4
0.0
266
1868
216
184
1080
8.0
284
104
1868
0.8
286
1080
ELO (Chulbot)
ELO (PPE)
Im=ELO (Chatbot)
mBELO (PPE)
D-info
168
480
1868
200
40
1063
364
480
1864
266
1063
step
step
step
step
Figure 3: This line chart shows A-Info and D-Info for each dataset and ELO method.
optimality methods, where the Top- 10 samples were selected in each sampling round (i.e., 10).
The experimental results are presented in Figure and Table
We can observe that our D-optimality method not only maintains its original performance but also shows improvements, where total 0.5% gain is observed compared to the time-improved version. This suggests that model pairs with higher D-Info are indeed prioritized for selection, supporting the effectiveness of our D-Info design. This result confirms the effectiveness of our improved D-optim. the appendix
we also present performance on larger-scale datasets.
8

---

Published as a conference paper at ICLR 2026
10.0
7.5
5.0
2.5
Random
Nearest
Ada-Pair
A-optim
D-optim
Heatmap of model pair sampling frequency for different selection methods. Each cell represents the number of battles between Model (ranked by ability) and Model (ranked by ability).
Figure 4:
Performance on Fisher Information Gain. Figure shows the variation of A(S)) = (r)-)
(A-Info) and D(S,) Is, (r) (D-Info) as the number of samples increases. As shown in Theorem 2 both A-Info and D-Info exhibit linear growth. The experimental results confirm that under most strategies, their growth is indeed linear. Furthermore, our method achieves the fastest improvement rate (steepest slope) in both A-Info and D-Info. In contrast, Ada-Pair shows similar information gain to the Random method. This suggests that our method, despite using estimated abilities, enhances information gain from samples, improving evaluation efficiency.
Visualization of Sampling Distribution. To visualize the sampling distribution model pairs selected by each strategy, we sorted the models by true abilities and used heatmaps represent the sampling frequency of each pair. Figure shows that D-optim selects model pairs with similar true abilities more often (darker colors near the diagonal), while ensuring diverse coverage. In contrast, Ada-Pair samples more uniformly across all pairs, and Nearest focuses too much on similar models.
Interestingly, the A-optimality method results many samplings involving specific model, since this model's ability is fixed in the ELO system. This creates unique role for the model pair selec- tion: The A-optimality method, while ensuring sparsity (i.e., most pairs have battled), prefer to use this model as an "anchor" to estimate others' abilities. As shown in Figure 3 this method actually leads to high A-Info, but may introduce bias and impact the Arena's 's sustainability. An improvement could be to fix each model sequentially, calculate A-Info for each, and average the results, but this would increase complexity. In contrast, D-optimality is minimally impacted by this constraint by ELO, making D-Info more suitable metric for Arena-based evaluation.
10,N--00
K-10.N-50
K- 100
K-10,N-200
Time Cost per Step
Origial D-opttn
4-
D-opim
4-
10-
4
2-
5-
2-
100
200
500
1000
100
200
500
1000
100
500
1000
100
200
500
1000
Nums Samples
Nums of Samples
Nums Samples
Nums of Samples
Figure 5: Time costs for Original D-optimality and Improved D-optimality Methods the number of samples and models increases.
Performance on Time Cost. Figure 5 reports the time required for each sample selection as the number of samples grows, with the red line marking second. Although the D-optimality method does not always achieve the absolute minimum time, its cost remains far lower than that of LLM API calls or human annotations realistic settings. Given the relatively low computational cost of our method, in practice, ever possible. These results suggest that our method is efficient in practice and can be seamlessly incorporated into existing arena-based evaluation platforms.
leverage computation to reduce the reliance human annotation wher-
CONCLUSION
In this work, addressed the efficiency bottleneck in arena- based LLM evaluation. By leveraging the asymptotic normality of ELO-based ability estimation and introducing Fisher information as guiding principle, proposed adaptive model-pair selection strategies grounded in A-optimality

---

Published as a conference paper at ICLR 2026
and D-optimality. These methods systematically reduce redundant battles while preserving ranking reliability, thereby accelerating evaluation and lowering annotation costs. Extensive experiments on both simulated and real-world datasets confirm that framework consistently outperforms existing approaches in efficiency and robustness. Beyond improving scalability for current arenas, approach can be regarded as general and flexible toolkit that be readily integrated into future evaluation platforms. We believe this work opens promising direction toward information- efficient, scalable, and reliable assessment of large language models. future work, an important direction is to incorporate instance- factors into the m- m-ELO framework, paving the way for fully adaptive evaluation across test instances, models, and even annotators.
ACKNOWLEDGMENT
This work was supported by grants from the National Key Research and Development Program of China (Grant No. 2024YFC3308200), the National Natural Science Foundation of China (U25B2072, 62502486, 62507010), the Key Technologies R&D Program of Anhui Province (No. 202423K09020039) the Fundamental Research Funds for the Central Universities and the Anhui Provincial Higher Education Digital Transformation Project.
ETHICS STATEMENT
This research does not involve any human subjects, animals, or sensitive data. The proposed methods and experiments focus solely on the evaluation of LLMs using publicly available datasets. We ensure that all data used our experiments are publicly accessible, adhering to ethical standards and relevant data usage policies. Furthermore, our work does not raise any concerns related to bias, fairness, or social impact, as focused on improving the efficiency of model evaluations a controlled, academic setting. In summary, our research does present any ethical issues, and are committed to conducting future work in line with ethical best practices and standards.
8 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we provide the complete implementation of our methods, along with all necessary code and data, GitHub repository. The repository includes scripts to replicate the experiments presented in the paper, including data preprocessing, model training, and evaluation procedures. All hyperparameters. random seeds, and environment configurations specified to ensure consistent results across different platforms. The repository can be accessed at the following link halktttst//itttutt
Lww.-iuuuu--- http:/iuuurrrrtapptttitiataa Arena
REFERENCES
Paul CH Albers and Han de Vries. Elo-rating as a tool in the sequential estimation of dominance strengths. Animal behaviour, pp. 489-495, 2001.
Siavash Ameli, Siyuan Zhuang, Ion Stoica, and Michael W Mahoney. A statistical framework for ranking based chatbots. arXiv preprint arXiv:2412.18407, 2024.
Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. Elo uncovered: Robustness and best practices in language model evaluation. Advances in Neural Information Processing Systems, 17:106135-1061 2024.
Róbert Busa-Fekete, Eyke Hüllermeier, and Balázs Szörényi. Preference-based rank elicitation using statistical models: The case of mallows. In International conference on machine learning, pp- 1071-1079. PMLR, 2014.
NN Chan. A-optimality for regression designs. Journal of Mathematical Analysis and Applications, 87(1):55-00, 1982.
Souray Chatterjee and SR Srinivasa Varadhan. The large deviation principle for the erdös-rényi random graph. European Journal of Combinatorics, 10077,000--117, 2011.
10

---

Published as a conference paper at ICLR 2026
Mingyue Cheng, Yucong Luo, Jie Ouyang. Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, et al. A survey on knowledge-oriema retrieval- aagganlte generation. arXiv preprint (rXiv:2503310677 2025a.
Mingyue Cheng, Jie Ouyang, Shuo Yu, Ruiran Yan, Yucong Luo, Zirui Liu, Daoyu Wang, Qi Liu, and Enhong Chen. Agent-rl: Training powerful Ilm agents with end-to-end reinforcement learn- ing. arXiv preprint arXiv:2511. 14460, 2025b.
Mingyue Cheng, Daoyu Wang, Qi Liu, Shuo Yu, Xiaoyu Tao, Yuqian Wang, Chengzhong Chu, Yu Duan, Mingkang Long, and Enhong Chen. Mind2report: A cognitive deep research agent for expert-level commercial report synthesis. arXiv preprint arXiv:2601 2xXV:2661.00879, 2026.
Herman Chernoff. Sequential design of experiments. Springer, 1992.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Sto- ica. Chatbot arena: An open platform for evaluating lims by human preference. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
htt./formmmmii 3MN8GKNY2I
Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. arXiv preprint urXiv:2304.0915/ 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Rémi Coulom. Computing "elo ratings" of move patterns in the game of go. ICGA journal, 30(4): 198-208, 2007.
Joel Franklin. Matrix theory. Courier Corporation, 2000.
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph Gonzalez, and Ion Stoica. How evaluate reward models for rlhf, 2024.
URL https: // http//ixvvvv././////.. /tt.s///211. 10..1488 14872
Roy Frieden. Science from Fisher information, volume 974. Citeseer, 2004.
Roger Guimera, Marta Sales-Pardas and Luís Nunes Amaral. Modularity from fluctuations in random graphs and complex networks. Physical Review E Statistical, Nonlinear, and Soft Matter Physics, 70(2):025101, 2004.
Ruijian Han, Rougang Ye, Chunxi Tan, and Kani Chen. Asymptotic theory of sparse bradley-terry model. 2020.
Trevor Hastie and Robert Tibshirani. Classification by pairwise coupling. Advances in neural information processing systems, 10, 1997.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 0000000033000.... 2020.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy KeiWei Lee. LIm-adapters: An adapter family for parameter-efffiient fine-tuning of large language models. arXiv preprint 2rXiv:2304.0193. 2023.
Zhonghua Jiang, Kui Chen, Kunxi Li, Keting Yin, Yiyun Zhou, Zhaode Wang, Chengfei Lv, and Shengyu Zhang. Acckv: Towards efficient audio-video lims inference via cross-calibration cache optimization. arXiv preprint arXiv:2511.11106. 2025.
and
RC St John and Norman Draper. D-optimality for regression designs: a review. Technometrics, 17(1):15-23, 1975.
11

---

Published as a conference paper at ICLR 2026
Charles Royal Johnson. Positive definite matrices. The American Mathematical Monthly, 77(3): 259-264, 1970.
Erich Kaltofen and Gilles Villard. On the complexity of computing determinants. Computational complexity, 13(3)91-130,, 2005.
Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023.
Alex Kipnis, Konstantinos Voudouris, Luca M Schulze Buschoff, and Eric Schulz. metabench- a sparse benchmark of reasoning and knowledge in large language models. arXiv preprint arXiv:2407.12844, 2024.
Arie Kruglanski and Icek Ajzen. Bias and error in human judgment. European Journal of &ocial Psychology, 13(1):1-44, 1983.
Ravi Kumar, Prabhakar Raghavan, Sridhar Rajagopalan, Sivakumar, Andrew Tomkins, and Eli Upfal. Stochastic models for the web graph. In Proceedings 41st Annual Symposium on Founda- tions of Computer Science, pp. 57-65. IEEE, 2000.
Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, Connor Holmes, Cheng Li, and Yuxiong He. Deepspeed data efficiency: Improving deep learning model quality and training efficiency via efficient data sampling and routing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18490-18498, 2024a.
Jiatong Li. Renjun Hu, Kunzhe Huang, Yan Zhuang, Qi Liu, Mengxiao Zhu, Xing Shi, and Wei Lin. Perteval: Unveiling real knowledge capacity of Ilms with knowledge-invariant perturbations. arXiv preprint arXiv:2400 1904. 2024b.
Kunxi Li, Zhonghua Jiang, Zhouzhou Shen, ZhaodeWang ZhaodeWang, Chengfei Lv, Shengyu Zhang, Fan Wu, and Fei Wu. Madakv: Adaptive cedality-perceptinn kv cache eviction for efficient multimodal long-context inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13306-13311 2025.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.00 10, 2022.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024b.
Zirui Liu, Zhuang Yan, Qi Liu, Jiatong Li, Yuren Zhang, Zhenya Huang, Jinze Wu, and Shijin Wang- Computerized adaptive testing via collaborative ranking. In Neural Information Processing Sys- lems, 2024c. URL httpt:t////.
:/ap:::://ppp/ a/ppi semanti cscho ar org/ CCoppu 27 62596259999
Ziru Liu, Jiatong Li, Yan Zhuang, Qi Liu, Shuanghong Shen, Jie Ouyang, Mingyue Cheng, and Shijin Wang. am-elo: A stable framework for arena-based Imm evaluation, 2025. URL https: 03475
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yan- song Tang, and Weizhu Chen. Arena learning: Build data flywheel for lims post-training via simulated chatbot arena. arXiv preprint arXiv:2407. 20022447.0062., 2024.
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul PPP Grasman, and Eric-Jan Wagenmak- ers. A tutorial on fisher information. Journal of Mathematical Psychology, 80:40-55, 2017.
12

---

Published as a conference paper at ICLR 2026
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale. arXiv preprint NrXfi:2309.04564 2023.
Rui Min, Tianyu Pang, Chao Du, Qian Liu, Minhao Cheng, and Min Lin. Improving your model ranking on chatbot arena by vote rigging. In ICLR 2025 Workshop on Foundation Models in the Wild, 2025.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229 2024.
Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? a call for multi-prompt IIm evaluation. Transactions of the Association for Compu- tational Linguistics, 12:933-949, 2024.
Onorato Timothy O'Meara. Introduction to quadratic forms, volume 117. Springer, 2013.
Tingyue Pan, Jie Ouyang, Mingyue Cheng, Qingchuan Li, Zirui Liu, Mingfan Pan, Shuo Yu, and Qi Liu. Paperscout: An autonomous agent for academic paper search with process-aware sequence-level policy optimization. arXiv preprint arXiv:2601.1002 2026.
Rajvardhan Patil and Venkat Gudivada. A review of current trends, techniques, and challenges in large language models (Imms). Applied Sciences, 14(5):2074, 2024.
Radek Pelánek. Applications of the elo rating system in adaptive educational systems. Computers & Education, 98: 98:169-179, 2016.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuckai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint 244.::9002.19999 2024a.
Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj. Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuckai Sun, and Mikhail Yurochkin. Efficient multi multi-prompt evaluation of Ilm.. arXiv preprint 1rXiv:2405.17202, 2024b.
Rao and Lawrence L Kupper. Ties in paired-comparison experiments: generalization of bradley-terry model. Journal of the American Statistical Association, 62(31)): 1967.
Jorma J Rissanen. Fisher information and stochastic complexity. IEEE transactions on information theory, 42(1):40-47) 1996.
Gayathri Saranathan, Mahammad Parwez Alam, James Lim, Suparna Bhattacharya, Soon Yee Wong, Martin Foltin, and Cong Xu. Dele: Data efficient Ilm evaluation. In ICLR 2024 Work- shop on Navigating and Addressing Data Problems for Foundation Models, 2024.
Gordon Simons and Yi-Ching Yao. Asymptotics when the number of parameters tends infinity in the bradley-terry model for paired comparisons. The Annals of Statistics, 27(3):1041-1060, 1999.
Lucas Spangher, Tianle Li, William Arnold, Nick Masiewicki, Xerxes Dotiwalla, Rama Kumar Pasumarthi, Peter Grabowski, Eugene Ie, and Daniel Gruhl. Chatbot arena estimate: towards generalized performance benchmark for Ilm capabilities. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 3: Industry Track), pp. 1016-1025, 2025.
Balázs Szörényi, Róbert Busa-Fekete, Adil Paul, and Eyke Hüllermeier. Online rank elicitation for plackett-luce: A dueling bandits approach. Advances neural information processing systems, 28, 2015.
Kelly Tang, Wei-Lin Chiang, and Anastasios N. Angelopoulos. Arena explorer: A topic modeling pipeline for Ilm evals & analytics, 2025.
Kangyu Wang, Hongliang He, Lin Liu, Ruiqi Liang, Zhenzhong Lan, and Jianguo Li. Inclusion arena: An open platform for evaluating large foundation models with real-world apps. arXiv preprint a50iv:2508.1 /452, 2025.
13

---

Published as a conference paper at ICLR 2026
Xianquan Wang, Likang Wu, Shukang Yin, Zhi Li, Yanjiang Chen, Hufeng Hufeng, Yu Su, and Qi Liu. I-am-g: Interest augmented multimodal generator for item personalization. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 21303-21317, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266 95290, 2024b.
Samson Wanyonyi, Ayubu Okango, and Julius Koech. Exploration of d-, ar, i-and g- optimality criteria in mixture modeling. 2021.
Hermann Weyl. Inequalities between the two kinds of eigenvalues of linear transformation. Pro- ceedings of the national academy of sciences, 35(7):408-41) 1949.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances Neural Information Processing Systems, 36: 34201-34227, 2023.
Yanbin Yin, Zhen Wang, Kun Zhou, Xiangdong Zhang, Shibo Hao, Yi Gu, Jieyuan Liu, Somanshu Singla, Tianyang Liu, Eric P. Xing, Zhengzhong Liu, Haojian Jin, and Zhiting Hu. Decentralized arena via collective IIm intelligence: Building automated, robust, and transparent Ilm evaluation for numerous dimensions, 10 URthttts::/dde arena mww.maatt..........m...m meaatrii.. org/
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint 1755:3990.007830, 2019.
Yang Zhang, Cunxiang Wang, Lindong Wu, Wenbo Yu, Yidong Wang, Guangsheng Bao, and Jie
Tang. Uda: Unsupervised debiasing alignment for pair-wise Ilm-asaa-judge. 2025. htRpps::::
//api. sespi.ssmmanicccool.... :ataaaarrr.rccoooaa10:00000000
Liammin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your Ilm an evaluation benchmark cheater. arXiv preprint arXiv:231 1.09964, 2023.
Yan Zhuang, Qi Liu, GuanHao Zhao, Zhenya Huang, Weizhe Huang, Zachary Pardos, Enhong Chen, Jinze Wu, and Xin Li. A bounded ability estimation for computerized adaptive testing. Advances in Neural Information Processing Systems, 36, 2023.
Yan Zhuang, Qi Liu, Zachary Pardos, Patrick C Kyllonen, Jiyun Zu, Zhenya Huang, Shijin Wang, and Enhong Chen. Position: Ai evaluation should learn from how we test humans. In Forty- second International Conference on Machine Learning Position Paper Track, 2025a.
Yan Zhuang, Junhao Yu, Qi Liu, Yuxuan Sun, Jiatong Li, Zhenya Huang, and Enhong Chen. Ef- ficient benchmarking via bias-bounded subset selection. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025b.
14

---

Published as a conference paper at ICLR 2026
A PROOFS OF THEOREM
Theorem 1. Consider the Erdös-Rényi Graph G(N, if IN = w( ), then the infomation N
matrix (r) almost surely positive definite.
implies that the graph G(N, is almost surely con- nected (Guimera al. 2004 Chatterjee & (atuddaan2201] we next prove the following: if there ex- ists a path between any two models i and J. then the information matrix is positive definite. Assume IN and consider the remaining variables r "..""",,N-11. Consider the log- likelihoodd function for records S:
Proof. Given that IN W Log
Is(r) = C²
Σ (KSWW,))
(ePj(ee-e,)Cee - e,)
where is the i-st standard basis vector and let eN O. Let the number of matches between model i and model j be dij and define Oij = P.y Pjs. then the quadratic form (O' OO'Meara 2013) for (r) can be expressed
N -1
N-1
IIs(()a
Σ ISiCiSN-1
atylz, - x,,)²
+
a
+
Σ ==1
ayNaz
1=1
Note that 0ij 2 therefore:
aIs (r)x²) T 2 0.
The following proves that the condition for the equality of this inequality can only be achieved at O.
Suppose there exists a non-zero vector á = (&1, ... -1) such that aIs(r)) = O. Without loss of generality, assume that è * 0. Then, since the graph formed by the pairwise battles among the models is connected, it means that there must exist path (cg 1, C1 c2.".. Ck-1, C₁ N) that connects 1 and N. Then accc4+1 > O, from which we can deduce that #1 tee Since Ck-1 and N are connected, it indicated that dcg IN 0. Therefore, de. = 0, and thus L 0, which is contradiction! So xIs(r)) x 0 if and only if x is the zero vector. Therefore,
fe----
Is (r) is a positive definite matrix (Johnson 1970).
B PROOFS OF THEOREM 2
Lemma 2. Let 0 X
< X < S Xyy denote the eigenvalues of the information matrix Is, (r),
then Xy = (()..
Proof. Without loss of generality, we can assume that model and j compete against each other at time f + 1. It is not hard to see that the information increment C² Pe(ee e,)(e; e,)_ is rank-1 1 matrix and its only positive eigenvalue satisfies that () < < 20 P. < C². Therefore, by Weyl's Inequality for matrix perturbations (Weyl. 1949 Franklin 2000), the eigenvalues of the 1 < XNy + S Xy C By recursively expanding
information matrix at time (+1 satisfy X < A
this incquality, we obtain that XN- S A + + < ... N. C2, i., = =(t).
Theorem 2. Let A(S,) = (r) L. and D(S)) = [T. then it can be proved that (I)A(S.) = O()) =
Proof. Since tr (I,. (r) (r) x
x 2 and [T. = XGAS ... X S (M
we can derive the following bounds for the optimality criteria:
X For the A-optimality criterion A(S.) = -r( < N
For the D-optimality criterion D(S.) II. (r)) X/y By Lemma 2, we know X = O(t), substituting this into the above inequalities yields A(SS) = 0(t)
and D(S.) O())
15

---

Published as a conference paper at ICLR 2026
C DETAILS OF EXPERIMENTS
C.I STATISTICS OF THE DATASET
In addition to the two datasets (Chatbot and PPE) mentioned in the main text, have also included a larger dataset Arena-100k (Chiang et al. 2024 lang et al. 2025) in the appendix. The Arena- 100k dataset comprises conversation data collected from June 2024 to August 2024, well as English human preference evaluations, which were used to develop Arena Explorer. Each record in datasets includes question ID, the names of the two competing models, their full conversation transcripts and the annotator's vote. The statistics of the datasets can be seen in Table
Table 3: Statistics of the dataset
Dataset
Chatbot
PPE
Arena-
#Modelss
20
20
55
#Response logs
33000
16038
106134
#R #Response logs per model
1650.0
801.9
1929.7
#Response logs per model pair
173.68
89.10
71.47
C.2 DETAILS OF EXPERIMENTAL SETTING.
In this experiment, 100 sample pairs will be randomly selected during the model initialization phase to initialize ELO. and the performance of different selection strategies will be recorded at steps 100, 200, 500, and 1000. After a selection strategy chooses a model pair, we consider two methods to simulate the competition result: (1) Conducting without-replacemment random sampling on the real competition results in the dataset; (2) Simulating based on the real model abilities. If the abilities of the two models are r and (C(rr - r))),, then each battle will return with probability of p², 0 with probability of p)² and 0.5 with a probability of 2p(1 p). As for ELO rating system, we the hyperparameter of ELO log 10 and epochs 50 for each battle m-ELO. 400
All experiments described in this paper can be executed on one 8G NVIDIA GeForce RTX 4060.
C.3 PERFORMANCE ON LARGE DATASET
To further validate the effectiveness of the D-optimality method, we also conducted experiments on a larger dataset, namely Arena- 1rona-100k. The key difference from the previous experiments lies in the following: as the number of models increased to 55, the number of initial samples we used was also increased to 1000. All other experimental settings remained unchanged. The experimental results
are presented Figure and Table
Arena 1 190k Dataset
Arana 100k Simulation
Areza 100k Dataset
Arena: 10tk Simulation
Ranie
0.58
0.66
0.52
0.56
0.54
0:815
0.00
0.82
0.350
0.78
0,82
0.835
0.50
0.800
Test Stop
Stop
Test Step
Step
Figure 6: This line chart illustrates the pairwise metrics for Arena-100k dataset and each ELO method. Here, "Dataset" indicates that the competition results are randomly sampled from the dataset, while "Simulation" denotes that the results are simulated based the real model abilities. The error bars represent the standard deviation.
can be observed that with an increase in the number of models, the errors introduced by the tradi- tional FL.O method into the information matrix become amplified, leading to inaccurate information estimation. This explains why the D-optim method exhibits superior performance specifically in terms of m-ELO. However, at step 1000, the D-optim method achieves the best performance across all experimental settings. Therefore, overall speaking, as long as the current abilities of the models can be effectively estimated, the D-optimality method remains fully effective even in scenarios with a relatively large number of models.
100k
16

---

Published as a conference paper at ICLR 2026
Table 4: The Performance of pairwise for Different Selection Algorithms. Each experimental sult represents the average of five experiments and four selection steps, the bold font represents significant improvement in statistics compared the baseline.
Is Simu
ELO Method
Method
Total
Real
Simu
ELO
m-ELO
Random
0.8000
0.8499
0.8164
0.8335
0.8249
Neurest
0.8083
0,8415
0.8133
0.8365
0.8249
Ada-Pair
0.8138
0.8486
0.8236
0.8388
0.8312
A-optim
0.8047
0.8438
0.8145
0.8340
0.8243
D-optim
0.8132
0.8561
0.8225
0.8468
0.8346
D USAGE OF LLM
According the rules about LLM use by the ICLR 2026 conference, when we worked on this paper, we only used LLMs to polish the text. This includes making the flow of language smoother and improving how well sentences connect logically. We did not use LLMs for key steps like coming up with main points, making research content, or handling data. Also, we made sure that the polishing process does not change the main ideas or weaken the academic strictness of the original text, and we fully follow the standards for proper LLM use set by ICLR 2026.
E CASE STUDY OF FISHER INFOMATION MATRIX
To facilitate better understanding of our method, we directly visualize the changes in the informa- tion matrix throughout the experimental process. Specifically, we adopt the D-optimality method: in each iteration, a model pair is selected, and the model ability estimates are updated in real time. Below, we present the information matrix of 5 iterations under the condition of 100 initial samples:
0.2
0.0
-0.2
-0.2
00.
-0.2
-0.4
2.3
0.D
0.0
0.0
0.0
0.0
0.0
0.9
-0.5
0.0
-0.4
0.2
0.0
-0.2
0.0
0.0
6.0
-0.2
-0.2
0.u
2.0
-0.2
0.D
0.D
-0.5
0.0
0.9
-0.2
0.0
0.0
0.0
-0.2
-0.2
0.0
0.0
0.0
-0.2
-0.2
0.0
-0.5
-0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
-0.2
-0.2
-0.2
0.0
0.0
0.0
0.0
-0.2
0.D
1.6
-U.2
0.0
0.9
0.0
0.0
0.0
D.0
0.0
-0.2
-0.4
0.0
0.0
-0.2
0.0
0.0
-0.5
0.0
0.D
0.2
2.7
-0.5
D.0
0.0
-0.4
-0.2
-0.4
0.0
-0.2
0.0
0.0
0.0
0.0
0.D
-0.5
0.0
0.0
00.2
0.0
0.0
-0.4
0.0
0.0
0.2
0.5
0.2
0.2
0.D
D.D
0.0
0.2
D.D
2.7
0.2
0.6
D.O
0.0
0.0
0.0
0.0
-0.2
0.0
0.2
0.D
0.D
D..4
U.D
U.D
D..4
0.0
0.0
0.2
2.8
-0.4
D.O
0..2
-0.7
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
D.4
0.0
0.0
0.0
0.2
0.22
0.0
0.0
0.0
0.5
0.0
0.2
0.5
U.D
0.0
0.0
0.0
0.9
0.0
0.2
0.0
0.2
1.6
0.0
0.0
0.0
0.0
0.0
0.2
0.2
0.2
-.2
0.2
0.2
-0.2
0.0
0.0
0.0
-0.7
0.0
0.5
0.0
3.1
0.0
-0.2
0.0
0.0
0.0
0.4
0.0
0.0
0.0
0.2
0.0
0.0
0.44
0.0
0.0
00.2
0.22
0.2
-02
0.0
0.0
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.2
0.0
0.0
D.0
0.0
0.0
-0.2
0.0
0.9
-0.2
0.2
0.2
0.2
0.0
0.0
0.2
0.0
-.2
0.0
0.0
0.0
0.0
D.0
0.0
0.0
0.0
002
0.2
2.1
+
Step 1:(17,4,1)
3.7
0.4
-.2
-0.2
0.0
0.0
0.0
0.0
0.5
0.2
0.2
00.5
0.0
-0.5
-0.
-02
0.0
0.0
-0.2
2.3
1
0.0
0.0
2.0
-021
0.0
0.0
-0.5
0.0
0.0
0.2
0.0
0.0
0.0
0.2
0.2
0.0
0.0
0.2
D.2
0.0
0.2
29
0.0
0.2
0.0
-022
0.2
0.2
0.4
0.0
D.0
0.5
0.2
0.0
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
D.5
0.0
0.G
0.2
D.G
6.
0.
02
02
OZ
0.0
0.0
0.0
-02
1.6
0.2
0.0
0.0
0.0
0.0
0.4
0.0
0.0
0.0
-0.5
0.0
0.0
0.2
2.7
-0.5
0.0
0.0
-0.4
-0.2
0.4
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
0.0
0.0
0.5
2.7
0.2
0.2
0.0
0.0
0.0
0.0
0.0
004
004
0.0
-022
0.2
0.5
-0.
021
0.0
0.0
0.0
0.2
0.0
27
0.2
-0.6
0.0
0.0
0.0
0.0
0.0
0.2
0.2
0.0
0.0
0.4
0.0
0.0
0.4
0.0
0.0
0.2
2.8
-0.4
D.O
0.2
0.7
0.0
0.0
0.0
0.0
0.5
004
0.0
0.0
0.2
0.0
0.2
0.0
D.O
D.G
0.4
3.0
0.2
0.0
0.0
0.2
02
0.0
0.0
-02
0.4
0.0
-,2
-02
0.5
0.0
0.
0.51
0.0
0.0
0.0
0.0
0.0
-02
0.0
0.2
0.0
0.0
00
-0.2
0.2
0.2
0.21
022
0.2
0.2
0.0
0.0
0.0
-0.7
0.0
0.5
0.0
3.1
0.0
02
0.0 0.0
-.2
0.0
0.0
0.0
0.4
0.0
-0.4
0.0
0.0
0.0
-0.2
0.0
0.0
0.0
2.2
-
0.0
0.0
0.0
02
02
0.0
0.0
0.0
0.6
022
0.0
0.0
0.0
0.0
0.0
02
0.0
002
0.2
02
0.2
0.0
0.0
02
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
02
+
Step 2:(17,13,0)
17

---

Published as a conference paper at ICLR 2026
0.0
-0.2
0-5.5
-0.4
233
0.0
0.0
0.0
0.0
0.0
0.0
-0.4
0.0
00
-02
0.0
2.0
-0.2
0.0
0.0
-0.5
0.0
0.0
-0.2
0.0
0.0
0.
02
02
0.0
00
0.0
-02
0.2
0.0
-0.2
2.9
0.0
-0.2
0.0
-0.2
-.2
-.2
-0.4
0.0
0.0
0.2
0.0
0.0
1-.
0.0
0.0
0.0
0.0
-0.
0.0
1.6
0.2
0.0
0.0
0.0
0.0
0.0
0.0
02
004
0.0
0.0
-0
0.0
0.0
-0.5
0.0
0.D
0.2
2.7
-0.5
0.0
0.0
-0.4
0.2
0.4
00
0.2
0.0
0.0
0.0
0.0
0.0
0.0
-.2
0.0
0.0
0.5
2.7
0.2
0.2
0.0
0.0
DO
0.0
0.0
0.4
044
0.0
0.2
0.2
0.5
-0.2
0.2
0.0
0.0
0.2
2.7
0.2
0.6
0.0
0.2
-0.
0.0
0.0
0.4
0.0
0.0
0.4
0.0
0.0
02
2.7
0.44
0.0
-
0.2
0.0
000
0.0
0.0
-0.5
-0.4
0.0
0.0
0.2
0.0
0.0
0.0
0.6
0.4
3.0
0.2
0.0
0.0
0.2
02
0.0
D.5
0.0
02
035
0.D
0.0
0.0
0.0
0.D
02
02
0.0
00
0.2
-0
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.7
0.5
0.0
0.21
0.0
0.0
0.
0.0
02
0.0
0.0
044
0.0
0.0
0.0
02
00
02
0.
00
0.0
0.0
0.0
02
-
02
0.0
0.0
0.0
0.0
-
02
0.0
0.0
0.
02
00
T.5
0.2
02
0.
0.6
0.0
-02
Step 3:(8,4.0)
3.7
-0.4
0.2
0.22
0.0
0.0
0.0
0.0
00.5
002
-0.2
-0.5
0.0
-0.5
-0.2
-.2
0.0
0.0
-0.2
-0.4
2.3
0.0
0.0
0.0
0.0
0.0
0.0
0.0
00.5
0.0
-0.4
-0.2
0.0
-0.2
0.0
0.0
0.0
-0.2
0.2
2.9
0.0
0.0
0.2
-02
-0,5
-0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
62
0.0
0.0
02
0.0
02
02
0.2
00
0.0
0.0
0.0
030
0.7
0.0
16
0.0
0.5
0.0
0.3
0.0
6.0
--
004
0.0
8.0
002
2.7
-0.
0.0
0.0
0.0
0.2
U.D
0.5
2.7
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.44
0.4
0.0
-
0.5
0.0
0.0
0.21
0.21
0.0
0.0
0.2
9
0.0
0.0
0.0
0.4
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
0.0
0.0
0.41
U.0
0.0
0.4
0.0
0.9
0.2
2.7
0.4
0.0
0.2
0.7
0.0
0.0
0.0
0.0
-0.5
0.4
0.0
0.0
0.2
0.0
0.2
0.0
0.0
0.6
0.4
3.0
-02
0.0
0.0
-0.
0.2
0.0
0.0
00.5
0.0
0.0
0.2
0.0
0.0
-0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.7
0.0
0.5
0.0
3.1
0.0
0.2
6.0
0.0
0.2
0.0
0.0
0.0
0.2
0.4
0.0
0.44
0.0
0.0
0.0
-02
0.0
0.0
0.0
2.2
-02
-0.2
0.0
-0.2
0.0
0.0
0.0
0.0
0.0
D.0
0.0
-.2
0.0
0.2
0.2
0.2
0.2
0.0
U.0
0.2
0.0
0.2
0.9
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
0.2
2.1
Step 4:(1,,5,1)
3.7
-0.4
-0.2
-0.2
0.0
0.0
0.0
0.0
0.5
0.2
0.2
0.5
0.0
-0.5
0022
0.2
0.0
0.0
00.2
0.0
0.0
0.4
0.2
0.0
2.0
0.2
0.5
0.0
0.0
0.0
0.0
0.2
0.0
0.2
2.8
0.0
0.21
0.0
0.2
0.2
0.2
0.4
0.0
D.0
0.5
0.2
0.0
0.0
0.2
0.0
0.0
0.0
0.0
0.0
1.4
0.0
0.0
0.0
0.2
0.0
0.0
-02
0.0
0.0
0.2
-02
0.2
-.2
0.0
502
6,2
0.0
0.G
0.2
00
502
-02
0.D
0.D
0.5
0.0
0.0
0.7
0.3
0.0
00.4
0.2
0.4
6.00
0.2
0.0
0.0
8.0
0.0
0.0
0.0
0.0
-0.
0.0
0.0
0.5
2.7
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0..
0.4
0.0
-0.2
0.0
0.2
0.0
0.2
2.6
0.6
0.0
0.0
100
0.2
0.0
0.0
0.4
0.0
0.0
--4
0.0
0.0
0.2
2.7
0.4
D.0
0.2
0.7
0.0
0.0
0.0
0.5
0.4
0.0
0.0
0.2
0.0
0.2
0.0
0.0
D.G
0.4
3.0
0.2
0.0
0.0
0.2
02
0.0
0.0
0.5
0.0
0.2
0.5
0.0
0.0
D.0
0.0
0.0
0.
002
022
0.2
0.2
0.2
0.2
0.21
0.0
D.9
0.0
0.7
D.0
0.5
0.0
3.1
0.0
021
6.0
0.2
0.0
0.0
0.0
0.2
0.0
0.4
0.0
0.0
0.0
0.2
0.0
0.0
0.0
2.2
-021
-22
0.0
0.0
02
02
0.0
0.0
0.0
-02
0.0
0.0
00
6.0
02
000
166
-02
0.2
0.2
-.2
0.0
0.0
0.2
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
-02
0.2
Step (:(.2.00.
3.7 00.4'-02''02
0.0
0.0
0.0
0.0
-0.5
-0.
-02
0.5
0.0
-0.5
-02
-02
0.0
0.0
-0.2
0.4
0.0
122
62
02
0.6
0.5
0.0
D.
0.6
0.0
D.0
00
0.
0.0
0.0
8.
0.2
0.0
0.2
28
0.0
0.2
0.0
0.2
0.2
0.2
04
0.0
0.0
0.5
02
0.0
0.0
-.2
0.0
0.
0.6
0.0
62
0.6
-62
0.5
0.0
0.0
0.0
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
00.4
0.0
-02
-02
0.0
0.0
0.5
00
0.0
2.6
0.5
0.0
0.0
0.4
0.2
0.4
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.5
0.0
0.0
0.0
0.2
0.0
0.0
0.0
0.0
0.0
-02
-051-021-0220000000000
0.0
0.2
0.0
2.6
0.2
-0.6
0.0
0.0
0.0
0.0
0.0
0.2
0.0
-02 0.0
0.0
-04
0.0
0.0
022
2.6
-0.4
0.0
0.2
00.7
0.0
0.0
0.0
0.0
-04
0.0 -02 0.0
0.
0.4
0.0
0.4
0.0
-0.2
1.8
0.2
0.0
0.0
0.0
-0. 0.0
-02
0.0
0.0
0.0
0.0
0.0
-0.
0.0
0.2
0.0
0.0
-02
0.0
022
002 022
0.2
0.0
0.0
0.7
0.0
-
0.5
0.0
3.0
0.0
02
0.0
0.0
-02
0.0
-02
0.0 0.0
0.0
0.4
0.0
0.2
0.0
0.2
-02
0.0 0.0
0.0
0.2
02
0.2
0.0
0.0
0.0
0.2
0.0
0.0
0.0
0.2
-02
0.0
1.6
-02
0.2 -0,2 -022
0.2
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.2
-02
2.1
18


Reviews:
[{"review_id": "2i0dCIz3iY", "reviewer": "Reviewer_xuUo", "summary": "This paper addresses the inefficiency of current arena-based evaluations for Large Language Models (LLMs), such as Chatbot Arena. At present, these arenas depend on random or exhaustive model pairings for human preference annotation, which results in significant redundant evaluations and a waste of resources.\n\nTo address this, the paper proposes an adaptive model-pair selection algorithm. The core of this method leverages the statistical properties of ELO rating estimation (asymptotic normality) and introduces Fisher Information as a metric to actively select the most informative (i.e., highest \"value\") model pairs. Specifically, the paper proposes: A-optimality: Aims to minimize the total variance of the model ability estimations. And D-optimality: Aims to maximize the determinant of the Fisher Information Matrix (FIM) to globally reduce the uncertainty in model rankings.", "questions": "A-optimality demonstrates a severe sampling bias in the experiments, showing an extreme tendency to select pairs involving the \"anchor\" model. Why does D-optimality manage to avoid this issue?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The problem formulation is significant and compelling.\n\nThe method is validated effectively through experiments.", "weaknesses": "The paper's theory is entirely established upon the m-ELO framework, with the FIM being derived from its likelihood function. However, in the experiments, the A/D-optimality strategies, which are based on this specific FIM, are applied to both m-ELO and traditional ELO. This is problematic because traditional ELO is an iterative update algorithm and does not directly optimize a global log-likelihood function.\n\nThe robustness of the heuristic experiments needs further validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lv3P5kacyx", "reviewer": "Reviewer_Mho6", "summary": "This paper introduces a method to improve the efficiency of model comparisons in a chatbot arena style setting consisting of repeated pairwise comparisons between models. Specifically, it uses the Fisher information of a matrix containing information about previous comparisons to guide selection of model pairs to consider. They introduce two specific metrics using Fisher Information, and show on real-world data that using them for model pair selection improves efficiency while still being robust and reliable.", "questions": "1. In the visualization section, you mention that D-Optimality might just be better in general in the arena setting than A-optimality. Is there an intuition behind what scenarios A-optimality might be better?\n2. How much impact does using a smaller initialization set have?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. I think the problem of improving the efficiency and reliability of pairwise evaluations is incredibly compelling yet under-explored. There is a good amount of work exploring the efficiency problem when considering large benchmarks, but there are fewer that look at the \"arena\" style comparisons, which are nonetheless very popular. \n2. The experiments are thorough and consider real-world data at a reasonable scale. Multiple baseline methods are compared against, as well as multiple variants of ELO. \n3. The paper contains a rigorous theoretical argument for their approach.", "weaknesses": "1. While I think the baselines are well-chosen, I would have liked to have seen some more comparisons with other active learning methods. I'd especially be interested in comparisons regarding how much data is required for initialization for other methods, and if this new method addresses the cold start problem in a much more efficient way. \n2. The relative performance gains don't seem too substantial considering the additional cost of the method. I think the authors address the caveats in their comparisons well, but I think some further analysis would be warranted to get a finer-grained understanding of the trade-offs between additional computational resources required at scale for the method and annotation volume. \n3. I think one major assumption made here is a fixed prompt distribution. Intuitively, I'd expect the specific test instance chosen to have a major impact as well on determining the relative quality of models. I understand this is likely beyond the scope of this paper, but I think some comparisons with methods that perhaps only consider this factor would be useful, since it could then help compare the impact of properties of the test input and properties of the model pairs themselves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CrpJ23lo3l", "reviewer": "Reviewer_d3Aw", "summary": "The paper introduces an information-theoretic approach to Arena-based LLM evaluation, applying Fisher information and optimal experimental design (A-/D-optimality) to select the most informative matchups.", "questions": "1. The explanation for Figure 5 (Line 468 - Line 473) is confusing. Can you explain more details?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is a elegant extension of optimal design principles to LLM ranking tasks. The inclusion of Top-K concurrent battles and adaptive scheduling shows good awareness of practical constraints in large-scale evaluation pipelines.\n2. The experiments demonstrate that the information-efficient pairing strategy achieves faster convergence and higher ranking consistency with fewer matches. The information-based metrics (A-info and D-info) are intuitive and interpretable for monitoring evaluation efficiency.", "weaknesses": "1. The “ground truth” ranking is derived from the same model used in the method, introducing potential circularity. It would strengthen the paper to include an external or human-verified ranking baseline to validate real-world correlation.\n2. The asymptotic normality and information growth results assume an Erdős–Rényi random graph of matchups. However, the real algorithm chooses pairs adaptively based on prior results, violating the independence assumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)