am-ELO: A Stable Framework for Arena-based LLM Evaluation
Authors: Zirui Liu
Jiatong Li
Yan Zhuang
Qi Liu
Shuanghong Shen
Jie Ouyang
Mingyue Cheng
Shijin Wang
Venue: ICML 2025 spotlightposter (2025)
Keywords: Large Language Models
Evaluation
Chatbot Arena
ELO Rating System
Primary area: general_machine_learning->evaluation

Abstract: Arena-based evaluation is a fundamental yet significant evaluation paradigm for modern AI models, especially large language models (LLMs). Existing framework based on ELO rating system suffers from the inevitable instability problem due to ranking inconsistency and the lack of attention to the varying abilities of annotators. In this paper, we introduce a novel stable arena framework to address these issues by enhancing the ELO Rating System. Specifically, we replace the iterative update method with a Maximum Likelihood Estimation (MLE) approach, m-ELO, and provide theoretical proof of the consistency and stability of the MLE approach for model ranking. Additionally, we proposed the am-ELO, which modify the Elo Rating’s probability function to incorporate annotator abilities, enabling the simultaneous estimation of model scores and annotator reliability. Experiments demonstrate that this method ensures stability, proving that this framework offers a more robust, accurate, and stable evaluation method for LLMs.

am-ELO: A Stable Framework for Arena-based LLM Evaluation
Zirui Liu
Jiatong Li
Yan Zhuang'
Qi Liu #12
Shuanghong Shen 2 Jie Ouyang
Mingyue Cheng
Shijin Wang"
Abstract
1050
Anno 9965
Anno 13046
Arena-based evaluation is fundamental yet sig- nificant evaluation paradigm for modern Al mod- els, especially large language models (LLMs). Ex- isting framework based on ELO rating system suf- fers from the inevitable instability problem due to ranking inconsistency and the lack of attention to the varying abilities of annotators. In this paper. we introduce novel stable arena framework to address these issues by enhancing the ELO Rat- ing System. Specifically, we replace the iterative update method with Maximum Likelihood Es- timation (MLE) approach, m- m-ELO, and provide theoretical proof of the consistency and stability of the MLE approach for model ranking. Addi- tionally, we proposed the am-ELO, which modify the Elo Rating's 's probability function to incorpo- rate annotator abilities, enabling the simultaneous estimation of model scores and annotator relia- bility. Experiments demonstrate that this method ensures stability, proving that this framework of- fers a more robust, accurate, and stable evaluation method for LLMs.
Anno 257
Anno 3820
1025
1000
ELO Score
975
950
925
wiz-13b
koala-13b vicuna-7b
gpt-13b
mpt-7b
Figure 1. An example of ELO score. The error bar represents the standard deviation and the error line represents the difference between the maximum minimum value and the mean value. The line chart represents the ELO scores estimated from records the specific annotator.
assessing the relative capabilities of competitors in games, forms the thcoretical foundation for the evaluation systems in most existing model arenas (Bai et al., 2022; Boubdir et al., 2023).
A significant issue with the current ELO method is its insta- bility, which can be attributed to two main factors: 1) From an algorithmic perspective, the existing ELO method treats the data as dynamic, making the results highly sensitive to the order in which the data is presented (Aldous, 2017: Li et al., 2024; Zhang et al., 2024a). In other words, when the same records are shuffled and re-evaluated, the ELO method often yields inconsistent scores. For instance, shown in Figure I, the significant error (highlighted in gray) compli- cates the comparison of models with similar abilities. 2) The judgment of human annotators varies across different as- pects such as quality, relevance, and importance of texts. For example, in the line chart in Figure 1, different annotators provide inconsistent ELO scores for each model. However, the arena-based evaluation paradigm, which involves human participation, overlooks these individual differences among humans (Welinder & Perona, 2010; Raykar & Yu, 2011).
1. Introduction
The rapid advancement of large language models (LLMs) (Jin et al., 2024b; Ouyang et al., 2025; Cheng et al., 2025) has led to the proliferation of "model tremaas-platffoe designed to compare and evaluate multiple models, identi- fying their relative strengths and weaknesses (Chiang al., 2024). These arenas play a critical role in driving inno- vation and shaping the deployment of cutting-edge LLMs across diverse applications. The ELO rating system (Elo, 1967), a well-cstablished methodology for quantitatively
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, Hefei, China "Institute of Arti- ficial Intelligence. Hefei Comprehensive National Science Center, Hefei, China "IFLYTEK Co., Ltd, Hefei, China. Correspondence to: Qi Liu <qiliugl wwt.usss..co.. edu.en>.
Ignoring this variability introduces biases and instability into the evaluation process, further undermining the cred- ibility of both the results and the decisions derived from
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
competing directly with others. Current research in this area can generally be divided into three key categories: Battle Scenarios, Annotators, and Ranking Systems.
ELO (C
Iterative
Record
ELO Score
Battle Scenario The classic battle scenario is exemplified by the Chatbot Arena (Chiang al., 2024), in which models respond to the same question and annotators compare their outputs. However, this approach are susceptible to the inher- ent biases of the annotators. To address this issue, several studies have incorporated multiple models working collab- oratively to generate and evaluate responses, enabling iter- ative improvements (Zhao et al., 2024). Notable examples of this approach include LLMChain (Bouchiha et al., 2024) and ChatEval (Chan et al., 2024). While such strategies of- fer increased fairness, they come with trade-offs, including higher computational costs and potential instability.
MLE
m-ELO
Annotator Ability O
am-ELO (Our proposal)
Figure The traditional iterative ELO method and our proposed am-ELO method based MLE,
them (Eickhoff, 2018). These instabilities diminish the in- terpretability and practical value of ELO scores, eroding confidence in the conclusions drawn from such evaluations, particularly when they are used to inform high-stakes deci- sions regarding model deployment or research directions.
Annotator In arena-based evaluation, the comparison of results typically involves human annotators (Cheng et al., 2024) or highly capable LLMs, such as GPT-4 (Achiam et al., 2023) and Claude (Anthropic). Additionally, some researchers have explored the use of specialized referee models for this task, such as PandaLM (Wang et al., 2023), JudgeLM (Zhu et al., 2023), and Auto-J (Li et al., 2023), which are designed to enhance the evaluation process.
In this work, we propose a novel stable arena framework to address these shortcomings. As illustrated in Figure 2, to mitigate the inconsistencies in ELO scores, we introduce maximum likelihood estimation (MLE)-driven ELO rating method, referred to m-ELO. By deriving the theoretical properties of this reformulation, we demonstrate that the proposed method produces consistent results without alter- ing the fundamental principles of the original ELO method. Furthermore, to account for variability in annotator perfor- mance, we propose an annotator ability-aware enhancement method for ELO (am-ELO), grounded in psychometries (Morizot et al., 2009; Furr, 2021). By modifying the ELO probability function, we estimate the annotator's 's ability and adjust their contribution accordingly, leading to more ac- curate and equitable aggregation of evaluation results.
Ranking Systems for LLM Evaluation Ranking systems play crucial role in arena-based LIM evaluation (Busa- Fekete et al., 2014; Szörényi et al., 2015; Chernoff, 1992). Among the existing approaches, many arena- based methods rely on the ELO Rating System to model LLMs' capabilities (Coulom, 2007: Pelánek, 2016). The ELO rating system, grounded in the Bradley-Terry model (Hunter, 2004; Rao Kupper, 1967), is widely used in competitive games (Sisma- nis, 2010; Ebtekar & Liu, 2021) to predict the likelihood of one competitor outperforming another based on their rela- tive abilities. However, due to its dynamic nature, which is tailored for traditional competitive games, the ELO system introduces instability in LLM evaluation. To mitigate this instability, existing approaches typically perform multiple random shuffles of the annotated dataset and calculate ELO scores for each iteration (Sismanis, 2010). The statistical summary, such as the mean or variance of the scores across these shuffles, is then used as the final evaluation metric. Although this strategy provides a practical solution, it does not fundamentally resolve the inconsistency introduced by the sequential updates in the ELO method.
Through experiments on real-world datasets, we demon- strate that our framework effectively models annotators while ensuring the consistency of ELO scores. Furthermore, in simulation experiments,
method not only identifies anomalous annotators but also reduces the inconsistency of ELO scores to 30% compared to the traditional ELO method. This indicates that our approach effectively mitigates the instability inherent in the traditional ELO method.
2. Background and Related Work
Arena-based evaluation is an important subfield within the broader domain of L.LM evaluation. Unlike traditional evalu- ation paradigms (Zellers et al., 2019; Hendrycks et al., 2020; Cobbe et al., 2021; Liang et al., 2022; Jin et al., 2024a), which typically assess a model's performance against prede- fined benchmarks, arena-based evaluation involves models
3. Preliminary
Arena-based Evaluation is a highly anticipated method in
LLMs evaluation, where models are compared head-to-hean on benchmarks or datasets and the results are annotated by evaluators. Let S = {(i,j, k, W.))li, E [N], & E [M]]
2

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
Algorithm 1 The Traditional ELO Rating System
4. Better Performance estimation with ELO
Input: Dataset S, Scaling Factor K, Init Score Ranit
Earlier, we introduced the traditional ELO method and high- lighted its key challenges, including ranking inconsistencies and the lack of consideration for annotator variability. To address these issues, this section presents stable arena framework with improvements to the ELO method.
Initialize: Set of Scores RS; <- 0, Score of Models
R2 < Rinst
Calculate ELO Score:
for (i. J. Waz) E S do
R - Rg + K (We - P(Rs. R2))
R2 + R3 + K (Wj+- P(Rj, Ra))
4.1. MLE for ELO (m-ELO) Estimation
end for
Output: ELO Score (Ri,-.. Rr)
The traditional ELO rating estimation method is based on an iterative algorithm, and the results are highly dependent on the order of the samples. This explains why ELO rat- ings often lack consistency. Inspired by the insensitivity of maximum likelihood estimation (MLE) to the sample or- der, we propose an MLE-driven ELO estimation algorithm, termed m-ELO. Specifically, for the record dataset S, its log- ong-likelihoo function can be expressed follows:
represent the comparative dataset we have collected, where N is the number of models and M is the number of annota- indicates that model and model j engaged in a battle, and annotator k provided the result Wi. Specifically, Wij 1 indicates that model i won the battle, Wi, 0 indicates that model won, and Wi = 0.5 indicates a tie. The goal of the arena-based eval- uation is to estimate the ranking scores R = (Ri,-... Rx) for the models based on the record S.
tors. Each element (i. j. k, W)
In L Σ Wiy In P(R,, R2) + Wf In P(Rj. R.),
(b..WW,)ES
(2)
where P(R;, R2) 1-6 -0R The result of the MLE method, (Ri, Ri...... R,y, can be obtained by solving the extreme point of the ligeliiel function using gradient descent. Specifically, for any given model n E [NV] the gradient of the log-likelihood function with respect to its rating R2 is:
ELO Rating System The ELO rating system is a widely used method for ranking competitors based on pairwise com- parisons. In the ELO system, each competitor (or model) is assigned rating R, which represents its relative strength. When two models, i and j. compete, their respective rat- ings, R2 and Rj. are used to calculate the expected prob- ability of each outcome: P(R,, R2)) = P(W,, = 1) = 1te -in where C' is a constant that scales the differ- ence in ratings. After observing the actual outcome of the match, the ratings are updated as follows:
1 OR,
Σ C(W,) P(Rn, R,)), (w.....))ES
(3)
By comparing Equations 1 and 3, observe that the two formulas share consistent structure. This highlights the essence of the ELO algorithm: it performs gradient descent K with a learning rate of δ on the MLE for each annotated sample. Gradient descent based on individual samples rarely converges, which reveals a key shortcoming of the tradi- tional ELO method.
R = K (Wy- P(R,, R2)), R3 R K (W,; - P(R,. R.)),
(L)
Convergence Analysis Although the estimation results of the MLE method are not influenced by the sample order, an- other important consideration is whether the log- ligeliiel function has only one extreme point. If multiple extreme points exist, it could still lead to inconsistencies in the ELO rankings. Unfortunately, because ELO scores are relative, is clear that if (Ri, Ri..... Rty) is an extreme point, then (Ri + E, RS + e,. ,Ry + E) is also an extreme point, Thus, the extreme points of the log- englliellhood function not unique. However, when we fix the score of one of the models, we obtain the following theorem (Zermelo, 1929);
where K is scaling factor that controls the magnitude of rating changes. The pseudo-code for this process is shown in Algorithm 1. However, the existing ELO method is iterative and highly sensitive to the order of the data. This is irrational for LLMs' evaluation because evaluation can be seen as a static process (Zhan et al., 2024). Specifically, the errors introduced by the ELO method arise from the algorithm's is dynamics rather than the data itself, which undermines the statistical significance of the ELO scores for many models.
Moreover, current algorithms do not account for differences in annotator abilities. They treat all annotators as if they have the same ability C, mixing annotation records ran- domly. This assumption can introduce bias and instability into the evaluation process.
Theorem 4.1. Assume Ro = 0 and |S] is sufficiently large, then the log-likelihood function In L with respect to (Rg...., Ry) is a concave function and has at most one extreme point.
3

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
Drawing from Theorem 4.1, can assert that the ELO score obtained through the MLE method is relatively sta- ble between models, meaning that the difference in ability between any two models remains stable.
To estimate the parameters of the probability function, we consider its logarithmic likelihood function similarly:
Wiz In P(R, R5(02) + W In P(R, Rg (k)) (i..A,,;)es
Σ
Replacing the iterative method with the MLE approach makes the ELO method more flexible. Additionally, it al- lows us model annotator abilities during the evaluation process. In the next section, we will adopt ideas from psy- chometrics to propose feasible modeling approach and analyze its interpretability.
(5)
After modifying the probability function, we need to account for both the ELO scores of the models R = (Ri,..., Rx) and the abilities of the annotators θ = (91..... 0xx) during gradient descent. For a model n E [N] and annotator m E [M], the gradients of In L to them can be expressed as:
4.2. Annotator Modeling m-ELO (am-ELO) Estimation
1lnL Σ
On mj - P(Rn, (2))
2Rn (s...
Although ability modeling is not commonly seen in LLM evaluation, many ability modeling methods have been de- veloped in education and psychometries (Liu et al., 2021: Wang et al., 2022; Zhang et al., 2024h; Zhuang et al., 2022;
0l
E
(R, - R,), - P(R., Ry)0m))
d0m.
(.j...WayEE
(6)
Liu et al., 2024). One prominent method is Item Response Theory (IRT) (Embretson & Reise, 2013; Zhu et al., 2022: Nguyen & Zhang, 2022; Polo et al.. 2024). IRT posits that an examinee's 's performance on a test depends solely its ability 0 and the properties of the questions. The standard model is the two wwo-paaamttrr logistic (2PL) model, defined as: (0) = Plu-- = 1) I+c 150-57 where Vj = 1 in- dicates a correct response question j, and aj and by E TR represent the discrimination and difficulty of question j.
This method allows us to simultaneously estimate the an- notators' abilitics during the MLE process. Beyond the concept of discrimination introduced by the improved prob- ability function, we should also explore the practical signifi- cance of this ability estimation in the context of the arena. Through analysis, we find that the estimated annotator abil- ity 0μ exhibits the following two properties:
Theorem 4.2. Given that 0 represents the ability of annota- tors estimated by am-ELO, the following conclusions can be drawn:
As noted, the parameter a in IRT can be interpreted as the discrimination parameter. Similarly, in the ELO method, the fixed value C can also be understood as the discrimination parameter. To account for annotator variability, we replace the fixed value C' the probability density estimation with parameter 6g that is specific to annotator k:
(1) two annotators label the same set of samples Wejs W with abilities 0₁ and O2 (02 > 01), ), then:
Σ - -R,)Wa < (K..
E
(R, (R-RR)Wer R))
(.A.DDES
1
P(R,, R2 + e RR)
1f < O, for each positive sample (i,j, k,k,, 1) of anno- tator k, its loss 8in1
< 0, and for each negative sample (i. k. 0) of annotator k, 0lml 0.
This new formulation has the following properties:
OR,
BR.
Maintain symmetry: The symmetry to the model's abilities R& and R2 is preserved even after modifying the constant C' to an annotator-relate related parameter f,. such that P(Rs, Rz) + P(R,, Rx) 1
From Theorem 4.2, it is evident that the annotator abilities derived from MLE have practical significance. Specifically, Ei., R, 1R,-R, RR, Wa,, can be interpreted as the correla- tion between the annotations Wi, and the rankings R; R,- Theorem 4.2 (1) implies that a higher annotator ability cor- responds to a greater value of E(.s, W., W.,)es (R, -R,,,,,,,, W. meaning that a larger 0g indicates that the annotations from annotator k are more consistent with the overall rankings. Meanwhile, Theorem 4.2 (2) suggests that an annotator with negative ability might annotate inconsistently or arbitrarily, and am an- can identify these anomalous annotators.
Discriminative ability (0, > 0): When the abili- ties of two models are identical, the change in win probability caused by small variations in ability val- ues is positively correlated with annotator's ability
0g- = 4 (P((
aR: R.=r. Therefore, the annotator's 's abil- ity 0g represents the maximum discriminative ability.
A Anomalous annotator (0,- < 0): When the discrim- inative ability 0z. it is observed that for any model with greater ability than model j. annotator l perceives the probability of model winning as less than 0.5. This indicates that it is an anomalous annotator.
Normalization Although this method has strong in- terpretability for modeling annotators, is not diffi- cult to observe that, for such an optimization problem, if (Ri, Ri,, ".., , an extreme point, then
4

---

Ln: A Stable Framework for Arena-based LLM Evaluation
Algorithm 2 The am-ELO Rating System
Building on the improvements to the ELO method discussed earlier, we introduce the Stable Arena Framework, novel paradigm for arena-based evaluation, as detailed in Algo- rithm 3. To ensure more robust evaluations, we carefully screen the annotated data both before and after applying the am-ELO method. Specifically, upon incorporating new annotation samples, we first filter out annotators who have fewer than 6 annotation records. This is crucial because annotators with fewer records tend to produce less reliable results. However, this does not imply permanent exclusion; once such annotators accumulate a sufficient number of annotations, their records will be reconsidered.
Input: Dataset S, Learning Rate a, Epoch Epoch Initialize: Score of Models R and annotators' ability θ for I. 1 to Epoch do
Calculate MLE: In L <- MLE(R, (,)5)
Optimize: R <- R +
8ln θ 6
L 08
OK
Normalization: θ 4- 1
e
end for
Output: ELO Score and annotators' ability (R, ().
Algorithm 3 The Stable Arena framework
Input: Learning Rate a, Epoch Epoch, Ability Thresh- old e.
After evaluating both models and annotators, we further refine the process by filtering annotators based on their estimated abilities. Annotators with negative ability values, or those with ability values below a threshold c, are deemed detrimental to the evaluation process. For these annotators, we either issue warnings or exclude them entirely from further evaluations. Moreover, since a higher 0 indicates greater consistency between the annotations and the overall ranking, the L.LM evaluation platform can reward annotators proportionally their demonstrated abilities.
Initialize: Dataset S <- 0, Data quantity threshold 8. while True do
S SUS. Snew
for k= = to M do
6% H((i Wa KH
end for
S' - [i.,j, k W. 10g > 8}
(R. Q) < am-aam--aaa ELO(S', a, Epoch)
(Ra.... RRy)= = R
Output: EL.O Score (R1, 1., RN)
(0,,-.. ,0x) =
5. Experiments
Filter annotators: S + [(i.). K W,, >
end while
In this section, we introduce and compare the performance of our proposed method with the traditional FL.O method in predicting annotation results, highlighting the superior modeling capability of am- An-.O.. Additionally, we demon- strate the limitations of the traditional ELO method through comparison of model rankings produced by various ELO methods and a case study, Next, we validate the convergence of the ELO rankings generated by our method, further rein- forcing the validity of our approach for evaluating LLMs. Finally, to assess the stability of the ELO method, we apply four different strategies to perturb the annotators. Our re- sults show that our method not only maintains stability in the tests but also effectively identifies anomalous annotators, emphasizing the superiority of our approach.
(aR .RN, .RR 2 07. 1 (ir) is also an extreme point. Thus, when a < 0, the model score ranking will be com- pletely reversed, leading to potential instability. To mitigate this issue, we impose a constraint on the annotator's ability:
0 + 02 + .... 0m 11 1.
(7)
From Theorem 4.2 (2), we know that 0g corresponds to users who annotate normally. The significance of this normalization operation is essentially based on the assump- tion that the majority of annotators in the group are labeling responsibly (Nowak & Rüger, 2010). Based on this as- sumption, we determine whether the model rankings should follow the original order or be reversed.
5.1. Dataset
We conduct experiments real annotation dataset, Chat- bot (Zheng et al., 2023), which was collected from 13,000 distinct IP addresses in the Chatbot Arena between April and June 2023. The dataset consists of 33,000 curated con- versations with pairwise human preferences. Each entry includes a question ID, the names of two models, their full conversation transcripts, the annotator's vote, and its ID, Due the requirement for MLE in this experiment, indi- vidual samples may introduce instability. Consequently, excluded annotator samples with fewer than 50 annotated records. The statistical information of the filtered dataset is shown in Table
4.3. Stable Arena Framework
Algorithm 2 presents the pseudo-code for the am-ELO al- gorithm. The am-ELO algorithm performs gradient descent (Ruder, 2016) on the negative log-likelihood function over the entire dataset find the extreme point, ultimately return- ing both the model scores and annotator abilities. Specif- ically, when considering only the m-ELO algorithm, the concavity of its log-likelihood function enables the use of Newton's 's method (Galántai, 2000; Kelley, 2003) during optimization. This allows for dynamic adjustment of the learning rate, thereby improving convergence efficiency.
5

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
1.0
am-ELO
0.9
0.8
m-ELO
ELO Scare (Nom)
0.6
ELO
0.8
0.4
Los
0.7
0.2
0.0
0.6
0.2
0.5
0.4
spt-4
claude-vI
gpt 3-rarbo
claude-ing mande-ins-vl ins-v1
palm-2
vicuma-1 1.3b
guanaco-33 333
wizardlm-I 13b
koala-1 1.3b
vicuma-77 7b
gpt- 13b
mpt- 7b-chat
Ilama- 13b
alpaca- 13b
RWKV V
oasst- 12b
fastchaf- Michatt-1539 3b
chargim-6h
stable-7b
dolly-v2-11
Figure 3. The result of each LLMs on different evaluation method. Specifically, the line chart represents the normalized ELO scorest (ranging from to 1) of each LLM under different evaluation methods. The chart represents the Loss (log-like likelihood function) each LLM's match records under different evaluation methods.
Table J. Statistics of the dataset
Table 2. The Performance of ELO method for prediction.
Dataset
Chatbot
Method
MSEJ
AUC1
#Annotators
42
ELO
0.1238 0.0031
0.74921 0.0068
#Models
20
m-ELO
0.34
0.0029
0.7503+ + 0.0066
#Response logs
4321
am- am-ELO
0.1208 0.0034
0.7581 0.0067
#Response logs per annotator
102.88
#Response logs per model
216.05
#Response logs per model pair
22.74
Better-
0
207
144
28
0
koala-13p
46
10
147
5.2. Setting
vicuna-7b
In this experiment, we consider baseline model, the tradi- tional ELO method, alongside two methods we proposed: m-ELO and am-ELO. For the iterative ELO method, we perform repeated experiments by shuffling the dataset 1000 times and averaging the results. The MIE is solved using the gradient descent (GD) approach with learning rate of 0.1 and a fixed number of 2000 iterations. The code can be found in the github: https: / http//ggttttttt/t//////////t thub. com/ aerdata-ustc// amm
gpt-13b-
11
0
13
Worse
0
42
11 0
Figure 4. The heatmap shows the number of victories in battles between various models (Three models with similar abilities, koala- 13b, vicuna-7b, gpt-13b, and better or worse models than them). Each number in the figure represents the times the row model wins the column model battle.
censt//ammEAi ELO.
5.3. Result and Case Study
also demonstrates that the improved probability function effectively models the annotators.
The bar chart in Figure 3 presents the mean log-likelihood loss for each method. As shown, the loss difference be- tween m-ELO and ELO, which share the same probability function, is minimal, while the loss for am am-ELO is signifi- cantly lower than the other two methods. This indicates that am-ELO demonstrates better fitting ability. Furthermore, as shown in Table 2, am-ELO significantly outperforms the other two baseline models in prediction tasks, suggesting that am-ELOO exhibits superior generalization ability. This
Meanwhile, the line chart in Figure illustrates the ELO scores obtained from the three ELO methods. It is clear that the ranking trends of our proposed methods align closely with the traditional EL.O method. However, there are some differences in the rankings of specific models, such as koala- 13b, vicuna-71 7b, and gpt-13b.
analyze these models with similar abilities, we categorize
6

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
0.70
1.0
all converge to a local minimum within limited number of iterations. The loss at convergence for m-ELO is nearly identical to that of the traditional ELO, which is expected since both methods share the same probability estimation function. This once again demonstrates that m-ELO and traditional ELO are essentially equivalent. Moreover, am- ELO (w/o Norm) converges the fastest, followed by am- ELO, with m-ELO being the slowest. This is because am- ELO has more adjustable parameters compared to m-ELO, and am-ELO (w/o Norm) benefits from fewer constraints during the gradient descent process. However, as seen from the consistency, am-ELO (w/o Norm) quickly converges to different local minima, and its consistency stabilizes at 0.4. This suggests that the five outputs of this method exhibit two ordered sequences and three reversed sequences CC++ Cg
Consistency
L
0.65
0.60
0.4
1000
2000
1000
2000
Epoch
Epoch
am-ELO
am - (w/o Norm)
i-ELO
ELO
Figure 5. The Loss and Consistency of the evaluation method at each epoch on the Chatbot dataset.
the remaining models into two groups based on their ELO scores: "Better" and "Worse", representing models that are better worse than the aforementioned models. We visualize the number of matches between these models. As shown in Figure 4, each number represents the number of times the model in the row defeated the model in the column. For example, the first row and third column indicate that vicuna-7b lost to "Better" models 148 times. From this, we observe that although the head-to-head records between koala-13b and vicuna-7b do not differentiate their abilities, both models defeated the same number of "Better" models. Meanwhile, 7bbuna-7blun lost to fewer "Better" and "Worse" models. Based on this result, we conclude that vicuna-7b is stronger than koala-130 13b, which aligns with the rankings provided by both am-ELO and m-ELO.
0.4). On the other hand, am-ELO not only achieves stable rankings after sufficient gradient descent iterations but does so more efficiently than m-ELO. This demonstrates that the proposed am-ELO method strikes balance between convergence and efficiency.
5.5. The Stability of Various ELO Methods
Since directly verifying the stability of the am-EL.O method during the evaluation process is challenging, we use simula- tion experiments to introduce perturbation to the annotators. Specifically, we perturb the annotators' results using four strategies to simulate the presence of anomalous annotators that may occur during testing:
However, due to Koala- 133's large number of victories over "Worse" models, the traditional ELO method disproportion- ately weighs these victories during the scoring process, ul- timately ranking Koala-138 higher than Vicuna-7B. This issue suggests that avoiding strong opponents and repeatedly defeating weaker ones could artificially inflate a model's ranking, which is an undesirable outcome.
If model A wins, the result will have 50% chance of being changed to "Tie" and a 50% chance of being changed to "model wins", vice versa.
Random:
Equal: All results changed to "Tie".
Flip: If model A wins, the result will be flipped to "model wins", and vice versa. The outcome "Tie" remains unchanged.
5.4. The Convergence and Efficiency of ELO Methods
In this subsection, we discuss the convergence and efficiency of the proposed am-ELO. Our comparison methods not only three mentioned model but also am-EL.O (w/o Norm), where normalization is not performed during training. To analyze the convergence and efficiency of the results ob- tained by each evaluation method, we record the loss (Loss) during the gradient descent process. Additionally, we per- form five random initializations of the model parameters and calculate the consistency of the rankings (Consistency) (Hastic& & Tibshirani, 1997) of the ELO scores output by these five models at each epoch. It should be noted that the iterative process of the traditional ELO method differs from the gradient descent approach of MLE. Therefore, we directly record the final output loss and consistency for the traditional ELO method. The results are shown in Figure 5.
Mixed: A random selection is made from the first three perturbation strategies for each instance.
These perturbations mimic scenarios where intentional mis- labeling occurs in annotations. Considering that the majority of annotators in the arena will annotate normally, the num- ber of perturbations in our simulation experiment will not exceed half of the total number of annotators. We expect stable scoring method to have two key properties: (1) should produce ELO rankings consistent with those with- out perturbations, and (2) it should identify the anomalous annotators. The ground truth for the consistency of the ELO score is the pairwise comparison between the ELO rankings with and without perturbations, and the ground truth for identifying anomalous annotators is the F1-score
As observed from the loss, the three MI.E-bsed methods
7

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
Random
Equal
Flip
Mixed
1.00
Consistency
0.95
0.95
0.90
0.96
0.90
0.8
0.85
0.10
0.10
Annocater Abilioes
0.05
0.05
0.05
0.05
0.00
0.00
0.00
0.00
-0.05
0.05
-0.05
-0.05
-0.10
-0.10
-0.10
20
40
20
40
20
40
20
40
Ratio of Pertrbation (%)
Ratio of Pesturbution (%)
Ratio of Perturhation (%)
Ratio of Perturbation (%)
um- ELO
m-E1.O
ELO
Positive Anmotators
Negative Annotators
Figure 6. This figure contains four line charts and four seatter plots, corresponding to the ELO score consistency under the four types of perturbation, as well as the changes in annotator abilities obtained from am-
am-ELO ELO as the level of perturbation increases.
Random
Equall
have ability scores below 0, indicating they are identified as noise points. Additionally, Figure 7 presents the F1- scores for detecting perturbations under thresholds of 0 and 0.005. Under different perturbations, the recognition accu- racy reached 90% when 0, and even up to 95% when = 0.005. These results demonstrate that our method ef- fectively detects perturbations, models the annotators, and maintains the consistency of results, thereby alleviating the problem of ELO inconsistency.
1.00
0.95
Flip
Mixed
1.00
0.95
6. Conclusion
In this study, we explored the instability of the ELO method in the context of LLM evaluation, emphasizing its impact on the reliability of evaluation outcomes. To address this issue, we introduced the Stable Arena Framework, which utilizes the MLE approach for ELO rating estimation and incorporates annotator ability parameters into the probabil- ity function. Our experiments demonstrated that am-FL.O not only achieves more stable convergence but also effec- tively identifies anomalous annotators, resulting in rankings that are more aligned with human intuition. These findings suggest that our approach can significantly reduce the insta- bility of ELO, enhancing the credibility and robustness of LLM evaluation, while providing a more stable and easily implementable framework for arena-based evaluation.
0
10
20
40
10
20
30
41
Ratio bation (5.)
Ratio Perturistion (5)
C- 0.008
Figure 7. The line chart of F1-score for outlier detection at different thresholds under the four types of perturbation.
(Chen & Lin, 2006) of the annotators' abilitics obtained from am-ELO. higher Fl-score indicates better accuracy in detecting the perturbations.
The line charts in Figure 6 show the relationship between the ratio of perturbations and the consistency of ELO scores. We observe that am- am-ELO maintains higher consistency across various types of perturbations. Specifically, aside from the fact that the "Equal" perturbation itself is unlikely to af- feet rankings, leading to high consistency across all ELO methods, in the other three perturbation scenarios, am-ELO reduces the inconsistency rate to 30% compared to m-ELO or traditional EL.O. Meanwhile, the scatter plot at the bot- tom of Figure 6 shows the changes in annotator abilities under each perturbation. Red dots represent annotators who were normal, while green dots represent those who were anomalous. is clear that nearly all anomalous annotators
However, our method has certain limitations. Specifically, the dimensions of annotator modeling are somewhat simplis- tic, as primarily captures the annotator's 's discriminatory ability and consistency with other annotators. This makes it challenging to fully capture the annotator's broader ca- pabilities. In future work, we aim refine the design of annotator ability dimensions to better leverage crowdsoure- ing for arena-based evaluation.
8

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
Acknowledgements
els: The case of mallows. In International conference on machine learning, pp. 1071-1079. PMLR, 2014.
This research was supported by grants from the National Key Research and Development Program of China (Grant No. 2024YFC3308200) the National Natural Science Foun- dation of China (62337001), the Key Technologies & D Program of Anhui Province (No. 202423k090200339, China Postdoctoral Science Foundation (Grant No. 2024M760725) and the Fundamental Research Funds for the Central Uni- versities.
Chan, C., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. Chateval: Towards better Ilm-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL https: htpp//peeeeeeeeeeeeeeeeeeeeeeeee net/ forum?id Forumiid-FOeppscUU
Chen, Y.-W. and Lin, C.-J. Combining svms with various feature selection strategies. Feature extraction: founda- tions and applications, pp. 315-324, 2006.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cheng, M., Zhang, H., Yang, J., Liu, Q. Li, L., Huang, X., Song, L., Li, Z., Huang. Z., and Chen, E. Towards personalized evaluation of large language models with anonymous crowd-sourcing platform. In Companion Proceedings of the ACM Web Conference 2024, pp. 1035- 1038, 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 23Xiv:2003.08774.4 2023.
Cheng, M., Luo, Y., Ouyang, J., Liu, Q. Liu, H., Li, L., Yu, S., Zhang, B., Cao, J., Ma, J., A survey on knowledge-oriented auugentel generation. arXiv preprint arXiv:2503.1060 10677, 2025.
Aldous, D. Elo ratings and the sports model: A neglected topic in applied probability? 2017.
Chernoff, H. Sequential design of experiments. Springer, 1992.
Anthropic, S. Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet. URL https: //api.semant www.seeeaniieeeeeeeeeeeeeeeeeeee ieschhlar....... org/CorpusID; 273639283.
Chiang, W., Zheng, L.,, Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M. I., Gonzalez, J. E., and Stoica, Chatbot arena: An open platform for evaluating Ilms by human preference. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: /ttp:://///peeeveee... wwwpeeiwwwwwwwweeeeeeeeeeeeeeeee forum? 1d=3MN8GKNyZ1.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXivv2204.05882. 2022.
Boubdir, M., Kim, E.,, Ermis, B., Hooker, S., and Fadace, M. Elo uncovered: Robustness and best practices in language model evaluation. arXiv preprint arXiv:2311.17295, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1411 2021.
Bouchiha, M. A., Telnoff, Q., Bakkali, S., Champagnat, Y. LImchain: Blockchain-basse reputation system for shar- ing and evaluating large language models. In 2024 IEEE 48th Annual Computers, Software, and Applica- tions Conference (COMPSAC), pp. 439-448, 2024. doi: 10.1 1.1100900MPSA61 10w.11........ 202.00067..
Coulom, R. Computing "elo ratings" of move patterns in the game of go. ICGA journal, 30(4):198-22)) 208, 2007.
R., Rabah, M., Coustaty, M., and
Ebtekar, A. and Liu, An elo-like system for massive mul- tiplayer competitions. arXiv preprint arXiv:2101.00400, 2021.
Eickhoff, C. Cognitive biases in crowdsourcing. In Pro- ceedings of the eleventh ACM international conference on web search and data mining, pp. 162-170, 2018.
Boyd, S. and Vandenberghe. L. Convex optimization. Cam- bridge university press, 2004.
Busa-Fekete, R., Hüllermeier, E., and Szörényi, B. Preference-based rank elicitation using statistical mod-
Elo, A. E. The proposed usef rating system, its development, theory, and applications. Chess life, 22(8):242 224, 1967.

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
Embretson, S. E. and Reise, S. P. Item response theory. Psychology Press, 2013.
Liu, Z., Yan, Z., Liu, Q. Li, Zhang, Y., Huang, Z., Wu, J., and Wang. S. Computerized adaptive testing via collabo- rative ranking. In Neural Information Processing Systems, 2024. URL https: app. semant cescholsh.r org/ CorpusTD:
Furr, R. M. Psychometries: an introduction. SAGE publica- tions, 2021.
Galántai, A. The theory of newton's method. Journal of Computational and Applied Mathematics, 124(1-2):25- 44, 2000.
Morizot, J., Ainsworth, A. T., and Reise, S. P. Toward modern psychometrics. Handbook of research methods in personality psychology, 407,2007
Hastie, T. and Tibshirani, R. Classification by pairwise coupling. Advances in neural information processing systems, 10, 1997.
Nguyen, D. and Zhang, A. Y. A spectral approach to item response theory. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. Curran Associates, Inc., 2022.
Hendrycks, D., Burns, C.. Basart, S., Zou, A., Mazeika, M.. Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint 20Xiv:2009.03300 2020.
Nowak, S. and Rüger, S. How reliable are annotations via crowdsourcing: study about annetannnttaor agreement for multi- image annotation. In Proceedings of the international conference on Multimedia information re- trieval, pp. 557-566, 2010.
Hunter, D. R. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32(():384-400 2004.
Jin, Y., Choi, M., Verma, G., Wang, J., and Kumar, S. Mm- soc: Benchmarking multimodal large language models in social media platforms. In ACL, 2024a.
Ouyang. J., Pan, T., Cheng, M., Yan, R., Luo. Y., Lin, J., and Liu, Q. Hoh: A dynamic benchmark for evaluating the impact of outdated information on etrreval--uggennt generation, 2025. URL https: //tpsi////xvvv..//a/ www.iiiiiigiiooooeoooooooooooooe /bs/ 25033 04800.
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., and Wang, J. Agentreview: Exploring peer review dynamics with Ilm agents. In EMNLP, 2024b.
O' Meara, O. T. Introduction quadratic forms, volume 117. Springer, 2013.
Johnson, C. R. Positive definite matrices. The American Mathematical Monthly, 77(3):259-22 1970.
Pelánek, R. Applications of the elo rating system in adaptive educational systems. Computers & Education, 98:169- 179, 2016.
Kelley, C. T. Solving nonlinear equations with Newton's method. SIAM, 2003.
Polo. F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., and Yurochkin, M. tinybenchmarks: evaluating Iims with fewer examples. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, net, 2024. URL https: openreview net / forum? i (0==Am13FFpff
Li, C., Shi, L., Zhou, C., Huan, Z., Tang, C., Zhang. X., Wang, X., Zhou, J., and Liu, S. merge sort based rank- ing system for the evaluation of large language models. In Bifet, A., Krilavičius, T., Miliou, I., and Nowaczyk, S. (eds.), Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track, pp. 240-255, Cham, 2024. Springer Nature Switzerland. ISBN 978-3- 031-70378-2.
Rao, P. and Kupper, L. I.. Ties in paired-comparist ex- periments: A generalization of the bradley-terry model. Journal of the American Statistical Association, 62(317): 194-204, 1967.
Li, J., Sun, S., Yuan, W., Fan, R.-Z., Zhao, H., and Liu, Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470. 2023.
Raykar, V. C. and Yu, S. Ranking annotators for crowd- sourced labeling tasks. Advances in neural information processing systems, 24, 2011.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., al. Holistic evaluation of language models. arXiv preprint ar 2 1.09//19111100 10, 2022.
Ruder, S. An overview of gradient descent optimiza- tion algorithms. ArXiv, abs/1609.04747, 2016. URL https://api weappssssattcccclarr...mmm CorpusID:1748526
Liu, Q. Huang, Z., Yin, Y., Chen, E., Xiong. H., Su. Y., and Hu, G. Ekt: Exercise- knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering. 33(1):100-115, 2021.
Shi-gu, Application of lagrange mean value theorem. 2014. URL http:/pp....aannscccoooooo
org//oorpusTD:
10

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
Sismanis, Y. How won the" chess ratings-elo the rest of the world" competition. arXiv preprint arXiv: 2010.
Zhao, Q. Wang, J., Zhang, Y., Jin, Y., Zhu, K., Chen, H., and Xie, X. Competeai: Understanding the competition behaviors in large language model-based agents. In ICML, 2024.
Szörényi, B., Busa-Fekkte, R., Paul, A., and Hüllermeier, E.
Online rank elicitation for plackett-luce A dueling ban- dits approach. Advances in neural information processing systems, 28, 2015.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.., Li, Z., Li, D., Xing, E. P. Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt- bench and chatbot arena, 2023.
Thacker, W. C. The role of the hessian matrix in fitting mod- els to measurements. Journal of Geophysical Research: Oceans, 94(C3):6177-6196 1989.
Zhu, L., Wang, X., and Wang, X. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint 2023.
Wang, F., Liu, Q.. Chen, E., Huang, Z., Yin, Y., Wang, S., and Su, Neuraled: general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering, 2022.
Zhu, Z.,, Arthur, D., and Chang, H.-H. new person-fit method based on machine learning edm in education. British Journal of Mathematical and Statistical Psychol- ogy, 75(3):616 637, 2022.
Wang, Y., Yu, Z., Zeng, Z., Yang, L., Wang. C.,, Chen, H., Jiang, C., Xie, R., Wang, J., Xie, X., al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. arXiv preprint arXiv:2306.05087, 2023.
Zhuang. Y., Liu, Q.. Huang, Z., Li, Z., Shen, S., and Ma, H. Fully adaptive framework: Neural computerized adaptive testing for online education. Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 4734- 4742, 2022.
Welinder, P. and Perona, P. Online crowdsourcing: rating annotators and obtaining cost-effective labels. In 2010 IEEE Computer Society Conference on Computer Vision and Puttern Recognition-Worrksopps. 2010.
Workshops, pp. 25-32. IEEE,
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: arXiv: 1905.07830, 2019.
Zermelo, E. Die berechnung der turnier-rrgebnisse als ein maximumproblem der wabrscheinlichkeisechhnu Mathematische Zeitschrift, 29(1):436 460, 1929.
Zhan, J., Wang, L., Gao, W., Li, H., Wang, C., Huang, Y., Li, Y., Yang. Z., Kang, G.. Luo, C.. et al. Evaluatology: The science and engineering evaluation. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 4(1):100162, 2024.
Zhang, Y., Zhang, M., Yuan, H., Liu, S., Shi, Y., Gui, T., Zhang, Q., and Huang, X. Llmeval: preliminary study how evaluate large language models. In Proceed- ings of the AAAI Conference Artificial Intelligence, volume 38, pp. 19615-19622, 2024a.
Zhang, Z., WU, L., Liu, Q., Liu, J.-Y., Huang, Z., Yin, Y., Yan, Z.,, Gao, W., and Chen, E. Understand- ing and improving fairness in cognitive diagnosis. Sci. China Inf. Sci., 67, 2024b. URL https: //api. semanticscholl oomaatticccolla..ooooc/ooooosp org/CorpusII: 269473652.

---

am-ELO: A Stable Framework for Arena-based LLM Evaluation
A. Proofs of Theorem 4.1
Proof. Assume Ri = 0 and consider the remaining variables (B2,... Rx). For each sample (i, Wiy), consider the log-like likeliihoodd function in for this sample is given by:
In Way la
P(Rj. R.).
The second-order partial derivatives of In are:
0 In/
- R2)(1 - P(R; R))),i+
0R
0 Inl OR,OR,
C2 R i.)
Now, let the number of matches between model and model j be 6, and define aij = O,, R,)(1 - ((R,,R)) For the Hessian matrix (Thacker, 1989) of the log likelihood function 0² In its quadratic form (O Meara, 2013) can be expressed as:
ØROR"
8 InnL L
Σ
aig x,z)
Σ
an
2
Σ
anzar..
OROR
1=
==2
Note that azj 2 0, therefore:
02 In L x OROR
5..
The equality holds if and only if I, 0, ie. x 0. Since the quadratic form is strictly negative for all non-zero 8 In L. is negative definite (Johnson, 1970). This implies that the log-likelihood function In concave. Therefore, In can have at most one extreme point (Boyd & Vandenberghe, 2004), ensuring the uniqueness of the maximum likelihood solution.
vectors x, the Hessian matrix BRER"
B. Proofs of Theorem 4.2
Proof. (1) For annotators and 2, the following formula can be obtained from Equation 6:
i In
E
(R, - RW, P(R,, Ry (1))
00,
(LS.W,)ES'
0 0mL
Σ
(R4 R,) P(R,, 02))
802
Since
0lm 00;
0ln 0, the difference between the two equations can be obtained:
06g
Σ
(R, (W - W())
Σ
(R4 (R), - P(R,, R, (2)))
(L.). JES'
(iJ.W,))S
According to the Lagrange mean value theorem (Shi-gu, 2014), the following derivation can be derived:
Σ
(R, - R.,
-02)
(KJ.W,))
Due to P.y(Ee)) > 0 and 01 < 0z:
Σ
(R- - R2)(W W'y) <0
(KS.W.,)ES'
(2) Because of 0 < P(R;, R2 (x) < and 0x < 0, for each positive sample (i.j, k, 1) of annotator k, we have 0lni
aR, = 01(1 P(R,, Rx) < 0. Similarly, for each negative sample (i.j, ki 0) of annotator k. we have
Bin! ole 00 P(R, Rz)0s)) 0.
12


Reviews:
[{"review_id": "mbGMYhsHat", "reviewer": "Reviewer_QUoD", "summary": "The paper introduces a novel stable arena framework, am-ELO, for evaluating LLMs using an enhanced ELO rating system. The authors address the instability issues in the traditional ELO method by replacing the iterative update approach with a MLE method, termed m-ELO. They further propose am-ELO, which incorporates annotator abilities into the ELO rating system, allowing for simultaneous estimation of model scores and annotator reliability. The paper provides theoretical proofs of the consistency and stability of the MLE approach and demonstrates through experiments that am-ELO offers a more robust, accurate, and stable evaluation method for LLMs compared to the traditional ELO system.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Regarding the questions you raised, we have carefully considered each point and have made following responses:\n\n> **Q1**: One potential weakness is the simplicity of the annotator ability modeling, which primarily focuses on discriminatory ability and consistency.\n\nThank you for your suggestion. Indeed, our work has primarily focused on proposing a stable framework that can simultaneously model both annotators and models rather than comprehensively modeling the annotators. In our subsequent work, we will conduct research on how to comprehensively model the annotators while evaluating the model's capabilities.\n\n> **Q2**: There are a few minor typos and formatting issues that could be addressed in the final version. For example, the descriptions of some formulas are not particularly clear in Section 4.2\n\nThank you for your kind reminder. We will correct these issues and refine the formulas in subsequent versions.\n\n> **Q3**: How does the proposed am-ELO framework perform in scenarios, especially the number of annotators is very large\n\nThis is a highly worthy question for discussion. In Comment 2 of Review vzb8, the relationship between time complexity and precision was discussed, indicating that this complexity can be neglected when compared to the inference of large-scale models.\n\nAs the number of annotators gradually increases, since each annotator is associated with only a single parameter, the impact on the total number of parameters of the entire model is relatively small. Moreover, with the help of a GPU, the results can be easily computed.\n\n> **Q4**: Could the authors discuss potential limitations of the proposed method when applied to highly imbalanced datasets, where some models are significantly stronger or weaker than others?\n\nWe believe that the methods we proposed are hardly affected by the dataset, especially the m-ELO method. Theorem 4.1 proves that the MLE of this method has at most one extreme point. When there is a significantly stronger model, there is usually no extreme point, or rather, the extreme point occurs at infinity. Although the MLE does not converge in this case, we can still obtain a stable ranking.", "claims_and_evidence": "The claims made in the paper are well-supported by clear and convincing evidence. The authors provide theoretical proofs for the stability and consistency of the MLE approach (Theorem 4.1) and demonstrate the practical significance of annotator ability modeling (Theorem 4.2). The experimental results, including the comparison of log-likelihood losses and the stability of ELO scores under different perturbation strategies, further validate the claims. The paper also includes a case study that highlights the differences in model rankings between the proposed methods and the traditional ELO method, reinforcing the superiority of am-ELO.", "methods_and_evaluation_criteria": "am-ELO, are well-suited for the problem of LLM evaluation in arena-based settings. The use of MLE to replace the iterative ELO update method addresses the instability issue caused by the order of data presentation. The incorporation of annotator abilities into the ELO system is a significant improvement, as it accounts for the variability in human judgment, which is often overlooked in traditional ELO systems.", "theoretical_claims": "Yes, the authors provided detailed proofs for Theorems 4.1 and 4.2, and upon inspection, no issues were found.", "experimental_designs_or_analyses": "Yes, the authors have demonstrated through prediction tasks that am-ELO has good fitting and generalization abilities. Multiple tests have shown that am-ELO can converge and obtain unique results. Simulation experiments have shown that am-ELO can effectively identify disturbances. The design and conclusions of these experiments are very reasonable.", "relation_to_broader_scientific_literature": "The paper is well-situated within the LLM evaluation. In my opinion, this article is an extension of Chatbot Arena. The incorporation of annotator abilities draws from psychometrics and Item Response Theory (IRT), which are well-established in educational assessment.", "essential_references_not_discussed": "Perhaps the current work on annotator modeling in crowdsourcing can provide some ideas for the paper.", "other_strengths_and_weaknesses": "The paper's strengths lie in its originality and significance. The proposed am-ELO framework addresses a critical issue in LLM evaluation by incorporating annotator abilities and providing a stable ranking system. The theoretical proofs and experimental results are convincing and demonstrate the practical utility of the proposed methods. One potential weakness is the simplicity of the annotator ability modeling, which primarily focuses on discriminatory ability and consistency. Future work could explore more nuanced dimensions of annotator capabilities to further enhance the evaluation framework.", "other_comments_or_suggestions": "The paper is well-written and clearly presents its contributions. However, there are a few minor typos and formatting issues that could be addressed in the final version. For example, the descriptions of some formulas are not particularly clear in Section 4.2", "questions_for_authors": "1. Changing the iterative algorithm to the MLE method undoubtedly increases the computation time. So I am curious that how does the proposed am-ELO framework perform in scenarios where the number of annotators is very large, and how scalable is the method in such cases?\n2. Could the authors discuss potential limitations of the proposed method when applied to highly imbalanced datasets, where some models are significantly stronger or weaker than others?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aNjHVir2Wx", "reviewer": "Reviewer_Nkad", "summary": "This paper introduces am-ELO, an evaluation framework designed to enhance the ELO rating system for evaluating LLMs through arena-based comparisons. Traditional ELO systems exhibit instability mainly due to their sensitivity to data ordering and their failure to account for variations in annotator expertise, resulting in inconsistent and potentially biased evaluation outcomes. To resolve these issues, m-ELO replaces the traditional iterative ELO method with an MLE-based approach, providing theoretical guarantees for consistency and stability in model rankings. In addition, am-ELO extends m-ELO by explicitly modeling annotator abilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our manuscript. We sincerely appreciate your time and effort in evaluating our work, and we will explain your questions and suggestions one by one below:\n\n> **Q1**: The evaluation criteria still lack a detailed sensitivity analysis.\n\nWe've conducted sensitivity analysis on **scale factor K** and **minimum number of annotations**. Results will be added to future paper versions.\n\n| ELO         | K=0.5  | 1      | 4(Standard) | 10     |\n| ----------- | ------ | ------ | ----------- | ------ |\n| Consistency | 0.9916 | 0.9811 | 0.9637      | 0.9305 |\n| MSE         | 0.1473 | 0.1368 | 0.1238      | 0.1225 |\n| AUC         | 0.7426 | 0.7443 | 0.7492      | 0.7505 |\n\nAs the K increases, the model demonstrates enhanced data fit. However, this improvement in fit comes at the cost of reduced consistency.\n\n| Annotation | 10         | 20         | 30         | 40         | 50         |\n| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n| ELO        | 0.9695     | 0.9637     | 0.9768     | 0.9726     | 0.9637     |\n| m-ELO      | **1.0000** | **1.0000** | **0.9979** | **0.9968** | **1.0000** |\n| am-ELO     | 0.8642     | 0.9305     | 0.9568     | 0.9979     | **1.0000** |\n\nThis shows traditional ELO gives inconsistent results across parameter changes, while m-ELO converges towards consistency, highlighting its greater stability.\n\n> **Q2**: The analysis fails to adequately address the impact of noisy or sparse annotation datasets on MLE stability.\n\nYou’re right—our theory doesn't directly counter the impact of noisy or sparse annotation datasets on MLE stability. But our analysis shows m-ELO has at most one maximum regardless of the dataset, maintaining stability as seen in Table above. In contrast, am-ELO, which models annotators explicitly, is sensitive to sparse data.\n\nIn section 4.3, we proposed selecting annotators with enough annotations. As shown in Table above, this strategy effectively lessens the negative impact of sparse datasets on MLE stability.\n\n> **Q3**: Experiments on one dataset provide limited evidence of method robustness.\n\nCurrently, open-source arena datasets are scarce. The Chatbot Arena platform is one of the few that offer public data. One NeurIPS 2024 paper also used just one real-world dataset[1]. Besides the dataset in our study, there’s the MTBench dataset[2]. Here’s its statistical information:\n\n| Dataset        | Chatbot/MTBench |\n| -------------- | --------------- |\n| #Annotators    | 42/7            |\n| #Models        | 20/6            |\n| #Response logs | 4321/1044       |\n\nHowever, after filtering, MTBench had a severely **limited number of annotators and models**. This scarcity made it inadequate for fully validating the stability of our research method.\n\nDespite this, MTBench is still valuable for demonstrating the superiority of our am-ELO modeling:\n\n| Method | MSE(Chatbot/MTBench)  | AUC(Chatbot/MTBench)  |\n| ------ | --------------------- | --------------------- |\n| ELO    | 0.1238/0.1120         | 0.7492/0.7738         |\n| m-ELO  | 0.1234/0.1097         | 0.7503/0.7785         |\n| am-ELO | **0.1208**/**0.1088** | **0.7581**/**0.7936** |\n\nWe’ll incorporate this finding in later paper versions.\n\n> **Q4**: There’s no clear justification that artificial perturbation methods accurately represent realistic annotator behavior.\n\nWe designed artificial perturbations to stress-test the ELO. By setting extreme perturbations, we explored the system's robustness.\n\nIn LLM evaluation, real annotator behavior is uncertain. Extreme perturbations can better expose ELO's weaknesses and help evaluate its stability.\n\nGoing forward, we'll explore artifact perturbation. First, we'll collect a larger real dataset to develop realistic perturbation methods. Second, we'll conduct online testing of am-ELO to validate its real-world effectiveness.\n\n> **Q5**: No baseline comparison beyond the traditional ELO.\n\nCurrently, ELO algorithms are widely used in arena platforms like Chatbot Arena. Modified ELO algorithms used in traditional competitive scenarios, such as ELO++[3], incorporate temporal information. But this conflicts with the static nature of LLM evaluations in our paper. Shuffling the dataset for these methods yields unreliable temporal information, making them unsuitable.\n\nCrowdsourcing and arena scenarios differ fundamentally. Arena scenarios lack ground-truth annotation values, while crowdsourcing assume their existence[4]. So, applying annotator-modeling techniques directly to the arena is inappropriate.\n\nIn future work, we'll focus on evaluating our am-ELO method in other arena scenarios to enhance evaluation comprehensiveness.\n\nReference:\n\n[1] Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. 2023.\n\n[2] Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. 2023.\n\n[3] How I won the \"Chess Ratings - Elo vs the Rest of the World\" Competition. 2010.\n\n[4] Learning from Crowds with Annotation Reliability. 2023.", "claims_and_evidence": "The paper points out the instability of the existing iterative ELO method in terms of ordering (and unreliable annotations). The proposed am-ELO method effectively reduces the instability by removing the data ordering issue and leveraging the reliability of each annotator. Both theoretical proofs and empirical evidence across the paper convincingly support these claims.", "methods_and_evaluation_criteria": "The proposed methods (m-ELO and am-ELO) and their evaluation criteria appropriately address the identified problems. The evaluation utilizes Chatbot Arena, extensive perturbation experiments, and robust statistical measures (consistency, MSE, AUC, F1 scores) that effectively assess the methods’ stability and reliability.\n\nHowever, the evaluation criteria still lack a detailed sensitivity analysis regarding some parameters, such as annotator counts or the scale factor K. Also, it only compares slightly outdated models.", "theoretical_claims": "It seems the theoretical proof sufficiently supports the authors' claim, but I'm not sure. One limitation is that the analysis does not adequately address the potential impact of noisy or sparse annotation datasets on MLE stability.", "experimental_designs_or_analyses": "While the experiments seem reasonable, the below limitations could be addressed more.\n\n- Experiments on one dataset (Chatbot Arena) provide limited evidence of method robustness.\n- Stability experiments rely on artificial perturbations, lacking clear justification that these perturbation methods accurately represent realistic annotator behavior.\n- No baseline or state-of-the-art comparison beyond the traditional ELO was considered, missing an opportunity to compare am-ELO with other advanced ranking or annotator-modeling methods (e.g., advanced crowdsourcing or Bayesian methods).", "relation_to_broader_scientific_literature": "The paper clearly situates itself within the broader literature on model evaluation, annotator reliability, and statistical ranking methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper is well-written and easy to follow.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RVIa61U3Hg", "reviewer": "Reviewer_3Foe", "summary": "The paper focuses on Arena-based LLM evaluation. The main algorithmic ideas include enhancing the ELO Rating System. It replaces the iterative update method with a MLE approach (m-ELO), which is more stable as it is insensitive to sample order. The am-ELO is also proposed, which modifies the ELO probability function to incorporate annotator abilities. The main findings are that the proposed methods can effectively model annotators, identify anomalous annotators, and reduce the inconsistency of ELO scores. Experimental results show that am-ELO outperforms the traditional ELO method in prediction tasks, with a lower loss and higher generalization ability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! Regarding the questions you raised, we have carefully considered each point and have made the following responses:\n\n> **Q1**: The annotator modeling in the paper is somewhat simplistic. It mainly focuses on the annotator's discriminatory ability and consistency with other annotators and may not fully capture the broader capabilities of annotators.\n>\n> **Q2**: How do you plan to extend the annotator modeling to better capture the diverse capabilities of annotators? \n\nThank you for your suggestion. Indeed, our work has primarily focused on proposing a stable framework that can simultaneously model both annotators and models, rather than comprehensively modeling the annotators. In our subsequent work, we will conduct research on how to comprehensively model the annotators while evaluating the model capabilities.\n\n> **Q3**: Can the am-ELO method still be applicable in scenarios where the annotator is not a human but Judge LLM?\n\nThis is indeed an issue we plan to research in the future. We believe that this method is applicable to both human annotators and Judge LLMs. In fact, by using a combination of human and Judge LLM annotations, we can potentially reduce annotation costs and evaluate the capabilities of Judge LLMs. However, we haven't yet figured out how to validate the effectiveness of this method.\n\n> **Q4**: Is there any other baseline method for identifying abnormal annotators?\n\nIn Arena systems, historical annotation records are typically used to identify abnormal annotators through hypothesis testing [1]. However, these hypotheses often rely on a rather strong assumption, that is, all annotators in the historical records are normal. Unfortunately, it is extremely difficult to verify this assumption. The premise of am-ELO is only that most annotators are normal annotators, and the premise for use is simpler, which is also the advantage of am ELO compared to hypothesis testing methods.\n\nIn future work, we plan to focus on identifying and comparing our am-ELO method on other scenes specifically tailored to the large-scale arena evaluation context [2]. This will enhance the comprehensiveness of our evaluation and more accurately position the contributions of our research.\n\n[1] Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. 2024.\n\n[2] Decentralized Arena via Collective LLM Intelligence: Building Automated, Robust, and Transparent LLM Evaluation for Numerous Dimensions. 2024.", "claims_and_evidence": "The claims are generally supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem. The m-ELO and am-ELO methods address the instability issues in the traditional ELO method, which is crucial for accurate LLM evaluation. The use of real-world datasets like Chatbot for evaluation is appropriate, as it reflects the practical scenario of LLM comparison. The evaluation metrics such as MSE, AUC, loss, and consistency of rankings are well-chosen to measure different aspects of the methods' performance, including prediction accuracy, goodness-of-fit, and stability.", "theoretical_claims": "The correctness of the proofs for theoretical claims has been checked. For Theorem 4.1, the authors prove that when fixing the score of one model, the log-likelihood function with respect to is a concave function and has at most one extreme point. This is done by calculating the second-order partial derivatives of the log-likelihood function and showing that the Hessian matrix is negative definite. For Theorem 4.2, the authors prove the properties of annotator abilities. The proofs are logical and based on sound mathematical reasoning.", "experimental_designs_or_analyses": "The soundness/validity of the experimental designs and analyses has been checked. In the experiments, the authors compare the proposed methods with the traditional ELO method. They use appropriate baselines and perform multiple random initializations and repeated experiments (shuffling the dataset 1000 times for the traditional ELO method). The way they record the loss during the gradient descent process and calculate the consistency of rankings is reasonable for analyzing the convergence and efficiency of the methods. The perturbation strategies in the stability experiments are well-designed to simulate real-world annotation anomalies.", "relation_to_broader_scientific_literature": "The paper improves upon the widely-used ELO rating system, which is the foundation for many existing model arena evaluation systems. The proposed methods address the instability issues and lack of annotator ability consideration in previous works, and the use of MLE and psychometric concepts for annotator modeling is an extension of relevant research in statistics and psychometrics.", "essential_references_not_discussed": "There are no essential references that are not currently cited/discussed in the paper.", "other_strengths_and_weaknesses": "**Strengths:**\n\nOriginality: The combination of MLE and annotator ability modeling in the ELO-based evaluation framework is novel. It provides new solutions to the long-standing problems of instability and annotator variability in LLM evaluation.\n\nSignificance: The proposed methods can improve the reliability and accuracy of LLM evaluation, which is of great significance for the development and deployment of LLMs. It helps to make more informed decisions in model selection and research directions.\n\nClarity: The paper is well-written. The algorithms, theoretical proofs, and experimental results are clearly presented, making it easy for readers to understand the research content.\n\n**Weaknesses:**\n\nThe annotator modeling in the paper is somewhat simplistic. It mainly focuses on the annotator's discriminatory ability and consistency with other annotators, and may not fully capture the broader capabilities of annotators.", "other_comments_or_suggestions": "See the question below.", "questions_for_authors": "1. In the am-ELO method, how do you plan to extend the annotator modeling to better capture the diverse capabilities of annotators? A more comprehensive answer could further strengthen the potential of this research. If the authors have clear plans or ideas, it would enhance the value of this work for future research.\n\n2. Can the am-ELO method still be applicable in scenarios where the annotator is not a human but Judge LLM.\n\n3. This article identifies abnormal annotators by screening those with negative ability. Is there any other baseline method to do this? What are the advantages of am ELO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYhRBzDg8Q", "reviewer": "Reviewer_vzb8", "summary": "The paper addresses challenges in ranking consistency and the variety of annotator abilities in arena-based evaluation of LLMs. They develop an enhanced ELO framework that replaces iterative updates with Maximum Likelihood Estimation (m-ELO). They prove theoretically that this MLE approach provides consistent and stable model rankings. The authors further extend their work with am-ELO, which factors in annotator abilities to simultaneously evaluate both model performance and annotator reliability. Experimental results validate their framework's effectiveness in stability and robustness compared to traditional ELO-based evaluation methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your high appreciation of the contribution and novelty presented in our paper. Your positive feedback means a lot to us. We also appreciate your valuable suggestions and questions for the computational complexity and experiments aspect of the paper. Here are the responses to each of your comments:\n\n> **Q1**: This paper might benefit from a discussion on other related work.\n\nWe sincerely appreciate your suggestion. We have referred to these work, which have been very enlightening for us. We will consider incorporating these related works into our article in future versions.\n\n> **Q2**: While the paper demonstrates the robustness of am-ELO through perturbation experiments, it would be useful to see how the method performs in real-world scenarios with varying levels of annotator quality.\n\nThank you very much for your question. We've discussed on the motivations behind using perturbations in experiments and plan to collect real-world datasets and test am-ELO online in future work, **which can be seen in Comment 5 for reviewer Nkad**.\n\n> **Q3**: How does he computational complexity of the proposed am-ELO method compare to the traditional ELO method?\n\nThank you for your question. The time complexity of am-ELO is equivalent to that of the MLE gradient descent, which is $O(1/\\epsilon^2)$ where $\\epsilon$ is the calculation accuracy of the GD method. However, since the MLE of m-ELO is a concave function, its time complexity is sub - linear, specifically $O(1/\\epsilon)$ [1].\n\nIn the experiments, there was no significant difference in the efficiency between am-ELO and m-ELO when running on a GPU. Moreover, as tools for  LLM evaluation, their inference costs are far lower than those of large - model inference. Hence, these costs are entirely acceptable.\n\n> **Q4**: Can the authors provide insights into how the proposed framework could be extended to handle more complex evaluation scenarios, such as multi-class or multi-label evaluations?\n\nThank you for your question. In my opinion, this paradigm can be applied to more complex annotation scenarios, such as multi-label evaluation, as long as **the probability density function can be reasonably defined**.\n\n> **Q5**: The lack of a detailed sensitivity analysis regarding some parameters.\n\nWe sincerely appreciate your suggestion. We assert that due to the convergence required by m-ELO and am ELO, **the learning rate and epoch are almost unaffected**. In addition, we’ve carried out sensitivity analysis experiments on two parameters. The results of these experiments will be integrated into subsequent versions of our paper.\n\nRegarding scale factor K (Learning Rate) specifically, both m-ELO and am-ELO necessitate extensive training and exhibit insensitivity to this scale factor. Therefore, we focused our analysis on the consistency of the traditional ELO method across different values of K. The findings are presented in Table 1:\n\nTable 1\n\n| ELO         | K=0.5  | K=1    | K=4 (Standard) | K=10   |\n| ----------- | ------ | ------ | -------------- | ------ |\n| Consistency | 0.9916 | 0.9811 | 0.9637         | 0.9305 |\n| MSE         | 0.1473 | 0.1368 | 0.1238         | 0.1225 |\n| AUC         | 0.7426 | 0.7443 | 0.7492         | 0.7505 |\n\nOur analysis reveals that as the scale factor K increases, the model demonstrates enhanced data fit. However, this improvement in fit comes at the cost of reduced consistency.\nFurthermore, we carried out a hyperparameter sensitivity experiment on the **minimum number of annotations per annotator**. The results of this experiment are detailed as follows:\n\nTable 2\n\n| Annotation | 10     | 20     | 30     | 40     | 50         |\n| ---------- | ------ | ------ | ------ | ------ | ---------- |\n| ELO        | 0.9695 | 0.9637 | 0.9768 | 0.9726 | 0.9637     |\n| m-ELO      | 1.0000 | 1.0000 | 0.9979 | 0.9968 | 1.0000     |\n| am-ELO     | 0.8642 | 0.9305 | 0.9568 | 0.9979 | **1.0000** |\n\nAs our research indicates, both the traditional ELO and m-ELO show resilience to the variation of annotation count parameters. However, their performance patterns diverge significantly. The traditional ELO consistently exhibits inconsistent results regardless of parameter fluctuations. In contrast, m-ELO demonstrates a tendency to converge towards consistency, underscoring its enhanced stability.\nRegarding am-ELO, it’s true that it’s highly sensitive to the annotation count. When faced with sparse annotations, the consistency of am-ELO is indeed compromised. But we’ve developed a practical solution. By implementing a screening process for annotators, we can effectively address this issue. This screening process ensures that only reliable annotators with sufficient annotations are included in the analysis, thus improving the consistency of am - ELO. \n\nReference:\n\n[1] Introductory Lectures on Convex Optimization: A Basic Course. 2014.", "claims_and_evidence": "The claims made in the submission are well-supported by clear and convincing evidence. The authors identify instability as a significant issue in the traditional ELO rating system used for evaluating LLMs and propose a new framework, am-ELO, to address this problem. The claims are backed by both theoretical analysis and empirical experiments. The authors provide mathematical proofs for the stability and consistency of their proposed methods and demonstrate through experiments the effectiveness of their methods.", "methods_and_evaluation_criteria": "The authors replace the iterative update method of the traditional ELO system with a MLE approach, which is theoretically sound and shown to be more stable. Additionally, the incorporation of annotator abilities into the evaluation process through am-ELO is a novel and meaningful enhancement. The evaluation criteria used are appropriate for assessing the stability and robustness of the proposed methods.", "theoretical_claims": "I have checked the correctness of the proofs for the theoretical claims. Specifically, the proof for Theorem 4.1 is sound. This ensures the stability of the ELO scores obtained through the MLE method. Also, the proof for Theorem 4.2 is logically consistent and demonstrates the practical significance of the estimated annotator abilities.", "experimental_designs_or_analyses": "The experimental designs and analyses are valid and well-executed. The authors conducted experiments on a real-world dataset and compared the performance of the traditional ELO method with their proposed methods. The results show significant improvements in terms of lower loss values and higher consistency in model rankings. Additionally, the stability tests using perturbation strategies (Random, Equal, Flip, Mixed) effectively demonstrate the robustness of the am-ELO method.", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely related to the broader scientific literature on LLM evaluation and ranking systems. The authors build upon the well-established ELO rating system and enhance it using principles from psychometrics and maximum likelihood estimation. The paper cites relevant prior work, such as the use of the ELO system in competitive games and its application to LLM evaluation. The proposed methods address the instability issues in the traditional ELO system and incorporating annotator abilities, which is a novel contribution in this domain.", "essential_references_not_discussed": "The paper appears to cover most relevant prior work. However, it might benefit from a discussion on other ranking systems used in machine learning, such as the Plackett-Luce model [1], to further contextualize the contributions. Additionally, recent work on robust evaluation of LLMs, such as [2, 3], could be cited to provide a more complete picture of the current landscape.\n\n[1] Robin L Plackett. The analysis of permutations. Applied Statistics, 1975.\n\n[2] Yiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang, Srijan Kumar. MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms. In Proceedings of ACL 2024\n\n[3] Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, Yang You. MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures. In Proceedings of NeurIPS 2024", "other_strengths_and_weaknesses": "Strengths: \n\n1. The paper is well-written and clearly presents the problem, proposed methods, and experimental results. \n\n2. The incorporation of annotator abilities into the evaluation process is a significant innovation that addresses a critical limitation of existing methods. \n\n3. The experiments are thorough and demonstrate the effectiveness of the proposed framework.\n\nWeaknesses: \n\n1. The paper could benefit from a more detailed discussion on the computational complexity of the proposed methods, especially in large-scale scenarios. \n\n2. While the paper demonstrates the robustness of am-ELO through perturbation experiments, it would be useful to see how the method performs in real-world scenarios with varying levels of annotator quality.", "other_comments_or_suggestions": "The paper could include a section on the potential applications of the proposed framework beyond LLM evaluation, such as in other competitive ranking scenarios.\n\nThe authors might consider discussing the limitations of their approach, such as the assumptions made about annotator behavior and the potential impact of these assumptions on the evaluation results.", "questions_for_authors": "1. How does the computational complexity of the proposed am-ELO method compare to the traditional ELO method, especially in scenarios with a large number of models and annotators?\n\n2. Can the authors provide insights into how the proposed framework could be extended to handle more complex evaluation scenarios, such as multi-class or multi-label evaluations?\n\n3. How sensitive are the results to the choice of the learning rate and the number of epochs in the gradient descent process? Are there any guidelines for selecting these hyperparameters?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses challenges in ranking consistency and the variety of annotator abilities in arena-based evaluation of LLMs. They develop an enhanced ELO framework that replaces iterative updates with Maximum Likelihood Estimation (m-ELO). They prove theoretically that this MLE approach provides consistent and stable model rankings. The authors further extend their work with am-ELO, which factors in annotator abilities to simultaneously evaluate both model performance and annotator reliability. Experimental results validate their framework's effectiveness in stability and robustness compared to traditional ELO-based evaluation methods.

The claims made in the paper are well-supported by clear and convincing evidence. The combination of MLE and annotator ability modeling in the ELO-based evaluation framework is novel. It provides new solutions to the long-standing problems of instability and annotator variability in LLM evaluation.
The paper also includes a case study that highlights the differences in model rankings between the proposed methods and the traditional ELO method, reinforcing the superiority of am-ELO. The paper is well-written. The algorithms, theoretical proofs, and experimental results are clearly presented, making it easy for readers to understand the research content.
Decision: Accept (spotlight poster)