Elo Uncovered: Robustness and Best Practices in Language Model Evaluation
Authors: Meriem Boubdir
Edward Kim
Beyza Ermis
Sara Hooker
Marzieh Fadaee
Venue: NeurIPS 2024 poster (2024)
Keywords: Elo Rating System
Language Model Evaluation
Reliability
Robustness
Reproducibility
LLMs Ranking
Primary area: evaluation
TL;DR: This paper probes the Elo rating system in LLM evaluations, revealing its inherent volatility and providing empirical guidelines for ensuring robust and accurate model ranking in real-world scenarios.

Abstract: In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such as chess, is increasingly being used to evaluate Large Language Models (LLMs) through "A vs B" paired comparisons.
However, while popular, the system's suitability for assessing entities with constant skill levels, such as LLMs, remains relatively unexplored. 
We study two fundamental axioms that evaluation methods should adhere to: reliability and transitivity. 
We conduct an extensive evaluation of Elo behavior across simulated and real-world scenarios, demonstrating that individual Elo computations can exhibit significant volatility.
We show that both axioms are not always satisfied, raising questions about the reliability of current comparative evaluations of LLMs.
If the current use of Elo scores is intended to substitute the costly head-to-head comparison of LLMs, it is crucial to ensure the ranking is as robust as possible.
Guided by the axioms, our findings offer concrete guidelines for enhancing the reliability of LLM evaluation methods, suggesting a need for reassessment of existing comparative approaches.

Elo Uncovered: Robustness and Best Practices in Language Model Evaluation
Meriem Boubdir
Edward Kim Cohere edward@cohere.com com
Beyza Ermis Cohere For Al
Cohere For AI
meri re com
beyza@cohere.com
Sara Hooker
Marzieh Fadaee Cohere For Al
Cohere For AI
sarahocker@cohere.com
marzieh@coheer..o. com
Abstract
In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such chess, is increasingly being used to evaluate Large Language Models (LLMs) through "A B" paired comparisons. However, while popular, the system's 's suitability for assessing entities with constant skill levels, such as L.LMs, remains relatively unexplored. We study two fundamen- tal axioms that evaluation methods should adhere to: reliability and transitivity. We conduct an extensive evaluation of Elo behavior across simulated and real world scenarios, demonstrating that individual Elo computations volatility. We show that both axioms are not always satisfied, raising questions about the reliability of current comparative evaluations of LLMs. If the current use of Elo scores is intended to substitute the costly head-to-head comparison LLMs. it is crucial to ensure the ranking is robust as possible. Guided by the axioms, our findings offer concrete guidelines for enhancing the reliability of LLM evaluation methods, suggesting need for reassessment of existing comparative approaches.
exhibit significant
Introduction
In the rapidly evolving field of Natural Language Processing (NLP), the task of accurately and reliably evaluating LLMs has become increasingly challenging [32 10 50
26 43). Human feedback has emerged as an indispensable tool in this performance assessment process, serving as a qualitative metric that captures nuances that automated scoring mechanisms often fail to address 2 3 4 50) evaluations, highly valuable to the overall progress the NLP field, typically adopt "A B" comparative setup, turning evaluations into zero-sum game between language models. Pairwise comparisons, however, are fundamentally difficult to scale for large pools of models, due to the quadratic growth of comparisons required. Fortunately, this paired feedback structure (56) naturally lends itself to the Elo rating system, originally designed for ranking chess players (including those who have never before played each other) for better matchmaking (16).
13 12
These
Under the Elo rating system, players' skills are indicated by an Elo rating, where higher ratings indicate higher skill, and all players can be ranked best to worst using this scalar Elo rating. In the standard formulation (see Section 2). player rated at 1800 has 10: 1 odds of winning against a player rated at 1400. After match, the winner takes rating points from the loser in zero-sum fashion [6]. Thus, with the Elo rating system, we can efficiently integrate subjective human feedback on paired "A vs 8" language model completions into structured and unified rating system to assess the performance of language models.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ProbLA B)
ProdcA
Vinn ratos
Naaves =3
Morro -1
Mperes
1-775
14:00
M
1:05
DooK
1:40
30
1375
1900
1325
Neam = 100
Mgoro 100
Moerra 100
Elo catin
1400
100
1900
Number of played games
Figure 1: Impact of win probabilities and permutation sampling on Elo ratings: Com- paring Model and Model B across three different win probabilities (Prob(A beats B) (0.6, 0.55, 0.51)) with two levels of permutation sampling (N = 1 and Npemns 100). The top row displays the observed win rates, the middle one the Elo ratings with single permutation, and the bottom one the mean and standard error of the mean (SEM) of Elo ratings across 100 permutations.
The principles of Elo rating have proven to be resilient and adaptable due its dynamic adjustments, relative rating focus, consistency across skill levels, and simplicity and transparency. As result, the Elo rating system has found diverse applications, from predicting sports events outcomes
25 31 53,, and facilitating matchmaking in massively multiplayer online games like StarCraft
and Dota 115 44
35 [7). to its recent in the evaluation of LLMs 2 3 50 13
12
54 33
However, to-date there has not been a comprehensive examination of the compatibility of Elo scores and LLMs evaluation.
Unlike dynamic competitors that evolve over time, LLMS have static capabilities and operate in constrained by preset number of
time-agnostic context. In this setting, evaluations of LLMs turns, as is the case with tournament timelines or predefined match sequences. Morcover, the ordering of matches significantly influence the final Elo scores and, consequently, model rankings. This oversight is particularly concerning, given the direct impact Elo system rankings on both research
directions and real-world applications in NLP well as its widespread adoption 2 55 57 29
50 13 12
54. 33].
This study aims to close this research gap by adopting an axiomatic approach and scrutinizing both the reliability and limitations the Elo rating system when applied to LLMs. We study two fundamental axioms that evaluation methods should adhere to: reliability and transitivity. Through theoretical and empirical analyses grounded in collected human feedback data, our contributions provide comprehensive understanding of when and how to reliably employ the Elo system for LLM evaluation, thus offering valuable guidelines for researchers and practitioners the NLP field.
We find that Elo ratings for LLMs are highly sensitive to the order of comparisons and the choice of hyperparameters. Moreover, desirable properties such as transitivity are not always guaranteed and can be unreliable unless there is comprehensive human feedback data for all unique pairwise compurisons among models in the feedback pool. The sensitivity of Elo ratings becomes more pronounced when dealing with models that exhibit similar performance levels. We illustrate the best practices for addressing Elo rating sensitivities by offering guidelines for hyperparameter selection and matchmaking scenarios.

---

Implications of our work As LLMs rapidly advance, evaluation leaderboards are gaining popularity assess the performance of newly introduced models using Elo scores. Elo can also be used in the learning framework of L.LMs to produce a ranking of models and their outputs for preference training. No research has explored the nuances of using Elo scores to compare L.MMs, which, unlike chess, exhibit static capabilities and operate in time-agnostic manner. We show that Elo rating does not always satisfy two critical
and transitivity-leaddee not accurate. Our research offers guidelines for reliable and robust implementation of Elo scores when comparing LLMs. Deviation from our recommendations could result in inaccuracies when ranking LLMs, particularly in situations where model performances closely matched, and Elo score differences are minimal (a common occurrence in many real-world scenarios).
to rankings of models that
2 Elo Algorithm Explained
We provide the standard mathematical formulation of the Elo algorithm [16], contextualized to the setting of LLM evaluation. this formulation, M be a set of models, and each model i E M assigned an initial numerical Elo rating Rg. For each match between two models, we calculate the expected score, then update the ratings both models as follows:
2.1 Expected Score Computation
For a given paired zero-sum match-up between two models and B (A, M), each with respective pre-match ratings RA and Rg, the expected scores EA and Eg (i.e., match outcomes) are computed as:
1
EA 1 RA3/400 and EB + 10 - Re)/400
In this context, the factor of 400 [16] precisely adjusts the sensitivity of the expected score to differences in ratings. 400-point advantage in ratings translates to 10: odds in favor of the higher-rated model, providing an interpretable metric for performance comparison. For evenly matched models (Ra = RB), both EA and EB equate to 0.5, reflecting a 50:50 win probability for both models.
2.2 Rating Update Mechanism
Following each match, the Elo ratings are updated based on the observed win-loss outcome. The rating adjustment for each model is dictated by the equation:
RA R. + K(SA -EA)
(2)
Here, SA represents the actual score achieved by model A, which can take on either the value 0 for a loss or for win. Model B's Elo rating is updated via the same method. The K-factor serves as variable hyperparameter adapt the rate of change in rating to different scenarios. A higher K-factor results in larger changes in the Elo score after each match-up, making the scoring more sensitive individual results. A lower K-factor, in contrast, makes the Elo ratings more stable, with smaller changes after each match. chess, the K-factor is usually set to 16 for masters and to for novice players.
3 Desirable Properties of Elo
The objective of using Elo scores rank models establish comparative understanding of the performance hierarchy among them. When incorporating a new model into an already ranked list, only a limited number pairwise annotations are required to determine The ability to infer model's 's relative performance compared to all previous models in the list relies on the robustness of the scoring method and the transitive property the ranking system. We describe these desirable properties through two axioms: transitivity and reliability.
position in ranking.

---

3.1 Axiom 1: Transitivity
desirable property of any rating system is transitivity because ensures consistency and logical coherence in how entities are ranked or rated. Transitivity in this context means that if player A beats player B, and player B beats player C, then player A is expected to beat player C. If the ranking of large language models exhibits transitivity, we can deduce their comparative performance without the need for direct head-to-heal evaluations between every pair of models. The central assumption in developing various leaderboards for comparing language models is that the rankings adhere the principle of transitivity [57].
While Elo's design inherently assumes transitivity, our synthetic data which are derived from realistic scenarios, uncovers certain circumstances that violate this assumption. Such anomalies can affect the final ranking of language models and their relative performance assessments.
3.2 Axiom 2: Reliability
We consider aspects of reliability:
Sensitivity to ordering: Unlike chess or time-bound sports where match sequences are structured, in LLM evaluations all matches occur independently and in parallel, amplifying the sequence's influence on final model ranking. In this context, each match represents the performance comparison specific prompt. If the prompts
between two models model happens to perform better on the initial set of prompts, it may gain an advantage in subsequent comparisons due to the cumulative effect of its early success. This inherent variability prompts to investigate the extent to which match-up ordering affects the robustness of Elo ratings.
presented in specific order, and one
Sensitivity to hyperparameters: The sensitivity of hyperparameters can compromise the robustness of Elo scores leading to inconsistent rankings. Evaluating and understanding this sensitivity is crucial for building evaluation frameworks that maintain consistency across diverse models. In this work, we evaluate the sensitivity of Elo performance to one key hyperparameter, the K-factor. This factor acts scaling constant in the Elo rating system, pivotal for updating ratings after each matching. essentially determines how quickly model's rating converges to what can be considered its "true" skill. While conventional applications like chess use standard K K-factor values, these may not be directly applicable in the context of evaluating LLMs due to the unique characteristics and requirements of this domain.
Synthetic Human Feedback
Given the costly and time-consuming nature of human evaluations, studying the Elo system's 's hehavior under various scenarios becomes challenging. To circumvent these limitations, we first validate the properties of Elo using synthetic data generation via Bernoulli processes to simulate various human feedback scenarios. In Section 6 we extend these evaluations include real-world human feedback. This time-agnostic and independent setup of LLM evaluations resembles a Bernoulli process [6]. sequence of independent experiments, each yielding simple "win" or "loss" outcome, representing one model outperforming another. We use this setting to control the characteristics of the distribution and evaluate the different desirable properties of rating system.
In this controlled setting, our primary objectives include testing the transitivity whether a consistently higher-rated model outperforms those with lower ratings in all scenarios. Additionally, in studying the reliability axiom, we explore how the Elo scores are affected by the order in which models are compared and the sensitivity to hyperparameter adjustments, particularly the K-factor. This synthetic setup offers a robust platform to dissect and understand the dynamics of the Elo rating system in the context of LLM evaluations, without the constraints and limitations of relying solely on real-world human feedback.
4.1
The Bernoulli Analogy
Pairwise comparisons in LLM evaluation draw parallels with the foundational principles of the Bernoulli experiment in probability theory. This section studies the similarity between human feedback-basee evaluations and the Bernoulli experiment's principles.

---

Preliminaries A Bernoulli trial is random experiment with exactly two possible outcomes, "success" or "failure". The outcomes adhere the probability condition:
P(success) + P(failure) =1
Here, the random variable X' denotes the outcome, where X' implies success, and X signifies failure. The probabilities associated with these outcomes are given by:
P(x=1) p P(x = 0)
with 0 S S 1, the "success" probability.
Mapping to Human Feedback When comparing two models, and B, across pairwise evaluations, the setup aligns with a Bermoulli process. This process comprises sequence of independent and identically distributed (i.i.)) Bernoulii trials. To frame this analogy, designate win probability, P(Awin), to model A. Leveraging a Bernoulli random variable, X, as a means to simulate synthetic human feedback, we proceed as follows:
1. A sample is drawn from X' using PAwwi.).
2. If X' = 1, feedback suggests a preference for model A.
3. Otherwise, model B is favored.
Extending to Multiple Players Given a finite set of n distinct models M, their pairwise compar- isons can be formulated as:
n!!
n(n 1)
2 21(n 2)
2
This formula yields (2) unique pairs (A, B) where A, A. M and + B. For each pair, a Bernoulli process comprising multiple Bernoulli experiments is conducted to discern which model performs better over a sequence of trials.
4.2 Synthetic Data Generation
Building upon the Bernoulli process analogy, when conducting multiple independent evaluations between two models, the distribution of the number of times one model is preferred over the other naturally follows a binomial distribution. For N pairwise comparisons, the relation is:
N P (1- (MM
P(k; N) =
(6)
where P(k: is the probability of one model being preferred times of evaluations. p the success probability and (N) is the binomial coefficient, representing the number of ways to choose & successes from N trials.
5 How Robust Are Elo Scores?
This section describes rigorous stress tests designed to investigate whether the two axioms, presented in Section are satisfied in this evaluation framework. We focus critical desirable properties of ranking mechanism that should (1) be insensitive to match-up ordering, (2) not overly sensitive to hyperparameters like the K- -ffactor, and (3) preserve properties of transitivity. Subsequently, we provide empirically grounded guidelines for a safe and interpretable application of Elo ratings.
5.1 Impact of Ordering on Elo Ratings
Experimental Setup
To quantify the effect of match-up ordering, we generate a baseline sequence of Ngumes 1000 match outcomes between models and (see Equation (6) reflecting the scale typical of LLM evaluations via human feedback. We hold N Ngames constant for the entirety of our study maintain consistency. From this baseline, we derive Npemss distinet permutations, each involving a complete reshuffling of the initial sequence to simulate various chronological orders in which the games might unfold. It important to note that we are not generating new match outcomes for each permutation; instead, we simply reorder the existing data to explore the potential impact of different
5

---

(a) Elo Scores for Single Sequence
(b) Elo Scores Averaged Over 100 Permutations
400
02
300
304
360
2C0
07
167
332
151
173
175
ProbiA beatss B)
100
ProbfA beats
100
100
0,55
0.59
-200
-200
Bl
0.1
-200
-400
K-factor
K-factor
Figure 2: Final Elo scores difference (SA SB) function of K-factor and N perms Positive values reflect the expected ranking where Model is superior to Model B, while negative values indicate discrepancy, falsely suggesting that Model B has higher Elo score than Model A. We compare between a single sequence of outcomes and averages over Mperms = 100 unique permutations.
match-up sequences. For each reordered sequence, we update the Elo ratings RA and Rø according to equation 2 resetting both ratings to initial value of 1400 at the start of each permutation. Finally, we compute average Elo ratings per match across all Npems permutations, ensuring a robust analysis that takes into account the full range of possible match-up orders.
We repeat this process to generate baseline sequences and their respective reorderings for set of selected winning probabilities enabling us to inspect ratings' behavior under various real-world scenarios. Npemss is varied from a minimum of to a maximum of 10k, providing a robust sample size for statistical analysis (see Figure 3. Subsequently, we compute the average Elo ratings per match across all permutations. These averages, RA and Rg. particularly for Nppenss = and Npams = 100, visualized to offer insights into the stability of the ratings, shown in Figure [
Key Findings Our analysis underscores the inter- play between winning probability P(Awin) and the number of different orderings Npemo on the stability of Elo ratings after each update. For P(Awin) 2 0.6,
1460
Photii beats (0) 0
PhomiA bears I0 0 ProBiA
ProbiA
Eo Score
Elo ratings demonstrate high stability; additional re-
sults for P(AAin) 0.65 and beyond are available
3420
Appendix B On the other hand, for P(Awin) = 0.5,
ratings exhibit significant instability for a single se- quence. As depicted in Figure] 1 when both mod- els have win probabilities around 0.5, Elo ratings frequently intertwine, making it challenging to dis- cern clear performance difference between the two. The instability plateaus as N exceeds 100, result- ing in stabilized Elo ratings that align closely with the preset winning probabilities. For instance, P(Awm) = 0.55, the average Elo rating for Model A, R.A, consistently exceeds that for Model B, Rg. when averaged across multiple permutations, reflecting an accurate performance-basee ranking of these models. These observations validate concerns highlighted earlier, emphasizing the critical role N Npems for a reliable interpretation of Elo ratings in I.M evaluations. Elo-based evaluations, the sequence model comparisons can significantly influence the final Elo scores, particularly in scenarios with models of similar quality, where this effect is magnified.
1380
10'
10"
10'
Myano
Figure 3: Variation of Model A's average Elo score with increasing number of permutations N Nperms for different probabilities of Model winning (P(A beats B)). Error bars indicate standard errors of the mean.
5.2 Sensitivity to Hyperparameters
Experimental Setup
We extend our previous approach by conducting tests across range of winning probabilities and multiple K K-factor values (1, 8, 16, 32, 64). We compute and compare the

---

Table 1: Investigation of Elo score reliability in capturing true model hierarchies across varying configurations. Scenarios explore the transitive relationship A >> B and B > C = A > C. The star (*) indicates cases where the Elo score fails accurately reflect the expected hierarchy of models. R represents models with similar performance; >>
indicates that model significantly outperforms
the other one.
Elo-based Models Ranking Configuration
Scenario
Model
1,
N 100,
N 1,
16
N
16
A
1539.43
1528.50 0.35
1650.93
1584.78 1 3.09
A
B
B
1390.47
1410.33 = 0.54
1381.17
1406.48 3.23
B
1270.10
1261.17 + 0.33
1167.90
1208.74 2.71
A
1502.09
1495.92
1509.08
1526.04 3.03
A B
1337.48
1342.70* + 0.53
1379.00
1340.83 2.83
B
1360.42
1361.38* 0.38
1311.92
1333.13 2.68
A
1437.97
1433,84* 0.41
1440.31
1460.22 + 2.90
A
1455.10
1453.84* 0.61
1481.04
1452.87 3.25
B C
C
1306.93
1312.32 + 0.34
1278.65
1286.91 2.72
0
A
1426.33
1419.73 + 0.36
1407.44
1432.26 2.93
A
B
1390.47
1393.29 + 0.59
1386.17
1392.75 3.04
B
1383.20
1386.99 + 0.41
1406.39
1374.99 + 3.12
average Elo scores SA and S3 for Ngames 1000 and Npems {1, 100). The differences between
these final averages for Model A and Model are summarized in Figure 2 to assess the stability and expected ranking between the two models.
Key Findings As shown in Figure 2 notable instability is observed model rankings based on the final Elo scores when consider single sequence of paired comparisons (i.e., Npemss = 1), especially for winning probabilities nearing 0.5. This instability markedly exacerbated at higher K-factors. In contrast, the picture changes when coupling higher K-factors with raising the number of permutations to at least 100. Higher K-factors, in this multi-permutation scenario, speed up the differentiation between models' Elo scores, enabling faster convergence their true skill levels. This yields much more stable and reliable model rankings. It is noteworthy that this faster convergence is observed to be more reliable for higher winning probabilities, which corresponds skewed win rates in real-world scenario.
5.3 Transitive Properties of Elo Scores
Experimental Setup
The transitivity property of the Elo scores is defined as:
A B and B>C = A>C
To test the transitivity property, we design four distinct scenarios that model real-world conditions:
Model A beats model B and model beats model C' both with high win probabilities (Pwin 0.75).
E Model A beats model B with high win probability (Pwin 0.75), model B beats model C with a win probability close to 0.5 (Pwin 0.51).
A Model A beats model B with a win probability close to 0.5 (Pwin = 0.51), model B beats modelC with high win probability (P win 0.75).
2 Model A beats model B with a win probability of 0.54, model B beats model C' with a win probability of 0.51.
In each of these scenarios, we simulate matches for paired comparisons "A vs. B" and "B vs. C" and then rearrange these matches in arbitrary order form baseline sequence. This approach mimics how Elo ratings are computed for online leaderboards in the evaluation of large language We then analyze whether Elo scores maintain the expected model hierarchies.
models 54 33
Key Findings The outcomes from all four scenarios, detailed in Table demonstrate the per- formance of Elo-based rankings across various configurations. In scenarios where there is clear disparity between models (c.g., $)) Elo ratings accurately reflect the expected hierarchy. However, in
7

---

claude-1
cleude-1
32
gpt-4-0613
got-4-0314
Elc Scores Diffrrnce
16
claude-1
claude-1
10
9pt-4-0613
got-4-0314
Kffactor
Kractorr
claude-1
cpt-4-0613
qpt-4-0314
claude-1
cleude-1
gpt-4-0613
gut-4-0314
10000
10000
Npers
Mypess
(a) Experiment: Claude-1 vs. Claude-2.1
(b) Experiment: GPT-4-0314 vs. GPT-4-0613
Recorded Win rates: 0.59 0.41
Recorded Win rates: 0.51 vs 0.49
Figure 4: Elo score differences (SA Ss) across varying K-factors and Nperms Positive values in the heatmap indicate that the expected ranking is maintained (Model A outperforming Model B), while negative values suggest ranking inversion, where Model B appears to outperform Model A, contrary to the actual win rates. Each cell's label indicates the model with the higher Elo score.
more complex cases such H and A. where one model significantly outperforms second, which turn is closely matched with third, the rankings become less stable, challenging the assumption of transitivity. We observe once again that varying the number of permutations (Nyerrs = 1 vs. Npemss = 100) and the K-factor plays a critical role in stability. In the I and & scenarios, with Npanss 100 and K
notice discrepancies in the models' rankings. This contrasts with K 16, where rankings are more consistent and accurate. The slower updates from 1 suggest this setting may be too conservative to capture transitive relations quickly, leading to inconsistencies.
Validation on World Human Feedback
Building on the insights gained from synthetic data experiments, this section extends the validation of the Elo rating system to real-world human feedback. Our objectives twofold: first, to ascertain how the properties demonstrated using synthetic data generalize to real human annotations, and second, to evaluate the Elo rating system's 's utility for assessing LLMs in practical settings.
use the LMSYS Chatbot Arena dataset (34), an open-source collection of human preference data derived from unique users' interactions with two distinct models responding to set of user- defined prompts. To align with our methodology from synthetic data analysis, tie outcomes have been excluded from this analysis to focus specifically on the implications of win-loss dynamics. We select pairs of models (A B) from the initial dataset that feature at least 300 non-tie comparisons. This threshold ensures statistical robustness and allows us to include cases where win rates are closely contested, which lead to sensitive ratings. These pairs predominantly involve models from the GPT-4 fam-
Experimental Setup
Table 2: Win rates per evaluated model across selected paired comparison exper- iments.
Experiment
Win Rates
GPT-4-0314
0.51
GPT-4-0613
0.49
Claude-1
0.59
Claude-2.1
0.41
GPT-4-0314
0.65
Claude-2.1
0.35
GPT-4-0613
0.61
ily [42) and the Claude family (1] A comprehensive list of model pairs is included in Appendix C under Table and subset discussed here is shown in Table The recorded win rates primarily exhibit skewed preferences, with the exception of the GPT-4-0314 vs. GPT-4-0613 pairing, indicating comparable performance levels (see T2b Given the variable number of evaluations per pair in the original dataset, we standardize this by sampling fixed number, Nample for each pair to align with the controlled conditions used in synthetic analyses. When sampling to Naamlee. we ensure that the resulting win rates accurately represent the original dataset's findings, providing a faithful evaluation of recorded model perfor- mance. This standardization facilitates more reliable comparison and assessment of the Elo rating
Claude-2.1
0.39
GPT-4-1106-previis
0.86
GPT-4-0613
0.33

---

M
6PR-62334
10C 4:100
Marym10000
Kab.d
g56-4-2133
K-32, Nay--100
X-1, Pope 2ae.-110000 10000
K- Nace 100
Kel, 10000
2400 Food
1200
Lioe
K-16. M00r0
Co A.r 100
K-16, Mprii1
K-8 Ne
Npive110
K-166,
K-16,
GPT-4-0314 GPT-4-0613 and GPT-4-0613 vs. Claude-2.1
(b) GPT-4-1106-preeviee GPT-4-0613 and GPT-4-0613 vs. Claude-2.1
Recorded Win rates: 0.51 vs 0.49 and
Recorded Win rates: 0.67 vs 0.33 and
0.61 0.39
0.61 0.39
Figure 5: Elo scores (SA. SB and Sc) for three models at different configurations of N Nperms {1, 100, 10000} and K-factor = {11 8, 16]. The intersections score lines in in5aainnicaat fluctuat- ing relative rankings. highlighting inconsistency especially pronounced among models with close performance levels. contrast, demonstrates more stable relative rankings in conditions where win rates more skewed.
system under real-world conditions. In line with previous analyses, continue to explore the influence variations in N Aperms = {1, 100, 10000) and the K-factor (ranging from to 36) on Elo score robustness and reliability. We examine scenarios where model decisively outperforms another (e.g., Claude-1 vs. Claude-2.1) and cases where models are nearly evenly matched (e.g.. GPT-4-0314 vs. GPT-4-0613).
Key Findings Our analysis of real-world human feedback data confirms that the stability of Elo ratings is influenced by disparities in win rates, analogous to win probabilities synthetic data, and by the choice of hyperparameters K -fccoor and Npems- In cases where the models show clear difference in performance as indicated by their win rates, such as the Claude-1 vs. Claude-2.1 experiment, Elo ratings remain notably consistent across different K K-factors and N perms configurations (see Figure 4ap. On the other hand, cases like the GPT-4-031 vs. GPT-4-0613 experiment where win rates are closely matched, the Elo rating system exhibits higher volatility at N Nperms 1 but gains stability with larger Npers settings (100 and 10000), especially at lower K-factors (see Figure 4b).. The magnitude of Elo score differences in these experiments illustrates that larger K-factor and N Npems values can amplify or reduce the perceived performance gap between models, reflecting the critical role of these parameters in evaluation sensitivity.
Regarding the conservation of transitivity, our findings indicate that this property is not universally maintained across real-world human evaluations and synthetic scenarios (see Section (5) The relative rankings of models with similar performance levels are particularly sensitive to the choice of hyperparameters. Consequently, one should exercise caution in drawing conclusions from the Elo scores, especially in the absence of extensive paired comparison data as required by the combination formula These observations consistent with the trends from our synthetic data experiments.
Related Work
Several works have proposed improvements to the Elo rating system. Variants such as Glicko 18 19 20 and "rueSkill" 24 39 have incorporated more complex statistical methods into the original Elo framework, to address some the limitations of the Elo rating system, particularly in the context of games with more than two players or teams, games with more complex outcomes

---

than just win or loss. There is also ongoing research into the efficacy of these systems in diverse and dynamic environments [TT 77. Prior work has demonstrated some limitations of Elo in maintaining transitivity, especially in non-transitive cyclic games such as rock-paper-seissors and StarCraft IT
52). However, our work diverges by focusing on the reliability of Elo applied to large language model systems. To date, there has not been comprehensive evaluation in this context.
Independent from Elo, numerous studies have explored how sensitivity hyperparameters can undermine the generalization of findings A 36 23 27 9 in machine learning. This forms part of wider body work that considers which factors influence reliability and reproducibility 21
Notable directions includes studies on the impact of random seeds [40]
37 [T], model design 28 47), the use of data parallelism [45]. hardware [58] and test set construction
choices 46
48
43
49
30
38
Our work is complementary to these efforts, providing rigorous evaluation of the
impact of key hyperparameters and experimental settings on Elo performance.
8 Empirical Guidelines for Robust Elo-based Evaluation of LLMs
In this section, we distill essential practices for enhancing the reliability of Elo-based evaluation of language models. These guidelines, derived from empirical findings, differ notably from some conventional Elo settings and have significant implications for current real-world applications:
Achieving Score Stability: To obtain stable and reliable Elo ratings, it's recommended to run numerous permutations, ideally with Npems 2 100. This approach significantly improves the consistency of outcomes over single or fewer permutations commonly used.
Adjusting the K-factor: A smaller K-factor may reduce significant rating fluctuations when models have closely matched win rates.
Rapid Convergence for Clear Winners: When there is a clear performance disparity between models, a higher K-factor accelerates the alignment of Elo ratings with the models' "true" perfor- mance levels. This is in stark contrast to traditional uses of Elo ratings, where one-size-fits-al K-factor is frequently applied.
Transitivity is not guaranteed: The assumption that (A beats B and B beats C' implies A >> C)) is not consistently valid in Elo ratings. This is particularly invalid when models have similar performance levels, challenging a common assumption in many Elo-based evaluations.
These guidelines serve as empirically grounded recommendations to improve the robustness and interpretability Elo-based evaluations for LLMMs. Following these best practices will help in yielding more reliable conclusions on models' performance via human judgment.
Conclusion and Limitations
This paper presents a comprehensive study on the reliability of the Elo rating system for evaluating LLMs through human feedback within an axiomatic framework. We identify various factors that influence the robustness of Elo ratings and provide guidelines for their effective application in real-world scenarios. While our findings establish an essential foundation, they are by no means exhaustive. Future work could extend the present study by considering tie outcomes and adopting multi-category Bernoulli synthetic data to more closely simulate the varied landscape of human feedback. Such extensions could yield additional insights into the convergence properties of the Elo rating system in the fast-evolving field of language models.
10 Impact Statement
The implications of our work are significant in fields relying on LLMs for dacision-making content generation, and more. Improving the evaluation methods of LLMs contributes to the development of Al systems that are more reliable and trustworthy. This research also holds the potential to influence evaluation practices in other sectors that employ the Elo rating system, broadening its relevance and utility. However, it also emphasizes the need for cautious, informed application of Elo ratings to prevent misinterpretation or reliance on Elo-based rankings, particularly when the performance of models is comparable. As LLMs become more integrated into societal frameworks, ensuring the robustness and reliability of their evaluation mechanisms paramount to fostering ethical, beneficial AI advancements.
10

---

References
[1]
Anthropic. Model card and evaluations for claude models. https:/ www- cdn
2023.
[2]
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds. Danny Hermandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. general language assistant laboratory for alignment, 2021.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Satfield-Dodde Danny Hermandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022.
[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli. Dustin Li. Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton. Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodd Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.
[5] Lorena A. Barba. Terminologies for reproducible research, 2018
[6] Jakob Bernoulli. Ars conjectandi, opus posthumum. Accedit Tractatus de seriebus infinitis, et epistola gallicé scripta de ludo pilae reticularis. Thurneysen Brothers, Basel, 1713.
[7] Quentin Bertrand, Wojeiech Marian Czarnecki, and Gauthier Gidel. On the limitations of the elo, real-world games are transitive, not additive. In Francisco Ruiz, Jennifer Dy, and Jan- Willem van Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Reseurch, pages 2921. PMLR, 25-27 Apr 2023.
[8] John J. Binder and Murray Findlay. The effects of the bosman ruling on national and club teams europe. Journal of Sports Economics, 13:107-129, 2009.
[9] Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. Accounting for variance in machine learning benchmarks. In A. Smola, A. Dimakis, and I. Stoica, editors, Proceedings of Machine Learning and Systems, volume 3, pages 747-769, 2021.
[10) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation large language models, 2023.
[11] Arman Dehpanah, Muheeb Faizan Ghori, Jonathan F. Gemmell, and Bamshad Mobasher. Evaluating team skill aggregation in online competitive games. 2021 IEEE Conference on Games (CoG), pages 01-08, 2021.
[12] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint axXiv:2305.14314 2023.
11

---

[13] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, Enhancing chat language models by scaling high-qualin instructional conversations, 2023.
[14] Chris Drummond. Replicability is not reproducibility: Nor is it good science. In Evaluation Methods for Machine Learning Workshop at the 26th International Conference on Machine Learning, ICML, 2009.
[15] Aram Ebtekar and Paul Liu. Elo-mmr: A rating system for massive multiplayer competitions. In Proceedings of the Web Conference 2021, WWW '21, page 1772-1784, New York, NY, USA, 2021. Association for Computing Machinery.
[16] Arpad E. Elo. The Rating of Chessplayers, Past and Present. Arco Pub., New York, 1978.
[17] ESL. Ranking dota2 esl pro tour.
[18] Mark E Glickman. A comprehensive guide to chess ratings. American Chess Journal, pages 59-102, 1995.
[19] Mark Glickman. Parameter estimation large dynamic paired comparison experiments. Applied Statistics, pages 377-394, 1999.
[20] Mark E Glickman. Example of the glicko-2 system. Boston University, pages 1-6, 2012.
[21] Steven N. Goodman, Daniele Fanelli, and John A. loannidis. What does research repro- ducibility mean? Science Translational Medicine, 8/341):341ps12-34ppsll 2016.
[22] Odd Erik Gundersen and Sigbjørn Kjensmo. State of the art: Reproducibility in artificial intelligence. In AAAI, 2018.
[23] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. CoRR, abs/1709.06560. 2017.
[24] Ralf Herbrich, Tom Minka, and Thore Graepel. TrueskillTM; bayesian skill rating system. In
B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
[25] Lars Magnus Hvattum and Halvard Artzznn Using elo ratings for match result prediction in association football. International Journal of Forecasting, 26(3):460-470, 2010.
[26] Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models, 2023.
[27] Rudolf Kadlec, Ondrej Bajgar, and Jan Kleindienst. Knowledge base completion: Baselines strike back. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 69-74, Vancouver, Canada, August 2017. Association for Computational Linguisties.
(28) Wei-Yin Ko, Daniel D'souza, Karina Nguyen, Randall Balestriero, and Sara Hooker. Fair- ensemble: When fairness naturally emerges from deep ensembling, 2023.
[29] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment, 2023.
[30] Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Sebastian Ruder, Dani Yogatama, Kris Cao, Tomas Kocisky, Susannah Young, and Phil Blunsom. Pitfalls of static language modelling, 2021.
[31] Christoph Leitner, Achim Zeileis, and Kurt Hornik. Forecasting sports tournaments by ratings of (prob)abilities: comparison for the euro 2008. International Journal of Forecasting, 26(3):471-481) 2010. Sports Forecasting
12

---

[32] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang. Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models, 2022.
[33] Yen-Ting Lin and Yun-Nung Chen. LLM-Eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models, 2023.
[34] Wei lin Chiang, Lianmin Zheng, Lisa Dunlap, Joseph F. Gonzalez, lon Stoica, Paul Mooney, Sohier Dane, Addison Howard, and Nate Keating. Lmsys chathot arena human preference pre- 2024.
dictions. https://kagg com/ hwtm/ccomet.../ t ons httaiiin///aaaa-caaabb-o-rrrrree Kaggle.
[35]
Liquipedia. Elo rating liquipedia starcraft brood war wiki.
[36]
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study, 2018
[37] Pranava Madhyastha and Rishabh Jain. On model stability as a function of random seed. In Proceedings of the 23rd Conference Compulational Natural Language Learning (CoNLL), pages 939, Hong Kong, China, November 2019. Association for Computational Linguistics.
[38] Gábor Melis, Chris Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. ArXiv, hass/1707.05589 2018.
[39] Tom Minka, Ryan Cleven, and Yordan Zaykov. Trueskill 2: An improved bayesian skill rating system. Technical Report MSR-TR-2018-- Microsoft, March 2018
[40] Prabhat Nagarajan, Garrett Warnell, and Peter Stone. The impact of nondeterminism on reproducibility in deep reinforcement learning. In 2nd Reproducibility in Machine Learning Workshop at ICML, July 2018.
[41] Roman Novak, Yasaman Bahri, Daniel A. Abolatia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. In International Conference on Learning Representations, 2018
[42] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaire Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings. Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao. Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lppss, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene. Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Eukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis,
13

---

Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Man- ning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine Mcleavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pan- tuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard. Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Kata- rina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang. Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
[43] Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. On the challenges of using black-box apis for toxicity evaluation in research, 2023.
[44] April M. Reid. Elo rating system for video games explained.
[45] Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the effects of data parallelism on neural network training. 2019.
[46] Gil I. Shamir, Dong Lin, and Lorenzo Coviello. Smooth activations and reproducibility in deep networks, 2020.
[47] Luisa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. Llm see, 11m do: Guiding data generation target con-differentiable objectives, 2024.
[48] Robert R. Snapp and Gil I. Shamir. Synthesizing irreproducibility in deep networks. CoRR, abs/2102. 2021.
[49] Anders Søgaard, Sebastian Ehert, Jasmijn Bastings, and Katja Filippova. We need to talk about random splits. In Proceedings the Association for Computational Linguistics: Main Volume, pages 1823-1832, Online, April 2021. Association for Computational Linguistics.
the 16th Conference of the European Chapter of the
[50] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexan- der W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong. Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron
14

---

Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta. Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison- Christopher Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rívera, Clemencia Siro, Colin Raffel, Courtney Asheraft, Cristina Garbacea, Damien Sileo, Dan Gar- rette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito. Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong- Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Eliza- beth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang. Erkut Erdem, Ernie Chang. Ethan A Chi, Ethan Dyer, Ethan Jerzkk, Ethan Kim, Funice Engefu Manyasi, Eygenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang. Gon- zalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mchta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Keriion, Jacob Hilton, Jachoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou. Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orrllo Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markent, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo. Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contteras-Ochantee Louis Phillip Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros-Colón Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael An- drew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starriut, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T. Nanyun Peng, Nathan Andrew Chi, Naycon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts. Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar. Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung. Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang. Peter W Chang. Peter Eckers- ley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Scungjae Ryan Lee, Ryan Stovall, Ryan Techan Rylan Yang. Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruet- ter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop
15

---

Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny. Tobias Gerstenberg, Trenton Chang. Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
[51] Cecilia Summers and Michael Dinneen. Nondeterminism and instability in neural network optimization. In International Conference on Machine Learning, 2021.
[52] Nelson Vadori and Rahul Savani. Ordinal potential-based player rating, 2023.
[53] Ben P. Wise. Elo ratings for large tournaments of software agents in asymmetric games. ArXiv, abs/2105.0ne 0..009 2021.
[54] Yuxiang Wu, Zhengyao Jiang, Akbir Khan, Yao Fu, Laura Ruis, Edward Grefenstette, and Tim Rocktäschel. Chatarena: Multi-agent language game environments for large language models. https://github: com http://aaaee/caaaaeeeaaaaae
2023.
[55] Yining Ye, Xin Cong, Yujia Qin, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Large language model autonomous decision maker, 2023.
[56] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback, 2023.
[57] Lianmin Zheng, Wei-Linn Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
[58] Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. ArXiv, abs/2106.11872, 2021.
16

---

A Extension to Multiple Outcomes
For scenarios where outcomes can extend beyond wins and losses, such as tie option, one could make use of the multinomial distribution for the synthetic data generation process. For the three outcomes; win, loss, and tie, one sample according to the distribution:
P (Jbwin Masss Phics N, Pwin Ploss Prie)
N!
Tuans Priee Pwin Pioss Pise
(8)
Tdwin Mijss Maie
B Impact of Ordering on Elo Ratings: Skewed Win Rates
We summarize our findings on the impact of match sequences on Elo ratings for winning probabilities Prob(A A beals B) 2 0.65.
Model
Modal
ProbjA
ProbIA B)
ProbiA B)
ProbiA B)
Win ratos
02
Mpurna
Mpores
1900
mama maman
lw ym
100
Mferva 100
Ngams 100
Nperva 100
Meerve 100
1000
5900
1000
133
200
890
680
100
200
430
680
205
380
400
189
200
36
Number played
Figure 6: Impact win probabilities and permutation sampling on Elo ratings: Com- paring Model A and Model B across three different win probabilities (Prob( A beats B) 0.9, 0.8, 0.7, 0.65) with two levels of permutation sampling (Nperms and N Npems = 100). The top row displays the observed win rates, the middle row illustrates Elo ratings with single permutation. and the bottom row shows the mean and standard error the mean (SEM) of Elo ratings across 100 permutations.
C Chatbot Arena Human Preference Data Preparation
For the experimental validation of the Elo rating system using real-world data, we utilize the LMSYS dataset from (34]. We first vizualize the first 100 unique paired comparisons sorted descending order by the number of recorded evaluations. The distribution of tie vs. non-tie outcomes is shown in Figure 7 To refine the dataset for our analysis, we exclude tie results. focusing exclusively on win-loss dynamics. The remaining dataset is further filtered to identify pairs with at least 300 non-tie comparisons. This threshold of 300 allows us to encompass a broad spectrum of comparison scenarios, ranging from skewed win rates (ModelA >> Modely) to closely matched (Model, ~ Modelg). These selected paired comparisons are depicted in Figure
17

---

tco 2oodd apt S-tur 31
1goe-4 1-44-1100 1106 preview, openchat 15
Outcome
fgot
(etture 3ib, 345 chalt
ye
Non-Tie Tie
ogpt (peninni motral 8x76 instructioo
zam 21
(daudee 1. e sard m- 2011
a 0013. vicune 3361
icwudeinnant vicuna 7bi
(llmmer-11 pwim-21
Idluude tuk.
fapt 4 0125 preview misro modiumi
ineoaboychie 75, sirue 13bi
3penchat
(gentri get 3, 3.5 06131
Otama-2 rob-chat pote 2-7h ampoye-0 betal
(get turbo 2015, lama 2: 700 chatt
Ilama 2.76 chat. mistral 7b indtrctt
10 Th-nsmuccio).
(chentte wattee
(gah-3 S-turoo 0013. spenchat-
ulama-2 2-70b-hhi vicunaa 1261
A vs B Match
(gpe-3 Siturbo-- 0613 septun 70 betel
mistral instruct
(ast b-turo 0615. gpt tur 12001
tenate vitume
4:0344
couce pct-4 061
Imistral Th ratruct aeppye 76 betal
(geniii doder got 3.5 tu to 0613
(cleade-2 1. modreelle (earrwidxdb---tttit 7o
igemin dey apl get 0613
(clunde bt qp--4- peop
(lama 2. 708 chat, motrol 3x70 instrcttv
Idlude (cledde-2.. n preveat
(gemini pm dec-nol qpr
instruttie
laok-d b-tut bo- 100b mocral-tss
ombstral med moerol 8x76 in 11
0314,
100--prvicee
(gui S-tu 100, misia anatiaml
(optc) 00-1 3
ictsude
06131
(gpt: 3.5 to 06: mizro modiumi
(spi-1 Setrron-001 vitne 1lbi
Idbude instant got tu 1106)
(apt- previse mosre
(gpt-4-0114 326-4-06131
(coude-2.1 g2t-4
idiedde de- caetee2.1 31
(dbude 2.). 4 06131
qpt-4-11 (e-pnevioi
200
400
600
800
1000
Counts
Figure 7: Initial Distribution of Tie vs. Non-Tie Outcomes: A visual overview of the first 100 paired comparisons from the LMSYS dataset ordered by evaluation sizwe
18

---

lapt 008-4-0334, sp: 4-613)
(cloude 23
meditu mixtral- BcTo Tn-iintruct--00..
54.5to0---511 mistral-meduuu))
(gpl-9-0314, gpt-4-1 106-pracin))
(Hama-2- 3b-chai, vicuma-133)
igpt preview
Itamaa 2-700-char, vicune- 3390
(gpt-3.5
Igpe- 3.5 -i-tubbb---33, vicume-330)
vs B Match
(cluude instant 1, gpt turbo 1106)
mictral medium)
(gpt-4 0013, got-4 1-1---100 130 prestew)
(qpt-3.5 5-turto--1106 mintrrl to inatruuttttt...
faot-4-1 1106-groxdww, mixtral
3)
(clauce 2.1, (314)
1.pt 0513, got-4:1106 preview)
(clouce-211 2.1, gpt-4-00133
got-4-1 108-peactea))
Outcome
(claddee,, spi-3 54urbo-0e11))
Non-Tie
Tie
Normalized
Figure Normalized Tie vs. Non-Ti Tie outcomes for A model comparisons with at least 300
evaluations.
19

---

Table Win Rates for Paired Model Evaluations: This table presents the initial match counts and win rates for model comparisons from 34 where each pair has at least 300 matches, excluding
ties. Results from fixed sample size of 300 are shown demonstrate model performance under controlled sampling conditions.
Experiment
Original Size
Win Rates (%)
Sample Size
Sampled Win Rates (%)
gpt-4-0314
50.93
51.00
324
300
gpt-4-0613
49.07
49.00
gpt-4-1106-preieww
67.01
67.00
673
300
gpt-4-0613
32.99
33.00
g01-4-1106-preieww
81.25
81.33
528
300
gpt-3.5-turbo-0613
18.75
18.67
8pt---110-ppreieew
86.18
86.33
434
300
gpt-3.5-turbo-1100
13.82
13.67
claude-1
58.60
58.67
471
300
claude-2.1
41.40
41.33
gpt-4-0314
65.26
65.33
331
300
claude-2.1
34.74
34.67
gpt-4-0613
61.16
61.00
569
300
claude-2.1
38.84
39.00
g06-4-110-prrview claude-2.1
75.21
75.33
823
300
24.79
24.67
claude-instant-1
56.86
57.00
306
300
Ept-3.5-turbo-1066
43.14
43.00
106---1106p
71.92
72.00
317
300
mistral-medium
28.08
28.00
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately outline the core investigation into the Elo rating system's reliability and transitivity properties forL.MM, aligning with the empirical results presented, which validate and support these claims.
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This study probes the Elo rating system, highlighting its vulnerabilities without considering the impact of "tie" outcomes on Elo scores convergence. It shows its limitations for the most simplistic setting of pairwise comparison A and proposes for future work an extension of tie outcomes to better model real-world scenarios.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our study does not derive theoretical results. Our research primarily employs stress testing on synthetic data and subsequent validation through real-world experiments to evaluate the robustness of Elo scores.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We detail the process of generating synthetic human feedback data and refer to similar experimental setup used to collect real-world human feedback. Although results based on human judgment are inherently difficult to replicate exactly, the main focus of study is the reliability and robustness of the Elo rating system. This system requires only a sequence of outcomes (wins, losses, and ties, represented as I, O, or 0.3, respectively) compute scores, facilitating the reproduction of our analysis based on these simplified inputs.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
22

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide thorough details on generating synthetic human feedback data, key component of our experiments. We also outline our method for using LMSYS human preference data, which is publicly accessible. We also outline the steps for computing Elo scores based on existing literature. These comprehensive instructions should enable other researchers effectively replicate our experiments and verify our results. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nip) cc/
public gui CodeSu/mm /iddi ss Lonolli cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (ht //nips. cc/ htte//ublccggguuccoo /nipss.cc//ubliccuu...e
/oodeeeieeeeeeaeeeeeoeeee CodeeeeeeSSeeee Submi ss ssisnooPicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: In our robustness study the Elo rating system, provide detailed descrip- tions of our experimental setup, including how synthetic data is generated. We explore and visualize the effects of wide range of hyperparameters, such the number of permutations (N perms)) and the K-facton to assess their impact on Elo score stability. We compare lan- guage models in the 7B 12B parameters range, selected due to hardware computational limitations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
23

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our paper includes error bars represent the uncertainty in experimental outcomes, specificaly using standard errors of the mean (SEM) across different permutations and hyperparameter settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We generated completions in batches sized between and 50, depending on the size of each model evaluated, using an Nvidia A 1100 GPU with 40GB memory for efficient computation. Inference was performed in 6 setting to reduce memory usage, as deatailed in referenced work.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://eurriss........ http:///eerrrppscccc////c//ccccc
Answer: [Yes]

---

Our research adheres to the NeurlPS Code of Ethics in all respects. We ensured fair compensation for human evaluators involved the model pairwise comparisons. The pool of evaluation prompts includes open-source datasets (SODA and the Public Pool of Prompts (P3)), and were selected to avoid any harmful content.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses several positive societal impacts, including the enhance- ment of AI system reliability and trustworthiness through more robust evaluation methods for large language models (LLMs). It highlights how these improved methods contribute to the development of safer and more accurate language models. We also address potential negative impacts by cautioning against the misinterpretaaion and over-reliance on Elo ratings, which might lead to suboptimal performance assessments. This underscores the need for exhaustive and fair evaluation mechanisms in AI development.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any models or datasets, so there are no associated risks.
Guidelines:
The answer NA means that the paper poses such risks.
25

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use open-source datasets and properly credit their creators by citing the original papers. The specific versions and licenses of these datasets are acknowledged, ensuring compliance with their terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw thoode com / dataaaaasss asets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [Yes]
26

---

Justification: Details regarding the instructions for human feedback experiments and evalua- tion format based on established methodologies in the field. As for the compensation. while we cannot provide specific details, we can guarantee that fair compensation practices were followed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification: No potential risk within our experimental setup.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "2lQ99KxjhQ", "reviewer": "Reviewer_jqvp", "summary": "This paper examines  how well the Elo rating system works for evaluating LLMs based on human feedback. The authors first check how stable and reliable Elo ratings are under different settings and point out key factors that affect these ratings. They study how changing the order of matches and adjusting settings like the K-factor can influence the consistency of Elo scores. Using both synthetic and real-world data, the paper shows that Elo ratings become more reliable with more match permutations and the right K-factor adjustments.\n\nThe paper also discusses the transitivity of Elo scores, showing that this property doesn't always hold for models with similar performance levels. The authors suggest practical tips for more reliable Elo-based evaluations, like running many permutations for stable scores and adjusting the K-factor based on how different the models' performances are. They highlight the limitations of current Elo-based evaluation methods and propose future research directions, such as considering ties and using more complex data simulations to better mimic real human feedback scenarios.", "questions": "1. If I want to automatically evaluate $k$ models to get their final Elo ratings, do you think the size of $k$ will affect the robustness of the Elo rating?\n2. Why does Dolly-v2-7b have a slightly higher win rate than Dolly-v2-12b in Table 2?", "limitations": "The paper discusses its limitations in Section 8. My personal view on the limitations can be found in the weaknesses mentioned above.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Despite Elo rating being a popular method for evaluating LLMs, there is a lack of analysis on its robustness. This paper provides a comprehensive investigation, helping us better understand and implement Elo rating for LLMs.\n2. The authors explore the reliability and transitivity of Elo rating under different configurations. Their definitions and formulations are clear and reasonable.\n3. Experiments are conducted on both synthetic data and real human feedback, offering valuable insights for future Elo rating applications.", "weaknesses": "**Major Issue:**\n\nMy biggest concern is the relatively small amount of real data used in the experiments. Only 500 prompts and three model pairs are included, with just one pair having similar performance (Dolly-v2-7b vs. Dolly-v2-12b). Given that Chatbot Arena currently includes dozens of LLMs and has released a large amount of preference data, the volume of real data in this study seems insufficient.\n\n**Minor Issues:**\n1. As the authors mentioned, not considering ties appears to simplify the widely used \"win/tie/lose\" practice.\n2. The empirical guidelines provided may have limited applicability since most papers on automatic evaluation of LLMs report win rates rather than Elo ratings. Also, running more than 100 permutations seems costly and lacks practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer [ jqvp] for their helpful feedback and observations, noting that our study **“offering valuable insights for future Elo rating applications”**. We thank R  jqvp for noting the comprehensive nature of the evaluation, **“on both synthetic and real human feedback”**, and involving **many ablation variants** such as order of matches and K-Factor. We take this opportunity to address R jqvp concerns and feedback below.\n\n> My biggest concern is the relatively small amount of real data used in the experiments. Only 500 prompts and three model pairs are included, with just one pair having similar performance (Dolly-v2-7b vs. Dolly-v2-12b). Given that Chatbot Arena currently includes dozens of LLMs and has released a large amount of preference data, the volume of real data in this study seems insufficient.\n\nWe thank the reviewer for the suggestion and agree that additional data will strengthen our findings. We ran additional experiments during this rebuttal to include this new dataset (see shared response above for results).\n\n> As the authors mentioned, not considering ties appears to simplify the widely used \"win/tie/lose\" practice.\n\nThanks for the comment. In the case of a tie, the adjustment in Elo rating points is typically smaller, as ties are considered less informative about the relative strengths of the models. Ties are useful to upper and lower bound the pool of models (with very hard prompts that lead to \"both bad\" ties and vice versa) and otherwise they slow down the eventual convergence to true rankings.\n\n> The empirical guidelines provided may have limited applicability since most papers on automatic evaluation of LLMs report win rates rather than Elo ratings. Also, running more than 100 permutations seems costly and lacks practicality.\n\nReporting win rates for model evaluation is only feasible when we have a small number of models to compare because we need to play every two combinations of models against each other. Additionally, to get a reliable signal of performance, they are often run on various benchmarks and a large number of prompts. The advantage of the Elo score is that we can obtain a ranking of a large number of models and position a newcomer model well in this relative ranking with only a few pairs of games instead of covering every head-to-head evaluation. We note that widely used leaderboards such as Chatbot Arena [1] use Elo due to these advantages, so our study is timely at illustrating the limitations.\n\n> If I want to automatically evaluate k models to get their final Elo ratings, do you think the size of k will affect the robustness of the Elo rating?\n\nYes, we believe so. With a larger k we increase the reliability of the Elo rating because this system has more information to work with. More head-to-head comparisons help smooth out anomalies and provide a more accurate representation of each model's true performance. Elo ratings might converge more slowly because each model must be compared against a larger number of other models. However, once they do converge, the ratings are likely to be more accurate. A large and diverse prompt pool will also stabilize the Elo scores, especially regarding real-world performance. If the prompt pool used for Elo diverges from IRL prompts, the Elo scores will not accurately reflect actual performance.\nWe will update section 4 in the manuscript to include this detail.\n\n> Why does Dolly-v2-7b have a slightly higher win rate than Dolly-v2-12b in Table 2?\n\nWhile surprising, this is consistent with the evaluation performance self-reported by the model authors over 7 benchmarks: OpenBookQA, ARC (easy), WinoGrande, HellaSwag, ARC (challenge), PIQA, and BoolQ. In the model cards, it’s shown that the average performance of the 7B model and 12B model is 0.573 and 0.567 respectively [2]. \n\n[1] https://arena.lmsys.org/\n[2] https://huggingface.co/databricks/dolly-v2-7b\n\n---\n\nWe hope that we were able to satisfy the reviewer’s asks by running additional experiments on a much larger dataset and clarifying the details in question. If so, we ask that you consider raising your score to reflect and please do let us know if anything requires further clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vObVaOBbeM", "reviewer": "Reviewer_NZSD", "summary": "Evaluation plays an important role in LLM research. Previous works have assessed the performance using the Elo rating system, which is designed for ranking players in games. This paper shows that Elo rating does not always satisfy two critical axioms, reliability and transitivity. So the rankings of the models are not accurate. Transitivity is whether a consistently higher-rated model outperforms those with lower ratings in all scenarios. Reliability has two aspects, sensitivity to ordering and sensitivity to hyperparameters, how the Elo scores are affected by the order in which models are compared and the sensitivity to hyperparameter adjustments, particularly the K-factor.\n\nThe authors first validate the properties of Elo using synthetic data generation via Bernoulli processes. Then extend the validation to include real-world human feedback. The findings are that the stability of Elo ratings is influenced by disparities in win rates, analogous to win probabilities in synthetic data, and by the choice of hyperparameters K-factor and Nperms. Finally the authors provide empirical guidelines for robust Elo-based Evaluation.", "questions": "1. In line 93 you mention the K factor, but it is hard to understand for the reader unfamiliar with the Elo rating.\n\n2. How can the Elo rating be generalized to multiple-task evaluation?\n\n3. Why not conduct experiments on more widely used API-based LLMs, such as ChatGPT, Gemini, and Claude-3?", "limitations": "This paper has no potential negative societal impact of their work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The evaluation of LLMs plays an important role in LLM research. The reliability of the Elo rating is an interesting and meaningful research topic.\n\n2. The authors highlight two Axioms for Elo-based evaluation, which is reasonable and useful.\n\n3. This paper verifies the findings from both synthetic human feedback and real-world human feedback.\n\n4. This paper provides several empirical guidelines for robust elo-based evaluation.", "weaknesses": "1. The writing is not very friendly to the reader unfamiliar with the Elo rating.\n\n2. This paper only conducts experiments on three LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer [ NZSD] for their helpful feedback and observations, noting that our study is **“an interesting”** and **“meaningful research topic”** and find our axiom definitions **“reasonable and useful”**. We take this opportunity to address R NZSD concerns and feedback below.\n\n> The writing is not very friendly to the reader unfamiliar with the Elo rating. In line 93 you mention the K factor, but it is hard to understand for the reader unfamiliar with the Elo rating.\n\nThank you for your feedback. We agree that this could be more kind to the reader, we will provide a more in-depth introduction to the Elo rating system in the final manuscript, including the impact of the $K$-factor on it. \n\n> How can the Elo rating be generalized to multiple-task evaluation?\n\nThis is indeed an exciting question. One method would be to just run experiments per task and obtain an “Elo per task\" rating and then average these ratings across different tasks. We want to note that even the single task setup currently used with LLMs is complex enough since the \"game\" (prompt) changes every round and the win condition is subjective (due to human judgment). However adding a multi-task angle to capture the complexity of various domains and human feedback would be interesting.\nLastly, there are more sophisticated rating methods such as Trueskill that can be useful for such scenarios. TrueSkill [1] is a Bayesian ranking system that can handle multiple tasks using a vector of skill parameters, each representing a different task.\n\n> Why not conduct experiments on more widely used API-based LLMs, such as ChatGPT, Gemini, and Claude-3?\n\nWe thank the reviewer for this question. To start, we want to explain how the core of our findings are model-agnostic. Relative differences and win rates are all that matter when we are studying the dynamics of Elo. With introducing more powerful models, the results could be reduced to one of the two cases we already have: model pairs that are close in performance or model pairs that have a large performance gap. The only new dynamics that may occur with many SOTA models is saturating Elo scores at the top.\nBut that isn't very relevant here since it's easy to still find prompts that all LLMs are not good at so we haven't found a saturation point yet.\nIntroducing each new model necessitates a comprehensive comparison with all previous models, which can quickly become overwhelming. The primary reason for not using API-based LLMs is their potential for behind-the-scenes updates that can alter performance over the evaluation time and impact replicability [2]. Hence, for this treatment, we restricted consideration to open weights which were highly performant and we have control over when collecting preference feedback over a period of time. \nLastly, to cover a larger model list we looked at the Chatbot Arena dataset with 20 models and present the findings in the shared response of this rebuttal. We confirm our previous findings with this dataset.\n\n\n[1] Ralf Herbrich, Tom Minka, and Thore Graepel. 2006. TrueSkill™: a Bayesian skill rating system. In Proceedings of the 19th International Conference on Neural Information Processing Systems (NIPS'06). MIT Press, Cambridge, MA, USA, 569–576.\n\n[2] Luiza Pozzobon, Beyza Ermis, Patrick Lewis, Sara Hooker. 2023. On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research\n\n---\n\nWe hope that we were able to satisfy the reviewer’s asks by running additional experiments on a much larger dataset and clarifying the details in question. If so, we ask that you consider raising your score to reflect and please do let us know if anything requires further clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zv38IjisWr", "reviewer": "Reviewer_9SNU", "summary": "Updated after response: I think this paper could provide some valuable contributions to the community, but has real presentational issues. The authors fixed some of them that had an easy fix  (which I pointed out) during the rebuttal. However, I believe the presentational standards (and therefore the understandabiluty of the contribution) are not up to par for this conference, and I thus stand by my borderline reject recommendation. \n\nThis paper studies the Elo rating system, recently popular in the context of ranking LLMs. Specifically they study two axioms that evaluation methods should adhere to: reliability and transitivity. They find that Elo ratings for LLMs are highly sensitive to the order of comparisons and the choice of hyperparameters, and that desirable properties such as transitivity are not always guaranteed.\n\nIn their first experiment, the authors focus on order sensitivity. They conduct a controlled experiment with synthetic human feedback, in which they compare various model pairs with different probabilities of winning a pairwise comparison.\n\nThey show that, with only order of comparisans (which, if I understand correctly, would be the actual scenario in which we deploy this?), if model A has a probability 40<P<60 to win  a pairwise comparison with model B, even 500 games are not enough to converge to the conclusion that model A is better than model B. With 100 orders, the difference is clear after few games, though (semi) convergence happens only after around 80.  For smaller differences (51%), even 100 orders do not reliably show the difference between models. For differences of P(a_win > 60) Elo results are stable (?) (I can't fully match this statement in the text with the figures).\n\nNext, the authors investigate the impact of K-factor (which, I learned from Wikipedia, is the maximum number of points a ranking can be updated after a match). They show that higher K-factors eacerbate instability for single orderings for models that have similar performances, but that faster convergence is observed for higher K-factors when models diverge more.\n\nThen, the authors investigate transitivity, using a set of three models with different respective win probabilities. They find that when there is a clear disparity between models, Elo ratings accurately reflect the expected hierarchy, but in more complex cases the rankings become less stable, challenging the assumption of transitivity. K-factor and number of orderings play a key roal in the stability of the rankings.\n\nThe authors confirm several of their findings using real human feedback data. And wrap-up their paper with a list of recommendations for using Elo ratings.", "questions": "- If I understand correctly, the number of permutations is the number of game orders, why not use a word that matches that closely, e.g. 'game orders' or 'orderings'?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The topic studied in this paper (Elo ratings for LLMs) is important, and the experiments are extensive. The paper provides several interesting lessons for using Elo ratings for LLMs that I believe can (positively) influence how we compare models in rankings based on pair-wise comparisons.", "weaknesses": "While I think this paper could potentially be very impactful and has many interesting results, in its current state it does not seem ready for publication. It took me excessively long to review, because the presentation is rather messy, and not clear at several points.  Some of those things are easily fixable, e.g.:\n- The Elo rating system is briefly explained in an appendix, but given the importance of the concept it should be in the main text (if space is needed, I think equation 5 describing transitivity can safely be moved to the appendix).\n- Some explanation of the K-factor should also be included. It is sort of described, but not really explained or defined, not even in the appendix as far as I can see).\n- Maybe I missed it, but I don't think it is listed how Elo ratings are most commonly used for LLMs, is it with only a single ordering?\n- Table 1 is somewhat difficult to read, why not make a plot similar to Figure 1 but with multiple lines for the different models?\n- In the introduction -- impliations of our work -- it is listed that the work is particularly important in situations where model performances are closely matched, which is a common occurrence in many real-world scenarios, whereas in 4.2 it is listed that the result is relevant because skewed win-rates are common in real-world scenarios (or maybe that is not what is meant, if not I am not sure why it is mentioned that there are skewed win-rates in real world scenarios)\n\nMore problematic are the unclarities concerning the results, for example:\n- It is not clear with what K-factor the main experiment (4.1) is conducted (or at least I couldn't find it)\n- in 4.1 it is stated that for P(A beats B) >= 0.6 Elo results are stable; I can't match this result with Figure 1 (middle column), where there seems to be quite some variation for P(a beats B) = 0.55 as well as P(a beats B) = 0.6\n- The idea of comparing a single sequence with 100 different orderings is generally strong, but doing only one single sequence and one series of permutiations is not very informative in this case: what we actually want to know is how often a single sequence or an average over 100 sequences would give us the right results. Perhaps this single sequence was a very unlucky one among many that give us the correct result, or the 100 permutations were instead 100 lucky permutations. To really make the point, we'd need the variation between single sequences, and multiple selections of 100 permutations. (similar for Figure 4)\n- In 4.2 it is listed that faster convergence is observed for higher K-factors, but I don't understand the evidence for that. I don't think this is visible from Figure 3, and I couldn't find an appendix plot either.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer [9SNU] for their helpful feedback and their observations noting that our study is **“important”** and the experiments are **“extensive”**. We also hugely appreciate the reviewer’s note that this paper provides **“several interesting lessons for using Elo ratings for LLMs”** that **“can (positively) influence how we compare models”**. We take this opportunity to address R 9SNU helpful feedback and questions below:\n\n> Table 1 is somewhat difficult to read, why not make a plot similar to Figure 1 but with multiple lines for the different models?\n\nWe thank the reviewer for the helpful feedback to make this table more readable and understandable. We will take the great suggestion from this reviewer into account to update the manuscript. To clarify, the main takeaway from Table 1 is ​​the red star values for which we observe the cases where the Elo score fails to accurately reflect the expected hierarchy of models. \n\n> In the introduction -- implications of our work -- it is listed that the work is particularly important in situations where model performances are closely matched, which is a common occurrence in many real-world scenarios, whereas in 4.2 it is listed that the result is relevant because skewed win-rates are common in real-world scenarios (or maybe that is not what is meant, if not I am not sure why it is mentioned that there are skewed win-rates in real world scenarios)\n\nThanks for pointing out this unclarity in our manuscript. We believe that both scenarios (closely matched models and skewedly performing models) are equally important and they both occur in real-world model comparisons as is evident on the Chatbot Arena ranking. Our paramount goal in this work is to study the reliability and robustness of comparison methods such as Elo across all scenarios. We will clarify this detail in both the introduction and section 4.2.\n\n> It is not clear with what K-factor the main experiment (4.1) is conducted (or at least I couldn't find it)\n\nThanks for pointing out this missing point. We use $K=16$ for all experiments where we don’t explore different values of $K$. We will make this clear in the manuscript.\n\n> It is not clear with what K-factor the main experiment (4.1) is conducted (or at least I couldn't find it) in 4.1 it is stated that for P(A beats B) >= 0.6 Elo results are stable; I can't match this result with Figure 1 (middle column), where there seems to be quite some variation for P(a beats B) = 0.55 as well as P(a beats B) = 0.6*\n\nWe thank the reviewer for giving us the opportunity to clarify the results presented in Section 4.1. \n\n- $P(A \\text{ beats } B) > 0.6$: The Elo results for models where the win probability is $P(A \\text{ beats } B) > 0.6$ are presented in Figure 6 (and not Figure 1). There we observe that with both one sequence and 100 permutations (row 2 and row 3 respectively in Figure 6), the Elo convergence is stable, i.e., the rankings of the models don’t change over the number of games played. \n\n- $P(A \\text{ beats } B) = 0.6$ and $P(A \\text{ beats } B) = 0.55$: In Figure 1 we observe a slight instability with one one sequence but no change of model orders with increasing the number of games ($N_{perm} = 100$): model A achieves a higher Elo score with the first few games and stays on top as we increase the number of games.\n\n- $P(A \\text{ beats } B) = 0.51$: we observe a different pattern where depending on how many games we conduct the model order changes: at 100 model B is better than A and at 250 model A is better than model A.\nWe make sure that the Figures 1 and 6 are consolidated and the description of the figures are more clear in the final manuscript.\n\n> The idea of comparing a single sequence with 100 different orderings is generally strong, but doing only one single sequence and one series of permutations is not very informative in this [...]\n\nReviewer [9SNU] raised a valid concern about the potential bias from comparing one sequence with $N$ permutations, as the single sequence could be unusually lucky or unlucky. To address this, we conducted new experiments using a dataset with 33k prompts, as detailed in the shared response section above. \n\n> In 4.2 it is listed that faster convergence is observed for higher K-factors, but I don't understand the evidence for that. I don't think this is visible from Figure 3, and I couldn't find an appendix plot either.\n\nWe would like to thank the reviewer for pointing out this vagueness in Figure 3. We will clarify the convergence details in this figure by adding the time dimension and showcase the speed of convergence more clearly.\nTo explain the current figure we don’t currently show the traditional notion of convergence, but rather show stability. Here we present the final Elo score differences $(SA − SB)$ for different parameter values (different $K$-factors and win probability scenarios). For all matches model A beats model B (albeit with different probabilities of win) and a positive value for this difference is desired. For larger $K$ values, we observe larger score differences $(SA − SB)$ indicating that the system is more stable and confident in predicting model A is better than model B.\nNonetheless, we will ensure that the final manuscript is clear on this detail.\n\n> If I understand correctly, the number of permutations is the number of game orders, why not use a word that matches that closely, e.g. 'game orders' or 'orderings'?\n\nThat is true. We mainly use permutation because of its convention of use as a mathematical definition to describe a number of orderings and common use in combinatorics. We will use “game orders” and will define this term more explicitly and clearly in the final manuscript. \n\n\n---\n\nWe hope that we were able to clarify the reviewer’s questions and concerns. If so, we ask that you consider raising your score to reflect. If you have any further questions or concerns during the rebuttal period, please let us know. We are happy to provide any necessary clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lUTCgTdrlG", "reviewer": "Reviewer_hqxB", "summary": "This paper conducts an empirical study on the reliability and transitivity of the Elo rating system when evaluating Large Language Models. By conducting experiments on both simulated data and real-world scenarios, this paper suggests that certain parameter setting guidelines should be followed to ensure the stability of evaluation results when applying the Elo rating system to LLM assessments. The paper provides insights and guidance for future research and applications of the Elo rating system in various LLM evaluation contexts.", "questions": "My main concerns are listed in Weaknesses", "limitations": "I think that this paper has no significant limitations apart from what is mentioned by the authors in Section 8.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper provides a detailed experimental analysis of the practicality of the currently common Elo rating system used for evaluating the performance of Large Language Models (LLMs).\n\n- Through appropriate experimental setups, this paper explores the impact of various factors on the Elo rating system results, including the number of sequence arrangements, the number of games, the K value, and win rates.\n\n- By combining results from simulated experiments and real-world scenarios, this paper demonstrates good reliability in its findings.", "weaknesses": "- This paper explores the experimental results with up to three players. It would be more meaningful if the conclusions could be further generalized to settings with more players.\n\n- In reality, $N_{perms}$ can be set very large without obvious computational cost to stable the evaluation results. Therefore, in multiple experiments, I would prefer to see results with settings like $N_{perms}$ = 10k, similar to those in Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer [ hqxB] for their positive feedback including observing our **“detailed”** and **“appropriate”** experiment setup as well as noting the breadth of ablation including **“sequence arrangements, the number of games, the K value, and win rates”**. We appreciate the comment that the rigor of the analysis provides **“good reliability”** in the findings. We take this opportunity to address R hqxB helpful feedback and questions below:\n\n> This paper explores the experimental results with up to three players. It would be more meaningful if the conclusions could be further generalized to settings with more players.\n\nWe thank the reviewer for this suggestion. Introducing each new model necessitates a comprehensive comparison with all previous models, which can quickly become overwhelming. To provide an overview of the landscape, we targeted a wide range of models with different architectures (encoder-decoder and decoder-only), training data (public, private), and model sizes (3B to 100s of billions). Additionally, our simulated experiments are devoid of any model artifacts and potential biases. This approach helps contextualize and establish our findings within the real world.\nLastly, in the shared response above, we added analysis of a recently released dataset including 20 models (such as GPT-4, Claude-v1, and Palm-2) and observed confirmation of our findings in the manuscript. We will update the final manuscript with these results.\n\n> In reality, Nperms can be set very large without obvious computational cost to stable the evaluation results. Therefore, in multiple experiments, I would prefer to see results with settings like Nperms = 10k, similar to those in Figure 2.\n\nWe agree with the reviewer that the cost of large Nperm is offline and we can increase this number to an extent. We compute and update Elo ratings for \"$N_{\\text{pairwise games}} \\times N_{perm}$\" instances. As the Number of pairwise games ($N_{\\text{pairwise games}}$) grows by increasing the number of models and the number of prompts in the dataset, running a high number of permutations will slow down the evaluation process. More importantly, our study shows that we do not need to run unnecessarily high numbers of permutations. As shown in Figure 2 in the paper, we observe stable results with $N_{perm} = 100$ with minuscule standard errors of the mean and conclude that there is no need to run a higher number of permutations. With the new Chatbot Arena dataset added in the shared response section of the rebuttal, we have 33k prompts and observed stable results with $N_{perm} = 1000$. \n\n---\n\nWe hope that we were able to clarify the reviewer’s questions. If so, we ask that you consider raising your score to reflect. If you have any further questions or concerns during the rebuttal period, please let us know. We are happy to provide any necessary clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the thoughtful and positive feedback from the reviewers. 

We are encouraged that the reviewers found the paper comprehensive and helpful and the experimental setup clear and detailed [jqvp, hqxB] **“This paper provides a comprehensive investigation, helping us better understand and implement Elo rating for LLMs**.

We are also glad that the reviewers found the problem we try to address to be well motivated [NZSD, 9SNU]: **“The reliability of the Elo rating is an interesting and meaningful research topic”** and **“The topic studied in this paper (Elo ratings for LLMs) is important, and the experiments are extensive”**

Reviewers also positively commented on the usefulness and clarity of our formulations [jqvp, NZSD] **“Their definitions and formulations are clear and reasonable”** and **“highlight two Axioms for Elo-based evaluation, which is reasonable and useful”**.

We are happy that the rigor and reliability of verifying our findings through both synthetic human feedback and real-world scenarios are appreciated [jqvp, NZSD, hqxB]: **“By combining results from simulated experiments and real-world scenarios, this paper demonstrates good reliability in its findings.”**

We are heartened by the reviewers' excitement about our findings and guidelines for a more robust evaluation of LLMs and acknowledgment of the impact this paper can have [NZSD, 9SNU]:  **“The paper provides several interesting lessons for using Elo ratings for LLMs that I believe can (positively) influence how we compare models in rankings based on pairwise comparisons”.**

--- 

Below we take this opportunity to clarify some of the shared concerns between reviewers:

**1. On the details of Elo score, in particular, the explanation of the $K$-factor**

[NZSD] and [9SNU] raised valuable points asking for an explanation of the $K$-factor in the Elo formula to be part of the main paper and not in the appendix. The $K$-factor is a crucial parameter that determines how much a player's rating changes after each game and controls the sensitivity of the rating system to new results.
We agree that a more detailed introduction to the Elo score would significantly enhance understanding of the analyses in the paper, as well as clarify the roles of various parameters. We will make changes to the final manuscript to address this concern. 

**2. Strengthening the findings with a more comprehensive real-world experiment**

Reviewers raised valid and valuable concerns regarding the potentially low volume of real data in this study [jqvp], asked about increasing the number of models in the pool of comparison [hqxB and NZSD], the possibility of running into lucky sequences when the dataset is small [9SNU].

To address these concerns, we used the opportunity to run new experiments for this rebuttal using the preference dataset released by Chatbot Arena [1] which contains 33k conversations between 20 models and preference annotations from 13k users. In the manuscript, we studied the Elo score robustness to rank models with two changes: variations in the number of permutations and the $K$-factor.   

Kindly see the attached PDF for the results of these experiments. Figure A1 shows the pairwise winrate plot of the 20 models in this dataset.


**2.1. Impact of $K$-factor on Elo ranking:**

Out of the 20 models, 19 change their ranking position depending on different values of $K$, with the largest position difference being 6 (observed for llama-13b and alpaca-13b). On average, the rankings shift by 0.94 positions when different values of $K$ are used. Figure A2 in the attached PDF presents the heatmap highlighting this change in the models' rankings.


**2.2. Impact of number of permutations on Elo ranking:**

For a fixed $K$-factor value ($K=4$, as recommended by Chatbot Arena), we study the impact of the order of matches on computing the Elo score. We perform 10 runs of Elo computation, where the only difference is the shuffled order of matches in each run. We observe that the rankings change from one run to the next. Using one run as a reference and comparing the rankings of each shuffled run to this reference, we find that, on average, 11.7 out of the 20 models change positions in the rankings with each shuffling of orders. Figure A3 in the attached PDF presents the results.


When we shift from calculating the Elo score in a single run (the current standard in the field) to performing 10 random permutations of the sequence and averaging the Elo scores, we observe an average difference of 2.9 in rankings. Performing 100 permutations reduces this difference to 0.9 compared to the 10-shuffling run. The difference becomes minimal (0.1) when increasing from 100 to 1000 permutations. This finding is consistent with our observations in the synthetic scenario and the small dataset analyzed in the manuscript. We observe that the number of permutations required to arrive at a reliable and stable Elo ranking depends on the size of the dataset and the distribution of winrates: we achieved consistent Elo scores with 100 permutations on our smaller dataset (ranking 4 models) and 1000 permutations to achieve stability with the Chatbot Arena dataset (ranking 20 models).
See Figure A4 in the attached PDF, which presents the results.




[1] huggingface.co/datasets/lmsys/chatbot_arena_conversations

Meta-review: This paper evaluates the effectiveness of the Elo rating system for assessing large language models (LLMs) using human feedback. The authors investigate the stability and reliability of Elo ratings under various conditions, identifying key factors that impact these scores, such as match order and the K-factor setting. Through experiments with both synthetic and real-world data, they demonstrate that Elo ratings become more consistent with increased match permutations and appropriate K-factor adjustments. Additionally, the paper reveals that Elo score transitivity often fails for models with similar performance, suggesting that Elo-based evaluations can be unreliable in certain cases. To improve reliability, the authors recommend running numerous permutations and adjusting the K-factor according to model performance differences. The paper also addresses the limitations of Elo-based methods and suggests future research directions, including incorporating ties and using more complex simulations to better reflect real-world human feedback.

While the reviews and the author-reviewer discussions are quite extensive, the most critical concern of the two lower reviews are with respect to the writing. Unfortunately, I cannot identify with the authors' argument that this concern is overweighted, since writing is extremely important in scientific communication and the concern was raised by more than one reviewer. I am hence reluctant about this paper and my recommendation is given with a low confidence, even though I am leaning to the positive since the paper has the potential to become quite impactful.
Decision: Accept (poster)