Title: Elo Uncovered: Robustness and Best Practices in Language Model Evaluation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Elo Rating System

Language Model Evaluation

Reliability

Robustness

Reproducibility

LLMs Ranking
Abstract: In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such as chess, is increasingly being used to evaluate Large Language Models (LLMs) through "A vs B" paired comparisons.
However, while popular, the system's suitability for assessing entities with constant skill levels, such as LLMs, remains relatively unexplored. 
We study two fundamental axioms that evaluation methods should adhere to: reliability and transitivity. 
We conduct an extensive evaluation of Elo behavior across simulated and real-world scenarios, demonstrating that individual Elo computations can exhibit significant volatility.
We show that both axioms are not always satisfied, raising questions about the reliability of current comparative evaluations of LLMs.
If the current use of Elo scores is intended to substitute the costly head-to-head comparison of LLMs, it is crucial to ensure the ranking is as robust as possible.
Guided by the axioms, our findings offer concrete guidelines for enhancing the reliability of LLM evaluation methods, suggesting a need for reassessment of existing comparative approaches.

Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

Abstract

In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such as chess, is increasingly being used to evaluate Large Language Models (LLMs) through 'A vs B' paired comparisons. However, while popular, the system's suitability for assessing entities with constant skill levels, such as LLMs, remains relatively unexplored. We study two fundamental axioms that evaluation methods should adhere to: reliability and transitivity . Weconduct an extensive evaluation of Elo behavior across simulated and real-world scenarios, demonstrating that individual Elo computations can exhibit significant volatility. We show that both axioms are not always satisfied, raising questions about the reliability of current comparative evaluations of LLMs. If the current use of Elo scores is intended to substitute the costly head-to-head comparison of LLMs, it is crucial to ensure the ranking is as robust as possible. Guided by the axioms, our findings offer concrete guidelines for enhancing the reliability of LLM evaluation methods, suggesting a need for reassessment of existing comparative approaches.

1 Introduction

In the rapidly evolving field of Natural Language Processing (NLP), the task of accurately and reliably evaluating LLMs has become increasingly challenging [32, 10, 50, 26, 43]. Human feedback has emerged as an indispensable tool in this performance assessment process, serving as a qualitative metric that captures nuances that automated scoring mechanisms often fail to address [2, 3, 4, 50, 13, 12]. These human-centered evaluations, highly valuable to the overall progress of the NLP field, typically adopt an 'A vs B' comparative setup, turning evaluations into a zero-sum game between language models. Pairwise comparisons, however, are fundamentally difficult to scale for large pools of models, due to the quadratic growth of comparisons required. Fortunately, this paired feedback structure [56] naturally lends itself to the Elo rating system, originally designed for ranking chess players (including those who have never before played each other) for better matchmaking [16].

Under the Elo rating system, players' skills are indicated by an Elo rating , where higher ratings indicate higher skill, and all players can be ranked best to worst using this scalar Elo rating. In the standard formulation (see Section 2), a player rated at 1800 has 10:1 odds of winning against a player rated at 1400. After a match, the winner takes rating points from the loser in a zero-sum fashion [16]. Thus, with the Elo rating system, we can efficiently integrate subjective human feedback on paired 'A vs B' language model completions into a structured and unified rating system to assess the performance of language models.

Beyza Ermis Cohere For AI

Marzieh Fadaee

Cohere For AI

Model A

Model B

Prob(A beatsB)=0.6

Prob(A beats B)=0.55

Prob(A beats B) =0.51

0.6

0.5

Win rates

0.4

0.3

0.2

0.1

0.0

Nperms = 1

Nperms = 1

Nperms = 1

1475

1450

Elo rating

1425

1400

1375

1350

1325

Nperms = 100

Nperms = 100

Nperms = 100

1440

MS

1420

Elo rating

1400

Y0YYVX/0O

1380

1360

0

100

200

300

400

500

0

100

200

300

400

500

0

100

200

300

400

500

Number of played games

The core principles of Elo rating have proven to be resilient and adaptable due to its dynamic adjustments, relative rating focus, consistency across skill levels, and simplicity and transparency. As a result, the Elo rating system has found diverse applications, from predicting sports events outcomes [8, 25, 31, 53], and facilitating matchmaking in massively multiplayer online games like StarCraft II and Dota [15, 44, 35, 17], to its recent use in the evaluation of LLMs [2, 3, 4, 50, 13, 12, 54, 33]. However, to-date there has not been a comprehensive examination of the compatibility of Elo scores and LLMs evaluation.

Unlike dynamic competitors that evolve over time, LLMs have static capabilities and operate in a time-agnostic context. In this setting, evaluations of LLMs are not constrained by a preset number of turns, as is the case with tournament timelines or predefined match sequences. Moreover, the ordering of matches can significantly influence the final Elo scores and, consequently, model rankings. This oversight is particularly concerning, given the direct impact of Elo system rankings on both research directions and real-world applications in NLP as well as its widespread adoption [2, 3, 55, 57, 29, 4, 50, 13, 12, 54, 33].

This study aims to close this research gap by adopting an axiomatic approach and scrutinizing both the reliability and limitations of the Elo rating system when applied to LLMs. We study two fundamental axioms that evaluation methods should adhere to: reliability and transitivity . Through theoretical and empirical analyses grounded in collected human feedback data, our contributions provide a comprehensive understanding of when and how to reliably employ the Elo system for LLM evaluation, thus offering valuable guidelines for researchers and practitioners in the NLP field.

We find that Elo ratings for LLMs are highly sensitive to the order of comparisons and the choice of hyperparameters. Moreover, desirable properties such as transitivity are not always guaranteed and can be unreliable unless there is comprehensive human feedback data for all unique pairwise comparisons among models in the feedback pool. The sensitivity of Elo ratings becomes more pronounced when dealing with models that exhibit similar performance levels. We illustrate the best practices for addressing Elo rating sensitivities by offering guidelines for hyperparameter selection and matchmaking scenarios.

Implications of our work As LLMs rapidly advance, evaluation leaderboards are gaining popularity to assess the performance of newly introduced models using Elo scores. Elo can also be used in the learning framework of LLMs to produce a ranking of models and their outputs for preference training. No research has explored the nuances of using Elo scores to compare LLMs, which, unlike chess, exhibit static capabilities and operate in a time-agnostic manner. We show that Elo rating does not always satisfy two critical axioms-reliability and transitivity-leading to rankings of models that are not accurate. Our research offers guidelines for reliable and robust implementation of Elo scores when comparing LLMs. Deviation from our recommendations could result in inaccuracies when ranking LLMs, particularly in situations where model performances are closely matched, and Elo score differences are minimal (a common occurrence in many real-world scenarios).

2 Elo Algorithm Explained

We provide the standard mathematical formulation of the Elo algorithm [16], contextualized to the setting of LLM evaluation. In this formulation, let M be a set of models, and each model i ∈ M is assigned an initial numerical Elo rating R i . For each match between two models, we calculate the expected score , then update the ratings of both models as follows:

2.1 Expected Score Computation

For a given paired zero-sum match-up between two models A and B ( A,B ∈ M ), each with respective pre-match ratings R A and R B , the expected scores E A and E B (i.e., match outcomes) are computed as:

In this context, the factor of 400 [16] precisely adjusts the sensitivity of the expected score to differences in ratings. A 400 -point advantage in ratings translates to a 10 : 1 odds in favor of the higher-rated model, providing an interpretable metric for performance comparison. For evenly matched models ( R A = R B ) , both E A and E B equate to 0 . 5 , reflecting a 50:50 win probability for both models.

2.2 Rating Update Mechanism

Following each match, the Elo ratings are updated based on the observed win-loss outcome. The rating adjustment for each model is dictated by the equation:

Here, S A represents the actual score achieved by model A , which can take on either the value 0 for a loss or 1 for a win. Model B's Elo rating is updated via the same method. The K -factor serves as a variable hyperparameter to adapt the rate of change in rating to different scenarios. A higher K -factor results in larger changes in the Elo score after each match-up, making the scoring more sensitive to individual results. A lower K -factor, in contrast, makes the Elo ratings more stable, with smaller changes after each match. In chess, the K -factor is usually set to 16 for masters and to 32 for novice players.

3 Desirable Properties of Elo

The objective of using Elo scores to rank models is to establish a comparative understanding of the performance hierarchy among them. When incorporating a new model into an already ranked list, only a limited number of pairwise annotations are required to determine its position in the ranking. The ability to infer a model's relative performance compared to all previous models in the list relies on the robustness of the scoring method and the transitive property of the ranking system. We describe these desirable properties through two axioms: transitivity and reliability .

3.1 Axiom 1: Transitivity

A desirable property of any rating system is transitivity because it ensures consistency and logical coherence in how entities are ranked or rated. Transitivity in this context means that if player A beats player B , and player B beats player C , then player A is expected to beat player C . If the ranking of large language models exhibits transitivity, we can deduce their comparative performance without the need for direct head-to-head evaluations between every pair of models. The central assumption in developing various leaderboards for comparing language models is that the rankings adhere to the principle of transitivity [57].

While Elo's design inherently assumes transitivity, our synthetic data which are derived from realistic scenarios, uncovers certain circumstances that violate this assumption. Such anomalies can affect the final ranking of language models and their relative performance assessments.

3.2 Axiom 2: Reliability

We consider two aspects of reliability:

Sensitivity to ordering: Unlike chess or time-bound sports where match sequences are structured, in LLM evaluations all matches can occur independently and in parallel, amplifying the sequence's influence on final model ranking. In this context, each match represents the performance comparison between two models on a specific prompt. If the prompts are presented in a specific order, and one model happens to perform better on the initial set of prompts, it may gain an advantage in subsequent comparisons due to the cumulative effect of its early success. This inherent variability prompts us to investigate the extent to which match-up ordering affects the robustness of Elo ratings.

Sensitivity to hyperparameters: The sensitivity of hyperparameters can compromise the robustness of Elo scores leading to inconsistent rankings. Evaluating and understanding this sensitivity is crucial for building evaluation frameworks that maintain consistency across diverse models. In this work, we evaluate the sensitivity of Elo performance to one key hyperparameter, the K -factor. This factor acts as a scaling constant in the Elo rating system, pivotal for updating ratings after each matching. It essentially determines how quickly a model's rating converges to what can be considered its 'true' skill. While conventional applications like chess use standard K -factor values, these may not be directly applicable in the context of evaluating LLMs due to the unique characteristics and requirements of this domain.

4 Synthetic Human Feedback

Given the costly and time-consuming nature of human evaluations, studying the Elo system's behavior under various scenarios becomes challenging. To circumvent these limitations, we first validate the properties of Elo using synthetic data generation via Bernoulli processes to simulate various human feedback scenarios. In Section 6 we extend these evaluations to include real-world human feedback. This time-agnostic and independent setup of LLM evaluations resembles a Bernoulli process [6], a sequence of independent experiments, each yielding a simple 'win' or 'loss' outcome, representing one model outperforming another. We use this setting to control the characteristics of the distribution and evaluate the different desirable properties of a rating system.

In this controlled setting, our primary objectives include testing the transitivity axiom-whether a consistently higher-rated model outperforms those with lower ratings in all scenarios. Additionally, in studying the reliability axiom, we explore how the Elo scores are affected by the order in which models are compared and the sensitivity to hyperparameter adjustments , particularly the K -factor. This synthetic setup offers a robust platform to dissect and understand the dynamics of the Elo rating system in the context of LLM evaluations, without the constraints and limitations of relying solely on real-world human feedback.

4.1 The Bernoulli Analogy

Pairwise comparisons in LLM evaluation draw parallels with the foundational principles of the Bernoulli experiment in probability theory. This section studies the similarity between human feedback-based evaluations and the Bernoulli experiment's principles.

Preliminaries A Bernoulli trial is a random experiment with exactly two possible outcomes, 'success' or 'failure'. The outcomes adhere to the probability condition:

Here, the random variable X denotes the outcome, where X = 1 implies success, and X = 0 signifies failure. The probabilities associated with these outcomes are given by:

with 0 ≤ p ≤ 1 , the 'success' probability.

Mapping to Human Feedback When comparing two models, A and B , across N pairwise evaluations, the setup aligns with a Bernoulli process. This process comprises a sequence of independent and identically distributed ( i.i.d ) Bernoulli trials. To frame this analogy, we designate a win probability, P ( A win ) , to model A . Leveraging a Bernoulli random variable, X , as a means to simulate synthetic human feedback, we proceed as follows:

A sample is drawn from X using P ( A win ) .

If X = 1 , feedback suggests a preference for model A .

Otherwise, model B is favored.

Extending to Multiple Players Given a finite set of n distinct models M , their pairwise comparisons can be formulated as:

̸

This formula yields ( n 2 ) unique pairs ( A,B ) where A,B ∈ M and A = B . For each pair, a Bernoulli process comprising multiple Bernoulli experiments is conducted to discern which model performs better over a sequence of trials.

4.2 Synthetic Data Generation

Building upon the Bernoulli process analogy, when conducting multiple independent evaluations between two models, the distribution of the number of times one model is preferred over the other naturally follows a binomial distribution. For N pairwise comparisons, the relation is:

where P ( k ; N,p ) is the probability of one model being preferred k times out of N evaluations. p is the success probability and ( N k ) is the binomial coefficient, representing the number of ways to choose k successes from N trials.

5 How Robust Are Elo Scores?

This section describes rigorous stress tests designed to investigate whether the two axioms, presented in Section 3, are satisfied in this evaluation framework. We focus on critical desirable properties of a ranking mechanism - that it should (1) be insensitive to match-up ordering, (2) not be overly sensitive to hyperparameters like the K -factor, and (3) preserve properties of transitivity. Subsequently, we provide empirically grounded guidelines for a safe and interpretable application of Elo ratings.

5.1 Impact of Ordering on Elo Ratings

Experimental Setup To quantify the effect of match-up ordering, we generate a baseline sequence of N games = 1000 match outcomes between models A and B (see Equation 6), reflecting the scale typical of LLM evaluations via human feedback. We hold N games constant for the entirety of our study to maintain consistency. From this baseline, we derive N perms distinct permutations, each involving a complete reshuffling of the initial sequence to simulate various chronological orders in which the games might unfold. It is important to note that we are not generating new match outcomes for each permutation; instead, we simply reorder the existing data to explore the potential impact of different

(a) Elo Scores for a Single Sequence

298

326

315

328

329

-300

-200

1

Prob(AbeatsB)

149

197

242

302

357

S

-100

Difference

0.6

60

49

37

9

-39

-0

-100

Scores

0.55

31

38

40

62

107

-200

Elo

0.51

-3

-49

-72

66-

-112

-300

1

8

K-factor

16

32

64

(b) Elo Scores Averaged Over 100 Permutations

400

304

390

393

404

426

-300

B

151

170

173

174

175

-200

Prob(A beatsl

-100

Difference

0.6

77

87

87

88

89

-0

-100

Scores

0.55

26

24

24

26

30

-200

-300

0.51

14

14

15

18

24

-400

1

8

16

32

64

K-factor

match-up sequences. For each reordered sequence, we update the Elo ratings R A and R B according to equation 2, resetting both ratings to an initial value of 1400 at the start of each permutation. Finally, we compute average Elo ratings per match across all N perms permutations, ensuring a robust analysis that takes into account the full range of possible match-up orders.

We repeat this process to generate baseline sequences and their respective reorderings for a set of selected winning probabilities enabling us to inspect ratings' behavior under various real-world scenarios. N perms is varied from a minimum of 1 to a maximum of 10k, providing a robust sample size for statistical analysis (see Figure 3). Subsequently, we compute the average Elo ratings per match across all permutations. These averages, ¯ R A and ¯ R B . particularly for N perms = 1 and N perms = 100 , are visualized to offer insights into the stability of the ratings, as shown in Figure 1.

Key Findings Our analysis underscores the interplay between winning probability P ( A win ) and the number of different orderings N perms on the stability of Elo ratings after each update. For P ( A win ) ≥ 0 . 6 , Elo ratings demonstrate high stability; additional results for P ( A win ) = 0 . 65 and beyond are available in Appendix B. On the other hand, for P ( A win ) ≈ 0 . 5 , ratings exhibit significant instability for a single sequence. As depicted in Figure 1, when both models have win probabilities around 0 . 5 , Elo ratings frequently intertwine, making it challenging to discern a clear performance difference between the two. The instability plateaus as N perms exceeds 100, resulting in stabilized Elo ratings that align closely with the preset winning probabilities. For instance, at P ( A win ) = 0 . 55 , the average Elo rating for Model A , ¯ R A , consistently exceeds that for Model B , ¯ R B ,

1480

Score[SA]

1460

Prob(A beats B) = 0.7

Prob(A beats B) = 0.55

Prob(A beats B)=0.6

Prob(A beats B)=0.51

1440

Elo

1420

1400

1380

10

10°

10

10

Nperms

when averaged across multiple permutations, reflecting an accurate performance-based ranking of these models. These observations validate our concerns highlighted earlier, emphasizing the critical role of N perms for a reliable interpretation of Elo ratings in LLM evaluations. In Elo-based evaluations, the sequence of model comparisons can significantly influence the final Elo scores, particularly in scenarios with models of similar quality, where this effect is magnified.

5.2 Sensitivity to Hyperparameters

Experimental Setup We extend our previous approach by conducting tests across a range of winning probabilities and multiple K -factor values ( 1 , 8 , 16 , 32 , 64 ). We compute and compare the

average Elo scores ¯ S A and ¯ S B for N games = 1000 and N perms = { 1 , 100 } . The differences between these final averages for Model A and Model B are summarized in Figure 2 to assess the stability and expected ranking between the two models.

Key Findings As shown in Figure 2, notable instability is observed in model rankings based on the final Elo scores when we consider a single sequence of paired comparisons (i.e., N perms = 1 ), especially for winning probabilities nearing 0.5. This instability is markedly exacerbated at higher K -factors. In contrast, the picture changes when coupling higher K -factors with raising the number of permutations to at least 100. Higher K -factors, in this multi-permutation scenario, speed up the differentiation between models' Elo scores, enabling faster convergence to their true skill levels. This yields much more stable and reliable model rankings. It is noteworthy that this faster convergence is observed to be more reliable for higher winning probabilities, which corresponds to skewed win rates in a real-world scenario.

5.3 Transitive Properties of Elo Scores

Experimental Setup The transitivity property of the Elo scores is defined as:

To test the transitivity property, we design four distinct scenarios that model real-world conditions:

K Model A beats model B and model B beats model C both with high win probabilities ( P win = 0 . 75 ).

R Model A beats model B with a high win probability ( P win = 0 . 75 ), model B beats model C with a win probability close to 0.5 ( P win = 0 . 51 ).

B Model A beats model B with a win probability close to 0.5 ( P win = 0 . 51 ), model B beats model C with a high win probability ( P win = 0 . 75 ).

N Model A beats model B with a win probability of 0.54, model B beats model C with a win probability of 0.51.

In each of these scenarios, we simulate matches for paired comparisons ' A vs. B ' and ' B vs. C ' and then rearrange these matches in an arbitrary order to form our baseline sequence. This approach mimics how Elo ratings are computed for online leaderboards in the evaluation of large language models [54, 33]. We then analyze whether Elo scores maintain the expected model hierarchies.

Key Findings The outcomes from all four scenarios, detailed in Table 1, demonstrate the performance of Elo-based rankings across various configurations. In scenarios where there is a clear disparity between models (e.g., K ), Elo ratings accurately reflect the expected hierarchy. However, in

(a) Experiment: Claude-1 vs. Claude-2.1

Recorded Win rates

:

0.59 vs 0.41

(b) Experiment: GPT-4-0314 vs. GPT-4-0613

Recorded Win rates

:

0.51 vs 0.49

60

3

claude-1

claude-1

40

Difference

claude-1

claude-1

20

K-factor

Scores

8

claude-1

claude-1

20

Elo

40

claude-1

claude-1

60

1

10000

Nperms

60

3

gpt-4-0613

gpt-4-0314

40

Difference

gpt-4-0613

gpt-4-0314

20

K-factor

Scores

8

gpt-4-0613

gpt-4-0314

-20

Elo

40

gpt-4-0613

gpt-4-0314

60

1

10000

Nperms

more complex cases such as R and B , where one model significantly outperforms a second, which in turn is closely matched with a third, the rankings become less stable, challenging the assumption of transitivity. We observe once again that varying the number of permutations ( N perms = 1 vs. N perms = 100 ) and the K -factor plays a critical role in stability. In the R and B scenarios, with N perms = 100 and K = 1 , we notice discrepancies in the models' rankings. This contrasts with K = 16 , where rankings are more consistent and accurate. The slower updates from K = 1 suggest this setting may be too conservative to capture transitive relations quickly, leading to inconsistencies.

6 Validation on Real-World Human Feedback

Building on the insights gained from synthetic data experiments, this section extends the validation of the Elo rating system to real-world human feedback. Our objectives are twofold: first, to ascertain how the properties demonstrated using synthetic data generalize to real human annotations, and second, to evaluate the Elo rating system's utility for assessing LLMs in practical settings.

Experimental Setup We use the LMSYS - Chatbot Arena dataset [34], an open-source collection of human preference data derived from unique users' interactions with two distinct models responding to a set of userdefined prompts. To align with our methodology from synthetic data analysis, tie outcomes have been excluded from this analysis to focus specifically on the implications of win-loss dynamics. We select pairs of models (A vs. B) from the initial dataset that feature at least 300 non-tie comparisons. This threshold ensures statistical robustness and allows us to include cases where win rates are closely contested, which can lead to more sensitive ratings. These pairs predominantly involve models from the GPT-4 family [42] and the Claude family [1]. A comprehensive list of model pairs is included in Appendix C under Table 3, and a subset discussed here is shown in Table 2. The recorded win rates primarily exhibit skewed preferences, with the

exception of the GPT-4-0314 vs. GPT-4-0613 pairing, indicating comparable performance levels (see Table 2). Given the variable number of evaluations per pair in the original dataset, we standardize this by sampling a fixed number, N sample, for each pair to align with the controlled conditions used in synthetic analyses. When sampling to N sample, we ensure that the resulting win rates accurately represent the original dataset's findings, providing a faithful evaluation of recorded model performance. This standardization facilitates a more reliable comparison and assessment of the Elo rating

(a) GPT-4-0314 vs. GPT-4-0613 and

GPT-4-0613 vs. Claude-2.1

Recorded Win rates

:

0.51 vs 0.49

and

0.61 vs 0.39

(b) GPT-4-1106-preview vs. GPT-4-0613 and

GPT-4-0613 vs. Claude-2.1

Recorded Win rates

:

0.67 vs 0.33

and

0.61 vs 0.39

K=1, Nperm=1

gpt-4-0314

K=32,Nperm=10000

K=1,Nperm=100

gpt-4-0613

claude-2.1

K=32, Nperm=100

K=1,Ngerm=10000

500

1400

1300

1200

1100

K=32,Ngerm=1

K=8,Nperm=1

K=16, Nperm=10000

K=8.Mperm=100

K=16, Nperm100

K=8,Nperm=10000

K=16, Nperm=1

K=1, Nperm=1

gpt-4-1106

K=32,Nperm=10000

K=1,Mperm=100

gpt-4-0613

claude-2.1

K=32,Npem=100

K=1,Nperm=10000

1500

1400

1300

1100

1200

K=32,Ngerm=1

K=8,Nperm=1

K=8. Mperm=100

K=16, Nperm=100

K=8,Nperm=10000

K=16, Nperm=1

system under real-world conditions. In line with our previous analyses, we continue to explore the influence of variations in N perms = { 1 , 100 , 10000 } and the K -factor (ranging from 1 to 36) on Elo score robustness and reliability. We examine scenarios where one model decisively outperforms another (e.g., Claude-1 vs. Claude-2.1) and cases where models are nearly evenly matched (e.g., GPT-4-0314 vs. GPT-4-0613).

Key Findings Our analysis of real-world human feedback data confirms that the stability of Elo ratings is influenced by disparities in win rates, analogous to win probabilities in synthetic data, and by the choice of hyperparameters K -factor and N perms. In cases where the models show a clear difference in performance as indicated by their win rates, such as in the Claude-1 vs. Claude-2.1 experiment, Elo ratings remain notably consistent across different K -factors and N perms configurations (see Figure 4a). On the other hand, in cases like the GPT-4-0314 vs. GPT-4-0613 experiment where win rates are closely matched, the Elo rating system exhibits higher volatility at N perms = 1 but gains stability with larger N perms settings (100 and 10000), especially at lower K -factors (see Figure 4b). The magnitude of Elo score differences in these experiments illustrates that larger K -factor and N perms values can amplify or reduce the perceived performance gap between models, reflecting the critical role of these parameters in evaluation sensitivity.

Regarding the conservation of transitivity, our findings indicate that this property is not universally maintained across real-world human evaluations and synthetic scenarios (see Section 5). The relative rankings of models with similar performance levels are particularly sensitive to the choice of hyperparameters. Consequently, one should exercise caution in drawing conclusions from the Elo scores, especially in the absence of extensive paired comparison data as required by the combination formula 5. These observations are consistent with the trends from our synthetic data experiments.

7 Related Work

Several works have proposed improvements to the Elo rating system. Variants such as Glicko [18, 19, 20] and TrueSkill ™ [24, 39] have incorporated more complex statistical methods into the original Elo framework, to address some of the limitations of the Elo rating system, particularly in the context of games with more than two players or teams, or games with more complex outcomes

than just win or loss. There is also ongoing research into the efficacy of these systems in diverse and dynamic environments [11, 7]. Prior work has demonstrated some limitations of Elo in maintaining transitivity, especially in non-transitive cyclic games such as rock-paper-scissors and StarCraft II [7, 52]. However, our work diverges by focusing on the reliability of Elo applied to large language model systems. To date, there has not been a comprehensive evaluation in this context.

Independent from Elo, numerous studies have explored how sensitivity to hyperparameters can undermine the generalization of findings [41, 36, 23, 27, 9] in machine learning. This forms part of a wider body of work that considers which factors influence reliability and reproducibility [21, 22, 5, 14]. Notable directions includes studies on the impact of random seeds [40, 37, 51], model design choices [46, 48, 43, 28, 47], the use of data parallelism [45], hardware [58] and test set construction [49, 30, 38]. Our work is complementary to these efforts, providing a rigorous evaluation of the impact of key hyperparameters and experimental settings on Elo performance.

8 Empirical Guidelines for Robust Elo-based Evaluation of LLMs

In this section, we distill essential practices for enhancing the reliability of Elo-based evaluation of language models. These guidelines, derived from our empirical findings, differ notably from some conventional Elo settings and have significant implications for current real-world applications:

Achieving Score Stability : To obtain stable and reliable Elo ratings, it's recommended to run numerous permutations, ideally with N perms ≥ 100 . This approach significantly improves the consistency of outcomes over single or fewer permutations commonly used.

Adjusting the K -factor : A smaller K-factor may reduce significant rating fluctuations when models have closely matched win rates.

Rapid Convergence for Clear Winners : When there is a clear performance disparity between models, a higher K-factor accelerates the alignment of Elo ratings with the models' 'true' performance levels. This is in stark contrast to traditional uses of Elo ratings, where a one-size-fits-all K-factor is frequently applied.

Transitivity is not guaranteed : The assumption that ( A beats B and B beats C implies A > C ) is not consistently valid in Elo ratings. This is particularly invalid when models have similar performance levels, challenging a common assumption in many Elo-based evaluations.

These guidelines serve as empirically grounded recommendations to improve the robustness and interpretability of Elo-based evaluations for LLMs. Following these best practices will help in yielding more reliable conclusions on models' performance via human judgment.

9 Conclusion and Limitations

This paper presents a comprehensive study on the reliability of the Elo rating system for evaluating LLMs through human feedback within an axiomatic framework. We identify various factors that influence the robustness of Elo ratings and provide guidelines for their effective application in real-world scenarios. While our findings establish an essential foundation, they are by no means exhaustive. Future work could extend the present study by considering tie outcomes and adopting multi-category Bernoulli synthetic data to more closely simulate the varied landscape of human feedback. Such extensions could yield additional insights into the convergence properties of the Elo rating system in the fast-evolving field of language models.

10 Impact Statement

The implications of our work are significant in fields relying on LLMs for decision-making, content generation, and more. Improving the evaluation methods of LLMs contributes to the development of AI systems that are more reliable and trustworthy. This research also holds the potential to influence evaluation practices in other sectors that employ the Elo rating system, broadening its relevance and utility. However, it also emphasizes the need for cautious, informed application of Elo ratings to prevent misinterpretation or reliance on Elo-based rankings, particularly when the performance of models is comparable. As LLMs become more integrated into societal frameworks, ensuring the robustness and reliability of their evaluation mechanisms is paramount to fostering ethical, beneficial AI advancements.

References

Anthropic. Model card and evaluations for claude models. https://www-cdn. anthropic.com/5c49cc247484cecf107c699baf29250302e5da70/ ModelCardClaudev2_with_appendix_v1.pdf , 2023.

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment, 2021.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.

Lorena A. Barba. Terminologies for reproducible research, 2018.

Jakob Bernoulli. Ars conjectandi, opus posthumum. Accedit Tractatus de seriebus infinitis, et epistola gallicé scripta de ludo pilae reticularis . Thurneysen Brothers, Basel, 1713.

Quentin Bertrand, Wojciech Marian Czarnecki, and Gauthier Gidel. On the limitations of the elo, real-world games are transitive, not additive. In Francisco Ruiz, Jennifer Dy, and JanWillem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 2905-2921. PMLR, 25-27 Apr 2023.

John J. Binder and Murray Findlay. The effects of the bosman ruling on national and club teams in europe. Journal of Sports Economics , 13:107-129, 2009.

Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. Accounting for variance in machine learning benchmarks. In A. Smola, A. Dimakis, and I. Stoica, editors, Proceedings of Machine Learning and Systems , volume 3, pages 747-769, 2021.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models, 2023.

Arman Dehpanah, Muheeb Faizan Ghori, Jonathan F. Gemmell, and Bamshad Mobasher. Evaluating team skill aggregation in online competitive games. 2021 IEEE Conference on Games (CoG) , pages 01-08, 2021.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 , 2023.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.

Chris Drummond. Replicability is not reproducibility: Nor is it good science. In Evaluation Methods for Machine Learning Workshop at the 26th International Conference on Machine Learning, ICML , 2009.

Aram Ebtekar and Paul Liu. Elo-mmr: A rating system for massive multiplayer competitions. In Proceedings of the Web Conference 2021 , WWW '21, page 1772-1784, New York, NY, USA, 2021. Association for Computing Machinery.

Arpad E. Elo. The Rating of Chessplayers, Past and Present . Arco Pub., New York, 1978.

ESL. Ranking - dota2 - esl pro tour.

Mark E Glickman. A comprehensive guide to chess ratings. American Chess Journal , pages 59-102, 1995.

Mark E Glickman. Parameter estimation in large dynamic paired comparison experiments. Applied Statistics , pages 377-394, 1999.

Mark E Glickman. Example of the glicko-2 system. Boston University , pages 1-6, 2012.

Steven N. Goodman, Daniele Fanelli, and John P. A. Ioannidis. What does research reproducibility mean? Science Translational Medicine , 8(341):341ps12-341ps12, 2016.

Odd Erik Gundersen and Sigbjørn Kjensmo. State of the art: Reproducibility in artificial intelligence. In AAAI , 2018.

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. CoRR , abs/1709.06560, 2017.

Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill™: A bayesian skill rating system. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems , volume 19. MIT Press, 2006.

Lars Magnus Hvattum and Halvard Arntzen. Using elo ratings for match result prediction in association football. International Journal of Forecasting , 26(3):460-470, 2010.

Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models, 2023.

Rudolf Kadlec, Ondrej Bajgar, and Jan Kleindienst. Knowledge base completion: Baselines strike back. In Proceedings of the 2nd Workshop on Representation Learning for NLP , pages 69-74, Vancouver, Canada, August 2017. Association for Computational Linguistics.

Wei-Yin Ko, Daniel D'souza, Karina Nguyen, Randall Balestriero, and Sara Hooker. Fairensemble: When fairness naturally emerges from deep ensembling, 2023.

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations - democratizing large language model alignment, 2023.

Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Sebastian Ruder, Dani Yogatama, Kris Cao, Tomas Kocisky, Susannah Young, and Phil Blunsom. Pitfalls of static language modelling, 2021.

Christoph Leitner, Achim Zeileis, and Kurt Hornik. Forecasting sports tournaments by ratings of (prob)abilities: A comparison for the euro 2008. International Journal of Forecasting , 26(3):471-481, 2010. Sports Forecasting.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models, 2022.

Yen-Ting Lin and Yun-Nung Chen. LLM-Eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models, 2023.

Wei lin Chiang, Lianmin Zheng, Lisa Dunlap, Joseph E. Gonzalez, Ion Stoica, Paul Mooney, Sohier Dane, Addison Howard, and Nate Keating. Lmsys - chatbot arena human preference predictions. https://kaggle.com/competitions/lmsys-chatbot-arena , 2024. Kaggle.

Liquipedia. Elo rating - liquipedia starcraft brood war wiki.

Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study, 2018.

Pranava Madhyastha and Rishabh Jain. On model stability as a function of random seed. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) , pages 929-939, Hong Kong, China, November 2019. Association for Computational Linguistics.

Gábor Melis, Chris Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. ArXiv , abs/1707.05589, 2018.

Tom Minka, Ryan Cleven, and Yordan Zaykov. Trueskill 2: An improved bayesian skill rating system. Technical Report MSR-TR-2018-8, Microsoft, March 2018.

Prabhat Nagarajan, Garrett Warnell, and Peter Stone. The impact of nondeterminism on reproducibility in deep reinforcement learning. In 2nd Reproducibility in Machine Learning Workshop at ICML , July 2018.

Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha SohlDickstein. Sensitivity and generalization in neural networks: an empirical study. In International Conference on Learning Representations , 2018.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis,

Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.

Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. On the challenges of using black-box apis for toxicity evaluation in research, 2023.

April M. Reid. Elo rating system for video games explained.

Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the effects of data parallelism on neural network training, 2019.

Gil I. Shamir, Dong Lin, and Lorenzo Coviello. Smooth activations and reproducibility in deep networks, 2020.

Luísa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. Llm see, llm do: Guiding data generation to target non-differentiable objectives, 2024.

Robert R. Snapp and Gil I. Shamir. Synthesizing irreproducibility in deep networks. CoRR , abs/2102.10696, 2021.

Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. We need to talk about random splits. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 1823-1832, Online, April 2021. Association for Computational Linguistics.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karaka¸ s, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron

Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Sw˛ edrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop

Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research , 2023.

Cecilia Summers and Michael J. Dinneen. Nondeterminism and instability in neural network optimization. In International Conference on Machine Learning , 2021.

Nelson Vadori and Rahul Savani. Ordinal potential-based player rating, 2023.

Ben P. Wise. Elo ratings for large tournaments of software agents in asymmetric games. ArXiv , abs/2105.00839, 2021.

Yuxiang Wu, Zhengyao Jiang, Akbir Khan, Yao Fu, Laura Ruis, Edward Grefenstette, and Tim Rocktäschel. Chatarena: Multi-agent language game environments for large language models. https://github.com/chatarena/chatarena , 2023.

Yining Ye, Xin Cong, Yujia Qin, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Large language model as autonomous decision maker, 2023.

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback, 2023.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.

Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. ArXiv , abs/2106.11872, 2021.

A Extension to Multiple Outcomes

For scenarios where outcomes can extend beyond wins and losses, such as a tie option, one could make use of the multinomial distribution for the synthetic data generation process. For the three outcomes; win, loss, and tie, one sample according to the distribution:

B Impact of Ordering on Elo Ratings: Skewed Win Rates

We summarize our findings on the impact of match sequences on Elo ratings for winning probabilities Prob ( A beats B ) ≥ 0 . 65 .

Model A

Model B

Prob(A beats B)=0.9

Prob(A beats B)=0.8

Prob(A beats B)= 0.7

Prob(A beats B)=0.65

0.8

Win rates

0.6

0.4

0.2

0.0

Nperms = 1

Nperms = 1

Nperms = 1

Nperms = 1

1600

1500

mMm

1300

1200

Nperms = 100

Nperms = 100

Nperms = 100

Nperms = 100

1600

1500

Elo

1300

1200

100

200

300

400

500

100

200

3004005000100

300400

Number ofplayed games

C Chatbot Arena Human Preference Data Preparation

For the experimental validation of the Elo rating system using real-world data, we utilize the LMSYS dataset from [34]. We first vizualize the first 100 unique paired comparisons sorted in descending order by the number of recorded evaluations. The distribution of tie vs. non-tie outcomes is shown in Figure 7. To refine the dataset for our analysis, we exclude tie results, focusing exclusively on win-loss dynamics. The remaining dataset is further filtered to identify pairs with at least 300 non-tie comparisons. This threshold of 300 allows us to encompass a broad spectrum of comparison scenarios, ranging from skewed win rates (ModelA ≫ ModelB) to closely matched (ModelA ≃ ModelB). These selected paired comparisons are depicted in Figure 8.

(claude-2.1,gpt-3.5-turbo-0613)

(claude-1, gpt-4-1106-preview)

Outcome

(gpt-4-1106-preview,openchat-3.5)

(gpt-3.5-turbo-0613,yi-34b-chat)

Non-Tie

(vicuna-33b,yi-34b-chat)

(gemini-pro,mixtral-8x7b-instruct-v0.1)

(gpt-3.5-turbo-0613,palm-2)

Tie

(claude-instant-1, wizardlm-70b)

(claude-instant-1,yi-34b-chat)

(gpt-4-0613,vicuna-33b)

(gpt-3.5-turbo-1106,pplx-7b-online)

(claude-instant-1,vicuna-7b)

(llama-2-13b-chat, palm-2)

(claude-instant-1, tulu-2-dpo-70b)

(gpt-4-0125-preview,mistral-medium)

(llama-2-70b-chat,pplx-7b-online)

(oasst-pythia-12b,vicuna-13b)

(llama-2-70b-chat,openchat-3.5)

(gemini-pro,gpt-3.5-turbo-0613)

(llama-2-70b-chat,pplx-70b-online)

(llama-2-7b-chat,zephyr-7b-beta)

(gpt-3.5-turbo-0613,liama-2-70b-chat)-

(gpt-4-0613,mistral-medium)

(claude-instant-1,ilama-2-70b-chat)

(gpt-3.5-turbo-0613,wizardm-70b)

(llama-2-7b-chat, mistral-7b-instruct)

(mixtral-8x7b-instruct-v0.1, solar-10.7b-instruct-v1.0)

(claude-1,claude-2.0)

(gpt-3.5-turbo-0613,openchat-3.5)

(gpt-3.5-turbo-1106,pplx-70b-online)

(llama-2-70b-chat,vicuna-13b)

(gpt-3.5-turbo-0613,pplx-70b-online)

B Match

(gpt-3.5-turbo-0613,zephyr-7b-beta)

(llama-2-13b-chat,mistral-7b-instruct)

(gpt-3.5-turbo-0613,gpt-3.5-turbo-1106)

(koala-13b,vicuna-13b)

(gpt-3.5-turbo-0613,gpt-4-0314)

(claude-1,gpt-4-0613)

(claude-1,gpt-4-0314)

(mistral-7b-instruct,zephyr-7b-beta)

A

(gemini-pro-dev-api,gpt-3.5-turbo-0613)

(llama-2-13b-chat,zephyr-7b-beta)

(claude-2.1,claude-instant-1)

(claude-2.1,mixtral-8x7b-instruct-v0.1)

(gemini-pro-dev-api,gpt-4-0613)

(claude-instant-1,vicuna-33b)

(claude-2.0,gpt-4-1106-preview)

(gemini-pro,gpt-4-1106-preview)

(llama-2-70b-chat,mixtral-8x7b-instruct-v0.1)

(claude-instant-1,gpt-3.5-turbo-0613)

(claude-2.0,gpt-4-0613)

(gemini-pro-dev-api,gpt-4-1106-preview)

(gpt-3.5-turbo-0613,mixtral-8x7b-instruct-v0.1)

(gpt-3.5-turbo-1106,mixtral-8x7b-instruct-v0.1)

(llama-2-13b-chat,vicuna-13b)

(mistral-medium,mixtral-8x7b-instruct-v0.1)

(gpt-3.5-turbo-1106,mistral-medium)

(gpt-3.5-turbo-0613,gpt-4-0613)

(claude-1,gpt-3.5-turbo-0613)

(gpt-3.5-turbo-0613,mistral-medium)

(gpt-4-1106-preview,mixtral-8x7b-instruct-v0.1)

(llama-2-70b-chat,vicuna-33b)

(claude-instant-1,gpt-3.5-turbo-1106)

(gpt-4-1106-preview.mistral-medium)

(claude-2.1,gpt-4-0314)

(gpt-3.5-turbo-1106,gpt-4-1106-preview)

(claude-1,claude-2.1)

(gpt-3.5-turbo-0613,gpt-4-1106-preview)

(claude-2.1,gpt-4-0613)

(gpt-4-0613,gpt-4-1106-preview)

(claude-2.1,gpt-4-1106-preview)

800

00

Counts

(gpt-4-0314, gpt-4-0613)

(claude-1, claude-2.1)

(mistral-medium, mixtral-8x7b-instruct-v0.1)

(gpt-3.5-turbo-0613,mistral-medium)

(gpt-4-0314, gpt-4-1106-preview)

(llama-2-13b-chat, vicuna-13b)

(gpt-4-1106-preview,mistral-medium)

(llama-2-70b-chat, vicuna-33b)

(gpt-3.5-turbo-0613,mixtral-8x7b-instruct-v0.1)-

(gpt-3.5-turbo-0613,vicuna-33b)-

A vs B Match

(gpt-3.5-turbo-0613, gpt-4-0613)

(claude-instant-1, gpt-3.5-turbo-1106)

(gpt-3.5-turbo-1106, mistral-medium)

(gpt-4-0613,gpt-4-1106-preview)

(gpt-3.5-turbo-1106,mixtral-8x7b-instruct-v0.1)-

(gpt-4-1106-preview, mixtral-8x7b-instruct-v0.1)-

(claude-2.1, gpt-4-0314)

(gpt-3.5-turbo-0613, gpt-4-1106-preview)

(claude-2.1, gpt-4-0613)

(gpt-3.5-turbo-1106, gpt-4-1106-preview)

Outcome

(claude-1, gpt-3.5-turbo-0613)

Non-Tie

Tie

(claude-2.1, gpt-4-1106-preview)-

0.0

0.2

0

0.8

1.0

Normalized Rate

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately outline the core investigation into the Elo rating system's reliability and transitivity properties forLLMs, aligning with the empirical results presented, which validate and support these claims.

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This study probes the Elo rating system, highlighting its vulnerabilities without considering the impact of "tie" outcomes on Elo scores convergence. It shows its limitations for the most simplistic setting of pairwise comparison A vs B and proposes for future work an extension of tie outcomes to better model real-world scenarios.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our study does not derive theoretical results. Our research primarily employs stress testing on synthetic data and subsequent validation through real-world experiments to evaluate the robustness of Elo scores.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We detail the process of generating synthetic human feedback data and refer to a similar experimental setup used to collect real-world human feedback. Although results based on human judgment are inherently difficult to replicate exactly, the main focus of our study is on the reliability and robustness of the Elo rating system. This system requires only a sequence of outcomes (wins, losses, and ties, represented as 1, 0, or 0.5, respectively) to compute scores, facilitating the reproduction of our analysis based on these simplified inputs.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide thorough details on generating synthetic human feedback data, a key component of our experiments. We also outline our method for using LMSYS human preference data, which is publicly accessible. We also outline the steps for computing Elo scores based on existing literature. These comprehensive instructions should enable other researchers to effectively replicate our experiments and verify our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In our robustness study of the Elo rating system, we provide detailed descriptions of our experimental setup, including how synthetic data is generated. We explore and visualize the effects of a wide range of hyperparameters, such as the number of permutations ( N perms ) and the K-factor, to assess their impact on Elo score stability. We compare language models in the 7B to 12B parameters range, selected due to hardware computational limitations.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our paper includes error bars to represent the uncertainty in experimental outcomes, specifically using standard errors of the mean (SEM) across different permutations and hyperparameter settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We generated completions in batches sized between 8 and 50, depending on the size of each model evaluated, using an Nvidia A100 GPU with 40GB memory for efficient computation. Inference was performed in a Bfloat16 setting to reduce memory usage, as deatailed in referenced work.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research adheres to the NeurIPS Code of Ethics in all respects. We ensured fair compensation for human evaluators involved in the model pairwise comparisons. The pool of evaluation prompts includes open-source datasets (SODA and the Public Pool of Prompts (P3)), and were selected to avoid any harmful content.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our paper discusses several positive societal impacts, including the enhancement of AI system reliability and trustworthiness through more robust evaluation methods for large language models (LLMs). It highlights how these improved methods contribute to the development of safer and more accurate language models. We also address potential negative impacts by cautioning against the misinterpretation and over-reliance on Elo ratings, which might lead to suboptimal performance assessments. This underscores the need for exhaustive and fair evaluation mechanisms in AI development.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any models or datasets, so there are no associated risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use open-source datasets and properly credit their creators by citing the original papers. The specific versions and licenses of these datasets are acknowledged, ensuring compliance with their terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: Details regarding the instructions for human feedback experiments and evaluation format are based on established methodologies in the field. As for the compensation, while we cannot provide specific details, we can guarantee that fair compensation practices were followed.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [No]

Justification: No potential risk within our experimental setup.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper conducts an empirical study on the reliability and transitivity of the Elo rating system when evaluating Large Language Models. By conducting experiments on both simulated data and real-world scenarios, this paper suggests that certain parameter setting guidelines should be followed to ensure the stability of evaluation results when applying the Elo rating system to LLM assessments. The paper provides insights and guidance for future research and applications of the Elo rating system in various LLM evaluation contexts.

STRENGTHS:
- This paper provides a detailed experimental analysis of the practicality of the currently common Elo rating system used for evaluating the performance of Large Language Models (LLMs).

- Through appropriate experimental setups, this paper explores the impact of various factors on the Elo rating system results, including the number of sequence arrangements, the number of games, the K value, and win rates.

- By combining results from simulated experiments and real-world scenarios, this paper demonstrates good reliability in its findings.

WEAKNESSES:
- This paper explores the experimental results with up to three players. It would be more meaningful if the conclusions could be further generalized to settings with more players.

- In reality, $N_{perms}$ can be set very large without obvious computational cost to stable the evaluation results. Therefore, in multiple experiments, I would prefer to see results with settings like $N_{perms}$ = 10k, similar to those in Figure 2.

QUESTIONS:
My main concerns are listed in Weaknesses

LIMITATIONS:
I think that this paper has no significant limitations apart from what is mentioned by the authors in Section 8.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUpdated after response: I think this paper could provide some valuable contributions to the community, but has real presentational issues. The authors fixed some of them that had an easy fix  (which I pointed out) during the rebuttal. However, I believe the presentational standards (and therefore the understandabiluty of the contribution) are not up to par for this conference, and I thus stand by my borderline reject recommendation. \n\nThis paper studies the Elo rating system, recently popular in the context of ranking LLMs. Specifically they study two axioms that evaluation methods should adhere to: reliability and transitivity. They find that Elo ratings for LLMs are highly sensitive to the order of comparisons and the choice of hyperparameters, and that desirable properties such as transitivity are not always guaranteed.\n\nIn their first experiment, the authors focus on order sensitivity. They conduct a controlled experiment with synthetic human feedback, in which they compare various model pairs with different probabilities of winning a pairwise comparison.\n\nThey show that, with only order of comparisans (which, if I understand correctly, would be the actual scenario in which we deploy this?), if model A has a probability 40<P<60 to win  a pairwise comparison with model B, even 500 games are not enough to converge to the conclusion that model A is better than model B. With 100 orders, the difference is clear after few games, though (semi) convergence happens only after around 80.  For smaller differences (51%), even 100 orders do not reliably show the difference between models. For differences of P(a_win > 60) Elo results are stable (?) (I can't fully match this statement in the text with the figures).\n\nNext, the authors investigate the impact of K-factor (which, I learned from Wikipedia, is the maximum number of points a ranking can be updated after a match). They show that higher K-factors eacerbate instability for single orderings for models that have similar performances, but that faster convergence is observed for higher K-factors when models diverge more.\n\nThen, the authors investigate transitivity, using a set of three models with different respective win probabilities. They find that when there is a clear disparity between models, Elo ratings accurately reflect the expected hierarchy, but in more complex cases the rankings become less stable, challenging the assumption of transitivity. K-factor and number of orderings play a key roal in the stability of the rankings.\n\nThe authors confirm several of their findings using real human feedback data. And wrap-up their paper with a list of recommendations for using Elo ratings.\n\nSTRENGTHS:\nThe topic studied in this paper (Elo ratings for LLMs) is important, and the experiments are extensive. The paper provides several interesting lessons for using Elo ratings for LLMs that I believe can (positively) influence how we compare models in rankings based on pair-wise comparisons.\n\nWEAKNESSES:\nWhile I think this paper could potentially be very impactful and has many interesting results, in its current state it does not seem ready for publication. It took me excessively long to review, because the presentation is rather messy, and not clear at several points.  Some of those things are easily fixable, e.g.:\n- The Elo rating system is briefly explained in an appendix, but given the importance of the concept it should be in the main text (if space is needed, I think equation 5 describing transitivity can safely be moved to the appendix).\n- Some explanation of the K-factor should also be included. It is sort of described, but not really explained or defined, not even in the appendix as far as I can see).\n- Maybe I missed it, but I don't think it is listed how Elo ratings are most commonly used for LLMs, is it with only a single ordering?\n- Table 1 is somewhat difficult to read, why not make a plot similar to Figure 1 but with multiple lines for the different models?\n- In the introduction -- impliations of our work -- it is listed that the work is particularly important in situations where model performances are closely matched, which is a common occurrence in many real-world scenarios, whereas in 4.2 it is listed that the result is relevant because skewed win-rates are common in real-world scenarios (or maybe that is not what is meant, if not I am not sure why it is mentioned that there are skewed win-rates in real world scenarios)\n\nMore problematic are the unclarities concerning the results, for example:\n- It is not clear with what K-factor the main experiment (4.1) is conducted (or at least I couldn't find it)\n- in 4.1 it is stated that for P(A beats B) >= 0.6 Elo results are stable; I can't match this result with Figure 1 (middle column), where there seems to be quite some variation for P(a beats B) = 0.55 as well as P(a beats B) = 0.6\n- The idea of comparing a single sequence with 100 different orderings is generally strong, but doing only one single sequence and one series of permutiations is not very informative in this case: what we actually want to know is how often a single sequence or an average over 100 sequences would give us the right results. Perhaps this single sequence was a very unlucky one among many that give us the correct result, or the 100 permutations were instead 100 lucky permutations. To really make the point, we'd need the variation between single sequences, and multiple selections of 100 permutations. (similar for Figure 4)\n- In 4.2 it is listed that faster convergence is observed for higher K-factors, but I don't understand the evidence for that. I don't think this is visible from Figure 3, and I couldn't find an appendix plot either.\n\nQUESTIONS:\n- If I understand correctly, the number of permutations is the number of game orders, why not use a word that matches that closely, e.g. 'game orders' or 'orderings'?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEvaluation plays an important role in LLM research. Previous works have assessed the performance using the Elo rating system, which is designed for ranking players in games. This paper shows that Elo rating does not always satisfy two critical axioms, reliability and transitivity. So the rankings of the models are not accurate. Transitivity is whether a consistently higher-rated model outperforms those with lower ratings in all scenarios. Reliability has two aspects, sensitivity to ordering and sensitivity to hyperparameters, how the Elo scores are affected by the order in which models are compared and the sensitivity to hyperparameter adjustments, particularly the K-factor.\n\nThe authors first validate the properties of Elo using synthetic data generation via Bernoulli processes. Then extend the validation to include real-world human feedback. The findings are that the stability of Elo ratings is influenced by disparities in win rates, analogous to win probabilities in synthetic data, and by the choice of hyperparameters K-factor and Nperms. Finally the authors provide empirical guidelines for robust Elo-based Evaluation.\n\nSTRENGTHS:\n1. The evaluation of LLMs plays an important role in LLM research. The reliability of the Elo rating is an interesting and meaningful research topic.\n\n2. The authors highlight two Axioms for Elo-based evaluation, which is reasonable and useful.\n\n3. This paper verifies the findings from both synthetic human feedback and real-world human feedback.\n\n4. This paper provides several empirical guidelines for robust elo-based evaluation.\n\nWEAKNESSES:\n1. The writing is not very friendly to the reader unfamiliar with the Elo rating.\n\n2. This paper only conducts experiments on three LLMs.\n\nQUESTIONS:\n1. In line 93 you mention the K factor, but it is hard to understand for the reader unfamiliar with the Elo rating.\n\n2. How can the Elo rating be generalized to multiple-task evaluation?\n\n3. Why not conduct experiments on more widely used API-based LLMs, such as ChatGPT, Gemini, and Claude-3?\n\nLIMITATIONS:\nThis paper has no potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines  how well the Elo rating system works for evaluating LLMs based on human feedback. The authors first check how stable and reliable Elo ratings are under different settings and point out key factors that affect these ratings. They study how changing the order of matches and adjusting settings like the K-factor can influence the consistency of Elo scores. Using both synthetic and real-world data, the paper shows that Elo ratings become more reliable with more match permutations and the right K-factor adjustments.\n\nThe paper also discusses the transitivity of Elo scores, showing that this property doesn't always hold for models with similar performance levels. The authors suggest practical tips for more reliable Elo-based evaluations, like running many permutations for stable scores and adjusting the K-factor based on how different the models' performances are. They highlight the limitations of current Elo-based evaluation methods and propose future research directions, such as considering ties and using more complex data simulations to better mimic real human feedback scenarios.\n\nSTRENGTHS:\n1. Despite Elo rating being a popular method for evaluating LLMs, there is a lack of analysis on its robustness. This paper provides a comprehensive investigation, helping us better understand and implement Elo rating for LLMs.\n2. The authors explore the reliability and transitivity of Elo rating under different configurations. Their definitions and formulations are clear and reasonable.\n3. Experiments are conducted on both synthetic data and real human feedback, offering valuable insights for future Elo rating applications.\n\nWEAKNESSES:\n**Major Issue:**\n\nMy biggest concern is the relatively small amount of real data used in the experiments. Only 500 prompts and three model pairs are included, with just one pair having similar performance (Dolly-v2-7b vs. Dolly-v2-12b). Given that Chatbot Arena currently includes dozens of LLMs and has released a large amount of preference data, the volume of real data in this study seems insufficient.\n\n**Minor Issues:**\n1. As the authors mentioned, not considering ties appears to simplify the widely used \"win/tie/lose\" practice.\n2. The empirical guidelines provided may have limited applicability since most papers on automatic evaluation of LLMs report win rates rather than Elo ratings. Also, running more than 100 permutations seems costly and lacks practicality.\n\nQUESTIONS:\n1. If I want to automatically evaluate $k$ models to get their final Elo ratings, do you think the size of $k$ will affect the robustness of the Elo rating?\n2. Why does Dolly-v2-7b have a slightly higher win rate than Dolly-v2-12b in Table 2?\n\nLIMITATIONS:\nThe paper discusses its limitations in Section 8. My personal view on the limitations can be found in the weaknesses mentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}