A Statistical Framework for Ranking LLM-based Chatbots
Authors: Siavash Ameli
Siyuan Zhuang
Ion Stoica
Michael W. Mahoney
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models (LLMs)
Paired Comparison
Statistical Ranking
Human Preferences
Chatbot Arena
Logistic Regression
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce a rigorous statistical framework for ranking large language models (LLMs) using crowdsourced comparisons, improving accuracy for ties, wins, and losses beyond current methods like Elo.

Abstract: Large language models (LLMs) have transformed natural language processing, with frameworks like Chatbot Arena providing pioneering platforms for evaluating these models. By facilitating millions of pairwise comparisons based on human judgments, Chatbot Arena has become a cornerstone in LLM evaluation, offering rich datasets for ranking models in open-ended conversational tasks. Building upon this foundation, we propose a statistical framework that incorporates key advancements to address specific challenges in pairwise comparison analysis. First, we introduce a factored tie model that enhances the ability to handle ties—an integral aspect of human-judged comparisons—significantly improving the model's fit to observed data. Second, we extend the framework to model covariance between competitors, enabling deeper insights into performance relationships and facilitating intuitive groupings into performance tiers. Third, we resolve optimization challenges arising from parameter non-uniqueness by introducing novel constraints, ensuring stable and interpretable parameter estimation. Through rigorous evaluation and extensive experimentation, our framework demonstrates substantial improvements over existing methods in modeling pairwise comparison data. To support reproducibility and practical adoption, we release leaderbot, an open-source Python package implementing our models and analyses.

Published as a conference paper at ICLR 2025
A STATISTICAL FRAMEWORK FOR RANKING LLM- BASED CHATBOTS
Siavash Ameli
Siyuan Zhuang
ICSI and Department of Statistics University of California, Berkeley samel Bameli@berse @eeereele........... edu edu
Department Computer Science University of California, Berkeley
Ion Stoica
Michael W. Mahoney
Department of Computer Science University of California, Berkeley istoicalcs berkeley. edu
ICSI, LBNL, and Department of Statistics University of California, Berkeley
mmahoney@stat..eeteee edu
ABSTRACT
Large language models (LLMs) have transformed natural language processing, with frameworks like Chatbot Arena providing pioneering platforms for evaluating these models. By facilitating millions of pairwise comparisons based on human judgments, Chathot Arena has become cornerstone in LLM evaluation, offering rich datasets for ranking models in open-ended conversational tasks. Building upon this foundation, we propose statistical framework that incorporates key ad- vancements to address specific challenges in pairwise comparison analysis. First, we introduce factored tie model that enhances the ability handle ties-an in- tegral aspect of human-judged comparisons Siigiiniiiinn improving the model's s fit to observed data. Second, we extend the framework to model covariance be- tween competitors, enabling deeper insights into performance relationships and facilitating intuitive groupings into performance tiers. Third, we resolve optimiza- tion challenges arising from parameter non-umin uonuiiqueness by introducing novel con- straints, ensuring stable and interpretable parameter estimation. Through rigorous evaluation and extensive experimentation, our framework demonstrates substan- tial improvements over existing methods in modeling pairwise comparison data. To support reproducibility and practical adoption, release leaderber rbot, open-source Python package implementing our models and analyses.
INTRODUCTION
The rapid advancement of large language models (LLMs) has transformed natural language process- ing, enabling breakthroughs across diverse tasks. As these models evolve, the need for effective evaluation methods becomes crucial for fostering innovation and ensuring that LLMs align with hu- man preferences. Traditional benchmarks, such as MMLU (Hendrycks et al., 2021) and HvmanEval (Chen al., 2021), play an important role in assessing specific capabilities of LLMs. However, they often fall short in capturing the nuanced, real-world interactions characteristic of open-ended conversational tasks, particularly those seen in chatbot applications.
To address this gap, crowdsourced evaluation platforms have emerged, with Chatbot Arena (Chiang al., 2024: Zheng et al., 2023) standing out as a pioneering framework. By facilitating millions of pairwise comparisons between LLMs based on human judgments, Chatbot Arena has become one of the largest and most credible datasets (Zheng et al., 2024) for chatbot evaluation. Its design more closely reflects the open-ended nature of chatbot usage, providing unparalleled diversity and robustness in assessing model performance. its first year, the platform amassed over two million than 150 state-of-the-art models, gaining adoption by leading institutions such as OpenAI, Google, and Hugging Face. This unparalleled scale and impact have solidified Chatbot Arena cornerstone in the L.M evaluation ecosystem.
votes across

---

Published as a conference paper at ICLR 2025
The ranking methodology employed in Chatbot Arena relies the Elo rating system (Zermelo, 1929; Bradley & Terry, 1952), which is well-suited for competitive settings with clear win-loss outcomes. However, the Elo system does not account for ties notable portion of human-judged comparisons-oov or for modeling deeper relationships hetween competitors, such as correlations in performance. Addressing these aspects presents an opportunity build upon the success of Chatbot Arena, enhancing its analytical capabilities while retaining its foundational strengths.
In this paper, we propose statistical framework that extends the foundational Elo-based approach employed in Chatbot Arena. Our contributions include:
1. Incorporating ties: Ties, where two competitors are judged to perform equally, are a key feature of pairwise comparisons. To model ties, we integrated well-established methods by Rao & Kupper (1967) and Davidson (1970), which define tie probabilities based on axiomatic assumptions. These methods have been applied in various contexts, from sports analytics to marketing and beyond (see Appendix A). While these frameworks offer a solid foundation for paired comparisons, applying them directly to the Chatbot Arena dataset, with its extensive pairwise comparisons, highlighted the need for further adaptation: these models yielded errors exceeding 10% when applied to tie outcomes.
To address these challenges, we propose novel factored tie model, which generalizes these frameworks by uncovering latent structures in tie patterns across pairs of competitors. This factor analysis substantially improves model fit, reducing errors in fitting tie data by two orders of magnitude. Notably, this enhancement also extends to win/loss predictions, achieving comparable improvements. Details of tie modeling, including a background and our generalizations, are provided in Section 2.
Incorporating covariance: We extend paired comparison models by introducing Thurstonian rep- resentations to capture covariance structures between competitors (Section 2.4), enabling deeper exploration of relationships beyond rankings, such as correlations in performance. Building on our theoretical analysis of covariance's s structural non-uniqueness and equivalence classes (Ap- pendices B and C), we show that derived metrics from covariance, such as dissimilarity metrics, are unique and provide interpretable insights. These metries enable visualization techniques to un- cover latent patterns, such as performance trends, and clustering techniques to group competitors into performance tiers based on relative strengths.
3. Addressing optimization challenges through constraints: Paired comparison models often ex- hibit parameter non-uniqueness due to structural symmetries in the model, resulting in equivalent parameter configurations that leave the likelihood invariant. These symmetries lead to valid but in- distinguishable solutions, compromising convergence stability during likelihood optimization. To resolve these issues, we introduce novel constraints that regularize the parameter space, ensuring stable and interpretable parameter estimation (Section 2.5 and Appendix C.4).
In addition to developing our framework, we conducted comprehensive evaluations and analyses to validate its performance and interpret its results. Empirical evaluations (Section 3) demonstrate the model's 's fit to observed data, supported by extensive experimentation (Appendix D) on model selection, goodness-of of-tit and prediction metrics. Detailed inference analyses (Section 4) further explore competitor relationships, offering visual insights through clustering and ranking similarity analyses (Appendices C and E). We also examine the relationship between LLM characteristics and their performance scores in Appendix F. Furthermore, Appendix A highlights the framework's potential applicability beyond ranking and chatbot evaluations other areas of machine learning,
To support reproducibility and broader adoption, we provide leaderbot, an open-source Python package implementing our statistical framework with tools for data processing, model fitting, and visualization. This ensures that all results in this paper are fully reproducible (Appendix G).
2
STATISTICAL MODEL
2.1 PROBLEM STATEMENT
Consider a paired-comparison experiment involving > competitors (here, LLM chatbots), indexed by the set V = (1....,m) Let E S [(i.j} j E V} denote the set of unordered pairs of competitors that have been compared in the experiment. We assume the graph @VV, E) is connected.
2

---

Published as a conference paper at ICLR 2025
We define the m x matrix W W= [wsg] where Wij represents the frequency with which competitor i wins against competitor j. and W) represents the frequency with which loses to j. Similarly, define the symmetrie x matrix T [Byll where t denotes the frequency of ties between competitors and j. with i;; = =(j- We set Waj = Wgg = lij = O whenever {i.j} & E to reflect the absence of comparisons between competitors and j. The total number of comparisons between competitors i and j is denoted by Mij. where Big = wij + y + tay. The triple (g, W, T) constitutes the input data for our problem.
Our objective is to rank the competitors based on their performance in the overall comparisons. To formalize this in a probabilistic framework, we define P(a > {i,j]) as the probability that competitor i wins against competitor j. and P(i ~ j {i,j}) as the probability that and j tic. For notational simplicity, we often denote these probabilities by Po--s and Panj, respectively.
A broad class of parametric models (which we will discuss in detail later) assumes the existence of a score array æ = (#i,.... Fmm) E R" which defines the ranking. Specifically, the ranking inferred by bijection from V to itself that orders the scores 24, such that æ; 2j implies is ranked higher than j. denoted by the binary relation i
The score vector æ forms part of the model's parameters, denoted by 0, which also includes other parameters governing the probability of each outcome. A common approach for estimating these parameters is the maximum likelihood method. The likelihood function L(O G, W, T) is defined as the product of multinomial distributions for each compared pair {i,}} E E, given by P P
May! Wiyhwy,ltss!
II (NJE
(0)P") 3 (0)P2) (0)P22(
(I)
The parameter estimate 0" is then obtained by maximizing the log-likelihood function E(O) = log (()), i.e., 0" argmax, (()).
2.2 PROBABILISTIC MODELS
parametric model for the above probabilitics must satisfy two fundamental axioms. First, by the law of total probability, we have Parj + Pi-j Pi.,y 1. Second, the model should respect the concept of transitivity in ranking, though in a probabilistic setting. Rather than standard transitivity, where > j and j > k imply > we adopt the principle of stochastic transitivity (see, e.g., Shah et al. (2017y; Shah & Wainwright (2018)).
In particular, we are interested in strong stochastic transitivity, which states that if Pa-; 2 and Pj+ }- A key sub-class of strong stochastic transitivity, and the
Pj=k 2 3. then Pi>k 2 max[Pi-j. (isyyPjjkke focus of this work, is linear stochastic transitivity. This property is characterized by the existence of an increasing comparison function F Porj- = ((x,))
[0, 1] and a merit function ( R + R, such that
In the following subsections, we describe several common models of paired comparison that satisfy these properties.
2.2.1 BRADLEY-TERRY MODEL.
One of the most widely used models for paired comparison was first introduced by Zermelo (1929) and later rediscovered by Bradley & Terry (1952), leading to the model being named after them. The Bradley-Terry model forms the basis of the well-known Elo rating system, which extensively used by the World Chess Federation. In this model, the probabilities of win and loss are given by
P(i > j (i.j)) ==
Ti
and P(i < ((li((i))))
T
n; + nT
क, +
where म; = ". This model assumes that x --a I. follows a logistic distribution, as shown by
P(z, - 0) 1te -w
(3)
Variants of this model, such as the one proposed by Glenn & David (1960), assume a standard normal distribution instead of the logistic. However, the logistic distribution is typically preferred in paired comparison settings due to its heavier tails, which provide better for real-world data, and its computational advantages and tractability (Böckenholt, 2001).
3

---

Published as a conference paper at ICLR 2025
2.2.2 MODELS WITH TIES
The Bradley-Terry (BT) model does not account for ties (i.e., Ping = 0), making it more suited for balanced paired comparisons, such as zero-sum sum games, However, in our application of ranking LLM chatbot agents, ties frequently occur, which poses challenges for applying the BT model directly.
Previous work, such as Chiang et al. (2024), addressed this by treating tie as halfway between a win and loss, modifying the outcome matrix as W < W + JT. While this approach provides a straightforward way to handle ties, other extensions of the BT model incorporate ties through axiomatic frameworks, which we explore in the following sections.
One such generalization is the model of Rao & Kupper (1967), which modifies the logistic distribu- tion to account for ties. The resulting probabilities for win, loss, and tie are given by
P(i > (1,5)) = P(x.- - J. > n) T; + VJJ
n
(4a)
P(i xj (i, = P(z. I3 < -n) =
(4b)
1/1,; TT
n. (v² P(i ~ j (i.j)) = P(1x, 2,1 <n) = (,, + VT,) (UTTT nT)
(4c)
where == e" 2 and 17 2 is a threshold parameter to be optimized. In this model, if the difference between competitors scores is less than the threshold 1j. the judge is unable to distinguish between competitors and j. resulting in a tie. Setting 7) = O reduces this model to the standard BT model.
Another extension of the BT model, introduced by Davidson (1970) and based on the axiom of choice of Luce (1959), models ties differently;
P(i jl (i.i)) =
T Ti + JTy + 3
(5a)
j T. + TT. + |
P(i < (i,j)) =
(5b)
V n
P(i P(i~il = +
(5c)
where y = e" and 17 E R is a threshold parameter for tie to be optimized. Here, setting 7) -00 reduces this model to the BT model. Note that in both the Rao-Kupper and Davidson models, the probability of tie increases as the difference in scores decreases.
2.3 A GENERALIZATION FOR MODELS WITH TIES
The original Rao- Kho-KK and Davidson models each employ a single parameter for ties, V. However, as our numerical results will demonstrate, one tie parameter is insufficient to capture the complexity of ties across all pairs of competitors. To address this, we propose generalization of these models by incorporating additional parameters, which, to our knowledge, a novel extension.
In this generalized model, the tie parameter v = c" is replaced with a pair-specific parameter My = =,",, where i, V. subject to the symmetry condition Pi = Uy.- This modification introduces [E] parameters, potentially leading to overfitting. To mitigate this, we propose reduced than treating all My independent parameters, we introduce the following factor model to construct H by
model. Let the symmetric m x m matrix H = [myy] represent the pairwise tie parameters. Rather
G T + DG JJJ
0 < Krin I m. Kiio 0.
H
(6)
where G
[gky] is an m x kesn matrix of the new parameters Dij- + - [o:y] is an m X hri constant matrix of rank kue m consisting of predefined basis column vectors that will be discussed below, and J is the m x m matrix of all ones. The rank of H is max(1. min min(m (2k m)) containing max(1 mkkj) parameters; and thus choosing knj allows us strike balance between the goodness
4

---

Published as a conference paper at ICLR 2025
of fit and the complexity of the model. Note that the case ktte 0 reverts to the original Rao- Rao-Kupper or Davidson models where single parameter Ti = 17 is used.
To further motivate the construction in (6), the pairwise tie parameters Tly (when Keje * 0) can be expressed as
Ti - 9: 0 + 9, Φ,:
where 9, (9:1 1.. Pikk..) and Φ; (6 . Pikk.e) ) the i-th rows of G and 4, respectively. This formulation emphasizes the additive nature of our proposed model, where contributions from competitors i and j are independently and symmetrically combined. While designing this generaliza- tion, we also considered adopting a multiplicative structure (c.g., H = AAT), ) as is commonly used in factor analysis. However, we found it unsuitable for this application: low contributions from one competitor's s tie threshold parameters would disproportionately suppress the pairwise tie parameter "ly. leading to an ineffective model for ties. By contrast, our additive design reflects the cumulative nature of ties, which we found aligns naturally with the structure of pairwise comparisons and is particularly effective for modeling shared thresholds.
To ensure 4 is of full rank, design & with orthogonal column vectors. This can be achieved, for instance, by orthogonalizing a randomly generated matrix using Gram-Schmidt orthogonaliza- tion. However, for reproducibility and to avoid using randomly generated matrices, recommend constructing + using discrete orthogonal polynomials with respect to a uniform weight over equally spaced points (Baik et al., 2007), which inherently orthogonal. Such matrices be generated, for instance, by discrete Legendre polynomials, the Hadamard transform (when v is power of 2), discrete Chebyshev polynomials (Corr al., 2000), the discrete cosine transform (DCT). For simplicity, we choose the discrete cosine transform of the fourth type (DCT-IV) basis, where the elements Pig are given by
01 =
cos m
π
(2i - (2i- i 1..,...,mm and j = 1, 1....,,,uu (8)
4m
2.4 INCORPORATING COVARIANCE USING THURSTONIAN REPRESENTATIONS
fundamental approach to modeling paired comparisons was introduced by Thurstone (1927) through the laws of comparative judgment, laying the foundation for psychometric choice mod- cling from statistical perspective. this section, we build on Thurstone's 's multivariate discriminal process to incorporate covariance structures into the paired comparison models discussed carlier, enriching the modeling framework with covariance as an additional parameter.
Thurstonian models assume that the score variables æ are stochastic processes defined by æ 14 + e, where μ E Rdh the deterministic component representing the mean of the process, and e is zero-mean random component with covariance Σ .o where J; == cov(c, (g) often referred to as the comparative dispersion. In this model, fa and serve as the parameters, with μ determining rankings by reflcting the expected performance of competitors.
We note that in this model, the difference between scores, which is central to the previously mentioned models, is also stochastic process: 2; I ,4 My + Eij. Here, Eiy has variance S = [syy] defined as $iy == var(e:)). ) and given by 8ij = σu + J5 - 20 This is commonly referred to as the discriminal dispersion (Heiser & de Leeuw, 1981).
The original Thurstonian model assumes that x follows joint normal distribution, meaning that I; 2y also has normal distribution. Let x, - Pij p4 My, and Viş == (æiy (24j-M))) SSj-- It can be shown that Pays, P(z,, > 0) (22g) (Madeu-Ollvaree 1999), where & is the cumulative standard normal distribution, and
M4
/8ij
Although the Thurstonian discriminal process is typically applied using normal distributions, we extend this process to the models previously introduced, including the Bradley-Terry model (which
'This follows from var(z; -2,) var(z.) var(z,) 200v(2,;21))
5

---

Published as a conference paper at ICLR 2025
uses the logistic distribution) and the Rao- Kup-er and Davidson models (which incorporate ties). To our knowledge, these extensions have not been explored in the literature.
To generalize the Bradley- Terry model with the discriminal process, assume Vij follows logistic distribution, yielding
P(i > j (i.))) 1te and P(i ((i 1
(10)
We can similarly extend the Rao-Kupper and Davidson models. The probabilities in (4) and (5) can be expressed in terms of the logit function log(T./my) Tij. which represents the quantile of the logistic distribution. To incorporate the Thurstonian model, we replace zij with Zig. Thus, the Rao- Kupper model becomes
> P(i>j((,j)) = 1 (19)-19))
(11a)
P(i ((.x)((ijj})
(11b)
+ e 113)-1111)*
2015
P(i ~ j| (i.j3)
(11c)
(1+e
(1+ (
where in the above we also used 7i to account for the generalized tie thresholds introduced in Section 2.3. Similarly, the Davidson model becomes
1
P(i - (1 = 1 +e tss +e -tag-
(12a)
1
P(i ((-s)) = 1 -30 +e 112
(12b)
P(i~jl = 1 E ((nyt - (133
(12c)
These models introduce an additional m(m + 1)/2 covariance parameters ozj, which can lead to overparameterization. To address this, Thurstone (1927) proposed various constraints on the covariance matrix, while Takane (1989) suggested a factor model for covariance structure. We adopt the factor model employed by Böckenholt (1993); Maydeu-Olin Olivares & Böckenholt (2005), where the covariance is constructed as
(13) where D [day] is positive-definit diagonal matrix with d, 0. and A [Nay] is X koow matrix of rank kovv I m, containing the factor parameters Xij- By selecting licuvs we can balance model fit with complexity.
Σ = D AAT,
The covariance matrix Σ is inherently non-unique, with multiple covariance matrices correspond- ing to the same variance matrix S. forming an equivalence class. Additional details on the non- uniqueness, equivalence classes, and identifiability of covariance are provided in Appendix C.I.
2.5 IMPOSING CONSTRAINTS TO RESOLVE SYMMETRIES
When estimating the parameters of the models (such 0 (æ, G) the generalized tie models, or 0 = (μ.. G. D, A) in their Thurstonian representation) through the maximum likelihood method described in (1), we encounter computational issues due the non-uniqueness of the solution. This arises from the fact that the likelihood function remains invariant under certain transformations of the parameters, known as symmetries. These symmetries can result poor optimization behavior, such as large small parameter values, causing instability in the estimation process. To address these issues, we propose constraints on the log-likelihood function to climinate the problematic symmetries. While one of these symmetries has been addressed in prior work, the other two have not, to the best of our knowledge.
The first symmetry arises from the model's 's invariance under the transformation 2; H I, where c E is constant. This invariance implies that the values of x, (or μ; in the Thurstonian repre- sentation) not identifiable-oni their differences are meaningful (see Appendix B for detailed

---

Published as a conference paper at ICLR 2025
discussion parameter identifiability). Consequently, the parameters can shift uniformly without affecting the model's predictions. To address this symmetry, common approaches include fixing one of the score parameters, as demonstrated by Chiang al. (2024), or imposing other constraints on the parameter values. For example, the original Bradley-Terry model (Bradley & Terry, 1952) enforces the constraint Di=1 F 1. In our work, we adopt similar approach by setting the mean of the scores: Σ I; O, which corresponds Σ M4 0 in the Thurstonian representation. This constraint effectively eliminates the shift invariance, ensuring stable optimization of the score parameters (or jt).
The second symmetry, which has not been addressed in previous studies, involves scaling both the score and covariance parameters. Specifically, the Thurstonian models remain invariant under the transformation (14, Viy) + (4p,, (pp,, o.y) for any positive constant i, because the ratio 2ij in (9) remains unchanged. This symmetry can lead to parameters collapsing to very small values, causing numerical instability or underflow during optimization. To resolve this, we introduce the following constraint
trace ΣS 11
(14)
where Σ = PEP, - 11 is the centering operator with I as the identity matrix and 1 (l....; 1)T is a column vector of ones. This constraint ensures proper scaling of the covariance parameters and avoids collapse during optimization. A detailed explanation of this constraint provided in Appendix C.4.
The third symmetry pertains to the factor model parameters Aig. Using the factor model for covariance in (13), we can express the elements of the matrix $
and Sy du + d,f || X, i
S/s
where A; (Aa ) the i-th row of the matrix A, and || |l2 denotes the Euclidean norm. This expression reveals an invariance under translation Aıj + Xıg + c, which has not been addressed in the literature. To eliminate this translation symmetry, we impose the constraint
| = =0
(15)
This constraint fixes the column- wise mean of to zero, preventing the factor parameters from arbitrarily shifting.
EMPIRICAL EVALUATION OF STATISTICAL MODELS
In this section, we evaluate the statistical models introduced earlier using the Chatbot Arena dataset. As of September 2024, the dataset comprises m = 129 competitors, with |E| 3455 unique pairs. The total number of comparisons across all pairs is ((.j)E Thi 1,47,, 996, distributed as follows: 43.3% wins, 36.2% losses, and 20.4% ties.
We analyzed 30 configurations of the Bradley-Terry, Rao-Kupper, and Davidson models, detailed in Table D. in Appendix D.I. These configurations include both the original forms of the models and various generalizations introduced in this work, each assigned unique ID corresponding rows in the table (e.g., Model 1, Model 2, etc.). which we reference throughout this and subsequent sections. For example, Model corresponds to the Bradley-Terry model with ties treated as half a win and half loss, following Chiang et al. (2024), while Model 4 represents the original Bradley-Terry model without ties. Similarly, Models 7 and 19 correspond to the original Rao-Kupper and Davidson models, respectively, with the remaining configurations representing our proposed generalizations.
their
Details of the experimental setup. along with results model selection, goodness of fit, and gen- eralization performance, are provided in Appendix D. Here, we focus on visual comparisons of pairwise win/loss and tie matrices to evaluate how well the models replicate observed outcomes across competitor pairs and to highlight their predictive capabilities.
3.1 EVALUATING THE PREDICTION OF WIN/LOSS AND TTE MATRICES
Figure visualizes a subset of the win/loss matrix W and tie matrix T for the top 25 competitors predicted by four models. The first row of the figure shows win probabilities, while the second row
7

---

Published as a conference paper at ICLR 2025
(
(
(
-0
Origiaal
Original
Originl
Figure 1: Comparison of pair-specific win (first row) and tie (second row) probabilities among 25 competitors between observed data (first column) and model predictions: our generalized Rao- Kupper with factored tie model (second column), original Rao-Kupper (RK) with ties (third column), Bradley-Terry with ties as half win/loss (fourth column, (Chiang et al., 2024)), and original Bradley- Terry without ties (fifth column). The ordinate and abscissa are shared across panels, shown only for the left-most and top-most panels. Each row shares the same color range and colorbar.
shows tie probabilities. The leftmost column contains the observed matrices, while the subsequent columns represent predictions from: the generalized Rao- Kao-Kupper model with factored ties (second column, corresponding to Model 18), the original Rao-Kupper (RK) model with a single tie parameter (third column, corresponding to Model 7), the Bradley-Terry model with ties treated as half win and half loss (Chiang et al., 2024) (fourth column, corresponding to Model 1), and the original Bradley- Terry model without ties (fifth column, corresponding Model 4). Note that the Bradley-Terry models do not account for ties, so their tie probability matrices are absent. For consistency, the 25 competitors are ordered identically across all panels based on the ranking from Model 18.
The generalized Rao-Kupper model with factored ties (second column) exhibits fine-grained accu- racy, closely matching the observed matrices for both win/loss and tie outcomes. In contrast, the original Rao- Kao-Kupper model (third column) predicts the win matrix reasonably well but shows re- duced accuracy for ties. The Bradley-Terry models (fourth and fifth columns, with ties treated half win/loss and without ties, respectively) differ significantly in their win/loss matrices, they do not account for ties, resulting in noticeable discrepancies compared to the observed data.
To complement the pairwise probability comparisons presented here, Appendix D.5 evaluates the models using marginal probabilities of win, loss, and tie aggregated across all competitors.
4 RANKING AND COMPARISON OF LLM-BASED CHATBOTS
In this section, demonstrate how the proposed models provide insights into the competitive perfor- mance of chatbots. Section 4.1 focuses on ranking analysis, while Section 4.2 explores relationships between chatbots, building the covariance structures introduced in Section 2.4.
4.1 RANKING
pairwise comparison methods, the score parameters (or μ the Thurstonian representation) rank competitors, offering straightforward interpretation of performance, with higher scores indicating stronger competitors. For example, Figure 1 Appendix E.1 shows the scores of the top 50 chatbots, ranked by Model 18 in Table D.1 (generalized Rao-Kupper model with k = 3 and kise = 20).
A key question is how different statistical models influence these rankings. To address this, analyzed rankings produced by 12 selected models from Table D.1. Figure E.2 in Appendix E.2 visualizes the bump chart of these rankings across models of varying complexity. Each column
8

---

Published as a conference paper at ICLR 2025
represents a model, ordered from simpler models on the right to complex the left, and each row tracks a chatbot's rank across models.
Notably, top-ranked chatbots exhibit stable rankings across all models, indicating reliability regard- less of model complexity. In contrast, lower-ranked chatbots display greater variation, reflecting their sensitivity model selection.
To systematically analyze these differences, we computed the Kendall rank correlation (r) between all 30 models in Table D.I, shown in Figure E.3 in Appendix E.3. Kendall's measures the agreement between two rankings by evaluating the relative ordering of pairs, making it particularly well-suited for ordinal data. This analysis revealed that rankings across models highly correlated, but notable differences emerge based on model parameterizations.
To better interpret these differences, we applied hierarchical clustering to the Kendall correlation matrix, as detailed in Appendix E.4. The resulting ordering of models, displayed in Figure E.3, is determined directly by the optimal ordering suggested by the clustering algorithm, reflecting the structure it uncovered.
Interestingly, the inclusion or exclusion of the Thurstonian covariance factor emerges the primary driver of these clusters: models without covariance (indicated by X) form a distinct cluster, separate from those with covariance. Furthermore, within the covariance group, models are divided into two subgroups based on the complexity of their covariance structures: Keov = O and keov = 3. This highlights the strong influence of covariance parameters on ranking similarities.
While covariance prominently drives ranking- parameters- earlier results (see Section 3 and Appendix D) showed that generalized tie modeling significantly enhances model fit and predictive accuracy. These findings highlight complementary strengths: tie modeling improves fit, while covariance modeling shapes ranking patterns and distin- guishes competitors based on shared performance characteristics.
than other model features such as tie
4.2 EXPLORING COMPETITOR CORRELATIONS
Incorporating covariance into Thurstonian models allows for exploring relationships between com- petitors beyond simple rankings. While rankings provide ordinal insights, covariance structures quantify uncertainties and correlations, uncovering patterns that rankings alone cannot capture.
As detailed in Appendix C.I, the covariance matrix Σ is not unique, but its associated matrix S [sy]] representing the variance of score differences, is unique and interpretable. This ma- trix serves as a dissimilarity measure, capturing the relative uncertainty in performance comparisons. By normalizing score differences with /8,j. construct the matrix Z [Z:g] given in (9), which forms the basis for analyzing relationships among competitors.
To visualize these relationships, apply kernel PCA Z and project the data into three dimensions, as shown in Figure 2. Each point represents competitor, with distances reflecting their dissim- ilarities. We use squared exponential kernel, Pi exp(->2ī)) with 10 to improve interpretability in feature space. This spatial representation highlights groupings of competitors with similar performance patterns.
Complementing the kernel PCA visualization in Figure Appendix C.2 employs multidimensional scaling (MDS) as an alternative approach for interpreting the dissimilarity matrix Z. MDS directly translates pairwise dissimilarities into spatial distances, preserving relational structures reduced- dimensional space. Both KPCA and MDS uncover consistent spatial grouping patterns that align closely with rankings.
To systematically capture these spatial patterns, we applied hierarchical clustering to Z, as detailed in Appendix C.3. The clustering analysis organizes competitors into performance-based tiers. For example, top performers such as ChatGPT-4o Latest and Gemini 1.5 Experimental form distinct clusters, reflecting their dominance, while mid- and lower-ranked competitors stratify into meaningful subgroups. Notably, the optimal ordering in hierarchical clustering effectively retrieves the ranking order, even without direct access to individual scores. These results emphasize the value of covariance in analyzing relational patterns.

---

Published as a conference paper at ICLR 2025
12 pro goop....
0.004
pamito S-pm-a
eilene 766-775
hard Banejja-2agaamnn
20240702
0.002
reke flsh 21b
Eettraa 2.
aame 31 io.
fama 1.20
glmi 5116
nutruct
1108
0.000
commang
simed
21248001
large preview
rdee lisk
mjstrallarg--000 2007
-0.002
mistslaag 2002
provew
despintt 0522.
-0.007
Te prote
-0 008
0.008
0.010
réG flssh 20040722
-0.012
0.000
-0.014
-001
-0.002
0.000
8
-0.10
0.002
-0000
Figure 2: Kernel PCA projection of the distance matrix onto three dimensions, focusing on the top 40 competitors ranked by Model 18 of Table D.1. Circle size and color proportional to the competitors' scores.
CONCLUSION
Evaluating large language models (LLMs) through pairwise comparisons is essential for assessing their capabilities and limitations in practical applications. This paper presents a statistical frame- work that extends traditional methods by incorporating tie modeling. covariance structures, and constraints to address optimization challenges, enhancing interpretabilityy stability, and accuracy in LLM evaluations.
Our generalized tie model addresses long-standing limitations of Rao-Kupper and Davidson methods, reducing prediction errors by two orders of magnitude and improving across win, loss, and tie outcomes. By introducing covariance structures, our framework uncovers latent relationships among competitors, enabling clustering into performance tiers and enriching interpretability beyond rankings. Additionally, resolve optimization challenges through novel constraints, ensuring stable and unique parameter estimation.
A key insight from our analysis is that covariance structures not only enrich interpretability but also shape ranking consistency, underscoring their critical role in capturing nuanced relationships in pairwise comparisons. This finding highlights the broader utility of covariance-based approaches in complex evaluation tasks.
To validate framework, conducted rigorous empirical evaluations, demonstrating significant improvements in model fit and interpretability compared to existing methods. To support reproducibil- ity and broader adoption, provide leaderbot, open-source Python package implementing our statistical framework with tools for data processing, model fitting, and visualization.
Our work not only advances LLM evaluation but also provides robust foundation for analyzing pairwise comparison data in diverse domains. By bridging theoretical rigor with practical applica- bility, this framework opens avenues for ranking, inference, and interpretability in complex comparison tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors thank Anastasios Nikolas Angelopoulos and Tianle Li for their thoughtful feedback. MWM acknowledges support from the DOE, IARPA, NSF, and ONR.
REFERENCES
Baik, J., Kriecherbauer, T., Mcl McLaughlin, K. T.-R., Miller, P. D. (2007). Discrete Orthogonal Polynomials. (AM-164): Asymptotics and Applications (AM-164). Princeton University Press.
Bar-Joseph, Z., Gifford, K., & Jaakkola, (2001). Fast optimal leaf ordering for hierarchical clustering. Bioinformaticx, 17(suppl_1) 1), S22-S29.
Böckenholt, U. (1993). Applications of Thurstonian models to ranking data. In M. A. Fligner S. Verducci (Eds.). Probability Models and Statistical Analyses for Ranking Data (pp. 157-172). New York, NY: Springer New York.
Böckenholt, U. (2001). Thresholds and intransitivities in pairwise judgments: A multilevel analysis. Journal of Educational and Behavioral Statistics, 26(3), 269-2822
Böckenholt, U. (2006). Visualizing individual differences in pairwise comparison data. Food Quality and Preference, 17(3), 179-187. Seventh Sensometries Meeting, Davis, USA, 28-30 July 2004.
Bradley, A. & Terry, M. E. (1952). Rank analysis incomplete block designs: comparisons. Biometrika, 39(3-4), 324-345.
method of paired
Chen, M., Tworek, J., Jun, H., Yuan, Q. de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A.. Puri, R., Krueger, G., Petrov. M., Khlaaf, H. Sastry, G., Mishkin. P., Chan, B.. Gray. S., Ryder, N., Pavlov, M., Power, A., Kaiser, I.., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss Voss, A., Guss, H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever. I., & Zaremba, W. (2021). Evaluating large language models trained code. arXiv preprint arXiv:2107.033:
Chiang, W. L.. Zheng, L.. Sheng. Y., Angelopoulos. A. N.. Li, T., Li, D., Zhu, B.. Zhang, H., Jordan, M.. Gonzalez, E., Stoica, (2024). Chatbot Arena: An open platform for evaluating LLMs by human preference. Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research (pp. 8359-8388); PMLR.
Corr, P., Smith, F. Hanna, P., Stewart, D., & Ming J. (2000). Discrete Chebyshev transform a natural modification of the DCT. In Pattern Recognition, International Conference on, volume 3 (pp. 7154). Los Alamitos, CA, USA: IEEE Computer Saciety.
Davidson, R. R. (1970). On extending the Bradley-Terry model accommodate ties in paired comparison experiments, Journal of the American Statistical Association, 65(329). 317-328.
Elo, (1978). The Rating of Chessplayers,
and Present. New York: Arco Pub.
Endres, S. C., Sandrock, C., & Focke, W. W. (2018). A simplicial homology algorithm for Lipschitz optimisa- tion. Journal of Global Optimization, 72(2), 181-217.
Epoch AI (2022). Parameter, compute and data trends in machine learning. Accessed: 2024-11-22.
Gibbons, J. Chakraborti, S. (2003). Nonparametric Statistical Inference. Statisties, texthooks and monographs. Marcel Dekker Incorporated, fourth edition: revised and expanded edition.
Glenn, W. A. & David, H. A. (1960). Ties in paired-comparison experiments using a modified Thurstone- Mosteller model. Biometrics, 16(1), 86-109.
Glickman, M. E. (1995). A comprehensive guide to chess ratings. American Chess Journal, 3(1), 59-102.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2 edition.
Heiser, W.J. & de Leeuw, J. (1981). Multidimensional mapping of preference data. Mathématiques et Sciences Humaines, 73, 39-96.
11

---

Published as a conference paper at ICLR 2025
Hendrycks, D., Burns, C., Basart, S.. Zou, A., Mazeika, M., Song. D.. & Steinhardt, J. (2021). Measuring massive multitask language understanding. In International Conference on Learning Representations.
Hoffmann, J. Borgeaud. S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford. E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, F., Vinyals, O., Rac, J. W., & Sifre, I.. (2022). Training compute- optimal large language models. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '2 Red Hook, NY, USA: Curran Associates Inc.
Hopkins, D. J. & Noel, Н. (2022). Trump and the shifting meaning of "Conservative": Using activists'
pairwise comparisons measure politicians' perceived ideologies. American Political Science Review, 116(3), 1133-1140.
Hunter, D. R. (2004). MM algorithms for generalized Bradley-Term models. The Annals of Statistics, 32(1), 384 406.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08880 088361..
Karthik, S.. Coskun, H., Akata, Z.. Tulyakov, S., Ren, J., & Kag, A. (2024). Scalable ranked preference optimization for text-to-image generation. arXiv preprint arXiv:2410.1800
Kendall, (1938). new measure rank correlation. Biometrika, 30(1/2), 81-93.
Kendall, (1945). The treatment
ranking problems. Biometrika, 33(3), 239-251.
Liu, J., Ge, & Zhu, (2024). Reward learning from preference with ties. arXiv preprint 2rX.:22410005328.
Loewen, J., Rubenson, D., Spirling, (2012). Testing power of arguments referendums: Moadley-Tery approach. Electoral Studies, 31(1), 212 212-221. Special Symposium: Germany's 's Federal Election September 2009.
Luce, R. D. (1959), Individual Choice Behavior: A Theoretical Analysis. Oxford, England: Wiley.
Mardia, K., Kent, J., & Bibby, J. (1979). Multivariate Analysis. Probability and Mathematical Statistics series of monographs and textbooks. Academic Press.
Maydeu-Olivares, A. (1999). Thurstonian modeling of ranking data via mean and covariance structure analysis. Psychometrika, 64(3), 325-340.
Maydeu-Olivares, A. Böckenholt, U. (2005). Structural equation modeling of paired-comparrson and ranking data. Psychological Methods, 10(3), 285-304.
Newman, M. J. (2023). Efficient computation rankings from pairwise comparisons. Journal of Machine Learning Research, 24(238), 1-25.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2024). Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference Neural Information Processing Systems, NIPS '23 Red Hook, NY, USA: Curran Associates Inc.
Rao, V. & Kupper, L. (1967). Ties in paired-comparison experiments: A generalization of the Bradley- Terry model. Journal of the American Statistical Association, 62(317), 194-204.
Rothenberg, (1971). Identification in parametric models. Econometrica, 39(3), 577-591.
Schölkopf, B., Smola, A. J., & Müller, K.-R. (1999). Kernel principal component analysis, (pp. 327-352).
MIT Press: Cambridge, MA, USA.
Seber, G. & Wild, C. (2005). Nonlinear Regression. Wiley Series in Probability and Statistics. Wiley.
Shah, N. B., Balakrishnan, S., Guntuboyina, A., & Wainwright, M. J. (2017). Stochastically transitive models for pairwise comparisons: Statistical and computational issues. IEEE Trans. Inf. Theor, 63(2), 934-959.
Shah, N. B. & Wainwright, M. J. (2018). Simple, robust and optimal ranking from pairwise comparisons. Journal Machine Learning Research, 18(199), 1-38.
Süderströö, T. Stoica, P. (1989), System Identification. Prentice-Hall Software Series. Prentice Hall.
12

---

Published as a conference paper at ICLR 2025
Takane, Y. (1989). Analysis of covariance structures and probabilistic binary choice data. In G. de Socte, H. Feger, K. C. Klauer (Eds.), New Developments Psychological Choice Modeling, volume 60 Advances in Psychology (pp. (30), North-Holland,
Thurstone, L. L. (1927). A law of comparative judgement. Psychological Review, 278-278-26
Wales, J. & Doye, J. P. K. (1997). Global optimization by Basin-Hoppiing and the lowest energy structures of Lennard--Joo -ones clusters containing up to 110 atoms. The Journal of Physical Chemistry 101(28), 5111-5116.
Williams, (2000). On connection between kemel PCA metric multidimensional scaling, Leen, Dietterich, & V. Tresp (Eds.), Advances Neural Information Processing Systems, volume 13: MIT Press.
Wu, T., Zhu, B., Zhang. R., Wen, Z., Ramchandran, K., Jiao, (2023). Pairwise proximal policy optimization: Harnessing relative feedback for LLM alignment. arXiv preprint arXiv:2310.00212
Wu, T., Zhu, B., Zhang, R., Wen, Z., Ramchandran, K., Jiao, (2024). Pairwise proximal policy optimization: Language model alignment with comparative RL. In First Conference on Language Modeling.
Zermelo, E. (1929). Die berechnung turnier-ergebnisse srechnung. Mathematische Zeitschrift, 29(1), 436-460.
maximumproblem der wahrscheinlichkeit-
Zheng, L., Chiang, W.-L., Sheng. Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Lin, Z., Xing, E., Gonzalez. E., Stoica, L, & Zhang. H. (2024). LMSYS LMSYS-Chat-IME Chat-lM: acgeescale real-world LLM conversation dataset. In The Twelfth International Conference on Learning Representations.
Zheng, L.,, Chiang, W.-L., Sheng. Y., Zhuang. S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023). Judging LLM-as-a-juue with MTT-bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36,
13

---

Published as a conference paper at ICLR 2025
Appendices
CONTENTS
A Broader Implications of Paired Comparison Methods
14
Unidentifiability of Parameters in Paired Comparison Models
15
B.1 Fisher Information and Identifiability: Background
15
B.2 Unidentifiability of Score Parameters
16
B.3 Identifiable Parametrization.
17
Covariance Model
17
C.I Non-Uniqueness and Equivalence Class of Covariance
17
C.2 Interpretation and Visualization of Covariance
18
C,3 Hierarchical Clustering of Competitor Performance
19
C.4 Constraint on Thurstonian Covariance Model
21
D Evaluation of Statistical Models
22
D.1 Details of Experimental Setup
22
D.2 Model Selection
23
D.3 Model Fit
24
D.4 Generalization Performance
26
D.5
Evaluating Marginal Probabilities Win, Loss, and Tie
26
Comparative Analysis of Ranking Variability
28
E.1
Representative Model Rankings and Score Distribution
28
E,2
Exploring Ranking Similarity Across Models
30
E.3 Quantifying Ranking Similarity Across Models
30
E,4 Identifying Ranking Similarities via Hierarchical Clustering
Relationship Between LLM Characteristics and Scores
33
Implementation and Reproducibility Guide
34
G.1 Model Training and Visualization
34
G.2 Model Evaluation: Fit and Consistency Metrics
35
G.3 Model Generalization: Performance Test Data
36
APPENDIX A BROADER IMPLICATIONS OF PAIRED COMPARISON METHODS
Paired comparison frameworks are foundational in many fields and have been widely applied in clas- sical and modern domains. In sports analytics, they are used to predict match outcomes and assess
14

---

Published as a conference paper at ICLR 2025
player performance in games such chess (Elo, 1978), tennis (Glickman, 1995), and soccer. Mar- keting applications include optimizing product offerings, advertisement placements, and analyzing consumer preferences. In psychometrics and behavioral studies, paired comparisons assess percep- tion and attitudes in response to visual or auditory stimuli. Similarly, in election studies and political science, they are employed rank candidates, analyze voting behavior, test referendum arguments (Loewen et al., 2012), and measure perceived political idcologies (Hopkins & Noel, 2022). Clinical research also uses paired comparisons to evaluate treatments and interventions in clinical trials and epidemiological studies. These diverse applications illustrate the versatility of paired comparison frameworks in extracting meaningful inferences from comparative data.
Recent advancements have extended paired comparison methods to machine learning, where they play a pivotal role in preference modeling and optimization. For instance, Reinforcement Learn- ing with Human Feedback (RI.F) uses paired comparisons to fine-tune large language models by ranking outputs based on human preferences, often employing the Bradley-Terry model for pref- erence quantification (Rafailov et al., 2024; Karthik al., 2024). Direct Preference Optimization (DPO) further refines this approach by aligning model outputs directly with human preferences without relying on scalar reward models (Wu et al., 2023). Additionally, methodologies like Pair- wise Proximal Policy Optimization (P3O) leverage relative feedback to enhance LLM alignment (Wu et al., 2024). Innovations such the integration of Rao-Kupper models have enabled paired comparison frameworks to incorporate ties, capturing ambiguous or neutral preferences in RI.FF settings (Liu al., 2024). These developments highlight the growing influence of paired comparison methods in machine learning and underscore the potential of our generalizations to enhance these frameworks further.
APPENDIX B UNIDENTIFIABILTTY OF PARAMETERS IN PAIRED COMPARISON MODELS
In this section, we discuss the challenge of estimating the uncertainties of scores z, in paired compar- ison models. Previous works, such as Chiang et al. (2024), have introduced methods for computing confidence intervals for scores using empirical approaches like bootstrapping. While these methods can be valuable in practice, we demonstrate that this problem is intrinsically ill-posed due to the unidentifiability of the score parameters. Specifically, the likelihood function depends only on score differences z; Zy rendering individual score estimates invariant under shift transformation. This invariance introduces non-uniqueness in the quantification confidence intervals.
This issue arises not from specific modeling choices but from structural characteristic of models based on linear stochastic transitivity, where probabilities take the form F(z, - xz) (see Section 2.2). To analyze this limitation rigorously, we examine the Fisher Information Matrix (FIM) of the like- lihood function. In Appendix B.1, 1, we provide a background on the FIM and its role in parameter identifiability. In Appendix B.2, show that the score parameters are unidentifiable due to the singularity of the FIM. Finally, in Appendix B.3, we propose reparameterizations that result in identifiable quantities.
B. FISHER INFORMATION AND IDENTIFIABILITE BACKGROUND
The Fisher Information Matrix (FIM) quantifies the amount of information that the likelihood func- tion carries about the parameters of interest. can be derived from the gradient of the log-likelihood function, known as the informant vector, 2 or equivalently, from the negative Hessian matrix of the log-likelihood under mild regularity conditions:
F())= = [V & @ Vo(()) E V]o(((())
(B.1)
The FIM measures the curvature of the likelihood function around the estimated parameters, reflecting the precision of the parameter estimates. A sharper likelihood function implies higher confidence in the parameter estimates. Formally, the Cramér Rao bound establishes a theoretical lower bound for the covariance of the parameter estimates (see e.g., Söderström & Stoica (1989)):
cov (0) > F (0)
(B.2)
Commonly referred to as the score but this term is avoided here prevent confusion with the score
parameters z;-
15

---

Published as a conference paper at ICLR 2025
This lower bound is often used to derive estimates of parameter uncertainty. For example, assuming the approximation var(O.) 2 F Ils. the confidence interval for can be estimated as
20; f., var (0,).
(B.3)
1] where aan-m is the critical value from the Student's I-distribution for a confidence level a E [o. with n- m degrees of freedom, being the number of data points and m the number of parameters. This approach yields conservative estimate of the variance of (, reflecting the Cramér-Rao bound. Alternative methods, such as bootstrapping, may provide more practical confidence intervals in certain cases.
When the FIM ill-conditioned or singular, however, the parameter estimation problem becomes ill-posed. In such cases, the uncertainty bounds A0; become unbounded undefined. This occurs when likelihood function exhibits invariance under certain transformations of the parameters, leading to parameter redundancy. We formally define parameter identifiability and its connection to the FIM below.
Definition B.1 (Rothhebbrg,, 1971, Definitions 1, 2, and 3)). Two parameter vectors @ and 6' are said be observationally equivalent if ((0) E(0"). A parameter vector 0 locally identifiable there exists an open neighborhood around containing no other O' that is observationally equivalent to 0. If 0 is not observationally equivalent to any other parameter vector in the entire domain of the likelihood function, is said be globally identifiable.
Theorem B.1 ((Rothenberg, 1971, Theorem 1)). Suppose the Fisher Information Matrix F is continuous function of O. Let 6" be regular point of meaning F(0) has constant rank in neighborhood of Then, 0" is locally identifiable if and only if F F(0) is non-singular.
The above theorem establishes that the FIM plays central role in determining parameter identifia- bility (see also (Seber Wild, 2005, Sections 3.4 and 8.4)).
B.2 UNIDENTIFIABILITY OF SCORE PARAMETERS
We now focus the identifiability of the score parameters, in paired comparison models. For simplicity, we limit the analysis to æ, though the results extend naturally to other parameters. We prove that the FIM for æ is singular for likelihood functions satisfying the shift invariance property. While the invariance property trivially implies unidentifiability by definition, analyzing the FIM reveals deeper insights into the parameter space. Specifically, it identifies the null space causing unidentifiability and highlights subspaces suitable for well-defined reparametrizations. as explored in the next subsection.
Propusition B.1. Let the log-likelihood function C² (R"" .R) satisfy the shift invariance property
f(x + c1) {(x), C C.R.
(B.4)
Then, the corresponding Fisher Information Matrix F(x) is singular where rank(F(æ)) S m 1. with 1 (the vector of ones) in its null space.
Proof. From (B.4) we have
0k(x + cl)
df(x) = 0.
(B.5)
dc
j=1 dx,
On the other hand, summing over all columns of the Hessian, H == Va VI((),, and using (B.5) yields
8 0l(())
a
H((x)
Σ Σ
-0,
Vi =i=1 1,...,m
(B.6)
dr,On,
Ox;
0x;
Hence, H has a zero row sum, implying H1 O. Therefore, 1 lies in the null space of H. and by extension, F(x) is singular with rank of at most m 1.
The singularity of F(x) confirms the unidentifiability of the score parameters æ. rendering the quantification of their uncertainties fundamentally ill-posed. As a result, the lower bounds from the Cramér-Rao inequality in (B.2) become unbounded, making the confidence interval such in (B.3) undefined. In the next section, we analyze the structure of F(x) to identify subspaces where meaningful parameter estimation is possible.
16

---

Published as a conference paper at ICLR 2025
B.3 IDENTIFIABLE PARAMETRIZATION
We now address which quantities are identifiable through the FIM. Specifically, any reparameter- ization within the range of the FIM identifiable. Let N denote the null space of the FIM and No its orthogonal complement. Suppose f 0* a local minima of the likelihood function. The FIM, when restricted to Not, positive definite, and any parameterization within this subspace identifiable.
In the case of pairwise comparison with the optimal solution a = x assuming (ank(F(x")) = m 1. we have Naa.. == span(1). The projection operator onto N is given by
P= 11
(B.7)
which the centering matrix that converts x" the mean-zero vector H" == Px E N A representation of this reparameterization is provided by the (m 1) x m forward differencing matrix A I + Na with entries A,, 1, A,,+1 -1, and otherwise. Using A, x be transformed into y = Ax", where each component y x r represents the difference between adjacent elements of x²" This reparameterization lies entirely in No making y" identifiable and enabling meaningful quantification of its uncertainty.
Thus, in paired comparison models we consider, only differences in scores provide meaningful inference. the next section, we explore this in the context of Thurstonian covariance parameters.
APPENDIX C COVARIANCE MODEL
In Section 2.4 expand the inclusion of covariance via Thurstonian model. We recall that, in Thurstonian model, the score parameters are assumed to be stochastic with x; = #4 + where e, is the stochastic component with the covariance Σ [o,;] where dj = cov(E, (c,; 5, Furthermore, defined the matrix = [Rig] where 8,y = dg + グj 20,5 representing the variance of x; Ty--
In this section, we provide detailed analysis of the covariance matrix Σ and its associated matrix S. In particular, in Appendix C.I, we explore the identifiability, particularly how Σ is inherently non- unique while S remains unique and identifiable. In Appendix C.2 we present how interpret and visualize these matrices. In Appendix C.3, we examine hierarchical clustering of competitors based on the dissimilarity matrix, uncovering performance tiers and relationships. Finally, Appendix C.4 we discuss constraints that allow stable identification of covariance during optimization of likelihood.
C.1 NON- UUNQUNQQUNNES AND EQUIVALENCE CLASS OF COVARIANCE
We begin by noting that the likelihood function in the Thurstonian models we presented depends on the function of ziy defined in (9), which itself depends siy- That is, S is observable quantity, while Σ is a latent variable. Below, we formalize the relationship between these two matrices and the equivalence class of covariance matrices that share the same S.
Let S denote the space of symmetric m x m matrices and S be the the space of hollow symmetric matrices where all diagonal elements are zero. Define the map S S that associates covariance matrix Σ with the matrix S, given by
S = S(E) = diag (E)1 + diag T -2
(C.1)
where diag(E) Σ) is vector containing the diagonal elements of Σ. This relation corresponds to $ty 03 + و3 20, in matrix form.
As we will show momentarily, the map S is non-injective, for each S, there exist non-unique covariance matrices Σ differing by elements in the kernel of S, all of which map to the same S. Consequently, the preimage of S defines the equivalence class of covariance matrices producing the same S, given by
Σ = S (S) E'' E SM
S(E') = S}
(C.2)
This equivalence class partitions T modulo the kernel of S, denoted as Snh /Sr( We now formalize this structure.
17

---

Published as a conference paper at ICLR 2025
Proposition C.1 (Equivalence Class of Covariance). The map S :3" - S.. defined in (C.1), is surjective, non-injeed non-injective linear transformation. Its kernel is given by
ker(S) = {vl1 + lo [
(C.3)
Consequently, the quotient space 3"h /kee())) represents the space of equivalence classes of covari- ance matrices of the form
Σ
[ + v1 + l 1 R.",,
(C.4)
where all elements of Σ]] map to the same matrix S under S.
Proof. To determine (S), consider E' E EST such that S(E') 0. From (C.1), follows that
diag 11 + diag E')T - 2E' 0.
Rearranging, we find
1 (diag(E") + diag(E')))
E' = 2
implying that any E' ker(S) must of the form given in (C.3). The equivalence class [>]
(S) is obtained by adding elements of ker(S) to representative Σ, yielding (C.4).
To show S is surjective, observe that for any S S the matrix - s" satisfies S(- ((-15) S. Hence, every S e 3 has at least one preimage, proving surjectivity.
As demonstrated in Proposition C.I, the covariance matrix E is not unique, as adding any sym- metric rank-one matrix to it leaves the likelihood invariant. Consequently, Σ is not identifiable by Definition B.1.
C.2 INTERPRETATION AND VISUALIZATION OF COVARIANCE
While the covariance matrix E is not unique, the matrix unique and identifiable. This makes the preferred object for interpreting relationships between competitors. Unlike Σ, which represents similarity, $ plays the role of a dissimilarity mairix, as commonly explored paired comparison lit- erature (Maydeu-Olivarrs Olivares & Böckenholt, 2005; Böckenholt, 2006). In fact, under suitable conditions, can be interpreted as a distance matrix.
For S to qualify distance matrix, it must be non-negative. This holds if and only if the doubly- centered covariance matrix, defined as PPP
(C.5)
is positive semi-definite Here, P is the centering matrix from (B.7). The matrix Σ represents the covariance of mean-centered scores, & Pæ. Importantly, E E [Σ] and is unique within the equivalence class [Σ]. Specifically, for any Σ', Σ" [Σ], holds that PEP PE"P = Σ. This ensures that Σ is well-defined and serves canonical representation of the covariance structure.
matrix S is called a Euclidean distance matrix if there exist spatial points PI..... P.. such that 8; lp: 1l Mardia et al. (1979, Theorem 14.2.1) guarantees that S is a Euclidean distance matrix if and only if Σ is positive semi-definite. In our setting, this condition is always satisfied, as we enforce the factor covariance model in (13) as
- D AAT
(C.6)
where D is positive diagonal matrix (D 0), ensuring that Σ is positive definite. Consequently, D is positive semi-definite, making S Euclidean distance matrix.
This property enables meaningful visualization of using multi-dimensional scaling (MDS). MDS (see e.g., (Mardia et al., 1979, Chapter 14) or (Seber & Wild, 2005, Section 5.5)) constructs set of points in two- or three-dimensional space such that their pairwise distances approximate the distances in S. This approach is particularly suitable for visualizing S or similar distance matrices derived from covariance models.
Here, the matrix Z the distance matrix, defined in (9), where 2g (x; - x1,) /Say. capturing both score differences and dissimilarities derived from covariance. The dissimilarity repre- sented by Z is visualized in Figure C.1 using MDS, showing the first principal coordinates in
18

---

Published as a conference paper at ICLR 2025
Mubidimenioonll Scaling
07-
chaipe--o
-0.30
0.6
no0 fish 20046022
des de
diezsod 22upi 2021 ge penvin
E21- 101-4--ttn.300....
0.16
dbude connet
nierol oage 2407
331 DI
sigh prr
oabii 3124
0.14
1100
311---3228---11
61---0013 4,003
de 0038
25247722
S
gereii 5 5.00.pp
-0.00
0 that
0.05
iikaciaah
22.-22..
02
Figure C.I: MDS projection of the distance matrix onto two dimensions, focusing on the top 50 competitors ranked by Model 18 of Table D.1. In the scatter plot, circle size and color are proportional to the competitors' scores.
two-dimensional space. Interestingly, the spatial arrangement of points in the plot not only reflects the pairwise dissimilarities but also aligns well with the overall ranking of competitors, effectively capturing their relative scores. This highlights the ability of MDS to extract meaningful patterns from the dissimilarity matrix alone, without access to the values of the scores.
A companion approach to MDS is kernel PCA (Schölkopf et al., 1999; Williams, 2000), applied earlier in Section 4.2 and visualized in Figure 2. Both techniques aim to uncover relationships in swwer-dimensionll spaces, with kernel PCA operating on points in a feature space and MDS working directly with pairwise distances. These methods are dual representations: PCA identifies principal components of the data, while MDS identifies principal coordinates of distance matrix (Mardia et al., 1979, Section 14.3). Notably, both the kernel PCA in Figure (projected into three dimensions) and the MDS in Figure C.1 (projected into two dimensions) rely on the same dissimilarity matrix Z. offering consistent visual representations of the relationships between competitors.
C.3 HIERARCHICAL CLUSTERING OF COMPETITOR PERFORMANCE
To complement the analysis of dissimilarity using PCA and MDS, we applied hierarchical agglomer- ative clustering (Hastic et al., 2009, Section (4.3.12) with optimal leaf ordering (Bar-Joseph et al., 2001) to the dissimilarity matrix Z. We recall that the matrix Z incorporates both score differences and dissimilarities derived from Thurstonian covariance in generalized models. This integration of covariance within the model enhances the interpretability of the clustering results by capturing both the performance levels of competitors and their correlation structure.
The clustering analysis focuses on the top 100 competitors ranked using Model 18 from Table D.I. The resulting dendrogram, shown in Figure C.2, reveals a hierarchical structure that organizes com- petitors into distinct tiers. Interestingly, despite the clustering algorithm having access only to the dissimilarity matrix Z, which encodes score differences (but not the values the scores) and eerariance-dervee uncertainties, the clustering order closely aligns with the competitors' rankings. Each cluster roughly corresponds contiguous range of ranks, albeit with minor variations, demon- strating the ability of the clustering method to infer meaningful performance groupings solely from relative differences.
This hierarchical structure provides additional insights into the relationships between competitors, suggesting a natural stratification into performance- based tiers. Tier includes the two leading models, ChatGPT-40 Latest (as of September 2024) and Gemini 1.5 Pro Experimental, reflecting their dominance. The remaining 98 models form Tier II, which is further divided into two subgroups: IA (green theme) and TB (red theme). These subgroups are further refined into smaller clusters:
19

---

Published as a conference paper at ICLR 2025
202402222222
claude 3-sonnet- 2024-0244.
reka-core 20240501
1lama-3-70 70s instruct
nemotron-4- sorrronn--400--- 3-000--nsss....
bard-jan-24-gee -e-iiii pro
deepseek. v2 (--ap--- 0628
deepsrek- wereek--ndeer-re cade---dd----- aade--220070 220024
grmini-1 1-5--aass-- s-flash-apl... aash-app....
gomma-2-22 2-2---2--b- it
reka teka-core- 20240722
quent 7-7--bbbrrrr
command ummmand-er replus
reke: Maaa--ash--0000
gin-4-05- 4--0-00000
Eot 00125 proview
claude 3-opus- 20240229
gemma 22----bbi
Aereeeee cader--p 12r-4/2
gin gim-4-0116 4-016
yi-large
jillarge preview
got 1ot-4-11 11106- preview
laam- 3. 11-7Oh--nnn rub-instruct
got- 1ol-4-turber 20--222244 04-09
com
rete flasfflhhh previes-2....
Oren more 0428
4itt44 4-0314
que
muma--33 3. 1-80- -ebb-mttrrct
mistral large-2417 2407
doude
athene- 70b 70b-0725
gemini- 1 5 5-pro 5-pro-api-O.. api-O...
llama-3 1-.1--4000 1-111-----mmm 11-44-0instrecc
owen Iwenl 1.5, 1110b-chat
Ret- 4-0013
gemini-1. 5.5-pro-app 5-pro-api-0514 api-0514
flash- 1ee-22b-2000
24022
mistral-lar Marge-2402
Tier Hal
10 70
got gpt-4o-2024400 100-2024-08-00 000-0008-0000 -06
gemini- gemini-advancnt aaaii--dvaaceddddom 0514
yi (1115-344 34bb--34b-hh char
claude 1-udee3-5--aa sonnert-..
command-r
Tier 14
Tier 14s2
got- 4o-mini-2022-07-11
Mama- 3-8b---------- Bb-insttuct
mistral-medii serraa-meelumm
mistral-
3.
gpt-4o
20-4--2222-0--13
reka-flash 21- 2022222
gemini-1.5-praat
5-pro-xxx0--
Tier I
chatgot-4o-
la-tat
mixtral-88 bo22b-iieeee
Saphyr-oroa
claude:1
Ter
98
99
100
vicuna- 136
codellama- 34b-in-----sn-uuct
codellama-70bb 70b-instruct
1416-A35.
qwen1. L
mistral-nect
Vir
97
pplr: 7b-online
zephyr- zephyr-7b-alpp 7b-alph
1.5-32b-chat
gomini
gomini
Tier tle
llama-22 2- 1--13bh-hhr chat
pro-dev-app
claude-2.0
phi-3-mini 1--33--mii---e 4k-instru...
11t-3-medium----e 4k-inst
33
g
quen 5. cbat
daude-2.11
Starlin
61
Fia1
solar- 10, 70 io-iintrruct....
mpt: 30b chat
Ept-3 5
wizardim- 13b
mistral-7u 7b ii--instruct-...
S-tubbo-0613
gemini-pro
34-44
3k.
8
delphinn-22 2.1.2.2..mmmtta.. i-mmsttr......
Quen1.5-1440 5-.5--14 1--14bhhhtt
gamma 2amma-22-2bl
Quent
8
Ilamaz llama--70b (11-700-sten
openchas
cluude- iost-iistaatt11
wizardi witzrrlm-77 706
spt 3.:335ttu 35-turh-11 1o-111 1106 pple 706 70b-onlin on-iiiiee
1.1 L-7b-it
2-2b-4t
spe 5-ttrbuuru 0125
tulu-2 2-l-22--p--00 d-p----770 706
3 S-burbo 0314
8p0-3
dbrz-
openchat 5-0106 vicuna 33b
openhermet mentieeees--... 2.552.5 5.5miistr... starting (m----- ne---b-oa alpha snow flake arctic-inss.. llama-2 70b chat derpseak flm- 67b-chat nou: hermes 2-mixtra...
S-turbo-0129
ins-insttuuttt-o previes
hhi-3-small-88 in-iinttucct
coll-stterimm-ee chat
Figure C.2: Hierarchical clustering of the top 100 competitors based on the distance matrix Z derived from Model 18 of Table The clustering reveals performance tiers, with Tier consisting of the top two competitors (ChatGPT-4 Latest and Gemini 1.5 Experimental) and Tier II further subdivided into groups representing decreasing performance levels. This structure highlights the relationships and relative strengths among competitors.
AA is split into HA, (dark green) and II, (light green), while Igg is split into HB, (light red) and HR, (dark red).
The meaningfulness of these groupings arises from the use of Z, which integrates performance scores with covariance-based dissimilarities-aa capability enabled by incorporating Thurstonian models in our framework. This analysis complements rankings by uncovering hierarchical structures and relational patterns among competitors.
While our analysis focuses on hierarchical clustering derived from the statistical properties of the dissimilarity matrix, alternative clustering approaches, such as those leveraging semantic relation- ships (e.g.. embeddings or linguistic features), could provide complementary insights into model relationships. Future research could explore these directions, integrating semantic and statistical perspectives to uncover deeper insights into competitor relationships.
20

---

Published as a conference paper at ICLR 2025
C.4 CONSTRAINT ON THURSTONIAN COVARIANCE MODEL
In Section 2.5 we presented constraints to resolve the symmetry of the likelihood functions with respect transformations of the parameters. Here, provide further detail the second symmetry presented therein. Specifically, we recall that the models are invariant under the transformation (iy) + (tx,, ox)) for an arbitrary > 0.
One approach resolve the arbitrariness of the parameters introduced by such translation to impose the constraint
C = 2m
Si 1,
(C.7)
is arbitrary, chosen for convenience we will explain momentarily. Since Si represents the variance of the difference Tij = H2 - Ty, this constraint ensures that the total variance of all random processes zi is fixed. We will further show that this constraint can be directly expressed in terms the covariance matrix E.
where the constant 2m
Proposition C.2. The constraint in (C.7) equivalent to
trace 1,
(C.8)
where Σ = PEP the doubly-centered covariance matrix given in (C.5), and -I-- 11T the projection matrix defined in (B.7).
Proof. Recall from (C.1) that the elements 8ij of the matrix S are related to E by
S = ddiag(E)1 + diag(E)) 2E.
(C.9)
On the other hand, the constraint in (C.7) can be written equivalently as
1SS..
(C.10)
2m
Substituting (C.9) into (C.10) and noting that 1 diag(22 = trace E), we obtain
1
C = trace Σ)
1TEL.
(C.11)
m
Next, using the cyclic property of the trace, we have trace ( = trrcc (PEP) = trace traco(EP2) Since P is idempotent (p² = P), this simplifies to
1 m
trace Σ) tracc Σ)
trace E11T).
(C.12)
Moreover, by the cyclic property of the trace, we have trace( Σ11 E1.. Substituting this into (C.12), we find that trace(E) equals C expressed in (C.11). Thus, the constraint (C.7) is equivalent to trace( Σ) = 1, completing the proof.
By using the factor model (13), the constraint C can also be expressed directly in terms of the diagonal matrix D and the factor matrix A. Additionally, for optimization purposes, we provide the derivatives of C with respect these variables. The following proposition formalizes these results.
Proposition C.3. If the covariance matrix Σ is expressed in the factorized form Σ D AAT then the constraint C in (C.7) becomes
c-(1-1)
(1-1 m
trace(D) || 1
1
(C.13)
m
where || || denotes the Frobenius norm. Furthermore, the derivatives of C are given by
aC =
I,
(C. 14a)
ØD
aC 2PA.
(C.14b)
0A
21

---

Published as a conference paper at ICLR 2025
Proof. Substituting the factorized form Σ = D + AAT into (C.12), we have
1 m
trace Σ) (race(DD trace( trace(AA) (AAT) trace DI11)
trace( IAT
(C.15)
The second term simplifies as trace( trace(AAT) trace(D111) D'D1 (ra since D is diagonal. Also, for the fourth term, trace( AAT I = || Substituting these results into (C.15), we obtain (C.13). The derivatives with respect to D and A follow directly.
(racc(AAA) = || | For the third term,
L
APPENDIX D EVALUATION OF STATISTICAL MODELS
In this section, we systematically evaluate the statistical models introduced in Section 2, focusing on how well they capture empirical data. We begin by detailing the experimental setup used for training and evaluating these models in Appendix D.1. This is followed by analyses of model selection, goodness of to observed outcomes, and generalization performance on unseen data, as detailed in Appendices D.2 to D.4, respectively. Additionally, Appendix D.5 complements the pairwise evaluations in Section 3.1 by analyzing marginal probabilities, offering an aggregated perspective on win, loss, and tie probabilities across competitors.
D.1 DETAILS OF EXPERIMENTAL SETUP
The models used in our analysis are listed in Table D.I. Rows to 6 include the Bradley-Terry (BT) model and its variants, rows 7 to 18 cover the Rao-Kupper model and its extensions, and rows 19 to 30 represent the Davidson model and its variants. For the BT model, we analyze two forms of the dataset. In rows to 3, we modify the input matrices to incorporate ties by treating each tie as half a win and half a loss, i.e., W f- W + 1 T. with row I following the approach by Chiang et al. (2024), In rows 4 to 6, we did not modify W. We recall that in both cases, the BT model does not account for ties, meaning T is effectively ignored.
Rows 1, 4, 7, and 19, marked with (*) in Table D.1, represent the standard versions of the BT, Rao- Kupper, and Davidson models as found in the literature. All other rows correspond to the generalized models proposed in this work, with features detailed in the third and fourth columns of the table.
The third column represents the covariance configuration. In our analysis, we considered three cases for covariance: no covariance (X), Keov = 0, and 3. The symbol "X" indicates that the model does not include covariance structure, excluding the Thurstonian representation. The other two cases employ the factored covariance structure Σ = D + AAT as given in (13), where kcov refers to the rank (i.e., the number of columns) of A. The case keov 0 implies a diagonal covariance matrix, i.e., Σ- = D, where the matrix A is absent.
The fourth column represents the tie configuration. We considered five cases: no tie (X), hise 0, 10,, and 20. The symbol "X" indicates that the model does not account for ties, while hese = O corresponds
the original Rao-Kupper and Davidson models with single tie parameter. For keie > 0, our generalized tie model is employed, where krie represents the rank (i.e., the number of columns) of the matrix G in the factor model for ties, defined in (6).
The fifth column represents the number of parameters in the model. All models include m parameters for the score vector æ, where m 129 corresponds the number of competitors in the Chatbot Arena dataset. Models with factored covariance structure include m additional parameters for the diagonal matrix D and m kevy parameters for the elements of A. Similarly, models incorporating ties add one parameter when kije O, or m x kis parameters for the elements of G when kiti > 0.
We trained these models (except for models and 4) by maximizing the likelihood function (1) using the BFGS optimization method, while satisfying the constraints in Section 2.5. This optimization method requires both the loss function - -(()) and its Jacobian - which we analytically derived with respect to all parameters for each model and provided during training. To ensure consistency, we used a tolerance level of tol 10- for convergence. Parameters were initialized as follows: scores ø were initialized randomly while ensuring their sum is zero, diagonals of D were set to m -1 and all other parameters were initialized to zero. Training time for each model, using an AMD EPYC 7543 processor with 32 cores, is shown in the last column of Table D.I.
22

---

Published as a conference paper at ICLR 2025
Model Features
Cross Entropy
Num. Param.
Training
ID Model
Cov. (ko
Tie (Hisie)
-(())
Win
Loss
Tic
Time
Bradley-Terry (with tie data)
X
129
0.6554
0.3177
0.3376
2.3
X
258
0.6552
0,3180
0.3371
3.8
X
645
0.6549
0.3178
0.3370
34.1
4
Bradley-Tery Terry (without tie data)
X
x
129
0.6351
0.3056
0.3295
0.0
5
x 258
0.6346
0.3059
0.3287
1.7
6
x 645
0.6342
0.3057
0.3285
27.5
7
Raa-KKppa Kao-KKuperr
X
D
130
1.0095
0.3405
0.3462
0.3227
5.8
8
X
1
258
1.0106
0.3401
0.3459
0.3245
6.9
9
X
10
1419
1.0055
0,3404
0,3455
0.3196
208.1
10
X
20
2709
1.0050
0.3403
0,3455
0.3192
396.9
11
0
259
1.0092
0.3408
0.3457
0.3228
8.4
12
1
387
1.0103
0.3404
0.3454
0.3245
7.5
13
1548
1.0052
0,3407
0.3449
0,3196
293.7
2838
I 1.0048
0.3406
0.3449
0.3193
664.9
646
1.0089
0.3405
0.3457
0.3227
36.0
774
1.0100
0.3400
0.3454
0,3245
36.9
1935
1.0049
0,3403
0,3449
0,3196
363.5
3225
1.0044
0.3403
0.3449
0.3193
817.3
+19
Davidson
0
130
1.0100
0.3409
0.3461
0.3231
6.0
20
258
1.0077
0.3413
0.3466
0.3198
10.5
21
X
10
1419
1.0057
0.3404
0.3456
0.3197
253.2
22
20
2709
1.0052
0.3404
0.3455
0.3193
602.8
23
0
259
1.0098
0.3411
0.3455
0.3231
8.7
24
1
387
1.0074
0,3415
0.3460
0.3200
8.3
25
10
1548
1,0055
0.3407
0,3451
0.3197
286.9
26
20
2838
1.0050
0.3407
0.3450
0.3194
665.1
27
3
0
646
1.0094
0,3410
0.3453
0.3231
34.6
28
1
774
1.0070
0,3412
0.3460
0.3199
35.8
29
1935
1.0051
0,3407
0.3448
0.3197
366.4
30
3225
1 1.0047
0.3405
0.3448
0.3194
804.9
Table D.1: Configurations and training details of the 30 statistical models used throughout the analysis. These models are referenced by their ID various sections of the paper. Rows marked with (*) represent the prior models, while unmarked rows correspond to the generalized models proposed in this work.
Models in rows 1 and were trained using the iterative of Newman (2023), which offers notable speed advantages over conventional maximum likelihood estimation. MM algorithms have also been extended to certain generalizations of the Bradley-Tere model, as shown by Hunter (2004). Whether MM methods are directly applicable to the more complex generalized models proposed in this work remains open question and warrants further investigation.
(MM) algorithm
To verify the robustness of our approach, tested global minimization methods, including basin- hopping (Wales Doye, 1997) and simplicial homology global optimization (SHGO) (Endres et al., 2018). While these global methods required significantly longer computation times, they produced solutions consistent with those found by BFGS, confirming the reliability of local optimization for this problem.
D.2 MODEL SELECTION
To evaluate model fit, we computed the negative log- ligeliiel (NLL) and the cross-entropy losses for win, loss, and tie outcomes, shown in the sixth ninth columns of Table D.1.
The cross-entropy loss quantifies how well the predicted probabilities align with the empirical proba- bilities derived from observed data. For given pair { ). the empirical probabilities are computed
23

---

Published as a conference paper at ICLR 2025
Wij
Pi (i,5))
Wji
Pi <i (8,1)) =
(i ~ (4,51)
where nij == Waj + w + biy is the total number of matches between competitors and j. The overall cross-entropy losses, H, for win, loss, and tie outcomes calculated as
Hwin (0) =
Σ (WjERR
Pi > ((,5)) log (P(i > j ((,)))(O) P((,j) E), (D.2a)
Hoss (0) Σ
Pi ((i<)((ú))) log (P(i <j (i))))(O)) P(ti,]) E), (D.2b)
(iJEEE
Hgie (0) = Σ p ( (i,j)) log (P(i ~ j (i) P(tij}} E),
(D.2c)
(WEE
where P(fi,5} E) is the probability of observing a match for the pair {i,j}. given by
P((i.)) E) = Rij where n ==
Σ
(D.3)
(A,))E
The cross-entropy losses for win, loss, and tie outcomes at 0", the optimal parameter values, are reported in the seventh, eighth, and ninth columns of Table D.1, respectively.
The sixth column reports the negative log-likelihood (NLL) of the models at 0", where
1 Σ Wey log (P)) + w/y log (P:z, (0)) tig log (Prw, (0))
(())
(D.4)
(1E)EE
This log- ligeliidl slightly differs from the expression in (1), as it excludes a constant term and is scaled by 1, where is defined in (D.3). Notably, the NLL here is equivalent to the sum of the cross-entropy losses for win, loss, and tie outcomes:
- {()) = Wwin (0) + Hosss (0) H (0).
This relationship can be verified by summing the values in the seventh, eighth, and ninth columns to obtain the value in the sixth column.
Since the BT models do account for ties, their NLL values are lower than those of tie-inelusive models. This is because the NLL of BT models includes only two terms, Hwin and Hosss while tie-inclusive models also include Hiie- Consequently, direct comparison of NLLs between these model types is not meaningful.
On the other hand, when comparing individual cross-entropy losses, Hwin and Hosss observe that the BT models achieve lower values compared to non-BT models. This discrepancy arises because the BT models predict two outcomes (win and loss), yielding only one independent output, as the probability of loss complements the probability of a win. In contrast, the Rao- Kupper and Davidson models predict three outcomes (win, loss, tie), resulting two independent output variables. Thus, the BT models fit one-dimensional output space, while the other models fit a two-dimensional space. Although the BT models achieve lower cross-entropy losses, the complexity of the Mho-Kupper and Davidson models offers richer predictions.
Within each model category, we observe that increasing the rank know for covariance or kigse for tie models consistently improves the NLL and cross-entropies, indicating better fit. Further evaluation metrics, including goodness-of-fe fit and generalization performance, are provided Appendix D.3 and Appendix D.4, respectively.
D.3 MODEL FIT
To assess model fit, we computed multiple metrics, including root-mean square error (RMSE), Kullback-Leibler (KI.) divergence, and Jensen-Shannon (JS) divergence, presented in Table D.2.
24

---

Published as a conference paper at ICLR 2025
Model Features
RMSE
Divergence (x (x10))
ID
Model
Cov. (Rooo)
Tie (Aiss)
Win
Loss
Tie
All
KLD
JSD
Bradley- Terry (with tie data)
X
29.7
29.7
29.7
1.49
0
x
26.2
26.2
26.2
1.42
0.42
3
X
17.4
17.4
17.4
1.30
0.39
Bradley-Tene
x
35.1
35.1
35.1
1.82
0.52
(without tie data)
31.5
31.5
31.5
1.71
0.49
17.3
17.3
17.3
1.58
0.46
Rao- Rao- Kuop-Kupper
48.2
69.9
103.5
77.3
3.32
0.92
x
46.4
67.8
99.2
74.3
3,45
0.91
10
34.1
34.2
23.1
30.9
2.63
0.73
10
34.3
32.2
16.8
28.8
2.35
0.65
46.5
67.9
103.6
76.4
3.23
0.90
12
43.5
66.8
99.4
73.5
3.36
0.89
13
0
10
29.8
31.6
22.7
28.3
2.55
0.70
14
0
20
30.4
29.1
16.7
26.1
2.26
0.63
15
49.0
61.7
104.7
75.6
3.09
0.86
16
48.6
58.7
100.9
73.0
3.18
0.84
17
20.0
21.2
22.1
21.1
2.42
18
20
18.7
18.9
15.8
17.9
2.12
0.59
19
Davidson
X
0
51.0
71.8
109.8
81.3
3.41
0.94
20
44.4
63.3
90.1
68.6
2.99
0.82
21
10
37.1
39.6
25.7
34.7
2.69
0.75
37.7
37.1
17.2
32.1
2.50
0.70
49.4
70.5
109.9
80.6
3.32
0.92
24
0
41.1
62.4
91.4
68.1
2.94
0.81
25
0
10
32.8
37.7
27.0
32.8
2.73
0.76
26
20
35.7
32.6
18.8
30.0
2.56
0.72
27
55.1
61.1
111.0
79.8
3.18
0.89
46.5
50.0
65.5
2.80
29
10
20.8
22.0
25.0
22.7
2.57
0.72
30
3
20
19.1
19.0
17.1
18.4
2.43
0.68
Table D.2: Goodness-of-fi metrics, including root-mean-square error (RMSE). Kullback-Leibler divergence (KLD), and Jensen-Shannon divergence (JSD), for the 30 statistical models introduced in Table D.1. Rows marked with (+) represent the prior models, while unmarked rows correspond to the generalized models proposed in this work.
Soot-Mean-Square Error (RMSE). The RMSE quantifies the deviation between the predicted and observed match frequencies. For each pair {i,j} E E, the predicted frequencies for win, loss, and outcomes are computed as:
ر = naj Pig
wif = naj Pegsi
Egj nhis Panei
The weighted squared errors given by
Tij (Wij (.gg)
win
Σ
nij (w) w 12
eoss
(LiEE
Σ
Thj (L.j 12)
(LEE
The above errors weighted by n,y n, the probability of observing match for the pair { as given in (D.3), to account for imbalances in the number matches per pair. The square roots of these quantities (ewing Cosss etie) are reported in the fifth to seventh columns of Table D.2. The overall
25

---

Published as a conference paper at ICLR 2025
RMSE, Call, is shown in the eighth column, where Call is defined the average of Ewin= a and etie Similar to earlier trends, BT models show lower errors due to their simpler output space. However, in each category, models with higher keov values for covariance and kiie for ties demonstrate notable improvements in model accuracy.
2
Divergence Metries, To further evaluate model fit, we computed the KL and JS divergences between the predicted and empirical probability distributions. To define divergences, we treated the three outcomes of win, loss, and tie classes of discrete probability mass functions with Pij == (Pirj: Pizg: Pinz) and Poj = s Prn)) The KI. divergence defined as
DKL
1
Σ
po, log
pe 1
+ Pig log
P
+ Plag log
[Pins] Pinj
E
Pirs!
Piaj
{iJ)EE
Accordingly, the IS divergence is defined by
Djs(P (P PP) (DKI,(P'I)M) (KI.(PllM)),
where for each pair {i,j), defined the mixture distribution M2 (P: + P.j). The JS divergence is symmetric and bounded between O and 1, making it intuitive for interpretation.
The KL and JS divergences IS values indicate better model fit. Notably, models incorporating covariance and tie factor models yield better results in terms of divergence, reaffirming the effectiveness of these extensions.
shown in the ninth and tenth columns of Table D.2. Lower KL and
D.4 GENERALIZATION PERFORMANCE
To evaluate the models' generalization performance, trained each model on 90% of the data and tested predictions on the remaining 10%. with the data randomly split into training and test sets. Results for the weighted RMSE are presented in the fifth to eighth columns of Table D.3, while the KI. and JS divergences are shown in the ninth and tenth columns, respectively. The definitions of RMSE, KL divergence, and JS divergence remain the same as those in Appendix D.3, but here, these metrics are computed on the test data rather than the training data. These generalization results complement the training results presented in Table D.2, highlighting how model performance extends to unseen data.
These metrics reveal that increasing the number of parameters, such as kçov for covariance or kit for tie factor models, initially enhances generalization performance. However, beyond certain threshold, additional parameters lead to diminishing returns in generalization, even as training fit improves (Table D.2). This phenomenon is attributed to overparameterization, where the model becomes too complex relative to the amount data available.
The ratio of model parameters (fifth column of Table D.I) to the number of data points (EE) 3455, the number of pairs) provides insight into this overparameterization. Models with kise = to kije 10 maintain balanced parameter-to-dant ratio, achieving strong generalization and fit, as reflected in the RMSE values in Table D.3. In contrast, models with kite = 20 approach the upper limit of this ratio, resulting in improved training fit but reduced generalization, signaling the onset of overfitting.
These results highlight the importance of balancing model complexity with data availability to achieve optimal generalization performance.
D.5 EVALUATING MARGINAL PROBABILITIES OF WIN, LOSS, AND TIE
The errors in previous sections were calculated based on pairwise probabilities, such as Po- with errors averaged over all pairwise comparisons in E. Here, we assess the marginal probabilities for each competitor, which represent the overall likelihood winning, losing, or tying against all other competitors. Specifically, denote these probabilities respectively by P(i V (1) F.).
26

---

Published as a conference paper at ICLR 2025
Model Features
RMSE
Divergence (x10²)
Model
Cov. (koow)
Tie (kiss)
Win
Loss
Tie
All
KLD
JSD
Bradley- Terry
X
27.4
27.4
1.46
0.41
2
(with tie data)
x
27.6
27.6
27.6
1.48
0.41
3
x
27.1
27.1
27.1
1.55
0.43
4
Bradley-Tene (without data)
X
x
30.0
30.0
30.0
1.74
0.48
X 30.3
30.3
30.3
1.77
0.48
X
30.4
30.4
30.4
0.54
7
Rao- Kao-Kupper
X
0
54.5
29.1
67.4
52.8
3.16
0.88
8
x
1
49.9
41.9
74.3
57.0
3.31
0.87
9
10
26.3
38.6
32.1
32.7
3.17
0.85
29.0
56.3
66.4
53.0
4.14
1.00
0
52.9
31.5
67.8
52.9
3.19
0.88
12
46.8
45.5
75.0
57.4
3.31
0.87
13
25.9
38.5
31.6
32.4
3.10
0.84
14
20
30.2
54.3
64.6
51.7
4.66
1.06
15
0
50.3
33.8
68.1
52.6
3.31
0.90
16
51.6
42.1
75.0
57.9
3.59
0.93
28.5
34.9
32.3
32.0
3.25
0.87
18
20
35.9
59.6
67.1
55.8
4.71
1.07
* 19
Davidson
X
0
54.7
30.8
70.0
54.3
3.28
0.91
20
X
1
43.1
24.6
42.7
37.8
2.76
0.77
21
27.6
41.8
31.4
34.2
2.95
0.81
28.6
65.5
73.4
3.42
0.93
0 54.0
32.8
70.2
54.5
3.31
0.92
24
1
44.2
26.2
45.0
39.4
2.91
0.81
25
10
26.9
40.0
28.6
32.3
3.06
0.84
26
20
31.0
68.3
80.4
63.5
3.51
0.96
27
0
52.5
34.2
70.2
54.3
3.40
0.93
1
40.7
28.2
44.0
2.87
0.79
29
3
10
33.6
32.7
28.5
31.6
3.31
0.89
30
3 20
32.8
71.6
83.3
66.2
3.70
1.00
Table D.3: Generalization performance of the 30 statistical models introduced Table D.I, evaluated on test data using 90/10 train-test split, including root-mean-soe got-mean-square error (RMSE), Kullback-Leine divergence (KLD), and Jensen-Shannon divergence (JSD). Rows marked with (+) represent the prior models, while unmarked rows correspond to the generalized models proposed in this work.
P(i < V (0) E), and P(i ~ V {i) E), which are defined by
P(i V [1) E) = Σ P(i Po-:51t.)))
(D.5a)
(1))EE()
P(i V E) Σ
P(i Pii ((i
(D.5b)
(1)))B))
P(i V E) = Σ P(i))(
(D.5c)
(1))EE()
where E(1) = fe E Eli e} represents the set of edges incident to the vertex i V, and P( E) the probability of observing a match for the pair {i,j}, as given by (D.3). For brevity, we denote the marginal probabilities in (D.5a) to (D.5c) by P.-V_ PLov_,> and P P.WV_ respectively, where V_; V (i).
Here, we evaluate the goodness fit the models by comparing the predicted marginal probabili- ties of winning, losing, and tying for each competitor against their corresponding empirical marginal probabilities. Figure D.I illustrates the marginal probabilities for selected set of models. Specifi- cally, the first two rows of the figure show results from the BT models (Models and of Table D.I,
27

---

Published as a conference paper at ICLR 2025
with and without modified input data), while rows 3 to 5 correspond to the Rao- Rao-Kupper models (Model 7 as the standard model with one tie parameter corresponding to kise 0, Model 8 with factored tie model and kue 1, and Model 10 with factored tie model and Kije 20). Results for the Davidson models are omitted as they closely resemble those of the Rao-Kupper models under similar conditions.
The left, middle, and right columns in the figure show the marginal probabilities of win (Pi>v_.). loss (Pi-v_,), and tie (P,w._,), respectively. Each row shares the same legend, shown only in the rightmost column. The abscissa represent competitors ordered by their rank Model 18. The colored curves represent predicted marginal probabilities for each model, with red-themed curves for BT models and green-themed curves for Sloo-uuperr models. The black curve represents the empirical marginal probabilities from the observed data, though in some cases, it may overlap with the colored curves. The relative error between model predictions and empirical probabilities is presented in the sixth row, using the same color scheme for consistency. Key observations are as follows:
First, the top two rows of the figure, the BT model predictions (red curves) noticeably deviate from the empirical probabilities (black curve). This is because the BT models do not account for ties, resulting in a different distribution of win and loss probabilities. To provide a fair comparison, we compare BT model predictions with adjusted empirical probabilities, represented by the dotted black curve, which excludes ties. Accordingly, the relative error for BT models in the sixth row is computed against these adjusted probabilities. By contrast, Rao- Kao-Kupper models are compared directly with empirical probabilities from the input data, which include ties.
Second, in the last of the figure, we observe that the errors for BT models are generally lower than those of the Rao- Kupper models. This is due to the smaller output dimension of the BT models, which Rao-Kupper fit only win and loss outcomes. If BT models were compared to the actual empirical probabilities (solid black curves), their error would be higher than that of the Rao-Kupper models. However, such a comparison would be unfair, as BT models are trained modified data that does not account for ties.
Finally, the original Rao-Kupper model with single tie parameter (Model 7, shown in the third row) exhibits errors on the scale of 1 to 10, making it impractical for real-world applications, particularly in predicting ties. However, our generalized Rao-Kupper models, which incorporate factored tie models (Models 8 and 10, shown in the fourth and fifth rows), demonstrate a substantial improvement in accuracy. This enhancement not only elevates the prediction of ties but also improves the prediction of win and loss outcomes by one to two orders of magnitude. This result is significant, as it brings the Kupper and Davidson models back into practical relevance, offering richer predictions for win, loss, and tie outcomes- uunlke the BT without compromising on accuracy.
APPENDIX E COOPPRAAIIEE ANALYSIS OF RANKING VARIABILITY
This section expands on the ranking comparisons presented in Section 4, offering a detailed analysis of how rankings vary across statistical models with different parameter configurations. Visualizations include representative model's ss score distribution (Appendix E, a comparison rankings across models (Appendix E.2), and Kendall's correlation matrix (Appendix E.3) to assess ranking simi- larities. Hierarchical clustering (Appendix E.4) further uncovers structural patterns in model-based rankings, highlighting the distinct effects of covariance and tie parameters on ranking alignment.
E. REPRESENTATIVE MODEL RANKINGS AND SCORE DISTRIBUTION
The score plot in Figure E.1 illustrates the rankings and scores of chatbots, using Model 18 from Table D. as an example. This generalized Rao-Kupper (RK) model incorporates covariance (koow = 3) and tie factor parameters (keie 20), achieving strong goodness-of- fit metrics. Each bar represents a chatbot, ordered from the highest to the lowest score, showcasing their relative performance.
Other models with similar configurations, such as the generalized Davidson model (e.g., Model 30), produce comparable score distributions and rankings. Appendix E.2 examines ranking similarity and variability across selection of models with diverse parameterizations.
28

---

Published as a conference paper at ICLR 2025
Win (Model 1)
Loss (Model 1)
Tie (Model 1)
80%
80%
30%
Probability
www Mnn pheramnn
40%
10%
15%
Obarcved Duta (Moditie)) Medel Pred ctainttion
0%
S0
1250
40
80
129
40
80
129
Win (Model 4)
Loss (Model
Tie (Model 4)
80%
30%
Probabiltty
m M phaneea Mmm
40%
40%
15%
(Modilie)) Modill Prediction
0%
129
10
129
10
80
129
(Model 7)
(Model 7)
Tie (Madel 7)
80%
30%
Probabiiity
40%
40%
15%
Model Prediction
0%
129
129
40
80
129
(Model 8)
(Model 8)
Tie (Model 8)
80%
30%
Probabliity
WWw mw M w
40%
40%
15%
Mudel
0%
0%
129
Win (Model 10)
Loss (Model 10)
(Model 10)
80%
30%
Probabilry
40%
40%
15%
Medel
0%
0%
0%
129
Win (All Models Above)
Loss (All Models Above)
Tie (All Models Above)
100%
109%
hamm
104%
100%
100%
mm
Relazive Error of Probabiltt
mm
10º%
m
100%
w Mamm//AA
myp No S Hyp M
10-1%
10 806
10-1%
Momonm a
10- 259
10
Model (wort Modried Data)
Modd (wuns Modifid Data)
Model
Modiee
Modie 10
10-35%
10-1%
10-09%
20
80
129
10
80
129
10
80
129
Rank
Rank
Rank
Figure D.I: Comparison of predicted (colored curves) and empirical (black curves) marginal prob- abilities of win (left), loss (middle), and tie (right) for selected models. First and second rows: BT models, third row: original Rao- Kupper with tie factor kiie 0, fourth and fifth rows: generalized Rao- Kup-er with tie factors kete and Aie = 20. Sixth row: relative errors for rows one to five, calculated between predicted and empirical probabilities.
29

---

Published as a conference paper at ICLR 2025
0.38
0.16
0.14
0.12
0.10
S2zt vou worge
Doose no t' cutr MerorrO0
1220020 snp ow
R010 plopaates
out po oo- y2- peaq
0250 upt
0100
Dhappo paiappp-oodddie epodiap
11009000 -er
1550 7
soru s Tru
901
08 TE ruey
muss
Figure E.I: Competitors ranked by their scores according to Model 18 from Table D.I.
E.2 EXPLORING RANKING SIMILARITY ACROSS MODELS
The bump chart in Figure E.2 visualizes how rankings vary across 12 selected models from Table D. Each column represents a model, arranged from simpler models on the right to more complex ones on the left, while each row tracks a chatbot's rank across models.
To comprehensively explore ranking similarity, we selected 12 models with diverse configurations. These include baseline models without covariance or tie factor generalizations (Models 1, 4, 7, and 19), models incorporating covariance (kcov = 0, Models 3 and 6). models introducing tie factor parameters (kgje 20, Models 10 and 22), and the most complex models combining both covariance
(keov = 3) and tie factors (Rite = 20, Models 18 and 30).
The rankings of top competitors remain consistent across all models, reflecting stable performance at the top. In contrast, discrepancies grow more pronounced at lower ranks, where models with additional parameters for ties and covariances produce differing results. Notably, similar models- especially those with comparable configurations tend to yield similar rankings, with occasional exceptions. A detailed analysis of ranking similarities is provided in Appendix E.3.
E.3 QUANTIFYING RANKING SIMILARITY ACROSS MODELS
While the bump chart in Appendix E.2 provides a visual overview of ranking shifts across models, quantifying the degree of similarity between these rankings requires statistical correlation measures. A variety of methods are available, including Pearson's 's correlation, Spearman's 's p. and Kendall's r (Kendall, 1938). Pearson's correlation is most suitable for assessing linear relationships between continuous variables, while Spearman's 's rank correlation generalizes it for monotonic relationships in ordinal data. However, neither offers as direct an interpretation for pairwise ranking comparisons as Kendall's t, which evaluates the ordering of pairs directly, making it particularly well-suited for ordinal data.
Kendall's ranking correlation quantifies the agreement between two ranking orders by comparing the relative ordering of pairs of objects. Given two score vectors, x" (af,... ,,",) and x (af...., .. from the p-th and g-th models, respectively, Tpy reflects the extent to which the pairwise orderings are concordant. Two pairs, (x,, a:)) and (+) x.)) are concordant if they maintain the same relative ordering- i.e., either x! x² and x! x/., or x!! x² and xa > =²/. Conversely, they are discordant if the orderings are reversed. This concordance criterion can be expressed as sgn(a? - x²)) sgn
2²)) 1 for concordant pairs, and -1 for discordant pairs.
The Kendall correlation Tpy is defined as the difference between the probabilities of concordant and discordant pairs and can be empirically obtained by computing the difference of counts for all
30

---

Published as a conference paper at ICLR 2025
Model 30
Model 18
Model 28
Model 14
Model 22
Model n0
Model 6
Model 3
Model 19
Model 7
Model 4
Model
Model 30
Model 18
Model
Modal 34
Model 22
Model 30
Model
Model
Model 19
Model
Model
Model
ch lo-gpt--o--lttes latest
starling-Im mb.....
d
gemini-1 pro......
g.5 turbo....
got-4o-2024.00 2024-003--
phi-3-medium-4..
mixtral-Bx7b-i...
cluddo3-5-son....
gemini-pre
gemini-advance...
yi-34b-chat
11-45-3.14455....
claude-instant-1
-01-4--202-08--6
wizardim-70b
gemini-1.5-pro. 5-pro...
gemini-1. p-pro..
apt-3 09e-3.5-turiie
gpt-4-turbo-20.
gpt-
dbox
mistrallargge...
dpo-70b
Ilama-3.-770b...
phi-3- 1hil3-small-B8
got-4-1106-pre.....
(lama-2-70b-chat
cluude-3-opus....
vicuna-33b
c01-4-0125-pre....
openchat-3.5-010
yrelarge-revveww
starling-lm-bb...
reka-core2024....
nous-hermes-2....
gemini-1.5-fla...
snowflake-arct...
deepseekv2-app...
Ilama2-70b-ste...
gemmm-2-27b-1t
pply 706 online
codpsee..oddrr...
deepseek- lim
yillarge
openchat-3.5
nemetron-4-340..
gemma-1 1-7b-it
glm-4-0520
openhormes-2.55 2.5..
llama-3-70bii....
27
gpt-3.5-turie 5-turbo...
hard-jan-24-ge...
28
mistral-Tb-ie
3-oone..
29
qwenl 5-7b-chat
reka-core-202k
30
solar-10, 7b-in...
90
reka 20s..220....
31
wizardlm-133 13b
91
command-r plus
32
dolphin-2 2.1...
92
qien2 2onn2-2bbinn inst...
33
Ilama 13b- chat
93
gemma-2-9b-it
eepbyr-7b-beta
94
gpt-4-0314
35
phi-3-mini-4kk...
95
glm-4-0116
36
mpt- 30b-chat
96
ewen max 0428
37
phi 1ei-3-mine mini- Ik-
97
claude 3 haiku...
38
pale 7b online
98
deepseek-coder-22 coder-v2
39
bodellama-34k.
99
reka fish-flas-.pr...
40
zophyr- ab-alpha
100
Ilama-3.1 1.1-b............
41
vicuna-13b
101
gpt-4-0613
42
guanaco-33b
102
quen1 5-110b-hha
falcon-180lschat
103
mistraHlarge....
qwen-14b-chat
104
reke-flash-21b...
codellama-70b-...
105
yi-1.5-34b-chat
1am--2-76-chhat
106
Ilama-3-bbinns...
gemma-7b-it
107
1laude 1
phi- phi-3-mini 128...
108
mistral-medium
stripedlyenain...
109
commandir
vicuna-7b
110
qwan1.5 7-ab-chht
palm-2 111
f1sh.
mistral-7b-inn....
112
Bx22
almo- 7b-instruc
113
gemini-pro-eev...
gemma-1
114
claude-2.0
gemma-2b-it
orphyn-orpo-14...
qwenl 5-4b-chat
116
koala-13b
gwen1.5.
chatglm3-6b
gemma
sptfall-
2.1ude-211
mpt-7b-chat
120
Figure E.2: Bump chart comparing chatbot rankings across 12 statistical models, with Model
representing the Elo-based ranking method used in Chiang et al. (2024). Models are arranged with increasing complexity from right to left. Lines track changes in ranking for each chatbot.
31

---

Published as a conference paper at ICLR 2025
9656
97%
989
99%
100%
Kendall T Correlation
Model
Aa
DV
MQ
RX
XX
XX
Va
118
YN
R8
18
AQ
AC
X
DV
WH
RR
AC
AC
MQ
X2
Cov (K...)
X
X
X
X
0
0
0
0
0
3
3
3
3
3
3
3
Tie (Nua)
10
20
X
0
X
20
X
0
10
20
X
10
10
20
20
10
0.0
Linloge Dist.
0.2
0.6
0.8
1.0
Figure E.3: Kendall т ranking correlation matrix for models in Table D.I. The table below cate- gorizes models by configuration: model type (first row), covariance factor kioov (second row), and tie factor krie (third row). In the first row, code names BT, RK, and DV denote Maddey--Teryy. Rao-Kupper, and Davidson models, respectively, with BT indicating the Bradley-Terry model treat- ing ties as half wins and half losses. The model order is determined by hierarchical clustering on Kendall's correlation values, highlighting two main clusters based the presence of covariance, with a further division within the covariance group based on factor koov- A dendrogram below the table illustrates this clustering.
concordant and discordant pairs, normalized by the total number of pairs, (7). as
sgn(x² - x²)) sgn(a - x9)).
(E.1)
Pg
m
Isicism
This correlation ranges from 1, where Tn indicates identical rankings, and Tpg implies complete reversal ranking order (i.e., of < ah implies zς! > 2² and vice versa). The probability that pairwise order x. < atf in one ranking aligns with x? < <x in another is { + 1) (Gibbons Chakraborti, 2003, p. 410).
In this analysis, we compute the Kendall correlation matrix T [Tpp] p. 1,... 30, between each pair of models in Table D.I using Kendall's 7-b method, which also accounts for ties in the scores (Kendall, 1945).
Figure E.3 shows the resulting T matrix, where each cell represents the Kendall correlation between two models. We present only the lower-triangular half of this symmetric matrix for clarity. An adjacent table below the matrix describes each model's type (first row), covariance factor koov (second row), and tie factor his (third row). The codes BT, RK, and DV represent Bradley-TTery, Rao- Kupper, and Davidson models, respectively, while * denotes the Bradley-Tery model with ties treated as half win and half loss. Across our 30 models, the Kendall correlation ranged from 0.96 to 1, indicating overall similarity in rankings but with distinguishable differences driven by model
32

---

Published as a conference paper at ICLR 2025
parameter variations. Further insights into these parameter-driver distinctions emerge by reordering the correlation matrix, as detailed in the next section.
E.4 IDENTIFYING RANKING SIMILARITIES VIA HIERARCHICAL CLUSTERING
To better interpret the distinctions between models, we performed hierarchical agglomerative cluster- ing (Hastie et al., 2009, Section 14.3.12) on the distance matrix J - T, where J is matrix of all ones, converting into a dissimilarity measure. Using optimal leaf ordering (Bar-Joseph et al., 2001), this clustering reorders the rows and columns T to reveal natural groupings based on ranking similarity across models.
The ordering of models shown in Figure E.3 is directly the arrangement produced by hierarchical clustering, visualized in the dendrogram below the figure, which reveals distinct block structure in the T matrix. The clustering first divides the models into two main groups: those without covariance (indicated by columns 1 to 10, shown by the yellow branch) and those with covariance (columns 11 to 30). Within the group of models with covariance, further subdivision occurs, with models having kicov
(columns 11 to 20, shown by the orange branch) forming a distinct sub-block from those with kcovv 3 (columns 21 to 30, shown by the red branch). This hierarchical structure highlights that the presence and type of covariance parameter lcoo are primary factors influencing ranking similarity, more so than the tie factor kie-
While covariance modeling prominently influences ranking consistency, earlier results (see Section and Appendix D) demonstrated that our generalized tie modeling significantly enhances inference and predictive accuracy. This dual impact-covariance structure shaping ranking alignment and generalized tie modeling improving model fit and accuracy-illustrates the complementary strengths of these two generalizations paired comparison models.
APPENDIX F RELATIONSHIP BETWEEN LLM CHARACTERISTICS AND SCORES
This section explores the relationship between the scores derived from our ranking framework and three key characteristics large language models: the number of parameters, computational budget (FLOPs), and dataset size. Using data from Epoch AI (2022), which provides these attributes for various LLMs, we matched these models with those evaluated in our analysis. To ensure reliability, filtered the data include only models with confident values for these characteristies. Among the matched models, 31, 27, and 24 LLMs had available and confident values for the number of parameters, FLOPs, and dataset size. respectively. Figure F.l presents scatter plots illustrating these relationships, with the abscissa displayed on logarithmic scale and dashed regression lines capturing the linear trends. The scores, shown on the ordinate, are derived from our generalized Rao- Kupper model corresponding to model 18 of Table D.1.
Table F.1 summarizes the results independent OLS regressions performed for the logarithm of each characteristic against the scores. The table presents the number of LLMs included in each regression, along with the regression coefficients and their standard errors, p-values, coefficient of determination (IR?), and Pearson correlations (r). The results demonstrate consistent positive associations between all three characteristies and model scores. Computational budget (FLOPs) and dataset size exhibit the strongest associations, with R² 0.59 and 0.77 for FLOPs, and R² 0.50 and 0.70 for dataset size. The number of parameters shows weaker association, reflected by R² 0.15 and r = 0.38, though it remains statistically significant.
Table F.1: Regression results examining the relationship between LLM characteristics and scores.
Variable
# LLMs
Coeff' (x (0100)
p- VaVuu
R²2
Pearson
Number of Parameters
31
1.76
(=0.79)
3.4144%
0.15
0.38
Training Compute (FLOPs)
27
2.98
0.50)
0.0003%
0.59
0.77
Dataset Size
24
5.70
(=1.23)
0.0124%
0.50
0.70
33

---

Published as a conference paper at ICLR 2025
Effect of Number of Parameters
Effect of Training Compute Budget (FLOPs)
Effect of Dataset Size
Matri
Larea 22-22
Uaee 31-238 Mada
36-4088
Loeo 4353
Gannna to
Paka Auu
Liar 200
Buu Codeng 22g
Seen
Gannra 90
Ques M Rousyy Messn
2425
Despdesio V2
Flush
Gereay 270
Lixe
Doennt T2a
Llara 36
Uimra Stp
DARK
Larra Sim
2.50
Rodter Cesen zulliaee
Steediee Luge ort
Oven
30P1:306 (har))
date 2018
Gyen-188 1985
Grewna
Srewns TR
GLM
0154-78
OL/6-1D
55%
9966
10th
10
10"1
Number of Parameters
Training Compute Budget (FLOPs)
Dataset Size
Figure F.l: Scatter plots showing the effect of LLM characteristics scores. The abscissa represents the logarithmic scale of the characteristics: (left) number of parameters, (middle) computational budget (FLOPs), and (right) dataset size. The ordinate (y-axis) is shared across all panels and shown only for the left panel. Each point is labeled with the corresponding LLM name, and the regression lines are shown in dashed black.
Our findings align with prior studies on the scaling laws of LLMs. Kaplan et al. (2020) emphasize that model performance depends strongly on scale, encompassing model size, dataset size, and com- putational budget. Hoffmann et al. (2022) further highlight the importance of balancing these factors, demonstrating that compute and dataset size play pivotal roles in maximizing performance within fixed budgets. Consistently, our analysis shows that scores improve across all three characteristics, with particularly strong trends observed for computational budget and dataset size.
The comparatively modest association observed for the number of parameters in our analysis reflects the diminishing returns noted the literature when model size is increased without proportional scaling of compute and data resources. This observation underscores the importance of balanced scaling for optimal performance, as emphasized by prior studies.
Given the sparsity of available data for proprietary models in analysis, these conclusions should be interpreted with caution. Further studies incorporating more comprehensive datasets could provide additional insights into the interplay between computational budget, dataset size, and the number of parameters in driving LLM performance.
APPENDIX G IMPLEMENTATION AND REPRODUCIBILITY GUIDE
We developed a Python package leade rbot that implements the methods presented in this paper. The package allows users to reproduce the numerical results, evaluate model fit, and explore model generalization performance. Below, we provide examples of using Leaderbot for common tasks such as model training, evaluation, and visualization. The full documentation, including further functionality and customization options, is available online.
G.1 MODEL TRAINING AND VISUALIZATION
Listing G.I demonstrates the basic usage of leaderbot for training a statistical model and visu- alizing results. In this example, we replicate Model 23 from Table D. using the Davidson class, which includes both tie modeling and covariance. The model is instantiated with parameters kaov = for covariance and kiie 0 for the tie factor model. Once instantiated, the model is trained using the BFGS optimization method on the dataset. While leaderbot. ships with the dataset used in
3 leaaerbott is available for installation from PyPl at hattts::/pppp/ http:://ppppppoppoopseeeeeeeeeee Leaeee//eeddrbott.... Documentation and usage instructions can be found at https: //1 leaderbat..org The source code is available on GitHub https: : http://ttttt.cooooo wwwoo///uuua//////o//o/// leddebbt...
34

---

Published as a conference paper at ICLR 2025
this paper, the load function also allows users provide URL load external dataset. See documentation for details.
After training, users perform inference and prediction, compute the loss function and Jacobian at the optimal parameters or any specified parameter values, and generate leaderboard tables. The package also includes visualization tools replicate figures such as the match matrix (Figure 1), kernel-PCA and MDS plots (Figures and C.1), and hierarchical clustering (Figure C.2), among others.
Listing G.I: Basic usage leaderbot for model training and visualization of results.
Tnstail leaderbet with "pip install leaderbot"
import leaderbot as lb
Load the default dataset shipped with the package data data load
Create a Davidson model with covariance factor Rcov = (diagonal covariance) and tie factor Aiio =0. This corresponds to Model 23 in Table D.1 model 1b. lb. (data, k_cov=0, k_tie-0)
Train the model model.tran (method= BEGS', max bax_iter=1500 col=le-8)
Make inference and prediction
probabilities model. (data) prediction = model.predict (data)
Compute the loss function -E(0), its Jacobian -8E(0)/00, and Bessian -VoV) loss, Jac model. (retarn_ 20 hess model.fisher
Plot marginal probabilities (similar to Figure D.1)
25 Generate a plot for compotitor scores (simiter to Figure R.1)
model.plot_scoree (max_ rank_5)
Rank competitore based their scores, print leaderboard rank = model. rank ()
30
model. leader.radd
Visualize correiation using Kernel PCA projected in 3D space (similar to Figure 2)
Use method-' mds! generate an plot (similar Pigure c.1)
model.map_dde distance (max_rank-40, method- kpea', dim-'
35
Generate match matrix plot for observed and predicted win/loss probabilities and tie probabilities (similar to Figure model sidel.mateen .ww.tcmmaartttttt.cceaaaa (max_rank=25, win_range= (0.2, 0.6], tie_range=(0.15 0.4])
40 Perform hierarchical clustering for the top 100 compet itors (simi Tar to Pigure 0.2) 1odel cluster max__aank=100)
G.2 MODEL EVALUATION: FIT AND CONSISTENCY METRICS
Listing G.2 demonstrates the evaluation of model fit and consistency for five selected models, chosen from the broader set of 30 models discussed in Table D.1, including the original Bradley-Terry model as well as original and generalized versions the Rao-Kupper and Davidson models.
In this example, each model is trained on the full dataset to evaluate goodness of fit. The script produces a bump chart similar to Figure E,2, comparing the rankings generated by the five models. Additionally, it provides tables similar Table D.1 and Table D.2, displaying model selection and
35

---

Published as a conference paper at ICLR 2025
goodness-of-fe metrics. These metries enable users to analyze and compare model consistency in training performance.
Listing G.2: Evaluating model fit and consistency metries across models in leaderbot.
import leaderbot 1b
Load detaset
data - ib..ata. load ()
List models 2, 12, 23, and from Table D.1 models
1b. 1b.models. Seredees..Baddeyyee Braddeyyerryyydeeeee (data, k_cov=Nene),
1b. 1b.models. BradB wradeeyeerryydae (data, k_cov=0),
1b. lb.models. m...meels (data, k_cov=0, k_tie=0), lb.models Rac.lsseaaeppprrree (data, k_oov=0, k_tie=1), 1b. lb.medels.e Davidsoo (data, k_cov=0, k_tie-0),
1b. lb.models.. Davidson (data, k_cov=,, k_tie-1)]
Pre-train the models for model models: model.train!)
Compare model rankings, generating bomp chert like Pigure F.2 1b. eval haaluateee cowpare_mankeeeeco (models, rank_range= [1, 60])
Evaluate model-selection metrics, similar to Table D.1 matrics - ab.evaluate.moder selectiono (models, xeport-True)
Svalcate models for goodness of fit, similar Table D.2 gof_metrics lb.evaluate wow.ttess.oooorsso_ssesesese fftfitffff... (models, matric=' RMSE', report=True)
G.3 MODEL GENERALIZATION: PERFORMANCE ON TEST DATA
Listing G.3 demonstrates the evaluation of model generalization using 90/10 train-test split, where the same five models from the previous listing are trained on 90% of the data and tested on re- maining 10%. The resulting RMSE, KLD, and JSD metrics, displayed in table similar to Table D.3, offer insight into each model's 's predictive accuracy and robustness on unseen data.
Listing G.3: Evaluating model generalization using train-test split in leaderbor rbot.
import leaderbot lb
Load detaset
data - .b.data.load ()
split data into training and test sets
training_datal tatt_ 1b. data.appiii (data, test_ratio seed-20)
List models 1, 2, 11, 12, 23, and 24 from Table D.1
models
lb.medels. Srredees..raaleeee. Bradleyeeryy (training data, k_cov-None), www.mdeess.Bradeyylle..
(training_date, data, k_cov-C), lb. lb.models. Rackelpee (trainingg_aadd k_cov-0, k_tie-0), lb. ab.models. www.meels....uupeerreeeeee Rocxupper
k_co==0, k_tie=l), 1b. wwdelmlll.... Saweelss.DDiiddonttttttoo (taiiiingggaiaaaaaaa k_oo=0, k_tie=0), 1b. ab.models... Dww.lsa.aidooooooaao (taiiiiooottraiiing_dddaaaaa k_sov=0, k_tie=1)]
Eva 1 late models for genera 1 ization on test deta, similar to Table D.3
gan_metrics 1b.evaluate. general ization (models, s, test test_dattet dat_ddaat__daaaaeeee _attatt,d_
train=True, metric=/RMSE' report-Irue)
36


Reviews:
[{"review_id": "nCi0LeNG17", "reviewer": "Reviewer_ztF7", "summary": "This paper presents a statistical framework for ranking LLM-based chatbots. It addresses the limitations of existing methods like the Elo rating system by incorporating ties and covariance structures. The proposed apply well-established statistical models to properly account for ties within an axiomatic framework and introducing factor analysis. Experiments on the Chatbot Arena dataset show improved accuracy and insights into chatbot rankings and relationships.", "questions": "1. Is this method only useful to chatbot arena or it is useful to all Elo ratings?\n2. We need another figure to see the ranking difference of original chatbot arena and this methods. Also the figure 1 showing the result of PCA analysis is confusing.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 1, "strengths": "1. The proposed method can account for ties in an axiomatic framework. This improves not only tie prediction but also enhances win-loss inference.\n2. The mathematical analysis is comprehensive and thorough.\n3. This work has an open source python package which is easy to use.", "weaknesses": "1. For model ranking, it has no \"ground truth\". Therefore, it is hard to convince others under this method, the ranking is more accurate.\n2. The evaluation is highly dependent on the Chatbot Arena dataset which makes this work a slight improvement on chatbot arena. Therefore, the impact of this work is limited.\n3. As for chatbot arena, a simple enough ranking rule is more important if the users are common users. This work will make the rule too complicated for them to understand and then reduce the impact of chatbot arena.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KBMZU9u7mx", "reviewer": "Reviewer_FnUn", "summary": "The authors propose an improved statistical model for ranking large language models (LLMs) on the Chatbot Arena dataset to enhance the traditional Elo rating system. Paper points out that the Elo system has limitations in handling ties and capturing relationships between models. To address these issues, paper introduces the Rao & Kupper and Davidson models, as well as a novel factor model to better capture the complexity of ties between different models, thereby improving prediction performance. Finally, they provide a Python package, “Leaderbot,” to reproduce the statistical model and support further experiments.", "questions": "1. In the treatment of pairs and unique pairs, have the authors considered introducing semantic clustering to their model, as opposed to solely using statistical techniques? Additionally, could they clarify the reasoning behind not exploring semantic relationships within pairs?\n2. Have the authors considered methods other than PCA to further analyze the correlations between models in the results? The current approach does not sufficiently reveal the underlying correlations between LLMs in an intuitive way.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors identify symmetry issues in the likelihood function of traditional models, which could lead to instability in parameter estimation. To address this, they propose symmetry constraints that ensure stable parameter estimation, thereby enhancing the model’s optimization performance and interpretability;\n2. They effectively address the issue of ties, which is a limitation of the existing Elo system, and make optimizations to handle this;\n3. They provide a Python package that allows for the reproduction of the paper's results and supports further research.", "weaknesses": "1. The authors use multiple models to rank models, showing that high-ranking models exhibit greater consistency than lower-ranking ones. However, they do not provide further analysis, such as examining the specific characteristics of models that initially show ranking inconsistencies;\n2. The authors’ work focuses on optimizing the ranking model but lacks subsequent analysis. Additional insights, such as a more in-depth examination of correlations between LLMs or an analysis of the differences between Leaderbot rankings and Chatbot Arena Elo rankings in relation to model characteristics, would enhance this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2G5IzqJquM", "reviewer": "Reviewer_cyAq", "summary": "This paper addresses a novel problem in the evaluation of large language model (LLM)-based chatbots by proposing an advanced statistical framework that builds on existing methods used in the Chatbot Arena setting. The framework integrates well-established models, such as Rao-Kupper and Davidson, to account for ties in a rigorous axiomatic framework, and introduces Thurstonian representations to model covariance structures between competitors. These additions allow for more nuanced insights into chatbot rankings and performance consistency. A Python package, leaderbot, is also provided for reproducibility and ease of experimentation.", "questions": "1. The paper mentions that high-ranking models show greater consistency than lower-ranked ones. Could you expand on this observation? For example, are high-ranking models generally larger or more sophisticated in architecture? And at what ranking position does this shift in consistency typically occur?\n2. How is the effectiveness of handling ties quantified across different statistical models? Is there an analysis of which models are more favorable to certain LLMs, and if so, what might explain these differences?\n\nIf these questions can be addressed, the paper could potentially be rated one point higher.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. **Pioneering Approach**: This paper tackles a novel and important problem in LLM evaluation, presenting a unique perspective on using advanced statistical models to refine the ranking process in chatbot comparisons. Its approach to systematically modeling ties and latent structures sets a new precedent for evaluating LLM-based chatbots and offers fresh insights that go beyond traditional ranking methods.\n2. **Innovative Use of Thurstonian Representations**: By integrating Thurstonian models and introducing covariance structures, the paper offers a groundbreaking method for capturing relationships among LLMs. This enables a deeper analysis of model consistency and performance that extends beyond simple rankings, marking a notable advancement in chatbot evaluation techniques.", "weaknesses": "1. While the paper introduces an alternative framework, it does not clearly discuss why Arena’s Elo-based approach is insufficient beyond the issue of ties. Additional insight into Arena’s limitations in capturing competitive dynamics or certain statistical shortcomings would strengthen the argument for this new model.\n2. The authors mention consistency in high-ranking models and variability in lower-ranking models, but do not explore further distinctions within these groups. For example, identifying specific characteristics (e.g., model size) associated with high consistency could provide more actionable insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)