Prediction-Powered Ranking of Large Language Models
Authors: Ivi Chatzi
Eleni Straitouri
Suhas Thejaswi
Manuel Gomez Rodriguez
Venue: NeurIPS 2024 poster (2024)
Keywords: Large language models
LLM evaluation
LLM ranking
Primary area: evaluation

Abstract: Large language models are often ranked according to their level of alignment with human preferences---a model is better than other models if its outputs are more frequently preferred by humans. One of the popular ways to elicit human preferences utilizes pairwise comparisons between the outputs provided by different models to the same inputs. However, since gathering pairwise comparisons by humans is costly and time-consuming, it has become a common practice to gather pairwise comparisons by a strong large language model---a model strongly aligned with human preferences. Surprisingly, practitioners cannot currently measure the uncertainty that any mismatch between human and model preferences may introduce in the constructed rankings. In this work, we develop a statistical framework to bridge this gap. Given a (small) set of pairwise comparisons by humans and a large set of pairwise comparisons by a model, our framework provides a rank-set---a set of possible ranking positions---for each of the models under comparison. Moreover, it guarantees that, with a probability greater than or equal to a user-specified value, the rank-sets cover the true ranking consistent with the distribution of human pairwise preferences asymptotically. Using pairwise comparisons made by humans in the LMSYS Chatbot Arena platform and pairwise comparisons made by three strong large language models, we empirically demonstrate the effectivity of our framework and show that the rank-sets constructed using only pairwise comparisons by the strong large language models are often inconsistent with (the distribution of) human pairwise preferences.

Prediction-Powered Ranking of Large Language Models
Ivi Chatzi
Eleni Straitouri
Max Planck Institute for
Max Planck Institute for Software Systems Kaiserslautern, Germany estraitouri dww.-sws.......... org
Software Systems Kaiserslautern, Germany ichatzi@mpi- sws org
Suhas Thejaswi
Manuel Gomez Rodriguez
Max Planck Institute for Software Systems Kaiserslautem, Germany the aswi@mii-sss org
Max Planck Institute for Software Systems Kaiserslautern, Germany manuelgr0mpi. -wws org
Abstract
Large language models are often ranked according to their level of alignment with human preferences-a model is better than other models if its outputs are more fre- quently preferred by humans. One of the popular ways to elicit human preferences utilizes pairwise comparisons between the outputs provided by different models to the same inputs. However, since gathering pairwise comparisons by humans is costly and consmmnn,,,,g, it has become common practice gather pairwise comparisons by strong large language model-a model strongly aligned with human preferences. Surprisingly, practitioners cannot currently measure the un- certainty that any mismatch between human and model preferences may introduce in the constructed rankings. In this work, we develop a statistical framework bridge this gap. Given (small) set of pairwise comparisons by humans and a large set of pairwise comparisons by a model, our framework provides a rank-set-a set of possible ranking positions-for each of the models under comparison. More- over, it guarantees that, with a probability greater than or equal to user-specified value, the rank-sets cover the true ranking consistent with the distribution of hu- man pairwise preferences asymptotically. Using pairwise comparisons made by humans in the LMSYS Chatbot Arena platform and pairwise comparisons made by three strong large language models, we empirically demonstrate the effectivity of our framework and show that the rank-sets constructed using only pairwise comparisons by the strong large language models are often inconsistent with (the distribution of) human pairwise preferences.
Introduction
During the last years, large language models (LLMs) have shown remarkable ability to generate and understand general-purpose language [1]. As result, there has been an increasing excitement in their potential to help humans solve variety of open-ended complex tasks across many application domains such as coding [2], healthcare [3] and scientific discovery [4], to name a few. However, evaluating and comparing the performance of different LLMs has become very challenging [5]. The main reason is that, in contrast to traditional machine learning models, LLMs can solve a large number of different tasks and, in many of these tasks, there is not unique, structured solution. As
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

consequence, there has been a paradigm shift towards evaluating their performance according to their level of alignment with human preferences- model is better than other models if its outputs are more frequently preferred by humans [6-10].
One of the most popular paradigms to rank set of L.LMs according to their level of alignment with human preferences utilizes pairwise comparisons [10-17]. Under this paradigm, each pairwise comparison comprises the outputs of two different models picked uniformly at random to an input sampled from given distribution of inputs. Moreover, the pairwise comparisons are used to rank the models with a variety of methods such as the Elo rating [18-22], the Bradley- Terry model [10, 17, 23] or the win-rate [12, 17, 23]. While it is widely agreed that, given sufficiently large set of pairwise comparisons, higher (lower) ranking under this paradigm corresponds to better (worse) human alignment, there have also been increasing concerns that this paradigm is too costly and time- consuming to be practical, especially given the pace at which models are updated and new models are developed.
To lower the cost and increase the efficiency of ranking from pairwise comparisons, has become a common practice to ask a strong LLM-a model known to strongly align with human preferences- perform pairwise comparisons [24-33]. The rationale is that, if a model strongly aligns with human preferences, then, the distributions of pairwise comparisons by the model and by the human should in principle match [24, 27, 34]. Worryingly, there are multiple lines of evidence, including our experimental findings in Figure 3, showing that the rankings constructed using pairwise comparisons made by a strong LLM are sometimes different to those constructed using pairwise comparisons by humans [12, 14-16, 19, 35, 36]. questioning the rationale above. In this work, we introduce a statisti- cal framework to measure the uncertainty in the rankings constructed using pairwise comparisons made by a model, which may be introduced by mismatch between human and model preferences or by the fact that we use a finite number of pairwise comparisons.
Our contributions. Our framework measures uncertainty using rank-sets-sen of possible ranking positions that each model can take. If the rank-set of model is large (small), it means that there is high (low) uncertainty in the ranking position of the model. To construct the rank-sets, our framework first leverages a (small) set of pairwise comparisons by humans and large set of pairwise comparisons by a strong LLM to create a confidence ellipsoid. By using prediction-powert this confidence ellipsoid is guaranteed contain the vector of (true) probabilities that each model is preferred over others by humans- the win-rates-aa wwith a user- eser-spec coverage probability 1 - a. Then, it uses the distance between this ellipsoid and the hyperplanes under which pairs of models have the same probability values of being preferred over others to efficiently construct the rank-sets. Importantly, we can show that, with probability greater than or equal to 1 - at the constructed rank-sets are guaranteed to cover the ranking consistent with the (true) probability that each model preferred over others by humans asymptotically. Moreover, our framework does not make any assumptions on the distribution of human preferences nor about the degree of alignment between pairwise preferences of humans and the strong LLM. Experiments on pairwise comparisons made by humans in the LMSYS Chatbot Arena platform [28) and pairwise comparisons made by three strong LLMs, namely GPT 3.5, Claude and GPT 4, empirically demonstrate that the rank-sets constructed using our framework are more likely to cover the true ranking consistent with (the distribution of) human pairwise preferences than the rank-sets constructed using only pairwise comparisons made by the strong LLMs. An open-source implementation of our methodology as well as the data on pairwise preferences of strong LLMs used in our experiments are available at setttiitioo--oweee----aaaee
inference [37-39],
Further related work. Our work builds upon recent work on prediction poweeeddd inference, ranking under uncertainty, and ranking of LLMs.
Prediction-pouue poweerdd inference [37-39] is recently introduced statistical framework to obtain valid p-values and confidence intervals about a population-level quantity such as the mean outcome or regression coefficient using a small labeled dataset and large unlabeled dataset, whose labels are imputed using a black-box machine learning model. However, our work is the first to use peediction-poweree inference (as a subroutine) construct rank-sets with coverage guarantees. In this context, it is worth acknowledging that very recent work by Suad-Falcon et al. [40] has used peediction-poeece inference to construct (single) rankings, rather than rank-sets. However, their rankings do not enjoy coverage guarantees with respect to the true ranking consistent with (the distribution of) the human preferences. Moreover, an independent, concurrent work by Boyeau

---

et al. [23] has also used prediction-powee powered inference to construct (single) rankings based on the estimated coefficients of Bradley-Tere model. However, the estimated coefficients come with large. overlapping confidence intervals, which would have led to uninformative rank-sets, had the authors used them to construct rank-sets.
The vast majority of the literature on ranking under uncertainty has focused on confidence intervals for individual ranking positions [41-47]. Only recently, a paucity of work has focused on joint measures of uncertainty for rankings [48-51]. Similarly as in our work, this line of work also seeks to construct rank-set with coverage guarantees. However, in contrast to our work, it estimates the quality metric (in our work, the probability that an LLM is preferred over others) and the confidence intervals separately for each of the items (in our work. LLMs) using independent samples. As a consequence, it needs to perform multiple comparison correction to create the rank-sets.
In recent years, there has also been flurry of work on ranking L.MSs using benchmark datasets with manually hand-crafted inputs and ground-truth outputs [52-58]. However, it has become increasingly clear that oftentimes rankings derived from benchmark datasets do not correlate well with rankings derived from human preferences- an improved ranking position in the former does not lead to an improved ranking position in the latter [12-14, 17, 26]. Within the literature on ranking LLMs from pairwise comparisons, most studies use the Elo rating system [18-22], originally introduced for chess tournaments [59]. However, Elo-based rankings are sensitive to the order of pairwise comparisons. as newer comparisons have more weight than older ones, which leads to unstable rankings [15]. To address this limitation, several studies have instead used the Bradley-Terry model [10. 17. 23]. which weighs pairwise comparisons equally regardless of their order. Nevertheless, both the Elo rating system and the Bradley-Tery model have faced criticism, as pairwise comparisons often fail to satisfy the fundamental axiom of transitivity, upon which both approaches rely [15, 60). Recently, several studies have used the win-rate [12. 17, 23], which weighs comparisons equally regardless of their order and does not require the transitivity assumption, but requires humans to make pairwise comparisons between every pair of models. In our work, we build upon the win-rate and lift the above requirement by using pairwise comparisons made by strong LLM.
2 LLM Ranking under Uncertainty
Let M be set of large language models (LLMs) and P(Q) be a distribution of inputs on a discrete
set of imputs Q. Moreover, assume that, for each input P(Q)." each model m E M may provide an output r (R] Q = q) from discrete of outputs R. Further, given two outputs r, r' E R from two different models, the (binary) variables w, w' ~ P(W, W" Q = g. R = r, indicate whether a human prefers r over r' (w 1, w' 0) or viceversa (w 0, w 1). In the case of a tie, then w = w' = O. In what follows, we use m(r) and m(r') to denote the models that provide outputs r and r' respectively, and without loss of generality, we assume that the output is shown first. Then, our goal is to rank all models according to the (empirical) probability On that their outputs are preferred over the outputs of any other model picked uniformly at random.
r')
To this end, we start by writing the probability Om as an expectation over the distribution of inputs, outputs and pairwise preferences:
8m
Σ
EQ
E ER-P Ew |1, R, R']
k-1
(mm
(I)
BR.PA FW [ (W' IW'IQ,1 R, R']]
where note that the order of the pairs of outputs is picked at random. Next, following previous work [48, 50]. we formally characterize the ranking position of each model m E M in the ranking induced by the probabilities 6m using rank-set [l(m), u(m)). where
7(m) = 1 + Σ
160m < 6ma and u(m) = k
Σ
1(em. > b,
MEMim
(EM)(m
are the lower and upper ranking position respectively and smaller ranking position indicates better alignment with human preferences. Here, note that it often holds that Om + 0,2 for all in E M') {m} and then the rank-set reduces to a singleton, i.e., 1(m) = u(m).
We denote random variables with capital letters and realizations of random variables with lower letters.
3

---

In general, we cannot directly construct the rank-sets as defined above because the probabilities 6n are unknown. Consequently, the typical strategy reduces first gathering pairwise comparisons by humans to compute unbiased estimates of the above probabilities using sample averages and then construct estimates [((m), a(m) of the rank-sets using Eq. 2 with 6. rather than 0m- Under this strate- gy, if the amount of pairwise comparisons we gather is sufficiently large, the estimates the rank-sets will closely match the true rank-sets. However, since gathering pairwise comparisons from humans is costly and consuming, it has become a very common practice to gather pairwise comparisons w u' by strong LLM, rather than pairwise comparisons
w' by humans [12-14, 28, 29, 31, 61- 64], and then utilize them to compute unbiased estimates of the probabilities 0. that the outputs provided by each model is preferred over others by the strong LLM, which can be written in terms of expectations follows:
0m k:
BQ
[Ewl [iIWIQ, R, R']
(EM
(3)
Pm. I (Q,
In general, one can only draw valid conclusions about using (an estimate of) θ if the distribution of the pairwise comparisons by the strong LLM P(W, W 1 = q, the distribution of pairwise comparisons by the humans P(W, any q E Q and r, r, E R. However, there are multiple lines of evidence showing that there is a mismatch between the distributions, questioning the validity of the conclusions drawn by myriad of papers. In what follows, we introduce statistical framework that, by complementing a (large) set of N + pairwise comparisons vi, w' by strong LLM with small set of pairwise comparisons w, w' by humans, is able to construct estimates (i(m), a(m) of the rank-sets with provable coverage guarantees. More formally, given a user-specified value a E (0, 1), the estimates the rank-sets satisfy that
R R = R' =r) closely matches = q,
R' = r') for
lim IP (n [(m),, u(m)] S ((mm), i(m)] ME
2 1 11-6
To this end, we will first use prediction- inference [37, 38] to construct a confidence ellipsoid that, with probability - a, guaranteed contain the (column) vector of (true) probabilities = (f M.... Then, we will use the distance between this ellipsoid and the hyperplanes under which each pair of models m, m E M have the same probability values of being preferred over others, to efficiently construct the estimates [((m), ú(m) of the rank-sets.
3 Constructing Confidence Regions with Prediction- Inference
Let the set Dx (gg Ti, T/, m((r), ( ws w comprise pairwise comparisons by strong LLM to N inputs and the set D, = [(: Tey T/ m(rr), m(?)), wa, w/, w. " comprise pairwise comparisons by the same strong LLM and by humans to inputs, with < N. In what follows, for each pairwise comparison, we will refer to the models m(r) and m(r') that provided the first and second output using one-hot (column) vectors and m', respectively. Moreover, to summarize the pairwise comparisons in D and D,.. we will stack the one-hot vectors m and m' into four matrices, My and M'y for Dx and M, and M for Dn, where each column corresponds one-hot vector, and the indicators w and i into six (column) vectors, un and wn for D and th,, w W., and w', for D...
We assume that each model E .M participates in at least one pairwise comparison in both Dy and Dn..

---

Algorithm I: It estimates and E using prediction-powereed inference.
Input: k. DN. Dn
Output: 0,
Un, VNN, MN, M + SUMMARIZE(DN,
Wn, Wh. Wn; w.:: Mn. M,, =-UUMARIZE(Dn;K)
À + LAMBDA UN V MN, MM. Wn: w/n Way ai Mn, M;)
// Algorithm 4
(LM + MK) IN) (1)) (Mx ADON + MN Xwx(x)
b + [a ((M, + M)),)) @I,) (Ma(Xw. - w.) + M. (Aai), - w/))
+ a a
1/ prediction-porentt estimate (Eq. 5)
((1a(200 - M @ MN + (Ix(Aely - MM a) M2) M. b) o M. + (1ε(Au0', w/ M, b) M. // estimate of covariance (Eq. 7)
(1x(200 wr
Σ + XI AA + + BBBB'
10 return 0,
Then, building upon the recent framework of prediction-powenn powered inference [37], we compute an unbiased estimate & of the vector (true) probabilities
θ-
L ((MN + M2x IN) ② IIs)
(MN AWn + M'N >eo'w)
1 ((Mn + M,) 1n) C Lk)
(M,()u, w.) M2 (Aui, 2o,,) (5)
where 1a denotes dddimensioonll column vector where each dimension has value 1 and Tı denotes k-dimensional identity matrix. Here, note that the first term utilizes the pairwise comparisons by the strong LLM from y to compute an unbiased estimate the vector probabilities Ó defined in Eq. 3 using sample averages. and the second term b utilizes the pairwise comparisons by the strong LLM and by humans from D,, to compute an unbiased estimate of the difference of probabilities defined in Eqs. 1 and 3, also using sample averages. The parameter E [o, 1] weighs the comparisons tù úò differently than the comparisons w, w. Details why this can be useful and on the selection of X are in Appendix B.2.
Further, as shown in Angelopoulos et al. [38], the difference of probabilities Ô 0 converges in distribution to a k-dimensional normal (0, E), where Σ = E[ () T] and thus the confidence region
Co 6 (x- (x-o)²
Σ
x 0) s 1
(6)
X
where Σ is an empirical estimate of the covariance matrix Σ using pairwise comparisons from D and D,,. i.e.
E = No
AAT + n
BB",
with
A ((1c(>wx MNa)T) o My + (1x(20'y MN a) 3 My), (14(Ain, w, M, b) o M, + (1=(>e0, w/, M, b) の My
and x is the quantile of the x² distribution with h degrees of freedom, satisfies that P C.) a.
(8)
Algorithm summarizes the overall procedure to compute á and E, which runs in O(12(N + n)) time.
5

---

Algorithm/ 2: It constructs ((m), {(m)] for all m E M
Input: M. DN. Dn,
Output: {[((m), i(m))mmM
k< <- |
0, O.E-
for m do CONTIDENCE- DN, Dn)
1/ Algorithm
[(m) + L, ū(m) e- k
for mh M {m) do
d - Am-
V2 V
Em, - 2mm mx
// Eq. 10
> 0 and 0m 0n then
[(m) [(m) +1
else d and 8m A. then
10
ú(m) + ū(m)
return ([(m),, ([mm)]mmEE
4
Constructing Rank-Sets with Coverage Guarantees
For each pair of models m, m E M such that m o m, we first define hyperplane H,, S R as follows:
Hm,m = {ac E R Im = }.)
(9)
Then, for each of these hyperplanes Hm we calculate the distance d(Ca; Hm,m) between Hm. and the confidence region Co defined by Eq. 6, i.e.,
10m - -0ml
(mmm + Em, )
d(Ca. Hm.
(10)
V2
where E is the empirical covariance matrix defined by Eq. 7.
Now, for each pair of models m, m' M, can readily conclude that, if the distance d(Ca. Hm,n) > 0, then, the confidence region Ca either lies in the half-space of R where I'm > z if θ.. > 9n or it lies in the half space of R* where Im Fm if 6m 6 Building upon this observation, for each model rank-sets [[(m), u(m)]:
M, construct the following estimates (i(m), ū(m) of the
((m) = 1+
Σ
1d(Can, Hm,m) > 0} 110m. 5
(EMi(m)
(11)
a(m) = k
Σ
1dd(Co, Hmmm) > 0} 1(0m mm
mEM)(()
Importantly, using a similar proof technique as in Lemma 1 in Neuhof and Benjamini [48], we show that the above rank-sets estimates enjoy provable coverage guarantees with respect the rank-sets [mm),, u(m)] induced by the probabilities 0 that the outputs of each model is preferred over any other model by humans (proven in Appendix A):
Theorem 4.1 The estimates ((m)), a(m) of the rank-sets defined by Eq. 11 satisfy that
( [mm), u(m)] S [((m), ú(m)) 2 1 MEM
lim IP
(12)
Algorithm 2 summarizes the overall procedure to construct the rank-sets (((m), a(m)) for all m E M. which rums in ( n)).

---

5 Experiments
We apply our framework to construct rank-sets for 12 popular LLMs using pairwise comparisons made by humans in the LMSYS Chathot Arena platform² and pairwise comparisons made by three strong LLMs. We show that the rank-sets constructed using our framework are significantly more likely to cover the true ranking consistent with (the distribution of) human pairwise preferences than the rank-sets constructed using only pairwise comparisons made by the strong LLMs.
Experimental setup. Our starting point is the Chatbot Arena dataset [12], which comprises 33,481 pairwise comparisons made by 13,383 humans about the responses given by 20 different L.LSs to 26,968 unique queries. In what follows, we refer to each pair of responses to a query by two different LLMs and the query itself as an instance. As an initial pre-processing, we filter out any instance whose corresponding query is flagged toxic multiturn. Then, we gather pairwise comparisons made by three strong LLMs, namely and Claude-3-Opuse 0pus-202402299 (CL3), about all the prre-procsssed) instances from the Chatbot Arena dataset. To this end, we use (almost) the same prompt as in Zheng et al. [12], which instructs each strong L.LM to output option 'A' ('B') it prefers the response of first (second) L.M,, or option 'C' if it declares tie. Further, we filter out any instances for which at least one strong L.LM provides a verbose output instead of 'A', 'B', or 'C', and focus on a set of LLMS with at least 96 pairwise comparisons between every pair of LLMs in the set. After these pre-processing steps, we have 14,947 instances comprising 13,697 unique queries and 12 different LLMs and, for each instance, we have one pairwise comparison made by a human and three pairwise comparisons by the three strong LLMs. Refer to Appendix for more information regarding the 12 LLMs, the number of pairwise comparisons between every pair of LLMs, and the prompt used to gather pairwise comparisons made by the three strong LLMs.
1...5-ttrb--0222 (GPT3.5) OPT-4-01125prrviee (GPT4)
To draw reliable conclusions, in each experiment, we construct rank-sets 1,000 times and, each time, we use a random set of N + n = 6,336 instances with an equal number of instances per pair of models, out of the 14,947 instances. The values of N and n vary across experiments and they define two random subsets, also with an equal number of instances per pair of models.
Methods. In our experiments, we construct rank-sets using the following methods:
a) BASELINE: it constructs (unbiased) rank-sets using the pairwise comparisons made by hu- mans corresponding to the random set of N + n instances via Algorithms 2 and 3, shown in Appendix B.1. The constructed rank-sets are presumably likely to cover the true rank-sets.
b) LLM GPT4, LLM GPT3.5 and LLM CL3: they construct (possibly biased) rank-sets using the pairwise comparisons made by one of the three strong LLMs corresponding to the random set of N + n instances via Algorithms 2 and 3.
c) PPR GPT4, PPR GPT3.5 and PPR CL3: they construct (unbiased) rank-sets using pairwise compar- isons made by one of the three strong LLMs corresponding to the random set of N + n instances and pairwise comparisons made by humans corresponding to the random subset of instances via Algorithms 1 and 2.
d) HUMAN ONLY:
it constructs (unbiased) rank-sets using the pairwise comparisons made by humans corresponding to the random subset of n instances via Algorithms 2 and 3.
In the above, note that a). b) and d) use linear regression to construct confidence region Ca using only pairwise comparisons by either humans or strong LLM via Algorithm 3, which runs in ON + n))) in a)-b) and in O(k?n) in d), and then use this confidence region to construct the rank-sets via Algorithm 2.
Quality metrics. Since the true probabilities are unknown, we cannot compute the true rank-sets of the 12 LLMS under comparison, which presumably may be singletons. As result, we cannot estimate the (empirical) coverage probability-the probability that the rank-sets constructed using the above methods cover the true rank-sets, which Theorem 4.1 refers to. To overcome this, assess the quality of the rank-sets using two alternative metrics: rank-set size and baseline intersection probability. Here, smaller (larger) rank-set sizes and larger (smaller) intersection probabilities are better (worse). The baseline intersection probability is just the (empirical) probability that the rank-sets [i(m). û(m)]
https://chat.Immsss..oo
7

---

2.4
HUMAN ONLY
1.0
3.0
LLM GPT3.5
PPR GPT3.5
Avergne manlk size
LLM CL3
PPR C13
0.9
2.5
LLM Gerd
PER GPT4
2.0
0.8
5-310
2.0
1.5
0.7
0.5
0.6
0.7
0.8
Buseline intersoction probubility
1.0
10
IOP
0.6
Beseline intersection probability
Figure 1: Average rank-set size against baseline intersection probability for rank-sets constructed using only pairwise comparisons by a strong LLM (LLM GPT4, LLM GPT3,5 and LLM CL3), only pairwise comparisons by humans (HUMAN ONLY), and pairwise comparisons by both a strong LLM and humans (PPR GPT4, PPR GPT3.5 and PPR CL3) for different values of a and n = 990. Smaller (larger) average rank-set sizes and larger (smaller) intersection probabilities are better (worse). In all panels, 95% confidence bars for the sen size are not shown, as they are always below 0.02.
PPR GPT4
PPR CL3
PPR GPT3.5
3000
Average reak pank-aat sime 12
2000
odtiomo- o
essouate o
extonesso a
1000
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
1.00
Baseline intersection probability
Baseline intersection probability
Baseline intersection probability
Figure 2: Average rank-set size against baseline intersection probability for rank-sets constructed using pairwise comparisons by both strong LLM and humans for different values of n and a. Smaller (larger) average rank-set sizes and larger (smaller) intersection probabilities are better (worse). In all panels, 95% confidence bars for the rank-set size are not shown, as they always below 0.04.
constructed using one of the above methods intersect with the rank-sets (i(m). ū(m)] constructed using the BASELINE method, i.e,, TP (Nmem [i(m), ū(m) n (i(m), ú(m)) ) Intuitively, we expect that the larger the baseline intersection probability, the larger the coverage probability since the BASELINE method uses a large(r) number of pairwise comparisons by humans to construct (unbiased) rank-sets and thus it is expected to approximate well the true rank-sets. Further, note that the baseline intersection probability tells us how frequently there exists at least one single ranking covered by both one of the above methods and the BASELINE method. In Appendix D.2, we experiment with an alternative metric, namely baseline coverage probability, which is the (empirical) probability that the rank-sets constructed using one of the above methods covers the rank-sets constructed using the BASELINE method. In Appendix E, we additionally evaluate our framework in a synthetic setting where the true rank-sets are known, allowing to compute the coverage probability and rank-biased overlap (RBO) [65].
Quality of the rank-sets. Figure I shows the average rank-set size against the baseline intersection probability for rank-sets constructed using all methods" except BASELINE for different a values' and n 990. The results show several interesting insights. First, we find that rank-sets constructed using only pairwise comparisons by strong LLM (LLM GPT4, LLM GPT3.5 and LLM CL.3) achieve much lower baseline intersection probability. even orders of magnitude lower, than those constructed using only pairwise comparisons by humans (HUMAN ONLY) or using both pairwise comparisons by a
'In Appendix D.I, we include a version this figure with three panels, where each panel contains results for one strong L.I and confidence regions for the rank-set sizes. Sa {0.4.0 0.3, 0.25, 0.2, 0.15, O. 0.075, 0.05, 0.025, 0.01).
8

---

BASELINE
LLM GPM4
PPR Grr4
FastChat ChatGLM
1.0
Alpaca
0.9
Empiricell polabbly
Pythia
RVKK
0.6
Koul
PaLM
Vieuna
0.4
GPT 3.5
Claude 1-1
0.2
Claure
GPT
4 6 10 101112
1011
6 101112
0.0
Ranking position
Ranking position
Ranking position
Figure 3: Empirical probability that each ranking position is included in the rank-sets constructed by BASELINE, LLM GeT4 and PPR GPT4 for each of the LLMs under comparison. all panels, n 990 and a 0.05. Larger (smaller) dots indicate higher (lower) empirical probability.
CPT
Claude 1
Vieuna
PaLM
1.0
Empiriol poduibitt
0 1.75
GPT4
0.5
LLM GPr4
0
BASELINE
5
2
0.0
Ranking position
Ranking position
Ranking position
Ranking position
Figure 4: Empirical probability of each rank-set constructed by BASELINE, LLM GPT4 and PPR GPT4 for GPT 4 (left), Claude 1 (middle left), Vicuna (middle right) and PaLM (right). In all panels, 990 and a = 0.05.
strong LLM and humans (PPr GPT4, PPR GPT3.5 and PPR C1.3). This suggests that the distributions of pairwise comparisons by strong LLMs and humans actually different, questioning the rationale used by an extensive line of work that proposed using only pairwise comparisons by strong LLMs to rank LLMs [12, 25-29, 31]. Second, find that rank-sets constructed using both pairwise comparisons by two of the strong LLMs and humans (PPR GrT4 and PPR CL.3) achieve a better trade-off between average rank-set size and baseline intersection probability than those constructed using only pairwise comparisons by humans (HUMAN ONLY). This suggests that pairwise comparisons by strong LLMs valuable if they complemented with (a few) pairwise comparisons by humans. Third, we find that, among the three strong LLMs, GPT stands our as the best performer.
Figure 2 shows the average set size against the baseline intersection probability for rank-sets constructed using PPR GPT4, PPR GPT3.5 and PPR CL3 for different values n and a (the same values in Figure 1). The results show that the trade-off between rank-sets and baseline intersection probabilities improves rapidly as the number of pairwise comparisons by humans n increases but with diminishing returns.
Structure of the rank-sets. In this section, we take closer look the structure the rank-sets constructed using BASELINE, LLM GPT4 and PPR GPT4. In Appendix D.3, we include additional results for all other methods.
First, we compute the empirical probability that each ranking position is included the rank-sets constructed by BASELINE, LLM GPr4 and PPR GPT4 of each of the LLMs under comparison. Figure summarizes the results for π = 990 and a = 0.05, which reveal several interesting insights. We find that there lower uncertainty regarding the ranking position of each model for LLM GPT4 than for PPR GPT4. However, for LI.M GPT4, the ranking position with the highest probability mass differs from BASELINE
out of LLMs, including the top-2 performers. In contrast, for PPR GPT4, only differs from BASELINE in out of 12 LLMs. This questions once more the status quo, which proposed using only pairwise comparisons by strong LLMs rank LLMs [12, 25-29, 31].
'n {66, 132, 198, 462, 990, 1452, 1980, 2442, 2970).

---

Next, we compute the empirical probability of each rank-set constructed by BASELINE, LLM GPT4 and PPR GPT4 for each of the LLMs under comparison. Figure summarizes the results for GPT Claude 1. Vicuna and PaI.M 2 for n 990 and a = 0.05. In agreement with the findings derived from Figure 3, we observe that the distribution of rank-sets constructed by LIM GPT4 is more concentrated than the distribution of rank sets constructed by PPR GPT4, However, the rank-sets with the highest probability mass constructed by LLM GPT4 coincide with those constructed by BASELINE much less frequently than those constructed by PPR GPT4. Refer to Appendix D.3 for qualitatively similar results for other LLMs.
6 Discussion and Limitations
In this section, we highlight several limitations of our work, discuss its broader impact, and propose avenues for future work.
Data. Our framework assumes that the queries and the pairwise comparisons made by humans and the strong LLMs are drawn i.i.d. from fixed distributions. In future work, it would be very interesting to lift these assumptions and allow for distribution shift. Moreover, our framework assumes that the pairwise comparisons made by humans truthful. However, an adversary could have an economic incentive to make pairwise comparisons strategically in order to favor specific model over others. In this context, it would be interesting extend our framework so that it robust strategic behavior.
Methodology. Our framework utilizes rank-sets as a measure of uncertainty in rankings. However, in case of limited pairwise comparison data, rank-sets may be large and overlapping, reducing their value. In such situations, it may be worthwhile to explore other measures of uncertainty for rankings beyond rank-sets. Further, to measure the level of alignment with human preferences, our framework utilizes the win-rate-the the probability that the outputs of each model are preferred over the outputs of any other model picked uniformly at random. However, if we need to rank k LLMS and k is large, win-rate may be impractical since, to obtain reliable estimates, we need to gather O(k?)) pairwise comparisons made by humans. Finally, our framework constructs rank-sets with asymptotic coverage guarantees, however, it would be interesting to derive PAC-style, finite-sample coverage guarantees.
Evaluation. We have showcased our framework using pairwise comparisons made by humans in single platform, namely LMSYS Chatbot Arema, and pairwise comparisons made by just three strong LLMs. As a result, one may question the generalizability of the conclusion derived from the rank-sets estimated using our framework. In this context, it is also important to acknowledge that, in LMSYS Chatbot Arena, the queries are chosen by the humans who make pairwise comparisons and this may introduce a variety of biases. Therefore, it would be interesting to apply our framework to human data from other platforms.
Broader Impact. Our framework rank LLMs according to their level of alignment with human preferences- LLM is ranked higher than others if its outputs are more frequently preferred by humans. However, in many application domains, especially in high-stakes scenarios, may be important to account for other important factors such as accuracy, faimesss bias and toxicity [57].
Conclusions
We have introduced a statistical framework to construct ranking of a collection of LLMs consistent with their level of alignment with human preferences using small set of pairwise comparisons by humans and large set of pairwise comparisons by strong LLM. Our framework quantifies uncertainty in the ranking by providing rank-set-a set of possible ranking positions-foo for each of the models under comparison. Moreover, guarantees that, with a probability greater than or equal to user- sper-specffic value, the rank-sets cover the ranking consistent with the (true) probability that each model is preferred over others by humans asymptotically. Finally, we have empirically demonstrated that the rank-sets constructed using our framework are more likely to cover the true ranking consistent with (the distribution of) human pairwise preferences than the rank-sets constructed using only pairwise comparisons made by the strong LLMs.
Acknowledgments and Disclosure of Funding
Honez-Rodrrguez acknowledges support from the Furopean Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement No. 945719).
10

---

References
[1] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang, Sparks of Artificial General Intelligence: Early experiments with GPT-4, arXiv preprint arXiv:2303. 22 2023.
[2] Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. Reading Between the Lines: Modeling User Behavior and Costs in Al-Assisted Programming. In Proceedings of the Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2024.
[3] Claudia E Haupt and Mason Marks. Al-Generated Medical Advice GPT and Beyond. Journal of American Medical Association, (399(16))339113333 2023.
[4] Bernardino Comera-Pareded Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical Discoveries from Program Search with Large Language Models. Nature, 625(7995):468-7777
2023.
[5] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang. Philip S. Yu, Qiang Yang, and Xing Xie. A Survey on Evaluation of Large Language Models. ACM Trunsuctions on Intelligent Systems and Technology. 2024.
[6] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In Proceedings of the International Conference on Learning Representations. ICLR, 2021.
[7] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484-13555 13508. Association for Computational Linguistics, 2023.
[8] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems, pages 27730-21 277444. Curran Associates, Inc., 2022.
[9] Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng. Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning Large Language Models with Human: A Survey, arXiv preprini arXiv:2307. 12966, 2023.
[10] LMSYS. Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings. https://lmsys.org/ 2023. Online; accessed 21 May 2024.
[11] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford Alpaca: An instruction instructioo-foolooiing LLaMA model. http://u...oootttttuaaaaaba stanford stanttdo alpaca, 2023. Online; accessed 21 May 2024.
https://github..cmm
[12] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as---Judge with MT-Bench and Chatbot Arena. Advances Neural Information Processing Systems, data track, pages 46595446623,, Curran Associates, Inc., 2023.
[13] Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative Judge for Evaluating Alignment. In Proceedings of the International Conference on Learning Representations. ICLR, 2024.
[14] Ruosen Li, Teerth Patel, and Xinya Du. PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations. Transactions on Machine Learning Research (TMLR), 2024.
[15] Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arXiv preprint arXiv:2311. 2023.
11

---

[16] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Senevi- ratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman Blaise Agüera y Arcas, Dale Webster, Greg Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajko- mar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large Language Models Encode Clinical Knowledge. Nature, 620(7972): July 2023.
[17] Wei-Lin Chiang, Lianmin Zheng. Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang. Michael I. Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. In Proceedings the International Conference on Machine Leurning. PMLR, 2024.
[18] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds Danny Hernandez. Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark. Sam McCandlish, Chris Olah, and Jared Kaplan. A General Language Assistant as a Laboratory for Alignment. arXiv preprint wxXiv:2122.00811, 2021.
[19] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized L.LMs. In Advances in Neural Information Processing Systems, pages 10088-10115. Curran Associates, Inc., 2024.
[20] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hermandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2204. 05862, 2022.
[21] Yuxiang Wu, Zhengyao Jiang, Akbir Khan, Yao Fu, Laura Ruis, Edward Grefenstette, and Tim Rocktäschel. ChatArena: Multi- Agent Language Game Environments for Large Language 2023.
Models. http:///tuub..com//ccaaaeteeeeee
[22] Yen-Ting Lin and Yun-Nung Chen. LLM-Eval: Unified Multi-Dimensional Automatic Eval- uation for Open-Domain Conversations with Large Language Models. In Proceedings of the Workshop on NLP for Conversational AI, pages 47-58. Association for Computational Linguistics, July 2023.
[23] Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael Jor- dan. AutoEval Done Right: Using Synthetic Data for Model Evaluation. arXiv preprint arXiv:2403.0m 07708, 2024.
[24] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. Large Language Models Can Accurately Predict Searcher Preferences. In Proceedings of the International ACM SIGIR Con- ference on Research and Development in Information Retrieval, pages 1930-1940. Association for Computing Machinery. 2024.
[25] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are fair evaluators. In Proceedings of the Association for Computational Linguistics, pages 9440-9450. Association for Computational Linguistics, 2024.
[26] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vieuna: An Open-Soure Chatbot Impressing GPT-4 with *0%* ChatGPT Quality. https://vicumalmmsss.rrg 2023. Online; accessed 21 May 2024.
[27] Cheng-Han Chiang and flung-yi Lee. Can Large Language Models Be an Alternative to Human Evaluations? In Proceedings of the Association for Computational Linguistics (Volume 1: Long Papers). pages 15607-1563) Association for Computational Linguistics, 2023.
[28] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-BBender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. In Proceedings of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165-14178, Toronto, Canada, 2023. Association for Computational Linguistics.
12

---

[29] Yidong Wang, Zhuohao Yu, Zhengran Zeng. Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang. Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. In Proceedings of the International Conference on Learning Representations. ICLR, 2024.
[30] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is ChatGPT a General- Porerall-Puroseeeeeee Natural Language Processing Task Solver? In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1339-1384. Association for Computational Linguistics, 2023.
[31] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba. Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. In Advances in Neural Information Processing Systems, pages 10039-30066 Curran Associates, Inc., 2024.
[32] Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1504-1518 1518. Association for Computational Linguistics, 2024.
[33] Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators. arXiv preprint arXiv:2403. 16950. 2024.
[34] Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. Preference Proxies: Evaluating Large Language Models in Capturing Human Preferences in Human-Al Tasks. In Proceedings of the ICML Workshop The Many Facets of Siftference-Basele Learning. 2023.
[35] Tim R Davidson, Veniamin Veselovsky, Martin Josifoski. Maxime Peyrard, Antoine Bosselut, Michal Kosinski, and Robert West. Evaluating Language Model Agency Through Negotiations. In Proceedings of the International Conference on Learning Representationss ICLR, 2024.
[36] Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large Language Models are Zero-Shot Rankers for Recommender Sys- tems. In European Conference on Information Retrieval, pages 364-381. Springer, 2024.
[37] Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana
Zrnic. Prediction-Powereed Inference. Science, (666666)) 2023.
[38] Anastasios N. Angelopoulos, John Duchi, and Tijana Zrnic PPI++: Efficient Prediction-
Powered Inference. arXiv preprint arXiv:231 1.0 2023.
[39] Tijana Zrnic and Emmanuel Candès. 22322083121, National Academy Sciences, 2024.
Pww.iiriioooooweeiecree Inference. volume 121, page
[40] Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An Auto- mated Evaluation Framework for Retrieval-Augmented Generation Systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 338-354, 2024.
[41] Oscar Lemmers, Jan AM Kremer, and George F Borm. Incorporating Natural Variation into IVF Clinic League Tables. Human Reproduction, 1355)11359 2007.
[42] Harvey Goldstein and David J Spiegelhalter. League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance. Journal of the Royal Statistical Society. Series A (Statistics in Society), 159())335 1996.
[43] Peter Hall and Hugh Miller. Using the Bootstrap to Quantify the Authority of an Empirical Ranking. The Annals of Statistics, 37:3929-390 -3959, 2009.
[44] E Clare Marshall, Colin Sanderson, David J Spicgelhalter, and Martin McKee. Reliability of League Tables of In Vitro Fertilisation Clinics: Retrospective Analysis of Live Birth Rates. British Medical Journal, 1116661146:1101111 1701-1755 1998.
[45] Tommy Wright, Martin Klein, and Jerzy Wieczorek. Ranking Populations Based on Sample Survey Data. Statistics, page 12, 2014.
[46] Minge Xie, Kesar Singh, and Cun-Hui Zhang. Confidence Intervals for Population Ranks in the Presence of Ties and Near Ties. Journal of the American Statistical Association, 104(486):775-
788, 2009,
13

---

[47] Shumpu Zhang, Jun Luo, Li Zhu, David G Stinchcomb, Dave Campbell, Ginger Carter, Scott Gilkeson, and Eric Feuer. Confidence Intervals for Ranks of Age-Adjusted Rates across States or Counties. Statistics in Medicine, 1):1833-1866, 2014.
[48] Bitya Neuhof and Yuval Benjamini. Confident Feature Ranking. In Proceeding of the ICML workshop on Spurious Correlations, Invariance and Stability. PMIR, 2023.
[49] Justin Rising. Uncertainty in Ranking. arXiv preprint arXiv:2107.03459 2021.
[50] Diaa Al Mohamad, Jelle Goeman, and Erik W van Zwet. Simultancous Confidence Intervals for Ranks with Application to Ranking Institutions. Biometrics, 78(1):238-22) 2022.
[51] Martin Klein, Tommy Wright, and Jerzy Wieczorek. A Joint Confidence Region for an Overall Ranking of Populations. Journal of the Royal Statistical Society Series C: Applied Statistics, 69(3):589-66) 606, 2020.
[52] Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, An- drea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang. Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts. In Proceedings of the Association for Compulational Linguistics: System Demonstrations, pages 93-104. Association for Computational Linguistics, May 2022.
[53] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned Language Models are Zero-Shot Learners, In Proceedings of the International Conference on Learning Representations. ICLR, 2022.
[54] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA:
A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Papers), pages 4149-4158. Association for Computational Linguistics, 2019.
[55] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-Tash General- ization via Natural Language Crowdsourcing Instructions. In Proceedings of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470-3493 Dublin, Ireland, May 2022. Association for Computational Linguistics.
[56] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan. Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.033744 2021.
[57] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu. Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi. Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023.
[58] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, and Adam Roberts. The Flan Collection: Designing Data
14

---

and Methods for Effective Instruction Tuning, In Proceedings of the International Conference on Machine Learning, pages 22631-22648, PMLR, Jul 2023.
[59] Arpad E. Elo. The USCF Rating System: Is Development, Theory, and Applications. United States Chess Federation, 1966.
[60] Quentin Bertrand, Wojeiech Marian Czarnecki, and Gauthier Gidel. On the Limitations of the Elo, Real-World Games are Transitive, Not Additive. In International Conference Artificial Intelligence and Statistics, pages 2905-2921 PMLR, 2023.
[61] Adian Liusie, Potsawee Manakul, and Mark F. Gales. LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models. In Proceedings of the European Chapter of the Association for Computational Linguistics (Volume
1: Long Papers), pages 139-151. Association for Computational Linguistics, 2024.
[62] Zongjie Li, Chaozheng Wang, Pingchuan Ma. Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and Merge: Aligning Position Biases Large Language Model Based Evaluators. arXiv preprint arXiv:2310.01432, 2023.
[63] Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang. and Xiaojun Wan. Human-like Summarization Evaluation with ChatGPT. arXiv preprint 23X4:23004.02554 2023.
[64] Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang. Juanzi Li, and Lei Hou. Benchmarking Foundation Models with Manguage--Moellass --aeddl------- Systems. Curran Associates, Inc., 2024.
Examiner. In Advances in Neural Information Processing
[65] William Webber, Alistair Moffat, and Justin Zobel. Similarity Measure for Indefinite Rankings. ACM Transactions on Information Systems, 28(4):20:1-200 2010.
15

---

A Proof of Theorem 4.1
Note that Eq. 12 holds if and only if:
lim IP (3m E M: M
[1(m), g [((m))(m))) S a
(13)
Therefore, to prove the theorem, is sufficient to prove that Eq. 13 holds. Now, to prove that Eq. 13. we first show that the probability on the left hand side of the above equation is smaller than or equal the probability P(O Ca).
To this end, first note that, if for least one model m M, we have that i(m) > [(m) or û(m) < u(m), then it holds that
n [(m), u(m)] & [((m), ū(m)). mEM
Next, without loss of generality, assume that, for model m, we have that i(m) > 1(m). In this case, from Eqs. 11 and 2 get:
Σ
1fd(Ca, Hm.m) > 0} 160m. < dm)
Σ
1{0m fmh
MmM
mEM)(m)
which means that there must be at least one model m E M such that Im < 2m Vo E Ca and Om Om, which implies that 0 & Ca. As result, we can immediately conclude that,
lim P (3m E M: [(m),, u(m)) & (((m), ú(m))) < lim P(O & C.)
This concludes the proof.
16

---

B
Algorithms
B.1
Algorithm 3
In this section, we present pseudocode implementation of the algorithm to construct confidence regions using linear regression.
Algorithm 3: It estimates and Σ using linear regression.
Input:
Output: 0, Σ
w. w. M, M' + SUMMARIZE(D,. k)
0 1n (IM 1)) 0IA) oT)) (M v M' .w)
// linear reg. estimate
A
(1 - M T)T) o M + (Ic(00 MMT 0)7) o MM)
-M
+ 1o AA
// estimate of covariance
return 0, Σ
Note that, in Algorithm θ and Σ can alternatively be computed by calling Algorithm 3 instead of Algorithm 1. Algorithm 3 runs in (K)) time.
B.2 Algorithm to set A
The parameter E [0, 1] weighs the comparisons wù, r0 by the strong LLM differently than the comparisons w, w' by humans. This way, if the strong LLM's preferences are strongly aligned with human preferences, the pairwise comparisons by the strong LLM can be weighed close or equally with the pairwise comparisons by humans. Conversely, if the strong LLM's preferences are not well aligned with human preferences, the pairwise comparisons úù r'' can be weighed lower than the pairwise comparisons w'. Following Angelopoulous et al. [38], we select the À that minimizes tr(E), where Σ = E[(Ø- (0-0))))))))) -))
tr ( TI + N tr (En)
(14)
where E is the sample covariance matrix of the estimate the pairwise comparisons by the strong LLM NN, and E,, is the sample covariance matrix of the estimates of Ó and 0 computed via Algorithm 3 using the pairwise comparisons by the strong LLM and by humans respectively in dataset D,,. Detailed computation of À is shown in Algorithm 4, which runs in 2 + n)).
computed from Algorithm using
Algorithm 4: It computes À
Input: kt Dx, D.. Output: X
W., Wn. wn. w', M., MA SSUMARRIZED....)
ān 1x ((M, + 1.)
+ TR
(M, w. + M2 w²,))
an 11 ((M, + MM) 1.))
(M, + M' w/,)
Ån < ((1c(wn M àm)
② M.
(1a(1', M. àn) T) 2 MM)
A, < (11{(wn M an)
@ M. + (1x(w,, M an T) 3 My)
Σn + A.A,1
-, ΣN +- CONFIDENCE HEIDDENNE-ELLIPSSODD... Dw) + " m(E..)
1/ Algorithm 3
O+N m(EN)
return A
17

---

[System]
Act as an impartial judge and evaluate the responses of two assistants to the user's question displayed below. Your evaluation should consider factors such as relevance, helpfulness, accuracy, creativity and level of detail of their responses. Output your final verdict by strictly following this format: 'A' if the response of assistant is better, "B' the response of assistant is better, and 'C' for a tie. Do not give any justification or explanation for your reaponse.
[User]
Question:
(Query)
Response of assistant [Response A}
Response of assistant B: {Response B}
Listing 1: The prompt used for gathering pairwise preferences of strong LLMss.
C Additional Details of the Experimental Setup
In this section, we provide the implementation details and computational resources used to execute experiments discussed in Section 5. Our algorithms are implemented Python 3.11.2 .2 pro- gramming language using NumPy and SciPy open-source libraries for efficient matrix operations. Further, we use the matplotlib package to facilitate visualizations of our results. The complete details of the software requirements can be found in the source code provided part of the sup- plementary materials. Our experiments are executed on a compute server equipped with 2 x AMD EPYC 7702 processor with 64 cores per processor and TB of main memory. is worth noting that, our experiments are not resource intensive and can be executed on a standard desktop computer or a laptop.
Pairwise comparisons and preprocessing. Using the dataset from LMSYS Chatbot Arena?, we gathered pairwise comparisons of three strong LLMs via API calls to OpenAI API version 2024-02-15- -----iiiie- for GPT 3.5 and GPT 4, and Anthropic API version 2023-06-01 for Claude 3. To this end, we use (almost) the same prompt as in Zheng et al. [12] that instructs each strong LLM to output option 'A' ('B') if prefers the response of first (second) model, or option 'C' if it declares a tie. The prompt used to obtain pairwise preferences of is available in Listing 1. preprocess
dataset by filtering out instances- an instance by two different models and the query itself-for which at least one strong LLM returned a verbose output instead of 'A', 'B' or 'C', and choose a set of 12 popular large language models for our experiments. The chosen models, along with their specific versions, are listed in Table 1. In Figure 5, we show the number of pairwise comparisons per each pair of these chosen models after completing all preprocessing steps.
pair responses
query
The user prompts licensed under CC-BY-4.0, while the model outputs licensed under CC-BY-NC-400.
18

---

Table 1: The names and versions of the 12 popular large language models considered for experiments after preprocessing the Chatbot Arena dataset.
Large Language Model Version
GPT4
GPT
Claude
Claude vl
Claude 1-1
Claude Instant vl
GPT 3.5
GPT 3.5 turbo
Vicuna
Vicuna 13B
PaLM
PaLM 2
Koala
Koala 13B
RWKV
RWKV 4 Raven 14B
Pythia
OpenAssistant Pythia 12B
Alpaca
Alpaca 13B
ChatGLM
ChatGL 6B
FastChat
FastChat T5 3B
Viema
712
305
283
170
368
261
243
306
208
187
169
Koala
282
396
285
242
184
GPT 3.5
305
342
0
209
201
200
288
263
182
232
187
135
GPT
282
273
272
177
Pythia
470
3916
291
273
0
302
1263
210
236
207
165
122
Alpaca
368
345
250
253
123
Claude
261
285
288
272
263
250
0
256
161
169
179
125
RWKV
2413
225
263
262
210
194
256
141
147
156
CbstGLM
306
2/12
182
195
236
253
161
1419
0
142
109
96
CastChat
206
251
232
192
207
202
169
147
142
127
108
PaLM 2
187
184
187
177
155
123
179
156
100
127
0
L68
Claude
169
144
135
L34
122
115
125
129
96
108
168
Vieina
Roala
GPT 3.5
GPT 4
Pythia
Alpaca
Clande
RWKY
ChaGLM
Chat
P 2
Clande 1-1
Figure 5: The number of pairwise comparisons per each pair of models after all preprocessing steps.
19

---

D Additional Experimental Results
In this section, we discuss additional experimental results that were omitted from the main paper due to space limitations.
D.1 Quality of the Rank-sets
In Figure 6, show more detailed analysis of the average rank-set size plotted against the baseline intersection probability, with individual plots for each strong L.LM: GPT4 (top); CL.3 (middle) and GPT3.5 (bottom), for n 990 and different a values. 8 All observations made in Figure apply to Figure 6 as well. Moreover, it is evident that among the three strong LLMs, GPT 4 clearly outperforms the others, achieving higher baseline intersection probability and returning rank-sets with smaller average size. Specifically, in Figure 6 (a), the difference between average ret size between HUMAN ONLY and PPR GPT4 significantly pronounced, while the gap gradually narrows in subsequent plots for PPR CL3 and PPR GPT3.5.
D.2 Baseline Coverage Probability
In this subsection, we investigate more conservative quality metric, namely baseline coverage probability, which is the (empirical) probability that the rank-sets [[mm), ú(m)) constructed by any method cover the rank-sets [(mm), ū(m) constructed using the BASELINE method, ie..
n 1 (((m)) S (i(m),, (((m))))m)))) MEM
(15)
The baseline intersection probability, which we discussed in Section 5 and illustrated in Figure 1, is a less conservative metric compared to the baseline coverage probability. While the latter represents the probability that all rank-sets are covered by the BASELINE rank-sets, the former only considers the probability that the rank-sets intersect. Thus, achieving high baseline coverage probability is difficult, particularly when the BASELINE rank-sets are larger.
Quality of the rank-sets when considering the baseline coverage probability. In Figure 7, we show the average renk-set size against the baseline coverage probability for rank-sets constructed using all methods (except BASELINE) for 990 and different values of a (the same values as in Figure 6). Immediately, we notice that the baseline coverage probability of all methods is very low. For rank-sets constructed using pairwise comparisons only by a strong LLM (LLM GPT4, LLM CL3 and LLM GPT3.5), the baseline coverage probability is close to (or exactly) zero. Rank-sets constructed using only pairwise comparisons by humans (HUMAN ONLY) or rrediction-poweete a strong LLM (PPR GrT4, PPR CL3 and PPR GPT3.5) achieve better baseline coverage probability. However, the difference in performance among these methods is not clear to distinguish, which motivated us to consider the baseline intersection probability metric for our experimental results in Section 5.
ranking using
In Figure 8, we show the average rank-set size against the baseline coverage probability for rank-sets constructed using PPR GPT4, PPR GPT3.5 and PPR CL3 for different values 9 of n and a (the same values as in Figure 6). Similarly as Figure 2, results show that there a trade-off between average rank-set size and the baseline coverage probability, which improves rapidly as the number of pairwise comparisons by humans n increases, with diminishing returns.
$ E (0.4, 0.25, 0.2,0.15, 0.15, 0.1,0.075, 0.05, 0.025, 0.01) 9n {66, 132, 198, 462, 990, 1452, 1980, 2442, 2970).
20

---

2.4
HUMAN ONLY
1.0
3.0
LLM GPT4
PPR GPT4
Averago rull-act sizo
2.5
2.2
2.0
0.8
2.0
1.5
0.5
0.7
0.8
Baseline intersection probsbility
1.0
10
10
10
10
0.6
Baseline intersection probubility
(a) GPT
HUMAN ONLY
3.0
LLM CL3
PPR CL.3
Averago rank--act size
2.5
2.0
6:08
1.3
0.7
0.5
0.6 Baseline intersection probability
0.7
0.8
1.0
10
10 Baseline intersection probability
10
TD
0.6
(b) Claude 3
HUMAN ONLY
3.0
LLM GPT3.5
PPR GPT3.5
Averago rank-sec sizo
2.5
2.0
0.8
1.5
0.7
0.5
0.6 Baseline intersection probability
0.7
0.8
1.0
10
10-1 Baseline intersection probubility
10
10
0.6
(c) GPT 3.5
Figure 6: Average rank-set size against baseline intersection probability for rank-sets constructed us- ing only pairwise comparisons by strong LLM: LLM GPT4 (top), LLM CL.3 (middle) and LLMM GPT3.5 (bottom), only pairwise comparisons by humans (HUMAN ONLY), and pairwise comparisons by both strong LLM and humans (PPR GPT4 top, CL3 middle and PPR GPT3.5 bottom) for different values and n 990. Smaller (larger) average rank-set sizes and larger (smaller) intersection probabilities are better (worse). The shaded region shows 95% confidence interval for the rank-set size of all rank-sets sets among all 1, 000 repetitions.
21

---

HUMAN ONLY
3.0
LLM GPT3.5
PPR GPT3.5
Averago ratil-set size
LLM CL.3
PPR CL3
0.9
2.5
LLM GPT4
2.2
PPR GPT4
2.0
0.8
000200
0.7
0.10
0.15
0.20
0.25
Baseline coverage probability
1.0
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.6
Boseline coverage prbubilitt
Figure 7: Average rank-set size against baseline coverage probability for rank-sets constructed using only pairwise comparisons by strong LLM (LLM GPT4 LLM GPT3.5 and LLM CL3), only pairwise comparisons by humans (HUMAN ONLY), and pairwise comparisons by both strong LLM and humans (PPR GrT4, PPR GrT3.5 and CL3) for different values and n 990. Smaller (larger) average rank-set sizes and larger (smaller) coverage probabilities are better (worse). In all panels, 95% confidence bars for the rank-set size are not shown, as they are below 0.02.
PPR GPT4
PPR CL3
PPR GPT3.5
3000
Average rank--oo cank--att size
mipttre o
2000
doo
glutt oo
cooss unce
1000
15
crotto Saaoooo
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
Baseline coverago probebility
Baseline coverage probability
Bascline coverage probalility
Figure 8: Average rank-set size against bascline coverage probability for rank-sets constructed using pairwise comparisons by both a strong LLM and humans for different n and a values. Smaller (larger) average rank-set sizes and larger (smaller) coverage probabilities are better (worse). In all panels, 95% confidence bars for the set size are not shown, as they are below 0.04.
22

---

BASELINE
LLM GPT
LLM CL.3
LLM GPT3.5
FastChat
1.0
ChatGLM
Alpaca
Pythia
RWKV
Koola
0.8
PaLM
Viema
GPT 3.5
SUO FUODOO
Claudo 14
Empirial prllabiity
Clande
0.6
GPT
123 12345678 910111
12345678 67661111
12345678
12345678 910111
HUMAN ONLY
Peit GrT4
PeR CL3
PPR GPT3.5
ChatGLM
FostChat
..000
.0000
G0000
.0000
3000
.0000
00300
.0000
0.4
Alpaca
Pythia
-00009
100009
00000
RWKV
.0006
0000
0000R
00008
Knola
PaLM
0.2
Viemm
GPT
Claude 1-1
Claude
GPT
2 2346678 910111
12345678 910112 Ranking position
12345678 910112
12345678 910111
0.0
Ranking position
Ranking position
Ranking position
Figure 9: Empirical probability that each ranking position is included in the rank-sets constructed by all methods for each of the LLMs under comparison. In all panels, n = 990 and a = 0.05. Larger (smaller) dots indicate higher (lower) empirical probability.
D.3 Structure of the Rank-sets
In this subsection, we take closer look the structure of the rank-sets constructed all methods. We compute the empirical probability that each ranking position is included in the rank-sets constructed by all methods, each of the LLMs under comparison. The results summarized in Figure 9 for = 0.05 and = 990.
Empirical probability of ranking positions. Our first observation is that, the ranking positions of each model in LLM GPT4 and LLM CL3 have lower uncertainty compared to PPR GPT3.5 and PPR CL3, respectively. However, LLM GPT3.5 exhibits higher uncertainty compared to PER GPT3.5. Nonetheless, the ranking positions with the highest probability mass in LLM GrT4, LLM Ct.3 and LLM GPT3.5 significantly deviate from the BASELINE. Specifically, the ranking position with highest probability mass differs for 7 out of 12 models. contrast, for PPR GPT4, PPR CL3 and PPR GPT3.5 only differs from the BASELINE for 3 out of 12 LLMs. These findings once again question the rationale of relying solely on pairwise comparisons by strong LLMs to rank LLMs [12, 25-29, 31]. Our second observation is that there no significant difference in the uncertainty in ranking positions across HUMAN ONLY, PPR GPT4, PPR CL.3 and PPR GPT4. However, the distribution of probability mass across different ranking positions differs slightly among these methods. This observation is clearly seen in PPR GPT4, where the Alpaca model has zero probability mass for position 6.
Empirical probability of rank-sets. Next, we compute the empirical probability of each rank-set constructed by all methods and for each of the LLMs under comparison with n = 990 and a 0.05. The results are summarized in Figure 10. Consistent with the observations from Figure 9, we note that the distribution rank-sets constructed by LLM GPT4 is more concentrated than those constructed by other methods. Conversely, LLM GPT3.5 exhibits more spread-out distribution of rank-sets, indicating higher uncertainty in its ranking positions. This observation is consistent across all LLMs considered for ranking. Despite the
concentrated distributions of rank-sets for LLM GrT4 and LLM CL3, we observed that the ranking positions with the highest probability mass often differed from those of the BASELINE, with discrepancies observed in 7 out of 12 models. On the contrary, the rank-sets constructed by PPR GPT4, PPR CL.3 and PPR GPT3.5 exhibit distributions that are neither excessively spread out highly concentrated. But the rank-sets with the highest probability mass constructed by these methods coincide with those constructed by BASELINE more frequently than their LLM only counterparts. Once again, these findings underscore our argument that rank-sets obtained using only pairwise comparisons of strong LLMs are not very reliable.
23

---

GPT
BASELINE
LLM GrT4
LLM CL.3
LLM GPT3.5
Empirical probobiitty
1234567851012
12345678 910112
1234567801md
123 6466790112
0.5
HUMAN ONLY
PPR GPT4
PPR. CL.3
PPR GPT3.5
0.25
0.0
123466799111
2 4847668800 9100112112
1 4234667899 9101112
T23 1234567891112
Ranking position
Ronking position
Ranking position
Ranking position
Claude 1
BASELINE
LLM GPT4
LLM CL3
LLM GPT3.5
1.0
0.75
Empirical probablity
2 12345677 9100112
T234567 9101112
123 1234567891019
2 674456777 8910112
0.5
HUMAN ONLY
PPR GrT4
PPR CL3
PPR GrT3.5
0.25
I 2 3 4 45577890112 Ranking position
2 4 5 8 9101112 Ranking position
2 4 66678910111 Ranking position
T 2 4 675011112
Ranking position
(b) Claude
Figure 10: Empirical probability of each rank-sct constructed by all methods for all 12 models (one model per sub-figure). In all panels, n 990 and a = 0.05.
24

---

Claude 1-1
BASELINE
LLM GrT4
LLM CL.3
LLM Grr3.5
1.0
0.75
Empirical probability
1234 910112
1234567591111
0.5
HUMAN ONLY
PPR GPr4
PPR CL.3
PPR GPT3.5
0.25
T 234567880 17688181112
123 035536666 1.66678 9101112
2 3 6 6 910112
T 2 3 43665 910
0.0
Ranking position
Roniiing position
Ranking position
Ranking position
(c) Claude 1-1
GPT 3.5
BASELINE
LLM GPT4
LLM CL3
LLM GPT3.5
1.0
0.75
Empirical probability
1234567891012
1234567891011
1234567891012
12345 173466775 9100122
0.5
HUMAN ONLY
PPR GPT4
PPR CL.3
PPR GPT3.5
0.25
0.0
2 3 4 8
2 5.6 9101112 Ranking position
123 3.3. 4 6.76773 910112 Ranking position
1 2 4 115566 9 112
Ranking position
Ranking position
Figure 10 (cont.)
25

---

Vicuna LLM CL3
BASELINE
LLM GPr4
LLM GPT3.5
1.0
Empiriol probabilitty
1 4 3334578911112
8234667899
123 4 6 917781111
123 4 678
0.5
HUMAN ONLY
PPR GPT4
PPR CL3
PPR GPT3.5
0.25
123
123 78 9111112
T23 133450 456688888 910112
1234567 9101112
0.0
Ranking position
Ranidig position
Ranking position
Ranking position
PaLM 2
BASELINE
LLM GPT4
LLM CL3
LIM GPT3.5
1.0
0.75
Empirical probbiiity
1 123 123467781mn1
1234567
610in2
12345678 610112
123 4 5 6 9100m
0.5
HUMAN ONLY
PPR GPT4
PPR CL3
PPR GPT3.5
3 4 4 1910112
2 3 456 8 9101112
1234 4 6 9101112 Ranking position
223 033466799000 9101112 Ranking position
Ranking position
Ranking position
Figure 10 (cont.)
26

---

Koala LLM CL3
BASELINE
LLM GPr4
LLM GPT3.5
1.0
Empiriod probabilitty
1 4 4547776811110
12346778 78 9101112
12346678 9334667991112 910111
123 4 5 67 9910122
0.5
HUMAN ONLY
PPR GPT4
PPR CL3
PPR GPT3.5
0.25
123 23456 78 9101112
T2345678 910112
T234567 57656788 9101112
0.0
Ranking position
Ranidig poeition
Ranking position
Ranking position
RWKV LLM CL3
BASELINE
LLM GPT4
LIM GPT3.5
1.0
0.75
Empirical probbility
123486788 110ii
12345678 610in2
12345678 610in2
1234567 910m
0.5
HUMAN ONLY
PPR GPT4
PPR CL3
PPR GPT3.5
0.25
12345678 8990112
1234667 78 910112
12345678 9
1234567 $ 91011
Ranking position
Ranking position
Ranking position
Ranking position
Figure 10 (cont.)
27

---

Pythia LLM CL3
BASELINE
LLM GPT4
LLM GPT3.5
1.0
0.75
Empiriod probabilitty
1234566 1234567891011
12345678 9101112
12346678 910111
1234567
6719911122
0.5
HUMAN ONLY
PPR GPT4
PPR CL3
PPR GPT3.5
0.25
T2346678910 112
12345678 901677891000 101112
T2345678 910112
12345678 9
0.0
Ranking position
Ranidigg position
Ranking position
Ranking position
Alpaca LLM CL3
BASELINE
LLM GPT4
LIM GPT3.5
1.0
0.75
Empirical probbility
123 13346677 123486785101 910ii
1234667 610in2
12345678 610in2
1234567 910mi
0.5
HUMAN ONLY
PPR GPT4
PPR CL3
PPR GPT3.5
0.25
7990012
1234667 78 910111
12345678 9
12345678 91111 Ranking position
Ranking position
Ranking position
Ranking position
Figure 10 (cont.)
28

---

ChatGLM
BASELINE
LLM GPT4
LLM CL3
LLM GPr3.5
1.0
0
Empirieul probability
123 023356788 910:112
12346678 9100112 9101112
12345678 910112
12345 6 910011
HUMAN ONLY
PPR GPT4
PPR CL3
PPR GPT3.5
0.25
1234567891011
T2345678 910112
T2345678 8 910112
T2345678 91011
0.0
Ranking position
Ranidng position
Ranking position
Ranking position
FastChat LLM CL3
BASELINE
LLM GPT4
LLM GPT3.5
1.0
0.75
Empirioal probabiiig
T234567 9
T234567890 910112
T2345678 9101m
1234567 111
0.5
HUMAN ONLY
PPR GPT4
PPR CL3
PPR GPT3.5
0.25
22345678 901112
12346678 910112
12346678 910111
0345678910111
0.0
Ranking position
Ranking position
Ranking position
Ranking position
Figure 10 (cont.)
29

---

E Synthetic Experiments
In this section, we evaluate our framework in a synthetic setting where the true rank-sets of the models are known. This allows us to validate the theoretical coverage guarantee of Theorem 4.1 without relying on proxy metrie, and also compute the rank-biased overlap (RBO) [65].
Experimental setup. We consider ht = 8 models and, in each experiment, we generate a random vector of true win probabilities 0 (Eq. 1), which induces the true rank-sets of the models, 10 To obtain win probabilities Ô (Eq. 3), we add random noise the vector and then re-normalize Ó. We sample the noise from a Uniform(-u, u) distribution, where u a parameter we manually set to simulate different aligmment levels between strong LLMs and human preferences.
larger u indicates greater difference between & and 0, meaning the L.M is less aligned with human preference, and vice versa. In our experiments, we set u E {0.05, 0. 0.3) to simulate three different strong LLMs.
To draw reliable conclusions. for each experiment, we create rank-sets 300 times, each time using different set of n + N = 50,000 simulated pairwise comparisons by humans and the three strong LLMs, with equal number of pairwise comparisons per pair of models. We ensure that each model provides the first and second response to an equal number of pairwise comparisons. Let m, and my be the two models comparison, first, we generate number z E (0, 1) uniformly at random. For the human outcome, if x 2 20... then the response of model m, preferred (w 1). Similarly, for the strong LLM outcome, if < 2 mm.. then the response of model m, is preferred (ι 1). In every comparison we set w' E 0.
pairwise comparison, with m. giving the first response. For each pairwise
Coverage probability. Using the generated pairwise comparisons, we compute rank-sets in a similar way as described in Section 5, with a = 0.1. Since the true rank-sets are known, we can compute the (empirical) coverage probability, shown in Figure 11. The results show that the coverage probability increases with n, consistent with Theorem 4.1. Further, the coverage probability is greater when is smaller, indicating that our method achieves better results when the strong L.L.M is more aligned to human preference.
Ganneb overlap (RBO). For each method, we obtain a ranking ी by sorting the models in descending order of their Ô values. We then compute the RBO metric relative to their true ranking T as follows:
RBO p)) (1
Σ p
Ta
i
iE[K]
where T3 and T: represents the top models in ranking 7 and T, respectively, and p E [0, is a chosen parameter. When 1, all models are weighed equally. As we decrease p. more emphasis is given to the top-ranked models, and at p = O, only the top-ranked model is considered. Figure 12, we compare RBO values we increase the number of human pairwise comparisons for p 0.6. The results show that increasing n improves RBO across all methods. Additionally, combining human pairwise preferences with strong LLM further improves RBO values, demonstrating that our method performs better than those solely relying on strong LLM preferences. We repeated our experiments with multiple values of p E [0. 1] and observed significant variation in the results.
In our experiments, we generate true win probabilities @ with unique values, so the rank-sets are always singletons, resulting in unique true rank for each model.
30

---

1.0
0.9
Coverage probability
0.8
0.7
0.6
TARGET
0.5
PPR 0.05
PPR 0.1
0.4
PPR 0.3
HUMAN ONLY
0.3
0
2000
5000
10000
15000
20000
72
Figure 11: Empirical coverage probability of the rank-sets constructed using only n synthetic pairwise comparisons by humans (HUMAN ONLY) and using both synthetic pairwise comparisons by humans and N + synthetic pairwise comparisons by one out of three different simulated strong LLMs (PPR 0.05, PPR 0.1 and PPR 0.3) with a 0.1 and N + 50000. Each of the strong LLMs has a different level of alignment with human preferences controlled by noise value (0.05, 0.1, 0.3). The dashed line indicates the a target coverage. The empirical coverage of the rank-sets constructed using only N + synthetic pairwise comparison by one of the same three strong LLMs (not shown the figure) is 0.38 (u 0.05). 0.13 (u 0.1) and 0.0 (u 0.3).
1.0
0.9
RBO
0.8
PPR 0.05
LIM 0.05
0.7
PPR 0.1
LIM 0.1
PeR 0.3
LLM 0.3
HUMAN ONLY
0.6
2000
5000
10000
15000
20000
n
Figure 12: Average rank-biased overlap (RBO) of rankings constructed by ordering the empirical win probabilities i estimated using only + n synthetic pairwise comparisons by one out of three different simulated strong LLMs (LLM 0.05, LLM 0.1 and LLM 0.3), only synthetic pairwise comparisons by humans (HUMAN ONLY), and both n synthetic pairwise comparisons by humans and N synthetic pairwise comparisons by out of the same three strong L.MSs (PPR 0.05, PPR 0.1 and PPR 0.3) for a 0.1 and N + n 50000. Each of the strong LLMs has different level of alignment with human preferences controlled by a noise value computed with respect to the true ranking constructed by ordering the true win probabilities O, for p 0.6. The shaded region shows 95% confidence interval for the RBO among all 300 repetitions.
{0.05, 0.1, 0.3). RBO was
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification:
We provide the proof of Theorem 4.1 in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide pseudocode implementation of all the algorithms in Appendix B and describe the setup used in our experiments in Section 5 and Appendix C. In addition, we have released the code and data at powered-ranking.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
33

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification;
We have released the code and data at http://giihub.com/eewwrrs
baaminngprrdictio--owwweer powered-ranking
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please
the NeurIPS code and data submission guidelines Ihttps://nips.c//pbbiii httppuuuiiuuuuussooododuumsssrrs
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for benchmark).
open-source
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https://nips.co htttps://niss.cc/puull///////
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe the training and test details in Section 5 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars in Figure 6 in Appendix D.I, and Figure 12 in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on computer resources in Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.. http:pppsscccpuuccccctiiieuueeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative social impacts of our work in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release any data or models that have high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned in Appendix C and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets. paperswithoode.cmm/ie licenses for some datasets. Their licensing guide can help determine the license of dataset.
has curated
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code and data that have released at https//gtthab.com/Newwks. contains documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "VaM5xZUtjL", "reviewer": "Reviewer_dge6", "summary": "The paper tackles an interesting problem of evaluating ranking large language models automatically using a strong LLM as alternative to human preference estimates. The work primarily focuses on modelling uncertainty in such a ranking generated when compared to the distribution of human preference rankings. Since, pairwise comparison by humans are cumbersome and pairwise comparisons by strong LLMs are not completely consistent with human preferences, the authors propose a framework that improves upon pairwise ranking by strong LLMs. The authors propose a prediction power inference based framework to construct rank sets that provide coverage guarantees with respect to the true ranking consistent with human preferences.", "questions": "Did the authors also try few-shot prompting the strong LLMs by showing few examples on how to judge the responses ? Would be interesting to see if this leads to any change in final observation and insights.\n\nFor a small sample set would it be possible to empirically test coverage guarantees without the other metric being the proxy assumption?", "limitations": "Due to the inherent limitations of the benchmark, this work may also not be representative of tasks that require reasoning over long form inputs and specifically complex reasoning tasks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe work tackles an important problem concerning evaluation of ranking LLMs in an automated manner with respect to limited human preferences. The work discusses in details the drawbacks of existing ranking approaches and provides a statistically grounded framework (prediction powered inference) that works well in face of scarcity in human preference annotations.\n2.\tThe authors perform extensive evaluation on chatbot arena and propose two measures, namely rank set size and baseline intersection probability.\n3.\tThe proposed framework is useful for modelling uncertainty when using LLMs as judges and can also be applied to other scenarios such as modelling uncertainty in LLM  driven relevance judgements for offline evaluation of retrieval.", "weaknesses": "1.\tWhile the authors perform extensive evaluation, it might be a good idea to also test on other benchmarks like MT-bench or AlpacaEval related to approximation of human judgements. While authors already discuss the generalization aspect in limitations with regards to this, I would like to add it would also help address the concern regarding selection bias of test set and evaluators. Additionally, due to input limitations the benchmark may also not be representative of tasks that require reasoning over long form inputs and specifically complex reasoning tasks. Hence the leaderboard may only weakly correlate with real-world performance on these tasks. Additionally a minor point is that the metrics considered for rating response “relevance , helpfulness , accuracy , creativity and level” as shown in prompt might also change depending on the task: For instance when evaluating on a benchmark akin to QA tasks where precise information is needed creativity may not be a valid metric anymore. Hence evaluating on more benchmarks would help give a clearer picture on usefulness of the proposed framework.\n\n2.\tWhile it is appreciated that the work provides the proof for theoretical coverage guarantees, the claim regarding the coverage guarantees made in Introduction and beginning of section 4 should be revisited as true rank-sets (true probabilities unknown) cannot be computed for LLMs. The Baseline intersection probability is a weak approximation for coverage guarantees. Though the authors argue that baseline method approximates well the true rank sets due to being constructed from large number of human pairwise comparisons this might not necessarily hold due to selection bias, distribution shift and various other factors.  Without further evidence the claim that  Baseline intersection probability is a good approximation for true coverage measure is not well supported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Other benchmarks]** Since our computational framework is rather generic, it is true that it could be readily applied to other benchmarks. However, this would require significant funding and time, and given the on-going discussions about the reliability of LLM evaluation, we believe the NeurIPS community may benefit more from our computational framework if published early. Further, given the rapid development of both new LLMs and benchmarks,  we also feel that any claim of superiority of an LLM over others based on the estimated rank-sets on (static) benchmark data may be quickly outdated and thus have limited value. Under **[Baseline method]**, we discuss selection bias of the test set and the evaluators.\n\n**[Reasoning tasks]** The goal of our experiments is to showcase and validate our computational framework, and not to make a comprehensive evaluation of LLMs across different tasks. Therefore, we do not claim that the conclusions derived from the rank-sets estimated using data from the LMSYS Chatbot Arena platform correlate with real-world performance on reasoning over long form inputs or complex reasoning tasks. We will clarify this in the revised version of the paper.\n\n**[Metrics and tasks]** We agree with the reviewer that each type of task may need to be evaluated using a different set of metrics. However, since our computational framework is generic and does not make any assumption about the metrics considered for rating each response, we do not find any reason for our framework not to be applicable. That said, as discussed under **[Other benchmarks]**, conducting experiments on other tasks and/or benchmarks would require significant funding and time and, given the on-going discussions about the reliability of LLM evaluation, we believe the NeurIPS community may benefit more from our computational framework if published early.\n\n**[Baseline method]** We would like to clarify that our claim is that the baseline intersection probability is a reasonable proxy for the coverage with respect to the true rank-sets induced by the **distribution of queries used by users at LMSYS** and the **distribution of human preferences of the users at LMSYS**. In this context, we would further like to clarify that, in Appendix D.2, we have also verified that, using a more conservative baseline metric, our conclusions also hold.\n\nRelatedly, we do acknowledge that, if the users at LMSYS are not representative of the target query distribution and distribution of human preferences, the estimated rank-sets both by our method and the baseline method may be inaccurate (due to selection bias and distribution shift). However, studying such a setting is left as future work. We will clarify this in the discussion of the limitations in the revised version of the paper.\n\n**[Few-shot prompting]** We did not try few-shot prompting of the strong LLMs but instead used (almost) the same prompt used by Zheng et al. [12]. We agree with the reviewer that it would be interesting to investigate and optimize the type of prompting used to elicit pairwise comparisons by strong LLMs. However, this is a research question on its own and is also left as future work.\n\n**[Empirically test coverage guarantees]** During the rebuttal period, we have carried out an additional evaluation of our framework in a synthetic setting. Since in the synthetic setting, the true ranking is known, we have been able to validate the theoretical coverage guarantees without using a proxy metric. Refer to the (general) author rebuttal and attached pdf for more details. We will include this additional evaluation in an Appendix in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TBC97ztQHI", "reviewer": "Reviewer_G9Nw", "summary": "- Focuses on uncertainty in rankings using a small set of human pairwise comparisons and a large set of model estimated comparisons using a concept of rank sets. A rank set is a set of ranks a specific model can take. A large rank set indicates high uncertainty in ranking position and vice-versa a small set implies a confident rank assessment. The method works by constructing a confidence ellipsoid which in turn using methods of prediction powered inferences (methods to estimate confidence intervals when you have a small set of labeled gold standard data and a large set of machine labeled data). The paper employs the methods to rank 12 LLMs using data from LMSYS Chatbot Arena.", "questions": "NA", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Evaluations of LLMs is particularly challenging and estimating ranking of models for specific tasks an important area. This paper makes a good contribution towards this by investigating ranking uncertainty using prediction powered inferences. The overall setting (small set of human-annotated data and a large machine labeled dataset) is realistic and therefore the work lends itself to practical use as well. \n\n- The methodological contributions are interesting in itself and the concept of using rank sets to characterize uncertainty intriguing. \n\n- The exposition and presentation of material is good. Some of the plots are well structured and intuitive to grasp (Figure 3 in particular is well crafted).", "weaknesses": "- The main weakness is on evaluation. The paper proposes two metrics: rank-set size and baseline intersection probability. Small rank-set sizes are better - presumably as the confidence in ranks is better, and baseline intersection probability - large is better as it supposes the baseline method is closer to the true ranking. First: The paper should really report more standard ranking metrics such as precision/recall @ k, RBO, MAP or Normalized Discounted Cumulative Gain (NDCG). To factor in rank sets, you could use most probable ranking from your rank sets to compute these. Second: the absence of true rankings makes the empirical aspects less convincing. A synthetic experiment where true rankings are known perhaps would make sense to empirically demonstrate the main claims.\n\n- Use of machine labels just appear to widen the confidence bounds (e.g. Figure 3). The base conclusions on ranking appears to remain the same. In this instance, therefore, it remains unclear what the value of machine labeled information is. \n\n- Some additional insights into rank sets would add to the paper. For instance, how stable are rank sets to minor changes in $\\mathcal{M}$? Uncertainty in rank for one model depends the overall set of models being considered, so this would be interesting to study. There are possibly other facets, such as how rank sets can be used in practice or the quality of the machine labeled data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Evaluation metrics]** In our work, we focus on rank-sets as a measure of uncertainty in rankings and thus our experimental evaluation aims to assess the quality of the rank-sets estimated using our method and several baselines. In this context, we think that the ranking metrics proposed by the reviewer provide little information about the quality of the rank-sets. More importantly, we are unsure how to operationalize them in our setting. On the one hand, precision/recall @ k, MAP and NDCG are used in settings in which there is a set of items to be ranked and, for each item, one can measure whether an item is relevant or is not relevant. However, in our setting, it is difficult to measure whether a model is relevant or is not relevant based on win-rates. On the other hand, RBO is used in settings in which there is a ground-truth ranking. However, in our setting, we do not have access to a ground-truth ranking.\n\nThat being said, following up on the reviewer's comment, we have carried out an additional evaluation of our framework in a synthetic setting where the ground-truth ranking is known. Then, in this synthetic setting, we have computed RBO using the most probable ranking, as suggested by the reviewer, as well as the empirical coverage. Refer to the (general) author rebuttal and attached pdf for more details. We will include this additional evaluation in an Appendix in the revised version of the paper.\n\n**[Synthetic experiments with true rankings]** As discussed in the previous response, during the rebuttal period, we have carried out an additional evaluation of our framework in a synthetic setting. Since in the synthetic setting, the true ranking is known, we have been able to validate the theoretical coverage guarantees without using a baseline. Refer to the (general) author rebuttal and attached pdf for more details. We will include this additional evaluation in an Appendix in the revised version of the paper.\n\n**[Value of machine labeled information in Figure 3]** In Figure 3, note that, by using machine labeled information, PPR GPT4 allows us to draw the same conclusions as BASELINE using significantly fewer human comparisons ($n$ vs. $N+n$). In Figure 1, the value of the machine labeled information is perhaps more apparent. Therein, the results show that PPR GTP4 achieves narrower confidence bounds (y axis) and higher baseline intersection probability (x axis) than HUMAN ONLY, a baseline that uses the same number of human comparisons but no machine comparisons.\n\n**[Additional insights]** In our work, we do not aim to make a comprehensive empirical evaluation of rank-sets as an uncertainty measure or the quality of machine labeled data. Therefore, we leave the study of the sensitivity of rank-sets to minor changes in $\\mathcal{M}$, practical uses of rank-sets, and the quality of machine label data as interesting venues for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uq8StgG544", "reviewer": "Reviewer_EPH6", "summary": "This paper proposes a statistical framework to rank a collection of LLMs according to how well their output aligns with human preferences. This framework does this using a small set of human-obtained pairwise comparisons from LMSYS Chatbot Arena platform and a larger set of pairwise comparisons by a \"strong\" LLM and additionally provides an uncertainty estimate by giving a set of rankings for each LLM being compared. This study shows that, with at least probability threshold the user can set, the predicted rankings will eventually become increasingly likely to match the true order in which humans would prefer the models. The authors perform several experiments to empirically demonstrate the valitdity of the proposed framework.", "questions": "see weaknesses.", "limitations": "limitations have been discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The framework is clearly explained and the paper is easy to follow\n- The paper studies an interesting problem that focuses on ranking LLM in the context of scarcity of gathered pairwise comparisons by humans\n- The empirical evaluation is thorough", "weaknesses": "- unless i misunderstood something, the small set of human pairwise comparisons has length = 1. Although, the potential bias and truthfulness of the human pairwise comparisons has been discussed in the limitation section, i think that it could be interesting to explore the potential error propagation in the rank sets from erroneous human comparisons.\n- it is not clear to me, how the self-recognition [1] problem can be tackled with this framework. It has been shown that LLMs have non trivial capability of recognizing their own generation. Would this not be the case for one of the strong LLMs? wouldn't they tend to rank their generation higher? This fact combined with the previous question might affect the generazalization ability of this framework.\n\n\n[1]: Panickssery, Arjun, Samuel R. Bowman, and Shi Feng. \"Llm evaluators recognize and favor their own generations.\" arXiv preprint arXiv:2404.13076 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Small set of human pairwise comparisons]** The small set of human pairwise comparisons has length = $n > 1$ and we evaluate the performance of our computational framework for different values of $n$ in Figure 2. \n\n**[Erroneous human comparisons]** We agree that it would be interesting to explore the potential error propagation in the rank sets from erroneous human comparisons due to, e.g., bias, lack of truthfulness or strategic behavior, however, as pointed out in the limitation section, this is left as future work.\n\n**[Self-recognition problem]** If the strong LLM tends to rank its generation higher due to self-recognition, this is corrected by our method, as PPR accounts for possible biases of the strong LLM using the small set of $n$ pairwise comparisons by humans. In fact, in our experiments, Claude 3 suffers from the self-recognition problem and our framework corrects for such bias. More specifically, in Figure 9 in Appendix D.3, LLM CL3 prefers to rank the two Claude 1 models higher than the baseline but PPR CL3 corrects for this bias.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jc4DbQVYUP", "reviewer": "Reviewer_Fevo", "summary": "The paper studies uncertainty estimate in the LLM ranking problem, where the task is to rank LLMs based on their response quality. Ideally the labels should be from humans, but due to the cose, people use models such as GPT-4 as auto raters. There lacks a good study of uncertainty estimation in the problem. The paper applies prediction powered inference (PPI) to construct a rank set for each candidate LLMs. Experiments are conducted on the Chatbot Arena data. It is shown that the PPI approach work as intended to produce reasonable rank set size vs accuracy (when comparing with an oracle method using only human data) trade-offs.", "questions": "See above.", "limitations": "The authors list several concrete limitations. Most are treated as future work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The application of PPI to constructing rank set from pairwise comparisons from LLM evaluators is an interesting and timely application to the reviewer.\n\nThe proposed algorithms look sound by following PPI.\n\nOverall the experiments demonstrate some desired behaviors of the proposed approach. Some analysis, such as the structure of the rank-sets, are interesting.", "weaknesses": "Overall I am positive of the paper as I feel the problem is important and applying PPI is a good proposal. However, the reviewer is not enthusiastic enough to give a higher rating due to the following concerns:\n\nThe paper is a more or less straightforward application of PPI to rank set construction. The theoretical properties shown in this paper mostly strictly follow those of PPI. Thus, the depth of this work is not substantial enough to warrant a higher rating in terms of novelty and technical depth.\n\nThe rank set is at dataset level, which may not be very useful in practice. For example, compared to a standard usage of PPI on a numeric metric, which will provide a concrete internal. For the paper, people end up with a discrete rank set for each candidate model. While it provides some uncertainty by looking at the set size, one may still end up wondering how useful that is. For example, if model 1 has rank 2,3,4, and model 2 has rank 3,4,5. It provides some information that model 1 seems better, but the reviewer is not sure how useful it really is (e.g., by how much? - as the guarantee is at the set level). \n\nThere are several points about the experimentation that the reviewer is not certain about\n- As the authors acknowledged, only one dataset is used so the generalization is less clear. F\n- The baseline / ground truth still needs some processing, such as a regression fit. This is different from standard tasks where human ground truth are given without the need to process anything. Thus the reviewer is not fully convinced how solid the conclusions are, e.g., “questioning the rationals used an extensive line of work that … (use LLM rankings for evaluation)”\n- Figure 1 is not entirely convincing - at least the PRP methods do not achieve pareto frontier here. The reviewer understands the argument about the x axis, still, for a pareto problem, one may not really argue one method is better than the other if it is not pareto optimal.\n\nThe flow of the paper may be improved. The reviewer was puzzled about the methods and algorithm 1,2,3 when reaching to the experiments. Figure 1 is not very easy to interpret.\n\nSome other minor limitations / future work, some are discussed in the paper: iid assumption - in practice, there could be a bias, e.g. using active learning to send data to human.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Application of PPI]** To the best of our knowledge, existing work has always applied PPI to construct confidence intervals for numerical quantities. In contrast, our work is the first to apply PPI to construct rank-sets and does so in a very timely domain, LLM evaluation. \n\n**[Rank set vs numeric metric]** We would like to first point out that rank-sets have been used in the literature as measures of uncertainty in rankings (see, for example, [48, 50]). In our setting, we believe rank-sets may provide useful information—the rank-sets tell the practitioner to what extent the difference in win-rates across models, after taking into account the uncertainty summarized by the confidence ellipsoid, are sufficient for each model to be ranked above or below other models. That being said, a practitioner may decide to use our framework to obtain not only estimates of the rank-sets but also estimates of the win-rates, which is a numeric metric, and the confidence ellipsoid used to construct the estimates of the rank-sets. We will clarify this in the revised version of the paper.\n\n**[One dataset / generalization]** Since our computational framework is rather generic, it is true that it could be readily applied to other benchmarks. However, this would require significant funding and time and, given the on-going discussions about the reliability of LLM evaluation, we believe the NeurIPS community may benefit more from our computational framework if published early. Further, given the rapid development of both new LLMs and benchmarks, we also feel that any claim of superiority of a LLM over others based on the estimated rank-sets on (static) benchmark data may be quickly outdated and thus have limited value. \n\n**[Baselines]** Since the regression fit is just an empirical estimate of a set of means (the win-rates), we do not find any reason to raise doubts about the baseline. Nevertheless, during the rebuttal period, we have carried out an additional evaluation of our framework in a synthetic setting. Since in the synthetic setting, the ground truth ranking is known and given, we do not need to rely on any processing to validate the theoretical coverage guarantees. Refer to the (general) author rebuttal and attached pdf for more details. We will include this additional evaluation in an Appendix in the revised version of the paper.\n\n**[Pareto frontier]** In the conclusions we have drawn from Figure 1 in lines 264-278, we do not (mean to) claim that the PPR methods achieve pareto frontier against all baselines. In lines 266-272, we claim that, **in terms of baseline intersection probability**, PPR methods are better than baseline methods using only (the same number of) comparisons by strong LLMs. In lines 273-278, we claim that PPR methods indeed achieve pareto frontier against a baseline method (HUMAN ONLY) using the same number of human comparisons as PPR methods since they are better both in terms of baseline intersection probability and size of the rank-sets. To avoid any misunderstanding, in the revised version of the paper, we will explicitly highlight that the baseline methods using only comparisons by strong LLMs GPT 4 and Claude 3 return smaller rank-sets than the PPR methods.\n\n**[Flow of the paper]** Following the reviewer’s suggestion, we will improve the flow of the paper and the description of Figure 1 in the revised version of the paper. \n\n**[Limitations]** Following the reviewer's suggestion, we will expand the discussions of the limitations and highlight that, in practice, active learning may be used to gather human pairwise comparisons.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their careful and insightful comments, which will help improve our paper. We include point-by-point responses to each reviewer in individual rebuttals. Moreover, in what follows, we provide details of an additional evaluation of our framework using a synthetic setting, which we have conducted during the rebuttal period. We refer to this evaluation in responses to three of the reviewers in the individual rebuttals (dge6, G9Nw, Fevo). The results of this evaluation are attached as a one page pdf.

Since in the synthetic setting, the true ranking is known, we have been able to validate the theoretical coverage guarantees without using a proxy metric, and we have also computed rank-based overlap (RBO) using the most probable ranking, as suggested by reviewer G9Nw. We will include this additional evaluation in an Appendix in the revised version of the paper. In the following paragraphs, we elaborate on this synthetic experimentation.

Initially, we set the number of models to $k=8$. In each experiment, we generated a random vector of true win probabilities $\boldsymbol{\theta}$ (Eq. 1), which induces a true ranking of the models. To generate the win probabilities $\boldsymbol{\breve{\theta}}$ (Eq. 3), we added random noise to the vector $\boldsymbol{\theta}$, then re-normalized $\boldsymbol{\breve{\theta}}$. The noise was sampled from a $Uniform(-u,u)$ distribution, where $u \in (0,1)$ a parameter we manually set to simulate different alignment levels of strong LLMs to human preference. Intuitively, the larger the value of $u$, the larger the difference between $\boldsymbol{\theta}$ and $\boldsymbol{\breve{\theta}}$, and the less aligned the strong LLM is to human preference. In our experiments, we set $u=\\{0.05, 0.1, 0.3\\}$, simulating three different strong LLMs.

To draw reliable conclusions for each experiment, we created rank-sets $300$ times, each time using a different set of $N+n=50 000$ simulated pairwise comparisons by humans and the three strong LLMs, with an equal number of pairwise comparisons per pair of models.

Let $m_a$ and $m_b$ be the two models participating in a pairwise comparison, with $m_a$ being the model that gave the first response. We ensure that each model provides the first and second response to an equal number of pairwise comparisons. For each pairwise comparison, we generate uniformly at random a number $x \in (0,1)$.  For the human outcome, if $x < 2 \theta_{m_a}$ then the response of model $m_a$ is preferred ($w=1$). Similarly, for the strong LLM outcome, if $x < 2 \breve{\theta}_{m_a}$ then the response of model $m_a$ is preferred ($\hat{w}=1$). In every comparison we set $w’ = \hat{w}' = 0$.

From the generated pairwise comparisons, we computed rank-sets in a similar manner as section 5 in our paper, using $\alpha=0.1$. Since the true ranking is known, we can compute the coverage, shown in Figure 1 in the attached pdf. The coverage increases with $n$, in agreement with our theoretical result in Theorem 4.1. 

By sorting the models in descending order of the estimated $\boldsymbol{\hat{\theta}}$, we obtain the most probable ranking of each method and can compute the RBO of each method’s ranking with respect to the true ranking, shown in Figure 2 in the attached pdf. We can see that combining pairwise comparisons of humans and a strong LLM results in the highest RBO values.

Meta-review: This paper studies how to rank Large Language Models (LLMs) automatically, using a strong LLM as an alternative to human preferences. It achieves this by focusing on the uncertainty that arises in such rankings compared to the distribution of human preference rankings. 

The reviewers identified the importance of the problem, the statistically-grounded framework, extensive evaluation, broad applicability, clarity of presentation, and the novelty of the methodology all as strengths of the paper.

To further improve the paper, the authors should consider testing their approach on a wider range of benchmarks, tightening up their theoretical justifications, and so on. The reviewers left a number of potentially interesting suggestions that the authors should carefully consider.
Decision: Accept (poster)