Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation
Authors: Jasper Dekoninck
Maximilian Baader
Martin Vechev
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
model evaluation
rating
Primary area: foundation or frontier models, including LLMs
TL;DR: We present a new multivariate rating system for LLMs that is able to capture biases in judgments and converges faster to the optimal ratings.

Abstract: Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs). However, current rating systems suffer from several important limitations: first, they fail to account for biases that significantly influence evaluation results, second, they require large and expensive preference datasets to obtain accurate ratings, and third, they do not facilitate meaningful comparisons of model ratings across different tasks. To address these issues, we introduce Polyrating, an expressive and flexible rating system based on maximum a posteriori estimation that enables a more nuanced and thorough analysis of model performance at lower costs. Polyrating can detect and quantify biases affecting human preferences, ensuring fairer model comparisons. Further, Polyrating can reduce the cost of human evaluations by up to $41$% for new models and up to $77$% for new tasks by leveraging existing benchmark scores. Lastly, Polyrating enables direct comparisons of ratings across different tasks, providing a comprehensive understanding of an LLMs' strengths, weaknesses, and relative performance across different applications.

Published as a conference paper at ICLR 2025
POLYRATING: A COST-EFFECTIVE AND BIAS-AWARE RATING SYSTEM FOR LLM EVALUATION
Jasper Dekoninck, Maximilian Baader, Martin Vechev
Department of Computer Science ETH Zurich, Switzerland desperrdekoninck, mbaader, martin. martin.vechevjeiin vechev)@@if..tteee .et.n..eth..cchh
ABSTRACT
Rating-baac based human evaluation has become an essential tool to accurately eval- uate the impressive performance of large language models (LLMs). However, current rating systems suffer from several important limitations: first, they fail to account for biases that significantly influence evaluation results, second, they re- quire large and expensive preference datasets to obtain accurate ratings, and third, they do not facilitate meaningful comparisons of model ratings across different tasks. To address these issues, we introduce POLYRATING, an expressive and flex- ible rating system based on maximum a posteriori estimation that enables more nuanced and thorough analysis of model performance at lower costs. POLYRAT- ING can detect and quantify biases affecting human preferences, ensuring fairer model comparisons. Further, POLYRATING can reduce the cost of human eval- uations by up to 41% for new models and up to 77% for new tasks by leverag- ing existing benchmark scores. Lastly, POLYRATING enables direct comparisons ratings across different tasks, providing a comprchensive understanding an LLMs' strengths, weaknesses, and relative performance across different applica- tions.
INTRODUCTION
Large language models (LLMs) have become powerful tools across a wide range of tasks, sometimes
even outperforming human experts (AT@ 2024 Anil et al., [0233] Anthropic 2024 OpenAl 2023). To evaluate and compare the performance of L.Mss, various benchmarks (C lark et al. 2018
Cobbe al. 2021 Hendrycks et al. [022I and evaluation frameworks (Gao et al. 2023 Liang al, 2022) have been developed. These benchmarks aim to provide a comprehensive evaluation of LLM capabilities across tasks such as code completion, mathematical problem-sole solving, and multilingual understanding. However, the reliability of these benchmarks to accurately estimate model perfor- mance has been questioned due to various concerns about data contamination (Dekoninek et al.) 2024 Zhang et al. 22024), errors in ground-truth solutions (Gema et al. 2024), and the discrepancy between benchmarks and real-world performance (Lin et al. 2024).
Ratings for LLMs To evaluate LLMs more accurately in real-world scenarios, recent works have made use of rating-based evaluations with human or LLM-based judges (Chiang et al., 20246)
Dubois et al., 2024 Lin al., 20244. These ratings reflect the relative performance of LLMs on specific tasks and are used to construct leaderboards indicating their real-world performance. As shown in Fig. ratings are derived from preference datasets containing samples consisting of query Q, response by two models (0) and (1),, judge J, and judgment indicating the pre- ferred response r. We illustrate human preference datasets for several tasks like code-based (<)), mathematical (E and Chinese (fE) questions, along with preference dataset using an LLMM-based judge (e)). Current methods fit each task separately using maximum likelihood estimation (MLE) to obtain ratings Re for each model m, that predict the judge's 's preferences as accurately as possible.
Limitations of Current Rating Systems However, current rating systems suffer from several crit- ical limitations. First, is widely recognized that judges are influenced by biases that significantly affect their preferences (Hosking al. 2023 Wu and Aji 2023: Shi et 2024 Chen al. 2024)
20e4
Code available at https:///ithub. wwwp:tthwwooo.......ee.....a.tt wtt.:///ttttss/ooororttiiiii

---

Published as a conference paper at ICLR 2025
Standard Approach
Preference Datasets
POLYRATING
</>
</> Qu 1o 0 Jim) Qe 1o) my? d2.rz)
MLE
MLE
MLE
MLE
V
17
R
R
R FE
R &
16
On mu mai Sac
MAP
1m2 + R
R
R
Speee--peeiffic
Shared
:
Continuous Features
+
mk
Rb: </>
Rk
Rk
Rk
CA
Figure I: Overview of POLYRATING. Given preference datasets of n samples for models over various tasks, the standard approach needs to fit separate and independent ratings for each task and cannot leverage continuous features. In contrast, POLYRATING fits a single linear model for all tasks and can leverage continuous features. Attribution in App. A
expressive enough to capture these biases, leading unfair com- parisons of performance. Second, obtaining human annotations is very expensive. However, current systems are sample inefficient and do not take measures reduce costs. This inefficiency makes it hard for resource-consione constrai-dd LLM practitioners to use human evaluation for their tasks. Finally, the wide applicability of L.MMs requires comprehensive evaluation system to compare model per- formance across tasks. Yet, current rating systems suffer from shift-invariance, meaning that rating optimality preserved when shifting all ratings by an arbitrary constant. For instance, all code-based ratings R can be shifted upwards by 40 points while maintaining optimality, making the ratings on this task much higher than those on other tasks and rendering direct comparisons meaningless.
Yet, current rating systems
This Work: POLYRATING
address these limitations, we introduce POLYRATING, an expres- sive rating system designed to model shared continuous features and biases influencing judge pref- erences. As illustrated in Fig. [ POLYRATING fits all preference datasets simultancously using maximum a posteriori (MAP) estimation, i.e. using maximum likelihood estimation with additional priors on all parameters. These priors enable more robust estimation model ratings and act as regularizer to prevent overfitting. Specifically, for each model m, base rating Rm is optimized to reflect overall performance across tasks. Task-specific modifiers f are then added to this base rating to derive ratings for individual tasks. Additionally, shared parameters across all models Obias capture the influence of features such as answer length and readability on preferences.
Benefits of POLYRATING POLYRATING addresses previous limitations by design. By modeling shared features, POLYRATING is the first rating system that can quantify biases affecting judge pref- erences by estimating their impact on model ratings. For instance, we find that answer length bias boosts ratings significantly by 41 points in the Chatbot Arena (Chiang et al., 2024b). Additionally, POLYRATING improves sample efficiency and reduces the costs of evaluations for new tasks by up to 77% when collecting 10000 human annotations. POLYRATING can also leverage LLM-based evaluations or traditional benchmarks to obtain ratings for human evaluation, allowing us reduce its cost by respectively 38% and 41% when collecting 10000 samples. Furthermore, POLYRATING not shift-invariant, enabling the construction of a leaderboard that offers detailed insights into each LLM's performance across different tasks, unlike previous approaches. Finally, provide convergence guarantees and prove the optimality POLYRATING.
Main Contributions
In summary, our main contributions
Introducing ATING, a multivariate rating system based on MAP estimation ($3) Detecting the influence of several judge biases in human and LLM-based evaluations and for the first time estimating their effect on model ratings ($4.1).
Demonstrating that POLYRATING improves sample efficiency, reducing the cost of human evaluation by up to 77% for new tasks and by up to 41% for new models ($4.2).
Providing a multivariate leaderboard using POLYRATING, enabling relative model perfor- mance comparisons across tasks $4.3)
2

---

Published as a conference paper at ICLR 2025
2 RATING SYSTEMS
In this section, we introduce the necessary notation to formalize rating systems for LLMs.
Preference Datasets A rating system requires the availability of preference dataset, which con- sists of n games that capture the preferences of judge. In language model evaluation, a game consists of query Q. two language models mg and m,, and judge J. The result r of the game is determined by the judge's 's preference for one of the completions and is if my beats mo, denoted as my > mg, and if mg m.. Thus, we can represent game g as a tuple (Q, mo, mi, r).
Rating System For given set of k models, rating system assigns a score " E R to model ma indicating the relative skill of the model on the task. With these scores, the probability that m. wins against my can be computed using the Bradley-Terry model (BT-modell) (Bradley and Terry 1952):
%
P(m, > myl 73)
+ %
In most rating systems the scores are parametrized using the exponential function % cxp( R, /400) where R2 is the rating of m, and 400 is constant used to scale ratings (Elo 2008 Glickman 2002
Rating Optimization To determine the ratings of the models, a rating system aims maximize predictive capabilities for the observed outcomes of the games. The maximum likelihood estimate for these observed outcomes in the BT-model can be found by minimizing the logistic loss
C(D,R) R) = - - (t log P(gm. > 9mo RR) + (1 9.) log Pgmoo > -9m, RR)
E(
(1)
BED
for dataset of games D (g 9n) and ratings R
(Ri,.... . Re). Thus, the optimal rat- ings can be obtained by computing arg minR L(D,R), To obtain ratings for specific tasks, the optimization performed separately for each task on the task-specific dataset D.
Incorporating Draws However, the BT-mmoeel ignores the possibility of draws in games. Fol- lowing the approach by the Chatbot Arena (Chiang et al., (024b), we can generalize the BT-model by setting the outcome 9. equal to 0.5 for draws to obtain model ratings that can incorporate these draws. Although we explore several alternatives to the BT-modell that more explicitly model draws in App. B we found they did not offer significant advantages in practice.
3 POLYRATING
We now introduce POLYRATING, a multivariate rating system specifically designed for language model evaluation. This section first outlines the four design goals for effective LLM rating system and then explains POLYRATING and how it meets these objectives.
3.1 DESIGN GOALS FOR LLM RATING SYSTEM
1) Quantify Biases Both human and LLM- based judges are influenced by biases that affect their preferences (Chen al. 2024 Hosking al. 2023 Shi et al. 2024 Wu and Aji 2023). A robust rat- ing system must capture and quantity these biases to ensure fair comparisons
LLM performance, regardless of the judge. Current rating systems predominantly univariate and are therefore unable to include the necessary extra parameters to capture these biases. Only AlpacaEval (Dubois et al.] 2024) accounts for length bias, where the length of the model answer significantly influences the judge's preference. However, its fitted coefficient is not directly interpretable and does not quantify the bias's influence on ratings.
2) Leverage Existing Information Leveraging existing information can make a rating system more sample efficient and reduce evaluation costs. This information can come from LLae- evaluations traditional benchmarks, both of which indicate model performance and should there- fore give valuable information that can improve sample efficiency. However, current rating systems start from scratch for each new task and model.
3

---

Published as a conference paper at ICLR 2025
3) Task Comparability Univariate rating systems cannot directly compare ratings across tasks. This is due to shift-invariiace, meaning that adding a constant с E IR to all ratings does not change their optimality. Specifically, the loss of ratings R dataset D is invariant under the transforma- tion R + R c, i.e., L R) = LDD, R + c). Therefore, if R, and R2 are the optimal ratings for two different tasks, the rating difference for specific model RT R2, be shifted by any con- stant e without losing optimality. This makes it impossible to see how many rating points a model gains or loses from one task to another. A good rating system should eliminate this issue, allowing accurate performance comparisons of the same LLM across different tasks.
4) Optimality Rating systems aim to predict the outcomes of games as accurately as possible. It can be shown that in the limit of infinite data, the univariate rating system presented in obtaann the highest possible accuracy. While infinite data is not practically achievable, any new rating system should retain this optimality.
3.2 POLYRATING
Modeling Features Features can be continuous, like the length of answer, or discrete, like the task of the query. We model each feature as a function f: G x (0, 1} + TR that maps a game g E G and boolean to number quantifying the presence of this feature. The boolean i specifies whether the model for which we want to compute the feature is the first or second model in the game.
Modeling Ratings is essential to model ratings as a function of these features to capture bi- ases and measure task-specific ratings. For this purpose, we first note that ratings must be game- dependent to incorporate game-dependent features like query task or answer length. Furthermore, judge-specific biases are model-independent and must therefore be measured using parameters that are shared across all models. Finally, it is important that ratings remain interpretable and practical for leaderboards. Therefore, the rating for model in game g is modeled using the linear model
Rd (g)
Plass
Σ aofi [om, = m]) + Σ BT f/ [9mi = mm).
Here, Rhige E IR is the base rating, and d' are the number of features in the respective sums, ay E R is the weight for feature fj and is shared across all models, 55TT E RR is the model-specific weight for feature f/j and [[...] is the indicator function. Importantly, Eq. 2 serves as the key element of POLYRATING that enables it to incorporate all design goals. Indeed, the shared parameters measure the biases in the judge's preferences, addressing the first design goal. Furthermore, to obtain task- specific ratings, we introduce a task-specific feature Ma (g. i) = [g E Dima.], where Dusk represents the set of all task-related queries. We then define the task-specific ratings as Rm Rm + - While this assumes no additional task-specific features are used, the definition can be casily extended to incorporate them, ensuring that ratings remain adaptable varying evaluation contexts.
Optimization Objective We perform MAP estimation with normal prior the weights a, and 8 with mean O and deviations oj and of respectively. This leads to the optimization objective
(az)² 2
(Sm
Crau( (D, Rinee a. a C(to),
(g) + Σ
(3)
2
2
DED
m=1j=1
where Roase is the vector of base ratings, Q is the vector of shared weights, B3r is the matrix of model- specific weights, and ratings are computed using Eq. 2 In contrast to the loss function presented in $2 the dataset can contain games from different tasks.
Using MAP estimation instead of MLE allows POLYRATING to incorporate information from ex- isting tasks through its priors, thereby improving sample efficiency and reducing evaluation costs. Furthermore, evaluations with LLM-based judges can be considered as distinct task and can there- fore be used to improve sample efficiency. Traditional benchmarks can also be reinterpreted as preference datasets, where question indicates a preference for models that answered correctly over those that did not. If the models are both (in)correct, the judge has no preference. We can therefore leverage all this information, ensuring that POLYRATING satisfies the second design goal.
4

---

Published as a conference paper at ICLR 2025
However, while traditional benchmarks often predict ranking similar to human preference data, their absolute performance measurements do not always align well with human ratings. For example, an easy benchmark will cluster all models close together because weaker models can answer most questions correctly, making it impossible for stronger models to achieve very high win rates against them. This makes POLYRATING less effective, since the ratings from human preference data spread further apart than those from these easy benchmarks. To address this issue, we introduce a hyperparameter that rescales the observed win rates of traditional benchmarks. This hyperparameter is optimized on the training data to realign POLYRATING with the human preference data, allowing for a more effective combination of the two. Full details on this parameter are A
Lastly, POLYRATING removes shift- invariance. Indeed. task-specific ratings R R + 5 cannot be arbitrarily shifted compared to each other, since the priors on the task-specific weights (x B3 ensure that Cran (D Rome: ax, (3) * Lran D) Rause a, B + c) for any non-zero constant c. This
enables accurate comparisons of model performance across tasks and fulfills the third design goal.
Optimization In App. D we show that the optimization objective from Eq. (3) convex and twice differentiable with respect to Riuse a and B, allowing us to use standard optimization techniques to optimize the loss with respect to these parameters. Specifically, we use Newton's method for the modcl-specific parameters and L-BFGS for the shared parameters. Furthermore, prove in App. D that, under weak assumptions, the obtained ratings converge to the ratings that maximize predictive capabilities and thus fulfill the final design goal.
Rating Uncertainty We compute pivotal intervals using bootstrapping to obtain rating uncertain- ties (Tibshirani, 1984). Specifically, given the actual estimate of the ratings R and n bootstrap estimates R1, R1...., R,, the pivotal interval for confidence is [R R(1)) R - R(./2)), where R(1-0/2) and R((/22) are respectively the 1 - a/2 and a/2 quantiles of the bootstrap estimates. For brevity, we report the 2o confidence intervals obtained using bootstrapping in $ with detailed pivotal intervals reported in App.
4 EVALUATION
We perform series of experiments with POLYRATING that showcase its ability to quantify the influence of biases on the ratings of the models ($4.1). its improved sample efficiency for various use-cases (44.), and its ability obtain reliable and comparable multivariate leaderboards ($4.3)
4.1 BIAS DETECTION
We use POLYRATING quantify the influence of biases in both human and LLM- evaluation using a public subset of the Chatbot Arena (Chiang et al. (2024)) and Wildbench (Lin et al., 2024). This enables an accurate estimation of the effects of these biases on model ratings and allows us to compare the influence of these biases between human and LLM M--bedde judges.
Biases We briefly explain the measured biases and refer to App. fooo full overview of all biases along with their functional form. We include the well-known length bias (Dubois al.) [024]
Singhal et al. 2023), which measures bias with respect to the length of the completion, and position bias (Shi et al. 2024 Wang et al. 2023), which measures bias with respect to the order of the models in the game. We further use classifiers (Babakov et al.] 2023 Camacho-collados et al.) (022) to
evaluate the influence of formality and sentiment of the model's 's output. Finally, we compute the repetitiveness of the answer by measuring the number of unique tokens in the completion and check the influence of the readability of an answer using the Flesch Reading Ease score (Kincaid et al.) 1975). We model all of these biases shared features when fitting POLYRATING, allowing to accurately estimate their influence on the resulting ratings.
Results Table [ shows the effects of these biases on both human and LLM-bbsed judges. We present both the coefficient Obias and the average influence this coefficient has model ratings for given queries, i.c. E,( (a If 0) frizx 1))). To put these numbers into context, the difference between the first and the tenth best models in the Chatbot Arena is 50 rating points.
5

---

Published as a conference paper at ICLR 2025
Table 1: Fitted coefficients for the biases and their average influence on model ratings for both human and LLM-based evaluation. The functional form of fuius used for each bias can be found in App. The influence computed as By (Obius [ (g. 0) fuas 1)))
(a) Human Evaluation
(b) LLM-based Evaluation
Bias
Coefficient (a)
Influence (B)
Bias
Coefficient (a)
Influence (IE)
Length
130. 737.7
40.8
Length
251.87
48.48
Position
2.70=2.4
2.70+2.4
Position
37.53+1.1
37.53_1.1
Formality
- -119.89211.3
-1..17+1.4
Formality
37.5646.9
4.31=0.8
Sentiment
57. 17.4
7.90
Sentiment
4.31+6.5
0.43_0.7
Repetitiveness
22.10
4.6441.8
Repetitiveness
75.0448.3
9.12-1.0
Readability
72.93
10.75+1.8
Readability
32.56+s.0
3..22
Discussion for Length and Position Bias We recover prior results on length and position bias for both human and LLM-based judges (Dubois et al., 2024] [Singaa al.] [2023] [Shi et al., 2024] Wang et al. (023). Length bias is significant in both paradigms, though more so for LLM-based judges. This explains why length-controlling techniques achieve higher correlation with human judges (Dubois et al. 2024). In contrast, position bias is only significant for L.M.-based judges. Furthermore, in contrast to prior work, we can now estimate the effects of these biases on the ratings of the models, with position bias gaining model around 38 rating points when using an LLM-based judge and length bias gaining 41 and 48 rating points for human and LLM-based judges respectively.
Discussion for Other Biases The other biases reveal interesting patterns. First, all biases differ significantly between human and LLM-based judges. For instance, while readability increases the rating of a model by 11 points for human judges, decreases the rating by points for LLM- based judges. This indicates that LLM--basd judges prefer denser text, while human judges prefer more readable text. Furthermore, we find that sentiment and formality have significant influence on human judges, gaining models 8 and 15 rating points respectively. In contrast, LLM- judges are indifferent to sentiment and not influenced as much by formality. Lastly, we find that repetitiveness decreases rating on average by 4 rating points for human judges, while for LLM- based judges, increases the rating by points.
4.2 IMPROVED SAMPLE EFFICIENCY
We demonstrate that POLYRATING is more sample-efficient compared to traditional univariate ap- proaches and therefore reduces the cost of human evaluation. In App. C we show that slight adjustment of POLYRATING additionally enables an improved sample efficiency when evaluating new model versions.
Dataset We use the full Chatbot Arena dataset (Chiang et al. 2024b), which contains over one million questions across various tasks. Each question answered by two models and judged by the human that posed the question. We use the tasks that contain Chinese, code- basedb and hard questions for this experiment and refer for full description of these tasks App.
New Task We first showcase the improved sample efficiency when obtaining ratings for a new task. For this purpose, we vary the number of available questions from the task and compute the logistic loss with respect to hidden test set. In this process, and in all further experiments of this subsection, POLYRATING is allowed to use all questions in the dataset that do not belong to the task. Thus, we model the rating of a model yn for game y as Rm + Pask [g E D where Dusk is the dataset of all games from the task. The standard deviation of the prior on 9 is determined by running cross-vale validation on the current training set.
Results for the three tasks are shown in Fig. 2 We find that POLYRATING converges much faster than the univariate baseline. Measuring efficiency improvement as the fraction of extra samples the univariate baseline requires to obtain the same loss as POLYRATING when collecting 10000 samples, find that POLYRATING improves efficiency by 58%. 38%, and 77% for respectively the Chinese, code-based, based, and hard questions. Thus, POLYRATING can cut the cost of obtaining ratings for new tasks by up to fourfold.

---

Published as a conference paper at ICLR 2025
Logistic Loss
Logitiic Loss
L Logistic Loss
Unioriate Baseline
Univariate Daseiie
Univoriate Beseline
POLYRATING
10
POLTRATING
PoLIRXIING
Constant facing
Conetoot Rating
Contant Rating
10-3
10-2
10-1
10-3
10
10-3
10030
300000
00000
100 001
20000
304033
00000
1:000
200000 # Somples
10000
of Samples
# Samples
(a) Chinese questions
(b) Code-based questions
(c) Hard questions
Figure2 2: Comparison between POLYRATING and univariate baseline for different tasks. The x-axis shows the number of samples of the task the rating systems are using. The logistic loss shown is normalized by subtracting the loss of the best possible rating for that task. The grey horizontal line indicates the loss of rating system that assigns the same rating to all models.
Logistic Loss
Logistic Loss
Logistic Loss
10-1
Constont Rating
Univariate Busefie
Conetant Roting
10P
POINRATING
10-2
Uniioriate Rosaline
Univariate Roweine POLYRATING
POLVIATTNG
10-
Constart Rating
10-3
20000
304033
30000
1000M 12000
Samples
dt Samples
Samples
(a) LLM- based evaluation
(b) Traditional benchmark
(c) Decomposable task
Comparison between POLYRATING and the univariate baseline when leveraging informa- tion from existing benchmarks. For the left and middle plot, the 2-axis shows the number of human annotations used. For the right plot, the x-axis shows the amount of samples from the Chinese code task. The logistic loss is normalized by subtracting the loss of the best possible rating.
Figure 3:
LLM-based Judge Improves Sample Efficiency LLM-based preferences much cheaper to obtain than human preferences. Therefore, LLM-based judges are often used to obtain model rat- ings, despite being less reliable than human judges. However, POLYRATING can further leverage these ratings to converge faster to the ratings corresponding to human judges. Specifically, we model the rating of model for game g as
RT (g) = Riase + 01 [g E DILMI log(length(eg...)) + 3r" [g E DLLM]
where length(gy... ) the length of the models' completion for the given question. We use the public dataset from Wildbench (Lin et al. 2024) to obtain our LLM-based evaluation. Fig. 3(a] shows the logistic loss on a test set of the Chatbot Arena for a varying amount of human annotations. We find that POLYRATING converges faster to the optimal ratings than the univariate baseline. Specifically, the increase in sample efficiency when collecting 10000 human annotations is 38%.
Classical Benchmarks Improve Sample Efficiency Obtaining results from classical benchmarks is even cheaper than LLM-based evaluations. We leverage these benchmarks to increase sample efficiency. To do so, we first convert a benchmark to a preference dataset based on model accuracy as explained in |3 We then model rating as Rig + 3 [g E Doenchmark] and apply POLYRATING to increase sample efficiency for human evaluation. We specifically use the HixEvvllHard (Ni et al.) 2024) benchmark and show results in Fig. (bb] We find that POLYRATING significantly increases the sample efficiency. Specifically, when collecting 10000 samples efficiency increases by 41%.
Decomposable Task Some tasks can be decomposed into two or more subtasks. If these subtasks are prevalent in the dataset, but their combination is not, POLYRATING can obtain more reliable ratings for the combined task. For example, Chinese code-based questions are rare in the Chatbot Arena, but both Chinese questions and code-based questions are prevalent. For this example, we can model the rating of model for game g as
RT (g) = Rmase + 1.Th [ E Detinese] ] + AT [9 E Dewde]] + 5.gh [gg E Dende n Dehinese].
Fig. B(c) shows the logistic loss for varying numbers of Chinese code-based questions from the Chatbot Arena. POLYRATING's logistic loss at the start is almost as low as the univariate baseline's at the end, with a sample efficiency improvement of 25% when obtaining 10000 samples.
7

---

Published as a conference paper at ICLR 2025
Several models in multidimensional leaderboard fitted using POLYRATING. The given rank of the models indicates its rank in the complete leaderboard shown in Table in App.
Table 2:
Rank
Model Name
Rating
English
Chinese
Hardness
Code
1
gpt-4o-2024-05-13
1297 24.4
132744
328.7
1347.5
1548.0
2
2aaule-2--ssomee222244000
1286_7.1
29410.7
19413.8
14411.
44412.6
12
vi-large-preview
1237 24.3
427.6
3728.5
1727.6
1148.3
26
1187 #2.9
67+G.u
51
1ta.s
-10
56
mixtral-8x77 -airal-8877--nstuuut--....
1114
2547.1
37
1046.8
447.6
Table Several models in the leaderboard fitted using a unidimensional approach where each task is fitted separately. Modifiers for the tasks are shown instead of the fitted rating to make comparison with POLYRATING casier. The complete leaderboard is shown in Table in App. F
Rank
Model Name
Rating
English
Chinese
Hardness
Code
-pt-4o-2024-05-13
1283=a.2
1940.2
12.
5
1347.8
Blaudc-3-5-sonnee 20000020
12.7
23
45.
11
35411.3
10
parrge-peeview
1233
-19
84
747.1
1148.3
17
ilama-3-700-
1202
22:4.1
3218.0
515.3
016.3
53
1114=0.0
020.0
020.0
020.0
010.0
4.3
MULTIVARIATE LEADERBOARD
We now compare separately fitted univariate leaderboards with a multivariate leaderboard fitted using POLYRATING. Since the univariate approach is shift-invariant, we need fix the shifting constant for each task. We follow the approach of the Chatbot Arena and set the constant by fixing the rating of to provide comparable ratings for the models in the leaderboard.
to 1114 4 for all tasks. We will show that this approach fails
Results Table and Table show the ratings of several models in the leaderboard fitted using POLYRATING and univariate approach respectively. For full overview of all models, we refer to App. By examining the modifiers computed by POLYRATING, we immediately see the downside of the univariate approach. The Mixtral model performs significantly worse, resp. better, the Chinese, resp. English, task compared to base rating. Therefore, fixing the shifting constant using Mixtral results in significant ratings shifts for these tasks making cross-task comparisons impossible.
This effect is most apparent in the Chinese task. 97 of the 1 14 models included in the dataset gain rating for this task in the univariate approach, even though most models were not specifically trained contrast, POLYRA ATING shows that only 67 models gain rating, and
for the Chinese language. that the only ones do so significantly are models trained by Chinese model providers, such Yi, Qwen and GLM.
Inspecting the English task, we find that POLYRATING indicates that top models tend to lose rating for this task, while bad models tend to gain rating. This is in line with expectations as older and worse models were predominantly trained on English data, making them more suitable for this task and thus increase their rating. However, the univariate approach simply shows that more than 100 models lose rating for this task, with no discernable pattern.
RELATED WORK
Ratings
Rating systems have been used across various domains, such as sports
Elo 2008 Glick-
man
2002
Shelopugin and Sirotkin
2023 Sismanis 2010 Vaz et al., 2012). gaming (Herbrich movies Talaltinis and Stephanides 2022) and recommenda-
et al.
2007
Dangauthier et al. 2007
tion systems (Adomavicius et al. 2005 Chen et al. 2018 Kong et al. 2019). The widely recognized
Elo rating system (Elo 2008) and its extensions such as Glicko Glickmman 2002) are generic uni- variate systems based on the BT-model (Bradley and Terry 1952) that are widely applicable. Fur- thermore, various rating systems have been developed for specific cases and areas. For example, Elo++ (Sismanis) 2010) was specifically designed for chess, and TrueSkill (Herbrich et al. 2007]
Dangauthier al. 2007 has been further developed specifically for multiplayer online games.
8

---

Published as a conference paper at ICLR 2025
Ratings for LLMs Preference datasets for LLMs have become common to evaluate model pabilities in areas lacking ground-truth benchmarks. The most popular one is the Chatbot Arena (Chiang et al. 2024b), which contains over million user queries and evaluates models in various tasks such as code, math, and multilingual understanding. Wildhench (Lin et al. 2024), MT--eecch (Zheng et al. 2023), and AlpacaEval (Dubois al., 2024) are LLMM-bssed evaluation frameworks that have gained attention. Among these, AlpacaEval is the only one that applies length-control bias similar POLYRATING obtain higher correlation with human judges. However, this fitted bias is not directly interpretable and is less generic than the approach used in POLYRATING. Addi- tionally, POLYRATING employs priors on various terms to improve sample efficiency and eliminate shift-invariance, which AlpacaEval lacks. Therefore, AlpacaEval cannot be used to obtain any of the benefits of POLYRATING that were presented in $4
Multivariate Rating Systems Multivariate rating systems have been used before in recommen-
dation systems (Chen et al. 2018 Adomavicius et al. 2003 Kong et al, 2019 Abdi et al. 202T).
These developed systems extensions to the more classical Elo (Elo 2008 and Glicko (Glick- man 2002) rating systems. However, they are not directly applicable to LLM evaluation, as they do not take into account the specific biases and dependencies that are present in LLM evaluation. Fur- thermore, the limited numbers of models allow us to build an exact optimization algorithm, unlike in recommendation systems where approximate algorithms are necessary due to the high number of rated players (or products). These approximate algorithms are not suitable for LLM evaluation, as shown in App. B Furthermore, these systems do not include priors on the ratings, which are crucial for the sample efficiency of POLYRATING.
Biases in Human and LLM-Based Evaluation Several works have examined biases in both hu- man and LLM-based evaluations (Hosking et al., 2023; Clark et al. 2021 Wang et al. 2023 Wu and Aji, 2023 Shi et al. 2024 Chen et al. 2024; Singhal et al., 2023). Typically, these studies introduce biases model answers to observe their impact judge preferences (Wu and Aji 2023 Chen et al. 2024 Singhal et al. 2023 Wang et al. 2023). Additionally, they also investigate bias by asking more specific questions to the judges, rather than simply asking their preference (Hosking jet al. 2023; Wu and Aji (023). These techniques, however, do not apply to existing datasets and require additional annotations for specifically crafted answers. In contrast, POLYRATING can be directly applied to existing datasets without further annotation.
6 LIMITATIONS
We briefly discuss the limitations of POLYRATING. First, while POLYRATING provides a way to measure model strengths and weaknesses, these comparisons are relative to the other models in the leaderboard and do not provide an absolute measure of model performance. For instance, if all models in the leaderboard perform well on one task, and poorly on another, the leaderboard will not reflect this absolute weakness. Instead, it will only show weaknesses relative to the average performance of the models. This is a fundamental limitation of rating systems and cannot be solved by any system that works solely based preference data. To obtain absolute measures of model performance additional data sources, such as traditional benchmarks, are required.
Furthermore, POLYRATING still requires significant manual inspection and tuning since users must determine the modeling parameters and functions that constitute the rating, a process that can be time-consuming. A more automatic discovery of interesting and relevant dimensions, especially for bias detection, would help mitigate this issue.
7 CONCLUSION
We introduced POLYRATING, a multivariate rating system specifically designed for language model evaluation. POLYRATING enables more comprehensive evaluation of LLMS by capturing biases and dependencies on both continuous and categorical features in the evaluation. We demonstrated the existence and influence of several biases, such as length and position bias, and compared these biases between human and LLM-based judges. Furthermore, we showed that POLYRATING can leverage existing data to increase sample efficiency by 41% and reduce the costs of human eval- uations for new tasks by up to 77%. Finally, we showed that POLYRATING can provide more reliable performance comparison of the same language model across different tasks by solving the shift-invan hhfttinnvariane of the ratings across multiple dimensions.

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have included our code in the supplementary material with instructions how to run and reproduce all the results presented the paper. Furthermore, App. App.Dcootaais detailed proofs of all theoretical statements made in the paper, particularly with respect to the optimality of POLYRATING.
REFERENCES
Solmaz Abdi, Hassan Khosravi, and Shazia Wasim Sadiq. Modelling learners in adaptive educa- tional systems: A multivariate glicko-based approach. In Maren Scheffel, Nia Dowell, Srecko Joksimovic, and George Siemens, editors, IAK'21: 1 11th International Learning Analytics and Knowledge Conference, Irvine, CA. USA, April 12-16, 2021, pages 497-503. ACM, 2021. doi: 041144//444813,,44888888 URL httph:t////ooiii org/10. 1145/3448139
Gediminas Adomavicius, Ramesh Sankaranarayanan, Shahana Sen, and Alexander Tuzhilin. In- corporating contextual information in recommender systems using a multidimensional approach. ACM Trans. Inf. Syst., 23:103-1 145-145,,, 01 2005, doi: 10. 10.115//0057709.055577
Al@Meta. Llama model card. 2024. URL https::/ggithu..comm/te
main/MODEL_ CARD.md
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ay- oub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roclofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Laksh- Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312.11800 2023. doi: 10.48550/ARXIV.332..11
Anthropic. Model card and evaluations for claude models, 2024. URL https://ww// files. http:/ppp...cmm/rrouuctttto/aaaa
Card- 2.
Nikolay Babakov, David Dale, Ilya Gusev, Irina Krotova, and Alexander Panchenko. Don't lose the message while paraphrasing: A study on content preserving style transfer. Elisabeth Métais, Farid Meziane, Vijayan Sugumaran, Warren Manning, and Stephan Reiff-Marganiec, editors, Nat- ural Language Processing and Information Systems, pages 47-61, Cham, 2023. Springer Nature Switzerland. ISBN
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: The method of paired comparisons. Biometrika, 39(3-4):324 345, 12 1952. ISSN 0006-3444. doi: 10.1093/ biomet/39.3-44 -...//333..4..4.... URL httt::::://o..rog/gppgtt 1ttp:///0...1///mmm///tt/ 3ttp:///iooett9.......... 39.33-4..344
Jose Camacho-collados, Kiamehr Rezaee, Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthe- nis Antypas, Joanne Boisson, Luis Espinosa Anke, Fangyu Liu, and Eugenio Martinez Camara. TweetNLP: Cutting-edge natural language processing for social media. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pages 38 49, Abu Dhabi, UAE December 2022. Association for Computational Linguis- tics. URL https::/aclantthololo org.y.o.rrr/oooo wwwpo/2...22....oo emn demos.5
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or Ilms as the judge? A study on judgement biases. CoRR, abs/2402. 10669, 2024. doi: 10.48550/ARXIV. 2402.10669, URL httttttss//:iiorrr
Pei-Yu Chen, Yili Hong, and Ying Liu. The value of multidimensional rating systems: Evidence from a natural experiment and randomized experiments. Manag. Sci., 64(10):4629 4647, 2018. doi: 10.1287/MNSS.220
uRI.https:://// //tttps:///ooi.rr//1/// 1287/ 2017 2882
10

---

Published as a conference paper at ICLR 2025
Wei-lin Chiang, Lianmin Zheng, Lisa Dunlap, Joseph E. Gonzalez, lon Stoica, Paul Mooney, Sohier Dane, Addison Howard, and Nate Keating. Lmsys chatbot arena human preference predictions, 2024a. /tttsss/////a comm:/comptttiiinss/ssssyyyyeeee chatbot arena
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos. Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez, and Ion Sto- ica. Chatbot arena: An open platform for evaluating Ilms by human preference. CoRR, URL
abs/2403.0413 044412,, 2024b. doi: 1044850/ARXXV.2403..44 11xiv.2403.600 84132
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7282-7296. Association for Computational Linguistics, 2021. doi: N0.18633/1/2221ACL-LL00 act- 565
URL https://doi.org//D 18633/1/22211
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the A12 reasoning challenge. ArXiv preprint, abs/1803.05457, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14166 2021.
Rémi Coulom. Whole-history rating: A bayesian rating system for players of time-varying strength. In H. Jaap van den Herik, Xinhe Xu, Zongmin Ma, and Mark H. M. Winands, editors, Com- puters and Games, 6th International Conference, CG 2008, Beijing, China, September 29 Oc- tober 1, 2008. Proceedings, volume 5131 of Lecture Notes in Computer Science, pages 113- 124. Springer, 2008. doi: 10. 10.1007/978-3-5000 3-540-87608- 87608-31 978-3-540 5-0----40-8768-- 31
11. 1118-11_1.1 URL https::/doi...... http:///oooorr/1100/ 1007/
Pierre Dangauthier, Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill through time: Revisiting the history of chess. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances in Neural Information Processing Systems 20, Pro- ceedings of the Twenty-First Annual Conference on Neural Information Processing Sys- tems, Vancouver. British Columbia, Canada, December 3-6, 2007, pages 337-344. Cur- ran Associates, Inc., 2007. URL bttps://procedding..eerrpp http:///peeeeen......rrro./../rs cc/pappr/20//0////snn
-asars/s7755s.ttaa-aaattttt.... Aww.tbatttt...............w html.
Roger R. Davidson. On extending the bradley-terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329):317-328, 1970. doi: 10.
1080/01621459.19999 1970.. 10481082
Jasper Dekoninck, Mark Niklas Müller, Maximilian Baader, Mare Fischer, and Martin T. Vechev, Evading data contamination detection for language models is (too) easy. CoRR. abs/2402.008833, 2024. doi: 10.48550/ARXI1.4402.222 02823
URL. httpp:://di. wrt.a/////g....5////kkk 2402.
Yann Dubois, Balázs Galambosi, Percy Liang and Tatsunori B. Hashimoto. Length-controllee
alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404.00475. 2024. doi: 10448500ARXIV22444004477
httpst/t:://ddt www.:oggg/////... 48550/arXiv wwwpxxii...4..... 22442 2404.84475
A.E. Elo. The Rating of Chessplayers: Past and Present. Ishi Press International, 2008. ISBN 9780923891275.
Leo Gao, Jonathan Tow, Baber Abbasi. Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter. Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang
11

---

Published as a conference paper at ICLR 2025
Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023.
Aryo Pradipta Gema, Joshua Ong Jun Leang. Giwon Hong. Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are we done with mmlu? CoRR, 2etp/22460004422 2024. doi: 10.4855//RXXVV24006 04127. URL https://doi. org/10 485100./885////1 2406. 64127
Mark E. Glickman. Parameter Estimation in Large Dynamic Paired Comparison Experiments. Jour- nal of the Royal Statistical Sociery Series C: Applied Statistics, 48(3):377-394, 01 2002. ISSN 0035-9254.
doi: 00111111467--88760000
URL https:///oi org/10. 1111/1117111911 9876.
08159
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proc. of ICLR, 2021.
Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill(tm): bayesian skill rating sys- tem. In Advances in Neural Information Processing Systems 20, pages 569-576. MIT Press, January 2007. URL
http://wwwwmccooff.coooooo www.o//enns/reeeeeeaeeeeeeeeeeee /ttp
trueskil tmmm- a- bayesian ski ll- rating- system/
Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard. CoRR, abs/2309.16349, 2023. doi: 1044850/AARXVV22300.666.. URL https:://oo..org/10... arXiv.2309.12
J Peter Kincaid, Robert P Fishburne Jr, Richard I. Rogers, and Brad S Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. 1975.
Demei Kong, Jun Yang, Hongchen Duan, and Siyue Yang. Helpfulness and economic impact of multidimensional rating systems: Perspective of functional and hedonic characteristics. Journal of Consumer Behaviour, 19, 12 2019. doi: 10.1002/cb. 1796.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yük- sekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tat- sunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. CoRR, abs/2211 11.011111110000 10, 2022. doi: 10,48550/ARXIV.211.. 1,000900 1.09110. WtLLhttps::/// httpp:://oii.or///1..o arXiv 2211. 09110
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking Ilms with challenging tasks from real users in the wild. arXiv preprint 24t.:220000047700, 2024.
Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from LLM benchmark mixtures. CoRR, abs/2406.06563 06565, 2024. doi: 10.48550/ARXIV.44060000
URL http:////oooor//1..444.4
17Xiv.2406.0655
OpenAl. GPT-4 technical report. CoRR, 0itx/2333.08774, 2023. doi: 10448550WaxXxv2233..877..
V. Rao and L. L. Kupper. Ties in paired-comparison experiments: A generalization of the bradley- terry model. Journal of the American Statistical Association, 62(317): 1967. ISSN 01621459, URL
Andrei Shelopugin and Alexander Sirotkin. Ratings of european and south american football leagues based 2 with modifications. 2023.
12

---

Published as a conference paper at ICLR 2025
Lin Shi, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic investigation of position bias pairwise comparative assessments by llms. CoRR, abs/2406.07791, 2024. doi: 10.4850/ARXIV.206..77777
URL https:://o..org//////oo 48550/ 2406 87791
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way go: Investigating length conrelations in RLHF. CoRR, abs/2310.03711 2023. doi: 10.48500ARRXV2231000711
48550/arXiv URL htKLttts////00..rr///1LL http://bbo...rr11....85555////// http://rxx.v.......... 2310..0376
Yannis Sismanis. How won the "chess ratings elo the rest of the world" competition. CoRR,
abs/1012.4571, 2010.
Kyriacos Talattinis and George Stephanides. Ratingslib: A python library for rating methods with applications. Softw. Impacts, 14:1 14:100416, 2022. doi: 10.1016/.SIMPX.2222..0006 URL
https://6o1.org//30 1016/] simpa. 2622. 160416
Robert Tibshirani. Bootstrap confidence intervals. Stanford University. Department of Statistics. Laboratory for Computational.
1984.
Luís Vaz, David Carreras, and Wilbur Kraak. Analysis of the effect of alternating home and away field advantage during the nations rugby championship. International Journal of Performance Analysis in Sport, 12 2012. doi: 10.1080/24748668 10..611666...
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. CoRR, abs/2305. 17926, 2023. doi: 1044850/ARKXI223055.177
URL http:: wttp////......//.......i... 2305 17926
Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models. CoRR, abs/2307.03025 2023. doi: 1148550/ARXXIV.237..000000 URL /tdps:://oi.
hTg/10.48558//4
.38255
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination large language model performance on grade school arithmetic, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Bric P. Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging lmm-as-a-jddge with mt-bench and chatbot arena. CoRR, abs/2306.055 056385, 2023. doi: 10.
13

---

Published as a conference paper at ICLR 2025
A ATTRIBUTION
We provide attribution for the icons used in Fig. here. The code icon was obtained from flati- con.com and created by Royyan Wijaya. The Chinese icon obtained from flaticon.com and created by Freepik. The bot icon was obtained from fllticon.com and created by Nuriali. The math icon was obtained from flaticon.com and created by widphic. The length icon was obtained from fireepik.com and created by Surang Lineal. Finally, the readability icon was obtained from freepik. and created by Generie Detailed Outline.
B ALTERNATIVE RATING SYSTEMS
This section explores several alternatives the exponential rating system that solves the MLE of the logistic loss function, as discussed in $ Specifically, we evaluate two extensions to the BT-model and one alternative inspired by the accuracy metric commonly used in benchmarks. We then com- pare these alternatives with the exponential rating system terms of their predictive performance and demonstrate that their added complexity does not result in better predictions.
All models discussed here are compatible with POLYRATING and can be used as substitutes for the MLFE-based BT-model used in $
Rao-Kupper Model Rao and Kupper (1677) extend the BT-model to explicitly account for
probability of draw by introducing parameter 0 R. 0 2 1:
P(i > j1(111))
%
7 0y
P(j 117)17))
7
7s + -
(62 - 1)
Pii (1mi)m))))) (ni + 07: - 07g)
can be shown that this model follows from the hypothesis that a judge cannot tell the difference between two answers if the quality of the answers is close to each other.
Davidson-Model rameter 0 E R, 0 2 0:
Davidson 1970) propose a similar modification include draws, using pa-
P(i > jla, =
7/s ?/ 0 7775
P(j (/44)75)
7s Y/ + 7/ + 0 /775
P(i~ (1m)))))))))) 7/ + 7/ + 0 1777
Accuracy-Based Model Both extensions to the BT-model presented above still model ratings using an exponential function. However, for LLMs, could be beneficial to use ratings directly comparable to standard benchmark accuracies. Benchmarks where, for given question Q, model m, defeats model my if m, answers correctly and m2 does not. draw occurs if both answer correctly incorrectly, and otherwise ma wins.
be viewed series of games
14

---

Published as a conference paper at ICLR 2025
Loss of Different Rating Systems
Constant Rating
10
Davidion (9 13
Fous Kupper (9 v
Bracley
Accuracy Glidko
10-3
10.0
12.5
15.0
17.5
of Samples 10
Figure 4: Logistic loss for all four alternatives on the Chatbot Arena dataset for various sizes of the training set.
Let D denote the accuracy function on benchmark D. If we model draws as 0.5 points for each model, the win rates can be expressed as:
P(i > j|ms, m³) = Acc - Accp(m2)
2
P(j > 1/mm, my) Amccc((m2) Amc)(mi))
To adapt the BT-model to this accuracy-based approach, modify it as follows:
P(i - j|R,, R2) min
1, max
0, 1 + R₁ R))
1
P(j > i|R, Rτ)
min
1, max
0,
(1 R - R.)) Ri))
where the min and max functions ensure probabilities remain within the [0, 1] range. Fitting this model on a standard accuracy-based benchmark by minimizing the logistic loss from Eq. ① would exactly recover the benchmark accuracies (up to constant shift). In contrast, the exponential used in the standard BT-model would ensure the benchmark would not exactly recover the accuracies. Thus, the ratings obtained with this model would more directly comparable with accuracies from standard benchmarks.
Comparison Comparing these models is challenging because the Rao-Kupper and Davidson mod- els include an additional draw prediction. For predictive purposes, we are only interested in the logistic loss L from Eq. (I) to determine whether the additional complexity of the Rao-Kupper and Davidson models reduces the value of L on an unknown test set. Using data from the Chatbot Arena (Chiang et al.] 2024b), we compute L for various training set sizes. For the Davidson and Rao- Mno-Kupper models, we add 0.5P(i ~ jlys. 7r)) to both P(i > 5/% 7') and P(j > i/77,
First of all, as expected. Using approximate systems for LLM evaluation is not recommended, as these systems were designed for time-varying, large-scale rating systems for multiple million players.
that the approximate Glicko system GGlickman 2002) performs by far the worst,
Results are shown in Fig. The Roa- Moa-Kupper model performs the worst, while the accuracy-bass model is only slightly worse than the remaining two. Finally, both the Davidson and BT-model per- form almost identically. Due to the extra complexity of the Davidson model and the more frequent of the BT-model for LLMss decided the BT-model default for POLYRATING.
15

---

Published as a conference paper at ICLR 2025
C
MODEL VERSIONS
Models are iteratively improved through time. Evaluating these iterations is essential for tracking performance changes and understanding the impact of updates. For developers who create multiple versions of a model simultancously or in quick succession, it is essential to evaluate the relative performance of these versions cost-effectively as possible. However, this process be com- putationally expensive, especially when human evaluations are used. This section describes how POLYRATING enables more efficient evaluation of model versions.
C.I INCORPORATING MODEL VERSIONS
Model versions evolve over time, much like human capabilities in competitive games. Rating sys- tems often reflect this evolution by introducing time-dependent factors, such as the method proposed by CCulom (2008), which uses dependenn Bradley-Terry (BT) model. This approach incor- porates a Gaussian prior on consecutive ratings, ensuring they do not shift arbitrarily over time:
Re+1 Rs NN, o0)
Unlike human capabilities, model updates are irregular. Some models may never be updated, while others evolve rapidly. However, Coulom (2008) assumes that the ratings are updated at regular intervals. To remove this assumption, we modify the prior by only updating ratings upon a new release. For two versions 2 and U22 we impose regularizing prior on the base rating:
R
R ~ o0)
This adjustment enables faster convergence by constraining rating shifts between versions without requiring regularity. Using this prior, the loss function for evaluating versions becomes:
(Ro
R )2
vversson (D, Ruase: a. () = Lian( D, Rhuses at (] +
Σ
(4)
20
Here, VM represents all pairs of consecutive model versions. Thus, if model has versions U1...., Un. then {(11, {211, (vn-1, C VM. Note that we only consider versions of mi- nor updates. For major updates (e.g., GPT-3.5 to GPT-4), the update is not included in VM as the rating of the new model will likely be substantially higher and is not related to the previous model anymore. This loss function can be optimized using the same techniques described in } Therefore, no additional adjustments are needed to evaluate model versions using POLYRATING.
C.2 EVALUATING MODEL VERSIONS
Logistic Loss
10-1
We evaluate whether POLYRATING can effectively reduce the evaluation costs associated with assess- ing new model versions. To do this, we adopt an experimental setup similar to the one described in 44 Using the Chatbot Arena dataset, we first iden- tify models with multiple versions to construct the set VM. Out the 129 models in the dataset, 43 subsequent versions of earlier models. We then split the dataset into two parts: one containing games involving these subsequent versions, and the other including all remaining games. As before, the lat- ter set is used for both the univariate method and POLYRATING, while the former set is further divided into training and test splits.
Constant Rating
10->
Univariate Baseline
POLYRATING
10-3
0
10000
20000
30000
40000
of Samples
Figure 5: Convergence rate of the univari- ate method and POLYRATING when evalu- ating model versions. The x-axis represents the number of games available in the train- ing set associated with the subsequent ver- sions, while the y-axis represents the loss of the univariate method and POLYRATING
Next, we vary the number of games available in the training set for the subsequent versions and compare how quickly the univariate method and POLYRAT- ING converge on the test set. This approach allows us to evaluate how efficiently the methods handle new models, assuming the first version of each model has already been evaluated. The results of this experiment shown in Fig. 5 When obtaining 10,000 samples, POLYRATING shows 38% improvement in sample efficiency compared to the univariate method. These results indicate that POlYRATING significantly reduces the evaluation costs associated with new model versions.
16

---

Published as a conference paper at ICLR 2025
D PROOFS
We provide the proofs for the theorems mentioned in the main text here.
We first prove the convexity of the optimization objective in Eq. (3)
Theorem 1 (Convexity of the Optimization Objective). The optimization objective in Eq. convex and twice differentiable.
is
Proof. Twice differentiability follows immediately from the twice differentiability of the logistic loss and the squared penalty term. To show convexity, we make use of the following well-known facts about convex functions:
The sum of two convex functions is convex.
The composition of convex function with affine function is convex.
Since the logistic loss f(x) - log(1 + exp(=)) is convex, and since POLYRATING relies on a linear combination of parameters in the loss function, the logistic loss is convex in these parameters. The squared penalty term is also convex, as it is a sum of squared terms with a positive quadratic coefficient. The sum of two convex functions is convex, so the optimization objective is convex. O
Further, we show the optimality of POLYRATING by showing it converges to the same optimal rating as the univariate approach when fitted on multiple tasks at the same time.
For this purpose, suppose we have a task for which we want to obtain separate rating. Specifically, let D be a dataset of games between models. Let D-tak C D, resp. Dusk C D, be the set of games not belonging to, resp. belonging to, the task of interest. We show that as [D] - oc, the rating obtained by individually fitting the tasks is equivalent to the rating obtained by fitting all tasks simultaneously using POLYRATIN Intuitively, the extra prior term in POLYRATING will be of less importance as the number of games in the task of interest increases, and the ratings will converge to the same optimal rating.
Theorem (Equivalence of Ratings). Let D be a set of i.i.d. games between models mo. mM-1- Let Dresk C D and D-sask C. D be as defined above. Let Ranks resp. R-uak be the rating obtained by fitting the games in D Drusks resp. D sank using the optimal univariate rating system. Let R' be the rating obtained by fitting all games in D simultaneously using POLYRATING with the formula R (g) = R + -. [g E Duskl and define Rim R' + -Tm Finally, let the priors on respectively Rask and T be (0, Renc and R' mol will, up to a constant difference, converge to the same optimal rating R if all optimal ratings are finite. Similarly, R-ask and R' -aak will, up to a constant difference, converge to the same optimal rating Rijo if all optimal ratings are finite.
and (0, Then, D-mask] - and ( + cc.
To prove the theorem, first need several lemmas.
Lemma (Shift-Invarance of Optimal Ratings). Let be a set of games between models mo; ... M where there exists one model that has played all other models least once. If Rj and R2 both minimize the logistic loss L(D,R) R.) for D, then Ry - R2 is constant vector.
Proof. Without loss of generality, we can assume that the first model is the model that has played all other models at least once. We first note that for any constant E R, it holds that L(D, R +-++c c) C(D, R). Therefore, we can assume that the first element of each vector, namely R and R are both zero by applying a constant shift to both. We now show that R. R2.
rk-1)) )) is strictly convex. Since strictly convex functions have unique minimum, this implies that R₁ R2. We show strict convexity by computing the Hessian and showing that it is diagonally dominant with strictly positive diagonal elements. By Gershgorin circle theorem, this implies that the Hessian cannot have eigenvalues equal to zero, and is therefore positive definite. Since the Hessian is positive definite, the function is strietly convex, and the result follows.
We do so by proving that the function F(m = L(D,
We compute the diagonal terms of the Hessian of F. We denote by D(s)) all games where one model is my and the other model is my. We slightly change the notation such that the game result
17

---

Published as a conference paper at ICLR 2025
9r (i.)) indicates whether m, lost, matter the order of the models. We define zo and drop the division by 400 for convenience. We have:
1-1k-1 ΣΣ
F(z
Σ
9r log( exp(a, 2.))
1=0 gE)()
Thus,
FFF
fr (1 expl22 r;))² + (1 gr) exp(o; xx) (1 exp(x2; 221)2
explx, - xz)
Σ
dx?
j=0 (E.)
exp(-2;)
exp(x;)
O²F F
Ir (1 (x+xxp(=x;)) -r- (1+ exp(x,))))
Σ
dr,Or,
(c.o)
Since all terms in the thermore, the Hessian is diagonally dominant as the last sum is the sum over all off-diagonal terms in the same column and the first sum strictly positive since m, has played at least one game against mg. Thus, the Hessian is positive definite, and the function is strictly convex.
positive, the diagonal terms of the Hessian strictly positive. Fur-
Lemma (Limit Exists and Is Finite). Let be a set of f.idd games between models mo. Furthermore, assume all ratings are bounded. Then,
m m
lim 1D|+
min L(D.R) R)
(5)
D
R
almost surely uniformly converges
Furthermore, lim arg min L(D, R)
R)).
(6)
[D| R.Ro=0
almost surely exists and converges the optimal rating.
Proof. Let Dn be the first n games in D. We show that the functions C, 12* - IR defined by C, (R) LD,, R) converge uniformly (C,g, R)).
Thus, for any given > O we need to prove the existence of an N such that for all n > N and all R. (R) E,y (L(g, R)) e. Let > be chosen arbitrarily. We can group games with the same models together in the notation for Ln- More specifically, let w.y denote the weight of the coefficient associated with log(1 + exp(- -R + R)))). Then we can write:
(n) w i. log(1 exp(-RR RTT + RT"))
(R)
Furthermore, we can write the expected value of the loss as:
& k
Ey(L( Σ>
() P(i ~ 5)) P(g D(.)) log(1 exp(-R," + R)))
jm1
k P. log(1 + exp(-R." + RT())
where ~ denotes a draw.
Thus, obtain:
(n)
(R.
= Ey(g(g ()),
Pij
log(1 + exxp(-R R,Th + +R))
71
i= j=1
18

---

Published as a conference paper at ICLR 2025
(n) By the strong law of large numbers, the weights wij /n converge almost surely the expected value of the weights, i.. P.y. Furthermore, since the ratings are finite, log(1 + exp + RT) can be bounded by a constant B. Thus, for any e > 0, there exists an N such that for all n > N, Wi
Paol
for all i,j. Then, almost surely.
BR
12) W
L R
(C(g,
P.j
B
1=1 j=1
Bk2
=,
BB
proving the first part of the lemma.
For the second part, we note that Lemma iimpiie that arg ming R, Ln (R.) has unique solution. By uniform convergence limit of the minimizers of L, is the minimizer of the expected loss, and the result follows.
compact domain of continuous functions L., thus have that the
Now, we can prove Theorem 2
Proof. We prove that Rusk and R'ust will converge to the same optimal rating Risk assuming that Reakk0 = R' us = R = which can be assumed due to shift-invariance. The other implication is proven equivalently.
Let D(r) mnkk denote the first n elements of Drask and (x)) the optimal solutions found when fitting using pask Note that leave the size of Dussk in the sequence unspecified since it does not matter for this part of the proof. We note that POLYRATING optimizes the loss
c(a) (D(m) R') L(C R' Rusk)) + L(D(()) R' task) + task
c(u)
1
1
R taakk.
d
20
20
f=0
tack
with optimal solution ((),, Suppose now that R Re(n) does not converge Rüsk Then there exists subsequence and a 5 O such that (2.)) can assume this subsequence the full sequence.
5. Without loss of generalization, we lask R
lusk: R) uniformly converges to the function (R) = Eq.. [tasks R)) which has Ris as minimizer. By Lemma [ L is continuous and has a unique minimum that satisfies R = 0. Therefore, there exists an e >> 0 such that for all ratings R with Rg the following true:
By Lemma 2 we know that 1 L(D(M)
R - Rin 8 ('(R) L" L" >E
(8)
Since R (1) Lask converges to R task by Lemma
we know there an no > 0 such that for each n >> Aln-
L ("Riiusk) E 4"
Lask
Furthermore, due to the uniform convergence the loss, there exists an ni > such that for all n1 and all R.
C(D R) L- (R) task
<
(10)
4'
Finally, there exists an ng such that for all n > 122.
d B²
d d+ 1 4B²
(11)
n
20 task
n 20
4
where the upper bound for all ratings.
However, we can now define ((n) = ((()) (i and 3r(n) (n) R((n) -takk
-tnkk The inequalities above
mak
imply that for all >> max(ng, n1, R2), c(a) (D)m) Ru((1)) [(n) (D(m) R ((), since
19

---

Published as a conference paper at ICLR 2025
1 (m) C(D R (ask)) task
1
-(D( task: Rask) > L (R' (R'usk)
-Raskss)) 4
4
L. (R' lnsk)
-L" (Ri( + L. (Rosk) - L
- (Ress)
2
4
where the first inequality follows from Eq. 10 and the last from Eq. (8) and Eq. (9. Since Eq.
11
ensures that the difference the bias term can most differ by e/4, we find Full (D(m), R"("))
c(n) (D(m), R (())). Therefore, R (x) task cannot be the optimal solution, which is contradiction to the
optimality of R((n), Therefore, R takk must converge to R
E EXPERIMENTAL DETAILS
In this section, provide detailed descriptions of the biases and tasks used in experiments. In Table 4 we describe the biases used $4.1 in more detail. In Table we describe the tasks used in $ more detail. Finally, note that any using POLYRATING took most 6 hours on single CPU, even for huge datasets with a million samples, 100 models and 10 tasks.
We also briefly explain how adjust the win rates of traditional benchmarks improve sample efficiency human evaluation, as discussed in $4.2 As detailed in the accuracy- model in
App. B the win rate of mi over m2 in traditional benchmark can be written
P(i > jim; my) = [ Amc)(m1) AccD(m22)
P(j > ilms, my) = 1 Accp(m22 Accp(mi))
where Acep the accuracy function.
Benchmarks often exhibit significant variation in accuracy differences between models. For in- stance, in some benchmarks, models may have closely aligned accuracies, while in others, the differences may be substantial. This variation affects the win rate estimates between models. To address this, we introduce parameter W, which allows adjust the scale of win rates. The adjusted win rates are modeled as:
W Acc)(m)) - Acco(m2)) 2
P(i > j/m,, mj) = min 1,
P(j > i/m; m2) P(i > mmm jlm;,,;).
This adjustment ensures that can control the scale of win rates, mitigating the issue of varying accuracy differences. We optimize the hyperparameter using human evaluations from the training data. Specifically, we fit univariate rating model using win rates for a given W on the classical benchmark and evaluate the logistic loss the resulting ratings on the training data. The parameter with the lowest logistic loss is selected. Importantly, do not any unknown test data during this optimization process, ensuring that our approach can be applied practical scenarios without compromising the integrity of the evaluation.
20

---

Published as a conference paper at ICLR 2025
Table 4: Overview of all biases used in 4.1 The table contains description of the bias and functional form of the bias. Scaling constant were introduced in these functional forms to ensure that all biases output values within the same order of magnitude.
Bias
Description
Functional Form
Length
Measures the length of model answer for given question.
f(g. i) 10g1o (length( (gp)))
Position
Computes order of the model the game.
f(g, = [i=
Formality
Computes the formality of an answer computed by popular formality classifier (Babakov 2023).
f(g i) MMym, (gp)))
Sentiment
Computes the sentiment of an answer computed by popular sentiment classifier
f(g, 1) Migm, (gp))2
Camacho-collados al. 2022
Repetitiveness Computes the repetitiveness of the answer by computing the percentage of non-unique words in the answer.
f(g, 5 of repested worde in gim, (gp)
yom, (ty)
Readability
Computes the Flesch Reading Ease score (Kincaid et al. 1975) of answer.
f(g. i) = min(1, max(O,
Flesch (0p.
100
Table 5: Overview of all tasks used in $4 Arena (Chiang et al. 2024b)
For each task. the same data as the actual Chatbot
Task
Description
English Chinese Hardness
Questions that are English.
Questions that are in Chinese.
Questions that are considered hard by the Chatbot Arena. These are questions that classified as being in at least six of the following seven categories: specific, requires domain knowledge, complex, requires problem-solving, requires creative thinking, requires technical accuracy, real-world question. Questions that require code to be answered. Whether the judge is language model.
Code LLM
Table 6: Fitted coefficients for biases and their average influence on the ratings of the models for both human and LLM-based evaluation. The functional form of fuis used for each bias can be found in App. E The influence is computed as Eg (Criias fof 0) foas (g. 1)]) and indicates the average influence the bias has the rating of models for specific games. Errors shown 95% pivot intervals computed using bootstrapping.
(a) Human Evaluation
(b) L.LM-based Evaluation
Bias
Coefficient (a)
Influence (E)
Coefficient (a)
Influence (B)
Length
130.74
40.84
Length
251.87*7.
48.48+1.4
1.3
Position
2.70+2.3
2.70+2.3
Position
37.33
37.53
Formality
119.89
-15.17+1.5
Formality
37.56 6.7
4.81
Sentiment
57.42 10.9
10.1
7.90+1.4
Sentiment
4.31 +6.1
0.43+066
D.7
Repetitiveness
22.10
4.64+1
Repetitiveness
75.04 88.
9.12
7.3
Readability
72.93*11.0
11.6
10.75+16
Readability
-32.56
3.92+10
21

---

Published as a conference paper at ICLR 2025
F DETAILED RESULTS
In Table 6 we show Table | with the adjusted confidence intervals computed using pivot intervals instead of 2a intervals.
The full multidimensional leaderboard fitted using POLYRATING on Chatbot Arena data (Chiang al. 20246) be found in Table 7 can be found in Table g
The full leaderboard fitted using unidimensional approach
Table 7:
Leaderboard of human evaluation with modifiers fitted with POLYRATING. Indicated devi-
ations are 95% pivot intervals determined using bootstrapping.
Rank
Model Name
Rating
English
Chinese
Hardness
Code
1
8pt-4o-2024-05-13
1297 4.5
13 13.44.3
10.3
3-82
13 13+10.0
15 +0.4
2
(omude-3---somet-202222
1286 6.7
-22
19+15.0
14+13.6
44+
3
geminniaadvanccee-5514
1285 4.5
-25
8+92
3+10.3
2+11.
4
gemini-l.5-pmoo-pp4444444
127311
20+
19+10.0
15+10.0
11+113
5
claude33-opuu-222240229
1273
39
146.0
15
2.9
12
6
bard-jan-24-emmini-po
1271 12.6 12.6
+12.4
-881177
16.9
25 25
25.8
43 45..16.
11.7
11 +19.
18.4
7
gpt-4-1106-preveew
1265
1111
3
9
11 +9.3
4.8
8
1264 4.4
244.1
5479
14.8
12 12. 10.8
-85
-48
5.5
9
gp1-4-trbbo-2024--0--09
1258 +2.8
3.4
3436
4+74
11+8,9
38
15.
10
gp1-4-0125-preview
1255 4-.6
5 543
1+73
1409.0
249.5
gemini-l.-ffass-app-0111
1243+4.7
21_10.5
8
9+.
15 10.8
12
yi-large-preview
1237 -44
1-166
371
17+10.3
11 11+11.0
3.8
13
gemma-2-277-it
1232 -1.4
10
+25.2
116.0
13
13.6
7 +1..9
115
14
yi-large
1222 +8.8
7-7
15.5
9 177.9
16.5
11
+16.4
10.4
21 +18.2
12.5
15
nemoorron-4-340b-nnstue
1222+6.5 +6.5
7.2
133
18.1
7 12.4
1+.3.0
11.3
8.5
+4177
9.4
16
claude-3ssommeet222242222
1220
t28
15+59
483
26 +.22
366
17
command-r-plus
1214+3.3
18
6 6467
10.0
49.
37
18
gpt-4-0314
1213+4.5
13447
23
9
19
rcka-core-02240501
1212
11
g+..1
2 4.6
20
Saude-3-haiku-2224400
1209
30
36160
10t8.
16
21
gemma-2-9b-it
1204 10.7
17
-17.5
1211
+21.4
3 3+18.
12.6
11 +0.3
16.3
22
glm-4-0520
1202 9.6
+0.
49
17.8
19
16.1 16.7
12 17.4
15.8
17.7
11.1
12.5
23
gpt-4-0613
1191 3.8
2243
41 6.5
47.6
18
486
19.5
37
9.5
377
42
24
claude-1
1190 8.5
+79
18+ 16.9
19+13.3
14.8
25
reka-faahh-previee2220001111
1188
5 5-15.0
10
7-10.
26
llama---7b-iistruee
1187
67
51
1778
10
27
qwen-max-042.
1187 5.4
355.4
11.
11+11.
8 ++2.3
8.7.
28
qwen2-72b-instruct
1182+6.0 6.0
12.9
68+119
11.
13+11.6
6.8
3-7.
29
gemini-pro-dev-api
1182 7.2
38772
23
11
23
13.6
87
30
deepseck-coder-v22
1181 +9.4
9.3
344
31
49.6
16.6
1771
17.5
43
+16.5
61 +8.11
11.5
31
1176 +6. 6.9
12+6.9
115
13.0
1+13.4
32
command-r
1175+42
15
12775
+3.
9
reka- 212-2222hh22
1170
16+
12
6111.
5
34
claude-2.0
1161+10.8
26
8
30.99
12 13.0
35
mistral-large-2402
1163+4.0 4.2
2
32+77
20+000
10 +9.9
5.3
36
gpt-3.5-turbo-0314
1162+19.5 19.5
21.0
199
-5.0
g
2221
14
37
qwenl.5-1100--caat
1161 +4.9
124
58111.1
+0.1
11 11.12.5
550
7.8
38
gpt-3.5-turb-06113
1161 +6.
41449
37
13+10.5
21 122
6.3
39
claude-2.1
1156 +6.3
40.
45412.
10+10.8
22+11.8
40
mistral-next
1153+110
-0
50
14+14.9
8+17.7
41
mistral-medium
1153
21+11.1
++11.4
11+13,2
22

---

Published as a conference paper at ICLR 2025
Table 7: (continued)
Rank
Model Name
Rating
English
Chinese
Hardness
Code
42
milxtrll-8x22--mstttrtet
1152
5 5
718.
12
8_5.9
43
Ilam----b-instruct
1150 1150*3.5 3.5
48
41
-16+8.5
3.4
4
44
glm-4-0116
1149 10.7
45 45.
15.8
76 19.5
18.7
21
12
45
qwenl.5-72b-chat
1148 4.6
114477
-1066
5819
1-47
19
46
gpt-3.5-turbo-0125
1147 114
38 +3.5
4617.
11 19.22
19
3.8
10.1
47
oepiyr-orpo-441-AA33
1143 11.
+122
27+213
2 2
2
12.2
19.4
22
16.2
18.9
gemini-pro
1139+14.
7+146
2 2
23+
7422
49
claude-instant-1
1134
13
15
T-88
3-10.
50
wizardIm-70b
1129 1129+12-7
18.0
5
30
28.0
19717.7
13.6
26 16.3
51
1126 5.0
13+52
3
13
5.7
11
6.7
52
qwenl.5-32b-chat
1126 +8.8
5
69-111
8+117
22 13.3
6.0
-1,77
10.4
7.1
8.9
53
yi-1.5-34l--caat
1126 5.0
63+6.9
-2.2
1031121
5+12.6
7.6
+135
8.9
54
phi-3-mediin-44-nnssnccc
1126 4.3
1247.
11.8
7413.0
21
133.5
4.6
5 +15.2
11.1
55
tulu-2-dpo-70b
1122 1122+13-4
+124
72-28.7
1
mixtral--x7b-instruet-t...
1114
25-10.2
37174
10
4-4.8
57
openchat-3.5-0166
1114
348.5
3
17 17-11.5
58
qwenl.5-14b-chat
1112 6,5
10
57
10.10.3
8 122
-1-90
59
1amms-770bsteer/m-chhtt
1111 20.6
20.2
3 25.7
19144
28 +3677
33.8
13
25.0
19.7
52 +5.8
21.6
60
starling-Im--b--edd
1111 7.7
711
19
35
+300
7.7
18 18
133.7
10.5
1lab----0
1108 5.8
447
10,3
10.
10.4
18
10.4
58
15 11.1
62
gpt-3.5-turbo-1 1166
1106 8.7
36 48.6
14.7
62
33
844
14.7
20.3 15.3
20
11.5
vicuna-33b
1105
1746.6
27.36.3
12.6
18 18+14.1
64
phi-3-small--k-innsee
1103
27+7.6
16+12.8
13+13.5
5-94
65
openchat-3.5
1101 -2299
5-18
2-,3.3
2 2
23.5
66
dbrx-instruct-prexiew
1101 +5.3
25 4499
4
5 3.44
17 G.8
12.4
5.3
10.8
67
yi-34b-chat
1101 +5.1
36 +
36
94.16.5
6+140
6 10.0
+1.5
7.9
14.4
16.5
988
68
starting-Im-77balpha
1099 11.0
10.4
17
17.00.0
19.0
-22
1181
12.6
69
gemma-l.1-7b-it
1097
1495.9
11.4
10
922
3+12.7
70
pplx-70b-online
1095 +12.8
11
13 134314
39 +20.3
32 12.9
loppseo--Iin-66b-chat
1092
0
0-216
46 464
- -00
10 22.9
72
nous-hermess2-mixxttlk-8770
1090 20.2
25
+18.3
26 26435.4
42 42+21.5
17 23.0
18.6
24.8
37.1
16.5
21.5
73
qwenl.5-7b-chat
1086 13.4
13.7
19.7
72
85.5.
24.
16.0
74
wizardlm-133
1083 14.8
14.1
5-9966
+124
27.5
11.9
13 +300
19.1
75
Lama-2-13b-che
1081 7.9
7.3
12
58
18.1
15.0
- 100
+129
7.7
+15.0
10.4
76
gwen-l4b-chat
1081 + 15.3
16.1
35
+16.
20.3
+353
31-7
+22
17
17.9
33 20.4
25.2
vicuna-13b
1077 6.7
452
48.2
14..5
15+13.5 13.8
115.2
78
openhermes-2255-miilh
1075 +5..1
+14
13
19.77
23.2 23.8
79
2hi-3--mni-122kk-nnttt
1072 6.3
0
11.9
2-116
0
23 8.7
80
codellama-344-instee
1070 12.0
13.1
18.
57429.6
13+10.3
13.2
5 1-.7.3
81
phi-3-mini-4K-instruce
1068 +62
5.7
28+62
12.5
24+13.5
13.0
15 12.1
7.5
10 10
82
solar-
1077b-innstrrct-v1..0
1064 17.8
18.2
27+179
23
2
213
83
1
1060 + 25.1
24.5
30 1446
15 154
37.3
0 +2.88
31 +33.5
28.0
84
vicuna-7b
1058 + 16.1
36 +157
37 37
28.5
3 +0.3
18 +21
21.4
85
falcon-180b-char
1056+29.7
3
22
25
25
4
86
mistral-7hhinssmuut--0..2
1054+
55
12.8
6_127
6
3-3
0
87
zephyr-7b-alpha
1051 -6.8 26.8
22
28.7
14-37
18 18+34.
25.8
2 2
88
zephyr-7b-heta
1049 11.0
39 16.5
42+25.7
22
13 188.
11.5
12.9
89
gemma-1.1-2biit
1044+9.0
8.6
2
161
-
19.
11.0
16.0
25
166
12.3
90
mpt-30b-chat
1041 #20.5
21.3
31 10.8
21
389
373
12
+9.1
22.5
20
31.8
30.0
codellama-70b-instmen
1039 23.2
29 26.1
2261
4366
8 33.3
++3.0
30.3
34.2
28.6
28.99
23

---

Published as a conference paper at ICLR 2025
Table 7: (continued)
Rank
Model Name
Rating
English
Chinese
Hardness
Code
92
pplx- dil--bb-nniine
1038
14.6
35 1
21.0
21
27.8
-17
15.4
21 -217
18.3
93
lama-2-7b-chat
1036
45+7.9
15.3
28" 19.0
22 -22 14.4
-31
+16.4 +1..4
12.3
94
guanaco-33b
1035 21.6
30
17
-12
-53127
27.0
26.1
95
gemma-7b-it
1029 1029-11: +113
11.0
-16.5
38 18.7
16.5
11 11
11.6
7-18.7
96
stripedhyema-nous-76
1028 + 15.3
19.6
21 +1414.1
16 16+35.2
32.9
-19
19.2
19.2
16.2
10 -10118
+24.2
18.6
97
qwenl.5-4b-chat
1026 13.0
23+
16
1372
4 +19.3
10:7
18.2
19.0
18.2
15.4
mistral--b-instruet
1008 + 11.9
32+115
26 27.6
26.8
18.1
0+10.2
palm-2
997 197.
43
29
+17.
18 18
100
gemma-2h-it
995 14.7
13.9
20 20.9
33 24.6
19.7 19.7
25.0
101
olmo-7b-instruet
995 995 11.3 13.3
+134
59
18.3
54
30
14.0
8
16.6.7 16.7
+21.9
102
RWKV-4-Raven144D
971 19.5
+17.6
30
+17.5
28
30.0
23
23.6
744.
19.5
23.4
28.6
17.6
21.3
103
koala-13h
967 16.6
17.4
31
+16.6
22.0
25.5
24.0
16+1
15.0
5 +21
16.9
104
alpaca-1.3b
955 + 19.8
18
96
428
62
+213
78 21.8
27.9
17.7
24.9
28.4
16.3
105
chatglm3-6b
946 - 16.3
33 33+173
113+30.0 30.00
113
+21.8
2 21.6
106
mpt-7b-chat
944
8 +
37
37
+241
7 3
26.0
107
chatglm2-6b
930 19.2
20.7
30 30+21
67
38 27.1
108
spt4all-13b-snon
924 924 27.33
39 29.2
8 31.9
10 24.2
21 32.2
109
oastt-pythia-12b
912 912 +1.6
10 10 21.2
166.6
63
63
+21.3
12 26.1
18.1
25.1
18.4
20.0
110
fastchat-15-3b
879 879-20. 19.6
42+18.6
-4.8
108
108
26.6
36
+23.6
16.6
90
28.8
24.5
111
chatglm-6b
874+
11818.1
199
19.8
18
22.4
11 24.4
19.1
24.3
28.7
19.5
23.1
112
dolly-v2-12b
856 856+22.7
4219
15
421.9
27.6
8
30.9
266.
58
30.8
24.2
1lama-13b
853+24.0
26
4-24
8 33.2
27.7
36
426.9
89 32.6
26.1
114
stablelm-tune teet--uned--ae aeeddalpha-7bb
843+19.6
19222
30
-13-20.5
32 4279
Table 8: Leaderboard of human evaluation with modifiers when all fitted separately using a uni- dimensional approach. For the four tasks, modifiers shown indicate the deviation from the main rating to make comparison with POLYRATING easier. Indicated deviations are 95% confidence intervals determined using bootstrapping.
Rank
Model Name
Rating
English
Chinese
Hardness
Code
1
gpt-40-2004-05-13
1283
2.8
19
4.4
52+9.0
9.2
5+5.
13+6.3
6.7
2 claude-3-fssomnet-222420022
1267 4.1
23
6.9
45
11
3519.1
-42
6.9
12.1
3
gemini-advanccod-05144
1261 2.9
12.8
-28+44 44.8
69
++6.2
4+7.0
5.1
4 geminn-l.5--ro--pp-00544
1259 +27
-25 14.8
761
5+6.1
8
2.9
5
gpt-4-uurbo-22244044-99
1252+2.3
-12+3.7
50+74
5+5.1
14+6.0
101-4-110-ppreveww
1248+2.3
17+
52+75
0+1.
8
7
gemmi-llsspr---p-44400pymyyvev
1247+2.6
16.433
55
1555..
15
8
claud-3--ppus-222442229
1245+2.1
+31
70166
3-43
6
9
1p1-4-0125ppreview
1243+2.2 244
- 16 -6+337
3.6
53472
744
5.0
6.2
10
lrrlagge-preview
1233+2.6 2.8
19
44
84+89
7
11
11
gemini-1.5--ashh-pp-0051..
1226 +3.0
3.0
25 +..
66
+6.0
9 9+
7.3
12
yi-large
1215 +4.4
166453
58-1
6+1.22
18+11.9
15.1
13
gemma-2-276.2.
1210
+0+1
54
18+11.7
5 H44.
14
bard-jan-24-gent
1207 1207+4.7
+5
59.24.1
22.5
-11
35
13.3
15
glm-4-0520
1206
15
90
10t10.7
13+
16
nomoorron-43300hinnstme
1204 1204+3. +36
3.9
22
64
62+123
12.4
8.2
7 ++12
17
llama-3. 3-700-instruce
1202+2.
22+3.
32+65
5+15
5.5
221
34
5.3
18
chade-33-oomee222244229
1198 +2.2
2.2
23435
3.4
48461
6.
1
1745.4
5.1
19
reka-cone-20240501
1194+2.4 +2.4
20+
633
552
6.2
2.4
24

---

Published as a conference paper at ICLR 2025
Table (continued)
Rank
Model Name
Rating
English
Chinese
Hardness
Code
20
command-r-plus
1188
25
6311.
-.0
-21
21
gpt-4-0314
1188 2.5
23 1.2
54+8.
9+
10
8.3
22
qwen-max-0423
1185 2.0
-17 4.9
105 11.3
11.4
3
71
23
qwen2-72b-instruect
1184 -1.2-3.2
+3.3
-17
106
10.6
10.4
6699
118.6
-86
24
claudc33-haika-222243307
1181 +2.
-33
33 7.0
1+4.7
10
2.2
25
gemma-2-9bbit
1180
24+10..
56+19.9
100
-17
6.0
10.1
19.5
13.4
-16.1
26
deepseek-coder-v2
1177
28
80+
37+1033
61+114
glm-4-0116
1173
2-0.1
99
14
19
28
1wenl..-110b-ccat
1165
13
96 10.1
4+63
10
29
gpt-4-0613
1165 2.4
+22
18+3.
25176
3-5.0
3
30
reka-flash-preieee-22244.
1164 4.1
22
52 52
14
3
13.5
8.5
31
yi-1.5-34b-chat
1158+33 3.3
5+5.7
110 11.0
6187
3.4
-.8
11.1
2-80
32
reka-ffath-211-2224022----iiiii
1155 8.7
+3.6
19
H61
16 11.0
12
6 644
6.7
10.9
8.8
33
mistral-lange-222
1155 +2.5
9 440
18+79
10
14
34
Ilama-3-55
Hama-3-Sb-instruct
1154
11
15
16
6+54
35
1151
16
91
5.5
11
36
claude-1
1150 -38
21
49 13.7
25
12
37
command-r
1149 2.5
2.5
26 4.3
66 66 7.5
72
300
5.6
-33
6.0
38
tekk-flash-21b-2224022
1147 3.3
3.1
20
40
133
4
99
39
mistral-medium
1146 2.8
22
9
20 49.
145.0
8
10.4
40
mixtral-&x22b-instruee-
1146 2.7
10-12
40 135
g+53
9 9
27
gemini-rro-dev-pi
1136
26 163
+126
27
33.+10.0
42
claude-2.0
1133
19 1.
58+216
1-.0.1
4+
43
qwenl.5-32b-chat
1132 3.4
21
104
4+73
16
44
zephyr-orpo- 141b-A35b-v..1
1128 +63
6.0
g 10.6
22 18.8
18.1
-11
+14.5
16.1
5
17.2
45
mistral-next
1127 1 4.5
16.5.5
127172
3+9.8
1+1
4.5
6.9
17.
9-1
-118
46
phi-3-medium-4k-innrruuee
1125 4.1
+4.1
7
36 12.2
11.9
9
10 +9.9
9.2
47
gpt-3.5-turbo-0113
1120
21
43.
4
16
qwenl.5-14k5
1119
19.
89 498
147.
445
49
starling-Im--7b-oar
1119
12+6.
66
10.3
1
13
50
1
1118
22
32 32
2
15
51
yi-34b-chat
1116 3.9
3.8
9
116 14.7
14.
12
92.0.7
52
gemini-pro
1115 +6,2
16
52 30
+294
30 13.5
-11
-193
53
mixtral-&kxhh-isstuut--..
1114 +0.0
+0.00
+0.0
0
54
gpt-3.5-turbo00125
1113+2.4
240
1+5.4
12+6.1
22
7.7
6.1
6,5
55
claude-instant-1
1112+3.9
1616.1
38+175 17.5
4+87
2+113
56
wizardlm-70b
1110
11
10
+1+129
38+15.7
57
gpt-3.5-urro-03314
1110
26
95 30.5
9 9
8 21.1
58
dbrx-instruct-previerr
1106 31
2.8
2+1
30+82
2+62
15+74
59
phi-3-small-Bk-insttur
1104 +3.9
3.8
269
-
116
3+88
10.0
60
tulu-2-dpo-70k
1103 61
6.2
11 185
33
5t113
+1170
9.3
snowflake-arcti--instnn
1099 29
20
1.7
43492
-33
-19
28
48
9.6
62
openchat-3.55
1099
15 +63
48 14.0
-13+81
6 +11.5
2amb-hh-
1097 2.7
1
395
-50600
22+
64
vicuna-33b
1095
7-5.5
g
9-12.
-27
+2.
65
starling-Im--7baalphaa
1093
7-7.8
20
-17
-10
66
1emma-1.1--7bit
1090 3.3
3.3
39
-13+7.2
8-5.4
9.8
8.80
67
nous-hermes:
2-1ixttral--ee mixxtaal-&77b-dpo
1087 7.6
6.6
+114
48.
-35+15
4+187
68
lamma2-70b-stertmm-hhll
1083 + 7.9
8.5
122
12.6
11
447
35.5
-35
59
122
21.
69
openchat-3.5
1080
14+78
53
26.8
-14+12.1
244 +154
25.7
18.3
25

---

Published as a conference paper at ICLR 2025
Table (continued)
Rank
Model Name
Rating
English
Chinese
Hardness
Code
70
deepseck-llm-660--e
1080
-15
103+34.
33.8
-14+15.3
15.5
2
10.7
71
openhermes-2.5-misee
1080
6
8 8+34.
-00 15.1
14.8
18 20.2
18.9
72
qwenl.5-7b-chat
1079 6.3
-.3
-221
122 22.7
23.8
-10 12.9
13.7
13
17.17.
73
pplx-70b-online
1077 5.9
-6.1
-22
54 12.66
30.5
-49
-3.7
-15.55
74
mistral-7h-insttuuttv0222
1075 +3.77
7
8 1+.12
6+13
0
3.5
75
gpt-3.5-turb--1106
1073 +3.9
18+62
10+17.
20
25+10.8
3.8
6.9
19.0
9.3
10.3
76
phi-3-mmini4Kk-inssuett
1072 1072+3.7
3+6.1
12+12.0
8+7.1
16+8.4
77
llama-2-13--ch
1068
6-3
17-15.4
19
16.
78
solar-
10.7--inttrutt
1067
1-1111
5
0 0
35.5
9
-10_1.1
16
+-9.8
79
21pphin-.2.1-mmitrra-
1066 11.3
11.4
5-1
44
-15
-0 34.8
80
wizardlm-13b
1063 5.6
+6.0
28 28
37
8.
26.0
14.3
81
7ephyr-7b-heta
1057 +5.2
2
+24
29
-24
5.0
-71
-413
82
phi-3-mini-128k-innsse
1054 3.6
3.4
15 167
17-114 11.
- 74
15+
23 9.2
83
vicuna-135
1050 +
19
63 16.5
-2+
-15
84
mpt-30b-chat
1050
5 5-.1511
3 3
2-25.0
21
30.3
85
codellama-34b-instmaett
1049
13
17 29-4
199
8 17.2
86
zephyr-7b-alpha
1048 10.99 10.9
6 16.4
1 68.44
27 27.
14
32.8
87
codellama-70binnste
1047 13.5
14.6
3 22.0
2228
62
39..8
37
28.6
2 36.9
+4.4
88
pplx-7b 7b-onln
1045 6.6
+64
6+02
47 33.3
3318
28 144
17.
89
gemma-7b-
1043 +..
5.2
84333
64 16.0
15.6
4+11,8
7 +13.7
12.8
90
llama-2-766 2-7b-chat
1043 +0.
163
11
3249..8
15.6
qwen-14h-che
1041 16.8
21
1103
94434.7
166
16 1
92
falcon-180b-cee
180b-chat
1039 15.3
12 12_21.2
6 749.9
-32+32.
22
93
guanaco-33b
1037 9.4
111777
8 37.8
35.
69
27.9
94
gemma-1.1-2b-44
1034 4.7
4.5
15 +8.0
54 13.9
- 155
-15 +10.2
8 11.1
8.6
138
10.6
95
stripedhyena-nous
1024 6.8
g 999
+99
+36.3
-66 14.3
21 +18.8
6.8
36.7
-17.9
96
olmo-7b-instruct
1021 +6.1
0 +99
62 184
26 13.2
+22.
5 +17.3
6.0
9.7
18.5
97
mistral-7b-instruet
1016 +3.
+8.1
26.1
25.7
- -1011,
12.6
+15.3
palm-2
1012 5.7
1487
69 4306
-
22+16.
99
vicuna-7b
1012
20
30
-0
100
--bbhhat
1003 +5.5
30+32
92 92+15.8
- 21+
111
5.8
101
gemma-22 2b-it
1000 6.7
6.6
11
67
- 12 14.9
0 18.4
102
koala-13b
971 66.0
6.6
6199
31 25.7
43
13.4
30
+18.7
18.9
103
chatglm3-6b
962 962+7.8 7.9
7.7
11:22
+11.
159
+33.1
35.2
711.
9 22.6
104
gpt4all-13b-snoozy
941 +1.
+18.2
6 48.6
+
28
35.3
12,2
17.0
44.9
28.5
37.5
chatglm2-6b
936 9..0
74149
120
29
23.2
106
mpt-7b-chat
935 +85
15_11.9
73+340
32.0
36 20.6
28
107
RWKV-4-Raven-I1
929
19
29 29131.6
36 18
30 21.2
108
alpaca- 13b
911 +70
13+103
65 30.9
88
117
7.2
17.7
109
oasst-pythia-12b
902 +7.
g+116
41 41
18 18
26 26+ +2.4
110
chatglm-6b
889+7.9 -7.
6.8
25+108
252+30.0
26.6
9
16.4
1 +22.5
11.33
21.6
111
fastchat-15-3b
879 7.9
++117
22 172.3
66
+88.
19.4
117 125.8
25.2
112
trablemm-tuncd-alppa--
851 +9.7
113113
52435.8
-17
10
dolly-v2-12b
828
202
53535.2
-27+21.
78
114
1lama-13b
806 806
-3
17 16.2
-7.7
-131 +35.4
10.6
26


Reviews:
[{"review_id": "rNcvEnXDl8", "reviewer": "Reviewer_6vXX", "summary": "The authors propose, POLYRATING, an large language model evaluation system. POLYRATING can detect and quantify biases that affect human preferences, improve sample efficiency, reduce evaluation costs, and enable cross-task model scoring comparisons. It achieves a significant improvement on the sample efficiency, reducing the cost of human evaluation by up to 77% for new tasks and 41% for new models.", "questions": "Will POLYRATING also be appliable to other domains that use large language models, such as computer vision, information retrieval, etc.? How is the cost for the adaptation to these domains?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- POLYRATING is a multivariate rating system that can detect and quantify biases affecting human and LLM-based evaluations of LLMs.\n- Such effective system reduces the cost of human evaluation by up to 77% for new tasks and 41% for new models, which is meaningful and practical in the real-world applications.\n- The paper is easy to follow.\n- Comprehensive experiments are conducted, demonstrating the effectiveness of the proposed method.", "weaknesses": "Null", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uiIOWJMVu2", "reviewer": "Reviewer_2PJr", "summary": "The paper addressed that current rating systems for LLMs suffer from several limitations: biases, human annotations, and shift-invariance.\\\nThen, the authors propose POLYRATING, a multivariate rating system based on MAP estimation.\\\nPOLYRATING uses maximum a posteriori estimation (MAP) to estimate model ratings across tasks, including base ratings and task-specific adjustments.\\\nIt also incorporates shared parameters to model factors like answer length and readability that affect preferences.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The notations and equations throughout the paper are clear, well-formulated, and add rigor to the methodology.\n2. The justification for POLYRATING is robust, with a well-explained rationale for the multivariate approach.\n3. Extensive experiments demonstrate the effectiveness of POLYRATING, showing its superiority over traditional rating methods in several key areas.", "weaknesses": "1. The modeling features are somewhat ambiguous; it is unclear which specific features are applied for each task, or the criteria used for feature selection. More detailed information on feature selection per task would be helpful.\n2. POLYRATING employs both model-shared and model-specific weights. It would be useful to clarify how the system initializes weights for newly added models and whether it can efficiently update ratings for new models without requiring a full retraining of the rating system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y6UenRISPe", "reviewer": "Reviewer_iGCp", "summary": "This paper introduces Polyrating, a novel multivariate rating system designed specifically for evaluating Large Language Models (LLMs). The system addresses three key limitations of current rating approaches: bias awareness, cost efficiency and cross-task comparability. Polyrating uses maximum a posteriori estimation with shared parameters to quantify and account for various biases in human and LLM-based evaluations. The system demonstrates up to 77% cost reduction for new tasks and 41% for new models by leveraging existing benchmark scores. It also enables direct rating comparisons across different tasks by solving the shift-invariance problem inherent in traditional rating systems. The paper provides thorough empirical validation using public datasets like Chatbot Arena and theoretical proofs of the system's optimality.", "questions": "1. What is the computational complexity of the optimization process and how does it scale with the number of models and tasks?\n2. How sensitive is the system to the choice of priors? Could you provide guidelines for selecting appropriate priors?\n3. How does the system handle rapid changes in model capabilities over time?\n4. How does the system handle extremely sparse preference data for new tasks or models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper makes good practical contributions by introducing a novel framework for LLM evaluation. Fundamental limitations in current approaches are well addressed, providing strong theoretical foundations with proper convergence guarantees and demonstrating substantial practical improvements.\n2. The authors present comprehensive empirical validation across multiple datasets with comparison against baseline approaches and clear ablation studies showing the contribution of each component.\n3. The significant practical impact demonstrated (up to 77% cost reduction for new tasks and 41% for new models by leveraging existing benchmark scores) offers immediate positive value to the field.\n4. The paper is generally well written with clear problem motivation and logical flow from theory to implementation to results.", "weaknesses": "1. There is limited discussion of computational complexity and scalability analysis. The paper could include runtime analysis for both training and inference phases of Polyrating, and recommended batch sizes and optimization strategies for large-scale deployment. Similarly, a short discussion of how the MAP optimization scales with number of models (k), tasks (d) and samples (n) would be useful.\n\n2. The hyperparameter selection process for standard deviations sigma_j and sigma_j' could be better explained. The relationship between dataset size and optimal prior values should be analyzed and sensitivity analysis showing how different prior choices affect final ratings would also be valuable.\n\n3. The system assumes static model capabilities but the paper should address rapid model changes with, eg: analysis of rating stability over time, methods for updating ratings incrementally as new data arrives, guidelines for maintaining rating consistency across model versions, etc.\n\n4. The paper doesn't explore well on how the system performs with extremely sparse preference data (cold-start scenarios). The authors could analyze the minimum number of preferences needed per task/model pair. Also, methods for handling completely missing task-model combinations should be detailed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4yWWI3ehPa", "reviewer": "Reviewer_1REa", "summary": "This manuscript introduces Polyrating, a multivariate rating system for evaluating Large Language Models (LLMs). The authors aim to address several key challenges in current LLM evaluation methods, including accounting for biases, reducing the cost of human evaluations, and enabling meaningful comparisons of model performance across different tasks. Polyrating uses maximum a posteriori (MAP) estimation to incorporate different tasks and biases into the rating process. Experiments are conducted on publicly available human preference data to demonstrate the effectiveness of the proposed system.", "questions": "1. Line 233: Why are different optimization methods (Newton's method and L-BFGS) used for different parameters? What's the rationale?\n\n2. How are sample efficiency improvements (e.g., \"38%\" in line 361) calculated?\n\n3. In Table 2 and 3, several models are chosen to illustrate the differences in task comparability between Polyrating and a unidimensional approach. The following questions arise:\n\n3-1: Line 410: What specifically makes Mixtral's performance \"very different\"? The tables don't clearly show this. Providing a more precise explanation or visualization of these differences is necessary to support this claim.\n\n3-2: Line 413: The Chinese task analysis uses proprietary models with undisclosed training details and the Mixtral model, which, due to its specific role in the Chatbot Arena setup, is not suitable for this comparison. Please consider revising the experimental setup to enhance the validity of the comparisons.\n\n4. While existing multivariate rating systems are mentioned, their performance isn't compared to Polyrating. Include empirical comparisons and highlight performance differences would significantly strengthen the manuscript.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors clearly identify important limitations in existing LLM evaluation approaches, specifically focusing on the influence of biases, the high cost of human evaluation, and the lack of comparability across tasks. The proposed Polyrating is designed to directly address these challenges, providing a clear motivation for the work.\n\n2. The manuscript provides a theoretical analysis of Polyrating, demonstrating its convergence properties. This provides a strong theoretical foundation for the proposed system and strengthens the confidence in its practical applicability.", "weaknesses": "1. The manuscript lacks a comparison with existing multivariate rating systems. Empirical comparison and discussion of adaptations for LLMs are needed.\n\n2. Experimental evidence to support some of the claimed benefits of Polyrating, is insufficient. Model selection for task comparability is questionable. The efficiency analysis lacks clarity on calculation methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)