Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation

Authors: Jasper Dekoninck, Maximilian Baader, Martin Vechev
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, model evaluation, rating

Abstract:
Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs). However, current rating systems suffer from several important limitations: first, they fail to account for biases that significantly influence evaluation results, second, they require large and expensive preference datasets to obtain accurate ratings, and third, they do not facilitate meaningful comparisons of model ratings across different tasks. To address these issues, we introduce Polyrating, an expressive and flexible rating system based on maximum a posteriori estimation that enables a more nuanced and thorough analysis of model performance at lower costs. Polyrating can detect and quantify biases affecting human preferences, ensuring fairer model comparisons. Further, Polyrating can reduce the cost of human evaluations by up to $41$% for new models and up to $77$% for new tasks by leveraging existing benchmark scores. Lastly, Polyrating enables direct comparisons of ratings across different tasks, providing a comprehensive understanding of an LLMs' strengths, weaknesses, and relative performance across different applications.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729916760816

thread_type:
official_review

<thread_object>
creator: reviewer_1rea

<content>
summary:
This manuscript introduces Polyrating, a multivariate rating system for evaluating Large Language Models (LLMs). The authors aim to address several key challenges in current LLM evaluation methods, including accounting for biases, reducing the cost of human evaluations, and enabling meaningful comparisons of model performance across different tasks. Polyrating uses maximum a posteriori (MAP) estimation to incorporate different tasks and biases into the rating process. Experiments are conducted on publicly available human preference data to demonstrate the effectiveness of the proposed system.

strengths:
1. The authors clearly identify important limitations in existing LLM evaluation approaches, specifically focusing on the influence of biases, the high cost of human evaluation, and the lack of comparability across tasks. The proposed Polyrating is designed to directly address these challenges, providing a clear motivation for the work.

2. The manuscript provides a theoretical analysis of Polyrating, demonstrating its convergence properties. This provides a strong theoretical foundation for the proposed system and strengthens the confidence in its practical applicability.

weaknesses:
1. The manuscript lacks a comparison with existing multivariate rating systems. Empirical comparison and discussion of adaptations for LLMs are needed.

2. Experimental evidence to support some of the claimed benefits of Polyrating, is insufficient. Model selection for task comparability is questionable. The efficiency analysis lacks clarity on calculation methods.

questions:
1. Line 233: Why are different optimization methods (Newton's method and L-BFGS) used for different parameters? What's the rationale?

2. How are sample efficiency improvements (e.g., "38%" in line 361) calculated?

3. In Table 2 and 3, several models are chosen to illustrate the differences in task comparability between Polyrating and a unidimensional approach. The following questions arise:

3-1: Line 410: What specifically makes Mixtral's performance "very different"? The tables don't clearly show this. Providing a more precise explanation or visualization of these differences is necessary to support this claim.

3-2: Line 413: The Chinese task analysis uses proprietary models with undisclosed training details and the Mixtral model, which, due to its specific role in the Chatbot Arena setup, is not suitable for this comparison. Please consider revising the experimental setup to enhance the validity of the comparisons.

4. While existing multivariate rating systems are mentioned, their performance isn't compared to Polyrating. Include empirical comparisons and highlight performance differences would significantly strengthen the manuscript.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710621492

thread_type:
official_review

<thread_object>
creator: reviewer_igcp

<content>
summary:
This paper introduces Polyrating, a novel multivariate rating system designed specifically for evaluating Large Language Models (LLMs). The system addresses three key limitations of current rating approaches: bias awareness, cost efficiency and cross-task comparability. Polyrating uses maximum a posteriori estimation with shared parameters to quantify and account for various biases in human and LLM-based evaluations. The system demonstrates up to 77% cost reduction for new tasks and 41% for new models by leveraging existing benchmark scores. It also enables direct rating comparisons across different tasks by solving the shift-invariance problem inherent in traditional rating systems. The paper provides thorough empirical validation using public datasets like Chatbot Arena and theoretical proofs of the system's optimality.

strengths:
1. The paper makes good practical contributions by introducing a novel framework for LLM evaluation. Fundamental limitations in current approaches are well addressed, providing strong theoretical foundations with proper convergence guarantees and demonstrating substantial practical improvements.
2. The authors present comprehensive empirical validation across multiple datasets with comparison against baseline approaches and clear ablation studies showing the contribution of each component.
3. The significant practical impact demonstrated (up to 77% cost reduction for new tasks and 41% for new models by leveraging existing benchmark scores) offers immediate positive value to the field.
4. The paper is generally well written with clear problem motivation and logical flow from theory to implementation to results.

weaknesses:
1. There is limited discussion of computational complexity and scalability analysis. The paper could include runtime analysis for both training and inference phases of Polyrating, and recommended batch sizes and optimization strategies for large-scale deployment. Similarly, a short discussion of how the MAP optimization scales with number of models (k), tasks (d) and samples (n) would be useful.

2. The hyperparameter selection process for standard deviations sigma_j and sigma_j' could be better explained. The relationship between dataset size and optimal prior values should be analyzed and sensitivity analysis showing how different prior choices affect final ratings would also be valuable.

3. The system assumes static model capabilities but the paper should address rapid model changes with, eg: analysis of rating stability over time, methods for updating ratings incrementally as new data arrives, guidelines for maintaining rating consistency across model versions, etc.

4. The paper doesn't explore well on how the system performs with extremely sparse preference data (cold-start scenarios). The authors could analyze the minimum number of preferences needed per task/model pair. Also, methods for handling completely missing task-model combinations should be detailed.

questions:
1. What is the computational complexity of the optimization process and how does it scale with the number of models and tasks?
2. How sensitive is the system to the choice of priors? Could you provide guidelines for selecting appropriate priors?
3. How does the system handle rapid changes in model capabilities over time?
4. How does the system handle extremely sparse preference data for new tasks or models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716517804

thread_type:
official_review

<thread_object>
creator: reviewer_2pjr

<content>
summary:
The paper addressed that current rating systems for LLMs suffer from several limitations: biases, human annotations, and shift-invariance.\
Then, the authors propose POLYRATING, a multivariate rating system based on MAP estimation.\
POLYRATING uses maximum a posteriori estimation (MAP) to estimate model ratings across tasks, including base ratings and task-specific adjustments.\
It also incorporates shared parameters to model factors like answer length and readability that affect preferences.

strengths:
1. The notations and equations throughout the paper are clear, well-formulated, and add rigor to the methodology.
2. The justification for POLYRATING is robust, with a well-explained rationale for the multivariate approach.
3. Extensive experiments demonstrate the effectiveness of POLYRATING, showing its superiority over traditional rating methods in several key areas.

weaknesses:
1. The modeling features are somewhat ambiguous; it is unclear which specific features are applied for each task, or the criteria used for feature selection. More detailed information on feature selection per task would be helpful.
2. POLYRATING employs both model-shared and model-specific weights. It would be useful to clarify how the system initializes weights for newly added models and whether it can efficiently update ratings for new models without requiring a full retraining of the rating system.

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723268493

thread_type:
official_review

<thread_object>
creator: reviewer_6vxx

<content>
summary:
The authors propose, POLYRATING, an large language model evaluation system. POLYRATING can detect and quantify biases that affect human preferences, improve sample efficiency, reduce evaluation costs, and enable cross-task model scoring comparisons. It achieves a significant improvement on the sample efficiency, reducing the cost of human evaluation by up to 77% for new tasks and 41% for new models.

strengths:
- POLYRATING is a multivariate rating system that can detect and quantify biases affecting human and LLM-based evaluations of LLMs.
- Such effective system reduces the cost of human evaluation by up to 77% for new tasks and 41% for new models, which is meaningful and practical in the real-world applications.
- The paper is easy to follow.
- Comprehensive experiments are conducted, demonstrating the effectiveness of the proposed method.

weaknesses:
Null

questions:
Will POLYRATING also be appliable to other domains that use large language models, such as computer vision, information retrieval, etc.? How is the cost for the adaptation to these domains?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731922875826

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
$\newcommand{R}{\textcolor{green}{6vXX}}$
$\newcommand{S}{\textcolor{blue}{2PJr}}$
$\newcommand{T}{\textcolor{purple}{iGCp}}$
$\newcommand{U}{\textcolor{purple}{1REa}}$
We thank the reviewers for their high-quality reviews and insightful questions. We are particularly happy to hear they consider our paper well-written and easy to follow ($\R, \S, \T$), that the justification for Polyrating is strong ($\S, \U$), and that it is effective ($\R, \S, \T$). We could not identify any common questions between the reviewers, and will only include a comprehensive overview of the changes made to the paper in response to the reviews.

**Experiment on Model Versioning (Appendix C)**  
We included an experiment showcasing that Polyrating can also improve convergence of the rating of new model versions in Appendix C. For developers who create multiple versions of a model simultaneously or in quick succession, it is essential to evaluate the relative performance of these versions as cost-effectively as possible. However, this process can be computationally expensive, especially when human evaluations are used.  By applying an extra prior between the ratings of two model versions, one can easily extend Polyrating to improve sample efficiency for new model versions. Specifically, we converge 38% faster when obtaining 10,000 samples for new model versions.

**Clarification Exclusion Multivariate Rating Systems (L438-442,L446-452)**  
In the revised version, we explain more thoroughly in our Related Work section why prior works on multivariate rating systems are inapplicable to the experiments performed in our paper.

**Expanding the Discussion in Section 4.3 (L405-420)**  
We expanded the discussion in Section 4.3 to more clearly refer to overall trends, which more convincingly show the benefits of Polyrating. For instance, the only models that gain significant rating points for the Chinese task are those trained by Chinese model providers. In contrast, the univariate approach shows no such pattern.

**Clarification Efficiency Metric (L320)**  
We clarified how we computed the efficiency improvements for Polyrating when we first introduced them in the experiments. Specifically, in L320 we now indicate that the reported number is the fraction of extra samples the univariate baseline requires to obtain the same loss as Polyrating.

**Inclusion Glicko in Figure 4 (L803-806)**  
We include Glicko in Figure 4 to show that approximate systems are ineffective when used for language models, and perform much worse than the exact systems based on the BT-Model used by Polyrating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733307433725

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all four reviewers for their constructive comments and questions, which have significantly helped us improve our work. We are particularly encouraged by the reviewers' recognition of our paper's strengths, as summarized below:
$\newcommand{R}{\textcolor{green}{6vXX}}$
$\newcommand{S}{\textcolor{blue}{2PJr}}$
$\newcommand{T}{\textcolor{purple}{iGCp}}$
$\newcommand{U}{\textcolor{orange}{1REa}}$

**Significant Improvement**
- The presented experiments rigorously demonstrate the effectiveness of Polyrating ($\S,\R,\T$).
- The measured improvement over prior work is substantial and practical in real-world applications ($\R,\S,\T$).

**Good Presentation**
- The paper is well written with a clear logical flow ($\T,\S,\R,\U$).
- The limitations of prior approaches are clearly identified and directly addressed by Polyrating ($\U,\T,\S$).

**Theoretical Guarantees**
- The theoretical analysis of Polyrating provides strong theoretical foundations ($\U,\T$).

We acknowledge that our initial submission had areas for improvement. In response to the reviewers’ thorough reviews, we have provided the following additional information and experiments which we incorporated in the revision of the paper:

- **Model Versioning**: We showcased that Polyrating can be used to improve the convergence rate by 38% for different versions of the same model.
- **Exclusion Multivariate Rating Systems**: We clarified that existing multivariate rating systems are inapplicable to the settings evaluated in our experiments.
- **Improved Result Discussion**: We expanded the discussion in Section 4.3 to more clearly refer to overall trends, which more convincingly show the benefits of Polyrating.
- **Clarification Efficiency**: We clarified the way in which we compute efficiency improvements.

We additionally clarified the following significant points to individual reviewers:
- **Applications Beyond LLMs**: Polyrating can be applied to areas outside of LLMs, as long as preference datasets are available.
- **No Retraining Needed and Limited Computational Complexity**: New models can be added on the go, and the maximum time spent for a single run was 6h on a single CPU core.

Once again, we deeply appreciate the valuable feedback and guidance provided by the reviewers.

Best regards,  
The authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper focuses on improving the performance scoring system for LLMs and proposes a new rating system called Polyrating based on maximum a posteriori estimation. All four reviewers agreed that this paper exceeds the ICLR 2024 acceptance threshold, with a clear presentation and theoretical support. However, reviewers also pointed out some further weaknesses, including: 1) more discussion should be conducted on the efficiency of the proposed method and its use cases; 2) A clearer description of the method's details is necessary. During the rebuttal phase, the authors addressed most of these issues, and it is believed that these concerns will be resolved and reflected in the final version.