UniCBE: An Uniformity-driven Comparing Based Evaluation Framework with Unified Multi-Objective Optimization

Authors: Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: evaluation, efficient, scalability, accuracy, convergence

Abstract:
Human preference plays a significant role in measuring large language models and guiding them to align with human values. Unfortunately, current comparing-based evaluation (CBE) methods typically focus on a single optimization objective, failing to effectively utilize scarce yet valuable preference signals. To address this, we delve into key factors that can enhance the accuracy, convergence, and scalability of CBE: suppressing sampling bias, balancing descending process of uncertainty, and mitigating updating uncertainty.
Following the derived guidelines, we propose UniCBE, a unified uniformity-driven CBE framework which simultaneously optimize these core objectives by constructing and integrating three decoupled sampling probability matrices, each designed to ensure uniformity in specific aspects. We further ablate the optimal tuple sampling and preference aggregation strategies to achieve efficient CBE.
On the AlpacaEval benchmark, UniCBE saves over 17% of evaluation budgets while achieving a Pearson correlation with ground truth exceeding 0.995, demonstrating excellent accuracy and convergence. In scenarios where new models are continuously introduced, UniCBE can even save over 50% of evaluation costs, highlighting its improved scalability.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730526693594

thread_type:
official_review

<thread_object>
creator: reviewer_8srk

<content>
summary:
This paper presents UNICBE, a new framework for comparing-based evaluation (CBE) to better align large language models with human preferences. Unlike traditional CBE methods that focus on single objectives, UNICBE addresses sampling bias, manages uncertainty, and optimizes preference signals through three specialized sampling probability matrices. Tested on the AlpacaEval benchmark, UNICBE achieves high accuracy with a Pearson correlation over 0.995 and reduces evaluation costs by 17%, with savings exceeding 50% when evaluating new models, highlighting its efficiency and scalability.

strengths:
This paper tackle an important question in LLM evaluation and provide a sound solution to it. The experiments are extensive and convincing with strong results showing the advantage of the proposed methods.

weaknesses:
I don't see any major weakness of the paper, just the presentation can be improved, especially lack in explaining why the method is better than others in an intuitive and easy to follow way.

questions:
The authors argue that to avoid bias, the budget should be allocated uniformly, if so how could this method be more sample efficient than random? I guess the reason is if model A is much better than B and model B is much better than C, then it's not necessary to compare A and C a lot. But if thats the reason why this method is more sample-efficient, it would be contradictory to the uniform assumption. Could the authors provide more insight into this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730528057207

thread_type:
official_review

<thread_object>
creator: reviewer_8lcx

<content>
summary:
The paper presents UNICBE, a uniformity-driven comparing-based evaluation (CBE) framework designed to optimize model evaluation across three primary objectives: accuracy, convergence, and scalability. Existing CBE method such as random allocation, method used in ARENA and ALPACAEVAL fails to maximize these aspects simultaneously. 
UNICBE addresses this by simple yet effective approach to promote uniformity across sample and models. The proposed method incorporates three decoupled sampling probability matrices, each of them derived to ensure uniformity in accuracy, convergence, scalability to newly added models. Comprehensive experiments and ablation studies across multiple datasets highlight UNICBE's cost-efficient performance in model evaluation.

strengths:
UNICBE introduces a novel approach to CBE by balancing three critical objectives, accuracy, convergence, and scalability, representing a straightforward yet impactful optimization techniques to advance model evaluation.
While the framework’s calculations and derivations to promote uniformity are relatively simple, its contributions are substantial given the current reliance on labor-intensive evaluation processes. 
The quality of the work is supported by extensive empirical analysis across diverse models, benchmarks, and settings, with extensive testing to support the assumptions in Section 3 and performance validation across multiple configurations in Section 5.4. 
Clarity is evident in the structured presentation of key concepts and experimental results, with details in experiment or notations clearly addressed. This paper's significance lies in its contribution to large language model evaluation, providing a scalable and efficient methodology that aligns closely with human preference signals, a crucial advancement for iterative model assessments.

weaknesses:
The novelty of balancing accuracy, convergence, and scalability needs further justification, as similar uniform sampling strategies have been discussed in prior works that highlight the uniformity, such as Vabalas et al. (2019) for sampling biases, which could diminish its uniqueness. 

Although the experiment of MT-Bench is based on human evaluator, larger portion of the evaluation is relied on AlpacaEval, as larger number of models and samples are used for the evaluation with AlpacaEval. The reliance on GPT-4 and GPT-3.5-turbo as evaluators, while useful, could benefit from validation against human judgments or additional LLMs, such as Claude, to establish greater reliability and generalizability across evaluator types. 

Minor details, but the readability of all figures could be enhanced by widening the lines in each plot, which would improve clarity and interpretation for readers.

questions:
As the UniCBE is based on three matrix, $P^{acc-l}, P^{con-l}, P^{sca-l}$, each targeting different goal of accuracy, convergence, scaliability, can user steer between those by adding hyperparameter for each matrix? Would it be also possible to quantify it through experiment?

While scalability is addressed by sequentially adding models, the paper could enhance this section by incorporating real-world scenarios, where models enter and exit dynamically, further proving UNICBE’s robustness in evolving benchmarks.

The given choice of greedy sampling over probabilistic sampling and Bradley-Terry model over Elo rating system appears significant to the framework’s success. Could the authors conduct a small experiment to demonstrate that UniCBE maintains its effectiveness across different sampling and aggregation settings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605564397

thread_type:
official_review

<thread_object>
creator: reviewer_pbe8

<content>
summary:
This paper studies to explore the use of preference signals with comparing-based evaluation (CBE). It proposes a unified uniformity-driven framework that can achieve CBE with better accuracy. Experiments show the proposed method saves over 17% of evaluation budgets compared to the random sampling baseline.

strengths:
1. The study of using preference signals with lower evaluation budgets is important research direction with huge potentials for benefiting future research and development

2. This paper proposes a framework to achieve CBE with better accuracy, convergence and scalability.

weaknesses:
1. The paper can be improved with more performance comparison with existing work and state-of-the-art performance.

2. Expect more statistical and experimental conclusions with the proposed CBE method for the scenario of large-scale preference learning.

questions:
None.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730627374480

thread_type:
official_review

<thread_object>
creator: reviewer_k9da

<content>
summary:
The paper presents UniCBE, a uniformity-driven comparing-based evaluation (CBE) framework designed to improve the efficiency and accuracy of evaluating large language models (LLMs) based on human preferences. UniCBE optimizes three main objectives simultaneously: reducing sampling bias, enhancing convergence by managing uncertainty, and ensuring scalability when new models are introduced. Compared to baselines like random, Arena, and AlpacaEval, UniCBE demonstrates the lowest error and highest correlation between the ground truth evaluation results.

strengths:
- Comparing-based evaluation is an important problem in LLM evaluation. The proposed method solved a non-trivial problem.
- This paper improves the comparing-based evaluation from three perspectives (accuracy, convergence, and stability). It provides a solid theory foundation, and the experiment results also demonstrate the efficiency of its proposed method.
- Their experiments cover three dimensions well in their proposed method. They also provide ablation studies on each variant. In general, most of the variants play their roles well.

weaknesses:
- The motivation is not enough clear. Why the previous method cannot perform well in both accuracy, convergence, and scalability?
- The runtime of the previous CBE method ($O(NM^2)$) is one of the major limitations, and the author starts from this limitation as one of the motivations for the proposed method. However, they lack the runtime analysis for the UniCBE but only an approximate number for saving time when compared to the previous method.
- While UniCBE shows promising results for scenarios with periodically introduced new models, it may be less efficient in highly dynamic, real-time evaluation settings where new models or samples are constantly introduced at high frequencies.

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721470826

thread_type:
official_review

<thread_object>
creator: reviewer_d5wd

<content>
summary:
This paper introduces UNICBE, a uniformity-driven framework for comparing-based evaluation (CBE) that simultaneously optimizes multiple objectives in language model assessment. Current CBE methods suffer from limitations related to bias, slow convergence, and lack of scalability.

strengths:
Efficient evaluation methods are essential for increasingly large-scale LLMs, and this work addresses relevant limitations in conventional CBE.

The integration of accuracy, convergence, and scalability in a uniformity-driven approach is a valuable contribution, setting UNICBE apart from single-objective methods.

weaknesses:
The framework assumes preference signals (particularly from automated judges like GPT-4) are consistent with human judgment, a potentially risky simplification given known limitations in automated preference evaluations.

The formulation of multi-dimensional sampling matrices and their interaction in optimizing accuracy, convergence, and scalability may be overly complex for practical implementations and difficult to interpret for further tuning or adjustment.

questions:
1. How does UNICBE perform when preference signals are less reliable, as is often the case with models lower than GPT-4 or inconsistent human annotations?

2. Could the authors elaborate on how UNICBE would handle scenarios with dynamic preference priorities, where, for example, accuracy is weighted more heavily than convergence?


3. To what extent could the uniformity constraints in the sampling matrices be relaxed while maintaining cost-effectiveness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733111239169

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Sincere Gratitude to All the Reviewers

comment:
**We sincerely thank all the reviewers for their time and effort in reviewing our manuscript. All the reviewers provided valuable suggestions and actively engaged in timely and constructive discussions, setting an excellent example for our community.** Based on their valuable suggestions, we have incorporated the following updates into the revised version:

- Validated the performance of UniCBE under more dynamic and realistic scenarios, while extending the preference budget from 8000 to 10000. This provides stronger support for the application of UniCBE in large-scale real-world scenarios.  
- Investigated the performance of UniCBE under varying levels of uniformity constraints. This serves as a more fine-grained ablation study to verify the validity of the uniformity optimization objective.  
- Explored the effects of adjusting the weights of different optimization objectives, demonstrating the flexibility of UniCBE and offering insights for more customized use cases.  
- Included experimental results using Qwen-Plus as the judge, further validating the generalization capability of UniCBE.  
- Expanded the discussion on related works concerning sampling bias, which supports and clarifies the motivation of our study.  
- Elaborated on the intuitive motivation behind UniCBE, making it easier for future readers to understand our work.

We believe these additions have significantly improved the quality of our work. We also look forward to seeing more researchers explore comparison-based evaluation in the future, achieving ongoing progress in the efficiency and accuracy of evaluation signals.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a comparing-based evaluation (CBE) framework which considers at the same time multiple objectives, such that it can leverage scarce signals. In that way, it can become more efficient while considering the standard factors of LLM CBE: accuracy, convergence and scalability. 

The paper is well motivated: it addresses a timely and important problem in a way that intuitively makes sense. This research also opens up the way for further work for LLM CBE under this new viewpoint. 

This paper has been reviewed by five experts and it has benefited from fruitful discussion between authors and reviewers. A few initial concerns were clarified during the rebuttal, as it was acknowledged by the reviewers. The only substantial remaining concern is about the scale of the experiments being relatively small; however, overall the experiments have been quite satisfactory.

Therefore, overall this paper contains solid research and it will likely be of interest to the ICLR community.