Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

Authors: yinhong  liu
han  zhou
zhijiang  guo
ehsan  shareghi
ivan  vulić
anna  korhonen
nigel  collier
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 13:43:32

Abstract: Large Language Models (LLMs) have demonstrated promising capabilities as automatic evaluators in assessing the quality of generated natural language. However, LLMs still exhibit biases in evaluation and often struggle to generate coherent evaluations that align with human assessments. In this work, we first conduct a systematic study of the misalignment between LLM evaluators and human evaluation, revealing that existing calibration methods aimed at mitigating biases of LLMs are insufficient for effectively aligning LLM evaluators. Inspired by the use of preference data in RLHF, we formulate the evaluation as a ranking problem and introduce Pairwise-preference Search (PairS), an uncertainty-guided search method that employs LLMs to conduct pairwise comparisons locally and efficiently ranks candidate texts globally. PairS achieves state-of-the-art performance on representative evaluation tasks in long-form generations and demonstrates significant improvements over direct scoring. Furthermore, we provide insights into the role of pairwise preference in quantifying the
transitivity of LLMs and demonstrate how PairS benefits from calibration using debiased pairwise evaluations.

Main paper: 
 introduction: 
 Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; Anil et al., 2023a; b) have achieved promising instruction-following abilities, enabling an efficient adaptation pipeline to solve complex natural language tasks with minimal or no labelled data (Kojima et al., 2022; Zhou et al., 2024c) . This unique capability stems from supervised training LLMs with instruction-following data (Wei et al., 2022; Chung et al., 2022) and reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Stiennon et al., 2020) . In the RLHF training paradigm, a reward model is aligned with human preferences based on ranked comparison data (Ouyang et al., 2022) . It enhances LLMs' alignment with human values (Touvron et al., 2023) , thereby generating responses that better assist humans and abide by human values (Dai et al., 2024) .Recently, LLMs have emerged as competent and inexpensive assessors for the evaluation of various aspects, such as coherence, fluency, consistency and hallucination, of text quality (Chen et al., 2023; Zeng et al., 2024; Zheng et al., 2024) . Put simply, LLMs can be seen as an appealing reference-free evaluation paradigm for generative tasks while avoiding expensive human-labeling costs (Liu et al., 2023a) . However, the predictions from LLM evaluators are highly sensitive to the prompt design (Zhou et al., 2023; Zheng et al., 2024) and even biased by multiple sources, including positional bias (Wang et al., 2023b) , verbosity bias (Saito et al., 2023) , and contextual bias (Zhou et al., 2024b) . These biases still hinder fair and trustworthy applications of LLM evaluators, resulting in inconsistency and misalignment with human judgements.To mitigate biased predictions from LLMs, calibration techniques have been developed to ensure less biased outcomes (Liu et al., 2023b; Li et al., 2023a; Zhou et al., 2024b) . Motivated by this line of work, we first conduct a systematic analysis of the effectiveness of calibration techniques in aligning LLM evaluators in direct scoring (Figure 1 , middle) with human judgements. We find that existing calibration methods are insufficient for aligning LLM evaluators, even when provided with supervised data. We discover that misalignment in evaluation is not primarily due to biased priors over evaluation scores, but rather from misaligned evaluation standards, i.e., the likelihood of LLM evaluators defined in Equation (1). Therefore, we are motivated to explore a new evaluation paradigm with LLMs to facilitate a more aligned judgement.Inspired by how LLMs are aligned through a reward model with preference data in RLHF, we argue that LLM evaluators can produce more aligned and inherent evaluations by leveraging their pairwise ranking abilities. Recent work has started exploring this intrinsic property by prompting LLMs with a pair of candidates (Wang et al., 2023b; Chen et al., 2023; Qin et al., 2023; Li et al., 2024) . However, the complexity and scalability of evaluating with pairwise preference have been largely overlooked. To produce a preference ranking for evaluation, existing methods require exhaustive pairwise comparisons, as they ignore the transitivity assumption1 (Qin et al., 2023) . This makes the evaluation procedure intractable as the number of comparisons grows quadratically (Zheng et al., 2024; Liusie et al., 2024) .In this work, we shed new light on the full potential of pairwise preference in LLM evaluators and its alignment with human judgements. We propose Pairwise-preference Search (PAIRS), an uncertainty-guided search method that efficiently estimates the Maximum Likelihood Estimate (MLE) of preference rankings by searching through an uncertainty-pruned pairwise comparison space. PAIRS is more effective and query-efficient than existing pairwise baselines while substantially outperforming traditional direct scoring evaluations. We extensively examine the applicability of PAIRS in representative evaluation tasks, including summarization and open-ended generation. We show that PAIRS produces more robust and transitive evaluations compared with the greedy search baseline. Lastly, we provide further insights into how pairwise preference can be used to quantify the transitivity ability of LLM evaluators and how it benefits from calibration.In sum, we provide the following contributions: 1) We present a systematic analysis of the limitations of calibration in aligning direct scoring LLM evaluators with human judgements.We formulate the evaluation as ranking from the perspective of transitivity and propose The score histograms on evaluating the coherence in HANNA (Chhun et al., 2022) and SummEval (Fabbri et al., 2021) .We present the scores from gold human evaluations, LLMs, and LLMs after calibrations.The histograms can be interpreted as estimated score prior distributions via marginalization.PAIRS, an uncertainty-guided pairwise preference search method that efficiently estimates MLE rankings. 3) We demonstrate that PAIRS attains unique scalability in aligning LLM evaluations. We also provide insights into the transitivity and calibration of pairwise preferences in LLM evaluators. 
on the limitations of calibration in aligning llm evaluators: 
 Misalignment Between LLM Evaluators and Human Judgements. The prediction of scorebased LLM evaluators can be expressed as a posterior predictive distribution p M (s|y; x, I):EQUATIONwhere x and y form an input-output pair (e.g., a source text and its summary, respectively), I denotes an instruction prompt, and s represents the model's rating score in its evaluation.Similarly, we can express the human posterior predictive distribution as p H (s|y; x, I). For simplicity, we omit the x and I terms hereafter. When utilizing LLMs to perform Likert scale evaluations (Likert, 1932) , it is known that there exists a mismatch between the score distributions of LLM ratings and human annotations (Liu et al., 2023b) . We start by observing and interpreting the same misalignment phenomenon, as shown in Figure 2 .Since the posterior distribution consists of two components, the likelihood, and the score preference prior, we hypothesise that the discrepancy between the posterior distributions of LLMs and humans mainly arises from a misalignment between their respective likelihoods. To verify this hypothesis, we conduct a preliminary experiment by calibrating the LLM's score preference prior, p M (s) to match the human prior, p H (s), described in what follows.Score-Based LLM Evaluator Calibration. Following previous calibration methods (Zhou et al., 2024b; Jiang et al., 2023b) , we first assume that the likelihood terms for both the LLM and humans are the same, p M (y|s) = p H (y|s). Upon this assumption, the human posterior distribution p H (s|y) can be expressed as:EQUATIONWe can estimate LLM's preference prior p M (s) by marginalizing the model's predictions, i.e., p M (s) = ∑ y p M (s|y)p(y). Unlike previous work that assumes a uniform prior for p H (s) (Zhou et al., 2024b) , we directly estimate it by marginalizing human annotations to achieve a more accurate estimation. The calibration is performed by scaling the model's predictions according to Equation (2).In Figure 2 , the experimental results show that even after calibration, the gap persists between the posterior distributions of the LLM and human evaluators. Specifically, the Mean Absolute Error (MAE) shifts from 1.62 to 1.16 on HANNA and from 0.78 to 0.86 on SummEval, indicating notable misalignment. This suggests that the discrepancy is not primarily due to the misaligned priors but rather stems from the differences in the likelihood of LLM evaluators.In other words, the assumption that LLM evaluators and humans have similar evaluation likelihoods becomes invalid, i.e., p M (y|s) ̸ = p H (y|s). The likelihood term reflects expected output candidates for a given score, which can be interpreted as the underlying evaluation standards. LLMs learn their standards from pretraining data, potentially diverging from the human evaluation standards. Hence, these findings motivate efforts to reduce the gap between evaluation standards.3 Uncertainty-Guided Pairwise-Preference Search 
evaluation as ranking: 
 Inspired by the fact that the RLHF training stage of most recent LLMs utilizes pairwise preference or preference ranking data (Touvron et al., 2023) , we propose that evaluation with pairwise comparisons could be a promising solution to address the above issues. We conclude the motivation: The difference between LLM and human evaluation standards is smaller when performing pairwise comparison, compared to rating with scores.When using an LLM to perform a pairwise comparison, we can compute the preference probabilities P(y i ≻ y j ) from output logits, where ≻ denotes the preference relation "is preferred to". However, LLM evaluators do not guarantee transitivity property, e.g. given an LLM's pairwise preferences y i ≻ y j and y j ≻ y k , it is not necessarily true that the LLM will have y i ≻ y k . The problem of finding the optimal preference ranking from non-transitive pairwise comparison probabilities is known as the Rank Aggregation problem (Dwork et al., 2002) . We formulate this problem as a Maximum Likelihood Estimation (MLE) problem.Let y 1:N be a set of N output candidates to be ranked. For a pair of outputs (y i , y j ), we have the probability P(y i ≻ y j ) that output y i is preferred over output y j . We denote a permutation (ranking) of these candidates as π = (π 1 , π 2 , . . . , π N ), where π i is the index of the i-th ranked candidate, i.e., y π i is the i-th ranked candidate output in π. The ranking π imposes a total order on y 1:N , such that y π 1 ≻ y π 2 ≻ . . . ≻ y π N . The goal is to find a ranking of the output candidates that maximizes the likelihood of observing pairwise probabilities:EQUATIONwhere L NT stands for the non-transitive likelihood,2 which does not make any transitive assumptions for the pairwise comparison probabilities. The MLE ranking of L NT is also equivalent to the Kemeny-Optimal Ranking (Kemeny, 1959) . However, finding the optimal ranking is challenging in two aspects:1. Computing L NT requires a complete set of O(N 2 ) pairwise comparisons.2. The exact solution to the rank aggregation problem is NP-hard (Dwork et al., 2001) , meaning that there is no known polynomial-time algorithm to solve it optimally.If LLM evaluators are assumed to possess transitivity property, then the problem of finding the MLE ranking reduces to a sorting problem, which can be solved by standard sorting algorithms such as merge sort or quick sort with only O(N log N) pairwise comparisons.However, strict transitivity is a strong assumption. To relax the strict transitivity assumption to some extent and find a better trade-off between computational complexity and performance, we propose simplifying the non-transitive likelihood in Equation ( 3) by assuming compositional stochastic transitivity (Latta, 1979; Oliveira et al., 2018) . The derivation of this simplified likelihood is provided in Appendix §C.2:Algorithm 1 Uncertainty-Guided PAIRS-beam 1: Objective: Find the ranking that maximizes the transitive likelihood L T . 2: Hyper-parameters: Beam size, beam size; Uncertainty threshold, U h . 3:Inputs: Two sorted input sub-arrays a = [A 1 , A 2 , . . . , A L ] and b = [B 1 , B 2 , . . . , B R]. 4: Output: A merged sorted array. 5: Initialize: Initialize a Beam B with the first candidate cand. Each cand keeps track of its choice trajectory, two pointers i and j for a and b, and a cumulative likelihood of its trajectory. 6: for k = 1 to L + R do 7:Create another empty beam B ′ . 8:for cand in B do 9: L T (π) = ∏P(A i ≻ B j ), U A i ,B j ← LLM(A i , B j ) 10: if U A i ,B j > U h then 11: B ′ ← B ′ ∪ cand.update(A i ) 12: B ′ ← B ′ ∪ cand.update(B j ) 13: else if P(A i ≻ B j ) ≥ 1 2 then 14: B ′ ← B ′ ∪ cand.update(A i ) 15: else 16: B ′ ← B ′ ∪ cand.update(B j ) 17: end if 18: end for 19: B ′ ← sort(B ′ , key = L T ) 20: B ← B ′ [:EQUATIONwhere L T stands for the transitive likelihood, and the MLE ranking is denoted as π * = arg max π L T (π). The traditional merge sort algorithm employs a divide-and-conquer approach, merging two sorted subarrays at each operation step. This merge operation can be interpreted as a greedy search procedure that aims to find the MLE ranking π * under the transitive likelihood. We refer to this approach as PAIRSgreedy. However, the non-transitive nature of LLMs suggests that optimization based on local probability may not lead to the optimal solution. Therefore, we also propose PAIRSbeam, a heuristic beam search algorithm that improves upon the greedy search merge operation by considering both candidates at each step.The PAIRS algorithm is described in Algorithm 1, and a demonstration of its workings is provided in Figure 3 and Figure 4 . Intuitively, while the PAIRS-greedy chooses the preferred items between the heads of two subarrays in a greedy fashion, PAIRS-beam main-tains a likelihood L T beam that keeps track of the top-k trajectory candidates. By also considering less preferred items between the two heads, PAIRS explores a broader search space and potentially finds a better ranking.To further improve the efficiency of the beam search, we introduce a pruning mechanism based on the LLM's uncertainty. When the uncertainty of the LLM's pairwise comparison is lower than a threshold U h , we prune the less preferred candidate from the beam. This uncertainty-based pruning restricts the search space, making the beam search more efficient and achieving a better balance between computational complexity and ranking performance.To represent the uncertainty between two items, we follow a commonly used approach in previous works (Wan et al., 2023a; b) that employs entropy as the uncertainty measurement.Hence, the uncertainty between a pair of comparisons can be expressed below:U(y i , y j ) = H(y i , y j ) = -P(y i ≻ y j ) log(P(y i ≻ y j )) -P(y j ≻ y i ) log(P(y j ≻ y i )). The theoretical complexity of PAIRS-beam ranges from O(N log N) to O(N 2 ) pairwise comparisons, as shown in Figure 5 . The upper bound occurs when the uncertaintybased pruning is disabled, and the beam size is sufficiently large to explore all potential trajectories during the merge operation.Conversely, the lower bound corresponds to PAIRS-greedy. However, for large instances, even the complexity of O(N log N) becomes computationally expensive and infeasible. To address this issue, we propose a two-stage scaling method that leverages the granularity of Likert scale annotations.In the first stage, we sample and rank a subset of the evaluation set, referred to as the anchor set. The size of the anchor set can be statistically determined by accommodating a tolerable error margin (Cochran, 1977) (Further details and empirical evidence in Appendix §F.3). In the second stage, we use binary search to find the positions of the remaining samples relative to the anchor set, efficiently determining their appropriate positions in the overall ranking. As depicted in Figure 5 , the second stage of the scaling method results in O(N) complexity, a significant improvement over the original complexity. This two-stage scaling approach allows PAIRS to efficiently handle large evaluation sets while maintaining the benefits of beam search and uncertainty-based pruning. The scaling variant strikes a balance between complexity and ranking performance, making it suitable for real-world applications involving large-scale datasets.Transitivity. Both PAIRS-direct and PAIRS-beam assume transitivity to some extent, implying that an LLM with better transitivity can perform more accurate and more efficient evaluations. In other words, we expect better-performing LLMs to exhibit more transitive behaviour when making pairwise comparisons. Consequently, we propose that transitivity can serve as an effective measure of an LLM's ability as an evaluator. By quantifying and comparing the transitivity of different LLMs, we can gain insight into their relative performance and suitability for evaluation tasks. To validate this hypothesis, we provide complementary experiments and analysis in Section 5.4.Although pairwise preference inherently aligns LLM evaluations with human judgement, pairwise prompting can still suffer from misaligned preference priors, e.g., positional bias and contextual bias (Zhao et al., 2021; Zhou et al., 2024b) . Therefore, we propose applying calibration on top of PAIRS to ensure debiased PAIRS evaluations. It is worth noting that, unlike calibrating the score-based LLM evaluators, which requires external information on the distribution of human preference prior, as shown in Section 2, pairwise comparisons can fully leverage a uniform human prior, due to the random nature of pairwise sample selections. 
related work: 
 LLMs as Evaluators. Recently, LLMs have been increasingly used to automatically evaluate various natural language generation tasks (Zhong et al., 2022; Chen et al., 2023; Wang et al., 2023a; Tan et al., 2024) . LLM evaluators enable reference-free or reference-guided metrics while being cheaper alternates to human evaluation in a diverse task set, such as summarization (Shen et al., 2023) , machine translation (Kocmi & Federmann, 2023) , open-ended story generation (Chiang & Lee, 2023) , dialog response generation (Liu et al., 2024a) and instruction-following (Zeng et al., 2024) . Unlike traditional reference-based evaluation metrics (Papineni et al., 2002; Liu et al., 2024b) , LLM evaluators are usually equipped with evaluation criteria prompts and generate scores to access the quality of the candidate text (Liu et al., 2023a) . This forms the scoring-based LLM evaluator that enables easy adaption to various tasks by simple instructions (Fu et al., 2023) . To fairly examine various LLM evaluation abilities, efforts have also been devoted to developing meta-evaluation benchmarks (Wang et al., 2023b; Zhang et al., 2023; Zheng et al., 2024) and prompt optimization (Zhou et al., 2024a) .Pairwise Preference. Preference data have been widely used in training the reward model to reflect human preferences in RLHF (Ouyang et al., 2022) . For example, Llama-2 has leveraged binary preference with 'chosen' or 'rejected' responses in its reward modelling (Touvron et al., 2023) . Following the preference idea, using pairwise comparisons in evaluating the LLM outputs has also been studied to some extent recently (Chen et al., 2023; Shen et al., 2023) , where Wang et al. (2023b) find that pairwise comparison exhibits better human correlations than traditional scoring-based LLM evaluators. However, Zheng et al. (2024) point out the intractability of pairwise comparisons due to its quadratic query complexity (Liusie et al., 2024) , which significantly limits its practicability. Qin et al. (2023) propose pairwise ranking with sorting but ignore the important transitivity of LLM predictions. In our work, we address the aforementioned issues by searching in a tractable pairwise search and transitive space based on model uncertainty. 
experimental setup: 
 Datasets. We evaluate PAIRS using three commonly used meta-evaluation benchmarks for two types of NLG tasks: abstractive summarization and creative story generation. Following previous research, we adopt two evaluation strategies, namely sample-level and datasetlevel Spearman correlations, as detailed in Appendix §E.1. For summarization task, we select News Room (Grusky et al., 2018) and SummEval (Fabbri et al., 2021) , consistent with prior studies (Zhong et al., 2022; Fu et al., 2023) . For the story generation task, we include HANNA (Chhun et al., 2022) . 3 We refer to Appendix §D for more details.To cover a good range of LLMs, we select two open-source models: Mistral 7B (Instruct-v0.1) (Jiang et al., 2023a) and Llama-2-chat 7b (Touvron et al., 2023) , and two closed-source models: GPT-3.5-turbo (-1106) and GPT-4-turbo (-preview) (OpenAI, 2023) .For open-source models, we calculate the preference probability by normalizing the logits of choice 'A' and 'B'. For GPT-3.5 and GPT-4, we return top-5 log-probabilities, which can be normalized to find the preference probability. 1 : LLM evaluation results. We report Spearman correlations on all datasets in terms of the following evaluation aspects: coherence (CH), fluency (FLU), relevancy (RE), informativeness (IN), consistency (CON), surprise (SU), and complexity (CX). For PAIRS-beam, the experiments were conducted with a beam size of 1000 and an uncertainty threshold at 0.6.Baselines. We consider two types of reference-free baselines: optimized and zero-shot evaluators. Optimized evaluators, UniEval (Zhong et al., 2022) and BARTscore (Yuan et al., 2021) , mitigate the evaluation standard gaps using annotated data, making them specialized for a specific dataset or domain but not transferable. We treat them as strong but unfair metrics and use their reported scores as a reference. For zero-shot evaluators, we re-implement the following baselines: BERTScore (Zhang et al., 2020) , G-Eval (Liu et al., 2023a) , and GPTScore (Fu et al., 2023) . We report direct scoring and G-Eval as the main baseline for fair comparisons.Performance Gap between Direct Scoring and PAIRS. As shown in Table 1 , PAIRS, as a zero-shot general evaluator, achieves state-of-the-art (SOTA) or near-SOTA performance on all datasets, even when compared with the optimized metrics UniEval and BARTScore, which are trained on dataset-specific data. In general, PAIRS outperforms direct scoring and G-Eval baselines in most aspects, with some exceptions, such as the consistency (CON) aspect of SummEval. We attribute this to the concentrated human scores distribution, with 86.7% summaries labelled with a score of 5. These characteristics make it difficult for pairwise comparisons to yield meaningful comparisons and consequent rankings.Impact of PAIRS on Different (Families of) LLMs. PAIRS particularly benefits the performance of smaller LLMs. For Mistral 7B and Llama-2 7B, we observe that pairwise comparison methods generally outperform direct scoring on News Room and SummEval. Notably, Mistral 7B achieves performance comparable to GPT-3.5-turbo on News Room. We hypothesize that this is due to their potentially similar quality of instruction tuning. Llama-2 7B struggles to effectively rate summaries with scores but demonstrates much better performance when comparing summaries. For HANNA, both small models do not benefit much from pairwise comparison, possibly due to the difficulty of understanding the long-context prompts or the presence of length bias. Conversely, direct scoring performs relatively well compared to summarization tasks. We believe this is because story generation evaluation does not depend on input, making the task more straightforward. For the two closed-source GPT models, PAIRS generally outperforms direct scoring and G-Eval baselines on all three datasets, indicating that PAIRS can achieve better performance more consistently for more capable models.Benefits of Uncertainty-Guided Search. PAIRS-beam generally exhibits higher correlations and lower Standard Error (SE), as shown in Table 2 , than PAIRS-greedy on all datasets, indicating its ability to more robustly produce better rankings. Interestingly, the performance difference between PAIRS-greedy and PAIRS-beam is minimal for GPT-4-turbo. We attribute this to GPT-4-turbo's better transitivity, allowing the greedy search to robustly find nearoptimal rankings. There are also a few cases where PAIRS-beam cannot outperform PAIRSgreedy. This could occur when the LLM's preference standards, the likelihood term, do not perfectly match with human annotations.PAIRS is an efficient pairwise rank aggregation method. We evaluate our method against two baseline rank aggregation techniques: 1) Win-loss rate and 2) ELO rating, which can both aggregate rankings from any number of pairwise comparisons. To assess their efficiency as aggregation algorithms, we calculate the correlations between human annotations and rankings predicted by PAIRS and the baselines using varying numbers of comparisons. The comparisons were random sampled without replacement. We also include the calibrated PAIRS, which requires double the computation by considering both permutations for each comparison pair.Figure 6 demonstrates that PAIRS is more efficient than the baselines. PAIRS-greedy typically requires only about 30% of the comparisons to achieve performance similar to ELO rating. The calibrated PAIRS generally outperforms the upper bound of the baselines, where all N(N -1) pairwise comparions are utilized. We attribute the strong performance of PAIRS to its leveraging of the transitivity assumption of LLMs to identify the most informative comparison pairs, which provides a natural efficiency advantage. While win-rate and ELO aggregation methods are designed to handle systems or players with variable performance, making them more effective at dealing with changes, our task involves ranking constant response candidates on a specific aspect. In this context, sorting-based algorithms that exploit transitivity have inherent advantages. Table 3 : Calibrating PAIRS. Mean of Spearman correlations (CH aspect) over 10 runs of PAIRS-greedy, w. and w/o calibration.Transitivity. The transitivity assumption plays a crucial role in the effectiveness of LLMs as evaluators, impacting the speed and consistency with which they achieve rankings. Hence, it is a good measurement of the capability of LLMs as evaluators. We quantify transitivity through the analysis of correlation variations, specifically the Standard Error. We conducted 10 runs of both PAIRS-greedy and PAIRS-beam with different random seeds.For the HANNA dataset, we sample and shuffle the same subset of 200 stories. Table 2 shows that PAIRS-beam generally demonstrates better correlations and lower SE than PAIRSgreedy, which indicates that it is more robust. We notice that GPT-3.5-turbo consistently exhibits fewer errors across all datasets, suggesting that it can produce more consistent evaluations with itself, thus possessing better transitivity. Furthermore, different datasets show varying levels of errors, which we believe indicates the transitivity difficulty of each dataset. For example, compared to other datasets, HANNA has relatively higher errors, which means that, for the same base models, its optimal rankings are more difficult to find. This observation is expected as HANNA, being a creative story-writing dataset, has naturally more subjective human annotations, causing challenges for LLM alignment.Calibration. Unlike direct scoring, pairwise comparison has a known preference prior, which is uniform. We perform the same calibration method as in the previous work by Zheng et al. (2023) , which averages the probabilities of both permutations of the candidates. Table 3 demonstrates that calibration consistently improves the performance of Mistral 7B and Llama-2 7B. However, calibration on GPT-3.5-turbo yields relatively small improvements, likely due to its heavily skewed output probabilities (also shown in the example in Figure 7 , Appendix §F.2). The over-confident predictions make the language model difficult to calibrate. 
conclusion: 
 In this work, we have first conducted a systematic analysis of LLMs used as direct scorers for evaluation in generative tasks. The analysis has revealed a misalignment between LLM and human evaluators, which cannot be effectively mitigated by applying calibration techniques. Inspired by RLHF, we have then proposed the use of pairwise comparisons which should produce evaluation rankings that are inherently more aligned to human judgements. We have formulated the evaluation as a ranking problem from the perspective of transitivity and proposed PAIRS, an efficient, scalable, and uncertainty-guided ranking method that navigates the pairwise comparison space and estimates the MLE rankings.Extensive experimental results have demonstrated that PAIRS is more aligned with human annotations and achieves SOTA performance compared to direct scoring and domainoptimized baselines. Furthermore, we have shown that PAIRS can also be used to measure LLM's transitivity.The second product can be simplified by noting that P(j ≻ i) = 1 -P(i ≻ j), so we get:L(π) = ∏ i,j:π(i)<π(j) P(i ≻ j) × ∏ i,j:π(i)>π(j) P(j ≻ i).Finally, since the comparisons are assumed to be independent, the second product can be absorbed into the first one, giving us the likelihood function for the rank aggregation problem: L(π) = ∏ i,j:π(i)<π(j) P(i ≻ j). 
 
 Appendix: In this section, we will show how we derive the transitive ranking likelihood, Equation ( 4), from the non-transitive ranking likelihood, Equation (3), given the compositional stochastic transitivity assumption.Compositional stochastic transitivity (Fishburn, 1973; Latta, 1979; Oliveira et al., 2018) states that P(a ≻ c) = H(P(a ≻ b), P(b ≻ c)), where H : R → [0, 1] is some strictly-monotone and symmetric composition rules. We use the notation of P a,b to represent P(a ≻ b). Therefore, we can have P a,c = λP a,b + λP b,c , where λ ∈ [0, 0.5].For a ranking π 1:N , where π 1 ≻ π 2 ≻ ... ≻ π N , of length N, there are Nk pairwise comparisons with distance k, e.g. P π i ,π i+k . Each comparison probability with distance k can be expressed by comparison probabilities with distance 1 using k -1 coefficients:EQUATIONThe number of λ coefficients has a scale of O(N 3 ). The non-transitive ranking likelihood can be expressed by probabilities of comparisons with distance 1.EQUATIONIf letting µ 1 = µ 2 = ... = µ N-1 , we can have around O(N 2 ) constraint equations, expressed by λs. As the dimension of λ coefficients is much larger than the number of constraint equations, we argue that there should always be a set of λ ∈ [0, 0.5] satisfying the constraints.Given ranking information, we can convert it to scores with any desired score distribution, via Quantile or Cumulative Density Function (CDF) matching. The desired score distribution can be either an assumed human score prior, for example, a normal distribution is a simple and suitable guess on prior, or an estimated distribution from validation samples.To match the CDF of variable Y to variable X, we need to find a transformation function g such that Y = g(X). The goal is to make the CDF of Y equal to the CDF of X.The CDF matching transformation is given by:EQUATIONwhere F -1 Y is the inverse of the CDF of Y. For example, if we assume the score prior distribution is [10%, 20%, 40%, 20%, 10%] for scores 1 to 5, then we can simply convert a ranking into scores by assigning the smallest 10% with score 1, the smallest 10% -30% with score 2 and so on.Summarization Tasks: We select SummEval (Fabbri et al., 2021) and NewsRoom (Grusky et al., 2018) for our experiments. SummEval consists of 100 input source texts, each paired with 16 summary candidates generated by different LMs. The dataset is annotated on four aspects: coherency (CH), fluency (FLU), consistency (CON), and relevancy (RE). NewsRoom includes 60 input source texts and 7 output summaries for each sample. It is annotated on the aspects of coherence (CH), fluency (FLU), relevance (RE), and informativeness (IN).Story Generation Task: We select HANNA (Chhun et al., 2022) , which contains 1056 creative story writings generated from 96 prompts collected from WritingPrompt. Three representative aspects are selected: coherence (CH), surprise (SU), and complexity (CX). We concatenate the story prompts with the story bodies, allowing the stories to be compared without input prompts. Due to the computational cost, we use the scaling variant of PAIRS with an anchor size of 100.Following previous works (Liu et al., 2023a; Zhong et al., 2022; Fu et al., 2023) we used two evaluation strategies, Sample level and Dataset level. For the sample level, the correlation values are calculated across the multiple candidates to the same input and then averaged across all input samples. For the dataset level, the correlation is calculated across the entire dataset.Mistral-7B Llama2-7B ρ # Q ρ # Q greedy 30.4 9,762 30.4 9,065 beam 32.0 24,733 31.1 24,006 beam-scaled 29.3 7,571 29.5 7,438 Table 4 : Performance trade-off of the scaling variant. We compare the Spearman correlations (ρ) and numbers of the model query (# Q) for PAIRS-greedy, PAIRS-beam and PAIRS-beamscaled on the Coherence aspect of the HANNA dataset. The beam size is set to 100, the anchor size is set to 100, and the uncertainty threshold is set to 0.67.Because of the binary search in the second stage of the scaling variant, we expect there will be performance loss compared to PAIRS-greedy and PAIRS-beam. To investigate the actual trade-off, we conduct an ablation study using Mistral-7B and Llama2-7B. The results presented in Table 4 illustrate the trade-off between performance and computational cost. PAIRS-beam-scaled achieves comparable performance to PAIRS-greedy, while requiring fewer model queries. We note that the computational complexity of PAIRS-greedy grows at a rate of O(N log N), whereas PAIRS-beam-scaled will grow at a rate of O(log 100), making it more computationally efficient as the problem size increases.Different base models have different logit distributions. Mistral 7B and Llama2 7B tend to have symmetric and bell-shaped distributions, which means a large portion of the probabilities lie within the range of [0.3, 0.7], as shown in Figure 4 . However, as shown in Figure 7 , the probability distribution is different for GPT-3.5-turbo. Therefore, their optimal uncertainty thresholds would be different. The Likert scale score rating can be treated as a five-category classification problem. Therefore the problem of anchor size estimation becomes estimating the sample size for estimating a categorical distribution. This can be calculated by the method of 'proportion sample size calculation' The formula for the sample size calculation for a single proportion is:n = (z 2 × p × (1 -p)) e 2 ,where n is the sample size, z is the z-score corresponding to the desired confidence level, p is the expected proportion of the most common category, and e is the desired margin of error. To account for multiple categories, the sample size is multiplied by a design effect (de f f ), which typically ranges from 1.5 to 2.n adj = n × de f f ,where n adj is the adjusted sample size for multiple categories, n is the sample size calculated for a single proportion, and de f f is the design effect. The design effect accounts for the increased variability in the data due to the multiple categories and the potential clustering of responses within categories.If we take the desired confidence level as 0.8, the expected proportion of the most common category p as 0.4, error margin e as 7% and de f f as 1.5, then the sample size required for estimating five categories distribution is around 120.Figure 8 presents empirical evidence for selecting an appropriate anchor set size. To determine the optimal size, we conducted an experiment where we randomly sampled different numbers of annotations from the HANNA (coherence) dataset and calculated the Kullback-Leibler (KL) divergence between the score distributions of the sampled sets and the entire set. For each anchor set size, we repeated the sampling process 100 times and computed the standard errors. The results suggest that an anchor set size between 100 and 200 provides a reasonable balance between computational efficiency and representativeness of the full dataset, as the KL divergence stabilizes within this range.The term transitivity in this work refers to mathematical transitivity, not linguistic transitivity. It is a property stating that if A is preferred to B and B is preferred to C, then A is preferred to C.This likelihood can be derived from the general learning to rank likelihood in Appendix §C.1.We make the first application of pairwise preference to evaluate over thousands samples. We use an anchor size of 100 on PAIRS-beam-scaled in HANNA. 


Decision: This paper presents a new technique for using LLMs as evaluators, by formulating the evaluation problem as a ranking problem, and using LLMs to conduct pairwise comparisons to create a global ranking. LLM-based evaluation is a very popular technique being used in industry today and this work addressing how to improve it is very timely. Reviewers were positive about the clarity of writing, motivation, comprehensive experiments on open and closed source models (though larger models can be added), ablations and strong empirical results.

Reviewers mainly had minor clarification comments and some suggestions on presentation. The title of the paper may be slightly misleading because it mentions human judgement, while there is no direct comparison with human annotators in the paper, so the authors may want to revise it.

[comments from the PCs] As much as possible, please follow the AC request regarding the title to avoid confusing readers.

Reviews: review: paper_summary: paper_summary: - This paper proposes a pairwise-preference search method that estimates MLE of preference rankings by searching through an uncertainty-pruned pairwise comparison space.
- The proposed method is more effective than traditional direct scoring evaluations and existing pairwise baselines.
strength_weakness: strength_weakness: - The main LLM evaluation results with the proposed method are not always better than the direct scoring method, which makes me doubt about the motivation in 3.1.
- Some results in Table 1 are close to the baselines. Are those statistically significant?
- Experiments can be more comprehensive if different size of open-source models (e.g. 70B size models). Also, it is curious if the larger models perform similarly.
questions: questions: - Any reason for the beam size of 1000, which seems to be very large number?
- What if the beam size is smaller for the main experiments? Also, what about the correlation between the inference time and the different beam sizes?
paper_summary: paper_summary: This paper develops a search-based search algorithm called PAIRS, which searches example pairs for LLM pairwise judgment to create a global ranking of examples. The basic idea is to convert the pairwise judgments conducted so far to prune the pairs for which the judgements can be inferred. Experiment results show that PAIRS can produce judgments that are more correlated with human judgments than a scoring-based method G-Eval.
strength_weakness: strength_weakness: - (W1) Baseline methods are missing. It is not clear if the proposed solution is reasonably effective for the purpose. More specifically, on (1) search and (2) aggregation. For (1), the most naive way is to use randomly selected pairs with the same rank aggregation method. For (2), other aggregation methods such as the Elo score, which is commonly used for LLM ranking (eg., Chatbor Arena).
- (W2) The motivation for Evaluation as Ranking is unclear. The main purpose of evaluation in most cases is to compare two (or multiple systems) to judge which one is better (best). In that sense, the global ranking discussed in the paper may not be the best option for evaluation.For example, the SummEval dataset used to 1-5 ratings for manual evaluation and the granularity is much more fine-grained by the proposed method.
- (W3) The cost of the proposed method is expensive. The computational complexity for scoring-based methods such as G-Eval is O(N). It is not clear if it’s worth paying extra cost. One option is to conduct experiments under the same budget (ie., just conducting N pairs to be fair against scoring-based methods) and discuss if the pairwise judgment approach can be more effective.
- (W4) Section 2 is misleading, as the limitations discussed in the section are not  addressed by aggregating pairwise judgments. If it is, the paper needs to show that the rating distribution is better aligned with that of human judgments (which is not straightforward, as the judgment is global ranking)
questions: questions: Please respond to the concerns raised in Reasons to Reject.
paper_summary: paper_summary: This paper presents an LLM evaluation framework with pairwise comparisons (Qin et al. 2023). Pairwise comparisons require $O(N^2)$ evaluations where $N$ is the number of outputs from LLMs. So, this paper explores the use of transitivity of preferences to reduce the number of evaluations into $O(N \log N)$ and presents a sorting algorithm for LLM outputs, inspired by the merge sort algorithm. This study also proposes using beam search for merge operations because transitivity may not always hold. The experimental results demonstrate that the proposed method has higher correlations with human annotations and performs better than direct scoring and domain-optimized baselines.
strength_weakness: strength_weakness: None
questions: questions: The title includes "aligning with human judgment." I was expecting to see how the proposed method aligns LLM judgments with humans. However, this study does not directly align human judgment with that of an LLM evaluator. In fact, this paper gave up calibrating LLM evaluators directly for alignment, but obtained a better alignment as a result of utilizing pairwise comparisons. So I felt that this phrase in the title is a bit misleading. Is it right that evaluator LLMs were not trained to align human pairwise preferences?

Where can I find the numbers of pairwise comparisons of the PAIRS-greedy, PAIRS-beam, and Qin et al. (2023)?
paper_summary: paper_summary: This paper develops a method for automatically ranking a set of text responses (e.g., summaries of source texts) using LMs, while training to maintain alignment with human rankings.

Problem background: Simply prompting an LM to score texts according to criteria can result in significant misalignment with human judgements. Estimating the marginal human score distribution p_H(s) and using that to calibrate the LM scores does not fully resolve the issue, implying that the model's likelihood p_M(y|s) is also misaligned from the human likelihood.

Method: the proposed method (PAIRS) instead focuses on using pairwise comparisons between candidate texts to produce a global ranking. This is formulated as searching over an ordering (permutation) that maximizes the likelihood; the definition in Eq 4 is designed to considers two algorithms for finding this ordering. The first is a merge-sort-like algorithm that merges two sorted lists of responses by greedily choosing the next candidate from either list that locally maximizes the likelihood. They then introduce a beam-search-like alternative of this procedure that tracks the top-k beams of candidate rankings at any time.

The experiments show that PAIRS achieves strong correlation with gold annotations on several natural language generation benchmarks.
strength_weakness: strength_weakness: * The text in 3.2 is not very clear at explaining how PairS-beam works, and the figure/algorithm need to be carefully studied to infer how it works without much supporting text. The caption or text could be expanded to make this easier to understand.
* The results could benefit from more clarity about the actual real-world cost (in terms of # of model calls) of each method in the experiments. Do the proposed methods use more model calls than direct scoring?
questions: questions: * What is the beam size used for PairS-beam(-scaled) in Figure 5? Does that matter at all for the number of model calls?
* If I understand correctly, the scaling variant of Pairs is linear in number of samples to be ranked, is that asymptotically the same as direct scoring?

Comments: title: Rebuttal 2
Reminder
Thanks
Thanks
Thanks
Reminder
Reminder
Reminder
Re: Rebuttal by Authors
Following up discussion towards the rebuttal
Following up discussion towards the rebuttal
comment: We thank the reviewer for the feedback and have addressed them below.


1. Pairwise Baselines

This ablation study with other two pairwise baselines using a different format than our result Table 1. In this table, 50, 100, 240(max) represents the number of randomly selected candidate pairs used to calculate the win rate or the ELO rankings. For SummEval dataset, there are 16 candidates for each datapoint, resulting in a total of 240 pairs. (N^2, including swapped comparisons and excluding self-comparison.) 

For PairS-Greedy, we cannot control the exact number of comparisons needed for the ranking. By averaging the results over 10 repetitions, we use 39.2 and 44.3 comparisons to achieve the global rankings for Mistral-7B and GPT-3.5.
When using calibrated PairS-Greedy, (where each comparison uses the average probabilities of pairwise comparison with permutated positions), the number of comparisons are 41.5*2 and 46.3*2. The reason we mention calibrated result is that the 240 total comparison pairs include comparisons with swapped positions. 

Both uncalibrated and calibrated PairS-Greedy, our method are significantly more efficient than the win rate and ELO baselines. Presenting these results in a table may not be optimal, and we plan to use line figures in the formal revision to better illustrate the findings.

---
2. Ranking as evaluation is not motivated  

> If you compared M systems for N pairs, wouldn’t the computational complexity be O(M^2 * N^2)?

We believe the reviewer has misunderstood the setup of pairwise ranking. **We are not comparing the whole LLM systems, but a list of candidates.** The candidates can be generated by different systems, but we only evaluate the quality of those candidates. If we have M candidates to rank, we can sample up to M*2-M pairs. What we meant by "as the number of candidates is typically much larger than the number of score categories", is M is usually larger than the scale of score categories (5 for likert scale). Therefore, providing the ranking of M candidates is more fine-grain than scoring.

---
3. pointwise evaluation and G-Eval

We acknowledge the statement in G-Eval paper. We believe the reason why G-Eval cannot handle API model efficiently is beyond the discussion of our paper. In our paper, what we claim is pointwise evaluation methods generally have complexity of O(N) and our argument is still hold: pointwise evaluations can only gain marginal performance with more computations.

---
In conclusion, we believe our experiments and ablation studies provide a comprehensive evaluation of the proposed PairS method. We have compared with multiple baselines including both pointwise and pairwise evaluation methods. Our results show that PairS is more efficient and effective than pairwise baselines and achieves higher human agreement than both pointwise and pairwise baselines. 

Furthermore, we have clarified and provided explicit support for the motivation behind ranking evaluation. We also have address reviewer's misunderstanding regarding our discussion of the limitation of direct scoring in Section 2. 

We have made a sincere effort to respond to the reviewer's concerns and to provide additional evidence and clarifications to support our claims. In light of our response, we respectfully request that the reviewer consider improving their score, as we believe our work makes a valuable contribution to the field of LLM evaluator.
Thank you for your responses and for sharing additional experiments. Apologies for the delayed reply.

Here are my follow-up questions for each of the points in your both responses. 

>> Pairwise Baselines
>
> Uncalibrated PairS-greedy achieves the same performance as the baselines using only 34% and 31% of the comparisons. Calibrated PairS-greedy, which doubles the number of comparisons, can sometimes outperform baselines that use O(N^2) comparisons.

Thank you for sharing the additional experiment results and the observations. 

I cannot draw the conclusion from the table. Could you add an explanation for the table in a similar style as the paper? What are these numbers? What are 50, 100, 240(max)?
What do 23.0(39.2) and 36.5(41.5*2) mean for PairS-G?

>> Ranking as evaluation is not motivated
>
> as the number of candidates is typically much larger than the number of score categories

If you compared M systems for N pairs, wouldn’t the computational complexity be O(M^2 * N^2)? Can N pairs be sampled from M systems and can offer the same level of ranking quality as that for 2 systems? I think this point is not discussed or clarified in the paper.

>> PairS may be expensive
>
> We acknowledge that pairwise evaluations are generally more expensive than pointwise methods. However, we would like to highlight:
> 
> 1. Performance should be considered alongside cost. We show that PairS outperforms most pointwise methods, such as G-Eval, which only obtain marginal gains with increased computation.
> 2. PairS is highly efficient compared to pairwise baselines.

> G-Eval uses a weighted average of 20 inference repetitions
Thank you for sharing the additional experiment. 

The G-Eval paper called the API multiple times because the GPT API does not return token probababilities. The method uses the weighted average of tokens (e.g., 1-5) based on their probabilities, which can be computed by a single inference. Below is the description from the G-Eval paper.

```
For GPT-4, as it does not support the output of token probabilities, we set ‘n = 20, temperature = 1, top p = 1’ to sample 20 times to estimate the token probabilities.
```

>> The limitation in Section 2
>
> It is important to note that this paper does not focus on calibration,

Thanks for clarifying. Then, I believe Section 2 is misleading and the point should be further clarified. This is rather a presentation issue and I don't have any follow-up questions.
Dear Reviewer zw92,

As the discussion period is ending, we would like to remind you to take a look at our response if you haven't. We are eager to know whether our response has addressed your concerns: we'd be grateful if you could reconsider the rating if that's the case. Otherwise, we are always happy to answer further questions that you may have.

Best,

Authors
We thank the review for confirming all questions have been answered.

As for the inference speed, the numbers in the bracket are averaged numbers of model forward calls. We believe it can represent the inference speed. The actual inference time depends on the LLM and GPU used. For example, in our case, it takes about 7s to perform a PairS-greedy (around 40 model calls) with Mistral-7B and batch size of 5, using RTX3090.
Thank you to the authors for their response, the additional ablation on beam size and the result of different size are helpful. I am still curious of inference speed vs number of beam sizes but my questions have mostly been clarified.
Thank you to the authors for their response, the additional ablation on beam size is helpful. I feel that my questions have largely been clarified and have no further questions.
Thank you for your kind message. We truly appreciate the time and effort you have put into reviewing our work. Thank you for your constructive feedback and for being a part of this collaborative effort to advance our field of study.

Best regards,

Authors
Dear Reviewer 6K4e,

We thank you once again for your feedback! As the discussion period is ending, we wonder if you could take a look at our response if you haven't? We are eager to know whether our response has addressed your concerns: we'd be grateful if you could improve the rating if that's the case. Otherwise, we are always happy to answer further questions that you may have.

Best,

Authors
Dear Reviewer zw92,

We thank you once again for your feedback! As the discussion period is ending, we wonder if you could take a look at our response if you haven't? We are eager to know whether our response has addressed your concerns: we'd be grateful if you could reconsider the rating if that's the case. Otherwise, we are always happy to answer further questions that you may have.

Best,

Authors
Dear Reviewer VjHC,

We thank you once again for your feedback! As the discussion period is ending, we wonder if you could take a look at our response if you haven't? We are eager to know whether our response has addressed your concerns: we'd be grateful if you could improve the rating if that's the case. Otherwise, we are always happy to answer further questions that you may have.

Best,

Authors
Thank you for the response. It's my pleasure to contribute to the work in some way through the reviewing process.
We thank the reviewer again for the insightful feedback. We also provide some further discussion on the performance on Larger LLMs:

> Performance on Larger LLMs

While computational resources prevent us from conducting larger-scale evaluations, we additionally include results from Llama2-13B, and we will endeavor to include larger-scale LLMs in future work in addition to the large-scale GPT4 experiments that we have conducted :

||PairS-greedy|
|-|-|
|Llama2-7B|15.3+-1.6|
|Llama2-13B|17.9+-1.1|
We thank the reviewer again for the valuable suggestions. We hope our complementary experiments have addressed all the reviewer’s concerns. We hope that in light of the response, the reviewer could consider improving their score.
We also provide some further discussion to support our rebuttal

---
> Pairwise Baselines  

PairS leverages the transitivity assumption of LLMs, which gives it a natural efficiency advantage. Win-rate and ELO aggregation methods are designed to handle systems or players with non-constant performance, making them more effective at dealing with changes. However, in our case, we are ranking the exact same aspect of constant response candidates, which means sorting-based algorithms that exploit transitivity have inherent benefits.

---
> Ranking as evaluation is not motivated  

To support our motivation claim in Sec 3.1, we conducted a complementary experiment using 50 randomly sampled summary candidate pairs for each source text in SummEval. We calculated the 3-way agreement rate accuracy between direct scoring and human scores, as well as between pairwise comparisons and human scores. The results consistently show that pairwise comparisons have better agreement with human judgements than scoring, across all three selected LLMs.
|Model|Scoring|Pairwise|
|-|-|-|
|Mistral-7B|38.7|45.5|
|Llama2-7B|39.6|43.0|
|GPT-3.5| 42.3|48.1|

Moreover, ranking-based evaluation has already been successfully employed in various real-world applications, such as re-rankers for data selection, output reordering, and recommendation systems.

---
> PairS may be expensive  

More discussion about the cost of pointwise methods, such as G-Eval.   
G-Eval uses a weighted average of 20 inference repetitions, resulting in total computations of 20N. In comparison, PairS-greedy requires Nlog2(N) in the worst case. Specifically, for the 16-candidate SummEval example mentioned above, the factor is only 2.5N.