Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
Authors: Yinhong Liu
Han Zhou
Zhijiang Guo
Ehsan Shareghi
Ivan Vulić
Anna Korhonen
Nigel Collier
Venue: COLM (2024)
Keywords: LLM evaluator; pairwise comparison; human alignment
TL;DR: We discuss the motivation of pairwise preference, formulate the LLM evaluation as a ranking problem and introduce a search-based ranking method that achieves sota performance.

Abstract: Large Language Models (LLMs) have demonstrated promising capabilities as automatic evaluators in assessing the quality of generated natural language. However, LLMs still exhibit biases in evaluation and often struggle to generate coherent evaluations that align with human assessments. In this work, we first conduct a systematic study of the misalignment between LLM evaluators and human evaluation, revealing that existing calibration methods aimed at mitigating biases of LLMs are insufficient for effectively aligning LLM evaluators. Inspired by the use of preference data in RLHF, we formulate the evaluation as a ranking problem and introduce Pairwise-preference Search (PairS), an uncertainty-guided search method that employs LLMs to conduct pairwise comparisons locally and efficiently ranks candidate texts globally. PairS achieves state-of-the-art performance on representative evaluation tasks in long-form generations and demonstrates significant improvements over direct scoring. Furthermore, we provide insights into the role of pairwise preference in quantifying the
transitivity of LLMs and demonstrate how PairS benefits from calibration using debiased pairwise evaluations.

Published as a conference paper at COLM 2024
Aligning with Human Judgement: The Role of Pairwise Pref- erence in Large Language Model Evaluators
Yinhong Liu!*
Han Zhou!*
Zhijiang Guo
Ehsan Shareghi
Ivan Vulic
Anna Korhonen Nigel Collier!
UUniversity of Cambridge Moonahh University [y1535, h2416, zg283, iv250, alk23, nhc30) @ca.mac..... uk
www.:shhhhehheoonssttioo edu
Abstract
Large Language Models (LLMs) have demonstrated promising capabili- ties as automatic evaluators in assessing the quality of generated natural language. However, LLMs still exhibit biases in evaluation and often strug- gle to generate coherent evaluations that align with human assessments. in this work, we first conduct a systematic study of the misalignment between LLM evaluators and human evaluation, revealing that existing calibration methods aimed mitigating biases of LLMs are insufficient for effectively aligning LLM evaluators. Inspired by the use of preference data in RLHF, we formulate the evaluation as a ranking problem and in- troduce Pairwise-prefern prifeeeeeeeeeeeeeeeeeee Search (PAIRS), an method that employs LLMs to conduct pairwise comparisons locally and efficiently ranks candidate texts globally. PAIRS achieves state-of-ther oh-tte-artttttt per- formance on representative evaluation tasks in long-form generations and demonstrates significant improvements over direct scoring. Furthermore, we provide insights into the role of pairwise preference in quantifying the transitivity of LLMs and demonstrate how PAIRS benefits from calibration using debiased pairwise evaluations.
search
Introduction
Large Language Models (LLMs) (Brown al., 2020; OpenAI, 2023; Anil al., 2023a;b) have achieved promising instruction-folloiing abilities, enabling efficient adaptation pipeline to solve complex natural language tasks with minimal or no labelled data (Kojima et al., 2022; Zhou et al., 2024c). This unique capability stems from supervised training L.MSs with instruction-folloiing data (Wei et al., 2022; Chung et al., 2022) and reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Stiennon et al., 2020). In the RLHF training paradigm, a reward model is aligned with human preferences based on ranked comparison data (Ouyang et al., 2022). It enhances LLMs' alignment with human values (Touvron et al., 2023), thereby generating responses that better assist humans and abide by human values (Dai et al., 2024).
Recently, LLMs have emerged as competent and inexpensive assessors for the evaluation of various aspects, such as coherence, fluency, consistency and hallucination, of text quality (Chen et al., 2023; Zeng et al., 2024; Zheng et al., 2024). Put simply, LLMs can be seen as an appealing enference-free evaluation paradigm for generative tasks while avoiding expensive human-labeling costs (Liu et al., 2023a). However, the predictions from LLM evaluators are highly sensitive to the prompt design (Zhou et al., 2023; Zheng et al., 2024) and even biased by multiple sources, including positional bias (Wang et al., 2023b), verbosity bias (Saito et al., 2023), and contextual bias (Zhou et al., 2024b). These biases still hinder fair and trustworthy applications of LLM evaluators, resulting in inconsistency and misalignment with human judgements.
To mitigate biased predictions from LLMs, calibration techniques have been developed to ensure less biased outcomes (Liu et al., 2023b; Li et al., 2023a; Zhou et al., 2024b). Motivated
Eqqul contribution. Code is available at https: https://githhbb com/ http:/abbbddelll/Paiii..

---

Published as a conference paper at COLM 2024
RLHF
Scoring
Pairwise Preference
definition
Generate a concise summary for the given text passage,
Score
definition
Evaluate coherence the following summary In to 5.
samples
[A, B, C, D,
B
Collect human preference
Append sample with the scorling criteria
Select pairs to be evaluated
This is
this week,..
This is
this week,
This
LLM Evaluator
Train the reward model
Generate the score
Generate preference
Reward Model
Evaluator
RLHF Finetuning
Collect
Select the next until the rank is sorted
Preference Search
highest
scores
[A<B<D<C...)
Figure 1: Illustration of RLHF (left), direct scoring with LLM evaluators (middle), and pairwise preference search (PAIRS; right). Pairwise preference data have been utilized to train the reward model to align the LLM in RLHE Leveraging this idea, PAIRS reframes the traditional scoring-basee evaluations into series of pairwise comparisons between selected candidate pairs which are then used to form the final ranking.
by this line of work, we first conduct systematic analysis of the effectiveness of calibration techniques aligning LLM evaluators direct scoring (Figure 1, middle) with human jud judgements. We find that existing calibration methods are insufficient for aligning LLM evaluators, even when provided with supervised data. We discover that misalignment in evaluation is not primarily due to biased priors over evaluation scores, but rather from misaligned evaluation standards, i.e., the likelihood of LLM evaluators defined in Equation motivated to explore new evaluation paradigm with LLMs to facilitate a more aligned judgement.
(1). Therefore,
Inspired by how LLMs are aligned through reward model with preference data in RLHF, we argue that LLM evaluators can produce more aligned and inherent evaluations by leveraging their pairwise ranking abilities. Recent work has started exploring this intrinsic property by prompting LLMs with pair of candidates (Wang et al., 2023b; Chen et al., 2023; Qin et al., 2023; Li et al., 2024). However, the complexity and scalability of evaluating with pairwise preference have been largely overlooked. To produce a preference ranking for evaluation, existing methods require exhaustive pairwise comparisons, as they ignore the transitivity assumption (Qin et al., 2023). This makes the evaluation procedure intractable as the number of comparisons grows quadratically (Zheng et al., 2024; Liusie et al., 2024).
In this work, shed new light on the full potential of pairwise preference in LLM evaluators and its alignment with human judgements. We propose mairwise-preferent Search (PAIRS), an uncertainty-guided search method that efficiently estimates the Maximum Likelihood Estimate (MLE) of preference rankings by searching through an uncertainty-prunee pairwise comparison space. PAIRS more effective and query-efficient than existing pairwise baselines while substantially outperforming traditional direct scoring evaluations. We extensively examine the applicability of PAIRS in representative evaluation tasks, including summarization and open-ended generation. We show that PAIRS produces more robust and transitive evaluations compared with the greedy search baseline. Lastly, we provide further insights into how pairwise preference can be used to quantify the transitivity ability of LLM evaluators and how it benefits from calibration.
In sum, we provide the following contributions: 1) We present systematic analysis of the limitations of calibration in aligning direct scoring LLM evaluators with human judgements. 2) We formulate the evaluation as ranking from the perspective of transitivity and propose
The term transitivity in this work refers to mat mathematical transitivity, not linguistic transitivity. It property stating that if A is preferred to B and B preferred to then A is preferred to C.
2

---

Published as a conference paper at COLM 2024
HANNA (Coherence)
SummEval (Coherence
0.7
0.6
6.5 tu/bo
Mistral-78
GPT-3.5 calibrated
Mistral-78 callbrated
Dencity
0.4
0.2
0.1
Score
Figure 2: LLM evaluations are misaligned with human judgements. The score histograms evaluating the coherence in HANNA (Chhun et al., 2022) and SummEval (Fabbri et al., 2021). We present the scores from gold human evaluations, LLMs, and LLMs after calibrations. The histograms can be interpreted as estimated score prior distributions via marginalization.
PAIRS, an uncertainty-guided pairwise preference search method that efficiently estimates MLE rankings. We demonstrate that evaluations. We also provide insights into the transitivity and calibration of pairwise preferences in LLM evaluators.
PAIRS attains unique scalability in aligning LLM
On the Limitations of Calibration in Aligning LLM Evaluators
Misalignment Between LLM Evaluators and Human Judgements. The prediction score- based LLM evaluators can be expressed as a posterior predictive distribution (syy, I):
PM(ssy;x,11 1) PM(yls;,1I) x (M,,;,,
(1)
Likelihood
Prior
where and y form input-output pair (e.g., source text and its summary, respectively), denotes an instruction prompt, and s represents the model's rating score in its evaluation. Similarly, we express the human posterior predictive distribution simplicity, we omit the x and I terms hereafter. When utilizing LLMs to perform Likert scale evaluations (Likert, 1932), is known that there exists mismatch between the score distributions of LLM ratings and human annotations (Liu et al., 2023b). We start by observing and interpreting the same misalignment phenomenon, as shown in Figure
(s);,, I). For
Since the posterior distribution consists of two components, the likelihood, and the score preference prior, we hypothesise that the discrepancy befween the posterior distributions of LLMs and humans mainly arises from misalignment between their respective likelihoods. To verify this hypothesis, we conduct a preliminary experiment by calibrating the L.M''s score preference prior, (s) to match the human prior, pH (s), described in what follows.
Ccore-Based LLM Evaluator Calibration. Following previous calibration methods (Zhou et al., 2024b; Jiang et al., 2023b), we first assume that the likelihood terms for both the LLM and humans are the same, P. (yls))) = P# (y]s). Upon this assumption, the human posterior distribution (sly) can be expressed as:
(sly) = (H(s)) (H (.((s) (s
&
(2)
(H
We can estimate LLM's preference prior (s) by marginalizing the model's predictions, iee, (s) = Σy (M) Unlike previous work that assumes a uniform prior for PH(s) (Zhou et al., 2024b), we directly estimate by marginalizing human annotations to achieve a more accurate estimation. The calibration is performed by scaling the model's predictions according to Equation (2).
In Figure 2, the experimental results show that even after calibration, the gap persists between the posterior distributions of the L.I and human evaluators. Specifically, the Mean Absolute
3

---

Published as a conference paper at COLM 2024
Error (MAE) shifts from 1.62 to 1.16 HANNA and from 0.78 to 0.86 on SummEval, indi- cating notable misalignment. This suggests that the discrepancy is not primarily due to the misaligned priors but rather stems from the differences in the likelihood of LLM evaluators. In other words, the assumption that LLM evaluators and humans have similar evaluation likelihoods becomes invalid, i.c., P. (y)s) + Px (y]s)) The likelihood term reflects expected output candidates for given score, which can be interpreted as the underlying evaluation standards. LLMs learn their standards from pretraining data, potentially diverging from the human evaluation standards. Hence, these findings motivate efforts to reduce the gap between evaluation standards.
3
Uncertainty-Guided Pairwise-Preference Search
3.1 Evaluation as Ranking
Inspired by the fact that the RLHF training stage of most recent LLMs utilizes pairwise preference or preference ranking data (Touvron et al., 2023), we propose that evaluation with pairwise comparisons could be promising solution to address the above issues. We conclude the motivation: The difference between LLM and human evaluation slandards is smaller when performing pairwise comparison, compared to raling with scores.
When using an LLM to perform a pairwise comparison, we can compute the preference probabilities P(y; > yj) from output logits, where > denotes the preference relation "is preferred to". However, LLM evaluators do not guarantee transitivity property, e.g. given LLM's pairwise preferences yi > jj and y; > Yı, it is not necessarily true that the LLM will have > y/- The problem of finding the optimal preference ranking from non-transitive pairwise comparison probabilities is known the Rank Aggregation problem (Dwork al., 2002). We formulate this problem as a Maximum Likelihood Estimation (MLE) problem.
Let !1:N be a set of N output candidates to be ranked. For a pair of outputs (y:, we have the probability ply > yi) that output y/ is preferred over output yj- We denote permutation (ranking) of these candidates as π = (TT, , where πτ; is the index of the i-th ranked candidate, i.e., Vr, is the i-th ranked candidate output in π. The ranking imposes total order on VI,N, such that Yn, > 1/Tr >... > VTy- The goal is to find a ranking of the output candidates that maximizes the likelihood of observing pairwise probabilities:
(T)(T) = II P(yn, > Vnz)),
where LN stands for the non-transitive likelihood, 2 which does not make any transitive assumptions for the pairwise comparison probabilities. The MLE ranking of CNT is also equivalent to the Kemeny-Optima Ranking (Kemeny, 1959). However, finding the optimal ranking is challenging in two aspects:
1. Computing LN requires a complete set of O(N2) pairwise comparisons.
2. The exact solution to the rank aggregation problem is NP-hard (Dwork et al., 2001), meaning that there no known polynomial-time algorithm to solve it optimally.
If LLM evaluators are assumed to possess transitivity property, then the problem of finding the MLE ranking reduces to sorting problem, which can be solved by standard sorting algorithms such as merge sort or quick sort with only O(N log N) pairwise comparisons.
However, strict transitivity is a strong assumption. To relax the strict transitivity assumption to some extent and find a better trade-off between computational complexity and perfor- mance, we propose simplifying the non-transitive likelihood in Equation (3) by assuming compositional stochastic transitivity (Latta, 1979; Oliveira et al., 2018). The derivation of this simplified likelihood is provided in Appendix SC.2:
2This likelihood can be derived from the general learning rank likelihood in Appendix SC.1.
4

---

Published as a conference paper at COLM 2024
Input subarrays
Algorithm Uncertainty-GGudee PAIRS-beam
Output merged array
Pairs
1: Objective: Find the ranking that maximizes the transitive likelihood LT.
AD
A2
B1
2: Hyper-parameters: Beam size, beam size; Un- certainty threshold, Un,
PairS-beam
3: Inputs: Two sorted input sub-arrays a
P(BT-M0)
RB0-88-200-0661
A2., and b = [B,, B2,...... Bg].
4: Output: A merged sorted array.
BO
P(DT-A))
B1
5: Initialize: Initialize Beam B with the first candidate cand. Each cand keeps track of its choice trajectory, two pointers and j for a and b, and cumulative likelihood of its trajectory.
POBO-A0
PVND-11)
P(BD-AC-B1-ATT-O..
PJAT-B1J
A1
6: for k 1 to L R do
Create another empty beam B'. for cand in do
RED--AO-AT-81--005
P(ND-B0)
P(BD-A1)
8:
9:
P(A, > UA,B, + (LM B,)
AD
10:
if UA B, > Un then
11:
B'' + U (aand.updatefa.
PJAT-B0)
12:
13' + B' Camd.uppdate(B)
13:
else if P(A, Bj) 2 1 then
PAA--83)
A2
14:
B B' Cand.update(AA)
PYAD-AT-A2-B0--0.33
15:
else
16:
B- B U cand.update((B)
17:
end if end for
Beam candidate
Filtered by likelhhood
Filtered by uncertainty
18:
19:
B' + =sort(B', key LT)
20: B - B: beammmsst Seea__size]
Figure Visualization of PAIRS in ranking with a beam size of 2 on the same evaluation examples in Figure 4.
21:
end for
22:
return .rajeerryyy
CT(TT= II P(yn, ymp1),
(4)
where L1 stands for the transitive likelihood, and the MLE ranking is denoted as π
arg max, (TTT).
3.2 Uncertainty-Guider Pairwise-Preferrece Search
B0
B1
B2
B3
The traditional merge sort algorithm em- ploys divide-and-conquer approach, merg- ing two sorted subarrays each operation step. This merge operation can be interpreted as a greedy search procedure that aims to find the ML.E ranking πt under the transitive like- lihood. We refer to this approach as PAIRS- greedy. However, the non-transitive nature of LLMs suggests that optimization based on local probability may not lead to the optimal solution. Therefore, we also propose PAIRS- beam, heuristic beam search algorithm that improves upon the greedy search merge oper- ation by considering both candidates at each step.
A0
A1
A2
arbay
A3
A4
-0
A5
002
Sub-array B
Figure 4: Illustration of PAIRS with beam size of 2. Heatmap values show P(A; > B,) by Mistral 7B, deciding the search direc- tion. Yellow-bordered lines highlight com- parisons whose uncertainties are above Un- Solid lines represent greedy search paths, while dashed lines denote alternative paths by other beam candidates.
The PAIRS algorithm is described in Algo- rithm 1, and demonstration of its workings is provided in Figure and Figure 4. Intu- itively, while the PAIRS-greedy chooses the preferred items between the heads of two sub- arrays in greedy fashion, PAIRS-bean main-

---

Published as a conference paper at COLM 2024
tains likelihood LT beam that keeps track of the top-k trajectory candidates. By also considering less preferred items between the two heads, PAIRS explores a broader search space and potentially finds better ranking.
To further improve the efficiency of the beam search, we introduce a pruning mechanism based the LLM's uncertainty. When the uncertainty of the LLM's pairwise comparison is lower than a threshold Un, we prune the less preferred candidate from the beam. This uncertainty-based pruning restricts the search space, making the beam search more efficient and achieving better balance between computational complexity and ranking performance.
To represent the uncertainty between two items, we follow commonly used approach in previous works (Wan et al., 2023a;b) that employs entropy as the uncertainty measurement. Hence, the uncertainty between a pair of comparisons can be expressed below:
yyy H(y.. vj) = Ply: > yj) log(P(y: > yj) - Ply; > yi) log > yi)). (5)
3.3 A Scaling Variant of PAIRS
The theoretical complexity of PAIRS-beam ranges from O(N logN) N) to O(2)) pairwise comparisons, as shown in Figure 5. The upper bound occurs when the uncertainty- based pruning is disabled, and the beam size is sufficiently large to explore all poten- tial trajectories during the merge operation. Conversely, the lower bound corresponds to PAIRS-greedy. However, for large instances, even the complexity of ONN log N) becomes computationally expensive and infeasible. To address this issue, we propose two-stage scaling method that leverages the granularity of Likert scale annotations.
10a
O(N))
Number of model calls
105
BarS-beam
PairS-greedy, O(NlogN)
PairS beami-scaled
10a
O(N)
10a-
10*
10a
200
400
800 1000 1200 1400 Number of samples
600
In the first stage, we sample and rank sub- set of the evaluation set, referred to as the anchor set. The size of the anchor set can be statistically determined by accommodating
Figure 5: PAIRS is scalable. Complexity plot for different pairwise preference algorithms.
tolerable error margin (Cochran, 1977) (Further details and empirical evidence in Ap- pendix $F.3). In the second stage, we binary search to find the positions of the remaining samples relative to the anchor set, efficiently determining their appropriate positions in the overall ranking. As depicted in Figure 5, the second stage of the scaling method results in O(N) complexity, a significant improvement over the original complexity. This two-stage scaling approach allows PAIRS to efficiently handle large evaluation sets while maintaining the benefits of beam search and uncertainty-based pruning. The scaling variant strikes balance between complexity and ranking performance, making it suitable for real-world applications involving large-scale datasets.
3.4 On the Benefits of Pairwise-Preferent Search
Transitivity. Both PAIRS-direct and PAIRS-beam assume transitivity some extent, im- plying that an LLM with better transitivity can perform more accurate and more efficient evaluations. In other words, we expect better-performing LLMs to exhibit more transitive behaviour when making pairwise comparisons. Consequently, we propose that transitiv- ity can serve as an effective measure of an LLM's ability as an evaluator. By quantifying and comparing the transitivity of different LLMs, we can gain insight into their relative performance and suitability for evaluation tasks. To validate this hypothesis, we provide complementary experiments and analysis in Section 5.4.
Calibration. Although pairwise preference inherently aligns LLM evaluations with human judgement, pairwise prompting can still suffer from misaligned preference priors, e-g.

---

Published as a conference paper at COLM 2024
positional bias and contextual bias (Zhao et al., 2021; Zhou et al., 2024b). Therefore, propose applying calibration on top of PAIRS to ensure debiased PAIRS evaluations. It worth noting that, unlike calibrating the score-based LLM evaluators, which requires external information on the distribution of human preference prior, as shown in Section 2, pairwise comparisons fully leverage uniform human prior, due to the random nature of pairwise sample selections.
4 Related Work
LLMs as Evaluators. Recently, LLMs have been increasingly used to automatically evaluate various natural language generation tasks (Zhong et al., 2022; Chen et al., 2023; Wang et al., 2023a; Tan et al., 2024). LLM evaluators enable reference-free metrics while being cheaper alternates to human evaluation in a diverse task set, such as summarization (Shen et al., 2023), machine translation (Kocmi & Federmann, 2023), open-ended story generation (Chiang & Lee, 2023), dialog response generation (Liu et al., 2024a) and instructionfoolloiing (Zeng al., 2024). Unlike traditional reference-based evaluation metrics (Papineni et al., 2002; Liu et al., 2024b), LLM evaluators are usually equipped with evaluation criteria prompts and generate scores to access the quality of the candidate text (Liu et al., 2023a). This forms the scoring-based LLM evaluator that enables easy adaption to various tasks by simple instructions (Fu al., 2023). To fairly examine various LLM evaluation abilities, efforts have also been devoted to developing meta-evaluation benchmarks (Wang et al., 2023b; Zhang al., 2023; Zheng et al., 2024) and prompt optimization (Zhou et al., 2024a).
reference-guided
Pairwise Preference. Preference data have been widely used in training the reward model to reflect human preferences in RLHF (Ouyang et al., 2022). For example, Llama-2 has leveraged binary preference with 'chosen' or 'rejected' responses in its reward modelling (Touvron al., 2023). Following the preference idea, using pairwise comparisons in evalu- ating the LLM outputs has also been studied to some extent recently (Chen et al., 2023; Shen et al., 2023), where Wang al. (2023b) find that pairwise comparison exhibits better human correlations than traditional scoring-based LLM evaluators. However, Zheng et al. (2024) point out the intractability of pairwise comparisons due its quadratic query complexity (Liusie et al., 2024), which significantly limits its practicability. Qin et al. (2023) propose pairwise ranking with sorting but ignore the important transitivity of LLLM predictions. In our work, we address the aforementioned issues by searching in a tractable pairwise search and transitive space based on model uncertainty.
5 Experiments
5.1 Experimental Setup
Datasets. We evaluate PAIRS using three commonly used meta-evaluation benchmarks for two types of NLG tasks: abstractive summarization and creative story generation. Following previous research, we adopt two evaluation strategies, namely sample-level and dataset- level Spearman correlations, as detailed in Appendix $E.1. For summarization task, we select News Room (Grusky et al., 2018) and SummEval (Fabbri et al., 2021), consistent with prior studies (Zhong et al., 2022; Fu et al., 2023). For the story generation task, we include HANNA (Chhun et al., 2022). 3 We refer to Appendix SD for more details.
Models. To cover a good range of LLMs, we select two open-source models: Mistral 7B (Instruct-v0.1) (Jiang et al., 2023a) and Llama-2-chat 7b (Touvron al., 2023), and two closed-source models: GPT-3.5-turbo (-1106) and GPT-4-turbo (-preview) (OpenAI, 2023). For open-source models, we calculate the preference probability by normalizing the logits of choice ''' and 'B'. For GPT-3.5 and GPT-4, we return top-5 log- fog-probabilities, which can be normalized to find the preference probability.
We make the first application of pairwise preference to evaluate over thousands samples. We use anchor size of 100 Paleee in HANNA.
7

---

Published as a conference paper at COLM 2024
News Room
SummEval
HANNA
Models
CH
RE
IN
FLU
CH
FLU
CON
RE
CH
SU
CX
Other Metrics
BertScore
0.15
0.16
0.17
0.28
0.19
0.11
0.31
0.22
0.30
GPTScore (d02)
0,31
0.35
0,26
0.31
0.28
0.31
0.38
0.22
0.22
0.25
0.08
UniEval (single best)
0.54
0.43
0.47
0.46
BARTScore (CNN)
0.65
0.56
0.61
0.64
0.45
0.35
0.38
0.35
Mistral 7B
Scoring
0.32
0.39
0.20
0.26
0.23
0.19
0.37
0.19
0.30
0.26
0.37
G-Eval
0.36
0.36
0.24
0.39
0.25
0.20
0.39
0.25
0.34
0.25
0.39
PAIRS-greedy
0.57
0.52
0.53
0.44
0.25
0.08
0.28
0.29
PAIRS-beam
0.55
0.53
0.48
0.48
0.28
0.24
0.30
0.27
0.29
0.27
0.31
Llama-2-chat
Scoring
0.02
0.14
-0.02
0.01
0.12
0.06
0.18
0.11
0.25
0.17
0.33
G-Eval
0.05
0.14
-0.15
0.04
0.15
0.07
0.23
0.20
0.31
0.17
0.35
PAIRS-greedy
0.38
0.41
0.25
0.16
0.16
0.28
0.20
PAIRS-beam
0.43
0.43
0.37
0.28
0.17
0.18
0.31
0.24
0.29
0.19
0.17
GPI-3.5-turbo
Scoring
0.47
0.46
0.50
0.32
0.28
0.49
0.25
G-Eval
0.45
0.45
0.45
0.52
0.30
0.29
0.47
0.27
0.28
0.18
0.39
PAIRS-greedy
0,56
0.55
0.53
0.45
0.35
0.23
0.41
0.27
PAIRS-beam
0.56
0.57
0.52
0.49
0.42
0.37
0.45
0.29
0.38
0.29
0.47
GPT-4-turbo
Scoring
0.55
0.54
0.57
0.60
0.44
0.42
0.46
0.53
0.35
0.28
0.38
G-Eval
0.58
0.55
0.57
0.58
0.45
0.42
0.45
0.54
0.34
0.31
0.41
PAIRS-greedy
0.62
0.59
0.62
0.59
0.53
0.45
0.47
0.58
PAIRS-beam
0.64
0.61
0.60
0.53
0.48
0.59
0.44
0.36
0.49
Table 1: LLM evaluation results. We report Spearman correlations all datasets in terms of the following evaluation aspects: coherence (CH), fluency (FLU), relevancy (RE), informa- tiveness (IN), consistency (CON), surprise (SU), and complexity (CX). For PAIRS-beam, the experiments were conducted with beam size of 1000 and an uncertainty threshold at 0.6.
Baselines. We consider two types of reference-free baselines: optimized and zero-shot evaluators. Optimized evaluators, UniEval (Zhong et al., 2022) and BARTscore (Yuan et al., 2021), mitigate the evaluation standard gaps using annotated data, making them specialized for a specific dataset or domain but not transferable. We treat them as strong but unfair metrics and use their reported scores as a reference. For zero-shot evaluators, we re-implement the following baselines: BERTScore (Zhang et al., 2020), G-Eval (Liu et al., 2023a), and GPTScore (Fu al., 2023). We report direct scoring and G-Eval as the main baseline for fair comparisons.
5.2 Main Experiments
Performance Gap between Direct Scoring and PAIRS. As shown in Table 1, PAIRS, zero-shot general evaluator, achieves state-of-the-art (SOTA) or near-SOTA performance all datasets, even when compared with the optimized metrics UniEval and BARTScore, which are trained on dataset-specific data. In general, PAIRS outperforms direct scoring and G-Eval baselines in most aspects, with some exceptions, such the consistency (CON) aspect of SummEval. We attribute this to the concentrated human scores distribution, with 86.7% summaries labelled with score of 5. These characteristics make difficult for pairwise comparisons to yield meaningful comparisons and consequent rankings.
Impact PAIRS on Different (Families of) LLMs. PAIRS particularly benefits the per- formance of smaller LLMss. For Mistral 7B and Llama-2 7B, we observe that pairwise comparison methods generally outperform direct scoring on News Room and SummEval. Notably, Mistral 7B achieves performance comparable to GI'T-3.5-turbe hypothesize that this is due their potentially similar quality of instruction tuning. Llama-2 7B struggles to effectively rate summaries with scores but demonstrates much better perfor- mance when comparing sunmaries. For HANNA, both small models do not benefit much from pairwise comparison, possibly due to the difficulty of understanding the long-context
News Room. We
8

---

Published as a conference paper at COLM 2024
Mistral-7B
GPT-3.5-turbo
40-
Spearman correlation
35
25-
20
Win Caseline
Win baseline
ELO baseline
ELC baseline
15
PairS- PairS-beam beam
PairS-beam
rairr-greedy
PairS-gredd
10
PairS-beam-caallle beam-calibrated
PairS SairS-beule aean-caa lorated
5
Pairs greedy calibratec
Pairs greedy callbraced
50
100
150
200
100
150
200
Number of comparisons
Number comparisons
Figure 6: PAIRS-greedy and PAIRS-bean are more efficient than other pairwise rank aggregation methods. Both diagrams are evaluating on the coherence of SummEval. Each datapoint contains 16 summary candidates, with the number of pairwise comparisons ranging from to 240. PAIRS-beam curves show results for beam sizes 1, 2, 5, 10, 20, 50, and 100, without uncertainty pruning. When beam size is 1, PAIRS-beam and PAIRS greedy are equivalent.
prompts or the presence of length bias. Conversely, direct scoring performs relatively well compared to summarization tasks. We believe this is because story generation evaluation does not depend on input, making the task more straightforward. For the two closed-source GPT models, PAIRS generally outperforms direct scoring and G-Eval baselines on all three datasets, indicating that PAIRS can achieve better performance more consistently for more capable models.
Benefits of Uncertainty-Guided Search. PAIRS-beam generally exhibits higher correlations and lower Standard Error (SE), as shown in Table 2, than PAIRS-greedy on all datasets, indicating its ability to more robustly produce better rankings. Interestingly, the performance difference between PAIRS-greed, and PAIRS-beam is minimal for GPT-4-turbo. We attribute this to GPT-4-turbo's better transitivity, allowing the greedy search robustly find near- optimal rankings. There are also few cases where PAIRS-beam cannot outperform PAIRS- greedy. This could occur when the LLM's preference standards, the likelihood term, do not perfectly match with human annotations.
5.3 Efficiency Comparison with Rank Aggregation Baselines
PAIRS is efficient pairwise rank aggregation method. We evaluate method against two baseline rank aggregation techniques: 1) Win-loss rate and 2) ELO rating, which can both aggregate rankings from any number of pairwise comparisons. To assess their efficiency as aggregation algorithms, we calculate the correlations between human annotations and rankings predicted by PAIRS and the baselines using varying numbers of comparisons. The comparisons were random sampled without replacement. We also include the calibrated PAIRS, which requires double the computation by considering both permutations for each comparison pair.
Figure 6 demonstrates that PAIRS more efficient than the baselines. PAIRS-greedt typically requires only about 30% of the comparisons to achieve performance similar to ELO rating. The calibrated PAIRS generally outperforms the upper bound of the baselines, where all N(N 1) pairwise comparions are utilized. We attribute the strong performance of PAIRS to its leveraging of the transitivity assumption of LLMs to identify the most informative comparison pairs, which provides natural efficiency advantage. While win-rate and ELO aggregation methods are designed to handle systems or players with variable performance, making them more effective at dealing with changes, our task involves ranking constant response candidates on a specific aspect. In this context, sorting-bast based algorithms that exploit transitivity have inherent advantages.

---

Published as a conference paper at COLM 2024
Models
News Room
SummEval
HANNA
Models
News Room
SummEiva
HANNA
M-7B greedy
55.7_1.22
25.6.10.89
33.213.2
M-7B
55.6
25,6
33.0
M-7B beam
54.7
27.6.10.2
33,8.2.9
calibrated
57.7
L-7B greedy
39.5=1.66
14.9++194
29.4+2.7
L-7B
38,9
15.6
29.2
L-7B beam
43.2.1..33
16.0.11.3
33.911.5
calibrated
40.5
16.2
31.5
G-3.5 greedy
54.6=1.57
36.2440.88
33.742.2
G-3.5
54.2
36.8
37.6
G-3.5 beam
55.9=0.72
41.7+0.43
36.2+1.0
calibrated
56.7
38.5
37.8
Table 2: Quantifying transitivity. Mean and std. of Spearman correlations (CH aspect)
Table 3: Calibrating PAIRS. Mean of Spear- man correlations (CH aspect) over 10 runs PAIRS-greedy, w. and w/ calibration.
over 10 runs with different random seeds.
5.4 Ablation Studies
Transitivity. The transitivity assumption plays crucial role in the effectiveness of LLMs as evaluators, impacting the speed and consistency with which they achieve rankings. Hence, it is a good measurement of the capability of LLMs as evaluators. We quantify transitivity through the analysis of correlation variations, specifically the Standard Error. We conducted 10 runs of both PAIRS-greedy and PAIRS-beam with different random seeds. For the HANNA dataset, we sample and shuffle the same subset of 200 stories. Table shows that PAIRS-beam generally demonstrates better correlations and lower SE than PAIRS- greedy, which indicates that it is more robust. We notice that GPT-3.5-turbo consistently exhibits fewer errors across all datasets, suggesting that it can produce more consistent evaluations with itself, thus possessing better transitivity. Furthermore, different datasets show varying levels of errors, which we believe indicates the transitivity difficulty of each dataset. For example, compared to other datasets, HANNA has relatively higher errors, which means that, for the same base models, its optimal rankings are more difficult to find. This observation is expected as HANNA, being a creative story-writing dataset, has naturally more subjective human annotations, causing challenges for LLM alignment.
Calibration. Unlike direct scoring, pairwise comparison has known preference prior, which is uniform. We perform the same calibration method as in the previous work by Zheng al. (2023), which averages the probabilities of both permutations of the candidates. Table demonstrates that calibration consistently improves the performance of Mistral 7B and Llama-2 7B. However, calibration on GPT-3.5-turbo yields relatively small im improvements, likely due to its heavily skewed output probabilities (also shown in the example in Figure 7, Appendix $E2). The over-confident predictions make the language model difficult to calibrate.
Conclusion
In this work, have first conducted systematic analysis of LLMs used direct scorers for evaluation in generative tasks. The analysis has revealed a misalignment between LLM and human evaluators, which cannot be effectively mitigated by applying calibration techniques. Inspired by RLHF, we have then proposed the use of pairwise comparisons which should produce evaluation rankings that are inherently more aligned human judgements. We have formulated the evaluation as ranking problem from the perspective of transitivity and proposed PAIRS, an efficient, scalable, and imcertainty-guudee method that navigates the pairwise comparison space and estimates the MLE rankings. Extensive experimental results have demonstrated that PAIRS is more aligned with human annotations and achieves SOTA performance compared to direct scoring and domain- optimized baselines. Furthermore, we have shown that PAIRS can also be used to measure LLM''s transitivity.
ranking
10

---

Published as a conference paper at COLM 2024
Acknowledgements
The work is supported by the UK Research and Innovation (UKRI) Frontier Research Grant EP/Y031350/1 (the UK government's funding guarantee for ERC Advanced Grants) awarded to Anna Korhonen at the University of Cambridge. The work has been supported in part by a Royal Society University Research Fellowship (no 221137; 2022-) awarded to Ivan Vulić, and by the UK EPSRC grant EP/T02450X/1.
References
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gon- zalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. Xiv preprint Xri::2332.111800 2023a.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arviv preprint ar arXi::2305 10403, 2023b.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Berbert-VVsss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, viriual, 2020.
Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. Exploring the use of large language models for reference-free text quality evaluation: An empirical study. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Findings of the Association for Computational Linguistics: IJCNLP- AACL 2023 (Findings), pp- 361-374, Nusa Dua, Bali, November 2023. Association for Computational Linguistics. URL httpt:tttcccaaaaeee http:::///lllahhogggoloo .rrgllooo.ooo/o/// 2rg//023. findings- ijcnlp. 32.
Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 7794-5836, Gyeongju, Republic of Korea, October 2022. International Committee Computational Linguistics. URL mttps://aclaantholgg 2022 1.ling11.500
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607-15631, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ 200333/11/2222 /2w23.acl-oong..o.... URL httpp::/accladeee /wwpaaaaaaaaagyooa/a///oo org/ acl acl-long.877 870.
11

---

Published as a conference paper at COLM 2024
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Urrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Con- ference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4299-4307, 2017. URL ppwoee/d//.occe/e.......... neur neurips. cc/ cc/paper/2000 /201 //hash/ d5e2c0adad503cc91 2et.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. arXio preprint arXin:2210.11416, 2022.
William G. Cochran. Sampling Techniques, 3rd Edition. John Wiley, 1977. ISBN 0-471-16240-X.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https: //tttrri//ppereviee.. net/Forum2id-TyFFPKe
Cynthia Dwork, Ravi Kumar, Moni Naor, and D Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th International Conference on World Wide Web, pp. 613-622. ACM, 2001.
Cynthia Dwork, Ravi Kumar, Moni Naor, and D. Sivakumar. Rank aggregation revisited. 2002. URL https: //ai. semant inrpranticccollar..oomm orgg/CrrpsIDD:133 139767991.
Alexander R. Fabbri, Wojciech Kryšciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summ Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409, 2021. doi: 10.1162 tt3//
URL
wwtpoogg..rgg///// org/2021. wttgg/222.taacl1... tac1-1.24.
Peter C Fishburn. Binary choice probabilities: on the varieties of stochastic transitivity. Journal of Mathematical psychology, 10(4):327-352, 1973.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. ar Xiv preprint ar erXi:22302.04166 2023.
Max Grusky, Mor Naaman, and Yoav Artzi Newsroom: A dataset of 1.3 million sum- maries with diverse extractive strategies. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp- June 2018. Association for Computational Linguistics. doi: 10.18653 //////v1/11111/0 1065. URL https: / http::s:/aallathollgyyy wwwphhoooooo.oo////8//// 8-NN1881065 1065.
708-719 New Orleans, Louisiana, June
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023a.
Zhongtao Jiang, Yuanzhe Zhang, Cao Liu, Jun Zhao, and Kang Liu. Generative calibration for in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 2312-2333, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653/ /22/2223 findings-emmll.115 1552. URL https::
org/ findings-emnnn emniii..gseelll.111....... 152. 152.
John G Kemeny. Mathematics without numbers. Daedalus, 88(4):577-511, 1959.
Tom Koemi and Christian Federmann. Large language models are fhate-of-thh-ate evalua- tors of translation quality. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassen- hove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 193-203,
12

---

Published as a conference paper at COLM 2024
Tampere, Finland, June 2023. European Association for Machine Translation. URL nttps://aclanthologg org/2023 eamt-1.19.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Sanmi Koyejo, S. Mohamed, A. Agar- wal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Robert B Latta. Composition rules for probabilities from paired comparisons. The Annals of Statistics, pp. 349-371, 1979.
Chengzu Li, Han Zhou, Goran Glavaš, Anna Korhonen, and Ivan Vulić. On task performance and model calibration with supervised and self-ensembled in-context learning. ar preprint ar Xit:2312.13772,, 2023a.
Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, and Chongyang Tao. Leveraging large language models for nlg evaluation: A survey. ar aiviv preprint ar Xix::2401.07103 2024.
Zongje Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in large language model based evaluators. arXiv preprint ar Xix:::2310.014 01440,,,11.... 2023b.
Rensis Likert. A technique for the measurement of attitudes. Archives of psychology, 1932.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2522, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 00.1663//112023.mmmp// //aclanthology, org 0rg/2023. e000233.mmlppmai.... 153.
URL https:
Yinhong Liu, Yimai Fang, David Vandyke, and Nigel Collier. Toad: Task-oriented automatic dialogs with diverse response styles. aiviv preprint ar drXiv:2402.10133 2024a.
Yinhong LIu, Yixuan Su, Ehsan Shareghi, and Nigel Collier. Unlocking structure mea- suring: Introducing PDD, an automatic metric for positional discourse coherence. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 2: Short Papers), pp. 92-100, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10.18653/ 100/663331/222 -1/2024.nacclshortt URL https://aclantthll// wrw.hholloooooo//22244444 mhacl-short.9
Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Calibrating llm-based evaluator. arXiv preprint drXie:2309.13308, 2023b.
Adian Liusie, Potsawee Manakul, and Mark Gales. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume Long Papers), pp. 139- 151, St. Julian's, 's Malta, March 2024. Association for Computational Linguistics. URL https: / http:://aallanhologyy wwwpthhoolgg.oooo////ee
eacl-long.
IFD Oliveira, S Zehavi, and O Davidov. Stochastic transitivity: Axioms and models. Journal of Mathematical Psychology, 85:25-35, 2018.
OpenAI. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systemss, 35:27730-27777 277444, 2022.
13

---

Published as a conference paper at COLM 2024
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 033115/1073083.00733333 URL https: httpp:://aclaatholoogyy org/P02-1048
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity the order of options in multiple-choice questions. arXiv preprint ar Xit:2308..11488 2023.
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. Large language models are effective text rankers with pairwise ranking prompting. ar preprint arXiv:2306.11 17563, 2023.
Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbasity bias in preference labeling by large language models. arx preprint ar arXit:2310.10011 2023.
Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. Large lan- guage models are not yet human-level evaluators for abstractive summarization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Com- putational Linguistics: EMNLP 2023, pp. 2215-4233 Singapore, December 2023. Associ- ation for Computational Linguistics. doi: 10.18653 vv h/////122000 12/022...nniin findinns--mmll 2778. URL org/2023....22oo findings-emnlp 278.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. arXiv preprint ar X10:22401.15042, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. Better zero-shot reasoning with self-adaptive prompting. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Pp. 3493-3514, Toronto, Canada, July 2023a. Association for Computational Linguis- tics. doi: 10.18653 (t.18633vv1/ /t//////2222///////// wttp://...nningssaa...e. findings-acl.22 findings- aels-acl.20 216.
URL https:/ haap::://cllnhhooogyy wtw.ollloyyyr.g/222//// org/2023.
Xingchen Wan, Ruoxi Sun, Hootan Nakhost, Hanjun Dai, Julian Eisenschlos, Sercan Arik, and Tomas Pfister. Universal self-adaptive prompting. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 7437-7462, Singapore, December 2023b. Association for Computational URL https: http///aaaaaaaloo///aao org/
Linguistics. doi: 10.18653 1118653/v1/200 2023. ee3..emllpmmannn.... 461.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is ChatGPT good NLG evaluator? a preliminary study. In Yue Dong, Wen Xiao, Lu Wang, Fei Liu, and Giuseppe Carenini (eds.), Proceedings of the 4th New Frontiers in Summarization Workshop, pp. 1-11, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653 https://aclanthhllgg org. newsum-1.1.
URL
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXio preprint ar arXit:2305.17922 2023b.
14

---

Published as a conference paper at COLM 2024
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2022. URL https://openrevieew
net/forru?ideggZrrGooo..
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263-27277 2021.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024. URL https:// /ttp:////pp//eeeeeeeeeeee net/forumidd= trØKidwPLc.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https: http:::///ppeeeviieeeee wwweeereeeee.eeeeorrriee..
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper Ilm networks are fairer Ilm evaluators. arv preprint ar arXiv:2308.0153 2023.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Im- proving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pp. 12697-12706,
2021.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Bric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards unified multi-dimensional evaluator for text generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2023-2038, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ org/ 2022. emnlp-main. 131.
v1 11/2022.emmlpmmain..1 131. URL https: /tattts://aclaatthoinn
Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. Survival of the most influen- tial prompts: Efficient black-box prompt search via clustering and pruning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Pp. 13064-10333 13077, Singapore, December 2023. Association for Computational Linguistics. URL https::/aalaatthology org/ 2023. findings-emm emnlp. 870.
Han Zhou, Xingchen Wan, Yinhong Liu, Nigel Collier, Ivan Vulić, and Anna Korhonen. Fairer preferences elicit improved human-aligned large language model judgments. arXiv preprint ar xrXio:2406.11377 2024a.
Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine A Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In The Twelfth International Conference Learning Representations, 2024b. URL https://openreviiew net httefforrummii--LFFlok.....
Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. Autopeft: Automatic con- figuration search for parameter-effiiient fine-tuning. Transactions of the Association for Computational Linguistics, 12:525-542, 2024c.
15

---

Published as a conference paper at COLM 2024
A
Additional Related Work
Calibrating LLM Evaluators. Though previous studies have shed light on using LLMs as evaluators, LLMs have been exposed their sensitivity to prompt designs (Zhao al., 2021; Zhou et al., 2023), and even biased from various factors, including but not limited to positional bias (Wang et al., 2023b; Pezeshkpour & Hruschka, 2023), verbosity bias (Saito et al., 2023), and contextual bias (Zhou et al., 2024b). Calibration methods are then proposed to mitigate these biases by criteria optimization (Liu al., 2023b), context decomposition (Li et al., 2023b), order permutation (Wang et al., 2023b; Zheng et al., 2023), ensembling (Li et al., 2023a) and batching (Zhou et al., 2024b). However, we show that calibrations are yet insufficient in aligning LLM evaluators with humans.
B Future Work
By formulating the evaluation ranking problem, we provide a playground for future work to develop better searching algorithms. PAIRS is type of Breadth-First Search (BFS) method. However, under different transitivity assumptions, there may be better trade-off between performance and efficiency. We encourage future work to explore and develop novel ranking algorithms that can optimize this trade-off.
Furthermore, future work can investigate alternative scaling approaches. For instance, by examining the performance of LLMs on selecting the best candidate from set of K options, instead of performing pairwise ranking, the search efficiency could potentially be improved from (log2N) to (loogNN). The improvement in efficiency could significantly impact the scalability and practicality of the search-based method, especially when dealing with large-scale datasets or real-world applications.
C Derivations
C.1 Rank Aggregation Likelihood Derivation
In the general setting of learning to rank, we have set of items and set of pairwise comparisons between these items. Each pairwise comparison indicates which of the two items is preferred or ranked higher. The goal is to find ranking of the items that is most consistent with these pairwise comparisons.
The likelihood function for the general ranking problem can be written
[(T)
II P(i j(r)) (LfECC
Here, C is the set of all pairwise comparisons, and P(i > j| is the probability of observing the comparison 'item i is preferred over item j' given the ranking π.
In the rank aggregation problem, and that the probability of observing a comparison 'item i is preferred over item j' given by P(i > i), which is provided as input. Therefore, the likelihood function for the rank aggregation problem can be derived from the general likelihood function by making the following substitution:
assume that the pairwise comparisons are independent
((t (1- 11-P(i>)) 1)
T( < π(j) 7f Λ(j).
P(i )(
In other words, if the ranking is consistent with the comparison 'item is preferred over item j', then the probability of observing this comparison is simply P(i > i). Otherwise, the ranking π is inconsistent with the comparison, the probability is 11 - P(i > j)-
Substituting this into the general likelihood function, we get:
L(T)
II
P(i i) x
II
(1- P(i > 1))
(f)(((
16

---

Published as a conference paper at COLM 2024
The second product can be simplified by noting that P(j > i) 1 1 P(i j), get:
((r))
II P(i > f) X
II
P(j i).
Finally, since the comparisons are assumed to be independent, the second product can be absorbed into the first one, giving us the likelihood function for the rank aggregation problem:
[(x)
II
P(i j).
C.2 Transitive Ranking Likelihood Derivation
In this section, we will show how we derive the transitive ranking likelihood, Equation (4), from the non-transitive ranking likelihood, Equation (3), given the compositional stochastic transitivity assumption.
Compositional stochastic transitivity (Fishburn, 1973; Latta, 1979; Oliveira et al., 2018) states that P(a > c) H(P(a > b), P(b > c)), where H IR + [0, 1] is some strictly-oontooee and symmetric composition rules. We use the notation of Pa.s to represent P(a > b). Therefore, we can have Pa.= = APap + A.per where À E [0,0.5].
For ranking 71,,, where 71 > 722 comparisons with distance k, e.g. PI.,TE Each comparison probability with distance k can be expressed by comparison probabilities with distance using k coefficients:
> TTN, of length N, there are N - k pairwise
PT.,i4 = A PT + Py TH
P + A2P 2, +... AR-1 A + 2 A
(6)
The number of Å coefficients has scale of O(N3). The non-transitive ranking likelihood can be expressed by probabilities of comparisons with distance 1.
(T) p(TI.N) = MiPr + M2Pm2.ng + +... P
(7)
If letting 14 #/2 MN-1, we can have around O(N2) constraint equations, expressed by As. As the dimension À coefficients is much larger than the number constraint equations, argue that there should always be set of À E [0,0.5 satisfying the constraints.
C.3 Ranking to Scores Conversion via Quantile Matching
Given ranking information, we can convert it to scores with any desired score distribution, via Quantile or Cumulative Density Function (CDF) matching. The desired score distribution can be either an assumed human score prior, for example, normal distribution is simple and suitable guess on prior, or an estimated distribution from validation samples.
To match the CDF of variable Y to variable X, need to find transformation function g such that = g(X). The goal is to make the CDF of equal the CDF of X.
The CDF matching transformation is given by:
Y Fr F= (Fx(X))
(8)
where Fy is the inverse of the CDF Y.
For example, if we assume the score prior distribution is [10%, 20%, 40%, 20%, 10%] for scores 1 to 5, then we can simply convert ranking into scores by assigning the smallest 10% with score 1, the smallest 10% 30% with score and so on.
D Datasets
Summarization Tasks: We select SummEval (Fabbri et al., 2021) and NewsRoom (Grusky et al., 2018) for experiments. SymmEval consists of 100 input source texts, each paired
17

---

Published as a conference paper at COLM 2024
with 16 summary candidates generated by different LMs. The dataset is annotated on four aspects: coherency (CH), fluency (ELU), consistency (CON), and relevancy (RE). NewsRoom includes 60 input source texts and output summaries for each sample. It is annotated the aspects of coherence (CH), fluency (FLU), relevance (RE), and informativeness (IN).
Story Generation Task: We select HANNA (Chhun al., 2022), which contains 1056 creative story writings generated from 96 prompts collected from WritingPrompt. Three representative aspects are selected: coherence (CH), surprise (SU), and complexity (CX). We concatenate the story prompts with the story bodies, allowing the stories to be compared without input prompts. Due to the computational cost, we use the scaling variant of PAIRS with an anchor size of 100.
E Implementation Details
E.1 Evaluation Strategies
Following previous works (Liu et al., 2023a; Zhong et al., 2022; Fu et al., 2023) used two evaluation strategies, Sample level and Dataset level. For the sample level, the correlation values are calculated across the multiple candidates to the same input and then averaged across all input samples. For the dataset level, the correlation calculated across the entire dataset.
F
Additional Experiments
F1
Scaling Variant Performance Loss
Mistral-7B
Llama2-7B
Methods
# Q
# Q
greedy
30.4
9,762
30.4
9,065
beam
32.0
24,733
31.1
24,006
beam-scaled
29.3
7,571
29.5
7,438
Table 4: Performance trade-off of the scaling variant. We compare the Spearman correlations (p) and numbers of the model query (# Q) for PAIRS-greedy, PATRS-beam and PAIRS-beam- scaled on the Coherence aspect the HANNA dataset. The beam size is set to 100, the anchor size set 100, and the uncertainty threshold set 0.67.
Because of the binary search in the second stage of the scaling variant, expect there will be performance loss compared to PAIRS-greedy and PAIRS-beam. To investigate the actual trade-off, conduct an ablation study using Mistral-7B and Llama2-7B, The results presented in Table 4 illustrate the trade-off between performance and computational cost. PAIRS-beam-scales achieves comparable performance to PAIRS-greedy, while requiring fewer model queries. We note that the computational complexity of PAIRS-greedy grows at rate of O(N log N), whereas PAIRS-beam-scaled will grow at rate of O 100), making it more computationally efficient as the problem size increases.
F2 Uncertainty Threshold for Different LLMs
Different base models have different logit distributions. Mistral 7B and Llama2 7B tend to have symmetric and bell-shaped distributions, which means a large portion of the probabilities lie within the range of [0.3, 0.7], shown in Figure 4. However, shown in Figure 7, the probability distribution different for GPT-3.5-turbo. Therefore, their optimal uncertainty thresholds would be different.
18

---

Published as a conference paper at COLM 2024
81
B2
B3 B4
B5
B6
87
3.0
Coherence Surprise Complexity
A1
A2
divergence
2.0
A3
0.6
1.5
A4
1.0
-04
0.5
Sub-array
Initial data size
Figure 7: Illustration of PAIRS with a beam size of Heatmap values show P(A; Bj) by GPT-3.5-turbe The rest of the settings are the same as Figure 4
Figure 8: Empirical evidence for anchor set size selection. The error bars show the mean and standard deviation of the KL divergence between the score distributions of randomly sampled anchor sets (100 repetitions) and the full set. The results suggest that an- chor set size of 100 yields relatively small error.
F3 Theoretical and Empirical Evidence for Choice of Anchor Set Size
The Likert scale score rating can be treated as a five-category classification problem. There- fore the problem of anchor size estimation becomes estimating the sample size for estimating a categorical distribution. This can be calculated by the method of 'proportion sample size calculation' The formula for the sample size calculation for single proportion is:
(22 x (2-xpx(1-))
e2
where n is the sample size, z is the z-score corresponding to the desired confidence level, is the expected proportion of the most common category, and e the desired margin of error. To account for multiple categories, the sample size is multiplied by design effect ff which typically ranges from 1.5 to 2.
Radj = n x deff.
where Madj is the adjusted sample size for multiple categories, is the sample size calculated for single proportion, and deff is the design effect. The design effect accounts for the increased variability in the data due to the multiple categories and the potential clustering of responses within categories.
If we take the desired confidence level as 0.8, the expected proportion of the most common category p as 0.4, error margin as 7% and deff as 1.5, then the sample size required for estimating five categories distribution is around 120.
Figure 8 presents empirical evidence for selecting an appropriate anchor set size. To determine the optimal size, we conducted an experiment where we randomly sampled different numbers of annotations from the HANNA (coherence) dataset and calculated the Kullback-Leibler (KL) divergence between the score distributions of the sampled sets and the entire set. For each anchor set size, we repeated the sampling process 100 times and computed the standard errors. The results suggest that anchor set size between 100 and 200 provides a reasonable balance between computational efficiency and representativeness of the full dataset, the KI. divergence stabilizes within this range.
19

---

Published as a conference paper at COLM 2024
G
Prompt Templates
G.1 Prompts
Prompt Template Example for Pairwise Comparison
Evaluate and compare the coherence of the two following summaries.
Source text: ssource_text_inpure
Summary A: [summary_]]
Summary [summary __]
Question: Which summary
more coherent?
If the summary A is more coherent, please return 'A" If the summary B is more coherent, please return 'B' You must only return the choice. Answer: [output]]
Table 5: A prompt template example for pairwise comparison of the coherence aspect in the summarization task.
Prompt Template Example for Direct Scoring
Evaluate the coherence of the following summary.
Source text: Sbource_text_inppt
Summary: [summary]
Please rate on a scale from to 5, where represents very low coherence and 5 indicates excellent coherence. You must only return an int score. Score: [output]
Table 6: A prompt template example for direct scoring of the coherence aspect in the summarization task.
20


Reviews:
[{"review_id": "pJASYgyT8M", "reviewer": "Reviewer_VjHC", "summary": "- This paper proposes a pairwise-preference search method that estimates MLE of preference rankings by searching through an uncertainty-pruned pairwise comparison space.\n- The proposed method is more effective than traditional direct scoring evaluations and existing pairwise baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback while acknowledging the quality of our work. We hope that in light of the response that addresses all the reviewer’s concerns, the reviewer could consider improving their score.\n\n---\n>PairS performance vs Direct scoring, and the motivation in 3.1.\n\nWhile our main results in Table 1 show that **PairS generally outperforms direct scoring**, we acknowledge that there are a few exceptions. These variations are expected due to challenges in LLMs' capability to handle long-context input and their positional bias.\nThe exceptions occur in the consistency aspect of SummEval, as explained in Section 5.2,  which is caused by the highly concentrated distribution of human scores, and in the performance of Llama2-7B and Mistral-7B on HANNA, which we attribute to the difficulty in understanding long-form creative writing.\n\nTo further support our motivation stated in Section 3.1, we conducted a complementary experiment by randomly sampling 50 summary candidate pairs for each source text in SummEval and calculating the 3-way *agreement rate accuracy* between direct scoring vs. human scores and pairwise comparison vs. human scores. The results show that pairwise comparisons have *consistently* better agreement than scoring.\n|Model|Scoring|Pairwise|\n|-|-|-|\n|Mistral-7B|38.7|45.5|\n|Llama2-7B|39.6|43.0|\n|GPT-3.5| 42.3|48.1|\n\n---\n> Significance test on Table 1 results: \n\nThank you for raising this point. To address this, we conducted z-tests on 30 repetitions for cases where direct scoring and PairS have relatively similar performance. The results show that PairS-greedy significantly outperforms direct scoring for Mistral-7B on SummEval coherence (p<0.01). For GPT-4, the differences are significant on fluency (p<0.001) and weakly significant on consistency (p<0.15). Furthermore, the difference is significant for GPT-3.5 on relevance (p<0.001).\n\n---\n> Beam size, # of model calls and performance:\n\nThe search space of the candidate ranking is determined by both the beam size and the uncertainty threshold. A large beam size allows for more ranking permutations to be considered when the uncertainty threshold is fixed.\nWe provide the following ablation study with an uncertainty threshold of 0.6 and reporting Spearman corr. along with the number of model calls in ():\n\n|Beam Size |Mistral-7B|Llama2-7B|\n|-|-|-|\n|1|22.9 (39.3)|13.5 (34.2)|\n|5|23.5 (73.7)|15.1 (70.5)|\n|20|23.8 (91.1)|16.6 (91.5)|\n|100|25.3 (103.8)|16.8 (94.6)|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The writing is clear to understand.\n- The analysis of the limitations of calibration in aligning LLM evaluators is well presented and the proposed PAIRS method is also well described.\n- Experiments are achieved with open-source and close-source models on multiple datasets.\n- Ablation studies are conducted.", "reasons_to_reject": "- The main LLM evaluation results with the proposed method are not always better than the direct scoring method, which makes me doubt about the motivation in 3.1.\n- Some results in Table 1 are close to the baselines. Are those statistically significant?\n- Experiments can be more comprehensive if different size of open-source models (e.g. 70B size models). Also, it is curious if the larger models perform similarly.", "questions_to_authors": "- Any reason for the beam size of 1000, which seems to be very large number?\n- What if the beam size is smaller for the main experiments? Also, what about the correlation between the inference time and the different beam sizes?", "extra_scores": {}}, {"review_id": "C1wNj4yIiU", "reviewer": "Reviewer_zw92", "summary": "This paper develops a search-based search algorithm called PAIRS, which searches example pairs for LLM pairwise judgment to create a global ranking of examples. The basic idea is to convert the pairwise judgments conducted so far to prune the pairs for which the judgements can be inferred. Experiment results show that PAIRS can produce judgments that are more correlated with human judgments than a scoring-based method G-Eval.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for insightful reviews and the acknowledgement of the novelty and presentation of our work. \n\n---\n> Pairwise Baselines\n\nWe agree that pairwise baselines can better demonstrate our method's efficiency. We propose comparing PairS with two baselines: 1) Win-loss rate and 2) ELO rating, both using randomly sampled pairs without replacement.\nWe present the mean and std of Spearman corr. over 10 repetitions, vs # of comparisons\n|||50|100|240(max)|\n|-|-|-|-|-|\n|Mistral-7B|Win-rate|18.7|22.5|26.8|\n||ELO|22.7|23.6|27.1|\n||PairS-G|23.0(39.2)|36.5(41.5*2)||\n|GPT-3.5|Win-rate|27.5|36.6|40.3|\n||ELO|35.6|38.0|40.2|\n||PairS-G|39.3(44.3)|39.9(46.3*2)||\n\nUncalibrated PairS-greedy achieves the same performance as the baselines using **only 34% and 31% of the comparisons**. Calibrated PairS-greedy, which doubles the number of comparisons, can sometimes outperform baselines that use O(N^2) comparisons.\n\n---\n>Ranking as evaluation is not motivated\n\nWe respectfully disagree. \nRanking offers a much finer granularity compared to score rating, as the number of candidates is typically much larger than the number of score categories. Even in rare cases where the candidate number is smaller, ranking still provides valuable information. As discussed in Sec 3.1, scoring suffers from challenges in aligning evaluation standards across raters.\n\n---\n>PairS may be expensive\n\nWe acknowledge that pairwise evaluations are generally more expensive than pointwise methods. However, we would like to highlight:\n1) Performance should be considered alongside cost. We show that PairS outperforms most pointwise methods, such as G-Eval, which only obtain marginal gains with increased computation.\n2) PairS is highly efficient compared to pairwise baselines.\n\n---\n>The limitation in Section 2\n\nIn Section 2,  we show that calibration cannot effectively bridge the gap between predictive posterior distributions. This serves as empirical evidence that leads to the conclusion that there is yet a gap between the likelihood terms of Eq.1., which motivates us to investigate pairwise evaluation to mitigate the gap in evaluation standards. \n\nIt is important to note that this paper does not focus on calibration, and we are not showing that our method can better eliminate the bias in **model’s prior**. Instead, our experiments are designed to show that we achieve better final evaluation performance, which can be interpreted as a smaller gap between the **posterior terms**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- (S1) Aggregating pairwise LLM judgments into a global ranking is an interesting idea and not well studied. LLM-as-a-Judge is an active research area and the paper could contribute to the research community.\n- (S2) The paper is generally well-written.", "reasons_to_reject": "- (W1) Baseline methods are missing. It is not clear if the proposed solution is reasonably effective for the purpose. More specifically, on (1) search and (2) aggregation. For (1), the most naive way is to use randomly selected pairs with the same rank aggregation method. For (2), other aggregation methods such as the Elo score, which is commonly used for LLM ranking (eg., Chatbor Arena).\n- (W2) The motivation for Evaluation as Ranking is unclear. The main purpose of evaluation in most cases is to compare two (or multiple systems) to judge which one is better (best). In that sense, the global ranking discussed in the paper may not be the best option for evaluation.For example, the SummEval dataset used to 1-5 ratings for manual evaluation and the granularity is much more fine-grained by the proposed method.\n- (W3) The cost of the proposed method is expensive. The computational complexity for scoring-based methods such as G-Eval is O(N). It is not clear if it’s worth paying extra cost. One option is to conduct experiments under the same budget (ie., just conducting N pairs to be fair against scoring-based methods) and discuss if the pairwise judgment approach can be more effective.\n- (W4) Section 2 is misleading, as the limitations discussed in the section are not  addressed by aggregating pairwise judgments. If it is, the paper needs to show that the rating distribution is better aligned with that of human judgments (which is not straightforward, as the judgment is global ranking)", "questions_to_authors": "Please respond to the concerns raised in Reasons to Reject.", "extra_scores": {}}, {"review_id": "oooUfRfo3K", "reviewer": "Reviewer_VVGt", "summary": "This paper presents an LLM evaluation framework with pairwise comparisons (Qin et al. 2023). Pairwise comparisons require $O(N^2)$ evaluations where $N$ is the number of outputs from LLMs. So, this paper explores the use of transitivity of preferences to reduce the number of evaluations into $O(N \\log N)$ and presents a sorting algorithm for LLM outputs, inspired by the merge sort algorithm. This study also proposes using beam search for merge operations because transitivity may not always hold. The experimental results demonstrate that the proposed method has higher correlations with human annotations and performs better than direct scoring and domain-optimized baselines.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and for recognizing the well structured presentation, novelty and superiority of this work to the research community. We appreciate the suggestions for listing the number of pairwise comparisons in our work to further enhance the presentation. Following this additional experiment, we hope our response could sufficiently address the reviewer’s concerns.\n\n---\n> A better title of \"aligning with human judgment.\"\n\nThank you for your suggestions. We are aware that the current title may overlap with the line of work in fine-tuning LLMs or calibration for a better alignment with human values. Our original intent was to highlight PairS' ability to leverage a smaller gap in evaluation standards between humans and LLMs. We will endeavour to figure out a more appropriate title for future versions. \n\n---\n> Number of pairwise comparisons\n\nThank you for the suggestion to report the actual number of model calls. We agree the actual number of model calls can be more helpful than the computational complexity. We conducted an additional ablation study on beam size selection, keeping the uncertainty threshold at 0.6. We will include the Spearman correlation results along with the number of model calls in ():\n\n|Beam Size |Mistral-7B|Llama2-7B|\n|-|-|-|\n|1|22.9 (39.3)|13.5 (34.2)|\n|5|23.5 (73.7)|15.1 (70.5)|\n|20|23.8 (91.1)|16.6 (91.5)|\n|100|25.3 (103.8)|16.8 (94.6)|\n\nWe will include this ablation study with a detailed figure in the later version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ This paper is well-structured and written.\n+ This paper proposes a reasonable algorithm for making pairwise approaches feasible for LLM comparisons.\n+ The experiments demonstrate the superiority of the proposed approach, which has a great impact on LLM evaluation and development.", "reasons_to_reject": "None", "questions_to_authors": "The title includes \"aligning with human judgment.\" I was expecting to see how the proposed method aligns LLM judgments with humans. However, this study does not directly align human judgment with that of an LLM evaluator. In fact, this paper gave up calibrating LLM evaluators directly for alignment, but obtained a better alignment as a result of utilizing pairwise comparisons. So I felt that this phrase in the title is a bit misleading. Is it right that evaluator LLMs were not trained to align human pairwise preferences?\n\nWhere can I find the numbers of pairwise comparisons of the PAIRS-greedy, PAIRS-beam, and Qin et al. (2023)?", "extra_scores": {}}, {"review_id": "SBiFvwYBba", "reviewer": "Reviewer_6K4e", "summary": "This paper develops a method for automatically ranking a set of text responses (e.g., summaries of source texts) using LMs, while training to maintain alignment with human rankings.\n\nProblem background: Simply prompting an LM to score texts according to criteria can result in significant misalignment with human judgements. Estimating the marginal human score distribution p_H(s) and using that to calibrate the LM scores does not fully resolve the issue, implying that the model's likelihood p_M(y|s) is also misaligned from the human likelihood.\n\nMethod: the proposed method (PAIRS) instead focuses on using pairwise comparisons between candidate texts to produce a global ranking. This is formulated as searching over an ordering (permutation) that maximizes the likelihood; the definition in Eq 4 is designed to considers two algorithms for finding this ordering. The first is a merge-sort-like algorithm that merges two sorted lists of responses by greedily choosing the next candidate from either list that locally maximizes the likelihood. They then introduce a beam-search-like alternative of this procedure that tracks the top-k beams of candidate rankings at any time.\n\nThe experiments show that PAIRS achieves strong correlation with gold annotations on several natural language generation benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback while acknowledging the novelty, effectiveness and simplicity of our method. We appreciate your valuable suggestions in further discussing the real-world cost. With the additional results with respect to beam size and model calls, we hope that our response sufficiently addresses the reviewer’s concerns and the reviewer could consider improving their score.\n\n---\n> Presentation in Section 3.2\n\nWe sincerely appreciate the time and effort you have invested in carefully examining our algorithm and the corresponding demo presented in Algo.1, Fig.3 and 4. We acknowledge that our method is intricate and our presentation may not have been optimally structured to facilitate a clear understanding. \nWe will expand the text and captions to improve the clarity of our work.\n\n---\n> Beam size, number of model calls and performance\n\nWe agree that detailed information on the number of model calls can help readers better understand the complexity of our methods and baselines.\nNumber of model calls is affected by many factors, including beam size, uncertainty threshold, number of candidates to rank and the distribution of model’s output logits.\nTherefore, we conducted an additional ablation study on beam size selection, keeping the uncertainty threshold at 0.6. We include the Spearman correlation results along with the number of model calls in ():\n\n|Beam Size |Mistral-7B|Llama2-7B|\n|-|-|-|\n|1|22.9 (39.3)|13.5 (34.2)|\n|5|23.5 (73.7)|15.1 (70.5)|\n|20|23.8 (91.1)|16.6 (91.5)|\n|100|25.3 (103.8)|16.8 (94.6)|\n\nAs mentioned in Section 3.4, direct scoring methods generally have complexity of O(N), while existing pairwise baselines are generally O(N^2). PairS provides an O(NlogN) solution and a trade-off between performance and complexity between O(NlogN) and O(N^2). \n\nIn conclusion, PairS is more efficient than pairwise baselines and utilizes computation more effectively than some score-based evaluation methods. For instance, G-Eval uses a weighted average of 20 inference repetitions, resulting in a total computation of 20N. In comparison, PairS-greedy requires Nlog2(N) in the worst case. Specifically, for the 16-candidate SummEval example mentioned above, the factor is only 2.5N.  \n\n---\n> Asymptotic complexity of PairS scaling variant:\n\nIn the second stage of PairS-beam-scaled, the complexity is N*log2(Anchor size). With an anchor size of 200, the complexity is 7.6N, which is comparable to some direct scoring method such as G-Eval.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is novel to my knowledge, seems relatively straightforward to implement, and the experiments appear to demonstrate strong empirical results.", "reasons_to_reject": "* The text in 3.2 is not very clear at explaining how PairS-beam works, and the figure/algorithm need to be carefully studied to infer how it works without much supporting text. The caption or text could be expanded to make this easier to understand.\n* The results could benefit from more clarity about the actual real-world cost (in terms of # of model calls) of each method in the experiments. Do the proposed methods use more model calls than direct scoring?", "questions_to_authors": "* What is the beam size used for PairS-beam(-scaled) in Figure 5? Does that matter at all for the number of model calls?\n* If I understand correctly, the scaling variant of Pairs is linear in number of samples to be ranked, is that asymptotically the same as direct scoring?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a new technique for using LLMs as evaluators, by formulating the evaluation problem as a ranking problem, and using LLMs to conduct pairwise comparisons to create a global ranking. LLM-based evaluation is a very popular technique being used in industry today and this work addressing how to improve it is very timely. Reviewers were positive about the clarity of writing, motivation, comprehensive experiments on open and closed source models (though larger models can be added), ablations and strong empirical results.

Reviewers mainly had minor clarification comments and some suggestions on presentation. The title of the paper may be slightly misleading because it mentions human judgement, while there is no direct comparison with human annotators in the paper, so the authors may want to revise it.

[comments from the PCs] As much as possible, please follow the AC request regarding the title to avoid confusing readers.
Decision: Accept