RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Reference-based Evaluation
In NLG evaluation, a standard approach is the comparison of annotator-provided gold-standard references with the generated response. Established heuristics, such as the N-gram overlap metrics ROUGE Lin ( 2004 ) and METEOR Banerjee and Lavie ( 2005 ) , have extensively been applied for assessing summarization and machine translation respectively. Recently, the paradigm has evolved to incorporate embedding-based methods like BERTScore Zhang et al. ( 2019 ) , which not only compares generated texts with references, but also factors in semantic considerations beyond word overlap.
Tailored NLG Evaluation Approaches
Tailored approaches have been proposed for assessing specific properties of generated texts. For example, question-answering systems are used for summary consistency assessment Wang et al. ( 2020 ); Scialom et al. ( 2021 ) to probe information consistency. For Dialogue quality assessment, the language model probability from a DiaoloGPT system is used as a proxy for response quality Mehri and Eskenazi ( 2020b ) . A survey for NLG evaluation methods was conducted by Celikyilmaz et al. ( 2020 ) .
Zero-shot LLM Evaluation
Given the current capabilities of LLMs such as ChatGPT and GPT4, the zero-shot ability of these systems for a wide range of tasks, including NLG evaluation, has been investigated. Existing works have looked at using LLM to evaluate open-ended story generation and adversarial attacks Chiang and Lee ( 2023 ) and using ChatGPT to score the quality of texts along a certain axis Wang et al. ( 2023 ); Kocmi and Federmann ( 2023 ) , demonstrating that ChatGPT can be used in a zero-shot setting and achieve reasonable performance.
LLM Pairwise Comparisons
Pairwise comparative judgement Thurstone ( 1927 ) has been a popular approach of assessing candidates for exams, however where typically human assessors are used. Investigating the ability and application of pairwise comparisons via LLMs has been relatively underexplored, with concurrent work using pairwise rankings for information text retrieval Qin et al. ( 2023 ) and separately for assessing LLM-based chat assistants on open-ended questions where outputs are compared to that of a baseline system Chiang et al. ( 2023 ); Zheng et al. ( 2023 ) .
Notation
In this work, we investigate using LLM comparative judgements for NLG assessment. Assume that there is a context d (e.g., a text passage or dialogue) and a set of N candidate responses, x_{1:N} . For a given attribute (e.g., coherence, consistency, fluency) the N candidates have true underlying scores, s_{1:N} . As scores often only have relative meaning, in this work only the ranks of the candidates will be evaluated. The objective is therefore to accurately predict the true ranks, r_{1:N} , of the candidate scores. In comparative assessment, one uses pairwise comparisons to determine which of the two input responses is better. Let y_{ij}\in\{0,1\} represent the true outcome of whether x_{i} is higher ranked than x_{j} , such that y_{ij}=\mathbbm{1}(s_{i}>s_{j}) . Here, an LLM is used to model the probability that response i is better than response j , p_{ij} ,

 p_{ij}=P(y_{ij}|x_{i},x_{j},d) (1)

Which can alternatively be converted into hard decisions, \hat{y}_{ij} , by selecting the most likely outcome.

\hat{y}_{ij}=\begin{cases}1,&if $p_{ij>0.5$}\ 0,&otherwise\end{cases} (2)

Let \mathcal{C}=\{c_{k}\}_{k=1...R} represent a set of comparisons, where R is the total number of comparisons, and each comparison c=(i,j) indicates the indices of the two considered candidate responses. For example, the set of all possible comparisons, \mathcal{C}=\{(i,j)\;|\;i,j\in[1...N],i\neq j\} , could be used, or alternatively a smaller subset of comparisons.
Prompt Design
To leverage the emergent ability of LLMs, we use comparative prompts that probe a model to decide which of the two candidates is better. Let T be a prompt template that converts candidate responses x_{i} and x_{j} as well as context d into an output text, prompt \mathcal{P}=T(x_{i},x_{j},d) . This work aims to find a simple, general and robust assessment method, and as such extensive prompt engineering is not in the scope of this work (despite possible performance gains). We evaluate two simple and suitable prompts in our initial investigations. Our prompts for comparative assessment are shown in Figure 2 .
Comparative Decisions
A central aspect of LLM comparative assessment is the methodology of getting comparative decisions. In this section, we consider two approaches for leveraging LLMs for comparative assessment; First for when one has output token-level probabilities (Prompt-Based Classifier), and second for when only the output texts are available.

Prompt-Based Classifier :
If one has access to the output probabilities, an efficient method to get probability estimates of the predictions is to leverage prompt-based classifiers. Let P_{\theta}(w|x) represent an LLM’s conditional language model distribution of the output sequence w given the input text x . For prompt-based classifiers, the LM probabilities of specific label words ( w_{k} ) are used as a proxy for the class decisions Liusie et al. ( 2023 ) . For example in summarization assessment, given a prompt \mathcal{P} ending in ‘… which summary is better’, one can set w_{i} =‘ Summary A ’ and w_{j} =‘ Summary B ’ and define the probability that response i is better than response j as:

p_{ij}=\frac{P_{\theta}(w_{i}|\mathcal{P})}{P_{\theta}(w_{i}|\mathcal{P})+P_{\theta}(w_{j}|\mathcal{P})} (3)

Text Generation :
Alternately, if only limited API access is available, one can sample responses from the conditional LM given the input prompt \mathcal{P} ,

\tilde{w}^{(k)}\sim P_{\theta}(w|\mathcal{P}) (4)

Let f(\tilde{w})\in\{0,1\} be a function that maps the text response to the comparative decision. By generating K samples from the LLM, one can estimate the comparative probability p_{ij} by looking at the fraction of the samples that selects x_{i} over x_{j} .

 p_{ij}=\frac{1}{K}\sum_{k=1}^{K}f(\tilde{w}^{(k)}) (5)
Comparisons to Ranks
Although the full set of possible comparisons yields the most information for the rankings, this requires R\!\!=\!\!N(N\!\!-\!\!1) comparisons, which can be computationally expensive. For computational efficiency, we can consider 3 different comparison selection strategies: random, no-repeat and symmetric. For random , comparisons are randomly selected from the set of all possible comparisons. For no-repeat , if (x_{i},x_{j}) is selected then (x_{j},x_{i}) will not be selected. For symmetric , if (x_{i},x_{j}) is selected, then (x_{j},x_{i}) will also be selected.

Given a set of selected comparisons \mathcal{C} and weights of a comparative assessment system \bm{\theta} , one can generate a predicted rank ordering \hat{r}_{1:N} of the candidate responses. A simple but effective approach is to sort the candidates by win-loss ratio,

\hat{s}_{i}=\frac{\#wins of x_{i}}{\#comparisons involving x_{i}} (6)

which can then be ordered to convert the scores into predicted ranks \hat{r}_{1:N} .
Debiased Comparative Assessment
Let \tilde{y}_{ij} represent the outcome of the comparison when considered in the opposite ordering, such that \tilde{y}_{ij}=1-\hat{y}_{ji} . For a positionally unbiased comparator, reversing the ordering should have no impact on the outcome of the comparison

\tilde{y}_{ij}=\hat{y}_{ij}\qquad\forall\;(i,j)\in[1...N],i eq j (7)

Systems may, however, have systematic positional biases and could for example favor the first position over the second position. To quantify the level of systematic bias, one can determine P(A) , the prior associated with the first position, and P(B) the prior for the second position. This can be estimated for a given set of comparisons by using the statistics over all comparisons, and by calculating the fraction of times that each position is selected.

P(A)=\frac{\sum_{i,j\in\mathcal{C}}\hat{y}_{ij}}{|\mathcal{C}|}\quad P(B)=\frac{\sum_{i,j\in\mathcal{C}}\tilde{y}_{ij}}{|\mathcal{C}|} (8)

When using a symmetric comparative set \mathcal{C} , for an unbiased system, both P(A) and P(B) should be 0.5 and any large deviation is symptomatic of positional bias. To address possible positional bias, one may reweight system probabilities, \hat{p}_{ij} , through

\hat{p}_{ij}=\frac{\alpha\cdot p_{ij}}{\alpha\cdot p_{ij}+(1-p_{ij})} (9)

where \alpha\in\mathbb{R}^{+} is a weight that can be set such that P(A)=P(B)=0.5 . Reweighting in this fashion is equivalent to,

\hat{y}_{ij}=\begin{cases}1,&if $p_{ij>\tau$}\ 0,&otherwise\end{cases} (10)

where \tau\in[0,1] is a decision threshold corresponding to \alpha , set such that P(A)=P(B)=0.5 .
Datasets
To investigate the general applicability of comparative assessment, we cover a range of standard NLG evaluation tasks and datasets as follows:

SummEval Fabbri et al. ( 2021 ) is a summary evaluation benchmark of 100 passages, each with 16 machine-generated summaries. Each summary is evaluated for coherency ( COH ), consistency ( CON ), fluency ( FLU ), and relevancy ( REL ).

Podcast Manakul and Gales ( 2022 ) is for benchmarking podcast summary assessment methods. It contains 179 podcasts each with 15 abstractive summaries. Each summary was evaluated for its overall quality on a 4-point scale.

TopicalChat with the USR annotations Mehri and Eskenazi ( 2020b ) is for benchmarking dialogue evaluation. It includes 60 dialogue contexts and six system responses per context. These responses were assessed on coherency ( COH ), continuity ( CNT ), engagingness ( ENG ), and naturalness ( NAT ).

WebNLG Gardent et al. ( 2017 ) is for benchmarking data-to-text evaluation methods. It contains 223 semantic triple groups, each paired with outputs from 8 triple-to-text generation systems. These texts were evaluated for fluency ( FLU ), grammar ( GRA ) and semantic equivalence ( SEM ).
Base Large Language Models (LLMs)
We investigate two families of open-source instruction-tuned LLMs. The first system is FlanT5 Chung et al. ( 2022 ) , T5 Raffel et al. ( 2020 ) that have been instruction tuned on a diverse set of 1000 NLP tasks Wang et al. ( 2022 ) . The second system is Llama2-chat Touvron et al. ( 2023 ) , which is Llama2 tuned on instruction datasets. We investigate a range of model sizes; 220M, 770M, 3B and 11B for FlanT5, and 3B and 13B for Llama2.
Baselines
The NLG evaluation methods can be categorized into reference-based and reference-free . Reference-based methods compare the output against the reference such as n-gram metrics (e.g., BLEU Papineni et al. ( 2002 ) and ROUGE Lin ( 2004 ) ), or embedding based metrics (e.g., BERTScore Zhang et al. ( 2019 ) ). In contrast, reference-free methods compare the generated texts against the original source (or context for generation) directly.
Bespoke Methods
Bespoke methods require a specific data which could be supervised labels (e.g., human judgements for the summaries) or data for model training (e.g., question-answering). Although bespoke methods could work in a similar domain (e.g., developed for summarization, but applied on dialogue generation), they are not as general as zero-shot methods.

UniEval Zhong et al. ( 2022 ) convert NLG evaluation into Boolean QA problem. This method uses pre-defined schemes for selected aspects (e.g., coherence) and generates synthetic data to fine-tune a T5 system for NLG assessment. References are used for particular aspects (e.g. relevancy), and schemes/systems are bespoke for a particular attribute (though a sequentially trained system that scores multiple attributes is also explored).

QuestEval Scialom et al. ( 2021 ) and MQAG Manakul et al. ( 2023a ) are QA-based approaches for assessing consistency in summarization tasks. QuestEval uses extracted answer spans while MQAG represents information using multiple-choice questions. Both methods are reference-free.

Longformer-SFT : For podcast summarization, we follow Manakul and Gales ( 2022 ) in using a Supervised Fine-Tuned longformer Beltagy et al. ( 2020 ) as a baseline. The input is the document and the summary, and human judgement is used as the supervised target label at training, and the performance is reported using 5-fold cross-validation.
Zero-shot Methods
Zero-shot methods can be applied generally to any task without further training or fine-tuning. Comparative assessment is a zero-shot method.

GPTScore Fu et al. ( 2023 ) evaluates texts using conditional language model scores. By conditioning the language model on instruction and context, GPTScore assumes that it will assign a higher probability to a high-quality generated text.

Prompt Scoring .

Another baseline is prompt-scoring. With this approach, for a particular attribute, the LLMs is asked to assess the response quality between 1-10. Simple prompts are used with the general templates shown in Figure 3 . Prompt-scoring is run for all open-source LLMs considered (FlanT5 and Llama2), and is used as the main baseline to compare comparative assessment against. During generation, the maximum generation length is set to 5 and the temperature is set to 1.0. Similarly, ChatGPT prompt-scoring has recently been proposed in Wang et al. ( 2023 ); Kocmi and Federmann ( 2023 ) , which we also include as a baseline where applicable.

G-Eval Liu et al. ( 2023 ) As an extension to prompt-scoring, G-Eval extends standard prompt scoring by using detailed prompts and then generating a continuous score by calculating the expected score over a score range (e.g. 1-5 normalized by their probabilities). We apply G-Eval to the various base LLMs and contrast performance to the other approaches for SummEval, since the prompts for different attributes have been made publically available. 1 1 1 https://github.com/nlpyang/geval
Methodology
Each LLM is used for both prompt-scoring and comparative assessment. For the main comparative assessment results, we consider the full set of possible comparisons, where all pairs of candidates in both permutations are compared by the framework. Comparisons are made using the prompt-based classifier (as described in § 3.3 ) using the prompt templates shown in Fig. 2 , where the system outputs a probability for Response A and Response B. The winner of the comparison is the response with the highest probability, where candidates are then ranked in order of the win-ratio (as described in § 3.4 ). For Llama2, comparative prompts are appended with ‘ Answer: ’ while scoring prompts end with ‘ Score: ’. The spearman correlation between predicted scores and human judgements is used as the performance metric.
Positional Bias
We investigate whether the comparative prompts have any implicit positional bias, and whether systems prefer the first/second position. Table 5 shows the fraction of comparisons that selected the candidate in the first position for SummEval. Since all comparisons in both permutations are considered, this fraction should be 0.50 for an unbiased system. However, we observe considerably high bias, with some set-ups even selecting the first option 80% of the time. Further, we observe that larger systems appear to be more susceptible to bias than smaller systems, which may explain the similarity in performance for the 3B and 11B/13B systems in the previous main results. Similar results for other datasets are provided in Appendix A.2
Debiasing
The previous section demonstrates that comparative assessment exhibits positional bias which may impact system decisions. We therefore investigate whether debiasing can improve evaluation performance. Table 6 shows standard and debiased LLM comparative assessment performance for the considered tasks and scores, with WebNLG SEM and Podcast omitted due to the required emergent ability and large context length respectively. We observe that debiasing can lead to performance boosts, where we note that the prompts which have a high bias (seen in Table 5 and Table 9 in the appendix) benefit most from debiasing. In particular, for TopicalChat we observe large gains for the FlanT5-11B system, which enables state-of-the-art performance. To explain why debiasing can lead to large performance boosts, consider a very biased system where the first response is always selected as better. Although over both permutations the system is unbiased for any comparison, the bias in the system will cause the system to assume that all candidates are of the same quality. By reducing the bias of each comparison, the system may be able to pick up subtler quality differences between the samples.
Comparative Accuracy
One can also measure the accuracy of the comparative system at a comparison level. Table 7 shows the pairwise comparison accuracy for Summeval, over all candidate pairs where the true score of the candidate response varies. We observe accuracies between 60-80% across all tasks and observe that debiasing can substantially increase accuracy. This highlights that LLMs are able to compare the quality of responses fairly well, though the moderately sized LLMs may not always select the best response (with respect to labels).
Self-Consistency
SummEval has 16 summaries per context which leads to 240 possible comparisons. If one were to instead randomly sample N outputs and consider all N\!\cdot\!(N\!-\!1) comparisons, how consistent would the rankings with the subset of systems be with respect to the final predicted rankings? Table 8 illustrates the self-consistency measured by the accuracy when comparing pairs, and demonstrates that even when using few outputs, the model is very consistent to the final rankings that would be achieved by using many more examples.
Subset of Comparisons
Due to O(N^{2}) number of comparisons required for the full comparison matrix, it might be practical to only consider a subset of comparisons. Figure 4 shows the downstream Spearman correlation for SummEval coherency, when averaged over 50 runs, for different comparison selection strategies. Of the three schemes, we observe that for small R (i.e. less than half the total number of comparisons) selecting comparisons with no repeats leads to a marginal improvement over random selection. Further, by using the symmetric selection scheme, despite the number of comparisons being half that of no-repeat (although each comparison is done twice, once in each permutation), interestingly there is only a performance difference of 1 in terms of Spearman. Finally, we observe that debiasing can be very effective in efficient set-ups, and leads to larger benefits when the number of comparisons is small. Equivalent plots for other tasks/scores can be found in Appendix A.1 .
Conclusions
This paper investigates LLM comparative assessment, a simple zero-shot approach to NLG evaluation. We demonstrate that for moderately sized LLMs, comparative assessment outperforms absolute scoring, and is an effective automatic assessment, achieving near state-of-the-art performance for a range of NLG evaluation tasks. Furthermore, we show that LLMs are prone to have positional bias that could impact their decisions, however, we introduce a simple debiasing approach that leads to performance boosts, especially for biased systems.
Limitations
Computational Cost . The comparative assessment framework with the full set of comparisons uses N\cdot(N-1) comparisons, which for large N can be computationally prohibitive. This paper investigated datasets with at most 16 candidates, and may not scale when more candidates are required.

Base LLMs . The empirical findings are for LLMs of up to 13B parameters. By using larger models (with 100B+ parameters) one may expect further performance improvements. However, due to API costs and the O( N^{2} ) number of comparisons, results are limited to open-source LLMs.

Selection of the subset of comparisons . For our comparison selection scheme, this work only considered static selection schemes. Future work may investigate dynamic selection schemes, either by considering sorting algorithms or ELO competition schemes, and methods similar to those studied in information retrieval by Qin et al. ( 2023 ) .
Ethics Statement
For some tasks/datasets, comparative assessment could be ineffective and have poor generalisation over the task. Deploying machine learning classifiers in real-world classification settings has many associated risks, and careful analysis should be made before deploying such systems. Misuse/overconfidence in the approach may lead to mistrust of users towards LLM solutions.
Acknowledgements
This paper reports on research supported by Cambridge University Press & Assessment (CUP&A), a department of The Chancellor, Masters, and Scholars of the University of Cambridge. This research is further supported by the Cambridge International & St John’s College scholarship.
B.1 Initial Results
Table 11 presents initial results for FlanT5-3B on Summeval, comparing the maximum likelihood ranking to the win ratio approach. The initial finding was that performance was similar between the two conversion schemes. However, it’s worth noting that minimizing the objective function poses intractability challenges, necessitating an approximate greedy search. For the sake of simplicity, our main paper focused on the win-ratio method, while future research may explore more advanced conversion strategies.

Pembroke’s going up
Figure 1: Prompt Scoring v.s. Comparative Assessment. Comparative Assessment prompts an LLM to compare candidates in a pairwise manner, and the comparisons are subsequently converted into scores or ranks.; Prompt Scoring v.s. Comparative Assessment. Comparative Assessment prompts an LLM to compare candidates in a pairwise manner, and the comparisons are subsequently converted into scores or ranks.
Figure 2: Comparative prompt template 1 and 2. When assessing different attributes, only the attribute is changed (e.g., consistent \rightarrow engaging) and for response assessment, the word ‘summary’ is replaced with ‘response’.; Comparative prompt template 1 and 2. When assessing different attributes, only the attribute is changed (e.g., consistent \rightarrow engaging) and for response assessment, the word ‘summary’ is replaced with ‘response’.
Figure 3: Scoring template 1 and template 2. Only the attribute is changed (e.g., consistent \rightarrow engaging) and response description (‘summary’ \rightarrow ‘response’) for different tasks.; Scoring template 1 and template 2. Only the attribute is changed (e.g., consistent \rightarrow engaging) and response description (‘summary’ \rightarrow ‘response’) for different tasks.
Figure 4: FlanT5-3B performance for SummEval COH when a subset of the comparisons are selected by either random, no-repeat or symmetric (as described in § 3.4 ). For no-repeat, each pair is compared once, hence has a smaller maximum R .; FlanT5-3B performance for SummEval COH when a subset of the comparisons are selected by either random, no-repeat or symmetric (as described in § 3.4 ). For no-repeat, each pair is compared once, hence has a smaller maximum R .
(a) FlanT5-3B, SummEval, CON; (b) FlanT5-3B, SummEval, FLU; (c) FlanT5-3B, SummEval, REL; (d) FlanT5-11B, TopicalChat, COH; (e) FlanT5-11B, TopicalChat, ENG; (f) FlanT5-11B, TopicalChat, NAT; (g) FlanT5-11B, SummEval, REL; (h) Llama-chat-7B, SummEval, CON; (i) Llama-chat-13B, SummEval, FLU; Assessment Performance when only a subset of comparisons are considered (extending the results of Figure 4 ). Multiple different base LLMs, datasets and scores and displayed.
Spearman correlation coefficient for SummEval , averaged over both prompts per system (for prompt-scoring and comparative). † ChatGPT performance is quoted from Wang et al. ( 2023 ) , which use more detailed scoring prompts.
Spearman correlation coefficient for Podcast .
Spearman correlation coefficient for TopicalChat . † ChatGPT is prompted using our prompt-scoring prompts.
Spearman correlation coefficient for WebNLG . ∗ Quoted from the NLI method with the backoff template in Dušek and Kasner ( 2020 ) .
Positional bias P(A) for both prompt templates, for various systems in the comparative setup on SummEval.
Spearman correlation coefficient on different aspects of the NLG evaluation tasks, averaged over all prompts considered, using all pairs and ordering considered (i.e. full matrix comparisons).
Accuracy of the comparative systems, at a comparison level, for SummEval.