Introduction
With the current rapid advances in generative AI, pre-trained models are increasingly utilized in a range of NLP tasks, necessitating reliable evaluations of these models. Human evaluation, where annotators critically assess the quality of the outputs of natural language generation (NLG) systems, has been the gold standard approach Lita et al. ( 2005 ); Belz and Reiter ( 2006 ); Lai and Tetreault ( 2018 ); Fabbri et al. ( 2021 ) . However, human evaluation has its drawbacks, and is notably labor-intensive, time-consuming, and costly. As such, automating the evaluation process and assessing NLG systems without human intervention is highly desirable.

Though there has been considerable progress in automatic evaluation methods, many proposed approaches have certain restrictions that limit their effectiveness. A large body of existing work use evaluation methods designed for particular tasks and attributes Mehri and Eskenazi ( 2020a ); Rei et al. ( 2020 ); Manakul et al. ( 2023b ) , for example, measuring the consistency of summaries Wang et al. ( 2020 ); Manakul et al. ( 2023a ) . Though effective within their domain, these approaches are not extensible to different NLG aspects and cannot be used by practitioners wishing to evaluate systems on inputs or properties that are less common.

The recent development in the emergent abilities of LLMs Wei et al. ( 2022 ) has enabled LLMs to achieve impressive zero-shot performance for a slew of language tasks. This has led to general prompt-based assessment approaches, such as prompt-scoring where an LLM is probed to score outputs on a particular aspect Wang et al. ( 2023 ); Kocmi and Federmann ( 2023 ) . These approaches are often only effective with massive LLMs with 175B+ parameters, which may limit the applicability of the approach, especially when access is limited to API access.

With the insight that for humans, it is often easier to select which of two options is better than it is to score options independently, we question whether pairwise comparisons may be more effective at leveraging the impressive emergent ability of LLMs. In this work, we consider LLM comparative assessment, where an LLM is prompted to compare pairs of NLG candidates and predict which one is better. We demonstrate empirically that comparative assessment performs much better than prompt-scoring for FlanT5 and Llama style models, and enables moderate-sized open-source LLMs to achieve near (or above) state-of-the-art performance across a range of NLG language tasks, for a diverse set of attributes. Our approach is general and can be applied to a diverse range of tasks and textual attributes, is simple and requires minimal prompt engineering. Further, we demonstrate that pairwise LLM comparisons often exhibit strong positional biases, where the ordering of candidates impacts the decisions. We introduce a simple debiasing method and empirically illustrate that debiasing can provide further performance improvements, especially when large biases are present.

Our contributions are 1) We are the first work that comprehensively analyzes pairwise comparative assessment for NLG evaluation; 2) We demonstrate that comparative assessment is far more effective than prompt-scoring for moderately-sized LLMs, and yields performance that is state-of-the-art for particular attributes; 3) We demonstrate that positional bias impacts comparative decisions, and introduce a method to debias LLMs which leads to performance boosts, especially when only a subset of comparisons are considered.