PairEval: Open-domain Dialogue Evaluation Metric with Pairwise Comparisons

Authors: chaehun  park
minseok  choi
dohyun  lee
jaegul  choo
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 07:22:02

Abstract: Building a reliable and automated evaluation metric is a necessary but challenging problem for open-domain dialogue systems. 
Recent studies proposed evaluation metrics that assess generated responses by considering their relevance to previous dialogue histories. 
Although effective, these metrics evaluate individual responses directly rather than considering their relative quality compared to other responses.  To handle this, we propose  PairEval, a novel dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations.  Our metric is built on top of open-sourced and moderate-size language models, and we make them specialized in pairwise comparison between dialogue responses. Extensive experiments on multiple benchmarks demonstrate that our metric exhibits a higher correlation with human judgments than baseline metrics. We also find that the proposed comparative metric is more robust in detecting common failures from open-domain dialogue systems, including repetition and speaker insensitivity. The codes and models will be publicly available after the paper is accepted.

Main paper: 
 introduction: 
 Open-domain dialogue systems aim to interact with users by generating natural and engaging responses for a given dialogue history. In this field, building an accurate automatic evaluation metric to judge the quality of such generative systems is an important but challenging task (Liu et al., 2016) . The difficulties partly originate from the one-to-many nature of daily conversations, where a single conversation can be continued with many different follow-up utterances. This variety in dialogues makes traditional reference-based metrics, which compare generated responses to a limited set of known answers (i.e., references), do not correlate well with human judgments (Liu et al., 2016) . The lack of reliable evaluation metrics impedes real-world deployment of open-domain dialogue systems.Researchers have proposed various evaluation metrics to achieve reliable assessments of dialogue systems. Initial studies enhance reference-based metrics by adopting distributed and contextualized representations by neural networks (Liu et al., 2016; Zhang et al., 2019) . Subsequent studies introduce reference-free metrics that employ prediction models to assess responses based on their relevance to the previous dialogue context (Tao et al., 2018; Lowe et al., 2017; Mehri & Eskenazi, 2020a; Zhang et al., 2021; Zhong et al., 2022) , showing a higher correlation with human judgments. Moreover, prompting-based metrics have been presented to properly transfer the instruction-following and zero-shot capabilities of large language models (LLMs) for dialogue evaluation (Zhang et al., 2023; Lin & Chen, 2023) .We hold that the dialogue evaluation should aim to assign differentiated scores to responses by considering their relative quality. In other words, evaluation metrics should make calibrated scores such that they are appropriately aligned with human evaluations. Numerous correlation metrics used in the meta-evaluation of evaluation metrics like Spearman (Zar, 2005) or Kendall rank correlations (Kendall, 1955) reflect this intuition. Therefore, we argue that assessing the target responses by considering their relative appropriateness with other ones is a meaningful process to make more reliable evaluation metrics. In this regard, several studies evaluate responses by considering their relative quality (Sato et al., 2020; Liusie et al., 2024) . However, these approaches usually require exhaustive comparison operations or human-crafted candidate responses, which may not always be available.In this paper, we propose PAIREVAL, a novel open-domain dialogue evaluation metric with comparative assessments. Our metric assesses the individual responses by comparing their quality against only a limited number of comparison responses. Instead of relying on a commercial or proprietary LLM, our metric is built on top of a moderate-size and open-sourced LLM (Touvron et al., 2023) . To elicit the comparative ability of LMs, we devise a simple but effective learning strategy with a public dialogue corpus. Experiments on multiple benchmarks show that PAIREVAL outperforms previous evaluation metrics, and sometimes even shows higher performance than metrics with a powerful proprietary LLM. Further analysis demonstrates that the pairwise evaluation approach is robust and effective in capturing common failures (e.g., repetitive outcomes) in dialogue systems. 
related work: 
 Evaluation Metrics for Open-domain Dialogue Systems Traditional metrics like BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004 ) that measure the N-gram overlap between generated responses and answers show a low correlation with human judgments (Liu et al., 2016) . Several studies use embedding models to consider the semantic similarity between candidate responses and answers (Liu et al., 2016; Zhang et al., 2019) . However, these reference-based metrics rely on a set of known answers for similarity comparison, making it hard to consider the wide semantic space of follow-up utterances for a single conversation. To tackle this problem, recent studies proposed numerous reference-free metrics that directly predict the relevance of a generated response to the given dialogue history (Tao et al., 2018; Mehri & Eskenazi, 2020a; b; Zhong et al., 2022; De Bruyn et al., 2022) . Specifically, neural classification or regression models are usually trained to distinguish relevant responses from irrelevant ones. These predictive metrics have shown meaningful progress along with pre-trained language models (Ghazarian et al., 2019; Mehri & Eskenazi, 2020a; Zhong et al., 2022) , data augmentation strategies (Gupta et al., 2021; Park et al., 2021) , and advanced training algorithms (Huang et al., 2020; Zhang et al., 2021) . In contrast, our work focuses on a comparative evaluation to consider the relative quality between responses for a more reliable evaluation. In open-ended text generation tasks, Pillutla et al. (2021) propose a corpus-level evaluation metric, namely MAUVE, which compares model-generated text distributions with human-written ones using divergence frontiers. To the best of our knowledge, Liusie et al. (2024) propose a pioneering step that introduces a comparisonbased evaluation approach for various natural language generation tasks, including an open-domain dialogue generation task. However, they find that exhaustive comparison between candidate responses is needed for reliable quality estimation of dialogues, resulting in an undesirable computational overhead. Our work overcomes this challenge by specializing in moderate-size LLMs for pairwise comparison between responses. Sato et al. (2020) consider the relative rank of generated responses against other false candidates, and a response selection model is employed for this purpose. Though promising, their evaluation approach requires human evaluation to filter out correct responses from a candidate pool. In contrast, we show that even a limited number of comparisons are enough to ensure a reasonable correlation with human judgment.The pairwise comparison approach has been widely explored from various perspectives, including preference learning (F ürnkranz & H üllermeier, 2003) , recommendation (Beutel et al., 2019) , reinforcement learning (Xu et al., 2020) , and retrieval systems (Qin et al., 2023) . Recent studies that consider the human-aligned behaviors of LMs proposed to reflect human preference in the form of comparisons over multiple model generations (Ouyang et al., 2022; Bai et al., 2022 ). An LLM-based pairwise evaluation has also been increasingly adopted to build system-level ranking information (Boubdir et al., 2023; Zheng et al., 2024) . This work evaluates individual responses by comparing their relative quality against the limited number of responses.  
method: 
 We propose PAIREVAL, a new reference-free dialogue evaluation metric based on a pairwise comparison. Our metric assesses responses by comparing their quality against a small number of comparison examples. The comparison examples are derived from humanwritten conversations in a dialogue corpus. Fig. 1 depicts the overall pipeline of PAIREVAL.In this work, we focus on the turn-level and reference-free evaluation of open-domain dialogue systems. Given a dialogue history h that consists of multiple utterances between two speakers, a dialogue system outputs a single utterance r as a follow-up response. The evaluation metric M considers how suitable the generated response is as the next utterance for the given dialogue history and makes an evaluation score s = M(h, r). The performance of the metric is usually measured by computing the correlation between human judgments A = {a 1 , a 2 , ..., a L } and metric scores S = {s 1 , s 2 , ..., s L } when dialogue systems make L individual responses R = {r 1 , r 2 , ..., r L } for their dialogue histories H = {h 1 , h 2 , ..., h L }.Fig. 1 illustrates the evaluation process of PAIREVAL. The metric is designed to judge the quality of a generated response by considering its relative quality against other comparison responses. We use an LLM along with a carefully designed text prompt to perform a pairwise comparison between two responses. Given a pair of conversations in the form of a text prompt, the LM is asked to choose the better response. The probabilities of the target response being better than each comparison example are then aggregated and utilized as its final score. The generated response receives higher metric scores as it is predicted as better than the other responses. To construct a group of comparison examples, we use a few randomly sampled conversations from a public dialogue corpus. This makes our metric to be a practical and efficient solution for dialogue evaluation. Formally, let (h i , r i ) denote a target conversation that consists of a generated response r i under evaluation and its corresponding dialogue history hi . Let C = {(h c 1 , r c 1 ), (h c 2 , r c 2 ), ..., (h c N , r c N )} represent N different comparison examples.We first make a pair of a target conversation with every comparison example {(h c j , r c j )} N j=1 , resulting in N different pairs of conversations. Each pair is then converted into an input text of an LLM by replacing the placeholder of our text prompt x ij = T(h i , r i , h c j , r c j ). Given the text prompt x ij , the LM (θ) is asked to choose a response of higher quality. To precisely acquire the LM's prediction in the form of predictive probability, we assign a single label word for each response (i.e., "A" and "B"), and regard the probability of each label word as a score of the response allocated by the LM. The probabilities that a generated response r i is in better quality s ij = P("r i is better than r j "|x ij , θ) are stored during every comparison against c j . The averaged probability of r i after every competition is then used as the final evaluation score of the target response. For the aggregation of probabilities with different comparison examples, we simply use an average operation (s i = ∑ N j=1 s ij ). In practice, since LLMs are known to be sensitive to ordering in the prompt (Wang et al., 2023) , we infer the LMs twice for a single conversation pair with swapped orders and use the averaged probability.Have you seen the latest episode of our favorite show? Not yet, I've been swampe d with work this week.When you do, we need to talk about it. it's a game-changer!Could you help me to translate this se ntence?Let's talk about it after you get on the new gardening tips. Although recent LLMs have shown impressive instruction-following and task generalization abilities, applying moderate-size LLMs directly for pairwise dialogue evaluation tasks described in Section 3.2 may make it hard to ensure reasonable evaluation results. For instance, the training examples that require LMs to compare two text inputs and choose the better one would occupy only a small portion of the entire training dataset. To tackle this problem, we propose an intuitive training strategy to make LMs specialize in pairwise comparisons for evaluation. Specifically, we construct synthetic training examples that instruct an LM to compare two responses of different quality -positive and negative responsesand choose the better one. To obtain a positive response along with its dialogue history, we simply use usual conversations that consist of a dialogue history and its follow-up utterances presented in the ordinary dialogue corpus. For reliable construction of negative ones, we explore two different types of negative responses with varied difficulties for evaluation: random and adversarial. The random negative responses are the utterances of different dialogue histories that are arbitrarily sampled from the dialogue corpus. The Adversarial negative responses are written by human annotators to exhibit high superficial relevance to a dialogue context but are not suitable as a follow-up utterance of the context. This adversarial response enables an LM to identify the inappropriateness of a response by capturing more subtle errors rather than relying on the superficial similarity against a dialogue history. In this work, we use human-written negative responses released by Sai et al. (2020) for the adversarial responses. Fig. 2 illustrates an example of responses with different types. Given a paired example with positive and negative responses, we construct an input text of an LM with pairwise evaluation prompt T. We then finetune the LM to predict the label word of a positive response correctly. The location of positive and negative examples in an input text is randomly decided to avoid unintended positional bias. 
experiments: 
 The section presents our experimental setups (section 4.1), main results (section 4.2), and further analysis (section 4.3) as follows.We evaluate the performance of each evaluation metric by comparing its outcome scores against scores assigned by human annotators. To this end, multiple meta-evaluation datasets with human annotations are used for experiments. Each instance in the dataset consists of (1) dialogue history, (2) generated response from a dialogue system, (3) answer response, and (4) response quality score annotated by human annotators. The details of each meta-evaluation dataset are as follows.DailyDialog-Zhao (Zhao et al., 2020) consists of 900 evaluation instances from multiple different dialogue systems. The dataset originates from the DailyDialog corpus (Li et al., 2017) , serving both as a training corpus of dialogue systems and the source of dialogue histories. The "overall" aspect of the example is used as a human annotation score. The inter-annotator agreement is measured by Kripendorff's alpha, with a result of 0.8. ConvAI2-USR and TopicalChat-USR are datasets released by Mehri & Eskenazi (2020a) and consist of 300 and 360 instances, respectively. Each dataset is derived by ConvAI2 (Dinan et al., 2020) and TopicalChat (Gopalakrishnan et al., 2019) dialogue corpus, respectively. Three dialogue systems are used as a dialogue system, and the "Overall Quality" score is used in our experiments. The inter-annotator agreements are measured by Spearman correlation with a result of 0.66 and 0.72 for each dataset. DailyDialog-GRADE and ConvAI2-GRADE are released by Huang et al. (2020) , each based on the DailyDialog and ConvAI2 datasets. DailyDialog-GRADE and ConvAI2-GRADE both contain 300 instances from two and four systems, respectively. FED (Mehri & Eskenazi, 2020b) The LM trained on the DailyDialog dataset is evaluated on the remaining meta-evaluation datasets.The following baseline metrics are considered in our experiments. For better readability, we group them into reference-based and reference-free metrics. Further details are in A.1. BLEU-2 (Papineni et al., 2002) measures the n-gram overlap similarity of a generated response against an answer (reference) response. BERTScore (Zhang et al., 2019) uses contextualized embeddings of a pre-trained LM (Liu et al., 2019) for similarity comparison. BLEURT (Sellam et al., 2020 ) is pre-trained on synthetic datasets for a better evaluation of machine translation systems.Reference-free Metrics USR (Mehri & Eskenazi, 2020a ) is trained to distinguish the original follow-up response of a dialogue history from randomly sampled ones. DEnsity (Park et al., 2023) calculates the distance of a generated response from the distribution of relevant responses on the feature space of an LM. FED (Mehri & Eskenazi, 2020b) and FULL (De Bruyn et al., 2022) both measure the likelihood of predefined follow-up utterances to estimate the quality of a generated response. UniEval (Zhong et al., 2022) adopt intermediate training on multiple datasets for an evaluation of various natural language generation tasks. LLMEval (Lin & Chen, 2023 ) is a prompt-based metric that leverages a proprietary LLM (claude-v1.3) for a dialogue evaluation task.Besides, we also introduce DIRECTEVAL, which uses the same training examples and the LLM as PAIREVAL. Instead of performing a pairwise comparison, however, this metric directly predicts the appropriateness of a generated response. Specifically, we ask the LM to predict the quality of a generated response ("Is the above response a good response to the given conversation?"), following previous studies (Gupta et al., 2022; Liang et al., 2023) . The probability of a target response (P("Yes")) is then used as a metric score. This metric is designed to validate the effectiveness of pairwise comparison by controlling the training configuration to be roughly the same as PairEval. We use Llama-2-7b-chat (Touvron et al., 2023) as an LM of both PAIREVAL and DI-RECTEVAL. The LM is finetuned with LoRA (Hu et al., 2021) for 1 epoch, and the r and α of LoRA is set to 16 and 8, respectively. AdamW (Loshchilov & Hutter, 2018) optimizer is used for optimization, and the initial learning rate is set to 1e-4. The batch size is set to 16. The number of comparison examples (N) is set to 3 and is randomly sampled from the validation set of the DailyDialog corpus. Note that these comparison examples are used across all meta-evaluation datasets. In all experiments, a single 3090 RTX GPU with 24GB of memory is used. Regarding the limited computational resources, we use 4-bit quantization (Dettmers et al., 2024) during the evaluation of DIRECTEVAL and PAIREVAL.We use the Pearson correlation coefficient (r) and Spearman's rank correlation coefficient (ρ) to measure the correlation between human judgments and metric scores. Table 1 shows the experimental results of all metrics. Overall, PAIREVAL achieves the highest performance in four and three out of six datasets in the Pearson and Spearman correlations, respectively. Our metric works well on dialogues from TopicalChat-USR and FED that are not used during the finetuning stages. Moreover, PAIREVAL sometimes even shows a higher correlation than LLMEval which relies on a powerful proprietary LLM. Among the other baseline metrics, UniEval shows reasonable performance. DIRECTEVAL, a metric that uses the same training dataset source as PAIREVAL, also shows competitive performance across multiple benchmarks. However, it generally performs worse than PAIREVAL, especially in the case of the Pearson correlation coefficient. We believe these results confirm the effectiveness of the pairwise comparison approach for dialogue evaluation.We conduct further analysis to understand PAIREVAL comprehensively as follows. To probe the validity of finetuning described in Section 3.3, we replace the finetuned LM in PAIREVAL with a pretrained one and observe the results. For experiments, we use a randomly sampled single conversation from a DailyDialog validation split as a comparison example. We report the mean and standard deviation of multiple runs (15) with different comparison examples. As shown in Table 2 , finetuning on pairwise evaluation examples greatly contributes to the performance of PAIREVAL. Furthermore, PAIREVAL with the finetuned LM is more robust to the changes of comparison examples, confirming its validity.We 3 . We first observe that the increased number of comparison examples usually contributes to a better correlation with human judgments. Regarding the different types of comparison examples, using the examples in the meta-evaluation datasets usually offers better performance. However, access to such test examples may not always be available. Therefore, we believe that using randomly sampled conversations from a dialogue corpus can be a reasonable option.We also explore the more exhaustive evaluation case, where a single evaluation sample is compared against all other samples in a meta-evaluation dataset to induce the final score. In other words, if we have M different responses under evaluation, the number of an LM inference becomes O(M 2 ). Results in Table 4 show that conducting an exhaustive comparison between all evaluation examples contributes to the increased correlation. For instance, In the ConvAI2-USR dataset, the correlations of PAIREVAL are increased from 66.5 and 71.1 to 70.0 and 71.9 in Pearson and Spearman correlation, respectively. Nevertheless, when considering the efficiency of an evaluation process, relying on a few (N ≤ 3) comparison examples can be a reasonable choice. Recent studies have reported that an LLM-based comparative evaluation usually suffers from positional bias, where an LLM exhibits spurious preference over instances in a certain position in the input texts (Wang et al., 2023; Zheng et al., 2024) . To analyze the impacts of such bias on performance, we locate the target evaluation conversation in the First and Second positions of an input prompt respectively, and observe the changed performance. Besides, we also report the original scores of PAIREVAL (Both) that use an averaged probability when the target response is located in First and Second positions. A single conversation is used as a comparison example for experiments. As shown in Table 5 , both zero-shot and finetuned LMs usually suffer from position bias to some extent. This problem is more significant in a zero-shot setting, where the max value of a correlation difference between the First and Second is 18.1 (41.4 and 23.3 in FED) . The finetuned LM looks relatively free from such issues, where a maximum correlation difference is 4.9 (57.7 and 62.6 in TopicalChat-USR).We verify the effectiveness of human-written adversarial examples described in Section 3.3. Table 6 presents the results of PAIREVAL when the LM is finetuned with and without human-written hard negative responses. For a fair comparison, we use the same number of training examples for both cases by replacing the hard negatives with random ones. From the results, we confirm that adversarial examples generally contribute to increasing the correlation of PAIREVAL with human evaluation. One remaining issue is that the creation process of adversarial responses involves a human annotation process, which is not always available and is not scalable. In this regard, a dataset generation strategy with LLMs can be an efficient and effective alternative (Yoo et al., 2021; Liu et al., 2022; Lee et al., 2022) . We leave such explorations as our future work.Khalid & Lee (2022) report that automated metrics for dialogue evaluation often struggle to give higher scores to correct responses rather than to adversarially curated and incorrect ones. We probe whether our pairwise comparative approach can alleviate such issues and can accurately identify subtly corrupted responses. To this end, we use a meta-evaluation dataset proposed by Khalid & Lee (2022) . The dataset is created by manipulating the original human-written response into various corruption strategies. Each evaluation example in the dataset consists of a (1) dialogue history, (2) original response, and (3) corrupted response. The metric should assign a higher score to the original response than the corrupted one. For experiments, we compare the performance of PAIREVAL and DIRECTEVAL. PAIREVAL compares two responses -original and corrupted -by locating them in a single input text, while DIRECTEVAL evaluates two responses individually. As shown in Table 7 , PAIREVAL achieves higher accuracy in most attack types, confirming its robustness on dialogue evaluation.Metric Visualization with Scattered Predictions Fig. 3 presents a prediction of selected metrics along with human judgments in the DailyDialog-GRADE dataset. The x and y axes of each point represent human judgments and metric scores, respectively. Among the baseline metrics, reference-based metrics like BLEU and BLEURT fail to assign discriminated scores as human ones. Reference-free metrics like USR and DirectEval show a positive correlation with human judgments. PAIREVAL shows a strong correlation with human evaluations. The code and models are available at https://github.com/ddehun/PairEval.https://huggingface.co/Elron/bleurt-tiny-512https://github.com/Tiiiger/bert_scorehttps://github.com/Shikib/fedhttps://github.com/tanyuqian/ctc-gen-evalhttps://github.com/maximedb/fullhttps://github.com/maszhongming/UniEvalhttps://github.com/ddehun/DEnsity/tree/master 
 
 Appendix:  


Decision: This paper proposes a new metric for evaluating open-domain dialog quality via paired comparison to a different dialog, showing higher correlation with human judgments of quality compared to several reference-free and reference-based metrics.

The idea of an in-between evaluation metric that uses some baseline, but not from the exact conversation context, is valuable. Additionally, the reviewers found the experiments to be comprehensive. However, the main concern is that the majority of the improvement seems to come from the specific language model used rather than the pairwise design choices, which is evident in comparing directEval and PairedEval. Since the core contribution of the paper is "a new comparative evaluation paradigm," as the authors put it, this limits the applicability of this work. While results in general setups are quite mixed when comparing directEval and PairedEval, PairedEval seems to perform slightly better in an adversarial setup.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel dialogue evaluation metric for assessing one response's quality by comparing the target dialogue with a set of different dialogues.  Specifically, the method requires the model to compare the target dialogue with each of the dialogues in the set and use the accumulated comparison score as the final evaluation for the target dialogue.  The method is based on moderate-sized open-source language models. Experiments on multiple test sets show the effectiveness. However, some statements are confusing and there are a few questions about the method.
strength_weakness: strength_weakness: 3.1 This paper needs to explain more deeply why this comparative method would lead to better evaluation results. Specifically, why comparing with other dialogue can help the model to evaluate dialogue quality? 

3.2 On page 2, the authors state "However, they find that exhaustive comparison between candidate responses is needed for reliable quality estimation of dialogues, resulting in an undesirable computational overhead.". This is why they use moderate-sized open-source models, but the method in this paper requires fine-tuning the model and multiple calls to the model for one evaluation. The undesirable computational overhead still exists. A computation cost comparison between the LLM-based method and this paper should be conducted.

3.3 On page 2, the authors state "However, these approaches usually require exhaustive comparison operations or human-crafted candidate responses, which may not always be available.". Since you state that human-crafted data has the above drawbacks, why do you still conduct experiments with human-crafted adversarial data (from previous research)?
questions: questions: 4.1 This article uses artificially constructed adversarial data (from previous research). Does this impact the migration of the method in this article to other domains? Such as DSTC-9 data?

4.2 Apart from using a relatively large 7B model, what is the essential difference between your pairwise training method and the previous method that distinguishes positive and negative examples (such as DynaEval by Zhang et al., 2021)? 

4.3 Why using an exhaustive comparison did not entail big improvements (Tables 3 and 4)?

4.4 The experimental results are different in Tables 3 and 6 when N=1. Why the differences?
paper_summary: paper_summary: The paper proposed a method *PairEval* for dialog response evaluation.

*PairEval* lies between commonly used reference-based metrics and reference-free metrics. The core idea and novelty is to make the assessment decision (1) dependent on some baseline examples, while (2) avoiding the direct use of a reference response of the conversation of interest.
* The first design makes the assessment results more discriminative, especially when all evaluation examples use the same set of baseline examples, as is done in this work.
* The second design alleviates the influence of the one-to-many mapping problem in open-domain dialogs.

The whole idea is great. Experiment settings are mostly reasonable and comprehensive.

The only problem is that the proposed method *PairEval* does not improve much over the actual baseline *DirectEval*.
* Although the authors also compare it against a number of baseline metrics from prior works and *PairEval* is much stronger, the baseline *DirectEval* also outperforms other baselines a lot. It is fair to say that most of the improvement comes from using llama-2-7b-chat as the base model, and the gap between *PairEval* and *DirectEval* is less obvious.
* To be more specific, firstly *PairEval* and *DirectEval* have close Spearman's $\rho$ (63.1 vs. 61.8). Secondly, the larger gap of Pearson's $r$ (58.5 vs. 49.8) is likely to be largely reduced if we use another simple baseline which is a fine-tuned llama-2-7b-chat that regresses the output score.

Overall, the proposed method is promising and the results are positive, but I would expect more performance gain from the method. I suggest considering the following to make this work stronger.
* Make a more delicate design on the choice of baseline examples. Since all evaluation examples will share a small set of baseline examples, curating the baseline example set is not going to be very costly. Additionally, maybe we can get aspect-based evaluation and interpretability as a bonus by carefully choosing baseline examples.
* Include a fine-tuned regression model based on llama-2-7b-chat as a baseline.
* Analyze how the choice of evaluation dataset causes very different experimental results. For example, *PairEval* outperforms *DirectEval* significantly on *DailyDialog-GRADE* but it is close to or underperforms *DirectEval* on other datasets.

A minor suggestion is to include the results of non-exhaustive comparison in Table 4 for easy reading.
strength_weakness: strength_weakness: * Marginal improvement over the actual good baseline (*DirectEval*).
* Lack of investigation into the performance discrepancy on different datasets.
paper_summary: paper_summary: The authors propose a new metric to evaluate open-domain dialogue via pairwise comparisons of responses in different dialogues. They use an LLM to judge the system's response along with a number of randomly sampled utterance-response examples (with different context) and use the scores to calculate a final score for the system response.
strength_weakness: strength_weakness: May not be very generalizable since it requires training (e.g. the authors use human-written negative responses)
questions: questions: If you need to fine-tune a model for Pairwise evaluation, how different is this to reference-free / LLM as a judge approaches? I understand the need to bypass proprietary LLMs but we now have access to very powerful open-source LLMs that can be used as judges. It would be nice to see a comparison against PairEval.

How do you handle topic switches, humour, sarcasm, or other cases where the response may not seem to match but could be actually good, funny, etc?

How difficult is it to design the LLM prompt (Section 3.2)? If one needs to replace the LLM with a newer / different model, how much effort is it to re-design the prompt?

Do you think this approach could work on other domains e.g. math or coding? What adjustments would need to be made?

Given that your metric needs to be trained, have you tried using a larger LLM or training with more data?
paper_summary: paper_summary: The authors introduce PairEval a dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations. For this purpose they fine-tune a LLM to conduct the specific pair-wise dialogue evaluation task proposed. Evaluation results show better correlation with human evaluations than other metrics used in the evaluation. I think this paper presents an interesting idea and pushes forward the state of the art in open dialogue evaluations. COLM participants and the research community will benefit from the ideas and results presented here. However, I still feel additional work is needed to get a much better assessment of the benefits of the proposed technique. For instance, it would be interesting to understand the range of variability of achieved results by changing the comparative examples used in the evaluations, as well as the training data used to fine-tune the LLM. How much semantic similarity and/or topic overlap are there between the data used for fine-tuning and the evaluation data sets? Also, how human evaluations correlate to other human evaluations? I think an opportunity is missed here to understand better the complexity of the task for humans and get a better sense on how far or how close the proposed technique is bringing us to human evaluations.
strength_weakness: strength_weakness: I would probably want to see more exploratory analysis on the items described in the questions to authors, although I do not feel this is actually a reason to reject, given that this is a conference paper.
questions: questions: It would be interesting to understand the range of variability of achieved results by changing the comparative examples used in the evaluations, as well as the training data used to fine-tune the LLM. How much semantic similarity and/or topic overlap are there between the data used for fine-tuning and the evaluation data sets? Also, how human evaluations correlate to other human evaluations? I think an opportunity is missed here to understand better the complexity of the task for humans and get a better sense on how far or how close the proposed technique is bringing us to human evaluations.

Comments: comment: Thank you for the response. The overall improvement is still marginal so I will keep my score unchanged.
Thanks for your response to my comments. I am looking forward to seeing the information about inter-annotator agreement in the camera ready version of the paper, as well as future research work addressing some of the issues discussed.