PairEval: Open-domain Dialogue Evaluation Metric with Pairwise Comparisons
Authors: ChaeHun Park
Minseok Choi
Dohyun Lee
Jaegul Choo
Venue: COLM (2024)
Keywords: Open-domain dialogue, Dialogue evaluation, Evaluation metric
TL;DR: We introduce a new evaluation metric with comparative assessments for open-domain dialogue systems.

Abstract: Building a reliable and automated evaluation metric is a necessary but challenging problem for open-domain dialogue systems. 
Recent studies proposed evaluation metrics that assess generated responses by considering their relevance to previous dialogue histories. 
Although effective, these metrics evaluate individual responses directly rather than considering their relative quality compared to other responses.  To handle this, we propose  PairEval, a novel dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations.  Our metric is built on top of open-sourced and moderate-size language models, and we make them specialized in pairwise comparison between dialogue responses. Extensive experiments on multiple benchmarks demonstrate that our metric exhibits a higher correlation with human judgments than baseline metrics. We also find that the proposed comparative metric is more robust in detecting common failures from open-domain dialogue systems, including repetition and speaker insensitivity. The codes and models will be publicly available after the paper is accepted.

Published as a conference paper at COLM 2024
PAIREVAL: Open-domain Dialogue Evaluation with Pairwise Comparison
ChaeHun Park
Minseok Choi
Dohyun Lee
Jaegul Choo
KAIST AI
[ddehun, minseok. choi, aiclaudev, jchoo) ac.kr
Abstract
Building reliable and automated evaluation metric is a necessary but challenging problem for open-domain dialogue systems. Recent studies proposed evaluation metrics that assess generated responses by consid- ering their relevance to previous dialogue histories. Although effective, these metrics evaluate individual responses directly rather than consid- ering their relative quality compared to other responses. To handle this, we propose PAIRE EVA, novel dialogue evaluation metric for assessing responses by comparing their quality against responses in different con- versations. PAIREVAL is built on top of open-sourced and moderate-size language models, and we make them specialized in pairwise comparison between dialogue responses. Extensive experiments on multiple bench- marks demonstrate that our metric exhibits exhibits a higher correlation with human judgments than baseline metrics. We also find that the proposed compara- tive metric is more robust in detecting common failures from open-domain dialogue systems, including repetition and speaker insensitivity.
1 Introduction
Open-domain dialogue systems aim to interact with users by generating natural and gaging responses for a given dialogue history. In this field, building an accurate automatic evaluation metric to judge the quality of such generative systems is an important but chal- lenging task (Liu et al., 2016). The difficulties partly originate from the one-to-many nature of daily conversations, where a single conversation can be continued with many different follow-up utterances. This variety in dialogues makes traditional reference-based metrics, which compare generated responses to a limited of known answers (i.e., references), do not correlate well with human judgments (Liu et al., 2016). The lack of reliable evaluation metrics impedes real-world deployment of open-domain dialogue systems.
Researchers have proposed various evaluation metrics to achieve reliable assessments of dialogue systems. Initial studies enhance reference-based metrics by adopting distributed and contextualized representations by neural networks (Liu et al., 2016; Zhang et al., 2019). Subsequent studies introduce reference-free metrics that employ prediction models to assess responses based on their relevance to the previous dialogue context (Tao et al., 2018; Lowe et al., 2017; Mehri & Eskenazi, 2020a; Zhang et al., 2021; Zhong et al., 2022), showing a higher correlation with human judgments. Moreover, prompting -aased based metrics have been presented to properly transfer the instruction-following and zero-shot capabilities of large language models (LLMs) for dialogue evaluation (Zhang al., 2023; Lin & Chen, 2023).
We hold that the dialogue evaluation should aim to assign differentiated scores to responses by considering their relative quality. In other words, evaluation metrics should make cali- brated scores such that they are appropriately aligned with human evaluations. Numerous correlation metrics used in the meta-evaluation of evaluation metrics like Spearman (Zar, 2005) or Kendall rank correlations (Kendall, 1955) reflect this intuition. Therefore, we argue that assessing the target responses by considering their relative appropriateness with other
The code and models available httttttp///// htt.ghtttbbhccogoo wwt.u./com//eeee///eee///e reval.

---

Published as a conference paper at COLM 2024
ones is meaningful process to make more reliable evaluation metrics. In this regard, several studies evaluate responses by considering their relative quality (Sato et al., 2020; Liusie et al., 2024). However, these approaches usually require exhaustive comparison operations or human-crafted candidate responses, which may not always be available.
In this paper, we propose PAIREVAL, a novel open-domain dialogue evaluation metric with comparative assessments. Our metric assesses the individual responses by comparing their quality against only a limited number of comparison responses. Instead of relying on a commercial or proprietary LLM, our metric is built on top of a moderate-size and open-sourced LLM (Touvron et al., 2023). To elicit the comparative ability of LMs, we devise a simple but effective learning strategy with public dialogue corpus. Experiments on multiple benchmarks show that PAIREVAL outperforms previous evaluation metrics, and sometimes even shows higher performance than metrics with powerful proprietary LLM. Further analysis demonstrates that the pairwise evaluation approach is robust and effective in capturing common failures (e.g., repetitive outcomes) in dialogue systems.
2 Related Work
Evaluation Metrics for Open-domain Dialogue Systems Traditional metrics like BLEU (Papineni et al., 2002) ROUGE (Lin, 2004) that measure the N-gram overlap be- tween generated responses and answers show a low correlation with human judgments (Liu et al., 2016). Several studies use embedding models to consider the semantic similarity between candidate responses and answers (Liu et al., 2016; Zhang et al., 2019). However, these reference-based metrics rely on a set of known answers for similarity comparison, making it hard to consider the wide semantic space of follow-up utterances for a single conversation. To tackle this problem, recent studies proposed numerous reference-fres metrics that directly predict the relevance of generated response to the given dialogue history (Tao et al., 2018; Mehri Eskemazi, 2020a;b; Zhong et al., 2022; De Bruyn et al., 2022). Specifically, neural classification or regression models are usually trained to distinguish relevant responses from irrelevant ones. These predictive metrics have shown meaningful progress along with pre-trained language models (Ghazarian et al., 2019; Mehri & Eskenazi, 2020a; Zhong et al., 2022), data augmentation strategies (Gupta et al., 2021; Park et al., 2021), and advanced training algorithms (Huang et al., 2020; Zhang et al., 2021). In contrast, our work focuses on a comparative evaluation to consider the relative quality between responses for more reliable evaluation. In open-ended text generation tasks, Pillutta et al. (2021) propose a corpus-level evaluation metric, namely MAUVE, which compares model-generan text distributions with human-written ones using divergence frontiers. To the best of our knowledge, Liusie et al. (2024) propose pioneering step that introduces a comparison- based evaluation approach for various natural language generation tasks, including an open-domain dialogue generation task. However, they find that exhaustive com parisiis between candidate responses is needed for reliable quality estimation of dialogues, resulting in an undesirable computational overhead. Our work overcomes this challenge by spe- cializing in moderate-size LLMs for pairwise comparison between responses. Sato et al. (2020) consider the relative rank of generated responses against other false candidates, and response selection model is employed for this purpose. Though promising, their evaluation approach requires human evaluation to filter out correct responses from a candidate pool. In contrast, we show that even a limited number of comparisons are enough to ensure a reasonable correlation with human judgment.
Pairwise Comparison with Large Language Models The pairwise comparison ap- proach has been widely explored from various perspectives, including preference learn- ing (Fürnkranz & Hüllermeier, 2003), recommendation (Beutel et al., 2019), reinforcement learning (Xu et al., 2020), and retrieval systems (Qin et al., 2023). Recent studies that consider the human-aligned behaviors of LMs proposed to reflect human preference in the form of comparisons over multiple model generations (Ouyang et al., 2022; Bai et al., 2022). An LLM-based pairwise evaluation has also been increasingly adopted to build -leee ranking information (Boubdir et al., 2023; Zheng et al., 2024). This work evaluates individual responses by comparing their relative quality against the limited number of responses.
2

---

Published as a conference paper at COLM 2024
Evaluation Example
Pairwise Text Prompt
Comparison & Aggregation
Dialogue History
Instruction: Given two conversations and responses, which response better conversation?
Did you finish the book lent you last week?
lold! plot twist the end unexpected.
Conversation
Parwii Prompt
Response
Idialoque hestozy_ __Al
Response
Sound interesting! Can you borrow ton ggh??
=eeaponae_At
LLM
Comparison Examples
Conversation
ddlaloque_hhhatoo h_scory__
Dialogue History
Response
heard there's is new cafe downtown.
[seaponae__e
P(A7)-0.17
Response
Which response better for the convensation? Just answer "Response A or "Response 8". Response
I've been eager to their artisan coffee.
Figure 1: The overall illustration of PAIREVAL.
3 Method
We propose PAIREVAL, a new reference-free dialogue evaluation metric based on a pairwise comparison. Our metric assesses responses by comparing their quality against small number of comparison examples. The comparison examples are derived from human- written conversations in a dialogue corpus. Fig. 1 depicts the overall pipeline of PAIREVAL.
3.1 Task Formulation
In this work, we focus on the turn-level and reference-free evaluation of open-domain dialogue systems. Given a dialogue history h that consists of multiple utterances between two speakers, dialogue system outputs a single utferance r as a follow-up response. The evaluation metric M considers how suitable the generated response is as the next utterance for the given dialogue history and makes an evaluation score $ = M(h,r) r).. The performance of the metric is usually measured by computing the correlation between human judgments A = {#1, (2,,,,, and metric scores S = {s,, $2, when dialogue systems make individual responses R f2., } for their dialogue histories H {h, h2 ...,,....
3.2 Dialogue Evaluation with Pairwise Comparison
Fig. illustrates the evaluation process of PAIREVAL. The metric is designed to judge the quality of a generated response by considering its relative quality against other comparison responses. We use an LLM along with a carefully designed text prompt to perform pairwise comparison between two responses. Given a pair of conversations in the form of text prompt, the LM asked to choose the better response. The probabilities response being better than each comparison example are then aggregated and utilized as its final score. The generated response receives higher metric scores as it is predicted as better than the other responses. To construct a group of comparison examples, we use few randomly sampled conversations from a public dialogue corpus. This makes our metric to be practical and efficient solution for dialogue evaluation.
the target
Formally, let ri) denote target conversation that consists of a generated re- sponse 7; under evaluation and its corresponding dialogue history h4, Let C (9), (hξ, r8),.., (Hy { represent N different comparison examples. We first make pair of target conversation with every comparison example {kC r)) resulting in N different pairs of conversations. Each pair is then converted into an input text of an Given the text prompt X;j, the LM (0) is asked to choose a response of higher quality. To precisely acquire the LM's prediction in the form of predictive probability, we assign single label word for each response (i.e., "A" and "B"), and regard the probability of each label word score of the response allocated by the LM. The probabilities that a generated response r; is in better quality sij P("r; is better than r;" |xij. 0) are stored during every comparison against
LLM by replacing the placeholder of our text prompt x;j T(h; r;, hf,, rf).
3

---

Published as a conference paper at COLM 2024
Cj. The averaged probability of ri after every competition is then used as the final evaluation score of the target response. For the aggregation of probabilities with different comparison examples, we simply use an average operation (s; -}= Stj). In practice, since LLMs are known to be sensitive to ordering in the prompt (Wang et al., 2023), we infer the LMs twice for single conversation pair with swapped orders and the averaged probability.
3.3 Making LMs Specialized in Pairwise Comparison
Although recent LLMs have shown impressive instruction-folloiing and task generalization abil-
ities, applying moderate-size LLMs directly for pair- wise dialogue evaluation tasks described in Sec-
Dialogue History
Have seen the latest eplcode our favorite show?
tion 3.2 may make it hard to ensure reasonable evalu- ation results. For instance, the training examples that
yet, been swamped with work this week.
Original Positive Response
When you we
g'amaammedii changer
require LMs to compare two text inputs and choose
Randomly Sampled Negative Response
the better one would occupy only a small portion of
Could you me
sentence?
the entire training dataset. To tackle this problem, we propose an intuitive training strategy make LMs
Human-crafted Adversorial Response
Let's talk about it after you get on the new ga raddinn lbs.
specialize in pairwise comparisons for evaluation.
Responses in different Figure 2:
Specifically, we construct synthetic training examples that instruct LM to compare two responses of different quality - positive and negative responses-
types to finetune a LM in PAIREVAL.
and choose the better one. To obtain positive response along with its dialogue history, we simply use usual conversations that consist of a dialogue history and its follow-up utterances presented in the ordinary dialogue corpus. For reliable construction of negative ones, we explore two different types of negative responses with varied difficulties for evaluation: random and adversarial. The random negative responses are the utterances of different dialogue histories that are arbitrarily sampled from the dialogue corpus. The Adversarial negative responses are written by human annotators to exhibit high superficial relevance to a dialogue context but are not suitable as a follow-up utterance of the context. This adversarial response enables an LM to identify the inappropriateness of a response by capturing more subtle errors rather than relying on the superficial similarity against dialogue history. In this work, we use human- negative responses released by Sai et al. (2020) for the adversarial responses. Fig. illustrates an example of responses with different types. Given a paired example with positive and negative responses, we construct input text of an LM with pairwise evaluation prompt T. We then finetune the LM to predict the label word of a positive response correctly. The location of positive and negative examples in an input text is randomly decided to avoid unintended positional bias.
4 Experiments
The section presents our experimental setups (section 4.1), main results (section 4.2), and further analysis (section 4.3) as follows.
4.1
Setup
4.1.1 Dataset
Meta-Evaluation Dataset We evaluate the performance of each evaluation metric by comparing its outcome scores against scores assigned by human annotators. To this end, multiple meta-evaluation datasets with human annotations are used for experiments. Each instance in the dataset consists of (1) dialogue history, (2) generated response from dialogue system, (3) answer response, and (4) response quality score annotated by human annotators. The details of each meta-evaluation dataset are follows.
DailyDialog-ZZ Zhao (Zhao et al., 2020) consists of 900 evaluation instances from multiple different dialogue systems. The dataset originates from the DailyDialog corpus (Li et al.,
4

---

Published as a conference paper at COLM 2024
2017), serving both as training corpus of dialogue systems and the source of dialogue histories. The "overall" aspect of the example is used as a human annotation score. The inter-annotator agreement is measured by Kripendorff's s alpha, with result of 0.8. ConvA12- USR and TopicalChal- USR are datasets released by Mehri & Eskenazi (2020a) and consist of 300 and 360 instances, respectively. Each dataset is derived by ATn (Dinan et al., 2020) and TopicalChat (Gopalakrishnan et al., 2019) dialogue corpus, respectively. Three dialogue systems are used as a dialogue system, and the "Overall Quality" score is used in our experiments. The inter-annotator agreements are measured by Spearman correlation with result of 0.66 and 0.72 for each dataset. are released by Huang et al. (2020), each based on the DailyDialog and Conv A12 datasets. DailyDialog-GRRe -RADE and ConvAI2-GRADE systems, respectively. FED (Mehri & Eskenazi, 2020b) contains 375 instances of conversations between human speakers or human and dialogue systems. The "Overall" aspect of the examples is used for evaluation. The inter-annotator agreement is measured by Spearman correlation with a result of 0.82.
SallyDiillo-GRADDD
and ConDAI2-GRADE
both contain 300 instances from two and four
Training Dataset We finetune an LLM with two widely used open-domain dialogue corpus DailyyDialo (Li et al., 2017) and ComoAl2 (Dinan et al., 2020) -individually, resulting in two different versions of PAIREVAL. Both corpora contain multi-turn conversations between two speakers, and the number of conversations is 13,118 and 17,818, respectively. To construct positive response sets, we use randomly sampled utterances along with their previous dialogue history in the train split of a dialogue corpus. For random negative response sets, we use randomly sampled utterances from a dialogue corpus. During the training of the LM on the DailyDialog corpus, we also use human-written adversarial responses released by Sai et al. (2020) as an adversarial negative type. The number of training examples is set to 80k and 65k for on the ConvA12 da dataset is used for the evaluation of CommA12-RR ADDDE and ConZA12-USR, The LM trained on the DailyDialog dataset is evaluated on the remaining meta-evaluation datasets.
and ConvAl, respectively. The LM trained
4.1.2 Baselines
The following baseline metrics are considered in our experiments. For better readability, we group them into reference-based and reference-free metrics. Further details are in A.1.
Reference-based Metrics BLEU-2 (Papineni et al., 2002) measures the n-gram overlap similarity of a generated response against an answer (reference) response. BERTScore (Zhang et al., 2019) uses contextualized embeddings of a pre-trained LM (Liu et al., 2019) for similarity comparison. BLEURT (Sellam et al., 2020) is pre-trained on synthetic datasets for a better evaluation of machine translation systems.
Reference-free Metrics USR (Mehri & Eskenazi, 2020a) is trained to distinguish the original follow-up response of dialogue history from randomly sampled ones. DEn- sity (Park et al., 2023) calculates the distance of a generated response from the distribution of relevant responses on the feature space of an LM. FED (Mehri & Eskenazi, 2020b) and FULL (De Bruyn et al., 2022) both measure the likelihood of predefined follow-up utterances to estimate the quality of a generated response. UniEval (Zhong al., 2022) adopt interme- diate training on multiple datasets for an evaluation of various natural language generation tasks. LLMExal (Lin & Chen, 2023) is a prompt-based metric that leverages a proprietary LLM (claude-v1 3) for a dialogue evaluation task.
Besides, we also introduce DIRECTEVAL, which uses the same training examples and the LLM as PAIREVAL. Instead of performing a pairwise comparison, however, this metric directly predicts the appropriateness
generated response. Specifically, we ask the LM to predict the quality of a generated response ("Is the above response a good response to the given conversation?") following previous studies (Gupta et al., 2022; Liang et al., 2023). The probability of target response (P("Yes"") is then used as metric score. This metric is designed to validate the effectiveness of pairwise comparison by controlling the training configuration to be roughly the same as PairEval.
5

---

Published as a conference paper at COLM 2024
Methods
GRADE
DailyDialog
GRADE
ConvAT2
USR
TopicalChat
USR
Avg.
Notrence fased Motris
(Papineni 2002)
52.3
31.5
BLEURT (Seliam al, 2020)
17.5
12.
31.
28.7
16.9
33.3
29.9
11.7
41.7
Seoee Zhang of (2019)
12.9
36.1
29.4
26,0
27.7
33.3
28.3
46.9
46.2
Ng tie MarAs
FED (Mehn Eskemari, 2020h)
2.6
0.1*
-9.0"
-8.5"
-6.1-
-4.6"
-2.0"
-0.7*
-7.1"
-6.9
11.9
9.4"
-1.
-1.9
USK (Metri Iskanazi 2020aa
27.5
23.5
08.8
51.6
40.3
40.0
60.9
08.1
10.7
32.5
11.4
11.7
28.2
34.5
CTC (Ding et al, 2021)
-15.1
-14.7*
-5.6"
-9.2"
3.9"
6.1"
17.1
08.3
39.8
36.3
16.1
19.5
19.4
14.0
FULL (Do Bruyn et 2072))
-8.8
-10.2
5.1"
6.07
22.07
23.8"
7.5
8.9
-4."
-7.2
47.3
50.
11.3
12.0
UniEval (7hong al., 2002)
7.6"
5.7"
30.9
90.1
46.6
47.9
65.1
65.9
62.0
64.5
29.8
20.0
46.9
47.7
DRessity (Park all, 2023)
30.3
29.5
56.8
57.0
48.0
48.6
57.0
63.0
163
24.7
24.5
2144
38.8
40.7
DIRECTEVAL
05.6
62.4
20.0
55.5
56.2
58.3
75.0
39.8
73.9
11.
50.1
19.8
61.
PAREVAL (Ours)
17.5
55.6
56.6
66.5
71.1
70.8
72.2
51.9
52.0
58.5
1. Chen, 2023)
53.3
51.5
49.0
49.9
Table 1: The correlations between automatic metrics and humans on meta-evaluation datasets. r and p denote Pearson correlation and Spearman's rank correlation coefficient, respectively. All values with > le-5 are marked with The highest and the second highest scores in each column are marked in bold and underline, respectively. For PAIREVAL VAL, we report the averaged scores over five runs with different comparison examples.
DailyDialog Zhao
DailyDialog GRADE
TopicalChat USR
FED
LM
14
a
u
0
Zero-shot
16.2
5.0
31.1
3.5
42.6
2.8
44.6
3.6
Fine-tuned
55.5
1.1
70.5
0.4
71.9
0.5
51.2
1.1
Table 2: Mean (μ) and standard deviation (o) results of fifteen runs with different random comparison examples (N - 1).
4.1.3 Implementation Details
We use Llama- 2-7b-chat (Touvron et al., 2023) as LM of both PAIREVAL and Di- RECTÉVAL. The LM is finetuned with LoRA (Hu et al., 2021) for epoch, and the r and of LoRA is set to 16 and 8, respectively. AdamW (Loshchilov & Hutter, 2018) optimizer is used for optimization, and the initial learning rate is set to le-4. The batch size is set to 16. The number of comparison examples (N) is set to 3 and is randomly sampled from the validation set of the DailyDialog corpus. Note that these comparison examples are used across all neta-evaluation datasets. In all experiments, a single 3090 RTX GPU with 24GB of memory is used. Regarding the limited computational resources, we use 4-bit quantization (Dettmers al., 2024) during the evaluation of DIRECTEVAL and PAIREVAL.
4.2 Main Results
We use the Pearson correlation coefficient (r) and Spearman's rank correlation coefficient (p) to measure the correlation between human judgments and metric scores. Table shows the experimental results all metrics. Overall, PAIREVAL achieves the highest performance in four and three out of six datasets in the Pearson and Spearman correlations, respectively. Our metric works well on dialogues from TopicalChat-USR and FED that are not used during the finetuning stages. Moreover, PAIREVAL sometimes even shows higher correlation than LLMEval which relies on a powerful proprietary LLM. Among the other baseline metrics, UniEval shows reasonable performance. DIRECTEVAL, metric that uses the same training dataset source as PAIREVAL, also shows competitive performance across multiple benchmarks. However, it generally performs worse than PAIREVAL, especially in the case of the Pearson correlation coefficient. We believe these results confirm the effectiveness of the pairwise comparison approach for dialogue evaluation.
4.3 Analysis
We conduct further analysis to understand PAIREVAL comprehensively as follows.

---

Published as a conference paper at COLM 2024
DailyDialog GRADE
DialyDialog Zhao
C2 GRADE
ConvA12 USR
TopicalChat USR
FED
Comporison Examples from Dinlogue Carpus (Random)
N
1
42.2
56.1
57.0
70.6
48.3
56.5
61.3
20.9
64.5
72.1
46.2
50.9
45.5
56.1
6088
70.7
49.0
56.3
66.3
71.1
69,9
72.3
50.6
52.0
3
47.8
55.8
62.9
70.7
51.1
56.6
66.5
71.1
70.8
72.2
51.9
52.0
Comporison Examples from Meta- Eonluntoonn Datasel (Test)
50.9
55.8
573
70.3
46.7
55.8
71.5
72.1
N
2
54.1
56.4
65.3
71.0
50.2
56.6
63.7
72.0
20.1
72.7
51.1
51.0
54.9
56.5
67.1
71.0
53.9
56.2
Table 3: Results with changed number (N) and type (Random and Test) of comparison examples. The highest score within the same type of comparison example is underlined. The highest score among all configurations is further highlighted in bold. All results are averaged scores over five runs with different comparison examples.
DailyDialoy GRADE
ConvA12
Methods
USR
p
Zero-shot
16.6
17.3
30.1
29.8
Fine-tuned
55.8
56.4
70.0
71.9
Table 4: Results with an exhaustive comparison between meta-evaluation examples.
4.3.1 Impacts of Finetuning on Correlation and Stability of PairEval
To probe the validity of finetuning described in Section 3.3, we replace the finetuned LM in PAIREVAL with a pretrained one and observe the results. For experiments, we use randomly sampled single conversation from a DailyDialog validation split as a comparison example. We report the mean and standard deviation of multiple runs (15) with different comparison examples. As shown in Table 2, finetuning on pairwise evaluation examples greatly contributes to the performance of PAIREVAL. Furthermore, PAIREVAL with the finetuned LM more robust to the changes of comparison examples, confirming its validity.
4.3.2 Changed Number and Types of Comparison Examples
We next analyze the impacts of different configurations for comparison examples in PAIREVAL by (1) reducing the number of comparison examples (N) and (2) using a few examples in the target meta-evaluation dataset (Test) instead of dialogue corpus (Random). We report the averaged results of five runs with different comparison examples. Finetuned LMs are used for the experiments, and results shown in Table 3. We first observe that the increased number of comparison examples usually contributes to a better correlation with human judgments. Regarding the different types of comparison examples, using the examples in the meta-evaluation datasets usually offers better performance. However, access such test examples may not always be available. Therefore, we believe that using randomly sampled conversations from dialogue corpus can be reasonable option.
We also explore the more exhaustive evaluation case, where a single evaluation sample is compared against all other samples in meta-evaluation dataset to induce the final score. In other words, if we have M different responses under evaluation, the number an LM infer- ence becomes (M2). Results in Table 4 show that conducting an exhaustive comparison between all evaluation examples contributes to the increased correlation. For instance, In dataset, the correlations of PAIREVAL are increased from 66.5 and 71.1 to 70.0 and 71.9 in Pearson and Spearman correlation, respectively. Nevertheless, when considering the efficiency of an evaluation process, relying on few (N < 3) comparison examples can be reasonable choice.
the Conv
7

---

Published as a conference paper at COLM 2024
DailyDialog
DailyDialog GRADE
TopicalChat USR
FED
LM
Position
Zhao
p
p
First
7.4
7.2
24.2
27.1
33.0
35.5
41.4
47.0
Zero-shot
Second
15.4
19.4
26.5
30.2
35.4
40.0
23.3
33.5
Both
14.3
15.9
31.2
32.2
39.8
42.2
42.3
44.5
First
40.8
54.3
55.8
69.5
57.7
73.2
45.3
50.6
Finetuned
Second
40.9
56.8
54.8
70.8
62.6
70.6
45.4
50.1
Both
57.0
70.6
64.5
72.1
46.2
50.9
Table 5: Analysis of position bias results with single random comparison example (N 1). The Position denotes a location of evaluation examples in an input prompt. We report the averaged results of five runs with different comparison examples.
DailyDialog GRADE
DailyDialog Zhao
TopicalChat USR
Comparison Example
Hard Negative
FED
p
36.9
39.6
62.4
69.2
69.0
69.9
45.3
45.0
Random
50.8
56.4
66.4
70.7
68.9
72.4
52.4
51.6
37.5
38.2
63.8
68.7
70.9
72.5
43.4
43.4
Test
49.0
55.5
64.5
71.1
71.1
71.8
53.3
51.8
Table 6: Results with ablating human-written hard negative for LM finetuning (N 1). The Test denotes comparison examples from each meta-evaluation dataset.
4.3.3 Position Bias
Recent studies have reported that an LLM-based comparative evaluation usually suffers from positional bias, where an LLM exhibits spurious preference over instances a certain position in the input texts (Wang et al., 2023; Zheng et al., 2024). To analyze the impacts of such bias on performance, we locate the target evaluation conversation in the First and Second positions of an input prompt respectively, and observe the changed performance. Be- sides, we also report the original scores of PAIRÉVAL (Both) that use an averaged probability when the target response is located in First and Second positions. A single conversation is used as a comparison example for experiments. As shown in Table 5, both zero-shot and finetuned LMs usually suffer from position bias to some extent. This problem is more signifi- cant in a zero-shot setting, where the max value of a correlation difference between the First and Second is 18.1 (41.4 and 23.3 in FED). The finetuned LM looks relatively free from such issues, where a maximum correlation difference 4.9 (57.7 and 62.6 in TopicalChat-USR)
4.3.4 Impacts of Human-Written Adversarial Negatives for LM Finetuning
We verify the effectiveness of human-written adversarial examples described in Section 3.3. Table 6 presents the results of PAIREVAL when the LM is finetuned with and without human-written hard negative responses. For fair comparison, we use the same number of training examples for both cases by replacing the hard negatives with random ones. From the results, we confirm that adversarial examples generally contribute to increasing the correlation of PAIREVAL with human evaluation. One remaining issue is that the creation process of adversarial responses involves a human annotation process, which is not always available and is not scalable. In this regard, a dataset generation strategy with LLMs can be an efficient and effective alternative (Yoo et al., 2021; Liu et al., 2022; Lee et al., 2022). We leave such explorations as our future work.
4.3.5 Robustness Adversarially Manipulated Responses
Khalid & Lee (2022) report that automated metrics for dialogue evaluation often struggle to give higher scores to correct responses rather than to adversarially curated and incorrect ones. We probe whether our pairwise comparative approach can alleviate such issues and can accurately identify subtly corrupted responses. To this end, we meta-evaluation
8

---

Published as a conference paper at COLM 2024
Adversarial Attack Type
Pronoun Usage
Named Entities
Speaker Sensitiveness
Method
Entailment
Contradict
DIRECTÉVAL PAIREVAL
93.9
95.7
94.9
79.0
95.5
92.3
98.6
94.9
95.6
96.4
(a) Evaluation results with different adversarial attack types.
Adversarial Attack Type
Vocabulary Diversity
Bad Paraphrase
Method
Repetition
Entrainment
Dullness
Avg.
DIRECTÈVAL
71.9
79.9
91.9
82.1
97.5
88.2
PAIREVAL
98.4
97.0
95.1
96.2
97.1
96.2
(b) (cont'd) Evaluation results with different adversarial attack types.
Table 7: Accuracy of Different Metrics on Adversarial Attack The accuracy is higher if the original response receives a higher score than corrupted one by metric. Thee higher score for each column is highlighted in underline.
BLEU
BLEURT
USR
DEnsity
DirectÉval
PairEval
3.0
6.8
6.7
0.0
0.462
0.30
0.25
0.75
1.00
030 825
2.50
675
0.0
0.5
8:00 025
0.75
2000
0.0 0.25
0.5 615
2.00
0.23 850 0.75 1.00
Figure 3: Scatter plots between human judgments and metric scores on the DailyDialog- Grade dataset. Each point indicates a response, and the and y values of each point indicate denote human and metric scores, respectively. We add noise sampled from (0, 0.03?) ) to human scores for better visualization. The red line indicates a linear regression.
dataset proposed by Khalid & Lee (2022). The dataset is created by manipulating the original human-written response into various corruption strategies. Each evaluation example in the dataset consists of (1) dialogue history, (2) original response, and (3) corrupted response. The metric should assign a higher score to the original response than the corrupted one. For experiments, we compare the performance of PAIREVAL and DIRECTEVAL. PAIREVAL com- pares two responses original and corrupted - by locating them in a single input text, while DIRECTEVAL evaluates two responses individually. As shown in Table PAIREVAL achieves higher accuracy in most attack types, confirming its robustness on dialogue evaluation.
4.3.6 Qualitative Results
Metric Visualization with Scattered Predictions Fig. 3 presents a prediction of selected metrics along with human judgments in the BailyDialogGRRAD dataset. The x and y axes of each point represent human judgments and metric scores, respectively. Among the baseline metrics, reference-based metrics like BLEU and BLEURT fail to assign discriminated scores as human ones. Reference-free metrics like USR and DirectEval show a positive correlation with human judgments, PAIREVAL shows strong correlation with human evaluations.
Case Study Table 8 presents two selected examples from meta-evaluation datasets and their evaluation results automatic metrics and human annotators. For both examples, EvIREVAL. evaluates generated responses closely to human scores.

---

Published as a conference paper at COLM 2024
Children's books require writing and more illustrations. would be worth it for
extra 500 copies though
History
Haha agrre with you. Do you ever buy stuff from Amazon? do you know why Jeff Bezos
put clock into mountain lol
Reference
A: I buy things from Amazon all the time. lan't that the clock that's supposed to run for 10,000 years
Response
A: have not, what is about?
Evaluation
Human 3.0(0.56)
BLEU-2
BERTScore 0.84(0.82)
FULL 0.76(0.72)
UniEval 0.76(0.72)
DirectEval 0.99(0.78)
PairEval 0.47(0.65)
Score(Rank)
0.0(0.95)
A: What do you think about the equipment our company?
History
have to equip our company with a network of work stations at every office and show room
possible.
Reference
A: I also realized this issue. And office automation is essential.
Respanse
How you ensure quality?
Evaluation
Human
BLEU--
BERTScore
FULL (.67()
UniEval 0.61(0.71)
Directlival 0.98(0.28)
Pairfival 0.33(0.2)
Score(Rank)
3.9(0.09)
0.0(1.0)
0.86(0.36)
Table 8: An example of TopicalChat-USR and HaallyDiallg-rrrra Grade datasets with evaluation results of selected metrics. We also report the rank score of each metric inside the parentheses, where the rank of the metric score is divided by the total number of evaluated examples.
Conclusion
In this work, we propose PAIREVAL, new comparative evaluation paradigm for assessing the quality of individual responses by considering their relative quality against a few comparison examples. encourage moderate-size and open-source LLMs to be specialized for pairwise comparison. Experiments on multiple evaluation benchmarks demonstrate that PAIREVAL correlates well with human judgments, confirming its effectiveness and validity. Although effective, PAIREVAL inevitably introduces multiple LLM calls for single evaluation. This problem would be amplified as we use a larger set of comparison examples. Our future work should address efficient way to find an optimal and small number of comparison examples.
Acknowledgement
We highly appreciate the reviewers' insightful comments on our manuscript. This work was supported by Institute for Information & communications Technology (rOTotion(IITP) grant funded by the Korea government(MSIT) (No.RS-2019-I11900
Artificial Intelligence Graduate School Program(KAIST) and the National Research Foundation of Korea(NRF) grant funded by the Korea governmen((MSIT) (No. NRF-2022R142255000009 and No.
2022R1A5A7083908)
References
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXio preprint arXiv:2204.05882 2022.
Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, et al. Fairness in recommendation ranking through pairwise comparisons. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 2212-2220 2019.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language foolkit. O'Reilly Media, Inc.", 2009.
Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadace. Elo uncovered: Robustness and best practices in language model evaluation. Xrv preprint arXit:2311.172295 2023.
10

---

Published as a conference paper at COLM 2024
Maxime De Bruyn, Ehsan Lotfi, Jeska Buhmann, and Walter Daelemans. Open-domain dialog evaluation using follow-ups likelihood. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin- Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedinss of the 29th International Conference on Compu- fational Linguistics, pp. 496-00, Cyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https: hta///aaaaaahllleee wwwp:/gg.....gg/.// coling-1.400
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Compression, transduction, and creation: A unified framework for evaluating natural language gen- eration. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Melhods in Natural Language Processing, pp. 7580-7605, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. http//2eeemmmnmmmmeeeee https://aclantholl/
URL
org/ 2021. emnlp sannp--main.. mamai.n.... 599.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Adoances in Neural Information Processing Systems, 36, 2024.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/vb N19-1423. URL https: / /aclanthooooyyyyy....occcoooolaa org NN1 1423..
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, al. The second conversational intelligence challenge (convai2). In The NeurIPS'18 Competition, pp. 187- 208. Springer, 2020. 2020.
Johannes Fürnkranz and Eyke Hüllermeier. Pairwise preference learning and ranking. In European conference machine learning, pp. 145-156. Springer, 2003.
Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. Better automatic evalua-
tion of open-domain dialogue systems with contextualized embeddings. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pp. 82-89, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/vi / URL https: http://aaaaaanatoaaaeee org/ -w 2-231
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. Topical-Chat: To- wards Knowledge-Groundee Open-Domain Conversations. In Proc. Interspeech 2019, pp. 1891-1895, 2019. doi: 10.21437 10.21437/Inters Inntesspeech2019-33773 URL http: // /xx dww..xoooo.ooolo..ooooo 21437/Interspeen 2019-20122000000 3079.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Prakhar Gupta, Yulia Tsvetkov, and Jeffrey Bigham. Synthesizing adversarial negative Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3867-3883, Online, August 2021. URL
responses for robust response ranking and evaluation.
Association for Computational Linguistics. doi: 10.18653/ v12332vv1222/// http///////222....nnnna....3333 org/ 2021 http:/2222.....nsssnsefe.e findings-acil 338.
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey Bigham. Instructdial: Improving zero and few-shot generalization in dialogue through instruction tuning. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, p0.505-15 505-525, 2022.
11

---

Published as a conference paper at COLM 2024
Edward] J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. GRADE: Auto- matic graph-enhanced coherence metric for evaluating open-domain dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2230-9240, Online, November 2020, Association for Computational Lin- guistics. doi: 10.18653/ /v155/1v/// 1tt///20....mmmmaaiie. 2020. emw.emmannmm..n..mmm 742.
URL https: httpaaaaaaaaat......a.... org/
Maurice G Kendall. Rank correlation methods. new york: Hafner, 1955. Manuscript received 3/30, 65, 1955.
Baber Khalid and Sungjin Lee. Explaining dialogue evaluation metrics using adversarial behavioral analysis. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5871-5883, Seat- tle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022.naacl-main m3..
Linguitt
URL https:://///s http:///aaaaaallgaaeeee org/ 2022. naacl iataiii.4400 430.
Nyoungwoo Lee, ChaeHun Park, Ho-jin Choi, and Jaegul Choo. Pneg: Prompt-based negative response generation for dialogue response selection task. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Pp. 10692-10703, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https:tt////m.. / aaclllllll anthologyy org. /trg22000 2022 emn emnlp-main mamm.inn.. 733..
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 986-995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL https:
/ aclallaaaa wwc.annnhllogyy. org/ 10///1177 I17-1093
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summa- rization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computa- tional Linguistics. URL htt /aal:///cllaat hol oollloy...o.. /000///044000 1004-1033,
Yen-Ting Lin and Yun-Nung Chen. L.M--val: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. In Yun-Nung Chen and Abhinav Rastogi (eds.), Proceedings of the 5th Workshop on NLP for Conver- sational AI (NLP4ConoAl 2023), pp. 47-58, Toronto, Canada, July 2023. Association 10000ooailp.....111...... URL https:
for Computational Linguistics. doi: //aclanthology. wwwpoogg.o...gggeeeeoe 2023 m1p4convaii11.5
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6826-6847, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653 v26652vv12022.211 http:////2....nnsseeeemme finnings-eennp..p. 508. URL https: http://aaaaaaaaaaaaaa/aa org 200/222...222222 www.nniinggnnnnmmm 1tt..00..........g.ge........... 508.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsuperviss insupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on
Empirical Methods in Natural Language Processing, pp. 2122-2132, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/ 20118633vv1/DD/ /t///v11/Dv/////// URL https: / //cll antho .rrglorrg/iii 6-
12

---

Published as a conference paper at COLM 2024
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zer,, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint ar XrXie:1907.11692, 2019.
Adian Liusie, Potsawee Manakul, and Mark Gales. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume Long Papers), pp. 139- 151, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL https:
org/ 2024. eacl-long.........
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Connerencc on Learning Representations, 2018.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. Towards an automatic Turing test: Learning evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1116-1126, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653 /v1 /v/1//111// URL https: https////aaaaattlogg...o.o
11033
Shikib Mehri and Maxine Eskenazi. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 681-707, Online, July 2020a. Association for Computational Linguistics. doi: 10.18653. v1 / 2200//200clmmmaaai 2020.acl-10 in.in.. 64.
URL https: / /acclacll/ anthology. org/
Shikib Mehri and Maxine Eskenazi. Unsupervised evaluation of interactive dialog with dialogpt. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 225 235, 2020b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for
automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 003115511070083.07731 URL Prgg/P2220100
https: https/://aaaattthhhaeee
ChaeHun Park, Eugene Jang, Wonsuk Yang, and Jong Park. Generating negative samples by manipulating golden responses for unsupervised learning of response evaluation model. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1525-15555 Online, June 2021. Association for Computational Linguistics. doi: 10.18653 /v133vll///v /t////222222aa/ naacl-main.1 120. URL https:tt..... anlaarhooll
ogy ogg.llloooooooo.oo....oo 2022 naacl maai i 120.
ChaeHun Park, Seungil Lee, Daniel Rim, and Jaegul Choo. DEnsity: Open-domain dialogue evaluation metric using density estimation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL. 2023, Pp. 14222- 14236, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 http/////222//////// findings-acl..8% http://////222..nnnnnsdl....
URL https: / aacaaaaaanooooooooceeeaoa org/2023.
findings www.innnsscc.... acl.896.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Adounces in Neural Information Processing Systems, 34: 4816-4828, 2021.
13

---

Published as a conference paper at COLM 2024
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. Large language models are effective text rankers with pairwise ranking prompting. arviv preprint (rXio22366117633 2023.
Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. Transactions of the Association for Computational Linguistics, 8:810-827, (200-827,,0 2020. org 2020 tac1-1..2.
doi: 10.1162 a 00347. URL https: /ttttp::/aacaaathtoooo
Shiki Sato, Reina Akama, Hiroki Ouchi, Jun Suzuki, and Kentaro Inui. Evaluating di- alogue generation systems via response selection. In Dan Jurafsky, Joyce Chai, Na- talie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 593-599, Online, July 2020. Association for Computational Linguistics. doi:
htt////1112020llll///1 aaccantoolll ogy. wegy.org... wwt.orgg//000 2020. acl -aamminnn i..in..5.5.. 55.
URL https:
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892 Online, July 2020. Association for Computational Linguis- URL
tics. doi: 10.18653 v11 v1/l////22002 a0l...00l0...iin.....maaaa acl-maing 7.4.
ogy.og/2000
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint #XXiv:2307.09288 2023.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926 2023.
Jun Xu, Zeng Wei, Long Xia, Yanyan Lan, Dawei Yin, Xueqi Cheng, and Ji-Rong Wen. Reinforcement learning to rank with pairwise policy gradient. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 509-518, 2020.
Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. comprehensive assessment of dialog evaluation metrics. In Wei Wei, Bo Dai, Tuo Zhao, Lihong Li, Diyi Yang, Yun-Nung Chen, Y-Lan Boureau, Asli Celikyilmaz, Alborz Geramifard, Aman Ahuja, and Haoming Jiang (eds.), The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pp. 15-33, Online, November 2021. Association for Computational Linguistics. doi: 10.18653/
000023/v1/222111/ v112221.eanncs-1..
URL https: http://llaaanttcocc
org/ 2021 eancs-1.3.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyoung Park. GPT3Mix: Leveraging large-scale language models for text augmentation. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2021, pp. 2225-2239, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: (0.18653/v1/200 /tt/////////222.nnnnnnneeee///// findings-ennlp.199 192. URL https: /aa/aalannttt antholl wwwethhhooy......o. org/ 2021. f inndingg-emrn emn 1p p. 192.
Jerrold H Zar. Spearman rank correlation. Encyclopedia of Biostatistics, 2005.
Chen Zhang, Yiming Chen, Luis Fernando D'Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. DynaEval: Unifying turn and dialogue level evaluation. In Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), Pp. 5676-5689, Online, August 2021. Association for Computational Linguis- 441. URL https: / /acclanthooogyy wwwpehholoo..rrr//222222222 org/2021.
tics. doi: 10.18653/v1. acl- acl-long. 441.
14

---

Published as a conference paper at COLM 2024
Chen Zhang, Luis Fernando D'Haro, Yiming Chen, Malu Zhang, and Haizhou Li. A com- prehensive analysis of the effectiveness of large language models as automatic dialogue evaluators. arXiv preprint ar arXfo:2312.15407, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics: System Demonstrations, Pp. 270-278, 2020.
Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. Designing precise and robust dia- logue response evaluators. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 26-33, Online, July 2020. Association for Computational Linguistics. doi: 20111665//122220aaa/ 2020. acl-main.4.. 4.
URL https: /aw.aaaannnnno.....a.... org/
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Bric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng
Ji, and Jiawei Han. Towards unified multi-dimensional evaluator for text generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, PP. 2023-2038, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ v1 21/2022.emnlpp 131. URL https: btttps://alaaaee /ww.aaaaaaaanoo...../oa 2022 emnlp-main
131.
15

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Baseline Details
We present further implementation details for baselines.
BLEU-2 (Papineni et al., 2002): We use BLEU-2 score implemented in Natural Language Toolkit (NLTK) library (Bird et al., 2009).
BLEURT (Sellam et al., 2020): We use an Elron//leuutttinyy---- face Hub.2
in Hugging-
BERTScore (Zhang et al., 2019): We use an official implementation" by the authors with RoBERIa-large (Liu al., 2019).
FED (Mehri & Eskenazi, 2020b): We use an official implementation" by the authors with DialoGPT (Zhang al., 2020).
CTC (Deng et al., 2021): We use official models and implementation by the authors.
USR Mehri & Eskenazi (2020a): We use USR-Retrieval for the overall experiments. For results in TopicalChat-USR and ConvAI2-USR, we cite the results in Mehri & Eskenazi (2020a). For the FED dataset, we cite results reported by Yeh et al. (2021). For other meta-evaluation datasets, we BERT (Devlin et al., 2019) finetuned on a response selection task.
FULL (De Bruyn et al., 2022): We use an official implementation" by the authors.
UniEval (Zhong et al., 2022): We use use official model and implementation' by the authors.
DEnsity (Park et al., 2023): We use official models and implementation by the authors.
A.2 Additional Qualitative Results
We present more visualized predictions of selected metrics other datasets in Fig. 4, Fig. 5, and Fig. 6. From the results in DailyDialog-Zhhan Zhao dataset (Fig. 4), we observe that PAIREVAL often outputs predictions concentrated certain range (around 0.5). We believe such behaviors can be alleviated through post-hoc calibration techniques like temperature scaling (Guo et al., 2017) if necessary.
16

---

Published as a conference paper at COLM 2024
BLEU
BLEURT
UniEval
DEnsity
DirectEval
PairEval
5.5
0.4
0.0
Figure 4: Scatter plots between human judgments and metric scores on the DailyDialog Zhao dataset. The indicators are the same as Fig.
BLEU
BLEURT
UniEval
DEnsity
DirectEval
PairEval
0.2
0.5
6.4
0.2
11
525
035
676
8:00 025 155
6.50
ais
300
625
nto
035
1.00
325
250
6m
310
025
6.00
671
191
800 6.5
05t
071 170
Figure 5: Scatter plots between human judgments and metric scores on the TopicalChat-USR dataset. The indicators are the same Fig. 3.
FULL
UniEval
DEnsity
DirectEval
PairEval
0.8
52
05221002
0.25x00.22
0.26x
2,696406
m 9, 9,79
310
cás
375
0do
ads
1.00
6.00
6ās
3.50
1.00
s
162
s55
6ão ads
Figure 6: Scatter plots between human judgments and metric scores on the FED dataset. The indicators are the same as Fig. 3.
17


Reviews:
[{"review_id": "Z1ELge5YG2", "reviewer": "Reviewer_P95M", "summary": "This paper introduces a novel dialogue evaluation metric for assessing one response's quality by comparing the target dialogue with a set of different dialogues.  Specifically, the method requires the model to compare the target dialogue with each of the dialogues in the set and use the accumulated comparison score as the final evaluation for the target dialogue.  The method is based on moderate-sized open-source language models. Experiments on multiple test sets show the effectiveness. However, some statements are confusing and there are a few questions about the method.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thorough review and positive support.\n\n**W1**\n\nOur metric is designed on top of the assumption that relative judgments often yield more consistent and reliable assessments than absolute judgments. Recent studies with comparative evaluation support our assumption (Zheng et al. 2023;  Liusie et al. 2024). When adopting such a comparative approach for dialog evaluation, the straightforward way is to utilize multiple responses for the target context, which may not always be available. Our metric makes a detour by comparing the target response against the ones derived from other dialogue contexts.\n\n**W2**\n\nWe also recognize that PairEval still requires more computational cost than the pointwise evaluation approach. However, we enabled PairEval to conduct a reliable evaluation with only a few comparisons rather than an exhaustive one. Regarding the fine-tuning cost, we kindly note that this training is only performed once before the evaluator's deployment and is not required after that. In other words, as the number of evaluated samples increases, the computational benefits from not performing exhaustive comparisons will exceed the initial fine-tuning cost.\n\n**W3**\n\nWe note that human-crafted responses are required during the training process rather than at evaluation time. This benefit of PairEval eliminates the need for human-crafted data during evaluation. We will further clarify these benefits in our camera-ready version.\n\n**Q1** \n\nWe believe the evaluation results on TopicalChat-USR and FED in Table 3 can address your concern, where the model trained on the DailyDailog dataset is evaluated on other datasets. \n\n**Q2**\n\nThe key difference between PairEval and previous metrics lies in its comparative assessment process to elicit the final judgments. Please note the superiority of PairEval does not solely originate from adopting the large model. PairEval performs better in evaluation datasets and adversarial responses than DirectEval, which uses the same LM and different assessment processes.\n\n**Q3**\n\nThe limited improvements of exhaustive comparison can be attributed to the aggregation strategy. As the number of comparisons increases, the impact of a new one on the overall result diminishes, leading to a saturated score. Our future work will devise more advanced strategies like de-emphasizing unreliable comparison outcomes.\n\n**Q4**\n\nTable 3 reports the averaged scores with different sets, while Table 6 reports the single run.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "2.1 The paper is generally well-written and easy to understand.\n\n2.2 Extensive experiments are conducted and the results show the effectiveness of the method.", "reasons_to_reject": "3.1 This paper needs to explain more deeply why this comparative method would lead to better evaluation results. Specifically, why comparing with other dialogue can help the model to evaluate dialogue quality? \n\n3.2 On page 2, the authors state \"However, they find that exhaustive comparison between candidate responses is needed for reliable quality estimation of dialogues, resulting in an undesirable computational overhead.\". This is why they use moderate-sized open-source models, but the method in this paper requires fine-tuning the model and multiple calls to the model for one evaluation. The undesirable computational overhead still exists. A computation cost comparison between the LLM-based method and this paper should be conducted.\n\n3.3 On page 2, the authors state \"However, these approaches usually require exhaustive comparison operations or human-crafted candidate responses, which may not always be available.\". Since you state that human-crafted data has the above drawbacks, why do you still conduct experiments with human-crafted adversarial data (from previous research)?", "questions_to_authors": "4.1 This article uses artificially constructed adversarial data (from previous research). Does this impact the migration of the method in this article to other domains? Such as DSTC-9 data?\n\n4.2 Apart from using a relatively large 7B model, what is the essential difference between your pairwise training method and the previous method that distinguishes positive and negative examples (such as DynaEval by Zhang et al., 2021)? \n\n4.3 Why using an exhaustive comparison did not entail big improvements (Tables 3 and 4)?\n\n4.4 The experimental results are different in Tables 3 and 6 when N=1. Why the differences?", "extra_scores": {}}, {"review_id": "xHpkEUPNTx", "reviewer": "Reviewer_JUxr", "summary": "The paper proposed a method *PairEval* for dialog response evaluation.\n\n*PairEval* lies between commonly used reference-based metrics and reference-free metrics. The core idea and novelty is to make the assessment decision (1) dependent on some baseline examples, while (2) avoiding the direct use of a reference response of the conversation of interest.\n* The first design makes the assessment results more discriminative, especially when all evaluation examples use the same set of baseline examples, as is done in this work.\n* The second design alleviates the influence of the one-to-many mapping problem in open-domain dialogs.\n\nThe whole idea is great. Experiment settings are mostly reasonable and comprehensive.\n\nThe only problem is that the proposed method *PairEval* does not improve much over the actual baseline *DirectEval*.\n* Although the authors also compare it against a number of baseline metrics from prior works and *PairEval* is much stronger, the baseline *DirectEval* also outperforms other baselines a lot. It is fair to say that most of the improvement comes from using llama-2-7b-chat as the base model, and the gap between *PairEval* and *DirectEval* is less obvious.\n* To be more specific, firstly *PairEval* and *DirectEval* have close Spearman's $\\rho$ (63.1 vs. 61.8). Secondly, the larger gap of Pearson's $r$ (58.5 vs. 49.8) is likely to be largely reduced if we use another simple baseline which is a fine-tuned llama-2-7b-chat that regresses the output score.\n\nOverall, the proposed method is promising and the results are positive, but I would expect more performance gain from the method. I suggest considering the following to make this work stronger.\n* Make a more delicate design on the choice of baseline examples. Since all evaluation examples will share a small set of baseline examples, curating the baseline example set is not going to be very costly. Additionally, maybe we can get aspect-based evaluation and interpretability as a bonus by carefully choosing baseline examples.\n* Include a fine-tuned regression model based on llama-2-7b-chat as a baseline.\n* Analyze how the choice of evaluation dataset causes very different experimental results. For example, *PairEval* outperforms *DirectEval* significantly on *DailyDialog-GRADE* but it is close to or underperforms *DirectEval* on other datasets.\n\nA minor suggestion is to include the results of non-exhaustive comparison in Table 4 for easy reading.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thorough review and positive support.\n\n**Weaknesses** \n> Marginal improvement over the actual good baseline (DirectEval). Lack of investigation into the performance discrepancy on different datasets.\n\nWe recognize that the performance of PairEval and DirectEval is sometimes mixed. Furthermore, the meta-evaluation datasets used in our work have quite different characteristics, leading to varied tendencies across different metrics. We think Tables 6 and 7 in Section 4.3.5 can address your concern, where PairEval is superior to penalizing adversarial responses compared to DirectEval. Moreover, incorporating your valuable suggestions, including advanced training algorithms or comparison example selection strategies, will enhance PairEval's performance.\n\n**Suggestion 1**\n> Make a more delicate design on the choice of baseline examples. Since all evaluation examples will share a small set of baseline examples, curating the baseline example set is not going to be very costly. Additionally, maybe we can get aspect-based evaluation and interpretability as a bonus by carefully choosing baseline examples.\n\nAs you pointed out, our current work does not delve into choosing the optimal comparison examples for better correlation. Instead, we conducted experiments with quite simple and easily replicable comparison example selection strategies (e.g., Random or Test in Section 4.3.2). We believe the essential future step to advance PairEval is finding better comparison examples by considering factors like target examples and LM’s initial prediction. We appreciate your insightful suggestions.\n\n**Suggestion 2**\n> Include a fine-tuned regression model based on llama-2-7b-chat as a baseline.\n\nWe appreciate your dedicated comments! Similar to your suggestions, during our preliminary study, we trained LLM with different levels of appropriateness labels (i.e., positive:1, human-written adversarial: 0.5, and randomly sampled: 0), inspired by Ye et al. (2021). However, this training objective does not always show a better correlation, so we decided to use only discrete label space. Regarding these results, we suspect constructing a more nuanced label space or dynamically adapting its value with external feedback can be a meaningful direction. We acknowledge your helpful comments on improving our work.\n\n[References]\n\nYe et al. Towards Quantifiable Dialogue Coherence Evaluation. ACL2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A novel dialog response evaluation method that combines the advantages of reference-free and reference-based metrics while avoiding their demerits.\n* Mostly comprehensive experiments and analyses.", "reasons_to_reject": "* Marginal improvement over the actual good baseline (*DirectEval*).\n* Lack of investigation into the performance discrepancy on different datasets.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HyyRdhTFZQ", "reviewer": "Reviewer_qHB5", "summary": "The authors propose a new metric to evaluate open-domain dialogue via pairwise comparisons of responses in different dialogues. They use an LLM to judge the system's response along with a number of randomly sampled utterance-response examples (with different context) and use the scores to calculate a final score for the system response.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thorough review and positive support.\n\n**W1**\n\nWe recognize that utilizing human-written examples can harm the adaption of our method to a new domain or dataset. To tackle this, we can consider replacing human annotators with an LLM for a more efficient data acquisition process. We also think the results on TopicalChat-USR and FED datasets in Table 1 can alleviate your concerns, as PairEval shows superior performance even when the training and evaluation datasets are not exactly matched.\n\n**Q1**\n\nWe believe that our work's key contribution lies in proposing a new comparative evaluation paradigm for assessing the quality of individual responses. When determining the quality of a single target response, we propose to consider its relative quality compared to responses derived from other dialogs. Although our current work adopts a fine-tuning approach to specialize LMs for comparative operations, more powerful LMs may not require a training process. \n\n**Q2**\n\nOur manuscript mainly focuses on evaluating the overall quality of a response rather than its various aspects. Suppose we extend PairEval to assess responses in such fine-grained aspects. In that case, comparison responses of varied quality from different aspects can be utilized, followed by a proper finetuning strategy to adapt PairEval into evaluation from diverse perspectives.\n\n**Q3** \n\nWe do not put much effort into crafting the LLM prompt in PairEval for a fair comparison and reproducibility. If the LLM should be changed into new ones, we believe one can use mostly the same prompt as the original one.\n\n**Q4**\n\nAlthough PairEval is mainly designed for open-domain dialogue evaluation, however, with proper adjustment,  we believe that our comparative evaluation approach can be extended to other tasks. To achieve this, we should define desirable characteristics in model outputs by considering the nature of the tasks of interest. For instance, in coding, one may focus on executability, accuracy, and reliability. Based on this consideration, we can construct comparative examples with varied levels of quality from diverse perspectives.\n\n**Q5**\n\nSince our manuscript focuses on the proof-of-concept of the comparative approach for dialogue evaluation, we did not conduct experiments with other LLMs or different datasets. The experiments were designed to utilize open-sourced LLMs and publicly available datasets. Our future study can be extended to leverage more powerful resources.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting methodology to judge open-ended responses that can be useful to the community\n\nThe paper is clear and easy to replicate\n\nThe metric is thoroughly evaluated including against human raters, outperforming metrics from related work (except LLMEval)", "reasons_to_reject": "May not be very generalizable since it requires training (e.g. the authors use human-written negative responses)", "questions_to_authors": "If you need to fine-tune a model for Pairwise evaluation, how different is this to reference-free / LLM as a judge approaches? I understand the need to bypass proprietary LLMs but we now have access to very powerful open-source LLMs that can be used as judges. It would be nice to see a comparison against PairEval.\n\nHow do you handle topic switches, humour, sarcasm, or other cases where the response may not seem to match but could be actually good, funny, etc?\n\nHow difficult is it to design the LLM prompt (Section 3.2)? If one needs to replace the LLM with a newer / different model, how much effort is it to re-design the prompt?\n\nDo you think this approach could work on other domains e.g. math or coding? What adjustments would need to be made?\n\nGiven that your metric needs to be trained, have you tried using a larger LLM or training with more data?", "extra_scores": {}}, {"review_id": "o1Heb9nR3H", "reviewer": "Reviewer_HoEo", "summary": "The authors introduce PairEval a dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations. For this purpose they fine-tune a LLM to conduct the specific pair-wise dialogue evaluation task proposed. Evaluation results show better correlation with human evaluations than other metrics used in the evaluation. I think this paper presents an interesting idea and pushes forward the state of the art in open dialogue evaluations. COLM participants and the research community will benefit from the ideas and results presented here. However, I still feel additional work is needed to get a much better assessment of the benefits of the proposed technique. For instance, it would be interesting to understand the range of variability of achieved results by changing the comparative examples used in the evaluations, as well as the training data used to fine-tune the LLM. How much semantic similarity and/or topic overlap are there between the data used for fine-tuning and the evaluation data sets? Also, how human evaluations correlate to other human evaluations? I think an opportunity is missed here to understand better the complexity of the task for humans and get a better sense on how far or how close the proposed technique is bringing us to human evaluations.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thorough review and positive support. We hope our responses address all the concerns.\n\n**Question 1**\n> It would be interesting to understand the range of variability of achieved results by changing the comparative examples used in the evaluations, as well as the training data used to fine-tune the LLM.\n\nWe acknowledge that it is an important analysis to investigate the changed performance of PairEval with different comparison examples. For instance, in Figure 4, PairEval can make predictions concentrated on a specific range if most evaluated responses are considered lower quality than their compared response. Our future work should present a more dedicated comparative example selection strategy to reduce the number of comparisons and increase the correlation with human judgments.\n\n**Question 2**\n> How much semantic similarity and/or topic overlap are there between the data used for fine-tuning and the evaluation data sets? Also, how human evaluations correlate to other human evaluations? I think an opportunity is missed here to understand better the complexity of the task for humans and get a better sense on how far or how close the proposed technique is bringing us to human evaluations.\n\nAmong the meta-evaluation datasets used in our experiments, TopicalChat-USR and FED can be regarded as different datasets from the ones used in fine-tuning (i.e., DailyDialog and ConvAI2). Here, evaluation metrics for TopicalChat are required to consider not only a dialogue history and response but also background knowledge from external sources like Wikipedia.\nAmong the meta-evaluation datasets used in our experiments, TopicalChat-USR and FED can be regarded as different from those used in fine-tuning (i.e., DailyDialog and ConvAI2). Here, evaluation metrics for TopicalChat should consider not only a dialogue history and response but also background knowledge from external sources like Wikipedia.\n\nRegarding correlations within the human annotators, we can refer to inter-annotator agreement scores reported by meta-evaluation datasets. For instance, Zhao et al. (2020) released a meta-evaluation dataset called DailyDialog-Zhao, and inter-annotator agreement is measured by Kripendorff's alpha, with a result of 0.8. We will include more details about the inter-annotator agreement scores of each dataset in our camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think this paper presents an interesting idea and pushes forward the state of the art in open dialogue evaluations. COLM participants and the research community will benefit from the ideas and results presented here.", "reasons_to_reject": "I would probably want to see more exploratory analysis on the items described in the questions to authors, although I do not feel this is actually a reason to reject, given that this is a conference paper.", "questions_to_authors": "It would be interesting to understand the range of variability of achieved results by changing the comparative examples used in the evaluations, as well as the training data used to fine-tune the LLM. How much semantic similarity and/or topic overlap are there between the data used for fine-tuning and the evaluation data sets? Also, how human evaluations correlate to other human evaluations? I think an opportunity is missed here to understand better the complexity of the task for humans and get a better sense on how far or how close the proposed technique is bringing us to human evaluations.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new metric for evaluating open-domain dialog quality via paired comparison to a different dialog, showing higher correlation with human judgments of quality compared to several reference-free and reference-based metrics.

The idea of an in-between evaluation metric that uses some baseline, but not from the exact conversation context, is valuable. Additionally, the reviewers found the experiments to be comprehensive. However, the main concern is that the majority of the improvement seems to come from the specific language model used rather than the pairwise design choices, which is evident in comparing directEval and PairedEval. Since the core contribution of the paper is "a new comparative evaluation paradigm," as the authors put it, this limits the applicability of this work. While results in general setups are quite mixed when comparing directEval and PairedEval, PairedEval seems to perform slightly better in an adversarial setup.
Decision: Accept