RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue
Year: 2023

Abstract:
Evaluating open-domain dialogue systems is challenging for reasons such as the one-tomany problem, i.e., many appropriate responses other than just the golden response. As of now, automatic evaluation methods need better consistency with humans, while reliable human evaluation can be timeand cost-intensive. To this end, we propose the Reference-Assisted Dialogue Evaluation (RADE) approach under the multi-task learning framework, which leverages the pre-created utterance as reference other than the gold response to relief the one-tomany problem. Specifically, RADE explicitly compares reference and the candidate response to predict their overall scores. Moreover, an auxiliary response generation task enhances prediction via a shared encoder. To support RADE, we extend three datasets with additional rated responses other than just a golden response by human annotation. Experiments on our three datasets and two existing benchmarks demonstrate the effectiveness of our method, where Pearson, Spearman, and Kendall correlations with human evaluation outperform stateof-the-art baselines.

heading: 1 Introduction
text: Open-domain dialogue system, which focuses on non-goal-oriented chitchat, may converse on a broad range of arbitrary topics. Recent years have witnessed rapid advances in natural language generation (Zhang et al., 2019b; Roller et al., 2021; Zhao et al., 2023), boosting the development of opendomain dialogue systems. Conversations with such systems resemble human-human interactions as various responses might fit the context, given that users often do not have a specific goal beyond enjoying the conversation. Evaluating these conversations is thus challenging because of the socalled one-to-many problem (Chan et al., 2021; Ji et al., 2022); see Figure 1 where three candidate
∗ Corresponding author.
responses with different semantics fit the context while there is only one golden response.
The most common practice of dialogue evaluation is done with reference-based metrics, which compare the generated response with a pre-created response, commonly referred to as the golden standard (Ji et al., 2022). The reference-based metrics calculate the similarity between the generated and gold responses at either lexical level (e.g., ROUGE (Lin, 2004), BLEU (Papineni et al., 2002)) or semantic level (e.g., BERTScore (Zhang et al., 2019a), ADEM (Lowe et al., 2017)). However, these metrics ignore the one-to-many nature of open-domain dialogues. As illustrated at the bottom of Figure 1, the generated response “Amazon is good but expensive ...” expresses the opposite semantics to the golden response “I shop online...” and is therefore considered a non-good response by the reference-based metrics. Therefore, these metrics may need a higher consistency with humans. Recently, multi-reference methods and referencefree methods are proposed to address the drawback of reference-based metrics. The former explicitly annotates multiple references for dialogue (Eric et al., 2021), whereas the latter discards the golden response in the evaluation and achieves high cor-
12856
relations with human judgments (Mehri and Eskenazi, 2020c; Huang et al., 2020). However, drawback still exists in these two classes of methods. Multi-reference methods are costly and hard to generalize to different datasets, while referencefree methods are often unstable and vulnerable to data-induced biases1.
To overcome the weakness of existing evaluation methods and further resolve the one-to-many problem, we propose a new technique, namely Reference-Assisted Dialogue Evaluation (RADE). RADE considers the pre-created response as a reference instead of the golden standard.
To support RADE, we design a new human annotation task to extend existing datasets, which includes metric decompose and pairwise annotation, where a pre-scored golden response is paired with generated responses for rating following a unified rating score. The final scores are arrived at by aggregating ratings with a weighted sum from different sub-metrics. The human annotation collects labels for three high-quality datasets with 10,112 dialogues, which correspond to three downstream open-domain dialogue system tasks, i.e., chitchat, empathetic dialogue, and personal chat. These multi-domain datasets make RADE more robust when generalizing to cross-domain evaluation scenarios while having a better task-specific performance.
We propose a RADE model under the multitask learning framework for automatic evaluation based on the newly collected datasets. Specifically, RADE first explicitly encodes the relation between dialogue context and generated response with reference assistance. Then RADE discriminates whether the reference or response fits the context better and predicts the scores for each utterance. To relieve the one-to-many problem, we augment RADE with a joint response generation task where RADE learns to generate the reference responses to better perceive the range of candidate responses.
Extensive experiments on our three benchmarks demonstrate that RADE achieves the best correlations with human judgment. We also examine two existing USR benchmark (Mehri and Eskenazi, 2020c) where RADE outperforms the state-of-the-
1The data-induced biases included two aspects: (1) Noise collected in data/annotations, (2) The reference-free models tend to favor the underlying models’ outputs and those from similar models or trained with similar datasets. (Khalid and Lee, 2022; Deutsch et al., 2022)
art methods, e.g., pushing the Pearson correlation coefficient to 48% (6.8% absolute improvement) and Spearman correlation coefficient to 46.6% (4.3% absolute improvement). Experiments also verify the generalizability of our proposed method.
Our contributions can be summarized as follows: (1) We propose the reference-assisted evaluation method, i.e., RADE, for open-domain dialogue evaluation; (2) We design a new human annotation task and collect three new dialogue evaluation datasets; (3) Experiments on our benchmarks and two existing benchmarks verify the effectiveness and robustness of the proposed methods; (4) We release three new benchmarks and the pre-trained evaluation model to facilitate future research on dialogue evaluation.

heading: 2 Related work

heading: 2.1 Reference-based dialogue evaluation
text: Previous reference-based methods compare the generated response with the pre-created response at the lexical or semantic level. Lexical-level metrics, e.g., ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005), count the n-gram overlap between the candidate response and the reference response. These methods usually correlate poorly with human evaluation results due to the lexical mismatch problem (Liu et al., 2016). Semantic-level metrics evaluate address lexical mismatch problem by calculating similarity with high-dimension embeddings. For example, Sharma et al. (2017) measures the embedding distance between golden and generated response. Ghazarian et al. (2019) and Zhang et al. (2019a) enhance the text representation using the large pretrain model, which has shown exemplary performance in capturing semantic similarity. However, they suffer from the one-to-many problem when evaluating open-domain dialogues since responses with various semantics may fit the dialogue context.
Recent works tend to relieve this drawback by annotating multiple references for dialogue, commonly referred to as multi-reference methods (Li et al., 2017; Sai et al., 2020), which are costly and hard to generalize to agnostic scenarios. The proposed RADE aims to consider the pre-created response as a candidate instead of the golden standard to address the one-to-many problem of dialogue evaluation.

heading: 2.2 Reference-free dialogue evaluation
text: The reference-free methods are gaining more attention as they correlate more with human judgment only with the dialogue context and response. For example, MAUDE predicts the score of dialogue using pre-trained language models, GRADE (Huang et al., 2020) evaluates the coherence of dialogues with the augmentation of the commonsense graph, EMS (Chan et al., 2021) enhances the dialogue evaluation by capturing the representation of the context and response in latent space. Some methods further decompose the evaluation of responses into multiple perspectives (Mehri and Eskenazi, 2020a,c; Phy et al., 2020), such as relevance, fluency, and engagingness, then aggregate the overall score from different sub-metrics with a weighted average. However, some recent studies (Khalid and Lee, 2022; Deutsch et al., 2022) reveal that the referencefree methods are vulnerable to data-induced biases and inherently biased toward models which are more similar to their own. In contrast, this paper proposes a reference-assisted approach, which enhances the robustness of the model using reference responses as a benchmark.

heading: 3 Task Formulation
text: In this work, we propose two tasks: (1) extending the existing datasets by human annotation, and (2) leveraging the rated references collected in (1) to enhance automatic evaluation.
Human annotation Human annotation aims to extend existing datasets with multiple rated responses to facilitate automatic evaluation. Given a dialogue context c, which is always paired with a golden response (denoted as reference) rh, we employ the generation models, e.g., BlenderBot (Roller et al., 2021), to generate one more response ra. We then assign a fixed overall score or derive from existing datasets to the reference as sh. The annotators are instructed to rate ra as sa, following the same scale while taking the reference as a benchmark. The annotators are also asked to revise the reference score sh if sh is inappropriate.
Automatic evaluation Given a dialogue context c, the proposed RADE learns to evaluate the response ra with the assistance of reference rh under the multi-task learning framework. The first task explicitly models the relation between reference and response and discriminates which fits the con-

heading: Relevance †:
text: Whether the response matches dialogue context semantically.
Engagingness†: Whether the response is engaging or interesting rather than rigid template.
Fluency†: Whether the response is fluent and natural throughout the conversation.
Understandability‡: Is there any external knowledge contained in the response.
Emotional-awareness‡: Whether the agent capture the emotion of user and support empathic support.
Personality-awareness‡: Whether the response conforms to given personality.
Table 1: Criteria in human annotation. Metrics with † are general metrics for all dialogue tasks, while metrics ‡ are metrics for specific dialogue tasks (e.g., understandability for chitchat, emotion-awareness for emotional dialogue and personal-awareness for personal chat).
text better. The scores of reference and response are predicted simultaneously. And the second task enhances the score prediction task by implicitly estimating the distribution of candidate responses.

heading: 4 Human Annotation
text: Our human annotation task aims to rate the candidate responses following a pre-scored reference as a benchmark. Since there are multiple perspectives to assess the response, we simplify by sorting the possible aspects into two categories: the general view and the task-specific view. As listed in Table 1, the former contains relevance, engagingness, and fluency, which are suitable for all dialogue agents. And task-specific criteria consist of understandability, emotional awareness, and personality awareness, which correspond to chitchat dialogue, emotional dialogue, and persona dialogue. We annotate rates on each metric and calculate the overall rating score by weighting these sub-metrics. Specifically, the weights are obtained based on the preference of users (see section A.1.3 for more details).

heading: 4.1 Data preparation
text: We consider three datasets to extend: • DSTC– ChitChat (ChitChat) (Hori and Hori, 2017), a chitchat dataset collected from Twitter, each example derived from the conversation between a customer and an agent. • Empathetic Dialogues (EmpaDial) (Rashkin et al., 2019), which consists of 25k dialogues grounded in emotional situations.
• PersonaChat (Zhang et al., 2018), a real-world dataset consisting of 10k dialogues where each participant plays the part of an assigned persona.
Then, we collect model-generated responses using the following seven well-performing dialogue models on these datasets: BlenderBot (Roller et al., 2021), DialoGPT (Zhang et al., 2019b), KEMP (Li et al., 2020b), MoEL (Lin et al., 2019), MIME (Majumder et al., 2020), EmpDG (Li et al., 2020a), PersonaGPT (Tang et al., 2021). The train-dev-test of collected datasets are split as Chitchat (1490/300/300, 5/1/1), Empathetic Dialogue (3022/500/500, 6/1/1), and Persona Chat (3000/500/500, 6/1/1). More details of these models are available in Appendix A.1.1.

heading: 4.2 Human annotation detalis
text: We hire 40 annotators for data annotation. Following a five-scale standard, they are asked to label sub-metrics as listed in Table 1. The five-scale allows the annotators to factor in their subjective interpretation of the extent of success or failure of a system’s response to satisfy a user’s request. The dialogue context, rated reference response, and corresponding score are provided in each example. At least three annotators are required for each example. We annotated about 10k dialogues for the three datasets, and the statistics of the collected datasets are listed in Table 2. The ratings achieve reasonable inter-annotator agreements with Fleiss Kappa scores of 0.540, 0.554, and 0.533 on three datasets, respectively. More details about the annotation guideline and details are provided in Appendix A.1.2.

heading: 5 Reference-Assisted Automatic Evaluation
text: We propose RADE, a Reference-Assisted Automatic Dialogue Evaluation method under the
framework of multi-task learning. Compared with reference-based methods that evaluate based on the distance between the golden and generated response, the proposed RADE explicitly discriminates whether the reference or candidate response fits the dialogue context better. To relieve the one-to-many problem, we augment RADE with a joint response generation task, which aims to perceive the range of feasible candidate responses. To improve the performance of RADE with the limited dataset, we propose a two-stage training strategy, including cross-domain pre-training and task-specific finetune.

heading: 5.1 Model architecture
text: The architecture of RADE is illustrated in Figure 2, which comprises a posterior encoder, a regression layer, and a candidate response generator.
Posterior encoder. The posterior encoder encodes the dialogue context c, reference response rh, and model-generated response ra into hidden representation. In particular, we first concatenate c, rh and ra together into X with a specific token [SEP]:
X = {c [SEP] rh [SEP] ra} (1)
Then the concatenated sequence is fed into a transformer-based encoder to get the representation H ∈ R|X|×d:
H = Encoder(X), (2)
where d is the hidden size of encoder, |X| is the length of sequence X .
Regression layer. The regression layer aggregates the representation H and predicts the scores of both reference and candidate response simultaneously. Specifically, a pooling layer aggregates the token-level representation into a sequence-level representation: h ∈ Rd×1:
h = Pooling(H) (3)
Then, a feedforward network takes h as input to predict the score of both reference and candidate response:
(ŝh, ŝa) = FeedForward(h), (4)
where ŝh and ŝa denote the predicted score of rh and ra, respectively.
Candidate response generator. To relieve the one-to-many problem, we devise a candidate response generator to perceive the range of feasible candidate responses (Chan et al., 2021). Specifically, a Transformer-based generator learns to generate reference responses autoregressively for a specific context. We first encode the dialogue context c using a encoder:
ĥ = Encoder (c), (5)
where the Encoder shares the same parameters with the posteriori encoder in Eq. (2). Then, we apply a Transformer-based decoder Decoder to model the generation probability of reference response rh:
P (rh|c) = T∏
t=1
Decoder(r (t) h |r (<t) h , ĥ), (6)
where T denotes the length of rh. Compared with the previous reference-free methods, which estimate the relation between context and response only with the knowledge acquired from their training data, RADE explicitly takes the pre-created response as a benchmark to reduce the data-induced bias when generalizing to agnostic scenarios. Moreover, different from existing reference-based methods, which use the pre-created response as the golden standard without considering the semantic diversity of the response, we relieve the one-to-many problem via auxiliary response generation tasks. The share encoder enhances the capability of context represen-
tation which augment the performance of scorepredicting task through multi-task learning.

heading: 5.2 Two-stage training
text: The neural-based model has been proven prone to data-induced bias, but it is costly to annotate a large dataset in every specific task. Therefore, we propose a two-stage strategy that includes: (1) cross-domain pre-training, and (2) task-specific fine-tuning, keeping a tradeoff of performance between in- and cross-domain. As shown in Figure 2 (right), we pre-train our model based on existing human-annotated datasets from different downstream tasks of open-domain dialogue to improve the generalizability (Ye et al., 2021a). Since the cross-domain datasets suffer from domain gaps and no pair-wised score, we finetune our model in the next stage with newly collected task-specific datasets.
Cross-domain pre-training. The pre-training datasets contain 54,438 dialogue-level examples collected from different downstream tasks, covering a wide range of domains (see more details in Table 7). For learning the coarse-grain judgment of generated response without human-annotated reference scores, our model is first pre-trained by minimizing a new cross-domain pre-training loss LCross. Concretely, the LCross is composed of scoreprediction loss and generation loss, which can be formulated as:
LCross = LMSE(ŝa, sa) + LGEN, (7)
where ŝa and sa denote the human-annotated score and the predicted score of the candidate response and LMSE(ŝa, sa) = (ŝa − sa)2. LGEN is the response generation loss, which is defined as:
LGEN = − logP (rh|c), (8)
where P (rh|c) is the generation probability of rh defined in Eq. (6).
Task-specific finetuning. We next finetune our model with newly annotated datasets to enhance the performance when evaluating task-specific dialogue agents. The optimize objective LIn is composed of score-prediction loss, generation loss, and pair-wised ranking loss, which can be formulated as:
LIn =LMSE(ŝa, sa) + LMSE(ŝh, sh)+ LGEN + LPR (9)
where LMSE(ŝa, sa) and LMSE(ŝh, sh) are MSE score-prediction loss of reference response and candidte response, respectively. LGEN is the generation loss as defined in Eq. (8). LPR is the pair-wise ranking loss defined as:
LPR = −g(sh, sa) log eŝa
eŝh + eŝa , (10)
in which g(sh, sa) is a labeling function defined as:
g(sh, sa) = { 0, sh ≥ sa 1, sh < sa
(11)
The LPR is introduced to assure that the rank order of the predicted scores satisfies the preannotated order. Compared to reference-free models that inherently favor outputs from their underlying models or those trained on similar datasets, RADE is specifically optimized to align with human intentions and effectively alleviate this bias.

heading: 6 Experimental Setup

heading: 6.1 Dataset and evaluation metrics
text: We mainly conduct experiments on the three datasets annotated in Section 4. We further evaluate the models on two existing benchmarks, USRTopicChat and USR-PersonaChat (Mehri and Eskenazi, 2020c), to examine the generalizability of our method. The evaluation metrics include Pearson (r), Spearman (ρ), and Kendall (τ ) correlation, which measures the linear relationship, monotonic relationship, and the ordinal association between
automatic evaluation and human evaluation, respectively2. We abbreviate the Pearson, Spearman, and Kendall correlation as r, ρ, and τ for simplicity.

heading: 6.2 Implementation details
text: We initialize the parameters of the encoder and decoder with BART (Lewis et al., 2019), a Transformer-based pre-trained model. BART is well-suited to our proposed model because it is capable of both text representation tasks and text generation tasks. We optimize the model using Adam optimizer with parameters β1 = 0.98, β2 = 0.97, and the learning rate of 5e−5. The model is trained up to 10 epochs, and we tune the hyper-parameters and pick the checkpoint on the development set. The training of the model can be done within 5 hours using two 2080Ti GPUs. We denote the RADE model that pre-trained on cross-domain datasets as RADE (PT), and the model that further finetuned on task-specific data as RADE (TS).

heading: 6.3 Baselines
text: We compare our method with two types of baselines: reference-based and reference-free methods.
The reference-free baselines include: DialoRPT (Gao et al., 2020a), which trained on largescale social media feedback data to predict rankingbased scores; GRADE (Huang et al., 2020), which enhances the contextualized representations via topic-level commonsense graphs and predicts the score using a regression module; FED (Mehri and Eskenazi, 2020a), an unsupervised dialogue evaluation model based on DialogGPT; UniEval (Zhong et al., 2022), which evaluates the response from multiple perspectives; QuesEval (Scialom et al., 2021), which evaluates the fact-based text using summarizing asks.
The reference-based baselines include: RUBER (Tao et al., 2017), an unsupervised evaluation metric considering the similarity of the response with dialog context and reference; BERTScore (Zhang et al., 2019a), which employs BERT to greedily match the response and the ground truth at the token level; BLEURT (Sellam et al., 2020), which is a BERT-based model pre-trained with millions of synthetic examples; BARTScore (De Bruyn et al., 2020), which weights the log-likelihood of the generated response as the score. We also test three reference-based lexicallevel metrics: ROUGE-L, BLEU-2, and METEOR.
2We use SciPy (https://scipy.org/) to calculate the scores.
Moreover, we implement two reference-based baselines, BERTMLP and BARTMLP, which are trained with the same human-annotated datasets as RADE, and provide a reasonable comparison with our proposed model. Specifically, we obtain the text representations of the dialogue using BERT or BART and then feed the representations into a multi-layer perception to calculate the scores. For a more comprehensive analysis, we also fine-tune the two strongest baselines, QuantiDCE and GRADE, on our cross-domain datasets as well as our selfcollected datasets, respectively.

heading: 7 Results and Analysis

heading: 7.1 Experimental results
text: Overall performance. Table 3 shows the experimental performance for all methods. Overall, RADE achieves the best performance in three benchmarks in terms of all metrics. Concretely, the pre-trained model RADE (PT) gets better or comparable correlation with human judgment than the best baseline method on three dialogue tasks.
The task-specific model RADE (TS), fine-tuned with the newly collected reference-assisted data, establishes a new state-of-the-art by improving the performance by about 30% on average compared to RADE (PT). For example, RADE (TS) gets r = 0.601, ρ = 0.569 in the ChitChat domain, and pushes r to 0.863 (0.314 absolute improvements), τ to 0.685 (0.287 absolute improvements) in EmpaDial domain. This result suggests that training with in-domain datasets is critical to enhancing the task-specific evaluation capability of RADE. For a more comprehensive comparison, we also train the two strongest baselines (QuantiDCE and GRADE) with our cross-domain and self-collected datasets, respectively. And the result and analysis are provided in Appendix A.2.3.
Generalizability. We find that the performance of the reference-free method varies dramatically across domains. For example, GRADE and QuantiDCE, trained in the chitchat domain, achieve high correlations with human judgment in ChitChat and EmpaDial but perform poorly in PersonaChat. The
result indicates that the contextual representation capabilities of unsupervised methods are limited by their training data and, therefore, are prone to datainduced bias, decreasing their performance when employing agnostic scenarios. In contrast, the gap between the proposed RADE (PT) methods across different domains is relatively small. These results indicate that RADE has better generalizability than reference-free methods due to the assistance of reference and the proposed cross-domain training strategy.
Results on USR benchmarks. We further examine our methods on two USR datasets (Mehri and Eskenazi, 2020c) to verify the efficiency and robustness of RADE when generalizing to existing dialogue evaluation benchmarks. The results are listed in Table 4. Experiments show that RADE, which has not explicitly trained on these datasets, achieves better or comparable results to previous supervised methods. See Appendix A.2.4 for more results and details.

heading: 7.2 Ablation study
text: We perform an ablation study to investigate the influence of different components in our methods. We examine two ablative variants: (1) w/o LPR: we remove the ranking-based loss LPR to verify its effectiveness (w/o LPR); (2) w/o LGEN: we remove the LGEN to verify training with response generation task jointly can improve the predicting correlation with human judgment.
Table 3 presents the results. Overall, the variants of our methods show a decreased performance compared to the base model. For example, Pearson drops 0.10, 0.09, and 0.07 in three benchmarks, respectively, after the LPR is removed. This result indicates that ranking-based loss can enhance performance by explicitly building the relation be-
tween response and reference. After removing the LGEN, the correlation in all benchmarks has a prominent decrease, e.g., Spearman correlation drops by 0.15, 0.10, and 0.09, respectively. The results suggest that the auxiliary response generation task improves the representation capability of our method and relieves the one-to-many problem.

heading: 7.3 Case study
text: Our case studies demonstrate that RADE is more consistent with human judgment than baselines. Details about our case studies are available in Appendix A.2.5.

heading: 7.4 Qualitative analysis
text: To explain more intuitively, we show the scatter plots against human judgments for different automatic evaluation methods (i.e., RADE, GRADE, BERTScore, METEOR) on the EmpaDial dataset in Figure 3. As shown in Figure 3 (a), our method RADE achieves a stronger correlation with human judgment than the other methods. Figure 3 (d) illustrates that METEOR scores are zero or extremely low for the most response. It results from the oneto-many nature of open-domain dialogue, and word overlapping occasionally occurs. Figure 3 (c) suggests that the BERTScore scores are mainly concentrated in the range of 0.3-0.6, indicating no significant differentiation between the different responses. Figure 3 (b) shows that GRADE achieves a better
correlation with human judgments. However, the distribution of GRADE predicted scores is concentrated in the high-scoring band, resulting in a low distinction of responses; RADE uses reference as a benchmark and thus has a more balanced distribution of predicted scores.

heading: 8 Discussions
text: The impact of the training data scale. To explore the minimum data scale required for our method, we train RADE using different amounts of randomly sampled annotated data. We observe a minor degradation in RADE’s performance as the amount of data decreases. For example, when training on 2,400 examples from the EmpatheticDialogue dataset, RADE(TS) achieves Pearman’r=0.837 and Spearman’rho=0.829; whereas with 1,200 examples, it obtains Pearman’r=0.807 and Spearman’rho=0.806. All results are averaged over three runs. Moreover, we find that RADE outperforms all baselines with only 800 training examples in three datasets, respectively.
The difference between golden and candidate Responses. Golden response refers to a scenario where there is only one correct response, and any different response is given a low score. For example, BERTScore calculates the cosine similarity between the golden and model-generated response. However, Candidate responses implies that there can be multiple correct answers, which is more flexible and human-intuitive. And RADE is optimized to align with this human intention using generative and pairwise-ranking loss. If more references are available, the RADE can consider multiple valid responses to make more reliable evaluations. To achieve this, we can concatenate model-generated responses with different references. However, due to the limitation of our datasets, we concatenate one reference and model-generated response, which are then fed to the encoder.
Employing RADE when the reference response is not available. Considering the reference is not always available in real-world scenarios, we design two alternatives to enable RADE, i.e., constructing a pseudo-reference via retrieval or generative method. We verify the two solutions on the FED dataset and the details can be found in Appendix A.3.

heading: 9 Conclusion
text: We have presented a new reference-assist dialogue evaluation (RADE) method to address the one-tomany problem when evaluating open-domain dialogue systems. RADE evaluates the response generated by open-domain dialogue agents with the assistance of reference response. In addition, we have curated the reference-assisted dialogue evaluation datasets by expanding three existing datasets via a pairwise human annotation. The extended datasets contain over 10K dialogues. Extensive experiments on three extended datasets and two existing benchmarks have verified the effectiveness and robustness of the proposed methods and their generalizability.

heading: Limitations
text: The main limitation of this paper is the need for human-labeled reference responses. We will explore automated or human-machine collaboration methods to reduce the cost of annotation in the next stage. Another limitation is that we need to explore whether other auxiliary tasks can also enhance the performance of score prediction. In the future, we also plan to reproduce the proposed method for other, less resource-rich languages.

heading: Ethics Statement
text: The paper proposes a dialogue evaluation method, which is intended to evaluate open-ended dialogue on topics such as books and movies. A new dataset is developed using some existing dialogue systems, such as DialoGPT, which are trained on large-scale web data that is known to contain biased or discriminatory content. The datasets that we trained on may also include subjective knowledge (comments on movies) that may express the bias of the writers.

heading: A Appendix

heading: A.1 Human Evaluation Details

heading: A.1.1 Details for Data Preparation
text: We first employ the generation models to generate one more response for our human annotation proposed in Section 3. The annotators are instructed to rate the newly generated responses. Specifically, we employ the following generation model:
• Blenderbot (Roller et al., 2021): Blender is a conversational agent based on the large-scale model that mainly focuses on generating personal, engaging, knowledgeable, and empathetic responses.
• DialogGPT (Zhang et al., 2019b): DialogGPT is a large, tunable neural conversational response generation model.
• KEMP (Li et al., 2020b): KEMP is an emotional dialogue agent enhanced with a knowledge-enriched context graph.
• MoEL (Lin et al., 2019): MoEL is an emotional dialogue agent based on encoderdecoder architecture. MoEL softly combines the response representation from different decoders, each focusing on one type of emotion.
• MIME (Majumder et al., 2020): MIME is an empathetic dialogue model considering polarity-based emotion clusters and emotional mimicry.
• EmpDG (Li et al., 2020a): EmpDG is a multiresolution empathetic chatbot enhanced by exploiting user feedback.
• PersonaGPT (Tang et al., 2021): PersonaGPT is a GPT2-based open-domain dialogue agent designed to generate personalized responses.
As shown in Table 5, we extend the DSTC dataset with Blenderbot and DialoGPT, the Empathetic Dialogue dataset with KEMP, MoEL, MIME and EmpDG; the Persona-Chat dataset with Blenderbot and PersonaGPT.
Since Roller et al. points out the length of the utterances is crucial to human judgments, i.e., too short responses are seen as dull, we only sample the example with at least two turn interactions with an average length of utterance no more than 25
vocab. And we randomly split the train-dev-test of collected datasets as Chitchat (1490/300/300, 5/1/1), Empathetic Dialogue (3022/500/500, 6/1/1), Persona Chat (3000/500/500, 6/1/1).

heading: A.1.2 Annotation Guideline
text: Table 6 provides detailed instructions for the annotators to s help them understand the setting of our annotation task.

heading: Annotation Guideline

heading: A.1.3 User Study
text: The dialogue can be evaluated from multiple perspectives. Some perspectives are universal to assess all dialogue agents, e.g., fluency, and relevance, while the other metrics are only used for task-specific dialogue agents. For example, the emotion-aware is a critical property for empathetic dialogue but is less important for persona dialogue. Therefore, we first simplify by sorting the possible aspects into two categories, i.e., the general view and the task-specific view. The former contains rel-
evance, engagingness, and fluency, while the latter consists of understandability, emotion-aware, and personality-aware, which correspond to chitchat dialogue, emotional dialogue, and persona dialogue. To understand the relation between sub-metrics and overall quality, we conduct a user study to learn their preference for different sub-metrics. Specifically, we invite 20 experts and 80 users, each of whom is asked to select the four most important ones from the sub-metrics. The results are listed in Figure 4. The approval rates reflect the user preference for different sub-metrics, which can be used as a weight to calculate the overall score. Moreover, we apply the softmax function on these weights to make them more interpretable.

heading: A.2 Experiment Details

heading: A.2.1 Datasets for Pre-train Stage
text: Our training process includes two stages, e.g., cross-domain pre-train and task-specific finetune. We first pre-train the model on diverse opendomain dialogue datasets as listed in Table 7 with the objective Lcross. The next stage relies on taskspecific dataset with the objective Lin (see in section 5).
These datasets are collected from https:// github.com/e0397123/dstc10_metric_track, which contain a variety of open-domain dialogue, such as emotional dialogue, personalized dialogue, knowledge-grounded dialogue, and chitchat. Every example in the datasets contains the dialogue context, response generated by dialogue agent, pre-created reference response, and the score of generated response which has been annotated for at least three people from several perspectives. We use cross-domain datasets for pre-training to improve the robustness and generalisability of the models across different evaluation scenarios.

heading: A.2.2 Experimental Details on Our Benchmarks
text: We show the details of our automatic evaluation experiments in Table 9. The BERTScore and BLEURT are computed based on the large version of Roberta. As in Section 6, we implement two reference-based baselines, BERTMLP and BARTMLP, using the same human-annotated datasets as RADE for training, and provide a reasonable comparison with our proposed model. Specifically, the BERTMLP is built on the base version of BERT (Devlin et al., 2018), while the BARTMLP is built on the base version of BART (Lewis et al., 2019).

heading: A.2.3 More Fair Comparison after Training
text: For a fair analysis, we pre-train the two strongest baselines (QuantiDCE and GRADE) with our cross-domain dataset. GRADE achieves Pearman’r=0.383, 0.378, -0.122, and QuantiDCE achieves Pearman’r=0.408, 0.522, 0.238 in the ChitChat, EmpatheticDialogue, and Personachat datasets. However, our proposed RADE(PT) remains the best results (Pearman’r=0.472, 0.650, 0.386). We further fine-tune GRADE and QuantiDCE with our self-collected datasets for a more comprehensive analysis. GRADE achieves Pearman’r=0.413, 0.430, -0.013, and QuantiDCE achieves Pearman’r=0.458, 0.589, 0.278 in three datasets, underperforming the proposed RADE(TS) (Pearman’r=0.601, 0.863, 0.470).
We skip pre-training/fine-tuning four baselines for the following reasons: (1) UniEval and QuestionEval have been pre-trained on multiple datasets across various domains. (2) The FED metric is unsupervised (cf. Shikib Mehri et al.) (3) The DialoRPT has been trained on a sizeable humanfeedback dataset (133M) covering various domains. These analyses validate the superiority of our method.

heading: A.2.4 Results on Existing Benchmarks
text: We further examine three existing benchmarks, i.e., USR-TopicalChat, USR-PersonaChat and GradeDailyDialogue to verify the efficiency and robustness of RADE when generalizing to agnostic scenarios. USR-TopicalChat and USR-PersonaChat datasets are collected to assess dialog evaluation metrics, with examples containing the dialogue context, reference, response and corresponding scores, which three people have annotated. The GradeDailyDialogue contains high-quality open-domain conversations about daily life including diverse topics. And the results are summarized in Table 8.
The experimental results show that RADE outperforms the state-of-the-art reference-free and reference-based methods on the USR-TopicalChat dataset. For example, we push the Pearson correlation to 48.0% (7% definite improvement) and Spearman correlation to 46.6% (4% absolute improvement). Moreover, RADE shows a stronger correlation with human judgment than existing
reference-based methods on the second dataset. It achieves comparable, even better results with the reference-free methods except for USL-H. The results demonstrate that our pre-trained model is more robust even under agnostic scenarios.
We also compare the two existing methods, and the results suggest a similar phenomenon as 3. Firstly, the reference-free methods achieve better consistency than reference-based methods, i.e., the former has the highest result of r = 41.2%, ρ = 42.3% while the latter gets r = 34.2%, ρ = 34.8% on the USR-TopicalChat dataset. However, the reference-free methods suffer from more significant variance. For example, the MAUDE gets r = 0.345% and ρ = 0.298% on the USRPearsonChat dataset but gets r = 0.044% and ρ = 0.083% on the USR-TopicChat dataset. It indicates that reference-free methods are more vulnerable and prone to data-induced bias.

heading: A.2.5 Case Study
text: To explain more intuitively, we show examples of automatic evaluation and them with human judgment in Table 10, 11, 12, suggesting that the scores of our methods are closer to human ratings.

heading: A.3 Presudo reference
text: Since the original FED does not provide the reference response, we construct a pseudo-reference via retrieval or generative method. The former retrieves reference from a curated response corpus based on our cross-domain datasets via BM25 with the dialogue context as the query. The latter generates via a large language model GPT-3 based on the dialogue context. The results show that RADE(PT) obtains Pearman’r=0.381 and Spearman’rho=0.368 with the retrieved reference while achieving Pearman’r=0.343, Spearman’rho=0.347 with generative reference, outperforming the stateof-the-art baseline (QuantiDCE, Pearman’r=0.319, Spearman’rho=0.323).
To further validate the generalizability of our method, we evaluate our proposed RADE(PT) on another challenging benchmark, GRADEDailydialogue. Our RADE(PT) achieves Pearman’r=0.356 and Spearman’rho=0.370 with 5% and 2% relative improvements compared to stateof-the-art baseline, indicating that our method can generalize to more challenging benchmarks.
Topic Library of Alexandria
Context User1: The library of Alexandria had a unique way of gathering books by making all ships visiting give their books for copy. User2: They must have had a impressive collection! User1: How unfair, they would copy their books give them back the copy and keep the originals.
Reference I guess that is true. Do you think we will ever have a centralized digital library of all our works and knowledge?
Response That is exactly why they didn’t stick around and stay put. I wish I could work somewhere where I could share the workload.
Evaluation Human (1-5) QuantiDCE (1-3) GRADE (0-1) METEOR (0-1) BERTScore (0-1) RADE (1-5)
3.67 3.00 0.26 0.17 0.48 3.54
Evaluation Human (1-5) QuantiDCE (1-3) GRADE (0-1) METEOR (0-1) BERTScore (0-1) RADE (1-5)
3 3.00 0.45 0.04 0.57 3.64
Evaluation Human (1-5) QuantiDCE (1-3) GRADE (0-1) METEOR (0-1) BERTScore (0-1) RADE (1-5)
2.98 2.7 0.29 0.34 0.61 2.64
Table 12: An example of the EmpatheticDialogue dataset evaluated by five automatic and human evaluation methods.
Personality (1): I like to donate time and money to charity. (2): I work as a computer programmer. (3): i make over 200 , 000 per year.
Context
User1: Hi, how are you ? tell me something about yourself! User2: Well, I love going fishing with my four attractive daughters every day. User1: Sounds fun! I enjoying volunteering and donating to charities. User2: Cool maybe you’d like to run a charity at my new race track. I race cars! User1: Sounds exciting! I am a computer programmer, which pays over 200k a year.
Reference Would you like to marry one of my four attractive daughters? I will sell one.
Response wow! That’s a lot of money. Do you have any hobbies?
Evaluation Human (1-5) QuantiDCE (1-3) GRADE (0-1) METEOR (0-1) BERTScore (0-1) RADE (1-5)
3.25 2.87 0.02 0.06 0.53 2.91
Table 13: An example of the PersonaChat dataset evaluated by five automatic and human evaluation methods, showing that our methods have a stronger correlation with human judgment than the other methods.
Personality (1): I like cupcakes and reading. (2): I work at a grocery store. (3): My favorite color is blue.
Context
User1: Hi! what are you up to ? I am doing what I like best, eating cupcakes! User2: Hi I am winding down from a day at work. User1: So am I. The local grocery store is where I work. What about you? User2: I also work in the retail industry at the local department store! User1: Other than eating cupcakes, reading is also what I like to do to wind down.
Reference I like to read also and play with my dog. Do you have a pet?
Response What do you do for fun? My girlfriend and I go to the lake a lot.
Evaluation Human (1-5) QuantiDCE (1-3) GRADE (0-1) METEOR (0-1) BERTScore (0-1) RADE (1-5)
2.75 3.00 0.01 0.22 0.58 2.79
Table 14: An example of the PersonaChat dataset evaluated by five automatic and human evaluation methods. This example shows that our methods have a stronger correlation with human judgment than the other methods.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations
3 A2. Did you discuss any potential risks of your work? Section Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 6 and 7
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 6
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 7
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 6
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4 and Appendix A
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix A
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4 and Appendix A
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section 4
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Section 4
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: In the future, we also plan to reproduce the proposed method for other, less resource-rich languages