DiQAD: A Benchmark Dataset for Open-domain Dialogue Quality Assessment
Authors: Yukun Zhao
Lingyong Yan
Weiwei Sun
Chong Meng
Shuaiqiang Wang
Zhicong Cheng
Zhaochun Ren
Dawei Yin
Venue: EMNLP 2023 Findings (2023)
Keywords: Dataset
Dialogue Evaluation Dataset
Dialogue Quality Assessment
Dialogue Quality Evaluation
Dialogue Quality Benchmark
Dialogue Evaluation

Abstract: Dialogue assessment plays a critical role in the development of open-domain dialogue systems.
Existing work are uncapable of providing an end-to-end and human-epistemic assessment dataset, while they only provide sub-metrics like coherence or the dialogues are conversed between annotators far from real user settings.  
In this paper, we release a large-scale dialogue quality assessment dataset (DiQAD), for automatically assessing open-domain dialogue quality.
Specifically, we (1) establish the assessment criteria based on the dimensions conforming to human judgements on dialogue qualities, and (2) annotate large-scale dialogues that conversed between real users based on these annotation criteria, which contains around 100,000 dialogues.
We conduct several experiments and report the performances of the baselines as the benchmark on DiQAD.
The dataset is openly accessible at \url{https://github.com/yukunZhao/Dataset_Dialogue_quality_evaluation}.

DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue Assessment
Yukun Zhao 1.2+
Lingyong Yan
Weiwei Sun'
Chong Meng
Shuaiqiang Wang2
Zhicong Cheng² Zhaochun Ren Dawei Yin "Shandong University, Qingdao, China
Baidu Inc., Beijing, China
'Leiden University, Leiden, The Netherlands
(anaayyukuno2,yanninnnnn @baidu...... com, www.wweewwee@@mmaeee com chengzhi (aae00le@iiaieeeee @baidu. com z. rene Liacs wwraaaccs.eeideee Leidenuniv..... nl, rindawei@acm org
Abstract
[When Yao Ming retire?
Dialogue assessment plays critical role in the development open-domain dialogue systems. Existing work are uncapable of providing end-to-end and human human-epistemic epistemic assessment dataset, while they only provide sub-metrics like coherence or the dialogues are conversed between annotators far from real user settings. In this paper, release large-scale dialogue quality assessment dataset (DiQAD), for automatically assessing open-domain dialogue quality. Specifically, (1) establish the assessment criteria based on the dimensions conforming human judgements qualities, and (2) annotate large-scale dialogues that conversed between real users based on these annotation criteria, which contains around 100,000 dialogues. We conduct several experiments and report the performances of the baselines the benehmark DiQAD. The dataset is openly accessible at https: //githull com/ yukunzaao Dattttstea.atatt..taa.aaaaaaaae Di wtwllouuusggllltyyeeeeeeeeee
2011 SUBE HE
"MEN"" AU TENES 2T A #R HE RUMAANNE X
[On
announced his retirement. he recalled basketball career and expressed his gratitude, and give special thanks to the Rockets fans in English)
200, (July 20th, 2011.
Common Response
Good Response
dialogue
Figure 1; achieves higher quality due to its richer informativeness.
Both responses are satisfied, but the right one
feedback (Kelly al., 2009; Li al., 2017b), which is often expensive and less reproducible. Other studies (Zhao et al., 2017: Gupta et al., 2019; Sai et al., 2020) relcase dialogues with references, which evaluate dialogue responses by comparing the generated responses with manually annotated references (Yang et al., 2015; Rajpurkar et al., 2016; Clark and Etzioni, 2016; Papineni al., 2002). However, their performance usually lim- ited reference coverage. Reference-free evalua- tion (Ghazarian et al., 2022: Sinha al., 2020; Sai et al., 2020: Zhao et al., 2020) uses crafted samples to train evaluation models due to the lack of anno- tations, which only capture coarse-grained metrics like coherence.
1 Introduction
Open-domain dialogue system (ODS) is quite pop- ular in artificial intelligence (Serban al., 2016; Huang et al., 2020; Bae et al., 2022), especially with the remarkable performance achieved in large language models (LLMs) (Wei et al., 2022; Ouyang et al., 2022; OpenAI, 2023). Dialogue assessment is critical to the development of open-domain dia- logue systems (Deriu et al., 2021) as it guides what constitutes good conversational dialogues.
On the other hand, most previous settings are less capable of end-to-end evaluation to determine whether the models perform like real humans. For example, the assessment datasets (Mehri and Es- kénazi, 2020; Gopalakrishnan al., 2019) focus on evaluating particular aspects of dialogues, e.g.. coherence diversity, while ignoring the overall quality. As shown Figure 1, the previous set- tings consider both responses as satisfied ones and ignore their informativeness differences, lacking
However, open-domain dialogue assessment still remains a challenging task (Deriu et al., 2021; Le et al., 2023). On the one hand, open-domain dialogue assessment is complicated and costly. A straightforward solution is the human-based method, i.c., recruiting human evaluators to inter- act with the dialogue systems and provide their
"Contributed equally.
"Co-corresponding authors.

---

higher requirements like real humans. Furthermore. previously released datasets (Young al., 2022;
language models (Ouyang et al., 2022: OpenAI, 2023), and they are usually classified into task- Komeili et al., 2022: Gopalakrishnan et al., 2019; Sun et al., 2021a) consist of dialogues conversed between recruited annotators and bots or between annotators, which varies from practice. As a conse- quence, the evaluating models learned from these datasets are far from optimal.
oriented and open-domain dialogue systems (Ni et al., 2022). The task-oriented dialogues aim to solve specific tasks in a certain domain, which are supported by pipeline systems (Li et al., 2017 7a; Cheng et al., 2020; Williams et al., 2017: Golo- vanov et al., 2019) or end-to-end systems (Ni et al., 2022; Le et al., 2020; Wang et al., 2019; He et al., 2020). The open-domain dialogue system consists of chat-oriented systems aiming to converse with users without task and domain restrictions (Tao et al., 2019; Feng et al., 2020; Xu et al., 2020; Song et al., 2020; Míao et al.. 2019; Wu et al.. 2021), and conversational
In this paper, we release large-scale dataset DiQAD (Dialogue Quality Assessment Dataset) for open-domain dialogue quality assessment, which aims to learn an atomically evaluating model to guide the dialogue system in producing both satis- fied and high-quality responses. Specifically, we first establish unified human-epistemii quality crite- ria for open- open-domain dialogues based on 6 quality di- mensions: grammaticality, relevance, consistency, empathy, proactivity, and informativeness (Finchh and Choi, 2020). We set the quality label as scale holistic score (from to 2) by considering the above dimensions, identifying higher quality dialogues that provide more comprehensive infor- mation and actively engage in the conversations using score of 2. After that, we collect sub- stantial number of real dialogues from online conversation platform and hire experienced annota- tors annotate them following the above quality criteria. total, DiQAD contains around 100k dia- logues with around 0.8 million dialogue utterances across 6 common domains.
systems which are developed to answer specific questions (Liu et al., 2019; Sun et al., 2019; Qu et 2019).
Dialogue Evaluation There are two directions for dialogue evaluation: human evaluation and automatic evaluation. The human evaluation in- volves recruiting experts to test a dialogue system and then collecting questionnaires (i.e., user rat- ings) (Kelly et al., 2009; Ashwin et al., 2018; 88; Li et al., 2017b). For the dialogues with predefined tasks, metrics like cask-completion rate and task- completion cost are calculated from user interac- tions (Walker et al., 1997; Bodigutla et al., 2019) or user simulators (Schatzmann et al., 2007; Schatz- mann and Young, 2009; Zhang and Balog, 2020; Sun et al., 2022) to evaluate the dialogues.
On the DiQAD, we conduct extensive experi- ments to verify the performance of different mod- cls. Additionally, the cross- cross-domain experiments DiQAD show that the best model learned on our dataset can generalize to unseen domain dialogues. The contributions of this paper are twofold:
Another way for automatic evaluation is evaluate the quality of dialogue contents One is reference-based evaluation, which measures the similarity between the generated responses and the ground-truth one, including correctness metrics such as MAP, MRR, EM, F1, and accuracy (Yang et al., 2015; Rajpurkar et al., 2016; Clark and Et- zioni, 2016). Word-overlap statisties like BLEU, ROUGE, and METEOR (Papineni et al., 2002: Lin, 2004; Banerjee and Lavie, 2005), well neural metrics (Sato et al., 2020: Tao et al., 2018; Zhao et al., 2023), are calculated. However, may suffer from the one-to-many problems (Zhao et al., 2017), even with multi-references (Gupta et al., 2019; Sai et al., 2020). Besides, recently released datasets are dialogues that occur between recruited anno- tators and bots (Young et al., 2022), or between annotators themselves (Wang et al., 2021: Smith et al., 2020; Komeili et al., 2022; Gopalakrishnan et al., 2019), or happen in certain scenarios like English practice (Li et al., 2017b). The conver-
This paper defines the rules for assessing di- alogue quality and constructs a large-scale dialogue evaluation dataset. To the best of our knowledge, this is the first large-scale dataset focused dialogue quality assessment.
This paper conducts benchmark experiments DiQAD and finds that recent large lan- guage models (e.g., ChatGPT, ChatGLM) are less capable of discriminating high-quality di- alogues.
2 Related Work
Dialogue System Dialogue systems are quite popular, with the remarkable performance of large

---

sational topics and contents are handcrafted and limited compared real users' settings.
hends the user's feelings and appropriately reacts to emotional expressions (e.g., appeas- ing).
The other one is reference-free evaluation, which train a classification model using users' rat- ings (Liang et al.. 2020) or crafted samples (Ghaz- arian et al., 2022; Sinha et al., 2020; Sai et al., 2020; Zhao et al., 2020) for coarse-grained evalu- ation like coherence due to the short of annotated labels. Recent work (Mehri and Eskénazi, 2020; Deriu al., 2020; Bao al., 2019; Sun et al., 2021a; Le et al., 2023) use annotated samples to train the evaluation model. The annotations for open-domain dialogues are desirable, but the pre- vious work only focuses on sub-type of dialogues such as task-oriented dialogues (Sun et al., 202 aa). For open-domain dialogue annotation, they (Sun al.. 2021a: Bodigutla al.. 2019; Mehri and Eskénazi, 2020; Gopalakrishnan et al., 2019) eval- uate some sub-metries like coherence, diversity, empathetic or factuality, lacking of compound and human-epistemic high-quality evaluation and the data size is limited.
Whether the respondent re- sponds actively, provides useful extensions, and moves the dialogue to new topics.
Proactivity:
Informativeness: Whether the responses pro- vide useful, specific, and sufficient informa- tion.
Based on the above dimensions, we conduct a 3-scale (0, 1, 2) quality assessment. A dialogue is set to at least answer the user's question; thus, the quality is at least binary category of 1 and 0 to dis- tinguish whether the user's question is resolved or not. Furthermore, aim to identify high-quality dialogues that guide future dialogue generation to- wards higher quality, like humans. To achieve this, we introduce an additional category of indicating a high-quality rating.'
The quality assessment includes turn-level and dialogue-level evaluation. We provide dialogue- level quality evaluation criteria. Specifically, we assess the quality of dialogue according to the fol- lowing instructions (The examples are shown in Table 1, and detailed instructions and more exam- ples are shown Appendix A.2):
3 Open-domain Dialogue Quality Assessment
This paper treats open-domain dialogue quality as- sessment as critical task, which differentiates higher quality dialogues based on real users' di- alogues, to guide the future dialogue generation towards more satisfying and human-epistemic re- sponses.
Low quality (score 0): The dialogue fails to meet the basic requirements, including gram- maticality, relevance, consistency, and empa- thy. As shown in Table I, the response of the first case is irrelevant to its question. Thus we label it 0-class directly without considering other metrics.
To this end, the quality assessment criteria are required to reflect higher quality for human cog- nition much possible. The human demands upon dialogue quality usually regarded tailing several ane-ggrined dimensions (Deriu et al., 2021; Smith et al., 2022). Inspired by fully ana- lyzed human-epistemic evaluation (Finch and Choi, 2020), we adopt the following 6 dimensions to establish our quality assessment criteria (see exam- ples in Table l):
Moderate quality (score 1): The dialogue can basically answer the user's question, pro- viding grammatical, relevant, consistent, and empathetic responses but fails to meet proac- tivity and informativeness. As the second case in Table the response is only reason that fails to supply sufficient information meet the informative requirement.
Grammaticality: Whether the utterances are fluent, readable, and free of grammatical and semantic errors.
High quality (score 2): The dialogue meets all the dimensions mentioned above. It is a higher requirement in addition to correctly an- swering questions on 1-class dialogue, which provides more comprehensive information and actively engages in conversations with
Relevance: Whether the responses logically match and are coherent with the questions.
Consistency: Whether the utterances provide consistent persona and no contradictions with the previously provided utterances.
'Our annotators show lower annotation agreement under more fine-grained, like 1-5 scale assessment for dialogue quality. We leave the 1-5 scale version for future work.
Empathy: Whether the respondent compre-

---

Dialogue
Score
Explanation
Q: Any suggestions for buying iPhone 14 or 14 pro? R: The iPhone is designed and marketed by Apple Inc. Apple is American multinational technology company... ...
Not relevant between buying and market
0
Q: My girlfriend is annoyed at my ex. R: Perhaps you remain affiliated with your ex. When you break up with your ex compl- etely, she would not be upset anymore.
Give only one reason that do not provide complete information
Q: Which products are manufactured by Foxconn?
R: Foxconn manufactures electronic products including the iPhone, MacBook, Dell, HP computers, and other consumer electronics.
Provide the all desired information.
2
R: Foxconn manufactures electronic products for major American, Canadian, Chinese, Finnish, and Japanese companies. is a multinational electronics contract manufacturer
Table I: Dialogue quality assessment examples following our criteria. The Q and R denote the question and response respectively. All contents are translated from Chinese.
the user, as shown in the third case in Table 1 and Figure in Appendix A.2.
Specifically, collect real dialogues in 6 do- mains from publicly accessible WenYiWen dia- logues set The domains we selected consist of emotion, digital, legal, education, car, and finance dialogues. Consequently, the collected dialogues for annotation are more than 100 thousand (The statistics of collected dialogues are shown in Sec- tion 4.3).
In the next section, we introduce how we build the DiQAD based on the above criteria.
4 DiQAD: A Large-scale Dataset for Dialogue Quality Assessment
In this section, we describe how we construct the DiQAD based on the above annotation instructions in detail. We first collect amounts of real dialogues from an online platform. Then, we recruit experi- enced annotators to annotate these dialogues. In the last part of this section, we will show the detailed statistics of DiQAD.
4.2 Quality Annotation
Next, we recruit dozens of experienced annota- tors to annotate the dialogue quality on a crowd- sourcing platform developed by ourselves.
The annotation process is an annotation- checking procedure, or repeated annotation- checking procedures if the annotation accuracy of the first round of annotation does not match the requirements. The dialogues are divided into multi- ple batches for annotation, with each batch consist- ing of 2000 samples. The annotators are divided into several groups, with each group annotating the batches of data.
4.1 Real Dialogue Collection
In the first step. we collect real dialogues in text from YiWen. Wen provides online con- versational services, in which users can chat with others and obtain required information from ex- perienced experts. Users can propose their ques- tions to the platform, and then experienced experts can browse them and choose their interested ones to start a dialogue. After that, the dialogue takes place between the user and the expert until the user obtains their satisfied information or the dura- tion reaches its limit. Wen YiWen provides domain and topic tags for user questions help experts select their related questions. The experts are in- vited from various professions, such as lawyers, university professors, college students, and car me- chanics. Besides, to attract enough experts, users are asked to pay for their questions, and the experts can receive cash rewards based on the dialogue quality.
For each dialogue, the quality label has been agreed upon and confirmed by annotators. The annotator is first presented with the whole dialogue text. To guarantee the labeling reliability, the read- ing duration of each dialogue is required to be no less than a fixed time. After that, annotators are asked to label the quality score following the anno- tation criteria in section 3 and the detailed instruc- tions in Appendix A.2. In addition to the quality labels, annotators are also asked give explicit reasons why the labels are decided to improve an- notation quality.
*For each YeWniWen dialogue, users are asked whether to open their dialogues to the public. If users agree to open, then the dialogue will hecome publicly accessible.
Phttps://ven baa.nbaaidie baidu.com

---

Operation of mobile device (19.1%)
K-12 education[13.5]
Affection (11.7%)
Digital (31.5%)
Interpers onal Relationships (8.5%)
Emotion (25.3%)
Bank loan (5.4%)
Education (22.1%)
Domestic appliance (4.5%)
Finance (8.6%)
Vehicle maintenance (4.3%)
Legal (6.)
Undergraduate education (3.6%)
Car (5.7
Marriage & family (3.6%)
Civil and commercial law (3.0%)
Others (8.6%)
(a)
(b)
Figure 2: The domain (a) and topic (b) distributions of DIQAD.
After annotating a batch of data, we have a pro- fessional quality controller to check the label accu- racy. (The roles and distribution of crowd-source annotators can be found in Appendix A.3). Our acceptance criterion is to achieve an annotation ac- curacy of 92% or above. More detailed information about annotation is shown in Appendix A.
Dataset
#insts. #uttrs, #tokens q-label
DailyDialogue
13k
7.9
114.7
N
BlendedSkillTalk
5k
11.3
155.4
N
NaturalConv
19k
20.1
244.8
N
FusedChat
10k
5.8
65.2
N
WizInt
10k
9.7
185.2
USR
660
9.3
180.2
Y
FED-Dial
125
12.7
113.8
Y
4.3 Dataset Statisties
TopicalChat
10k
21
399
Y
Finally, we obtain DiQAD, consisting 100k dia- logues across different domains. Its basic statis- tics are shown in Table 2. DiQAD contains over 100k dialogues, where each dialogue consists of around 8 utterances and 334 tokens on average. In Table 2, we also compare our proposed DiQAD with recent popular open-domain datasets, DailyDi- alogue (Li al., 2017b), BlendedSkillTalk (Smith al., 2020), NaturalConv (Wang et al., 2021), FusedChat (Young et al., 2022), WizInt (Komeili et al., 2022), USR (Mehri and Eskénazi, 2020), FED-Dial (Mehri and Eskenazi. 2020), and Top- icalChat (Gopalakrishnan et al., 2019), while the first 5 datasets have no quality annotation. Natural- Cony and DiQAD written in Chinese, and the others in English. The scale of DiQAD is larger, and compound human-epistemic evaluation real users' dialogues compared with all other datasets.
DIQAD
100k
7.9
334.1
Y
Table 2:
The statistics of DiQAD and other open- domain dialogue evaluation datasets. We list the number of dialogue instances, the mean number of utterances dialogue, the mean number of tokens per dialogue. and whether containing quality labels.
Experiments
In this section, we conduct and illustrate bench- mark experiments on DiQAD, including examin- ing the overall performance of benchmark models ($5.2), conducting detailed analysis based the best benchmark model ($5.3), comparing to other reference-fre free evaluation methods ($5.4), and per- forming hyperparameter analysis ($5.5).
5.1 Experimental Setup
The domain distributions of DiQAD are shown in Figure 2a. We see that the top 3 domains in DiQAD are digital, emotion, and education, with the smaller ones are finance, legal, and car. We uti- lize the built-in topic tags mentioned in section 4.1 and show the top 10 topics in DiQAD in Figure 2b. We see that DiQAD spreads over diverse topics such as the operation of mobile devices, affection, bank loans, etc., which can reflect the real users" settings to some extent.
Dataset. We randomly split DiQAD into train- ing, validation, and test sets, where the validation and test sets consist of 10,000 dialogues, and the remaining dialogues are used for training. For all models trained the training set, tune their parameters the validation and report the per- formance results the set. The final test set contains 1,958 samples classified as score 0, 3,628 samples classified as score 1, and 4,414 samples classified as score

---

Metrics. We employ the following metrics of baseline models': Acc, the accuracy of the model predictions: (ii) Unweighted average re- call (UAR), the arithmetic average of class-wing wise re- calls; (iii) Cohen's s Kappa coefficient (Kappa) (Co- hen, 1960), which we used to measure the agree- ment between model-predicted labels and human- annotated labels; (iv) Spearman's 's rank correlation coefficient (Spearman), the non-parametric mea- sure of the correlation between model predictions and human labels; (v) Pearson correlation coeffi- cient (Pearson), the measure of linear correlation between model predictions and human labels. Be- sides, we also report the precision, recall, and F1- score of model predictions.
For LLM-based methods, we use the instructions detailed in Appendix C for few-shot learning.
Implementation.
The neural models are opti- mized using the cross-entropy loss. We set the batch size to 16, the learning rate to 1e-5, use the AdamW optimizer to optimize parameters, em- ploy gradient clipping with a maximum gradient norm of 1.0, train up to 10 epochs, and select the best checkpoints based on performance on the val- idation set. Under default conditions, we use the ERNIE-base model and set the maximum input length to 512 tokens. We conduct analytical exper- iments on hyperparameters such as model size and maximum length in Section 5.5.
Benchmark Models. We use the following two types of models as our benchmark models: clas- sical methods, Transformer-base based methods, and LLM-based methods.
5.2 Overall Performance
The results of benchmark models are shown in Table 3. We can see that: (1) All transformer- based methods achieve higher performance than classical methods. This verifies the advantages of the Transformer-base based architecture in natural language processing. (2) Pre-trained models, espe- cially the Ernie model, outperform other methods by large margin. This is not surprising because pre-training is widely regarded as the ability to incorporate external knowledge into model param- eters. (3) LLMs-based methods, since they are not sufficiently trained on the dialogue assessment task, still have a significant gap compared to the training- based approach. Further exploration is needed on how to better utilize LLMs for dialogue evaluation tasks.
Classical methods: Naive Bayes (Bayes), Lo- gistic Regression (LR), XGBoost, and GRU.
Transformer-based methods: Vanilla Trans- former (Vaswani et al., 2017), a 4-layer Transformer, BERT (Devlin et al., 2019); MENGZI (Zhang et al., 2021), an efficiently pre-trained Transformer using a 300G Chi- nese corpus; ERNIE (Sun et al., 2021b), 12-layer Transformer encoder pre-trained on 4TB corpus consisting.
L..M-based methods: ChatGLM-6B (Du et al., 2022), an open-source Chinese LLM; and ChatGPT (Ouyang et al., 2022), an LLM developed by OpenAl and accessed through 3pt-3. 5-turbo-03300 API.
Since the Ernie model achieves the best results DiQAD, we use it as the backbone for detailed analysis in the next section.
5.3 Detailed Analysis
For Transformer-basee models, we use the same data format as the input. Specifically, given a dia- logue, we use "question: and "response:" to annotate the sentences from the questioner and respondent, respectively. We then splice the dia- logue into a long text and feed it into the models to get predictions. For Transformer-baseee models, we add two special tokens, [CLS] ] and [SEP], to the beginning and end positions after tokenization. Af- ter encoding the dialogue using Transformer-bases models, the embedding of the (CLS) token is fed into a 3-class classification layer to get predictions.
Cross-domain Generalizability Firstly, we ana- lyze the cross-domain generalizability of the model trained on DiQAD, We conduct comparison ex- periments by removing training samples from a particular domain during training and testing on that domain. The experimental results are shown in Table 4. For each domain, we also report Ernie's 's performance on it, where Ernie is trained with all training samples.
From Table 4, we can see that the model can still achieve comparable performance on domains not appearing in the training set. For example, all accuracy scores drop by less than 6% when the domain samples are removed from the training set.
WW use the implementation from SciPy (https: pppp:// // scipy. org/) calculate the Spearman and Pearson scores.

---

Acc
UAR
Kappa
Spearman
Pearson
Precision
Recall
Classical methods
Bayes
52.67
28.53
24.03
22.64
24.52
48.35
47.55
47.56
LR
60.98
32.99
36.83
37.66
39.60
57.74
54.98
55.21
XGBoost
63.37
34.56
40.54
42.22
43.35
62.06
57.59
58.21
GRU
61.34
33.72
37.68
39.98
41.15
60.02
56.11
56.53
Transformer- based methods
Transformer
64. 14.16
35.02
41.61
45.52
46.66
63.12
58.62
59.28
BERT
68.2
38,76
48.91
55.59
55.37
67.92
64.61
65.71
MENGZI
69.25
39.79
50.91
56.28
56,29
68.28
66.31
67.07
ERNIE
69.94
40.96
52.55
37.39
57.25
68.68
68.27
68.46
LLM-based methods
ChatGLM-6B
36.70
21.08
3,05
9.4 9.44
10.09
36.22
36,70
34.22
ChatGPT
39.30
21.88
4.70
12.83
12.51
37.61
36.47
36.28
ChatGPT + CoT
41.00
24.91
10.68
17.79
17.33
30.77
31.14
30.38
Table 3: Comparison of benchmark models DiQAD, The best results of each metric shown in bold.
Acc
UAR
Spcurman Pearson
original
+pos
+neg
Digital domain
0.9
ERNIE
63.89
38.40
50.14
50.08
63.90
0.8
w/o Digital
61.97
37.01
46.75
45.90
61.72
0.7
0.6
Emotion domain
ERNIE
77.30
40.49
61.14
61.48
68.95
0.5
w/l Emotion
72.14
38.74
56.74
56,97
64.25
0.4
0.3
Education domain
0.2
ERNIE
69.82
40.55
52.13
54.78
68.06
0.1
w/o Education
64.26
37.21
45.13
44.76
62.54
Legul domain
0
1
2
72.63
41.00
56.73
w/o Legal
72.00
40.23
59.94
58.65
68.41
Figure 3: Comparison of predicted score distributions when adding different sentiment expressions.
Car domain
ERNIE
67.71
38.56
47.93
47.03
64.49
-//o Car
66.67
38.84
48.92
48.06
64.32
the template "question: I don't 't think so. "" We denote the modified model as ERNIE+neg. The experimental results are shown in Table 5. We also demonstrate the prediction score changes when adding different sentiment expressions in Figure 3.
Finance domain
ERNIE
68.35
38.96
53.61
52.77
65.80
w/o Finance
67.41
37.75
51.30
50.33
64.43
Table 4: Cross-domain evaluation results. Each group in the table represents the results of specific domain.
From Figure and Table 5, we can see that the model is more likely to predict a low-quality label when the dialogue contains negative user expres- sions and tends to give higher quality if there is positive user feedback. As a result, the perfor- mance of ERNIE+pos decreases slightly compared the original Ernie (e.g., the accuracy by around 3and adding the negative expressions (i.e., ERRII++ne) makes the accuracy score dras- tically decline by more than 38These performance changes, especially on the negative sentiment, show that quality assessment in DiQAD is sensitive to sentiment expressions. We believe that the sensi- tivity to user sentiment can help dialogue systems evolve in direction that better satisfies the user.
We also find that the metric decline rates are related to the number of samples in that domain. For exam- ple, the performance decrease of the model is more significant when large domain (e.g., Emotion) is removed than small domain (c.g., Finance).
drops
To analyze the influ- ence of user sentiment expression on dialogue qual- ity assessment, we design two types of sentiment expressions and add them to the end of the origi- nal dialogues: (i) Positive sentiment: We use the template "question: ok thanks! response: you are welcome!'S 1*5 We denote the modified model as ERNIE +pos. (ii) Negative sentiment: We add
User Sentiment Influence
PIn Chincse: "question: {
response: !
IIn Chinese: "question:

---

Acc
UAR
Speurman Peurson
(a) Model
(b) Data size log #sample
(c) length
log #param
#token
FRNIR
69.94
40.96
$7.39
57.25
68.46
ERNIE-pos
66.99
36.63
50.60
51.12
62.51
298M
100%
75
ERNIE+neg
31.33
24.55
27.07
28.05
27.04
70
50%
70
7SM
Table 5: Comparison of evaluation results when adding different sentiment expressions.
68
27M
64
60
18M
Spearman
Pearson
7.2 7.7
8.2
2.9
3.9
4,9
0
1024
2048
Unsupervised methods
Figure 4: Hyperparameter analysis: (a) Model size, we demonstrate the Ernie performance with different num- bers of parameters: (b) Data size, we show the effects of the training data size on the Ernie performance; (c) Max length, we compare the Ernie performances with different maximum input lengths.
USR (Mehri and Eskénazi, 2020)
4.47
11.64
USR-MLM
2.61
3.83
USR-DR
3.83
11.61
USL-H (Phy al., 2020)
4.60
7.91
USL-VUP
0.65
-1.22
BARTScore (Yuan al., 2021)
24.73
32.77
5.5
Hyperparameter Analysis
Supervised methods
BLEURT (Sellam al., 2020)
48.37
49.96
We conduct hyperparameter analysis, and the sults are illustrated in Figure 4. The aspect we analyzed and the findings are: (a) Model size. We use ERNIE of different sizes (c.g.. 18M, 27M) and the results show that larger models achieve better performance. (b) Data size. We random sample portion of training data (e.g., 1%, 5%) and train the ERNIE- base model. We find that more data lead to better performance and the increase curve is basically in line with the logarithmic speed. (c) In- put length. We change the maximize input token number and test the performance difference. We find that increasing the maximum length can im- prove the effect when the length is less than 512, and there is almost no improvement after the length exceeds 512. This may be mainly because dialogue tokens in DiQAD are mostly less than 512.
P-Tuning (Liu et al., 2021)
53.03
53.02
Prompted ERNIE (3-shot)
15.09
28.34
finetuning
56.49
56.49
ERNIE
57.39
57.25
Table 6: Results of reference- free dialogue evaluation methods. The best results are shown in bold.
5.4 Comparison with Other Reference-free Assessment Methods
this section, we compare the performances of other reference-free assessment methods on DiQAD. There are usually two types of reference- free dialogue assessment methods: unsupervised methods and supervised methods. Unsuper- vised methods are usually learned without notated dialogues, including: USR (Mehri and Eskénazi, 2020), USI.-H (Phy al., 2020), BARTScore (Yuan et al., 2021). Supervised meth- ods are usually learned upon annotated dialogues, including: BLEURT (Sellam et al., 2020), P-
Conclusion
In this paper, we conduct a human-epistemic dia- logue quality assessment on real user conversations. We release large-scale dataset DiQAD for open- domain dialogue quality assessment, which con- tains around 100k dialogues and 0.8 million utter- ances in domains. We conduct several benchmark experiments on DiQAD to study the performances of classical, Transformer-bast based models and large language models on the dialogue quality assess- ment task. We hope the release of this dataset can inspire researchers in the community who lack such real and large-scale datasets, to evaluate real user dialogues to guide further improvement for the language models' generation.
Tuning (Liu et al., 2021), Prompted ERNIE (3- shot (Sun et al., 2021b).
The evaluation metric is Spearman and Pearson, following previous studies (Mehri and Eskénazi, 2020). The results are listed Table 6. We see that the unsupervised methods perform unfavor- ably. For example, USR and USL-H perform well on previous dialogue evaluation benchmarks but show low consistency with human assessment DiQAD. The main reason may be that in DiQAD, the user's questions are more difficult and the sys- tem needs to provide comprehensive responses to meet the user's needs, and simple metrics such as relevance and fluency are not well suited.

---

Limitations
References
The dialogue contents written in Chinese, and hence the dataset can only be used evaluate Chinese models. The impact may be restricted by its language. Besides, the benchmark models are absent from fine-tuning larger pre-trained mod- els such as ChatGLM (Du et al., 2022) and Vi- cuna (Chiang et al., 2023). We cannot promise the current benchmark provide the strongest base- line. We believe that larger models and more fine- grained tuning would achieve better performance. Finally, the quality annotation has been simplified 0-2 scale for higher labeling accuracy. More de- gree of labels may help to differentiate the dialogue quality more elaborately. We will study the above limitations in our future work.
Ram Ashwin, Prasad Rohit, Khatri Chandra, Venkatesh Anu, Gabriel Raefer, Liu Qing, Nunn Jeff, Hedayat- nia Behnam, Cheng Ming, Nagar Ashish, et al. 2018. Conversational ai: The science behind the alexa prize. arXiv: 03604..
Sanghwan Bae, Donghyun Kwak, Sungdong Kim, Donghoon Ham, Soyoung Kang. Sang- Woo Lee, and Woomyoung Park. 2022. Building Role Specified Open-Domain Dialogue System Leveraging Large- Scale Language Models. In NAACL-HLT, pages 2128-2150.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In ACL Intrinsic and Extrinsic Evaluation Measures for Machine Transla- tion and Summarization Workshop. pages 228-231
Siqi Bao, Huang He, Fan Wang. Rongzhong Lian, and Hua Wu. 2019. Know more about each other: Evoly- ing dialogue strategy via compound assessment. In ACL, pages 5382-5391.
Ethics Statement
We acknowledge the importance of the ACM code of Ethics and totally agree with it. We ensure that this work is compatible with the provided code, specifically in terms of providing the dia- logue dataset. We use the dataset in accordance with copyright terms and under the licenses of its provider.
Praveen Kumar Bodigutla, Lazaros Polymenakos, and Spyros Matsoukas. 2019. Multi-domain conversation quality evaluation via user satisfaction estimation.
Jianpeng Cheng, Devang Agrawal, Héctor Martínez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Shaona Ghosh, Dain Kaplan, Dimitri Kart- saklis, Lin Li, et al. 2020. Conversational semantic parsing for dialog state tracking. In In EMNLP, page 8107-8117.
Licensing We collect the dialogues that have been licensed from both the users and the respon- dent. All the dialogues have been granted pub- lish by their owner, i.e., the Wen YiWn platform.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See hrtps:/vvicua. Imsys. org (accessed 14 April 2023).
Personal Information We collect dialogue texts without user information. The collecting procedure obeys the privacy policy according to local laws.
Privacy All the dialogues have been ex- amined by the YiWen audit team one by one, to avoid pornography, offensiveness, profamity, pri- vacy, and toxicity risks.
Peter Clark and Oren Etzioni. 2016. My computer is
honor student-but how intelligent it? standardized tests as a measure of ai. Al Magazine, 37(1):5-12.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20:37 - 46.
Acknowledgements
This work was supported by the National Key R&D Program of China with grant the Natural Science
Jan Deriu, Alvaro Rodrigo. Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021. Survey evaluation methods for dialogue systems. Artificial Intelligence Review, 54(1):755-810.
No. N..2200YFB14067044 Foundation of China (62272274, 61902219, 61972234), the Natural Science Foundation of Shandong Province (ZR2021QF12)) We thank the Wen YiWen platform for providing the original dialogues and helping us to filter privacy and toxic dialogues.
Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos. Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliehak. 2020. Spot the bot: A robust and efficient framework for the evaluation conversational dialogue systems. ACL. page 3971-3984.

---

Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, page 4171-4186.
Cat P Le, Luke Dai. Michael Johnston. Yang Liu, Mar- ilyn Walker, and Reza Ghanadan. 2023. Improving open-domain dialogue evaluation with causal infer- ence model. arXiv preprint arXiv:2301.1337.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Gim: General language model pretraining with autoregres- sive blank infilling. In ACI. pages 320-335.
Hung Le, Doyen Sahoo. Chenghao Liu, Nancy Chen, and Steven CH Hoi. 2020. Uniconv: A unified con- versational neural architecture for multi-domain task- oriented dialogues. arXiv:2004.14307
Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017a. End-to-end task- completion neural dialogue systems. In IJCNLP. page 733-743.
Shaoxiong Feng. Xuancheng Ren, Hongshen Chen, Bin Sun, Kan Li, and Xu Sun. 2020. Regularizing dia- logue generation by imitating implicit scenarios. In EMNLP, page 3592-6604
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 201 Dailydialog: A manu- ally labelled multi-turn dialogue dataset. In IJCNLP, page 986-995.
Sarah E. Finch and Jinho D. Choi. 2020. Towards uni- fied dialogue system evaluation: A comprehensive analysis of current evaluation protocols. In SIGDIAL, page 236-245.
Weixin Liang, James Zou, and Zhou Yu. 2020. Beyond user self-reported likert scale ratings: A comparison model for automatic dialog evaluation. ACL, page 1363-1374.
Sarik Ghazarian, Nuan Wen, Aram Galstyan, and Nanyun Peng. 2022. Deam: Dialogue coherence amr-basee based semantic manipulations.
evaluation using amr- In ACI, page 771-785.
Chin- Yew Lin. 2004, Rouge: evaluation of summaries. In Text Summarization Branches Out, page 74-81.
A package for automatic
Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, and Thomas Wolf. 2019. Large-scale transfer learning for natural language generation. In ACL, pages 6053- 6058.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. P-tuning v2: Prompt tuning can be comparable fine-tuming universally across scales and tasks. ArXiv, abs/2110.07600
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinglang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Racfer Gabriel, Dilek Hakkami-Tinn and Amazon Alexa AI. 2019. Topical-chat: Towards knowledge-grounded open-domain conversations. In InterSpeech, pages 11995
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv:/907, /91.//977.1111 1699..
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019, Investigating evaluation of open-domain dialogue systems with human generated multiple references. In SIGDIAL, page 379-391.
Shikib Mehri and Maxine Eskenazi. 2020. Unsuper- vised evaluation of interactive dialog with dialogpt. In SIGDIAL, page 225-235.
Shikib Mehri and Maxine Eskénazi. 2020. Usr: An unsupervised and reference free evaluation metric for dialog generation. In ACL, page 681-707.
Wanwei He, Min Yang, Rui Yan, Chengming Li, Ying Shen, and Ruifeng Xu. 2020. Amalgamating knowl- edge from two teachers for task-oriented dialogue system with adversarial training. In EMNLP, pages 3498-3507.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. Cgmb: Constrained sentence generation by metropolis-hasting sampling. In AAAI, pages 6834-6842.
Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain dia- log systems. TOIS, 38(3).
Jinjie Ni, Tom Young. Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2022. Recent advances in deep learn- ing based dialogue systems: A systematic survey. arXiv:2105.0488755
Diane Kelly, Paul B Kantor, Emile L Morse, Jean Scholtz, and Ying Sun. 2009. Questionnaires for eliciting evaluation data from users of interactive question answering systems. Natural Language En-
OpenAl. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
gineering. 15(1):119-14)
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. NIPS, 35:27730-27744
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-augen augmented dialogue generation. In ACL, pages 8460-8478.

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In ACI, page 31 1-318.
Haoyu Song, Yan Wang. Wei-Nan Zhang, Xiaojiang Liu, and Ting Liu. 2020. Generate, delete and rewrite: A three-stage framework for improving per- sona consistency of dialogue generation. In ACL, page 5821-5831.
Vitou Phy, Yang Zhao, and Akiko Aizawa. 2020. De- construct to reconstruct configurable evaluation metric for open-domain dialogue systems. ArXiv, abs/201 .00483.
Weiwei Sun, Shuyu Guo, Shuo Zhang, Pengjie Ren, Zhumin Chen, Maarten de Rijke, and Zhaochun Ren. 2022. Metaphorical user simulators for evaluating task-oriented dialogue systems. arXiv:2204, 007663.
Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W Bruce Croft, and Mohit lyyer. 2019. Attentive history selection for conversational ques- tion answering. CIKM, page 1391-1400.
Weiwei Sun, Shuo Zhang. Krisztian Balog, Zhaochun Ren, Pengjie Ren, Zhumin Chen, and Maarten de Ri- jke. 2021a. Simulating user satisfaction for the eval- uation of task-oriented dialogue systems. In SIGIR, pages 2499-2506 2506.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for
machine comprehension of text. In EMNLP, pages 2383-2392.
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang. Peng Sun, Wei Liu, Xuan Ouyang. Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang. 2021b. Emni 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. ArXiv, abs/2 010.2107.0222 107.02137.
Ananya B Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M Khapra. 2020. Improving dialog evaluation with a multi-reference adversar- ial dataset and large scale pretraining. TACL. page 810-827.
Shiki Sato, Reina Akama, Hiroki Ouchi, Jun Suzuki, and Kentaro Inui. 2020. Evaluating dialogue gener- ation systems via response selection. In ACL, page 593-599.
Sun, Shuohuan Wang, Yukun Li, Shikun Feng- Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: En- hanced representation through knowledge integration. arXiv:1944.0222 092233.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-basee baaddd user simulation for bootstrapping a pomdp dialogue sys- tem. In NAACL, pages 149-152.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In AAAI
Jost Schatzmann and Steve Young. 2009. The hidden agenda user simulation model. JEEE-ACM T AUDIO SPE, 17(4):733-747
Chongyang Tao, Wei Wu, Can Xu, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019. One time of interaction may not be enough: Go deep with an interaction -oer--vveeeetteeeeeeee lection in dialogues. In ACL, pages 1-11.
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. Bleurt: Learning robust metrics for text gener- ation. In ACL. page 18812
network for response se-
Tulian Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Build- ing End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models. AAAI, pages 3776-3783
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv, abs/1706.03762.
Marilyn A Walker, Diane Litman, Candace Kamm, and Alicia Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. In ACL.
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang. Ryan Lowe, William Hamilton, and Joelle Pineau. 2020. Learning an unreferenced metric for online dialogue evaluation. In ACL, page 2430-2441.
Weikang Wang. Jiajun Zhang, Qian Li, Mei-Yuh Hwang, Chengqing Zong, and Zhifei Li. 2019, Incremental learning from scratch for task-oriented dialogue sys- tems. arXiv: / 7906.00000 04991.
Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022. Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for eval- uating dialogue agents. In Proceedings of the 4th Workshop on NLP for Conversational AI, pages 77- 97, Dublin, Ireland. Association for Computational Linguistics.
Xiaoyang Wang, Chen Li, Jianqiao Zhao, and Dong Yu. 2021. Naturalconv: A chinese dialogue dataset to- wards multi-turn topic-driven conversation. In AAAI. pages 14006-14014.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations.
Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can you put it all together: Evaluating conversational agents' ability to blend skills. In ACL, page 2030-

---

Jason D Williams, Kavosh Asadi, and Geoffrey Zweig, 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and rein- forcement learning. ACL, page 665-677.
Zeqiu Wu, Michel Galley, Chris Brockett, Yizhe Zhang, Xiang Gao, Chris Quirk, Rik Koncel-Kedziorsk Jianfeng Gao, Hannaneh Hajishirzi, Mari Ostendorf, al. 2021. controllable model of grounded re- sponse generation. In AAAI, pages 14085-14093.
Jun Xu, Haifeng Wang, Zheng-Yu Yu Niu, Hua Wu, Wanx- lang Che, and Ting Liu. 2020. Conversational graph grounded policy learning for tion generation. In ACT., pages 1835-1845.
conversa-
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain ques- tion answering. EMNLP, pages 2013-2018.
Tom Young. Frank Xing, Vlad Pandelea, Jinjie Ni, and Erik Cambria. 2022. Fusing trientrde and open- domain dialogues in conversational agents. In AAAI, volume 36, pages 11622-11629.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. ArXiv, abs/2106. 15200.
Shuo Zhang and Krisztian Balog. 2020. Evaluating con- versational recommender systems via user simulation. In KDD, pages 1512-1520.
Zhuosheng Zhang, Han Zhang, Keming Chen, Yuhang Guo, Jingyum Hua, Yulong Wang, and Ming Zhou. 2021. Mengzi: Towards lightweight yet inge- nious pre-trained models for chinese. ArXiv, abs/21 10.06696.
Kun Zhao, Bohao Yang. Chenghua Lin, Wenge Rong. Aline Villavicencio, and Xiaohui Cui. 2023. Evaluat- ing open-domain dialogues in latent space with next sentence prediction and mutual information. arXiv preprint arxiv:2305.1em /6967.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In ACL, page 654-664.
Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020. Designing precise and robust dialogue response eval- uators. In ACL, page 26

---

A Annotation Details
A.1 Annotation Process
We randomly sample the dialogues from YiWe platform that are licensed to be publicly available. We proceed to sample dialogues from six domains: emotion, digital, legal, education, car, and finance. After collect the dialogues from Wen YiWen platform, we employee crowd-soureing workers annotate the quality labels for the dialogues (0, 1, 2).
The annotation process is an imnotation-chekiing procedure, or repeated annotation-checking proce- dures (if the annotation accuracy of the first round of annotation does not meet the needs). The dialogues are divided into multiple batches for annotation, with each batch consisting of 2000 samples.
For each dialogue, the quality label have been agreed upon and confirmed by crowd-soures workers. After annotate a batch of data, we have a professional quality controller to check the the label accuracy. (The roles and distribution of crowd-source workers can be found in section A.3). Generally, a quality controller randomly samples 100 dialogues and check the label accuracy of this batch including overall annotation accuracy and the accuracy of each category (0, 1, 2). Our acceptance criterion is to achieve overall accuracy of 92% above, and the accuracy for each category no less than 90% for each batch of data.
the accuracy does not meet the thresholds, the entire batch of data is returned, and the existing labels are removed for re-annotation. This annotation and checking process is then repeated until the labeling accuracies surpass the thresholds. Based on the statistics of our annotation, approximately 31 % of batches require re-annotation, which means these data need two more rounds of annotation.
is worth noting that we continually annotate the dialogue quality data until now, And we sample 100k annotated dialogues for the open- open-domain dialogue quality assessment task for the community. We have applied the annotation in actual usage and were able to demonstrate real improvement when we use the data to train a dialogue quality model to determine the quality of real dialogues. The user satisfaction and user revisit ratio in Wen WenYiWen platform have been improved 56%, 23% relatively in the last year after deploying such dialogue quality evaluation.
A.2 Annotation Instructions
For each annotator, we provide instructions to enable them to perform the dialogue quality assessment task, as shown in Figure 5.
Each annotator take the form of a dialogue evaluation as shown in the following Figure 6. This task includes answer whether the information provided need to be verified through external knowledge and choose the reason why you
case of hallucinating problems, rate the quality score from -2, -1, 0, rate the score. The -1 and -2 are rated when there are privacies and personal information contained in the responses and questions respectively. We use annotators to avoid the ethics. We exclude the dialogues that score -2 and 11 for the consideration of ethics. This is the interface translated from Chinese, and the original Chinese interface is shown in Figure 10.
The detailed guidelines for scoring 1, and are shown in the following Figure 7, Figure Figure 9 respectively.
In this task, you evaluate the quality a dialogue provided here.
To correctly solve this task, follow these steps:
1. Read the dialogue carefully, be aware of the information in the questions and responses.
2. Decide whether the information supplied need to be verified through external knowledge.
3. Rate the dialogue quality from 0 scale by its grammaticality, relevance, empathy, proactivity and informativeness.
4. Choose the reasons why you score 0, 1 2.
Figure 5: The task instruction used for dialogue quality evaluation.

---

Dialogue:
Q1 How to retrieve forgotten bank card number?
A1: Hi. You can only retrieve it the bank counter for the customer service. The bank card code issued by various banks, and there are unified regulations for the numbers between different banks. The 6 digits of credit card number represent the issuing bank identification code, also known as the BIN (Bank Identification Number). Different BIN numbers symbolize different bank institutions and card levels. The central bank uniformly assigns business number ranges to commercial banks, and each bank has different number ranges. Once bank card assigned, the customer's basic information, credit limit, and transaction status promptly transmitted to the central bank to prevent money laundering and facilitate data aggregation and unified management A2: Hello?
number refers
central bank's database.
Q2: have forgotten my ICBC (Industrial and Commercial Bank of China Limited) card number.
A3: You still need to go to the according bank counter for help.
Is the information provided in the answers need to be verified through external knowledge?
1
0
Rate the quality score.
2
1
-2
Abandon-orr-phone-numeer
Choose the reason why you score 1.
Less informative Choose the reason why you score O.
Incomplete
Less proactivity
None
No/wrong answers
Not relevant
Not empathy
Grammatical Erro
Not consistent
None
Figure 6: The interface for annotators label the dialogue quality. This is translated version from Chinese.
A.3 Annotators
The annotators are full-time crowdsourcing annotators who continuously annotate the dialogues to serve the commercial Wen YiWen platform. We sample 100k dialogues to be published to the community for the dialogue quality evaluation task.
We maintain annotation team of approximately 26 employees, including 2 trainers, 4 quality- controllers, and 20 annotators. All of them have least colledge degree. The 20 annotators carry out specific annotations, while the muulity-contolleryy randomly review 100 annotations for each batch data after it has been annotated, assess annotatation accuracy against demands. Trainers are responsible for collecting cases inconsistent annotations, discussing and formulating the annotation metrics with us and training the reviewers and annotators using the annotation criteria.
B Baseline Details
Unsupervised methods usually learned without annotated dialogues, which include:
USR (Mehri and Eskénazi. 2020), which a holistic score for quality evaluation using the RoBERTa as the backbone. USR is composed of two sub-scores: USR-MLM and USR-DR, where USR-MLM is caleulated using the log-likelln ligellikehhod values of responses, and USR-DR finetunes the model with randomly sampled negatives.
USL-H (Phy et al., 2020) is composed of VUP, NUP, and MLM, in which NUP and MLM are basically the same as USR-DR and USR-MLM, respectively, and VUP finetunes the model with synthetic negatives.
BARTScore (Yuan et al., 2021), which uses the log-probability of BART generating answers conditioned questions as the score.
Supervised methods are usually learned upon annotated dialogues, including:

---

Dimension
Explanation
Case Study
No Answer
Do not provide answers
Wrong Answers
Provide the wrong answers through the annotators' verification.
Q: How many prefecturee-evel cities are there in Jiangxi Province?
A: About 23.
(Inconsistent with ground-truth answer 11, wrong answer)
Not relevant
The responses are not relevant with the questions.
Q: How many Universities in Zhuhai?
Zhuhai is a city in Guangdong province, and is 140 kilometres (87 miles) southwest of Guangzhou.
Not empathetic
The responses are offensive, don't appropriately reacts to emotional expressions.
am sick and went to the Zhejiang hospital, but have wait for notification for admission. Can wait until then?
A: Ask your doctor directly. You don't even know how get to the hospital as an adult?
Not consistent
Provide the inconsistent answers between different round of answers.
Q: An HP printer runs out of paper and cannot print even after the paper loaded, how should be resolved?
A: The sensor did not detect the presence of paper.
You need to restart the power.
not off the power...
Not Grammatical
The responses are not fluent, readable, and exists grammatical and semantic errors.
Q: My classmates in the always tease me.
When someone likes you, will to get your attention or intentionally pick you. It's actually way of expressing your feelings. That's the sentence meaning.
Figure 7: The detailed guidelines to rate 0 for the quality of dialogue.
Dimension
Explanation
Case Study
Not complete Informativeness
Not sufficient. Provide the basic answers, not complete information
Q: The characteristics of winter in Perth, Australia. Perth has a Mediterranean climate, with wet winters and long, hot summers. The hot weather during summer is caused by heatwaves blowing in from the eastern desert.
Not useful- Informativeness
Not specific and useful. Only give a direct answer the question, and don't provide any related extensions for helping understanding the answers.
Q: When the car dashboard displays an icon of small car with a key inside, what does it mean? A: That is the indicator light for the car's anti-the anti-theft system.
(Absence of explanation, why the system is active how to solve it
Not proactive
When the question is single noun or a relatively simple statement without clear demand, the respondent don't further explore user's needs but provides answers directly.
Q: Tsinghua University
is nationally renowned key university. It member of the C9 League, the Song League, the Association of Asian Universities, the Association Pacific Rim Universities the Tsinghua- Cambridge MIT Low Carbon University Alliance
Figure The detailed guidelines to rate the quality of dialogue.

---

Dimension
Explanation
Case Study
All
The question has been fully addressed. is an informative answer that meets the criteria of relevance, consistency, grammaticality, and empathy.
Q: Let someone do or let someone to do in English?
A: "let someone do" is more commonly used.
A: The verb "let" is a causative verb. After take an object, it is followed by the base form of the verb.
All
The answer completely addresses the emotional feelings of the questioner and provides proactive and comprehensive
Q: How to console someone when their loved one is sick?.
A: Can you give help? At this time, when you can't ' be of direct help, you can send a message of comfort like "Everything will be fine. Let me know there's anything can do to help." Q: Just send that message?
communication. And meets the criteria of relevance, consistency, grammaticality, and empathy.
A: You can also send "Don't worry, good fortune will come to your family. Take care of yourself well." Can send this?
A: It's even better and more thoughtful.
A: Just send it, don't hesitate too much.
Don't worry too much either.
Q: Okay. Thank you?
Figure 9: The detailed guidelines to rate 2 for the quality of dialogue.
RR: THRRPO 19700 WRRRRRSEEAAARRRSEE PONEESSEEE, RIARRE ESTARCRS.
BASKY
18GB
ZERE
Figure 10: The interface for annotators to label the dialogue quality in Chinese.
BLEURT (Sellam et al., 2020), in which BERT-large model is pre-trained on synthetic data and fine-tuned with regression loss;
P-Tuning (Liu et al., 2021), which optimizes the parameters of the prompt vector and freezes other parameters ERNIE;
Prompted ERNIE (3-shot) (Sun al., 2021b), which uses annotated dialogues as in-context examples to prompt an ERNIE model; we also test supervised finctuning setting by finetuning the parameters the prompted model on the training set.

---

C Prompts
Chinese prompt
USER
I 2 2 HI * [lassttaas x-J i
x in M
role (uservtassistaan) W
F:
-04 1
15t. #
25t. :
ASSISTANT
USER
l'role:: 'user', 2rt TTwT::/:.....A.  .........AAAA T 1
S #
ASSISTANT
3t:
USER
['role': 'user', " + 'role': 'assistatt 'content': : *
t { BEE 44
ft 18 S BL
ASSISTANT
USER
l'role": 'user', 'content': BITNE 'role: "assistant"
"content'; : T SRA.GATAAS.TTTNEEEEEEEEEEEEEEE E
B
ASSISTANT
3 2
USER
{ [(dialogue)

---

English prompt (translated version)
USER
will provide you a dialogue between user and assistant. The dialogue is saved in json format, in which each item include the role (user or assistant) and the message content, plcase rate the assistant performance based on the response quality, and the criterion as follow:
Score O, low quality.
Score I, moderate quality.
Score 2, high quality.
Note that you only need to output the number of the score, without any explanation
ASSISTANT
OK, please provide the dialogue and will rate scale of 0-2.
USER
['role": 'user', "content': 'News that does not exceed the cross. Hurry up! Hurry up!', 'role': "assistant', 'content': 'Even the shortest news can hardly be less than 40 words. No more than the cross is the Headline, shown below, flights at the Hong Kong airport are generally normal, small number of cancellations and partial delays, middle school student in Gamma Cygni was physically punished by the head teacher, and the Striated muscle tissue was dissolved. The Financial Committee fully increased its support for the real economy. Train tickets will be robbed tomorrow during the National Day holiday']
ASSISTANT
Score: 0
USER
('role': 'user', 'content': 'How much can I refund 20000 yuan after Ping An Insurance takes effect for days', 'role': 'asssisant', 'content': 'Hello, thank you very much for your question. We can offer a full refund as the 7-day validity period still a hesitation period, and refunds during the hcsitation period are full refunds. hope my answer can help you, and wish you a happy life!']
ASSISTANT
Score:
USER
['role": 'user', "content': 'Where can I find out if I have been prosecuted", 'role': 'assistant', 'content': 'If you usually need to retrieve information, you can choose to go the public security bureau or court to provide your identity and relationship with the case or party you want to inquire about, as well as your ID card and household registration book, and explain the reason for your inquiry (the court will ask you to fill out form or directly regis- ter it for you). You can bring these materials with you, and the court can help you investigate them.') 1
ASSISTANT
Score: 2
USER
{ (ddialogue)) }


Reviews:
[{"review_id": "NEAVQzjftz", "reviewer": "Reviewer_JK85", "summary": "", "questions": "OpenAssistant Conversations dataset (OASST1) is surprisingly absent in this paper. Is there a reason for this?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable feedback. The following are our responses to your concerns in order.\n\n**Ethics**: As we mentioned in the ethics statement section, all the dialogues we collect have been authorized by the WenYiWen platform. We collect dialogues that are licensed by both the users and the respondents to be publicly available on the internet. \nFurthermore, annotators are asked to determine whether there are privacy and personal information present besides the quality labels (0, 1, 2). Dialogues with privacy concerns are excluded before release, as mentioned in the Appendix A.2.\n\n**Annotators**: The annotators are full-time employees who have continuously annotated the dialogue quality for WenYiWen for over a year, and all of them hold a college degree at least. They have undergone training to annotate the dialogue quality to meet the requirements before the formal annotation. Thus, we regard them as the experienced annotators. Moreover, the quality controllers are those annotators with higher annotation accuracy.\n\nThe annotation team comprises 26 employees, including 14 females and 12 males. We promise to supply these information at the Appendix A.3 in the next revision.\n\n**Data Collection Process**: After we randomly sample the dialogues from WenYiWen platform that are licensed to be publicly available, we proceed to sample dialogues from six domains: emotion, digital, legal, education, car, and finance. Finally, we filtered out dialogues containing privacy or personal information. This is achieved through our annotation process, where a score of -1 or -2 is assigned if there are privacy or personal information in the questions or responses. We subsequently filter out the dialogues with scores of -1 or -2.\n\n**Response for Question 1**: Recently the open-assistants are constantly and rapidly improved and their response style are updated accordingly. The variety of open-assistants with their different machine generated styles, are different from real users settings. Therefore, we focus on the ultimate goal that enables machines generate human-epistemic high-quality responses. We directly annotated real human conversations for the dialogue quality assessment.\n\nThanks again for providing these insightful feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The amount of effort that has gone through developing this dataset is unmatched. This dataset is times larger than any other dataset in this domain. The authors claim users have to pay to talk to the experts; if true, I suspect the conversations are of high quality. Considering the conversations are in Chinese, this dataset is a valuable resource for future studies.", "reasons_to_reject": "This paper could benefit from providing more details and clarity about the data collection process and the annotators. The Data Card and Human Evaluation Data Sheet are missing from this paper. Providing this information will facilitate the use and reproducibility of this work.", "questions_to_authors": "OpenAssistant Conversations dataset (OASST1) is surprisingly absent in this paper. Is there a reason for this?", "paper_topic_and_main_contributions": "This paper introduces a large-scale dialogue quality assessment dataset (DiQAD) that could be extremely valuable for the training and evaluation of future dialogue systems. The dialogues are annotated on six unique dimensions: grammaticality, relevance, consistency, empathy, proactivity, and informativeness. Furthermore, dialouages are from a wide variety of topics. Ultimately, the authors evaluate several state-of-the-art models on this dataset and provide a baseline for future studies.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "N/A."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I found the paper slightly hard to read and understand, though I suspect this is due language barrier. I would suggest the authors have a native English speaker proofread the paper. For example, the language of the dataset should be mentioned earlier than on-page, line 345."}}, {"review_id": "VLHEpRPZ84", "reviewer": "Reviewer_uFZJ", "summary": "", "questions": "1. For experiments on user sentiment influence, how do you know a dialogue is positive or negative? I do not see this type of annotation in \n2. Does the collected dialogues not need any data cleaning process? If so, can you provide cleaning rules for this process?\n4. In lines 449-453, have you tried different prompts for LLMs to score dialogues? Like Chain of Thought method to guide LLMs? It is very surprising that LLMs have such low performance.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable feedback. The followings are our responses to your concerns in order.\n\n**Annotation with Reasons**: We annotate a holistic label fusing the six dimensions to align with human perspective for quality assessment. This mapping assigns distinct quality levels (e.g., 0, 1, 2) to the various dimensions. Moreover, each annotator is asked to provide explicit reasons when label (see Sections 4.2 and the Appendix A.2 for more details). \nThere are two advantages of  providing explicit reasons: (1)  they help annotators to improve annotating precision and (2) they can be used to determine which specific standards that are unsatisfied for low quality. These explicit reasons might be more suitable for the evaluation on generation-based models. \n\n**Response to Question 1**: In the detailed analysis of whether user responses' sentiments would impact our quality assessment, we distinguish positive and negative sentiments via key words (\"ok Thanks\" or \"I don’t think so\"). We append this identified emotional expressions to the original dialogue content to validate any changes in quality assessment. We conduct this analysis to determine whether our model has learned shortcuts for quality assessment or not.\n\n**Response to Question 2**: Our data is randomly sampled from WenYiWen platform and we only collect those dialogues licensed by both the users and the respondents to be publicly published on the internet. We then sample dialogues from six domains: emotion, digital, legal, education, car, and finance. Finally, we filtered out dialogues containing privacy or personal information. This is achieved through our annotation process, where a score of -1 or -2 is assigned if there is privacy or personal information in the questions or responses. We subsequently filter out the dialogues with scores of -1 or -2. \n\nThese descriptions are spread across Section 4.1, the ethics statement section, and Appendix A.2. We appreciate your feedback and we promise to provide a clearer of our data processing in Section 4.1 in the next revision.\n\n**Response to Question 3**: We utilized the explicit reasons and labels in the annotations for few chain-of-thought examples. We demonstrate a slight improvement in the quality assessment performance of the few-shot approach compared with the zero-shot approach.\n| Model | Acc | UAR | Kappa | Spearman | Pearson | Precision | Recall | F1 | \n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Ernie | 68.40 | 40.04 | 50.40 | 52.19 | 52.05 | 67.35 | 66.75 | 67.02 |\n| ChatGPT | 39.30 | 21.88 | 4.70 | 12.83 | 12.51 | 37.61 | 36.47 | 36.28 |\n| ChatGPT+CoT | 41.00 | 24.91 | 10.68 | 17.79 | 17.33 | 30.77 | 31.14 | 30.38 |\n\nThanks again for providing these valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces an annotated open-domain dialogue dataset with 100k dialogues included, and the annotated dialogues are good for dialogue evaluation research.", "reasons_to_reject": "1. The quality annotation process is confusing. There are six standard rules for annotators but only one annotation for each dialogue. For instance, grammaticality, relevance, consistency, and empathy are related to low quality. However, there is no annotation to know among the four rules, which one causes a dialogue to be annotated as low quality. \n2. The quality annotation dose not contribute to generation-based method. If researchers want to evaluate generation ability of a dialogue model, there is no labels for evaluating grammaticality, relevance, consistency, empathy, proactivity and informativeness of generated responses.", "questions_to_authors": "1. For experiments on user sentiment influence, how do you know a dialogue is positive or negative? I do not see this type of annotation in \n2. Does the collected dialogues not need any data cleaning process? If so, can you provide cleaning rules for this process?\n4. In lines 449-453, have you tried different prompts for LLMs to score dialogues? Like Chain of Thought method to guide LLMs? It is very surprising that LLMs have such low performance.", "paper_topic_and_main_contributions": "This paper introduces a large end-to-end open-domain dialogue dataset with quality annotations for dialogue evaluation research. The dialogues are collected from a public question/answer platform, covering six different domains. The authors propose to annotate the collected dialogues into three different quality levels (0, 1, 2 for low/moderate/high qualities) based on six standard rules: grammaticality, relevance, consistency, empathy, proactivity and informativeness. In total, there are 100k dialogues are annotated. Results of multiple baseline models are provided to serve as baselines for future dialogue evaluation research.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 030. conversational dialogues --> just \"dialogues\"."}, "extra_text": {}}, {"review_id": "3bmUM1xB90", "reviewer": "Reviewer_NmNL", "summary": "", "questions": "+ [Solved] Question A: In Lines 375-378, you wrote '1,958 samples classified as score 0, ..., and 4,414 samples classified as score 0'. It must be a fault and what exactly the numbers of samples scored as 0, 1, and 2?\n+ [Solved] Question B: In Table 4, why did the UAR, Spearman correlation, and Pearson correlation drop in legal and car domains when using these in-domain data?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable feedback. The followings are our responses to your concerns in order.\n\n**Our Contribution**: The primary purpose of our paper is to provide human-epistemic assessments of the open-domain dialogues in real users settings, rather than providing new metrics. Recently released datasets comprise dialogues that are conversed between recruited annotators and bots (e.g., FusedChat (Young et al., 2022)) or between annotators themselves (e.g., NaturalConv (Wang et al., 2021), TopicalChat (Gopalakrishnan et al., 2019), BlendedSkillTalk (Smith et al., 2020), WizInt (Komeili et al., 2022)), or happened in certain scenarios like English practice (e.g., DailyDialogue (Li et al., 2017b)). The conversational topics and contents are handcrafted and limited compared with the real users’ dialogues, making them less suitable for the real user dialogue evaluation. In contrast, we present a dataset of real user-to-human dialogues with human-epistemic quality labels. \n\nThe different lengths demonstrate that the dialogues in open assistants and dialogues in real-user scenarios do exhibit differences. The responses generated by open assistants have a distinct machine style, towards longer responses to enhance interactions with users. However, in real user dialogues, the goal should be to have as few turns as possible to meet the user's needs. With the rapid improvement of open assistants, our subsequent work will sample chinese dialogues from the known open assistants and annotate them using the existing process, thereby expanding the diversity of the dataset.\n\n**Response for Question A**: The testset contains 10,000 samples, among 1,958 of them are scored 0, 3628 samples are scored 1 and the reset 4,414 samples are scored 2. Thank you for pointing out this writing error and we will correct it in the next revision.\n\n**Response for Question B**: The metrics including UAR, spearman correlation, and pearson correlation in legal and car domains have slightly dropped, but the change is not significant. We have observed that the amount of data in these two domains is relatively small (around 5% to 6%). The model demonstrates good generalization for these two domains, with similar performance using in-domain and out-of-domain data. Additionally, the model trained has exhibited somewhat insufficient training. Therefore, adding or excluding training data of these two domains doesn't significantly impact the model's performance, and it's possible that some of the metrics for the in-domain performance are slightly lower.\n\nThanks again for you valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed large-scale annotated dataset for dialogue evaluation is certainly a good contribution to the NLP community.\n2. Empirical studies are extensive, involving various comparison methods and diverse evaluation metrics. The experimental analysis also provides some valuable conclusions for this field.", "reasons_to_reject": "1. The proposed aspects of human-centered evaluation are not novel. USL-H (Phy et al., 2020) has already considered Grammaticality, Relevance, Consistency, and Empathy. Informativeness is also a well-studied aspect in the field of knowledge-grounded dialogue systems. There are also many discussions on proactive dialogue systems (Liao et al., Proactive conversational agents, WSDM 2023 tutorial).\n\n2. The length of dialogues in the dataset is short (less than 512 tokens on average), which may not be suitable for the real-world scenario that requires long-term dependency.", "questions_to_authors": "+ [Solved] Question A: In Lines 375-378, you wrote '1,958 samples classified as score 0, ..., and 4,414 samples classified as score 0'. It must be a fault and what exactly the numbers of samples scored as 0, 1, and 2?\n+ [Solved] Question B: In Table 4, why did the UAR, Spearman correlation, and Pearson correlation drop in legal and car domains when using these in-domain data?", "paper_topic_and_main_contributions": "This paper proposes a large-scale Chinese dataset for dialogue evaluation called DiQAD containing 100k dialogues. Different from the previous work, they focused on human-centered evaluation with proposed six aspects: grammaticality, relevance, consistency, empathy, proactivity, and informativeness. And the dialogues are directly crawled from real-world scenarios instead of created by the annotators. After collecting the original dialogues, their annotation team annotates the dialogue-level quality in the six aspects. Since there are two annotator trainers and four quality controllers in the team, the quality of annotations can also be guaranteed. After constructing the dataset, the authors also conduct extensive experiments to evaluate various comparison methods for classifying the dialogue quality. This benchmark can also be considered as a contribution for future researchers.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "### Typos\n+ Line 394: two types -> three types\n+ Line 956: colledge -> college\n\n### Grammar\n+ Line 540: random -> randomly"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a large dataset (100k dialogues) for dialogue quality assessment; the dialogs are in Chinese and are annotated on six dimensions: grammaticality, relevance, consistency, empathy, proactivity and informativeness, and covers six different domains. The dataset is annotated manually despite its large size. The reviewers appreciate the analyses conducted on the dataset. The reviewers however point to additional work that should be cited and have questions about the annotation process and the annotators, to which the authors replied satisfactorily in the response (The annotators are full-time employees who have continuously annotated the dialogue quality for WenYiWen for over a year, and all of them hold a college degree at least. They have undergone training to annotate the dialogue quality to meet the requirements before the formal annotation. Thus, we regard them as the experienced annotators. Moreover, the quality controllers are those annotators with higher annotation accuracy. The annotation team comprises 26 employees, including 14 females and 12 males.)
Decision: Accept-Findings