Automatic Evaluate Dialogue Appropriateness by Using Dialogue Act
Authors: Bao Chen
Yuanjie Wang
Zeming Liu
Yuhang Guo
Venue: EMNLP 2023 Findings (2023)
Keywords: Automatic Dialogue System Evaluation; Dialogue Evaluation; Dialogue System
TL;DR: This paper evaluates the appropriateness of chatbot responses by comparing the similarity of dialogue act patterns between human-machine dialogues and human-human dialogues.

Abstract: Evaluation of dialogue systems requires assessing various aspects, among which appropriateness holds significance as a core element of communicative language competence. However, current evaluations heavily rely on human judgments, which are time-consuming, labor-intensive, prone to biases, and lacking objectivity. In this paper, we introduce Dialogue Act Appropriateness (DAA), a novel method that utilizes the underlying patterns of dialogue act transitions to evaluate the appropriateness of chatbot responses. We learn transition patterns from human-human dialogue corpora, evaluating chatbot appropriateness by measuring the similarity of their transition patterns to those observed in human-human dialogues. To validate DAA, we annotate a test dataset by manually evaluating the appropriateness of dialogues from multiple chatbot systems. The experimental results demonstrate a strong correlation between our evaluation metric and human ratings, establishing the reliability of DAA as a measure of dialogue appropriateness.

Automatic Evaluate Dialogue Appropriateness by Using Dialogue Act
Bao Chen' Yuanjie Wang' Zeming Liu 2
Yuhang Guo 1
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China
School of Computer Science and Engineering, Beihang University, Beijing, 100191, China {chenbao, guoyuhangJ@bit. edu. cn njwangyuanjiee0163.om com zmliu@buaa. cew.ouuaaaeed....o
Abstract
Appropriateness is fundamental characteris- tic language usage, serving as core element of communicative language competence and an evaluative criterion for language users or learners (Grundy, 2008). It encompasses the notion of using language in manner that is suitable, contextually relevant, and socially acceptable in various com- munication settings. Evaluating the appropriate- ness of language use crucial for assessing the proficiency and effectiveness of individuals' com- municative skills in both spoken and written forms of expression. However, the current assessment appropriateness mainly relies on human judgment, which is not only time-consuming and laborious, but also leads to potential bias and lack of objectiv- ity.
Evaluation of dialogue systems requires assess- ing various aspects, among which appropriate- ness holds significance communicative language competence. How- ever, current evaluations heavily rely human judgments, which are time-consuming, labor- intensive, prone to biases, and lacking objectiv- ity. In this paper, introduce Dialogue Act Appropriateness (DAA), a novel method that utilizes the underlying patterns of dialogue transitions to evaluate the appropriateness of chatbot responses. We learn transition patterns from human-human dialogue corpora, evaluat- ing chatbot appropriateness by measuring the similarity their transition patterns those observed in numan-human dialogues. To vali- date DAA, we annotate a test dataset by manu- ally evaluating the appropriateness of dialogues from multiple chatbot systems. The experimen- tal results demonstrate a strong correlation be- tween evaluation metric and human ratings, establishing the reliability of DAA measure dialogue appropriateness.
core element of
According to Fetzer (2004), the concept of ap- propriateness is closely tied to the effective exe- cution of dialogue acts. Appropriateness serves as necessary and sufficient condition for carry- ing out dialogue acts properly. These acts play a crucial role identifying the actions performed by utterances a conversation, shedding light on their intended meaning and purpose in terms of illocutionary force, as discussed by Searle (1970). By analyzing dialogue acts, researchers can gain a deeper understanding of the interactions between speakers and the objectives they aim to accomplish. This comprehension is further enhanced through the works of Allwood al. (1992): Stolcke et al. (2000), who emphasize the significance of dialogue acts in unraveling the functions, intended mean- ings, and purposes of utterances. Inspired by this understanding, we consider the dialogues in human- human corpora to exhibit high level of appropri- ateness, and the dialogue act pattern within these conversations is deemed suitable. Therefore, we can leverage human-humal dialogues guiding reference to assess the appropriateness of human- chatbot dialogues, considering the patterns of dia- logue act transitions.
1 Introduction
Automatic evaluation of text generation quality has long been a challenge the field of natural lan- guage processing (NLP) (Liu et al., 2016). When it comes to assessing the quality of dialogue sys- tems, the task becomes even more complex and multifaceted. Dialogue quality encompasses vari- aspects, including coherence, fluency, engage- ment, and appropriateness of responses (Mesgar al., 2020; Mchri and Eskenazi, 2020b; Jiang et al., 2023). Depending on the application and user requirements, different aspects of dialogue quality may need to be evaluated (See et al., 2019). Consequently, there is growing demand for di- verse evaluation metrics capable of capturing the different facets of dialogue quality.
"Correspoding author.
'The code and data is available at https:: comm
xbaê/DAA

---

Specifically, we propose an automatic evalua- tion method called Dialogue Act Appropriateness (DAA) to assess appropriateness based on dialogue acts. Initially, we employ a dialogue act classifier to categorize dialogue utterances into sequences of dialogue acts. Subsequently, we model the di- alogue act transition patterns in human-human di- alogue corpora to capture the typical patterns of dialogue act transitions, which serve as the refer- ence dialogue act patterns. Finally, we compare the dialogue act transition patterns of the chatbot's responses with the reference dialogue act patterns to measure the appropriateness of the chatbot's re- sponses.
2 Method
We posit that human-human dialogues exhibit a higher level of appropriateness in the use of di- alogue acts. Therefore, by measuring the simi- larity between the dialogue act patterns observed in human-chatbol dialogues and those in human- human dialogues, we can assess the appropriate- ness of chatbot responses. The process of DAA, as depicted in Figure 1, involves three key steps. Firstly, we classify the utterance sequences in a multi-tum dialogue into sequences of dialogue acts, enabling us to learn and compare the transition patterns of dialogue acts, Next, we model the tran- sition patterns of dialogue acts among interlocutors using a human-human dialogue corpus. This al- lows us to capture the inherent patterns of dialogue act transitions that occur in human conversations, serving as a reference for assessing appropriateness. Finally, we compare the dialogue act transition pat- terns observed in given human-chatbot dialogue with the learned patterns from the previous step, providing an appropriateness score based on the degree of similarity.
To the best of our knowledge, there is currently a lack of annotated dialogue datasets targeting prag- matic appropriateness. To facilitate the automated evaluation of pragmatic appropriateness and vali- date the efficacy of DAA, we curated an evaluation dataset. The dataset comprises conversation logs from six evaluation objects, including five popular chatbot systems and one human for comparison. Following the data collection process, seven anno- tators independently assigned ratings to evaluate the appropriateness of each turn of responses from the objects.
2.1 Dialogue Act Classification
We trained a dialogue act classifier to map the ut- terance sequence in dialogue to sequence of dialogue act labels, allowing us to model the de- pendency among dialogue acts. A context-aware dialogue act classifier enhances the accuracy of dialogue act classification, partially due the clas- sifier's utilization of dialogue act transition patterns present in DA training data. Our approuch involves evaluating the similarity of these transition features across distinct corpora, such as numan-human dia- logues and human-chatbot dialogues, to establish scoring criteria. So, we deliberately restrict the DA classifier from utilizing contextual information prevent potential biases arising from the classifier learning and incorporating prior transition patterns from the DA dataset. Specifically, we fine-tuned a transformer- based pre-trained model as our clas- sifier. We concatenated the [CLS] token with the utterance in the dialogue as the input to the pre- trained model, considering the final hidden state of the [CLS] token as the representation of the entire sequence. The representation hele of the [CLS] token underwent linear projection and tanh acti- vation, followed by softmax classifier to obtain the probabilities of dialogue act labels. We used cross- cross-entropy as the loss function in the training
Experimental results demonstrate strong cor- relation between the appropriateness scores gener- ated by DAA and the annotators. This indicates that DAA effectively captures the notion of appro- priateness in dialogues and aligns well with human perceptions.
Our contributions are summarized as follows:
We quantify human subjective evaluation of dialog appropriateness using an inter- pretable approach, providing a stable evalua- tion metric called Dialogue Act Appropriate- ness (DAA).
To evaluate the effectiveness of DAA and fa- cilitate further research, conduct human evaluations six different evaluation objects and build a dedicated test dataset for evalu- ating pragmatic appropriateness.
Experiments show strong correlation be- tween our proposed metric and human judg- ments, demonstrating the effectiveness of DAA in accurately assessing dialog appropri- ateness.

---

Dialogue act classification
Conventieralloponne
you living?
Question
like build then live them.
Statement-non-opimeee
B: That sounds like of fun
PLM
Statement-opinion
B: Do you have any hobbies that you do in your spane time?
Yes-No-Quosiion
A: like to bake stuff, like bread and cake and brownies.
Stateneni-non-oppniio
yummy!
Slatemenn--ppizion
conversation
dialogue act labels
Prresponse act context act)
Modelling dialogue
transition human-huumn dialogues
PLM
Evaluating appropriateness human-chathot dialogues
context act
inappropriate appropriate
response azt
Figure 1: An illustration of D The colored matrix represents the identified patterns of dialogue act transitions from human-human conversation corpora. Darker cells indicate a more frequent occurrence of these dialogue act transitions in human dialogues.
process.
terance of a turn usually serves as a continuation of the previous speaker's content, while the sub- sequent utterances typically complement the first utterance. Therefore, we do not consider the transi- tion relationships of dialogue acts within the same turn.
2.2 Modelling Dialogue Act Transition in Human-Human Dialogues
In this step, we utilize a well-trained dialogue act classifier to annotate dialogue act labels from human-human dialogues and calculate the con- ditional probability distribution of response acts given the dialogue history. A dialogue can be di- vided into turns, where two speakers take turns speaking. Odd turns and even turns are generated by different individuals. Each turn consists of mul- tiple utterances spoken by the same speaker. We denote the j-th utterance in the i-th turn of the dia- logue u Following the method described in section 2.1, we assign dialogue act label (DA) to utt.,--
Hence, we treat the last utterance of the speaker in turn, utt; as the context, and the first utterance of the next turn, utt;+1,1, as the re- sponse to this context. From a dialogue consist- ing of N turns, we obtain N - 1 context-respons. pairs. We then calculate the conditional probability P ((re(ppoo)))) D DA D(conlezz)) from human dialogues. Finally, establish transition proba- bility matrix for dialogue acts the turn boundary.
2.3 Evaluating Appropriateness in Human-Chathot Dialogues
Considering that an utterance often responds to the last utterance from the other speaker, we over- look the weak connections between dialogue acts in non -dj-adjacnnt turns. Our focus is on assessing the appropriateness of response acts relative to the dialogue act of the dialogue partner. The first ut-
To measure the appropriateness of a chatbot's 's re- sponses, we first the same dialogue act classifier as in section 2.2 obtain the sequence of dialogue acts in the dialogue. To focus on evaluating the chatbot's responses in the evaluation process, we mask out the human responses to the chatbot and only consider the chatbot's responses to humans.
AA detailed list of dialogue act lahels can he found at http:
/cmmppaag www.chhiiipt.. paristopherpott net/swda html#tags

---

We treat the last utterance of human turn as the context and the first utterance of the subsequent bot turn as the response this context. We then use conditional probability of dialogue act transition to measure the appropriateness of each context- response pair:
the context. Their responsibility was assess the appropriateness of the provided chatbot response, assigning an integer score within the range of 0 to 4, where 4 represents "very appropriate" and O signifies "very inappropriate"
We gathered 300 dialogues involving six entities, resulting in total of 2,716 responses. We em- ployed seven annotators who individually assigned turn-level scores to all the collected responses. The average score for each turn within a dialogue was used to as the dialogue-level score, and the average score of all dialogues pertaining to an object was used as the object's score.
DAA- P(DA(response:)) D D
Finally, the appropriateness the dialogue level is computed by taking the geometric mean of the appropriateness scores at the turn level, where N represents the number of context-response pairs:
To examine the inter-annotator consistency, we calculated the Pearson and Spearman correlation coefficients between the scores of one annotator and the mean scores of other annotators for dif- ferent chatbots. The average Pearson correlation coefficient was found to 96.19%, and the aver- age Spearman correlation coefficient was 92.65%.
N DAA N II DAA,
3 Evaluation Data Construction
While annotated data exists for overall appropriate- ness and semantic appropriateness, to the best of our knowledge, we did not come across any anno- tated data specifically designed for evaluating prag- matic appropriateness in conversations. In order to assess the effectiveness of DAA, we gathered con- versation logs from six objects with different levels of intelligence. Each response of these conversa- tions is manually annotated with score indicating its appropriateness.
4 Experimental Setup
4.1
Training Datasets
The dialogue act classifier utilized in our study is trained on the Switchboard dialogue act corpus (SWDA) (Stolcke al., 2000). The transition prob- ability matrices for dialogue act transfer were com- puted based on different human-human dialogue datasets, namely DailyDialog (Li et al., 2017), Per- sonachat (Zhang et al., 201 (), Reddit Corpus (Lee et al., 2019), and Cornell Movie-Dialoge Corpus Mainescu-Niculscc--Mee
3.1 Evaluation Objects
In our experiment, we selected five famous chat- bots as the evaluation objects: blender (Roller et al., 2021), cleverbot' jabberwacky". meena (Adiwar- dana et al., 2020) and mitsuku. Additionally, we included a human for comparison. We collected 50 dialogues from each object. All dialogues were sourced from publicly available data. The dia- logues of Meena, Mitsuku, and Human were ob- tained from Google Research', while the dialogues of Blender were soureed from the ParlAI platform
and Lee, 2011). These diverse datasets are chosen explore the impact of dialogue domains and styles.
4.1.1 Dataset for Dialogue Act Classification
The switchboard dialogue act corpus (Stolcke al., 2000) is widely used dataset for studying dia- logue act classification. It consists of telephone conversations recorded between two participants, covering wide range of topics and conversational styles. Each utterance in the corpus annotated with a dialogue act label, indicating the intention or function of the speaker's utterance. We chose the SWDA close resemblance daily conversation. To facil- itate our experiments, we utilized preprocessed version of the dataset, which can be found here?. After preprocessing, the dataset consisted of total
3.2 Manual Annotation
Following Mehri and Eskenazi (2020a), we em- ployed 5-point Likert scale for the manual quality assessment of appropriateness. In this process, we provided annotators with chatbot response and the preceding utterance of the conversational partner as
training data due to its content's
'https: www.eeewe...eeeeeeeeeeeeeeeeeeee com.eeicoonoessssioonnn http:://ww.jabberwaccyy con/
hoogle-rrseerchhttte///ssteeerre
'ht //gi thub. com/NathanDuran/ Switchhoordd-Crpus Corpus

---

of 199,740 utterances. The maximum length of an utterance in the dataset was 132, while the mean length was 9.62.
maximum input sequence length to 256 accom- modate the data. To extract appropriateness fea- tures human dialogues, we utilized the NLTK toolkit 10 to split each dialogue turn into utterances and annotated these utterances using dialogue act classifiers.
4.1.2 Human-Human Dialogue Datasets
Daily Dialog (Li et al.. 2017) is a high-quality multi-turn dialogue dataset that contains conver- sations about daily life. It is cleaner compared to other corpora and covers wide range of topics for casual conversation. The dataset includes total of 13,118 dialogues, with an average of 7.9 speaker turns per dialogue. The average number of tokens per dialogue is 114.7, and the average number of tokens per utterance is 14.6.
5 Results and Analysis
In this section, we present the correlation of DAA with human ratings and conduct a comparative anal- ysis with other appropriateness metrics, then ad- dress the following research questions (RQs): (1) Does DAA provide a stable and objective evalu- ation of dialogue appropriateness? (2) Is DAA
Personachat (Zhang et al., 2018) is crowd- sourced dataset collected through Amazon Mechan- ical Turk. In this dataset, each pair of speakers conditions their dialogue a given profile. The dataset consists of 1,097 dialogues with total of 162,064 utterances.
sensitive to the accuracy of the dialogue act classi- fier? (3) How does the domain of the training data impact the evaluation of dialogues? (4) How can interpret the appropriateness scores provided by DAA? (5) How does the direction of context- response pairs affect the performance DAA?
Reddit Corpus (Lee et al., 2019) is a dataset used Task 2 of the DSTC 8 Competition. The data is crawled from 1.000 relatively non-toxic sub- reddits on Reddit, covering a period of 12 months from November 2017 to October 2018. The train- ing set contains 5,085, 13 dialogues.
5.1 Correlation with Human Ratings
Following previous work (Jiang et al., 2022), we employed Pearson and Spearman correlation coef- ficients to assess the performance of the proposed method. The Pearson coefficient is used to measure the linear correlation between two continuous vari- ables, while the Spearman coefficient evaluates the statistical dependence between the rankings of two variables. Higher correlation coefficients indicate a stronger alignment between DAA and human rat- ings. Table 1 and Table present the Pearson and Spearman correlation coefficients between DAA and human ratings. We conducted extensive exper- iments using multiple pre- pre-trained models and cor- pora. The results demonstrate strong correlation between DAA and human evaluation, confirming the effectiveness of DAA. The performance of the base model is weaker compared to the large model, despite their similar classification accuracy. This difference can be attributed to the large model's 's ability to capture more comprehensive features, re- sulting in improved performance. The best perfor- mance was achieved by employing the RoBERTa- large model the dialogue act classifier and using the PersonaChat corpora.
Cornell Movie-Dialogs Corpus (Danescu- Niculescu-Mizii and Lee, 2011) is large metadata- rich collection fictional conversations extracted from raw movie scripts. It includes 220,579 con- versational exchanges and a total of 304,713 utter- ances.
4.2 Implementation Details
We employed the base and large versions of BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), BART (Lewis et al., 2020), and Ernie 2.0 (Sun al., 2020) as the dialogue act classifiers. This choice was made based on several factors, includ- ing their popularity the field, their strong perfor- mance in various natural language processing tasks, and their availability as pre-trained models. BART, BERT, and RoBERTa were obtained from Hugging- face Transformers while Ernie 2.0 was obtained from PaddlcHub". These transformer-batt els were fine-tuned for 10 epochs, and the classi- fiers that demonstrated superior performance on the validation set were selected for subsequent ex- periments. Given that the longest sentence in the SWDA dataset consists of 132 words, we set the
mod-
"https: http://ghhuu.cccogggggnnnnee//ee "https://githuu httpp/twww..coeeeeeeeeeeeeeeeeaa
wnttp://ww.nltk.

---

Model
Daily Dialog
Persona Chat
Reddit
Cornell
Turn-Level
Dialogue-Leve
System-Level
Method
p
r
r
e
Bart,
85.05
83.34
79.69
76.39
FED
14.06
12.99
16,50
11.81
55.29
42.86
C-PMI
4.51
4.72
2.83
2.61
13.21
8.57
Bert,
93.28
92.62
88.59
82.29
DAA
21.67
21.19
36.16
36.88
89.05
87.38
Ernie
90.95
79.38
84.40
79.59
RoBERTa;
91.45
86.43
85,49
88.15
Table 3: A comparison with the popular method FED
Bart;
92.54
93.22
88.67
83.15
(Mehri and Eskenazi, 2020a) and the recently proposed dataset. Here,
Bert,
92.83
92.54
91.28
86.58
C-PMI method (Ren al., 2023) r and represent the Pearson correlation coefficient and Spearman correlation coefficient, respectively.
Eriee
92.15
88.63
82.75
74.65
RoBERTa;
97.66
98.70
93.60
92.17
Table I: Pearson correlation coefficients (%) between human ratings and DAA. All results are statistically significant with p-value < 0.05
DAA
A-a1
97.73
96.90
A-a2
97.85
95.23
A-23
97.13
89.26
Model
Daily Dialog
Persona Chat
Reddit
Cornell
A-a4
96.68
97.81
Bart,
77.14
88.57
77.14
71,42
A-a5
97.41
96.73
Bert,
88.57
94.28
88.57
77.14
A-a6
97.33
99,40
Ernie,
94.28
82.85
88.57
88.57
A-a7
97.85
98.02
RoBERTa,
94.28
94.28
88.57
88.15
Average
97.43
96.19
Bart;
88.57
88.57
88.57
88.57
Bert; 88.57
88.57
88.57
88.57
Table 4: The Pearson correlation coefficients (%) be- tween the ratings of annotator a, and the mean ratings of other annotators. DAA included for comparison.
Ernie 94.28
94.28
94.28
71.42
RoBERTa; 94.28
94.28
88.57
88.57
Table
Spearman correlation coefficients (%) between human ratings and DAA. All results are statistically significant with p-value < 0.05
DAA method exhibits a relatively stable correlation with the mean ratings of other annotators, ranging from 96.68% to 97. 97.85%. In contrast, the human annotators demonstrate more significant fluctua- tions in their correlations with the mean ratings, ranging from 89 89.26% to 99.40%. These findings in- dicate that DAA can provide stable and objective evaluation dialogue appropriateness.
5.2 Comparison with Other Appropriateness Metries
Table 3, we compared DAA with a popular method FED (Mchri and Eskenazi, 2020a) and the recently proposed C-PMI method (Ren et al., 2023) on our dataset. Since these methods do not evaluate the aspect of pragmatic appropriateness, we con- ducted comparison using their semantic appropri- ateness dimension. Our proposed DAA approach achieved the highest correlation with human ratings across level, dialogue-Ieve level, and system-level evaluations. is worth noting that C-PMI has made improvements over the FED method, enhancing its performance in terms of semantic appropriateness (as reported in their paper). However, in the dimen- sion of pragmatic appropriateness, which is our focus, it exhibited performance decline.
5.4 RQ 2: The Effect of the Dialogue Act Classification Accuracy
We experimented with eight pre-trained models as dialogue act classifiers. The models demonstrate relatively high accuracy on both the validation set and test set when considering only single utter- ance input, without incorporating additional con- text features. Our focus lies in capturing the tran- sition patterns of dialogue behaviors rather than specific dialogue act labels. With an accuracy of at least 73%, all models are capable of annotating dialogue acts. To investigate the impact of classi- fier accuracy on DAA, we analyze the relationship between classifier accuracy and the correlation of DAA with human ratings. The Pearson correlation between DAA and different levels of classifier ac- curacy illustrated Figure 2. A linear fit was attempted, yielding an R-squared value of 0.063. This indicates that the effectiveness of DAA is not
5.3 RQ 1: Stability
Table presents the Pearson correlation coefficients between the ratings of annotator a, and the mean ratings of other annotators. We compare them with DAA ratings obtained from Roberta-large on the PersonaChat dataset. It can observed that the

---

significantly influenced by the accuracy of the dia- logue act classifier.
human turn, the human poses Yes-No-Question. but the bot's response veers off-topic by asking an unrelated question, leading to low ratings from both the human annotator and DAA. It noteworthy that the dialogue act classifier erroneously labeled the dialogue act of the third bot turn as "Wh- "Wh-Question" instead of the correct label. However, this misclas- sification does not impact the consistent ratings assigned by DAA, we employed the same di- alogue act classifier for annotating the chatbot's dialogue logs and for modeling appropriateness us- ing the training corpus. Thus, such classification errors are mitigated during the modeling process. Our DAA method allows for pinpointing specific inappropriate response utterances and provides a reason (inappropriate dia- logue act) for the assessment.
Dailydistng Cornil
Pearsin Coneraatinn
50-
85-
Bett.
Emity
Roberta,
Bart:
Bert
Roborta:
74.8
75.4
75.6
76.0
Classifler
Figure 2: The scatterplot illustrates the Pearson correla- tion of DAA with different test accuracy of dialogue act classifiers.
5.7 RQ 5: Ablation Study
Human-chatbot dialogues involve two types of turn transitions: from the bot to the human and from the human to the bot. As mentioned in Section 2.3, to focus on assessing bot responses, DAA evalu- ates only the bot's reply to human. In Table 6, we compare the effect of considering the direction of responses the performance of DAA. We mea- the effectiveness and stability of the models by calculating the mean and standard deviation of the Pearson and Spearman correlation coefficients between DAA's scores and human ratings. We evaluate the performance of both approaches on different corpora. The results demonstrate that con- sidering the direction of responses yields the best performance, maintaining the highest correlation with human ratings and exhibiting greater stability. Notably, DAA shows minimal performance loss when trained on the Reddit and Cornell corpora, both of which encompass domains distinct from the evaluation objects.
5.5 RQ 3: The Impact of Corpus Domain
It can be observed from Table and 2 that using DailyDialog and PersonaChat as training datasets for the dialogue act binary model yields stronger correlations compared to the Reddit Corpus and Comell Movie-Dialogs Corpus. The dialogue data in DailyDialog and PersonaChat is more closely aligned with chitchat, in contrast the other two datasets. This indicates that the domain of the training data has an impact on the evaluation. We can assess specific dialogues more effectively by employing corresponding corpus that belong to the same domain. For instance, if we intend to evaluate the quality of a healthcare chatbot, we can utilize a healthcare dialogue corpus the training data.
5.6 RQ 4: Case Study
In Table 5, we present snippet of a conversation log selected from the publicly available human- chatbot dialogue logs from (Adiwardana et al.. 2020). Due to the length of the original conver- sation, we have extracted a portion for illustration purposes. For each first response from the chatbot in a turn, we provide both manual ratings and DAA automatic ratings to assess its appropriateness. To facilitate comparison, all ratings are scaled within the range of 0 to 1. Overall, there is close cor- respondence between the manual ratings and the DAA scores. In the second human turn, the human poses a -us-to-Question, and the bot responds ap- propriately, resulting in high ratings from both the human annotator and DAA. However, in the third
Related Work
Early dialogue evaluation methods employed sta- tistical metries such as BLEU, METEOR, and ROUGE (Papineni et al., 2002; Banerjee and Lavie, 2005; Lin, 2004). These metrics measure word overlap between system- and ref- erence responses. Embedding- BERTScore (Zhang et al., 2020a) utilize contextual embeddings to compute the similarity between sys- tem responses and references. ADEM (Lowe et al., 2017) collects numan-scored data and employs hi-
methods like

---

Role
Sentence
Dialogue act label
luuman
DAA
Human:
hi
conventional-opennng
Bot:
Hi there.
Conventional
0.87
0.85
Human:
do you have plan thanksgiving weekend??
Yes-No-Question
Bot:
No don't have it.
Statemenf-non-opiniee
1.00
1.00
But do have answer every question..
Statement-non-opinion
Human:
do you have suggestion for the thanksgiving holidays?
Yes-No-Question
Bot:
Where would get it?
Signal-non non-understanding
0.04
0.01
But I do have answer to every question..
Statement--urtg nun-upinion
Human:
you know thanksgiving?
Tes-No-Question
Bot:
know very little ahout it.
Statement-non-opiniee
0.96
1.00
What kind of it do you like?
Wh-Quee Thh-Quettion
Human:
you like turkey?
Yes-No-Question
Bot:
Yes can a bit dry.
Statement-non-opinie
0.96
1.00
T only really eat at Christmas.
Statement-non-oppnioon
Table 5: An example of numan-chatoen dialogue intercepted from our dataset. The appropriateness of each first utterance from the bot turn is rated by the human and our DAA method, scaled from to I for comparison.
Daily Dialog
Persona
Reddit
Cornell
Method DAA
P
P
P
91.99=3.25
90.00+5.53 5.53 89.36+5.79
00.71
86.8124.29
87.85=4.46 38.57421.71
82.80 82.80=7.52 67.63=6698
82.87+5.58
w/o dir.
82.85+2.85
82.85
84.93+6.00
82.85#1 1.43
63.3629.61
47.85 13.83
Table 6: The mean and standard deviation Pearson correlation coefficient (r) and Spearman correlation coef- ficient (p) between DAA scores and human ratings when considering or not considering the direction of context- response pairs
erarchical RNN predict scores based on dialogue context, reference, and system response. RUBER (Tao et al., 2018) integrates referenced and unref- erenced indicators. However, these methods rely on pre-given references or manual scoring, which is impractical in dialogue evaluation due to the di- verse range of possible responses. In contrast, DAA does not require manual scoring pre-provided reference responses for evaluating dialogue quality.
lated the dialogue history to generate negative examples. DEAM (Ghazarian et al., 2022) ana- lyzes coherent error categories in real dialogues and designs targeted automatic rules for construct- ing negative examples. FineD-Eval (Zhang et al., 2022) supplements training data with additional model scores, such question-answer relevance scoring and natural language inference contradic- tion scores, obtain positive and negative samples, Compared to these methods, DAA directly mod- els human dialogues without the need for negative sample generation.
Some researchers have proposed unsupervised metrics to reduce reliance on manual annotations, such as GRADE (Huang et al., 2020). Dynal- val (Zhang et al., 2021), and QuantiDCE (Ye et al., 2021). These methods adopt a response- differentiating paradigm, using extensive human- to-human dialogues as positive examples and ap- plying heuristic perturbations to obtain negative examples. The models are trained to distinguish between paired positive and negative examples. However, these heuristic perturbations are often simplistic, such as random utterance order shuf- fling, limiting the model's ability to learn accurate decision boundaries. On the other hand, PONE (Lan al., 2020) performs negative example sam- pling based on semantic similarity with positive examples. Park
Some methods directly obtain scores based on the token probability distribution provided by lan- guage models. HolisticEval (Pang et al., 2020) uses GPT2 (Brown et al., 2020) to calculate the conditional probability of response given the di- alogue history. Mehri and Eskenazi (2020a) uti- lize DialoGPT (Zhang et al., 2020b) to pre-given follow-up utterances and take the likelihood scores. FULL (De Bruyn et al., 2022) improves upon Mehri and Eskenazi (2020a) by validating a large number of follow-up utterances and selecting the follow-up utterance that has the highest corre- lation with human evaluation. In comparison to these methods, we do not model token-level inputs. Instead, we model dialogue act sequences.
(2021) use a masked lan- guage model to replace some words in the response, considering the dialogue history, with words unre-
In the field evaluation, there is variability

---

in the definition of appropriateness. Lowe al. (2017) focus on the overall impression of the re- sponse. Young et al. (2018) define appropriateness as whether the response is appropriate in terms of grammar, topic, and logic. Liang and Li (2021) consider appropriateness to be synonymous with
domains at a low cost, by collecting unannotated dialogues from the relevant domain and automati- cally labeling them using DA classifier.
Ethics Statement
In this paper, we propose novel automatic evalu- ation method for assessing the appropriateness of dialogues. The study utilized publicly available dialogue corpora and human-chathor dialogue logs. The data collection process strietly adhered to le- gal and ethical requirements, ensuring privacy and consent. Participants were provided with compre- hensive information about the study's purpose, data collection procedures, and potential risks or ben- efits. They had the opportunity to ask questions and voluntarily provide their consent before par- ticipating. The collected data were used solely for research purposes.
the overall quality. Mehri and Eskenazi (2020a)
emphasize semantic appropriateness. Stasaski and Hearst (2023) associate appropriateness with dia- logue act. In this work, we adopt the definition of appropriateness from Stasaski and Hearst (2023) and leverage the dialogue act to measure the appro- priateness.
7 Conclusion
In this paper, we propose an interpretable and ef- fective method to assess the appropriateness of di- alogue agent responses by modeling dialogue act transitions. To facilitate the automated evaluation of pragmatic appropriateness, we annotated man- ually scored dataset consisting of 2,716 turn-level appropriateness ratings for six different agents, with evaluations provided by seven annotators. Our extensive experiments, using eight pre-trained lan- guage models and four human conversation cor- pora, demonstrate a strong correlation between our evaluation method and human ratings. These re- sults confirm the reliability and effectiveness of DAA in assessing the appropriateness of dialogue agent responses.
A Acknowledgments
Thanks for the insightful comments from reviewers. This work is supported in part by the National Key R&D Program of China (No. 2020AAA0106600 and the National Natural Science Foundation of China (No.U21B2009)
References
Daniel Adiwardana, Minh-Thanp Luong, David R So. Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang. Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards human-like open-domain chat- bot, ArXiv preprint, abs/2001..0997
Limitations
While our method demonstrates a strong correla- tion with human evaluations, this work has certain limitations. Firstly, like other fine-grained evalua- tion metrics, our approach focuses one aspect of a conversation while neglecting other aspects. Comprehensively evaluating all facets of dialogues using single method is a challenging task. To achieve more holistic evaluation, future research can explore the integration of semantic-based met- rics with our approach. Furthermore, constrained by the existing dialogue act classification datasets, our employed DA labels may not distinguish nu- anced intentions. It is important to note that our approach is not restricted to any specific dialogue act classification dataset and can seamlessly tran- sition to finer-grained dialogue act datasets in the future. Lastly, the performance this method in- fluenced by the dialogue corpus used. Fortunately, it is possible to migrate to dialogues different
Jens Allwood, Joakim Nivre, and Elisabeth Ahlsén 1992. On Semanties and Pragmatics of Linguistic Feedback. Journal of Semantics, 91)):1226.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACI. Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor. Michigan. Association for Computational Linguis- tics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Van Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish,

---

Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are shot learners.
Tian Lan, Xian-Ling Mao, Wei Wei, Xiaoyan Gao, and Heyan Huang. 2020. Pone: A novel automatic eval- uation metric for open-domain generative dialogue systems. ArXiv, abs/2004.02399.
Cristian Danescu- Maresscu-Nicuuesuu-MMe -izzll and Lillian Lee. 2011. Chameleons in imagined conversations: A new ap- proach to understanding coordination of linguistic style in dialogs. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguis- tics, pages 76-87, Portland, Oregon, USA. Associa- tion for Computational Linguistics.
Sungjin Lee, Hannes Schulz, Adam Atkinson, Jian- feng Gao. Kaheer Suleman, Layla El Asri, Mahmoud Adada, Minlie Huang. Shikhar Sharma, Wendy Tay, and Xiujun Li. 2019. Multi-domain cosp-comnl dialog challenge. In Dialog System Technology Chal- lenges 8.
Maxime De Bruyn, Ehsan Lotfi, Jeska Buhmann, and Walter Daclemans. 2022. Open-domain dialog evalu- ation using follow-ups likelihood. In Proceedings of the 29th International Conference on Computational Linguistics, pages 496-504, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 787 1871-7 Online, Association for Computa- tional Linguistics.
pre-training
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yanran Li. Hui Su. Xiaoyu Shen, Wenjie Li. Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
( (Volume I: Long Papers).
Anita Fetzer. 2004. Recontextualizing Context: Gram- maticality Meets Appropriateness. John Benjamins Publishing Company, Amsterdam//hilladepi
Hongru Liang and Huaqing Li. 2021. Towards standard criteria for human evaluation of chatbots: A survey. ArXiv preprint.
Sarik Ghazarian, Nuan Wen, Aram Galstyan, and Nanyun Peng. 2022. DEAM: Dialogue coherence evaluation using AMR-based semantic manipula- tions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers). pages 771-785, Dublin, Ire- land. Association for Computational Linguistics.
Chin-Yer Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Chia- Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Nose- worthy. Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods Natu- ral Language Processing, pages 2122-2132, Austin, Texas, Association for Computational Linguistics.
Peter Grundy. 2008 Doing Pragmatics. Routledge, New York.
Lishan Huang. Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020. GRADE: Automatic graph- enhanced coherence metric for evaluating open- domain dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9230-9240 Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Shaojie Jiang, Svitlana Vakulenko, and Maarten de Ri- jke. 2023. Weakly supervised turn-level engaging- ness evaluator for dialogues. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval, pages 258-268 ACM.
Ryan Lowe, Michael Noseworthy, Julian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceed- ings of the 55th Annual Meeting of the Association for Computationaal Linguistics ( (Volume I: Long Papers), pages 11 1116-1 1116-122,, Vancouver, Canada. Association for Computational Linguistics.
Zhihua Jiang, Guanghui Ye, Dongning Rao, Di Wang. and Xin Miao. 2022. Im^2: An interpretable and multi-category integrated metric framework for au- tomatic dialogue evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language P Processing, pages 11091-11103. Associa- tion for Computational Linguistics.
Shikib Mehri and Maxine Eskenazi. 2020a. Unsuper- vised evaluation of interactive dialog with DialoGPT.

---

In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225-235, 1st virtual meeting. Association for Computational Linguistics.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Humun Language Technologies, Volume 1 (Long and Short Papers), pages 1702-1723, Minneapolis, Minnesota. Association for Computa- tional Linguisties.
Shikib Mehri and Maxine Eskenazi. 2020b. USR: An unsupervised and reference free evaluation metrie for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681-707, Online. Association for Computational Linguistics.
Katherine Stasaski and Marti A. Hearst. 2023, Pragmat- ically appropriate diversity for dialogue evaluation. ArXiv preprint, abs/2304.02812.
Mohsen Mesgar, Sebastian Blicker, and Iryna Gurevych. 2020. Dialogue coherence assessment without ex- plicit dialogue act labels. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1439-1450, Online. Association for Computational Linguistics.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Eliza- beth Shriberg, Rehecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for au- tomatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339-377
Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yix- ian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of ppen-domain dialogue gener- ation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3619-3624 Online. Association for Computational Linguisties.
Yu Sun, Shuohuan Wang. Yu-Kun Li, Shikun Feng. Hao Tian, Hua Wu, and Haifeng Wang. 2020. ERNIE 2.0: A continual pre-training framework for language un- derstanding. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelli- gence Conference, IAAI 2020, The Tenth AAAI Sym- posium on Educcttinal Advances in Artificial Intel- ligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8968-888888888 088888975.484 AAAI Press.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. RUBER: unsupervised method for auto- matic evaluation of open-domale dialog systems. In Proceedings of the Thirty-Secone AAAI Conference Artificial Intelligence, (AAAI-18), the 30th inno- vative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 pages 722-729. AAAI Press.
ChaeHun Park, Eugene Jang. Wonsuk Yang, and Jong Park. 2021. Generating negative samples by manip- ulating golden responses for unsupervised learning of a response evaluation model. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 1525-1532 1534, Online. Association for Computational Linguistics.
Zheng Ye, Liucun Lu, Lishan Huang, Liang Lin, and Xiaodan Liang. 2021. Towards quantifiable dialogue coherence evaluation. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 2718-2729, Online. Association for Computational Linguistics.
Liliang Ren, Mankeerat Sidhu, Qi Zeng, Revanth Gangi Reddy, Heng Ji, and ChengXiang Zhai. 2023. C-PMI: Conditional pointwise mutual information for tum-level dialogue evaluation. In Proceedings of the Third DialDoc Workshop on Document- grounded Dialogue and Conversational Question Answering, pages 80-85, Toronto, Canada. Association for Com- putational Linguistics.
Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018 Augment- ing end-to-end dialogue systems with commonsense knowledge. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applicutions of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4970-4977 AAAI Press.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott. Eric Michael Smith, Y-Lan Boureau, and Jason We- ston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
John R. Searle. 1970. Speech acts: An essay in the philosophy of language. Language, 46:217.
Chen Zhang. Yiming Chen, Luis Fernando D'Haro. Yan Zhang, Thomas Friedrichs, Grandee Lee, and

---

Haizhou Li. 2021. DynaEval: Unifying turn and di- alogue level evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 5676-5689, Online. Association for Computational Linguistics.
Chen Zhang. Luis Fernando D'Haro, Qiquan Zhang, Thomas Friedrichs, and Haizhou Li. 2022. FineD- eval: Fine- Fine-grained automatic dialogue-level evalu- ation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3336-3355, Abu Dhabi, United Arah Emirates. Association for Computational Linguisties.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yizhe Zhang. Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT Large- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. Association for Computational Linguistics.


Reviews:
[{"review_id": "Hgyp6eheoC", "reviewer": "Reviewer_e6re", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review for acknowledging our proposed framework.\n\n**Reasons To Reject 1:  including an analysis of DAA with and without directive (dir) contexts**\n\nThank you for your suggestion, and we apologize for any confusion our abbreviation caused. \"dir\" indeed stands for \"direction\" rather than \"directive\".\nHuman-machine dialogues involve two distinct turn transition directions: from bot to human and from human to bot. As elucidated in Section 2.3, for the purpose of specifically evaluating bot responses, we have chosen not to consider human responses to chatbots, but rather only focus on chatbot responses to humans (thus considering the direction of responses). We utilize the last sentence of the human turn as context and consider the immediately succeeding sentence of the bot turn as the response to that context.\nThe results in Table 5 underscore the advantages of this approach, particularly in cross-domain settings (such as the Reddit and Cornell datasets), where our method experiences minimal performance loss. We have duly revised the relevant content in the paper (Section 5.6) to reflect these circumstances.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, the paper is very clear and well descriptive. Following reasons are listed below:\n\n\n 1. All experiments underwent rigorous statistical analysis, leading to a comprehensive comparison of eight distinct models across four human-human datasets.\n\n 2. The depiction of the DAA workflow is notably lucid and detailed, aiding in its understanding.\n\n 3. The results and discussions encompass a diverse range of dialogue evaluation domains.\n\n 4. A meticulously scored dataset, comprising 2716 turn-level appropriateness ratings, was created for six agents. This dataset involved evaluations from seven annotators.\n\n 5. The study encompassed extensive experimentation involving eight pre-trained language models and four human conversation datasets. Remarkably, a strong correlation emerged between the proposed evaluation approach and human ratings.\n\n 6. The obtained results affirm the reliability and effectiveness of DAA in evaluating the appropriateness of dialogue agent responses.\n\n7. The comprehensive evaluation of all dialogue facets using a single method is acknowledged as a complex and challenging endeavor.\n \n 8. Ethical statements are addressed properly.", "reasons_to_reject": "\nThe paper incorporates all essential experiments, yet there would be added value in including an analysis of DAA with and without directive (dir) contexts. This comparison could offer valuable insights into the effectiveness of DAA in differentiating dialogue nuances based on the presence or absence of directive elements.                ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The author proposed an unbiased and reliable evaluation metric grounded in Dialogue Acts. Additionally, a dedicated test dataset was developed to assess the practical appropriateness of dialogues. Through experimentation on four distinct datasets, a strong correlation between the proposed metric and human evaluations was identified, underscoring the accuracy of DAA in proficiently measuring dialogue appropriateness. Furthermore, the study included a sensitivity test and comprehensive analysis of interpretability, as well as directive context-response pairs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SYYn48hB6B", "reviewer": "Reviewer_sYa4", "summary": "", "questions": "A. Pragmatic appropriateness (as defined by simply a sequence of DAs i.e. illocution force) can not stand on its own. Simply replying to a question with an answer does not make for an appropriate dialogue if the answer is semantically irrelevant.\n\nB. One would expect that the DA schema would play an important role here. E.g., one would expect a [location] in response to a [where] question and a [time] in response to [when] question. But if DA schema bundles them together as [wh-question] then it will miss out on these distinctions.\n\nC. Can you also report DAA correlations for turn-level and dialogue-level?\n\nD. The DA classifier is trained in a context free way. But there have been many studies (e.g., Stolcke et al, 2000) that showed DA labeling is improved with knowledge of context and previous DAs. Why is context not taken into account in these studies?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review for acknowledging our proposed framework.\n\n**Reasons To Reject 1:  Evaluation is lacking**\n\nThank you for your suggestions. In the supplementary materials provided below, we have included a comparison with popular reference-free methods such as FED [4] and the recent C-PMI method [6]. Since these methods do not evaluate the aspect of pragmatic appropriateness, we conducted a comparison using their semantic appropriateness dimension. Notably, our proposed DAA approach achieved the highest correlation with human ratings across turn-level, dialogue-level, and system-level evaluations.\n\n| Level|Metric| Pearson| Spearman|\n| :--- |:---| -----| -----|\n| Turn   | FED   | 14.06 | 12.99 |\n| Turn   | C-PMI | 4.51  | 4.72  |\n| Turn   | DAA   | 21.67 | 21.19 |\n| Dialog | FED   | 16.50 | 11.81 |\n| Dialog | C-PMI | 2.83  | 2.61  |\n| Dialog | DAA   | 36.16 | 36.88 |\n| System | FED   | 55.29 | 42.86 |\n| System | C-PMI | 13.21 | 8.57  |\n| System | DAA   | 89.05 | 87.38 |\n\n**Reasons To Reject 2: Authors only report system wide correlations between the 6 systems, but not dialogue level or turn-level correlations**\n\nThank you for your suggestion. In the supplementary material provided for the previous question, we have reported the correlation between the DAA metric and human evaluations at both turn-level and dialogue-level.\nReasons To Reject 3-1: Many simplifying assumptions are used to define the notion of pragmatic appropriateness\n\nIn our research, we draw inspiration from the work of Stasaski and Hearst [5], who posit that the speech acts inherent in a conversation play a pivotal role in defining suitable responses. Similarly, in our study, we embrace this very assumption and utilize speech acts as a means to quantify appropriateness. \n\n**Reasons To Reject 3-2: pragmatic appropriateness cannot stand without semantic appropriateness**\n\nIn many cases, language users employ figurative language such as metaphors, idioms, and sarcasm to convey meanings that go beyond the literal sense of words. While these expressions might not be semantically appropriate when taken literally, they can be pragmatically appropriate when the context is understood. For example, saying \"It's raining cats and dogs\" doesn't make semantic sense, but in the context of expressing heavy rain, it is pragmatically appropriate.\nWhile semantic appropriateness lays the foundation for understanding words and phrases, pragmatic appropriateness goes beyond that by considering context, shared knowledge, and speaker intention\n\n**Reasons To Reject 3-3: Authors completely ignore that important aspect(semantic appropriateness) of evaluating dialogue**\n\nAs stated in the introduction section, existing methods evaluate specific aspects of dialogue quality, such as coherence, fluency, and engagement [1,2,3]. Our focus is specifically on evaluating pragmatic appropriateness.\n\n**Reasons To Reject 4: I cannot see this metric being adapted by the community without combining it with some form of semantic appropriateness**\n\nSame as the response to Question A.\n\n**Q.A: Pragmatic appropriateness can not stand on its own**\n\nSimilar to other fine-grained evaluation metrics, pragmatic appropriateness, as a specific aspect of dialogue quality, indeed cannot independently capture the overall quality of a dialogue.\nAs evidenced in FED (Table 5 from FED [4]), the overall quality of a dialogue is determined by a synthesis of multiple evaluative aspects. Even the most crucial aspect doesn't carry more than 20% importance.\n\n**Q.B: If DA schema bundles them together as [wh-question] then it will miss out on these distinctions**\n\nIn our study, we employed DA labels from the well-regarded SWBD-DAMSL dataset, which encompasses a comprehensive array of 43 distinct speech act labels. As far as our knowledge extends, this dataset represents the finest granularity of annotated speech acts presently available.\n\nIt's important to note that our method is not confined to any specific speech act classification dataset. In fact, it can seamlessly transition to higher-quality speech act datasets in the future.\n\n**Q.C: Can you also report DAA correlations for turn-level and dialogue-level?**\n\nAs provided in the response to rejection reason 1.\n\n**Q.D: The DA classifier is trained in a context free way. Why is context not taken into account in these studies?**\n\nA context-aware dialogue act classifier does indeed enhance the accuracy of DA classification. This enhancement, to a certain extent, emanates from the classifier's utilization of dialogue act transition features in DA training data.\nOur approach centers around the comparison of the similarity of these transition features across diverse corpora (i.e., human-human dialogues and human-machine dialogues) to formulate scoring criteria.\nWe deliberately restrict the DA classifier from utilizing contextual information to prevent biases that might arise from the classifier learning and incorporating prior transition feature patterns from DA dataset.\nWe apologize for any confusion caused and wish to highlight that we have addressed this aspect in Section 2.1 of the paper.\n\n**References**\n\n[1] Mohsen Mesgar, Sebastian Bücker, and Iryna Gurevych.2020. Dialogue oherence assessment without explicit dialogue act labels. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1439–1450, Online. Association for Computational Linguistics.\n\n[2] Shikib Mehri and Maxine Eskenazi. 2020b. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online. Association for Computational Linguistics.\n\n[3] Shaojie Jiang, Svitlana Vakulenko, and Maarten de Rijke. 2023. Weakly supervised turn-level engagingness evaluator for dialogues. In proceedings of the 2023 Conference on Human Information Interaction and Retrieval, pages 258–268. ACM.\n\n[4] Shikib Mehri and Maxine Eskenazi. 2020a. Unsupervised evaluation of interactive dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225–235, 1st virtual meeting. Association for Computational Linguistics.\n\n[5] Katherine Stasaski and Marti A. Hearst. 2023. Pragmatically appropriate diversity for dialogue evaluation. ArXiv preprint, abs/2304.02812\n\n[6] Liliang Ren, Mankeerat Sidhu, Qi Zeng, Revanth Gangi Reddy, Heng Ji, and ChengXiang Zhai. 2023. C-PMI: Conditional pointwise mutual information for turn-level dialogue evaluation. In Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pages 80–85, Toronto, Canada. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A dataset of appropriateness ratings (multiply annotated) for 2716 turn-level transitions from 6 different dialogue systems.", "reasons_to_reject": "* Evaluation is lacking. Authors list editing dialogue appropriateness metrics in Related work section, but do not empirically compare their proposed approach with any of them.\n* Authors only report system wide correlations between the 6 systems, but not dialogue level or turn-level correlations.\n* Many simplifying assumptions are used to define the notion of pragmatic appropriateness. But pragmatic appropriateness cannot stand without semantic appropriateness. Authors completely ignore that important aspect of evaluating dialogue. \n* I cannot see this metric being adapted by the community without combining it with some form of semantic appropriateness. Such a combination is neither proposed nor evaluated here.\n", "questions_to_authors": "A. Pragmatic appropriateness (as defined by simply a sequence of DAs i.e. illocution force) can not stand on its own. Simply replying to a question with an answer does not make for an appropriate dialogue if the answer is semantically irrelevant.\n\nB. One would expect that the DA schema would play an important role here. E.g., one would expect a [location] in response to a [where] question and a [time] in response to [when] question. But if DA schema bundles them together as [wh-question] then it will miss out on these distinctions.\n\nC. Can you also report DAA correlations for turn-level and dialogue-level?\n\nD. The DA classifier is trained in a context free way. But there have been many studies (e.g., Stolcke et al, 2000) that showed DA labeling is improved with knowledge of context and previous DAs. Why is context not taken into account in these studies?\n", "paper_topic_and_main_contributions": "Authors propose an automatic evaluation for dialogue appropriateness. Specifically they focus on the notion of pragmatic appropriateness which they argue is realized through dialogue acts. Authors posit that human-human dialogues are appropriate and hence the dialogue act patterns exhibited in those are an indicator of appropriateness.\n\nDialogue act (DA) schema used in this work is that from the Switchboard corpus which was annotated for DAs. Authors have trained a DA classifier based on this DA-annotated Switchboard data and use that DA classifier to learn DA transition patterns observed in various human-human corpora. The notion of transition pattern is further simplified to probability of response DA given previous utterance’s DA.\n\nThe proposed automatic metric, DAA, measures the degree to which a human-bot conversation agrees with the DA bigram model estimated from human-human corpus.\n\nAuthors have collected a set of human judgements from 7 raters for appropriateness for 2716 turn-level transitions from 300 dialogues from 5 different chatbots and human baselines. As evaluation of the proposed metric authors report correlation analysis at chatbot level.\n\nThe main contribution of this work are:\n* Proposed automatic evaluation metric for dialogue appropriateness based on estimating dialogue act transition bigram model.\n* A dataset of manually rated appropriateness by 7 raters for 300 dialogues from 6 different chat models (5 bots + 1 human).\n", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uPuluHntDP", "reviewer": "Reviewer_LjnF", "summary": "", "questions": "1.  Could you explain more about ignoring the direction of context-response?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review for acknowledging our proposed framework.\n\n**Reasons To Reject 1: the proposed method will be strongly impacted by the training domain**\n\n\nOur method encompasses the Dialogue Act (DA) classification phase and the DA sequence modeling phase.\n\nIn Section 5.3, we illustrate that the accuracy of the classifier has a relatively minor effect on the method's relevance. This is due to the fact that classification errors are uniformly applied to both the modeling corpus and the dialogues under evaluation. It's noteworthy that we leverage the transitions of DAs rather than specific DA labels.\n\nFurthermore, the modeling of DA sequences in human-human dialogues involves utilizing unlabeled dialogues specific to the corresponding domain. These dialogues are automatically labeled using the DA classifier, thereby eliminating the need for additional manual annotations.\n\nAs a result, it is feasible and cost-effective to adapt our method to different dialogue domains, requiring only minimal domain-specific unlabeled data.\n\n\n\n**Reasons To Reject 2-1:  it is unknown how the single 5-point Likert scale is annotated.**\n\nFollowing Mehri and Eskenazi (2020), we employed a 5-point Likert scale for the manual quality assessment of appropriateness. In this process, we presented annotators with both human utterances and bot responses at the turn level. Their task was to evaluate the appropriateness of the given bot response. They assigned an integer score ranging from 0 to 4, where 4 represents \"very appropriate\" and 0 signifies \"very inappropriate.\" \n\n\n\n**Reasons To Reject 2-2:  This single subjective human evaluation score is not very convincing for dialogue evaluation.**\n\nTo mitigate the inherent subjectivity inherent in individual human scores, we adopted an averaging approach. Specifically, we calculated the mean score assigned by annotators to the same response. This strategy served to temper the subjectivity in individual scores.\n\n\nFurthermore, following the methodology of Mehri and Eskenazi (2020) and as outlined in section 3.2 of our paper, we conducted computations of Pearson and Spearman correlations. These correlations, measuring the consistency between annotators' mean scores and the scores assigned by other annotators, yielded values of 0.9619 and 0.9265 respectively. These robust correlations attest to the high quality of human annotations.\n\n\n\n**Reasons To Reject 2-3:  Considering a new dimension of human evaluation, without any calibration, I cannot imagine the proposed method will still be highly correlated with human annotation**\n\nWe are pleased to confirm that the proposed method has demonstrated a high correlation with human annotation, as you rightly pointed out. This result underscores the effectiveness of our approach in capturing the intended dimension of evaluation. While your initial concern regarding introducing a new dimension without calibration was valid, our extensive experiments have been successful in establishing a strong alignment between our method and human judgments. This high correlation serves as a testament to the robustness and validity of our methodology.\n\n\n\n**Q.1: Could you explain more about ignoring the direction of context-response?**\n\nHuman-machine dialogues involve two distinct turn transition directions: from bot to human and from human to bot. As elucidated in Section 2.3, for the purpose of specifically evaluating bot responses, we have chosen not to consider human responses to chatbots, but rather only focus on chatbot responses to humans (thus considering the direction of responses). We utilize the last sentence of the human turn as context and consider the immediately succeeding sentence of the bot turn as the response to that context.\n\nThe results in Table 5 underscore the advantages of this approach, particularly in cross-domain settings (such as the Reddit and Cornell datasets), where our method experiences minimal performance loss. We apologize for any confusion that may have arisen and wish to inform you that we have made relevant amendments to the paper (Section 5.6).\n\n\n\n**References**\n\n[1] Shikib Mehri and Maxine Eskenazi. 2020a. Unsupervised evaluation of interactive dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225–235, 1st virtual meeting. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed evaluation method considers the simple perplexity-like metric for dialogue act sequence as the perplexity for a sentence. Experiments are very extensive.", "reasons_to_reject": "1. Similar to perplexity, the proposed method will be strongly impacted by the training domain on both the dialogue act classifier and the sequential pattern of the dialogue act. Hence, in real use cases, domain adaptation for dialogue act classifier and a very amount of in-domain human data are needed for the evaluation.\n2. it is unknown how the single 5-point Likert scale is annotated. This single subjective human evaluation score is not very convincing for dialogue evaluation. Also, considering a new dimension of human evaluation, without any calibration, I cannot imagine the proposed method will still be highly correlated with human annotation. Hence, this method could only be used as a preliminary evaluation of perplexity.  ", "questions_to_authors": "1.  Could you explain more about ignoring the direction of context-response?\n", "paper_topic_and_main_contributions": "This paper proposed a new metric to evaluate dialogue appropriateness via calculating the geometric mean of transition probabilities of dialog act context-response pair. The study is conducted on 2716 turn-level appropriateness ratings across 6 different agents with 7 annotators. Extensive experimental results show that the proposed DAA metric is strongly correlated with human ratings.  ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CmlmwwWvwr", "reviewer": "Reviewer_gNvL", "summary": "", "questions": "No questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review for acknowledging our proposed framework.\n\n**For Minor spelling errors:**\n\nThanks for your comments. We have proofread and revised the manuscript\nthoroughly according to your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths:\n• Human subjective evaluation of dialogue appropriateness is quantified by using an approach, that provides a stable evaluation metric - Dialogue Act Appropriateness.\n • To evaluate the effectiveness of the proposed method, human evaluations on 6 different dialogue agents are conducted, and a test dataset is built for evaluating dialogue pragmatic appropriateness.\n• Experiments using 8 pre-trained language models and 4 human conversation corpora show a strong correlation between the proposed metric and human judgments.\n", "reasons_to_reject": "The paper can be accepted.", "questions_to_authors": "No questions.", "paper_topic_and_main_contributions": "The paper concentrates on an important aspect of dialogue systems - appropriateness. A novel method is introduced to evaluate dialogue agent responses by modeling dialogue act transitions - Dialogue Act Appropriateness.  The method is validated, the experimental results demonstrate a strong correlation with manual evaluation. \nThe study utilizes publicly available dialogue corpora and human-chatbot dialogue logs.\n5 tables and 2 figures illustrate the discussion.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Nothing is missing.", "Typos_Grammar_Style_and_Presentation_Improvements": "Style is fine. \nMinor spelling errors, e.g.: work. our (line 581)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a metric for evaluating dialogue appropriateness, but reviewers have expressed several concerns. First, there is a notable lack of empirical comparison with existing dialogue appropriateness metrics, despite listing them in the related work section. The reported correlations between six systems are system-wide, with no exploration of dialogue or turn-level correlations. Additionally, the metric focuses on pragmatic appropriateness while ignoring semantic appropriateness.
Decision: Accept-Findings