Don’t Forget Your ABC’s: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems
Year: 2023

Abstract:
Despite tremendous advancements in dialogue systems, stable evaluation still requires human judgments producing notoriously high-variance metrics due to their inherent subjectivity. Moreover, methods and labels in dialogue evaluation are not fully standardized, especially for opendomain chats, with a lack of work to compare and assess the validity of those approaches. The use of inconsistent evaluation can misinform the performance of a dialogue system, which becomes a major hurdle to enhance it. Thus, a dimensional evaluation of chat-oriented opendomain dialogue systems that reliably measures several aspects of dialogue capabilities is desired. This paper presents a novel human evaluation method to estimate the rates of many dialogue system behaviors. Our method is used to evaluate four state-of-the-art open-domain dialogue systems and compared with existing approaches. The analysis demonstrates that our behavior method is more suitable than alternative Likert-style or comparative approaches for dimensional evaluation of these systems.

heading: 1 Introduction
text: Recent work in human-computer chat has made remarkable progress. Multi-turn open-domain (MTOD) models are capable of holding engaging conversations with humans (Roller et al., 2021; Adiwardana et al., 2020). However, there remain a number of challenges facing MTOD chatbots such as hallucinations (Shuster et al., 2021), commonsense violations (Zhou et al., 2021), and consistency issues (Nie et al., 2021). A significant obstacle for research that addresses these challenges is the difficulty in formulating an appropriate evaluation methodology due to the inherent subjectivity in determining chat quality (van Miltenburg et al., 2021). Since existing automatic evaluation metrics have been shown to be biased measures of chat quality (Liu et al., 2016; Sai et al., 2019; Deriu *Contributed equally to this work as first authors.
et al., 2022), evaluation using human judgments is standard, although the type of human judgments varies widely across works (Finch and Choi, 2020).
Overall, there are few works comparing and assessing the validity of various human evaluation methods. The result of this gap in the literature is that the relative sensitivity, interpretability, and importance of the metrics used to evaluate chat models are not well understood. A dimensional approach for evaluating chat models that measures different aspects of chat quality would surely aid progress (van Miltenburg et al., 2021). However, to our knowledge, no work has investigated the coverage of a comprehensive set of evaluation metrics. Consequently, existing chat model evaluation results provide an incomplete picture of the strengths and weaknesses of MTOD chatbots. This paper addresses these limitations of previous work through the following three contributions:
1. A novel, dimensional human evaluation method that measures the rate of chatbot behaviors impacting chat quality (Section 5).
2. A detailed validation of human evaluation methods, including likert scales and pairwise comparisons (Section 7).
3. A comprehensive evaluation of four MTOD chatbots using validated metrics (Section 8).
By presenting a detailed picture of MTOD chatbot performance and standard methods to evaluate them, we aid future work’s efforts to further understand and improve human-computer interaction. Our evaluation platform, analyses, and data are available at https://github.com/emorynlp/ ChatEvaluationPlatform.

heading: 2 Chatbots
text: To evaluate the strengths and weaknesses of MTOD models, we select the chatbots for our study using a
15044
two-stage process: (1) a literature review to identify chatbot candidates, and (2) a pilot evaluation to select the final set of bots for our full study.
Literature Review To promote diversity among the selected chatbots, we focus our review on four popular themes of the human-computer chat: (1) Knowledge-grounded chat, (2) Empathetic chat, (3) Self-consistent chat, and (4) General open-domain chat with large pre-training resources like Reddit. Candidate chatbots are selected from each theme using the following criteria:
1. The bot must demonstrate state-of-the-art performance in a task related to the theme.1
2. The implementation must be provided.2
3. The response latency of the bot must be <10 seconds using modern GPU hardware.
This review yields the 6 chatbot candidates in Table 1: Blender-Decode (Nie et al., 2021), Blender2 (Weston and Shuster, 2021), BART-FiD-RAG (Shuster et al., 2021), Emora (Finch et al., 2020), DukeNet (Meng et al., 2020), and CEM (Sabour et al., 2022). Appendix A presents details of our literature review and selection process.
Chatbot Selection A pilot evaluation using the 6 chatbot candidates is conducted in order to verify the multi-turn dialogue capability of the chatbot candidates. Appendix B provides details on the implementations of each chatbot candidate. 10 students majoring in Computer Science or Linguistics are invited to interact with randomly assigned chatbots in 3-5 text-based conversations,3 each of 1Note that selection occurred in October 2021. 2We accepted either a trained English model or codebase with a fully-specified procedure to replicate the model. 3We use the web interface provided by ParlAI (Miller et al., 2017) hosted on our local webserver.
which consisted of 30 turns.4 At the end of each conversation, students are asked to rate the quality from 1 (least) to 5 (most). Based on the pilot results (Table 1), DukeNet and CEM are excluded from our full study because they are unable to hold satisfying multi-turn conversations, despite their reasonable single-turn response generation capabilities. Appendix C shows example dialogues from these systems.

heading: 3 Conversation Collection
text: The conversation dataset used for the full study is collected using human interactors in a text-based conversation setting. 46 undergraduates are recruited as interactors. Each interactor is compensated with a $5 gift card for every 6 conversations, and allowed to complete up to 18 conversations. Conversations are collected remotely using ParlAI’s interactive web interface, and links to the web interface are sent to each interactor with instructions to be completed within 2 weeks.
For each link, the interactor completes two conversations with a random pair of chatbots, for a minimum of 30 turns per conversation. We impose a similar open-ended, topic-free chatting environment to Adiwardana et al. (2020). Interactors are asked to rate 8 dimensions (Table 3) of each conversation after its completion on a 1-5 Likert scale, and to select the higher-quality conversation along the same 8 dimensions after each conversation-pair (ties allowed). Our final conversation dataset includes 400 human-bot dialogues (100 dialogues per chatbot), averaging 30.3 turns per dialogue (11.3 tokens per user turn).

heading: 4 Evaluation Methods
text: For a comprehensive evaluation of MOTD chatbots, a robust dimensional evaluation of their chat capabilities is crucial (van Miltenburg et al., 2021). To have confidence that any evaluation metric yields useful information, its interpretability and sensitivity require validation. In addition, it is important to verify that each evaluation metric provides distinct information relative to the others.
Several previous works propose sets of evaluation metrics that could be used for a dimensional evaluation but with insufficient analyses to validate them. Finch and Choi (2020) present an exhaustive set of metrics based on a literature survey of human
4A “turn” is defined as ONE message from a single interactor.
evaluation methods, but do not quantitatively validate its interpretability, sensitivity, or per-metric distinctness. Mehri and Eskenazi (2020a) present a set of Likert metrics and analyze their relationship to overall dialogue quality, but do not validate the sensitivity or distinctness of the individual metrics. Mehri and Eskenazi (2020b) present 5 Likert metrics and evaluate their coverage with respect to explaining single response quality, but do not validate their sensitivity or distinctness.
Similarly, some works look to identify common chatbot errors. Sanguinetti et al. (2020) and Higashinaka et al. (2021) present error taxonomies empirically grounded by error analyses, but do not present distinctness or sensitivity results for their error categories. See and Manning (2021) identify errors for one dialogue model and analyze the impact of each error on overall quality but do not attempt to verify the generalizability of their results.
Furthermore, various works propose novel evaluation methods with varying degrees of validation of the reliability and effectiveness of such methods. Deriu et al. (2020) present Spot the Bot, a pairwise evaluation approach that uses survival analysis to rank bots based on self-chats, but do not directly compare to alternative methodologies other than for cost. Sedoc and Ungar (2020) apply Item-Response Theory (IRT) (Lord and Novick, 2008) to pairwise comparison dialogue evaluation, by using a latent variable Bayesian model to estimate both the ability of the evaluated systems and the informativeness of inputs in the static evaluation set. Their analysis of the utility of IRT for dialogue evaluation does not include comparisons to existing approaches or a dimensional focus since they exclusively consider overall response quality. Ji et al. (2022) propose a continuous-scale method for evaluating multi-turn dialogue systems with quality control measures for mitigating artifacts from human annotators. They validate their proposed method on various dialogue dimensions using replication studies, a sensitivity analysis, and a correlation analysis between dimensions, although they explicitly acknowledge that their set of dimensions is not intended to be comprehensive. Phy et al. (2020) assert 3 dimensions (understandability, sensibleness, and likability) are sufficient for capturing the quality of a dialogue and validate their claims using agreement, correlation analysis, and distinctness analysis on human annotations of their dimensions, although they are not applied to multi-turn dialogues.
Two studies, Li et al. (2019a) and Smith et al. (2022), compare pairwise comparison and Likert evaluation methods via a sensitivity analysis. However, neither of them target a high-coverage set of dimensional metrics, as their studies were limited to 4 and 3 metrics respectively. Lee et al. (2020) also investigates pairwise evaluation using the ChatEval platform. However, this is not a multi-turn evaluation setup and it does not target a dimensional analysis since the comparisons are based exclusively on the overall quality of the responses.
Overall, the relative validity of human evaluation metrics requires further investigation before a comprehensive and reliable dimensional evaluation of human-computer chat is achieved. Table 2 summarizes the goals and contributions of the previous evaluation works. Our study addresses all existing gaps by conducting a detailed validation study of 4 different human evaluation methods and a wide range of fine-grained metrics.

heading: 4.1 Selected Methods
text: Four human evaluation methods are chosen for our study. Since MTOD chat model evaluation is our goal, any domain- or approach-specific methods or single-response evaluation methods providing
chatbots with a specific context are excluded.5 We also focus on external human evaluation methods, where human evaluators judge conversations they do not participate in. Three of the selected methods represent popular approaches: Dialogue Likert, Turn Likert, and Comparative. The fourth method, ABC-Eval, is our novel evaluation approach.
Dialogue Likert Annotators provide dialoguelevel ratings from 1 (least) to 5 (most) for the 8 labels shown in Table 3. We use the dimension set proposed in Finch and Choi (2020) which results from a detailed survey of characteristics used in chat evaluation and has better coverage than alternatives like the set used in ACUTE-Eval (Li et al., 2019a). Bot-level metrics are calculated as the mean rating across all bot dialogues.
Turn Likert Annotators provide turn-level ratings on the same scale and labels as those used for Dialogue Likert. The dialogue-level metric is measured as the mean rating of a single dialogue’s turns. The bot-level metric is calculated as the mean rating of all turns in all bot dialogues.
Comparative Annotators select the dialogue in which chatbot responses better fit a label definition from a side-by-side pair of dialogues, also using the labels in Table 3. A “neither” option is allowed, only for cases where the evaluator cannot distinguish which dialogue was a better fit. Bot-level metrics are calculated as bot pair win/tie/loss proportions between pairing of their dialogues.
Behavior Classification: ABC-Eval Annotators provide binary labels on the turn-level indicating 5We do not include a turn-level comparative evaluation because controlled comparisons require comparing turns with identical historical contexts which is not viable for real human-bot dialogues like those used in this work.
the presence or absence of a particular chat characteristic. The included chat characteristics are defined in Table 4. Dialogue-level metrics are calculated as the proportion of turns that display the characteristic of the dialogue. Bot-level metrics are calculated as the proportions of turns that display the characteristic over all bot dialogues. ABC-Eval is described in detail next in Section 5.

heading: 5 ABC-Eval Design
text: We hypothesize that binary turn-level behavior labels provide more reliable and informative metrics for quantifying fine-grained aspects of chat quality than alternative approaches such as Likert or Comparative scoring. Our novel method, the Annotation of Behaviors in Chat Evaluation (ABC-Eval), is developed in three stages: (1) collecting a set of behavior label candidates, (2) developing and piloting our annotation instructions and procedure, and (3) selecting a subset of behavior labels based on the validation study results in Section 7.
Collecting Behavior Label Candidates Based on a review of recent work in chat-oriented dialogue modeling and evaluation, we identify characteristics of chatbot responses relevant to conversation quality. These characteristics include those presented as error cases, evaluation metrics, or desirable response features. We then curate binarized definitions of these characteristics to create an initial set of behavior label candidates, which are revised through an iterative piloting and development process. Due to its high coverage of error categories, Higashinaka et al. (2021) is the primary source of inspiration for many of our behavior labels. However, we improve upon their presented taxonomies by considering additional labels based on characteristics of chat presented by other work, and by further refining their error categories to improve average Inter-Annotator Agreement (Section 7.1). Table 4 presents the final set and definitions of the 16 candidate behavior labels used in our full study, along with selected works from our review that inspired their inclusion. Appendix D details in full our development process.
Annotation Procedure The ABC-Eval procedure includes 16 binary behavior labels divided between 8 independent annotation tasks (Table 4). In each task, human evaluators are provided with definitions and examples of the behavior labels associated with that task and asked to annotate every
chatbot turn in a given human-chatbot conversation with each behavior label. Evaluators complete these tasks using a custom web application based on the ParlAI evaluation interface (Appendix G).
Training and Screening To improve annotation consistency and detect poorly performing evaluators, we develop automated training sessions each annotation task inspired by van Miltenburg et al. (2021). Each session consists of 3 conversations that evaluators annotate using an identical procedure and web interface to the corresponding task. The 3 conversations used for each session are handcrafted by the authors to represent a variety of positive and negative examples of the behavior labels for the corresponding task (Appendix D). The gold annotations for each training conversation are hidden from evaluators during the annotation; however, after completing each training conversation, any disagreements between the evaluator’s annotations and gold labels are displayed along with an explanation to help the evaluator improve. We use the evaluator’s performance on the third conversation of each training session to screen evaluators, where performance is measured by the number of turns where their annotations disagree with gold labels. Evaluators are eligible to complete the work on a task if they make mistakes on fewer than 2 turns for the antisociality and uninterpretability tasks, or on fewer than 3 turns for the other 6 tasks.

heading: 6 Evaluation Study
text: Our full study consists of the collection of 40 labels per conversation. This collection was split into 18 independent evaluation tasks as follows:
• 8 ABC-Eval tasks, each composed of 1 to 4 labels as denoted by groupings in Table 4
• 1 Dialogue Likert task, composed of all 8 labels from Table 3 completed in random order
• 8 Turn Likert tasks, each composed of 1 label from Table 3
• 1 Comparative task, composed of all 8 labels from Table 3 completed in random order
The 18 evaluation tasks are posted on SurgeHQ’s annotation platform6 to be completed by dedicated remote workers (Surgers) with experience in NLP annotation. Each time an evaluator connects to one of our tasks, they are assigned a randomly selected conversation to annotate. We are allocated a group of 125 Surgers, chosen by a SurgeHQ employee based on high annotation performance on past projects. Evaluators are compensated per annotated conversation per task, at an estimated rate of $20/hr7. We allow evaluators to annotate up to 60 conversations per task. 6https://www.surgehq.ai; Appx. E details annotator selection. 7Per-task payment rates provided in Appendix F.
Our final evaluation dataset consists of 400 conversations, each with results for all 40 labels.8 Additionally, a randomly-selected subset of 100 conversations (and 50 of the conversation pairs) is evaluated a second time by a different Surger in order to measure IAA.

heading: 7 Metric Analysis

heading: 7.1 Interpretability
text: We measure the reliability of interpreting each metric’s annotation instructions by calculating IAA using our set of 100 double-annotated conversations (Figure 1). High agreement between annotators demonstrates that different people can reliably come to the same conclusions about how a metric’s definition applies to each chatbot response.
Our results suggest that the definitions of most ABC-Eval metrics can be interpreted more reliably than the definitions of most Dialogue Likert, Turn Likert, and Dialogue Comparison metrics. Likertstyle and comparison-style annotations appear to have similar interpretability, although Quac was a notable exception that produced higher agreement than Quad.

heading: 7.2 Importance
text: The importance of each metric is estimated by a predictive validity analysis that measures the extent, to which the metric can predict conversation quality (Figure 2). We use Quad and Quac from interactors that participated in the conversations (Section 3) to avoid cases where the same evaluator produced the quality label and explanatory metric. 8Only 192 of our 200 dialogue pairs were evaluated with Comparative labels due to a collection mistake 9Bias-corrected and accelerated confidence intervals with k =10,000 Monte Carlo case resamples. 10!Socb and !Intb’s confidence intervals are largely due to a low rate of positive examples (see Figure 4).
The predictive validity of each metric was measured by fitting univariate linear or logistic regression models to predict Quad or Quac, respectively. Quac was represented as a binary encoding, where 0 and 1 represent choosing the first and second conversation, respectively. We excluded any conversation pairs in which the interactor could not distinguish a difference in quality between conversations, and fitted models on the remaining set of 184 conversations. To use non-comparative predictors for predicting Quac, the difference in metric value between each pair of conversations was used.
Our results suggest that dialogue quality is substantially related to emotional understanding metrics (Emo, Empb, !Empb), relevance-related metrics (Rel, !Relb, Ignb), and consistency metrics (Con, !Selb, Redb, !Parb). Within these metric groupings, ABC-Eval metrics were overall more predictive of quality than their Likert or comparative analogs, while comparative metrics were least predictive of quality. Chatbots’ ability to express knowledge (Inf , Facb, !Facb Lifb, Prefb) was an overall poor predictor of quality; however, commonsense knowledge errors (!Comb) was one of the strongest predictors.

heading: 7.3 Sensitivity
text: We investigate the sensitivity of each metric using two analyses. First, we use the fitness of the univariate regression models described in the previous section as one source of evidence for metric sensitivity, since a metric must be sufficiently sensitive in order to distinguish conversations of low and high quality. Second, we follow Li et al. (2019a) and run hypothesis tests to count the number of statistically significant differences each metric is able to detect between the 6 pairings of our 4 chatbots (Table 5). To make results comparable, we
downsample the conversations used for hypothesis testing to 32 conversations per bot for our Dialogue Likert, Turn Likert, and ABC-Eval metrics to match the 32 conversation-pairs per bot-pair produced by our Comparative evaluation.
Our results show that the Likert evaluations were more sensitive than the Comparative evaluation for most labels. ABC-Eval metrics have a wide range of sensitivity, with some ABC-Eval metrics appearing to be more sensitive analogs of similar likert metrics. For example, the results suggest that !Selb and Redb are more sensitive than Con, that Facb and !Facb are more sensitive than Inf , and that Empb and !Empb are more sensitive than Emo. On the other hand, the likert-style Rel metric shows similar or slightly superior sensitivity compared to the analogous Ign and !Rel behavior metrics.

heading: 7.4 Coverage & Distinctness
text: We investigate the coverage and distinctness of our metrics via incremental validity analysis. For this analysis, we perform backwards stepwise regression that determines (1) the ability of an evaluation method as a whole to explain conversation quality, and (2) whether each metric contributes distinct information about quality above and beyond other metrics (Figure 3). Specifically, we fit a multivariate regression model for each of our 4 evaluation methods. These models are fit similarly to those
presented in Section 7.2, but include all non-quality metrics within an evaluation method as predictors. Then, we remove predictors from each model one at a time based on a beam search (k=100) of which removed predictor results in the smallest decrease in model fitness (adjusted R2 or adjusted pseudoR2). We perform this stepwise regression analysis twice to predict both Quad and Quac given by interactors, similar to our analysis in Section 7.2. Our results suggest that ABC-Eval has overall better coverage than other evaluation methods for explaining conversation quality. Furthermore, most ABC-Eval metrics that have a strong relationship with conversation quality appear to be appropriately distinct in the information they provide, especially !Empb, !Selb, Redb, !Relb, Empb, !Comb, and Ignb. Similar distinctness can also be seen in Turn Likert metrics, whereas dialogue-level metrics show relatively low distinctness.

heading: 7.5 Final ABC-Eval Metrics
text: Given the results of our metric analysis, we select the final set of ABC-Eval metrics bolded in Table 4. In our analyses, this final set had better interpretability (Section 7.1), a wider coverage of distinct characteristics of chat that impact quality (Section 7.2 and Section 7.4), and overall higher measurement sensitivity (Section 7.3) than alternative evaluation methods. Furthermore, the fi-
nal ABC-Eval metrics are less costly11 (a median of 15.2 min/dialogue) to collect than Turn Likert metrics (19.9 min/dialogue). Although dialoguelevel evaluations are least costly (2.8 min/dialogue for Dialogue Likert, 4.4 min/dialogue for Comparative), our results suggest that dialogue-level judgements may be ill-suited for dimensional evaluation, since the dialogue-level metrics we tested had worse coverage and distinctness (Section 7.4).

heading: 8 Chatbot Evaluation
text: To evaluate the strengths and weaknesses of our 4 selected chatbots, we present results for the 400 collected conversations across all ABC-Eval metrics (Figure 4 and Figure 5), Likert Dialogue metrics (Figure 6), Likert Turn metrics (Figure 8), and Comparative metrics (Figure 7). We focus our discussion on the final set of ABC-Eval metrics since they performed best in our metric analysis.
The results highlight the notable recent progress in human-computer chat, as the vast majority of chatbot turns are interpretable, relevant responses to the dialogue context. Less than 1% of re11See Appendix F for detailed cost results.
sponses have interpretability issues, and Blender2 and BART-FiD-RAG each achieve a relevant response rate of nearly 90%. Blender2 specifically is also able to incorporate factual knowledge into about 20% of its responses while hallucinating factual information at a remarkably low rate, less than 1%. Furthermore, the chatbots almost never produce responses with offensive language.12 The chatbots also show a high rate of emotional understanding, with 40% of their responses containing emotionally-appropriate reactions to the user.
Despite these strengths, our results also show several clear directions for improvement. Commonsense violations are present in about 15-20% of the bots’ responses. Consistency issues are prevalent across all bots: self-contradictions, partner contradictions, and redundancies appear in about 5% of the bots’ responses overall. Also, all chatbots have a substantial rate of violating natural dialogue structure: about 10% of responses are judged as ignoring the user, and depending on the chatbot,
12Note that our experiments are conducted with cooperative human interactors. Chatbots similar to those we test have been shown to reliably produce offensive language when responding to provocative inputs (Dinan et al., 2022).
around 10-20% of responses include irrelevant contributions to the dialogue. Additionally, 5-15% of the chatbots’ responses show a lack of empathy or other emotional misunderstandings. The reality of these observed rates of problematic behaviors is that, in most 30-turn conversations with these chatbots, a human interactor is likely to experience several issues that impact conversation quality.

heading: 9 Conclusion
text: As illustrated here, dialogue quality is a complex construct with many dimensions. Depending on the approach, dialogue systems can have markedly different weaknesses among these quality dimensions. Our research highlights several outstanding challenges, especially regarding the relevance, consistency, common-sensibility, and emotional understanding of chat model responses. Our analyses not only demonstrate that these four dimensions have a high impact on conversation quality, but also that current chatbots have substantial response error rates in these areas.
To efficiently address the challenges facing opendomain dialogue models, we need a reliable, dimensional evaluation method; however, our results show that popular evaluations such as dialoguelevel Likert and comparative methods may not be suitable. The presented ABC-Eval serves as a promising alternative in this direction. Although the popular dialogue-level likert evaluation method may be the most cost-effective and robust method for measuring overall dialogue quality, we recommend that researchers additionally use the final set of ABC-Eval metrics, or a subset relevant to their scientific goals, to evaluate specific strengths and weaknesses of new chat models. Overall, we hope future work can use insights from our study to make better-informed decisions about which evaluation method to use, and to tackle the challenges facing current chatbots.

heading: 10 Limitations
text: There are several characteristics of the presented analyses that limit the scope of conclusions that can be drawn. We discuss how each of these limitations affect the takeaways of our results below.
Number of Chatbots The generalizability of our metric analysis results (Section 7) is constrained by
the fact that we were only able to include conversations from 4 chatbots in our analyses. We did our best to choose chatbots representative of the field and seem to have selected a fairly diverse group of models (Section 8). However, it is possible that not all results we found in our metric analyses will generalize when evaluating other chat models. One possible example is the number of partner contradictions we observed among our 4 chatbots (Figure 4), which may be similar by coincidence. If other chatbot models indeed differ more substantially in partner contradiction rates, our sensitivity metric analysis may have underestimated the sensitivity of our partner contradiction metric (Section 7.3). In general, including a larger number of chatbots in a metric analysis will improve the chance that its results will apply to new chatbot models. Future work that performs metric analyses like those we presented, but with different chatbots than the 4 selected in this work, would aid further analysis of our results’ generalizability.
Use of Surgers as Evaluators We perform our analyses using only a single evaluator group (Surgers). This choice of evaluator group does not harm the replicability of our methods, as other researchers have access to use of SurgeHQ or similar third-party annotation companies. However, several other evaluator groups are more popularly used for chat model evaluation, such as university students and Amazon Mechanical Turkers (MTurkers). We attempted to carry out our study with three evaluator groups (see Appendix E for details), but were unable to proceed with student and MTurker evaluator groups due to time constraints. Consequently, it is unclear to what extent our metric analysis results will generalize to other choices of evaluator.
Number of Collected Conversations As with any study involving a sampling procedure, resource constraints limit the number of collected samples, which in turn limits the statistical power of the study’s analyses. Our study included 400 conversations, which provided more than adequate statistical power for most of our analyses. For example, our investigation of each metric’s predictive validity (Section 7.2) relied on a simple linear regression analyses. At a significance level of α=0.05, our 400 conversation samples would yield a statistical power of 1-β=0.80 to detect effect sizes of f2=0.142 by F-test for each metric’s regression. However, our analyses with the weakest statistical
power are our dialogue-level analyses that compare bots with only 100 samples per bot. At 100 samples per bot, and assuming a standard deviation of 1.0 Likert points,13 a two-tailed t-test of mean Dialogue Likert rating would have a statistical power of 1-β=0.80 to detect differences of an effect size of Cohen’s d=0.40. This is still a reasonable amount of statistical power, but leaves room for our study to produce inconclusive results when the true differences between chatbots are small.

heading: 11 Ethics Statement
text: The presented work aims towards improving the scientific methodology of chat model evaluation. To this end, we present a battery of analyses comparing several aspects of metric validity for four different evaluation methods (Section 7). Our results allow other researchers in the field to make betterinformed decisions regarding appropriate evaluation methodology in human-computer chat. To ensure replicability of our methods we publicly release the annotation software and chatbot implementations used to collect our conversation and evaluation data. Additionally, we provide full transparency in our analyses by releasing the code for all our presented analyses. Finally, to aid future research efforts in human-computer chat modelling and evaluation, we release an anonymized version of our conversation and evaluation data.
One ethical consideration involved in our work involved managing human workers in our data collection processes. All worker participation in our study was voluntary and involved zero subjective screening processes, with a complete description of worker tasks, workload, and timeframe provided before work was assigned. Workers could opt out of our study at any time for any reason. As compensation for work completed, we targeted a compensation rate of $10/hour for student14 and Amazon Mechanical Turk workers, and a rate of $20/hour for Surgers. We compensated on a pertask-completed basis to ensure timely completion of work, but verified that target hourly rates were reasonably approximated throughout the course of the study by measuring workers’ median task completion times (see Appendix F for details). These measures ensured that all human work in our study 13Smith et al. (2022) reports standard deviations of Likert
metrics between 0.8 and 1.3 14Students’ compensation is given as an Amazon Gift Card
for convenience; students are informed of this prior to any work being completed
was fair, transparent, and mutually-beneficial. Other ethical considerations arise in our study’s conversation collection. Unlike the collection of evaluation or annotation data, collecting interactive conversation data from human-computer interaction poses a small but meaningful risk that sensitive, damaging, or personally identifying information could get collected. We mitigated this risk in three ways. First, students were notified in multiple email communications and before each conversation that their conversations with our chatbots would be publicly released. Included in these notices was the instruction to refrain from releasing any personally identifiable or damaging information. Our instructions suggest that students fabricate personal information at any time during the conversations if it would make them feel more comfortable. Second, we hand-checked all 400 conversations to ensure the non-presence of any sensitive information. Third, we anonymize all data before public release. Our study’s collection and analysis of conversation data did not investigate interactors as human subjects, and we did not seek institutional review board approval.
Finally, there is a concern in our study about the potential of the chatbots to respond to student interactors with toxic, insensitive, or vulgar language. The data-driven nature of some of our evaluated chat models means the chatbots are prone to reflecting any biases, toxicity, and vulgarity present in the training data (see Dinan et al. (2022) for a quantitative analysis). A high rate of antisocial behaviors among our evaluated models could potentially make human interactors’ experience talking with the bots quite uncomfortable, and would poorly reflect on the research field’s potential for social good. To mitigate this risk, the authors extensively hand-tested all evaluated chat models, as well as conducting a pilot evaluation among the authors’ lab group. As confirmed further in our results (Section 8), our chatbots exhibited negligible rates of antisocial behavior.

heading: 12 Acknowledgements
text: Thank you to Bradley Webb, Scott Heiner, and the rest of the SurgeHQ team for their guidance in running our annotation projects on their platform. We are also grateful to our colleagues at Emory for their participation in piloting the bots and refining the annotation interfaces. Lastly, a thank you to our reviewers for their helpful feedback.

heading: A Chatbot Selection Details
text: This appendix discusses the details of our literature review for each of the four chosen research themes.
General Our work focuses primarily on opendomain chat. Large-scale language modeling using dialogue-like pretraining data can produce surprisingly human-like conversation on virtually any popular conversation topic (Roller et al., 2021). Of these approaches, we chose Blender2 (Weston and Shuster, 2021), which reportedly outperformed its previous iteration Blender (Roller et al., 2021) who had surpassed DialoGPT (Zhang et al., 2020) and Meena (Adiwardana et al., 2020).
There had been also several chatbots produced by the Amazon Alexa Prize Socialbot Grand Challenge (Ram et al., 2018) focusing on general, opendomain chat, most of which incorporate rule-based methods to ensure interesting and consistent responses (Khatri et al., 2018). Since these chatbots performed well in practice but lack comparison to SOTA data-driven models, we selected the bot with the all-time highest final score, Emora15 (Finch et al., 2020), as one of our candidates.
Knowledge Grounding chat with supplementary knowledge resources is a common way to improve engagingness and control the topic of conversation (Li et al., 2019b; Ye et al., 2020). ColV (Zhan et al., 2021) achieved SOTA performance in knowledgegrounded dialogue response generation on the popular WoW dataset (Dinan et al., 2019); however, no implementation was publicly available. DukeNet (Meng et al., 2020) and PIPM (Chen et al., 2020) report next-best performance in this task. DukeNet’s implementation was available while PIPM’s was not, therefore we selected DukeNet as a candidate.
BART-FiD-RAG also reported compelling performance for knowledge-grounded chat (Shuster et al., 2021), but did not compare to other SOTA models we identified. Since BART-FiD-RAG’s inclusion in ParlAI provided easy replication, we included it in our bot pilot.
Consistency Improving consistency of chatbot responses is noted as a challenge and addressed in several works (Welleck et al., 2019; Nie et al., 2021; Li et al., 2021). DECODE (Nie et al., 2021) reported SOTA performance for general inconsistency avoidance, improving upon an approach that used unlikelihood training with dialogue natural 15https://github.com/emora-chat/emora_ap3_parlai
language inference data (Li et al., 2020). Note that there were several works focusing specifically on persona consistency (Song et al., 2020; Kim et al., 2020; Song et al., 2021), which we did not consider due to their narrower contradiction scope.
Empathy Several works demonstrated the importance of emotional understanding in chat (Partala and Surakka, 2004; Prendinger and Ishizuka, 2005; Kim et al., 2021; Sabour et al., 2022). To provide contrast with our knowledge-grounded candidates, we selected CEM (Sabour et al., 2022), which reported SOTA results in empathetic response generation. Many related works investigated controllable emotional response generation (Song et al., 2019; Zhong et al., 2021), but we did not consider models requiring an emotion label as input.

heading: B Chatbot Implementation Details
text: For each selected candidate model, a brief overview of the implementation details required to use them as interactive models in this work is below:
Emora We implement a ParlAI agent using the interactive chatting mode provided for the Emora system (Finch et al., 2020).
BART-FiD-RAG An interactive chatting mode for BART-FiDRAG is provided through ParlAI. We use the default arguments, except for compressed-indexer-nprobe and beam-context-block-ngram which we set to 128 and 3, respectively, following internal testing by the authors before the pilot.
Blender2 An interactive chatting mode for Blender216 is provided through ParlAI; however, the internet-search augmented functionality as described in Komeili et al. (2022) is not provided. Although there is an available alternative internet search server17 that is compatible with Blender2, it increased the response latency of Blender2 such that it no longer satisfied our response latency requirement. As a result, we use the 3B parameter version of Blender2 that does not include the internet search functionality.
Blender-Decode We implement a custom agent in the Parlai framework following the bestperforming setup of Nie et al. (2021) of finetuning a consistency classification model and using it to
16https://parl.ai/projects/blenderbot2/ 17https://github.com/JulesGM/ParlAI_SearchEngine
rerank generated responses from Blender (Roller et al., 2021) such that the least contradictory response is selected.
DukeNet The published code for DukeNet is missing an interactive chatting mode, although a trained model on the multi-topic Wizard of Wikipedia (WoW) (Dinan et al., 2019) is provided. We modified the published code to support an interactive chatting mode. However, one major difference is that the relevant knowledge documents need to be retrieved in interactive chatting mode, whereas they are predetermined in the training and evaluation environments. To remedy this, we use a trained Dense Passage Retrieval model (Karpukhin et al., 2020) provided through Sentence-Transformers18 to select knowledge at each dialogue turn. We implement a ParlAI agent using the available DukeNet model and our custom interactive chatting mode.
CEM The published code for CEM is missing both an interactive chatting mode and a trained model. We trained a CEM model using the default arguments from the repository and modified the published code to support an interactive chatting mode. We implement a ParlAI agent using this trained model and our custom interactive chatting mode.
18https://www.sbert.net/docs/pretrained-models/dpr.html

heading: C Bot Pilot Examples
text: Tables 6 and 7 show two of our chatbot pilot (Section A) conversations, one from CEM (Sabour et al., 2022) and one from Blender2 (Weston and Shuster, 2021), that exemplify the difference between single-turn and multi-turn dialogue response generation models. The CEM model is trained to give an empathetic response to a dialogue context, and achieves good performance towards this goal. However, as shown in the example, this response policy does not translate well for multi-turn interaction with a human. By contrast, Blender2 is trained and evaluated specifically to achieve multi-turn dialogue.

heading: D Pilots and Development
text: The final ABC-Eval label set, annotation procedure, and software application are created using an iterative process of development and piloting. 14 students are invited to serve as evaluators for piloting the evaluation. To avoid overfitting the evaluation design, our pilots evaluated conversations collected between Blender (Roller et al., 2021) and one of the authors, and a new set of conversations was used for each pilot round. We ran 4 pilot rounds, making revisions after manually reviewing each round’s annotation.
Table 8 presents a summary of the major changes made in each pilot round and IAA metrics. It is important to note that each annotator performed all of the annotation tasks in one sitting in sequence for each pilot. These piloting rounds are not necessarily directly comparable to one another when taken as a whole, since the annotator groups and dialogues to be annotated varied between each round. Instead, we will discuss below the major takeaways afforded by different splits of the pilots that informed the final design of ABC-Eval.
Subtask Formulation The decision to format ABC-Eval into several small annotation subtasks, each with a tailored subset of the behavior labels, was made from the results of Pilot 1. In Pilot 1, we divided the initial set of annotation labels into 3 annotation subtasks each with 4-9 labels: errors, information usages (commonsense, world knowledge, etc.), and utterance types (request, presentation, etc.). Each annotator performed the annotation tasks in one sitting in sequence. The overall interannotator agreement was quite low (α = 0.18), which was concerning. Based on ad-hoc feedback from the pilot annotators, the consensus was that each subtask demanded an unreasonable cognitive load on annotators due to the large number of labels to keep track of.
For Pilot 2 we increased the number of annotation tasks such that each covered a small and related scope of behavior labels, with 1-4 labels per task. Table 8 shows the boost to interannotator agreement between Pilots 1 and 2. However, this agreement increase could have resulted from an increase in the quality of the annotators (as Pilot 2 was composed primarily of annotators with a graduate-level education whereas Pilot 1 was more evenly split
between annotators with an undergraduate-level education and graduate-level education). To remove this confound, we calculated the agreement in Pilots 1 and 2 when only considering graduate-level annotators. Although it was less dramatic, there remained an increase in agreement from 0.39 to 0.50, which encouraged the decision to maintain the smaller annotation subtasks. Dividing the annotation into tailored subtasks seemed to reduce the cognitive load on annotators, thus allowing them to perform more accurate annotations per task.
Training and Screening Manual analysis of the pilot annotations from Pilots 1 and 2 revealed some recurring annotation mistakes, arising from misunderstandings of the guidelines for the tasks. In an attempt to correct such misunderstandings, a training procedure was introduced for each task.
Each round of training consists of 1 curated conversation with ground-truth labels and explanations that are shown as feedback to the annotator after they complete the training round. We used the results of Pilots 1 and 2 in order to develop these curated conversations as follows:
1. Label Specifications: We constructed a label specification that consisted of a comprehensive enumeration of positive and negative cases of the label with the goal of defining a decision boundary that the annotators should strive towards. We especially focused on the utterances for which several of the annotators failed to produce labels that matched the ground truth annotations we had defined for each of the Pilots.
2. Training Conversation Selection: We selected 3 conversations between Blenderbot and a human (from a collection within our lab) for each label to be used as training conversations for it. This selection was manually done by ranking the conversations on their coverage of the label specification.
3. Training Conversation Modification: We heavily revised the selected conversations by hand to ensure that all of the cases identified in the specification were adequately represented, most often by inserting new utterances that corresponded to any underrepresented cases.
To evaluate the utility of this training process, a third pilot was conducted using 4 undergraduates.
We observed a general upwards trend in annotation performance between the training rounds for the annotators, suggesting that the training was aiding in the annotation accuracy for the annotators. The final agreements were 0.43 and 0.45 between all annotators and annotators who passed the training, respectively, on the annotated conversations.
Due to the small nature of this pilot, we are unable to conclude whether this difference is meaningful. However, ad-hoc feedback from the annotators suggested that the training rounds were useful towards their understanding of the tasks, although the amount of training did increase the overall workload of participation. Accordingly, the decision was made to treat each subtask independently, rather than require all subtasks to be completed for one dialogue in a single sitting for each annotator.
General Revisions Throughout each of these pilot rounds the annotation instructions, examples, and training rounds were updated based on manual review of the annotations in an attempt to correct any unclear or misleading information.

heading: E Evaluator Training and Screening
text: We attempted to use three different groups of evaluators for our full evaluation study:
Students Undergraduate students were recruited from the authors’ university via word-of-mouth and email advertisements sent to computer science, psychology, and quantitative methods departmental mailing lists.19
MTurkers Our 20 evaluation tasks were posted to the Amazon Mechanical Turk crowdsourcing platform.20
19Students were compensated with an Amazon gift card at the completion of the data collection. 20https://www.mturk.com/
Surgers Our 20 evaluation tasks were posted on SurgeHQ’s annotation platform21 to be completed by dedicated workers with experience in NLP annotation. A group of 125 Surgers were qualified to participate in our tasks, chosen by a SurgeHQ employee on the basis of high annotation performance on past projects.
All three groups were compensated per task per annotated conversation, at an estimated rate of $10/hr for Students and MTurkers, and $20/hr for Surgers. To check the viability of each worker group to produce evaluation data for our full study, we released a random 5 conversations out of our set of 400, to be fully evaluated by each worker group in each of our 8 ABC-Eval tasks. After a two week period, Surgers were the only worker group that were able to fully evaluate the 5 conversations in all 8 ABC-Eval tasks. This was due to an overall lack of participation from the Student group, and due to low training pass rates from the MTurk group (see Figure 9 for quantitative outcomes). Although worker group differences in work rate and training performance might be explained by the difference in compensation structure, we decided to proceed with the Surgers group only for our full study to collect our evaluation data in a timely manner.

heading: F Collection Cost
text: Compensation rates are based on per-task completion times from an internal pilot run. The rates per task paid to Surgers are shown in Table 9. We also present the real and theoretical costs for collecting each method included in our evaluation data (Table 10). As expected, turn-level annotation tasks are an order of magnitude more expensive to collect than dialogue-level tasks. Notably, the final set of ABC-Eval labels (Table 4) are, on average, less expensive to collect than turn-level Likert labels.
21https://www.surgehq.ai/

heading: G Evaluation Interfaces
text: Examples of the annotation interfaces for each annotation task of ABC-Eval are provided in Figures 10 - 17, and an example for the conversation collection interface is provided in Figure 21.
Examples of the annotation interfaces for Dialogue Likert, Turn Likert, and Comparative evaluations are provided in Figures 18, 19, and 20, respectively. The definitions that were shown to the annotators in the interface for each of the 8 dimensions of Dialogue Likert, Turn Likert, and Comparative are taken verbatim from Finch and Choi (2020).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 10

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 11

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sections 3, 5, and 6

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections 2 and 5

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sections 2, 4, 5, and 11 and Appendix A, B, D, G

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 11

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sections 3 and 6
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sections 3 and 6
C 3 Did you run computational experiments? Sections 3, 7, and 8

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 2 and Appendix B for models and parameters used. Computational budget (GPU hours) and computing infrastructure was not provided because experiments were of low-resource intensity (GPU only required for decoding steps of 300 conversations and the bulk of computational experiments had quick runtimes and did not require GPU infrastructure).
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values?
text: Computational experiments were mainly on evaluation methodologies which did not require hyperparameter tuning; Appendix B provides hyperparameter specifications for dialogue models used in this work

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sections 7 and 8
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? All code, software, data, and procedural descriptions for our study will be released at the time of publication. Our list of used packages is well-documented in the files, including settings used. Additionally, Appendix B reports model implementation details with parameter settings.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Sections 2, 3, and 6

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 3, 5 and Appendix D, E, G

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3, 6 and 11 and Appendix E, F

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 11
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Sec. 11

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 3 and 6

heading: Future Work
text: Finally, to aid future research efforts in human-computer chat modelling and evaluation, we release an anonymized version of our conversation and evaluation data. One ethical consideration involved in our work involved managing human workers in our data collection processes. All worker participation in our study was voluntary and involved zero subjective screening processes, with a complete description of worker tasks, workload, and timeframe provided before work was assigned. Workers could opt out of our study at any time for any reason. As compensation for work completed, we targeted a compensation rate of $10/hour for student14 and Amazon Mechanical Turk workers, and a rate of $20/hour for Surgers. We compensated on a pertask-completed basis to ensure timely completion of work, but verified that target hourly rates were reasonably approximated throughout the course of the study by measuring workers’ median task completion times (see Appendix F for details). These measures ensured that all human work in our study 13Smith et al. (2022) reports standard deviations of Likert
metrics between 0. 8 and 1. 3 14Students’ compensation is given as an Amazon Gift Card
for convenience; students are informed of this prior to any work being completed
was fair, transparent, and mutually-beneficial. Other ethical considerations arise in our study’s conversation collection. Unlike the collection of evaluation or annotation data, collecting interactive conversation data from human-computer interaction poses a small but meaningful risk that sensitive, damaging, or personally identifying information could get collected. We mitigated this risk in three ways. First, students were notified in multiple email communications and before each conversation that their conversations with our chatbots would be publicly released. Included in these notices was the instruction to refrain from releasing any personally identifiable or damaging information. Our instructions suggest that students fabricate personal information at any time during the conversations if it would make them feel more comfortable. Second, we hand-checked all 400 conversations to ensure the non-presence of any sensitive information. Third, we anonymize all data before public release. Our study’s collection and analysis of conversation data did not investigate interactors as human subjects, and we did not seek institutional review board approval. Finally, there is a concern in our study about the potential of the chatbots to respond to student interactors with toxic, insensitive, or vulgar language. The data-driven nature of some of our evaluated chat models means the chatbots are prone to reflecting any biases, toxicity, and vulgarity present in the training data (see Dinan et al. (2022) for a quantitative analysis). A high rate of antisocial behaviors among our evaluated models could potentially make human interactors’ experience talking with the bots quite uncomfortable, and would poorly reflect on the research field’s potential for social good. To mitigate this risk, the authors extensively hand-tested all evaluated chat models, as well as conducting a pilot evaluation among the authors’ lab group. As confirmed further in our results (Section 8), our chatbots exhibited negligible rates of antisocial behavior