SlowGuess/ACL_2020_OCR huggingface dataset:
Weixin Liang $^{1}$ , James Zou $^{1}$  and Zhou Yu $^{2}$
$^{1}$  Stanford University,  $^{2}$  University of California, Davis {wxliang, jamesz} @stanford.edu, joyu@ucdavis.edu
Open Domain dialog system evaluation is one of the most important challenges in dialog research. Existing automatic evaluation metrics, such as BLEU are mostly reference-based. They calculate the difference between the generated response and a limited number of available references. Likert-score based self-reported user rating is widely adopted by social conversational systems, such as Amazon Alexa Prize chatbots. However, self-reported user rating suffers from bias and variance among different users. To alleviate this problem, we formulate dialog evaluation as a comparison task. We also propose an automatic evaluation model CMADE (Comparison Model for Automatic Dialog Evaluation) that automatically cleans self-reported user ratings as it trains on them. Specifically, we first use a self-supervised method to learn better dialog feature representation, and then use KNN and Shapley to remove confusing samples. Our experiments show that CMADE achieves  $89.2\%$  accuracy in the dialog comparison task. Our implementation is available at https://github.com/Weixin-Liang/dialog.evaluation_CMADE.
Open-domain dialog system evaluation is one of the most difficult challenges in the dialog community. Open-domain chatbots have a user-centric goal: to provide human with enjoyable user experience. However, user experience is difficult to quantify due to bias and variance among different users. Previous research has optimized on automatic dialog evaluation metrics such as BLUE (Papineni et al., 2002), which measures the difference between the generated responses and the reference responses. Due to the contrast between the one-to-many nature of open-domain conversations and the limited number of available references, such met
rics correlate poorly with human judgments (Liu et al., 2016; Lowe et al., 2017; Novikova et al., 2017). Designing a fully automatic dialog evaluation metric is still an open research problem.
Currently, both academia and industry (Ram et al., 2018a; Li et al., 2019b; Liang et al., 2019) rely on human ratings to evaluate open-domain dialogs. Following the ubiquitous application of Likert scores in survey research like online reviews (Godes and Silva, 2012) and consumer satisfaction (Peterson and Wilson, 1992), a common practice of human evaluation on dialogs is to ask either a third-person rater or the chatbot user to report a Likert score. However, concerns have been raised about the validity of Likert score-based ratings. Kulikov et al. (Kulikov et al., 2018) observe high bias and variance of Likert scores. Such issue is more severe in real-world commercial dialog systems like Alexa social chatbot (Ram et al., 2018a; Venkatesh et al., 2018), because the real-world users have neither monetary incentive nor necessary annotation training to calibrate their ratings.
To explore the validity of Likert score based dialog evaluation, we first perform a large-scale data analysis of 3,608 collected real-world human-machine dialogs along with their self-reported Likert scale ratings from Amazon Alexa Prize Challenge (Ram et al., 2018a; Yu et al., 2019; Chen et al., 2018). One noticeable property of the ratings is its J-shape skew distribution: nearly half of the dialogs are rated with the highest Likert score. The prevalence of such extreme distribution of ratings has long been observed by the business research community in variable aspects of real-life (Schoenmüller et al., 2018; Godes and Silva, 2012; Hu et al., 2017; Zervas et al., 2015).
Although we could tell which dialog system is better by running statistical test on a large number of noisy ratings, it is difficult to locate dialogs with bad performance reliably to improve dialog system
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1363-1374 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
quality. In this paper, we take on the challenge of calibrating a large number of noisy self-reported user ratings to build better dialog evaluation models. We formulate the task as to first denoise the self-reported user ratings and then train a model on the cleaned ratings. We design CMADE (Comparison Model for Automatic Dialog Evaluation), a progressive three-stage denoising pipeline. We first perform a self-supervised learning to obtain good dialog representations. We then fine-tune CMADE on smoothed self-reported user ratings to improve the dialog representation while preventing the network from overfitting on noisy ratings. Finally, we apply data Shapley to remove noisy training data, and fine-tune the model on the cleaned training set. Our experiments show that CMADE is able to successfully identify noisy training data and achieves  $89.2\%$  in accuracy and 0.787 in Kappa on a test set with unseen expert-rated dialog pairs.
Open-domain dialog system evaluation is a long-lasting challenge. It has been shown that previous automatic dialog evaluation metrics correlate poorly with human judgments (Liu et al., 2016; Lowe et al., 2017; Novikova et al., 2017). A well-known reason is that these automatic dialog evaluation metrics rely on modeling the distance between the generated response and a limited number of references available. The fundamental gap between the open-ended nature of the conversations and the limited references (Gupta et al., 2019) is not addressed in methods that are lexical-level based (Papineni et al., 2002; Lin, 2004; Banerjee and Lavie, 2005), embedding based (Rus and Lintean, 2012; Forgues et al., 2014), or learning based (Tao et al., 2018; Lowe et al., 2017).
Given the aforementioned limitations, Likert-score based rating is the de-facto standard for current dialog research and social conversational systems such as in Amazon Alexa Prize Challenge (Yu et al., 2019; Chen et al., 2018). Various forms of evaluation settings have been explored to better measure human judgments. Single-turn pairwise comparison (Vinyals and Le, 2015; Li et al., 2016) is primarily used for comparing two dialog systems. Each system predicts a single utterance given the static "gold" context utterance from human-human logs. Although such A/B test setting is robust to annotator score bias, it cannot capture the multi-turn nature of dialogs. A more complete multi-
turn evaluation is typically measured with a Likert scale for the full dialog history, where either a third-person rater or the chatbot user (Pérez-Rosas et al., 2019) reports a Likert score on user experience (Venkatesh et al., 2018), engagement (Bohus and Horvitz, 2009) or appropriateness (Lowe et al., 2017). However, as observed in (Kulikov et al., 2018; Ram et al., 2018a; Venkatesh et al., 2018) Likert scores suffer from bias and variance among different users. Different from previous empirical observations, we conduct a large-scale quantitative and qualitative data analysis of Likert score based ratings. To address the issue of Likert scores, the Alexa team proposed a rule-based ensemble of turn-granularity expert ratings (Yi et al., 2019), and automatic metrics like topical diversity (Guo et al., 2018) and conversational breadth. ACUTE-EVAL (Li et al., 2019a) makes a small-scale attempt to use multi-turn pair-wise comparison to rank different chatbots. Given the ubiquity and simplicity of Likert scores based evaluation, instead of proposing an alternative measure, we take on the challenge of denoising Likert scores with minimal expert annotations introduced (one order of magnitude smaller). Different from (Li et al., 2019a), our proposed expert annotation scheme is for comparing the dialogs within the same chatbot.
3 Dialog Rating Study
The data used in this study was collected during the 2018 Amazon Alexa Prize Competition (Ram et al., 2018b). Our data contain long and engaging spoken conversations between thousands of real-world Amazon Alexa customers and Gunrock, the 2018 Alexa Prize winning social bot (Yu et al., 2019). The chatbot has 11 topic dialog modules including movies, books, and animals. One notable characteristic of the chatbot is its versatile and complex dialog flows which interleaves facts, opinions and questions to make the conversation flexible and interesting (Chen et al., 2018). At the end of each dialog, a self-reported Likert scale rating is elicited by the question "on a scale of one to five, how likely would you talk to this social bot again?"
We first filter out dialogs that have inappropriate content using keyword matching. We then select 3,608 ten-turn dialogs on movies, because movie dialogs are more coherent and diverse compared to other topics according to both real users and Amazon selected experts. We observe that dialogs
Figure 1: Schematic of the CMADE workflow. CMADE contains a three-stage training pipeline to denoise self-reported ratings to train an automatic dialog comparison model: learning representation via self-supervised dialog flow anomaly detection, fine-tuning with smoothed self-reported user ratings, denoising with data Shapley & further fine-tuning. The gray and blue rectangles in stage 1 represents system and user utterances. The red rectangle in stage 1 represents the randomly replaced system utterance for dialog flow perturbation. In stage 2 & 3, each ball represents a dialog in the training data. The number on each ball represents the dialog rating.
with more than eight turns are more meaningful and semantically versatile, while dialogs more than 10 turns exceed the max length limit of the BERT model (512 tokens). So we select dialogs that have ten turns. Our approach could support longer conversations by adopting a memory footprint efficient algorithm for self-attention to support sequences with thousands of tokens (Huang et al., 2019). We leave this to future work.
We aim to evaluate user experience for each dialog from the same chatbot of the same length. This is significantly more challenging than identifying which chatbot provides a better user experience on average since our problem setup requires us to capture more subtle difference in user experience.
3.2 Likert Score Based Evaluation
Rating	1	2	3	4	5
Count	386	404	566	664	1588
Fraction	10.7%	11.2%	15.7%	18.4%	44.0%
Table 1: The statistics of self-reported Likert scale ratings. The distribution is heavily skewed and noisy: nearly half of the dialogs are rated with score  $= 5$ .
J-Shape Skewness We perform a detailed analysis of the self-reported Likert scale ratings. As shown in Table 1, abnormally, nearly half of the dialogs are rated as five, which is the highest score. A similar skewed distribution is also observed in previous years' Alexa competition (Fang et al., 2018). In fact, the business research community has long observed the prevalence of the extreme distribution of reviews in which the reviews are heavily skewed to the positive end of the rating scale (known as
"J-shape") in online reviews (e.g., Amazon, Airbnb, Yelp) (Godes and Silva, 2012; Hu et al., 2017; Zervas et al., 2015), word of mouth (East et al., 2007) and consumer satisfaction (Peterson and Wilson, 1992; Danaher and Haddrell, 1996).
Comparison to expert ratings We randomly selected 50 dialogs rated score-5 and showed these to an expert, and our expert rated 27 of them with score-4 or less. The Alexa team (Venkatesh et al., 2018) has also reported that the inter-user agreement is quite low for their internal rating analysis. Such phenomena indicate that the self-reported Likert scale ratings are extremely noisy. Using such ratings cannot localize individual bad interactions. In addition, Likert score based evaluation also suffers from insensitivity issues. As observed by the Alexa team (Venkatesh et al., 2018) in multiple internal user studies, even though users evaluated multiple dialogs with the same score, they had a clear rank order among the dialogs.
The skewness, noisiness and insensitivity of the self-reported Likert scale rating make it a suboptimal dialog evaluation metric. In practice, we find that directly training a classifier (even for pre-trained BERT-based model) on the noisy self-reported Likert scale ratings suffers from underfitting. One of the Alexa Price Challenge team, Alana (Papaioannou et al., 2017) train a binary-classifier between successful dialogs (human rating 4 or 5) and unsuccessful dialogs (rating 1 or 2) with heavy hand-engineered features. They reach  $69.40\%$  accuracy on this binary classification problem, which is far from usable in real-world settings.
3.3 Pairwise Comparison Based Evaluation
Selecting the better dialog from two options is easier for a human evaluator than giving an absolute number like the Likert score, which requires the evaluator to maintain a consistent standard. People's perception is inherently relative, and pair-wise comparison is local and does not require the user to have global consistency. There are many other examples where humans find it easier to perform pairwise comparisons rather than providing direct labels (Simpson and Gurevych, 2018; Mailthody et al., 2019; Liang et al., 2018), including content search (Furnkranz and Hüllermeier, 2010), image retrieval (Wah et al., 2014; Feng et al., 2019), and age estimation (Zhang et al., 2017).
We randomly sample 400 dialog pairs for experts to annotate. We ask the question, "If you were the user, in which scenario would you be more likely to come back and talk to the system again?" We guide the experts to focus on the user experience rather than calibrating the performance of any specific module of the dialog system. Two researchers with conversational training experience annotated the data. The leading expert has been working in an Alexa competition team for more than one year with an emphasis on the user ratings. For each dialog pair  $(A,B)$ , they label 'A is better than  $B$  or 'B is better than  $A$  or cannot tell'. They reached a high inter-annotator agreement score (Cohen, 1968) with kappa  $\kappa = 0.83$ . To make sure that the dev & test is accurate, we throw away all "cannot tell" dialog pairs. We then study the correlation between Likert score based evaluation and pairwise comparison based evaluation.
3.4 Correlation Between User Ratings and Expert Ratings
Delta of Self-Reported Ratings (e.g., 5-1=4)	Δ=1	Δ=2	Δ=3	Δ=4
Disagreement Rate	0.45	0.383	0.220	0.157
Table 2: The correlation between the self-reported Likert scale ratings and our pair-wise comparison annotation. For a pair of dialogs, if the delta of self-reported Likert scale ratings is large, then they are more likely to align with the comparison results from experts.
To further analyze the self-reported Likert scale ratings, we also compare the annotated labels of the 403 dialog pairs with the self-reported Likert scale ratings of these dialogs. For each pair of dialogs,
we compare the pairwise comparison label and the delta between the self-reported Likert scale ratings of the two dialogs. Ideally, the dialog with a higher self-reported Likert scale rating should be the one that is annotated as having a better user experience in the pairwise comparison. We count the number and fraction of "disagreement" between the two types of ratings. Overall, roughly 1/3 of the dialog pairs disagree. As shown in Table 2, as the gap between the self-reported Likert scale ratings becomes larger, the disagreement between expert and self-reported ratings goes down. This suggests that if the difference between the two dialogs' Likert score is huge, they are more likely to be consistent with the comparison ratings.
Suppose the training set  $D_{train}$  consists of data points  $D_{train} = \{(x_i, y_i)\}_{1}^{N_{train}}$  where  $x_i$  is a dialog and  $y_i$  is the noisy self-reported user ratings. We define a strict partial order relationship  $\triangleright$  where  $x_i \triangleright x_j$  means that dialog  $x_i$  provides a better user experience than dialog  $x_j$ . Note that  $y_i > y_j$  does not always imply  $x_i \triangleright x_j$  since self-reported user ratings are noisy (§ 3.3, § 3.4). The test set  $D_{test}$  consists of  $N_{test}$  dialog pairs along with their binary pair-wise comparison labels  $D_{test} = \{(x_i^{test}, x_j^{test}, z_{i,j}^{test})\}_{i,j \in I^{test}}$ , where  $z_{i,j}^{test}$  is annotated by experts and indicates whether dialog A provides a better user experience than dialog B, i.e.,  $z_{i,j}^{test} = \mathbb{1}(x_i \triangleright x_j)$ . The development set  $D_{dev}$  has a similar structure.
Following the structure of the expert annotated pairs, we formulate our model  $M(\phi, f)$  as a pairwise dialog predictor with a similar architecture as RankNet (Burges et al., 2005). For a dialog pair  $(x_i, x_j)$ , the model predicts an un-normalized score  $o_i, o_j \in \mathbb{R}$  for each dialog:  $o_i = f(\phi(x_i))$  and  $o_i = f(\phi(x_j))$  where  $\phi$  is a dialog encoder that maps each dialog to a feature space and  $f$  is a linear transformation that converts each dialog feature into a real number  $o$ . We define a binary relationship  $\hat{v}$  where  $x_i \hat{v} x_j$  means that the model predicts that dialog  $x_i$  provides a better user experience than dialog  $x_j$ . We denote model's prediction of  $z_{i,j}$  as  $z_{i,j}^{\hat{v}}$  where  $z_{i,j}^{\hat{v}} = \mathbb{1}(x_i \hat{v} x_j)$ . We model the predicted posterior  $P(z_{i,j}^{\hat{v}} = 1) = P(x_i \hat{v} x_j)$  as:
$$
P (z _ {i, j} ^ {\hat {}} = 1) = P (x _ {i} \hat {>} x _ {j}) = \frac {1}{1 + e ^ {- (o _ {i} - o _ {j})}}
$$
Our goal is to reduce the noise of the self-reported user ratings (§ 3). Directly training a classification model using the noisy ratings leads to severe underfitting. To this end, we propose a three-stage training pipeline to denoise self-reported ratings to train an automatic dialog comparison model. Figure 1 describes the overall pipeline:
- In Stage 1, we learn dialog feature representation with a self-supervised dialog flow anomaly detection task.
- In Stage 2, we perform label smoothing to adjust the noisy self-reported ratings in the training set and fine-tune the dialog comparison model on the smoothed ratings.
- In Stage 3, we perform data Shapley (Ghorbani and Zou, 2019; Jia et al., 2019a) on the self-reported user ratings to identify and remove noisy data points. We then fine-tune the dialog comparison model on the cleaned training set.
5.1 Stage 1: Learning Representation via self-supervised dialog anomaly detection
Sys: What movie did you see?  
User: Spider man into the spider verse
Sys: Ah, I know about Spider man into the spider verse! I'm wondering. What would you rate this movie on a scale from 1 to 10?
Replaced Sys: Isn't it crazy how famous actors can get? Are you interested in talking more about Scarlett Johansson?
Table 3: A fake dialog example created by dialog flow perturbation in Stage 1. We perturb the dialog flow by replacing a system utterance (here the second Sys utterance in the table) with a random system utterance from the corpus (here the replaces Sys utterance) to generate a fake dialog. With high probability, the fake dialog is less appropriate than the origin one.
Having a good dialog representation is the first step towards denoising the data. Our primary goal in this stage is to train a dialog encoder  $\phi$  to learn good dialog feature representations for the following stages. Here  $\phi$  could be any sequence encoder that could encode a dialog and we use BERT (Devlin et al., 2019) in this paper.
For each dialog in the training set, we perturb the dialog flow to generate a fake dialog
and train the model to differentiate the fake dialog and the real one. Dialog flow is a user-centric measure of whether a conversation is "going smoothly" (Eskénazi et al., 2019). To perturb the dialog flow for each dialog  $x_{i}$ , we randomly replace a user utterance in  $x_{i}$  with a random user utterance from the training corpus  $D_{train}$ , yielding a perturbed dialog  $x_{i,face}$ . With high probability, the system utterance immediately following the replaced user utterance becomes inappropriate. Therefore, we incorporate  $\{(x_{i}, x_{i,face}, z = 1)\}$  into the training pairs. Similarly, we also randomly replace a system utterance and yield another perturbed dialog. We generate two perturbed dialogs for each dialog in the training set and thus  $2N_{train}$  real-fake dialog pairs in total. An example is shown in Table 3. We note that appropriateness is one of the most widely applied metrics of human evaluation on dialogs (Lowe et al., 2017). By learning to differentiate the perturbed dialog and the original one, we expect CMADE to learn a good dialog encoder  $\phi$  which maps dialogs with similar dialog flow close to each other in the feature space.
5.2 Stage 2: Fine-tuning with smoothed self-reported user ratings
Stage 1 only performs unsupervised learning and does not incorporate any supervision from human ratings. To obtain better dialog feature representations for Stage 3, Stage 2 fine-tunes  $\phi$  with supervision from the noisy self-reported user ratings. We adopt a simple yet effective label smoothing, inspired by (Szegedy et al., 2016; Nie et al., 2019), using the representation learned in Stage 1. A key assumption in Stage 2 is that dialogs with similar dialog flow provide a similar user experience. For each dialog  $x_{i}$ , we find its  $K$  nearest neighbors in the feature space defined by  $\phi$ . We use the average self-reported ratings of the  $K$  nearest neighbors as a smoothed rating  $y_{i}^{s}$  for  $x_{i}$ . To construct training dialog pairs, we randomly sample dialog pairs  $x_{i}$  and  $x_{j}$  and derive a pair-wise comparison label  $z_{i,j}^{s}$  by comparing the smoothed rating  $y_{i}^{s}$  and  $y_{j}^{s}$ :  $z_{i,j}^{s} = \mathbb{1}(y_{i}^{s} > y_{j}^{s})$ . We discard the pairs with equal  $y_{i}^{s}$  and  $y_{j}^{s}$ . To improve the dialog feature representation, we fine-tune the model  $M(\phi, f)$  on sampled dialog pairs along with the derived labels from comparing the smoothed scores  $\{x_{i}, x_{j}, z_{i,j}^{s}\}$ . We note that  $z_{i,j}^{s}$  depends solely on the noisy self-reported ratings in the training set and does not depend on the expert annotations. Theoretically, we could iter
ate between label smoothing and model fine-tuning since the fine-tuned model provides better dialog feature representation. In practice, we find that one iteration is enough to reach good prediction performance.
Label smoothing has led to state-of-the-art models in image classification (Szegedy et al., 2016), language translation (Vaswani et al., 2017) and speech recognition (Chorowski and Jaitly, 2017). Prior attempts in label smoothing (Szegedy et al., 2016; Vaswani et al., 2017; Chorowski and Jaitly, 2017; Muller et al., 2019) focus on categorical labels to prevent the network from becoming overconfident while we apply label smoothing on ordinal labels (i.e., Likert scores) to prevent the network from overfitting on noisy ordinal labels.
5.3 Stage 3: Denoising with data Shapley & further fine-tuning
In Stage 2, noisy ratings still have effect in the smoothed ratings for other data points. In Stage 3, we aim to identify and remove dialogs with noisy self-reported user ratings  $y_{i}$  with data Shapley value technique (Ghorbani and Zou, 2019; Jia et al., 2019a,b). Shapley value comes originally from cooperative game theory (Dubey, 1975). In a cooperative game, there are  $n$  players  $D = \{1,\dots,n\}$  and a utility function  $v:2^{[n]}\to \mathbb{R}$  assigns a reward to each of  $2^{n}$  subsets of players:  $v(S)$  is the reward if the players in subset  $S\subseteq D$  cooperate. Shapley value defines a unique scheme to distribute the total gains generated by the coalition of all players  $v(D)$  with a set of appealing mathematical properties. Shapley value has been applied to problems in various domains, ranging from economics (Gul, 1989) to machine learning (Cohen et al., 2005; Yona et al., 2019).