SlowGuess/ACL_2020_OCR huggingface dataset:
In our setting, given  $D_{train} = \{(x_i,y_i)\}_{1}^{N_{train}}$  we view them as  $N_{train}$  players. We could also view the utility function  $v(S)$  as the performance on the development set. The Shapley value for player  $i$  is defined as the average marginal contribution of  $\{(x_i,y_i)\}$  to all possible subsets that are formed by other users (Jia et al., 2019a):
$$
s_{i} = \frac{1}{N}\sum_{S\subseteq D_{train}\setminus \{x_{i}\}}\frac{1}{\binom{N - 1}{|S|}} [v(S\cup \{x_{i}\}) - v(S)]
$$
As suggested by the definition of data Shapley, computing data Shapley value requires an exponentially large number of computations to enumerate  $\mathcal{O}(2^{N_{train}})$  possible subsets and train the
model  $M$  on each subset, which is intractable. Inspired by (Jia et al., 2019a), CMADE tackles this issue by reducing the deep model  $M$  to a k-nearest neighbors (KNN) model and then apply the closed-form solution of shapley value on KNN. Using the feature extractor  $\phi$  trained in Stage 1 and Stage 2, we fix  $\phi$  and map all dialogs in the training data  $\{x_{i}\}_{1}^{N_{train}}$  to  $\{\phi(x_i)\}_{1}^{N_{train}}$ . We first define the utility function  $v(S)$  in a special case where the development set only contains one dialog pair  $(x_{p}^{dev}, x_{q}^{dev}, z_{p,q}^{dev})_{p,q \in I^{dev} = \{(1,2)\}}$ . In our setting, the development set contains dialog pairs annotated by experts. Given any nonempty subset  $S \subseteq D_{train}$ , we use the KNN Regressor to rate  $x_{p}^{dev}$  and  $x_{q}^{dev}$ . To do this, we compute  $\phi(x_{p}^{dev})$  and sort  $\{x_p\}_{1}^{N_{train}}$  based on their euclidean distance in the dialog feature space to  $x_{p}^{dev}$ , yielding  $(x_{\alpha_1^{(p)}}, x_{\alpha_2^{(p)}}, \ldots, x_{\alpha_{|S|}^{(p)}})$  with  $x_{\alpha_1^{(p)}}, \ldots, x_{\alpha_K^{(p)}}$  as the top-K most similar dialogs to  $x_{p}^{dev}$ . Similarly, we get  $(x_{\alpha_1^{(q)}}, x_{\alpha_2^{(q)}}, \ldots, x_{\alpha_{|S|}^{(q)}})$  with  $x_{\alpha_1^{(q)}, \ldots, x_{\alpha_K^{(q)}}}$  as the top-K most similar dialogs to  $x_{q}^{dev}$ . Based on the self-reported user ratings in the training data, we use the KNN Regressor to rate  $x_{p}^{dev}$  and  $x_{q}^{dev}$  as follows:
$$
\hat {y} _ {p} ^ {d e v} = \frac {1}{K} \sum_ {k = 1} ^ {\min  \{K, | S | \}} y _ {\alpha_ {k} ^ {(p)}} \tag {1}
$$
$$
\hat {y} _ {q} ^ {d e v} = \frac {1}{K} \sum_ {k = 1} ^ {\min  \{K, | S | \}} y _ {\alpha_ {k} ^ {(q)}} \tag {2}
$$
The model predicts  $\hat{z}_{p,q}^{dev} = 1$  if  $\hat{y}_p^{dev} > \hat{y}_q^{dev}$  and vice versa.
To obtain a closed-form solution to calculate Shapley value, instead of defining the utility function as the accuracy of the pair-wise prediction, we define the utility function as follows:
$$
v (S) = \left\{ \begin{array}{l l} \hat {y} _ {p} ^ {d e v} - \hat {y} _ {q} ^ {d e v}, & \text {i f} \quad z _ {p, q} ^ {d e v} = 1, \\ \hat {y} _ {q} ^ {d e v} - \hat {y} _ {q} ^ {d e v}, & \text {i f} \quad z _ {p, q} ^ {d e v} = 0 \end{array} \right. \tag {3}
$$
Theorem 1 Consider the utility function in Equation (3). Then the Shapley value of each training point  $s_m$  can be decomposed into two terms  $s_m^{(p)}$  and  $s_m^{(q)}$  which depend on  $x_p^{dev}$  and  $x_q^{dev}$  respectively.  $s_m^{(p)}$  and  $s_m^{(q)}$  can be calculated recursively as follows:
$$
\begin{array}{l} s _ {m} = \left\{ \begin{array}{l l} s _ {m} ^ {(p)} - s _ {m} ^ {(q)}, & \mathrm {i f} z _ {p, q} ^ {d e v} = 1, \\ s _ {m} ^ {(q)} - s _ {m} ^ {(p)}, & \mathrm {i f} z _ {p, q} ^ {d e v} = 0 \end{array} \right. \\ s _ {\alpha_ {N} ^ {(p)}} ^ {(p)} = \frac {y _ {\alpha_ {N} ^ {(p)}}}{N} s _ {\alpha_ {N} ^ {(q)}} ^ {(q)} = \frac {y _ {\alpha_ {N} ^ {(q)}}}{N} \\ s _ {\alpha_ {m}} ^ {(p)} = s _ {\alpha_ {m + 1}} ^ {(p)} + \frac {y _ {\alpha_ {m} ^ {(p)}} - y _ {\alpha_ {m + 1} ^ {(p)}}}{K} \frac {m i n \{K , m \}}{m} \\ s _ {\alpha_ {m} ^ {(q)}} ^ {(q)} = s _ {\alpha_ {m + 1} ^ {(q)}} ^ {(q)} + \frac {y _ {\alpha_ {m} ^ {(q)}} - y _ {\alpha_ {m + 1} ^ {(q)}}}{K} \frac {\operatorname* {m i n} \{K , m \}}{m} \\ \end{array}
$$
With Theorem 1, the Shapley value calculation could be finished in  $\mathcal{O}(N\log N)$  time. The above result for a single point in the development set could be readily extended to the multiple-testpoint case. In our experiment, with such optimization, the Shapley value calculation takes less than 5 seconds to finish. Theorem 1 comes primarily from (Jia et al., 2019a,b) and we extends their results of vanilla KNN regressor (Jia et al., 2019a) to our pairwise testing setting.
By applying the Shapley technique to the data, we identify noisy training data points which contribute negatively to the performance and remove them from the training set. Similar to Stage 2, to construct training dialog pairs, we randomly sample dialog pairs  $x_{i}$  and  $x_{j}$  from the cleaned training set and derive  $z_{i,j}$  by comparing the self-reported rating  $y_{i}$  and  $y_{j}$ . We then further fine tune the model from Stage 2. Theoretically, we could iterate between Stage 2 and Stage 3 multiple times while in practice one iteration is enough.
5.4 Towards Scalable Pair-based Training
We use a similar factorization technique for pairwise ranking in LambdaRank (Burges et al., 2006) to speed up training. For Stage 2 and 3, we have  $\mathcal{O}(N^2)$  possible dialog pairs, which leads to quadratically increasing training time. Similar to LambdaRank (Burges et al., 2006), it is possible to calculate the exact gradient of  $\mathcal{O}(N^2)$  possible dialog pairs with  $\mathcal{O}(N)$  forwards and backpropagations. More specifically, we denote the possible input pairs during training at Stage 2 or Stage 3 as:  $D_{train}^{pair} = \{(x_i,x_j,z_{i,j})\}_{i,j\in I}$ . The total cost  $L$  for  $\mathcal{O}(N^2)$  possible dialog pairs is the sum of  $\mathcal{O}(N^2)$  cross-entropy costs:
$$
\begin{array}{l} L _ {i, j} = C r o s s E n t r o p y (\hat {z} _ {i, j}, z _ {i, j}) \\ L = \sum_ {(i, j) \in I} L _ {i, j} \\ \end{array}
$$
Theorem 2 We can compute  $\frac{\partial L}{\partial w_k}$  in  $\mathcal{O}(N)$  by factoring it into a weighted sum of  $\frac{\partial o_i}{\partial w_k}$  where the weight  $\lambda_i \in \mathbb{R}$  only depends on  $\{o_j\}$  and  $\{z_{i,j}\}$ . W.l.o.g., we assume  $z_{i,j} \equiv 1$ .
$$
\frac {\partial L}{\partial w _ {k}} = \sum_ {i} \lambda_ {i} \frac {\partial o _ {i}}{\partial w _ {k}}
$$
$$
\lambda_ {i} = \sum_ {j: (i, j) \in I} \frac {- 1}{1 + e ^ {o _ {i} - o _ {j}}} + \sum_ {j: (j, i) \in I} \frac {1}{1 + e ^ {o _ {i} - o _ {j}}}
$$
Here  $o_i = f(\phi(x_i)) \in \mathbb{R}$  and  $o_j = f(\phi(x_j)) \in \mathbb{R}$  are the outputs of the two branches of the model. Theorem 2 shows that instead of performing backpropagation for all possible pairs, we could first perform  $N$  forward passes to obtain  $\{o_j\}$  and then calculate  $\{\lambda_i\}$ . Calculating  $\{\lambda_i\}$  from  $\{o_j\}$  in Equation 5.4 takes negligible time since this stage does not involve any neural network operation. Finally, we calculate a weighted sum of  $\mathcal{O}(N)$  backpropagation and update the model parameters.
Model Setup We fine tune the pre-trained BERT (Devlin et al., 2019) to learn the dialog feature extractor  $\phi$ . We partition the 403 expert annotated dialog pairs into a 200-pair development set and a 203-pair test set. We set  $K = 50$  for both the KNN label smoothing in Stage 2 and the KNN Shapley value calculation in Stage 3.
Model Details The details of extending BERT to encode multi-turn dialogs are as follows. Each dialog is represented as a sequence of tokens in the following input format: Starting with a special starting token [CLS], we concatenate tokenized user and system utterances in chronological order with [SEP] as the separators for adjacent utterance. In other words, we represent each dialog as a sequence: [CLS],  $S_{1,1}$ ,  $S_{1,2}$ , ..., [SEP],  $U_{1,1}$ ,  $U_{1,2}$ , ..., [SEP],  $S_{2,1}$ ,  $S_{2,2}$ , ..., [SEP] where  $S_{i,j}$  and  $U_{i,j}$  are the  $j^{th}$  token of the system and user utterance in the  $i^{th}$  turn. Following BERT, we also add a learned embedding to every token indicating whether it comes from user utterances or system utterances.
Model Comparisons and Ablations We compare CMADE to its several ablations (Table 4) and evaluate the performance on the testing set, which is annotated by experts. We also report the kappa
No.	Model	Test Acc.	Kappa
κ	SE
(1)	BERT-Classification	0.581	0.161	0.049
(2)	BERT-Regression	0.640	0.280	0.048
(3)	BERT-Pairwise	0.730	0.459	0.044
(4)	BERT-Pairwise+Dev	0.749	0.499	0.043
(5)	Stage 2	0.755	0.509	0.043
(6)	Stage 2 + 3	0.764	0.529	0.042
(7)	Stage 3	0.714	0.429	0.045
(8)	Stage 1	0.620	0.241	0.048
(9)	Stage 1 + 3	0.788	0.628	0.039
(10)	Stage 1 + 2	0.837	0.673	0.037
(11)	CMADE	0.892	0.787	0.031
Table 4: Test accuracy and kappa agreement comparison among variants of CMADE.
agreement (Cohen, 1968) (kappa  $\kappa$  and Standard Error  $SE$ ) between the predicted output and the expert annotations. (1) BERT-Classification and (2) BERT-Regression fine tune the pre-trained BERT to perform a 5-class classification and regression respectively directly using the noisy self-reported ratings. To test BERT-Classification on dialog pairs, we apply the DEX trick (Rothe et al., 2015) to get a floating-point number of predicted rating and thus get rid of the cases when the model predicts the dialog pairs as tie. (3) BERT-Pairwise shares the same model architecture with CMADE. It constructs dialog pairs for training by randomly sample dialog pairs  $x_{i}$  and  $x_{j}$  and derive  $z_{i,j}$  by comparing the corresponding self-reported user rating  $y_{i}$  and  $y_{j}$ . We discard the pairs with equal  $y_{i}$  and  $y_{j}$ . (4) BERT-Pairwise+Dev augments (3) by adding the 200 expert annotated dialog pairs in the development into the training data. We also compare the variants of CMADE which skips one or two of the three stages.
Results Our first takeaway is that vanilla classification or regression formulation might not be the best way to formulate the problem of learning a dialog evaluation model. As shown in Table 4, pairwise architecture (BERT-Pairwise, 0.73) is better than classification (BERT-Classification, 0.53) or regression (BERT-Regression, 0.64) in this problem. Similar to our observation, the research community in computer vision has long observed that both vanilla classification and regression formulation has drawbacks in age estimation (Rothe et al., 2015; Niu et al., 2016; Zhang et al., 2017).
Our second takeaway is that denoising algorithm that is more aggressive usually makes stronger assumptions on the quality of feature representa
Figure 2: Removing training data with low Shapley value improves the performance of the KNN regressor.
tions. Therefore, it helps to create a denoising pipeline that starts with better feature representation learning and less aggressive denoising algorithm to learn better feature representation before applying the more aggressive denoising algorithms. As shown in Table 4, our three-stage denoising pipeline CMADE (Acc. 0.892) significantly outperforms all baselines by a large margin. Although (8) Stage 1 does not directly provide high accuracy (Acc. 0.620), the feature representation it learned is extremely important. Without Stage 1, both (5) Stage 2 (Acc. 0.755) and (6) Stage  $2 +$  Stage 3 (Acc. 0.763) perform worse.
Since the KNN label smoothing is performed on the feature space, we expect the smoothing performs worse without self-supervised dialog feature representation learning in Stage 1. However, they still work better than baseline (1) (2) (3) which are models that do not account for the noise in data. This is because we use the pre-trained BERT to initialize our dialog encoder  $\phi$  and thus  $\phi$  is still able to provide some useful features for Stage 2. In addition, we observe that denoising with data Shapley in Stage 3 requires better dialog feature representation. (7) Stage 3 (Acc. 0.714) performs even worse than BERT-Pairwise (0.730) without good representations to perform the Shapley denoising algorithm. Skipping Stage 2 also hurts performance (Acc. 0.788). However, it does not mean that Shapley denoising in Stage 3 is not powerful. We observe a large performance gain in applying stage 3 after stage 1 and stage 2 (Acc. 0.837 v.s. 0.892). Finally, we note that adding the expert annotated development set directly into the training data is much less efficient compared to using the development set for data Shapley to denoise. BERT-Pairwise+Dev only reaches an accuracy of 0.749.
Additional analysis We also present an analysis to show how Shapley denoising works as shown in
Figure 2. We examine the Shapley value for each training datum in Stage 3. We first show an example dialog with a negative Shapley value in Table 5. According to the Shapley value, we remove data points one by one starting from the least valuable to the most valuable. Each time, after the point is removed, we create new KNN regressor models  $K = 1, 5, 25, 50, 100$  on the remaining dialogs and labels and evaluate them on the test set with expert annotations. We extract the features of the remaining dialogs using the dialog encoder  $\phi$  tuned by Stage 1 and Stage 2. As shown in Figure 2, removing training data with low Shapley values increases the performance to a certain point before convergence for  $K$  of all choices. We observe a similar trend when re-training a model on the remaining data. In contrast, removing data randomly decreases the performance on the test set. In addition, larger  $K$  has a better performance, which validates the denoising effect of KNN with large  $K$ .
The ultimate chatbot evaluation metric should be user-centric, as chatbots are there to provide human with an enjoyable experiences. Previously Likert-score based self-reported rating is the de-facto standard for current dialog evaluation. However, our analysis indicates that self-reported dialog ratings are skewed (J-shape), noisy and insensitive due to bias and variance among different users. We propose a three-stage denoising pipeline CMADE to reduce self-reported ratings and, at the same time, build an automatic comparison-based automatic dialog quality predictor. CMADE's results highly correlate with expert judgments on pair-wise dialog comparison ratings (89.2% agreement, 0.787
User: the whole harry potter series
Sys: whoops i think i misheard you. can you repeat the movie name please?
User: all the harry potter series
Sys: ah, i don't know any movies by that name. sorry if i sometimes confuse you saying something else with a movie name ...
Sys: oh, okay. anyways, i really enjoyed watching black panther and wonder woman recently. what's your favorite superhero movie?
User: super man
Sys: what was that? can you please say the movie name again?
User: stop i gotta go
Table 5: An example dialog with noisy self-reported rating (Score-5) identified by data shapley.
Kappa).
We would like to sincerely thank ACL 2020 Chairs and Reviewers for their review efforts and helpful feedback. We thank Yu Li for his insightful guidance and support in shaping this project. We thank Boxin Wang for helpful discussions on data Shapley. We would also like to extend our gratitude to Yanbang Wang, Youzhi Tian, Weiyan Shi and Michihiro Yasunaga for their valuable feedback and suggestions.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In IEEvaluation@ACL, pages 65-72. Association for Computational Linguistics.
Dan Bohus and Eric Horvitz. 2009. Learning to predict engagement with a spoken dialog system in openworld settings. In SIGDIAL Conference, pages 244-252. The Association for Computer Linguistics.
Christopher J. C. Burges, Robert Ragno, and Quoc Viet Le. 2006. Learning to rank with nonsmooth cost functions. In NIPS, pages 193-200. MIT Press.
Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. 2005. Learning to rank using gradient descent. In ICML, volume 119 of ACM International Conference Proceeding Series, pages 89-96. ACM.
Chun-Yen Chen, Dian Yu, Weiming Wen, Yi Mang Yang, Jiaping Zhang, Mingyang Zhou, Kevin Jesse, Austin Chau, Antara Bhowmick, Shreenath Iyer, et al. 2018. Gunrock: Building a human-like social bot by leveraging large scale real user data. *Alexa Prize Proceedings*.
Jan Chorowski and Navdeep Jaitly. 2017. Towards better decoding and language model integration in sequence to sequence models. In INTERSPEECH, pages 523-527. ISCA.
Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213.
Shay B Cohen, Eytan Ruppin, and Gideon Dror. 2005. Feature selection based on the shapley value. In *IJ-CAI*, volume 5, pages 665–670.
Peter J Danaher and Vanessa Haddrell. 1996. A comparison of question scales used for measuring customer satisfaction. International Journal of Service Industry Management, 7(4):4-26.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1), pages 4171–4186. Association for Computational Linguistics.
Pradeep Dubey. 1975. On the uniqueness of the shapley value. International Journal of Game Theory, 4(3):131-139.
Robert East, Kathy Hammond, and Malcolm Wright. 2007. The relative incidence of positive and negative word of mouth: A multi-category study. International journal of research in marketing, 24(2):175-184.
Maxine Eskenazi, Shikib Mehri, Evgeniia Razumovskaia, and Tiancheng Zhao. 2019. Beyond turing: Intelligent agents centered on the user. CoRR, abs/1901.06613.
Hao Fang, Hao Cheng, Maarten Sap, Elizabeth Clark, Ari Holtzman, Yejin Choi, Noah A. Smith, and Mari Ostendorf. 2018. Sounding board: A user-centric and content-driven social chatbot. In *NAACL-HLT (Demonstrations)*, pages 96-100. Association for Computational Linguistics.
Zunlei Feng, Weixin Liang, Daocheng Tao, Li Sun, Anxiang Zeng, and Mingli Song. 2019. Cu-net: Component unmixing network for textile fiber identification. Int. J. Comput. Vis., 127(10):1443-1454.
Gabriel Forgues, Joelle Pineau, Jean-Marie Larchevêque, and RÉal Tremblay. 2014. Bootstrapping dialog systems with word embeddings. In Nips, modern machine learning and natural language processing workshop, volume 2.
Johannes Furnkranz and Eyke Hüllermeier. 2010. *Preference learning*. Springer.
Amirata Ghorbani and James Y. Zou. 2019. Data shapley: Equitable valuation of data for machine learning. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 2242-2251. PMLR.
David Godes and José C Silva. 2012. Sequential and temporal dynamics of online opinion. Marketing Science, 31(3):448-473.
Faruk Gul. 1989. Bargaining foundations of shapley value. *Econometrica: Journal of the Econometric Society*, pages 81-95.
Fenfei Guo, Angeliki Metallinou, Chandra Khatri, Anirudh Raju, Anu Venkatesh, and Ashwin Ram. 2018. Topic-based evaluation for conversational bots. CoRR, abs/1801.03622.
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P. Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. CoRR, abs/1907.10568.
Nan Hu, Paul A Pavlou, and Jie Jennifer Zhang. 2017. On self-selection biases in online product reviews. MIS Quarterly, 41(2):449-471.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2019. Music transformer: Generating music with long-term structure. In ICLR (Poster). OpenReview.net.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gürel, Bo Li, Ce Zhang, Costas J. Spanos, and Dawn Song. 2019a. Efficient task-specific data valuation for nearest neighbor algorithms. PVLDB, 12(11):1610-1623.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J. Spanos. 2019b. Towards efficient data valuation based on the shapley value. In AISTATS, volume 89 of Proceedings of Machine Learning Research, pages 1167-1176. PMLR.
Ilya Kulikov, Alexander H. Miller, Kyunghyun Cho, and Jason Weston. 2018. Importance of a search strategy in neural dialogue modelling. CoRR, abs/1811.00907.
Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and William B. Dolan. 2016. A persona-based neural conversation model. In ACL (1). The Association for Computer Linguistics.
Margaret Li, Jason Weston, and Stephen Roller. 2019a. ACUTE-EVAL: improved dialogue evaluation with optimized questions and multi-turn comparisons. CoRR, abs/1909.03087.
Yu Li, Kun Qian, Weiyan Shi, and Zhou Yu. 2019b. End-to-end trainable non-collaborative dialog system. arXiv preprint arXiv:1911.10742.
Weixin Liang, Kai Bu, Ke Li, Jinhong Li, and Arya Tavakoli. 2018. Memcloak: Practical access obfuscation for untrusted memory. In ACSAC, pages 187-197. ACM.
Weixin Liang, Youzhi Tian, Chengcai Chen, and Zhou Yu. 2019. MOSS: end-to-end dialog system framework with modular supervision. CoRR, abs/1909.05528.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In EMNLP, pages 2122-2132. The Association for Computational Linguistics.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. In ACL (1), pages 1116-1126. Association for Computational Linguistics.
Vikram Sharma Mailthody, Zaid Qureshi, Weixin Liang, Ziyan Feng, Simon Garcia De Gonzalo, Youjie Li, Hubertus Franke, Jinjun Xiong, Jian Huang, and Wen-Mei Hwu. 2019. Deepstore: In-storage acceleration for intelligent queries. In MICRO, pages 224-238. ACM.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. 2019. When does label smoothing help? CoRR, abs/1906.02629.
Allen Nie, Arturo L. Pineda, Matt W. Wright Hannah Wand, Bryan Wulf, Helio A. Costa, Ronak Y. Patel, Carlos D. Bustamante, and James Zou. 2019. Litgen: Genetic literature recommendation guided by human explanations. CoRR, abs/1909.10699.
Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. 2016. Ordinal regression with multiple output CNN for age estimation. In CVPR, pages 4920-4928. IEEE Computer Society.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In EMNLP, pages 2241-2252. Association for Computational Linguistics.
Ioannis Papaioannou, Amanda Cercas Curry, Jose L Part, Igor Shalyminov, Xinnuo Xu, Yanchao Yu, Ondrej Dusek, Verena Rieser, and Oliver Lemon. 2017. Alana: Social dialogue using an ensemble model and a ranker trained on user feedback. *Alexa Prize Proceedings*.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318. ACL.
Verónica Pérez-Rosas, Xinyi Wu, Kenneth Resnicow, and Rada Mihalcea. 2019. What makes a good counselor? learning to distinguish between high-quality and low-quality counseling conversations. In ACL (1), pages 926-935. Association for Computational Linguistics.
Robert A Peterson and William R Wilson. 1992. Measuring customer satisfaction: fact and artifact. Journal of the academy of marketing science, 20(1):61.
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Yi Pan, Han Song, Sk Jayadevan, Gene Hwang, and Art Pettigree. 2018a. Conversational AI: the science behind the alexa prize. CoRR, abs/1801.03604.