SlowGuess/ACL_2020_OCR huggingface dataset:
The data for this experiment is the test data of the Twitter conversation corpus. We extract contexts, true and false responses from the data. The true response is the ground-truth response  $(GT)$ . And the false responses are four types that are described in section 3.3 (SC, SP, SS, Rand).
We compare SSREM with RUBER and RSREM that compute the similarity between a context and a response. We take the unreferenced metric score in RUBER. And we take the output of the  $f$  function in RSREM and SSREM. We use the same trained models in section 5.
Figure 3 shows the results. The x-axis is the models, and the y-axis is the output of the unreferenced metric or  $f$  function. All models perform well on distinguishing between  $GT$  utterances and Rand utterances. But RUBER performs poor on identifying  $SC$ ,  $SP$ , and  $SS$ . And RSREM cannot identify false responses from  $SC$ . Finally, SSREM outperforms the other two models for identifying all cases. It also maximizes the difference between  $GT$  and Rand than the other two models. It is another clue for showing the effectiveness of using speaker sensitive negative samples.
One interesting result is that the output scores decrease from  $GT$  to Rand. It is the same observation about the differences of speaker sensitive utterances in section 3.1. And it also means that identifying  $GT$  and  $SC$  is a harder problem than  $GT$  and Rand pair. It is another evidence for why
we use speaker sensitive negative samples, as we discussed in section 3.3.
$SC$  consists of negative samples that are most difficult for the model to distinguish, so it makes sense to consider only  $SC$  negative samples. But we include  $SP$  and  $SS$  for the following two reasons. First, there are only a limited number of  $SC$  utterances because they must all come from the same conversation, whereas we need a pretty large number of negative samples to effectively train the model (Mnih and Teh, 2012). Second, we also sample from  $SP$  and  $SS$  because they represent different degree of similarity to the context utterances.  $SC$  utterances are from the same conversation, leading to decreased model generalization.
7 Experiment 3 - Applying New Corpus
In this section, we investigate the applicability of SSREM to a new conversation corpus. SSREM takes the speaker sensitive samples from Twitter. But there are many open-domain conversation corpora such as Movie scripts (Danescu-Niculescu-Mizil and Lee, 2011). Tao et al. (2018) run a similar experiment with RUBER, but they use the similar domain of data, Chinese online forum (Training from Douban and testing on Baidu Tieba). We choose the Movie scripts corpus because it is written by the script writers whereas Twitter is personal causal online conversations. We present the performance of SSREM on the new corpus.
7.1 Experiment Setup
First, we annotate 1,200 responses to the movie dialog corpus. We use HRED (Sordoni et al., 2015) rather than VHUCM. The next procedure of annotation is the same when we create human scores for Twitter conversation responses in section 4. Two hundred forty-four workers tagged all responses. But, 94 workers failed the attention check question, so we collect the 150 workers' answers. The inter-annotator Fleiss' kappa (Fleiss, 1971) for Movie is  $\kappa = 0.63$ . It is still consistent with the results in (Lowe et al., 2017) and annotated Twitter conversations. The bottom row in Table 3 shows the basic statistics of the annotated responses.
We run two experiments, comparing with human scores and identifying true and false responses. We use the same models in section 5. We use the Twitter conversation corpus to train RUBER, RSREM, and SSREM. And we test the models on annotated movie dialogs. Unlike the Twitter conversation cor
6382
Metric	Spearman	Pearson
BLEU	0.036 (0.378)	0.063 (0.124)
ROUGE	0.041 (0.322)	0.054 (0.191)
EMB	0.022 (0.586)	0.010 (0.815)
RUBER	0.004 (0.920)	-0.009 (0.817)
RSREM	0.009 (0.817)	0.024 (0.550)
SSSREM	0.132 (< 0.001)	0.119 (< 0.005)
Table 5: Correlation between human and model scores with Movie corpus. We compute Spearman and Pearson correlation coefficient.  $p$ -values are shown in brackets. SSREM shows higher correlation with human judgement than the other models.
pus, the movie dialogs have a short length of conversations. So we choose  $SC$  and Ran only to run the second experiment.
7.2 Results and Discussion
In the experiment on comparing with human scores on the movie dialogs corpus, Table 5 shows the results. First, BLEU, ROUGE, and EMB are not correlated with human scores. RUBER shows worse performance than testing on the Twitter corpus. RSREM performs better than RUBER and other baselines, but it also shows worse performance than testing on the Twitter corpus. Finally, SSREM outperforms all other methods for two correlations with low  $p$ -values. It shows the effectiveness of using speaker sensitive negative samples for the new corpus. Figure 2 shows the similar results by plotting scatter plots.
In the experiment on identifying true and false responses with the movie dialogs corpus, Figure 5 shows the results of the identification task. RUBER performs poor on distinguishing between  $GT$  and Rand statistically significantly. RSREM performs better than RUBER. And SSREM outperforms the other two models for identifying all cases in the new corpus.
In this paper, we presented SSREM, an automatic evaluation model for conversational response generation. SSREM looks at the context of the conversation and the ground-truth response together. We proposed negative sampling with speaker sensitive samples to train SSREM. We showed that SSREM outperforms the other metrics including RSREM that uses random negative samples only. We also showed that SSREM is effective in evaluating a
movie conversation corpus even when it is trained with Twitter conversations.
There are several future directions to improve SSREM. First, we can make SSREM more robust on adversarial attacks. Sai et al. (2019) shows limitations of ADEM on adversarial attacks such as removing stopwords and replacing words with synonyms. We investigated another type of the adversarial attack named copy mechanism that copies one of the utterances in the context as the generated response. All existing automatic evaluation methods including RUBER that compare the context and the response can be cheated by the copy mechanism. SSREM is also susceptible. However, SSREM is fooled less than other existing models because SSREM learns with negative samples from the set of utterances in the same conversation. SSREM learns to differentiate among utterances in the same context. We show this empirically with an experiment to identify true and false responses (Sec 6.2). When we look at the mean score for the context utterances that shows this copy mechanism compared to the mean score of the ground-truth response (GT), the mean score of context utterances is 0.07 higher by RUBER, but only 0.01 higher by SSREM. SSREM does not give lower scores for the context utterances than GT, but it is not as bad as RUBER. We will make SSREM more robust on the attacks.
Second, we can improve SSREM for a higher correlation with human judgement. We chose to approach SSREM with a classification loss because it is simple and widely used to estimate the models using negative sampling. Although the classification loss is simple, SSREM outperforms all existing automatic evaluation models. However, as Table 2 and Figure 3 are shown, each negative samples has different correlation with the context. We will use ranking loss (Wang et al., 2014; Schroff et al., 2015) to learn the difference among samples. Recently, Zhang et al. (2020) uses BERT (Devlin et al., 2019) to evaluate generated candidate sentences by comparing reference sentence. We used word embeddings to represent an utterance to the vector for the simplicity, but contextual embeddings are much better since it generates more context-related representation than word embeddings. We will use the contextual embedding to represent utterances.
Third, we can extend using SSREM to various conversation corpora such as task-oriented dialogues. We trained and tested SSREM on open-
(a) BLEU,coeff:0.007
(d) RUBER, coeff: -0.001
(b) ROUGE, coeff: 0.005
(e) RSREM, coeff: 0.008
(c) EMB, coeff: 0.002
(f) SSREM, coeff: 0.047
Figure 4: Scatter plot showing model against human scores with Movie corpus. We add Gaussian noise drawn from  $N(0,0.3)$  to the human scores to better visualize the density of points which is similar to (Lowe et al., 2017).
Figure 5: Difference of scores on various responses in Movie corpus. The range of the vertical error bar is a  $95\%$  confidence interval of the values among the responses. SSREM outperforms the other models for identifying true and false responses.
domain conversation corpora. However, contextual coherence between the input context and the generated text is important in multi-turn conversations. We will apply SSREM to various conversation tasks for evaluating the generated text automatically. We will explore these directions in our future work.
We would like to thank Jeongmin Byun<sup>3</sup> for building the annotation webpage, and the anonymous reviewers for helpful questions and comments. This work was supported by Institute for Information &
communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2017-0-01779, A machine learning and statistical inference framework for explainable artificial intelligence).
Selim Aksoy and Robert M Haralick. 2001. Feature normalization and likelihood-based similarity measures for image retrieval. Pattern recognition letters.
JinYeong Bak and Alice Oh. 2019. Variational hierarchical user-based conversation model. In Proceedings of the EMNLP-IJCNLP.
Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In Proceedings of the ICLR.
Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level bleu. In Proceedings of the Ninth Workshop on Statistical Machine Translation.
Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the ACL.
Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics.
<sup>3</sup>https://jmbyun.github.io
6384
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin.
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of the SIGdial.
Michael Gutmann and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the AISTATS.
Helen Hastie. 2012. Metrics and evaluation of spoken dialogue systems. In Data-Driven Methods for Adaptive Spoken Dialogue Systems. Springer.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the EMNLP.
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In Proceedings of the ICLR.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceedings of the ACL.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the NIPS.
Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the ICML.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the EMNLP.
Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In Proceedings of the ACL.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the ACL.
Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018. A hierarchical latent structure for variational conversation modeling. In Proceedings of the NAACL.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the EMNLP.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the NAACL.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the EMNLP-IJCNLP.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval.
Ananya B. Sai, Mithun Das Gupta, Mitesh M. Khapra, and Mukundhan Srinivasan. 2019. Re-evaluating adem: A deeper look at scoring dialogue responses.
F. Schroff, D. Kalenichenko, and J. Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the CVPR.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the AAAI.
Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and JianYun Nie. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the CIKM.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Rubber: An unsupervised method for automatic evaluation of open-domain dialog systems. In Proceedings of the AAAI.
Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2010. What is left to be understood in atis? In Proceedings of the IEEE Spoken Language Technology Workshop.
J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu. 2014. Learning fine-grained image similarity with deep ranking. In Proceedings of the CVPR.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of the ICLR.
6385