SlowGuess/NAACL_2021_OCR huggingface dataset:
We analyze our model, by performing experiments with some variations in making the negative samples to be used with the random negative sample: (1) drop-golden: Instead of following the steps of scoring, selecting, and replacing, we randomly drop some of the words in the golden response to create a negative sample, and use it with the random negative sample. (2) shuffle-golden: Instead of following the three steps, we randomly shuffle the words in the golden response to create a negative sample, and use it with the random negative sample. (3) score-w/o-history: We use the scoring function in Equation 1 without the first term, so that it only considers the probabilities within the sentence without the dialogue history. (4) select-random: Instead of using the scoring function proposed in Equation 1, we randomly select the words to be replaced. (5) replace-w-history: When replacing a word, we concatenate the dialogue history with the response so that the LM considers the dialogue history when replacing the masked words.
Table 2 shows the correlations between model predictions and human evaluations for the modified models above. Dropping or shuffling words in the golden response to make a negative sample shows similar or lower performance compared to using random responses (BERT-retrieval (random-1, random-2)). The correlation was lower when the dialogue history was not considered in the scoring process than when it was considered. We speculate that this is because it gives high scores not only to words important for the consistency of a conversation, but also to the words with low likelihoods in general. Randomly selecting the tokens shows lower correlation than using our proposed scoring function. Considering the dialogue history in the replacing process gives lower performance than when it is not considered. We speculate that providing the dialogue history makes predictions on the masked words that are more appropriate to the context, making the reconstructed response less
Figure 4: Some examples of cases in which our model predicted scores similar to human evaluations. GPT2 denotes the GPT2-coherence model. BERT-rtv. denotes the BERT-retrieval (random-1). All scores are normalized into the range of [0, 1].
appropriate as a negative sample.
Figure 4 shows some of the evaluation results of each model on the DailyDialog dataset. The responses in the first and second examples are appropriate to the given dialogue history as suggested by the high human score. BLEU-2 gives a score of 0
Figure 5: The POS tag distribution of the words in the original training corpus (left) and the selected words by our method (right).
because the response has no bi-grams shared with the golden response. RUBER and GPT2-coherence did not recognize the utterances as appropriate responses. BERT-retrieval (random-1) and BERT-retrieval (ours) gave relatively high scores to the responses, evaluating them as appropriate utterances. In the third example, the system response appears to be somewhat relevant to the given context because it includes some words ("chance", "future") relevant to the phrase "take part in the finals". A repetition of a phrase in this example ("to get a chance") is believed to have contributed to the low human evaluation score (0.12). The RUBER and BERT-retrieval (random) models appear to lack this intuition, and instead evaluate the response as appropriate, possibly because some words appear relevant. Our proposed model scored the response with a relatively low score of 0.15, which was close to the human score. In the fourth example, the response is not coherent, but because it begins with a sentence "Let me get a peek", it could have appeared as a coherent response to the previous dialogue about parking tickets. For this case, our proposed model and GPT2-coherence gave scores similar to human scores.
4.3 POS-tag distribution of selected words
We compute the Part-of-Speech (POS) tag distribution of selected words by our method and compare it with the original distribution of the DailyDialog corpus (Figure 5). As we can see, the VERB and NOUN tags are the most frequently selected (21.9% and 20.5%, respectively), and their ratio is increased than in the original corpus (18.3% and 16.7%, respectively). Meanwhile, the ratio of punctuation tag (. is highly decreased (from 21.3% to
Figure 6: Box plot of scores for each type of responses. The average scores of each type are 4.65, 2.51 and 1.19. The standard deviations for the scores of each type are 0.67, 1.27, and 0.41 (from left to right).
$12.1\%$ . We suspect that the likelihood of the punctuation tag is more affected by local information from a response rather than dialog history.
4.3.1 Are the generated samples actually inappropriate?
To see whether the negative samples generated by our method are actually inappropriate, we conducted a survey through Amazon Mechanical Turk (AMT). We selected 40 dialogue history examples and prepared three types of responses for each dialogue: 1) the golden response, 2) a negative sample generated by our method, and 3) a randomly selected negative sample from the corpus. For each dialog, 4 annotators were asked to score the quality of the three responses. Following Lowe et al. (2017), we asked the question "How appropriate is the response overall?" for each context-response pair, and the evaluation was conducted on a 5-point Likert scale. The Fleiss' kappa and Krippendorff's alpha for the annotations were 0.63 and 0.63, respectively.
Figure 6 shows the survey results. The mean scores of golden and random responses were 4.65 and 1.19, respectively. The mean score of our negative samples was 2.51. The standard deviations for the scores of each response type were 0.67, 1.27, and 0.41 for the golden response, our negative sample, and the random response, respectively. We see that these results do not guarantee that all the generated negative samples are inappropriate. What we can assume, however, is that our method of manipulating a golden response generates a negative sample that is more inappropriate than the golden response. Table 3 shows two examples of the three different types of responses for a given dialog history with their survey results.
4We use the NLTK POS tagger (https://www.nltk.org/book/ch05.html) with universal tagset.
Dialog History
A: Sir, would you like some dessert now?
B: Please show me the menu again.
A: Here you are sir. The chocolate cake is very delicious.
Golden: No, thanks. I don't like chocolate. I'd like strawberry pie. (5)
Ours: No, thanks. I don't have chocolate. I'll like some one. (1.5)
Random: I basically believe in science over theology.  
I mean, I (...) (1)
A: Could you tell me something about your family?
Golden: Ok. There are five people in my family, father, mother, elder brother, younger sister and I. (5)
Ours: Ok. There are five children in my family, father, mother, and brother, and father my me. (3.25)
Random: When do you want to move in? (1.25)
Table 3: Examples of three different types of responses for a given dialog history with their survey results. The highlighted words are newly generated by our method. The score of each response is underlined.
For a model learning to find the difference between appropriate and inappropriate responses, we speculate that the task of distinguishing the negative samples generated by our method from the golden responses would be more difficult than the task of distinguishing the randomly selected negative samples from the golden responses. We believe that this is because the generated negative samples can be inappropriate in more subtle ways than completely unrelated responses are. We suspect that learning with this more challenging setting have resulted in the performance gain that we discussed in Section 4.2.1. However, we believe that it will need a more in-depth semantic analysis on each of the cases, such as performing a more quantitative analysis (through an extensive human study, for instance) and further interpretation of the semantic relationships between the original golden responses and the modified negative samples according to the proposed method. We leave it as a future work.
In this paper, we proposed an automatic method for generating negative samples that can be used to train an unsupervised and unreferenced response evaluation model. We performed experiments to demonstrate that the proposed method can boost the unsupervised training of a response evaluation model. We analyzed the experiment results quantitatively, and examined some examples that show
the distinct characteristics of our proposed method.
This work was supported by Institute for Information and communications Technology Promotion (IITP) grant funded by the Korea government MSIT (No. 2018-0-00582, Prediction and augmentation of the credibility distribution via linguistic analysis and automated evidence document collection).
JinYeong Bak and Alice Oh. 2020. Speaker sensitive response evaluation model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6376-6385.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. 2019. Approximating interactive human evaluation with self-play for open-domain dialog systems. In Advances in Neural Information Processing Systems, pages 13658-13669.
Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 82-89.
Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1689-1701.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations.
Jiwei Li, Alexander H Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2016. Dialogue learning with human-in-the-loop. arXiv preprint arXiv:1611.09823.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, pages 986-995.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1116-1126.
Shikib Mehri and Maxine Eskenazi. 2020. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681-707.
Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of open-domain dialogue generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3619-3629.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog.
Ananya B Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. arXiv preprint arXiv:2009.11321.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892.
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 3776-3783.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 3295-3301.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104-3112.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Rubber: An unsupervised method for automatic evaluation of open-domain dialog systems. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 722-729.
Thomas Wolf, Victor Sanh, Julien Chaumont, and Clement Delangue. 2018. Transfertransfo: A transfer learning approach for neural network based conversational agents. In NeurIPS Workshop on Conversational AI.
Hanlu Wu, Tengfei Ma, Lingfei Wu, Tariro Manyumwa, and Shouling Ji. 2020. Unsupervised reference-free summary quality evaluation via contrastive learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3612-3621.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2204-2213.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with BERT*. In 8th International Conference on Learning Representations.
Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020. Designing precise and robust dialogue response evaluators. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 26-33.