Venue: NAACL 2021

# Measuring the 'I don't know' Problem

through the Lens of Gricean Quantity

Huda Khayrallah

Johns Hopkins University

huda@jhu.edu Joao Sedoc

New York University

jsedoc@stern.nyu.edu

###### Abstract

We consider the intrinsic evaluation of neural generative dialog models through the lens of Grice's Maxims of Conversation (1975). Based on the maxim of Quantity (be informative), we propose Relative Utterance Quantity (RUQ) to diagnose the 'I don't know' problem, in which a dialog system produces generic responses. The linguistically motivated RUQ diagnostic compares the model score of a generic response to that of the reference response. We find that for reasonable baseline models, 'I don't know' is preferred over the reference the majority of the time, but this can be reduced to less than 5% with hyperparameter tuning. RUQ allows for the direct analysis of the 'I don't know' problem, which has been _addressed_ but not _analyzed_ by prior work.

## 1 Introduction

Neural generative dialog models have a tendency to produce generic, safe responses, such as 'I don't know' [23, 4]. The repetition of such phrases is annoying to users, and contributes nothing to the conversation.

Evaluating chatbots is an active area of research, partly due to their open-ended nature [6, 8, 10, 11, 12]. To the best of our knowledge, no prior work focuses on _analyzing_ systems for generic, safe responses, such as 'I don't know.' While prior work [11, 12, 13, 14] the 'I don't know' problem, the lack of _analysis_ leaves it unclear if a method improves models by mitigating _this_ problem, or another.

One linguistic framework for analyzing conversations is Grice's Cooperative Principle (1975), which consists of Maxims of Conversation that function as guidelines for effective communication. Grice considered conversations between humans, but there has also been some exploration in NLP [1, 1, 13, 14].

We discuss each of the categories of maxims and the ways a chatbot might violate them in Table 1.

We propose a novel automatic diagnostic inspired by the Gricean Quantity maximum. Relative Utterance Quantity checks if the model favors a generic response (such as 'I don't know.') over the reference it was trained on for each prompt. We apply our diagnostic to a method designed to address this problem [13], and find that method does mitigate it, though not by as much as a hyperparameter search.

\begin{table}
\begin{tabular}{l|p{142.3pt}|p{142.3pt}|p{142.3pt}} \hline \hline Maxim & Definition & Violated by... & Prompt: What color is grass? \\ \hline Quantity & Be informative. & not answering a question (fully), or giving too much information. & I don’t know. \\ \hline Quality & Be truthful. & lying, or saying something without evidence. & Grass is purple. \\ \hline Relation & Be relevant. & off-topic responses. & I like pizza. \\ \hline Manner & Be clear, brief, and orderly. & disfluent responses & is green grass usually. \\ \hline \end{tabular}
\end{table}
Table 1: Gricean maxims, with examples of how they can be violated for the prompt ‘What color is grass?

Relative Utterance Quantity (RUQ)

If a system responds 'I don't know.' when it could have given a better or more informative answer, this is by definition a violation of Quantity. Based on this interpretation we propose a method for diagnosing the problem. We compare the model score of producing 'I don't know.' to the model score of producing the reference response. This can be done on the training data, or the test data. Particularly on the training data, we should expect the model to 'know' the data it was trained on and therefore score it higher than 'I don't know.'

We propose two diagnostic measures to compute the Relative Utterance Quantity of a model: (1) We plot the average model score for each token across sentences. We compare the original reference, beam search output, and two 'I don't know' (IDK) variants: 'I don't know.' and 'I don't know what to do.' allowing for the visualization of the relative gap in scores at different points in the sentence. (2) We compute the (length normalized) model score for 'I don't know.' and the reference of each training prompt, and count how many times the reference is preferred. We denote the later as RUQ score. Both generalize to other generic responses, as might be appropriate for other corpora or other languages.

If there are multiple references we would recommend comparing the lowest likelihood reference for RUQ score, since all valid references should be better than I don't know.

We note that RUQ captures some types of quantity violations, but not all violations of this maxim.

## 3 Data

Following Khayrallah and Sedoc (2020), we train and evaluate on DailyDialog Li et al. (2017),1 which consists of \(\sim\!80{,}000\) turns of English-learners practicing 'daily dialogues' in various contexts, e.g., chatting about vacation or food.

Footnote 1: As released by ParlAI Miller et al. (2017). The ParlAI release of DailyDialog is tokenized and lowercased. Following Khayrallah and Sedoc (2020) we detokenize and recase the DailyDialog data for training.

We also use Entropy-Based Data Filtering Csaky et al. (2019), which filters out high entropy utterances2 with the goal of removing generic ones. We use the recommended filtering threshold of 1.0 and 'IDENTITY' clustering. We filter based on their 'source', 'target', and 'both' settings. We consider 'target' as the baseline, as they find it works best. We denote models trained on DailyDialog as dd and models trained on Csaky et al.'s entropy filtered version as ef.

Footnote 2: Prompts that solicit many different responses and responses that can apply to many different prompts.

## 4 Evaluation Metrics

### Standard Automatic Metrics

We use the single-reference and multi-reference3 automatic evaluation framework for DailyDialog released by Gupta et al. (2019),4 which is computed using nlg-eval Sharma et al. (2017).5 We primarily consider multi-reference METEOR Lavie and Agarwal (2007); see Appendix A.7 for all metrics.6

Footnote 3: For RUQ, we only use the original single-reference.

Footnote 4: github.com/prakhargupta/multifreqval

Footnote 5: github.com/Maluuba/nlg-eval

### Human Evaluation

For human evaluation of the different systems we use crowdworkers on Amazon Mechanical Turk to judge the fluency, coherence, and interestingness of utterances on a 1-5 Likert scale (see Appendix A.4 for full details) for 100 randomly sampled evaluation set prompts. Four annotators judge the responses from all systems for each prompt in a single turn context. We remove any annotators with a linear Cohen's Kappa \(<\) 0.1 from the results.

## 5 Models

Following Khayrallah and Sedoc (2020), we train Transformer Vaswani et al. (2017) chatbots in fairseq using parameters from the Flores benchmark for low-resource MT Guzman et al. (2019): \(5\)-layer encoder and decoder, \(512\) dimensional embeddings, and \(2\) encoder and decoder attention heads. The default regularization parameters are \(0.2\) label smoothing Szegedy et al. (2016), \(0.4\) dropout, and 0.2 attention & ReLU dropout.

### Hyperparameter Sweep

Some kinds of regularization (e.g., label smoothing and subword vocabularies) are not universally used 

in dialog.8 Since we are concerned with the model over-fitting on IDK, we perform a hyperparameter sweep of regularization parameters, including SentencePiece (Kudo and Richardson, 2018) vocabulary size, learning rate, dropout, attention & relu dropout, and label smoothing.9

Footnote 7: For example popular toolkits for dialog (e.g., Hugging Face (Wolf et al., 2020) and ParlAI (Miller et al., 2017)) do not implement label smoothing.

Footnote 8: See Appendix A.1 for more hyperparameter details.

We denote models trained with the flores hyperparameters as base, and the best model from the hyperparameter searches for each data type (as selected by multiple-reference METEOR) as best.

We report the multi-reference METEOR scores for the base and best systems in Table 2.10 For the DailyDialog data we find that hyperparameter tuning can improve multiple-reference METEOR from 12.7 (dd-base) to 17.8 (dd-best).

Footnote 10: We report hyperparameters of these models and their performance on the full set of automatic metrics in § A.7.

We perform the same hyperparameter sweep after performing entropy filtering (Csaky et al., 2019) on the data, but we find that the best model is still dd-best. Without hyperparameter tuning, entropy filtering improves performance by \({\sim}\)0.5 on multi-reference METEOR, but the improvement by hyperparameter sweeping is much larger (5.1 points).11

Footnote 11: We note that Csáky et al. (2019)—who proposed entropy filtering and an observed a 1 BLEU point improvement from using it (we observed a 0.3 improvement in single reference BLEU)—did not use any subwords units; they used a total vocab size of 16k. Our 10 best systems all had Sentenepiecv vocab sizes of 2k, 4k, or sk, so perhaps this difference may explain the discrepancy between their results and our replication. We note that for the 3 metrics which we believe our evaluations are comparable—single reference Embedding Average Cosine Similarity, and single reference Vector Extrema Cosine Similarity—our baseline outperforms their results. The BLEU scores are not directly comparable because they report sentence BLEU, while we report corpus BLEU following Gupta et al. (2019).

We did a very thorough sweep (including values we expected to perform poorly), which led to some general takeaways:12 Using a subword vocabulary (of 4-8k) is helpful. (2) Label smoothing interacts with subword vocabulary size, but is also helpful.

Footnote 12: See chateval_org/RUQ for automatic metrics on the full hyper parameter sweep.

## 6 Relative Utterance Quantity

### RUQ Plots

We show plots for the four models in Figure 1. We plot the token normalized model score for reference and 'I don't know.' For additional comparison, we also plot the model scores for the

Figure 1: RUQ plots on the train (top) and test (bottom) data. We plot the token normalized model score for the reference (✗), the beam-search output (�⃝**), ‘I don’t know.’ (⃝**), and ‘I don’t know what to do.’ (�). Points are per (subword) token, and averaged over all prompts.

 

beam-search output and 'I don't know what to do.' Overall, we observe that for the base models the IDKs are higher probability than the reference, even on the training data. This is problematic, because the model is ranking a response that is not providing enough Quantity of information higher than the reference despite the fact that it should _'know'_ the training data. The relative difference in probabilities is much better in ddbest than ddbase, particularly on the training set. Simply entropy filtering the data alone does not fix the problem.

### RUQ scores

We summarize Quantity in a single statistic by counting how many times the reference has a higher probability than 'I don't know.' on the training data.

Entropy filtering improves how often the reference is preferred to 'I don't know.', but not by as much as the hyperparameter sweep does, see Table 3 for the RUQ scores on the training data.13 For both ddbase and efbase, IDK is preferred over the reference response the model was trained on over half of the time (71.5% for dd, 62.1% for ef).

Footnote 13: RUQ scores on the on the test data are reported in § A.7. The overall trend is same, but the absolute values lower.

### Human Evaluation

Table 4 shows human judgments of fluency, coherence, and interestingness.14 The models trained on DailyDialog have higher fluency and coherence, while the models trained on the filtered data have higher interestingness. For both kinds of data, the hyperparameter tuning (as selected by METEOR) improved interestingness. Fluency did not change. Coherence was reduced for the filtered models and improved for the base model. Improved RUQ may be reflected in either interestingness or coherence, but other factors can influence those judgments. Therefore, measuring RUQ directly is important to measuring progress on the IDK problem.

Footnote 14: § A.5 discusses head to head judgments. Models trained on the DailyDialog data are preferred over the filtered models, but there is no clear preference between base and best models.

## 7 Discussion

The relative RUQ rankings of the four systems we consider in this work are the same as the relative rankings by multi-reference METEOR, and ddbest (the single best model according to mulit-reference METEOR) is also the one with the highest RUQ score. Among all models in the hyperparameter sweep, RUQ is correlated with METEOR with Spearman's \(\rho\) of 0.9 but this drops to 0.6 when considering only the top 20 systems, demonstrating that RUQ and METEOR do not capture the same phenomenon. We note that RUQ on the training data does not require a particular (multi-reference) test set like most automatic evaluation metrics. RUQ simply diagnoses how well the model learned the training data compared to a generic response.

The model's relative preference of IDK over the (presumably) better reference response is not only a Quantity violation, but is also indicative of a fundamental problem with the models themselves, and should be fixed before decoding time (either by correcting the data, or by correcting the model).

Csaky et al. (2019) argue that the IDK problem is due to the one-to-many/many-to-one nature of dialog training data--if a single response applies to many different responses, it will become the canonical response. Therefore their entropy filtering method removes one-to-many/many-to-one pairs, by removing high entropy responses. While this data filtering reduces the problem, we found that the baseline model trained on the

\begin{table}
\begin{tabular}{l|c c c} \hline \hline  & Fluency & Coherence & Interestingness \\ \hline Human & 4.9 & 4.6 & 4.0 \\ \hline ddbase & **4.8** & 3.5 & 2.6 \\ ddbest & **4.8** & **3.8** & 2.7 \\ \hline efbase & 4.4 & 3.3 & 2.8 \\ ef-best & 4.4 & 3.1 & **3.3** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Average human judgement ratings on 1-5 pointwise scale for DailyDialog (dd) and the entropy filtered (ef) data. The result of the hyperparameter sweep is denoted best.

\begin{table}
\begin{tabular}{l|c c} \hline \hline training data & base & best \\ \hline DailyDialog & 28.5\% & 95.3\% \\ Entropy-Filtered & 37.9\% & 89.2\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Training data RUQ scores. Entropy filtering improves how often the reference is preferred to ‘I don’t know.’, but by less than the hyperparameter sweeps (which are denoted best).

 

entropy filtered data (ef-base) still preferred IDK over the reference the majority of the time, suggesting opportunities for future research on the IDK problem.

## 8 Related Work

Gricean Maxims in NLPGricean maxims have previously been discussed in NLP. bernsen2019generating examine the relationship between a new set of maxims for human-bot dialogs and relate them to Gricean maxims. They point out that these do not entirely overlap; however, the maxim of Quantity is preserved since unambiguous contributing responses are required in conversations in general. harabagiu2016predicently attempt to explicitly create an evaluation methodology using sets of primitive rules and WordNet. Our approach is different as RUQ is a diagnostic metric.

jwalapuram:2017:EM propose a Gricean dialog evaluation where humans rate performance on a Likert scale for each category. qwaider2017exploiting consider the quantity, relation, and manner maxims for ranking community question answers. They use other NLP tools to evaluate if the response has key elements or named entities (quantity/relation), has high semantic similarity (relation), and includes/excludes positive/negative polarity terms (manner).

Chatbot evaluationAutomatic evaluations for dialog typically measure lexical or semantic similarity between a produced response and a reference, under the assumption that the reference is a good response and responses similar to it will be good as well. Since there are often multiple valid responses to a prompt, this can be extended to multiple references too. In contrast, in our work we compare a model's score of a reference to a model's score of a generic response for directed analysis.

HUSE hashimoto2019predicually uses the model score combined with human judgments to evaluate diversity and quality, classifying a response as human- or machine-generated. Our work does not require human judgments, and compares the model score of a generic response to the reference response.

Mehri2020predicually also use scoring from a model. Whereas that work is using an external model, we propose an intrinsic diagnostic for a particular phenomenon. Each serves a different purpose, and an advantage of our method is our analysis does not require an external model, which might not be available in all languages and for all types of text.

Mitigating the IDK ProblemA variety of approaches have been proposed to _mitigate_ the IDK problem. These include active post-processing methods such as MMI li2016predicually, as well as training data filtration csaky2019predicually, reinforcement learning li2016predicually and unlikelihood training welleck2020predicually. In our work, we propose an intrinsic model diagnostic to _analyze_ the problem.

MmiMaximum Mutual Information was proposed as a 'Diversity-Promoting Objective Function' for dialog lci2016predicually. MMI-bidi encourages the prompt to be predictable from the response, by using a reverse direction model. We argue this was not diversity broadly speaking, but actually tackling a relevancy problem, since it is scoring how predictable the prompt is from the response.

li2016predicually demonstrate MMI improves performance, though recent work found that it does not always khoys2020predicually.

Copying in Machine Translationott2018universal found that copying was overrepresented in the output of RNN NMT. Using an analysis that inspired RUG plots they compare the score of the beamsearch output to that of the copied source. They also consider the probability at each position in the output, and find the model is unlikely to start copying; however, after starting to copy continuing to copy has high probability. We find IDK has a relatively high score from the start, though for some models the gap widens towards the end of the sentence.

## 9 Conclusion

We reframe the IDK problem as a violation of the Gricean maxim of Quantity, and introduce a new measure--Relative Utterance Quantity (RUQ)--which allows researchers to diagnose if their model is violating this particular conversational principle, and analyze methods that aim to address it.

We aim to encourage further discussion and research drawing on linguistic principles about discourse and pragmatics for analysis of dialog models 



## Acknowledgments

We thank Patrick Xia, Nathani Weir, Rachel Rudinger, and Claire Daniele for their helpful comments and feedback on the paper. We additionally thank the reviewers for their insightful comments.

This work was supported in part by DARPA KAIROS (FA8750-19-2-0034). The views and conclusions contained in this work are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.

## References

* O. Bernsen, H. Dybkjaer, and L. Dybkjar (1996)Cooperativity in human-machine and human-human spoken dialogue. Discourse Processes21 (2), pp. 213-236. Cited by: SS1.
* R. Csaky, P. Purgai, and G. Recski (2019)Improving neural conversational models with entropy-based data filtering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 5650-5669. External Links: Link, Document Cited by: SS1.
* J. Deriu, D. Tuggener, P. von Daniken, J. Campos, A. Rodrigo, T. Belkacem, A. Soroa, E. Agirre, and M. Cieliebak (2020)Spot the bot: a robust and efficient framework for the evaluation of conversational dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 3971-3984. External Links: Link, Document Cited by: SS1.
* G. Forgues, J. Pineau, J. La Archeveque, and R. Tremblay (2014)Bootstrap dialog systems with word embeddings. In Modern Machine Learning and Natural Language Processing at NeurIPS, Cited by: SS1.
* 58. Note: Brill, Leiden, The Netherlands Cited by: SS1.
* P. Gupta, S. Mehri, T. Zhao, A. Pavel, M. Eskenazi, and J. Bigham (2019)Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, Stockholm, Sweden, pp. 379-391. External Links: ISBN 978-3-319-

Proceedings of the Second Workshop on Statistical Machine Translation, Prague, Czech Republic. External Links: Link Cited by: SS1.
* J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan (2016)A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California, pp. 110-119. External Links: Document Cited by: SS3.4.
* J. Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, and J. Gao (2016)Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, pp. 1192-1202. External Links: ISBN 978-3-540-3213-1, Link, Document Cited by: SS1.
* M. Li, J. Weston, and S. Roller (2019)Acute-eval: improved dialogue evaluation with optimized questions and multi-turn comparisons. External Links: Link Cited by: SS1.
* Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu (2017)DailyDialog: a manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Taipei, Taiwan, pp. 986-995. External Links: Link Cited by: SS1.
* C. Lin (2004)ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, pp. 74-81. External Links: Link Cited by: SS1.
* Sh. Mehri and M. Eskenazi (2020)USR: an unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 681-707. External Links: Link Cited by: SS1.
* S. Mehri and M. Eskenazi (2020)USR: an unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 681-707. External Links: Link Cited by: SS1.
* A. Miller, W. Feng, D. Batra, A. Bordes, A. Fisch, J. Lu, D. Parikh, and J. Weston (2017)ParlAI: a dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Copenhagen, Denmark, pp. 79-84. External Links: Link Cited by: SS1.
* M. Ott, M. Auli, D. Grangier, and M. Ranzato (2018)Analyzing uncertainty in neural machine translation.. In proceedings of the 35th International Conference on Machine Learning, Vol. 80, Proceedings of Machine Learning Research, Stockholmsmassan, Stockholm Sweden, pp. 3956-3965. External Links: Link Cited by: SS1.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 311-318. External Links: Link Cited by: SS1.
* M. R. Qwaider, A. Alhakim Freihat, and F. Giunchiglia (2017)TrentoTeam at SemEval-2017 task 3: an application of price maxims in ranking community question answers. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, pp. 271-274. External Links: Link Cited by: SS1.
* V. Rus and M. Lintean (2012)A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, Montreal, Canada, pp. 157-162. External Links: Link Cited by: SS1.
* J. Sedoc, D. Ippolito, A. Kirubarajan, J. Thirani, L. Ungar, and C. Callison-Burch (2019)CatEval: a tool for chatbot evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), Minneapolis, Minnesota, pp. 60-65. External Links: Link Cited by: SS1.
* I. V. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau (2016)Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, New York, NY, pp. 3776-3783. External Links: Link Cited by: SS1.
* S. Sharma, L. El Asri, H. Schulz, and J. Zumer (2017)Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRRabs/1706.09799. External Links: Link Cited by: SS1.
* C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna (2016)Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826. External Links: Link Cited by: SS1.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all 

you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.

 

Appendix

### Hyperparameter Search

We sweep SentencePiece (Kudo and Richardson, 2018) vocabulary size (1k,4k,8k,16k), learning rate (1e-2, 1e-3, 1e-4), dropout (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6), attention & ReLU dropout (0.0, 0.1, 0.2, 0.3, 0.4), and label smoothing (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8).

### Standard Automatic Metrics

In SS A.7 we report the full automatic evaluation results of the 14 metrics across both the single reference and multi-reference evaluation from the the multi-reference automatic evaluation framework for DailyDialog released by Gupta et al. (2019),15 which is computed using nlg-eval16Sharma et al. (2017). This includes word-overlap metrics: BLEU Papineni et al. (2002), METEOR Lavie and Agarwal (2007), and ROUGE-L Lin (2004) as well as embedding based metrics: SkipThought Kiros et al. (2015), embedding average Forgues et al. (2014), vector extrema, and Greedy Matching Rus and Lintean (2012). For reading ease, we report metrics scaled between 0 and 100 rather than 0 and 1. See chateval.org/RUQ for automatic metrics on the full hyper parameter sweep.

Footnote 15: github.com/prakharguptaz/multifreqval

Footnote 16: github.com/Maluuba/nlg-eval

### Lexical Diversity

The Gricean maxims focus on ensuring cooperation between speakers, but there is more to a conversation than cooperation--especially in an open ended conversation that might be had with a chatbot. This is where additional desiderata may come in to play, such as interestingness. One (indirect) automatic way of measuring interestingness is lexical diversity Halliday (1989); Laufer and Nation (1995), by computing the n-gram type/token ratio Li et al. (2016a). We use the same spaCy17 tokenization used in the automatic evaluation scripts (SS A.2).18

Footnote 17: spacy.io

Footnote 18: github.com/Maluuba/nlg-eval

### Pointwise Human Evaluation

We presented Amazon Mechanical Turk workers the task with 4 prompts and all system responses along with a human reference. The annotators had a maximum time allotted of 20 minutes. Our criteria for inclusion were over 500 approved HITs, an approval rate over 98%, and location set to US. Each HIT was paid $0.15 with an overlap of 4 annotators per HIT.19 A screenshot of the HIT is in Figure 2.

Footnote 19: We aimed to compensate the crowdworkers fairly (§ 8 per hour) and did this by annotating a set of data ourselves to estimate the timing of the task

### Head to Head Human Evaluation

In addition to the point-wise evaluation, we also test head-to-head pairwise performance on the evaluation set of 480 unique prompt/response pairs, as shown in SS A.5. Models trained on the DailyDialog data outperform the filtered models, but there is no clear preference between base and best models.

### Dialog Models

We train Transformer conditional language models in fairseq using parameters from the flores20 benchmark for low-resource machine translation Guzman et al. (2019).

Footnote 20: [https://github.com/facebookresearch/floers/tree/5696dd4ef07e29977d5690d2539513afe6f2fe7f0](https://github.com/facebookresearch/floers/tree/5696dd4ef07e29977d5690d2539513afe6f2fe7f0)

We use a \(5\)-layer encoder and decoder, \(512\) dimensional embeddings, and \(2\) encoder and decoder attention heads. We regularize with \(0.2\) label smoothing, and \(0.4\) dropout. We optimize using Adam with a learning rate of \(10^{-3}\). We train 100 epochs, and select the best checkpoint based on validation set perplexity. We generate with a beam size of \(10\), and no length penalty.

Figure 3 shows the train command.

We train and evaluate on the DailyDialog corpus Li et al. (2017), as released by ParlAI Miller et al. (2017).21

Footnote 21: [https://github.com/facebockresearch/fluerali/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearchresearch/treebookresearch/treebookresearchresearch/treebookresearchresearch/treebookresearchresearch/treebookresearchresearch/treebookresearchresearch/treebookresearchresearchresearch/treebook](https://github.com/facebockresearch/fluerali/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/tree/facebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/treebookresearch/partial/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearch/treebookresearchresearch/treebookresearch/treebookresearchresearch/treebookresearchresearch/treebookresearchresearch/treebookresearchresearch/treebookresearchresearch/treebookresearchresearchresearch/treebook) 

Please Note
* You have to be an **English Native Speaker**.
* You have to complete the ratings for all responses. All fields are required.

### Informed Consent

This is a linguistic experiment performed at

If you have any question about this study, feel free to contact Participation in this research is voluntary. You have the right to withdraw from the experiment at any time. The collected data will be used for research purposes only. Personal data will be kept confidential and will not be shared with third parties.

## Instructions

In this task you will read a turn of a conversation. For each conversation you will see possible responses to the last turn generated by a computer program. The programs attempts to generate responses that are relevant, while also making an interesting contribution to the conversation.

For each prompt, you will read several system possible responses, and judge each response on its appropriateness on a 1 to 5 scale from not at all appropriate to externally appreciate. Appeepointness is defined as follows:

1. Fluency: Flows naturally and sounds like what an experienced speaker/writer of the language might say.
2. Coherence: Facts and topics are consistent, i.e. non-contradictory with previous parts of the conversation.
3. Interestingness: Contains rich, non-reptitious, and interesting information that adds to the conversation.

_Example:_ Below we show a prompt and several possible responses, along with suggested scores for each response.

**Prompt: What do you do for work?**

**Responses**: **Suspended scores**

I dateco. Flussy: 1. Coherence: 5. Interestingness: 5. This does not sound like a native speaker's response, but there are no contradictions and it answers the question. I like purchasics. Flussy: 5. Coherence: 1. Interestingness: 3. This is grammatical but not coherent is it does not answer the question and only some information was added to the conversation. I wock. Flussy: 4. Coherence: 4. Interintegrees: 1. This is grammated and mostly coherent, but no information is added to the conversation..

I wock as a teacher. Flussy: 5. Coherence: 5. Antiresponse: 5. An excellent on-topic response that does not contradict any previous statements.

Note ignore things like "i" or "this should be read as "Tt" and not decided points for grammar. There are ATTENTION CHECKS please make sure you select the number.

(% for prompts in prompts >>) (% set pornology = long %)

1. **Prampet:** (% for run in prompts-prompt %)

{{turn }}

(% endlor %)

(% for model in prompts-model%) (% set malcolup + kog +)

**system ( (methoding Index):):[ ( moded. response )]**

Figure 2: Instructions for AMT task.

\begin{table}
\begin{tabular}{l l|c c c} \hline \hline M1 & M2 & M1 & M2 & tie \\ \hline ef-best & dd-best & 38.5\% & 42.4\% & 19.1\% \\ ef-base & ef-best & 34.4\% & 34.0\% & 31.7\% \\ ef-base & dd-base & 37.0\% & 41.6\% & 21.4\% \\ dd-base & dd-best & 36.3\% & 36.5\% & 27.2\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Head to head comparison between various systems. Models trained on the DailyDialog data outperform the filtered models, but there is no clear preference between base and best models.

 

\ }

python train.py {  $DATADIR \  --source-lang src \  --target-lang tgt \  --seed 10 \  --save-dir $SAVEDIR \  --patience 50 --criterion label_smoothed_cross_entropy \  --label-smoothing 0.2 \  --share-all-embeddings \  --arch transformer --encoder-layers 5 --decoder-layers 5 \  --encoder-embed-dim 512 --decoder-embed-dim 512 \  --encoder-ffn-embed-dim 2048 --decoder-ffn-embed-dim 2048 \  --encoder-attention-heads 2 --decoder-attention-heads 2 \  --encoder-normalize-before --decoder-normalize-before \  --dropout 0.4 --attention-dropout 0.2 --relu-dropout 0.2 \  --weight-decay 0.0001 \  --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0 \  --lr-scheduler inverse_sqrt --warmup-updates 4000 --warmup-init-lr le-7 \  --lr le-3 --min-lr le-9 --no-epoch-checkpoints \  --max-tokens 4000 \  --max-epoch 100 --save-interval 10 --update-freq 4 \  --log-format json --log-interval 100 ``` 



\begin{table}
\begin{tabular}{c c|r r r|r} \hline \hline  & & \multicolumn{3}{c|}{Cosine Similarity} \\ Data & Params & SkipThought & Embed. Avg. & VectorExtrema & GreedyMatching \\ \hline dd & base & 65.3 & 86.3 & 50.6 & 71.3 \\ dd & best & **68.2** & **88.5** & **54.7** & **74.6** \\ ef & base & 64.9 & 86.9 & 50.2 & 71.3 \\ ef & best & 67.0 & 87.7 & 52.3 & 73.1 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Embedding based metrics on the single reference test set.

\begin{table}
\begin{tabular}{c l|r r} \hline \hline Data & Params & RUQ-train & RUQ-test \\ \hline dd & base & 28.5 & 12.2 \\ dd & best & **95.3** & **35.7** \\ ef & base & 37.9 & 15.5 \\ ef & best & 89.2 & 30.7 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Type/Token ratios.

\begin{table}
\begin{tabular}{l l|r r|r r|r r|r r|r r} \hline \hline  & & \multicolumn{3}{c|}{Average Max Sentence BLEU} & \multicolumn{3}{c|}{Corpus BLEU} & \multicolumn{3}{c|}{} & \multicolumn{3}{c}{} \\ Data & Params & BLEU1 & BLEU2 & BLEU3 & BLEU4 & BLEU1 & BLEU2 & BLEU3 & BLEU4 & METEOR & ROUGE \\ \hline dd & base & 15.3 & 7.6 & 5.6 & 4.5 & 12.9 & 6.3 & 4.1 & 3.0 & 6.7 & 20.6 \\ dd & best & **24.3** & **16.7** & **14.3** & **12.8** & **23.2** & **16.7** & **14.2** & **12.9** & **11.9** & **29.2** \\ ef & base & 15.9 & 7.4 & 5.2 & 4.1 & 15.8 & 7.5 & 4.7 & 3.3 & 7.2 & 20.4 \\ ef & best & 22.1 & 14.0 & 11.8 & 10.5 & 22.9 & 15.8 & 13.2 & 11.8 & 11.1 & 26.6 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Word-overlap based metrics on the single reference test set.