SlowGuess/NAACL_2022_OCR huggingface dataset:
We have verified that NLI Diversity is both able to capture semantic diversity and aligns with human judgements. We can additionally use NLI Diversity to define a straightforward desired diversity threshold,  $div_{thresh}$  for a set of model-generated responses,  $u_1, \dots, u_n$ . For example, we might intend there to be 10 Contradictions within the set. We propose a generation procedure, Diversity Threshold Generation, designed to iteratively increase the diversity of a set of responses for a conversation.
For a conversation, Diversity Threshold Generation begins by sampling  $n$  responses. We score the diversity of these responses using a diversity metric,  $div\_metric(u_1, \dots, u_n)$ . If the diversity score falls above  $div_{thresh}$ , the process is finished.
If, however, the score falls below  $div_{thresh}$ , we identify the model response which contributes least to the diversity score by calculating  $div\_metric(u_1, \dots, u_{n-1})$  for each sub-group of model responses of size  $n - 1$ . We discard the model response not present in the highest-scoring subgroup and resample a new response. We re-calculate  $div\_metric(u_1, \dots, u_n)$  and if  $div\_metric(u_1, \dots, u_n) > div_{thresh}$ , the process finishes. We continue resampling until the maximum cutoff of  $S$  is reached.
6 Evaluation of Diversity Threshold Generation Method
6.1 Models and Datasets
We experiment with two neural dialogue models, DialogGPT (700M parameters) (Zhang et al., 2020b) $^{4}$  and BlenderBot 1.0 (300M parameters) (Roller et al., 2021) $^{5}$ . We use the default Transformers implementation for each model (Wolf et al., 2020) and do not fine-tune them. Runtime was between 3 and 36 hours on one Titan-X GPU.
All experiments involve the dialogue model  $M$  generating 5 responses for each conversation. The maximum number of samples,  $S$ , is set to 20. All experiments are averaged over 10 trials for stability.
We evaluate each model on the development set of two public English conversational datasets: DailyDialog++ (1,028 conversations) (Sai et al., 2020;
Li et al., 2017) and EmpatheticDialogues (2,763 conversations) (Rashkin et al., 2019). DailyDialog++ includes 5 human-written responses per conversation, allowing for multi-reference comparison. We split each EmpatheticDialogues conversation at a random turn (consistent for all experiments) for generation. Since BlenderBot supports up to 128 positional embeddings, we pass in the last 128 tokens of the conversation for this condition.
We evaluate three diversity metrics: two semantic diversity metrics, Baseline NLI Diversity (Section 3) and Sent-BERT (Reimers and Gurevych, 2019; Tevet and Berant, 2021), and one lexical diversity metric, distinct-n (Li et al., 2016; Tevet and Berant, 2021). For Sent-BERT, we compute the average negative cosine similarity between BERT sentence embeddings for each pair of responses. Like Tevet and Berant (2021), for distinct-n, we compute the average distinct n-grams from  $n \in \{1, 2, 3, 4, 5\}$ .
Because Baseline NLI Diversity is more human-interpretable than Confidence NLI Diversity, we use this version for experimentation. For all NLI Diversity experiments,  $div_{thresh}$  is achieved when # Contradictions is greater than 10 out of a total of 20 pair-wise comparisons. For both Sent-BERT and distinct-n, however, we do not have a human-specifiable threshold. We use empirical thresholds measured from the sets of 5 human responses for each conversation in DailyDialog++. We choose the 90th percentile for  $div_{thresh}$  (0.98 and -0.179 for distinct-n and Sent-BERT, respectively).
We decode using nucleus sampling  $(p = 0.9)$ , as it has been shown to increase response diversity (Holtzman et al., 2020). However our method could be applied with other decoding procedures.
In order to robustly evaluate Diversity Threshold Generation, we measure both (i) whether Diversity Threshold Generation is able to generate more diverse sets of responses than was originally sampled and (ii) whether the increased diversity comes at the expense of decreased relevancy of the responses.
6.3 Diversity Results
We aim to measure whether the diversity of the 5 responses from  $M$  increases using Diversity Threshold Generation, compared to the initial 5 sampled responses. Diversity of the starting and ending sets of utterances is measured by Baseline NLI Diversity, distinct-n, or Sent-BERT. We also report the
4https://huggingface.co/transformers/ model_doc/dialogpt.html 5https://huggingface.co/transformers/ model_doc/blenderbot.html
Met-ric	Mo-del	Data-set	StartingDiv.	End-ingDiv.	Num.Sam-pled
Baseline NLI	DG	Daily	4.11	10.24	6.3
Emp	3.68	10.11	7.1
BB	Daily	-5.55	2.51	14.4
Emp	-8.90	-1.72	16.5
Distinct-n	DG	Daily	0.95	0.98	5.4
Emp	0.43	0.52	20.0
BB	Daily	0.61	0.80	20.0
Emp	0.52	0.71	20.0
Sent-BERT	DG	Daily	-0.26	-0.16	5.2
Emp	-0.28	-0.16	5.8
BB	Daily	-0.62	-0.40	19.0
Emp	-0.71	-0.52	19.7
Table 4: Diversity results of using Diversity Threshold Generation (with a  $div_{thresh}$  of 10# Contradictions for NLI, 0.98 for distinct-n, and -0.164 for Sent-BERT). Num. sampled has a maximum value of 20; DG is the DialogGPT model; BB is BlenderBot.
number of sampled utterances required to reach  $\text{div}_{\text{thresh}}$ .
Results for Diversity Threshold Generation are shown in Table 4. For every condition, we see an increase from starting to ending diversity; for NLI Diversity, this results in an average  $137\%$  increase. For most conditions, distinct-n requires more samples than Sent-BERT and Baseline NLI Diversity.
We can use the results of Diversity Threshold Generation to probe differences in the models. In our experimental setup, DialogGPT generates more diverse utterances across all conditions than BlenderBot. The models change by similar proportions from starting to ending diversity using the NLI metric. However, the starting diversity for BlenderBot is far lower than DialogGPT; the negative value for BlenderBot indicates that a large number of entailment predictions were present in the starting response set.
We can also examine differences between the datasets. For instance, we observe lower starting diversities for the Empathetic Dialogues dataset than for DailyDialog++ for both models. Additionally, the number of samples required for EmpatheticDialogues is consistently higher than for DailyDialog++. This is likely because  $div_{thresh}$
for both datasets was calculated using human responses from DailyDialog++, since EmpatheticDialogues does not include multiple human responses.
Sampled responses can be seen in Appendix B and results reporting the average overlap from starting to ending sets of responses is in Appendix C. Appendix D includes results using beam search instead of nucleus sampling, and Appendix E reports the stability of Diversity Threshold Generation.
6.4 Relevance Results
Since past work has documented a tradeoff between diversity and relevancy (Zhang et al., 2018), we also report results for the relevancy of the starting and ending sets of responses for Diversity Threshold Generation. We use two established relevancy metrics: BLEU Score (Papineni et al.,  $2002)^{6}$  and BERTScore (Zhang et al.,  $2020\mathrm{a})^{7}$ . We show results on DailyDialog++, which has multiple human-generated responses for comparison, which is more correlated to human judgements than single-reference evaluation (Gupta et al., 2019).
Results are shown in Table 5. The key takeaway is that the relevancy values remain virtually unchanged when using the Diversity Threshold Generation procedure, according to both BLEU score and BERTScore. The average percent difference is  $0.08\%$  for BertScore and  $1.1\%$  for BLEU.
Limitations. While NLI Diversity is highly correlated with human judgements of diversity, it is limited by the NLI model chosen. Compared to Sent-BERT, the dataset used to train the NLI model is limited in scope. While our experiments showed that an NLI model trained on more datasets (Combined) did not perform better than MNLI, future work can more explicitly explore the effect of more generalized data on NLI Diversity.
This work is limited by automatic evaluation metrics for diversity and relevance. Future work should conduct additional human validation of model responses. More work could also be done to examine cases where the model was not able to generate diverse set, such as when humans also find creating a diverse set of responses difficult.
Future Work. Our results showed Confidence NLI Diversity was highly correlated with both
<sup>7</sup>https://github.com/Tiiger/bert_score
Metric	Model	Starting BERT Score	Ending BERT Score	Starting BLEU	Ending BLEU
NLI	DG	0.862	0.862	0.317	0.318
BB	0.868	0.867	0.367	0.368
Distinct- n	DG	0.862	0.861	0.319	0.306
BB	0.867	0.867	0.366	0.367
Sent- BERT	DG	0.863	0.862	0.318	0.313
BB	0.868	0.867	0.366	0.366
Table 5: Results comparing starting and ending sets of responses from Diversity Threshold Generation to sets of human responses using two relevancy metrics, BERTScore and BLEU score.
human judgements and the diversity parameter, achieving state-of-the-art performance on a semantic diversity dataset. The ablation study deepened this finding, showing that NLI contradiction predictions are especially correlated with diversity. Future work can leverage this finding, e.g., by wording crowdworker instructions to ask for generation contradictory, rather than diverse, responses.
Our results also show that dialogue generation models are able to improve the diversity of a sampled sets of responses using Diversity Threshold Generation. Diversity Threshold Generation can be used to evaluate future models' capacity to generate multiple diverse responses.
Future work should compare the resulting diverse responses in a conversational context. Studies could be conducted where chatbot users or dialogue writers can choose the way they want the model to respond, similar to Clark and Smith (2021).
We propose a novel semantic diversity metric, NLI Diversity, which is highly correlated to human judgments. Confidence NLI Diversity achieves state-of-the-art results on measuring semantic diversity. We propose Diversity Threshold Generation to incentivize production of diverse sets of responses for a conversation. This results in more diverse sets of responses than originally sampled for multiple models, datasets, and metrics while maintaining relevancy, and can also be used to investigate a model's ability to produce diverse responses.
This work was supported by an AWS Machine Learning Research Award, an NVIDIA Corporation GPU grant, an AI2 Key Scientific Challenge
Proposal grant, and a National Science Foundation (NSF) Graduate Research Fellowship (DGE 1752814). We thank the anonymous ARR reviewers as well as Philippe Laban, Dongyeop Kang, Nate Weinman, and the Hearst Lab Research Group for their helpful comments.
Dhruv Batra, Payman Yadollahpour, Abner Guzman-Rivera, and Gregory Shakhnarovich. 2012. Diverse m-best solutions in markov random fields. In Computer Vision - ECCV 2012, pages 1-16, Berlin, Heidelberg. Springer Berlin Heidelberg.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Kris Cao and Stephen Clark. 2017. Latent variable dialogue models and their diversity. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 182-187, Valencia, Spain. Association for Computational Linguistics.
Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '98, page 335-336, New York, NY, USA. Association for Computing Machinery.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Elizabeth Clark and Noah A. Smith. 2021. Choose your own adventure: Paired suggestions in collaborative writing for evaluating story generation models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3566-3575, Online. Association for Computational Linguistics.
Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. 2019. Approximating interactive human evaluation with self-play for open-domain dialog systems. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA. Curran Associates Inc.
Kevin Gimpel, Dhruv Batra, Chris Dyer, and Gregory Shakhnarovich. 2013. A systematic exploration of diversity in machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1100-1111, Seattle, Washington, USA. Association for Computational Linguistics.
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 379-391, Stockholm, Sweden. Association for Computational Linguistics.
Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1689-1701, Minneapolis, Minnesota. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019. Outlier detection for improved data quality and diversity in dialog systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 517-527, Minneapolis, Minnesota. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Association for the Advancement of Artificial Intelligence (AAAI).
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. Transactions of the Association for Computational Linguistics, 8:810-827.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1702-1723, Minneapolis, Minnesota. Association for Computational Linguistics.
Haoyu Song, Wei-Nan Zhang, Jingwen Hu, and Ting Liu. 2020. Generating persona consistent dialogues by exploiting natural language inference. In *The Thirty-Fourth AAAI Conference on Artificial Intelligence*, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8878-8885. AAAI Press.
Katherine Stasaki, Grace Hui Yang, and Marti A. Hearst. 2020. More diverse dialogue datasets via diversity-informed data collection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4958-4968, Online. Association for Computational Linguistics.
Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation of diversity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326-346, Online. Association for Computational Linguistics.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen,
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Xinyuan Zhang, Yi Yang, Siyang Yuan, Dinghan Shen, and Lawrence Carin. 2019. Syntax-infused variational autoencoder for text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2069-2078, Florence, Italy. Association for Computational Linguistics.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018. Generating informative and diverse conversational responses via adversarial information maximization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 1815-1825, Red Hook, NY, USA. Curran Associates Inc.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. Association for Computational Linguistics.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texy- gen: A Benchmarking Platform for Text Generation Models, page 1097-1100. Association for Computing Machinery, New York, NY, USA.
Figure 3: Spearman's Correlation with  $95\%$  Confidence Intervals.
Metric	Model	Dataset	Utterance Overlap
NLI	DG	Daily	2.63
Emp	2.42
BB	Daily	1.78
Emp	1.73
Distinct- n	DG	Daily	2.89
Emp	0.87
BB	Daily	1.51
Emp	1.65
Sent- BERT	DG	Daily	3.11
Emp	3.0
BB	Daily	1.56
Emp	1.64
Table 6: Average utterance overlap from starting to ending set of responses using Diversity Threshold Generation on multiple models, datasets, and diversity metrics.
A Confidence Interval Analysis
We perform experimentation using bootstrapping to determine confidence intervals for conTest correlations to the diversity parameter. We sample a dataset of 110 elements (50% of the original conTest dataset's size) from conTest with replacement and compute corresponding Spearman's correlation values using the sampled dataset for Sent-BERT, Baseline NLI Diversity, Confidence NLI Diversity, and human judgements. We repeat this process 1,000 times for stability and calculate 95% Confidence Intervals. The full conTest correlation value plotted with these intervals can be seen in Figure 3. While the Confidence Interval values overlap
between all 4 conditions, the Confidence NLI Diversity distribution closely matches the human distribution.
B Sampled Responses
Table 7 shows randomly-sampled examples from the DailyDialog++ dataset, created using Diversity Threshold Generation with the DialoGPT model and NLI Diversity as the intended div_metric.
C Average Utterance Overlap
We measure the number of utterances which occur in both the starting and ending sets of responses, called utterance overlap. A high utterance overlap represents a set of responses which did not need to be significantly changed to reach  $div_{thresh}$ . For example, an utterance overlap of 4 indicates that only 1 response needed to be resampled (potentially multiple times) from the starting set to reach  $div_{thresh}$ . Results are seen in Table 6. Keeping in mind that higher Average Overlap indicates less resampling was needed, we note higher overlap for DialoGPT than BlenderBot 1.0 (with the exception of distinct-n and EmpatheticDialogues).
D Beam Search