Venue: NAACL 2022

# Semantic Diversity in Dialogue with Natural Language Inference

Katherine Stasaki and Marti A. Hearst

UC Berkeley

{katie_stasaski, hearst}@berkeley.edu

###### Abstract

Generating diverse, interesting responses to chitchat conversations is a problem for neural conversational agents. This paper makes two substantial contributions to improving diversity in dialogue generation. First, we propose a novel metric which uses Natural Language Inference (NLI) to measure the semantic diversity of a _set_ of model responses for a conversation. We evaluate this metric using an established framework Tevet and Berant (2021) and find strong evidence indicating NLI Diversity is correlated with semantic diversity. Specifically, we show that the contradiction relation is more useful than the neutral relation for measuring this diversity and that incorporating the NLI model's confidence achieves state-of-the-art results. Second, we demonstrate how to iteratively improve the semantic diversity of a sampled set of responses via a new generation procedure called Diversity Threshold Generation, which results in an average 137% increase in NLI Diversity compared to standard generation procedures.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 85 - 98

## 1 Introduction

Dialogue models often struggle to produce engaging utterances in conversations, tending to generate responses which are common in the training data, such as "OK," "Yean," or "I don't know" Li et al. (2016). While these responses are appropriate for a wide variety of contexts, their over-production can result in a dull conversation See et al. (2019).

An evaluation task has emerged that consists of measuring the diversity of chitchat model responses over a test set. While some past work uses human evaluation to measure model response diversity according to engagingness, specificity, or interestingness Li et al. (2016); See et al. (2019); Ghandeharioun et al. (2019), several automated metrics have also been proposed to measure diversity of model responses. Some metrics measure _lexical_ diversity, typically via n-gram overlap Li et al. (2016) or computing the BLEU score Zhu et al. (2018) among model responses generated from the test set. Other past work attempts to measure _semantic_ diversity via repurposing sentence similarity metrics Tevet and Berant (2021); Zhang et al. (2020); Cer et al. (2017).

We propose a new metric aimed at measuring semantic diversity by leveraging a Natural Language Inference (NLI) model to score a set of multiple dialogue model responses for a single conversation, as illustrated in Figure 1. NLI is a three-way classification task to determine whether one sentence entails, contradicts, or is neutral toward a second sentence. We hypothesize that a diverse set of responses for a conversation captures contradictory ways one could respond, which can be measured by the NLI model. We aggregate the contradiction, neutral, and entailment predictions among pairs of

Figure 1: Illustration of NLI Diversity using human responses from DailyDialog++. Contradictions are weighted by 1, entailments by -1, and neutrals by 0, so the score is \((2\times 1)+(3\times 0)+(1\times-1)=1\).



responses from the set and combine the predictions into a new diversity metric, called _NLI Diversity_.

We additionally explore two modifications of NLI Diversity. First, because the neutral prediction may be indicative of diversity, we propose Neutral NLI Diversity, where neutral predictions are weighted the same as contradiction predictions. Second, since our Baseline NLI Diversity method does not take into account the _confidence_ of the model's prediction, we propose Confidence NLI Diversity, which aggregates the probability mass of the model's predicted class instead of aggregating the number of predictions for each class.

We assess NLI Diversity using Tevet and Berant (2021)'s diversity metric evaluation framework, finding that NLI Diversity is highly correlated both with human judgments of diversity and with the _diversity parameter_, a gold standard diversity value used to generate the set of responses. Confidence NLI Diversity achieves state-of-the-art performance in terms of correlation with semantic diversity. Also, through an ablation study, we find positive, neutral, and negative correlations between human judgments and the number of contradiction, neutral, and entailment predictions, respectively.

We next explore the use of a dialogue model to generate a set of candidate responses with a minimum target level of semantic diversity, such as 10 Contradictions. Our new generation procedure, _Diversity Threshold Generation_, iteratively improves a set of model responses until this intended threshold is reached. If a set of sampled responses does not meet the intended threshold, the lowest-scoring response is thrown out and a new response is sampled until the diversity threshold is reached. We show this procedure results in a more diverse set of responses than the original sampled set, often with only a few resampled responses. Results of automated analysis shows relevancy is maintained from initial to final sets of responses.

In summary, our contributions are:

* A novel diversity metric, NLI Diversity, evaluated using Tevet and Berant (2021)'s framework, that measures semantic diversity and interrogates the relationship between Contradiction and Neutral predictions and diversity,
* Confidence NLI Diversity, a diversity metric which obtains state-of-the-art performance on semantic diversity,
* A new dialogue generation procedure, Diversity Threshold Generation, which continues sampling responses until an intended diversity threshold, defined using NLI Diversity, is reached,
* Experimental results indicating dialogue models are able to generate diverse responses using Diversity Threshold Generation with minimal loss in relevancy.

## 2 Related Work

Past work has explored lexical and semantic diversity metrics as well as ways of evaluating these metrics. We also draw from work in NLI and generating diverse sets of hypotheses.

### Measuring Model Response Diversity

Traditionally, a model's diversity has been measured in terms of its predictions over the test set (Li et al., 2016), which we call _Test Set Diversity_. In this setup, the model predicts one response for each conversation in the test set (containing \(n\) conversations), resulting in \(n\) predictions. The diversity measure is computed over these \(n\) predictions, resulting in a score over the entire test set.

The notion of diversity we investigate, however, measures the model's ability to generate a _set_ of responses for a single conversation (Zhang et al., 2019; Tevet and Berant, 2021), which we call _Multi-Response Diversity_. Instead of generating one response for each of the conversations in the test set, we evaluate a model's ability to generate \(m\) responses for each of the \(n\) conversations.

As shown by Tevet and Berant (2021), metrics which have been proposed in the _Test Set Diversity_ setting can still be applied in the _Multi-Response Diversity_ setting, however, by treating each set of \(m\) responses as its own "test set" and averaging over the \(n\) total sets.

### Diversity Metrics

Lexical diversity metrics measure differences in word choice, as opposed to diversity of content. Li et al. (2016) propose _distinct-n_, which measures the number of unique n-grams generated divided by the total number of n-grams generated in the _Test Set Diversity_ setting. Some past work has applied this metric to the _Multi-Response Diversity_ setting (Tevet and Berant, 2021). Cao and Clark (2017) propose examining the percent of unique _responses_ over the test set. Other past work has proposed using BLEU score over a set of model responses in the _Test Set Diversity_ setting (Zhu et al., 2018) 

Semantic diversity metrics, on the other hand, compare diversity of the content present in each response. Many of these measures are adapted from semantic similarity scores, since lower similarity can indicate higher diversity (Tevet and Berant, 2021). BERTScore measures the similarity of BERT embeddings for each token in two sentences (Zhang et al., 2020). Bert-STS assigns a score based on the semantic similarity of two sentences (Tevet and Berant, 2021). The Sent-BERT metric computes cosine similarity between BERT sentence embeddings (Reimers and Gurevych, 2019). Larson et al. (2019) propose identifying diverse paraphrases by identifying embedding outliers.

Other past work has used human evaluation to measure a model's diversity. Li et al. (2016) ask humans to choose the better of two responses based on specificity to the past conversation. See et al. (2019) ask humans to rank dialogue responses on a variety of factors, including interestingness and inquisitiveness. Tevet and Berant (2021) compare participants' ability to judge diversity of a set of responses in two ways: (i) by ranking one response as more diverse than a second response and (ii) by judging the diversity of a single response on a Likert scale, finding that participants were equally able to judge diversity in both conditions. They also find that human judges are better at distinguishing semantic diversity than lexical diversity.

Other past work has incorporated diversity metrics into the dialogue dataset creation pipeline. Stasaki et al. (2020) propose a method which measures the diversity of a crowdworker's contributions compared to a corpus, using that information to determine when to stop collecting data from the worker. This results in a more diverse dataset.

### Evaluation of Diversity Metrics

Tevet and Berant (2021) propose a framework to examine the reliability of diversity metrics. They propose the notion of a _diversity parameter_, which is used to generate a set of model responses, e.g., the \(p\)-value in nucleus sampling, which specifies the vocabulary probability distribution cutoff used to restrict sampling to the most-likely words whose combined likelihood \(\geq p\). If \(p\) is higher, the set of responses should have higher diversity, and vice-versa. This _diversity parameter_ is treated as a gold standard for a set of responses' diversity. Diversity metrics assign scores in the _Multi-Response Diversity_ condition and are evaluated in terms of correlation to the diversity parameter. They further propose two datasets to evaluate diversity metrics: one which includes model responses and contains varying levels of lexical diversity and one which is human-created and maintains high lexical diversity to allow focused evaluation of semantic diversity.

### Natural Language Inference

Natural Language Inference is a task aimed at predicting whether one sentence contradicts, entails, or is neutral towards a second sentence. Models for NLI are typically trained using one of two datasets: Stanford Natural Language Inference (SNLI) (Bowman et al., 2015) or Multi-Genre NLI (MNLI) (Williams et al., 2018). More recent datasets include FEVER (Thorne et al., 2018; Nie et al., 2019), adapted from a fact-checking dataset, and ANLI (Nie et al., 2020), collected in an adversarial human-in-the-loop procedure. With the rise of transformer architectures, models have achieved high performance on NLI tasks (Liu et al., 2019).

In a dialogue setting, NLI has been used to improve _consistency_ between a persona and model responses over the course of a conversation by integrating an NLI-based reward into a reinforcement learning training procedure (Song et al., 2020).

To our knowledge, however, NLI has not been used to measure the diversity of model responses in either the _Test Set Diversity_ or the _Multi-Response Diversity_ setting.

### Generating Diverse Sets of Hypotheses

While work has only recently begun to explore the task of generating multiple _dialogue_ responses to a conversation (Zhang et al., 2019; Tevet and Berant, 2021), past work has explored generating diverse sets of hypotheses in some other application areas. Carbonell and Goldstein (1998) explored using Maximal Mutual Relevance to reduce redundancy without sacrificing relevancy in document selection for summarization. Batra et al. (2012) proposed a greedy iterative algorithm to generate diverse, probable hypotheses for multiple vision tasks. Most related to our work is Gimpel et al. (2013), which applied Batra et al. (2012)'s approach to machine translation, generating a _set_ of translations instead of a single translation. In contrast to Gimpel et al. (2013), by holding the sampling procedure constant throughout the iterative process, our method can explore the extent to which diversity can be increased without altering standard decoding practices 

NLI Diversity Metric

We propose three diversity metrics in the _Multi-Response Diversity_ setting which leverage the predictions of an NLI model. Two metrics (Baseline and Neutral) aggregate the NLI model's class predictions and one metric (Confidence) aggregates the weight of these predictions.

### Baseline NLI Diversity

We propose a new metric, called _Baseline NLI Diversity_, which uses an NLI model's predictions to measure diversity. More formally, for a given conversation, \(c\), and a dialogue generation model \(M\), a set of utterances \(u_{1},...,u_{n}\) is produced by the model. Each pair of utterances is compared in both directions using an NLI model, \(NLI(u_{1},u_{2}),NLI(u_{2},u_{1}),...,NLI(u_{n},u_{n-1})\).

The NLI model predicts a distribution over the three potential classes: contradiction, neutral, and entailment. We take the argmax over these classes, resulting in a list of NLI predictions, \(NLIpreds(NLI(u_{1},u_{2}),...,NLI(u_{n-1},u_{n}))\) of size \(n(n-1)\). To produce an overall diversity score for \(NLIpreds(u_{1},...,u_{n})\), we assign each of these classes a value representing their diversity, denoted \(NLIscore(NLIpreds(u_{1},...,u_{n}))\).

We hypothesize that larger numbers of entailment predictions found in a set of model-generated utterances is indicative of a lack of diversity; similarly, larger number of contradiction predictions is indicative of a larger amount of diversity. Because we want a higher value of \(NLIscore\) to indicate higher diversity, we assign values as:

\[NLIscore=\begin{cases}1&\text{if contradiction}\\ 0&\text{if neutral}\\ -1&\text{if entailment}\end{cases}\]

The sum of the \(NLIscore\) values for the set of utterances results in the final NLI Diversity score, formally defined as:

\[Baseline~{}NLIDiversity=\\ \sum_{u_{i},u_{j}\in u_{1},...,u_{n}}NLIscore(NLIpred(NLI(u_{ i},u_{j}))\]

While the Baseline NLI Diversity metric aggregates all classes, we also investigate the separate number of entailment, contradiction, and neutral predictions in \(NLIpreds\), denoted # Entailment, # Contradiction, and # Neutral, respectively.

### Neutral NLI Diversity

Our primary hypothesis is that contradictions indicate diversity and entailments indicate lack of diversity. Because it is unclear what the role of neutrals might be, we explore a version of NLI Diversity which weights neutral and contradiction predictions as equally diverse. This metric is the same as Baseline NLI Diversity except the \(NLIscore\) used to assign values is:

\[NLIscoreneutral=\begin{cases}1&\text{if contradiction}\\ 1&\text{if neutral}\\ -1&\text{if entailment}\end{cases}\]

### Confidence NLI Diversity

Because the prior two NLI Diversity metrics do not incorporate the confidence of the NLI model's class predictions, we explore an additional metric which incorporates this value. Letting \(conf_{class}(u_{1},u_{2})\) represent the model's probability mass assigned to the predicted NLI \(class\) after \(softmax\), the function is defined as: \(NLIscoreconfidence=\)

\[\begin{cases}1\times conf_{con}(u_{1},u_{2})&\text{if contradiction}\\ 0&\text{if neutral}\\ -1\times conf_{ent}(u_{1},u_{2})&\text{if entailment}\end{cases}\]

Intuitively, instead of assigning a 1 value for a contradiction prediction, this metric assigns the probability of the contradiction class. Likewise, instead of a -1 for an entailment prediction, this metric assigns the negative probability mass of the entailment class.

## 4 Evaluation of NLI Diversity

We evaluate NLI Diversity by computing the correlation between the metric and both human labels and _diversity parameter_ labels. Below we first describe the models and data and then present the results of the evaluation.

### Models

We explore two NLI models: a Roberta-large model Liu et al. (2019) fine-tuned on the MultiGenre NLI (MNLI) Corpus Williams et al. (2018)1 and a Roberta-large model fine-tuned on a combination of MNLI, SNLI, FEVER, and ANLI2, both 

containing 300M parameters. We refer to these models as _NLI Diversity_ - _MNLI_ and _NLI Diversity_ - _Combined_, respectively. We do not employ additional fine-tuning of these models.

### Data

There are two different English datasets released to evaluate diversity metrics in Tevet and Berant (2021): _conTest_ and _decTest_, described in Table 1. The _conTest_ dataset is human-created and captures _content_, or _semantic_, diversity independent of _lexical_ diversity. Low-diversity examples in this dataset have high lexical diversity but low semantic diversity. This dataset was created by asking crowdworkers to generate sets of utterances with either low or high _semantic_ diversity using varied language, in order to keep a high level of _lexical_ diversity constant across both conditions.

The _decTest_ dataset includes model-generated responses, with diversity controlled by a decoding parameter, such as a temperature parameter. The dataset can include duplicate responses, and does not attempt to mediate lexical diversity; therefore, low-diversity examples in this dataset may reflect _low lexical_ as well as _low semantic_ diversity.

While the original dataset includes multiple generation tasks, we evaluate on the dialogue task, _respGen_, which is drawn from Reddit conversations Hashimoto et al. (2019)3. There are 200 conversations for each of _conTest_ and _decTest_ for the _respGen_ task, with multiple responses for each conversation (5 for _conTest_, 10 for _decTest_).

Footnote 3: In the data released from Tevet and Berant (2021), these files are called con_test_200withhdsrepsen.csv and dec_test_200withhdsrepsen.csv for _conTest_ and _decTest_, respectively.

### Diversity Parameter Correlation

The _diversity parameter_ from Tevet and Berant (2021) represents either a parameter directly used to generate responses via a dialogue model, such as \(p\) in nucleus sampling, or a binary value indicating whether crowdworkers were instructed to generate a high- or low-diversity set of responses. A measure which is able to capture diversity will be positively correlated with this diversity parameter.

Table 2 shows Spearman's correlations between NLI Diversity and the diversity parameter. On the _conTest_ semantic diversity dataset, Confidence NLI Diversity achieves the highest correlation of all metrics (0.62) and approaches human performance. Baseline NLI Diversity performs comparably to the top-performing automatic metric from Tevet and Berant (2021), at 0.59 correlation. We note the 95% confidence intervals overlaps between Baseline NLI Diversity, Confidence NLI Diversity, SentBERT, and human judgements, indicating a lack of significant differences (see Appendix A). Although Neutral NLI Diversity does relatively poorly on _conTest_ (0.24), it is the highest-performing NLI metric on _decTest_ (0.72), suggesting that incorporating neutral predictions may capture lexical instead of semantic diversity.

A histogram of Confidence NLI Diversity values for low and high semantic diversity sets of responses is shown in Figure 2. We note the lack of large overlap between the distributions of low and high semantic diversity data. In addition to

\begin{table}
\begin{tabular}{|l l|} \hline
**decTest** & Mixed Lexical Diversity; Mixed Semantic Diversity; Model Generated \\ \cline{2-3}
**Examples:** & “I think he is the most awesome guy ever” \\  & “He is the most awesome guy ever” \\ temp 0.55 & “The unemployment rate is lower than what it is” \\  & “No but it does make it more likely to be higher than what it is” \\ \hline \hline
**conTest** & High Lexical Diversity; Mixed Semantic Diversity; Human Generated \\ \cline{2-3}
**Examples:** & “Sorry, but I don’t agree.” \\ high lexical and & “I think you are wrong about that.” \\  & “Don be so judgemental, try to see things her way.” \\ high semanatic & “You are right that is insane.” \\  & “ 

the correlation results in Sections 4.3 and 4.4, this result indicates the Confidence NLI Diversity metric distinguishes between low and high semantic diversity.

The higher correlation to the diversity parameter leads us to choose NLI Diversity - MNLI instead of Combined for all further experimentation.

### Human Correlation

In this subsection, we examine the NLI Diversity metric's correlation to the human annotations collected by Tevet and Berant (2021). Each set of responses in _conTest_ and _decTest_ is scored by \(10\) annotators from \(1\) (not diverse at all) to \(5\) (very diverse) with half-point increments. We compute correlation with respect to the averaged rating.

In addition to NLI Diversity, we explore the prediction counts for each category. We expect that a higher # Entailment value will be negatively correlated with diversity because the more pairs of responses that entail each other, the more similar the set of responses is. Similarly, we expect that a higher # Contradiction value will be positively correlated with diversity. Since the NLI Diversity metric incorporates both # Entailment and # Contradiction, we would expect this metric to be highly correlated with human judgments as well.

Spearman's \(\rho\) rank correlation results between our metrics and the human diversity scores are shown in Table 3. The highest-performing correlation for lexical diversity is the Neutral NLI Diversity (0.69). The highest-performing semantic diversity correlation is Confidence NLI Diversity (0.64). Additionally, Baseline and Confidence NLI Diversity correlations are stronger when evaluating with the _conTest_ dataset than the _decTest_ dataset (an increase of 0.48 to 0.63 for Baseline MNLI and 0.41 to 0.64 for Confidence NLI), indicating these metrics are more correlated with human ratings of semantic diversity than lexical diversity.

Across both datasets, # Entailment is negatively correlated with diversity, # Neutral does not have a strong correlation, and # Contradiction is positively correlated, as hypothesized. This supports

Figure 2: Histogram of Confidence NLI Diversity for high and low semantic diversity examples.

 

our motivation to use NLI as a diversity metric.

## 5 Diversity Threshold Generation

We have verified that NLI Diversity is both able to capture semantic diversity and aligns with human judgements. We can additionally use NLI Diversity to define a straightforward desired diversity threshold, \(div_{thresh}\) for a set of model-generated responses, \(u_{1},...,u_{n}\). For example, we might intend there to be 10 Contradictions within the set. We propose a generation procedure, Diversity Threshold Generation, designed to iteratively increase the diversity of a set of responses for a conversation.

For a conversation, Diversity Threshold Generation begins by sampling \(n\) responses. We score the diversity of these responses using a diversity metric, \(div\_metric(u_{1},...,u_{n})\). If the diversity score falls above \(div_{thresh}\), the process is finished.

If, however, the score falls below \(div_{thresh}\), we identify the model response which contributes _least_ to the diversity score by calculating \(div\_metric(u_{1},...,u_{n-1})\) for each sub-group of model responses of size \(n-1\). We discard the model response not present in the highest-scoring subgroup and resample a new response. We re-calculate \(div\_metric(u_{1},...,u_{n})\) and if \(div\_metric(u_{1},...,u_{n})>div_{thresh}\), the process finishes. We continue resampling until the maximum cutoff of \(S\) is reached.

## 6 Evaluation of Diversity Threshold Generation Method

### Models and Datasets

We experiment with two neural dialogue models, DialoGPT (700M parameters) [23]4 and BlenderBot 1.0 (300M parameters) [8]5. We use the default Transformers implementation for each model [9] and do not fine-tune them. Runtime was between 3 and 36 hours on one Titan-X GPU.

Footnote 4: [https://huggingface.co/transformers/model_doc/dialogopt.html](https://huggingface.co/transformers/model_doc/dialogopt.html)

Footnote 5: [https://huggingface.co/transformers/model_doc/blenderbot.html](https://huggingface.co/transformers/model_doc/blenderbot.html)

All experiments involve the dialogue model \(M\) generating 5 responses for each conversation. The maximum number of samples, \(S\), is set to 20. All experiments are averaged over 10 trials for stability.

We evaluate each model on the development set of two public English conversational datasets : DailyDialog++ (1,028 conversations) [24, 25] and EmpatheticDialogues (2,763 conversations) [24]. DailyDiaLog++ includes 5 human-written responses per conversation, allowing for multi-reference comparison. We split each EmpatheticDialogues conversation at a random turn (consistent for all experiments) for generation. Since BlenderBot supports up to 128 positional embeddings, we pass in the last 128 tokens of the conversation for this condition.

### Metrics

We evaluate three diversity metrics: two semantic diversity metrics, Baseline NLI Diversity (Section 3) and Sent-BERT [19, 20], and one lexical diversity metric, distinct-n [16, 21]. For Sent-BERT, we compute the average negative cosine similarity between BERT sentence embeddings for each pair of responses. Like Tevet and Berant (2021), for distinct-n, we compute the average distinct n-grams from \(n\in 1,2,3,4,5\).

Because Baseline NLI Diversity is more human-interpretable than Confidence NLI Diversity, we use this version for experimentation. For all NLI Diversity experiments, \(div_{thresh}\) is achieved when # Contradictions is greater than 10 out of a total of 20 pair-wise comparisons. For both Sent-BERT and distinct-n, however, we do not have a human-specifiable threshold. We use empirical thresholds measured from the sets of 5 human responses for each conversation in DailyDialog++. We choose the 90th percentile for \(div_{thresh}\) (0.98 and -0.179 for distinct-n and Sent-BERT, respectively).

We decode using nucleus sampling (\(p\) = 0.9), as it has been shown to increase response diversity [10]. However our method could be applied with other decoding procedures.

In order to robustly evaluate Diversity Threshold Generation, we measure both (i) whether Diversity Threshold Generation is able to generate more diverse sets of responses than was originally sampled and (ii) whether the increased diversity comes at the expense of decreased _relevancy_ of the responses.

### Diversity Results

We aim to measure whether the diversity of the 5 responses from \(M\) increases using Diversity Threshold Generation, compared to the initial 5 sampled responses. Diversity of the starting and ending sets of utterances is measured by Baseline NLI Diversity, distinct-n, or Sent-BERT. We also report the 

number of sampled utterances required to reach \(div_{thresh}\).

Results for Diversity Threshold Generation are shown in Table 4. For every condition, we see an increase from starting to ending diversity; for NLI Diversity, this results in an average 137% increase. For most conditions, distinct-n requires more samples than Sent-BERT and Baseline NLI Diversity.

We can use the results of Diversity Threshold Generation to probe differences in the models. In our experimental setup, DialoGPT generates more diverse utterances across all conditions than BlenderBot. The models change by similar proportions from starting to ending diversity using the NLI metric. However, the starting diversity for BlenderBot is far lower than DialoGPT; the negative value for BlenderBot indicates that a large number of entailment predictions were present in the starting response set.

We can also examine differences between the datasets. For instance, we observe lower starting diversities for the Empathetic Dialogues dataset than for DailyDialog++ for both models. Additionally, the number of samples required for EmpatheticDialogues is consistently higher than for DailyDialog++. This is likely because \(div_{thresh}\) for both datasets was calculated using human responses from DailyDialog++, since EmpatheticDialogues does not include multiple human responses.

Sampled responses can be seen in Appendix B and results reporting the average overlap from starting to ending sets of responses is in Appendix C. Appendix D includes results using beam search instead of nucleus sampling, and Appendix E reports the stability of Diversity Threshold Generation.

### Relevance Results

Since past work has documented a tradeoff between diversity and relevancy Zhang et al. (2018), we also report results for the _relevancy_ of the starting and ending sets of responses for Diversity Threshold Generation. We use two established relevancy metrics: BLEU Score Papineni et al. (2002)6 and BERTScore Zhang et al. (2020a)7. We show results on DailyDialog++, which has multiple human-generated responses for comparison, which is more correlated to human judgements than single-reference evaluation Gupta et al. (2019).

Footnote 6: [https://www.nltk.org/_modules/nltk/translate/bleu_score.html](https://www.nltk.org/_modules/nltk/translate/bleu_score.html)

Footnote 7: [https://github.com/Tiiiger/bert_score](https://github.com/Tiiiger/bert_score)

Results are shown in Table 5. The key takeaway is that the relevancy values remain virtually unchanged when using the Diversity Threshold Generation procedure, according to both BLEU score and BERTScore. The average percent difference is 0.08% for BertScore and 1.1% for BLEU.

## 7 Discussion

**Limitations.** While NLI Diversity is highly-correlated with human judgements of diversity, it is limited by the NLI model chosen. Compared to Sent-BERT, the dataset used to train the NLI model is limited in scope. While our experiments showed that an NLI model trained on more datasets (Combined) did not perform better than MNLI, future work can more explicitly explore the effect of more generalized data on NLI Diversity.

This work is limited by automatic evaluation metrics for diversity and relevance. Future work should conduct additional human validation of model responses. More work could also be done to examine cases where the model was not able to generate diverse set, such as when humans also find creating a diverse set of responses difficult.

**Future Work.** Our results showed Confidence NLI Diversity was highly correlated with both

\begin{table}
\begin{tabular}{|c|l|p{28.5pt}|p{28.5pt}|r|r|r|} \hline  & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} \\ \cline{5-6} \multirow{-2}{*}{**Metric**} & \multicolumn{1}{c|}{**Mocric**} & \multicolumn{1}{c|}{**Dialog**} & \multicolumn{1}{c|}{**Sart-**} & \multicolumn{1}{c|}{**End-**} & \multicolumn{1}{c|}{**Num.**} & \multicolumn{1}{c|}{**Sampling**} \\  & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} \\ \hline \multirow{4}{*}{**Dataset**} & \multirow{2}{*}{DG} & Daily & 4.11 & 10.24 & 6.3 \\  & & Emp & 3.68 & 10.11 & 7.1 \\  & & Daily & \(-\)5.55 & 2.51 & 14.4 \\  & & Emp & \(-\)8.90 & \(-\)1.72 & 16.5 \\ \hline \multirow{4}{*}{**Dataset**} & \multirow{2}{*}{DG} & Daily & 0.95 & 0.98 & 5.4 \\  & & Emp & 0.43 & 0.52 & 20.0 \\ \cline{1-1}  & & \multirow{2}{*}{BB} & Daily & 0.61 & 0.80 & 20.0 \\ \cline{1-1}  & & Emp & 0.52 & 0.71 & 20.0 \\ \hline \multirow{4}{*}{**Dataset**} & \multirow{2}{*}{DG} & Daily & \(-\)0.26 & \(-\)0.16 & 5.2 \\ \cline{1-1}  & & Emp & \(-\)0.28 & \(-\)0.16 & 5.8 \\ \cline{1-1}  & & \multirow{2}{*}{BB} & Daily & \(-\)0.62 & \(-\)0.40 & 19.0 \\ \cline{1-1}  & & Emp & \(-\)0.71 & \(-\)0.52 & 19.7 \\ \hline \end{tabular}
\end{table}
Table 4: Diversity results of using Diversity Threshold Generation (with a \(div_{thresh}\) of 10 # Contradictions for NLI, 0.98 for distinct-n, and -0.164 for Sent-BERT). Num. sampled has a maximum value of 20; DG is the DialogGPT model; BB is BlenderBot.

 

human judgements and the diversity parameter, achieving state-of-the-art performance on a semantic diversity dataset. The ablation study deepened this finding, showing that NLI contradiction predictions are especially correlated with diversity. Future work can leverage this finding, e.g., by wording crowdworker instructions to ask for generation _contradictory_, rather than _diverse_, responses.

Our results also show that dialogue generation models are able to improve the diversity of a sampled sets of responses using Diversity Threshold Generation. Diversity Threshold Generation can be used to evaluate future models' capacity to generate multiple diverse responses.

Future work should compare the resulting diverse responses in a conversational context. Studies could be conducted where chatbot users or dialogue writers can choose the way they want the model to respond, similar to Clark and Smith (2021).

## 8 Conclusion

We propose a novel semantic diversity metric, NLI Diversity, which is highly correlated to human judgments. Confidence NLI Diversity achieves state-of-the-art results on measuring semantic diversity. We propose Diversity Threshold Generation to incentivize production of diverse sets of responses for a conversation. This results in more diverse sets of responses than originally sampled for multiple models, datasets, and metrics while maintaining relevancy, and can also be used to investigate a model's ability to produce diverse responses.

## Acknowledgements

This work was supported by an AWS Machine Learning Research Award, an NVIDIA Corporation GPU grant, an AI Key Scientific Challenge Proposal grant, and a National Science Foundation (NSF) Graduate Research Fellowship (DGE 1752814). We thank the anonymous ARR reviewers as well as Philippe Laban, Dongyeop Kang, Nate Weinman, and the Hearst Lab Research Group for their helpful comments.

## References

* ECCV 2012, Berlin, Heidelberg, pp. 1-16. External Links: ISBN 978-3-319-3319-3 Cited by: SS1.
* S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning (2015)A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 632-642. External Links: Link, Document Cited by: SS1.
* K. Cao and S. Clark (2017) Latent variable dialogue models and their diversity. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Valencia, Spain, pp. 182-187. External Links: Link, Document Cited by: SS1.
* J. Carbonell and J. Goldstein (1998)The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '98, New York, NY, USA, pp. 335-336. External Links: ISBN 978-3-642-4655-3 Cited by: SS1.
* D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia (2017) SemEval-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, pp. 1-14. External Links: Link, Document Cited by: SS1.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|} \hline
**Metric** & **Model** & **Starting** & **Ending** & **Start-** & **Ending** \\  & & **BERT** & **Score** & **BLEU** & **BLEU** \\ \hline \multirow{2}{*}{NLI} & DG & 0.862 & 0.862 & 0.317 & 0.318 \\  & BB & 0.868 & 0.867 & 0.367 & 0.368 \\ \hline Distinct-n & DG & 0.862 & 0.861 & 0.319 & 0.306 \\  & BB & 0.867 & 0.867 & 0.366 & 0.367 \\ \hline Sent- & DG & 0.863 & 0.862 & 0.318 & 0.313 \\ BERT & BB & 0.868 & 0.867 & 0.366 & 0.366 \\ \hline \end{tabular}
\end{table}
Table 5: Results comparing starting and ending sets of responses from Diversity Threshold Generation to sets of human responses using two relevancy metrics, BERTScore and BLEU score.

 

Elizabeth Clark and Noah A. Smith. 2021. Choose your own adventure: Paired suggestions in collaborative writing for evaluating story generation models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online. Association for Computational Linguistics.
* Ghandeharioun et al. (2019) Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. 2019. Approximating interactive human evaluation with self-play for open-domain dialog systems. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA. Curran Associates Inc.
* Gimpel et al. (2013) Kevin Gimpel, Dhruv Batra, Chris Dyer, and Gregory Shakhnarovich. 2013. A systematic exploration of diversity in machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA. Association for Computational Linguistics.
* Gupta et al. (2019) Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, Stockholm, Sweden. Association for Computational Linguistics.
* Hashimoto et al. (2019) Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
* Larson et al. (2019) Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019. Outlier detection for improved data quality and diversity in dialog systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
* Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California. Association for Computational Linguistics.
* Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
* Nie et al. (2019) Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Association for the Advancement of Artificial Intelligence (AAAI).
* Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nsli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, New York, NY. Association for Computational Linguistics.
* Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics.
* Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 300-325, Online. Association for Computational Linguistics 

Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. _Transactions of the Association for Computational Linguistics: Main Volume_, pages 326-346, Online. Association for Computational Linguistics.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remii Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lobest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Zhang et al. (2020a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Evaluating text generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Zhang et al. (2019) Xinyuan Zhang, Yi Yang, Siyang Yuan, Dinghan Shen, and Lawrence Carin. 2019. Syntax-infused variational autoencoder for text generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2069-2078, Florence, Italy. Association for Computational Linguistics.
* Zhang et al. (2018) Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018. Generating informative and diverse conversational responses via adversarial information maximization. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_, NIPS'18, page 1815-1825, Red Hook, NY, USA. Curran Associates Inc.
* Zhang et al. (2020b) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT : Large-scale generative pre-training for conversational response generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 270-278, Online. Association for Computational Linguistics.
* Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. _Texygen: A Benchmarking Platform for Text Generation Models_, page 1097-1100. Association for Computing Machinery, New York, NY, USA 



## Appendix A Confidence Interval Analysis

We perform experimentation using bootstrapping to determine confidence intervals for _conTest_ correlations to the diversity parameter. We sample a dataset of 110 elements (50% of the original _ConTest_ dataset's size) from _conTest_ with replacement and compute corresponding Spearman's correlation values using the sampled dataset for Sent-BERT, Baseline NLI Diversity, Confidence NLI Diversity, and human judgements. We repeat this process 1,000 times for stability and calculate 95% Confidence Intervals. The full _conTest_ correlation value plotted with these intervals can be seen in Figure 3. While the Confidence Interval values overlap between all 4 conditions, the Confidence NLI Diversity distribution closely matches the human distribution.

## Appendix B Sampled Responses

Table 7 shows randomly-sampled examples from the DailyDialog++ dataset, created using Diversity Threshold Generation with the DialoGPT model and NLI Diversity as the intended \(div\_metric\).

## Appendix C Average Utterance Overlap

We measure the number of utterances which occur in both the starting and ending sets of responses, called utterance overlap. A high utterance overlap represents a set of responses which did not need to be significantly changed to reach \(div_{thresh}\). For example, an utterance overlap of 4 indicates that only 1 response needed to be resampled (potentially multiple times) from the starting set to reach \(div_{thresh}\). Results are seen in Table 6. Keeping in mind that higher Average Overlap indicates less resampling was needed, we note higher overlap for DialoGPT than BlenderBot 1.0 (with the exception of distinct-n and EmpatheticDialogues).

## Appendix D Beam Search

We evaluate beam search's ability to generate diverse utterances using Diversity Threshold Generation for DailyDialog++ and NLI Diversity. To compare nucleus sampling to beam search, we generate 25 beams and consider these responses from most to least probable, i.e. if the 5 most likely beams do not satisfy the diversity threshold, we remove the lowest-scoring beam and replace it with the 6th most likely beam. We find the starting NLI Diversity for beam search is -5.05, the ending diversity is 5.35, and an average of 10.97 sampled utterances is required. While the NLI Diversity does improve from the starting to ending set of responses, beam search has a much lower ending diversity than nucleus sampling. While past work has confirmed that nucleus sampling is more _lexically diverse_ than beam search using Self-BLEU Holtzman et al. (2020), our results confirm that nucleus sampling is also able to generate more _semantically diverse_ utterances.

## Appendix E Stability of Procedure

We investigate the stability of Diversity Threshold Generation by measuring the number of samples

Figure 3: Spearman’s Correlation with 95% Confidence Intervals.





\begin{table}
\begin{tabular}{|p{108.4pt}|l|c|l|l|} \hline
**Context** & **Starting Responses** & **SD** & **Ending Responses** & **ED** \\ \hline
* Am I too abrupt? Did I’ scare you? Or you don’t want ... **Speaker 2:*
* No! I have been expecting this moment for a long time . I mean my answer is ’ yes’. **Speaker 1:*
* Oh, I’m so happy . I will try my best to love you and protect you . I promise. &
* Aww, stop, we both know you’re going to try to love me, you know? Thanks
* IT DHushing. I love you. Thank you. Thank you so much. It think you are a good woman. You can have free shipping if you want . I live a 5 dropoff fee that I pay for. In the beginning
* No. We just get it for free.So, no. &
* Hands you. Thank you. Thank you. Your a nice man. &
* Hands you. Thank you. Thank you as much. It's okay. I think you are a good woman. You are awesome. And I will try. And you are my best friend. Thank you, Sarah. &
* Hands you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. She is relieved now. I am at my desk. &
* Hands you. Thank you.


required before reaching \(div_{thresh}\) across multiple runs of the experiment. We present results for NLI Diversity, DailyDialog++, and DialoGPT and observe similar trends across all other conditions.

Figure 4 reports the number of resampled utterances required before reaching the intended number of contradictions. Each bar color represents a different run of the experiment. We do not observe a large difference in number of resamples required between runs of the same condition, indicating that the method is stable. The last bucket contains sets of responses which reached the maximum number of samples, \(S=20\), indicating \(div_{thresh}\) could not be reached.

## Appendix A

Figure 4: Histogram of number of samples required before reaching intended number of contradictions. Each bar color represents a different run of the experiment.