SlowGuess/NAACL_2022_OCR huggingface dataset:
We evaluate beam search's ability to generate diverse utterances using Diversity Threshold Generation for DailyDialog++ and NLI Diversity. To compare nucleus sampling to beam search, we generate 25 beams and consider these responses from most to least probable, i.e. if the 5 most likely beams do not satisfy the diversity threshold, we remove the lowest-scoring beam and replace it with the 6th most likely beam. We find the starting NLI Diversity for beam search is -5.05, the ending diversity is 5.35, and an average of 10.97 sampled utterances is required. While the NLI Diversity does improve from the starting to ending set of responses, beam search has a much lower ending diversity than nucleus sampling. While past work has confirmed that nucleus sampling is more lexically diverse than beam search using Self-BLEU (Holtzman et al., 2020), our results confirm that nucleus sampling is also able to generate more semantically diverse utterances.
E Stability of Procedure
We investigate the stability of Diversity Threshold Generation by measuring the number of samples
Table 7: Starting and ending responses with corresponding NLI Diversity for DialogGPT, randomly sampled from the DailyDialog++ dataset. SD = Starting NLI Diversity; ED = Ending NLI Diversity.
Figure 4: Histogram of number of samples required before reaching intended number of contradictions. Each bar color represents a different run of the experiment.
required before reaching  $div_{thresh}$  across multiple runs of the experiment. We present results for NLI Diversity, DailyDialog++, and DialogGPT and observe similar trends across all other conditions.
Figure 4 reports the number of resampled utterances required before reaching the intended number of contradictions. Each bar color represents a different run of the experiment. We do not observe a large difference in number of resamples required between runs of the same condition, indicating that the method is stable. The last bucket contains sets of responses which reached the maximum number of samples,  $S = 20$ , indicating divthresh could not be reached.