SlowGuess/ACL_2020_OCR huggingface dataset:
Model	PPL	F1	Token frequency classes
Freq	Med	Rare	Rarest
Human	-	-	.400	.300	.200	.100
MLE Baseline	11.4	.199	.491	.282	.157	.068
UL, α = 100	11.4	.200	.483	.289	.163	.063
UL, α = 101	11.9	.201	.459	.328	.154	.058
UL, α = 102	12.5	.190	.430	.335	.163	.071
UL, α = 103	14.4	.174	.399	.339	.188	.073
Table 4: Unlikelihood loss applied to vocabulary distributions. Stronger  $\alpha$  terms greatly shift probability mass from the most Frequent words to Medium and Rare words, at a small cost to PPL and F1. Frequent, medium, rare and rarest token classes are defined as the sets of tokens whose cumulative masses account for the top  $40\%$ , the next  $30\%$ , the next  $20\%$  and final  $10\%$  of tokens empirically generated by humans, respectively.
Figure 5: Vocabulary control with unlikelihood training: more probability mass is transferred from Frequent words to Rare words as we increase the  $\alpha$  weighting parameter. The maximum likelihood baseline is far from the human distribution.
from the Persona-Chat dialogue task (Zhang et al., 2018). We show examples from Dialogue NLI in
Figure 6: Dialogue NLI from (Welleck et al., 2019b).
Train	Test	Valid
Entailment	95k	4613	4959
Triple-Entailment	105k	5285	5481
Neutral	110k	5500	5700
Negatives	110k	5500	5700
Table 5: Dialogue NLI two utterance generation task dataset statistics.
Figure 6. The original data consists of sentence pairs  $(s_1, s_2)$  along with a label (E, N, or C), and was constructed by developing a schema and employing crowdworkers to label utterances with relation triples. The labels are then inferred from the triple representation.
We first transform the original classification dataset into a form useful for unlikelihood training of a generative dialogue model. We consider two setups: (i) a two utterance generation task; and (ii) a full dialogue generation task.
Two Utterance Generation Task We adapt the initial dialogue NLI dataset by using entailing and neutral training sentence pairs as plausible positive utterances, and contradicting pairs as negatives. That is, if a pair  $(s_1,s_2)$  from Dialogue NLI has label E or N, the example  $(\mathbf{x},\mathbf{y}) = (s_1,s_2)$  is added to  $\mathcal{D}^+$ , otherwise (label C) it is added to  $\mathcal{D}^{-}$ .
We consider two types of entailment: entailing sentence pairs that appear together in a dialogue in the original Persona-Chat dataset and are therefore natural ('entailment'), and those that only entail via their triple relations ('triple-entailment'). The latter are more challenging, noisier targets. Evaluation is performed by measuring the test set perplexity over the four target label types, where contradictions should have relatively higher perplexity. We additionally evaluate a selection accuracy task, where for each test example there are two candidate responses: a positive and a negative
(contradicting) statement. The candidate response with the lowest perplexity is considered to be the model's selection, and we measure the selection success rate. Evaluation is broken down by positive type (entailment, triple-entailment, neutral). Dataset statistics are given in Table 5.
Full Dialogue Task To evaluate in a more realistic setup that involves full dialogue rather than a single utterance, we take full Persona-Chat dialogues (Zhang et al., 2018) similar to Figure 6, and map back the dialogue NLI data to provide positive and negative continuations of the dialogue. We consider continuations as either triple entailing utterances, neutral utterances or contradictions - where the relation triple is used to match the existing persona or dialogue turns by the same speaker to induce the label. That is, an example  $(\mathbf{x},\mathbf{y})$  consists of a dialogue history  $\mathbf{x} = \{p_1,\dots ,p_k,u_1,\dots ,u_t\}$  and utterance  $\mathbf{y} = s_{2}$  where  $(s_1,s_2)$  is a sentence pair from Dialogue NLI, and at least one sentence in  $\mathbf{x}$  has the same relation triple as  $s_1$ . When the pair  $(s_1,s_2)$  is labeled as E or N in Dialogue NLI, the example  $(\mathbf{x},\mathbf{y})$  is added to  $\mathcal{D}^+$ , and otherwise it is added to  $\mathcal{D}^{-}$ .
Results Our MLE baseline obtains a perplexity of 11.4, in line with current best systems on this task (Lewis et al., 2019). Unfortunately, despite being good on such standard metrics, our baseline models fail at our coherence task. As seen in Table 6 for the two utterance task, the perplexity of contradicting utterances (12.5) is on average lower than for neutral (36.7) or triple-entailing utterances (17.5), although it is higher than entailing utterances. We believe this is due to contradicting utterances having high word overlap with the premise utterance, coupled with an inability to judge incoherence. Viewed as a selection task between utterances, picking the utterance with the lowest perplexity, this means the selection rates of non-contradicting utterances are very low, e.g. picking neutral utterances over contradicting utterances only  $18\%$  of the time. Even fully entailing utterances are only picked  $73\%$  of the time. Similar results are found on the full dialogue task as well, see Table 7.
Unlikelihood training brings large improvements in coherence metrics, whilst minimally impacting overall dialogue perplexity. After applying unlikelihood, perplexity for contradicting utterances has a clear signature, with very large av
Data + Model	Selection Accuracy	Perplexity
Entail	Tr.-E	Neutral	Entail	Tr.-E	Neutral	Contradict	ConvAI2
MLE Baseline	72%	41%	18%	8.54	17.5	36.7	12.5	11.4
UL (Dialogue NLI)	96%	85%	78%	9.1	26.6	39.4	248.9	11.9
Table 6: Test evaluation on the Dialogue NLI two utterance generation task, comparing standard likelihood (MLE) models trained on pushshift.io Reddit and ConvAI2 with unlikelihood loss NLI training. Results are broken down according to whether the premise and positive candidate are entailing, triple-entailing, or neutral (Entail, Tr.-E, Neutral). Selection Accuracy measures how often the model assigns lower perplexity to the positive candidate than to the negative candidate in the pair. Top two rows: for standard maximum likelihood models, the perplexity of contradicting utterances is lower compared to neutral or triple-entailing utterances (albeit higher compared to entailing utterances), showing partial failure at the coherence task. Bottom row: NLI Unlikelihood training yields large improvements on all coherence metrics, while minimally increasing overall perplexity.
Data + Model	Selection Accuracy (vs. Neg)	Perplexity
Triple-Entail	Neutral	Triple-Entail	Neutral	Contradict	ConvAI2
MLE Baseline	66.5%	36.8%	23.3	45.1	35.9	11.4
UL (Dialogue NLI)	89.0%	69.8%	21.5	40.3	63.5	11.8
Table 7: Test evaluation on the Full Dialogue NLI generation task. NLI unlikelihood training improves coherence metrics compared to likelihood (MLE) training. For UL, the triple-entailing or neutral candidates are assigned relatively lower perplexity compared to contradicting candidates, with higher selection accuracy for coherent labels.
Premise	Hypothesis	LMLE PPL	ULPPL
Yes, I love watching baseball and basketball. I do not like running though.	(C) I love running.	25.5	226.9
(E) I despise running.	29.9	9.4
Yes, I love watching baseball and basketball. I do like running though.	(E) I love running.	26.2	3.1
(C) I despise running.	42.8	247.1
We did too but working in real estate for 12 years . sucked up a lot of time	(E) I have been working as a real estate agent for the past 12 years.	3.9	3.8
(C) We did too but working in real estate for fifteen years sucked up a lot of time.	3.1	17.6
Figure 7: Example perplexities of a baseline maximum likelihood model  $(\mathcal{L}_{\mathrm{MLE}})$  and our unlikelihood trained model  $(\mathcal{L}_{\mathrm{UL}})$  when generating the provided hypotheses, given the premise. The maximum likelihood trained model assigns high probability (low perplexity) to contradictory generations, while unlikelihood does not.
erage values compared to entailing or neutral utterances, e.g. 248.9 vs. 9.1 for contradict vs. entail on the two utterance task. This converts to corresponding large increases in selection accuracy across all types on both tasks, e.g., an increase from  $18\%$  to  $78\%$  on neutral statements on the two utterance task, and from  $37.4\%$  to  $69.8\%$  on the full dialogue task.
Some example model predictions are given in Figure 7, comparing the MLE baseline and unlikelihood model perplexities of generating the given hypotheses. The likelihood model cannot differentiate between contradicting and entailing statements easily, while there are large perplexity differences for the unlikelihood model in these cases.
Generating consistent and coherent human-like dialogue is a core goal of natural language research. We studied several aspects that contribute to that goal, defined metrics to measure them, and proposed algorithms that improve them, mitigating some of the failings of maximum likelihood training, the current dominant approach. Our method defines objective functions under the umbrella of unlikelihood: during training, we wish to make inconsistent dialogue unlikely by lowering the probability of such events occurring. This makes generative models repeat themselves less, copy the context less, and use more rare words from the vocabulary - closer to matching human statistics. Further, utilizing supervised datasets with labeled
coherent and incoherent utterances and applying unlikelihood yields measurably improved levels of coherence with respect to the aspect measured, in this case contradiction. Future work could apply this same technique with other supervised data, e.g. correcting causal or commonsense reasoning errors (Zellers et al., 2019; Qin et al., 2019).
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2020. The second conversational intelligence challenge (ConvAI2). In *The NeurIPS '18 Competition*, pages 187–208, Cham. Springer International Publishing.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations.
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45-54. Association for Computational Linguistics.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pages 94-104, Copenhagen, Denmark. Association for Computational Linguistics.
Saadia Gabriel, Antoine Bosselut, Ari Holtzman, Kyle Lo, Asli Celikyilmaz, and Yejin Choi. 2019. Cooperative generator-discriminator networks for abstractive summarization with narrative flow. arXiv preprint arXiv:1907.01272.
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017,
System Demonstrations, pages 43-48, Vancouver, Canada. Association for Computational Linguistics.
Hiroaki Hayashi, Zeong Hu, Chenyan Xiong, and Graham Neubig. 2019. Latent relation language models. arXiv preprint arXiv:1908.07690.
Tianxing He and James Glass. 2019. Negative training for neural dialogue response generation. arXiv preprint arXiv:1903.02134.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv preprint arXiv:1905.01969.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: Improved dialogue evaluation with optimized questions and multi-turn comparisons. In Proceedings of the NeurIPS Workshop on Conversational AI.
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79-84, Copenhagen, Denmark. Association for Computational Linguistics.
Aigul Nugmanova, Andrei Smirnov, Galina Lavrentyeva, and Irina Chernykh. 2019. Strategy of the negative sampling for training retrieval-based dialogue systems. In 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pages 844-848. IEEE.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066.
Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5042-5052, Hong Kong, China. Association for Computational Linguistics.
4723
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1702-1723, Minneapolis, Minnesota. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019a. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019b. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning semantic textual similarity from conversations. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 164-174, Melbourne, Australia. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Model	PPL	F1	Repetition
Context	Label
Human	-	-	.160	.0006
MLE Baseline	8.3	.373	.582	.002
UL (Context only)	8.8	.345	.270	.001
UL (Label only)	8.3	.371	.645	.000
UL (Context + Label)	8.5	.358	.445	.003
A Repetition Control with Beam Search
The experiments on repetition and copying in the main paper were carried out with greedy decoding for simplicity. In this section we show that similar results hold with beam decoding as well. Using a beam size of 5, we take the same 4 models from Table 2 and compute metrics with beam instead. The results are given in Table 8 which show similar trends to before, except the baseline model using beam tends to suffer more from repetition, which is a known result (Holtzman et al., 2019). Note that we simply evaluated the same unlikelihood models as before, but we expect that better results could be obtained by performing sequence level unlikelihood training with beam search in the training loop, as well as choosing hyperparameters specifically with this kind of decoding being used to measure validation performance.
B Nucleus Sampling for Vocabulary control
Table 9 compares the MLE baseline, unlikelihood with increasing  $\alpha$  values, and Nucleus sampling (Holtzman et al., 2019) with hyperparameter  $p$  in terms of distribution and F1 score. The vocabulary unlikelihood fine-tuning shifts probability mass from the over-represented frequent words towards under-represented medium and rare words, with the effect strengthening as  $\alpha$  increases. At a small cost to perplexity and F1, the unlikelihood tuning reduced the overuse of common tokens by 9 points, matching the human rate, while improving the production of rare tokens by 3 percentage points.
Nucleus sampling is a popular method that can also produce generations closer to the human vocabulary distribution. It does this by sampling from the model's probability distribution rather
Table 8: Evaluation on the Wizard of Wikipedia task test set, comparing standard likelihood (MLE) with repetition unlikelihood loss training, where both methods use beam search (beam size of 5).
Model	PPL	F1	Token frequency classes
Freq	Med	Rare	Rarest
Human	-	-	.400	.300	.200	.100
MLE Baseline	11.4	.199	.491	.282	.157	.068
Nucleus p = 0.3	11.4	.180	.452	.315	.168	.064
Nucleus p = 0.4	11.4	.171	.440	.320	.172	.068
Nucleus p = 0.5	11.4	.160	.425	.322	.180	.072
Nucleus p = 0.6	11.4	.151	.411	.318	.192	.078
Nucleus p = 1.0	11.4	.141	.394	.302	.201	.101
UL, α = 100	11.4	.200	.483	.289	.163	.063
UL, α = 101	11.9	.201	.459	.328	.154	.058
UL, α = 102	12.5	.190	.430	.335	.163	.071
UL, α = 103	14.4	.174	.399	.339	.188	.073
Table 9: Unlikelihood loss applied to vocabulary distributions. Stronger  $\alpha$  terms greatly shift probability mass from the most Frequent words to Medium and Rare words, at a small cost to PPL and F1. Frequent, medium, rare and rarest token classes are defined as the sets of tokens whose cumulative masses account for the top  $40\%$ , the next  $30\%$ , the next  $20\%$  and final  $10\%$  of tokens empirically generated by humans, respectively. Nucleus sampling can also produce a distribution close to human with parameter  $p$  close to 1, but with larger losses in F1.
than using beam search, where the sampler restricts to the smallest set of tokens with total mass above a threshold  $p \in [0,1]$ . Small values of  $p$  are similar to greedy sampling. Increasing  $p$  yields distributions closer to human, but with large losses in F1 score, e.g.  $p = 0.5$  has a similar distribution to unlikelihood with  $\alpha = 10^2$  but the F1 scores are 0.160 vs. 0.190. This can be understood because maximizing likelihood during decoding yields better token accuracy than sampling (Welleck et al., 2019a), so the unlikelihood training approach to both use likelihood decoding and match the human distribution can obtain the best of both worlds.
Description of ConvAI2 vocabulary setup We follow (Li et al., 2019) and perform a pairwise comparison with full-length model conversations. We first collected 252 model-human conversations with each of the models (MLE baseline, and weights for  $\alpha$  of Unlikelihood, examples in 8). We then set up a pairwise-comparison using the software of (Li et al., 2019), using the same question ("Who would you prefer to talk to for a long conversation?") and use the exact same quality control question (a baseline greedy model without repetition control, versus a human). We collected ap
proximately 200 preferences per model comparison and filtered annotators who failed quality control.
Description of ELI5 repetition setup We follow (Fan et al., 2019) and perform a pairwise evaluation where human annotators were asked "which response answers the question better?" A screenshot of the UI is shown in Figure 9. Human evaluators were asked to rate a total of 5 questions, two of which were quality control annotations. The quality control examples contained the real human responses, along with model predictions: one question contained a baseline model, and one contained an unlikely model. Annotators which did not pick humans in quality controls were removed from the final setups. We collected 200 annotations comparing the baseline and the unlikelyhood model.
Results Evaluation results from all evaluated matchups are shown in Figure 10. We find our repetition-controlled ELI5 model significantly outperforms the MLE baseline. We find that two of the vocabulary repetition significantly outperform the MLE baseline. We compute significance with a two-tailed binomial test  $(p < .01)$ .
(d)
Figure 8: Examples of model-human conversations collected during human evaluation of the vocab unlikelihood models. Human utterances are in blue bubbles, model utterances are in white. Conversations (a) and (b) are from the baseline. Conversations (c) and (d) are from the  $\alpha = 10^{2}$  model and more frequently employ rarer words.
Figure 9: Screenshot of the Human Evaluator UI.
Figure 10: Complete Human Evaluation results. Human evaluators do not significantly prefer the  $\alpha = 10^{0}$  and  $\alpha = 10^{3}$  models over the baseline model.