SlowGuess/ACL_2020_OCR huggingface dataset:
Don’t Say That!  
Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Margaret Li<sup>1</sup>, Stephen Roller<sup>1</sup>, Ilia Kulikov<sup>2*</sup>, Sean Welleck<sup>2*</sup>  
Y-Lan Boureau<sup>1</sup>, Kyunghyun Cho<sup>1,2</sup>, Jason Weston<sup>1,2</sup>
$^{2}$ New York University
{margaretli,roller,ylan,kyunghyuncho,jase}@fb.com
wellecks@nyu.edu, kulikov@cs.nyu.edu
Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetitions within utterances, (iii) overuse frequent words, and (iv) at a deeper level, contain logical flaws. In this work we show how all of these problems can be addressed by extending the recently introduced unlikelihood loss (Welleck et al., 2019a) to these cases. We show that appropriate loss functions which regularize generated outputs to match human distributions are effective for the first three issues. For the last important general issue, we show applying unlikelihood to collected data of what a model should not do is effective for improving logical consistency, potentially paving the way to generative models with greater reasoning ability. We demonstrate the efficacy of our approach across several dialogue tasks.
Open-ended tasks such as dialogue reveal a number of issues with current neural text generation methods. In more strongly grounded tasks such as machine translation and image captioning, current encoder-decoder architectures provide strong performance, where mostly word-level decisions are often taken correctly by the model. However, critical failings are exposed in less constrained generation: reliance on repetitive copying and overuse of frequent words, and an inability to maintain logical coherence. The former shows the learning objective is faulty in that it cannot match simple statistics of the training data, while the latter touches more to the heart of artificial intelligence:
Figure 1: GPT-2 345M model completions can show lack of coherence, e.g. direct contradictions.
these models do not understand what they are saying. For example, Figure 1 shows how the 345M-parameter GPT2 model (Radford et al., 2019) can give high probability to contradictory generations.
In this work, we show how the recently introduced unlikelihood objective (Welleck et al., 2019a) can be generalized to remedy these problems. Unlikelihood is a technique developed for removal of repetition in language model completions, and works by adding an extra term to the objective that forces repetitions to have low probability, alleviating the degenerative problems highlighted in Holtzman et al. (2019). In fact, likelihood can be seen as a much more general framework, as we will see.
We first generalize unlikelihood to a different domain: dialogue, where we measure statistics of the training distribution in terms of contextual copies, within-utterance repeats, and vocabulary usage. We then develop loss functions that control these statistics, providing improved metrics on several tasks. Secondly, we show how the same tools can be used to address deeper semantic issues in such models. By leveraging existing natural language inference (NLI) data (Welleck et al., 2019b) as supervision against poor quality generations, we train models that assign low probability to generating incoherent and contradictory text. Overall, our approach yields more consistent dialogue models across several axes, and provides a
$^{\star}$ Work done while at Facebook AI Research (FAIR).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715-4728 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
promising framework for further advances.
Code and pre-trained models will be made available.†
2 Dialogue Unlikelihood Training
Dialogue Generation Dialogue generation consists in predicting an utterance  $\mathbf{y} = (y_{1},\dots ,y_{|y|})$  given a context  $\mathbf{x} = \{s_1,\ldots ,s_k,u_1,\ldots ,u_t\}$  that consists of initial context sentences  $s_{1:k}$  (e.g., scenario, knowledge, personas, etc.) followed by dialogue history utterances  $u_{1:t}$  from speakers who take consecutive turns.
Likelihood Training Given a dataset  $\mathcal{D} = \{(\mathbf{x}^{(i)},\mathbf{y}^{(i)})\}$  derived from a collection of human-human interactions, the standard approach to generative training for dialogue tasks is maximum likelihood estimation (MLE), that minimizes:
$$
\mathcal {L} _ {\mathrm {M L E}} ^ {(i)} (p _ {\theta}, \mathbf {x} ^ {(i)}, \mathbf {y} ^ {(i)}) = - \sum_ {t = 1} ^ {| y ^ {(i)} |} \log p _ {\theta} (y _ {t} ^ {(i)} | \mathbf {x} ^ {(i)}, y _ {<   t} ^ {(i)}),
$$
where  $\mathbf{x}^{(i)}$  is a gold context (dialogue history and initial context sentences) and  $\mathbf{y}^{(i)}$  is a gold next-utterance, and  $y_{t}^{(i)}$  is the  $t$ -th token of  $\mathbf{y}^{(i)}$ .
Likelihood-based (greedy or beam) decoding applied after training a model with this objective yields sequences with statistics that do not match the original human training sequence distribution.
Unlikelihood Training To control for such distribution mismatches, we employ the unlikelihood loss (Welleck et al., 2019a), generalizing it to our setting, and developing a particular form of the loss function for each type of mismatch.
The general form of the unlikelihood loss penalizes a set of tokens  $\mathcal{C}_t$  at each time-step,  $\mathcal{L}_{\mathrm{UL}}^{(i)}(p_{\theta},\mathcal{C}_{1:T},\mathbf{x},\mathbf{y}) =$
$$
- \sum_ {t = 1} ^ {| y |} \sum_ {y _ {c} \in \mathcal {C} _ {t}} \beta \left(y _ {c}\right) \log \left(1 - p _ {\theta} \left(y _ {c} \mid \mathbf {x}, y _ {<   t}\right)\right),
$$
where  $\mathcal{C}_t\subseteq \mathcal{V}$  is a subset of the vocabulary, and  $\beta (y_{c})$  is a candidate-dependent scale that controls how much the candidate token should be penalized. The overall objective in unlikelihood training then consists of mixing the likelihood and un-likelihood losses,
$$
\mathcal {L} _ {\mathrm {U L E}} ^ {(i)} = \mathcal {L} _ {\mathrm {M L E}} ^ {(i)} + \alpha \mathcal {L} _ {\mathrm {U L}} ^ {(i)}, \tag {1}
$$
where  $\alpha \in \mathbb{R}$  is the mixing hyper-parameter.
Likelihood tries to model the overall sequence probability distribution, while unlikelihood corrects for known biases. It does this via the set of negative candidates  $\mathcal{C}_t$  calculated at each step  $t$ , where we are free to select candidate generation functions depending on the biases to be mitigated. Likelihood pushes up the probability of a gold token  $y_{t}^{(i)}$  while unlikelihood pushes down the probability of negative candidate tokens  $y_{c}\in \mathcal{C}_{t}$ .
In Welleck et al. (2019a) the context  $\mathbf{x}$  consists of a ground-truth sequence  $(\mathbf{x} = \mathbf{x}^{(i)})$ , the target  $\mathbf{y}$  is either a ground-truth sequence  $(\mathbf{y} = \mathbf{y}^{(i)})$  or a model-generated sequence  $(\mathbf{y} = \hat{\mathbf{y}})$ , and the pertoken scale parameter  $\beta(y_c)$  is 1.
In this paper, we demonstrate how unlikelihood can be used as a general framework by applying it to the dialogue domain. We show how varying the contexts  $\mathbf{x}$ , targets  $\mathbf{y}$ , candidates  $\mathcal{C}$  and scaling  $\beta$  can be used to improve the coherence and language modeling quality of dialogue models. To do this, we now consider the different biases we wish to mitigate, and construct a specific unlikelihood loss for each in turn.
2.1 Repetition and Copying
Generative dialogue models are known to both (i) rely too much on copying existing context knowledge or dialogue history; and (ii) repeat themselves within individual utterances. To address this with unlikelihood, we define two types of negative candidate tokens which either appear in a repeating n-gram from the context or from the generated label itself,
$$
\mathcal {C} _ {t} ^ {\text {c o n t e x t - c o p y}} = \left\{ \begin{array}{l l} \{y _ {t} \} & y _ {t} \in \text {r e p e a t c o n t e x t n - g r a m} \\ \emptyset & \text {o t h e r w i s e}, \end{array} \right.
$$
$$
\mathcal {C} _ {t} ^ {\text {l a b e l - r e p e a t}} = \left\{ \begin{array}{l l} \{y _ {t} \} & y _ {t} \in \text {r e p e a t i n g l a b e l n - g r a m} \\ \emptyset & \text {o t h e r w i s e ,} \end{array} \right.
$$
where  $y_{t}$  is a token in a repeating context n-gram when  $y_{t}$  is part of an n-gram that already appeared in the context tokens  $x$ , and is in a repeating label n-gram when  $y_{t}$  is part of an n-gram that already appeared in  $y_{< t}$ . Given a ground-truth context  $\mathbf{x}^{(i)}$ , we apply these two forms of unlikelihood to a model-generated sequence  $\hat{\mathbf{y}}^{(i)}$ . In summary, we either apply the per-example loss
$$
\mathcal {L} _ {\mathrm {U L}} ^ {(i)} (p _ {\theta}, \mathcal {C} _ {1: | y |} ^ {\text {c o n t e x t - c o p y}}, \mathbf {x} ^ {(i)}, \hat {\mathbf {y}} ^ {(i)})
$$
$\dagger$  https://parl.ai/projects/dialogue_unlikelihood/
for controlling context copies, or
$$
\mathcal {L} _ {\mathrm {U L}} ^ {(i)} (p _ {\theta}, \mathcal {C} _ {1: | y |} ^ {\mathrm {l a b e l - r e p e a t}}, \mathbf {x} ^ {(i)}, \hat {\mathbf {y}} ^ {(i)}).
$$
for controlling label repeats. We also consider mixing the two losses to mitigate both issues.
2.2 Vocabulary Usage
Neural sequence models trained with maximum likelihood generate sequences with token distributions that differ from those of human text (Dinan et al., 2020; Holtzman et al., 2019). In particular, these models tend to produce high frequency tokens too often and low frequency tokens too rarely, where frequency is defined by the human token distribution.
We address this with unlikelihood by penalizing tokens according to the mismatch between the model and ground-truth unigram distributions. Specifically, we first maintain an empirical estimate of the model's unigram distribution  $p_{\mathrm{model}}(y_t)$  and the human distribution  $p_*(y_t)$ :
$$
p _ {\text {m o d e l}} (y _ {t}) = \frac {\operatorname {c o u n t} (y _ {t})}{| Y |},
$$
where  $Y$  is a collection of token predictions on a subset of training data  $\mathcal{D}'$  (e.g. the preceding  $k = 256$  batches), and  $\mathrm{count}(y_t)$  is the number of occurrences of  $y_t$  in  $Y$ . This is computed using model sequences  $(\mathbf{y} = \hat{\mathbf{y}})$ , defining  $Y$  as the collection of all tokens in all  $\hat{\mathbf{y}}$ .
We wish to push down the probability of tokens appearing too often, i.e. when  $p_{\mathrm{model}}(y_t) > p_*(y_t)$ . For the unlikelihood loss, each step's candidate is thus the current token,  $C_t^{\mathrm{identity}} = \{y_t\}$ , and each token's unlikelihood loss is scaled according to the mismatch between the approximated model and human distributions,
$$
\beta (y _ {c}) = p _ {\mathrm {m o d e l} (y _ {c})} \log \left(\frac {p _ {\mathrm {m o d e l}} (y _ {c})}{p _ {*} (y _ {c})}\right).
$$
The unlikelihood loss for a token  $y_{c}$  is non-zero when the token occurs more often in the model's estimated unigram distribution. In summary, the resulting per-example loss is
$$
\mathcal {L} _ {\mathrm {U L}} ^ {(i)} (p _ {\theta}, \mathcal {C} _ {1: | y |} ^ {\mathrm {i d e n t i t y}}, \mathbf {x} ^ {(i)}, \mathbf {y})
$$
where  $\mathbf{y}$  is a model-generated sequence.
2.3 Contradictions
Neural generation models appear fluent, especially when pre-trained on large datasets, but are still poor at understanding the language they produce. That is, they can produce logically or factually inaccurate, or contradicting statements (Welleck et al., 2019b; Zhang et al., 2018; Hayashi et al., 2019; Petroni et al., 2019). Here, we show how the unlikelihood objective can be used to train such models to assign low probability to inconsistent and contradictory utterances.
To do so, we assume the existence of training data of both positive and negative examples of coherent behavior. There is a raft of recent large-scale, high quality data that can be massaged into this form, from natural language inference (NLI) tasks (Bowman et al., 2015; Williams et al., 2018; Welleck et al., 2019b) to commonsense reasoning tasks (Zellers et al., 2019; Qin et al., 2019). Two collections of data can be derived from the labels of such a supervised task:
$$
\mathcal {D} ^ {+} = \left\{\left(\mathbf {x} ^ {(i)}, \mathbf {y} ^ {(i) +}\right) \right\}, \quad \mathcal {D} ^ {-} = \left\{\left(\mathbf {x} ^ {(i)}, \mathbf {y} ^ {(i) -}\right) \right\},
$$
where  $\mathcal{D}^+$  is coherent behavior, e.g. neutral or entailing data in NLI, and  $\mathcal{D}^{-}$  is incoherent behavior, e.g. contradictions. In general, many forms of this type of data can be collected, not just NLI, and it is also not necessary for the contexts  $\mathbf{x}^{(i)}$  to overlap as we have written here.
Standard likelihood training can then be performed on coherent data  $\mathcal{D}^+$ , while the unlikelihood objective is applied to  $\mathcal{D}^-$  as we wish to push down the probability of generating the incoherent response  $\mathbf{y}^{-}$  given a context  $\mathbf{x}$ . That is, given an incoherent pair  $(\mathbf{x},\mathbf{y}^{-})$  we use the loss
$$
\mathcal {L} _ {\mathrm {U L}} (p _ {\theta}, \mathcal {C} _ {1: | y |} ^ {\text {i d e n t i t y}}, \mathbf {x}, \mathbf {y} ^ {-}),
$$
where we penalize each token in the target  $(\mathcal{C}_t^{\mathrm{identity}} = \{y_t^{-}\})$ . Hence, the loss makes generating the contradicting sentences less likely.
Our work provides new applications of unlikelihood training (Welleck et al., 2019a), showing that unlikelihood offers a general framework for improving generative models, and in particular dialogue models. Outside of that work, the use of negative training in dialogue retrieval, rather than generation, has been previously extensively studied, see e.g. (Humeau et al., 2019; Nugmanova
et al., 2019). In the area of generative dialogue, a number of works have focused on improving the standard likelihood training approach. Closer to our work is that of He and Glass (2019) which developed the approach of negative training to prevent generic and malicious responses in dialogue models. In terms of improving repetition and specificity, a recent alternative approach is that of control (Fan et al., 2018; Ficler and Goldberg, 2017; Ghazvininejad et al., 2017; See et al., 2019). Nucleus sampling (Holtzman et al., 2019) can help to remove generic or repetitive utterances at the expense of accuracy, but was shown to be inferior to beam blocking, which in turn was shown to be inferior to unlikelihood in Welleck et al. (2019a).
In terms of dialogue coherence, Welleck et al. (2019b) showed that retrieval, but not generative models, could be improved with NLI as a rescorer, while Yang et al. (2018) multi-tasked with NLI. The work of Gabriel et al. (2019) has also studied improving narrative flow with a discriminative rescorer, but in that case for generated language. In our work, the improvements are tightly integrated into the training of the model itself.
In all of our experiments we employ a large pre-trained seq2seq Transformer (Vaswani et al., 2017) as our base model, which we then fine-tune for particular tasks with the objectives outlined in Section 2 and specified in each experiment below. Following previous work (Humeau et al., 2019), we pre-train our model on dialogue data, using a previously existing Reddit dataset extracted and obtained by a third party and made available on pushshift.io, training to generate a comment conditioned on the full thread leading up to the comment, spanning  $\sim 2200M$  training examples. Our Transformer model consists of an 8 layer encoder, 8 layer decoder with 512-dimensional embeddings and 16 attention heads, and is based on the ParlAI implementation of Miller et al. (2017). The model was trained with a batch size of 3072 sequences for approximately 3M updates using a learning rate of 5e-4, and an inverse square root scheduler. This pre-training took approximately two weeks using 64 NVIDIA V100s.
4.1 Repetition and Copying
We use the ConvAI2 persona-based dialogue (Zhang et al., 2018), Wizard of Wikipedia
Model	PPL	F1	Repetition
Context	Label
Human	-	-	.0223	.0004
MLE Baseline	11.4	.199	.1131	.0210
UL (Context only)	11.8	.194	.0330	.0069
UL (Label only)	11.4	.203	.0984	.0005
UL (Context & Label)	11.9	.193	.0352	.0023
Table 1: Evaluation on the ConvAI2 task valid set (test set is hidden), comparing standard likelihood (MLE) with context and label repetition unlikelihood loss training. The repetition types can be decreased depending on which type of unlikelihood loss is used, with minimal changes in perplexity and F1.
Model	PPL	F1	Repetition
Context	Label
Human	-	-	.160	.001
MLE Baseline	8.3	.368	.441	.014
UL (Context only)	8.8	.346	.229	.037
UL (Label only)	8.3	.371	.426	.001
UL (Context + Label)	8.5	.358	.313	.009
Table 2: Evaluation on the Wizard of Wikipedia test set, comparing standard likelihood (MLE) with context and label repetition unlikelihood loss training. The repetition types can be decreased depending on the type of unlikelihood loss used, while minimally impacting F1.
knowledge-grounded dialogue (Dinan et al., 2019) and ELI5 long-form question answering (Fan et al., 2019) datasets to evaluate the effect of using unlikelihood to reduce copying and repetition in model generated utterances. On each dataset, we fine-tune the pre-trained pushshift.io Reddit model, then evaluate by generating next-utterances for dialogue contexts from the test set (or validation in ConvAI2, as the test set is hidden). We use greedy decoding in our main experiments for simplicity and scalability, but we also obtained similar results with beam search, shown in Appendix A.
To measure label repetition in a sequence  $\mathbf{y}$ , we use the portion of duplicate n-grams:
$$
1. 0 - \frac {\left| \text {u n i q u e} \mathrm {n} \text {- g r a m s} (\mathbf {y}) \right|}{\left| \mathrm {n} \text {- g r a m s} (\mathbf {y}) \right|},
$$
and report the metric averaged over the examples. Label repetition increases from zero as the model generates more repeated n-grams. To measure context repetition, we measure the fraction of gen
Model	PPL	F1	Repetition
Context	Label
Human	-	-	.009	.010
MLE Baseline	21.0	.130	.033	.617
UL (Context only)	21.4	.163	.008	.322
UL (Label only)	21.4	.183	.015	.055
UL (Context + Label)	21.8	.184	.009	.078
Table 3: Evaluation on the ELI5 task test set, comparing standard likelihood (MLE) with context and label repetition unlikelihood loss training. The repetition types can be decreased depending on which type of un-likelihood loss is used, while improving F1.
erated n-grams that appear in the original context:
$$
\frac {\left| \mathrm {n - g r a m s} (\mathbf {y}) \cap \mathrm {n - g r a m s} (\mathbf {x}) \right|}{\left| \mathrm {n - g r a m s} (\mathbf {y}) \right|},
$$
and report the metric averaged over the examples. Context repetition increases when the model 'copies' n-grams from the context. To quantify language modeling quality, we use standard perplexity and F1 metrics.
We use the pre-trained model fine-tuned with MLE as the baseline, and compare it against the pre-trained model fine-tuned with copy and repetition unlikelihood (§2.1).
Results Results for ConvAI2 are shown in Table 1. We see that training unlikelihood using only-contexts or only-labels reduces their corresponding metrics dramatically compared to the MLE baseline. Training with both context- and label-repetition unlikelihood reduced both context repetitions (by  $69\%$ , .0352 vs. .1131) and label repetitions (by  $89\%$ , .0023 vs. .0210) compared to the MLE baseline, much closer to human levels, while keeping perplexity essentially constant.
Comparatively, the Wizard of Wikipedia MLE baseline experiences a much larger problem with context repetition, due to its tendency to copy grounded knowledge verbatim (Table 2).
Results for ELI5, shown in Table 3, show that it has an especially large problem with label repetition, and that label-unliikelihood is able to reduce the repetitions by  $91\%$  (.055 vs .617), while significantly boosting F1 (.130 to .182).
Figures 2 and 3 show perplexity as a function of label and context repeats respectively using un-likelihood on ELI5. The parameter  $\alpha$  can clearly control repeats smoothly, with only very high values resulting in increased perplexity.
Figure 2: ELI5: Perplexity vs. label repeats as a function of  $\alpha$  in the label unlikelihood objective.
Figure 3: ELI5: Perplexity vs. context repeats as a function of  $\alpha$  in the context unlikelihood objective.
Human Evaluation Finally, we perform a human evaluation using the same pairwise evaluation scheme as (Fan et al., 2019) performed on ELI5, comparing the MLE baseline to UL (Label only) which asks: Which response answers the question better? The evaluators are asked to consider both the readability and accuracy of the answer. Results are given in Figure 4 (left), showing a statistically significant improvement over the baseline (150 trials, two tailed binomial test,  $p < 0.01$ ). Further details are given in Appendix C.
4.2 Vocabulary Usage
We evaluate the ability of vocabulary unlikelihood (§2.2) to reduce the mismatch between model and human token distributions.
We use the ConvAI2 dataset, where our baseline is again trained using maximum likelihood. Starting with the baseline model, we then fine-tune several models using vocab unlikelihood at logarithmically interpolated values of  $\alpha \in [1, 1000]$ .
We partition the vocabulary into 'frequent', 'medium', 'rare', and 'rarest' using the human
unigram distribution computed with the ConvAI2 training set, corresponding to the sorted token sets whose cumulative mass accounts for the top  $40\%$ , the next  $30\%$ , the next  $20\%$  and the final  $10\%$  of usage, respectively. We evaluate a model by generating utterances given contexts from the ConvAI2 validation set, and compute the fraction of tokens within each class.
Results Figure 5 shows how the vocabulary distribution obtained after unlikelihood training is affected by the choice of mixing hyperparameter  $\alpha$  (Eq. 1): it can smoothly transition between the human training distribution and the MLE trained distribution ('Baseline'), which is far from the human one.
Table 4 compares the MLE baseline with un-likelihood with increasing  $\alpha$  values in terms of distribution and F1 score. The vocabulary unlikelihood fine-tuning shifts probability mass from the over-represented frequent words towards underrepresented medium and rare words, with the effect strengthening as  $\alpha$  increases. At a small cost to perplexity and F1, the unlikelihood tuning reduced the overuse of common tokens by 9 points, matching the human rate, while improving the production of rare tokens by 3 percentage points.
Human Evaluation Finally, we perform a human evaluation using the ACUTE-EVAL framework (Li et al., 2019), comparing the MLE baseline to UL for various  $\alpha$ . First, 252 human-bot conversations (8 turns each) are collected, and then models are compared pairwise by asking the question: Who would you prefer to talk to for a long conversation? For these experiments we compare with both methods generating using beam with context blocking of trigrams. Results are given in Figure 4 (right), showing a statistically significant improvement over the baseline according to humans (two tailed binomial test,  $p < 0.01$ ). Further details are given in Appendix C.
4.3 Contradictions
We use the dialogue natural language inference (NLI) task of Welleck et al. (2019b) to obtain labeled non-contradicting and contradicting dialogue sentence pairs to use in unlikelihood training (§2.3). Dialogue NLI contains utterances labeled as entailing (E), neutral (N) or contradiction (C), given a premise that is either a persona sentence (an initial context sentence describing a dialogue agent's personality) or another dialogue utterance
Figure 4: Human evaluation experiments for label un-likelihood on ELI5 (left), and vocabulary unli-kelihood on ConvAI2 for two values of  $\alpha$  (right). Unlikelihood significantly outperforms the MLE baselines.