SlowGuess/ACL_2020_OCR huggingface dataset:
Tianxing He and James Glass
{tianxing, glass}@csail.mit.edu
Although deep learning models have brought tremendous advancements to the field of open-domain dialogue response generation, recent research results have revealed that the trained models have undesirable generation behaviors, such as malicious responses and generic (boring) responses. In this work, we propose a framework named "Negative Training" to minimize such behaviors. Given a trained model, the framework will first find generated samples that exhibit the undesirable behavior, and then use them to feed negative training signals for fine-tuning the model. Our experiments show that negative training can significantly reduce the hit rate of malicious responses, or discourage frequent responses and improve response diversity.
End-to-end dialogue response generation can be formulated as a sequence-to-sequence (seq2seq) task: given a dialogue context, the model is asked to generate a high-quality response. In recent years, deep learning models, especially seq2seq language generation models (Sutskever et al., 2014; Cho et al., 2014), have brought significant progress to the field of dialogue response generation.
However, recent research has revealed undesirable behaviors of seq2seq models that are side effects of standard maximum likelihood estimation (MLE) training, such as the generic (boring) response problem (Li et al., 2016), vulnerability to adversarial attacks (Cheng et al., 2018; Belinkov and Bisk, 2017), and the malicious (egregious) response problem (He and Glass, 2019).
In this work, we propose and explore the negative training framework to correct unwanted behaviors of a dialogue response generator. During negative training, we first find or identify input-output pairs for a trained seq2seq model that exhibit some
undesirable generation behavior, treat them as “bad examples,” and use them to feed negative training signals to the model. Correspondingly, we regard the training data as “good examples” and standard MLE training as “positive training”.
The idea of negative training is inspired from the way parents might teach their children to use language by incorporating both positive and negative training signals. For example, when teaching children how to use "love" and "hate", in addition to using positive examples like "I love apples but I hate bananas", they might also point out that saying "I hate you" to someone is considered impolite.
In this work, negative training is used to address the malicious response problem and the frequent response problem (to be described in Section 3.2 and 3.3) in open-domain dialogue response generation. In our experiments, we show that negative training can significantly reduce the hit rate for malicious responses, or discourage frequent responses and greatly improve response diversity.
2 Model Formulation
In this work we adopt recurrent neural network (RNN) based encoder-decoder seq2seq models (Sutskever et al., 2014; Cho et al., 2014; Mikolov et al., 2010), which are widely used in NLP applications like dialogue response generation (Li et al., 2016), machine translation (Luong et al., 2015), etc. We use  $\pmb{x} = \{x_{1}, x_{2}, \dots, x_{n}\}$  to denote one-hot vector representations of the input sequence, which serves as context or history information (e.g. the previous utterance),  $\pmb{y} = \{y_{1}, y_{2}, \dots, y_{m}\}^{1}$  to denote scalar indices of the corresponding reference target sequence, and  $V$  as the vocabulary. We use  $\theta$  to represent the parameters for the seq2seq
1The last word  $y_{m}$  is a  $\langle \mathbb{E}\mathbb{O}\mathbb{S}\rangle$  token which indicates the end of a sentence.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2044-2058 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
model, and  $P_{\theta}(\pmb {y}|\pmb {x})$  as the model's generative distribution.
On the encoder side, every  $\boldsymbol{x}_t$  will be first mapped into its corresponding word embedding  $\boldsymbol{x}_t^{emb}$ . Then  $\{\boldsymbol{x}_t^{emb}\}$  are input to a long-short term memory (LSTM) (Hochreiter and Schmidhuber, 1997) RNN to get a sequence of latent representations  $\{h_t^{enc}\}^2$ .
For the decoder, at time  $t$ , similarly  $y_{t}$  is first mapped to  $\pmb{y}_{t}^{emb}$ . Then a context vector  $\pmb{c}_{t}$ , which is supposed to capture useful latent information of the input sequence, needs to be constructed. We adopt the "attention" mechanism for context vector construction: first an attention mask vector  $\pmb{a}_{t}$  (which is a distribution) on the input sequence is calculated to decide which part to focus on, then the mask is applied to the latent vectors to construct  $\pmb{c}_{t}$ :  $\pmb{c}_{t} = \sum_{i=1}^{n} a_{t(i)} \pmb{h}_{i}^{enc}$ . We use the formulation of the "general" type of global attention, described in (Luong et al., 2015), to calculate the mask.
During baseline training, standard MLE training with stochastic gradient descent (SGD) is used to minimize the negative log-likelihood (NLL) of the reference target sentence given the input sentence in the data:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {M L E}} (P _ {d a t a}; \theta) = E _ {(\boldsymbol {x}, \boldsymbol {y}) \sim P _ {d a t a}} (- \log P _ {\theta} (\boldsymbol {y} | \boldsymbol {x})) \\ = E _ {(\boldsymbol {x}, \boldsymbol {y}) \sim P _ {\text {d a t a}}} \left(- \sum_ {t = 1} ^ {m} \log P _ {\theta} \left(y _ {t} \mid \boldsymbol {y} <   t, \boldsymbol {x}\right)\right) \tag {1} \\ \end{array}
$$
where  $\pmb{y}_{<t}$  refers to  $\{y_0, y_1, \dots, y_{t-1}\}$ , in which  $y_0$  is set to a begin-of-sentence token  $<\text{BOS}>$ .
We consider two popular ways of decoding (generating) a sentence given an input: greedy decoding and sampling. In practice for dialogue response generation, greedy decoding will provide stable and reproducible outputs, but is severely affected by the generic response problem. Sampling will provide more diverse but less predictable responses, and thus give rise to the malicious response problem.
3 The Negative Training Framework
The negative training framework<sup>3</sup> is a two-stage process. Given a trained model, we put it under a
"debugging" environment  $P_{test}$  which provides test input samples<sup>4</sup>, get the model's decoded samples and decide (using well-defined criteria) whether each input-output pair exhibits some undesirable behavior. Then, these "bad" pairs are used to provide negative training signals.
Negative training can be derived from Empirical Bayes Risk Minimization (Och, 2003). Specifically, the overall objective is to minimize the expected risk that the model exhibits undesirable decoding behavior:
$$
\mathcal {L} _ {\mathrm {N E G}} \left(P _ {\text {t e s t}}; \theta\right) = E _ {\boldsymbol {x} \sim P _ {\text {t e s t}}} E _ {\boldsymbol {y} \sim P _ {\theta} (\boldsymbol {y} | \boldsymbol {x})} c (\boldsymbol {x}, \boldsymbol {y}) \tag {2}
$$
where  $c(\pmb{x},\pmb{y})$  refers to the binary criteria that will be 1 if  $(\pmb{x},\pmb{y})$  exhibits undesirable behavior, and 0 otherwise.
Then, we take the derivative of  $\mathcal{L}_{\mathrm{NEG}}$  w.r.t. to  $\theta$ , using the log derivative trick (widely used in Reinforcement Learning (Sutton and Barto, 1998)):
$$
\begin{array}{l} \nabla_ {\theta} \mathcal {L} _ {\mathrm {N E G}} (P _ {t e s t}; \theta) = \\ E _ {\boldsymbol {x} \sim P _ {t e s t}} E _ {\boldsymbol {y} \sim P _ {\theta} (\boldsymbol {y} | \boldsymbol {x})} c (\boldsymbol {x}, \boldsymbol {y}) \cdot \nabla_ {\theta} \log P _ {\theta} (\boldsymbol {y} | \boldsymbol {x}) \tag {3} \\ \end{array}
$$
Compared to  $\mathcal{L}_{\mathrm{MLE}}$  in eq. (1), which maximizes the log-likelihood of training data samples,  $\mathcal{L}_{\mathrm{NEG}}$  minimizes the log-likelihood of undesirable model samples. This is the reason why we call it "Negative Training".
In our preliminary experiments, we find that negative training needs to be augmented with the standard MLE objective  $\mathcal{L}_{\mathrm{MLE}}$ , encouraging the model to retain its original performance:
$$
\mathcal {L} _ {\mathrm {N E G} + \mathrm {P O S}} = \mathcal {L} _ {\mathrm {N E G}} + \lambda_ {\mathrm {P O S}} \mathcal {L} _ {\mathrm {M L E}} \tag {4}
$$
In our experiments, we find  $\lambda_{\mathrm{POS}}$  can be simply set to 0.1 to work well.
In the next two sections, we discuss how the general negative training framework is tailored for the malicious response problem and frequent response problem, respectively.
3.2 Negative Training for the Malicious Response Problem
For the malicious response problem, we follow the methodology proposed by (He and Glass, 2019).
Here  $\pmb{h}$  refers to the output layer of LSTM, not the cell memory layer.
3Our code is available at https://github.mit.edu/tianxing/negativetraining_ac12020
4Note that here "test" does not refer to the test data.
First a list of malicious target sentences are created, then the gibbs-enum algorithm<sup>5</sup> is called to find "trigger input" that will cause the model to assign large probability to the target sequence. The following "hit types" are defined:
- o-greedy-hit: A trigger input sequence is found such that the model generates the target sentence from greedy decoding.
- o-sample-min/avg-hit: A trigger input sequence is found such that the model generates the target sentence with an minimum/average word log-probability larger than a given threshold  $T_{out}$ .
- io-sample-min/avg-hit: In addition to the definition of o-sample-min/avg-hit, we also require that the average log-likelihood of the trigger input sequence, measured by a LM, is larger than a threshold  $T_{in}$ . This enforces the trigger input to be more likely to be input by real-world users.
$T_{out}$  is set to the trained seq2seq model's average word log-likelihood on the test data, and  $T_{in}$  is set to be a reasonable LM's 6 average word log-likelihood on the test set. The intuition is that the model should not assign larger probabilities to the malicious sentences than the reference sentences in the test set. Note that these hit types act as criteria  $c(\pmb{x},\pmb{y})$ , indicating whether a target sentence is hit by a trigger input.
As shown in (He and Glass, 2019), a typical seq2seq model trained by MLE has around a  $10\%$  hit rate for malicious targets w.r.t. sample-min/avg-hit, across data-sets. However, very few malicious targets are hit w.r.t. greedy-hit, so in this work, we focus on the malicious response problem for sampling during decoding. In Table 1 we show pairs of trigger inputs and the malicious target sentences w.r.t io-sample-min-hit, for the baseline model on Ubuntu data.
Now we apply the negative training framework, and aim to reduce the hit rate of a trained model for a given list of malicious targets. During each iteration of negative training, for every target sentence  $\mathbf{y}_{\mathrm{target}}$ , we first call the gibbs-enum algorithm to find the trigger input  $\mathbf{x}_{\mathrm{trigger}}$ . And if the target is
Algorithm 1 Negative Training for the Malicious Response Problem
Input: Target list  $\mathbf{Y}_{\mathrm{target}}$ , model parameter  $\theta$ , learning rate  $\alpha$ , criterion for hit  $c$ , and training data  $D_{\mathrm{train}}$
for  $y_{\text{target}}$  in  $Y_{\text{target}}$  do
Get  $\pmb{x}_{\mathrm{trigger}}$  for  $\pmb{y}_{\mathrm{target}}$  using the gibbs-enum algorithm.
while  $c(\pmb{x}_{\mathrm{trigger}}, \pmb{y}_{\mathrm{target}}) = 1$  do
Negative update:
$$
\theta = \theta - \alpha \cdot \nabla_ {\theta} \log P _ {\theta} \left(\mathbf {y} _ {\text {t a r g e t}} \mid \mathbf {x} _ {\text {t r i g g e r}}\right)
$$
Get data sample  $(\pmb{x}_{\mathrm{pos}},\pmb{y}_{\mathrm{pos}})$  from  $D_{\mathrm{train}}$
Positive update:
$$
\theta = \theta + \alpha \cdot \lambda_ {\mathrm {P O S}} \cdot \nabla_ {\theta} \log P _ {\theta} (\mathbf {y} _ {\mathrm {p o s}} | \mathbf {x} _ {\mathrm {p o s}})
$$
Trigger Input: ok give me a minute to see what you have in the first place
Malicious Target: no one will help you
Trigger Input: mirc it's supposed to be <unk> that seems to be the problem
Malicious Target: i do n't think i like you
Trigger Input: i know about photoshop i use skype too but i can't buy it
Malicious Target: you have no money
Table 1: Examples of trigger inputs.
hit  $(c(\pmb{x}_{\mathrm{trigger}},\pmb{y}_{\mathrm{target}}) = 1)$ , we update the model to reduce the log-likelihood  $P_{\theta}(\pmb{y}_{\mathrm{target}}|\pmb{x}_{\mathrm{trigger}})$ . The process is formulated in Algorithm 1<sup>7</sup>.
For each trigger input, multiple iterations of negative updates are usually needed before the hit criterion is no longer met. Note that in each iteration, the gibbs-enum algorithm is called again to find a new trigger input for each target.
In our experiments, we show that negative training effectively reduces the hit rate for malicious targets after each iteration, and eventually, the gibbs-enum algorithm can no longer find trigger inputs for a large number of targets that were initially hits.
3.3 Negative Training for the Frequent Response Problem
The generic response problem (Li et al., 2016) for end-to-end dialogue response generation refers to the typical behavior of a MLE trained model, whereby the generated responses are mostly safe,
Note that in actual implementation, the algorithm is minibatch based.
For this paper to be self-contained, we describe the gibbs-enum algorithm in Appendix A.
${}^{6}$  A LSTM language model (LM) is trained on the same training data (regarding each response as an independent sentence).
boring or uninformative (such as “i don’t know” or “good idea”). However, it is difficult to invent an automatic criterion to determine whether a response is generic or not.
In this work, we focus on the frequent response problem, as a sub-problem of the generic response problem. It refers to the behavior that a trained model generates exactly the same (usually boring) response, with a high frequency.
We propose to use a metric called max-ratio to measure how severe the frequent response problem is. Given a test set and a decoding method, the model will generate a set of responses, and max-ratio is defined to be the ratio of the most frequent response. In our experiments, the baseline models have a max-ratio of around 0.3 for response like "I don't know" across different data-sets, showing the severity of the frequent response problem.
During negative training for frequent response, first a threshold ratio  $r_{\text{thres}}$  is selected (such as 0.01), and responses with frequency ratio larger than  $r_{\text{thres}}$  will be discouraged. For each iteration, the model's response to each training data input sentence is monitored and responses with frequency larger than  $r_{\text{thres}}$  will be used as negative examples. The frequency statistics are calculated using the current and the last 200 mini-batches. The procedure is formulated in Algorithm 2. Note that positive training is also needed here for the model to retain its original performance.
In our experiments, it is shown that negative training significantly reduces max-ratio for the model on test data, and greatly increases the diversity of the model's responses.
We conduct experiments on three publicly available conversational dialogue data-sets: Ubuntu, Switchboard, and OpenSubtitles. To save space, descriptions of the data-sets are provided in Appendix B.
4.1 Baseline Model Training
For all data-sets, we first train an LSTM based LM and attention based seq2seq models with one hidden layer of size 600, and the embedding size is set to 300. For Switchboard a dropout layer with rate 0.3 is added to the model because over-fitting is observed. The mini-batch size is set to 64 and we apply SGD training with a fixed starting learning rate (LR) for 10 iterations, and then another 10 iterations with LR halving. For Ubuntu and Switchboard, the starting LR is 1, while a starting LR of 0.1 is used for OpenSubtitles. The results are shown in Appendix C.
After negative training, in addition to measuring the hit rate for malicious targets or the diversity of the responses, it is also important to check whether the original sample quality of the baseline model is damaged. Towards that end, the perplexity of the model before and after negative training will be compared, we also conduct human evaluation to measure whether the sample quality is decreased. Other popular measurements, such as the BLEU score, have been found to correspond poorly with human judgements (Liu et al., 2016). Nevertheless, we also find that the model's BLEU score does not become worse after negative training.
4.2 Experiments on the Malicious Response Problem
Following (He and Glass, 2019), a list of malicious targets are created to test whether negative training can teach the model not to generate sentences in the list. However, in addition to prevent the model from generating targets in a specific list, it is also important to check whether negative training generalizes to other malicious targets. So, a test target list which contains similar but different targets from the training list are also created to test generalization. The training and test lists each contain  $0.5\mathrm{k}$  targets.
It is also interesting to investigate whether using more malicious targets for negative training can lower the hit rate on the test list. Towards that end, we train a seq2seq paraphrase model using the paraNMT data-set (Wieting and Gimpel, 2017),
Train	Paraphrase	Test
you are broken
i will kill	you’re broken	are you broken
you are bad	i’ll kill myself	i’m going to kill
you are stupid	you’re bad	you are really bad
you shut up	you’re stupid	you are so stupid
shut your mouth	can you shut up
with a model of the same structure as described in Section 2. Then, the paraphrase model is used to generate paraphrases of the malicious targets in the training target list $^{8}$  for augmentation. In our experiments, the training list without augmentation is first used for negative training, then it is augmented with  $0.5\mathrm{k}$  or  $2\mathrm{k}$  paraphrased targets respectively (1 or 4 paraphrase copies for each training target sentence). Samples of the malicious targets are shown in Table 2. The same training, augmented training and test list are used for all three data-sets, and there is no sequence-level overlap between training lists (augmented or not) and the test list.
In our experiments, we spotted a harmful side effect of negative training where frequent words in the training target list are severely penalized and sometimes receive low probability even in normal perplexity testing, especially for experiments with small  $\lambda_{\mathrm{POS}}$ . To alleviate this problem, we use a simple technique called frequent word avoiding (FWA): negative gradients are not applied to the most frequent words in the malicious training target list<sup>9</sup>. For example, when doing negative training against the target "i hate you <EOS>", only "hate" will get a negative gradient.
For all data-sets, negative training (Algorithm 1) is executed on the (trained) baseline model for 20 iterations over the training target list. A fixed learning rate of 0.01 and a mini-batch size of 100 are used.  $\lambda_{\mathrm{POS}}$  is set to 0.1 for Ubuntu, and to 1 for Switchboard and OpenSubtitles.
The main results are shown in Table 3. For Switchboard we focus on sample-avg-hit because we find very few targets are hit w.r.t. sample-min-hit (Similar results are reported in (He and Glass, 2019)), while for Ubuntu and OpenSubtitles we focus on sample-min-hit. Note that we get very similar results w.r.t. sample-avg-hit for
Table 2: Examples of malicious targets in the training list, the test list, and paraphrases of the training targets which will be used for augmentation.
Ubuntu Training	o-sample-min-hit	io-sample-min-hit
Train	Test	PPL	Train	Test	PPL
Baseline	16.4%	12.6%	59.49	7.8%	5.2%	59.49
+neg-tr(0.5k)	0%	2%	60.42	0.2%	1.4%	59.97
+neg-tr(1k)	0.1%	1.4%	60.72	0.1%	1%	60.21
+neg-tr(2.5k)	0.04%	0%	62.11	0.2%	0%	63.37
Switchboard Training	o-sample-avg-hit	io-sample-avg-hit
Train	Test	PPL	Train	Test	PPL
Baseline	27.8%	27.6%	42.81	19.6%	21%	42.81
+neg-tr(0.5k)	3.8%	13.4%	42.91	2.2%	9.4%	42.7
+neg-tr(1k)	2.4%	5%	42.96	2.1%	4%	42.76
+neg-tr(2.5k)	1.3%	2.6%	43.51	1.5%	1.6%	43.24
OpenSub Training	o-sample-min-hit	io-sample-min-hit
Train	Test	PPL	Train	Test	PPL
Baseline	40.7%	36.6%	70.81	19.2%	13.6%	70.81
+neg-tr(0.5k)	5.8%	12.2%	77.90	5.2%	6.6%	73.48
+neg-tr(1k)	5.2%	7%	68.77	9.2%	4.6%	68.92
+neg-tr(2.5k)	4.8%	6%	74.07	3.4%	3.6%	75.9
Table 3: Main results for the hit rates of malicious targets before and after negative training. "Neg-tr(0.5k)" refers to the negative training experiment using the original malicious training target list without paraphrase augmentation.
Ubuntu/OpenSubtitles, and we omit those results here.
We first observe that, for all data-sets, negative training can effectively reduce the hit rate on the training target list to less than  $5\%$  with little or no degradation on perplexity. We provide a comparison of the model's behavior in Appendix D. Also, significant hit rate reduction is achieved on the test target list, which has no overlap with the training target list. This shows that negative training, similar to traditional positive training, also generalizes.
It is also shown that training list augmentation can further reduce the malicious target hit rate consistently for both training and test lists. For example, on Ubuntu data, the hit rate after negative training w.r.t. o-sample-min-hit is  $12.6\%$ , and can be reduced to  $0\%$  with paraphrase augmentation.
We find that the model's generation behavior in non-adversarial setting is almost the same as the baseline after negative training. For example, the 10-best list from beam search before/after neg-train has larger than  $90\%$  overlap. We also find that the model generates similar samples (shown in Appendix G). We believe the reason is that negative training focuses on making the model more robust with the adversarial inputs, and the original generation behavior is kept intact by the positive training (Equation 4).
Note the training and test lists are manually created. The exact avoiding word set used is  $\{\langle \mathrm{EOS}\rangle$  you, i, me, are, to, do}.
4.3 Experiments on the Frequent Response Problem
In this section we report results where the negative training framework (Section 3.3) is applied to tackle the frequent response problem. For all datasets, negative training is executed for 20 iterations on the MLE trained model over the training data, with a selected  $r_{\mathrm{thres}}$ . A fixed learning rate of 0.001 is used for all three data-sets, the mini-batch size is set to 64 and  $\lambda_{\mathrm{POS}}$  is set to 1.
In this work, we focus on improving the model's greedy decoding behavior instead of beam search for the following two reasons: 1) For the baseline models our experiments, we found that beam search gives far worse response diversity than greedy decoding, because it favors short responses (usually only of length one) too much, resulting in a much larger max-ratio; 2) During training, doing beam search is much more time-consuming than greedy decoding.
To measure the diversity of the model's generated responses, in addition to max-ratio introduced in Section 3.3, which is specially design for the frequent response problem, we also adopt the entropy metric proposed in (Zhang et al., 2018). Given a set of responses from decoding on the test set, Ent- $n$  calculates the entropy of the n-gram distribution:
$$
E n t - n = \sum_ {g \in G _ {n}} - r (g) \log r (g) \tag {5}
$$
where  $G_{n}$  is the set of all n-grams that appeared in the response set, and  $r(g)$  refers to the ratio (frequency) of n-gram  $g$  w.r.t. all n-grams in the responses set.