SlowGuess/ACL_2020_OCR huggingface dataset:
In our experiments with negative training, a harmful side-effect is spotted: during decoding, the model tends to output long and ungrammatical responses such as "i do n't know if it's a real valid deterrent crime crime yeah i 'm satisfied trying not to". We believe the reason is that the sentence end token  $<\mathrm{EOS}>$  gets over penalized during negative training (it appears in every negative example). So, we apply the same frequent word avoiding (FWA) technique used in Section 4.2, except that here only the negative gradient for  $<\mathrm{EOS}>$  is scaled by  $0.1^{10}$ .
In addition to the baseline model, we compare our proposed negative training framework against a
Ubuntu	r_thres	PPL	M-ratio	E-2	E-3
Test-set	N/A	N/A	1.1%	10.09	11.32
Baseline	N/A	59.49	4.4%	5.33	5.92
+GAN	N/A	59.43	4.7%	5.30	5.87
+MMI	N/A	N/A	4.5%	5.34	5.93
+neg-train	1%	59.76	1.2%	5.74	6.52
+neg-train	0.1%	60.06	1.3%	6.44	7.55
Switchboard	r_thres	PPL	M-ratio	E-2	E-3
Test-set	N/A	N/A	10.0%	8.61	9.65
Baseline	N/A	42.81	37.4%	2.71	2.42
+GAN	N/A	42.69	49%	2.66	2.35
+MMI	N/A	N/A	23%	5.48	6.23
+neg-train	10%	42.84	12.4%	3.86	4.00
+neg-train	1%	44.32	9.8%	5.48	6.03
OpenSubtitles	r_thres	PPL	M-ratio	E-2	E-3
Test-set	N/A	N/A	0.47%	9.66	10.98
Baseline	N/A	70.81	20%	4.22	4.59
+GAN	N/A	72.00	18.8%	4.08	4.43
+MMI	N/A	N/A	3.6%	7.63	9.08
+neg-train	1%	72.37	3.1%	5.68	6.60
+neg-train	0.1%	75.71	0.6%	6.90	8.13
Table 4: Main results of negative training with different  $r_{\text{thres}}$ , for the frequent response problem. Diversity metrics for the responses in the test data are also shown, "E-n"/"M-ratio" refer to the Ent-n/max-ratio metric.
GAN (Goodfellow et al., 2014a) approach, where a discriminator  $D$  is introduced and the generator  $G$  tries to fool the discriminator to believe its samples are real data samples:
$$
\begin{array}{l} \min  _ {G} \max  _ {D} V (D, G) \\ = \min  _ {G} \max  _ {D} \left\{E _ {\left(\boldsymbol {x}, \boldsymbol {y}\right) \sim P _ {d a t a}} \log D (\boldsymbol {x}, \boldsymbol {y}) + \right. \tag {6} \\ E _ {\boldsymbol {x} \sim P _ {d a t a}, \boldsymbol {y} \sim G (\cdot | \boldsymbol {x})} \log (1 - D (\boldsymbol {x}, \boldsymbol {y})) \} \\ \end{array}
$$
where the generator  $G$  refers to the seq2seq model  $P_{\theta}$ . The GAN framework is very attractive for tackling the generic response problem (Li et al., 2017; Zhang et al., 2018), because the discriminator can act as a critic to judge whether a response sample is boring. We describe the training details and hyper-parameter setting for the GAN approach in Appendix E.
We also provide an comparison to the MMI decoding (Li et al., 2016), which is a very popular work in this field. We implement MMI-antiLM for our models.
The experimental results are shown in Table 4. The experiment with best diversity result and nondegenerate sample quality are shown in bold. We first observe a large gap on the diversity measures between the baseline models and the test set, especially on Switchboard and OpenSubtitles data.
10We find that scal by zero will result in extremely short responses.
Switchboard	OpenSubtitles
Input: it ’ll cost about three hundred dollars for a stud	Input: captain you wanted to see me
Baseline: i think that ’s a good idea	Baseline: i ’m sorry
Neg-train: i think i would agree with that	Neg-train: i was in the hotel
Input: we want to breed her with a champion	Input: yes mr. brown could i
Baseline: i do n’t know	Baseline: i do n’t know
Neg-train: i think it was	Neg-train: i ’d like to introduce myself
Input: now these are long-haired	Input: leave it to me
Baseline: i do n’t know	Baseline: i ’m not going to leave you
Neg-train: i ’ve been in a very very good shape	Neg-train: you ’re taking the first step
Input: the other two are short hairs	Input: thank you mr. brown
Baseline: i do n’t know	Baseline: i ’m sorry
Neg-train: i ’m going to try to get it	Neg-train: i ’m happy to see you
Table 5: Greedy-decoding samples on the test data before and after negative training. The samples are consecutive (input of the next sample is the reference response for the previous one).
That indicates the severity of the frequent/generic response problem. Then, results of negative training with different  $r_{\mathrm{thres}}$  show that negative training can significantly increase response diversity, with little or no loss on PPL or BLEU score (shown in Appendix F) performance. For example, max-ratio is reduced by  $73.7\%$  and Ent-3 is increased by  $149\%$  for Switchboard data. Further, consistent improvement is achieved when a smaller  $r_{\mathrm{thres}}$  is used. However, sample quality will decrease (becoming too long or ungrammatical) when  $r_{\mathrm{thres}}$  is too small. The reason could be that when too much diversity is asked for, the model will go to extremes to provide diversity, resulting in degradation of sample quality.
Comparing to MMI, note that although on Switchboard/Opensubtitles MMI gives higher entropy, the max-ratio is not as low as the negative training result, which is the main focus of our work (the frequent response problem). We also find MMIs hyper-parameters are difficult to tune: the working set of hyper-parameters dont transfer well between data-sets. Further, for MMI in a lot of configuration tries the model gives ungrammatical output samples (this is problem is also mentioned in the paper (Li et al., 2016)). For the Ubuntu data, we can not even find a configuration that performs better than the baseline model.
Further, the vanilla GAN approach is not shown to be effective in our experiments. The reason could be that despite its discriminative nature, GAN training still feeds "positive" gradient for samples from the model (eq. (11) and eq. (12) in Appendix
E), which is not enough to prevent the model from generating them. We believe additional techniques (Zhang et al., 2018; Li et al., 2017) are needed for the GAN approach to be effective.
We show some model samples before and after negative training in Table 5. It is shown that negative training effectively discourages boring responses, and response diversity is improved. However, one limitation is observed that diversity does not necessarily lead to improvement on the informativeness of the response w.r.t. the input (sometimes the model generates a completely unrelated response). More samples for all three data-sets are included in Appendix G.
To rigorously verify negative training is not getting diversity when sacrificing the sample's quality, a human evaluation is conducted and results are shown in Table 6. It is observed that negative training wins by a significant margin for all three data-sets. This shows that, negative training does not damage the quality of the generated samples. Note that the human evaluation does not reflect the diversity of the model, because the raters only rate one response at a time.
The malicious response problem and the gibbs-enum algorithm to find trigger inputs (He and Glass, 2019) originates from a large body of work on adversarial attacks for deep learning models, with continuous input space (e.g. image classification) (Goodfellow et al., 2014b; Szegedy et al., 2013), or discrete input space (e.g. sentence classification, or
Data-set	Tie	Baseline	Neg-train
Ubuntu	64.6%	14.0%	21.3%
Switchboard	45.1%	18.3%	36.4%
Opensubtitles	58.3%	19.0%	22.6%
Table 6: Human Evaluation Results. For each dataset, 300 samples (input-output pairs) from the baseline model and the model after negative training, are evenly distributed to 4 English-speaking human evaluators. The evaluators are asked to pick a preferred sample, or report a tie. This evaluation is to check whether negative training has hampered the quality of the generation.
seq2seq models) (Papernot et al., 2016; Samanta and Mehta, 2017; Liang et al., 2018; Ebrahimi et al., 2017; Belinkov and Bisk, 2017; Chen et al., 2017). "Adversarial attacks" refer to the phenomenon that when an imperceptible perturbation is applied to the input, the output of the model can change significantly (from correct to incorrect). The trigger inputs found by the gibbs-enum algorithm, can be regarded as a type of "targeted attack", in which the attack triggers the model to assign large probability to a specific malicious target sentence.
Motivated by the works on adversarial attacks, various adversarial training strategies (Madry et al., 2017; Belinkov and Bisk, 2017; Miyato et al., 2016) have been proposed to make trained models more robust against those attacks. During adversarial training, the model is fed with adversarial examples and the correct labels. The negative training framework considered in this work differs from adversarial training in that, instead of asking the model to "do the right thing" (referred to as "positive training" in this work), the model is trained to "not do the wrong thing". To the best of our knowledge, this is the first work investigating the concept of negative training for dialogue response models, and the first proposed solution for the malicious response problem.
The malicious target list used in this work is very similar to the one used in (He and Glass, 2019). We propose to add a test target list to test the generalization of negative training. Further, we show that the training list can be effectively augmented by utilizing a paraphrase model.
In this work, we propose a definition for the frequent response problem, as a sub-problem of the generic response problem (Li et al., 2016). Much research work has devoted to alleviate the generic response problem in end-to-end dialogue response
generation, (Li et al., 2016) use the maximal mutual information (MMI) objective, and propose to utilize an auxiliary LM to penalize the generic response during decoding. Closely related to this work, sophisticated training frameworks based on GAN (Zhang et al., 2018; Li et al., 2017) have also been shown to be effective, where techniques such as variational information maximization or reward for every generation step (REGS) are proposed to improve GAN training. However, in our experiments it is shown that a vanilla GAN approach gives unsatisfactory results. Whether negative training $^{11}$  is complementary to these frameworks is worth investigating in future work.
Finally, note that the concept of negative training in this work is very different to the negative samples in word2vec training (Mikolov et al., 2013). The negative samples in word2vec training are used to prevent the training from being trivial, and is usually chosen randomly. In this work, the negative samples are carefully chosen to exhibit some particular undesirable behavior of the model, and is then used to correct such behavior.
In this work, we propose the negative training framework to correct undesirable behaviors of a trained neural dialogue response generator. The algorithm involves two major steps, first input-output pairs that exhibit bad behavior are identified, and then are used for fine-tuning the model as negative training examples. We also show that negative training can be derived from an overall objective (eq. (2)) to minimize the expected risk of undesirable behaviors. In our experiments, we apply negative training to the malicious response problem and the frequent response problem and get significant improvement for both problems.
Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. CoRR, abs/1711.02173.
Hongge Chen, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, and Cho-Jui Hsieh. 2017. Show-and-fool: Crafting adversarial examples for neural image captioning. CoRR, abs/1712.02051.
Minhao Cheng, Jinfeng Yi, Huan Zhang, Pin-Yu Chen, and Cho-Jui Hsieh. 2018. Seq2sick: Evaluating the
<sup>11</sup>Note that negative training is considerably easier to implement than the mentioned frameworks based on GAN.
robustness of sequence-to-sequence models with adversarial examples. CoRR, abs/1803.01128.
Kyunghyun Cho, Bart van Merrienboer, Calar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2017. Hotflip: White-box adversarial examples for NLP. CoRR, abs/1712.06751.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014a. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS'14, pages 2672-2680, Cambridge, MA, USA. MIT Press.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014b. Explaining and harnessing adversarial examples. CoRR, abs/1412.6572.
Tianxing He and James Glass. 2019. Detecting egregious responses in neural sequence-to-sequence models. In International Conference on Learning Representations.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746-1751.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In *NAACL HLT* 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 110-119.
Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. CoRR, abs/1701.06547.
Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018. Deep text classification can be fooled. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden., pages 4208-4215.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132. Association for Computational Linguistics.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. CoRR, abs/1506.08909.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421. Association for Computational Linguistics.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. CoRR, abs/1706.06083.
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pages 1045-1048.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS'13, pages 3111-3119, USA. Curran Associates Inc.
Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. 2016. Adversarial training methods for semi-supervised text classification. Cite arxiv:1605.07725Comment: Published as a conference paper at ICLR 2017.
Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL '03, Stroudsburg, PA, USA. Association for Computational Linguistics.
Nicolas Papernot, Patrick D. McDaniel, Ananthram Swami, and Richard E. Harang. 2016. Crafting adversarial input sequences for recurrent neural networks. In 2016 IEEE Military Communications Conference, MILCOM 2016, Baltimore, MD, USA, November 1-3, 2016, pages 49-54.
Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. CoRR, abs/1707.02812.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. CoRR, abs/1505.00387.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
Richard S. Sutton and Andrew G. Barto. 1998. Introduction to Reinforcement Learning, 1st edition. MIT Press, Cambridge, MA, USA.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. CoRR, abs/1312.6199.
Jörg Tiedemann. 2009. News from OPUS - A collection of multilingual parallel corpora with tools and interfaces. In N. Nicolov, K. Bontcheva, G. Angelova, and R. Mitkov, editors, *Recent Advances in Natural Language Processing*, volume V, pages 237-248. John Benjamins, Amsterdam/Philadelphia, Borovets, Bulgaria.
John Wieting and Kevin Gimpel. 2017. Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. CoRR, abs/1711.05732.
Lijun Wu, Yingce Xia, Li Zhao, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2017. Adversarial neural machine translation. CoRR, abs/1704.06933.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2016. Seqgan: Sequence generative adversarial nets with policy gradient. CoRR, abs/1609.05473.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018. Generating informative and diverse conversational responses via adversarial information maximization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 1815-1825. Curran Associates, Inc.
A The Gibbs-enum Algorithm for Finding Trigger Inputs
In this section, we briefly describe the gibbs-enum algorithm, we also refer readers to (He and Glass, 2019) for the intuition and full development of the algorithm. The goal of gibbs-enum is that given a (malicious) target sentence  $\mathbf{y}$  of length  $m$ , and a trained seq2seq model, we aim to find a trigger input sequence  $\mathbf{x}$ , which is a sequence of one-hot vectors  $\{\mathbf{x}_t\}$  of length  $n$ , to minimize the negative log-likelihood (NLL) that the model will generate  $\mathbf{y}$ . We formulate our objective function  $L(\mathbf{x}; \mathbf{y})$  below:
$$
L (\boldsymbol {x}; \boldsymbol {y}) = - \frac {1}{m} \sum_ {t = 1} ^ {m} \log P _ {\text {s e q 2 s e q}} \left(y _ {t} \mid \boldsymbol {y} _ {<   t}, \boldsymbol {x}\right) + \lambda_ {\text {i n}} R (\boldsymbol {x}) \tag {7}
$$
A regularization term  $R(x)$  is applied when looking for io-sample-min/avg-hit, which is the LM score of  $x$ :
$$
R (\boldsymbol {x}) = - \frac {1}{n} \sum_ {t = 1} ^ {n} \log P _ {L M} (x _ {t} | \boldsymbol {x} _ {<   t}) \tag {8}
$$
In our experiments we set  $\lambda_{in}$  to 1 when searching for io-sample-min/avg-hit, otherwise 0.
During gibbs-enum, every time we focus on a single index slot  $\boldsymbol{x}_t$ , and find the best one-hot  $\boldsymbol{x}_t$  while keeping the other parts of  $\boldsymbol{x}$  fixed:
$$
\underset {\boldsymbol {x} _ {t}} {\arg \min } L (\boldsymbol {x} _ {<   t}, \boldsymbol {x} _ {t}, \boldsymbol {x} _ {> t}; \boldsymbol {y}) \tag {9}
$$
Since the size of vocabulary  $|V|$  is finite, it is possible to try all of them and get the best local  $\boldsymbol{x}_t$ . But it is still costly since each try requires a forwarding call to the neural seq2seq model. To address this, gradient information is utilized to narrow the range of search. We temporarily regard  $\boldsymbol{x}_t$  as a continuous vector and calculate the gradient of the negated loss function with respect to it:
$$
\nabla_ {\boldsymbol {x} _ {t}} \left(- L \left(\boldsymbol {x} _ {<   t}, \boldsymbol {x} _ {t}, \boldsymbol {x} _ {> t}; \boldsymbol {y}\right)\right) \tag {10}
$$
Then, we try only the  $G$  indexes that have the highest value on the gradient vector. The procedure is formulated in Algorithm 3.
For hyper-parameters of gibbs-enum,  $T$  (the maximum number of sweeps) is set to 5,  $G$  (size of the set of indices for enumeration during each update) is set to 100, the algorithm is run 5 times with different random initializations and the trigger input with the best loss is returned. Note that larger hyper-parameters can give slightly higher hit rates, but will be more time-consuming.
Algorithm 3 Gibbs-enum algorithm
Input: a trained seq2seq model, target sequence  $\mathbf{y}$ , a trained LSTM LM, objective function  $L(\mathbf{x};\mathbf{y})$ , input length  $n$ , output length  $m$ , and target hit type.
Output: a trigger input  $x^{*}$
if hit type is in "io-hit" then
initialize  $x^{*}$  to be a sample from the LM
randomly initialize  $\pmb{x}^*$  to be a valid input sequence
for  $s = 1,2,\ldots ,T$  do
for  $t = 1,2,\dots ,n$  do
get gradient  $\nabla_{\pmb{x}_t^*}(-L(\pmb{x}_{< t}^*,\pmb{x}_t^*,\pmb{x}_{>t}^*;\pmb{y}))$  and set list  $H$  to be the  $G$  indexes with highest value in the gradient vector
for  $j = 1,2,\ldots ,G$  do
set  $x^{\prime}$  to be:
concat  $(\pmb{x}_{< t}^{*}$  ,one-hot  $(H[j])$ $\pmb{x}_{>t}^{*})$
if  $L(\pmb{x}'; \pmb{y}) < L(\pmb{x}^*; \pmb{y})$  then
$$
\operatorname {s e t} \boldsymbol {x} ^ {*} = \boldsymbol {x} ^ {\prime}
$$
if this sweep has no improvement for  $L$  then break
return  $x^{*}$
B Data-set Descriptions
Three publicly available conversational dialogue data-sets are used: Ubuntu, Switchboard, and OpenSubtitles. The Ubuntu Dialogue Corpus (Lowe et al., 2015) consists of two-person conversations extracted from the Ubuntu chat logs, where a user is receiving technical support from a helping agent for various Ubuntu-related problems. To train the baseline model, we select the first  $200\mathrm{k}$  dialogues for training (1.2M sentences / 16M words), and the next 5k dialogues for validation and testing respectively. We select the 30k most frequent words in the training data as our vocabulary, and out-of-vocabulary (OOV) words are mapped to the <UNK> token.
The Switchboard Dialogue Act Corpus  $^{12}$  is a version of the Switchboard Telephone Speech Corpus, which is a collection of two-sided telephone conversations, annotated with utterance-level dialogue acts. In this work we only use the conversation text part of the data, and select 1.1k dialogues for training (181k sentences / 1.2M words), 25 dialogues for validation and 25 dialogues for testing. We select the 10k most frequent words in the training data as our vocabulary.
We also report experiments on the OpenSubtitles data-set $^{13}$  (Tiedemann, 2009). The key difference between the OpenSubtitles data and Ubuntu/Switchboard data is that it contains a large number of malicious sentences, because the data consists of movie subtitles. We randomly select  $5\mathrm{k}$  movies for training (each movie is regarded as a big dialogue), which contains 5M sentences and 36M words, and 50 movies for validation and testing respectively. The  $30\mathrm{k}$  most frequent words are used as the vocabulary. We show some samples of the three data-sets in Appendix C.
For pre-processing, the text of all three data-sets are lower-cased, and all punctuation are removed. The maximum input sequence length is set to 15, with a maximum output sequence length of 20. Longer input sentences are cropped, and shorter input sentences are padded with  $\langle \mathrm{PAD} \rangle$  tokens.
C Data Samples and Baseline Perplexity Results