SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 4 summarizes the human evaluation results. We can see that the proposed approach is overall better than all baselines. Specifically, ND achieves better performance than Standard in terms of informativeness and relevance, and remains competitive in fluency. Compared with both NT and UL, our approach shows significant advantages, especially in fluency. It indicates that their punishment for high-frequency tokens or utterances will lead to
a serious non-fluency and inconsistency problem. We use Fleiss's kappa (Fleiss, 1971) to measure the inter-annotator agreement.
3.6 Experimental Analysis
We conduct extensive analysis on DailyDialog to investigate the effectiveness of the negative distillation in more details.
Models	Dist-2	Dist-3	LF	KL-2	BLEU-4
ND	.0678	.1447	.158	.26	.388
w/o Lpred	.0529	.1084	.145	.39	.397
w/o Latt	.0517	.1032	.138	.26	.392
w/o Lhid	.0365	.0677	.109	.62	.380
w/o Lneg	.0313	.0576	.102	.53	.395
Ablation study We study the effects of different negative distillation objectives by ablating the prediction layer distillation (w/o  $\mathcal{L}_{pred}$ ), the attention distillation (w/o  $\mathcal{L}_{att}$ ), the hidden state distillation (w/o  $\mathcal{L}_{hid}$ ), and the whole negative distillation (w/o  $\mathcal{L}_{neg}$ , i.e. Standard). The results in Table 5 show that all three proposed negative distillation objectives are useful for improving the diversity. The significant decline in w/o  $\mathcal{L}_{hid}$  indicates that the negative information in intermediate layers is very important for ND. w/o  $\mathcal{L}_{att}$  is better than w/o  $\mathcal{L}_{hid}$ , attributing to the more abundant information in hidden states.
Does source entropy work? To verify whether the source entropy filtering method can collect the generic responses, we select the top  $50\%$  and the bottom  $50\%$  of the sorted training set as  $\mathcal{D}_t$  and  $\mathcal{D}_b$ , respectively. Then we train  $N_t$  and  $N_b$  on the corresponding sub-sets. From Table 6, we can see that  $N_b$  outperforms  $N_t$  in all the diversity-related metrics, indicating the effectiveness of source entropy.
Table 5: Ablation studies of different negative distillation objectives in ND.
Models	Dist-1	Dist-2	Dist-3	LF
Nt	0.0024	0.0078	0.0134	0.0331
Nb	0.0040	0.0121	0.0215	0.0444
Can the negative knowledge be transferred? We take  $N_{t}$  and  $N_{b}$  as the negative teachers for the students  $S_{t}$  and  $S_{b}$ , respectively. Then we conduct negative distillation on both  $S_{t}$  and  $S_{b}$ . The
results in Table 7 demonstrate that  $S_{t}$  obtains more gains in diversity than  $S_{b}$ , indicating  $S_{t}$  gets rid of more negative knowledge. It can be further verified by the results of previous analysis that  $N_{t}$  has more negative knowledge than  $N_{b}$ .
Table 6: Effect of the source entropy filtering method.
Models	Dist-2	Dist-3	LF	KL-2	BLEU-4
St	0.0678	0.1447	0.158	0.26	0.388
Sb	0.0409	0.0844	0.097	0.40	0.386
Table 7: Effect of negative knowledge.
Models	Dist-2	Dist-3	LF	KL-2	BLEU-4
Standard	.0313	.0576	.102	.53	.395
ND (fixed α)	.0392	.0793	.123	.42	.386
ND	.0678	.1447	.158	.26	.388
Table 8: Effect of progressive distillation.
Models	Dist-1	Dist-2	Dist-3	LF
ND (random target)	0.0040	0.0109	0.0170	0.053
ND (hard target)	0.0136	0.0620	0.1344	0.139
ND (soft target)	0.0145	0.0678	0.1447	0.158
Table 9: Comparison of soft targets, hard targets, and random targets for negative distillation.
Study of soft target To evaluate the superiority of soft targets for negative distillation, we sample responses (i.e., hard target) by greedy search on the predictions of negative teachers for comparison. The results in Table 9 show that ND with soft targets can diversify the responses more effectively, demonstrating the advantages of richer negative information (e.g., the similarity between labels) in soft targets. What's more, we randomly select responses from the negative training set  $\mathcal{D}_N$  as negative targets. The sharp decline in performance proves that the negative teacher can produce targeted generic responses.
Effect of progressive distillation In order to verify the effectiveness of progressive negative distillation, we conduct negative distillation with fixed  $\alpha$ . The value is obtained by calculating the average of  $\alpha$  in Equation 15 across the convergence steps. The results in Table 8 demonstrate that the progressive distillation policy can help the student exploit negative knowledge more effectively. Besides, note that ND with fixed  $\alpha$  also outperforms the Standard model.
Models	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑	LF ↑	KL-1 ↓	KL-2 ↓	BLEU-3 ↑	BLEU-4 ↑
Standard	0.0060	0.0238	0.0455	0.068	0.92	0.62	0.375	0.372
NT	0.0077	0.0326	0.0640	0.083	0.76	0.70	0.349	0.334
UL	0.0059	0.0270	0.0570	0.075	0.82	0.95	0.321	0.294
CVAE	0.0022	0.0152	0.0579	0.026	1.60	1.94	0.144	0.087
FACE	0.0074	0.0305	0.0591	0.086	0.73	0.64	0.123	0.077
ND	0.0108	0.0427	0.0799	0.120	0.72	0.05	0.386	0.404
Table 10: Automatic evaluation results using beam search (size 5) on DailyDialog. Our approach (ND) outperform baselines on all the metrics.
Evaluation results with beam search He and Glass (2020) and Li et al. (2020a) choose greedy decoding due to its simplicity and higher diversity than beam decoding. However, we find that both NT and UL tend to generate long but non-fluent and incoherent responses. So we conduct beam search with adding the length penalty. Table 10 summarizes the results and it shows that both two baselines get better KL and BLEU scores than using greedy search due to shorter responses. ND outperform baselines on all the metrics, confirming the effectiveness of our method.
Input: The phone turns off by itself.
Standard: What's the problem?
NT: I see. Is a part of the most important things. And I'll be happy to keep you there. Is the most important thing for you...
UL: I see. Is not allowed to be a good thing. Birthday. Birthday. - hour are very important.....
ND: I am calling to tell you that he is in the right desk.
Input: Well, I bought this skirt here yesterday afternoon. But I got the wrong size. It's too small for my daughter. I wonder if I could refund it.
Standard: That's fine.
NT: That's fine. Your mind and address number. number. And date are available. And so on. Is very simple......
UL: I'm sorry, sir. But you've got to work overtime before you leave the contract. Service is very important. Service. Service usually be late.
ND: I think you have to pay the money.
Table 11: Examples of generated responses.
Case study Table 11 shows some cases generated by the proposed method and baselines. Standard prefers generic and meaningless responses. Both NT and UL tend to generate a short generic sentence followed by a incoherent and non-fluent subsequence. In contrast, ND can produce diverse and coherent responses.
Diversity Dialogue Learning There are two lines of work for solving the generic response prob
lem: One line promotes the diversity from positive view, which is outside of our work. Specially, previous work includes MMI (Li et al., 2016), GAN (Li et al., 2017a; Zhang et al., 2018), CVAE (Zhao et al., 2017), BT (Su et al., 2020), FACE (Jiang et al., 2019), AdaLabel (Wang et al., 2021), IAT (Zhou et al., 2021), and Nucleus Sampling (Holtzman et al., 2020). The other line alleviates the generic response problem using negative training. He and Glass (2020) regards frequent response problem as a sub-problem of the generic response problem and conduct negative update for the high-frequency responses during training. Li et al. (2020a) focuses on high-frequency tokens rather than tokens and punishes them by using the unlikelihood objective (Welleck et al., 2020). Both of them handle the generic response problem only from the angle of reducing frequency, thus can not capture all the features of generic replies.
Negative Training for Dialogue Learning Negative training for retrieval-based dialogue learning has been previously extensively studied (Humeau et al., 2020; Nugmanova et al., 2019), while we focus on the dialogue generation in this work. He and Glass (2020) uses negative training to prevent generic and malicious responses in dialogue models. Li et al. (2020a) generalizes unlikelihood to dialogue generation for improving repetition, specificity and coherence. Lagutin et al. (2021) proposes implicit unlikelihood training to minimize repetition. Our work proposes a new negative training paradigm aimed at improving the diversity of dialogue responses while avoiding the problem of poor consistency and fluency of previous work.
We present a novel negative training paradigm to improve the diversity of dialogue responses. It formulates the conventional negative training as a knowledge distillation process, which is rarely explored before. The negative teacher can produce
the corresponding generic and dull responses given any query, which naturally avoids problems that hinder previous negative training methods. Besides, we further boost the performance of negative distillation by exploiting richer information, i.e., multilevel features. Extensive experiments validate the superiority of our proposed method compared with prior negative training work.
A limitation of our work is that we only focus on the generic response problem. For future work, we will extend the proposed negative distillation to handle other generation problems, such as inconsistency and lacking personas or emotions.
We would like to thank the anonymous reviewers for their constructive comments. This work is supported by Beijing Natural Science Foundation (No.4222037, L181010) and National Natural Science Foundation of China (No.61972035). Kan Li is the corresponding author.
Reina Akama, Sho Yokoi, Jun Suzuki, and Kentaro Inui. 2020. Filtering noisy dialogue corpora by connectivity and content relatedness. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 941-958, Online. Association for Computational Linguistics.
Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 362-367, Baltimore, Maryland, USA. Association for Computational Linguistics.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Richard Csaky, Patrik Purgai, and Gábor Recski. 2019. Improving neural conversational models with entropy-based data filtering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5650-5669, Florence, Italy. Association for Computational Linguistics.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Tianxing He and James Glass. 2020. Negative training for neural dialogue response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2044-2058, Online. Association for Computational Linguistics.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Shaojie Jiang, Pengjie Ren, Christof Monz, and Maarten de Rijke. 2019. Improving neural response diversity with frequency-aware cross-entropy loss. In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 2879-2885. ACM.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online. Association for Computational Linguistics.
Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. 2019. NLNL: negative learning for noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 101-110. IEEE.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. 2019. Importance of search and evaluation strategies in neural dialogue modeling. In Proceedings of the 12th International Conference on Natural Language Generation, pages 76-87, Tokyo, Japan. Association for Computational Linguistics.
Evgeny Lagutin, Daniil Gavrilov, and Pavel Kalaidin. 2021. Implicit unlikelihood training: Improving neural text generation with reinforcement learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1432-1441, Online. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017a. Adversarial learning for neural dialogue generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2157-2169, Copenhagen, Denmark. Association for Computational Linguistics.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020a. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715-4728, Online. Association for Computational Linguistics.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017b. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Zuchao Li, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. 2020b. Data-dependent gaussian prior objective for language generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Ruotian Ma, Tao Gui, Linyang Li, Qi Zhang, Xuanjing Huang, and Yaqian Zhou. 2021. SENT: Sentence-level distant relation extraction via negative training. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6201-6213, Online. Association for Computational Linguistics.
Aigul Nugmanova, Andrei Smirnov, Galina Lavrentyeva, and Irina Chernykh. 2019. Strategy of the negative sampling for training retrieval-based dialogue systems. In IEEE International Conference on Pervasive Computing and Communications Workshops,
PerCom Workshops 2019, Kyoto, Japan, March 11-15, 2019, pages 844-848. IEEE.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1577-1586, Beijing, China. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-1958.
Hui Su, Xiaoyu Shen, Sanqiang Zhao, Zhou Xiao, Pengwei Hu, Randy Zhong, Cheng Niu, and Jie Zhou. 2020. Diversifying dialogue generation with non-conversational text. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7087-7097, Online. Association for Computational Linguistics.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818-2826. IEEE Computer Society.
Jörg Tiedemann. 2009. News from OPUS—A Collection of Multilingual Parallel Corpora with Tools and Interfaces.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. In ICML Deep Learning Workshop.
Pavlos Vougiouklis, Jonathon Hare, and Elena Simperl. 2016. A neural network approach for knowledge-driven response generation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3370-3380, Osaka, Japan. The COLING 2016 Organizing Committee.
Yida Wang, Yinhe Zheng, Yong Jiang, and Minlie Huang. 2021. Diversifying dialog generation via adaptive label smoothing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3507-3520, Online. Association for Computational Linguistics.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In
8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018. Generating informative and diverse conversational responses via adversarial information maximization. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 1815-1825.
Wangchunshu Zhou, Qifei Li, and Chenle Li. 2021. Learning from perturbations: Diverse and informative dialogue generation with inverse adversarial training. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 694-703, Online. Association for Computational Linguistics.
A Details for Implementations
Here are some implementation details of our experiments. Dropout (Srivastava et al., 2014) is used for the self-attention module, the feed-forward layer, and the activation layer, and the rate of all three is set to 0.1. We also use label smoothing (Szegedy et al., 2016) and the smoothing value is 0.1. The batch size is set to 256. We use the Adam optimizer (Kingma and Ba, 2015) and employ the warm-up (He et al., 2016) trick to adjust the learning rate during training. The warm-up steps  $s_{\mathrm{wp}}$  are 128000 and 256000 for DailyDialog and OpenSubtitles, respectively. The learning rate is computed as follows:
$$
l r = \frac {2 \cdot \min \left(\frac {1}{\sqrt {s}} , \frac {s}{\sqrt {s _ {\mathrm {w p}} ^ {3}}}\right)}{\sqrt {d _ {\mathrm {m o d e l}}}}, \tag {17}
$$
where  $lr$  is the learning rate at the  $s^{th}$  step of training and  $d_{\mathrm{model}}$  is the size of hidden states. We implement all approaches with Pytorch 1.7, and conduct all experiments on RTX 3090.
For NT, the threshold  $r_{\text{thres}}$  is set to 1% and the weight coefficient  $\lambda_{\text{POS}}$  is set to 1 as the authors'
suggestion. For UL, we search the mixing hyperparameter  $\alpha$  in [1, 10, 100, 1000] and 1000 is selected for its best performance. Both NT and UL are refined on the well-trained Standard model. For CAVE, we set the latent size with patience to 256 and 64 for DailyDialog and OpenSubtitles, respectively. And for FACE, we use the "output frequency" and "pre-weight" version as the author suggested.
We also compare the proposed method (ND) with AdaLabel (Wang et al., 2021), although AdaLabel alleviates the generic response problem from the perspective of target regularization instead of negative training. The results in Table 12 confirms the superior performance of our method for improving the diversity of generated responses.
Models	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑	LF ↑	KL-1 ↓	KL-2 ↓	BLEU-3 ↑	BLEU-4 ↑
AdaLabel	0.0100	0.0397	0.0757	0.105	0.89	0.59	0.097	0.061
ND	0.0145	0.0678	0.1447	0.158	0.65	0.26	0.381	0.388
AdaLabel	0.0065	0.0259	0.0476	0.064	1.11	0.83	0.109	0.069
ND	0.0108	0.0427	0.0799	0.120	0.72	0.05	0.386	0.404
Table 12: Comparing with AdaLabel by greedy search(Up) and beam search(Down) on DailyDialog. "↑" means higher is better. "↓" means lower is better.