SlowGuess/ACL_2020_OCR huggingface dataset:
To investigate the effectiveness and general applicability of the proposed framework, we instantiate our data manipulation framework on several state-of-the-art models for dialogue generation. The automatic evaluation results of our proposed learning framework and the corresponding vanilla models are listed in Table 2. Compared with the vanilla training procedure, the proposed data manipulation framework brings solid improvements for all the three architectures regarding almost all the evaluation metrics. Such improvements are consistent across both two conversation datasets, affirming the superiority and general applicability of our proposed framework.
We further compare our model with existing related methods. Not surprisingly, as shown in Table 3, our data manipulation framework outperforms the baseline methods on most of metrics. In particular, the improvement on Distinct metrics of our model is much greater, which implies that data manipulation effectively induce the neural dialogue model generating more diverse responses.
Table 3: Performance (%) of our approach instantiated on the naive SEQ2SEQ and the baseline approaches on (a) DailyDialog and (b) OpenSubtitles.
Opponent	Win	Loss	Tie	Kappa
Ours vs. SEQ2SEQ	45%	13%	42%	0.5105
Ours vs. Calibration	40%	9%	51%	0.4208
Ours vs. CVAE-GAN	37%	14%	49%	0.4063
Ours vs. Clustering	41%	12%	47%	0.4893
Table 4: The results of human evaluation on the test set of DailyDialog.
3.5 Human Evaluation
We use the DailyDialog as the evaluation corpus since it is more similar to our daily conversations and easier for annotators to make the judgement. Three graduate students are recruited to conduct manual evaluations. 100 test messages are randomly sampled. We present the input messages and the corresponding responses generated by our model and the comparison model to the annotators. The annotators are then required to compare the quality of these two responses (response<sub>1</sub>, response<sub>2</sub>), taking the following criteria into consideration: coherence, language consistency, fluency and informativeness, and evaluate among "win" (response<sub>1</sub> is better), "loss" (response<sub>2</sub> is better) and "tie" (they are equally good or bad). Note that cases with different evaluation results are labeled as "tie". Table 4 summarizes
6339
Dist-1	Dist-2	Dist-3	Intra-1	Intra-2	Intra-3	Ent-1	Ent-2	Ent-3	BLEU	Avg	Ext	Gre
Baseline	0.8570	4.0123	7.9559	88.509	94.727	96.844	6.7783	10.394	11.719	0.2146	65.200	46.355	67.344
w/ word-level augmentation	1.2205	6.0622	12.2620	89.916	95.265	96.627	6.9457	10.920	12.334	0.2657	65.315	46.821	68.025
w/ sentence-level augmentation	1.4702	6.7803	13.0910	91.309	95.772	97.397	7.0260	10.952	12.517	0.2721	66.788	47.464	67.911
Table 5: Ablation test (%) for word-level and sentence-level augmentations.
Dist-1	Dist-2	Dist-3	Intra-1	Intra-2	Intra-3	Ent-1	Ent-2	Ent-3	BLEU	Avg	Ext	Gre
Full model	2.0515	9.7186	18.9970	91.343	96.446	97.613	7.0858	11.121	12.545	0.3604	66.551	47.325	68.378
w/o weighting	1.8156	8.1939	15.9000	90.747	95.816	97.199	7.0976	11.130	12.731	0.5147	65.675	46.955	68.048
w/o augmentation	1.1456	5.4386	11.1140	86.399	92.293	94.825	6.8752	10.579	11.837	0.2002	64.937	46.540	67.541
w/o instance filter	1.8627	8.2850	15.9400	88.551	93.445	94.419	7.1440	11.305	12.823	0.2813	65.606	46.912	67.863
Table 6: Model ablation test (%) on DailyDialog.
Figure 4: Comparison of the training with data manipulation and vanilla training using SEQ2SEQ on the validation set of DailyDialog. Dist-1, Embedding Extrema and Ent-3 are denoted as "Distinct", "Embedding" and "Entropy", respectively.
human evaluation results. The kappa scores indicate that the annotators came to a fair agreement in the judgement. Compared with the baseline methods, our data manipulation approach brings about more informative and coherent replies.
3.6 Model Analysis
Learning Efficiency Figure 4 presents validation results along iterations when training the SEQ2SEQ model on DailyDialog. We observe that when training SEQ2SEQ using our framework, the initial learning speed is a bit slower than the standard vanilla training. However, our framework surpasses the vanilla training on the final stage. One reason is that, at the early stage, the data manipulation model takes some time to improve its manipulation skills. This may slow down the neural dialogue model learning. Once the manipulation skills are effective enough, the neural dialogue model may benefit from learning more effective
samples instead of those inefficient instances, and achieves better performance.
Examples with Different Augmentation Frequencies The data manipulation model selectively chooses samples to conduct data augmentation. To further glean the insights regarding which samples are favored by the augmentation model, we list examples with different augmentation frequencies in Figure 5. We notice that samples frequently augmented by the manipulation model are more reliable than those seldom augmented ones. Therefore, the dialogue model is able to learn from those effective instances and their synonymous variants.
Word-level vs. Sentence-level Augmentation In our framework, we implement two kinds of augmentation mechanisms. Word-level augmentation enriches the given samples by substituting words, while sentence-level augmentation paraphrases the original samples through back-translation. We evaluate their performances and report results in Table 5. Both augmentation mechanisms improve the performance over the vanilla SEQ2SEQ baseline, while sentence-level augmentation performs slightly better than word-level augmentation on most evaluation metrics. One possible reason is that sentence-level augmentation captures more paraphrasing phenomenon.
Ablation Study Table 6 presents the results of model variants, by ablating specific parts of the data manipulation model. Among different variants, without data augmentation, the performance degrades rapidly. Meanwhile, without weighting or instance filter also decreases the performance. This implies that the neural dialogue generation model not only benefits from more training samples but also reaps greater advantages from those effective rather than inefficient instances.
Figure 5: Examples with different augmentation frequencies. Instances with higher augmentation frequencies are more effective than those seldom augmented examples.
Distinct (Δ)	Embedding (Δ)	Entropy (Δ)	BLEU (Δ)
50% training data	0.8179 (+109.88%)	1.6860 (+2.54%)	0.4910 (+4.20%)	0.0768 (+56.64%)
100% training data	1.0865 (+71.62%)	0.2720 (+0.40%)	0.4750 (+3.90%)	0.1307 (+43.21%)
Table 7: Performance improvements regarding different sizes of training data on DailyDialog. Dist-1, Embedding Greedy and Ent-3 are denoted as "Distinct", "Embedding" and "Entropy", respectively.
Impact of Training Data Scale We explore the impact of training data scale on the data manipulation framework by comparing a model trained on half amount of the training data in DailyDialog. As presented in Table 7, with only  $50\%$  amount of training data, our model achieves a greater performance boost, which affirms the effectiveness and robustness of the proposed approach.
Existing approaches to improving neural dialogue generation models mainly target on building more powerful learning systems, using extra information such as conversation topics (Xing et al., 2017), persona profile (Song et al., 2019), user emotions (Zhou et al., 2018), or out-sourcing knowledge (Liu et al., 2018). Another popular framework for dialogue generation is variational autoencoder (Kingma and Welling, 2014; Zhao et al., 2017; Shen et al., 2017), in which a latent variable is introduced to benefit the dialogue model with more diverse response generation. Contrasted with previous researches, we investigate to improve the dialogue model from a different angle, i.e., adapting the training examples using data manipulation techniques.
Data augmentation is an effective way to im
prove the performance of neural models. To name a few, Kurata et al. (2016) propose to generate more utterances by introducing noise to the decoding process. Kobayashi (2018); Wu et al. (2019) demonstrate that contextual augmentation using label-conditional language models helps to improve the neural networks classifier on text classification tasks. Sennrich et al. (2016) boost neural machine translation models using back-translation. Xie et al. (2017); Andreas (2020) design manually-specified strategies for data augmentation. Hou et al. (2018) utilize a sequence-to-sequence model to produce diverse utterances for language understanding. Li et al. (2019); Niu and Bansal (2019) propose to generate sentences for dialogue augmentation. Compared with previous augmentation approaches for dialogue generation, augmented sentences in our framework are selectively generated using the pretrained models and the augmentation process is additionally fine-tuned jointly with the training of dialogue generation.
Regarding data weighting, past methods (Jiang and Zhai, 2007; Rebbapragada and Brodley, 2007; Wang et al., 2017; Ren et al., 2018; Hu et al., 2019) have been proposed to manage the problem of training set biases or label noises. Lison and Bibauw (2017) propose to enhance the retrieval-based dialog system with a weighting model. Shang et al. (2018) likewise design a matching network to calibrate the dialogue model training through instance weighting. Cai et al. (2020) investigate curriculum learning to adapt the instance effect on dialogue model training according to the sample complexity. Whereas our proposed framework learns to reweight not only the original training examples but also the augmented examples. Another differ
6341
ence is that, we directly derive data weights based on their gradient directions on a validation set, instead of separately training a external weighting model. Csaky et al. (2019) claim that high-entropy utterances in the training set lead to those boring generated responses and thus propose to ameliorate such issue by simply removing training instances with high entropy. Although data filtering is a straightforward approach to alleviate the problem of noisy data, the informative training samples remain untouched and insufficient. Whereas our method holds the promise of generating more valid training data and alleviating the negative noises in the meantime.
Note that either data augmentation or instance reweighting can be considered band-aid solution: simply augmenting all training data risks introducing more noisy conversations as such low-quality examples prevail in human-generated dialogues, whilst adapting the sample effect merely by instance reweighting is also suboptimal since effective training samples remain insufficient. The proposed learning-to-manipulate framework organically integrates these two schemes, which collectively fulfill the entire goal.
In this work, we consider the automated data manipulation for open-domain dialogue systems. To induce the model learning from effective instances, we propose a learnable data manipulation model to augment effective training samples and reduce the weights of inefficient samples. The resulting data manipulation model is fully end-to-end and can be trained jointly with the dialogue generation model. Experiments conducted on two public conversation datasets show that our proposed framework is able to boost the performance of existing dialogue systems.
Our learning-to-manipulate framework for neural dialogue generation is not limited to the elaborately designed manipulation skills in this paper. Future work will investigate other data manipulation techniques (e.g., data synthesis), which can be further integrated to improve the performance.
We would like to thank all the reviewers for their insightful and valuable comments and suggestions. This work is supported by the National Natural Science Foundation of China-Joint Fund for Ba
sic Research of General Technology under Grant U1836111 and U1736106. Hongshen Chen and Xiaofang Zhao are the corresponding authors.
Jacob Andreas. 2020. Good-enough compositional data augmentation. In ACL.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In *ICLR*.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
Hengyi Cai, Hongshen Chen, Cheng Zhang, Yonghao Song, Xiaofang Zhao, Yangxi Li, Dongsheng Duan, and Dawei Yin. 2020. Learning from easy to complex: Adaptive multi-curriculum learning for neural dialogue generation. In AAAI.
Richard Csaky, Patrik Purgai, and Gabor Recski. 2019. Improving neural conversational models with entropy-based data filtering. In ACL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In EMNLP.
Xiaodong Gu, Kyunghyun Cho, JungWoo Ha, and Sunghun Kim. 2019. Dialogwae: Multimodal response generation with conditional Wasserstein autoencoder. In ICLR.
Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. 2018. Sequence-to-sequence data augmentation for dialogue language understanding. In COLING.
Zhiting Hu, Bowen Tan, Ruslan Salakhutdinov, Tom M. Mitchell, and Eric P. Xing. 2019. Learning data manipulation for augmentation and weighting. In NeurIPS.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In ICLR.
Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in NLP. In ACL.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Diederik P. Kingma and Max Welling. 2014. Auto-encoding variational bayes. In ICLR.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In NAACL-HLT.
6342
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL.
Gakuto Kurata, Bing Xiang, and Bowen Zhou. 2016. Labeled data generation with encoder-decoder LSTM for semantic slot filling. In INTER-SPEECH.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In NAACL-HLT.
Juntao Li, Lisong Qiu, Bo Tang, Min Dong Chen, Dongyan Zhao, and Rui Yan. 2019. Insufficient data can also rock! learning to converse using smaller data with augmentation. In AAAI.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In IJCNLP.
Pierre Lison and Serge Bibauw. 2017. Not all dialogues are created equal: Instance weighting for neural conversational models. In SIGDIAL.
Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In LREC.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In EMNLP.
Shuman Liu, Hongshen Chen, Zhaochun Ren, Yang Feng, Qun Liu, and Dawei Yin. 2018. Knowledge diffusion for neural dialogue generation. In ACL.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair's wmt19 news translation task submission. In Proc. of WMT.
Tong Niu and Mohit Bansal. 2019. Automatically learning data augmentation policies for dialogue tasks. In EMNLP.
Umaa Rebbapragada and Carla E. Brodley. 2007. Class noise mitigation through instance weighting. In ECML.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. In ICML.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In AAAI.
Mingyue Shang, Zhenxin Fu, Nanyun Peng, Yansong Feng, Dongyan Zhao, and Rui Yan. 2018. Learning to converse with noisy data: Generation with calibration. In IJCAI.
Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi Niu, Yang Zhao, Akiko Aizawa, and Guoping Long. 2017. A conditional variational framework for dialog generation. In ACL.
Haoyu Song, Weinan Zhang, Yiming Cui, Dong Wang, and Ting Liu. 2019. Exploiting persona information for diverse generation of conversational responses. In *IJCAI*.
Bowen Tan, Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric P. Xing. 2019. Connecting the dots between MLE and RL for sequence generation. In ICLR Workshop.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017. Instance weighting for neural machine translation domain adaptation. In EMNLP.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional BERT contextual augmentation. In ICCS.
Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017. Data noising as smoothing in neural network language models. In ICLR.
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In AAAI.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. In ICLR.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In ACL.
Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional chatting machine: Emotional conversation generation with internal and external memory. In AAAI.
6343