SlowGuess/ACL_2020_OCR huggingface dataset:
Hengyi Cai†,§*, Hongshen Chen‡
Yonghao Song†, Cheng Zhang†, Xiaofang Zhao† and Dawei Yin††
†Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
$^{\S}$ University of Chinese Academy of Sciences, Beijing, China
‡Data Science Lab, JD.com, China
††Baidu Inc., China
caihengyi@ict.ac.cn, ac@chenhongshen.com,
{songyonghao, zhangcheng, zhaoxf} @ict.ac.cn, yindawei@acm.org
Current state-of-the-art neural dialogue models learn from human conversations following the data-driven paradigm. As such, a reliable training corpus is the crux of building a robust and well-behaved dialogue model. However, due to the open-ended nature of human conversations, the quality of user-generated training data varies greatly, and effective training samples are typically insufficient while noisy samples frequently appear. This impedes the learning of those data-driven neural dialogue models. Therefore, effective dialogue learning requires not only more reliable learning samples, but also fewer noisy samples. In this paper, we propose a data manipulation framework to proactively reshape the data distribution towards reliable samples by augmenting and highlighting effective learning samples as well as reducing the effect of inefficient samples simultaneously. In particular, the data manipulation model selectively augments the training samples and assigns an importance weight to each instance to reform the training data. Note that, the proposed data manipulation framework is fully data-driven and learnable. It not only manipulates training samples to optimize the dialogue generation model, but also learns to increase its manipulation skills through gradient descent with validation samples. Extensive experiments show that our framework can improve the dialogue generation performance with respect to various automatic evaluation metrics and human judgments.
Open-domain dialogue generation, due to its potential applications, is becoming ubiquitous in the community of natural language processing. Current end-to-end neural dialogue generation models (Li et al., 2016; Serban et al., 2017; Zhao et al.,
Figure 1: Data manipulation helps the dialogue model training by augmenting and highlighting effective learning samples as well as reducing the weights of inefficient samples.
2017) are primarily built following the data-driven paradigm, that is, these models mimic the human conversations by training on the large-scale query-response pairs. As such, a reliable training corpus that exhibits high-quality conversations is the crux of building a robust and well-behaved dialogue model.
Unfortunately, owing to the subjectivity and open-ended nature of human conversations, the quality of the collected human-generated dialogues varies greatly (Shang et al., 2018), which hampers the effectiveness of data-driven dialogue models: 1) Effective conversation samples are quite insufficient. To glean some insights on the data quality of dialogue corpus, we choose the query-relatedness to take a glimpse of the data quality. In dialogue corpus, some conversations are quite coherent, where the queries and responses are well-correlated, while others are not. Query-relatedness measures the semantic similarities between the query and its corresponding response in the embedding space and ranges from 0 to 1. When reviewing DailyDialog (Li et al., 2017), we find that only  $12\%$  conversation samples are of relatively high query-relatedness scores  $(>0.6)$ . Without adequate reliable training samples, the neural dialogue model is prone to converge to a sub-optimal point. 2) Meanwhile, noisy and even meaningless conversa
*Work done at Data Science Lab, JD.com.
6334
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6334-6343 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tion samples frequently appear. As Li et al. (2016) reported, “I don’t know” appears in over 113K sentences in the training corpus OpenSubtitles (Lison and Tiedemann, 2016). Such kind of noisy conversation data prevails in neural dialogue model training, and vitally impedes the model learning.
Therefore, effective dialogue learning requires not only more reliable learning samples, but also fewer noisy samples. In this work, as illustrated in Figure 1, we propose a novel learnable data manipulation framework to proactively reshape the data distribution towards reliable samples by augmenting and highlighting effective learning samples as well as reducing the weights of inefficient samples simultaneously. Specifically, to generate more effective data samples, the data manipulation model selectively augments the training samples in terms of both word level and sentence level, using masked language models such as BERT (Devlin et al., 2019) and back-translation (Sennrich et al., 2016) technique. To reduce the weights of inefficient samples from the original training samples and the augmented samples, the data manipulation model assigns an importance weight to each sample to adapt the sample effect on dialogue model training. It gives out higher importance weights to critical learning samples and lower weights to those inefficient samples. Furthermore, different from most previous data augmentation or data weighting studies (Li et al., 2019; Shang et al., 2018; Csaky et al., 2019), which are unaware of the target model states during augmentation or weighting, our data manipulation framework not only manipulates training samples to optimize the dialogue generation model, but also learns to increase its manipulation skills through gradient descent with validation samples.
We apply the proposed data manipulation framework on several state-of-the-art generation models with two real-life open-domain conversation datasets and compare with the recent data manipulation approaches in terms of 13 automatic evaluation metrics and human judgment. Experiment results show that our data manipulation framework outperforms the baseline models over most of the metrics on both datasets.
2 Data Manipulation for Neural Dialogue Generation
The proposed data manipulation framework tackles the problem of un-even quality data by inducing the
Figure 2: Overview of the proposed automated data manipulation framework for neural dialogue generation. At training step  $t$ , the data manipulation model augments and weights the training samples for the dialogue model learning.
model learning from more effective dialogue samples and reducing effects of those inefficient samples simultaneously. In particular, as illustrated in Figure 2, it manipulates and reshapes the data distribution for neural dialogue model learning in mainly three stages: First, each batch of training samples are selectively augmented to generate more variant samples; and then, all the samples, including the original samples and the augmented samples, are assigned with instance weights indicating their importance regarding current learning status; finally, the weighted samples are fed into the neural dialogue model to induce the model learning from more effective training instances.
Note that, although we describe the framework in three components for ease of understanding, in fact, the whole framework can be trained in an end-to-end manner. As a result, the data manipulation network is capable of not only manipulating training samples to optimize the dialogue generation model, but also learning to increase its manipulation skills through gradient descent with validation samples.
We first introduce the augmentation and weighting strategies for data manipulation in §2.1 and §2.2, and then describe how the neural dialogue generation model learns from the manipulated samples in §2.3. Parameters estimation for the data manipulation model is elaborated in §2.4.
2.1 Dialogue Augmentation
To induce the neural dialogue generation model to learn from more effective samples, we develop a gated data augmentation mechanism for the manipulation framework to selectively augment the learning samples.
Specifically, as shown in Figure 3, given a training sample, the manipulation framework first specifies whether to augment it or not through an in
6335
Figure 3: Illustration of the data manipulation model. During training, it takes the original batch samples as input, and generates the augmented data samples as well as the importance weights for dialogue model training.
stance filter, which can be implemented using a sigmoid gating function. Then, two levels of data augmentation are introduced, word-level contextual augmentation and sentence-level data augmentation, to augment the chosen sample accordingly.
2.1.1 Word-level Contextual Augmentation
As the name suggests, word-level augmentation enriches the training samples by substituting the words in the original sample (Figure 3 (a)). Here, we employ a masked language model, BERT (Devlin et al., 2019), to implement word-level augmentation. Given an original sentence, the language model first randomly masks out a few words. BERT then takes in the masked sentence and predicts the corresponding masked positions with new words.
A fixed pre-trained BERT may not generalize well for our data manipulation framework, because BERT is unaware of the dialogue learning status. To mitigate such defects, we further fine-tune BERT through backpropagation (more details in § 2.4). In particular, BERT is adapted to be differentiable by utilizing a gumbel-softmax approximation (Jang et al., 2017) when predicting substitution words.
2.1.2 Sentence-level Data Augmentation
Word-level data augmentation is quite straightforward. However, such kind of rewriting is limited to only a few words. In human dialogues, there exist various synonymous conversations with different sentence structures. To further diversify the expressions in conversion, we introduce the sentence-level data augmentation through back
translation as in Edunov et al. (2018); Yu et al. (2018), which trains two translation models: one translation model from the source language to target language and another backward translation model from the target to the source, as shown in Figure 3 (b). By transforming the expression styles across different languages, the augmented training samples are expected to convey similar information while with different expressions.
Similar to the fine-tuning strategy in word-level data augmentation, we also fine-tune the sentence-level data augmentation components to encourage the model to generate more effective samples for dialogue training. The gradients are backpropagated into the translation-based augmentation model, where a differentiable gumbel-softmax is utilized when predicting sentences using the translation model.
2.2 Data Weighting
Given the original training samples and the augmented samples, to deal with the problem of noisy instances, data manipulation model assigns an importance weight to each training sample regarding the learning status. In particular, the sample importance weights are approximated through a softmax function over the scores of these instances. A multilayer perceptron is employed to compute example scores, taking distributional representations of these instances as input. Each sample is converted into its corresponding distributional representation through a transformer-based encoder.
6336
2.3 Dialogue Generation with Data Manipulation
Conventionally, neural dialogue generation model is optimized with a vanilla negative log-likelihood loss using the training data  $\mathcal{D}$  with size  $N$ :  $\mathcal{L}_{\text{vanilla}} = \sum_{j=1}^{N} - \log p(\boldsymbol{y}_j | \boldsymbol{x}_j)$ , where each sample is treated equally. In our framework, we assign each sample with an importance weight and augment the original training set  $\mathcal{D} = \{(x_j, y_j)\}_{j=1}^N$  to  $\mathcal{D}' = \{(x_j, y_j)\}_{j=1}^{N'}$  regarding the learning status. To perform the weighted optimization with augmented training set  $\mathcal{D}'$ , we utilize a weighted negative log-likelihood loss function:
$$
\mathcal {L} _ {d m} = \sum_ {j = 1} ^ {N ^ {\prime}} - w _ {j} \log p (\boldsymbol {y} _ {j} | \boldsymbol {x} _ {j}), \tag {1}
$$
where  $w_{j}$  is the instance weight produced by the data manipulation network.
2.4 Parameter Estimation for Data Manipulation
The data manipulation network not only manipulates training samples to optimize the dialogue learning process, but also learns to increase its manipulation skills through gradient descent with validation samples. We formulate such joint learning process following a novel policy learning paradigm (Hu et al., 2019; Tan et al., 2019), where the manipulation framework is formulated as a learnable data-dependent reward function  $R_{\phi}(\boldsymbol{d} = \{\boldsymbol{x}, \boldsymbol{y}\}|\mathcal{D})$ , the dialogue model  $p_{\theta}(\boldsymbol{y}|\boldsymbol{x})$  is treated as a policy, the input  $\boldsymbol{x}$  as the "state", and the output  $\boldsymbol{y}$  as the "action". The reward function  $R_{\phi}(\boldsymbol{d}|\mathcal{D})$  is defined as:
$$
R _ {\phi} (\boldsymbol {d} | \mathcal {D}) = \left\{ \begin{array}{l l} w _ {i} & \text {i f} \boldsymbol {d} \text {i s a n a u g m e n t e d s a m p l e} \\ & \text {o f} \boldsymbol {d} _ {i} ^ {*} \text {o r} \boldsymbol {d} = \boldsymbol {d} _ {i} ^ {*}, \boldsymbol {d} _ {i} ^ {*} \in \mathcal {D} \\ - \infty & \text {o t h e r w i s e ,} \end{array} \right. \tag {2}
$$
where  $\phi$  denotes the parameter of data manipulation network and  $w_{i}\in \mathbb{R}$  is the importance weight associated with the ith data sample. In such formulation, a sample  $\pmb{d}$  receives a real-valued reward when  $\pmb{d}$  is an augmented sample, or  $\pmb{d}$  matches an instance in the original training set.
As depicted in Algorithm 1, the parameter  $\theta$  of the neural dialogue model and parameter  $\phi$  of the data manipulation network are alternatively optimized. Jointly optimizing the dialogue model and the manipulation network can be regarded as reward learning, where the policy  $p_{\theta}(\boldsymbol{y}|\boldsymbol{x})$  receives relatively higher rewards for effective samples and
Algorithm 1 Joint Learning of Dialogue Model and Data Manipulation Network
Input: The dialogue model  $\theta$ , data manipulation network  $\phi$ , training set  $\mathcal{D}$  and validation set  $\mathcal{D}^v$
1: Initialize dialogue model parameter  $\theta$  and data manipulation model parameter  $\phi$
2: repeat
3: Optimize  $\theta$  on  $\mathcal{D}$  enriched with data manipulation.
4: Optimize  $\phi$  by maximizing data log-likelihood on  $\mathcal{D}^v$ .
5: until convergence
Output: Learned dialogue model  $\theta^{*}$  and data manipulation model  $\phi^{*}$
lower rewards for those inefficient samples. More concretely, to optimize the neural dialogue model, at each iteration, mini-batch instances are sampled from the training set, and are then enriched through augmentation and weighting. The parameter  $\theta$  of the neural dialogue model is then updated with a weighted negative log-likelihood loss function in Eq.(1):
$$
\theta^ {\prime} = \theta - \alpha \nabla_ {\theta} \mathcal {L} _ {d m} (\theta , \phi), \tag {3}
$$
where  $\nabla_{\theta}\mathcal{L}_{dm}(\theta ,\phi)$  is the gradient of  $\theta$  with respect to the loss  $\mathcal{L}_{dm}$ , and  $\alpha$  is the step size. The parameter  $\phi$  of the data manipulation network is learned by taking a meta gradient descent step on validation samples (Ren et al., 2018). Equation (3) shows that  $\theta^{\prime}$  depends on  $\phi$ . Therefore, the manipulation model (i.e. the reward function  $R_{\phi}(d|\mathcal{D})$ ) can be optimized by directly backpropagating the gradient through  $\theta^{\prime}$  to  $\phi$ .
Dataset	Train	Valid	Test
DailyDialog	54,889	6,005	5,700
OpenSubtitles	64,000	8,000	8,000
Table 1: Data statistics of the experiment corpora.
3.1 Experiment Setup
Data We conduct experiments on two English conversation datasets: (1) DailyDialog (Li et al., 2017), a collection of real-world dialogues widely used in open-domain dialogue generation. This is a multi-turn dataset, and we treat each turn as a training pair in this work. The overlapping pairs are removed from the data set. (2) OpenSubtitles (Lison and Tiedemann, 2016), a group of human-human conversations converted from movie transcripts.
6337
80,000 instances are sampled from the original corpus and the data proportion for train-valid/test set is set to  $8/1/1$ , respectively. The dataset statistics are listed in Table 1.
Experimental Models To ascertain the effectiveness and applicability of our method, we implement the proposed data manipulation framework on following representative models: (i) SEQ2SEQ: a RNN-based sequence-to-sequence model with attention mechanisms (Bahdanau et al., 2015); (ii) CVAE: a latent variable model using conditional variational auto-encoder, trained with KLannealing and a BoW loss as in Zhao et al. (2017); (iii) Transformer: an encoder-decoder architecture relying solely on the attention mechanisms (Vaswani et al., 2017).
Comparison Models We also compare our approach with previous data augmentation or instance weighting methods: (i) CVAE-GAN (Li et al., 2019): a model that combines CVAE and GAN for augmenting the training data to generate more diversified expressions. (ii) Calibration (Shang et al., 2018): a calibration network measures the quality of data samples and enables weighted training for dialogue generation. (iii) Clustering (Csaky et al., 2019): it clusters high-entropy samples as noises and filters them out.
We adopt several widely used metrics (Liu et al., 2016; Li et al., 2016; Serban et al., 2017; Gu et al., 2019) to measure the performance of dialogue generation models, including BLEU, embedding-based metrics, entropy-based metrics and distinct metrics. In particular, BLEU measures how much a generated response contains n-gram overlaps with the reference. We compute BLEU scores for  $n < 4$  using smoothing techniques<sup>1</sup>. Embedding-based metric computes the cosine similarity of bag-of-words embeddings between the hypothesis and the reference. We employ the following three embedding metrics to assess the response quality: (1) Embedding Average (Avg): cosine similarity between two utterances, in which the sentence embedding is computed by taking the average word embedding weighted by the smooth inverse frequency sent\_emb(e) =  $\frac{1}{|e|}\sum_{\nu \in e}\frac{0.001}{0.001 + p(\nu)} emb(\nu)$  of words as in Arora et al. (2017). where  $emb(\nu)$
and  $p(\nu)$  are the embedding and the probability $^2$  of word  $\nu$  respectively. (2) Embedding Greedy (Gre): greedily matching words in two utterances based on the cosine similarities between their embeddings, and averaging the obtained scores, (3) Embedding Extrema (Ext): cosine similarity between the largest extreme values among the word embeddings in the two utterances. We use Glove vectors as the word embeddings. Regarding entropy-based metrics, we compute the n-gram entropy  $\mathbf{Ent - n} = -\frac{1}{|r|}\sum_{\nu \in r}\log_2p(\nu)$  of responses to measure their non-genericness, where the probabilities  $p(\nu)$  of n-grams  $(n = 1,2,3)$  are calculated based on the maximum likelihood estimation on the training data (Serban et al., 2017). Distinct computes the diversity of the generated responses. Dist-n is defined as the ratio of unique n-grams  $(n = 1,2,3)$  over all n-grams in the generated responses. Following Gu et al. (2019), we also report Intra-\{1,2,3\} metrics which are computed as the average of distinct values within each sampled response.
3.3 Implementation & Reproducibility
For word-level dialogue augmentation, we employ the pre-trained BERT-base language model with the uncased version of tokenizer. We follow the hyper-parameters and settings suggested in Devlin et al. (2019). The replacement probability is set to  $15\%$ . For back-translation in sentence-level dialogue augmentation, we use the Transformer model (Vaswani et al., 2017) trained on En-De and En-Ru WMT'19 news translation tasks (Ng et al., 2019). German and Russian sentences were tokenized with the Moses tokenizer (Koehn et al., 2007). The same hyper-parameters are used for the translation tasks, i.e., word representations of size 1024, dropout with 0.8 keep probability, feedforward layers with dimension 4096, 6 blocks in the encoder and decoder with 16 attention heads. Models are optimized with Adam (Kingma and Ba, 2015) optimizer using initial learning rate 7e-4. Regarding dialogue models implementation, we adopt a 2-layer bidirectional LSTM as the encoder and a unidirectional one as the decoder for both the SEQ2SEQ and CVAE. The hidden size is set to 256, and the latent size used in CVAE is set to 64. The transformer model for dialogue generation is configured with 512 hidden size, 8 attention heads and 6 blocks in both the encoder and decoder. The
${}^{2}$  Probability is computed based on the maximum likelihood estimation on the training data.
1https://www.nltk.org/_modules/nltk/translate/bleu_score.html
6338
Models	Dist-1	Dist-2	Dist-3	Intra-1	Intra-2	Intra-3	Ent-1	Ent-2	Ent-3	BLEU	Avg	Ext	Gre
(a)	SEQ2SEQ	0.9026	4.2497	8.4039	87.909	94.399	95.971	6.7263	10.381	12.036	0.2160	67.671	47.472	68.349
SEQ2SEQ (★)	1.3058	5.8408	11.2820	88.628	94.268	96.171	7.0253	11.018	12.726	0.3619	68.018	47.665	68.708
CVAE	0.9798	4.6095	9.0876	91.848	96.815	98.025	6.9184	10.740	12.365	0.2617	66.935	46.926	68.068
CVAE (★)	2.0683	9.0082	17.3260	93.301	97.418	98.323	7.0278	11.078	12.586	0.2954	66.363	46.955	68.424
Transformer	1.3489	5.9736	11.3310	87.725	94.170	95.944	6.9024	10.624	11.941	0.2342	65.305	46.223	67.419
Transformer (★)	2.4763	11.6270	21.4520	89.058	96.615	98.248	7.1556	11.320	12.956	0.4163	66.908	46.284	67.656
(b)	SEQ2SEQ	0.5695	2.9952	6.2377	96.200	97.754	98.355	6.5996	10.371	12.213	0.0078	55.912	40.320	57.664
SEQ2SEQ (★)	0.7285	3.6053	7.2580	95.938	97.829	98.561	6.8391	10.903	13.411	0.0210	58.105	41.113	59.551
CVAE	0.5493	2.9585	6.3159	78.534	90.028	98.864	5.8675	10.089	12.544	0.0019	54.508	41.262	62.139
CVAE (★)	1.0883	4.8967	9.7060	95.489	97.579	98.201	6.8952	10.902	12.200	0.0173	56.473	41.678	59.330
Transformer	0.7226	3.8053	8.3877	92.94	94.947	96.023	7.0361	11.091	11.832	0.0050	55.257	41.302	58.232
Transformer (★)	1.7264	6.8750	12.5770	94.223	97.204	98.055	7.0493	11.334	12.098	0.0110	55.219	40.701	59.081
Table 2: Automatic evaluation results (%) on (a) DailyDialog and (b) OpenSubtitles. “★” denotes that the model is trained using our proposed data manipulation framework. The metrics Average, Extrema and Greedy are abbreviated as Avg, Ext and Gre, respectively. The best results in each group are highlighted with **bold**.
Models	Dist-1	Dist-2	Dist-3	Intra-1	Intra-2	Intra-3	Ent-1	Ent-2	Ent-3	BLEU	Avg	Ext	Gre
(a)	Calibration (Shang et al., 2018)	0.7278	3.2265	6.0570	86.619	91.697	93.753	6.7827	10.439	11.867	0.1876	67.309	47.347	67.886
CVAE-GAN (Li et al., 2019)	0.6996	3.2448	6.4911	85.329	92.804	94.953	6.8184	10.425	12.260	0.2149	68.012	47.079	68.007
Clustering (Csáky et al., 2019)	0.6532	3.0747	6.2315	78.612	87.268	91.151	6.8554	10.436	12.358	0.2062	69.040	47.367	68.276
Ours	1.3058	5.8408	11.2820	88.628	94.268	96.171	7.0253	11.018	12.726	0.3619	68.018	47.665	68.708
(b)	Calibration (Shang et al., 2018)	0.5107	2.7129	5.6281	95.997	97.590	98.242	6.7281	10.625	12.322	0.0034	58.786	40.850	59.132
CVAE-GAN (Li et al., 2019)	0.5175	2.7843	5.8150	95.303	97.109	98.218	6.9186	10.747	12.592	0.0104	57.610	40.871	58.767
Clustering (Csáky et al., 2019)	0.4728	2.6349	5.3878	96.145	97.614	98.317	6.8789	10.869	13.271	0.0124	59.069	41.026	59.343
Ours	0.7285	3.6053	7.2580	95.938	97.829	98.561	6.8391	10.903	13.411	0.0210	58.105	41.113	59.551
hyper-parameters in the baseline models are set following the original papers (Li et al., 2019; Shang et al., 2018; Csaky et al., 2019).
3.4 Evaluation Results