SlowGuess/NAACL_2022_OCR huggingface dataset:
Yiwei Li, Shaoxiong Feng, Bin Sun, Kan Li
School of Computer Science, Beijing Institute of Technology
{liyiwei, shaoxiongfeng, binsun, likan}@bit.edu.cn
Generative dialogue models suffer badly from the generic response problem, limiting their applications to a few toy scenarios. Recently, an interesting approach, namely negative training, has been proposed to alleviate this problem by reminding the model not to generate high-frequency responses during training. However, its performance is hindered by two issues, ignoring low-frequency but generic responses and bringing low-frequency but meaningless responses. In this paper, we propose a novel negative training paradigm, called negative distillation, to keep the model away from the undesirable generic responses while avoiding the above problems. First, we introduce a negative teacher model that can produce query-wise generic responses, and then the student model is required to maximize the distance with multi-level negative knowledge. Empirical results show that our method outperforms previous negative training methods significantly.
In the past few years, data-driven response generation (Vinyals and Le, 2015; Shang et al., 2015; Vougiouklis et al., 2016) has achieved impressive performance, drawing continuously increasing attention from academia and industry. Conventionally, with the guidance of maximum likelihood estimation (MLE), neural dialogue models are expected to maximize the probability of generating the corresponding reference given any query. Unfortunately, due to the many-to-one phenomenon (see Table 1), a characteristic of the dialogue task (Csaky et al., 2019), these models are prone to produce safe but generic responses (e.g., I don't know (Li et al., 2016)), which sets an obstacle for the generative dialogue system to be deployed widely. Some researchers tried to redesign the objective
of models to meet the requirement of diverse responses instead of MLE, such as MMI (Li et al., 2016), AdaLabel (Wang et al., 2021), and IAT (Zhou et al., 2021). Besides, several studies (Kulikov et al., 2019; Holtzman et al., 2020) proposed more advanced decoding strategies to alleviate the problem of generic responses. Indeed, the above methods boost the diversity of responses by reminding the model what should be said.
However, inspired by negative training (Kim et al., 2019; Ma et al., 2021), we argue that it is also necessary to tell the dialogue model what not to say. To alleviate the problem of generic responses, He and Glass (2020) negatively updates the parameters when identifying the high-frequency responses. Li et al. (2020a) punishes the behaviors of generating repetitive or high-frequency tokens by using the unlikelihood objective (Welleck et al., 2020).
Although the negative-training based methods enhance the diversity of responses, there still exists two drawbacks: First, they regard high-frequency tokens or utterances as negative candidates. However, the high-frequency response problem is only a sub-problem of the generic response problem (He and Glass, 2020). It means that the responses that are low-frequency but generic will escape from punishment. Even worse, we have observed that some generic responses followed by a low-frequency but meaningless subsequence can avoid being identified as high-frequency, which inevitably sacrifices the fluency of responses (see Analysis). Second, these methods ignore the implicit negative knowledge in neural networks that characterizes negative candidates at multiple levels. We contend that it is more effective to conduct negative training with richer information (e.g., hierarchical representation).
To tackle the above problems and further improve the diversity of responses, we propose a novel negative training paradigm called Negative Distillation (ND). Conventional knowledge distil
<sup>1</sup>The code and preprocessed data are available at https://github.com/Yiwei98/dialogue-negative-distillation.
Human Language Technologies, pages 407 - 418
Query	Response	Entropy Score
1:	Oh, honey, you made a mistake.	I don’t know how to do it.	8.61
2:	Would you like regular car wash package?	I don’t know what you mean.	8.75
3:	I’m looking for the airport.
Can you tell me how to get there?	No, sorry. I don’t know.	10.69
4:	That’s cathy. She is pretty, isn’t she?
She likes dancing. Ask her to dance.	Well, I don’t know.	12.14
5:	It doesn’t matter. You gotta find what she’s interested in and go with that.	I don’t know . . .	6.82
Table 1: The many-to-one phenomenon in DailyDialog. All the above five queries have the same I don't know-like responses. The corresponding source entropy (Csaky et al., 2019) scores are much higher than the median score (0.92) of the whole training set. This phenomenon will lead to the generic response problem.
lation (KD) (Hinton et al., 2015; Jiao et al., 2020) takes the teacher as a positive role model and induces the student to imitate. Differing from that, we train the teacher as a negative role model and remind the student to get rid of those bad behaviors.
Specifically, we first collect a negative training set by using a filtering method called Source Entropy (Csaky et al., 2019). This filtering method can retrieve all many-to-one cases of the raw dataset. Note that the "one" is usually a generic response. Then, we train a dialogue model on the above subset as the negative teacher. Given queries, the negative teacher can provide a set of negative candidates (i.e., generic and dull responses) that the student is prone to generate, which avoids the first drawback mentioned before. Therefore, the student obtains query-wise bad behaviors for Negative Distillation. To conduct the negative update holistically, we design two negative objectives, including soft unlikelihood loss on the prediction layer and reverse square error on the intermediate layer. In this way, the negative distillation fully exploits multilevel negative knowledge to force the student to generate non-generic responses.
- We propose a novel and effective negative training paradigm called Negative Distillation. It constructs query-wise generic responses as the negative candidates.
- We design two negative objectives to utilize multi-level information to further boost the performance of negative distillation.
- We perform extensive experiments and detailed analysis to verify the effectiveness of
the negative distillation framework and the superiority compared with previous negative training methods.
In this section, we first introduce the negative teacher, then describe the negative distillation on the prediction layer and the intermediate layer, respectively, and finally present the progressive optimization objective. Algorithm 1 shows the whole training details.
Dialogue Generation with MLE Take  $Q = \{q_{1}, q_{2}, \dots, q_{T_{q}}\}$  and  $R = \{r_{1}, r_{2}, \dots, r_{T_{r}}\}$  as the (query, response) pair, where  $T_{q}$  and  $T_{r}$  represent the length of query and response, respectively. The generative dialogue model aims to learn a conditional probability distribution  $p_{\theta}(R|Q)$ . The maximum likelihood estimation (MLE) is usually used to train the model, which can also be expressed as minimizing the negative log-likelihood:
$$
\mathcal {L} _ {\mathrm {M L E}} = - \sum_ {i = 1} ^ {T _ {r}} \log p _ {\theta} \left(r _ {i} \mid r _ {<   i}, Q\right). \tag {1}
$$
Considering one characteristic of the dialogue task, i.e., allowing the response to be varied, the many-to-one phenomenon occurs in the dialogue corpora frequently. However, with the MLE-based training, this phenomenon will cause the model to produce generic responses.
Unlikelihood Training Unlikelihood (UL) loss (Welleck et al., 2020) is proposed for the model to address the problem of undesirable behaviors (e.g.,
repetitive or high-frequency tokens). It forces the model to minimize the probability of generating negative candidates, which is formulated as:
$$
\mathcal {L} _ {\mathrm {U L}} = - \sum_ {i = 1} ^ {T _ {r}} \sum_ {r _ {c} \in \mathcal {C} _ {t}}
$$
$$
\log \left(1 - p _ {\theta} \left(r _ {c} \mid r _ {<   i}, Q\right)\right), \tag {2}
$$
where  $\mathcal{C}_t$  consists of negative candidates (e.g., overuse frequent words) that are also a sub-set of the vocabulary.
Knowledge Distillation The traditional knowledge distillation (KD) usually transfers useful knowledge from a large and strong teacher network  $T$  to a small student network  $S$ . The distillation loss is used to align the soften predictions of the teacher and the student, denoted as  $f^{T}(x)$  and  $f^{S}(x)$ :
$$
\mathcal {L} _ {\mathrm {K D}} = \sum_ {x \in \mathcal {D}} L \left(f ^ {T} (x), f ^ {S} (x)\right), \tag {3}
$$
where  $L(\cdot)$  is a measurement function that calculates the distance of different probability distributions,  $x$  is the input text, and  $\mathcal{D}$  denotes the training set.
In this work, we replace the positive teacher in vanilla KD with a negative teacher, aiming to provide negative knowledge for the student to conduct negative training and avoid undesirable behaviors.
2.2 Negative Teacher
To improve the diversity of responses, the dialogue model should be told which responses are generic. For negative distillation, a negative teacher is required to produce possible generic responses given any query. In this work, we adopt the widely used Transformer (Vaswani et al., 2017) as the underlying model for both teacher and student. We introduce the Source Entropy filtering method (Csaky et al., 2019) to identify and collect the many-to-one cases for the negative training set. The source entropy is defined as:
$$
H _ {s r c} (r, \mathcal {D}) = - \sum_ {(q _ {i}, r) \in \mathcal {D}} p (q _ {i} | r) \log p (q _ {i} | r), \quad (4)
$$
where  $p(q_i|r)$  is the conditional probability calculated based on the relative frequency of (query, response) pairs,  $r$  is a response,  $q_{i}$  is the query corresponding to the response  $r$ , and  $\mathcal{D}$  represents
the raw training set. A higher source entropy indicates that the response  $r$  corresponds to more queries, i.e., the many-to-one problem is serious. We select the top  $50\%$  dialogue pairs  $(q,r)$  with a high source entropy as the negative training set  $\mathcal{D}_N$ , which contains a much higher proportion of generic responses than the raw training set.
After that, we train the teacher  $N$  on the negative training set  $\mathcal{D}_N$  by Equation 1. The teacher will naturally produce generic responses for any input query. More importantly, it will provide richer negative knowledge for the student, including soft logits in the prediction layer and implicit features in the intermediate layers.
2.3 Negative Distillation
In this section, we conduct the negative distillation for the student based on the multi-level negative knowledge.
ND for Prediction Layer The soften logits in the prediction layer contain more information than the ground-truth labels, such as the similarity between labels (Wang et al., 2021). Therefore, conventional KD transfers knowledge by narrowing the gap between the probability distributions of the teacher  $T$  and the student  $S$ :
$$
\mathcal {L} _ {K D} = - \sum_ {i = 1} ^ {T _ {r}} \sum_ {k = 1} ^ {| \mathcal {V} |} p _ {T} (r _ {i} = k \mid r _ {<   i}, Q)
$$
$$
\cdot \log p _ {S} (r _ {i} = k \mid r _ {<   i}, Q). \tag {5}
$$
As for negative distillation, the extra knowledge in soften logits of the negative teacher reflects how to generate dull responses based on the input query. Therefore, we propose a soft unlikelihood loss to maximize the distance between the predictions of the negative teacher  $N$  and the student  $S$ :
$$
\mathcal {L} _ {p r e d} = - \sum_ {i = 1} ^ {T _ {r}} \sum_ {k = 1} ^ {| \mathcal {V} |} p _ {N} (r _ {i} = k \mid r _ {<   i}, Q)
$$
$$
\cdot \log (1 - p _ {S} (r _ {i} = k \mid r _ {<   i}, Q)), \tag {6}
$$
where  $p_N$  and  $p_S$  are calculated by:
$$
p ^ {i} = \frac {\exp (z _ {i} / t)}{\sum_ {j} \exp (z _ {j} / t)}, \tag {7}
$$
where  $t$  is a temperature coefficient that is used to soften the probability distribution over words.
2Simply the same as Akama et al. (2020)
It should be emphasized that previous negative training methods only use the high-frequency words or phrases with one-hot representation as the targets, which ignores the rich information existing in the soften logits (e.g., the generic words have similar probabilities). In the Analysis section, we demonstrates the superiority of soften logits compared with hard targets (i.e., one-hot representation).
ND for Intermediate Layer In addition to the output knowledge from the prediction layer, there is also some implicit knowledge embedded in the intermediate layers, such as hidden states and attention matrices. To keep the student away from undesirable behaviors (i.e., producing generic responses) more effectively, we further consider the above knowledge into negative distillation. Specifically, the distance between features of the negative teacher and the student should also be increased. In this work, we propose a new measurement function, called mean reverse square error (MRSE), to calculate this distance:
$$
\mathcal {L} _ {M R S E} (\boldsymbol {A}, \boldsymbol {B}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \exp^ {- S E \left(\boldsymbol {A} _ {i}, \boldsymbol {B} _ {i}\right)}, \tag {8}
$$
where  $\mathbf{A}$  and  $\mathbf{B}$  are the feature matrices of the negative teacher and the student, respectively, and  $n$  is the number of elements of each matrix.
Due to the responses generating in the decoding phrase, we only conduct negative distillation on the intermediate layers of the decoder. For each decoder layer, the negative distillation objective of hidden states is defined as:
$$
\mathcal {L} _ {h i d} ^ {l} = \mathcal {L} _ {M R S E} \left(\boldsymbol {H} _ {N} ^ {l}, \boldsymbol {H} _ {S} ^ {l}\right), \tag {9}
$$
where  $\pmb{H}_N^l$  and  $\pmb{H}_S^l$  are the output hidden states of the  $l^{th}$  decode layer of  $N$  and  $S$ , respectively.
As the attention weights can learn substantial linguistic knowledge (Clark et al., 2019), it is beneficial for the student to further conduct negative distillation on the attention matrices, which is computed as follows:
$$
\boldsymbol {A} = \frac {\boldsymbol {Q} \boldsymbol {K} ^ {T}}{\sqrt {d _ {k}}}, \tag {10}
$$
$$
\operatorname {A t t e n t i o n} (Q, K, V) = \operatorname {s o f t m a x} (A) V, \tag {11}
$$
where  $Q, K,$  and  $V$  are the matrices of queries, keys, and values, respectively, and  $d_{k}$  is a scaling
factor. Following Jiao et al. (2020), the attention matrix  $\mathbf{A}$  is chosen to calculate the distance rather than its softmax version softmax(A). Similar to Equation 9, the negative distillation objective of attention matrices is formulated as:
$$
\mathcal {L} _ {\text {a t t}} ^ {l} = \mathcal {L} _ {\text {M R S E}} \left(\boldsymbol {A} _ {N} ^ {l}, \boldsymbol {A} _ {S} ^ {l}\right), \tag {12}
$$
where  $A_N^l$  and  $A_S^l$  are the attention matrices of the  $l^{th}$  decoder layer of  $N$  and  $S$ , respectively.
Algorithm 1 Negative Distillation
Input:  $\mathcal{D}$ : The raw training set;  $H_{src}$ : The Source Entropy filtering method;  $N$  and  $S$ : The negative teacher and the student.
1:  $\%$  Collection of negative training set.
2: [Data_entropy]  $\leftarrow$  Calculate_data_entropy(D,  $H_{src}$ ) using Eq.4
3: Index_list  $\leftarrow$  Sort([Data_entropy])
4:  $\mathcal{D}_N\gets$  Extract_top_data(D, Index_list,  $50\%$
5: % Training of negative teacher.
6: repeat
7: Optimize  $N$  by minimizing  $\mathcal{L}_{mle}(N)$  on  $\mathcal{D}_N$  using Eq. 1
8: until Convergence
9: % Negative distillation.
10: repeat
11: Optimize  $S$  by minimizing  $\mathcal{L}(S)$  on  $\mathcal{D}$  using Eq. 13
12: until Convergence
Output:  $S$ : The trained student.
2.4 Progressive Optimization
The overall loss, combining the above negative distillation objectives and the MLE objective, is denoted as:
$$
\mathcal {L} = (1 - \alpha) \mathcal {L} _ {m l e} +
$$
$$
\alpha \left(\mathcal {L} _ {p r e d} + \sum^ {l} \mathcal {L} _ {h i d} ^ {l} + \sum^ {l} \mathcal {L} _ {a t t} ^ {l}\right), \tag {13}
$$
where  $\alpha$  is a hyper-parameter that balances the importance of supervised learning and negative distillation. For negative distillation, it would be better that the student has the ability to say something before it is reminded of what not to say. Thus, we perform a progressive distillation that first warms up the negative distillation ratio and then colds it down gradually. Inspired by the derivative of sigmoid function:
$$
\sigma^ {\prime} (z) = \sigma (z) (1 - \sigma (z)) = \frac {e ^ {- z}}{(e ^ {- z} + 1) ^ {2}}, \tag {14}
$$
Datasets	Train	Valid	Test	Vocab
DailyDialog	68k	6.8k	6.8k	17,930
OpenSubtitles	200k	20k	10k	21,177
Table 2: Statistics of two dialogue datasets in the experiments.
which shows a trend of gradual rise-fall, we define the balance coefficient  $\alpha$  as:
$$
\alpha = \lambda * \frac {e ^ {- z}}{\left(e ^ {- z} + 1\right) ^ {2}}, \tag {15}
$$
where  $\lambda$  controls the peak value and  $z$  is calculated by:
$$
z (s) = \beta * (s - \gamma), \tag {16}
$$
where  $s$  is the training step, and  $\beta$  and  $\gamma$  control the telescopic and translation transformation, respectively.
In our experiments, two widely used dialogue datasets are employed to evaluate the proposed method: DailyDialog, which collects conversations that are similar to human daily communication (Li et al., 2017b), and OpenSubtitles, which consists of large-scale dialogues extracted from movie subtitles (Tiedemann, 2009). In this work, we focus on the single-turn dialogue generation, thus we pre-process these two datasets into the (query, response) pairs. Table 2 provides the statistics of both datasets.
We take the Transformer-based sequence-to-sequence model (Vaswani et al., 2017) as the underlying model for all approaches. Following the settings of Transformer in Csaky et al. (2019), both encoder and decoder contain 6 layers, in which the self-attention module has 8 attention heads and the number of feed-forward units is 2048. The size of hidden states is set to 512 and the dimension is 64 for query, key, and value. Please refer to Appendix A for more details.
For the proposed approach, both the negative teacher network and the student network have the same settings in terms of the network architecture and hyper-parameters.  $\lambda$  in Equation 15 is set to 4, making the peak value equal to 1.  $\gamma$  is 25600 and  $\beta$  is  $6 / \gamma$ . For the temperature coefficient  $t$ , we simply set it to 1.
We compare the proposed negative distillation (ND) approach with the standard Transformer, two existing negative training approaches and two extra diversity improving approaches:
- Standard The vanilla Transformer-based sequence-to-sequence model with the MLE-based training (i.e., the cross-entropy based loss).
- NT (Negative Training) (He and Glass, 2020) During training, it first counts the frequency of all generated utterances and then conducts the negative update based on the high-frequency utterances.
- UL (Unlikelihood Training) (Li et al., 2020a) Different from NT, it calculates the frequency of all generated words instead of utterances and penalizes the high-frequency words by introducing an unlikelihood loss term.
- CVAE (Zhao et al., 2017) A dialogue response generation model using conditional VAE to improve the diversity of generated responses.
- FACE (Jiang et al., 2019) It uses the frequency-aware cross-entropy loss to tackle the low-diversity problem.
All the baselines are performed with the same architecture and hyper-parameters as ours. Following He and Glass (2020); Li et al. (2020a), we use greedy search as the decoding strategy for all baselines and our method. We also evaluate the performance with beam search (size 5) and obtain similar results (see 3.6 for details). Details for baselines is described in Appendix B.
3.4 Automatic Evaluation
Metrics To evaluate whether negative distillation can effectively reduce the generic responses, we adopt Dist-{1,2,3} (distinct) (Li et al., 2016) to reflect the lexical diversity of the generated responses. It is a widely used metric that counts the proportion of unique unigrams/bigrams/trigrams. LF (low-frequency token ratio) (Li et al., 2020b) further measures the diversity of responses by calculating the ratio of low-frequency words in the generated responses. The threshold of low frequency is set to 100. Besides, it is necessary to verify whether
Models	Dist-1 ↑	Dist-2 ↑	Dist-3 ↑	LF ↑	KL-1 ↓	KL-2 ↓	BLEU-3 ↑	BLEU-4 ↑
Standard	0.0089	0.0313	0.0576	0.102	0.96	0.53	0.384	0.395
NT	0.0059	0.0293	0.0760	0.070	1.05	1.84	0.226	0.183
UL	0.0062	0.0319	0.0882	0.075	1.07	1.88	0.228	0.187
CVAE	0.0024	0.0201	0.0821	0.029	1.54	2.02	0.144	0.087
FACE	0.0113	0.0412	0.0763	0.127	0.83	0.57	0.099	0.063
ND	0.0145	0.0678	0.1447	0.158	0.65	0.26	0.381	0.388
Standard	0.0020	0.0071	0.0147	0.022	2.19	1.40	0.355	0.353
NT	0.0011	0.0045	0.0108	0.014	2.26	2.39	0.255	0.216
UL	0.0015	0.0060	0.0151	0.018	1.85	1.97	0.303	0.269
CVAE	0.0009	0.0055	0.0182	0.013	2.76	2.70	0.134	0.084
FACE	0.0020	0.0079	0.0166	0.023	2.03	1.56	0.353	0.339
ND	0.0027	0.0102	0.0218	0.029	2.10	1.23	0.355	0.355
Table 3: Automatic evaluation results using greedy search on DailyDialog (Up) and OpenSubtitles (Down). The best/second-best results are **bold** underlined. "↑" means higher is better. "↓" means lower is better.
vs. Models	Informativeness	Kappa	Relevance	Kappa	Fluency	Kappa
Win(%)	Tie(%)	Lose(%)	Win(%)	Tie(%)	Lose(%)	Win(%)	Tie(%)	Lose(%)
Standard	77.3	18.7	4.0	0.456	48.0	34.7	17.3	0.453	14.7	76.0	9.3	0.491
NT	31.3	38.7	30.0	0.669	54.7	32.0	13.3	0.421	91.3	8.0	0.7	0.497
UL	44.7	28.7	26.7	0.411	66.0	23.3	10.7	0.425	92.7	7.3	0.0	0.614
Table 4: Results of human evaluations on DailyDialog. Our framework has a higher win rate than baselines.
the models can ensure consistency while improving diversity. So we use KL- $\{1,2\}$  (KL divergence) (Csaky et al., 2019), which measures the distribution distance between the generated and the ground-truth responses, to reflect how well a model can approximate the ground-truth unigrams/bigrams distributions. BLEU (Chen and Cherry, 2014) is also reported and it measures n-gram overlap between the generated and the ground-truth references.
Results Table 3 shows the results obtained at the lowest point of the validation loss. We can see that our approach outperforms all baselines in diversity (Dist and LF) by a significant margin on both datasets, demonstrating that ND can effectively alleviate the generic response problem by using multi-level negative information. The KL and BLEU scores of ND are close to or better than Standard, which verifies that our method can maintain the consistency of responses while improving its diversity. To some extent, both NT and UL improve the diversity of words, especially for trigrams, but the low LF scores indicate that they reduce the high-frequency words but fail to increase the number of low-frequency's. What's worse, BLEU and KL-2 scores of above two and CVAE sharply decline. It suggests that previous negative training approaches and other methods for diversity enhancement may harm the consistency
and fluency of responses dramatically, which is not in line with the goals of the dialogue system. Our method obtains similar results with beam search. Please refer to 3.6 for details.
Apart from automatic evaluations, we conduct human evaluations to further verify the effectiveness of our method than previous negative training methods. We randomly select 50 samples from the test set of DailyDialog, and three well-educated annotators are invited to judge which of the responses generated by ND and baselines is better (i.e., win, tie or loss) in terms of informativeness, relevance, and fluency. Informativeness reflects how much the information related to the query is contained in the generated response. Relevance reflects how likely the generated response is coherent to its query. Fluency reflects how likely the generated response is produced by human.