SlowGuess/ACL_2020_OCR huggingface dataset:
Lei Shen $^{1,2}$  Yang Feng $^{1,2*}$
<sup>1</sup>Key Laboratory of Intelligent Information Processing,
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
{shenlei17z,fengyang}@ict.ac.cn
Emotion-controllable response generation is an attractive and valuable task that aims to make open-domain conversations more empathetic and engaging. Existing methods mainly enhance the emotion expression by adding regularization terms to standard cross-entropy loss and thus influence the training process. However, due to the lack of further consideration of content consistency, the common problem of response generation tasks, safe response, is intensified. Besides, query emotions that can help model the relationship between query and response are simply ignored in previous models, which would further hurt the coherence. To alleviate these problems, we propose a novel framework named Curriculum Dual Learning (CDL) which extends the emotion-controllable response generation to a dual task to generate emotional responses and emotional queries alternatively. CDL utilizes two rewards focusing on emotion and content to improve the duality. Additionally, it applies curriculum learning to gradually generate high-quality responses based on the difficulties of expressing various emotions. Experimental results show that CDL significantly outperforms the baselines in terms of coherence, diversity, and relation to emotion factors.
Infusing emotions into dialogue systems can make conversational agents more human-like and benefit the interaction between human and machine (Prendinger and Ishizuka, 2005; Prendinger et al., 2005; Partala and Surakka, 2004). In some real-life scenarios, we need to customize and control the agent's emotion so that the agent can express a specific one. For example, in psychological counseling, the agent is supposed to express sadness to
show the sympathy and also convey happiness to cheer the patient up.
1	q	It is very pleasant to have a cup of black tea with sugar on a cold day. (Happy)
r1	[Neural] It starts to cool down today.
r2	[Like] I will try, thanks for your advice.
r3	[Sad] I am frozen to death ...
r4	[Disgust] Winner is the worst season.
r5	[Angry] You know nothing!
r6	[Happy] I really like to drink black tea.
2	q	So pets live better than humans now... (Sad)
r1	[Disgust] You are so bad.
r2	[Happy] Haha, you too.
3	q	We should study hard. (Neural)
r	[Disgust] You are so bad.
4	q	Happy birthday, Xinxin. May you be more beautiful, find a good person and get married soon! (Happy)
r	[Happy] Haha, you too.
Table 1: Examples of emotion-controllable response generation (response emotions are denoted in brackets). Example 1 is one query and 6 emotional responses. Example 2 and 3 have different queries, but the responses generated with emotion "Disgust" are the same. Similar to Example 2 and 4 with emotion "Happy". The emotions of queries are marked in parentheses.
Recently, a framework called emotional chatting machine (ECM) (Zhou et al., 2018a) was proposed to address the emotion factor in a controlled manner, which focuses on generating a response with a specific emotion (Example 1 in Table 1). In the research field of emotion-controllable response generation, ECM and its successive methods (Colombo et al., 2019; Song et al., 2019) mainly represent the given emotion category as a vector and add it to the decoding steps to influence the procedure of response generation, which would aggravate the safe response problem. For the response generation task, safe response is notorious, as the model tends to produce some generic but meaningless responses, like "Thank you", "I don't know", "Yes",
* Yang Feng is the corresponding author.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 556-566 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
etc. Due to the constraint of emotion factors, the scale of proper responses shrinks, and the model is more likely to map any query to a frequently-occurring response in that emotion category. That is, given "Disgust", the response would be "You are so bad" in general, while given "Happy", it would be "Haha, you too" (Example 2 to 4 in Table 1).
Intuitively, for a good pair of query and response, they should be in a tight relationship and have equal qualities. Then, both the query-to-response mapping and response-to-query mapping would be easier and more natural. On the contrary, it is hard for a safe response to reach the original query through back-generation, neither on the content level nor the emotion level. At the same time, the difficulties of producing various emotions are different, especially in a noisy and uneven-quality dataset. Therefore, we can evaluate the response based on the feedback from the backward process to improve the coherence (Zhang et al., 2018; Cui et al., 2019; Luo et al., 2019b) and try to learn from easy to hard data to generate appropriate and emotion-rich responses.
In this paper, we propose a new framework for emotion-controllable response generation named Curriculum Dual Learning (CDL). We take the learning of response and query generation with emotions as a dual task, and use the duality to model the mutual relation between them. The forward and backward models are trained alternatively via reinforcement learning (RL). Rewards designed here aim to encourage both emotion expression and content consistency. Specifically, emotion expression can be either explicit (embodied in some obvious emotion words) or implicit (reflected by the organization of the entire sentence). For example, "I am happy to meet her again" is explicit with the word "happy", while "It seems like I have eaten the honey" is implicit, but the happiness can be felt when we consider the sentence as a whole. Based on these features, we use the accuracy of emotion classification of sentences and the proportion of emotion words as feedbacks for explicit and implicit emotions, respectively. For content consistency, we apply the reconstruction probability as the measurement of coherence (Section 3.1). Furthermore, in order to better utilize samples of multiple emotions from the noisy and uneven-quality dataset, we incorporate the curriculum learning (Section 3.2) into our dual learning framework (Section 3.3).
Experimental results on both automatic and human evaluations show that for a given query and an emotion category, our CDL can successfully express desired emotion as well as keep the response informative and coherent to the query.
For emotion-controllable response generation, given a query  $\mathbf{q}$  and an emotion category  $e_r$ , the goal is to generate a response  $r'$  that is not only meaningful, but also in accordance with the desired emotion.
Emotional Chatting Machine (ECM) (Zhou et al., 2018a) addresses the emotion factor using three new mechanisms: Emotion Category Embedding, Internal Memory, and External Memory. Specifically, 1) Emotion Category Embedding models the high-level abstraction of emotion expression by embedding emotion categories, and concatenates corresponding embedding to the input at each decoding step. 2) Internal Memory captures the change of implicit internal emotion states with read and write gates, 3) External Memory applies an external emotion vocabulary to express emotion explicitly, and finally assigns different generation probabilities to emotion and generic words. The loss function on one training sample  $(q,r)$ $(q = q_{1},q_{2},\dots,q_{n},r = r_{1},r_{2},\dots,r_{m})$  is defined as:
$$
- \sum_ {t = 1} ^ {m} \boldsymbol {p} _ {t} \log (\boldsymbol {o} _ {t}) - \sum_ {t = 1} ^ {m} q _ {t} \log (\alpha_ {t}) + | | \boldsymbol {M} _ {e, m} ^ {I} | |, \tag {1}
$$
where  $\boldsymbol{o}_t$  and  $\boldsymbol{p}_t$  are the predicted token distribution and gold distribution,  $\alpha_t$  is the probability of choosing an emotion word or a generic word,  $q_t \in \{0,1\}$  is the true choice between them in  $\boldsymbol{r}$ , and  $M_{e,m}^I$  is the internal emotion state at the last step  $m$ . The first term is the cross-entropy loss, the second one is used to supervise the probability of selecting an emotion or generic word, and the last one is used to ensure that the internal emotion state has been expressed completely once the generation is finished. Please refer to the original paper for more details.
3 CDL for Emotion-Controllable Response Generation
Since our CDL method is a combination of dual learning (DL) and curriculum learning (CL), we first present the main components of DL, including states, actions, policy and reward, then introduce
the plausibility of curriculum learning. Finally, we describe the training algorithm of CDL.
Figure 1: The architecture of dual learning.  $CLS$ ,  $M_{f}$  and  $M_{b}$  are emotion classifier, forward model and backward model, respectively. Red parts are for the forward process, while blue parts are for the backward process.
3.1 DL Architecture
The architecture of DL is illustrated in Figure 1. Both the forward model  $M_{f}$  and the backward model  $M_{b}$  are ECMs with independent parameters and are initialized according to the maximum likelihood estimation (MLE). CLS is a pre-trained classifier that calculates the score of implicit emotion expression.
In general,  $M_{f}$  generates a response  $r'$  for a given query  $q$  and emotion category  $e_{r}$ , and then obtains the reward  $R$  that consists of  $R_{e}$  from CLS and  $R_{c}$  from  $M_{b}$  (red parts in Figure 1). Similarly,  $M_{b}$  generates a query  $q'$  for a given response  $r$  and emotion category  $e_{q}$ , and obtains the reward  $R$  that consists of  $R_{e}$  and  $R_{c}$  from CLS and  $M_{f}$  (blue parts in Figure 1). These two models are trained alternatively via reinforcement learning (RL). Specifically, an action is the dialogue response to generate. The action space is infinite since arbitrary-length sequences can be generated. A state is denoted by the query, which is further transformed to a vector representation by the encoder. A policy takes the form of a GRU encoder-decoder and is defined by its parameters. Following the work of Li et al. (2016c); Zhang et al. (2018), we use a stochastic representation of the policy, i.e., a probability distribution over actions given states.
In order to encourage both content consistency and emotion expression, we introduce two rewards and use them to train  $M_{f}$  and  $M_{b}$ . The definition
of the two rewards for model  $M_{f}$  is introduced as follows<sup>1</sup>.
Reward for emotion expression For implicit emotion expression, a straightforward method is to employ the pre-trained classifier  $CLS$  to evaluate the emotion category of the generated response  $\boldsymbol{r}^{\prime}$  and use the classification accuracy as the reward:
$$
R _ {e _ {1} (\boldsymbol {q}, \boldsymbol {r} ^ {\prime})} = p \left(e _ {r} \mid \boldsymbol {r} ^ {\prime}; \varphi\right), \tag {2}
$$
where  $\varphi$  is the parameter of  $CLS$ , and it is fixed during training. For explicit emotion expression, the reward is formulated as:
$$
R _ {e _ {2} \left(\boldsymbol {q}, \boldsymbol {r} ^ {\prime}\right)} = n \left(w _ {e _ {r}}\right) / \left| \boldsymbol {r} ^ {\prime} \right|, \tag {3}
$$
where  $n(w_{e_r})$  is the number of emotion words belong to category  $e_r$ , and  $|\pmb{r}^{\prime}|$  is the length of  $\pmb{r}^{\prime}$ . Then, the emotion reward is defined as:
$$
R _ {e \left(\boldsymbol {q}, \boldsymbol {r} ^ {\prime}\right)} = R _ {e _ {1} \left(\boldsymbol {q}, \boldsymbol {r} ^ {\prime}\right)} + \lambda R _ {e _ {2} \left(\boldsymbol {q}, \boldsymbol {r} ^ {\prime}\right)}, \tag {4}
$$
where  $\lambda$  controls the relative importance of implicit and explicit rewards.
Reward for content consistency If the response are coherent and related to the query, it will be easier to reproduce the query via back generation. Inspired by Zhang et al. (2018); Cui et al. (2019); Luo et al. (2019b), we measure the coherence by means of reconstructing  $q$  conditioned on  $r'$ . Formally, the content consistency reward is defined as:
$$
R _ {c \left(\boldsymbol {q}, \boldsymbol {r} ^ {\prime}\right)} = p \left(\boldsymbol {q} \mid \boldsymbol {r} ^ {\prime}, e _ {q}; \eta\right), \tag {5}
$$
where  $\eta$  is the parameter of backward model  $M_{b}$  and it is fixed during the training of  $M_{f}$ .
Overall reward We use the weighted sum of the above two rewards as the final reward:
$$
R _ {(\boldsymbol {q}, \boldsymbol {r} ^ {\prime})} = R _ {c (\boldsymbol {q}, \boldsymbol {r} ^ {\prime})} + \gamma R _ {e (\boldsymbol {q}, \boldsymbol {r} ^ {\prime})}, \tag {6}
$$
where  $\gamma$  is a hyper-parameter that controls the tradeoff between  $R_{c(\pmb{q},\pmb{r}^{\prime})}$  and  $R_{e(\pmb{q},\pmb{r}^{\prime})}$ .
3.2 Curriculum Plausibility
Intuitively, learning from less noisy and even-quality dataset is simpler, but in this task, the data is inherently complicated as there are multiple emotions mixed in it. To better utilize the data, we integrate curriculum learning into the dual learning framework. The core of curriculum learning
Rewards for model  $M_{b}$  can be computed in a similar way, where  $q^{\prime},r,b$  and  $f$  replace  $r^\prime ,q,f$  and  $b$  respectively. Therefore, we omit them here for space limitation and brevity.
(Bengio et al., 2009) is to design an evaluation for complexity, and to provide the model with easy samples first, then gradually increase the difficulty. The curriculum is arranged by sorting each sample in training set according to a specific ranking standard.
Here, We reorder samples from easy, i.e., with high accuracy of emotion classification, to hard. We consider the classification accuracy after pretraining as an indicator of the learning order. Another intuitive way is to put emotionless samples (labelled as "Neural") first and then emotional ones, however, it exhibits poor performance in our experiments. At training step  $t$ , a batch of training samples is obtained from the top  $f(t)$  portions of the entire sorted training samples. Following Platanios et al. (2019) and Cai et al. (2020), we define the function  $f(t)$  as:
$$
f (t) \triangleq \min  (1, \sqrt {\frac {t \left(1 - c _ {0} ^ {2}\right)}{T} + c _ {0} ^ {2}}), \tag {7}
$$
where  $c_0^2$  is set to 0.01, which means that the model starts training using the  $1\%$  easiest training samples, and  $T$  is a hyper-parameter that represents the duration of curriculum learning (curriculum length). At the early stage of the training process, the model learns from the samples in the easy part of the curriculum, where there is only one emotion category. As the advance of the curriculum, the difficulty gradually increases, as complex training samples from more different categories appear. After training  $T$  batches, training sample of each batch is drawn from the whole training set, which is the same as the conventional training procedure.
3.3 Training of CDL
Optimization We use the policy gradient method (Williams, 1992) to find parameters that lead to a larger expected reward. For the forward learning process, the expected reward of the generated response  $r'$  and its approximate gradient are defined as:
$$
J (\theta) = \mathbb {E} \left[ R _ {\left(\boldsymbol {q}, \boldsymbol {r} ^ {\prime}\right)} \right], \tag {8}
$$
$$
\nabla_ {\theta} J (\theta) \simeq R _ {(\boldsymbol {q}, \boldsymbol {r} ^ {\prime})} ^ {\prime} \cdot \nabla_ {\theta} \log \left(p _ {\theta} \left(\boldsymbol {r} ^ {\prime} \mid \boldsymbol {q}, e _ {r}\right)\right), \tag {9}
$$
where  $\theta$  is the parameter of forward model  $M_{f}$ ,  $R_{(q,r^{\prime})}^{\prime} = R_{(q,r^{\prime})} - b_{f}$ , and  $b_{f}$  is the baseline value from the greedy search decoding method for  $M_{f}$ , which is used to reduce the variance of the estimation (Zaremba and Sutskever, 2015; Paulus et al., 2017). Analogously, for the backward learning process, the expected reward of the generated query
$q^{\prime}$  and corresponding approximate gradient are defined as:
$$
J (\eta) = \mathbb {E} \left[ R _ {\left(\boldsymbol {r}, \boldsymbol {q} ^ {\prime}\right)} \right], \tag {10}
$$
$$
\nabla_ {\eta} J (\eta) \simeq R _ {(\boldsymbol {r}, \boldsymbol {q} ^ {\prime})} ^ {\prime} \cdot \nabla_ {\eta} \log \left(p _ {\eta} \left(\boldsymbol {q} ^ {\prime} \mid \boldsymbol {r}, e _ {q}\right)\right), \tag {11}
$$
where  $\eta$  is the parameter of backward model  $M_{b}$ ,  $R_{(r,q^{\prime})}^{\prime} = R_{(r,q^{\prime})} - b_{b}$ , and  $b_{b}$  is the baseline value from the greedy search decoding method for  $M_{b}$ .
Algorithm 1 Curriculum dual learning algorithm for emotion-controllable response generation
Input: The training set  $\mathcal{D} = \{(q_i, e_{q_i}, r_i, e_{r_i})\}$  where each query-response pair is labelled with corresponding emotion labels  $e_{q_i}$  and  $e_{r_i}$
Output:  $M_{f}$  and  $M_{b}$
1: Pre-train  $M_{f}$  and  $M_{b}$  with  $(q_{i},r_{i},e_{r_{i}})$  and  $(r_i,q_i,e_{q_i})$ , respectively, based on Eq. 1
2: Pre-train CLS with  $(q_i, e_{q_i})$  and  $(r_i, e_{r_i})$
3: Sort training samples according to the ranking standard in Section 3.2 for both forward and backward learning process to get  $\mathcal{D}_f$  and  $\mathcal{D}_b$
4: for training step  $t = 1, \dots, T$  do
5: Train  $M_{f}$  6: Sample a batch  $B_{f_t}$  in  $\mathcal{D}_f$  based on Eq. 7
7: Sample  $(q,r,e_r)$  from  $B_{f_t}$
8: Generate response  $r'$  via  $M_f$
9: Compute reward  $R$  based on Eq. 6
10: Update  $\theta$  using  $R$  based on Eq. 9.
11: Teacher Forcing: Update  $\theta$  with  $(q,r,e_r)$
12:  $\triangleright$  Train  $M_{b}$
13: Sample a batch  $B_{b_t}$  in  $\mathcal{D}_b$  based on Eq. 7
14: Sample  $(r,q,e_q)$  from  $B_{b_t}$
15: Generate response  $q^{\prime}$  via  $M_{b}$
16: Compute reward  $R$  based on Eq. 6
17: Update  $\eta$  using  $R$  based on Eq. 11
18: Teacher Forcing: Update  $\eta$  with  $(r,q,e_q)$
19: end for
Teacher Forcing When  $M_{f}$  and  $M_{b}$  are trained with only the rewards from the dual tasks, the training process would easily collapse as it may find an unexpected way to achieve a high reward but fail to guarantee the fluency or readability of the generated text (Ranzato et al., 2015; Pasunuru and Bansal, 2018; Luo et al., 2019b). To stabilize the training process, after each update according to Eq. 9 or 11,  $M_{f}$  or  $M_{b}$  is exposed to real query-response pairs and is trained via MLE, which is also known as Teacher Forcing (Li et al., 2017; Lamb et al., 2016).
The training procedure of CDL is summarized in Algorithm 1. First, we use MLE to pre-train
$M_{f}$ ,  $M_{b}$  and  $CLS$  with query-response pairs and emotion labels in the training set. After the pretraining phase, we sort samples in the training set following the ranking standard in Section 3.2. For forward learning process, the ranking is based on responses, while for backward learning process, it is based on queries. Then, we can get two sorted training set  $\mathcal{D}_f$  and  $\mathcal{D}_b$  for each direction. Finally,  $M_{f}$  and  $M_{b}$  are optimized with rewards and the regularization of Teacher Forcing, alternatively.
In this section, we conduct experiments to evaluate our proposed method. We first introduce some empirical settings, including dataset, hyperparameters, baselines, and evaluation measures. Then we illustrate our results under both automatic and human evaluations. Finally, we give out some cases generated by different models and do further analyses over our method.
We apply our method on the corpus of NLPCC 2017 Emotional Conversation Generation Challenge $^{2}$ , namely NLPCC2017 Dataset, which is an extension version of the dataset collected by Zhou et al. (2018a). The provided dataset is already segmented into Chinese words. There are over 1 million query-response pairs, in which both the query and response are labelled with one emotion tag among "Happy", "Angry", "Disgust", "Sad", "Like" and "Neutral". The dataset has been tokenized into words. We randomly split the whole dataset into training/validation/test set with the number of 1,105,487/11,720/2,000. The detailed statistics of training set are shown in Table 2.
Emotion	Query	Response
Training	Happy	120,358	197,528
Angry	79,611	138,198
Disgust	184,427	197,428
Sad	128,482	179,215
Like	257,471	197,565
Neutral	335,138	195,553
1,105,487
Validation	11,720
Test	2,000
Table 2: Statistics of the NLPCC2017 Dataset. In the training set, we count the number of queries and responses for each emotion category.
4.2 Hyper-parameter Settings
The settings of both  $M_{f}$  and  $M_{b}$  follow the default implementation details of original ECM paper (Zhou et al., 2018a), where the encoder and decoder have 2-layer GRU structures with 256 hidden cells for each layer, the embedding size of words and emotion categories are set to 100, and the vocabulary size is limited to 40,000. The minimum and maximum sentence length is set to 3 and 30, respectively. We train a TextCNN-based classifier (Kim, 2014) and the classification accuracy reaches  $65.6\%$  on the test set, which has the similar performance with those used by (Zhou et al., 2018a) and (Song et al., 2019). Before curriculum dual learning, model  $M_{f}$  and  $M_{b}$  are pre-trained 10 epochs via MLE. The optimizer is Adam (Kingma and Ba, 2015) with 0.05 initial learning rate for pre-training and  $10^{-5}$  for curriculum dual learning. The batch size is set to 64.  $\lambda$  in Eq. 4 is 0.5,  $\gamma$  in Eq. 6 is 1 and  $T$  in Eq. 7 is 100k. During curriculum dual learning, training runs until the performance on validation set does not improve.
We compare our approach with four representative baselines: (1) S2S-Attn: The Seq2Seq model with attention mechanism as in Shang et al. (2015). (2) EmoEmb: A Seq2Seq variant which takes the embedding of emotion categories as additional input at each decoding position (Ficler and Goldberg, 2017; Li et al., 2016b). (3) EmoDS: An emotional dialogue system with lexicon-based attention and a word-based classifier (Song et al., 2019). (4) ECM: Emotional Chatting Machine proposed by Zhou et al. (2018a).
Additionally, we also conduct ablation study to better analyze our method as follows: (5) CDL-emo: CDL with emotion reward only; (6) CDL-con: CDL with content reward only, which is similar to the work of Zhang et al. (2018); (7) CDL-DL: CDL with both rewards but without curriculum learning.
4.4 Evaluation Measures
To better evaluate our results, we use both quantitative metrics and human judgements in our experiments.
4.4.1 Automatic Metrics
For automatic evaluation, we mainly choose four kinds of metrics: 1) Embedding scores (Average,
<sup>2</sup>http://coai.cs.tsinghua.edu.cn/html/challenge2017/
Method	Embedding Metrics	Diversity	BLEU Scores	Emotion Expression
Avg.	Ext.	Gre.	Coh.	Dist-1	Dist-2	BLEU-1	BLEU-2	Emo-acc.	Emo-word.
S2S-Attn	0.497	0.352	0.328	0.582	0.035	0.119	0.0424	0.0073	0.244	0.285
EmoEmb	0.532	0.381	0.356	0.594	0.040	0.133	0.0722	0.0164	0.693	0.436
EmoDS	0.623	0.427	0.403	0.603	0.050	0.174	0.0976	0.0282	0.746	0.527
ECM	0.625	0.433	0.405	0.607	0.052	0.177	0.1023	0.0332	0.753	0.562
CDL-emo (ours)	0.631	0.451	0.435	0.615	0.058	0.193	0.1162	0.0342	0.765	0.583
CDL-con (ours)	0.628	0.441	0.417	0.612	0.055	0.182	0.1059	0.0338	0.758	0.566
CDL-DL (ours)	0.635	0.452	0.431	0.630	0.062	0.217	0.1187	0.0353	0.794	0.615
CDL (ours)	0.642	0.457	0.438	0.635	0.065	0.221	0.1254	0.0370	0.823	0.620
Table 3: Automatic evaluation results for content and emotion measurements. The metrics Average, Extrema, Greedy, Coherence, Emotion-acc and Emotion-word are abbreviated as Avg., Ext., Gre., Coh., Emo-acc. and Emo-word., respectively.
Method	Like	Sad	Disgust	Angry	Happy	Overall
Con.	Emo.	Con.	Emo.	Con.	Emo.	Con.	Emo.	Con.	Emo.	Con.	Emo.
S2S-Attn	1.295	0.435	1.125	0.120	1.160	0.115	1.255	0.045	1.155	0.305	1.198	0.204
EmoEmb	1.290	0.630	0.990	0.225	1.125	0.295	1.220	0.220	1.275	0.400	1.180	0.354
EmoDS	1.375	0.685	1.210	0.395	1.200	0.340	1.225	0.345	1.260	0.535	1.254	0.460
ECM	1.375	0.690	1.205	0.425	1.205	0.325	1.240	0.385	1.255	0.590	1.256	0.483
CDL	1.395	0.700	1.245	0.565	1.235	0.490	1.250	0.525	1.305	0.630	1.286	0.582
Table 4: Human evaluation results. "Con." and "Emo." denote content and emotion, respectively.