SlowGuess/ACL_2020_OCR huggingface dataset:
Greedy, Extrema and Coherence $^{3}$  (Liu et al., 2016; Xu et al., 2018); 2) BLEU scores (Papineni et al., 2002) in 0 to 1 scale; 3) Dist-1, Dist-2 (Li et al., 2016a) and 4) Emotion-acc, Emotion-word (Zhou et al., 2018a; Song et al., 2019).
Embedding scores and BLEU scores are used to measure the quality of generated responses in terms of content relevance. Whereas, Dist-1 and Dist-2 are used to evaluate the diversity of responses<sup>4</sup>. Emotion-acc and Emotion-word are utilized to test the emotion expression. Specifically, Emo-acc is the agreement between the ground truth labels and the predicted labels through the TextCNN classifier trained before. Emo-word is the percentage of the generated responses that contain the corresponding emotion words. Since there are no multi-emotion ground truths in the test set, we only calculate the metrics between the ground truth, labelled emotion  $e$ , and the generated response given also label  $e$  for fair comparison.
4.4.2 Human Evaluation Settings
Inspired by Zhou et al. (2018a); Song et al. (2019), a human evaluation is conducted to better analyze the quality of generated responses. First, we randomly sample 200 queries from the test set. For
each method except S2S-Attn, they generate six responses for six emotion categories, while S2S-Attn generates top 6 responses from beam search decoding for each query. Then, we send the triples of (query, response, emotion) to three human annotators without order, and require them to evaluate each response on both content level and emotion level independently.
Content and emotion are measured by a 3-scale rating (0, 1, 2) and a 2-scale rating (0, 1), respectively. Evaluation from the content level assesses whether a response is fluent, coherent and meaningful for the query, and evaluation from the emotion level decides if a response reveals the desired emotion.
Now we demonstrate our experimental results on both automatic evaluation and human evaluation.
4.5.1 Automatic Evaluation Results
The automatic results are shown in Table 3. The top part is the results of all baseline models, and we can see that CDL outperforms the other methods on all metrics (t-test,  $p$ -value  $< 0.05$ ). The improvements of CDL on Coherence, Emotion-acc and Emotion-word are significant, indicating that it can enhance content consistency and emotion expression simultaneously. EmoS and ECM have similar performance, as both of them use the forward method to pay more attention on the emotion
3We use the pre-trained word embeddings based on Sina Weibo data from https://github.com/Embedding/ Chinese-Word-Vectors.
4We employ a popular NLG evaluation project available at https://github.com/Maluuba/nlg-eval for automatic evaluation.
Query	Method	Response	Translated Response
创意源于生活,看你对生
活的态度如何。
Creativity comes from life,
and depends on your
attitude.	S2S-Attn	生活需要改变心态。	Life needs a change of mindset.
ECM	Like
Sad
Disgust
Angry
Happy	希望有个好心情。
我也是很纠结的。
你太过分了。
你怎么还没睡觉?
哈哈,是啊。	Hope to have a good mood.
I am also very tangled.
You are too much.
how come you are still up?
Haha, yes.
CDL	Like
Sad
Disgust
Angry
Happy	希望每天都有好心情。
我觉得我很不淡定。
别说废话了。
你根本不懂生活!
开心是最重要的。	Hope to have a good mood every day.
I do not think I am calm.
Stop talking nonsense.
You do not understand life at all!
Happiness is the most important.
善良的人还是存在的。
Kind people still exist.	S2S-Attn	我不做好事。	I do not do good things.
ECM	Like
Sad
Disgust
Angry
Happy	是的,喜欢就好。
是啊,我很无辜的。
不是不是,不是好人。
你是什么意思????哈哈,你也是。	Yes, it is ok you like it.
Yeah, I am innocent.
No, no, not a good person.
What do you mean???Haha, you too.
CDL	Like
Sad
Disgust
Angry
Happy	你是不是在夸我?
可惜我不想做。
听起来像假话。
我一点也没觉得。
要对生活充满希望。	Are you praising me?
Unfortunately, I do not want to be.
It sounds like a lie.
I feel nothing at all.
Be hopeful about life.
我最喜欢的季节就是秋
天。
Autumn is my favorite
season.	S2S-Attn	我也很喜欢。	I like it, too.
ECM	Like
Sad
Disgust
Angry
Happy	我也喜欢秋天。
我也想念秋天。
你太过分啦。
你怎么不说我是爱?
哈哈,我也喜欢。	I also like autumn.
I also miss autumn.
You are too much.
Why don't you say that I love it?
Haha, I like it too.
CDL	Like
Sad
Disgust
Angry
Happy	秋天的天空非常美丽。
我很想念过去的夏天。
秋天的景色很难看。
你有病吧?
哈哈,要不要去秋游?	The autumn sky is very beautiful.
I miss the past summer very much.
The autumn scenery is ugly.
Are you insane?
Haha, do you want to go to the autumn tour?
factor. S2S-Attn can only generate fluent responses based on semantic mapping, but fail to express diverse responses.
The bottom part of Table 3 shows the results of our ablation study. Comparisons among CDL-emo, CDL-con and CDL show the effectiveness of the combined reward for both emotion expression and content consistency. In addition, we can find that with the support of curriculum learning, CDL can achieve better results than CDL-DL.
4.5.2 Human Evaluation Results
The results are shown in Table 4. CDL obtains the best performance (t-test,  $p$ -value  $< 0.05$ ) on both emotion expression (0.582) and content coherence (1.286). As we can see, there is no obvious difference between EmoS and ECM. Due to the insufficient training data of "Anger" (79,611 in queries and 138,198 in responses), S2S-Attn achieves the best content score for it, which is similar to the
results of Zhou et al. (2018a).
Table 5: Sample responses generated by S2S-Attn, ECM and CDL (original Chinese and English translation). The colored words are the emotion words corresponding to the given emotion category.
Method (%)	2-1	1-1	0-1	2-0	1-0	0-0
S2S-Attn	10.3	7.2	2.8	36.4	26.5	16.8
EmoEmb	21.8	12.6	7.5	24.6	15.3	18.2
EmoDS	28.7	15.6	4.0	22.7	13.5	15.5
ECM	27.1	12.7	4.5	23.5	15.4	16.8
CDL	32.5	17.6	4.1	17.7	12.8	15.3
Table 6: The percentage of responses in human evaluation of Content-Emotion scores. 2-1 means content score is 2 and emotion score is 1.
Results of emotion and content in Table 4 are independent. To better evaluate the overall quality of the generated responses, we present results in Table 6 by considering content and emotion scores simultaneously.  $32.5\%$  of the responses generated by CDL are annotated with Emotion score 2 and Content score 1, which shows that CDL is better at producing coherent as well as emotion-rich responses.
Agreements to measure the consistency among three annotators are calculated with the Fleiss' kappa (Fleiss and Cohen, 1973). Fleiss' kappa for content and emotion is 0.497 and 0.825, indicating "Moderate agreement" and "Substantial agreement", respectively.
4.6 Case Study
Table 5 shows the examples generated by S2S-Attn, ECM and CDL. As can be seen from it, for a given post, there are multiple emotion categories that are appropriate for its response in the conversation. S2S-Attn generates a response with a random emotion, while ECM and CDL can utilize the specific emotion label. Compared with ECM, CDL can generate both coherent and informative responses with any desired emotion. In addition, the emotion can be expressed in either explicit or implicit manner. For example, "你/根本/不懂/生活! (You do not understand life at all!)" express anger when we read this sentence as a whole, while "美丽 (beautiful)" or "开心 (happy)" are strong emotion words to represent "Like" or "Happy".
4.7 Further Analysis of CDL
Here, we conduct a further analysis to show some characteristics of this task and the effect of CDL. Emotion lexicon size and classification accuracy after pre-training of each category ( $N$  (correct prediction)  $\div$  category size) are listed in Table 7. We can see that the classification accuracy is not totally related to the emotion lexicon size, indicating the emotion expression is partially implicit or explicit. To better illustrate the learning efficiency of CDL, we plot the changes of Emotion-acc on the validation set. As shown in Figure 2, CDL accelerates the learning effectively and consistently outperforms CDL-DL.
Figure 2: Comparison of CDL and CDL-DL for Emotion-acc on the validation set.
Like	Sad	Disgust	Angry	Happy
Lex. Size	1,629	294	1,142	30	405
ACC (f)	0.653	0.691	0.609	0.736	0.818
ACC (b)	0.690	0.655	0.602	0.756	0.808
Table 7: Emotion lexicon size and classification accuracy after pre-training of each emotion category. "Lex.", "ACC(f)" and "ACC(b)" represent lexicon, classification accuracy of forward process and classification accuracy of backward process, respectively.
Responses generated by traditional open-domain dialogue systems are usually safe and generic. To produce diverse and informative responses, researchers tried to either import latent variables for model construction (Zhao et al., 2017; Serban et al., 2017; Shen et al., 2019) or utilize some extra knowledge, e.g., sentence types, personas, emotions, documents and knowledge triples/graphs (Ke et al., 2018; Li et al., 2016b; Zhou et al., 2018a; Meng et al., 2019; Zhou et al., 2018b; Niu et al., 2019). In this paper, we mainly touch on two branches of research: emotional response generation and dual learning in NLP.
5.1 Emotional Response Generation
Early studies have proven that dialogue systems with proper emotional expressions and reactions can directly improve user satisfaction (Prendinger and Ishizuka, 2005; Prendinger et al., 2005) and contribute to effective users' performance (Partala and Surakka, 2004). Polzin and Waibel (2000) and Polzin and Waibel (2000) apply rule-based methods to choose emotional responses from a conversation corpus, but those rules are hard to extend to large corpora. With the advent of deep learning, some researchers utilize neural networks to solve this problem (Ghosh et al., 2017; Hu et al., 2017; Zhou and Wang, 2018; Sun et al., 2018). Besides, the Valence, Arousal, and Dominance (VAD) lexicon (Warriner et al., 2013; Mohammad, 2018) is embedded to the sequence-to-sequence model (Sutskever et al., 2014) to provide extra affective information (Asghar et al., 2018; Zhong et al., 2019).
Responses generated by above studies can simply continue the emotion of the query. To generate emotion-controllable responses, Zhou et al. (2018a) address the emotion factor in large-scale conversations, and propose ECM to generate responses based on different given emotions. After that, Colombo et al. (2019) augment ECM with
VAD embeddings and modified the loss function and decoding procedure. Song et al. (2019) use lexicon-based attention and a word-based classifier to improve the ability of emotion expression.
5.2 Dual Learning in NLP
He et al. (2016) propose Dual Learning (DL) for machine translation first which consider the source to target language translation and target to source language translation as a dual task. After that, Tang et al. (2017) implement a dual framework for the question answering system. Both Zhang et al. (2018) and Cui et al. (2019) use similar idea in dialogue generation task to produce coherent but not safe responses, since they find that a more diverse and specific response usually has a higher probability of being transformed back to the given query. Luo et al. (2019b) and Luo et al. (2019a) exploit DL in unsupervised text style transfer to relieve the need of parallel data.
The differences between our method and those in Section 5.1 and Section 5.2 are: (1) We consider the emotion expression and content consistency simultaneously via a DL method. (2) Instead of regarding the query as an emotionless sentence, we utilize the emotion of query, which can help model the emotion shifting and coherence to improve the quality of response. (3) To better model the changes in emotion and content between the query and response, we combine the DL method with curriculum learning, which is known to improve the effectiveness and generalization.
In this paper, we propose a new framework Curriculum Dual Learning (CDL) for generating emotional responses in a controlled manner. Since existing methods in this field only focus on the emotion expression of target label but fail to consider the emotion of queries, the safe response problem deteriorates and hurts the content consistency. CDL utilizes two kinds of rewards to enhance emotion and content simultaneously via dual learning. Besides, with the support of curriculum learning, it can be more efficient. Experimental results show that CDL can generate fluent, coherent, informative as well as emotional responses.
This work was supported by National Key R&D Program of China (NO. 2017YFE0192900). We
sincerely thank the anonymous reviewers for their helpful and valuable suggestions.
Nabiha Asghar, Pascal Poupart, Jesse Hoey, Xin Jiang, and Lili Mou. 2018. Affective neural response generation. In European Conference on Information Retrieval, pages 154-166. Springer.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48. ACM.
Hengyi Cai, Hongshen Chen, Cheng Zhang, Yong-hao Song, Xiaofang Zhao, Yangxi Li, Dongsheng Duan, and Dawei Yin. 2020. Learning from easy to complex: Adaptive multi-curricula learning for neural dialogue generation. arXiv preprint arXiv:2003.00639.
Pierre Colombo, Wojciech Witon, Ashutosh Modi, James Kennedy, and Mubbasir Kapadia. 2019. Affect-driven dialog generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3734-3743.
Shaobo Cui, Rongzhong Lian, Di Jiang, Yuanfeng Song, Siqi Bao, and Yong Jiang. 2019. Dal: Dual adversarial learning for dialogue generation. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 11-20.
Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pages 94-104.
Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613-619.
Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2017. Affect-lm: A neural language model for customizable affective text generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 634-642.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820-828.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In Proceedings
of the 34th International Conference on Machine Learning-Volume 70, pages 1587-1596. JMLR.org.
Pei Ke, Jian Guan, Minlie Huang, and Xiaoyan Zhu. 2018. Generating informative responses with controlled sentence function. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1499-1508.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751.
Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In the 3rd International Conference on Learning Representations.
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, pages 4601-4609.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994-1003.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016c. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2157-2169.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132.
Fuli Luo, Peng Li, Pengcheng Yang, Jie Zhou, Yutong Tan, Baobao Chang, Zhifang Sui, and Xu Sun. 2019a. Towards fine-grained text sentiment transfer. In Proceedings of the 57th Annual Meeting
of the Association for Computational Linguistics, pages 2013-2022.
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. 2019b. A dual reinforcement learning framework for unsupervised text style transfer. arXiv preprint arXiv:1905.10060.
Chuan Meng, Pengjie Ren, Zhumin Chen, Christof Monz, Jun Ma, and Maarten de Rijke. 2019. Refnet: A reference-aware network for background based conversation. arXiv preprint arXiv:1908.06449.
Saif Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174-184.
Zheng-Yu Niu, Hua Wu, Haifeng Wang, et al. 2019. Knowledge aware conversation generation with explainable reasoning over augmented graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1782-1792.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Timo Partala and Veikko Surakka. 2004. The effects of affective interventions in human-computer interaction. Interacting with computers, 16(2):295-309.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 646-653.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1162-1172.
Thomas S Polzin and Alexander Waibel. 2000. Emotion-sensitive human-computer interfaces. In ISCA tutorial and research workshop (ITRW) on speech and emotion.
Helmut Prendinger and Mitsuru Ishizuka. 2005. The empathic companion: A character-based interface that addresses users' affective states. Applied Artificial Intelligence, 19(3-4):267-285.
Helmut Prendinger, Junichiro Mori, and Mitsuru Ishizuka. 2005. Using human physiology to evaluate subtle expressivity of a virtual quizmaster in a mathematical game. International journal of human-computer studies, 62(2):231-245.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1577-1586.
Lei Shen, Yang Feng, and Haolan Zhan. 2019. Modeling semantic relationship in multi-turn conversations with hierarchical latent variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5497-5502.
Zhenqiao Song, Xiaqing Zheng, Lu Liu, Mu Xu, and Xuan-Jing Huang. 2019. Generating responses with a specific emotion in dialog. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3685-3695.
Xiao Sun, Xinmiao Chen, Zhengmeng Pei, and Fuji Ren. 2018. Emotional human machine conversation generation based on seqgan. In 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), pages 1-6. IEEE.
I Sutskever, O Vinyals, and QV Le. 2014. Sequence to sequence learning with neural networks. Advances in NIPS.
Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, and Ming Zhou. 2017. Question answering and question generation as dual tasks. arXiv preprint arXiv:1706.02027.
Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior research methods, 45(4):1191-1207.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Xinnuo Xu, Ondrej Dusek, Ioannis Konstas, and Verena Rieser. 2018. Better conversations by modeling, filtering, and optimizing for coherence and diversity. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3981-3991.
Wojciech Zaremba and Ilya Sutskever. 2015. Reinforcement learning neural tuning machines-revised. arXiv preprint arXiv:1505.00521.
Hainan Zhang, Yanyan Lan, Jiafeng Guo, Jun Xu, and Xueqi Cheng. 2018. Reinforcing coherence for sequence to sequence model in dialogue generation. In *IJCAI*, pages 4567–4573.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654-664.
Peixiang Zhong, Di Wang, and Chunyan Miao. 2019. An affect-rich neural conversational model with biased attention and weighted cross-entropy loss. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7492-7500.
Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018a. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Thirty-Second AAAI Conference on Artificial Intelligence.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018b. Commonsense knowledge aware conversation generation with graph attention. In *IJCAI*, pages 4623-4629.
Xianda Zhou and William Yang Wang. 2018. Mojitalk: Generating emotional responses at scale. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1128-1137.