SlowGuess/NAACL_2022_OCR huggingface dataset:
which has decoders for each emotion and integrates outputs of the decoders according to predicted emotion distribution. (2) EmpDG (Li et al., 2020) uses emotional words and consists of an adversarial framework including a generator and discriminators which reflect the user feedback. (3) MIME (Majumder et al., 2020) is also a transformer-based generative model which mimics user emotion based on emotion grouping and uses stochastic sampling for varied responses. (4) MIME+Focused S1 and (5) Blender+Focused S1 (Kim et al., 2021) attach RSA Framework to MIME and Blender (Roller et al., 2021). Blender is a pretrained model with 90M parameters size, using an immense number of dialogues. It is finetuned on EmpatheticDialogues. Using distractors and Bayes' Rules, RSA Framework makes the models focus on certain parts of the post, such as emotion cause words when generating responses in the single-turn dialogues<sup>5</sup>. Implementation details about Emp-RFT and baselines are covered in Appendix A.1.
Automatic Evaluation. We evaluated the models, using the following three metrics: (1) Perplexity (PPL) (Vinyals and Le, 2015) measures how highly likely tokens are generated, which evaluates the overall quality of the model. (2) Distinct-n (Dist-n) (Li et al., 2016) measures how diverse the generated response is via the unique words within its n-gram. (3). We use BERTscore  $(\mathbf{F}_{\mathbf{BERT}})$  (Zhang et al., 2019) which measures token-level semantic similarities between the generated response and the gold response based on embeddings from BERT (Devlin et al., 2019).
Human Ratings. Human evaluations for the dialogues models are essential because of insufficient reliability on automatic metrics. We randomly sampled 100 test dialogues and asked 3 human workers to score models' generated responses on 1 to 5 point scale, following the four metrics (Rashkin et al., 2019): (1) Empathy measures whether the generated response understands the speaker's emotion and situation. (2) Relevance measures whether the generated response is coherent to the context. (3) Fluency measures whether the generated response is grammatically correct and readable. (4) Since we conclude that models generating generic responses are not empathizing to the speaker, we
5To make the models work in the multi-turn dialogues, the models are converted to take several utterances and to focus on emotion cause words of the last utterance.
Method	Automatic Evaluation	Human Evaluation
PPL	Dist-1	Dist-2	\(F_{BERT}\)	Empathy	Relevance	Fluency	Diversity
MoEL	38.04	0.44	2.10	0.11	3.25	3.73	3.49	2.85
EmpDG	37.29	0.46	2.02	0.14	3.30	3.76	3.57	3.11
MIME	37.09	0.47	1.91	0.13	3.23	3.78	3.53	2.83
MIME+Focused S1	36.43	0.52	2.21	0.15	3.34	3.84	3.65	3.15
Blender+Focused S1	13.21*	3.11*	4.38*	0.31*	3.69*	4.11*	4.05*	3.78*
Emp-RFT	13.59*	3.24*	4.59*	0.34*	3.78*	4.23*	4.11*	4.02*
w/o FTR	15.22	3.22*	4.49*	0.27	3.56	4.05	4.01	3.95*
w/o CP	13.89*	3.07	4.36	0.33*	3.74*	4.20*	4.05*	3.84*
w/o (CP+NEKD)	14.87	2.89	4.08	0.28	3.61	4.02	3.95	3.69
w/o (CP+NEKD+FCK)	15.45	2.75	3.86	0.23	3.51	3.89	3.83	3.50
MoEL	41.13	0.40	1.96	0.08	2.97	3.44	3.30	2.55
EmpDG	40.10	0.41	1.91	0.11	3.01	3.50	3.32	2.85
MIME	40.51	0.42	1.82	0.09	2.94	3.51	3.29	2.53
MIME+Focused S1	39.58	0.48	2.11	0.11	3.05	3.59	3.39	2.91
Blender+Focused S1	16.96	3.03*	4.19*	0.26*	3.43*	3.90*	3.88*	3.65*
Emp-RFT	14.71*	3.21*	4.48*	0.32*	3.66*	4.15*	4.01*	3.91*
w/o FTR	17.12	3.20*	4.40*	0.22	3.32	3.83	3.85	3.88*
w/o CP	15.12*	3.04*	4.28*	0.31*	3.62*	4.11*	3.96*	3.71*
w/o (CP+NEKD)	16.24	2.84	4.02	0.25*	3.50*	3.92*	3.84	3.61
w/o (CP+NEKD+FCK)	17.33	2.71	3.78	0.20	3.42	3.82	3.76	3.42
Table 2: Results of automatic evaluation and human ratings on all(top) and multi-turn(bottom) instances. * means superior results with  $p$  -value  $< {0.05}$  (sign test).
Emp-RFT vs.	Win (%)	Lose (%)	κ
MoEL	74.4/82.2	9.3/6.9	0.67/0.73
EmpDG	70.3/77.7	12.3/9.8	0.61/0.70
MIME	71.6/79.5	11.1/8.2	0.64/0.71
MIME+Focused S1	65.3/74.5	13.2/10.8	0.61/0.66
Blender+Focused S1	32.0/38.6	25.5/22.5	0.46/0.48
Table 3: Results of human A/B test. The results in front of and behind  $\prime$  are each on all instances and multi-turn instances. Fleiss' kappa  $\left( \kappa \right)$  denotes agreements among human workers,where  ${0.4} < \kappa  < {0.6}$  and  ${0.6} <$ $\kappa  < {0.8}$  indicate moderate and substantial agreements, respectively.
use Diversity to measure whether the generated response is non-generic.
Human A/B Test. We further conducted a human A/B test which provides stronger intuitions and higher agreements than human ratings, because this is carried with 3 human workers selecting the better response when given two generated responses (Sabour et al., 2021).
6.3 Analysis of Response Generation
We abbreviate feature transition recognizer, contrastive PPLM, next emotion and keywords detection, and fusing context with keywords as FTR, CP, NEKD, and FCK, respectively.
Automatic Evaluation Results. The overall automatic evaluation results are shown in the left part of Table 2. Emp-RFT performed exceedingly on all metrics except for PPL, which was
nearly the same as Blender+Focused S1. The improvements on other metrics indicated that our approach was effective for generating generally high quality and non-generic responses which were also semantically similar with the gold response. While the utilization of pretrained models yielded significant improvements compared to models only trained on EmpatheticDialogues, EmpRFT showed even greater performance when compared to Blender+Focused S1 endowed with more significant number of dialogues. In addition, due to utilization of FTR, Emp-RFT obtained remarkable results even on multi-turn instances, whereas, other models suffered due to their means of utilizing features for the context at the coarse-grained level.
Human Evaluation Results. In the right part of Table 2, Emp-RFT acquired the highest scores on all metrics, which demonstrated that all components of Emp-RFT helped generate responses that are empathetic, coherent to the context, and non-generic. Also, utilizing pretrained models showed significant improvements, especially on Fluency and Diversity scores. In Table 3, the generated responses from Emp-RFT were more preferred, which indicated Emp-RFT consistently outperformed other methods in various experiments. When observing at the models' performance difference between multi-turn instances and all instances, only Emp-RFT continued to perform consistently, whereas other models showed significant perfor
Method	Top-1 Acc	Top-5 Acc	macro-F1
CoMAE	41.12	80.09	39.61
Emp-RFT	42.08	80.78	40.39
Method	TL-P	TL-R	TL-F1
ConceptFlow	37.68	48.27	42.32
CG-nAR	44.62	61.94	51.87
Emp-RFT	45.35	63.15	52.78
mance drops under multi-turn instances. From this, we concluded that Emp-RFT continuously understood the dialogue flow.
Ablation Study. To better understand effects of each component in Emp-RFT, we conducted the ablation study. We gradually ablated each component within the response generation strategy in a hierarchical manner. (1)  $\mathbf{w} / \mathbf{o}$  FTR: Feature transition recognizer was disabled, which resulted in considerable drops on all metrics, especially on PPL,  $\mathrm{F_{BERT}}$ , Empathy, and Relevance scores on multi-turn instances, because Emp-RFT could not grasp the attention-needed features of utterance within multi-turn instances through FTR. (2)  $\mathbf{w} / \mathbf{o}$  CP: Contrastive PPLM was removed, which caused lower Dist-n and Diversity scores, because Emp-RFT could not actively use various  $\hat{k}^y$  when generating responses through CP. (3)  $\mathbf{w} / \mathbf{o}$  (CP+NEKD): Next emotion and keywords detection were disabled, which interfered with Emp-RFT's utilization of the next emotion and keyword. It dropped not only Dist-n and Diversity scores but also other metrics. (4)  $\mathbf{w} / \mathbf{o}$  (CP+NEKD+FCK): Fusing the context representation with keyword representations was disregarded. Since keywords were no longer emphasized for context modelling, such information could not get more attention when generating responses. It caused drops on all metrics, particularly on  $\mathrm{F_{BERT}}$ , Dist-n, and Diversity.
6.4 Analysis of Next Emotion and Keywords
We report the results in terms of NEKD in Table 4. Since all baselines have not conducted NEKD, we trained models showing promising results such as CoMAE (Zheng et al., 2021), ConceptFlow (Zhang et al., 2020) and CG-nAR (Zou et al., 2021) with EmpatheticDialogues. (More details are covered in Appendix A.2). Then, we compared EmpRFT to those models. Emp-RFT outperformed other models on all metrics, which proved Emp
RFT figured out which emotion and keywords were proper for generation.
Table 4: The results of next emotion(top) and keywords(bottom) detection. We used the metrics introduced in Table 1.
Emotion Label: Furious
Annotated Emotion: Annoyed → Apprehensive → Confident → Hopeful
u1: My roommate eats my food sometimes. This makes me so angry!
u2: You should get a mini fridge and put it in your room, with a lock on it.
u3: I think that's a great idea. I know where to get those fridges for cheap.
Gold: Yea man go for it, don't procrastinate.
MoEL: I am sorry you have to hear that. I hope it works out for you.
EmpDG: I agree with you. I think it isn't worth before you get it back.
MIME: I am sorry to hear that. I hope you don't have to deal with that.
MIME+Focused S1: I agree. I have a friend who is not to be a parent.
Blender+Focused S1: Roommates can be so annoying.
ˆy: Trusting,ˆy: procrastinate, safety, profit
Emp-RFT: Don't procrastinate. It makes your foods safe.
w/o FTR: I agree. I have a safety but spicy fridge.
Emotion Label: Proud
Annotated Emotion: Trusting → Proud → Grateful → Sentimental → Disappointed → Sentimental
u1: Hi, My girlfriend was recently let go of her job, but she put so much work in to finding a new job and got one with better pay!
u2: That's great! It was meant to happen then.
u3: Yes I agree, she was so happy when she found out the job was hers.
u4: I would have been too. I've never been fired. I always quit.
u5: She wasn't really fired, her position was no more. Schools don't have much room left for Home Ec teachers. Sad.
Gold: I remember taking Home Ec. I loved that class.
MoEL: I am sorry to hear that. I hope you can find a new job.
EmpDG: I am sorry you had that. I am not a fan of the past few years ago. I am not sure she had a close relationship.
MIME: I am sorry to hear that. I hope you find a new job.
MIME+Focused S1: I am sorry to hear that. I hope you find a new job.
Blender+Focused S1: She was paid for her hard work.
ˆy: Sentimental,ˆy: Ec, class, jobs, solution, remember
Emp-RFT: I remember the Ec class. It's a relief for her to find the job.
w/o FTR: I'm glad she found the solution to the job problem.
Emotion Label: Caring
Annotated Emotion: Annoyed → Caring → Sad → Caring
u1: My poor neighbor is suffering a lot without her husband.
u2: I suffer a lot too when my wife is gone. What happened to him?
u3: He passed away from cancer.
Gold: Ah, the evil cancer. Took my grandmother as well. I am sure he's off in a better place now.
ˆy: Sad,ˆy: condolences, cancer, grandmother, evil, lost
Emp-RFT: My condolences. I lost my grandmother because of the cancer.
w/o CP: I'm sorry to hear that. It's so hard to lose someone.
w/o (CP+NEKD): Oh no. I'm scary to get the cancer.
w/o (CP+NEKD+FCK): Oh no. I'm sorry to hear that.
Table 5: Model generations. The words marked in red and blue are keywords of the speaker utterance and listener utterance, respectively.
The cases from the models are shown in Table 5. In the first case, MoEL and MIME expressed regret, which was emotionally inappropriate to the context. All baselines except for MoEL failed to grasp the proper features within the context, and therefore generated incoherent responses. Especially, Blender+Focused S1 ignored the features of  $u^3$ . Since Emp-RFT understood the dialogue flow, it became attentive to not only the features of  $u^3$  but also those of  $u^1$ ,  $u^2$ , mentioning ('procrastinate', 'foods', 'safety'), which led to empathy and coherence. In the second case, all baselines couldn't understand the longer context, which resulted in im
proper empathy. Also, Blender+Focused S1 disregarded the features of  $u^5$ , and therefore overlooked the speaker's sadness. Emp-RFT fully comprehended why the speaker's happiness changed to the sadness. In both cases, without FTR, the responses of Emp-RFT were non-empathetic and incoherent because of dismissing appropriate features. In the third case, we report the case in terms of the response generation strategy. Without CP, NEKD, and FCK, Emp-RFT produced a generic response. With the utilization of FCK, Emp-RFT perceived the word 'cancer' in  $u^3$  but expressed excessive emotion by mentioning 'scary'. When Emp-RFT additionally conducted NEKD, Emp-RFT generated emotionally appropriate responses by mentioning 'sorry' and 'hard', and utilized the keyword 'lost'. Lastly, with CP, Emp-RFT generated a diverse response, actively using  $\hat{k}^y$ .
We proposed a novel approach that recognizes feature transitions between utterances, which led to understanding the dialogue flow and grasping the features of utterance that needs attention. Also, to make our model focus on emotion and keywords related to appropriate features, we introduced a response generation strategy including fusing context with keywords, next emotion and keywords detection, and contrastive PPLM. Experimental results showed that our model outperformed baselines, and especially, achieved significant improvements on multi-turn instances, which proved our approach was effective for empathetic, coherent, and nongeneric response generation.
We expect that our proposed approach does not suffer from ethical problems. The dataset we use in our work is EmpatheticDialogues which is English-based. The dataset is constructed by crowdsourcing with Amazon Mechanical Turk, which protects private user information (Rashkin et al., 2019). In addition, the dialogue dataset is anticipated not to have responses which include discrimination, abuse, bias, etc, because the robust collection procedure of EmpatheticDialogues ensures the quality of the dataset. Thus, we expect that models trained using the dataset, do not generate inappropriate responses which harm the users. However, we inform that our model utilizes a pretrained language model, which may produce inappropriate
responses. Lastly, we anticipate our model make potential users be interested and consoled by generating empathetic responses.
We thank all anonymous reviewers for their meaningful comments, and Hyeongjun Yang, Chanhee Lee and Sunwoo Kang of Yonsei University for their discussion and feedback about our work. This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP; Ministry of Science, ICT & Future Planning) (No. NRF2022R1A2B5B01001835). Also, this work was partly supported by the Institute of Information and Communications Technology Planning and Evaluation(IITP) grant funded by the Korean government(MSIT) (No. 2020-0-01361-003, Artificial Intelligence Graduate School Program (Yonsei University)). Kyong-Ho Lee is the corresponding author.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607.
Kenneth Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22-29.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. Eraser: A benchmark to evaluate rationalized nlp models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443-4458.
Nancy Eisenberg and Janet Strayer. 1987. Critical issues in the study of empathy.
Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613-619.
Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998-998.
Jun Gao, Yuhan Liu, Haolin Deng, Wei Wang, Yu Cao, Jiachen Du, and Ruifeng Xu. 2021. Improving empathetic response generation by recognizing emotion cause in conversations. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 807-819.
Xiaodong Gu, Kang Min Yoo, and Jung-Woo Ha. 2021. Dialogbert: Discourse-aware response generation via learning to recover and rank utterances. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12911-12919.
Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2021. Perspective-taking and pragmatics for generating empathetic responses focused on emotion causes. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2227-2240.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Qintong Li, Hongshen Chen, Zhaochun Ren, Pengjie Ren, Zhaopeng Tu, and Zhumin Chen. 2020. Empdg: Multi-resolution interactive empathetic dialogue generation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4454-4466.
Qintong Li, Piji Li, Zhaochun Ren, Pengjie Ren, and Zhumin Chen. 2022. Knowledge bridging for empathetic dialogue generation.
Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, and Pascale Fung. 2019. Moel: Mixture of empathetic listeners. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 121-132.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Bodhisattwa Prasad Majumder, Sudha Rao, Michel Galley, and Julian McAuley. 2021. Ask what's missing and what's useful: Improving clarification question generation using global knowledge. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4300-4312.
Navonil Majumder, Pengfei Hong, Shanshan Peng, Jiankun Lu, Deepanway Ghosal, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. Mime: Mimicking emotions for empathetic response generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8968-8979.
Lisong Qiu, Yingwai Shiu, Pingping Lin, Ruihua Song, Yue Liu, Dongyan Zhao, and Rui Yan. 2020. What if bots feel moods? In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1161-1170.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370-5381.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, et al. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 300-325.
Sahand Sabour, Chujie Zheng, and Minlie Huang. 2021. Cem: Commonsense-aware empathetic response generation. arXiv preprint arXiv:2109.05739.
Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff. 2020. A computational approach to understanding empathy expressed in text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263-5276.
Lei Shen, Jinchao Zhang, Jiao Ou, Xiaofang Zhao, and Jie Zhou. 2021. Constructing emotional consensus
and utilizing unpaired data for empathetic dialogue generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3124-3134.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations.
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869.
Shuohang Wang and Jing Jiang. 2017. A compare-aggregate model for matching text sequences.(2017). In ICLR 2017: International Conference on Learning Representations, Toulon, France, April 24-26: Proceedings, pages 1-15.
Haolan Zhan, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Yongjun Bao, and Yanyan Lan. 2021. Aug-menting knowledge-grounded conversations with sequential knowledge transition. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5621-5630.
Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2020. Grounded conversation generation as guided traverses in commonsense knowledge graphs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2031-2043.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Chujie Zheng, Yong Liu, Wei Chen, Yongcai Leng, and Minlie Huang. 2021. Comae: A multi-factor hierarchical framework for empathetic response generation. In *Findings of the Association for Computational Linguistics: ACL* 2021.
Yicheng Zou, Zhihua Liu, Xingwu Hu, and Qi Zhang. 2021. Thinking clearly, talking fast: Concept-guided non-autoregressive generation for open-domain dialogue systems. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2215-2226.
A Implementation Details
A.1 Empathetic Response Generation Models
We use the official codes of all baselines, and follow the implementations (MoEL 6, EmpDG 7, MIME 8, MIME+Focused S1 and Blender Focused S1 9).
$^{6}$ https://github.com/HLTCHKUST/MoEL  
 $^{7}$ https://github.com/qtli/EmpDG  
 $^{8}$ https://github.comDECLARE-lab/MIME  
 $^{9}$ https://github.com/skywalker023/focused-empathy