SlowGuess/NAACL_2022_OCR huggingface dataset:
We use both automatic as well as human evaluation metrics. It is required from a dialogue system that it should be able to generate task-specific and quality responses. Therefore, we evaluate our proposed system with respect to two types of evaluation metrics: task-specific and quality-specific. The task-specific rewards include persuasiveness strategy (PerStr) - percentage of utterances generated with persuasive strategy and emotion probability (EmoPr) - percentage of empathetic utterances generated. The quality-specific reward includes perplexity (PPL) - to evaluate the generated response quality and utterance length (LEN) - to evaluate the average number of tokens generated in an utterance.
We perform human evaluation by recruiting 20 unique users and asked them to interact with the model. Each user acted as a persuadee and our model as a persuader. Once the user has conversed with the model, s/he is asked to evaluate the model's generated responses with respect to task-specific and quality-specific metrics. The task-specific metrics include persuasiveness (Per), empathy (Emp) - checking persuasiveness and empathy factor in the dialogue based on the one-to-five positive integer scale (corresponding definitions of all values are given in Section A.4 of the Appendix) and donation probability (DonPr) - calculating percentage of time people donated. The quality-specific metrics include (Cons), (Fluen) and (N-Rep) to check the consistency (with the dialogue context), linguistic fluency and non-repetitiveness of the generated utterance in the dialogue.
We analyze the results of our proposed RL-based emotion and persuasive model (RL-Emo-Per) in
31-5 denotes the intensity scale from the lowest to highest degrees.
23 class labels were found to be  $58.17\%$  and  $67.44\%$ , respectively.
Model	PerStr	EmoPr	PPL	LEN
ARDM (Wu et al., 2019)	49.2%	-	12.45	15.03
RFI (Shi et al., 2020a)	51.2%	-	12.38	19.36
RL-Emo-Per	55.42%	58.1%	11.25	16.75
Table 1: Results of automatic evaluation.
Model	Per	Emp	DonPr	Const	Fluen	N-Rep
ARDM	2.33	-	0.50	3.95	4.17	3.17
RFI	2.98	-	0.61	4.17	4.41	3.50
RL-Emo-Per	3.91	3.51	0.68	4.59	4.62	3.89
Table 2: Results of human evaluation.
comparison to two baselines, ARDM (trained on MLE loss) (Wu et al., 2019) and RFI (fine-tuned an MLE loss language model in an RL setting) (Shi et al., 2020a). Automatic and human evaluation results are shown in Table 1 and Table 2, respectively.
Automatic evaluation: It can be seen in Table 1 that our proposed RL-based emotion and persuasive model (RL-Emo-Per) outperforms both the baselines, ARDM and RFI. RL-Emo-Per performs better in terms of PerStr with a significant difference of  $6.22\%$  and  $4.22\%$  from ARDM and RFI, respectively. Improvements in PerStr show that the responses generated by RL-Emo-Per are more persuasive when incorporated with empathy factor in the dialogue than the ARDM or RFI. It can also be observed that RL-Emo-Per obtains lower perplexity (PPL) than both ARDM and RFI with the difference of 1.2 and 1.13, respectively, showcasing that RL-Emo-Per models better probability distribution in generating the utterances. Further, as compared to ARDM, RL-Emo-Per generates longer sentences as is depicted by the LEN metric, but shorter than the RFI model. One of the reasons for this behaviour could be the way our reward function has been designed i.e. persuasion and emotion rewards force the agent to generate long meaningful persuasive and empathetic utterances whereas repetitive reward penalize the repetitive tokens in the sentences forcing the agent to generate shorter sentences. Lastly, results of EmoPr metric shows that RL-Emo-Per encourages the model to generate empathetic utterances. It can be due to the fact that emotion reward feedbacks force RL-Emo-Per towards generation of more empathetic utterances.
Human evaluation: As per the human evaluation results reported in Table 2, it is observed that our
proposed RL-Emo-Per model performs better than the baselines in terms of all the metrics. It can be inferred from the table that incorporation of consistency and repetitiveness rewards have played a critical role in achieving better consistency (Const), fluency (Fluen) and non-repetitiveness (N-Rep) scores of 4.59, 4.62 and 3.89, respectively, than the baselines ARDM (3.95, 4.17 and 3.17) and RFI (4.17, 4.41 and 3.50). Further, in terms of task-specific metrics i.e. persuasiveness (Per) and empathy (Emp), RL-Emo-Per yields the highest score of 3.91 and 3.51, respectively, implying that adding empathetic factor may engage users more in the dialogue. Lastly, it is seen that  $68\%$  times users agreed to donate which depicts that our model is able to effectively make connection with users to persuade for donation.
We present an example of user's interaction with our proposed RL-Emo-Per and ARDM models in Table 3 (some more examples are given in Section A.6 of Appendix), with depiction of human evaluation ratings. It can be observed, that our proposed RL-Emo-Per model first tries to build rapport to engage with the user. In the later utterances, it employs different persuasion strategies particularly: Personal-Related Inquiry, Source related Inquiry, Emotional Appeal, Credibility Appeal, Donation Information and Self-Modelling. Further, it also empathetically connects with the user by generating different emotion-grounded utterances, such as impressed, sentimental, caring and trusting. Lastly, it can also be seen in Table 3 that as compared to RL-Emo-Per the responses from the ARDM model are very plain and mostly non-empathetic.
Our proposed approach has certain limitations.
Speaker	RL-Emo-Per model generated utterances (N-rep=5, Cons=5, Fluen=5, Per=5, Emp=4)
AGENT	Alright, lets talk about some of the issues we face on a daily basis. How are you?
USER	alright, I am doing good.
AGENT	Do you think there are people who have been out of touch with the world for too long?
USER	Yes I think so, we are so involved in ourselves.
AGENT	You are right, I know. I feel like it has become so important to me to help others and to be a part of the solution. What do you think about that?
USER	I think it is very important to help people in need whenever one can.
AGENT	Do you like children?
USER	I do like children. My sister has three children and I love to play with them.
AGENT	Have you ever heard of the organization Save the Children?
USER	I have not heard about them.
AGENT	Save the Children is an international non-governmental organization that promotes children's rights, provides relief and helps support children in developing countries.
USER	Oh they are doing god's work. Which countries do they operate in?
AGENT	oh yes! They are doing all over the world. The amount of children dying every day due to lack of access to health care is staggering. Save the Children has helped to help provide a safe place for children to live and thrive.
USER	Do they operate here in India as well?
AGENT	Yes, they are active in the US, too.
USER	oh very well.
AGENT	So what are you going to do today?
USER	I am not sure if I will donate. I need more information about the charity.
AGENT	The research team will collect all donations and send it to Save the Children. Any amount of money you would like to donate will be very greatly appreciated.
USER	How much can I donate?
AGENT	I'm thinking $1.00.
USER	wonderful, that you are donating. I think I will also donate $1 dollar.
AGENT	Thank you so much, so much appreciated!
USER	You are welcome
AGENT	Your welcome, I hope you have a wonderful day!
USER	Thank you for the information. You too enjoy your day.
Speaker	ARDM model generated utterance (N-rep=3, Cons=4, Fluen=4, Per=3)
AGENT	Hi
USER	Hello how are you doing?
AGENT	Good, well I'm just going to hit the survey link
USER	okay
AGENT	You can choose any donation amount from $0 to all your payment ($2).
USER	What is this donation for?
AGENT	To help the children.
USER	that's a very good cause
AGENT	It's a very noble cause
USER	can you tell me more about the charity?
AGENT	Its called Save the Children and it help kids in war zones.
USER	How much can I donate?
AGENT	I am willing to donate $1
USER	oh nice. I guess I will donate as well.
AGENT	I'm glad you are willing to donate.
USER	Thank you
AGENT	How much do you like to donate?
USER	i will donate $0.5 dollars.
AGENT	ok cool
USER	thank you.
AGENT	you are very kind
USER	thanks
AGENT	you are very kind
Table 3: An example of user interaction with our proposed RL-Emo-Per and ARDM.
Sometimes our model generates out of the context entities, such as in reply to 'Do they operate here in India as well?', the model responds with 'Yes, they are active in the US, too'. It can be due to the fact that defined reward function can not possibly cover the crucial aspects of an ideal conversation due to the lack of world knowledge present in the model.
Development of persuasive dialogue agents to generate empathetic responses is still in its nascent stage due to the lack of modelling the changing attitudes of individuals. Further, generative models only with MLE loss may lead to exposure bias and tend to generate generic responses. Therefore, to
connect with end-users empathetically and generate goal oriented-responses, we propose here an RL-based dialogue generation framework adopting PPO method to fine-tune the model. To force the agent to generate more empathetic and persuasive responses, we define an efficient and effective reward function considering two generic rewards, viz. consistency and repetitiveness and two task-specific rewards i.e. emotion reward - which forces the agent towards empathetic responses and persuasive reward - which forces the agent to generate persuasive responses. Automatic and human evaluation results demonstrate that by just adding extra reward of emotion, our model is able to achieve state-of-the-art result in a complex task like persuasion, and generate consistent, non-repetitive,
empathetic and persuasive responses 4.
In future, we would like to model persuasion in healthcare domain considering factors, such as effectiveness (providing evidence-based persuasions to the needed) and safety (avoiding harm to people for whom the persuasion is intended).
To model persuasion and empathy we used publicly available datasets. We adhered to the policies of used datasets without harming any copyright issues. Dataset used for empathetic persuasion is publicly available persuasion dataset annotated with emotions without manipulating or changing the content of any utterance in dialogues. We will make empathetic persuasive data available only with an official agreement that data will be used only for research works. The dataset is annotated with human experts by our in-house regular employees in the research group and they are paid at par with the university norms. We have also got our data annotation process verified by our university review board. It is also to be noted that a similar annotation scheme could be used for coercion, manipulation, or other amoral activities. Further, it may persuade people to draw inconsistent conclusions with those that they would have reached by exercising their full judgement (Garsten, 2009). Therefore, to develop a persuasive conversational AI an ethical intention must be taken into account. In this work, we choose a simple task of persuading for donation to Save the Children connecting with the end-users empathetically. We tried here to build a 'well speaking dialogue agent for social good' so that the society may benefit at large by reaching to a large number of users for persuasion in a very less time. Lastly, generative models may lead to uninformative utterances due to absence of world knowledge, hence, it is required to model knowledge grounding or fact-verification. This study we will employ in our future work.
Kshitij Mishra acknowledges the support of the "Prime Ministers Research Fellowship (PMRF) Program of the Government of India. Asif Ekbal acknowledges the Young Faculty Research Fellowship (YFRF), supported by Visvesvaraya PhD
scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia).
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Inigo Casanueva, Paweł Budzianowski, Pei-Hao Su, Stefan Ultes, Lina Rojas-Barahona, Bo-Hsiang Tseng, and Milica Gašić. 2018. Feudal reinforcement learning for dialogue management in large domains. arXiv preprint arXiv:1803.03232.
Julio Castillo and Paula Estrella. 2012. Semantic textual similarity for mt evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 52-58.
Lu Chen, Zhi Chen, Bowen Tan, Sishan Long, Milica Gašić, and Kai Yu. 2019. Agentgraph: Toward universal dialogue management with structured deep reinforcement learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(9):1378-1391.
Arie Dijkstra. 2008. The psychology of tailoring ingredients in computer-tailored persuasion. Social and personality psychology compass, 2(2):765-784.
Marian Friestad and Peter Wright. 1994. The persuasion knowledge model: How people cope with persuasion attempts. Journal of consumer research, 21(1):1-31.
Bryan Garsten. 2009. Saving persuasion. Harvard University Press.
Christopher Hidey and Kathleen McKeown. 2018. Persuasive influence detection: The role of argument sequencing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541.
Qintong Li, Hongshen Chen, Zhaochun Ren, Zhumin Chen, Zhaopeng Tu, and Jun Ma. 2019. Empgan: Multi-resolution interactive empathetic dialogue generation. arXiv e-prints, pages arXiv-1911.
4The codes and data used in this research work can be obtained from given link: https://www.iitp.ac.in/~ai-nlp-ml/resources.html#EMP4G
Zhaojiang Lin, Peng Xu, Genta Indra Winata, Farhad Bin Siddique, Zihan Liu, Jamin Shin, and Pascale Fung. 2020. Caire: An end-to-end empathetic chatbot. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13622-13623.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
Stephanie M Lukin, Pranav Anand, Marilyn Walker, and Steve Whittaker. 2017. Argument strength is in the eye of the beholder: Audience effects in persuasion. arXiv preprint arXiv:1708.09085.
Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. arXiv preprint arXiv:1809.01984.
Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica: Biochemia medica, 22(3):276-282.
Mohsen Mesgar, Edwin Simpson, and Iryna Gurevych. 2020. Improving factual consistency between a response and persona facts. arXiv preprint arXiv:2005.00036.
Kshitij Mishra, Mauajama Firdaus, and Asif Ekbal. 2022a. Please be polite: Towards building a politeness adaptive dialogue system for goal-oriented conversations. Neurocomputing.
Kshitij Mishra, Mauajama Firdaus, and Asif Ekbal. 2022b. Predicting politeness variations in goal-oriented conversations. IEEE Transactions on Computational Social Systems.
Richard E Petty and John T Cacioppo. 1986. The elaboration likelihood model of persuasion. In Communication and persuasion, pages 1-24. Springer.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Weiyan Shi, Yu Li, Saurav Sahay, and Zhou Yu. 2020a. Refine and imitate: Reducing repetition and inconsistency in persuasion dialogues via reinforcement learning and human demonstration. arXiv preprint arXiv:2012.15375.
Weiyan Shi, Xuewei Wang, Yoo Jung Oh, Jingwen Zhang, Saurav Sahay, and Zhou Yu. 2020b. Effects of persuasive dialogues: testing bot identities and inquiry strategies. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13.
Weiyan Shi and Zhou Yu. 2018. Sentiment adaptive end-to-end dialog systems. arXiv preprint arXiv:1804.10731.
Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. 1999. Byte pair encoding: A text compression scheme that accelerates pattern matching.
Satinder Singh, Michael Kearns, Diane Litman, and Marilyn Walker. 1999. Reinforcement learning for spoken dialogue systems. Advances in neural information processing systems, 12:956-962.
Pei-Hao Su, Pawel Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. arXiv preprint arXiv:1707.00130.
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. arXiv preprint arXiv:1906.06725.
Qingyang Wu, Lei Li, and Zhou Yu. 2020. Textgail: Generative adversarial imitation learning for text generation. arXiv preprint arXiv:2004.13796.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2019. Alternating roles dialog model with large-scale pre-trained language models. arXiv preprint arXiv:1910.03756.
Diyi Yang, Jiaao Chen, Zichao Yang, Dan Jurafsky, and Eduard Hovy. 2019. Let's make your request more persuasive: Modeling persuasive strategies via semi-supervised neural nets on crowdfunding platforms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3620-3630.
Yinhe Zheng, Guanyi Chen, Minlie Huang, Song Liu, and Xuan Zhu. 2019. Personalized dialogue generation with diversified traits. arXiv preprint arXiv:1901.09672.
Yinhe Zheng, Rongsheng Zhang, Minlie Huang, and Xiaoxi Mao. 2020. A pre-training based personalized dialogue generation model with persona-sparse data.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9693-9700.
A APPENDICES
A.1 Classifier Implementation Details
Both the Emotion and Persuasion Classifier are trained using Roberta (Liu et al., 2019). It is a transformer based model with 24-layer, 1024-hidden units, 16-heads with a total of  $355\mathrm{M}$  parameters. The learning rate and the batch size are set to 2e-5 and 32, respectively, for both the classifiers. They are trained using AdamW optimizer (Loshchilov and Hutter, 2017) with a dropout rate of 0.1.
A.2 Merged Emotion Details
Some emotion labels behave in similar manner at the time of persuasion. Therefore, we combine nine such emotion classes to their corresponding overlapping emotions. Details of these merged emotions are shown below:
angry  $^+$  furious  $=$  angry
$sad + \text{devasted} = sad$
afraid + terrified = afraid
guilty + ashamed = guilty
apprehensive + anticipating = apprehensive
sentimental + nostalgic = sentimental
surprised + excited = surprised
annoyed + disgusted = annoyed
trusting  $^+$  Faithful  $=$  trusting
Distribution of emotion classes in our emotion annotated persuasive dataset is shown in Figure 3.
In order to find the right combination of weights for our reward function, we have performed experiments with different values of alphas on the validation set (10% of whole data). Finally, the combination yielding best perplexity is selected to train our proposed RL-based model. Perplexity obtained with these different weight combinations are shown in Table 4. It is also to be noted that weighted task-specific rewards such as persuasion and emotion based rewards yield better perplexity as compared to the weighted generic rewards such as consistency and repetitiveness. This validates the use of task-specific rewards. Further, it is also observed that when the generic rewards are not assigned any weight, it increases the perplexity. This also validates the use of generic rewards in our designed reward function. Hence, we can infer that all the sub-rewards contribute to generate better persuasive responses.
WEIGHT OPTIMIZATION
α1	α2	α3	α4	Perplexity
0.1	0.1	0.8	0.0	11.31
0.15	0.15	0.7	0.0	11.30
0.2	0.2	0.6	0.0	11.29
0.25	0.25	0.5	0.0	11.29
0.1	0.1	0.1	0.7	11.26
0.1	0.1	0.4	0.4	11.27
0.1	0.1	0.5	0.3	11.27
0.1	0.1	0.55	0.25	11.25
0.1	0.1	0.4	0.4	11.27
0.5	0.5	0.0	0.0	11.31
0.6	0.4	0.0	0.0	11.32
Table 4: Weight optimisation using different values of alpha.
A.4 One-Five Scale Explanation
All human evaluation are performed on one-five scale. Here, for each of the metrics i.e. Per, Emp, Cons, Fluent and N-Rep the score of 1, 2, 3, 4 and 5 denote low (such as not-persuasive, not-empathetic, inconsistent, not-fluent and repetitive), somewhat-moderate, moderate, somewhat-high and high (such as persuasive, empathetic, consistent, fluent and non-repetitive) scores, respectively.
A.5 Annotation Details
In order to cross verify the emotion label predicted by the emotion classifier trained on EMPATHATICDIALOGUES dataset, we hired three annotators. These annotators were proficient in English communicative skills with postdoctoral qualification. We employ human experts who are our in-house regular employees in the research group and they are paid at par with the university norms. The annotators were first asked to understand the underlying emotions in the EMPATHATICDIALOGUES dataset and then correct the predictions on PERSUASION-FORGOOD dataset if required. We also made sure that the names of people mentioned in the datasets were not uniquely identifiable.
A.6 Examples of User Interaction in the Proposed Model
In Table 5 and 6, we show the interaction of two different users with our RL model. It can be seen that  $\mathbf{Per} = 3$  in Table 5 depicting that our model may not be able to persuade user efficiently all the times.
Figure 3: Distribution of emotion classes for the annotated PERSUASIONFORGOOD dataset.
Speaker	RL-Emo-Per model generated utterances (N-rep=3, Cons=4, Fluen=4, Per=3, Emp=2)
AGENT	hi
USER	How is it going?
AGENT	fine, thank you
USER	What are I suppose to do?
AGENT	I'm going to ask you a question. How about donating some of your earnings to a charity called save the children.
USER	I have never donated to a charity.
AGENT	Have you heard of it?
USER	Not really. what do they do?
AGENT	They are an international non-governmental organization that promotes children's rights, provides relief and helps support children in developing countries.
USER	Where are they located?
AGENT	They are an international non-governmental organization that promotes children's rights, provides relief and helps support children in developing countries.
USER	i am not sure if I can trust them.
AGENT	I'm not sure about this either. Do you donate much to charities?
USER	I have donated once long back to Red Cross organisation.
AGENT	Ok, I'm going to give some of my earnings to them.
USER	Oh thats very nice of you. How much do you plan to donate?
AGENT	I will give half of my earnings.
USER	oh wow, how many dollars?
AGENT	I plan to donate two dollars.
USER	Is the amount enough?
AGENT	Yes it is enough.
USER	very well, I will also donate 10 cents then since I need the money for myself.
AGENT	Very well, thank you so much