SlowGuess/NAACL_2022_OCR huggingface dataset:
Azlaan Mustafa Samad *, Kshitij Mishra *, Mauajama Firdaus, Asif Ekbal
Patna, India
{kshitij_1921cs23,mauajama.pcs16,asif}@iitp.ac.in
azlaanmsamad@gmail.com
Persuasion is an intricate process involving empathetic connection between two individuals. Plain persuasive responses may make a conversation non-engaging. Even the most well-intended and reasoned persuasive conversations can fall through in the absence of empathetic connection between the speaker and listener. In this paper, we propose a novel task of incorporating empathy when generating persuasive responses. We develop an empathetic persuasive dialogue system by fine-tuning a Maximum Likelihood Estimation (MLE)-based language model in a Reinforcement Learning (RL) framework. To design feedbacks for our RL-agent, we define an effective and efficient reward function considering consistency, repetitiveness, emotion and persuasion rewards to ensure consistency, non-repetitiveness, empathy and persuasiveness in the generated responses. Due to lack of emotion annotated persuasive data, we first annotate the existing PERSUAION-FORGOOD dataset with emotions, then build transformer based classifiers to provide emotion based feedbacks to our RL agent. Experimental results confirm that our proposed model increases the rate of generating persuasive responses as compared to the available state-of-the-art dialogue models while making the dialogues empathetically more engaging and retaining the language quality in responses.
While conversing with persuasive dialogue agents, on top of fluent and meaningful response generation, a high quality conversation is often derived by understanding and acknowledging implied feelings towards the conversing partner. People are more likely to engage in the conversation when they are motivated with empathetic responses. These persuasive responses can be associated with differ
ent emotions in consonance with the way people perceive and think about the world. For instance, in Figure 1, while the strike-through response is persuasive, the green box response may be more engaging, as it connects with the end-user and acknowledges the underlying emotion of caring. In this work, we investigate different generic and task specific rewards to reinforce a dialogue agent to generate fluent, persuasive and empathetic responses.
Empathetic Persuasion
I just need more time to do my own research and think about it. I am not ready to donate right now.
Figure 1: Example of persuasion with underlying caring emotion.
You could make just a one-time donation: Only amount of 10 would change a life.
I really understand your worries. You could donate just one time. Only a little help may save the children as a whole. Would you reconsider for 10?
In recent studies on personalized conversational agents (Mazaré et al., 2018; Zheng et al., 2019; Wang et al., 2019; Zheng et al., 2020), it is suggested that adopting different human oriented chatbot identities or conversational strategies can significantly affect the responses of users and make the conversations more engaging. These dialogue agents greatly improved the user-targeted personalization. For instance, Shi and Yu (2018) include user sentiment to make an effective user-adaptive system. Li et al. (2019) takes both fine-grained token-level and coarse-grained sentence-level emotions to generate the responses. Mishra et al. (2022a) designed different rewards to reinforce politeness in a dialogue agent's responses. But, there is a subtle dependency between the different personalization techniques, such as empathy, sentiment and persuasion which can be used to generate better human-like responses. Therefore, we focus to incorporate emotions to generate more engaging and persuasive utterances.
* These authors are jointly first authors.
$^{\dagger}$ Work done during an internship at IIT Patna.
‡ Corresponding author: asif@iitp.ac.in.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 844 - 856
Due to the paucity of available data and dynamic nature of attitude and emotions of users in an ongoing dialogue, it is a hard task to model a personalized dialogue agent in a Supervised Learning (SL) framework which can generalize to different users in different situations. These MLE-loss based models tend to suffer from exposure bias. Therefore, lately researchers have focused on RL to fine-tune these models due to its ability to learn from user interactions and improve based on user's feedbacks in the form of rewards (Singh et al., 1999; Li et al., 2016; Casanueva et al., 2018; Chen et al., 2019; Mesgar et al., 2020). An RL based dialogue agent treats dialog planning as a sequential decision problem and focuses on long-term rewards to decide the next action which helps in enhancing the performance compared to the earlier systems (Su et al., 2017).
Recently, there had been an effort made by Shi et al. (2020a) to refine an MLE-loss based language model without user simulators to generate persuasive responses. They focused to penalize repetitive and inconsistent utterance generation when persuading a persuadee. Our current work focuses on incorporating emotions to engage the end users empathetically as well as to persuade them for donation. We first design a reward function consisting of generic rewards i.e. consistency and repetitiveness, and the task specific rewards i.e. emotion and persuasiveness to explicitly assess the quality of a generated response as per consistency, repetition, emotion and persuasion. We then train a policy via RL to maximize the score given by our reward function. The policy generates a response at each turn, and is updated using the Proximal Policy Optimisation (PPO) algorithm (Schulman et al., 2017) based on the reward assigned to the entire generated response by the defined reward function
As there is no relevant data that provides both emotion and persuasion, we extend the existing PERSUASIONFORGOOD (Wang et al., 2019) dataset to infuse empathy in the form of emotions. We assess the adequacy, fluency, empathy and persuasiveness of the generated responses from our RL-based model using both automatic as well human evaluation metrics. Our core contributions are four-fold:
1. To have persuasion with empathetic information, we manually annotate the PERSUASION-FORGOOD dataset with 23 different emotions.
2. We fine-tune transformers based pre-trained model to create robust and state-of-the-art models for emotion recognition and persuasive classification.
3. We propose an RL-based dialogue generation framework that makes use of two generic and two task specific rewards, to ensure fluency, non-repetitiveness, empathy and persuasiveness.
4. We use the automatic and human evaluation metrics to show that our RL-based system generates a response that is more consistent, fluent, empathetic and persuasive than the available state-of-the-art model (Shi et al., 2020a).
Historically, there had been attempts made to model persuasions. Petty and Cacioppo's Elaboration Likelihood Model (ELM) (Petty and Cacioppo, 1986) argues that a person's persuasion depends on the varying degrees of thoughts of processing information and persuasive context. Friestad and Wright's Persuasion Knowledge Model (PKM) suggests that there is a inter-relationship between scientific persuasion knowledge and everyday persuasion knowledge (Friestad and Wright, 1994). Further, Dijkstra (2008) suggests that incorporation of personal factors with the persuasive information can enhance individual's motivation towards persuasive messages.
Recently, due to the increasing need for social chatbots, modelling empathy and persuasion has attracted much attention in the community. Rashkin et al. (2018) have proposed a EMPATHETICDILOGUES dataset to generate empathetic dialogues grounded in emotional situations. To recognize user emotions and generate empathetic responses, Lin et al. (2020) developed an end-to-end dialogue system, CAiRE. Mishra et al. (2022b) predicted the politeness of utterances in goal-oriented conversations. While Hidey and McKeown (2018) modelled argument sequences in social media to predict the persuasiveness, the research reported in Yang et al. (2019) identified different persuasion strategies using a hierarchical neural network. Wang et al. (2019) proposed a multi-turn PERSUASIONFORGOOD dataset annotated with different persuasion strategies to model the persuasion classification. Using the same dataset, Shi et al. (2020b)
randomly assigned 790 participants to different conditions to conduct an online study that whether they can be persuaded by a chatbot for charity donation or not. Lukin et al. (2017) considered personality traits in single-turn persuasion dialogues and found that personality factors such as emotional arguments on social and political issues can affect belief change, with conscientious, and can convince more people.
Recently, Wu et al. (2019) trained a MLE loss based language model while Shi et al. (2020a) fine-tuned this MLE loss based model in an RL-framework to generate persuasive responses. But, these research works focused on generating persuasive responses alone, whereas the persuasion, in itself, covers a vast domain space with different end-user attitudes. Further, a persuasive utterance cannot ensure engagement of user in an ongoing dialogue unless the user is connected emotionally with the cause s/he is persuaded for. Therefore, our work focuses here on the stylistic and engaging dialogue generation by incorporating empathy with persuasion. To the best of our knowledge, there has been no prior research that incorporated emotions for persuasive dialogue generation.
3.1 Formal Definition
A multi-turn dialogue is defined as  $d = \{p_1^e, p_1^r, p_2^e, p_2^r, \dots, p_t^r, p_t^e\}$ , where  $p_t^r$  and  $p_t^e$  are the utterances of the persuader and persuadee at turn  $t$ . The two individuals take turns to respond where a turn comprises of multiple sentences. Each utterance of the persuader in the dialogue has two labels, one for emotion  $e = \{e_1^l, e_2^l, \dots, e_t^l\}$  and the other for persuasion strategy  $s = \{s_1^l, s_2^l, \dots, s_t^l\}$  expressed by it. Here,  $l$  represents the label associated with the persuader's utterance  $p_i^r$  at the  $i^{th}$  turn. The sets  $\mathbb{E} = \{e^{l_1}, e^{l_2}, \dots, e^{l_{n_1}}\}$  and  $\mathbb{S} = \{s^{l_1}, s^{l_2}, \dots, s^{l_{n_2}}\}$  contain the different labels for emotion and persuasion strategy, where  $n_1$  and  $n_2$  denote the number of emotion and persuasion strategy labels, respectively.
3.2 Proposed Methodology
We first initialize our proposed model  $p_{\theta}$  with a MLE loss pre-trained parameters  $q$  of ARDM model (Wu et al., 2019), then we fine-tune it by defining an efficient reward function in an RL framework. While fine-tuning, at each step RL-agent generates  $n$  candidate responses consider
ing the entire dialogue history. These generated responses are compared with the gold human response and are assigned rewards based on the quality of the generated candidates. The model is rewarded for generating responses encompassing emotion and persuasion strategy while penalised for inconsistent and repetitive responses.
Emotion and Persuasion Classification: In order to receive persuasion and emotion reward feedbacks for RL-agent, we fine-tune a pre-trained RoBERTa (Liu et al., 2019) model to build two classifiers, viz. emotion and persuasion strategy classifiers. First, we fed the sampled batches to the model to obtain contextual representations  $h_{< s >}$ . Then  $h_{< s >}$  is passed through a feed forward network which outputs a vector having scalar scores for all classes. Further, softmax function is applied to obtain the probability score of each class over all the classes. Lastly, highest probability score is chosen to represent the predicted class. The emotion (23 classes) and persuasion strategy classifiers (11 classes) achieve accuracy scores of  $58.13\%$  and  $73.2\%$ , respectively.
Reward: The reward function  $R$  is considered as a combination of multiple sub-rewards which serves to capture different aspects of an adequate response and assess the quality of the generated response candidates. The reward  $R$  consists of subrewards  $R_{1}$  for repetitiveness,  $R_{2}$  for consistency,  $R_{3}$  for empathy and  $R_{4}$  for persuasion. The final reward  $R$  is expressed as a weighted sum of these rewards as shown below:
$$
R = \alpha_ {1} R _ {1} + \alpha_ {2} R _ {2} + \alpha_ {3} R _ {3} + \alpha_ {4} R _ {4} \tag {1}
$$
Repetitiveness Reward: As pointed out by Shi et al. (2020a), the frequently occurring utterances in the dataset tend to be generated more by the models, and this repetition usually happens at the exact lexical level. Thus, we use Jaccard Score as a measure of similarity between the previous utterance  $p_{t-1}^r$  and the current generated response  $p_t^r$  based on unigrams. The sentences are first normalized using  $\mathrm{spaCy}^1$  and the generated score is then directly used as a sub-reward:
$$
R _ {1} = \frac {p _ {t - 1} ^ {r} \cap p _ {t} ^ {r}}{p _ {t - 1} ^ {r} \cup p _ {t} ^ {r}} \tag {2}
$$
Consistency Reward: In order to generate human-like responses, the Meteor score (Banerjee
1 https://spacy.io/
Figure 2: A skeleton of our overall system. Our architecture has two models: A Reference Model (RM) and an Active Model (AM). The RM is used for generating response candidate given a context (persuadee's utterance). It generates log probabilities for both the response candidates  $(p_{t,g}^{r},\log P_{t,g}^{old})$  and the Golden Human Response  $(p_t^r,\log P_t^{old})$ . Gold Human Response is the actual persuader's response present in the dataset. Rewards are then calculated for the generated candidates  $R_{t,g}$  while the reward for gold human response  $R_{c}^{r}$  is a constant. These are then stored in the buffer memory, and sampled during the training. After sampling, the batch is inputted to the AM which outputs the new log probabilities  $\log P_{t}^{new}$  for the PPO loss calculation and finally optimisation is performed only for the Active Model.
and Lavie, 2005) is computed between the generated response  $p_t^r$  (hypothesis) and the gold human response  $p_{g_t}^r$  (reference).
$$
R _ {2} = M E T \left(p _ {t} ^ {r}, p _ {g _ {t}} ^ {r}\right) \tag {3}
$$
We select Meteor score since it uses WordNet to match synonyms if exact match does not occur (Castillo and Estrella, 2012), and also because of its high correlation with human judgement in machine translation tasks (Banerjee and Lavie, 2005).
Emotion and Persuasion Reward: To design emotion and persuasion rewards, we use our emotion and persuasion strategy classifiers to predict the emotion and persuasion strategy of the generated candidates. These predicted labels are compared with ground truth emotion and persuasion strategy labels of the corresponding gold human response. The candidate with matching label is rewarded. For brevity, explanation is done in terms of emotion reward since both emotion and persuasion rewards are calculated in the exact same manner. In order to encourage emotion in the generated responses, the model is penalised for generating responses contradicting the gold human response
label and encouraged for matching it:
$$
R _ {3} = R _ {4} = \mathcal {P} _ {e _ {j}} (p _ {t, g} ^ {r}) - \beta \sum_ {i \in S \backslash \{e _ {j} \}} \mathcal {P} _ {i} (p _ {t, g} ^ {r}) (4)
$$
where  $\mathcal{P}_i(p_{t,g}^r)$  is the probability of the generated response  $p_{t,g}^r$  belonging to the class  $i$  where  $i\in S$  with  $S = \{e_1,e_2,\dots ,e_n\}$  being the set of all classes of size  $n$ . The term  $e_j$  in the above equation refers to the gold human response class at turn  $t$ .  $\beta$  is a scalar, which takes a value greater than or equal to 1. Increasing  $\beta$  would result in increased penalisation for contradiction.
Policy: Policy  $\mathcal{P}_{\theta}$  is defined as the probability of generating a sentence  $y$ . The probability of the text sequence of length  $L$  is the joint probability of all the tokens that make up the entire text sequence.
$$
\mathcal {P} _ {\theta} \left(y _ {1: L} | x\right) = \prod_ {l = 0} ^ {L} \mathcal {P} _ {\theta} \left(y _ {l} \mid y _ {<   l}, x\right) \tag {5}
$$
Proximal Policy Optimisation: Proximal Policy Optimisation (PPO) (Schulman et al., 2017) is a policy gradient optimisation method which deals with the issues of sensitiveness, instability etc. faced by some of the policy gradient methods.
It is chosen because of ease of implementation and good performance on previous text generation task (Wu et al., 2020). The policy gradient methods maximize the expected reward following a parameterized policy using gradient ascent:
$$
\nabla_ {\theta} J (\theta) = \mathbb {E} _ {y \sim \mathcal {P} _ {\theta}} [ \nabla_ {\theta} \log \mathcal {P} _ {\theta} (y) \hat {A} _ {y} ] \tag {6}
$$
PPO replaces the log term in the above equation with an importance sampling term and clipping is performed in order to restrict the model from moving too much away from the policy, thus preventing catastrophic forgetting. In our implementation, we use the clipped version of PPO:
$$
\begin{array}{l} L ^ {\mathrm {C L I P}} (\theta) = \hat {\mathbb {E}} [ \min (r _ {y} (\theta) \hat {A _ {y}}, \operatorname {c l i p} (r _ {y} (\theta), \\ \left. \left. 1 - \varepsilon , 1 + \varepsilon) \hat {A} _ {y}) \right] \quad (7) \right. \\ \end{array}
$$
Here,  $r_y(\theta)$  is the probability ratio of generating a response between new and old policies  $\mathcal{P}_{\theta}^{new} / \mathcal{P}_{\theta}^{old}$ .  $\varepsilon$  is a hyperparameter used to define the clipping range and  $\hat{A}_y$  is the estimated advantage which is obtained by normalizing rewards in our case. Our architecture uses two models, viz. A Reference Model and an Active model as shown in Figure 2. Both the models are initialized with the same pre-trained parameters  $q$ , but only one is fine-tuned using RL. The Reference Model is used for the sample collection step, where the generated candidates and the golden human responses are stored along with their respective rewards and probabilities  $\mathcal{P}_{\theta}^{old}$  in the buffer memory. During the training step, batch is sampled from the buffer memory and inputted to the Active Model to obtain the new probabilities  $\mathcal{P}_{\theta}^{new}$ . Finally, the loss is calculated as mentioned in equation 7 and the optimisation is performed.
$$
\theta_ {k + 1} = \underset {\theta} {\operatorname {a r g m a x}} \mathbb {E} _ {s, a \sim \mathcal {P} _ {\theta_ {k}}} [ L ^ {\mathrm {C L I P}} ] \tag {8}
$$
ARDM: Wu et al. (2019) uses a pre-trained large-scale language model to formulate both the persuader and persuadee utterances into a combined dialogue model:
$$
p (d) = \prod_ {t = 1} ^ {T} p _ {u} \left(u _ {t} \mid u _ {<   t}, s _ {<   t}\right) p _ {s} \left(s _ {t} \mid u _ {<   t}, s _ {<   t}\right) \tag {9}
$$
The terms  $p_u$  and  $p_s$  are the utterances of the user and the system at turn  $t$ . The model uses GPT-2 (Radford et al., 2019), one each for the system and
the user, and is trained to maximize the likelihood for the entire dialog model.
RFI: Shi et al. (2020a) proposed a model which does not require interaction with the environment and aims to learn the policy directly from the data, thereby, eliminating the use of user simulators. They used ARDM (Wu et al., 2019) as a pre-trained model and then fine-tuned it using RL based algorithm.
4 Datasets and Experiments
We experiment and analyze to what extent our RL-based fine-tuning improves the persuasive response generation through both automatic and human evaluation. First, the datasets used in our experiments are introduced (Section 4.1). Then the implementation details for the proposed RL-based system are provided in Section 4.2. Due to space restrictions, the implementation details of emotion and persuasion strategy classifier are given in the section A.1 of the Appendix. Finally, the details of automatic and human evaluation metrics are provided in Section 4.3.
We design our experiments on PERSUASIONFOR-GOOD (Wang et al., 2019) dataset consisting of 1,017 human to human conversations for donation to a charity organization named Save the Children.
In order to connect with the end-user empathetically and promote emotional responses, our RL-based system also needs emotion feedback of end-user to form its responses. Therefore, to annotate the PERSUASIONFORGOOD dataset with different emotion labels, we use EMPATHATICDIALOGUES dataset (Rashkin et al., 2018) consisting of  $25\mathrm{k}$  dialogues grounded with 32 different emotions.
First, to achieve a better class distribution in EMPATHATICDIALOGues dataset, we reduce the number of classes from 32 to 23 by merging those emotions which may work in similar way at the time of persuasion, such as terrified and afraid are merged into one emotion (details of all such combined emotions is given in section A.2 of the Appendix). Then, we fine-tune a pre-trained RoBERTa (Liu et al., 2019) based classifier on the reduced labelled EMPATHATICDIALOGUES dataset. It is observed that the classifier trained on 23 labels performed significantly better than that of the model trained on 32 labels<sup>2</sup>.
2The accuracy scores for emotion classifier with 32 and
This trained emotion classifier is used to predict the emotions of each utterance in PERSUASION-FORGOOD dataset. Out of these 1,017 dialogues, we choose 385 dialogues, and assign three annotators proficient in English communicative skills to perform manual cross-verification of the predicted emotions for these utterances. They are first asked to understand the underlying emotion in the EMPATHATICDIALOGUES dataset, and then cross-verify the emotion predictions of PERSUASION-FORGOOD dataset to annotate it with the right emotion in case of any errors. A reliable multi-rater Kappa (McHugh, 2012) agreement ratio of approximately  $72\%$  is observed in their annotation. Further, this annotated gold standard emotion persuasion dataset is used to train our persuasive emotion classifier which is, in turn, used to predict the emotions on-the-fly during RL-based training to form emotion reward.
We use PERSUASIONFORGOOD dialogue dataset to train two classifiers, viz. persuasion strategy classifier and persuasive binary classifier where the former is used to form persuasion reward and the latter predicts whether an utterance is persuasive or not during evaluation.
ARDM: We use OpenAI's two pre-trained GPT-2 medium models (Radford et al., 2019) with 345M parameters to model both the persuader and the persuadee. The model consists of 24-layers, 1024 hidden size with 16 heads. The tokenization of the words are carried out using Byte-Pair Encoding (Shibata et al., 1999). Depending on the persuader or the persuadee, their utterances are prefixed with "A:" or "B:" to generate responses under zero-shot condition and suffixed with "\n\nThe model is trained with a learning rate of 3e-5, using AdamW optimizer (Loshchilov and Hutter, 2017) with 100 warm-up steps and dropout rate of 0.1.\n\n
RL Fine-tuning: For fine-tuning using RL, we set the gold standard human reward to 10 and the number of generated candidate responses at each training step to be 2. This was done after experimenting initially with the values of 2, 4, 5 and 10. The values of  $\alpha_{1},\alpha_{2},\alpha_{3}$  and  $\alpha_{4}$  were chosen as 0.1, 0.1, 0.55 and 0.25. These values were selected after thorough experimentation of different combination of values for alphas as mentioned in Section
A.3 of the Appendix. The value of  $\beta$  is set to 2.0 for both the emotion and persuasion rewards. The generated candidate responses were decoded using the widely popular method of nucleus sampling (Holtzman et al., 2019) where  $p$  is 0.9 with a temperature  $T$  of 0.8. AdamW optimizer (Loshchilov and Hutter, 2017) is used for optimization with a learning rate of 2e-05. The value of  $\varepsilon$  is set to 0.2. The train and validation set split ratio is considered as 9:1 for the PERSUASIONFORGOOD dataset.