PAL to Lend a Helping Hand: Towards Building an Emotion Adaptive Polite and Empathetic Counseling Conversational Agent
Year: 2023

Abstract:
The World Health Organization (WHO) has significantly emphasized the need for mental health care. The social stigma associated with mental illness prevents individuals from addressing their issues and getting assistance. In such a scenario, the relevance of online counseling has increased dramatically. The feelings and attitudes that a client and a counselor express towards each other result in a higher or lower counseling experience. A counselor should be friendly and gain clients’ trust to make them share their problems comfortably. Thus, it is essential for the counselor to adequately comprehend the client’s emotions and ensure client’s welfare, i.e. s/he should adapt and deal with the clients politely and empathetically to provide a pleasant, cordial and personalized experience. Motivated by this, in this work, we attempt to build a novel Polite and empAthetic counseLing conversational agent PAL. To have client’s emotion-based polite and empathetic responses, two counseling datasets laying down the counseling support to substance addicts and crime victims are annotated. These annotated datasets are used to build PAL in a reinforcement learning framework. A novel reward function is formulated to ensure correct politeness and empathy preferences as per client’s emotions with naturalness and non-repetitiveness in responses. Thorough automatic and human evaluation showcases the usefulness and strength of the designed novel reward function. Our proposed system is scalable and can be easily modified with different modules of preference models as per need1.

heading: 1 Introduction
text: The global prevalence of mental health disorders is estimated to affect 20% of the world population (Holmes et al., 2018). The COVID-19 pandemic
∗Jointly first authors. 1Data and codes can be accessed at https://github.com/Mishrakshitij/PAL.git and AI-NLPML_PAL.
has further jeopardized the mental well-being of individuals (Paredes et al., 2021). Further, the limited availability of experts (Organization et al., 2021) and the overwhelming healthcare burden (Huffman et al., 2021) have created a substantial demand for counseling services. Counselors are faced with a diverse audience, including individuals struggling with substance addiction and crime victims, who often experience mental health challenges such as anxiety, depression, and post-traumatic stress (Eisen et al., 1992; Ross and Peselow, 2012; Kilpatrick et al., 1985; Kilpatrick and Acierno, 2003). Therefore, pertinent and timely counseling support is crucial. In light of this, it is imperative to explore the potential benefits that counseling could derive from the advancements in artificial intelligence (AI) and natural language processing (NLP).
In order to ensure effective counseling, active listening, combined with accurate interpretation of the client’s meaning and emotions is vital (Katz and McNulty, 1994). This demonstrates care and understanding, fostering a sense of being heard and valued, hence, facilitating deeper self-exploration for the client. The essential elements of effective support include politeness and empathy (Norcross,
12254
2002; Robert et al., 2011; Newbold et al., 2019). Previous studies indicate that politeness plays a critical role in facilitating supportive interactions (Feng et al., 2016) and aids in the disclosure of sensitive information (Bickmore and Picard, 2004; Lucas et al., 2014). Quantitative data supports the idea that empathetic interactions play a significant role in reducing issues in mental health support (Elliott et al., 2018) and are essential for building a strong therapeutic bond and connection (Robert et al., 2011).
Thus, a counseling conversational agent should adapt its support to users’ mental state whilst being polite and empathetic towards them. In Figure 1, the agent’s polite and empathetic response in the green box fosters higher user engagement and satisfaction compared to the counseling response in the red box. Furthermore, the agent’s response should align with the client’s emotions - offering reassurance and compassion when they are sad and showing understanding and providing encouragement when they are afraid.
This paper introduces a novel client’s emotionadaptive Polite and empAthetic counseLing dialogue system, named PAL to create a comfortable, engaging and cordial atmosphere during counseling. For this, we create a novel counseling conversational dataset in English utilizing the recently released EmoInHindi (Singh et al., 2022) dataset and annotate it with defined emotion, politeness, and empathy labels. We also annotate the Highquality and Low-quality Counseling Conversations dataset (Pérez-Rosas et al., 2019) with emotion, politeness and empathy label for our work. Finally, to build our system, we first warm-start with a supervised fine-tuning (SFT) of a language model on these datasets. Then, this trained SFT model is finetuned in a reinforcement learning (RL) framework using preference models to ensure preferences of utterance-emotion politeness (and empathy) consistency, adaptiveness, and correctness by designing six preference rewards.
The key contributions and/or the distinguishing attributes of our current work can be summarized as: (i) Prepared two large-scale counseling conversational datasets annotated with three distinct labels, viz. emotion, politeness and empathy; (ii) Proposed PAL - a novel emotion-adaptive Polite and empAthetic counseLing conversational agent fine-tuned in an RL setting; (iii) Designed a novel reward function consisting of six preference re-
wards and two generic rewards to generate engaging, fluent and interactive client’s emotion-adaptive polite and empathetic responses; (iv) Trained six transformer-based robust preference models comprising of two generation models and four classifiers to compute preference rewards; (v) Extensive automatic and human evaluation is performed with novel preference-specific and language qualityspecific metrics to demonstrate the effectiveness of the proposed system.

heading: 2 Related Work
text: Recent research has focused on developing dialogue systems for online mental health support, investigating approaches such as contextspecific adaptation, response diversity (PérezRosas et al., 2019; Zhang and Danescu-NiculescuMizil, 2020; Sharma et al., 2020b, 2021; Saha et al., 2022a), linguistic development of counselors (Zhang et al., 2019), conversational engagement patterns (Sharma et al., 2020a), moderation analysis (Wadden et al., 2021), therapeutic action detection (Lee et al., 2019), and cognitive restructuring identification (Pruksachatkun et al., 2019).
In our current work, we prioritize two crucial aspects of conversation, namely politeness and empathy, which are fundamental components of counseling (Norcross, 2002; Bickmore and Picard, 2004; Robert et al., 2011; Lucas et al., 2014). Previous research has demonstrated that perceiving conversational agents as caring and polite can enhance empathy and encourage the disclosure of personal and sensitive information (Kim et al., 2018; Mishra et al., 2022a; Samad et al., 2022; Mishra et al., 2023). For instance, Newbold et al. (2019) and Mishra et al. (2022b) applied Brown and Levinson’s politeness strategies (Brown et al., 1987) to create dialogue templates for mental health “mood log” and develop a polite persuasive dialogue system, respectively. The agent’s polite demeanor reflects a sense of care and personal investment, resembling a human companion.
Prior works (Ghandeharioun et al., 2019; Mieleszczenko-Kowszewicz et al., 2022) advocate incorporating emotion as an extra input in generation frameworks. This can be achieved by including emotion either during decoding phase or using it as a reward to assist models in generating responses that align with the user’s emotions. The role of empathy in therapy has been explored in (Morris et al., 2018; Sharma et al., 2020b), while studies
in (Sharma et al., 2021; Saha et al., 2022b) have investigated methods to induce empathy in mental health support conversations. Further, several research studies have endeavoured to understand and create computational techniques for recognizing empathy in various contexts, such as face-to-face therapy (Gibson et al., 2016) and text-based peerto-peer support systems (Sharma et al., 2020b).
Existing studies primarily focus on analyzing social media posts and activities to detect different mental health concerns and provide support to users. They often emphasize incorporating empathy into the agent’s responses to instill motivation and hope in the user. In contrast, our current work aims to develop a counseling dialogue agent that can adapt its politeness and empathy based on the user’s emotions.

heading: 3 Dataset
text: We perform the experiments on two different counseling datasets, viz. English translation of EmoInHindi (Singh et al., 2022) dataset and High-quality and Low-quality Counseling Conversations dataset (Pérez-Rosas et al., 2019).

heading: 3.1 English Translation of EmoInHindi
text: EmoInHindi is a Hindi conversational dataset comprising 1,814 dialogues focused on mental health and legal counseling for women and children victims of crimes like stalking, harassment, domestic violence, etc. For our present work, we translate each EmoInHindi dialogue to English, preserving Fluency (F) and Adequacy (A). The resulting dataset is named “en-EmoInHindi”. The translation is conducted in two phases with four experienced human translators who are native Hindi speakers
with post-graduate qualifications and relevant expertise.
In the first phase, we use Google Translate to translate the utterances from Hindi to English. The translators are asked to manually check and mark every translated utterance for both F and A on a scale of 1-52 (White and O’Connell, 1996). The average scores of F and A are found to be 3.23 and 3.10 respectively. In the second phase, to further
enhance translation quality, we select utterances with F or A scores of 1, 2, or 3 and instruct translators to post-edit them by referring to the original Hindi utterance. Subsequently, all four translators rate each corrected utterance on the 1-5 scale for both F and A. This process yields average F and A scores of 4.65 and 4.48, respectively, indicating good translation quality. In the scoring of fluency and adequacy, the inter-translator agreement has been found to be <69.4%, 66.4%> and <74.8%, 71.3%> in phases 1 and 2, respectively. The dataset statistics are given in Table 1. A couple of sample translations from EmoInHindi are shown in Figure 2. A few examples of erroneous utterances observed during translation are presented in Section A.1 of the Appendix.

heading: 3.2 High-quality and Low-quality Counseling Conversations Dataset
text: The High-quality and Low-quality Counseling Conversations (HLCC) dataset consists of 258 counseling interactions, with 154 interactions labeled as high-quality counseling and 104 labeled as lowquality counseling. The counseling is conducted using Motivational Interviewing (MI) and each counseling conversation has annotations on two microskills frequently used in the evaluation of MI counseling, namely reflective listening, and questions. The counseling sessions primarily cover different topics on behavior change, including smoking cessation, alcohol consumption, substance abuse, weight management, and medication adherence. The dataset statistics are given in Table 2.
2Fluency - 5: Flawless, 4: Good, 3: Non-native, 2: Disfluent, 1: Incomprehensible; Adequacy - 5: All, 4: Most, 3: Much, 2: Little, 1: None

heading: 3.3 Dataset Annotation
text: In order to build PAL, we annotate en-EmoInHindi and HLCC datasets with three distinct aspects, viz. emotion, politeness, and empathy, and name these datasets as EPE-enEIH and EPE-HLCC, respectively. Each task is carried out with the help of three annotators, two with Ph.D. degree in Linguistics and one with a Master’s degree3. All three annotators are proficient in English, sufficiently acquainted with labeling tasks and well-versed with the concepts of emotions, politeness and empathy.

heading: 3.3.1 Annotation for emotion
text: We annotate the client’s utterances in both datasets with emotion labels utilizing the EMPATHETICDIALOGUES dataset (Rashkin et al., 2019) comprising of 25K conversations grounded on 32 different emotions in two phases. In the first phase, we begin by reducing the number of emotions in the EMPATHETICDIALOGUES dataset from 32 to 11 by consolidating similar emotion labels into one emotion to obtain a better class distribution. This dataset with consolidated emotion labels will be called EmpDial-merged, hereafter. Table 3 showcases the specifics of the merged emotion labels.
Then, a pre-trained RoBERTa (Liu et al., 2019) based model is fine-tuned on EmpDial-merged dataset for emotion classification. We notice that the emotion classifier trained on 11 emotions outperformed the classifier built on 32 emotion labels by a substantial margin4. This trained emotion classifier is used to label each utterance in both the datasets with an appropriate emotion class.
In the second phase, we employ three annotators and brief them about each emotion class along with a few examples. Thereafter, we ask them to manually cross-check the predicted emotion label of the utterances in both the datasets and correct it in case of any error. This gives our gold-standard datasets. We observe a reliable multi-rater Kappa (McHugh, 2012) agreement ratio of 71.6% and 73.4% for HLCC and en-EmoInHindi datasets, respectively.

heading: 3.3.2 Annotation for politeness
text: Politeness annotation for the agent’s utterances in both HLCC and en-EmoInHindi datasets is carried out in two phases. In the first phase, each utterance is passed through the Stanford Politeness Classifier trained on Wikipedia requests data (Danescu-
3Annotators are paid as per institute norms. 4The accuracy of the classifiers trained on 32 and 11 emotion labels was found to be 58.17% and 77.46%, respectively.
Niculescu-Mizil et al., 2013). The classifier outputs a politeness score between 0 and 1 and classifies the utterance as polite if the score is >= 0.5, else impolite. In order to effectively adapt the conversation to the user’s mood and emotions during an ongoing dialogue, we fine-grain the politeness label as impolite, low_polite and highly_polite.
The utterances with a score < 0.5 are marked as impolite. The polite utterances with a score >= 0.5 and <= 0.65 are marked as low_polite and the utterances with a score > 0.65 are marked as highly_polite. The politeness score range for low_polite and highly_polite is determined empirically through inspection on a small subset of samples. In the second phase, we ask the same three annotators to manually cross-check the labels and correct the labels if found to be incorrect. A multirater Kappa (McHugh, 2012) agreement ratio of approximately 84% is observed among these annotators. Finally, majority voting is applied to decide the final politeness class of the given utterance.

heading: 3.3.3 Annotation for empathy
text: The agent’s utterances in both datasets are annotated with one of the ternary empathy labels, viz. non-empathetic, low_empathetic and highly_empathetic. To lessen the manual efforts, the empathy annotation proceeds in two stages. In the first stage, we randomly sample 100 and 600 dialogues from HLCC and en-EmoInHindi datasets, respectively and then ask all three annotators to manually annotate with the required empathy label. In the second stage, pre-trained RoBERTa-large (Liu et al., 2019) is fine-tuned on manually annotated samples to build an empathy classifier. Then, the remaining utterances in both datasets are passed through this classifier to predict the corresponding empathy label. Lastly, same annotators are asked to cross-verify the predicted labels and correct them, if needed. A reliable multi-rater Kappa (McHugh, 2012) agreement ratios of 78.2% and 77.1% in the first phase, and 86.5% and 84.3% in the second phase are observed for HLCC and en-EmoInHindi datasets, respectively.

heading: 4 Proposed Methodology
text: To build our proposed system, first, to foster natural language interaction between the counseling agent and the client, we get a warm-start by training a client’s emotion-informed dialogue model (EIDM) in a supervised learning framework. Then, to generate the client’s emotion-adaptive polite and
empathetic responses, EIDM is fine-tuned with a Proximal Policy Optimization (PPO) (Schulman et al., 2017) loss utilizing novel designed eight rewards. This end model is our proposed Polite and empAthetic counseLing conversational system PAL. To generate polite and empathetic responses as per user’s varying emotion, we use six preference models viz. two sequence-to-sequence generation models - Utterance-emotion-politeness (Guep) and Utterance-emotion-empathy (Guee) and four classifiers - client’s utterance+emotion politeness (Cuep) and empathy (Cuee) and agent’s utterance politeness (Caep) and empathy (Caee). These preference models are used to design rewards that are maximized to force the agent to generate the client’s emotion-adaptive polite and empathetic responses. The overall architecture of the proposed system is shown in Figure 3.
A multi-turn dialogue with a client’s and counseling agent’s utterances can be represented as d = {ca0, cl0, ..., cai, cli, ..., caT−1, clT−1}, where cai and cli give the counseling agent’s and client’s ith response out of T turns. Let the client’s current utterance with corresponding emotion, ek ∈ E = {e1, .., em} (m is the number of emotion classes), be represented as clei = [cli + cliek ]. Further, let ρca and ρcle representing agent’s and client’s utterances with corresponding emotions distributions, respectively. Now for a given context, an emotioninformed dialogue model (EMID) ρθ(d) is trained by minimizing the KL-divergence (Kullback and Leibler, 1951) between the predicted and ground truth utterances probability distributions. ρθ(d) can be expressed as:
ρθ(d) =
T−1∏
T=0
ρcle(cli|cle<i, ca<i)ρca(cai|cle<i, ca<i)
(1)

heading: 4.1 PAL
text: ρθ(d) is further fine-tuned in an RL framework with a PPO loss using a novel reward function R = β1R1 + β2R2 + β3R3 + β4R4 + β5R5 + β6R6+β7R7+β8R8, comprising of eight rewards R1, R2, R3, R4, R5, R6, R7, and R8, to generate client’s emotion-adaptive polite and empathetic counseling responses while keeping informative content intact. All rewards values are between 0 and 1 and β1+β2+β3+β4+β5+β6+β7+β8 = 1. These rewards are obtained using six preference models to ensure politeness and empathy preferences as per the client’s emotion. At each time step of training, for a given context, n-candidate responses are generated and are scored in terms of reward using these preference models. This score is further propagated in the policy loss as feedback to force the agent toward generating preferred highquality candidates. RL Policy and PPO loss details can be referred to in Section A.2 of the Appendix.

heading: 4.1.1 Prefernce Models and Rewards
text: The proposed reward function R consists of two types of rewards viz. (i) Preference rewards (R1, R2, R3, R4, R5, and R6) - to adapt politeness and empathy as per client’s emotion, and (ii) Generic rewards (R7, and R8) - to account for dialog flow consistency and diversity in generated responses. To design preference rewards, six preference models are trained and generic rewards are obtained using the BERT-based cosine similarity model.
Preference Models. R1 and R2 are designed by training two BART-large (Lewis et al., 2019) based seq2seq models - (i) Utterance-emotionpoliteness (Guep): It takes client’s utterance with its corresponding emotion clei as input and outputs agent’s utterance with its corresponding politeness [cai + caipoj ] (here poj ∈ Po = {po0, po1, po2}), (ii) Utterance-emotion-empathy (Guee): It takes client’s utterance with its corresponding emotion clei as input and outputs agent’s utterance with its corresponding empathy [cai + caieml ] (here, eml ∈ Em = {em0, em1, em2}). R3, R4, R5, and R6 are designed by training four RoBERTalarge (Liu et al., 2019) based classifiers - (i) client’s utterance+emotion politeness (Cuep): Input - clei , Output - caipoj (ii) client’s utterance+emotion empathy (Cuee): Input - clei , Output - caieml (iii) agent’s utterance politeness (Caep): Input - cai,
Output - caipoj (iv) agent’s utterance empathy (Caee): Input - cai, Output - caieml .
Utterance-emotion-politeness and Utteranceemotion-empathy consistency rewards. It is a challenging task to approximate the relationship between client’s dynamically changing emotion and the counselor’s politeness or empathy, as for each client’s utterance and its corresponding emotion, a counseling agent should change the way of counseling by applying correct politeness and empathy in its responses. Hence, to force the agent to employ emotion adaptive utterance-wise correct politeness (R1) at turn T , the difference between negative likelihood losses (NLL) produced by Guep when given with ground truth responses input-output pair (NLL(Ggruep)) and predicted responses input-output pair (NLL(Gpruep)) is taken. Similarly, for empathy, R2 is computed using Guee. The responses, deviating from ground truth response pair loss are penalized. R1 and R2 can be formulated as:
R1 = tanh(NLL((Ggruep)− α×NLL(Gpruep)) (2)
R2 = tanh(NLL((Ggruee)− α×NLL(Gpruee)) (3)
Politeness and Empathy-adaptive Rewards. To adapt varying politeness and empathy levels as per the client’s utterance and emotion, the agent needs to be forced towards correct politeness and empathy. Using Cuep and Cuee, we get the agent’s politeness and empathy probabilities for client’s utterance with emotion (clei) for ground truth (Cgruep
and Cgruee) as well as agent’s predicted responses (Cpruep and Cpruee), and take the difference between them. The greater the deviation of the predicted response politeness or empathy, the greater will be the penalization.
R3 = prob(Cgruep)− γ × prob(Cpruep) (4)
R4 = prob(Cgruee)− γ × prob(Cpruee) (5)
Politeness and Empathy Correctness Rewards. Generated response deviation from true politeness and empathy labels is restricted by computing politeness and empathy probabilities of ground truth agent’s utterances (Cgraep and Cgraee) and agent’s predicted utterances (Cpraep and Cpraee) and taking the difference between them.
R5 = prob(Cgraep)− δ × prob(Cpraep) (6)
R6 = prob(Cgraee)− δ × prob(Cpraee) (7)
where, α, γ, and δ work as the penalization factors, i.e. greater respective values will result in hard penalization5.
Dialogue Flow Consistency Reward. In achieving all six preferences, generated responses should be consistent with the context of the dialogue. Therefore, we take the cosine similarity of generated response (prT ) with context at turns T − 1 (ctxtT−1) and T (ctxtT ). To avoid similar response generation, the threshold value of 0.75
5The value of α, γ, and δ is taken as greater than or equal to 1.
is taken. Above the threshold value similarity, the generated responses are penalized.
R7 = (min(cos(prT , ctxtT ) +
cos(prT ,ctxtT−1) 2 ), 0.75)
2 (8)
Diversity Reward. To ensure an interactive and engaging counseling dialogue, diverse responses are crucial as the length of the dialogue increases. In this regard, we introduce the diversity reward R8, which takes into account two factors influencing dialogue quality: diversity, measured as the Jaccard similarity between prT and prT−1 at turns T and T − 1 (Jaccard, 1912), and the dialogue length.
R8 = (len(d))− ( prT−1 ∩ prT prT−1 ∪ prT ) (9)

heading: 5 Experiments
text: We compare the performance of our PPO lossbased model to three baselines: GPT-2-medium (Radford et al., 2019), a strong baseline ARDM (Wu et al., 2021) and EIDM. Due to space restrictions implementation details are given in Section A.3 of the Appendix. Evaluation Metrics. To evaluate the performance of the proposed system, PAL, both automatic and human evaluations are conducted. The generation preference models are assessed using Bleu score (BLEU) (Papineni et al., 2002), Perplexity (PPL) (Brown et al., 1992), METEOR score (MET) (Banerjee and Lavie, 2005), NIST score (NIST) (Doddington, 2002) and Rogue-2 F-1 score (R2-F1) (Lin, 2004). All four classifier preference models are evaluated in terms of Weighted Accuracy (W-ACC) and Macro-F1. In automatic evaluation, PAL efficacy is evaluated in terms of two aspects: (i) preference success - Emotion-politeness consistency (EPC), Emotion-empathy consistency (EEC), Politeness correctness (PC), and Empathy Correctness (EC)6, and (ii) response quality - Perplexity (PPL) and Response-length (R-LEN).
For human evaluation, we recruit three evaluators with postgraduate qualifications and relevant
6EPC, EEC, PC, and EC are computed by four respective classifiers in Table 5.
experience in the field7. Initially, each evaluator engages with the system seven times, with a different set of responses each time. These 21 humanevaluated interactions are then cross-verified by experts from government-run institutions to ensure evaluation quality. Upon passing verification, an additional 30 interactions (10 per evaluator) are evaluated, resulting in a total of 51 humanevaluated dialogues. Human evaluation metrics include Emotion-politeness consistency (EPC), Emotion-empathy consistency (EEC), Politeness correctness (PC), Empathy correctness (EC), Fluency (Flue), Non-Repetitiveness (N-Rep) and Consistency (Cons). All dialogue interactions are evaluated on an integer Likert scale of 1-58 9.

heading: 6 Results and Analysis
text: First, all six preference models results are shown i.e., two generation models and four classifiers. Thereafter, the results of the proposed system PAL are stated and compared with three baselines, viz. LM (GPT-2-medium), ARDM, EIDM and three variants of PAL, PAL-R (PAL with zero reward), PAL-GR (PAL without generic rewards) and PALPR (PAL without preference rewards).
For both the datasets - EPE-enEIH and EPEHLCC, Tables 5 and 4 show the evaluation results of all the preference models. Scores of PPL, BLEU, MET and R-2-F1 in Table 4 depicts that BARTlarge performed well enough to generate fluent and diverse sentences. It is also seen that all four RoBERTa-large based classifiers gave well scores of W-ACC and Macro-F1.

heading: 6.1 Automatic evaluation
text: Table 6 shows that proposed PAL gives better performance than the baselines: LM, ARDM, and
7Human evaluators are remunerated according to our institute’s policies.
81-5 denotes low to high. 9An inter-evaluator agreement ratio of 75.1% is observed
in evaluations.
EIDM as well as variants PAL-R, PAL-GR, PALPR in terms of all metrics. It can be observed that PAL achieves well scores in terms of preferences EPC, EEC, PC, and EC with a significant difference of <11.4, 9.3, 4.4, 5.6, 1.4, 2.7>, <11.3, 7.7, 3.3, 5, 1.8, 4.1>, <5.8, 4.4, 3.2, 3.8, 1, 2.1>, and <8.6, 5.4, 4, 5.5, 1.9, 3.4>, respectively as compared to LM, ARDM, EIDM, PAL-R, PAL-GR, and PAL-PR10 on EPE-enEIH. Hence, the design of our reward function in PAL is justified by its ability to encourage the agent to generate emotionadaptive, correct, polite, and empathetic responses aligned with the client’s preferences. Similarly, PAL outperforms all other models in terms of PPL and R-LEN. Hence, the influence of generic rewards can be attributed to the model’s ability to establish a natural and seamless connection with the user, leading to interactive and engaging interactions. Additionally, it is noteworthy that there is minimal disparity between the results of ARDM and GPT-2, while EIDM demonstrates substantial improvement over both of them. It could be due to the use of emotion aspects as well during training.
10All results are statistically significant. We perform a statistical significance test, Welch’s t-test (Welch, 1947), and it is conducted at 5% (0.05) significance level.
Further, scores of EPC, EEC, PC, EC for PALPR are lower than PAL-GR suggesting the importance of preference rewards. Similarly, scores of PPL and R-LEN are lower in absence of generic rewards, hence, showcasing their requirement as well. The obtained results strengthen our hypothesis that incorporating politeness and empathy aspects into the counseling dialogue system enables to create a more engaging, comfortable, cordial, and clientunderstanding environment.

heading: 6.2 Human evaluation
text: Human assessments are recorded in Table 7. It can be seen that as compared to the LM, ARDM, EIDM, and PAL-R, the proposed PAL yields better scores in terms of EPC, EEC, PC, EC, Flue, N-Rep and Cons with a difference of <0.75, 0.68, 0.47, 0.57>, <0.83, 0.75, 0.54, 0.47>, <0.93, 0.83, 0.52, 0.55>, <0.94, 0.88, 0.52, 0.49>, <1.04, 0.73, 0.27, 0.40>, <0.78, 0.65, 0.58, 0.49>, and <1.01, 0.83, 0.41, 0.49>, respectively on EPE-enEIH. The significance of all eight rewards is evident in generating accurate, natural, non-repetitive, emotion-adaptive, polite, and empathetic utterances. This indicates that the inclusion of preference and generic rewards contributes to the ability of PAL to establish a rap-
port with the client by generating engaging and interactive responses. A detailed discussion with interaction examples is given in Section A.5 of the Appendix.

heading: 6.3 Error Analysis
text: Our proposed model PAL performs well in terms of both preference and generic metrics but it is also seen that scores of EPC = 73.7% and EEC = 75.9% in automatic evaluation can be better. It can be due to the fact that PAL is trained as a dialogue model and due to finite memory capacity, it may not be able to memorize the old information and drop it with the increase in context as per turns. Similarly, low and high scores of PPL on EPE-enEIH and EPE-HLCC, respectively, can be due to the low and high variance in the datasets. As EPE-enEIH dataset is focused on a fixed topic of harassment of women and children, it has low variance, whereas EPE-HLCC dataset utterances have been taken from the transcripts of YouTube conversations between different clients and counselors covering a wide range of topics, resulting in high variance.

heading: 7 Conclusion and Future Direction
text: In this work, we proposed an emotion-adaptive polite and empathetic counseling conversational agent PAL to offer higher engagement in e-counseling sessions and resolve the issues faced by the clients. To this end, we prepared two counseling conversational datasets: EPE-enEIH and EPE-HLCC annotated with emotion, politeness and empathy labels. These datasets are used to train PAL in an RL framework. A novel reward function is designed to ensure the preferences of emotion-adaptive politeness and empathy in generated responses. Our extensive automatic and human evaluation results conclude the requirement and effectiveness of designed rewards considering all three aspects - emotion, politeness and empathy over supervised fine-tuned based models. It is found that PAL is able to efficiently generate engaging emotion-adaptive polite and empathetic responses. In future, we would like to incorporate domain and common-sense knowledge into the responses to facilitate more realistic and engaging counseling conversations.
Limitations
Our proposed model has also some limitations. First, the requirement of large memory power of
GPU (here, 40 GB) due to the use of GPT-2medium in the training of PAL. Further, weight optimization for each of the possible combinations of different rewards may lead to model training and validation time to months. Hence, some heuristic is adopted to choose some sets of combinations of reward weights. In case of continuous, short and direct responses during interaction like ‘Yes’, ‘I don’t know’, ‘No’, ‘2’, ‘Yeah’, the system first tries to counsel client by inquiring about their issue but after three or four turns it starts deviating and may generate repetitive or inconsistent responses. This can be due to the fact that the datasets which are used to train the PAL mostly consists of interactive dialogues with long utterances, hence model gets confused when treated with short and direct responses. Lastly, it is also observed that sometimes, model asks too many questions to the user. This may dissatisfy the user. Hence, the model should be forced to generate only relevant inquiries by discriminating the irrelevant inquiries. This opens up the door for future studies to build a counseling dialogue system.
Ethics Statement
We recognize that there are substantial stakes involved in developing computational models for use in counseling applications and this fact necessitates careful attention to ethical issues. It is crucial for a counseling dialogue system to render support and suggestions to the clients while maintaining their privacy. Thus, proper care has been taken to maintain the client’s privacy. Despite the fact that we have used publicly available counseling conversational datasets, the annotators pledged not to contact or deanonymize any of the users or share the data with others. Further, it is to be noted that this paper does not make any therapy recommendations or clinical diagnostic claims.

heading: Acknowledgements
text: Kshtij Mishra acknowledges the support received from the Indian government’s “Prime Minister’s Research Fellowship (PMRF) Program”. Priyanshu Priya acknowledges the financial support provided by the Department of Science and Technology, Ministry of Science and Technology, Government of India, through the Innovation in Science Pursuit for Inspired Research (INSPIRE) Fellowship. The authors also acknowledge partial support from the project titled "An Empathetic Knowl-
edge Grounded Conversational System for Mental Health Counseling and Legal Assistance," sponsored by IHUB Anubhuti, TIH, IIIT Delhi.

heading: A Appendix
text: A.1 Errors in English Translation of EmoInHindi
The translation of Hindi utterances to English causes several errors. Some of them are illustrated in Figure 4. One of the errors that we observe is the translation of Hindi phrases and idioms into English. For instance, as shown in example 1 in Figure 4, the text “ye bhains ke aage bin bajaana jaisa h” is translated to “it’s like barking in front of a buffalo”. Also, we observe that few named entities are translated into English as shown in Example 4, the named entity “Rakshak” is translated into “keeper” or “protector”. These errors are manually post-edited with the help of human translators to create a high-quality dataset.
A.2 RL Policy and PPO Loss
RL policy is a probability mapping function Pθ predicting a feasible response pr with t tokens (an action) for a given context (a state).
Pθ(pr1:t|x) = t∏
k=0
Pθ(grk|y<k, x) (10)
As only minor preferred meta-communicative language aspects (politeness and empathy) updates are needed keeping informative content same, PPO loss is selected to update the policy which results in low variance with respect to the old policy. Policy optimization can be formulated in three steps. First, gradient ascent is applied on loss function J(θ) to maximize the expected reward:
∇θJ(θ) = Er∼Pθ [∇θlogPθ(r)ÂR] (11)
Second, large deviations are restricted by replacing the log term with an importance sampling term and catastrophic forgetting is prevented using clipping:
LCLIP(θ) = Ê[min(prr(θ)Âr, clip(pry(θ), 1−ε, 1+ε)Âr)] (12)
Here, the probability ratio between the new and old policies is calculated as prr(θ) = Pnewθ /Poldθ . ε and Ây gives the clipping range and estimated advantage (normalized rewards), respectively. Finally, parameter updating is done as follows:
θk+1 = argmax θ E s,a∼Pθk [LCLIP] (13)
A.3 Implementation Details
All classifiers are built by fine-tuning the RoBERTalarge (Liu et al., 2019) model. To train generation preference models, BART-large (Lewis et al., 2019) having a bidirectional (BERT-like) encoder and an autoregressive (GPT-like) decoder is used. The Language model GPT2-medium (Radford et al., 2019), ARDM (Wu et al., 2021) and EIDM are trained in a supervised learning setting with AdamW optimizer (Loshchilov and Hutter, 2018), learning rate α = 2e−05 and seed_value = 10. Fine-tuning of trained EIDM is done in an RL setting by experimenting with the different number of candidate responses i.e., n = 2, 3, 4, 5, 10. n = 3 is selected as the final value as per the loss obtained. The generated candidates are decoded using nucleus sampling (Holtzman et al., 2019) with temperature T = 0.8 and probability p = 0.9.
The proposed system PAL is trained considering the seed_value = 10, human_reward = 10, max_candidate_length = 50, and AdamW optimizer (Loshchilov and Hutter, 2018) with a learning rate of α = 2e−05, ε = 0.2 and epochs = 20. The reward weight combination of 0.1, 0.1, 0.2, 0.2, 0.1, 0.1, 0.1, 0.1 are chosen as the final weights for β1, β2, β3, β4, β5, β6, β7, and β8, respectively. Lastly, for R1, R2, R3, R4, R5, and R6, the penalization factors α, γ and δ are set to 2.
A.3.1 Hardware Configuration The device configurations that we use for the experiment are: (a) GPU: A100-PCIE-40GB, (b) CUDA Support: CUDA 11.x (or later.) (c) GPU clocks: Base: 765 MHz, Boost: 1410 MHz (d) Memory Size: 40 GB (e) Memory Type: HBM2 (f) Memory clock: 1215 MHz (g) Bus Width: 5120 bits (h) Total board power: 250 W.
A.4 Weight Optimization We experimented with the various combination of weights (β1, β2, β3, β4, β5, β6, β7 and β8) to find the correct combination of weights for the reward function. We validate these rewards on 10% holdout data and finally, the combination of weights that gives the highest perplexity score is selected to train PAL. Table 8 shows the weights considered for optimization with the dataset EPE-enEIH. It can be inferred from the table that a better perplexity score is obtained when all the rewards are considered. Further, we observe that when any reward is eliminated, the perplexity score is dropped, which, in turn, demonstrates the significance of each reward.
A.5 Discussion A few examples of client’s interaction with our proposed system PAL and the baseline model EIDM is shown in Table 9 and Table 10. We notice from the examples that the user gave a better score to our
model PAL than EIDM in terms of all human evaluation metrics, viz. EPC, EEC, PC, EC, Flue, NRep and Cons. Thus, it can be concluded that the proposed system is able to incorporate politeness and empathy in the generated responses depending on the client’s underlying emotion, thereby showcasing friendliness and a caring attitude towards the client. Also, these results qualitatively show that PAL is able to generate more fluent, coherent, and engaging responses.
However, some limitations are also observed in the responses generated by PAL. For example, in Table 9, for the client’s utterance like “I am probably not going to stop smoking; I haven’t even considered it, and until today’s fantastic news at work, I had no intention of doing so.”, the model again responds with Okay, so where do you put yourself on a scale of 1 to 10?. This may be due to the fact that our designed reward function is unable to cover the significant aspects of common-sense knowledge that can probably map “no intention to quit” with the being on “1” on the scale of 1 to 10. Moreover, the counselor/agent also needs some domain knowledge to generate more informative response instead of generic one. For instance, in the counselor’s utterance “That’s really great! Well, there are now a lot of products available that can help you fight the urge to smoke. For example, there are products that can help you replace nicotine, which can help ease the withdrawal symptoms you might feel when quitting or cutting down. You could also have some medications that might take away even some of the desire to smoke. You can look ahead to whichever is feasible for you.”, a better response would be suggesting some appropriate medication like “Chantix is a medication for smoke cessation that you might consider if you want to quit.” instead of vaguely saying “You could also have some medications that might take away even some of the
desire to smoke.”. We may need relevant domain and/or common-sense knowledge to generate more realistic responses in counseling conversations.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
3 A2. Did you discuss any potential risks of your work? 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1

heading: 7 A4. Have you used AI writing assistants when working on this paper?
text: Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 9
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 9

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 4
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 3

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 3
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 3

heading: Future Work
text: In future, we would like to incorporate domain and common-sense knowledge into the responses to facilitate more realistic and engaging counseling conversations. Limitations
Our proposed model has also some limitations. First, the requirement of large memory power of
GPU (here, 40 GB) due to the use of GPT-2medium in the training of PAL. Further, weight optimization for each of the possible combinations of different rewards may lead to model training and validation time to months. Hence, some heuristic is adopted to choose some sets of combinations of reward weights. In case of continuous, short and direct responses during interaction like ‘Yes’, ‘I don’t know’, ‘No’, ‘2’, ‘Yeah’, the system first tries to counsel client by inquiring about their issue but after three or four turns it starts deviating and may generate repetitive or inconsistent responses. This can be due to the fact that the datasets which are used to train the PAL mostly consists of interactive dialogues with long utterances, hence model gets confused when treated with short and direct responses. Lastly, it is also observed that sometimes, model asks too many questions to the user. This may dissatisfy the user. Hence, the model should be forced to generate only relevant inquiries by discriminating the irrelevant inquiries. This opens up the door for future studies to build a counseling dialogue system