e-THERAPIST: I suggest you to cultivate a mindset of positivity and nurture uplifting thoughts
Authors: Kshitij Mishra
Priyanshu Priya
Manisha Burja
Asif Ekbal
Venue: EMNLP 2023 Main (2023)
Keywords: Dialogue Systems
Psychotherapy
Persona
Sentiment
Politeness
Interpersonal communication Strategy
Rewards
Reinforcement Learning
TL;DR: e-THERAPIST: a polite interpersonal psychotherapy dialogue system

Abstract: The shortage of therapists for mental health patients emphasizes the importance of globally accessible dialogue systems alleviating their issues. To have effective interpersonal psychotherapy, these systems must exhibit politeness and empathy when needed. However, these factors may vary as per the user's gender, age, persona, and sentiment. Hence, in order to establish trust and provide a personalized cordial experience, it is essential that generated responses should be tailored to individual profiles and attributes. Focusing on this objective, we propose e-THERAPIST, a novel polite interpersonal psychotherapy dialogue system to address issues like depression, anxiety, schizophrenia, etc. We begin by curating a unique conversational dataset for psychotherapy, called PsyCon. It is annotated at two levels: (i) dialogue-level - including user's profile information (gender, age, persona) and therapist's psychotherapeutic approach; and (ii) utterance-level - encompassing user's sentiment and therapist's politeness, and interpersonal behaviour. Then, we devise a novel reward model to adapt correct polite interpersonal behaviour and use it to train e-THERAPIST on PsyCon employing NLPO loss. Our extensive empirical analysis validates the effectiveness of each component of the proposed e-THERAPIST demonstrating its potential impact in psychotherapy settings.

e-THERAPIST: I suggest you to cultivate a mindset of positivity and nurture uplifting thoughts
Kshitij Mishra" and Priyanshu Priya* and Manisha Burja and Asif Ekbal Department of Computer Science and Engineering Indian Institute of Technology Patna, India
(kshitij_1921 ,r prinassshhhpp 2021 burja_222 1cs05, asif)eiitp.cc in
Abstract
the demand for counseling services continues rise, and the existing mental health workforce is struggling to meet the needs adcquately. Hence, dialogue systems possessing social influence skills like psychotherapy are crucial for expanding the ap- plication of technology to a wide range of realistic situations.
The shortage of therapists for mental health patients emphasizes the importance of glob- ally accessible dialogue systems alleviating their issues. To have effective interpersonal psychotherapy, these systems must exhibit po- liteness and empathy when needed. However, these factors may vary as per the user's gen- der, age, persona, and sentiment. Hence, in order to establish trust and provide a person- alized cordial experience. generated responses should be tailored to in- dividual profiles and attributes. Focusing on this objective, we propose e-THERAPIST. novel polite interpersonal psychotherapy dia- logue system to address issues like depression, anxiety, schizophrenia, etc. We begin by rating a unique conversational dataset for psy- chotherapy, called PsyCON. It annotated at two levels: (i) dialogue-leved including user's profile information (gender, age, persona) and therapist's psychotherapeutic approach; and (ii) utterance-level encompassing user's 's sen- timent and therapist's politeness, and interper- sonal behaviour. Then, we devise novel re- ward model to adapt correct polite interpersonal behaviour and use to train e- e-THERAPIST PsyCON employing NLPO loss. Our extensive empirical analysis validates the effectiveness of each component of the proposed e-THERAPIST demonstrating its potential impact in psychotherapy set- tings'.
Politeness has shown to be effective in driving smooth and engaging conversation during psy- chotherapy (Budiarta et al., 2021). However, polite- ness, in itself, encompasses various aspects (Kita- mura, 2000; Laplante and Ambady, 2003; Stephan et al., 2010), making it challenging instil in a dia- logue system. For instance, the degree of politeness in communication varies based on the gender and age of the person involved Daanecu-Niculesse Mizil et al., 2013; Mahmud, 2013; Firdaus et al.. 2022b); conversations with femalcs/clders often include more polite expressions than males/youths. Politeness can also be influenced by various pects of individual's personality traits (Gold- berg, 1993; Hirsh et al., 2010; Xafizovna and Bobo- qulovna, 2022).
essential that
Interpersonal behaviour advances the under- standing of the interpersonal dispositions associ- ated with psychological problems. The interper- sonal circumplex (IPC) model assesses these dispo- sitions and facilitates the comprehension of user's s social cognition, motivation, and behaviour (Locke et al., 2017). It indicates that adults and youth facing psychological issues are more likely to seek negative feedback and criticism in their interactions compared to older individuals. Further, females tend to initiate more interpersonal stressors than males, and their responses to these stressors often involve rumination and negative thinking patterns. The IPC model can also help identify patterns and relationships berween personality traits, thereby fostering an understanding of how individuals in- teract with others (Gurtman, 2009).
1 Introduction
The prevalence of psychological and mental disor- ders, such as depression, anxicty, stress and others, is increasing globally. Approximately 5% of the adult population worldwide is estimated to expe- rience depression (WHO, 2023). Consequently,
"The authors are jointly first authors.
'The dataset and code can be accessed at http://hhi...ommiiiiaaaaaattttte or http://www.iii...e..ihimmmnnnnne himle- THERAPIST
The incorporation of polite and interpersonal

---

Elderly Fermals Openness
Eiderly Main Opanness you here? (ssiphil
newly designed seven rewards in reinforcement learning (RL) setting (Casanueva et al., 2018; Mes- gar et al., 2020; Lambert and von Werra, 2023). This allows e- e-THERAPIST to learn and enhance their performance based on the rewards received through interactions with the environment. To build e- e-THERAPIST, we first curate a novel conversa- tional dataset, named PSYCON by employing well- designed prompts with manual interventions. Then, we annotate the user's utterances with sentiment, and agent's utterances with politeness and interper- sonal behavioural information. First, PSyCON is used to fine-tune a large language model (LLM) in a supervised setting. Then, we fine-tune this trained LLM in an RL framework incorporating novel re- ward function. This reward function ensures appro- priate psychotherapeutic approach, politeness, and interpersonal behaviour based on gender, age, per- sona and sentiment of the user along with context adequacy and fluency in the generated responses. This reward is used in a Natural Language Pol- icy Optimization (NLPO) loss (Ramamurthy et al., 2022) to optimize the model fine-tuned in super- vised setting. Finally, the performance of the pro- posed system is evaluated through both automatic metrics and human assessment.
T. Aery good momiin madam! (bnkith))
feeting
ue Good morring, I'm feeling too (negalive)
M. Im not teeting good these daye. 000x your help. Incoolieol
help
T: Im realy scrry to hear that. going on? sundinssinniing)
T. Sonry leam this want to hip you? jolnectin)
erything my lte no mobivation nothing.
right in my Itf. I don't have any inagelio))
I: but don't momy, togethnor we will ixptore solutons would beter Could you please tell me more shhar's bean (empathatic)
7: Okay, let's explore that Can you plaaee uxphen about aie
exce fancing these days?
Young Pomate Openness
Elderly Male Agresableness
T: Hello dear! I'm ready provide you with empathete support How are you [helatal)
bitngs you here today? (halota
feolnn lothorcto days, Pleas help mas (negulos))
Not aead, energetic. (negative)
I: anologine for the odd whuation you planse sisar delets about
my dear frand I presume you are cays. would lnon to help you if you rould (empaineti)
(empltll)
U You crunse. oun share with you. (ppattion)
apprectate your villngness share. Would mind
U:1 everything it going wrong in my Iife
overshalming sense that
current expedences? jundetataniing U: I've been strugaling fnnd any dishaartniig to look medivotion and fecting empply
onthutioom
in any activiiie. (ongativ))
T: Hay gin, underasand your aitutiin
understand
Atratagins that would be helphul if
will do our best help you. Leta your chrummttences. (empathetis)
quasstons" (ommuthstts)
Figure 1: Example demonstrating the polite and inter- personal behaviour of the therapist (T) based on the user's (U) gender, age, persona and sentiment. The text in green depicts the user's 's personal profile (gender, age and persona). The text in pink, blue and grey depict the user's sentiment, the therapist's polite and interpersonal behaviour, respectively.
In summary, the key contributions of our current work are summarized as follows: (i) Introduced e- THERAPIST, novel gender, age, persona and sentiment-aware polite and interpersonal dialogue system for psychotherapy, fine-tuned in an RL envi- ronment: (ii) Created novel conversational dataset for psychotherapy. PsyCON, and annotated at two distinct levels (a) dialogue-level with gen- der, age, persona, and psychotherapeutic approach (b) utterance-level information, viz. sentiment, po- liteness, and interpersonal behaviour information; (iii) Devised novel reward function incorporating five attribute-specific rewards and two response- quality rewards to generate engaging, fluent, and interactive responses tailored to the user's senti- ment, gender, age, and persona; (iv) Conducted ex- tensive empirical evaluation to test the efficacy of e- THERAPIST in terms of novel metrics, attribute- success and response-quality with respect to the strong baselines.
conduct into the dialogue agents establishes friendly and congenial environment, thereby, pro- viding a better personalized experience to users. The user's sentiment further facilitates the gener- ation of contextually correct polite (Firdaus et al., 2022a) and appropriate interpersonal behavioural responses. For instance, in Figure 1, for the "Young Female Openness" " sample (fourth utterance), the user expresses negative sentiment, thus agent's s behaviour in the form of imposing confronta- tion might make the user more angry or dissatisfied, Hence, should adapt helping, understanding or empathetic behaviour. Besides, a therapeutic dia- logue system needs to utilize various psychother- apeutic approaches (Smith, 1982; Snyder, 1945; Thorne, 1948; Howard et al., 1986) during the on- going conversation based on user's personal profile (gender, age and persona) to produce effective out- comes during psychotherapy as depicted in Figurel. 1, Driven by these considerations, in this work, we propose e- THERAPIST, novel gender, age, per- sona and sentiment-and aware polite and interpersonal dialogue system for psychotherapy.
2 Related Work
The issue of mental health disorders, which is a significant concern for public health (Jacob, 2012), has been the focus of previous research, including
To develop e- e-THERAPIST, we exploit the

---

computational studies. While depression has re- ceived the most attention, other mental illnesses like anxiety, schizophrenia, post-traumatic stress disorder, suicide risk, and self-harm have also been examined (Uban et al., 2021). A few stud- ies have examined the posts and blogs of users on social sites to detect depression (Yates et al., 2017; Tadesse et al., 2019), suicidal thoughts (Zirikly al., 2019; Cao et al., 2019), and other men- tal health issues (Xu al., 2020) using natural language processing (NLP) techniques. Some re- scarchers have also worked on developing "ther- apybots" and creating dialogue agents to provide therapeutic support (Fitzpatrick et al., 2017).
sentiment information, we propose a polite and interpersonal dialogue system for psychotherapy that generates responses in accordance to user's age, gender, persona and sentiment. To the best of our knowledge, ours is the first attempt that exploits politeness and interpersonal relationship to generate precise responses in dialogue systems for psychotherapy. Furthermore, our research pio- neers the exploration of how politeness and inter- personal relationships differ across individuals of varying gender, age groups, and personas within psychotherapeutic conversations.
Dataset
In recent times, generating empathetic responses in psychotherapeutic conversations has grown in popularity (Morris et al., 2018; Sharma et al., 2020). In order to help mental health supporters, (Sharma al., 2021) investigated empathy rewriting text generation task. The authors in (Saha al.. 2022) focused on generating sentiment-driven mo- tivational responses in mental health support dia- logues. A few studies have explored the role of politeness in improving the sense of empathy and compassion during conversation (Lucas et al., 2014; Kim et al., 2018). 8). The agent's courteous attitude conveys sense of concern and emotional involve- ment like human companion. Further, compre- hending and demonstrating proper interpersonal behaviour has proved its effectiveness in psychol- ogy to study interpersonal processes, personality traits, and relationship functioning (Kiesler and Auerbach, 2003; Pincus and Gurtman, 2006; Locke et al., 2017). Studies have suggested that good quality interpersonal relationships and behaviour are important for peoples' social functioning and mental health (Cremers al., 2021). Lately, the au- thors in (Firdaus et al., 2022b) have demonstrated that inculcating politeness in the agent based on the user's s personal profile, such as gender and age group makes the dialogue agent capable of iden- tifying subtle language changes while conversing with different users. Likewise, taking the users' persona into account while generating responses will further enhance the personalization quotient in the dialogue systems (Firdaus al., 2020; Nargund al., 2022; Ahmad et al., 2023; Zhao et al., 2023).
To develop e-THERAPIST, we create PSYCON, a novel high-quality conversational dataset for psy- chotherapy. We focus conversations considering the gender, age and persona of the user with the ul- timate purpose of enhancing mental health support in a personalized way and improving the overall outlook of people facing psychological issues.
3.1 Dataset Creation
PSYCON comprises interactions between the ther- apist and the user suffering from one of the seven most common psychological issues, viz. depres- sion, anxiety, stress, bipolar disorder, disruptive behaviour and dissocial disorders, post-traumatic stress disorder (PTSD), and schizophrenia (WHO, 2022). To minimize the requirement of expensive and scarce human resources, we create the dataset by utilizing knowledge present in LLMs like GPT-J model (Wang and Komatsuzaki, 2021). We create the dataset by prompting this GPT-J model fol- lowed by manual intervention to ensure quality control. The dataset creation process involves two steps: (i). Therapist-user dialogue creation, and (ii). Data cleaning and quality control.
3.1.1 Therapist- Dialogue Creation
We create therapist-user dialogues utilizing the fol- lowing steps:
Attaining the Seed Utterance. We require seed user utterances accompanied by a specific gender, age and persona followed by the seed utterance of the therapist adhering to designated psychother- apeutic approach begin the few-shot dialogue generation using GPT-J model. We refer sev- eral authentic websites, such as the World Health Organization (WHO, 2022), the National Mental Health Foundation (MHF, 2023), and the National
Inspired by the significance of politeness and interpersonal relationship in psychological support, together with the subtle change in these aspects with users' profiles (gender, age and persona) and

---

Alliance on Mental Illness (NAMI, 2023) under- stand the specific characteristics of different psy- chological issues. We also utilize the real user interactions from threads posted on different men- tal health-focused subreddits (e.g. r/depression) to gather the real experiences of the users facing psychological issues.
ing conversation's context, the contextual similar- ity is computed between them using BERT Fl (BSF1) (Zhang et al., 2019). The candidate with max (BSF1) is selected as the final response. These generated dialogues are then manually rated by the same six human experts for quality on a Likert scale of 1-low, 2-moderate, and 3-high. An inter-evaluator Kappa (McHugh, 2012) agreement ratio of 76.3% is observed among the experts. The prompt generating the maximum number of dia- logues with score of 3 is selected as the final prompt. A prompt example is shown in Section A.I of the Appendix. The remaining dialogues generated using selected prompt with a score of either 1 or 2 are manually corrected by the experts to ensure quality dialogues. In the second stage, the selected prompt along with the generated seed dialogues is given as input the GPT-J model, which generates the dialogues with n number of utterances an incremental way. The dialogues with utterances having BSF1 < 0.4 are filtered out for manual cross-verification by human experts who are requested to correct the dialogues as per the guidelines.
Further, to enhance user interaction and satis- faction, it is important for the therapist to respond politely and display uppropriate interpersonal be- haviour. Thus, we utilize the information gathered from the mentioned sources to create the seed utter- ances considering the variation in politeness and in- terpersonal behaviour across different genders, age groups and personas while complying with a par- ticular psychotherapeutic approach. This is done with the help of six human experts having post- graduate qualifications in English Linguistics and proficiency in politeness concepts, interpersonal be- haviour theory and psychotherapeuuti approaches under the guidance of leading psychotherapist from a government-run institution. The human ex- perts were given proper instructions for designing the seed utterances: (a) create a sequence of seed utterances for each possible combination of gen- der, age. persona and psychotherapeutic approach for particular psychological problem; (b) attempt seek information about the problem and make advances towards solution following designated psychotherapeutic approach according to the user profile; (c) formulate responses concerning the vari- ation of politeness and interpersonal hehaviour quo- tient in the responses based on the user profile; (d) ensure diversity of the seed utterances to increase user engagement and facilitate better communica- tion; (e) frame the responses displaying positive- ness and affirmativeness aiming to boost the user's morale. A few examples of seed utterances are given in Table in Section A. of the Appendix.
3.1.2 Data Cleaning and Quality Control After obtaining the entire conversational dataset, each dialogue is assessed in terms of humanness, profile consistency, and psychotherapeutic ap- proach consistency by the same set of experts. A score of 1-low, 2-moderate. or 3-high is assigned to each utterance. After obtaining the ratings, we observe an agreement ratio (McHugh, 2012) of 82.7%, R 85.3% and 89.6% for humanness, user pro- file consistency and psychotherapeutic approach consistency, respectively among these experts. All dialogues having utterances with a score of 0 for any of these three aspects are discarded from the dataset. The final dataset statisties given in Table 1.
Dialogue Generation. Once the seed utterances are finalized, the GPT-J model utilized to gener-
Metrics
Train
Validation
Test
# Diologues
816
102
102
of Unerances
19,568
2,692
2.111
ate new utterances. The overall dialogue genera- tion completes in two stages. In the first stage, we manually design three prompts, each consisting of instruction and designed seed utterance. For each of the three prompts, 40 dialogues created by feeding them to GPT-J. During generation, top-k sampling (Fan et al., 2019; Radford al., 2019) is employed, resulting in the generation of three candidate responses for each input utterance. To check alignment of the candidates with the ongo-
Arg. Ulterances per Dialogue
23.98
26.39
27.56
Table I: Dataset statistics of PsyCON.
3.2 Dataset Annotation
To construct e-THERAPIST, we obtain the anno- tation for PsyCON dataset at two distinct levels- (i). dialogue-level annotation aspects, viz. user pro- file information- gender: male (m) and female (f).

---

age: young(y), adult(a), and elderly(e), persona: openness (0), conscientiousness (C), extraversion (E), agreeableness (A) and neuroticism (N), and psychotherapeutie approach: directive (d), non- directive (nd) and eclectic (ec)²; (ii). utterance- level annotation aspects, viz. sentiment, politeness, and interpersonal behaviour. The annotation pro- cess involves the collaboration of three annotators, consisting of two Ph.D. holders in Linguistics and one individual with a Master's degree". All three annotators possess excellent proficiency in English, substantial experience in labeling tasks, and a thor- ough understanding of sentiment, politeness, and interpersonal behavioural aspects.
and Pa A,,,, respectively. Further, for each user utterance, u, TC2, corresponding sentiment s; = {negative, neulral, positive) is predicted by sentiment classifier SC. The concate- nated u, with 8; can be given as ug = [us + si]- Now, the Img is trained to predict output ~ t; given input x; [uā + + +4 4 +] [gs + are + Pi] in supervised learning setting. It can be given as:
1=T
(mo(
II
I
p(un a.)
k= 6-0
We call the trained lmmg as supervised learning fine-tuned language model (SLLM). Next, to end up with proposed e- e-THERAPIST, we fine-tune SLLM using our novel reward model in RL setting to generate user's profile and sentiment-aware po- lite responses with correct interpersonal behaviour in therapeutic conversations. The architecture of the proposed system is shown in Figure 2.
3.2.1 Sentiment, Politeness, and Interpersonal Behaviour Annotation
The user's and therapist's utterances in PsyCON are annotated with one of the ternary sentiment labels, viz. positive, negative and neutral, and po- liteness labels viz. polite, moderately_ pplite and impolite, respectively.
4.1 Reward Model
For given input zú, we generate set of n-W pos- sible candidates using bmg and score them using reward model. Our reward model consists of seven distinct rewards to adapt correct polite inter- personal behaviour in generated responses of Img- First reward R₁ focuses on the awareness of the user's gender and age. Secondly, Ra emphasizes the awareness of the user's persona. R& drives the model to follow the correct psychotherapeutic ap- proach. R4, ensures appropriate politeness per the user's sentiment. R₂ aims to adapt the inter- personal behaviour in accordance with the user's sentiment. Rg focuses on maintaining the plausibil- ity of the context and individual utterances. Lastly, R7 steers the generated responses to be fluent and diverse.
The different interpersonal behaviour labels fol- lowing the wooddimensional IPC model (Cremers in our work are: directing (Dg), helpful (HI), understanding (Ug), complaint (Ct), imposing (Ig), confrontational (CI), dissatis- fied (Dd) and uncertain (Un). This interpersonal behaviour annotation list has been extended to in- corporate one more label, namely empathetic (Em). considering the significance of empathy in therapy (Sharma et al., 2020, 2021; Saha et al., 2022). Due to space constraints, the description of all the nine interpersonal behaviour labels and the dataset an- notation procedures are provided in Section A.1 of the Appendix.
al., 2021) that
4 e- e-THERAPIST
All seven rewards can be categorized into two types of rewards, viz. Attribuie specifii Rewards (R1, R2, Ra, R4, Rş) reinforce different at- tributes of the user or therapist in generated sponses, and Response-qualits Rewards (Rg, R7) to enforce contextually fluent and adequate re- sponses in a conversation.
We warm start with a pre-trained language model GPT-2 medium (Radford al.. 2019) fine-tuned on PSYCON denoted by TCN. Each conver- sation in TCN can be represented by TC₁ {to: (o. -., 14, U4, ty ty 1 tú and u, give the therapist's 's and user's ,th utterance in the conver- sation, respectively, where 0 <= h < N for N number of conversations. For each TCp, the user's corresponding gender, age, and persona can be represented by Ok (m, f), ag {y,a,e a, é)
4.1.1 Attribute-specific Rewards
To design each of the R1, R2, R3. Ra and Rs, five different RoBERTa-large (Liu et al., 2019) based classifiers are fine-tuned. For R1 Gender-Age Classifier (GAC) pre- diets one of the six classes of gender and age GAC(tt)) = (fv, fe, my, ma, me),
"The dialogue-level information is obtained during Data Cleaning and Quality Control stage described in Section 3.1.2. "The annotators are compensated according to institutional guidelincs.

---

Step Pretraining language model (SLLM)
Step Reward model training
Step 3 Fine Tuning Reinforcenen Learning
fasls like have no molivation to do anything. I'm always tired and sad. can't concentrate on my
Lunsieraland how dilliall it experiencee lack of molivation and conclant leelings lineinsss and sodhees overshelming.
RL polccy per curnent policy canditatee resjorses are generated and quality
studies, and I've lost things enjoy
As per feadbacks of RM policy is optimized using NLPO kes
challanging feeling studies
way. you
(operness)
Yourg Femole
Sentiment Classifier
Losing intereet in acthities enjoy symesoo
emptinoss singgre valio feel it Let's work together ways bring that
inuly aeery that wxpur roiigg
depression.
(negative)
Gender-Age
RM
Awazenoss
SLLM
Ranking generated responsex employing
R2 Persona awareness roward Counseling roward
Politeness RS Interpersonal bahavior zeward Adequacy rluent and
RM
GPT-2 is fine-tuned using supervised learning
Quality-check the generated responses seven aspects and outputs score
loss Divecuity zeward
Figure Overall architecture of proposed system
where fy, fe, my, ma. me denote female- young, female-adult, female-elder, male- young, malc-adult, and male-elder, respec- tively. For R2, Persona classifier (PC) predicts one of the five personality traits PC((t) = {0, C, E, A, N). For Rg, a psychother- apeutic approach classifier CTC((t, + uil)
To formulate R4, and Rş, we penalize the re- sponses that do not adapt true politeness and in-
terpersonal behaviour per the sentiment of the user.
RA PoC(4, SC(w)) - PoC( +C(ss)) (5)
H5 HCC(t, + SC(u)) x HCC SC(()) (6)
(d,nd, (d, ec}. For R4, a sentiment classifier SC(u,) = {positive, neutral, negative) predicts the sentiment of the user's utter- ance and politeness classifier PoC(4) {impolite, moderately_politte polite) predicts the politeness of the therapist's utterance. Lastly, for Rs, an interper- sonal behaviour classifier IBC(4) [Dg, HI, Ug. Ct,, Ig, Cl, Dd, Un, Fm] pre- dicts one of the nine interpersonal behaviour labels To design each reward, we track the true class probabilities score from each of the classifiers. Ri and Ra penalize those responses which deviates from user profiles viz. gender, age and persona and are computed as:
acts as penalization factor in all of the rewards
R1, R2, R3. R4, and Rg.. The final attribute- specific reward can be written as: RA = 31 Ry R2 + RgR3 + R4 + R5
4.1.2 Response-quality Rewards
In Rg, the candidates deviating from context and user's utterance are penalized. computed using BERTScore-F1 (BSF1) (Zhang et al., 2020) be- tween (i) the true context input x; and generated (ii) user's utterance i, and generated yi- To avoid rewarding high similarities steering, threshold of is taken into account.
Rs = (in((BSS BSp (m..))))
RI = GAC((t) a GAC(E)
(2)
2
n2 PC(L))
PC(y:)
(3)
R7 ensures fluency and non-repctitiveness in the generated responses and is computed as the sum of the perplexity reciprocal and BSF1 between generated and 9:-1-
R3 penalize the responses deviating from the cor- rect psychotherapeuttic approach in the ongoing dialogue:
R3 CTC(I. wl) CTC(M v.))
(4)
Br
PPL BSF
(8)
'The accuracies of GAC, PC, SC, PoC, CT CC and / HHC are 89.4%, 89.7%, 91.2%, 95.6%, 92.3%, and 89.8% respectively.
"The value of a taken as greater than or equal
9 + 8g + 8g + 84 + 8s =

---

The response-quality reward can be written as: RR- = =1 R. + R2 Lastly, by combining RA and RR. we obtain our final normalized reward function as follows:
Evaluation Metries. To evaluate the perfor- mance of the proposed system, e-THERAPIST. both automatic and human evaluation are con- ducted. All the five classifiers, viz. GAC, PC, CTC, PoC, and IBC are evaluated in terms of Weighted Accuracy (W-ACC) and Macro-F1, In automatic evaluation, e- e-THERAPIS is evaluated W.I.t (i). attribute-success Gender-Age consis- tency (GAc), Persona consistency (P.), Psychother- apeutic approach correctness (CT.), Politeness cor- rectness (Poz) and Interpersonal behaviour correct- ness (IB,),, and (ii). language-quality Perplexity (PPL), BERTScore-FI (BSri), Response-length (R_LEN).
A 61 Ra + 6RR
7
The final score R is utilized in an RL policy loss NLPO (Ramamurthy et al., 2022) to provide feed- back, compelling the agent to generate high-quality candidates that align with the preferred outcomes. During RL-fine tuning, RL-policy is initialized with Te = lmg. NLPO basically implements masking policy Tgg copy of the current policy Th updated after m steps. Top-p tokens in vo- cabulary V contribute as valid-mask remaining are subjected to an invalid mask, i.e. selection prob- ability is set to zero for these tokens. Hence, this strikes a balance between incorporating more task- relevant information compared to the KL penalty derived from Ty. Due to space restrictions, details of NLPO are given in Section A.2 of the Appendix.
For human evaluation, we ask the same six ex- perts to evaluate the e- e-THERAPIST, Initially, each evaluator engages with the system 5 times, with different set of responses each time. These human-evaluated interactions are then cross- verified by psychotherapists from a government- run institution to ensure evaluation quality. Upon passing verification, the additional 90 interactions (15 per evaluator) are evaluated, resulting in total of 120 human-evaluate dialogues. Human evalua- tion metrics also includes GAc,, Pa. CTe, Po IB. as attribute-succen and Fluency (F), Consistency (C), and Non- Ron-Repetitiveness (NR).. All dialogue interactions are evaluated on an integer Likert scale of 1-59 10
5 Experiments
Baselines. We compare the performance of pro- posed THEERAPPIST with seven strong baselines LM (Radford et al., 2019), ARDM (Wu et al., 2021b): LMs trained alternatively for both user and therapist, GPT-Critic (Jang et al., 2022): im- proving LM through cloning of critic- Pritic-guided self- generated sentences during fine-tuning, SLLM: Fine-tuned LM in supervised setting with user profiles and sentiment, SLLM+PPO: Supervised PPO loss based fine-tuning, e- T-HERRPIST-:: e-THERAPIST with î = 0, e-THERAPIST-ASR: e- TTHERAPIST with R.A - 0 and T-ERAAPP RQR: e- T-THHRPPIST with R₂=00. 0.
6 Results and Analysis
Automatic Evaluation. Table 2 depicts that the proposed e-THERAPIST achieves better results w.r.t. all the eight baselines, viz. LM, ARDM, GPT-Critic, SLLM, SLLM+PPO, e- T-THERPISST- R. e- THERAPIST-ASR and e-TERRPISS-RQR in terms of all the seven metrics, viz. GAe. Pa, CT Poc. IBe, PPL, BSFI, and RLEN- Better results of SLLM compared to LM, ARDM, and RL- based GPT-critic highlight the importance of user's gender, age and persona profiles. Incorporation of these attributes steer the SLLM towards more interactive responses inherently. e-THERAPIST-R performs comparable to SLLMM as in the absence of any reward, RL policy acts Ty R Img.
Implementation. We experiment with different values of n [,,3, 3, 1, 5, 8). and found that = 3 yields the best performance. We use GPT-2 as the language model (LM) and context window size of 4. We employ top-k sampling with 20 as the decoding method for all the models in our work. For RL-based models, we determine the end values of coefficients 31 = 0.1, B2 = 0.2, B3- = 0.2, 84 = 0.2, 3:- = 0.3, 71 0.5 72 = 0.5, 51 = 0.75, and 82- 0.25 empirically. Due to space restrictions, we have included detailed implementations and weight optimization in sections A.3 and A.5 of the Appendix, respectively.
It can also be observed that in the absence of only attribute-spece
rewards, the performance
*GAc. Pc. CTc, Po., IB. computed by five respective classifiers with accuracies.
"1-5 denotes low to high. An inter-evaluator agrecment ratio of 72.3% is observed in evaluations.
Tn + 72- =1

---

Model
G..
Pc
CTC
1Hc
PPL
4Sp1
REEN
LM (Radford al., 2019)
78.4%
72.1%
79.5%
80.2%
73.6%
4.26
0.68
15.61
ARDM (Wu al., 2021b)
80.4%
73.3%
80.0%
81.5%
74.2%
3.57
0.74
16.82
GPT-Critic (Jang al., 2022)
80.7%
73.8%
80.6%
82.7%
73.1%
3.86
0.69
15.94
SLLMM
85.4%
80.1%
86.3%
84.6%
77.8%
3.26
0.81
19.79
e- THHERAPIST-R
85.1%
79.7%
86.8%
84.3%
77.5%
3.09
0.84
19.26
e- -THERAPIST-SEE
86.1%
80.8%
87.2%
86.2%
79.8%
3.06
0.87
20.12
--THERAPIST-RQR
87.5%
82.3%
88.7%
87.9%
80.5%
2.97
0.88
22.79
SLLM+PPO
89%
83.9%
91.5%
91.3%
82.3%
2.67
0.89
23.01
<-HHERAPIST
90.1%
84.1 %
92.6%
92.5%
83.4%
2.52
0.89
23.89
Table Automatic evaluation results
Model
GA2
P.
CT.
Po.
18:
F
NR
LM (Radford et al., 2019)
2.21
2.10
2.17
2.39
ARDM (Wu et al., 2021b)
2.74
2.77
2.81
2.79
2.83
2.29
GPT-Critic (Jang et al., 2022)
2.98
2.83
2.81
2.90
2.91
2.86
2.91
2.34
SLLM
3.67
3.80
3.75
3.41
3.89
3,44
3.21
c- THERAPIST-R
3.53
3.45
3.86
3.84
3.50
4.11
4.05
3.72
c- THERAPIST-ASH
3.75
3.70
4.01
3.91
3.72
4.32
4.27
3.82
c- THERAPIST-RQR
3.97
3.91
4.12
4.09
3.89
4.45
4.33
3,97
SLLM+PPO
4.10
4.06
4.38
4.30
4.01
4.55
4.50
4.05
e- TTEERAPIST
4.21
4.10
4.42
4.35
4.02
4.62
4.60
4.08
Table Human evaluation results.
THERAPIST-ASE sees a significant increase in scores of PPL. BSF1, and RLEN. but mini- mal change is attribute spriiutt-specificc metrics. This sup- ports the use of Response-quality rewards, due to which model tries engage the user with longer and contextually-adequaal responses. Similarly, an increase in specifcc metrics in case --THERAPIST-RQE supports the requirement attribute-specific rewards as well. It can also be inferred from both e-THERAPIST-AS. and e- THERAPIST-RQR results that the presence of each type of rewards affects other metrics positively but with minimal margin. This means that all rewards interact with each other and helps in achieving user- aware polite interpersonal therapy. SLLM+PPO is same as e- THERAPIST only with the difference loss. the performance of both the models, there little margin, but still, reflects that NLPO drives the model to generate longer responses with correct incorporation of politeness, interpersonal behaviour and psychotherapeutic approach. Au- tomatic evaluations support our hypothesis that knowledge of user's profile and reward modelling can play a significant role building a better psy- chotherapy dialogue system.
lines. Here, as well, SLLM and e-THERAPIST- R beat LM, ARDM, and GPT-Critic. Further, e- THERAPIST-ASR shows a small marginal increase on SLLM and e-THERAPIST-R. This implies that papponse-quliite rewards not enough to ensure user's profile and sentiment-aware therapeutic re- sponses. Notably, difference between performance of THERAPIST-RQR and SLLM+PPO of small margin. Similarly, minimal performance dif- ference can be seen between SLLM+PPO and e- THERAPIST as well. This leads to the argument that attribute-ppecific rewards play crucial role in adapting correct psychotherapeutie approach and interpersonal behaviour in generated responses while imbuing politeness in them. should also be pointed out that e-THERAPIST achieves much better scores w.r.t. all metries as compared to THERRAPITTRRR. This highlights the importance of response-quality rewards as well. Thus, it can be concluded that a robust psychotherapeutic di- alogue system can be built by striking balance between antribute-speciffe and response-quality wards. Employing both, the user be engaged in interpersonal therapeutic communication while also maintaining a cordial environment.
Human Evaluation. Results of human evaluation for e-THERAPIST are obtained in sync with the automatic evaluation metrics shown in Table For all the metrics, viz. GAg, Po. CT.,, Po.- IB F, C, and NR. e-THERAPIST achieves better scores as compared to all the eight base-
Conclusion
This work introduces e-THERAPIST. novel po- lite interpersonal dialogue system for psychother- apy. First, considering user profile information (gender, age, persona) and the therapist's psy-

---

chotherapeutic approach, new conversational dataset PSYCON is created by prompting GPT-J with manual interventions. Further, it annotated with user's sentiment and therapist politeness label and interpersonal behaviour. Then, psychother- apeutic dialogue system e- T-TERRPIST is devel- oped in an RL framework. To ensure the prefer- ences of gender, age, persona and sentiment-awar polite interpersonal psychotherapeutic responses, a novel reward function is designed consisting of seven rewards. Our results concludes the require- ment each of the reward to ensure polite and interpersonal psychotherapeutic responses tailored to user's profile and attributes, and eventually con- tributing to improved therapy experiences. In fu- ture, we would like to incorporate external knowl- edge facilitate more factual conversations.
Ethics Statement
Developing a psychotherapeutic dialogue system do take a step towards Responsible AI. But, same methodology can also be used to develop toxic or irresponsible natural language generation mod- els. Hence, we acknowledge that the significant importance of addressing ethical concerns when developing computational models for psychothera- peutic applications is needed. Given the sensitive nature of the subject, we prioritize safeguarding the privacy of users' personal data. It is important to clarify that this paper does not provide any clinical diagnostic assertions. We mainly focus to enrich the interactiveness of such dialogue system with the user better engagingness and therapy dialogee.. The dataset will be made available for research purposes with proper permissions.
Acknowledgements
Limitations
Kshtij Mishra acknowledges the support received from the Indian government's "Prime Minister's Research Fellowship (PMRF) Program". Priyan- shu Priya acknowledges the financial support pro- vided by the Department of Science and Technol- ogy, Ministry of Science and Technology, Govern- ment of India, through the Innovation in Science Pursuit for Inspired Research (INSPIRE) Fellow- ship.
e-THERAPIST comes with some limitations. Firstly, the training process requires substantial GPU memory capacity i.e. of 40 GB. Another challenge arises from the optimization of reward weights, which can significantly extend the train- ing and validation time. To address this, heuristic approaches are used to select specific combinations of reward weights. It is also observed, in the case of continuous, short, direct responses (e.g., 'Yes', II don't know', 'No', '2', 'Yeah') ') model initially at- tempts to provide therapy by inquiring about the user's issue; however, after a few turns, the model may generate repetitive or inconsistent responses. This is because the training data primarily con- sists of interactive dialogues with longer utterances, leading to confusion in handling concise inputs. It is also observed that continuous cat-of-the-contexx responses may drive the model towards generation of inadequate responses, as these large language models models inherently have knowledge of vast information in their memories.
References
Zishan Ahmad, Kshitij Mishra, Asif Ekbal, and Pushpak Bhattacharyya. 2023. Rptes: A reinforced persona- aware topic-guiding conversational system. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 3464-3476.
Tri Budiarta, Joko Nurkamto, Sumarlam Sumarlam, and Dwi Purnanto. 2021. Expressive speech acts of politeness in the counselling process. Langkawi: Journal of The Association for Arabic and English. 7(2):212-22
Lei Cao, Huijun Zhang, Ling Feng, Zihan Wei, Xin Wang, Ningyun Li. and Xiaohao He. 2019. La- tent suicide risk detection microblog via suicide- oriented word embeddings and layered attention. arXiv preprint
To overcome these limitations, future research could focus on refining and enhancing psychother- apeutic dialogue systems as well as language mod- els. This could involve addressing memory require- ments, reducing training time, improving the han- dling of short, direct and out-of-context responses, and ensuring the generation of relevant inquiries to enhance user satisfaction.
Inigo Casanueva, Pawet Budzianowski, Pei-Hao Su, Ste- fan Ultes, Lina Rojas-Barahon Bo-Hsiang Tseng, and Milica Gašié. 2018. Feudal reinforcement learn- ing for dialogue management in large domains. arXiv preprint arXiv: 1803.0332.

---

Jolien Cremers, Helena JM Pennings, Tim Mainhard, and Irene Klugkist. 2021. Circular modelling of cir- cumplex measurements for interpersonal behavior. Assessment, 28(2):585-600.
Youngsoo Jang. Jongmin Lee, and Kee-Eung Kim. 2022. GPT--critic: Offline reinforcement learning for end-to- end task-oriented dialogue systems. In International Conference on Learning Representations.
Cristian
Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. arXiv preprint arXiv:/306.660 6078.
Donald J Kiesler and Stephen M Auerbach. 2003. In- tegrating measurement control and affiliation studies of physician-patien interaction: the inter- personal circumplex. Social Science & Medicine, 57(9):107-1122.
Junhan Kim, Yoojung Kim, Byungjoon Kim. Sukyung Yun, Minjoon Kim, and Joongseek Lee. 2018. Can a machine tend to teenagers' emotional needs? study with conversational agents. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1-6.
Angela Fam, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. arXiv preprint arXiv:1902.01109
Mauajama Firdaus, Asif Ekbal, and Pushpak Bhat- tacharyya. 2022a. PoliSe: Reinforcing politeness us- ing user sentiment for customer care response gener- ation. In Proceedings of the 29th International Con- ference on Compulational Linguistics, pages 6165- 6175, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Noriko Kitamura. 2000. Adapting brown and levinson's 's 'politeness' theory the analysis of casual conversa- tion. In Proceedings of ALS2k, the 2000 Conference of the Australian Linguistic Society, pages 1-8.
Nathan Lambert and I. von Werra. 2023. Mlstrrtinn reinforcement learning from human feedback (rlhf).
Mauajama Firdaus, Arunav Shandilya, Asif F'kbal, and Pushpak Bhattacharyya. 2022b. Being polite: Mod- eling politeness variation in a personalized dialog agent. IEEE Transactions on Computational Social Systems.
Debi Laplante and Nalini Ambady. 2003. On how things said: Voice tone, voice intensity, verbal content, and perceptions of politeness. Journal of language and social psychology, 22(4):434 441.
Mauajama Firdaus, Naveen Thangavelu, Asif Flbba, and Pushpak Bhattacharyya. 2020. Persona aware re- sponse generation with emotions. In 2020 Interna- tional Joint Conference on Neural Networks (IJCNN), pages 1-8. IEBE.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1692.
Kathleen Kara Fitzpatrick, Alison Darey, and Molly Vierhile. 2017. Delivering cognitive behavior ther- apy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (woebot): a randomized controlled trial. JMIR mental health, 4(2)5
Kenneth D Locke, Liliane Sayegh, J Kim Penberthy. Charlotte Weber, Katherine Haentjens, and Gustavo Turecki. 2017. Interpersonal circumplex profiles of persistent depression: Goals, self- self-efficacy, problems. and effects of group therapy. Journal of Clinical Psychology, 73(6):595-611 611.
Nya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Confer- ence Learning Representations.
Lewis R Goldberg, 1993. The structure of phenotypic personality uaits. American psychologist, 48(1):26.
Michael B Gurtman. 2009. Exploring personality with the interpersonal circumplex. Social and personality psychology compass, 3(4):601-619,
Gale M Lucas, Jonathan Gratch, Aisha King, and Louis- Philippe Morency. 2014. It's only computer: Vir- tual humans increase willingness to disclose. Com- puters in Human Behavior, 37:94-100.
Jacob B Hirsh, Colin G De Xiaowen Xu, and Jordan B Peterson. 2010. Compassionate liberals and polite conservatives: Associations of agreeableness with political ideology and moral values. Personality and Social Psychology Bulletin, 36(5):655-66 6664.
Murni Mahmud. 2013. The roles of social status, age, gender, familiarity, and situation in being polite for bugis society. Asian Social Science, 95):58-77.
Mary McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282
George S Howard, Don W Nance, and Pennie Myers. 1986. Adaptive counseling and therapy: An integra- tive, eclectic model. The Counseling Psychologist, 14(3):363 442.
Mohsen Mesgar, Edwin Simpson, and Iryna Gurevych. 2020. Improving factual consistency between response and persona facts. arXiv preprint arXiv:2005.0003
KS Jacob. 2012. Depression: a major public health problem in need of a melti-sectoral response. The Indian journal of medical research, 136(4):537.
Mental Health Foundation MHE, 2023. Good mental health for all.

---

Robert R Morris, Kareem Kouddous, Rohan Kshir- sagar, and Stephen M Schueller. 2018. Towards an artificially empathic conversational agent for men- tal health applications: system design and user per- ceptions. Journal of medical Internet research, 1014).
Elena Stephan, Nira Liberman, and Yaacov Trope 2010. Politeness and psychological distance: a construal level perspective. Journal of personality and social psychology, 98(2):268.
Michael M Tadesse, Hongfei Lin, Bo Xu, and Liang Yang. 2019. Detection of depression-related posts in reddit social media forum. JEEE Access, 7:44883- 44893.
National Alliance on Mental Illness NAMI. 2023. Men- tal health conditions.
Abhijit Nargund, Sandeep Pandey, and Jina Ham. 2022. Par: Persona aware response in conversational sys- tems. In Proceedings of the 19th International Con-
Frederick C' ThToree. 1948. Principles directive coun- seling and psychotherapy. American Psychologist. 3(5): 160.
ference on Natural Language Processing (ICON). pages 50-54.
Ana Sabina Uban, Berta Chulvi, and Paolo Rosso, 2021. Understanding patterns of anorexia manifestations in social media data with deep learning. Proceed- ings of the Seventh Workshop on Computational Lin- guistics and Clinical Psychology: Improving Access, pages 224-236.
Aaron L Pincus and Michael B Gurtman. 2006. In- terpersonal theory and the interpersonal circumplex: Evolving perspectives on normal and abnormal per- sonality.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188)9
Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model.
World Health Organization WHO. 2022. Mental disor- ders.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu,
Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi. and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion.
World Health Organization WHO, 2023, Depressive disorder (depression).
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Sti- ennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021a. Recursively summarizing books with human feedback. arXiv preprint arXiv: 109.02..
Tulika Saha, Saichethan Reddy, Anindya Das, Sriparna Saha, and Pushpak Bhattacharyya. 2022. A shoulder to cry on: Towards a motivational virtual assistant for assuaging mental agony. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2436-2446 -239.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2021b. Alternating recurrent dialog model with large-scale pre-trained language models. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Compulational Linguistics: Main Volume, pages 1292-1301.
John Schulman, Philipp Moritz, Sergey Levine. Michael Jordan, and Pieter Abbeel. 2015. High-dimense continuous control using generalized advantage esti- mation. arXiv preprint arXiv: 1506:::5500.025 506.02438.
Ruziyeva Nilufar Xafizovna and Xolova Madina Bobo- qulovna. 2022. Politeness in literary works: An overview, Eurasian Research Bulletin, 7:200-206
Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. 2021. Towards facilitating empathic conversations in online mental health sup- port: A reinforcement learning approach. In Proceed- ings of the Web Conference 202/, pages 205-205.
Zhentao Xu, Verónica Pérez-Rosas, and Rada Mihalcea. 2020. Inferring social media users' mental health status from multimodal information. In Proceedings of the 12th language resources and evaluation con- ference, pages 6292-6299.
Ashish Sharma, Adam Miner, David Atkins, and Tim Al- thoff. 2020. A computational approach to understand- ing empathy expressed in text- mental health support. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 5263-5276.
Andrew Yates, Arman Cohan, and Nazli Goharian. 2017. Depression and self-harm risk assessment in online forums. arXiv preprint arXiv: 1709.01848
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- vating text generation with bert. arXiv preprint wrXiv:1904.09675.
Darrell Smith. 1982. Trends in counseling and psy- chotherapy. American Psychologist, 870))
William U Snyder. 1945, An investigation of the na- ture of non-directive psychotherapy. The Journal of General Psychology, 3332)3193-223.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert.

---

Sirui Zhao, Hongyu Jiang, Hanqing Tao, Rui Zha, Kun Zhang. Tong Xu, and Enhong Chen. 2023. Pedm: multi-task learning model for persona-aware emoji- embedded dialogue generation. ACM Transactions on Multimedia Computing, Communications and Ap- plications, 19(3s)11-11.
A
Appendix
A.1
Dataset
We provide the details of the dataset in this section.
A.1.1 Prompt Design
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint (oX::1909.08893.
begin
psychological Issue: depression
Ayah Zirikly, Philip Resnik, Ozlem Uzuner, and Kristy Hollingshead. 2019. Clpsych 2019 shared task: Pre- dicting the degree of suicide risk in reddit posts. NAACL HLT 2019, page
gender: female
age: young
persona: openness
psychotherapeutic approach: directive Technique
dialogue flow: problem description, self-care, coping strategies
topic:
"problem description"
User: Hi, I've been struggling with depression lately, and I'm seeking counseling help cope with it.
topic: "problem description selff-care
Agent: I'm here to assist It's great that you've recognized the need for support. In addition to counseling,
incorporating self-care practices into your daily routine can be beneficial for managing depression. Have you tried self- care activities before?
topic: "self-care"
User: I've dabbled in self-care, but I'm looking for more guidance on effective strategies to improve my mental well- being.
topic: "self-care"
Agent
Figure Example of one-shot version of prompt.
The proposed PsyCoN dataset comprises con- versations focusing on seven psychological issues and different gender, age, personas and psychother- apeutic approach. In order to create synthetic data focusing on these aspects, we prompt the GPT-J model utilizing four-shot prompt. A sample of the one-shot version of the prompt is provided in Figure 3. We provide the psychological issue, gen- der, age, persona, psychotherapeutic approach, di- alogue flow and the seed utterances for which the next utterance needs be generated. Table de- picts few examples seed utterances. The four- shot prompt adheres to a similar pattern with four examples in the input sequence. A few turns of a dialogue generated using this method are given in Table 6.
A.1.2 Dataset Annotation Details
The entire annotation process for sentiment, polite- ness and interpersonal behaviour labels proceeds

---

two stages to reduce manual efforts. First, we ran- domly sample 340 dialogues from the datasct and then ask all three annotators to manually annotate the user's utterances with the required sentiment label and the therapist's ss utterance with the relevant politeness and interpersonal behaviour labels. Sec- ond, three pre-trained RoBERTa-large (Liu et al., 2019) models are fine-tuned on manually annotated samples to build sentiment, politeness, and inter- personal behaviour classifiers. Then, the remaining utterances are passed through the respective clas- sifier to predict the corresponding label. Lastly, the same annotators are asked to cross-verify the predicted labels and correct them, if needed. A reli- able multi-rater Kappa (McHugh, 2012) agreement ratios of 85.2%, 77.1 % and 73.6% is observed in the first stage and 89.6%, 84.6% and 80.3% is ob- served in the second stage for sentiment, politeness and interpersonal behaviour labels, respectively.
of sequence-level rewards in the environment, we apply Then, regularization (Wu al., 2021a) is applied address the sparsity of sequence-leve re- wards in the environment. basically incorporates a token-level KL. penalty into the reward function to discourage the model from deviating too far from the last model Tg. It can be formalized as:
R((s, ai, v) = R((s, a.,y) v) ( (as (as/s:) (15)
Here, Å represents the regularized KL. re-
ward, y denotes the ground-truth predictions, K L(To(aa))) (ar [TTo(aa)s))))bnaa (t)](/mo(aa) is the KL divergence between (To(ae)ss) and (mg(ay)ss (a)]s:)). The KL coef- ficient 5 is dynamically adjusted following (Ziegler et al.. 2019). Now, we update the RL policy me by maximizing the PPO-Clip objective:
1
Tom+1 = argmax
| IT
A sample conversation with the annotation for sentiment, politeness and interpersonal behaviour is shown in Figure 4. The description of different interpersonal behaviour labels along with examples are provided in Table 5.
T
Σ TEDm t=0
min (Tr(0).ATOm. clip(r(()))
1- 1 + +el++ e) (TT)
xa(ar(es)
where ra(0) updated as follows:
Next, value function is
Tme (ax (sc)
A.2 NLPO
The RL--based parameterized control policy for e- THERAPIST can be defined as:
Von.. argming
Vo(s.) Rt:
Dm
: S - A(4)
(10)
(16)
For every m iterations, the parameterized masked policy is also updated as follows:
Here, Ty is a probability function that attempts to
select an action A in given state space S with goal to maximize long-term discounted rewards R over trajectory:
T p+1
(+1s Tom+s)
(17)
A.3 Implementation Details
E. PL n a())
(11)
All classifiers are fine-tuned using the RoBERTa- large (Liu et al., 2019). The language models GPT-2-medium (Radford et al., 2019), ARDM (Wu et al., 2021b), and SLLM are trained with cross-entropy loss. supervised learning setting, AdamW aptimizer(Loshchiingg and Hutter, 2018) is empoloyed with a learning rate of a = 2e 05 and
During RL fine-tuning, we initialize xo Img The value function /" and Q-value function Q" for policy Ne and estimated reward R are be computed as:
V Eas π ERR(er, (o.ter...,))) a.
(12)
seed_ calue = 10.
(s., = R(si, ar, u) T ( (1444) (13)
For SLLM+PPO and e-THERAPIST, train- ing is conducted with batch_siae = 8, 10, human_reward max www__candddaleeeeeegee 50, clip_ratic 0.2. discount_factor 0.95, number_ camber_of_stepp sseess 32000,
Considering V and Q-value function QT, we com- pute the advantage estimate T as:
10.
as, = (x, V
(14)
To stabilize the training, we approximate the advantage using Generalized Advantage Estima- tion(Schulman et al., 2015). To address the sparsity
= 640 and Adam W op-
timizer (Loshchilov and Hutter, 2018) with a learn- ing rate of a = 2e -05 E = 0.2 and epochs = 20.

---

Seed Utterances
Psychological Issue: PTSD
User: been struggling with PTSD lately. boen really tough cope with.
Gender: Male
Therapist: As llbeapist understand that PTSD can have signilicant imppcc your life. I'm here to help you navigate through it.
Age: Young
Persona: Openness
User: I'm a young male dealing with PTSD, and I'm looking for effective techniques to manage it.
Psychotherapeutic Approach: Eclectic
Can you suggest any approaches?
Psychological Issue: Depression
think I might he experiencing depression. Therapir: understand how challenging depression can be.
User: I've heen focling really down lutely. I
Gender: Fermale
Age: Adult
to provide support guidance.
Persona: Conscientiousees
User: I'm an adult female struggling with depression, and I've heard about the directive technique. Can you tell more about and could help me?
Psychotherapeutic Approach: Directive
Psychological Issuc: Stress
User: I think lot my stress comes from feeling overwhehmed by all my responsibilttess Thempttt: understand. Steess affect us various ways.
Gender: Female
Age: Elder
Can tell me more about what's been going on?
Persona: Extraversion
the extravented side, heing isolated during the pandemic has been particularly challenging for me. I'm used to being active and engaged with others.
User: I've always been
Psychotherapeutic Approach: Directive
User: It's been going for several months now. The intensity varies, but there are momeents when I fool completely overwhelmed by these memories.
Psychological Issue: PTSD
Gender: Male
Therapist: can undersiand how distressing and disruptive these symptomn can be, Have you sought any professional help or support for your PTSD symptoms?
Age: Adult
Persona: Agrecableness
Psychotheraeuttic Approach: Non- Non-directive
User: No, I haven't. L I've been besilant to reach out because I've always tried to handle things on my own.
Psychological Issue: Anxiety
User: Lately, I've been feeling overwhelmed by anxiety.
Gender: Male
Thempiir:: Thank you for sharing that with me. Anxiety be quite overwhelming. Can you describe what you'te re specifically experiencing?
Age: Young
Persona: Neuroticism
Psychothenaeutiic Approach: Non-directive
User: always been someone who tends to more neurotic, so anxiety really overwhelming.
Table 4: A few examples seed utterances.
Interpersonal behaviour
Definition
Example
Directing
Provides cleur instructions, assistance and guidance tn the users.
Okay, well let's start with something simple. Mayhe you can try going for walk outside every day.
Helpful
Active listening. providing reassurance and cmotional support to the users based on their circumstances order to motivate them.
One thing you can try is practising self-compassion to fight depression. That means treating yourself with she same kind- nest and understanding that you would offer to friend.
Empathetic
Conveys a sense of heing heard, valined and validated to create non-judg aon-judgmental and caring cavirenment so to understand and solve the user's 's problem.
That sounds really tough. Have you been experiencing these feelings for long time?
Undersianding
Reflects the ability to scknowledge and sccept others per- spectives.
It's completely understandable to have reservations. Take time and consider joining support group your
pace.
Comphant
Reflects the sense of accommodation, cooperation and contir- mation with the expectations and demands of users.
understand that you' 'e feeling upset about the way your friend treated you. Let's explore strategies to improve your communication and set boundaries in healthy way. can be challenging, but
Imposing
Reflects an altempt assert control over the user's emotions, thoughts, and experiences for their betterment.
that can be developed over time. I can recommend some exercises you
help you get started.
Confromtational
Reflects direct or assertive approach that confronts chal-
That sounds really challenging. It's important to remember
lenges the user's thoughts and emotions la make users under-
that your worth inn't sied to your jub performance. Have you
stand their self-defcating patterns or manipulations.
considered salking to your boss about how you' 're feeling? Perhaps, you feel comfortable, we can explore that topic further during our new session.
Dessotisfied
Reflects usual feeling of unhappiness with the user's behaviour or actions. Identifying the source of dissatisfaction focilitates making quicker progress towards the user's goals.
discontentment
Uncertain
Reflects the lack of clarity about the user's s perspectives and experiences. Recognizing the source of uncertainty fosters
I'm not quite sure, but would it be helpjul if we explored that topic further together?
healthy and supportive interactions.
Table 5: The definition and example of different interpersonal behaviour our annotated dataset.
A.3.1 Hardware Configuration
Memory Size: 40 GB, (e) Memory Type: HBM2, (f) Memory clock: 1215 MHz, (g) Bus Width: 5120 bits, (h) Total board power: 250 W.
The experimental setup includes the following de- vice configurations: (a) GPU: A PO0B
(b) CUDA Support: CUDA 11.x (or later.) (c) GPU
clocks: Base: 765 MHz, Boost: 1410 MHz, (d)

---

Psychological Issue
Post-traumatic sttess disorder
Gender
Male
Age
Young
Persona
Openness
Psychotherapeutic Approach
Eclectic
User: Hi, I've been dealing with PTSD recently and has been quite challenging.
Seed Utterances
Therapist: Hi, I'm sorry to hear that you ve been dealing with PTSD.
Could you provide me with more details about your experience?
User: Of course. I've been grappling with PTSD, and I' in uncertain about effective coping strategies.
Speaker
Generated Utterances
Therapist
Hello, nice meet you. How help you today?
User
Hi, I've been struggling with PTSD and not sure to cope with it.
Therapist
I'm sorry to hear that. Would you mind telling me more about your experiences with PTSD?
It started after witnessed traumatic event a few years ago.
User
Ever since then. I've had trouble sleeping and have intense flashbacks.
Therapist
That sounds really difficult. Have you tried any coping mechanisms the past?
User
I've tried breathing exercises meditation, they don't 't to help much.
Table 6: A sample dialogue from PSYCON generated using our method.
A.4 Classifier Results
Results of all five classifiers shown in Table 7. From this table, it can be inferred that RoBERTa- large based classifiers achieve better scores for W- ACC and Macro-F1 as compared to the BERT- large based classifiers.
BERT-large
RoBERT-large
Clasifier
W-ACC
Macro-F1
W-ACC
Macro-F1
GAC
0.882
0.871
0.894
0.873
0.860
SC
0.805
0,881
0.912
0.904
POC
0.947
0.942
0.956
0.943
0,872
0.752
0.898
0.872
Table
Evaluation results the classifiers.
A.5 Weight Optimization
To determine the optimal combination of weights for the reward function, we conducted exper- iments with different weight combinations of $1,32, B2, 83, B4, 35. 71. 72, 51, and ô2. These weights were validated using 20% hold-out dataset Psy- CON, and the combination that resulted in the high- est perplexity
was selected for training THERAPIST. Table presents the weights con- sidered for optimization using the PsyCON dataset. The table indicates that considering all the rewards leads to a better perplexity score. Additionally, re- moving any of the rewards results in a decrease in the perplexity score, highlighting the importance of each reward in the model.

---

WEIGHT OPTIMIZATION
B1
32
33
34
71
7/2
81
82
PPL
0.2
0.2
0.3
0.2
0.5
0.5
0.5
2.5261
0
0
0
0.8
0.2
0.55
0.45
0.6
2.556
0.1
0.1
0.1
0.7
0
0.1
0.9
0.9
0.1
2,6154
0
0
0
0
0
0
0
3.0963
0.1
0.2
0.2
0.2
0.3
0.35
0.65
0.45
0.55
2.5290
0.1
0.2
0.2
0.3
0.2
0.7
0.3
0.3
0.7
2.5361
0.2
0.1
0.1
0.4
0.2
0.8
0.2
0.2
0.8
0
0.15
0.15
0.3
0.3
0.1
0
1
0.25
0.75
2.5961
0.3
0.3
0.3
0.1
0
0
1
0
2,6182
0.2
0.2
0.2
0.2
0.2
0.5
0.5
0.5
0.5
2,58932
0.1
0.2
0.2
0.2
0.3
0.5
0.5
0.75
0.25
2.52131
Table 8: Weight optimization using different reward weight combinations.
User's Profile: Female Adult Extraversion Therapist's Psychotherapeutic Approach: Non-directive
T: Good allernoon. How ure you feeling Ioday? [ ddreellig polite)
U: I've been foeling really down lately. My depression has been affecting energy levels and my ability to engege with others. (nnetral
sorry to hear that you're fooling down experiencing a lack energy due to depression. Could you tell me more about how your depression is impacting your interactions with others? [
empathelic, palite )
U: Well, I've always been extroverted person, but lately, it's beon difficult to find the motivation socialize. feal exhausted and withdrawn, and it's affecting my relationships. negative
T: It sounds like your depression influencing your usual extraverted neture making challenging to engage with others. How does this change in behavior and lack of social interaction make you feel? [ (uudderstanding, moderately polite)
U: makes me feel Isolated and lonely. miss the connections used to have, but It's hard to find the energy reach and
participate social activities. (negative)
can understand how lack of social Interaction and feeling lsolated can contribute to your feelings of loneliness. Let's explore ways to gradually increase your engagement with others while considering your current energy levels. How do you envision addressing this? (understanding, polite)
Maybe
by reaching
close friend family member expressing how been feeling. might help
participate low-energy social activities, such as having a coffee going for walk with someone trust. (positive)
T: Those are great suggestions. Reaching out to someone you trust and engeging in low-energy social activities can help you reconnect with others pace that feels comfortable you. Remember be gentle with yoursell and lake step time. Hew does that with you? (understanding, moderately polite)
feels like a positive approach. eppreciate your guidance and understanding as navigate this. (positive).
T: I'm here to support you every step of the way. la there anything else you'd like to discuss today? (helpful, polite)
U: Actually, I've struggling with negalive thoughts and self-criticism. It's like constant stream of self-doubl and negalivily that (negative)
T: Negative thoughts and self celf-crilicism are common challenges for individuals with depression. Let's explore ways to address these pattemss and cultivate eelf-compaselon. How do you envielon addreseing this? (understandinng moderstely polite)
T: Those excellent strategies. Practicing mindfulness, challenging negalive thoughts, and journaling help you gain and effort, U: It feels like step in the right direction. appreciate your guidance and support as work through this. (positive)
insight into
your thinking patterns develop
compassionate perspective
yourself. Remember,
patient with yourself. How does that resonate with you? (understanding. moderately polite)
I'm here support you every of the way. is there anything else you'd like to discuss today? (helpful, polite) that everything now. Thank you for your understanding and kindness. (positive)
You're most welcome, Remember. I'm here provide support whenever you need it. Take care, and welll continue our work together in the next session. (helpful,, polite)
Figure 4: An example dialogue between therapist (T) and the user (U) from PSYCON. text highlighted purple denotes the sentiment label of the user's utterances. The text highlighted in pink denotes the interpersonal behaviour and politeness labels of the therapist's utterances.


Reviews:
[{"review_id": "fCdzRpJfad", "reviewer": "Reviewer_YcLm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. It is not clear how significant the results are. In addition, the effect of different components of the reward functions is not explored. While the authors explore by setting to 0 each R, RA, or RR, they do not address the effects of their components like R1-7.\n\nResponse1: We appreciate your observation regarding the significance of our results and the exploration of different rewards. While we agree that solely setting R, R_A, and R_R to 0 might not provide a complete understanding, we took a comprehensive approach to address this concern. Our experiment involved systematically varying weight values (β_1, β_2, β_3, β_4, β_5, γ_1, γ_2, δ_1, δ_2) assigned to each rewards. By doing so, we quantified the influence of each of the rewards and their combinations more comprehensively. The outcomes of these experiments, presented in Table 8 within the appendix, offer a holistic view of our findings. This approach allows us to draw more robust conclusions about the effects of each of the rewards on the overall model performance. \n\n2. Suggestion: I would probably recommend comparing your model against a bigger language model (like GPT-J), where you utilize a little shot-learning.\n\nResponse2: We appreciate the valuable suggestion to compare our model against a larger language model, such as GPT-J. While we acknowledge the significance of this approach, we regret to inform you that our computational resources were, unfortunately, limited in this regard. It's possible to do inference using GPT-J for dataset creation, but our ability to fine-tune with a GPT-J-like model was hampered by the constraints of a 40GB GPU. It became apparent that the scope of our input conversations, averaging 27 turns (even some exceeds to 45 turns) with around 30 tokens per response, resulted in a cumulative token count of approximately 810 tokens (maximum 1350 tokens). Regrettably, this far exceeded the capacity of our available resources. These constraints highlight the practical challenges we encountered during our research. Despite these limitations, we assure you that we have strived to maximize the potential of our available resources to produce meaningful results. On positive side, we have requested for resources and would try our best to show the experimental results with GPT-J in camera-ready version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The dataset collected is a valuable resource for future work, and the authors combine LLM prompting with expert interventions to lower the costs and control the quality of the dataset.\n- A novel dialogue system that utilizes a RL framework for fine-tuning.", "reasons_to_reject": "- It is not clear how significant the results are. In addition, the effect of different components of the reward functions is not explored. While the authors explore by setting to 0 each R, RA, or RR, they do not address the effects of their components like R1-7.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of dialogue systems for interactions between users and therapists by including the personalized information of the user, like gender, age, persona, and sentiment. For this purpose, they create a novel conversational dataset, “PSYCON,” that contains annotations in dialogue level like gender, age, persona, and psychotherapeutic approach. Furthermore, the dataset also contains utterance-level annotations like sentiment, politeness, and interpersonal behavior information.  Then, the paper introduces e-Therapist, an interpersonal dialogue system fine-tuned in a RL environment. The results show the improvement in dialogue experience by incorporating interpersonal attributes.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Suggestion: I would probably recommend comparing your model against a bigger language model (like GPT-J), where you utilize a few shot-learning."}, "extra_text": {}}, {"review_id": "qZ1Fdpl6Wk", "reviewer": "Reviewer_jwGD", "summary": "", "questions": "Also listed in previous section.\n\nC. The numbers in Table-1 are using the Roberta classifiers? (\nFrom 548, it appears to be one of the weighted accuracy or F1, please\nclarify). Then considering these classifiers were fine-tuned on smallish\ndatasets, how reliable are these numbers for comparing across the baselines?\n\nD. Not sure what is n in line 531 and why the selection is what it is.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. The motivating use-case and the name (e-Therapist) are not very representative. A therapist is expect to advice on how to address specific problem of the user with relevant therapy-grounded solutions. But both in the anecdotal examples, seed utterances, and from the overall reading of the paper it appears that the proposed model does not really go there (and only seems to be handling some sort of gathering preliminary information=problem elicitation state in therapy sessions).\n\nResponse1: We acknowledge the observation regarding our motivating use-case and the name \"e-Therapist.\" We understand the reviewer's perspective on how the use-case might primarily seem to involve preliminary information gathering. However, we would like to emphasize that while the PSYCON dataset does indeed encompass instances where therapists assist users in the initial stages of problem elicitation, it also encompasses diverse conversational contexts where therapists address specific issues of patients and guide them through various therapy sessions. For instance, we would like to highlight Conversation 1 from our sample dataset, available at the link: https://anonymous.4open.science/r/Anon-2AAD/. In this conversation, a therapist guides a patient through deep breathing exercises, offers relaxation techniques, and addresses the patient's sleep and nutrition patterns. This example illustrates that our e-Therapist model does encompass specific issue-based interactions as well. Example:\n\nPatient:\tNo, I haven't tried anything like that.\n\nTherapist:\tOkay, well let's start by practising some deep breathing right now. Take a deep breath in through your nose, hold it for a few seconds, and then slowly exhale through your mouth. Let's do that a few more times together.\n\nPatient:\tdeep breath in and exhale That does feel a bit better.\n\nTherapist:\tGood, I'm glad to hear that. You can use this technique anytime you're feeling stressed or overwhelmed. And I can also recommend some other relaxation exercises that might be helpful for you.\n\nPatient” \tThat would be great, thank you.\n\nTherapist:\tNow, let's also talk about some ways to improve your overall mental health. Have you been getting enough sleep and eating well?\n\nPatient:\tNot really. I've been having trouble sleeping and I haven't been eating very healthily.\n\nTherapist:\tOkay, those are two really important things to work on. Lack of sleep and poor nutrition can both contribute to feelings of depression and anxiety. Let's start by setting a goal to get at least 7-8 hours of sleep each night. And when it comes to nutrition, try to focus on incorporating more fruits, vegetables, and whole grains into your diet.\n\nPatient:\tThat sounds like a good plan.\n\nTherapist:\tGreat. And lastly, I want to touch on the importance of being kind to yourself. Remember that recovery from depression takes time and it's okay to have setbacks along the way. What's most important is that you keep working towards your goals and continue to take care of yourself.\n\n2. Although there is some mention of involvement of a Psychotherpist as a guide, the dataset creation was primarily by non-experts in Psychotherapy. Indeed they are Linguistic students, so while some generic politeness and sentiment annotations are OK, would it be OK to use Psychotherapeutic labels provided by these annotators? In the dataset creation, how were the cases selected from Reddit? Was any triaging done? These details are completely missing except for brief mentions in lines 258-259. I also find the anecdotal motivating examples a bit unrealistic and machine-like in Figure-1 (\"Hey girl/Hello dear\").\n\nResponse2: We understand the importance of expertise in psychotherapy when creating a dataset for training our model. While linguistic students were indeed involved in annotation, we actively collaborated with experienced psychotherapists to ensure the appropriateness and quality of the labels.  We have not done any triaging in terms of urgency, severity, or other likewise parameters, but we have extracted seed utterances from Reddit using keywords like depression, anxiety, stress, bipolar disorder, disruptive behaviour and dissocial disorders, post-traumatic stress disorder etc.. After extracting these seed utterances these are manually checked in terms of appropriateness ( of which 35% were dropped as they were not good enough to be treated as seed utterances and approx. 40% were linguistically corrected to align the flow of language as if a therapeutic conversation is starting). Additionally, we have taken the important step of incorporating three psychotherapy approaches (directive, non-directive, and eclectic) within the prompt generation, leading to diverse therapy conversations for each set of seed utterances. We will clarify this in camera-ready version by providing a more comprehensive explanation in the relevant section. \n\n3. The numbers in Table-1 are using the Roberta classifiers? ( From 548, it appears to be one of the weighted accuracy or F1, please clarify). Then considering these classifiers were fine-tuned on smallish datasets, how reliable are these numbers for comparing across the baselines?\n\nResponse3: The W-ACC and F1 scores of GAC, PC, SC, CTC, PoC, and IBC classifiers are computed on 80:20, train:test split of PSYCON dataset with manually annotated 340 dialogues which are shown in Table 7. In Table 1, we computed all the metrics based on W-ACC on a whole dataset of 1020 conversations with all annotated labels.\n\n4. Not sure what is n in line 531 and why the selection is what it is.\n\nResponse4: In this context, 'n' represents the number of candidate responses generated by the SLLM and subsequently rewarded using the reward R. To clarify further, these 'n' candidate responses encompass distinct variations of outputs for a given input. These candidates are then ranked based on the reward R. By optimizing the SLLM to generate responses that are highly ranked according to this reward, we create a mechanism that encourages the model to generate relevant outputs. This approach aids in ensuring that the model's generated responses align well with the desired outcomes as ensured by different rewards.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The paper presents an advancement in handling Psychotherapy-related\ndialog through its contributions towards creation of a relevant dataset\nas well as the incorporation of various conversational elements\nessential to Pschotherapy sessions in a dialog generation model via\nreinforcement learning.\n\nB. Overall the paper is well-written although there are too many\nabbreviations and terminology to make it somewhat hard to track and read. The motivation\nand the proposed methods have merit and the experiments are set up well\nto showcase their benefits compared to the state-of-the-art baselines for\nthis topic.\n", "reasons_to_reject": "A. The motivating use-case and the name (e-Therapist)\nare not very representative. A therapist is expect to advice on\nhow to address specific problem of the user with relevant\ntherapy-grounded solutions. But both in the anecdotal examples,\nseed utterances, and from the overall reading of the paper it appears\nthat the proposed model does not really go there (and\nonly seems to be handling some sort of gathering\npreliminary information=problem elicitation state in therapy sessions).\n\nB. Although there is some mention of involvement of a Psychotherpist\nas a guide, the dataset creation was primarily by non-experts\nin Psychotherapy. Indeed they are Linguistic students, so while some\ngeneric politeness and sentiment annotations are OK,\nwould it be OK to use Psychotherapeutic labels provided by\nthese annotators? In the dataset creation, how were the cases selected\nfrom Reddit?\nWas any triaging done? These details are completely missing except\nfor brief mentions in lines 258-259. I also find the anecdotal motivating examples a bit unrealistic and machine-like in Figure-1 (\"Hey girl/Hello dear\").\n", "questions_to_authors": "Also listed in previous section.\n\nC. The numbers in Table-1 are using the Roberta classifiers? (\nFrom 548, it appears to be one of the weighted accuracy or F1, please\nclarify). Then considering these classifiers were fine-tuned on smallish\ndatasets, how reliable are these numbers for comparing across the baselines?\n\nD. Not sure what is n in line 531 and why the selection is what it is.\n", "paper_topic_and_main_contributions": "\nThe paper presents a novel model for interactions\nin the Psychotherapy settings.\nTo this end, the authors suggest (based on previous works\nand other resources that for effective communication) the incorporation of\nuser factors such as age, persona (extroversion, neurotic etc), gender \nas well as specific\nPsychotherapeutic approaches\nadopted from (directive, non-directive, eclectic) into the interaction. Moreover, at the utterance-level it is\nimportant to be polite, account for sentiment in previous\nutterance and employ a variety of interpersonal\nbehavioral approaches (such as confrontational, helpful, empathetic etc).\n\nAs part of their contributions, they created a novel dataset for\nPsychotherapy conversations (using GPT-J) along with expert\ninputs and corrections as well as annotations at dialog and utterance\nlevels. In addition, they propose an RL-based model that uses\nrewards and penalties based on the various user and response aspects.\nComparison with competitive baselines shows the effectiveness of their\ne-Therapist model for incorporating the\nvarious politeness, Psychotherapeutic styles, sentiment constraints\nessential in Psychotherapy.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f2zkIKWNf3", "reviewer": "Reviewer_MZbm", "summary": "", "questions": "(1) On which part of the data are Attribute-specific Reward Models trained? How to split data to get input-label pairs? Is each turn of data used as training data to predict gender, age, and persona?\n\n(2) I am still confused about how to determine the weight of each reward, the authors only mentioned using heuristic approaches in the Limitations. Also, how stable is the training?\n", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. This paper lacks the elaboration and analysis statistics of some details. This paper claims the importance of politeness. The examples shown in Figure 1 are all differences in words and sentence patterns. This paper does not count the diversity of polite expressions in the collected data and the differences among different gender, age, and person. Furthermore, the presented examples show that politeness is not specific to gender, age, and persona. In terms of presentation, the term \"persona\" generally includes \"gender\" and \"age\".\n\nResponse1: Thank you for highlighting the importance of politeness diversity. We understand that our initial presentation might have fallen short in capturing the nuances of these factors. We have indeed conducted analyses to observe how politeness varies based on different attributes such as gender and age. For example, our findings show that conversations generated for Gender=female and Age=elderly tend to exhibit higher levels of politeness.  In our revised manuscript, we will provide a more thorough analysis of the diversity of polite expressions in our collected data, along with a nuanced exploration of differences across gender, age, and persona. We would also like to clarify here that by term “persona”, we mean here BIG FIVE OCEAN personality traits, of which one is considered to generate conversations. For instance, it has been found that for extraversion persona trait the generated conversations had been much more polite as compared to other traits. We will ensure clarity in our revised manuscript on “persona” aspect analysis as well. \n\n2. The paper misses essential references, e.g., ESConv, EmpatheticDialogues, AugESC, etc. The \"polite\" proposed in this paper is similar to empathetic phrases, and the \"interpersonal behaviour\" is similar to response strategies, but this paper does not compare the differences and advantages between the constructed dataset and existing datasets (e.g., ESConv, EmpatheticDialogues, etc.). In addition, the data construction method of this paper is similar to AugESC, but the difference and correlation with AugESC in construction methods are not compared.\n\nResponse2: We apologize for the oversight in missing references to existing works like ESConv, EmpatheticDialogues, and AugESC. In our revised manuscript, we will perform a comprehensive comparative analysis, highlighting the differences, similarities, and benefits of our dataset and approach in relation to these existing datasets. Specifically, our data construction method diverges from AugESC, particularly in terms of the prompt provided to the LLM. We fed <task_description> - consisting of task, psychological issue, and psychotherapeutic approach that had to be followed by Therapist in assisting the client, <seed_utterances>, and user profile aspects: gender, age, persona as prompt to the GPT-J. Lastly, polite and interpersonal behavior therapy annotations were done only after dialogue generation and cleaning steps, differing from AugESC, which used EmpatheticDialogues to extract out the negative emotion reflecting information and posts for dialogue completions. \n\n3. Ethical considerations. Some of the psychological issues involved in this paper have a higher level of ethical risk. Although the authors have given an ethical statement, limited by the ability of GPT-J, it is still difficult to guarantee the ethical risk issue of the data in a few-shot manner. Why not use a more secure and controllable data distillation model like ChatGPT?\n\nResponse3: We appreciate your concern about the ethical risks associated with psychological issues. We understand the potential ethical risks associated with data distillation, even with manual oversight. It is important to note that we opted for GPT-J due to its accessibility and availability for researchers. However, we are aware of the limitations of GPT-J and have taken steps to mitigate ethical risks by involving linguistics and psychology experts for manual validation and corrections. This process is done to ensure high quality psychotherapeutic conversation without any leakage of sensitive information. Lastly, we would provide our dataset only for research purposes after considering a proper request. We will revise our ethical considerations section to provide more clarity on the steps taken to mitigate ethical risks.\n\n4. Multiple fine-grained rewards are set in the model proposed in this paper, but there is no relevant experiment to verify how much each reward contributes to the overall performance.\n\nResponse4:  We acknowledge the need to validate the contributions of individual fine-grained rewards to the overall model performance. To achieve this, we have performed experiments with varying weights (β_1, β_2, β_3, β_4, β_5, γ_1, γ_2, δ_1, δ_2) for each reward. By systematically adjusting these weights and evaluating model performance, we can determine the impact of each reward on the overall outcome. Our results and analyses for these experiments are presented in Table 8 of the appendix to provide transparency and insights into the contribution of each reward.\n\n5. On which part of the data are Attribute-specific Reward Models trained? How to split data to get input-label pairs? Is each turn of data used as training data to predict gender, age, and persona?\n\nResponse5: We performed all experiments with a train:test split of 80:20 and whole data is annotated with required input-label pairs. Hence, attribute specific reward model is trained on this 80% of the data by tracking attribute reward R_A value for each turn, i.e. at each turn, for a generated Therapist’s response, its corresponding reward R_A is computed which is optimized to be maximized. We will provide a more detailed description of this process in our revised manuscript.\n\n6. I am still confused about how to determine the weight of each reward, the authors only mentioned using heuristic approaches in the Limitations. Also, how stable is the training?\n\nResponse6: Each reward weight’s value is kept between 0 to 1 such that all weights sum to 1. This gave us freedom to experiment with different weights for reward models R_A, R_R and R. \nTo determine the best combination of β_1, β_2, β_3, β_4, β_5, γ_1, γ_2, δ_1, and δ_2, we adopted a heuristic approach. We start with high values for R_R ( β_1, β_2, β_3, β_4, β_5) reward weights, and increase it until and unless model performance decreases. In subsequent steps on the model's performance decrease, we start giving values to R_A reward (γ_1, γ_2) weights with different combinations like, giving high values to only one single reward and checking the spike in decrease/increase in model’s performance. Then, we check the importance of R_A (δ_1) and R_R (δ_2). Lastly, the best performing combination of weights (β_1, β_2, β_3, β_4, β_5, γ_1, γ_2, δ_1, and δ_2) is selected. We will elaborate on our heuristic approach for setting reward weights and discuss the stability of training in our manuscript. We will ensure that our explanations provide clearer insights into these aspects.\n\n7. Missing References:\n[1] Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset [2] Towards Emotional Support Dialog Systems [3] AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation\n\nResponse7: Thank you for bringing up the missing reference. We apologize for any oversight and will promptly include all these references in camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper contributes a psychotherapy data resource distilled from GPT-J with fine-grained user-side and therapist-side annotations.", "reasons_to_reject": "(1) This paper lacks the elaboration and analysis statistics of some details. This paper claims the importance of politeness. The examples shown in Figure 1 are all differences in words and sentence patterns. This paper does not count the diversity of polite expressions in the collected data and the differences among different gender, age, and person. Furthermore, the presented examples show that politeness is not specific to gender, age, and persona. In terms of presentation, the term \"persona\" generally includes \"gender\" and \"age\".\n\n(2) The paper misses essential references, e.g., ESConv, EmpatheticDialogues, AugESC, etc. The \"polite\" proposed in this paper is similar to empathetic phrases, and the \"interpersonal behavior\" is similar to response strategies, but this paper does not compare the differences and advantages between the constructed dataset and existing datasets (e.g., ESConv, EmpatheticDialogues, etc.). In addition, the data construction method of this paper is similar to AugESC, but the difference and correlation with AugESC in construction methods are not compared.\n\n(3) Ethical considerations. Some of the psychological issues involved in this paper have a higher level of ethical risk. Although the authors have given an ethical statement, limited by the ability of GPT-J, it is still difficult to guarantee the ethical risk issue of the data in a few-shot manner. Why not use a more secure and controllable data distillation model like ChatGPT?\n\n(4) Multiple fine-grained rewards are set in the model proposed in this paper, but there is no relevant experiment to verify how much each reward contributes to the overall performance.\n\n[1] Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset\n[2] Towards Emotional Support Dialog Systems\n[3] AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation\n", "questions_to_authors": "(1) On which part of the data are Attribute-specific Reward Models trained? How to split data to get input-label pairs? Is each turn of data used as training data to predict gender, age, and persona?\n\n(2) I am still confused about how to determine the weight of each reward, the authors only mentioned using heuristic approaches in the Limitations. Also, how stable is the training?\n", "paper_topic_and_main_contributions": "This paper aims to construct a polite interpersonal psychotherapy dialogue system by considering the user's gender, age, persona, and sentiment. The authors used GPT-J as the base model to construct a psychotherapy dialogue dataset in a few-shot manner. The dataset is annotated with the user's profile, i.e., gender, age, and persona, at the dialogue-level, and is annotated with the user's sentiment and therapist’s politeness and interpersonal behavior at the utterance. Based on this dataset, the authors trained a response generation model on GPT-2 using fine-tuning and policy fine-tuning.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset\n[2] Towards Emotional Support Dialog Systems\n[3] AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation\n"}}]

Author rebuttal:

Meta-review: This work constructs a polite interpersonal psychotherapy dialogue system by conditioning system responses on the user's gender, age, persona, and sentiment.

Contributions:
 - The (synthetic) dataset collected is a valuable resource for future work.

Weakness:
 - As presented it is not clear how significant the results are, there is a lack of analysis of some details, and in relation to the main claim, which is the importance of politeness the examples do not seem to show a specific connection to gender, age, and persona.
Decision: Accept-Main