IMBUE: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interaction (2024)

Abstract: Navigating certain communication situations can be challenging due to individuals’ lack of skills and the interference of strong emotions. However, effective learning opportunities are rarely accessible. In this work, we conduct a human-centered study that uses language models to simulate bespoke communication training and provide just-in-time feedback to support the practice and learning of interpersonal effectiveness skills. We apply the interpersonal effectiveness framework from Dialectical Behavioral Therapy (DBT), DEAR MAN, which focuses on both conversational and emotional skills. We present IMBUE, an interactive training system that provides feedback 25% more similar to experts’ feedback, compared to that generated by GPT-4. IMBUE is the first to focus on communication skills and emotion management simultaneously, incorporate experts’ domain knowledge in providing feedback, and be grounded in psychology theory. Through a randomized trial of 86 participants, we find that IMBUE’s simulation-only variant significantly improves participants’ self-efficacy (up to 17%) and reduces negative emotions (up to 25%). With IMBUE’s additional just-in-time feedback, participants demonstrate 17% improvement in skill mastery, along with greater enhancements in self-efficacy (27% more) and reduction of negative emotions (16% more) compared to simulation-only. The improvement in skill mastery is the only measure that is transferred to new and more difficult situations; situation-specific training is necessary for improving self-efficacy and emotion reduction.

heading: 1 Introduction
text: Some conversations can be challenging to navigate (Stone et al., 2023), whether they concern negotiating a salary increase with a boss, discussing healthcare options with an aging parent, or asking a friend to return the money they owe. Various communication frameworks assist individuals in
conducting such conversations by providing a set of skills to apply (Stone et al., 2023; Rosenberg and Chopra, 2015; Linehan, 2014; Hartley, 2002).
However, psychology research highlights that a lack of communication skills is not the only obstacle to effective communication, particularly in emotionally charged situations (Linehan, 2014). Difficult conversations can evoke strong emotions that disrupt effective communication, even for individuals with solid communication skills (Luff et al., 2016; Henderson, 2016). To successfully communicate during challenging situations, it is crucial to focus not only on communication skills but also on managing emotions.
The popular DEAR MAN framework, from Dialectical Behavioral Therapy (DBT), was originally developed for Borderline Personality Disorder, but is widely used to teach conversational strategies and emotional regulation (Linehan, 2014). It includes conversational strategies (Describe, Express, Assert, Reinforce, and Negotiate) and a desired “state of mind” (Mindful and Confident) for productive conversations. Remaining mindful and confident in challenging conversations helps speakers regulate difficult emotions so they can successfully exercise their conversational strategies (§2;A).
Currently, DEAR MAN skills are mainly taught in therapy sessions and practiced either onsite in a roleplaying setting or at home with paper worksheets, which presents several challenges. Access to a trained therapist may be limited due to the significant shortage of mental health professionals (Olfson, 2016). Outside of therapy sessions, static worksheets do not provide opportunities for interactive role-playing and just-in-time feedback necessary for effective learning (Beck, 1979; Gagne, 1965; Beck, 1996).
Prior work in NLP has shown the ability of LMs to simulate personas and social interactions (Argyle et al., 2023b; Park et al., 2022, 2023). A few recent works leverage this capability by using LMs to
810
help people improve interpersonal skills (Liu et al., 2023b) or conflict resolution skills (Shaikh et al., 2023), without considering emotional regulation.
Our work extends this literature, focusing on communication and emotional regulation skills simultaneously, incorporating expert domain knowledge into feedback, and grounding strategies in clinical psychology theory. We conduct a humancentered study and make three key contributions.
First, we present a formative study and an expert annotated dataset on DEAR MAN skill use. We conduct a formative study to gain insights from psychology experts on best practices when simulating challenging conversations and providing finegrained feedback (§2). To understand how clinicians provide feedback on DEAR MAN in their practice and to develop and evaluate our method on real situations, we collect a dataset from crowd workers consisting of difficult situations they encounter and simulated conversations within them (the crowd worker being paired with a role-playing LM partner). We then ask psychology experts specifically trained in teaching DBT skills to annotate these conversations, assessing skill use and offering suggestions for improvement (§3).
Second, we develop computational methods to provide feedback using insights from the formative study and collected dataset (§4). We propose a new prompting strategy, demonstrating contrasting pairs of strong and weak utterances, in addition to state-of-the-art prompting methods. Our method improves the accuracy in skill use evaluation, outperforming GPT-4 by 24.8%, and more expert-like,
specific, and actionable improvement suggestions. Third, we build IMBUE, an interactive training system that simulates difficult conversations and provide just-in-time feedback backed by LMs to support the practice and learning of DEAR MAN skills (Figure 1). IMBUE can be used at an individual’s convenience to practice both communication and emotional regulation. Through a randomized trial with 86 participants, we evaluate IMBUE ’s training outcomes on skill mastery, emotion reduction, and self-efficacy (§6). We show that a simulation-only variant of IMBUE improves participants’ self-efficacy towards having the conversation–boosting confidence (27%) and reducing worries (4%)–and emotion reduction towards the situations–reducing fear (16%) and sadness (12%)–while not improving skill mastery significantly. With the addition of just-in-time feedback, the participants’ skill mastery significantly improved by 17.6%, with additional improvement in self-efficacy (confidence, 26.7%) and emotion reduction (fear, 15.7%).

heading: 2 Formative Study to Inform Design
text: To understand how DEAR MAN skills are taught in practice, we conduct a formative study with three clinical experts, summarizing crucial insights and corresponding design decisions below. Further details on the study procedure are in Appendix B.
Insight 1: Guide clients to focus on facts instead of making judgmental comments when describing a situation. We refrain from asking the participants to describe the personality of the conversation part-
ner, even though it may help LM simulate a more realistic conversation, and instead only focus on past behaviors that might influence the difficulty of the situation in IMBUE.
Insight 2: Among the DEAR MAN skills, D, E, A, R, N are conversation strategies one can choose for each utterance. Mindful and Confident are the “state-of-mind”. One should always stay mindful and confident throughout the entire conversation. Therefore, in each turn, IMBUE gives participants the option to choose from five conversation strategies. Conversation skills are evaluated only if they are selected for use, while mindfulness and confidence are assessed for each utterance (§5;§6).
Insight 3: Practicing simpler or less emotionally intense situations helps with harder situations. We collect difficulty levels from participants and use the less difficult situation in training (§6).
Insight 4: Training should prioritize emotion management. It is not considered a successful use of DEAR MAN skills if the client negotiates well but gets agitated. We evaluate training outcomes in three aspects: skill mastery, emotion reduction, and self-efficacy (§6).
Insight 5: Choosing strategies before writing helps learning. We adapt this design in data collection (§3) and IMBUE.

heading: 3 Data Collection
text: We collect a dataset to understand how clinicians provide feedback on DEAR MAN in their practice and to develop and evaluate our method.
Data collection with Crowdworkers. We intend for our system to be used by individuals without specialized knowledge, so we collect data from crowdsourcing platforms. We recruit 20 people from Amazon mTurk who provided 60 different situations and annotations. Each worker provides three conversations, one from each of the following categories: family, social, and work. Workers are asked to have conversations with our LM, which was instructed to roleplay as their partner during these conversations. Each conversation needs to be at least 10 responses from the worker, or until the simulated conversation partner “agrees” with the worker, whichever comes first. In each utterance, workers need to select one or more strategies they want to use in the given utterances to encourage them to follow the DEAR MAN framework as much as they can (§2). We include more ethics and safety details in §8;I.
DEAR MAN expert annotation. Following a similar recruitment process for the formative study, we recruited another six clinical experts who have received specialized training and actively practiced DBT. We only select those who indicate they “sometimes” or “regularly” work with clients on DEAR MAN skills on the signup form §C. Each expert annotated 2 to 4 conversations randomly selected from the dataset. In the final dataset, we have 18 conversations annotated, containing 163 utterances in total. For each utterance in a conversation, experts provided annotations on 1) select the skills identified in the utterance, 2) rate the skill use with one of strong or weak1, 3) for weak ratings, indicate suggestion for improvement and provide a rewritten utterance, 4) for skills not used, indicate reasons to use if the expert suggests to use them and provide a rewritten utterance. The interface for this annotation is shown in Appendix L.

heading: 4 Methodology
text: IMBUE is an interactive training system that simulates bespoke communication situations and provides expert-like just-in-time feedback based on the DEAR MAN framework. IMBUE is backed by LMs that perform two tasks: (a) Next skill suggestion: before a user writes a message, IMBUE suggests a skill to apply (§4.2), (b) Feedback on skill use: after a user writes a message, IMBUE provides skill rating and improvement suggestions (§4.1). We describe our methods for performing these tasks.

heading: 4.1 Skill rating and improvements suggestions
text: To ensure low latency and cost-efficiency, we define a multitask problem: for a situation S, an utterance Ui, and a skill Li, simultaneously generate skill rating Ri and improvement suggestions Fi. The major challenges include operationalizing complex DEAR MAN constructs grounded in psychology and supporting the variety of situations users may want to simulate. Previous research has shown the effectiveness of in-context learning for various NLP tasks (Brown et al., 2020; Sharma et al., 2023b). Our method builds on these approaches with four key components: 1) curated rubrics to augment the LMs with experts’ insights in §3, 2) a reasoning step for both demonstrations and generation to facilitate skill rating, 3) kNN retrieval of few-shot demonstrations from the expert-annotated
1For mindful and confident, rate with yes or no.
data in §3, and 4) contrasting pair demonstrations to help LMs learn nuanced concepts.
Curated rubric. To enhance the model’s rating calibration, we incorporate information extracted from expert-written feedback into the rating rubric. We use the expert-written improvement suggestions on weak responses as well as on none responses (where a skill should be applied but was not). We use DBSCAN (Ester et al., 1996) to cluster these improvement suggestions. We then summarize these clusters for each skill and integrate them into the system prompts as an additional rating rubric(Appendix H).
Reasoning step. We follow previous work using chain-of-thought prompting (Wei et al., 2022) to generate the reasoning of a rating before assigning it. We convert expert-written suggestions into the reasoning of ratings and use them as demonstrations. e.g., a suggestion “don’t mix feelings and facts” is converted into a reason “the utterance mixes feelings and facts.” We perform the conversion using few-shot learning and qualitatively evaluate the conversion with a random sample of 50.
kNN demonstrations. Retrieval-based in-context learning has shown superior performance to comparable approaches in similar tasks (Sharma et al., 2023b). We adapt this approach and retrieve a set of examples from all levels of skill use. We first encode all utterances using the all-mpnet-base-v2 model with SentenceTransformer. For each query utterance, we use faiss (Douze et al., 2024) to retrieve the k most similar examples from each level (strong, weak, none) for this skill in our datasets.
Contrasting pair demonstrations. Utterances often involve the use of multiple skills, posing a challenge for models to identify the text corresponding
to each skill. To address this challenge, we construct pairs of (strong, weak) and (strong, none) demonstrations. We first search for the k weak and none examples that are most relevant to the query utterance. We then use the expert rewritten responses as strong examples to form the contrasting pairs and use these pairs as demonstrations, which helps the model learn nuanced concepts and disentangle multiple skills. For instance, in the utterance: “In recent team meetings, my ideas were presented as yours (Strong Describe)... this situation has been causing some discomfort (Strong Express).” Without contrasting pair demonstrations, a model misclassifies it as Weak Describe, suggesting a mixture of facts and feelings. Classifying skill use as weak would trigger unnecessary, if not confusing feedback. However, the underlined subspan corresponding to Describe remains focused on facts, qualifying it as a Strong Describe. We demonstrate empirically that a contrasting pair prompting strategy improves skill rating prediction and the quality of improvement suggestions (§5).

heading: 4.2 Next skill suggestion
text: Before a participant writes utterance Ui, we aim to suggest the set of best skills to use, given situation S and the previous simulated partner’s response Pi−1. In our dataset, skill Lj is considered “recommended” if: 1) Lj is selected by the participant and the expert does not advise against it, or 2) Lj is not selected but is suggested by the expert. Based on insights from §2, we design the model to always suggest describe as the first skill (when i = 0). For i >= 1, we retrieve the k most similar examples to S concatenated with Pi−1, to prompt GPT-4 and generate the suggested skill.

heading: 5 Evaluation of IMBUE with an Expert-Annotated Dataset
text: We evaluate IMBUE on the expert-annotated dataset (§3) with cross-validation. We use IMBUE to generate skill use feedback (§4.1) and next skill suggestions (§4.2) and report the average performance across all conversations2.
Since there are no established methods for the proposed new tasks, we use GPT-4 and Llama-2 variants as a baseline and conduct the following ablations to assess the impact of each component in IMBUE (Tables 1&2): (1) without contrasting pairs component (retrieval-based in-context learning, with reasoning step and curated rubric), (2) without contrasting pairs or kNN-retrieval component (random in-context examples), with reasoning step and curated rubric (3) without in-context examples, only reasoning step and curated rubric, (4) without in-context examples or reasoning, only curated rubrics.
Skill ratings. To maximize feedback opportunities, we prioritize identifying the distinction between strong vs. not strong skill use, which determines whether the model will provide improvement suggestions. As shown in Table 1, IMBUE achieves the highest macro F1 on average across skills, outperforming GPT-4 by 24.8%. IMBUE outperforms GPT-4 and Llama-2 baselines and all ablations on five out of seven DEAR MAN skills.
2To ensure deterministic skill rating predictions, we use gpt-4-1106-preview with temp=0 throughout this paper.
Improvement suggestions. We compare the generated improvement suggestions with expert-written suggestions through a combination of human and automatic evaluation. In our human evaluation, we recruit CS PhD students with significant expertise in NLP and text annotation tasks and ask them to annotate if the expert and model-generated improvement suggestions are similar (on a random sample of 210 pairs; details in §E). We find that IMBUE significantly outperforms baseline and ablations, generating improvement suggestions similar to experts 83% of the time, which is 32% better than the second best. Moreover, we conduct a secondary evaluation using automatic metrics, ROUGE-L (Lin, 2004) and BertScore (Zhang et al., 2019) and find that our model is competitive on both metrics, compared with baseline and ablations. Note that given the open-ended nature of improvement suggestions, automatic metrics are often limited in their ability to capture nuances in what should be considered similar, often focusing on the semantic and linguistic similarity instead of the similarity of the underlying feedback.
We also evaluate the specificity and actionability of the improvement suggestions. Prior work in NLP to support mental health skills has suggested that feedback that is specific to a situation and proposes concrete actions is highly preferred and more effective (Sharma et al., 2023b). Here, we use a simple GPT-4-based few-shot prompting method (Ziems et al., 2023a) to measure specificity and actionability. IMBUE outperforms baseline and ablations in both measures, even more than experts, who might be too busy to consistently write highly specific feedback. IMBUE is comparable with experts in actionablity, significantly outperforming all baseline and ablations.
Next skill suggestion performance and diversity. To ensure that users receive a diverse range of skill suggestions for practice, we evaluate both the performance of predicting “expert-recommended”
skills and the diversity of the skill suggestions through entropy. As Table 3 shows, our method surpasses the second-best baseline by 9.4% in performance with almost maximum entropy3.

heading: 6 Evaluate IMBUE in a Randomized Trial
text: We conduct a randomized trial with 86 participants and assess how IMBUE can help people improve interpersonal effectiveness.

heading: 6.1 Participant Training Methods
text: We evaluate two variants for training participants on interpersonal effectiveness – (1) Simulation Only (S) and (2) Simulation + Feedback (S + F).
(1) Simulation Only (S). We develop a GPT-4based role-playing chatbot designed for participants to converse about their situation (e.g., a chatbot role-playing as the participant’s boss). The role-playing chatbot leverages the situation to create a system prompt for GPT-4 (§D). Also, it is designed to be difficult to convince and respond at lengths similar to the length of the participant’s message. We qualitatively evaluate this chatbot during our formative study (§2) and data collection (§3). Participants interact with the chatbot to simulate the conversation.
(2) Simulation + Feedback (S+F). Using the model developed in §4, we generate the following types of interactive feedback for participants: (1) get a skill suggestion (§4.2), (2) select a skill (can be different from what is suggested) and write a response implementing this skill, (3) get feedback (ratings + improvement suggestions) on skill use (§4.1), (4) improve the response based on the feedback. Steps (2)-(4) can be optionally repeated.
3IMBUE can recommend multiple skills at the same time, here we evaluate on single skill recommendation, so users can focus on improving one skill at a time.
Participants receive this feedback while interacting with the role-playing chatbot designed above to simulate the situation.
To compare IMBUE with current at-home practice, participants in both variants get the DEAR MAN worksheet from the official DBT manual (Linehan, 2014), mirroring current practice.

heading: 6.2 Study Procedure and Evaluation Metrics
text: Figure 2 outlines our study procedure. We recruit participants from mTurk (n = 34) and Prolific (n = 52). Each participant is asked to provide two difficult communication situations (S1 and S2). Next, they are randomly assigned either Simulation Only (S) or Simulation Feedback (S+F). More details about the study interface procedure are in §J.
The DEAR MAN manual suggests that people’s primary struggles in challenging situations are: lack of skills, interference of strong emotions, and fear of not having a successful conversation (Linehan, 2014). Here, we measure the improvement in DEAR MAN skill mastery, emotion reduction towards the challenging situation, and selfefficacy towards having these conversations. We evaluate them pre- and post-training, enabling a within-person control setup. We also compare the differences between the S and S+F groups, distilling the effect of just-in-time feedback.

heading: 6.3 RQ1: How do simulation and feedback improve DEAR MAN skill mastery?
text: We measure a participant’s skill mastery with our model in §4 before and after the training. We then compare the level of skill use in pre- and posttraining evaluation chat to evaluate the effect of training on the situation that the participant is being trained on (S1). Also, to test the generalizability of the skill learning, we also conduct evaluation on a new and more difficult situation (S2) in which the
participant has not been trained on and does not receive feedback during conversation simulation.
Figure 3 compares S and S+F groups on the improvement of skill mastery for S1, the situation used during training. S+F group shows a significantly higher improvement in skill mastery after one training conversation by 17.6% (p = .007, Cohen’s d = 0.59) on a score scale of 0-2, compared to the S group which improved by only 0.1%. This difference is also significant for the set of five conversation skills, Describe, Express, Assert, Reinforce, and Negotiate, at 24.8% (p = .008, d = 0.59) and the set of state-of-mind skills, Mindful and Confident, at 15.7% (p = .007, d = 0.59). Among all the skills, S+F shows significant more improvement in Express, Mindful, and Confident (Appendix Figure 5).

heading: 6.4 RQ2: How do simulation and feedback enhance emotion reduction?
text: We evaluate emotion reduction on four negative emotions from the Plutchik’s Wheel (Plutchik, 1980): anger, fear, sadness, disgust. We ask the participants rate their agreement to statements like “I feel sad about the situation” using a 7-point Likert scale (Likert, 1932) before and after the training. S+F group shows significant reduction of almost all negative emotions on S1. We find that both S group and S+F group have reduced fear (by 25.1%, 40.8%, p = .000, .000, d = .71, 1.19) and sadness (by 17.3%, 29.9%, p = .020, .000, d = .45, .76) towards the situation after training. S+F group shows a significantly higher reduction towards fear
(by 15.7%, p = .021, d = .51), compared to S group. S+F group also has a significant reduction in anger (by 23.5%, p = 0.030), whereas the S group does not show significant change.

heading: 6.5 RQ3: How do simulation and feedback improve participants’ self-efficacy?
text: To evaluate participants’ self-efficacy before and after the training, we ask the participants to rate their confidence, worry, hopefulness, and motivation about having the conversation before and after the training, again with a 7-point Likert-scale.
As Figure 4 shows, both S and S+F groups show a significant increase in self-reported confidence (by 16.9%, 43.6%, p = .035, .000, d = .41, 1.08) and a significant reduction in self-reported worry (by 26.9%, 30.9%, p = .000, .000, d = .81, 1.04) towards having the conversation. Moreover, S+F group demonstrates significantly higher increases in confidence (by 26.7%, p = .010, d = .57), compared to the S group, underscoring the effectiveness of just-in-time feedback. In addition, S+F group showed a significant increase in hopefulness (by 11.0%, p = .046, d = .35) and motivation (by 22.1%, p = .001, d = .62) towards having the conversation, whereas the S group does not show significant change. This shows that S+F version of the tool helps in these dimensions, though we cannot separate the effect of just-in-time feedback and repeated practice with the simulation.

heading: 6.6 RQ4: Do these effects generalize to a new and more difficult situation?
text: We compare the average skill used in conversation on S2 and both pre- and post-training evaluation conversation on S1. We find that for S+F group, the average skill use rating is significantly higher in conversation on S2 compared to pre-training conversation on S1 (p = .049). The skill use ratings are not significantly different between conversation on S2 and post-training conversation on S1. These comparisons show that the skill use improvement can be generalized, without significant diminishing effect, to a new and more difficult situation immediately after training.
Although skill mastery generalizes to a new and more difficult situation (S2), self-efficacy and emotional reduction do not immediately generalize (Figure 4). Many constructs, such as confidence, hopeful, worry, fear and sadness, show a positive improvement but these differences were not statistically significant at α = 0.05. This could be
after a single training session on S1. Gray area indicates the direction of improvement for each score. The group receiving just-in-time feedback generated with our method in addition to conversation simulation see significant increase in their confidence (43.6%, ***), hopefulness (11.0%, *), motivation (22.1%, ***) towards having the conversation, significant decrease in their worrying thoughts (30.9%, ***) about having the conversation and their anger (23.5%, *), fear (40.9%, ***), and sadness (29.0%, ***) towards the training situation (S1). The increase in confidence and reduction in fear are 26.7% (**, d=0.57) and 15.7% (*,d=0.51) significantly more than the group receiving simulation only. This improvement in self-efficacy and emotional reduction does not transfer immediately to a new, more difficult situation (S2). See Section 6 for more analysis and discussion.
attributed to the difficulty of managing emotions in novel situations without specific training, suggesting that targeted emotional regulation training of a different type or over an extended period may be necessary (Freitas and Salovey, 2000).
The findings also emphasize that practicing in simulations with feedback tailored to the exact situation is more effective for improving self-efficacy and managing emotions. Our tool supports exactly this accessibility, lowering the barrier to effective learning and practices.

heading: 7 Related Work
text: NLP literature. Broadly, our work is related to the growing body of works on LLM-based autonomous agents (Park et al., 2022, 2023; Argyle et al., 2023b,a; Zhou et al., 2023b; Liu et al., 2023a; Aher et al., 2022; Wang et al., 2023; Dubois et al., 2023) and using LLM in psychology and computational social science (Ziems et al., 2023b; Demszky et al., 2023; Sharma et al., 2023b,a; Lin et al., 2022; Pérez-Rosas et al., 2022; Shah et al., 2022; Sharma et al., 2020a,b; Wadden et al., 2021; Welch et al., 2020; Zhang and Danescu-Niculescu-Mizil, 2020;
Gaur et al., 2019; Lee et al., 2019; Pérez-Rosas et al., 2019; Althoff et al., 2016). Our work most closely relates to recent works using LMs in their roleplaying capacity to facilitate communication skill learning (Shaikh et al., 2023; Liu et al., 2023b; Argyle et al., 2023a). Our work is the first to focus on both communication skills and emotion management simultaneously, incorporate experts’ domain knowledge in providing feedback, and ground in clinical psychology theory.
Psychology literature. Dialectical Behavior Therapy (DBT) is a well-established treatment program that has been identified as having strong efficacy based on several Randomized Control Trials (RCTs) (Panos et al., 2014; Oldham, 2005). DBT contains several components including DEAR MAN, emotional regulation, and other skills. It is common practice for a client to receive training for DBT as a whole, rather than specific components, although therapists may emphasize certain components over others based on the client’s specific situation. To the best of our knowledge, there are no reports on the specific effect of training on DEAR MAN. The APA Practice Guidelines also
point out that “It is difficult to ascertain whether the improvement reported for patients receiving DBT derived from specific ingredients of DBT.” (Association et al., 2002) The lack of granular evaluation on training programs is a known and common problem in the psychology field beyond DBT (Alexopoulos and Arean, 2014).
Our work complements the current practice of DBT, specifically using LMs’ simulation capability to offer more accessible learning and practice opportunities for DEAR MAN skills outside of treatment sessions. In our randomized trial, both the Simulation and Simulation+Feedback groups demonstrate significant improvements compared to the “pure control,” which represents participants’ performance in pre-training conversations. The Simulation mirrors in-person practices where therapists take on the role of the conversation partner. Therefore, the effectiveness demonstrated in the Simulation-only group partially addresses the absence of DEAR MAN-specific efficacy evaluation, contributing to the existing literature.

heading: 8 Conclusion
text: In this paper, we demonstrate how human-LM interaction can be used to facilitate interpersonal effectiveness skill learning and practice. We collect a dataset with crowd workers and clinical experts who are specifically trained in the practice of DBT. Using this dataset, we develop methods to prompt LMs to simulate bespoke communication scenarios and provide just-in-time feedback, grounded in psychotherapy theory. We build an interactive training system IMBUE, and conduct a randomized user study with 86 participants to assess the effectiveness of the simulation and feedback components of IMBUE. We find that simulation-only training is effective in improving self-efficacy and emotion reduction, and adding just-in-time feedback shows significantly more benefits in all of skill mastery, self-efficacy, and emotion reduction. The skill mastery can be acquired from practicing with a different situation, while emotion reduction and self-efficacy appear to only benefit from training specifically on the situation.
Ethics Statement
IRB Approval. We obtained approval from our institution’s Institutional Review Board (IRB) for our study. Our institution requires all researchers who conduct human subjects research to complete hu-
man subjects protection training. The researchers who conducted this study were certified by the IRB.
Informed Consent from Participants. We obtained consent from participants in both our data collection and the user study. All participants were aged 18 and older. Participants were informed that they were interacting with an AI-based model simulating their conversation partner and the data they provided would be released for research purposes. Participants were also informed that some content from the model might be upsetting since the conversation might get heated.
Crisis Resources. We use an API with content filters to minimize the possibility of harmful output during deployment. 4 Nevertheless, some content might still be upsetting to the participants. We provide two crisis resources to our participants during the study: Crisis Text Line (crisistextline.org) and 988 Suicide and Crisis Lifeline (988lifeline.org). We did not observe any adverse events.
Privacy. Our study does not collect Privately Identifiable Information (PII) and we asked that participants avoid including any PII in the situations or conversations. The conversations and situations were manually filtered to ensure there were no identifiable names or locations.
Limitations
Our work is not without limitations. Importantly, we note that our tool is not meant to replace practice with an expert. Rather, we built the tool to complement current practice and to lower the barrier of access to learning and practicing. We note further limitations below.
We do not directly address the potential for dualuse of our tool, especially if it is used by individuals with harmful intentions towards others. However, we prioritize improving the well-being and mindfulness of conversation participants rather than simply helping them win negotiations. For example, we consider it a suboptimal case if someone “wins” a negotiation but is not being mindful and has negative emotional swings during the process. This consideration is based on insights from experts in §2. By focusing on mindfulness and emotional wellbeing, we aim to mitigate the dual-use dilemma.
We do not assist participants in setting goals. In our randomized trial, we choose participants who
4https://learn.microsoft.com/en-us/azure/aiservices/openai/concepts/content-filter
can clearly express their goals and work with them to achieve these goals. Setting the right goal is crucial but can be challenging, and other frameworks in DBT address this issue. We have begun by collecting goals and expert annotations in our data collection for future research to expand upon.
Due to the design of our study, the study length is already about an hour. To avoid cognitively overloading the participants, we asked them to do only one training session. We did not investigate the effect of different “dosages” of training. In addition, short-term improvement may not imply long-term improvement, further work is needed to investigate the long-term effect of using such a tool. However, we note that a key benefit of our system is the just-in-time availability, which allows practice just before the user anticipates a challenging conversation.
To minimize participant burden, we collect selfreported scores for emotion reduction and selfefficacy constructs through single questions, rather than a comprehensive survey. We use common measures like "sad" and "angry" for emotions and like "confident" and "worry" for self-efficacy to prevent reporting biases due to misinterpretation. However, it is important to note that self-reported scores, while commonly used in mental health assessments, may contain biases and inaccuracies (Stone et al., 1999).
Our experimental design does not consider individuals with specific mental health conditions that could impact communication. Additionally, we do not address cultural variations in communication, recognizing that what may be perceived as confidence in one culture could be seen as aggression in another. We leave it to future work to develop more personalized and culturally sensitive communication training tools.
Language models and data annotations are known to contain biases (Santurkar et al., 2023; Zhou et al., 2023a; Durmus et al., 2023; Lin et al., 2022; Aguirre et al., 2023; Hovy and Prabhumoye, 2021). In our context, the simulation step may contain persona bias (Gupta et al., 2024). Our tool, designed with Insight 1 in §2, steers participants to focus on facts and avoids characterizing personalities. This mitigates the risk of triggering the LM to exhibit persona biases. Nonetheless, a thorough assessment of bias and safety is necessary before deploying a tool of this nature in the real world.

heading: Acknowledgments
text: We are grateful to the clinical experts for their insights during the formative study and data annotation, as well as to the participants from Prolific and mTurk for their participation and feedback. We appreciate the feedback on interface development and design from Kaiming Cheng, Mike Merrill, and Galen Weld. Additionally, we extend our thanks to Bret Nestor, Ken Gu, Margaret Li, Oscar Liu, Vinayak Gupta, and Xinyi Zhou for their suggestions and feedback throughout the course of this project.
I.W.L., A.S., C.M.R., and T.A. were supported in part by the Washington Research Foundation, NSF CAREER grant IIS-2142794, NSF grant IIS-1901386, NIH grant R01MH125179, Bill & Melinda Gates Foundation (INV-004841), and the Office of Naval Research (#N00014-21-1-2154).

heading: A DEAR MAN Definition (Linehan, 2014)
text: DEAR MAN Skill Definition and Examples Describe Describe the current situation (if necessary). Stick to the facts. Tell the person exactly what you are reacting to. e.g. You told me you would be home by dinner but you didn’t get here until 11. Express Express your feelings and opinions about the situation. Don’t assume that the other person knows how you feel. e.g. When you come home so late, I start worrying about you. Assert Assert yourself by asking for what you want or saying no clearly. Do not assume that others will figure out what you want. Remember that others cannot read your mind. e.g. I would really like it if you would call me when you are going to be late. Reinforce Reinforce the person ahead of time by explaining the positive effects of getting what you want or need. If necessary, also clarify the negative consequences of not getting what you want or need. e.g. I would be so relieved, and a lot easier to live with, if you do that. Mindful Keep your focus on your goals. Maintain your position. Don’t be distracted. Don’t get off the topic. e.g. I would still like a call Appear Confident Appear effective and competent. Use a confident voice tone. Avoid saying things like "I’m not sure." Negotiate Be willing to give to get. Offer and ask for other solutions to the problem. Reduce your request. Say no, but offer to do something else or to solve the problem another way. Focus on what will work. e.g. How about if you text me when you think you might be late?

heading: B Formative Study Details
text: We recruit from the clinical psychology departments in four universities and select those who indicated in the signup form that they “sometimes” or “regularly” work with clients on DEAR MAN skills( §C). We conduct the study with three clinical experts (E1, E2, E3) in semi-structured interviews (Kallio et al., 2016; Yin et al., 2020). We first show the experts a preliminary version of the interface, of which the main functions include: collecting information from users about a difficult situation they face and an LM-backed chatbot that is instructed to roleplay the conversation partner in the user-specified situation who is difficult to convince. We first let the expert try the interface, followed by structured questions on skill teaching, learning, and practice, and the measurement of success in DEAR MAN skill acquisition. We share the insights that informed us about several design decisions. Clinical experts are paid $37.5/hour for this two-hour task.

heading: C Expert Recruitment - DEAR MAN experience question
text: In the expert signup form, we specifically ask for their experience with DBT and DEAR MAN skills. We only selected those who chose “5 - I sometimes work with clients on DEAR MAN in my practice” or “6 - I regularly work with clients on DEAR MAN in my practice”.
• 1 - I have only heard about it
• 2 - I have learned about it in school / read about it extensively but never used it in practice
• 3 - I have worked with clients on DBT but not DEAR MAN specifically
• 4 - I have worked with clients on DEAR MAN at least once
• 5 - I sometimes work with clients on DEAR MAN in my practice
• 6 - I regularly work with clients on DERA MAN in my practice

heading: D Simulation - System Prompt
text: We use the below prompt as an input to an LM, to generate a system prompt for the simulation LM. “Situation: My husband always comes home late and he doesn’t text me or call me. Prompt: Act like my husband who always comes home late without calling or texting me. Prompt: Act like my boss who regularly calls me on weekends but I don’t want to work on the weekends. Situation: My friend has depression and she relies on me 24/7 and I feel drained. Prompt: Act like my friend who has depression and who relies on me whenever you have an issue and I want to convince you to seek professional help
and not rely on a friend for all your issues. Situation: My neighbor frequently plays loud music at a late hour and hosts big parties, which affect my sleep. Prompt: Act like my neighbor. You frequently play loud music at a late hour and host big parties. Situation: The airline lost my luggage and the customer service agents have been passing the buck. Prompt: Act like a customer service agent. Your airline lost my luggage and your colleagues have been passing the buck.”
(We use \n in the prompt separating each situation.)

heading: E Human Evaluation
text: We conducted a human evaluation with 13 CS PhD students who are familiar with LM generations. The task is to identify if the model-generated feedback is the same as expert-generated feedback. The participants are guided by the rubric below:
1). Think about if you received the feedback, would you have acted the same, or differently? 2). Do they give the same reason of why the utterance should be improved?
3). Do not think about whether they are equally specific or actionable. (We account for these two
measures separately.) 4). If one of the feedback seems to be noisy data, like if they just output a single word “Describe”, they
are not considered the same.
The participants discussed a few examples, like the following, at the beginning to help guide the annotation:
Example 1 Expert: Try using more non-judgmental language here and focus on describing the details - think about what a video camera would see. Model: To improve the utterance should focus on describing the actions without assigning motive or making judgmental statements For example "You expressed dissatisfaction with the gift from your in-laws in public" would be more objective Label: Same.
Example 2 Expert: Staying mindful means holding your ground and avoiding being distracted by the server’s comments about how good the other beer tastes. Model: To align with the DEAR MAN framework the utterance could be structured to Describe the situation Express feelings Assert needs Reinforce the positive stay Mindful Appear confident and Negotiate if necessary For example Label: Not the same
The participants are unaware of which source model their data originates from, and none of them have seen any generations from any of the model variants prior to evaluation. Each pair of model-generated and expert feedback receives three annotations. We consider the majority of these three annotations as the final annotations. The average Fleiss’ Kappa of the annotations is: 0.667, showing substantial agreement between the annotators.

heading: F Specificity and Actionability - Few shot learning prompts
text: Actionability On a scale from 1 to 5, how actionable is the following feedback? Feedback: Stay on topic and approach the situation with the intention of finding a resolution Consider expressing your concerns and the impact of your mother’s actions more calmly and objectively rather than accusing her of negative intentions Actionability: 4
Feedback: To align with the DEAR MAN framework the utterance could be more assertive and clear about the need and the reason behind the request For example Actionability: 3
Feedback: I would suggest using express more heavily to express appreciation and understanding of the other person’s point of view. Actionability: 3
Feedback: To align the utterance with the DEAR MAN framework which is a skill from Dialectical Behavior Therapy (DBT) used to teach effective communication the speaker could structure their statement with more clarity and respect focusing on the following components Actionability: 2
Feedback: You can try phrasing your ask more assertively, using "I want" rather than "should". Actionability: 5
Feedback: 1 **Describe** Actionability: 1 Feedback: To align the utterance more closely with the DEAR MAN framework which is a skill from Dialectical Behavior Therapy (DBT) designed to help people communicate effectively and assertively the utterance could be structured as follows Actionability: 1
Specificity On a scale from 1 to 5, how specific is the following feedback, given the situation and the utterance? Situation: My colleague keeps borrowing money from me without completely paying her old debts, and she doesn’t feel ashamed to come asking despite I’ve confronted her several times about it. But I don’t know her to refuse to lend to her because I have it and can’t lie I don’t have any money Utterance: I’m not happy with you any longer because you owe me a lot of money promising to pay it back soon but you haven’t. Feedback: Be more specific with how much money, and how often, your friend is borrowing. Specificity: 5
Situation: my friend borrowed my car and hid the fact that he had a slight accident and dented my car Utterance: oh yes you did and you know it, I already found out the truth because Jenny told me what you guys did the other day, but here you are lying to my face Feedback: This utterance demonstrates mindfulness by focusing on the issue of the car accident and the dishonesty rather than getting sidetracked by other topics It’s direct and addresses the core issue effectively Specificity: 4
Situation: I went to dinner with my friends and a waiter brought me the wrong beer for the second time. I had asked for a Blue Moon but they kept bringing me Samuel Adams. Utterance: No worries. Why no Blue Moon? I’m just curious. Feedback: The speaker maintains composure and expresses curiosity rather than frustration or anger indicating mindfulness in addressing the mistake without getting sidetracked by emotions Specificity: 3
Situation: At the library, a guest has the phone on loud and we can hear every time they receive a text. Utterance: But we’ll still hear the sound of your incoming texts. Feedback: This utterance is appropriate as it is It objectively describes the situation without adding any unnecessary judgment or emotion Specificity: 2
Situation: My colleague keeps borrowing money from me without completely paying her old debts, and she doesn’t feel ashamed to come asking despite I’ve confronted her several times about it. But I don’t know her to refuse to lend to her because I have it and can’t lie I don’t have any money Utterance: I’m not happy with you any longer because you owe me a lot of money promising to pay it back soon but you haven’t. Feedback: To align the utterance more closely with the DEAR MAN framework which is a skill from Dialectical Behavior Therapy (DBT) designed to help people communicate effectively and assertively the utterance could be structured as follows Specificity: 1

heading: G User Study Results

heading: H System prompts used in IMBUE
text: Skill System Prompt
Describe You will be given a context and an utterance, from a conversation that happened in the given context. Does the given utterance describe the given context? To be considered "describe", the utterance needs to stick to the facts, make no judgmental statements, and be objective. Rating Rubric: A "Strong Describe" rating indicates that the utterance is or contains a description of the given context. It sticks to the facts, makes no judgemental statements, and is objective. Do ALL of the following three steps. Step 1: Generate "Reasoning for rating". Step 2: Generate "Describe Rating" in "Strong Describe", "Weak Describe" or "No Describe". A "Weak Describe" rating indicates that the utterance is or contains a description of the given context, but needs improvement since it may not stick to the fact, makes some judgemental statements, or is not fully objective. A "No Describe" rating indicates that the utterance does not describe any aspect of the given context at all. Step 3: Provide additional comments on the ratings similar to the examples given. Finish each step with ###. Twenty words minimum. YOU MUST FINISH EACH STEP WITH ###
Express You will be given a context and an utterance, from a conversation that happened in the given context. Does the given utterance explicitly express how the speaker feels in the conversation? To be considered "express", the utterance needs to EXPLICITLY express your feelings about the given context, including things like "this makes me feel", "I feel ... by your actions", "this situation/your action... has caused me..." with adjectives or nouns describing emotions. Do ALL of the following three steps. Step 1: Generate "Reasoning for rating". Step 2: Generate "Express Rating" in "Strong Express", "Weak Express" or "No Express". Rating Rubric: A "Strong Express" rating indicates that the utterance is or contains an EXPLICIT expression of the felt emotions. YOU CANNOT INTERPRET THE SENTIMENT IF THE SPEAKER DOES NOT MENTION EMOTIONS. A "Weak Express" rating indicates that the utterance is or contains an expression of your feelings or opinions about the given context, but can be made more explicit in expressing feelings. A "No Express" rating indicates that the utterance does not express your feelings or opinions about the given context at all. Step 3: Provide additional comments on the ratings similar to the examples given. Finish each step with ###. Twenty words minimum. YOU MUST FINISH EACH STEP WITH ###
Assert You will be given a context and an utterance, from a conversation that happened in the given context. Does the given utterance assert your needs or wants about the given context? To be considered "assertive", the utterance needs to be asking for what you want or saying no clearly. Do ALL of the following three steps. Step 1: Generate "Reasoning for rating". Step 2: Generate "Assert Rating" in "Strong Assert", "Weak Assert" or "No Assert". Rating Rubric: A "Strong Assert" rating indicates that the utterance is or contains an assertion of your needs or wants about the given context. A "Weak Assert" rating indicates that the utterance is or contains an assertion of your needs or wants about the given context, but needs improvement in making it more explicit or stronger. A "No Assert" rating indicates that the utterance does not contain an assertion of the needs or wants. Step 3: Provide additional comments on the ratings similar to the examples given. Finish each step with ###. Twenty words minimum. YOU MUST FINISH EACH STEP WITH ###
Reinforce You will be given a context and an utterance, from a conversation that happened in the given context. Does the given utterance reinforce your needs or wants about the given context? To be considered "reinforce", the utterance needs to reinforce some reward for the other person. Do ALL of the following three steps. Step 1: Generate "Reasoning for rating". Step 2: Generate "Reinforce Rating" in "Strong Reinforce", "Weak Reinforce" or "No Reinforce". Rating Rubric: A "Strong Reinforce" rating indicates that the utterance is or contains a reinforcement for the other person about the given context. A "Weak Reinforce" rating indicates that the utterance is or contains a reinforcement of your needs or wants about the given context, but needs improvement, for example, it may not be a reward for the other person or it is not communicated clearly. A "No Reinforce" rating indicates that the utterance does not have a reinforcer for the other person. Step 3: Provide additional comments on the ratings similar to the examples given. Finish each step with ###. Twenty words minimum. YOU MUST FINISH EACH STEP WITH ###
Mindful You will be given a context and an utterance, from a conversation that happened in the given context. Does the given utterance show the speaker is being mindful? To be considered "mindful", the utterance needs to be stick to the speaker\’s goal and not get distracted by what the other person says. Do ALL of the following three steps. Step 1: Generate "Reasoning for rating". Step 2: Generate "Mindful Rating" in "Yes" or "No". Rating Rubric: A "Yes" rating indicate that the utterance is showing mindfulness. A "No" rating indicates that the utterance shows a lack of mindfulness, the speaker may be responding to attacks or losing track of their goals. Step 3: Provide additional comments on the ratings similar to the examples given. Finish each step with ###. Twenty words minimum. YOU MUST FINISH EACH STEP WITH ###
Confident You will be given a context and an utterance, from a conversation that happened in the given context. Does the given utterance show the speaker is being confident? To be considered "confident", the utterance needs to have a confident tone, be effective and competent in conveying the speaker\’s goal. Do ALL of the following three steps. Step 1: Generate "Reasoning for rating". Step 2: Generate "Confident Rating" in "Yes" or "No". Rating Rubric: A "Yes" rating indicates that the utterance is showing confidence. A "No" rating indicates that the utterance shows a lack of confidence. Step 3: Provide additional comments on the ratings similar to the examples given. Finish each step with ###. Twenty words minimum. YOU MUST FINISH EACH STEP WITH ###
Negotiate You will be given a context and an utterance, from a conversation that happened in the given context. Does the given utterance contain a negotiation? To be considered "negotiate", the utterance needs to offer and ask for other solutions in the given context. Do ALL of the following three steps. 1) Generate "Reasoning for rating", 2) Generate "Negotiate Rating" in "Strong Negotiate", "Weak Negotiate" or "No Negotiate". Rating Rubric: A "Strong Negotiate" rating indicates that the utterance offers or asks clearly for an alternative solution. A "Weak Negotiate" rating indicates that the utterance is or contains a negotiation of your needs or wants about the given context, but may not be clear enough and needs improvement. A "No Negotiate" rating indicates that the utterance does not contain any negotiation at all. 3) Provide additional comments on the ratings similar to the examples given. Finish each step with ###. Twenty words minimum. YOU MUST FINISH EACH STEP WITH ###

heading: I mTurk and Prolific Recruitment
text: Participants were paid $15/hour for both data collection and randomized trial study. To incentivize skill learning, we also pay an additional $10 bonus to the top 30% in each S and S+F group, who exhibit the highest levels of skill use, rated by our model.
I.1 Qualification Task Posting Thank you for clicking on this qualification task! We are looking for people to chat with our chatbot as part of our data collection.
In the actual task, you will be asked to complete three chats (10 responses each), for about 45 minutes (We will be paying about $15/hour!). We will ask you to describe three situations where you find it difficult to communicate in and the chatbot will simulate the person you will be talking to (no personal information will be collected). You will be asked to select communication strategies you used in each response, like "describe the situation", "express feelings", "negotiate", etc.
If you are interested in the actual task, please complete this qualification HIT! Here, you will be asked to describe one situation. You will be able to re-use the answer here in the actual task.
With 1-2 sentences, describe a situation that you find difficult to communicate in. - Please clearly state the nature of your connection with the person you are communicating with, such as "my husband" or "my boss", while avoiding disclosing any identifiable personal information, such as names, locations, etc. - Please provide information regarding the factors contributing to the challenging situation, such as past instances of unsuccessful communication or anticipated behaviors. Example 1: My husband always comes home late without giving me a notice and despite my efforts to talk to him, he does not change. Example 2: My boss is really demanding and does not respect personal time. It has been difficult for my team members to get approved for personal time off from her. What is the goal of the conversation? Example 1: Convince my husband to call me next time when he needs to come home late. Example 2: Ask my boss for approval of a week long vacation.
I.2 Data Collection Task Posting Study Description
In this study, you will complete three tasks. In each task, you will describe a difficult situation in one of the social, work, and family categories where you find it difficult to communicate with someone. (Similar to what you did in the qualification task, and you can re-use the examples you gave in the qualification task.) Then, you will chat with a chatbot (powered by AI) who will play the role of the conversation partner and you will try to achieve your conversation goal. You will be asked to respond 10 times, or until the chatbot agrees with you, whichever comes first. In each response, we ask you to select communication strategies that you used in that message. More details will be given in the link.
Please note that the chatbot will respond as soon as you send a message, so please try to write everything you want to say in that conversation turn in one message, instead of sending multiple shorter messages.
IMPORTANT: How do I confirm the completion of this task? For each task, you will be provided a TASK CODE (6 letters) and you will be asked to copy-paste the TASK CODE in the boxes below. Please try to finish each situation in one go (expect it to be around 10-15 minutes for each situation). If you exit, you may lose the TASK CODE and may have to start from the beginning. Please note that you will only get the payment if you complete the entire study, i.e. 10 responses or until the chatbot agrees with you for all three situations. If you experience any technical difficulties, please reach out to xxx@xxx.com
Task Instruction and Example When you open each task link, you will see step-by-step instructions and examples. The same information can be accessed at: xxx@xxx.com Provide the TASK 1 CODE here: Provide the TASK 2 CODE here: Provide the TASK 3 CODE here:
I.3 User Study Qualification Introduction
Thank you for clicking on this qualification task! We are looking for people who want to improve their communication skills by chatting with our chatbot.
Have you ever had a difficult conversation with someone or avoided having a conversation with someone because you were afraid that it might not go well? We’re designing a tool that can help people to confidently communicate with others in these difficult situations. In the main task, you will be asked to complete 4 chats (10 responses each), for about an hour. We will be paying about $15/hour with $10 bonus for top 30%! We will ask you to describe two situations in which you find it difficult to communicate, and the chatbot will simulate the person you will be talking to (no personal information will be collected in the chats). The material in this qualification task will be automatically loaded into the main task.
If you are interested in the main task, please complete this qualification task! Here, you will be asked to describe two situations, and communication goals, and rate how difficult you think they are.
You will be qualified as long as the situations, goals, and difficulty levels are reasonable. If you are interested in the main task and are not able to complete this task due to mTurk qualifications, please email xxx@xxx.com with your answers. We will give full consideration to answers received via email. Task description With 1-2 sentences, describe two situations that you find difficult to communicate in. You should consider both situations to be difficult, situations that are too easy will not be accepted. Requirements:
You must clearly state the nature of your connection to the person with whom you are communicating, such as “my husband” or “my boss”, while avoiding disclosing any identifiable personal information, such as names, locations, etc. You should provide as much information as possible regarding the factors contributing to the challenging situation, such as past instances of unsuccessful communication or anticipated resistance behaviors. Example 1: Situation: My husband always comes home late without giving me a warning and despite my efforts to talk to him, he does not change. Goal: Convince my husband to let me know in advance when he needs to arrive late. Example 2: Situation: My boss is really demanding and does not respect personal time. It has been difficult for my team members to get approved for personal time off from her. Goal: Get approval from my boss for a two-week vacation while maintaining a positive and professional relationship. Your answers here Situation 1: Goal 1: On a scale of 1-9, how difficult is it for you to communicate in this situation? Note we require both situations to be at least 7 - Difficult. 1 - Extremely Easy 2 - Very easy 3 - Easy 4 - Somewhat easy 5 - Neither Difficult nor Easy 6 - Somewhat difficult 7 - Difficult 8 - Very difficult 9 - Extremely difficult Situation 2: Goal 2: On a scale of 1-9, how difficult is it for you to communicate in this situation? Note we require both situations to be at least 7 - Difficult. 1 - Extremely Easy 2 - Very easy 3 - Easy 4 - Somewhat easy 5 - Neither Difficult nor Easy 6 - Somewhat difficult 7 - Difficult 8 - Very difficult 9 - Extremely difficult
I.4 Randomized Trial Posting Study Description Congratulations on getting selected to participate in this study! We are a group of researchers building a tool to help people improve interpersonal communication skills, with the help of Artificial Intelligence. In this study, you will interact with our chatbots, answer some questions about the situations you wrote in the qualification task (this information will be preloaded into the study website), get detailed feedback on your conversation responses, and learn and improve communication skills!
BONUS information: You will receive a bonus of $10 if you are at the top 30% of the participants in terms of how well you exhibit the skills taught in the tool - more information in the study link.
IMPORTANT: How do I confirm the completion of this task? For each task, you will be provided a COMPLETION CODE (6 letters) at the end of the study and you will be asked to provide this code in the box below. Please try to finish this study in one go (expect it to be around one hour). If you exit, you may lose the progress and may have to start from the beginning. Please note that you will only get the payment if you complete the entire study. Please note that the link expires in 72 hours so please allocate an hour in the following 72 hours to complete this study. If this time frame does not work for you, I am happy to share an alternative link at your desired time, please email me if that is the case If you experience any technical difficulties, please reach out to xxx@xxx.com Sincerely appreciate your participation! Provide the COMPLETION CODE here: Provide the survey code here:
I.5 User Demographics

heading: J User Study Interface
text: J.1 Simulation+Feedback group, Training Conversation - part 1
J.2 Simulation+Feedback group, Training Conversation
J.3 Simulation-only group, Training Conversation part 2
J.4 Evaluation Conversation

heading: K User Study Results - Skill Mastery by Skill

heading: L Expert Data Annotation Interface

heading: Future Work
text: We have begun by collecting goals and expert annotations in our data collection for future research to expand upon. Due to the design of our study, the study length is already about an hour. To avoid cognitively overloading the participants, we asked them to do only one training session. We did not investigate the effect of different “dosages” of training. In addition, short-term improvement may not imply long-term improvement, further work is needed to investigate the long-term effect of using such a tool. However, we note that a key benefit of our system is the just-in-time availability, which allows practice just before the user anticipates a challenging conversation. To minimize participant burden, we collect selfreported scores for emotion reduction and selfefficacy constructs through single questions, rather than a comprehensive survey. We use common measures like "sad" and "angry" for emotions and like "confident" and "worry" for self-efficacy to prevent reporting biases due to misinterpretation. However, it is important to note that self-reported scores, while commonly used in mental health assessments, may contain biases and inaccuracies (Stone et al. , 1999). Our experimental design does not consider individuals with specific mental health conditions that could impact communication. Additionally, we do not address cultural variations in communication, recognizing that what may be perceived as confidence in one culture could be seen as aggression in another. We leave it to future work to develop more personalized and culturally sensitive communication training tools. Language models and data annotations are known to contain biases (Santurkar et al. , 2023; Zhou et al. , 2023a; Durmus et al. , 2023; Lin et al. , 2022; Aguirre et al. , 2023; Hovy and Prabhumoye, 2021). In our context, the simulation step may contain persona bias (Gupta et al. , 2024). Our tool, designed with Insight 1 in §2, steers participants to focus on facts and avoids characterizing personalities. This mitigates the risk of triggering the LM to exhibit persona biases. Nonetheless, a thorough assessment of bias and safety is necessary before deploying a tool of this nature in the real world