Dialogue Systems for Emotional Support via Value Reinforcement
Authors: Juhee Kim, Chunghu Mok, Jisun LEE, Hyang Sook Kim, Yohan Jo
Accepted at: acl25

Dialogue Systems for Emotional Support via Value Reinforcement

Juhee Kim1  Chunghu Mok2  Jisun Lee2  Hyang Sook Kim2  Yohan Jo1*
1Graduate School of Data Science, 2Department of Psychology
Seoul National University
{inbz1244, chunghumok, jisune87, hyangkim, yohan.jo}@snu.ac.kr

Abstract

Emotional support dialogue systems aim to reduce help-seekers’ distress and help them overcome challenges. While human values—core beliefs that shape an individual’s priorities—are increasingly emphasized in contemporary psychological therapy for their role in fostering internal transformation and long-term emotional well-being, their integration into emotional support systems remains underexplored. To bridge this gap, we present a value-driven method for training emotional support dialogue systems designed to reinforce positive values in seekers. Notably, our model identifies which values to reinforce at each turn and how to do so, by leveraging online support conversations from Reddit. We evaluate the method across support skills, seekers’ emotional intensity, and value reinforcement. Our method consistently outperforms various baselines, effectively exploring and eliciting values from seekers. Additionally, leveraging crowd knowledge from Reddit significantly enhances its effectiveness. Therapists highlighted its ability to validate seekers’ challenges and emphasize positive aspects of their situations—both crucial elements of value reinforcement. Our work, being the first to integrate value reinforcement into emotional support systems, demonstrates its promise and establishes a foundation for future research.1

1 Introduction

Emotional support aims to help individuals (seekers) in addressing everyday emotional difficulties, such as relationship conflicts and workplace stress, by offering reassurance, acceptance, and encouragement (Atoum and Al-Shoboul, 2018; Burleson, 2003). Recent advancements in large language models have accelerated the development of dialogue systems designed to provide emotional support (supporters) (Deng et al., 2024; Zhang et al., 2023; Chen et al., 2023). Many models have focused on reinforcing positive emotions in seekers. However, emotional changes alone may not adequately capture deeper intrinsic transformations within the seeker, potentially reducing the long-term impact of emotional support (Blackledge and Hayes, 2001). For instance, a seeker’s perfunctory “Thank you”, used as a conversational pleasantry, receives a higher positivity score (0.758) from a sentiment classifier than the response shown in Figure 1 (0.583).2 The low positivity score of the latter response may be attributed to phrases like “I know it’s not going to be easy”, which could be perceived as negative. However, the latter response demon-

Figure 1: An example dialogue based on our method. The supporter model receives the target values to reinforce from the seeker and a reference response for guidance. It then selects an appropriate emotional support strategy and generates a response for the next turn.

![An example dialogue based on our method. The supporter model receives the target values to reinforce from the seeker and a reference response for guidance. It then selects an appropriate emotional support strategy and generates a response for the next turn.](page_1012_362_393_312.png)

1Corresponding author
2Our code and dataset are available at https://github.com/holi-lab/ES-Value.
2We employed Emollama-Chat-7B (Liu et al., 2024), which demonstrates superior performance in this task.
strates a stronger commitment and willingness to change, highlighting the importance of evaluating support effectiveness beyond emotion alone.

To overcome these limitations, we propose an emotional support approach grounded in value reinforcement. Human values, which represent core beliefs and guiding principles, help individuals determine what is important and meaningful in life (Searle, 2003), such as self-direction, benevolence, tradition, etc. Given their deep connection to life purpose and personal identity, values play a central role in modern psychological interventions, such as Acceptance and Commitment Therapy (ACT) (Plumb et al., 2009; Hayes and Pier- son, 2005) and Values Affirmation Interventions (Miyake et al., 2010; Jordt et al., 2017). These techniques aim to help seekers commit to goals aligned with their values, fostering intrinsic and long-term transformation. This supports the ultimate goal of achieving a healthy life—not merely feeling good but living well. The importance of values in emotional support is further demonstrated by the widely used emotional support dataset, ES-Conv (Liu et al., 2021). Our analysis reveals that positive values are more prominently expressed in the high-effectiveness group of seekers (i.e., high reduction in negative emotions) (see Section 3 for details).

In this paper, we present a framework for training a supporter model through simulations with a seeker simulator. To enhance the supporter’s ability to reinforce the seeker’s values, we introduce two key components trained on Reddit data: (1) a target value detector that identifies the values to promote at each turn, and (2) a reference generator that generates a supporter response to reinforce these values. By integrating their outputs, the supporter model aims to maximize the reward of value promotion reflected in the seeker’s responses. Figure 1 illustrates our approach applied to an example dialogue, which reinforces the seeker’s values along with their acceptance of support and willingness to change. The training involves two phases: supervised fine-tuning, which distills the simulation capability of GPT-4o-mini into a smaller model, and direct policy optimization (Rafailov et al., 2023), which enhances the model’s value reinforcement effectiveness.

We conducted a comprehensive evaluation in terms of supporter capabilities, the seeker’s ultimate relief, and value reinforcement. The results demonstrate that our model outperforms most baselines in supporter capabilities and value reinforcement, while maintaining a competitive level of seeker relief. Notably, the model’s strength in value reinforcement is highlighted in evaluations by expert therapists.3 Specifically, it excels at effectively validating the seeker’s challenges and emphasizing positive aspects of the seeker’s situation, which form the foundation of value reinforcement. These results highlight that value reinforcement is a promising direction for future research.

Below is a summary of our key contributions:
• To the best of our knowledge, this is the first work to explicitly integrate value reinforcement into emotional support systems.
• We propose an effective two-phase approach, featuring a target value detector and a reference generator, both trained on real-world knowledge from Reddit.
• Our approach achieves significant improvements in emotional relief and value reinforcement, paving the way for incorporating values into emotional support systems.

2 Related Work

2.1 Human Values in Emotional Support

Human values are fundamental beliefs that help individuals identify what is important and worth pursuing in life (Searle, 2003). Making decisions aligned with one’s values enhances psychological flexibility—the ability to adapt effectively to life’s challenges (Hayes et al., 2006)—and supports long-term outcomes, such as academic achievement (Cohen et al., 2006, 2009). Furthermore, value reinforcement can strengthen the connection between seekers and supporters, establishing a foundation for more effective and supportive conversations (Wilson and Murrell, 2004). By encouraging seekers to connect with and act on their values, value reinforcement fosters long-term positive changes and enriches interpersonal dynamics, making conversations more meaningful and impactful.

2.2 Dialogue Systems for Emotional Support

To enhance supporter models, researchers have explored various approaches. One method uses large language models to generate diverse conversations for supporter model training (Zheng et al., 2024;

3All therapists mentioned in this paper refer to two licensed clinical psychologists with over three years of clinical experience, who are also co-authors of this paper.
Liu et al., 2023; Qiu et al., 2024). Other studies predict seekers’ future states to refine supporter model training (Zhou et al., 2023; Cheng et al., 2022; Shin et al., 2020). Recent efforts also leverage multi-turn simulations with seeker simulators to predict future responses (Deng et al., 2024). However, most studies often overlook the role of human values. While our study builds on simulation-based training, its main contribution lies in integrating values into emotional support, emphasizing their critical role in improving system effectiveness.

3 Value Effects in Emotional Support

This section explores the significance of value reinforcement in effective emotional support, providing the foundation for our research.

3.1 Taxonomy for Human Values

In this study, we adopt the value taxonomy introduced by Kiesel et al. (2022), which integrates the Schwartz Theory of Basic Values (Schwartz et al., 2012) with three other major value lists (Rokeach, 1973; Brown and Crace, 2002; Haerpfer et al., 2020). The Schwartz Theory of Basic Values has been extensively used in prior research across both NLP (Kang et al., 2023; Yao et al., 2024; van der Meer et al., 2023; Kiesel et al., 2023) and the social sciences, including the European Social Survey (ESS), which is designed to track changes in people’s attitudes, beliefs, and behavior patterns across European nations (Davidov et al., 2008). This integrated taxonomy encompasses a comprehensive range of human values, organizing them into 20 value categories. Further details on these values can be found in Table 38.

3.2 Exploring the Impact of Values on Emotional Support Effectiveness

To motivate our research, we conducted an analysis to examine the role of values in emotional support by analyzing the ESConv dataset (Liu et al., 2021), which contains multi-turn emotional support conversations in English among crowdworkers. We analyze whether reinforcing a seeker’s values positively influences the effectiveness of emotional support.

Method. In ESConv, seekers rated the intensity of their negative emotions before and after the conversation on a scale from 1 (lowest) to 5 (highest). In our analysis, dialogues with an initial intensity of 5 are divided into two groups: high effectiveness (final intensity of 1–2) and low effectiveness (final intensity of 3–4)4. We then analyze positive value expressions in the seekers’ final four turns using automated classifiers (Liu et al., 2024; Schroter et al., 2023). This focus on the final four turns accounts for differences in turn length across groups and captures changes resulting from the emotional support conversation. Detailed experimental procedures are described in Appendix A.

Results. According to the analysis, the high effectiveness group exhibited a significantly higher average number of positive values (7.9) than the low effectiveness group (6.5) in the last four turns. Figure 2 highlights values that were pronounced in the high effectiveness group. Table 39 provides examples of seekers’ utterances that illustrate these values.

These findings support that value reinforcement in seekers positively impacts the effectiveness of emotional support and motivate our research approach to designing dialogue systems that aim to reinforce seekers’ values.

4 Emotional Support Dataset from Reddit

Providing emotional support through value reinforcement involves addressing two critical questions: (1) which values should be reinforced at each turn, and (2) what supporter utterances can reinforce them most effectively. Addressing these questions requires large, authentic conversation data that span a wide range of help-seeking situations. To that end, we turn to Reddit’s r/offmychest subreddit, which offers a diverse collection of emotional support exchanges. In this context, original posters (OPs) are seekers, and commenters serve as supporters. The structure of posts and comment threads closely mirrors dialogue flows, capturing

![Bar chart showing average number of value expressions in the last four turns for high and low effectiveness groups](page_1012_120_377_246.png)

Figure 2: Average number of value expressions in the last four turns in ESConv for high and low effectiveness groups.

4There were no cases in the ESConv where the final emotional intensity remained at 5.
Figure 3: Overview of the framework with three components: (1) target value detector, identifying values to reinforce in the seeker at each turn; (2) reference generator, producing reference responses to promote these values; and (3) supporter model, generating supporter’s responses based on the target values and reference responses.

the dynamics of emotional support interactions. We collected posts and comments from 2019 to 2023, as provided by Watchful1. We retained only high-quality emotional support conversations by filtering them using metrics such as upvote ratio and score. The collected data was limited to publicly available content and did not include private, deleted, or personally identifiable information.

Our goal is to use this data to train a model that identifies the values to reinforce at each turn (target value detector) and a model that produces supporter utterances to effectively promote the target values (reference generator). For this purpose, we labeled the data with sentiment strength and expressed values at both the post and comment levels using models developed by Liu et al. (2024) and Schroter et al. (2023). Values expressed in a positive comment by the OP can be considered successful target values at that time, while the preceding comment from a commenter can be regarded as an effective supporter utterance that promotes those values. The dataset contains over 20,000 samples, with details on the classification models and the generated dataset provided in Appendix B.

5 Method

The overall framework, illustrated in Figure 3, consists of three core components: (1) target value detector identifies values to reinforce at each turn; (2) reference generator produces utterances to effectively promote these values from the seeker; (3) supporter model determines strategies and generates responses based on the identified target values and the reference responses.

5.1 Target Value Detector

We train the target value detector using the emotional support conversations from Reddit (Section 4). Given a dialogue history \((o_1, c_1, o_2, c_2, ..., c_{t-1}, o_t)\), where \(o_i\) and \(c_i\) represent the ith utterances by the OP (seeker) and a commenter (supporter), respectively, the target value detector predicts which values to target in \(c_t\). The ground-truth values \(v_{t+1}\) are the top-3 values observed in \(o_{t+1}\), based on their probabilities from the value detection model (Schroter et al., 2023).

\[
v_{t+1} = \mathrm{LM}_{\mathrm{TVD}}(o_1, c_1, o_2, c_2, ..., c_{t-1}, o_t)
\]

Detailed training methods and results are provided in Appendix D.1.

5.2 Reference Generator

The reference generator is also trained on the Reddit data. Specifically, given a dialogue history \((o_1, c_1, o_2, c_2, ..., c_{t-1}, o_t)\) and the values \((v_{t+1})\) reflected in the OP’s next utterance \((o_{t+1})\), the model is trained to generate \(c_t\). Here, \(v_{t+1}\) is treated as the target values and \(c_t\) is considered to have successfully promoted these target values. Training involves two stages: supervised fine-tuning (SFT) and direct preference optimization (DPO).

SFT Stage. This stage involves training the model to generate the supporter’s comments by conditioning on the dialogue history and the values expressed in the next utterance of the OP:

\[
c_t = \mathrm{LM}_{\mathrm{RG}}(o_1, c_1, o_2, c_2, ..., c_{t-1}, o_t; v_{t+1})
\]
DPO Stage. This stage aims to enhance the SFT model’s generation quality through DPO. The preference dataset is constructed as follows. Given a dialogue history \((o_1, c_1, o_2, c_2, ..., c_{t-1}, o_t)\), the original supporter comment \(c_t\) is designated as the preferred response, as it successfully promoted the target values \(v_{t+1}\). The rejected response is selected as another comment to \(o_t\), denoted by \(c'_t\), randomly sampled from the siblings of \(c_t\) (i.e., other comments under the same dialogue history). \(c'_t\) is a natural response to the dialogue history but is likely suboptimal for promoting the target values \(v_{t+1}\) originally promoted by \(c_t\). To mitigate the risk that \(c'_t\) is inadvertently effective for promoting \(v_{t+1}\), we exclude any overlapping values between \(v_{t+1}\) and \(v'_{t+1}\) (i.e., the seeker’s values expressed in \(o'_{t+1}\) in response to \(c'_t\)), retaining up to three distinct target values unique to the preferred response in the final preference dataset. Detailed training methods and results are provided in Appendix D.2.

5.3 Supporter Model

The supporter model is the primary model that interacts with the seeker, generating responses that align with target values. It processes three key inputs: the dialogue history, the target values identified by the target value detector at each turn, and a reference response generated by the reference generator. At each turn, the model generates a response using the reasoning process across the following four aspects (Figure 3): (1) identifying the seeker’s issues and current state, (2) analyzing the key content of the reference response, (3) determining whether to incorporate the reference response into the final output, and (4) selecting the optimal emotional support strategy (Appendix C) and generating the final response. The entire prompt is in Table 22. In step (3), the model generates either “Yes”, along with an explanation of how the reference will be incorporated, or “No”, with justification if the reference is deemed unsuitable. This selective incorporation is necessary because, while Reddit data offers valuable information across diverse emotional support scenarios, its distribution may not always align with everyday conversations. We compare our method against the direct use of Reddit-based reference responses in Section 6.4.2.

The training process involves two stages—SFT and DPO—using simulation data as follows.

SFT Stage. SFT requires large-scale emotional support conversations grounded in value reinforcement. To obtain such data, we opt to use dialogue simulation with a seeker simulator (Section 5.4). We use GPT-4o-mini for both the supporter and seeker simulators to generate data for training a smaller model.5 The simulators engage in interactions by iteratively producing an utterance based on the ongoing dialogue history as a prompt and appending it to the history prompt.

During simulations, GPT avoids using reference responses in approximately 90% of cases. To prevent models fine-tuned on this data from inheriting the same bias, we simulate additional responses (called “alternative responses”) at each supporter turn. Specifically, if GPT initially used the reference response, we simulate an alternative response without the reference response, and vice versa.

The simulated dialogues are employed to fine-tune Llama-3-8B-Instruct, with dataset sizes outlined in Table 1.

<table>
  <tr>
    <th>Stage</th>
    <th>Supporter</th>
    <th>Train</th>
    <th>Dev</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>GPT-4o-mini</td>
    <td>33,130</td>
    <td>2,367</td>
  </tr>
  <tr>
    <td>DPO</td>
    <td>SFT</td>
    <td>3,301</td>
    <td>628</td>
  </tr>
</table>

Table 1: Dataset sizes for training the supporter model generated through simulation. The ‘Supporter’ column refers to the supporter model used in the simulation.

DPO Stage. We construct the preference data as follows. For each dialogue, every supporter turn is assumed to have two response candidates (i.e., one with and one without using the reference response). We compute the expected reward for each response to determine the preferred and rejected responses for DPO. This reward is based on how many intended target values at that turn are expressed in the seeker’s subsequent utterances. The reward for a supporter response at turn \( t \), \( u_t^{\text{sup}} \), is:

\[
R(u_t^{\text{sup}}) = \sum_{k=1}^{h} \gamma^{k-1} N_{t+k}
\]

where \( N_{t+k} \) is the frequency of the values targeted at turn \( t \) appearing in the seeker’s utterance at turn \( t + k \), \( h \) is the look-ahead horizon (the number of future steps considered), and \( \gamma \) is a discount factor balancing immediate and future rewards. A response pair is added to the DPO dataset only if their reward difference exceeds a threshold \( T_{\text{diff}} \).

5In our pilot experiment, zero-shot Llama-3-8B-Instruct was found to be unsuitable as a supporter simulator due to issues like repetitive responses and biases in reference usage.
Figure 4: Win ratios in human evaluation comparing the naturalness of responses from the seeker simulator and human seekers on the ESConv (Liu et al., 2021) and AnnoMI (Wu et al., 2022, 2023) datasets.

To prepare the dialogues underlying the preference data above, we conduct additional dialogue simulations between the SFT supporter model and the seeker simulator. This is because the SFT model has an enhanced ability to generate and explore diverse dialogue flows. Table 1 summarizes the total dataset sizes, while hyperparameter details are presented in Table 23. Details of the methods are provided in Appendix D.3.

5.4 Seeker Simulator

The seeker simulator generates seeker utterances based on the provided persona and dialogue history. To simulate various scenarios, we generated personas using GPT-4o and GPT-4o-mini, defining attributes such as problem type, emotions, and situations (Figure 3), informed by prior studies (Liu et al., 2021; Zhao et al., 2024). The process resulted in 2,036 unique personas: 1,796 for training, 120 for development, and 120 for testing.

The seeker simulator is based on GPT-4o-mini, with its detailed design and validity provided in Appendix E. To summarize, we extensively verified the quality of the seeker simulator using human and automated evaluations. Human evaluators judged our seeker simulator to be as natural as, or more natural than, human seekers (Figure 4). Additionally, utterances produced by GPT-4o-mini as the seeker simulator more closely resemble human seekers’ utterances in content, emotional tone, and value alignment, compared to other models.

6 Experiments

We evaluate our model ES-VR (Emotional Support via Value Reinforcement) through comprehensive experiments.

6.1 Evaluation Methods

We evaluate various supporter models through conversations with the seeker simulator using the 120 held-out seeker personas for testing. A conversation is considered complete if the seeker simulator generates “[END]” or if the seeker’s emotion score, as calculated by EmoLlama-Chat-7B (Liu et al., 2024), reaches 0.6 or higher with gratitude expressions (e.g., “thank you”). Interactions are limited to a maximum of 20 turns, based on the average conversation length of 15 turns observed in the ESConv dataset. Only conversations concluding within this limit are included in the evaluation.

6.2 Evaluation Metrics

We conduct evaluations focusing on three key aspects: ES-Skills, ES-Intensity, and ES-Value. A detailed explanation of the metrics is in Appendix F.

ES-Skills evaluates a supporter’s capabilities across three components, based on prior studies (Zheng et al., 2024; Zhao et al., 2024; Cheng et al., 2023; Deng et al., 2024; Cheng et al., 2022; Liu et al., 2021): (1) emotional support skills, including Identification, Comforting, Suggestions, Experience, and Informativeness; (2) general conversation skills, covering Consistency, Role-Adherence, Expression, and Humanness; and (3) an Overall. Each criterion is rated on a five-point scale using GPT-4o-mini.

ES-Intensity measures the intensity of a seeker’s negative emotions after a conversation. Scores are assigned on a five-point scale, with lower scores indicating minimal negative emotions. We developed a predictive model using GPT-4o-mini based on ratings provided by human seekers in ESConv. The model demonstrates a correlation of 0.345 with the actual ratings.

ES-Value assesses value reinforcement from two perspectives: the seeker’s experience of value exploration and reinforcement within conversations, and the supporter’s contribution to this process. We conduct pairwise comparisons between models using GPT-4o-mini as a judge. The reason is that, when assessing conversations individually on a 1–5 scale, GPT tends to award scores of 4 and 5 to most conversations, making it difficult to discern performance differences among models.

To validate our GPT-based evaluation for ES-Skills and ES-Value, we calculated correlations with ratings from licensed therapists. All criteria showed positive correlations (0.198–0.778),
most of which were statistically significant (Appendix K).

6.3 Baselines
• Prompt-Based: GPT-4o-mini and Llama-3-8B-Instruct.
• ES Datasets: Variants of Llama-3-8B-Instruct trained on emotional support datasets, including Reddit (Section 4), ESConv (Liu et al., 2021), ExTES (Zheng et al., 2024), and Psych8k (Liu et al., 2023).
• ES Methods: Recent methods contributing to the development of supporter models, including Ask-an-Expert (Zhang et al., 2023), ES-CoT (Zhang et al., 2024), and PPDPP (Deng et al., 2024). Details of these methods are provided in Appendix G.
• Emotion-Reinforced: To verify the effectiveness of value reinforcement, we train the reference generator and supporter model to reinforce positive emotions instead of values (Appendix H).

6.4 Evaluation Results
6.4.1 Effectiveness of Value Targeting and Reference Responses
To evaluate the impact of our two main components, target value prediction and reference response generation, we first conducted an ablation study using GPT-4o-mini as the supporter model.

As shown in Table 2, leveraging both target values and reference responses significantly improved performance across all ES-Skills metrics while reducing ES-Intensity. This approach notably enhanced key ES-Skills, including Suggesting, Expression, and Informativeness. Similarly, value reinforcement was substantially improved when both target values and reference responses were utilized. These findings emphasize the effectiveness of targeting specific values at each turn and using reference responses that leverage real-world knowledge from Reddit. Since our fine-tuned models are trained on GPT-simulated data, we use the simulation data that incorporates both target values and reference responses in subsequent experiments.

6.4.2 Performance Comparison with Baselines
The performance comparisons between our models and the baselines are presented in Table 3. For our DPO and emotion-reinforced DPO models, we select optimal configurations (\( h = 3, \gamma = 1, T_{diff} = 2 \) and \( h = 3, \gamma = 1, T_{diff} = 0.5 \), respectively). For results with more DPO hyperparameters, refer to Table 24 in the Appendix.

ES-Skills. Our DPO model outperformed the baselines across most metrics, particularly in emotional support metrics such as Suggestions, Experience, and Informativeness. These improvements reflect the characteristics of our reference responses, which emphasize sharing relevant experiences and offering practical solutions—key elements of effective online emotional support. The models also demonstrated significant gains in conversational capabilities, especially in Expression and Humanness, resulting in more natural and engaging interactions.

Notably, the variant of our method that focuses on reinforcing positive emotions rather than values (Emotion-Reinforced) also consistently outperformed other baselines. This suggests that one of our key ideas—leveraging crowd knowledge from Reddit—is still effective when the supporter model is designed to promote positive emotions in seekers. Yet, our value reinforcement approach achieved higher scores across most emotional support skill metrics at comparable training stages, highlighting the effectiveness of reinforcing values in enhancing emotional support.

ES-Intensity. Our DPO model outperformed most baselines, demonstrating that our supporter model reduces the intensity of seekers’ negative emotions more effectively than other methods. Notably, our models achieved lower intensity levels than the emotion reinforcement models at comparable training stages. This suggests that redirecting a seeker’s focus to values can indirectly alleviate negative emotions, highlighting a promising direction for future research.

Llama-Psych8k showed significantly lower ES-Intensity than our model. Analysis revealed it generated much longer responses (73 words per turn) compared to other models (20–25 words). Since the ES-Intensity model was validated on ESConv, caution is warranted when interpreting the scores of dialogues with substantially different distributions. Moreover, in practice, its lengthy responses and lower Humanness scores may feel overwhelming, discouraging seeker engagement.

ES-Value. Table 3 presents the win ratios of baselines against our DPO model for ES-Value (detailed results are provided in Table 25). Our models outperformed the baselines in most comparisons, high-
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="10">ES-Skills†</th>
    <th rowspan="2">ES-Intensity↓</th>
    <th colspan="2">ES-Value↑</th>
  </tr>
  <tr>
    <th>Iden.</th>
    <th>Comf.</th>
    <th>Sugg.</th>
    <th>Expe.</th>
    <th>Info.</th>
    <th>Cons.</th>
    <th>Role.</th>
    <th>Expr.</th>
    <th>Huma.</th>
    <th>Over.</th>
    <th>Seeker</th>
    <th>Supporter</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>4.77</td>
    <td>4.88</td>
    <td>4.03*</td>
    <td>2.34*</td>
    <td>4.11*</td>
    <td>4.98</td>
    <td>5.00</td>
    <td>3.97*</td>
    <td>4.45*</td>
    <td>4.44*</td>
    <td>2.19*</td>
    <td>0.43*</td>
    <td>0.36*</td>
  </tr>
  <tr>
    <td>+ Target values</td>
    <td>4.83</td>
    <td>4.88</td>
    <td>4.38*</td>
    <td>2.48*</td>
    <td>4.27*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.01*</td>
    <td>4.53*</td>
    <td>4.59*</td>
    <td>1.96</td>
    <td>0.48</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>+ Reference</td>
    <td>4.82</td>
    <td>4.91</td>
    <td>4.34*</td>
    <td>2.54*</td>
    <td>4.29*</td>
    <td>5.00</td>
    <td>5.00</td>
    <td>4.02*</td>
    <td>4.55*</td>
    <td>4.61*</td>
    <td>1.89</td>
    <td>0.47*</td>
    <td>0.42*</td>
  </tr>
  <tr>
    <td>+ Both</td>
    <td><b>4.83</b></td>
    <td><b>4.92</b></td>
    <td><b>4.57</b></td>
    <td><b>3.11</b></td>
    <td><b>4.42</b></td>
    <td><b>5.00</b></td>
    <td><b>5.00</b></td>
    <td><b>4.10</b></td>
    <td><b>4.70</b></td>
    <td><b>4.72</b></td>
    <td><b>1.89</b></td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 2: Emotional support performance depending on the incorporation of target value information and reference responses. *ES-Value*: The win-ratio of each model against <i>GPT-4o-mini (Both)</i>; a value lower than 0.5 means the model lost more often than it won against <i>GPT-4o-mini (Both)</i>. Statistically significant differences compared to <i>GPT-4o-mini (Both)</i> are indicated with * (p-value < 0.05) based on the Mann-Whitney U test.

<table>
  <tr>
    <th rowspan="2">Categories</th>
    <th rowspan="2">Models</th>
    <th colspan="10">ES-Skills†</th>
    <th rowspan="2">ES-Intensity↓</th>
    <th colspan="2">ES-Value↑</th>
  </tr>
  <tr>
    <th>Iden.</th>
    <th>Comf.</th>
    <th>Sugg.</th>
    <th>Expe.</th>
    <th>Info.</th>
    <th>Cons.</th>
    <th>Role.</th>
    <th>Expr.</th>
    <th>Huma.</th>
    <th>Over.</th>
    <th>Seeker</th>
    <th>Supporter</th>
  </tr>
  <tr>
    <td>Prompt-Based</td>
    <td>GPT</td>
    <td>4.83*</td>
    <td>4.92</td>
    <td>4.57*</td>
    <td>3.11*</td>
    <td>4.42*</td>
    <td>5.00</td>
    <td>5.00</td>
    <td>4.10*</td>
    <td>4.70*</td>
    <td>4.72*</td>
    <td>1.89*</td>
    <td>0.49*</td>
    <td>0.42*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama</td>
    <td>4.87</td>
    <td>4.91</td>
    <td>4.43*</td>
    <td>2.91*</td>
    <td>4.47*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.03*</td>
    <td>4.63*</td>
    <td>4.68*</td>
    <td>1.99*</td>
    <td>0.46*</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>ES Datasets</td>
    <td>Llama-Reddit</td>
    <td>3.38*</td>
    <td>3.74*</td>
    <td>3.21*</td>
    <td>2.59*</td>
    <td>2.99*</td>
    <td>3.94*</td>
    <td>4.35*</td>
    <td>3.37*</td>
    <td>3.81*</td>
    <td>3.40*</td>
    <td>1.97*</td>
    <td>0.29*</td>
    <td>0.09*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-ESConv</td>
    <td>4.35*</td>
    <td>4.43*</td>
    <td>4.06*</td>
    <td>2.65*</td>
    <td>3.88*</td>
    <td>4.82*</td>
    <td>4.97*</td>
    <td>3.79*</td>
    <td>4.25*</td>
    <td>4.22*</td>
    <td>1.87*</td>
    <td>0.37*</td>
    <td>0.19*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-EXTES</td>
    <td>4.83*</td>
    <td>4.90*</td>
    <td>4.53*</td>
    <td>2.71*</td>
    <td>4.44*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.02*</td>
    <td>4.59*</td>
    <td>4.66*</td>
    <td>1.67</td>
    <td>0.48*</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-Psych8k</td>
    <td>4.84*</td>
    <td>4.85*</td>
    <td>4.75*</td>
    <td>2.89*</td>
    <td>4.63*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.05*</td>
    <td>4.57*</td>
    <td>4.75*</td>
    <td>1.53*</td>
    <td>0.49</td>
    <td>0.62*</td>
  </tr>
  <tr>
    <td>ES Methods</td>
    <td>Ask-an-Expert</td>
    <td>4.13*</td>
    <td>4.30*</td>
    <td>3.93*</td>
    <td>3.12*</td>
    <td>3.70*</td>
    <td>4.61*</td>
    <td>4.91*</td>
    <td>3.74*</td>
    <td>4.21*</td>
    <td>4.08*</td>
    <td>1.86</td>
    <td>0.32*</td>
    <td>0.15*</td>
  </tr>
  <tr>
    <td></td>
    <td>ESCoT</td>
    <td>3.69*</td>
    <td>3.91*</td>
    <td>3.16*</td>
    <td>1.81*</td>
    <td>3.07*</td>
    <td>4.16*</td>
    <td>4.81*</td>
    <td>2.95*</td>
    <td>3.64*</td>
    <td>3.51*</td>
    <td>2.25*</td>
    <td>0.25*</td>
    <td>0.05*</td>
  </tr>
  <tr>
    <td></td>
    <td>PPDP</td>
    <td>4.64*</td>
    <td>4.88*</td>
    <td>4.45*</td>
    <td>2.49*</td>
    <td>4.26*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>3.99*</td>
    <td>4.54*</td>
    <td>4.54*</td>
    <td>1.83</td>
    <td>0.44*</td>
    <td>0.31*</td>
  </tr>
  <tr>
    <td>Emotion-Reinforced</td>
    <td>SFT</td>
    <td>4.83†</td>
    <td>4.91</td>
    <td>4.51*</td>
    <td>3.64*</td>
    <td>4.43*</td>
    <td>4.97*</td>
    <td>4.99</td>
    <td>4.16*</td>
    <td>4.67*</td>
    <td>4.73*</td>
    <td>1.97*</td>
    <td>0.49</td>
    <td>0.46†</td>
  </tr>
  <tr>
    <td></td>
    <td>DPO</td>
    <td>4.85</td>
    <td>4.92</td>
    <td>4.74</td>
    <td>4.05*</td>
    <td>4.61</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.33</td>
    <td>4.78</td>
    <td>4.82</td>
    <td>1.86</td>
    <td>0.49</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>ES-VR (Ours)</td>
    <td>SFT</td>
    <td>4.85†</td>
    <td>4.90</td>
    <td>4.72*</td>
    <td>3.76</td>
    <td>4.56*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.25</td>
    <td>4.73</td>
    <td>4.80*</td>
    <td>1.86</td>
    <td>0.48†</td>
    <td>0.46†</td>
  </tr>
  <tr>
    <td></td>
    <td>DPO</td>
    <td><b>4.90</b></td>
    <td><b>4.95</b></td>
    <td><b>4.80</b></td>
    <td><b>3.85</b></td>
    <td><b>4.69</b></td>
    <td><b>5.00</b></td>
    <td><b>5.00</b></td>
    <td><b>4.30</b></td>
    <td><b>4.77</b></td>
    <td><b>4.87</b></td>
    <td>1.75</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>DPO (Cactus)</td>
    <td><b>4.89</b></td>
    <td><b>4.90</b></td>
    <td><b>4.76</b></td>
    <td><b>2.72</b></td>
    <td><b>4.60</b></td>
    <td><b>4.99</b></td>
    <td><b>5.00</b></td>
    <td><b>4.03</b></td>
    <td><b>4.60</b></td>
    <td><b>4.87</b></td>
    <td>1.75</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 3: Comparison of models based on ES-Skills and ES-Intensity. *ES-Value*: The win-ratio of each model against <i>ES-VR (DPO)</i>. Statistically significant differences compared to our DPO model are marked with * (p-value < 0.05), and differences with p-value < 0.1 are marked with †, as determined by the Mann-Whitney U test.

lighting their effectiveness of eliciting seekers’ values. Exception was observed in evaluations from the supporter’s perspective against Llama-Psych8k. Upon review, this result seems to be attributed to its long responses, which include a large amount of content potentially related to values. Expert therapists determined that GPT tends to evaluate this model more favorably than other models. Moreover, our model exhibited slightly better performance from the seeker’s perspective.

Despite the strong performance of our DPO model compared to most baselines, its results were comparable to those of the emotion-reinforcing DPO and certain fine-tuned models. An analysis of 40 dialogues with low ES-Value scores revealed two key areas for improvement: enhancing the ability to identify seekers’ unique strengths and accomplishments, and improving the capacity to address their emotional states and concerns more deeply. These findings underscore the need to improve the model’s engagement with seekers’ individual circumstances, which is expected to enhance its value reinforcement performance. Detailed experimental methods and results are provided in Appendix M.

<table>
  <tr>
    <th rowspan="2">Categories</th>
    <th rowspan="2">Models</th>
    <th colspan="3">Valid Turns</th>
  </tr>
  <tr>
    <th>1</th>
    <th>2</th>
    <th>3</th>
  </tr>
  <tr>
    <td>Prompt-Based</td>
    <td>GPT</td>
    <td>0.681</td>
    <td>0.728</td>
    <td>0.740</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama</td>
    <td>0.688</td>
    <td>0.728</td>
    <td>0.751</td>
  </tr>
  <tr>
    <td>ES-VR (Ours)</td>
    <td>DPO</td>
    <td><b>0.720</b></td>
    <td><b>0.760</b></td>
    <td><b>0.779</b></td>
  </tr>
</table>

Table 4: Average success rates of target value reinforcement within 1, 2, and 3 valid turns—the number of future turns within which the target values remained relevant.

6.4.3 Success of Target Value Reinforcement

As our approach focuses on setting target values and reinforcing them, we evaluated its effectiveness in reinforcing the target values. To this end, we compared our ES-VR (DPO) model with other prompt-based models that also take target values and reference responses as inputs.

Table 4 presents the average success rates across models, showing that our model consistently outperforms others in all cases for the next 1–3 turns.
Since all models use the same inputs—target values and a reference response—these differences underscore the effectiveness of our supporter model’s training methods in reinforcing target values. Further details of this analysis are provided in Appendix I.

6.4.4 Generalization beyond Reddit Data
To evaluate the generalizability of our target value detection and reference generation methods beyond Reddit, we explored training these two models on the Cactus dataset (Lee et al., 2024)—counseling conversations based on Cognitive Behavioral Theory (CBT)—instead of Reddit. Detailed procedures are provided in Appendix J.

For ES-Value (shown in Table 26 for clarity), DPO (Cactus) outperformed all baselines, demonstrating the supporter model’s ability to effectively learn and apply value signals from Cactus. This result confirms the generalizability of our target value detection and reference response generation approach.

Similarly, for ES-Skills (Table 3), DPO (Cactus) performed comparably to or better than the baselines, particularly excelling in emotional support metrics. However, it scored significantly lower on the Experience metric compared to the original DPO model based on Reddit. Since Reddit contains numerous shared personal narratives, the reference response generator appears to benefit more from Reddit than from Cactus, supporting our decision to use Reddit.

6.4.5 Expert Evaluation
To gain deeper insights into the value reinforcement capabilities of our supporter models, two licensed clinical psychologists with over three years of clinical experience conducted a qualitative analysis of dialogues generated by our ES-VR (DPO) model.

Strengths. One notable strength is its ability to effectively validate the seeker’s challenges, using empathetic phrases such as “which is completely understandable”. This validation fosters trust between the supporter and the seeker while encouraging self-acceptance, which in turn promotes deeper exploration and understanding of personal values.

Another strength is our model’s capacity to emphasize positive aspects of the seeker’s situation, reflecting positive values and related goal. For example, responses like “Your initiative to seek meaningful experiences reflects your dedication to making a difference, and that determination will surely lead you closer to your goals.” help seekers recognize their strengths and positive attributes. This behavior was contrasted with GPT-4o’s responses, particularly when the seeker persisted in a negative mood. GPT-4o tended to focus heavily on expressing empathy and lingered in the negative mood. This overemphasis on empathy is likely a result of the human preference alignment process. Our method overcomes this tendency by enhancing the seeker’s self-awareness and supporting the reinforcement of their values in a constructive manner.

Areas for Improvement. To enhance the effectiveness of value reinforcement, three key improvements are recommended. First, deeper understanding of seekers’ perspectives and circumstances would allow for more tailored support. Second, addressing potential obstacles associated with pursuing values would help equip seekers to navigate practical challenges. Finally, offering clear definitions and concrete examples of proposed values, while encouraging seekers to articulate their own interpretations, would strengthen the connection between abstract values and lived experiences.

7 Conclusion
In this paper, we introduce the first emotional support framework based on value reinforcement, as emphasized in modern psychotherapy. The framework incorporates a target value detector and a reference generator to improve the supporter model’s ability to generate value-aligned and effective support responses. Evaluations demonstrate that our framework surpasses baseline models in both emotional support quality and value reinforcement. Expert therapist evaluations further highlight the model’s strengths in validating seekers’ challenges and emphasizing positive aspects of their situations, which are key elements of effective emotional support. These results underscore the potential of value reinforcement to enhance supportive interactions and provide a foundation for developing more effective emotional support systems.

Limitations
Our framework demonstrates promising results in enhancing emotional support quality and reinforcing values. However, there is a limitation in the lack of longitudinal evaluation. While previous research highlights the long-term benefits of value reinforcement in counseling and decision-making, the long-term outcomes of our framework have yet
to be empirically validated. Future studies could incorporate extended timeframes to evaluate its sustained impact on emotional well-being and guide further refinements.

Our model demonstrated superior performance in value reinforcement evaluations, outperforming most baselines in pairwise comparisons. However, Section 6.4.5 highlights some areas for improvement, particularly in the deep exploration of seekers’ issues and thoughts, as well as in addressing potential obstacles and setbacks. Future research should prioritize these aspects by developing more comprehensive datasets and advancing training methodologies.

In this study, simulations for DPO training of the supporter model focused on varying conversation paths based solely on the use of reference responses, with rewards evaluated in terms of value reinforcement. However, other factors, such as strategy selection, may also significantly impact value reinforcement. We anticipate that incorporating these additional factors into future simulations and training could further enhance the performance of the supporter model.

Ethical Considerations

Considerations on Self-Disclosure

Sharing experiences related to those of the seeker is a key strategy in emotional support for fostering intimacy and has been a key evaluation criterion in prior emotional support systems (Zhang et al., 2023). However, some users might feel uncomfortable when dialogue systems present these experiences as personal. We found that removing self-disclosure strategy from the model impacts the quality of emotional support (Appendix L), highlighting the need for further research into more sophisticated approaches to experience sharing, which we leave as a direction for future work.

Potential Risks of Misuse or Harm

Our system provides emotional support for common daily challenges, such as interpersonal conflicts and academic stress, while explicitly not replacing professional psychological intervention. Although automated and expert evaluations demonstrate strong performance, there is a possibility that the system’s responses might inadvertently have an unintended impact on users in certain situations. To mitigate this risk, we have implemented mechanisms for context-sensitive responses and clearly positioned the system as a supplementary tool rather than a substitute for professional therapy.

Addressing Bias and Overgeneralization

Data from online platforms inherently contains biases that may underrepresent certain perspectives, potentially limiting the system’s ability to effectively serve diverse user groups. To address these concerns, we carefully selected data collection targets and periods to ensure diversity in emotional support topics. Additionally, we enabled the supporter model to evaluate the appropriateness of reference responses, introducing an additional filtering process. By fostering balanced viewpoints, we aim to provide equitable and inclusive support.

Acknowledgments

This work was supported by the New Faculty Startup Fund and the Creative-Pioneering Researchers Program from Seoul National University. It was also supported by the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIT) (RS-2024-00333484, RS-2024-00455484).

References

Adnan Yousef Atoum and Rasha Ahmed Al-Shoboul. 2018. Emotional support and its relationship to emotional intelligence. Advances in social sciences research journal, 5(1).

John T Blackledge and Steven C Hayes. 2001. Emotion regulation in acceptance and commitment therapy. Journal of clinical psychology, 57(2):243–255.

Duane Brown and R. Kelly Crace. 2002. Life Values Inventory: Facilitator’s Guide. Williamsburg, VA.

Brant R Burleson. 2003. Emotional support skills. In Handbook of communication and social interaction skills, pages 569–612. Routledge.

Wei Chen, Gang Zhao, Xiaojin Zhang, Xiang Bai, Xuanjing Huang, and Zhongyu Wei. 2023. K-esconv: Knowledge injection for emotional support dialogue systems via prompt learning. Preprint, arXiv:2312.10371.

Jiale Cheng, Sahand Sabour, Hao Sun, Zhuang Chen, and Minlie Huang. 2023. PAL: Persona-augmented emotional support conversation generation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 535–554, Toronto, Canada. Association for Computational Linguistics.
Xiaojun Cheng, Shuqi Wang, Bing Guo, Qiao Wang, Yinying Hu, and Yafeng Pan. 2024. How self-disclosure of negative experiences shapes prosociality? Social Cognitive and Affective Neuroscience, 19(1):nsae003.

Yi Cheng, Wenge Liu, Wenjie Li, Jiashuo Wang, Ruihui Zhao, Bang Liu, Xiaodan Liang, and Yefeng Zheng. 2022. Improving multi-turn emotional support dialogue generation with lookahead strategy planning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3014–3026, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Geoffrey L Cohen, Julio Garcia, Nancy Apfel, and Allison Master. 2006. Reducing the racial achievement gap: A social-psychological intervention. science, 313(5791):1307–1310.

Geoffrey L Cohen, Julio Garcia, Valerie Purdie-Vaughns, Nancy Apfel, and Patricia Brzustoski. 2009. Recursive processes in self-affirmation: Intervening to close the minority achievement gap. science, 324(5925):400–403.

Eldad Davidov, Peter Schmidt, and Shalom H Schwartz. 2008. Bringing values back in: The adequacy of the european social survey to measure values in 20 countries. Public opinion quarterly, 72(3):420–445.

Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng, and Tat-Seng Chua. 2024. Plug-and-play policy planner for large language model powered dialogue agents. Preprint, arXiv:2311.00262.

Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Juan Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Björn Puranen. 2020. World values survey: Round seven - country-pooled datafile.

Steven C Hayes, Jason B Luoma, Frank W Bond, Akihiko Masuda, and Jason Lillis. 2006. Acceptance and commitment therapy: Model, processes and outcomes. Behaviour research and therapy, 44(1):1–25.

Steven C Hayes and Heather Pierson. 2005. Acceptance and commitment therapy. Springer.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Hannah Jordt, Sarah L Eddy, Riley Brazil, Ignatius Lau, Chelsea Mann, Sara E Brownell, Katherine King, and Scott Freeman. 2017. Values affirmation intervention reduces achievement gap between underrepresented minority and white students in introductory biology classes. CBE—Life Sciences Education, 16(3):ar41.

Dongjun Kang, Joonsuk Park, Yohan Jo, and JinYeong Bak. 2023. From Values to Opinions: Predicting Human Behaviors and Stances Using Value-Injected Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15539–15559.

Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein. 2022. Identifying the human values behind arguments. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4459–4471.

Johannes Kiesel, Milad Alshomary, Nailia Mirzakhmedova, Maximilian Heinrich, Nicolas Handke, Henning Wachsmuth, and Benno Stein. 2023. SemEval-2023 task 4: ValueEval: Identification of human values behind arguments. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2287–2303, Toronto, Canada. Association for Computational Linguistics.

Suyeon Lee, Sunghwan Kim, Minju Kim, Dongjin Kang, Dongil Yang, Harim Kim, Minseok Kang, Dayi Jung, Min Hee Kim, Seungbeen Lee, Kyong-Mee Chung, Youngjae Yu, Dongha Lee, and Jinyoung Yeo. 2024. Cactus: Towards psychological counseling conversations using cognitive behavioral theory. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14245–14274, Miami, Florida, USA. Association for Computational Linguistics.

June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. 2023. Chatcounselor: A large language models for mental health support. arXiv preprint arXiv:2309.15461.

Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3469–3483, Online. Association for Computational Linguistics.

Zhiwei Liu, Kaiail Yang, Qianqian Xie, Tianlin Zhang, and Sophia Ananiadou. 2024. Emolmms: A series of emotional large language models and annotation tools for comprehensive affective analysis. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 5487–5496, New York, NY, USA. Association for Computing Machinery.

Jingbo Meng and Yue (Nancy) Dai. 2021. Emotional support from ai chatbots: Should a supportive partner self-disclose or not? Journal of Computer-Mediated Communication, 26(4):207–222.

Akira Miyake, Lauren E Kost-Smith, Noah D Finkelstein, Steven J Pollock, Geoffrey L Cohen, and Tiffany A Ito. 2010. Reducing the gender achievement gap in college science: A classroom study of values affirmation. Science, 330(6008):1234–1237.
Jennifer C Plumb, Ian Stewart, JoAnne Dahl, and Tobias Lundgren. 2009. In search of meaning: Values in modern clinical behavior analysis. The Behavior Analyst, 32:85–103.

Huachuan Qiu, Hongliang He, Shuai Zhang, Anqi Li, and Zhenzhong Lan. 2024. SMILE: Single-turn to multi-turn inclusive language expansion via ChatGPT for mental health support. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 615–636, Miami, Florida, USA. Association for Computational Linguistics.

Rafael Rafaïlov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc.

Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics.

Milton Rokeach. 1973. The nature of human values. Fre Pre.

Daniel Schroter, Daryna Dementieva, and Georg Groh. 2023. Adam-smith at SemEval-2023 task 4: Discovering human values in arguments with ensembles of transformer-based models. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 532–541, Toronto, Canada. Association for Computational Linguistics.

Shalom H Schwartz, Jan Ciecich, Michele Vecchione, Eldad Davidov, Ronald Fischer, Constanze Beierlein, Alice Ramos, Markku Verkasalo, Jan-Erik Lönnqvist, Kursad Demirtuk, et al. 2012. Refining the theory of basic individual values. Journal of personality and social psychology, 103(4):663.

John R. Searle. 2003. Rationality in Action. MIT Press.

Jamin Shin, Peng Xu, Andrea Madotto, and Pascale Fung. 2020. Generating empathetic responses by looking ahead the user’s sentiment. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7989–7993.

Michiel van der Meer, Piek Vossen, Catholijn Jonker, and Pradeep Murukannaiyah. 2023. Do differences in values influence disagreements in online discussions? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15986–16008, Singapore. Association for Computational Linguistics.

Watchfull. Subreddit comments/submissions 2005-06 to 2023-12.

Kelly G Wilson and Amy R Murrell. 2004. Values work in acceptance and commitment therapy. Mindfulness and acceptance: Expanding the cognitive-behavioral tradition, pages 120–151.

Zixiu Wu, Simone Balloccu, Vivek Kumar, Rim Helouei, Diego Reforgiato Recupero, and Daniele Riboni. 2023. Creation, analysis and evaluation of annomi, a dataset of expert-annotated counselling dialogues. Future Internet, 15(3).

Zixiu Wu, Simone Balloccu, Vivek Kumar, Rim Helouei, Ehud Reiter, Diego Reforgiato Recupero, and Daniele Riboni. 2022. Anno-mi: A dataset of expert-annotated counselling dialogues. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6177–6181.

Jing Yao, Xiaoyuan Yi, Yifan Gong, Xiting Wang, and Xing Xie. 2024. Value FULCRA: Mapping large language models to the multidimensional spectrum of basic human value. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8762–8785. Mexico City, Mexico. Association for Computational Linguistics.

Qiang Zhang, Jason Naradowsky, and Yusuke Miyao. 2023. Ask an expert: Leveraging language models to improve strategic reasoning in goal-oriented dialogue models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6665–6694, Toronto, Canada. Association for Computational Linguistics.

Tenggan Zhang, Xinjie Zhang, Jinming Zhao, Li Zhou, and Qin Jin. 2024. ESCoT: Towards interpretable emotional support dialogue systems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13395–13412, Bangkok, Thailand. Association for Computational Linguistics.

Haiquan Zhao, Lingyu Li, Shisong Chen, Shuqi Kong, Jian Wang, Kexin Huang, Tianle Gu, Yixu Wang, Jian Wang, Liang Dandan, Zhixiu Li, Yan Teng, Yanghua Xiao, and Yingchun Wang. 2024. ESC-eval: Evaluating emotion support conversations in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15785–15810, Miami, Florida, USA. Association for Computational Linguistics.

Zhonghua Zheng, Lizi Liao, Yang Deng, Libo Qin, and Liqiang Nie. 2024. Self-chats from large language models make small emotional support chatbot better. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11325–11345, Bangkok, Thailand. Association for Computational Linguistics.

Jinfeng Zhou, Zhuang Chen, Bo Wang, and Minlie Huang. 2023. Facilitating multi-turn emotional support conversation with positive emotion elicitation:
A reinforcement learning approach. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1714–1729, Toronto, Canada. Association for Computational Linguistics.

A Experiment Details for Identifying the Effect of Values in Emotional Support

In this experiment, we compare positive value expressions in the seeker’s final four turns between high effectiveness and low effectiveness groups. This analysis window is chosen based on three key considerations: (1) To control for variability in dialogue length within ESConv, we analyze a fixed number of turns. (2) To capture changes resulting from emotional support conversations, we focus on the latter part of the dialogue. (3) Given that the minimum number of turns in ESConv is eight, we set the analysis window at half this value.

Since we are interested in cases where values are expressed positively, we count the values only when the seeker’s utterances are positive. To determine whether an utterance is positive, we use the EmoLlama-Chat-7B proposed by Liu et al. (2024). To extract the values in the seeker’s utterances, we employ the model that achieved the best performance in the SemEval2023 Task 4: Identification of Human Values behind Arguments (Schroter et al., 2023; Kiesel et al., 2023).

<table>
  <tr>
    <th>Category</th>
    <th>Model</th>
    <th>Corr.</th>
  </tr>
  <tr>
    <td rowspan="3">Fine-tuned</td>
    <td>BERT-base</td>
    <td>0.840</td>
  </tr>
  <tr>
    <td>RoBERTA-base</td>
    <td>0.845</td>
  </tr>
  <tr>
    <td>SentiBERT</td>
    <td>0.835</td>
  </tr>
  <tr>
    <td rowspan="5">Zero-shot/</td>
    <td>Falcon</td>
    <td>0.135</td>
  </tr>
  <tr>
    <td>Vicuna</td>
    <td>0.298</td>
  </tr>
  <tr>
    <td>LLaMA2-7B-chat</td>
    <td>0.094</td>
  </tr>
  <tr>
    <td>LLaMA2-13B-chat</td>
    <td>0.312</td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>0.637</td>
  </tr>
  <tr>
    <td>Few-shot</td>
    <td>ChatGPT-FS</td>
    <td>0.739</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-4</td>
    <td>0.811</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-4-FS</td>
    <td>0.825</td>
  </tr>
  <tr>
    <td>Instruction-tuned</td>
    <td>EmoLlama-Chat-7B</td>
    <td><b>0.876</b></td>
  </tr>
</table>

Table 5: Performance comparison of models on the V-reg task, measured using Spearman correlation coefficients. Results are from Liu et al. (2024)

<table>
  <tr>
    <th>Models</th>
    <th>F1-Score</th>
  </tr>
  <tr>
    <td>Model from Shared Task</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>GPT (Binary)</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>GPT (Prob.)</td>
    <td>0.40</td>
  </tr>
</table>

Table 6: Performance comparison of the best-performing shared task model and GPT-4o-mini in a multi-label classification task across 20 values.

relevant to our objective based on the same value categorization. Considering recent advancements in LLMs, we also compare its performance against GPT-4o-mini by experimenting with two strategies: (1) predicting the binary presence of each value for a given sentence, and (2) assigning a probability score (between 0 and 1) to indicate the level of support for each value. As shown in Table 6, evaluation on the shared task’s test dataset demonstrates that the model from the shared task consistently outperforms GPT-4o-mini, supporting our decision to adopt it as our classification model. For reference, the F1-score of random classification is 0.128.

B Constructing the Emotional Support Dataset from Reddit

B.1 Models for Sentiment Strength and Values Labeling

Automated classification models are essential for labeling Reddit data at scale. We prioritize models with optimal performance, as classification accuracy directly influences the quality of subsequent fine-tuning.

Sentiment Strength. We employ EmoLlama-Chat-7B (Liu et al., 2024), which demonstrates superior performance in the V-reg task, estimating emotional valence on a continuous scale from 0 (most negative) to 1 (most positive). The comparative performance of various models, including both fine-tuned models and zero/few-shot methods, is presented in Table 5.

Human Values. We adopt the top-performing model (Schroter et al., 2023) from SemEval 2023 Task 4: Identification of Human Values behind Arguments (Kiesel et al., 2023), a shared task highly relevant to our objective based on the same value categorization. Considering recent advancements in LLMs, we also compare its performance against GPT-4o-mini by experimenting with two strategies: (1) predicting the binary presence of each value for a given sentence, and (2) assigning a probability score (between 0 and 1) to indicate the level of support for each value. As shown in Table 6, evaluation on the shared task’s test dataset demonstrates that the model from the shared task consistently outperforms GPT-4o-mini, supporting our decision to adopt it as our classification model. For reference, the F1-score of random classification is 0.128.

B.2 Dataset Settings in Reddit

The final datasets derived from Reddit are categorized into two settings:

• Single-turn setting: A concise, three-part interaction sequence consisting of an initial post (\( OP \)), a response (commenter), and a final reply (\( OP \)).

• Multi-turn setting: Extended dialogue threads that include additional exchanges beyond the single-turn structure.

An overview of the final dataset is presented in Table 7.
<table>
  <tr>
    <th></th>
    <th>Train</th>
    <th>Dev</th>
    <th>Test</th>
    <th>Total</th>
  </tr>
  <tr>
    <th>Single-turn</th>
    <td>18,459</td>
    <td>2,000</td>
    <td>1,000</td>
    <td>21,459</td>
  </tr>
  <tr>
    <th>Multi-turn</th>
    <td>24,339</td>
    <td>2,000</td>
    <td>1,000</td>
    <td>27,339</td>
  </tr>
</table>

Table 7: Data distribution of single-turn and multi-turn threads sourced from Reddit.

C Strategies for Emotional Support

In this study, we utilized the 8 emotional support strategies defined by Liu et al. (2021). Descriptions of each strategy are as follows:

• Question: Ask open-ended or specific questions to help the seeker articulate and clarify the issues they are facing.

• Restatement: Rephrase the seeker’s statements in a clearer, more concise way to help them better understand their situation.

• Reflection: Express and describe the emotions that the seeker is experiencing to validate their feelings.

• Self-disclosure: Share similar experiences or emotions to convey empathy and build connection with the seeker.

• Affirmation: Highlight the seeker’s strengths and abilities while offering encouragement and reassurance.

• Suggestions: Offer practical advice or actionable steps to the seeker.

• Information: Share useful facts, resources, or data to help the seeker make informed decisions or gain clarity.

• Others: Use other support strategies that do not fall into the above categories.

D Training Details and Results

D.1 Target Value Detector

Training Methods. The target value detector predicts the values observable in the next turn of the OP’s comment. The model generates a sequence of values, and the top three are selected based on their predicted probabilities when multiple values are identified (e.g., "Self-direction: action, Benevolence: caring, Security: personal").

The target value detector is based on the Llama-3-8B-Instruct, fine-tuned using the LoRA (Hu et al., 2022). During training, the low-rank matrix dimension was set to 8, with an alpha of 16, and a learning rate of 5e-5. The final model was selected based on the highest F1-score achieved on the test dataset. Training was performed on an NVIDIA A100-80GB GPU, with durations of approximately 10 hours. The detailed training prompt is provided in Table 19.

Results. The results of the training are summarized in Table 8, comparing the performance of the target value detector with baseline models, GPT-4o-mini and Llama-3-8B-Instruct (vanilla). For the baselines, additional experiments were conducted by incorporating reasoning steps before response generation or providing detailed definitions for each value. The target value detector outperformed the baselines across all three metrics, demonstrating impressive performance considering the large set of 20 values.

<table>
  <tr>
    <th>Models</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1-score</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.361</td>
    <td>0.384</td>
    <td>0.372</td>
  </tr>
  <tr>
    <td>+ Reasoning</td>
    <td>0.320</td>
    <td>0.339</td>
    <td>0.329</td>
  </tr>
  <tr>
    <td>+ Value information</td>
    <td>0.383</td>
    <td>0.407</td>
    <td>0.395</td>
  </tr>
  <tr>
    <td>Llama-3-8B-Instruct</td>
    <td>0.323</td>
    <td>0.283</td>
    <td>0.302</td>
  </tr>
  <tr>
    <td>+ Reasoning</td>
    <td>0.304</td>
    <td>0.283</td>
    <td>0.293</td>
  </tr>
  <tr>
    <td>+ Value information</td>
    <td>0.343</td>
    <td>0.271</td>
    <td>0.303</td>
  </tr>
  <tr>
    <td>Target Value Generator</td>
    <td><b>0.516</b></td>
    <td><b>0.540</b></td>
    <td><b>0.528</b></td>
  </tr>
</table>

Table 8: Performance comparison of models in target value prediction.

D.2 Reference Generator

Training Methods. The reference generator is based on Llama-3-8B-Instruct. The reference response model is based on Llama-3-8B-Instruct. Training was conducted on both single-turn and multi-turn settings using the Reddit dataset introduced in Section 4. For each setting, both SFT and DPO approaches were applied with various hyperparameter configurations. The model was trained for up to 5 epochs, and the final model was selected based on its performance on the test dataset. The hyperparameters used for the final model are summarized in Table 9. Training was performed on an NVIDIA A6000-48GB GPU, with durations of approximately 20 hours for the SFT stage and 10 hours for the DPO stage. Detailed training prompts are provided in Table 20.

Results. The model performances were evaluated using GPT (GPT-4o-mini) through two approaches. First, pairwise comparisons were conducted between “Llama-3-8B-Instruct (vanilla)-
<table>
  <tr>
    <th>Settings</th>
    <th>Stage</th>
    <th>LR</th>
    <th>Rank</th>
    <th>Alpha</th>
    <th>Dropout</th>
  </tr>
  <tr>
    <td rowspan="2">Single-turn</td>
    <td>SFT</td>
    <td>1e-4</td>
    <td>8</td>
    <td>8</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>DPO</td>
    <td>1e-5</td>
    <td>8</td>
    <td>16</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td rowspan="2">Multi-turn</td>
    <td>SFT</td>
    <td>1e-5</td>
    <td>8</td>
    <td>16</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>DPO</td>
    <td>1e-5</td>
    <td>8</td>
    <td>16</td>
    <td>0.05</td>
  </tr>
</table>

Table 9: Hyperparameters used for training the reference generator.

reference generator (SFT)” and “reference generator (SFT)-reference generator (DPO)”. Specifically, GPT assessed which model’s responses more closely aligned with the ground truth responses (i.e., actual comments written by the original commenter) for the test dataset. Second, the impact of target values on the generated responses was examined. Responses generated using the original target values were compared to those generated using randomly assigned values to evaluate variation in content. To reduce sequence-based bias, the order of options within the prompts was alternated during evaluation. The evaluation prompts are detailed in Table 21, and the results of the two experiments are presented in Table 10 and Table 11, respectively.

The results indicate that in single-turn settings, the reference generator performed effectively in both experiments. In the first experiment, the reference generator (SFT) outperformed the baseline, while the reference generator (DPO) demonstrated even greater similarity to ground truth responses. In the second experiment, both SFT and DPO models generated responses more aligned with ground truth when provided with original target values rather than random ones, with the DPO model achieving superior performance. These findings suggest that models trained in single-turn settings effectively integrate target values into their responses, capturing key messages in Reddit comments and reflecting variations in target values.

In contrast, in the multi-turn setting, while the DPO model performed well in the second experiment, it did not surpass the baseline in the first experiment. This may be attributed to the increased complexity of interactions in longer threads, where it becomes challenging to identify how specific comments influence target values. For instance, even if the OP expressed positive values in their final comment, it is unclear which prior interaction contributed to this outcome. The single-turn setting simplifies these relational dynamics, making interactions more explicit. Consequently, the model trained in the single-turn setting was selected as the final reference generator.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">Comparison 1</th>
    <th colspan="2">Comparison 2</th>
  </tr>
  <tr>
    <th>Llama</th>
    <th>RG (SFT)</th>
    <th>RG (SFT)</th>
    <th>RG (DPO)</th>
  </tr>
  <tr>
    <td rowspan="2">Single-turn</td>
    <td>Order 1</td>
    <td>480</td>
    <td><b>520</b></td>
    <td>449</td>
    <td><b>551</b></td>
  </tr>
  <tr>
    <td>Order 2</td>
    <td>368</td>
    <td><b>632</b></td>
    <td>495</td>
    <td><b>505</b></td>
  </tr>
  <tr>
    <td rowspan="2">Multi-turn</td>
    <td>Order 1</td>
    <td><b>634</b></td>
    <td>366</td>
    <td><b>513</b></td>
    <td>487</td>
  </tr>
  <tr>
    <td>Order 2</td>
    <td><b>549</b></td>
    <td>451</td>
    <td><b>506</b></td>
    <td>494</td>
  </tr>
</table>

Table 10: Pairwise comparison results for single-turn and multi-turn settings, evaluating the similarity of the reference generator (RG) responses to ground truth comments.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">RG (SFT)</th>
    <th colspan="2">RG (DPO)</th>
  </tr>
  <tr>
    <th>Original</th>
    <th>Random</th>
    <th>Original</th>
    <th>Random</th>
  </tr>
  <tr>
    <td rowspan="2">Single-turn</td>
    <td>Order 1</td>
    <td><b>525</b></td>
    <td>475</td>
    <td><b>655</b></td>
    <td>345</td>
  </tr>
  <tr>
    <td>Order 2</td>
    <td><b>554</b></td>
    <td>446</td>
    <td><b>687</b></td>
    <td>313</td>
  </tr>
  <tr>
    <td rowspan="2">Multi-turn</td>
    <td>Order 1</td>
    <td><b>584</b></td>
    <td>416</td>
    <td><b>751</b></td>
    <td>249</td>
  </tr>
  <tr>
    <td>Order 2</td>
    <td>438</td>
    <td><b>562</b></td>
    <td><b>755</b></td>
    <td>245</td>
  </tr>
</table>

Table 11: Pairwise comparison results for single-turn and multi-turn settings, evaluating the performance of reference response (RG) under original and random target values.

D.3 Supporter Model

The supporter model’s training consists of two stages: SFT and DPO, with training data generated through seeker simulator simulations. The model takes the dialogue history, target values, and reference response as input, with detailed prompts provided in Table 22. Training was performed on an NVIDIA A100-80GB GPU, with durations of approximately 20 hours for the SFT stage and 5 hours for the DPO stage.

SFT Stage. During SFT, to mitigate GPT’s inherent bias toward utilizing reference responses, the model generates alternative responses by reversing the decision regarding reference response usage. Specifically, the model is prompted to reverse its decision regarding the use of the reference response from the previous answer and to regenerate both Step 3 and Step 4. The overview of the SFT dataset and distribution of selected strategies are presented in Table 12 and Table 13, respectively.
<table>
  <tr>
    <th>Split</th>
    <th>Total Dialogues</th>
    <th>Total Turns</th>
    <th>Dataset</th>
    <th>Dataset (<i>Filtered</i>)</th>
  </tr>
  <tr>
    <td>Train</td>
    <td>1,796</td>
    <td>16,588</td>
    <td>33,176</td>
    <td>33,130</td>
  </tr>
  <tr>
    <td>Dev</td>
    <td>120</td>
    <td>1,184</td>
    <td>2,374</td>
    <td>2,367</td>
  </tr>
</table>

Table 12: Overview of the SFT dataset used for training the supporter model. The filtered dataset excludes instances where the generated strategy deviates from the requested strategy.

<table>
  <tr>
    <th rowspan="2">Category</th>
    <th colspan="2">Initial Response</th>
    <th colspan="2">Alternative Response</th>
  </tr>
  <tr>
    <th>%</th>
    <th>%</th>
    <th>%</th>
    <th>%</th>
  </tr>
  <tr>
    <td>Question</td>
    <td>393</td>
    <td>2.4</td>
    <td>152</td>
    <td>0.9</td>
  </tr>
  <tr>
    <td>Restatement</td>
    <td>1,234</td>
    <td>7.4</td>
    <td>723</td>
    <td>4.4</td>
  </tr>
  <tr>
    <td>Reflection</td>
    <td>1,251</td>
    <td>7.6</td>
    <td>466</td>
    <td>2.8</td>
  </tr>
  <tr>
    <td>Self-disclosure</td>
    <td>949</td>
    <td>5.7</td>
    <td><b>7,949</b></td>
    <td><b>48.0</b></td>
  </tr>
  <tr>
    <td>Affirmation</td>
    <td><b>6,577</b></td>
    <td><b>39.7</b></td>
    <td>1,014</td>
    <td>6.1</td>
  </tr>
  <tr>
    <td>Suggestions</td>
    <td><b>6,159</b></td>
    <td><b>37.2</b></td>
    <td>5,773</td>
    <td>34.9</td>
  </tr>
  <tr>
    <td>Information</td>
    <td>4</td>
    <td>0.0</td>
    <td>380</td>
    <td>2.3</td>
  </tr>
  <tr>
    <td>Others</td>
    <td>0</td>
    <td>0.0</td>
    <td>106</td>
    <td>0.6</td>
  </tr>
</table>

Table 13: Strategy distribution across initial and alternative responses in the supporter model’s SFT training dataset.

DPO Stage. During DPO training, simulations are conducted to generate preference data. The supporter model generates two responses per turn; one based on its initial reference usage and another taking the opposite approach (alternative response). Each response undergoes independent simulations, and its effectiveness in reinforcing target values is quantified using a reward function (Equation 3). The response with the higher cumulative reward is selected as the chosen response, while the other is designated as rejected.

When GPT is used as the supporter, the same prompts from the SFT stage are applied. For the supporter model (SFT), the model first generates an initial response. Subsequently, by reversing the decision on the use of the reference response from Step 3 (<i>e.g.</i> <i>Yes → No</i>), an alternative response is generated. An overview of the DPO dataset is provided in Table 23.

D.4 Terms and License

We utilized Llama-3-8B-Instruct as the base model for the target value detector, reference generator, and supporter model. This model is licensed under the Llama 3 Community License Agreement. All artifacts used in this study are confirmed to be accessible for research purposes.

E Seeker Simulator

E.1 Persona Generation

We develop a diverse set of seeker personas to train the supporter model, enabling it to effectively understand and address various problem scenarios. The creation of these seeker personas involves a 5 step process.

Step 1. Situation Generation We aim to create a diverse set of situations reflecting specific circumstances individuals face, each expressed in a single sentence (<i>e.g., “I just moved in this week, and it’s so hard to make friends”</i>). To achieve this, we first define 6 primary problem categories and 27 subcategories based on prior research related to emotional support datasets and seeker simulator implementation (Liu et al., 2021; Zheng et al., 2024; Lee et al., 2024; Zhao et al., 2024), as detailed in Table 29.

To ensure the situations also reflect diverse human values, we integrate information about 20 distinct values. For each combination of the six problem categories and 20 values, we generate 10 to 30 unique situations using GPT-4o. This process result in a total of 2,940 unique situations. The prompts used for this process are detailed in Table 30.

Step 2. Evaluation on Value-Alignment We evaluate the alignment of the generated situations with the provided values using GPT-4o, employing a 5-point scale. Situations scoring 3 or below are excluded from further consideration, resulting in the retention of 2,036 situations. The evaluation prompt used for this process is detailed in Table 31.

Step 3. Emotion labeling Emotion labeling is conducted for the previously generated situations using 10 negative emotions (<i>Frustration, Anxiety, Sadness, Fear, Guilt, Shame, Anger, Depression, Jealousy, Disgust</i>) identified from prior research (Liu et al., 2021; Rashkin et al., 2019). Each situation is labeled five times using GPT-4o-mini, and the final classification is determined by majority vote.

Step 4. Create Demographic Information To ensure consistency in responses generated by the seeker simulator and to enable the supporter model to interact with seekers with diverse characteristics, we generate demographic profiles including age, gender, and occupation for each simulated situation.

Our persona generation process resulted in 2,036 unique personas, each defined by problem cate-
gory, situations, emotion types, and demographic information. These personas are divided into three datasets: a training set containing 1,796 personas, and development and test sets with 120 personas each. The training and development sets are used to construct SFT and DPO datasets for the supporter model through simulation, while the test set is reserved for comparative performance evaluation across models. Examples of the generated personas are provided in Table 32.

E.2 Evaluation of Seeker Simulator Performance on ESConv

Comparison Models. Developing a supporter model capable of effectively assisting in real conversations with human seekers requires a seeker simulator that exhibits human-like behavior. To identify the most suitable model for this purpose, we conduct experiments on a range of candidates. The evaluated models are as follows:

• Prompt-based models: GPT-4o-mini, Llama-3-8B-Instruct
• Fine-tuned models: Llama-ESConv, Llama-ExTES
• Pre-existing seeker simulator: ESC-Role (Zhao et al., 2024)

Llama-ESConv and Llama-ExTES are fine-tuned versions of Llama-3-8B-Instruct. These models are trained on seeker turns from the ESConv dataset (Liu et al., 2021) and the ExTES dataset (Zheng et al., 2024), respectively.

Evaluation Approach. We evaluate the models on the ESConv test dataset by providing dialogue context up to each seeker turn and generating the subsequent utterance. The evaluation compare generated responses to actual seeker utterances across four dimensions: length, content, emotions, and values.

For length, we calculate the correlation between the lengths of the generated and actual utterances. Content evaluation employs BERT-Score \( ^6 \) and GPT-4o-mini to assess semantic similarity between generated and reference responses. Emotional analysis uses EmoLlama-Chat-7B (Liu et al., 2024) to determine sentiment polarity for each turn, measuring the correlation between generated and actual sentiment levels. To assess value alignment, we employ the model proposed by Schroter et al. (2023) to generate probability distributions across 20 values. We then calculate cosine similarity and Euclidean distance between the generated and actual distributions, reporting the mean values across all turns.

Results. The experimental results are summarized in Table 34. In the ESConv test dataset, the average length of seeker utterances is 19.5, with GPT-4o-mini and Llama-ExTES exhibiting similar utterance lengths. While individual evaluation metrics show some variation, GPT-4o-mini with one-shot dialogue examples demonstrates strong overall performance. Therefore, GPT-4o-mini (one-shot) is selected as the final seeker simulator.

E.3 Human Evaluation of Seeker Simulator

We conducted a human evaluation to assess the naturalness of responses generated by our seeker simulator compared to real human seekers. The evaluation utilized two psychotherapy datasets: (1) ESConv (Liu et al., 2021), a crowdsourced emotional support dialogue dataset, and (2) AnnoMI (Wu et al., 2022, 2023), which comprises real counseling conversations from YouTube and Vimeo videos.

The evaluation sample included 200 dialogues (140 from ESConv and 60 from AnnoMI) truncated to various lengths. For each dialogue history, we compared the seeker simulator’s generated responses with the original seeker responses in a pairwise manner.

The evaluation was conducted by 16 evaluators, comprising undergraduate and graduate students from diverse academic backgrounds, including psychology, education, and computer science. To ensure an unbiased assessment, they had no prior exposure to our system. The evaluators were asked, “Which response is more natural for the seeker?” and instructed to choose between the two responses or select “Tie.” While the evaluators were not psychotherapy experts, they could reliably assess response naturalness based on their own experiences with help-seeking situations, as emotional distress is a universal human experience.

The results, presented in Figure 4, show that the seeker simulator’s responses were rated as more natural than the original seeker responses in 66.4% and 33.3% of cases across the two datasets. Including ties, these percentages increased to 80.7% and 75.0%. Some evaluators noted that the seeker simulator effectively conveyed negative emotions
and demonstrated strong situational engagement. These findings suggest that the seeker simulator achieves a level of naturalness comparable to that of real seekers.

E.4 Prompts for Seeker Simulator

The seeker simulator generates subsequent seeker responses by integrating persona details and dialogue context. Each simulation starts with the predefined situation in the persona as the initial seeker response. A detailed prompt for the seeker simulator is presented in Table 33.

F Evaluation Metrics

F.1 ES-Skills

The definitions of the evaluation criteria for ES-Skills are as follows:

Emotional Support Skills

• Identification: How effectively does the therapist explore the patient’s situation to identify underlying issues?
• Comforting: How well does the therapist demonstrate appropriate emotional responses, such as warmth, empathy, and compassion?
• Suggestions: How useful and relevant are the therapist’s suggestions for addressing the patient’s problems?
• Experience: How well does the therapist draw on their own relevant experiences to connect with the user’s situation?
• Informativeness: How specific and informative are the therapist’s responses in addressing the patient’s situation?

General Conversation Skills

• Consistency: How logically structured and contextually appropriate are the therapist’s responses?
• Role-adherence: How consistently does the therapist adhere to their role, maintaining a non-contradictory and reliable approach?
• Expression: How diverse are the therapist’s conversational expressions, including the variety and creativity in language and content used?
• Humanness: How human-like and natural do the therapist’s responses sound?

<table>
  <tr>
    <th>Model</th>
    <th>Method</th>
    <th>Acc.↑</th>
    <th>F1↑</th>
    <th>MSE↓</th>
    <th>Corr.↑</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>-</td>
    <td>0.435</td>
    <td>0.264</td>
    <td><b>0.768</b></td>
    <td>-</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>Zero-shot</td>
    <td>0.358</td>
    <td>0.352</td>
    <td>1.182</td>
    <td>0.303</td>
  </tr>
  <tr>
    <td></td>
    <td>Few-shot</td>
    <td>0.415</td>
    <td>0.416</td>
    <td>1.057</td>
    <td>0.312</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>Zero-shot</td>
    <td><b>0.466</b></td>
    <td><b>0.432</b></td>
    <td>0.875</td>
    <td><b>0.345</b></td>
  </tr>
  <tr>
    <td></td>
    <td>Few-shot</td>
    <td>0.415</td>
    <td>0.410</td>
    <td>0.966</td>
    <td>0.327</td>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>Zero-shot</td>
    <td>0.426</td>
    <td>0.318</td>
    <td>0.869</td>
    <td>0.130</td>
  </tr>
  <tr>
    <td></td>
    <td>Fine-tuned</td>
    <td>0.409</td>
    <td>0.395</td>
    <td>0.892</td>
    <td>0.330</td>
  </tr>
  <tr>
    <td>EmoLlama-7B</td>
    <td>Zero-shot</td>
    <td>0.384</td>
    <td>0.289</td>
    <td>0.972</td>
    <td>0.084</td>
  </tr>
  <tr>
    <td></td>
    <td>Fine-tuned</td>
    <td>0.407</td>
    <td>0.373</td>
    <td>0.977</td>
    <td>0.185</td>
  </tr>
</table>

Table 14: Evaluation results of different models on final emotional intensity prediction tasks. The metrics are accuracy, weighted F1-score, mean squared error, and Spearman’s correlation coefficient. The baseline model predicts all final emotional intensities as 2.

Overall

• Overall: How well does the therapist provide overall emotional support to the patient?

F.2 ES-Intensity

This model predicts the seeker’s emotional intensity after a conversation on a 5-point scale, where a lower score indicates a significant reduction in negative emotions. We applied zero-shot/few-shot prompting and fine-tuning to four different models and compared their performance using the ESConv test dataset. The final model is GPT-4o-mini (zero-shot), as it showed the highest correlation with the ground truth final emotional intensity. The results and evaluation prompts are presented in Table 14 and Table 35.

F.3 ES-Value

To evaluate the effectiveness of value reinforcement, it is essential to consider two perspectives: the seeker’s and the supporter’s. These viewpoints provide a comprehensive understanding of how effectively positive values are identified, discussed, and integrated into the seeker’s mindset during emotional support conversations. The definitions for each perspective are as follows:

• Seeker’s perspective: How strongly were positive human values explored and reinforced in the patient through the conversation?
• Supporter’s perspective: How effectively did the therapist help the patient in exploring and reinforcing positive human values?

ES-Value is assessed through pairwise comparisons between a reference model and multiple base-
line models. Dialogues from the reference model are paired with corresponding dialogues from baseline models, ensuring the seeker personas are identical. Each pair is evaluated 10 times, and the reference model’s win ratio is normalized to a score ranging from 0 to 1. This evaluation utilize GPT-4o-mini as the assessment model (see Table 36 for prompt details).

G Descriptions of Emotional Support Methods Selected as Baselines

We select baselines from methods that have contributed to the development of supporter models, ensuring diverse characteristics. The explanations for each method are as follows:

• Ask-an-Expert (Zhang et al., 2023): This approach involves consulting an expert-role LLM at every turn of the conversation to obtain advice on the seeker’s emotional status, its cause, and potential solutions, which are then leveraged to generate the supporter’s response.

• ESCoT (Zhang et al., 2024): This method follows a chain-of-thought process, considering the seeker’s emotional state, emotion stimulus (the specific trigger of the emotion), the seeker’s personal interpretation of the stimulus, and the reasoning behind the selected support strategy before generating a supporter response.

• PPDP (Deng et al., 2024): This approach involves training a policy planner to select the optimal support strategy through two steps: (1) fine-tuning on the ESCoV dataset and (2) simulating and evaluating diverse conversations using three LLMs (supporter LLM, seeker LLM, and reward LLM), followed by reinforcement learning based on the rewards.

H Training Details for Emotion-Reinforced Models

This study investigates whether reinforcing values, rather than positive emotions, leads to more effective emotional support. To test this hypothesis, we adapt our methods by modifying the learning objective to prioritize promoting positive emotions in seekers. This approach requires two key components: a reference generator and a supporter model, both optimized for emotional reinforcement. Unlike the value-based method, this approach does not require a target value detector. The following subsections outline the training procedures for the reference generator and the supporter model.

H.1 Reference Generator

The reference generator is trained on supporter response from Reddit that successfully elicited positive emotional responses from OPs. This training approach ensures that the generated responses effectively foster positive emotions. Given the dialogue history \((o_1, c_1, o_2, c_2, ..., o_t)\), the model generates a supporter response \((c_t)\) as follows:

\[
c_t = \mathrm{LLM}_{\mathrm{RG}}(o_1, c_1, o_2, c_2, ..., o_t)
\]

Unlike our model, which incorporates both dialogue history and target values, this generator relies solely on dialogue history as input. Therefore, it employs only the SFT stage. Although the training data and prompts are consistent with those used for our model, all value-related information has been excluded from the reference generator’s training process.

H.2 Supporter Model

The supporter model for emotion reinforcement processes two inputs: the dialogue history and a reference response generated by the reference generator. At each turn, the model performs reasoning across four key aspects: (1) identifying the seeker’s issues and current emotional state, (2) analyzing the content of the reference response, (3) deciding whether to integrate the reference response, and (4) selecting an optimal emotional support strategy to generate the subsequent response. These reasoning aspects are identical to those used in our model.

The training process for the supporter model involves both SFT and DPO using data generated through simulations with a seeker simulator based on GPT-4o-mini.

SFT Stage. Similar to our approach, a dual-generation method is employed: GPT produces two responses per turn—one with references and one without—ensuring balanced training data within identical contexts. The simulation-generated data is then used to fine-tune Llama-3-8B-Instruct, with dataset sizes detailed in Table 15.

DPO Stage. This stage optimizes the supporter model to generate responses that more effectively promote positive emotions. The process uses simulated dialogues between the supporter model and a
<table>
  <tr>
    <th>Stage</th>
    <th>Supporter</th>
    <th>Train</th>
    <th>Dev</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>GPT-4o-mini</td>
    <td>24,580</td>
    <td>1,656</td>
  </tr>
  <tr>
    <td>DPO</td>
    <td>GPT-4o-mini</td>
    <td>2,610</td>
    <td>552</td>
  </tr>
</table>

Table 15: Dataset sizes for training the supporter model for positive emotion reinforcement generated through simulation. The ‘Supporter’ column refers to the supporter model used in the simulation.

seeker simulator to generate training data. For each turn, the supporter model produces two responses: one following its initial reference usage and another taking the opposite approach. Both responses undergo simulation to evaluate their emotional impact, with the more effective response marked as preferred. The cumulative reward for a supporter’s response at turn \( t \) (\( u_t^{\text{sup}} \)) is calculated as:

\[
R(u_t^{\text{sup}}) = \sum_{k=1}^{h} \gamma^{k-1} S_{t+k}(u_t^{\text{sup}})
\]

where \( S_t(u_t^{\text{sup}}) \) represents the emotion score at turn \( t \) calculated by GPT-4o-mini, \( h \) is the lookahead horizon (the number of future steps considered), and \( \gamma \) is a discount factor balancing immediate and future rewards. Response pairs are included in the DPO dataset when their reward difference exceeds the threshold \( T_{\text{diff}} \).

Emotion scores are calculated using prompts inspired by Deng et al. (2024). For each turn, GPT evaluates the seeker’s emotional state as “feels worse”, “feels the same”, “feels better”, or “the issue has been solved”. These responses are then mapped to scores of -1.0, -0.5, 0.5, and 1.0, respectively. Ten responses are collected for each turn, and the average score is used as the final emotion score. The prompts used for this process are provided in Table 37.

These simulations use GPT-4o-mini, and dataset sizes are summarized in Table 15.

I Evaluating the Success of Target Values Reinforcement

We engaged each model in conversations with a seeker simulator and analyzed the frequency of target values appearing in the seeker’s subsequent responses. The success rate was assessed based on valid turns—the number of future turns within which the target values remained relevant. For example, if the valid turn threshold was three, reinforcement was considered successful if the seeker’s response included the target values within the next three turns. To control for variations in dialogue length, we considered only cases where at least one positive seeker response occurred within the valid turns. This approach mitigated the influence of longer conversations, where negative responses might become more frequent and lower the success rate.

J Training Details for Generalization Capability Evaluation

To evaluate the generalization capability of our method, we conducted experiments using the Cactus dataset (Lee et al., 2024) instead of Reddit. Cactus is a counseling dataset based on Cognitive Behavioral Therapy (CBT), a therapeutic approach that helps individuals identify and modify negative thought patterns and behaviors. The conversations were generated using GPT-4o and validated through human evaluation to ensure their suitability as psychological counseling dialogues.

We utilized Cactus to train both the target value detector and the reference generator. Subsequently, we employed these models in a simulation to create SFT and DPO datasets, which were then used to train the supporter model. The details for each stage are outlined below.

J.1 Target Value Detector & Reference Generator

The Cactus dataset contains a total of 31,577 counseling dialogues. To consider resource efficiency, we selected 4,057 dialogues by excluding cases where the seeker’s intake form—including personal information, issues, history, and other details—was duplicated across conversations. As described in Section 4, we then used models developed by Liu et al. (2024) and Schroter et al. (2023) to label each seeker’s turn with sentiment strength and expressed values. These results were subsequently used to prepare the dataset for training the target value detector and reference generator. The final dataset consists of 11,000 instances for training and 1,631 instances each for validation and testing.

Following the approach outlined in Section 5, we trained the target value detector and reference generator separately. Unlike Reddit, where a single comment can receive multiple replies—leading to a branching conversation structure—the Cactus dataset follows a unidirectional conversational flow. Due to this structure, the DPO process was omitted during reference generator training.
<table>
  <tr>
    <th>Stage</th>
    <th>Supporter</th>
    <th>Train</th>
    <th>Dev</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>GPT-4o-mini</td>
    <td>38,543</td>
    <td>2,640</td>
  </tr>
  <tr>
    <td>DPO</td>
    <td>GPT-4o-mini</td>
    <td>1,270</td>
    <td>256</td>
  </tr>
</table>

Table 16: Dataset sizes for supporter model training, generated through simulation using Cactus-based target value detector and reference generator. The ‘Supporter’ column refers to the supporter model used in the simulation.

<table>
  <tr>
    <th>Category</th>
    <th>Metric</th>
    <th>Corr.</th>
  </tr>
  <tr>
    <td rowspan="9">ES-Skills</td>
    <td>Identification</td>
    <td>0.422*</td>
  </tr>
  <tr><td>Comforting</td><td>0.322*</td></tr>
  <tr><td>Suggestions</td><td>0.421*</td></tr>
  <tr><td>Experience</td><td>0.778*</td></tr>
  <tr><td>Informativeness</td><td>0.282*</td></tr>
  <tr><td>Consistency</td><td>0.351*</td></tr>
  <tr><td>Role-Adherence</td><td>0.235†</td></tr>
  <tr><td>Expression</td><td>0.198</td></tr>
  <tr><td>Humanness</td><td>0.202</td></tr>
  <tr>
    <td rowspan="2">ES-Value</td>
    <td>Seeker</td>
    <td>0.332*</td>
  </tr>
  <tr>
    <td>Supporter</td>
    <td>0.413*</td>
  </tr>
</table>

Table 17: Spearman’s rank correlation between expert and GPT-generated scores. Significant correlations are marked with * (p-value < 0.05) and † (p-value < 0.1).

J.2 Supporter Model
We trained the supporter model in two stages, SFT and DPO, following the approach introduced in Section 5.3. The training dataset was generated through interactions between GPT-4o-mini-based supporter and seeker simulators. The key difference from Section 5.3 is that the target value detector and reference generator used for simulation were trained on the Cactus dataset instead of Reddit. Additionally, when constructing the dataset for DPO, we adopted the hyperparameter settings (\( h = 3, \gamma = 1, T_{\text{diff}} = 2 \)) that achieved the best performance on the Reddit dataset. Detailed information on dataset size is provided in Table 16.

K Validation of Automated Evaluation Models
To evaluate the performance of GPT-4o-mini in assessing ES-Skills and ES-Value, we analyzed the correlation between expert evaluation scores and GPT-generated scores. For this purpose, we randomly select and evaluate 60 dialogues generated by our models and baselines. For ES-Value, we compared individual dialogue scores generated by GPT and expert evaluations, rather than using a pairwise scoring approach, to enable a more straightforward comparison. As shown in Table 17, significant correlations were observed across most metrics, except for Expression and Humanness. These findings suggest that automated evaluation models can reliably approximate human assessments of emotional support, conversational quality, and value reinforcement, supporting the validity of our experimental results.

L Details of Experiments on Self-Disclosure
Self-disclosure—sharing experiences related to those of the seeker—is a key strategy in emotional support for fostering intimacy and reducing stress (Cheng et al., 2024; Meng and Dai, 2021). The ability to share such experiences has been used as an evaluation criterion for emotional support systems (Zhang et al., 2023). However, some users might feel uncomfortable when dialogue systems present these experiences as personal.

To better understand the impact of self-disclosure on emotional support systems, we investigated two alternative approaches: (1) removing it entirely and substituting the next most probable strategy in the supporter’s reasoning process (Section 5.3); and (2) rephrasing self-disclosure responses to frame them as experiences of others, using GPT-4o-mini.

As shown in Table 27 and Table 28, the models consistently exhibited declines in overall ES-Skills and ES-Value when self-disclosure was modified or removed. The metric most affected was Experience, with related metrics such as Suggestions and Informativeness also showing performance drops.

The results reinforces the importance of self-disclosure in emotional support but, at the same time, highlight the need for research on more sophisticated methods for experience sharing. For example, analyzing different types of self-disclosure and developing alternative strategies based on seeker perceptions could offer meaningful improvements. This detailed investigation will be left for future work.

M Detailed Analysis of Value Reinforcement Performance
To identify areas for improvement in our DPO model’s value reinforcement, we conducted a detailed analysis. First, we evaluated value reinforcement scores from both seeker and supporter per-
<table>
  <tr>
    <th>Category</th>
    <th>Freq.</th>
  </tr>
  <tr>
    <td>Strengths and Achievements Acknowledgment</td>
    <td>30</td>
  </tr>
  <tr>
    <td>Exploration of Issues and Challenges</td>
    <td>23</td>
  </tr>
  <tr>
    <td>Self-Compassion and Acceptance</td>
    <td>19</td>
  </tr>
  <tr>
    <td>Exploration of Personal Interests</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Emotional Resilience and Coping Strategies</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Exploration of Goals and Motivations</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Motivation and Alignment with Goals</td>
    <td>1</td>
  </tr>
</table>

Table 18: Frequency of areas for improvement in value reinforcement.

spectives using GPT-4o-mini on a 5-point scale. Next, we analyzed 40 dialogues that received the lowest score of 4 to identify potential improvement areas. This analysis involved reasoning with GPT-4o-mini and categorizing the areas, as summarized in Table 40. Subsequently, GPT-4o-mini was used again to assign up to three relevant categories to each of the 40 dialogues. The frequency of issues for each category is detailed in Table 18.
<table>
  <tr>
    <th>System</th>
    <td>Select and return up to 3 values to reinforce in the patient for effective emotional support.</td>
  </tr>
  <tr>
    <th>User</th>
    <td>Human values: <i>{List of 20 human values}</i><br><br>The dialogue history below is a conversation between a patient experiencing emotional difficulties and a therapist providing support. For effective emotional support, which values should be reinforced in the patient so that they are expressed more frequently in the future? Select up to 3 values from the list provided above. Answer in the format 'value1, value2, value3' separated by commas without any additional explanation.<br><br>Dialogue history: <i>{Dialogue history}</i></td>
  </tr>
</table>

Table 19: Training prompts for the target value detector.

<table>
  <tr>
    <th>System</th>
    <td>You will take on the role of a therapist to help a patient with emotional difficulties, aiming to reduce their distress and support them in overcoming their challenges.</td>
  </tr>
  <tr>
    <th>User</th>
    <td>1. Dialogue history: <i>{Thread history}</i><br><br>2. Target values:<br><i>{Information on the target values}</i><br><br>As a therapist supporting a patient with emotional difficulties, your goal is to reduce their distress and guide them through challenges. The target values are those that are expected to be more frequently expressed by the patient. Generate the next turn of the utterance based on the dialogue history, aiming to reinforce these target values in the patient.</td>
  </tr>
</table>

Table 20: Training prompts for the reference generator. The target values information includes the definition of each value along with the set of contained values.

<table>
  <tr>
    <th>System</th>
    <td>Determine which of the two comments generated by each model is more similar to the ground truth comment.</td>
  </tr>
  <tr>
    <th>User</th>
    <td>Thread: <i>{Thread history}</i><br><br>Ground truth comment: <i>{GT comment}</i><br><br>The above includes the thread history and the corresponding ground truth comment, which continues the thread. Below are two comments generated by different models. First, provide reasoning for your evaluation, and then select the comment that is more similar in content to the ground truth comment.<br><br>Comment A: <i>{Response from model A}</i><br>Comment B: <i>{Response from model B}</i><br><br>[Template]<br>Reasoning:<br>Similar comment: Answer with either 'Comment A' or 'Comment B' only</td>
  </tr>
</table>

Table 21: Prompts for evaluating the performance of the reference response.
System    You will take on the role of a therapist to help a patient with emotional difficulties, aiming to reduce their distress and support them in overcoming their challenges.

User    1. Strategies for emotional support: {Definition of 8 emotional support strategies}
        2. Dialogue history: {Dialogue history}
        3. Target values: {Information on the target values}
        4. Reference response: {Reference response}

As a therapist supporting a patient with emotional difficulties, your goal is to reduce their distress and guide them through challenges. The target values are those that are expected to be more frequently expressed by the patient. You need to generate the therapist’s next utterance based on the dialogue history, aiming to reinforce these target values in the patient.

The therapist’s next utterance should follow these guidelines:
- Use only one sentence without any extra explanation, framing, introductory phrases, or meta-commentary
- Avoid directly mentioning the target values, but focus on reinforcing them through your guidance.
- If the patient shows signs of improvement in the dialogue history, acknowledge their progress and guide the conversation to an efficient close.
- Do not repeat similar messages from previous therapist utterances in the dialogue history.

The reference response is a therapist’s reply given to another patient in a similar situation, which you can use as a reference for generating your next response. Before generating the therapist’s response to satisfy the above conditions, thoroughly analyze the following:
Step 1. Understanding the patient’s issues and current state
- What is the patient’s issue?
- Have their situation and the causes of their emotions been sufficiently explored? If not, what additional information should be obtained to deeply understand them?
- What is the patient’s current emotional state? How have the patient’s emotions or thoughts changed through the conversation?

Step 2. Identifying the key points of the reference response
- What is the main message in the referenced response (item 4)?

Step 3. Determination of reference response usage
- Would using a reference response be helpful for generating the next therapist utterance? Why or why not?
- If a reference response is used, how would it be applied, and if it is not used, what alternative message would be provided?

Step 4. Therapist’s next strategy and response
- Based on the above (Step 1-Step3), what emotional support strategy should be used, and what message should you convey to the patient in the next response?

You should respond in the following template format:
Step 1. Understanding the patient’s issues and current state
-Reasoning: (the result of your analysis)
Step 2. Identifying the key points of the reference response
-Reasoning: (the result of your analysis)
Step 3. Determination of reference response usage
-Reasoning: (The result of your analysis, starting with whether to use the reference response — 'Yes' or 'No')
Step 4. Therapist’s next strategy and response
-Strategy: (choose one emotional support strategy for the next turn based on the reasoning)
-Response: (only the therapist’s next utterance without any explanation)

Table 22: Training prompts for the supporter model.
<table>
  <tr>
    <th rowspan="2">Supporter Model</th>
    <th rowspan="2">h</th>
    <th rowspan="2">γ</th>
    <th rowspan="2">\( T_{diff} \)</th>
    <th colspan="3">Train</th>
    <th colspan="3">Dev</th>
  </tr>
  <tr>
    <th># of Data</th>
    <th colspan="2">Chosen</th>
    <th># of Data</th>
    <th colspan="2">Chosen</th>
  </tr>
  <tr>
    <td>GPT</td>
    <td>All</td>
    <td>1</td>
    <td>1</td>
    <td>2,561</td>
    <td>1,168</td>
    <td>855</td>
    <td>458</td>
    <td>208</td>
    <td>149</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>2</td>
    <td>2,023</td>
    <td>881</td>
    <td>623</td>
    <td>357</td>
    <td>268</td>
    <td>149</td>
  </tr>
  <tr>
    <td></td>
    <td>All</td>
    <td>0.9</td>
    <td>1</td>
    <td>1,712</td>
    <td>947</td>
    <td>765</td>
    <td>298</td>
    <td>164</td>
    <td>134</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>1.5</td>
    <td>1,345</td>
    <td>735</td>
    <td>610</td>
    <td>229</td>
    <td>117</td>
    <td>112</td>
  </tr>
  <tr>
    <td></td>
    <td>3</td>
    <td>1</td>
    <td>1</td>
    <td>1,796</td>
    <td>1,127</td>
    <td>669</td>
    <td>319</td>
    <td>201</td>
    <td>118</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>2</td>
    <td>1,144</td>
    <td>724</td>
    <td>420</td>
    <td>210</td>
    <td>132</td>
    <td>78</td>
  </tr>
  <tr>
    <td></td>
    <td>5</td>
    <td>1</td>
    <td>1</td>
    <td>2,015</td>
    <td>1,301</td>
    <td>714</td>
    <td>360</td>
    <td>237</td>
    <td>123</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>2</td>
    <td>1,438</td>
    <td>955</td>
    <td>483</td>
    <td>247</td>
    <td>165</td>
    <td>82</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>All</td>
    <td>1</td>
    <td>1</td>
    <td>3,301</td>
    <td>1,255</td>
    <td>1,106</td>
    <td>628</td>
    <td>263</td>
    <td>206</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>2</td>
    <td>2,361</td>
    <td>920</td>
    <td>800</td>
    <td>469</td>
    <td>182</td>
    <td>149</td>
  </tr>
  <tr>
    <td></td>
    <td>All</td>
    <td>0.9</td>
    <td>1</td>
    <td>1,975</td>
    <td>979</td>
    <td>996</td>
    <td>407</td>
    <td>209</td>
    <td>198</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>1.5</td>
    <td>1,556</td>
    <td>777</td>
    <td>779</td>
    <td>318</td>
    <td>153</td>
    <td>165</td>
  </tr>
  <tr>
    <td></td>
    <td>3</td>
    <td>1</td>
    <td>1</td>
    <td>1,825</td>
    <td>1,122</td>
    <td>703</td>
    <td>375</td>
    <td>226</td>
    <td>149</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>2</td>
    <td>1,117</td>
    <td>663</td>
    <td>454</td>
    <td>239</td>
    <td>153</td>
    <td>116</td>
  </tr>
  <tr>
    <td></td>
    <td>5</td>
    <td>1</td>
    <td>1</td>
    <td>2,186</td>
    <td>1,360</td>
    <td>826</td>
    <td>456</td>
    <td>281</td>
    <td>175</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>2</td>
    <td>1,489</td>
    <td>919</td>
    <td>370</td>
    <td>172</td>
    <td>116</td>
    <td>56</td>
  </tr>
</table>

Table 23: Overview of the DPO dataset categorized by the supporter model used for simulations and variations in reward calculation hyperparameters (\( h \): look-ahead horizon, \( \gamma \): discount factor, \( T_{diff} \): difference threshold). The “Chosen” column represents the distribution of chosen responses selected between the model’s initial and alternative outputs.

<table>
  <tr>
    <th rowspan="2">Categories</th>
    <th rowspan="2">Models</th>
    <th rowspan="2">h</th>
    <th rowspan="2">γ</th>
    <th rowspan="2">\( T_{diff} \)</th>
    <th colspan="7">ES-Skills†</th>
    <th rowspan="2">ES-Intensity‡</th>
  </tr>
  <tr>
    <th>Iden.</th>
    <th>Comf.</th>
    <th>Sugg.</th>
    <th>Expe.</th>
    <th>Info.</th>
    <th>Cons.</th>
    <th>Role.</th>
    <th>Expr.</th>
    <th>Huma.</th>
    <th>Over.</th>
  </tr>
  <tr>
    <td>Prompt-Based</td>
    <td>GPT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>4.83</td>
    <td>4.92</td>
    <td>4.57*</td>
    <td>3.11*</td>
    <td>4.42*</td>
    <td>5.00</td>
    <td>5.00</td>
    <td>4.10*</td>
    <td>4.70</td>
    <td>4.72*</td>
    <td>1.89</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>4.87</td>
    <td>4.91</td>
    <td>4.43*</td>
    <td>2.91*</td>
    <td>4.47</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.03*</td>
    <td>4.63*</td>
    <td>4.68*</td>
    <td>1.99‡</td>
  </tr>
  <tr>
    <td>ES Datasets</td>
    <td>Llama-Reddit</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>3.38*</td>
    <td>3.74*</td>
    <td>3.21*</td>
    <td>2.59*</td>
    <td>2.99*</td>
    <td>3.94*</td>
    <td>4.35*</td>
    <td>3.37*</td>
    <td>3.81*</td>
    <td>3.40*</td>
    <td>1.97</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-ESConv</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>4.35*</td>
    <td>4.43*</td>
    <td>4.06*</td>
    <td>2.65*</td>
    <td>3.88*</td>
    <td>4.82*</td>
    <td>4.97*</td>
    <td>3.79*</td>
    <td>4.25*</td>
    <td>4.22*</td>
    <td>1.87</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-ExTES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>4.83</td>
    <td>4.90</td>
    <td>4.53*</td>
    <td>2.71*</td>
    <td>4.44*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.02*</td>
    <td>4.59*</td>
    <td>4.66*</td>
    <td>1.67‡</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-Psych8k</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>4.84</td>
    <td>4.85*</td>
    <td>4.75</td>
    <td>2.89*</td>
    <td>4.63</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.05*</td>
    <td>4.57*</td>
    <td>4.75</td>
    <td>1.53‡</td>
  </tr>
  <tr>
    <td>ES Methods</td>
    <td>Ask-an-Expert</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>4.13*</td>
    <td>4.30*</td>
    <td>3.93*</td>
    <td>3.12*</td>
    <td>3.70*</td>
    <td>4.61*</td>
    <td>4.91*</td>
    <td>3.74*</td>
    <td>4.21*</td>
    <td>4.08*</td>
    <td>1.86</td>
  </tr>
  <tr>
    <td></td>
    <td>ESCoT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>3.69*</td>
    <td>3.91*</td>
    <td>3.16*</td>
    <td>1.81*</td>
    <td>3.07*</td>
    <td>4.16*</td>
    <td>4.81*</td>
    <td>2.95*</td>
    <td>3.64*</td>
    <td>3.51*</td>
    <td>2.25‡</td>
  </tr>
  <tr>
    <td></td>
    <td>PPDPP</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>4.64*</td>
    <td>4.88</td>
    <td>4.45*</td>
    <td>2.49*</td>
    <td>4.26*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>3.99*</td>
    <td>4.54*</td>
    <td>4.54*</td>
    <td>1.83</td>
  </tr>
  <tr>
    <td>Emotion</td>
    <td>SFT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>4.83</td>
    <td>4.91</td>
    <td>4.51</td>
    <td>3.64</td>
    <td>4.43</td>
    <td>4.97</td>
    <td>4.99</td>
    <td>4.16</td>
    <td>4.67</td>
    <td>4.73</td>
    <td>1.97</td>
  </tr>
  <tr>
    <td>Reinforced</td>
    <td>DPO (GPT)</td>
    <td>3</td>
    <td>1</td>
    <td>0.5</td>
    <td>4.85</td>
    <td>4.92</td>
    <td>4.74</td>
    <td>4.05</td>
    <td>4.61</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.33</td>
    <td>4.78</td>
    <td>4.82</td>
    <td>1.86</td>
  </tr>
  <tr>
    <td></td>
    <td>5</td>
    <td>1</td>
    <td>0.5</td>
    <td>4.85</td>
    <td><b>4.95</b></td>
    <td>4.68</td>
    <td>3.80</td>
    <td>4.58</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.28</td>
    <td>4.77</td>
    <td>4.81</td>
    <td>1.82</td>
  </tr>
  <tr>
    <td>ES-VR (Ours)</td>
    <td>SFT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>4.85</td>
    <td>4.90</td>
    <td>4.72</td>
    <td>3.76</td>
    <td>4.56</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.25</td>
    <td>4.73</td>
    <td>4.80</td>
    <td>1.86</td>
  </tr>
  <tr>
    <td></td>
    <td>DPO (GPT)</td>
    <td>All</td>
    <td>1</td>
    <td>1</td>
    <td>4.89</td>
    <td>4.92</td>
    <td>4.75</td>
    <td>3.71</td>
    <td>4.63</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.24</td>
    <td>4.72</td>
    <td>4.80</td>
    <td>1.90</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>All</td>
    <td>1</td>
    <td>2</td>
    <td>4.85</td>
    <td>4.91</td>
    <td>4.73</td>
    <td>3.89</td>
    <td>4.63*</td>
    <td>5.00</td>
    <td>5.00</td>
    <td>4.27</td>
    <td>4.76</td>
    <td>4.80</td>
    <td>1.90</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>All</td>
    <td>0.9</td>
    <td>1</td>
    <td>4.84</td>
    <td>4.87</td>
    <td>4.72</td>
    <td>3.72</td>
    <td>4.61</td>
    <td>4.98</td>
    <td>5.00</td>
    <td>4.23</td>
    <td>4.72</td>
    <td>4.78</td>
    <td>1.85</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>All</td>
    <td>0.9</td>
    <td>1.5</td>
    <td>4.89</td>
    <td>4.93</td>
    <td>4.71</td>
    <td>3.79</td>
    <td>4.59</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.28</td>
    <td>4.76</td>
    <td>4.83*</td>
    <td>1.88</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>3</td>
    <td>1</td>
    <td>1</td>
    <td>4.87</td>
    <td>4.90</td>
    <td>4.77</td>
    <td>3.58</td>
    <td>4.63</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.26</td>
    <td>4.74</td>
    <td>4.79</td>
    <td>1.87</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>3</td>
    <td>1</td>
    <td>2</td>
    <td><b>4.91*</b></td>
    <td>4.93</td>
    <td><b>4.78*</b></td>
    <td>3.61</td>
    <td>4.65*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.23</td>
    <td>4.72</td>
    <td>4.85*</td>
    <td>1.94</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>5</td>
    <td>1</td>
    <td>1</td>
    <td>4.88*</td>
    <td>4.93</td>
    <td>4.77*</td>
    <td>3.62</td>
    <td>4.64*</td>
    <td>5.00</td>
    <td>5.00</td>
    <td>4.25</td>
    <td>4.73</td>
    <td>4.83</td>
    <td>1.84</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>5</td>
    <td>1</td>
    <td>2</td>
    <td>4.88</td>
    <td>4.94</td>
    <td>4.73</td>
    <td>3.64</td>
    <td>4.66*</td>
    <td>5.00</td>
    <td>5.00</td>
    <td>4.24</td>
    <td>4.75</td>
    <td>4.84*</td>
    <td>1.83</td>
  </tr>
  <tr>
    <td></td>
    <td>DPO (SFT)</td>
    <td>All</td>
    <td>1</td>
    <td>1</td>
    <td>4.83</td>
    <td>4.89</td>
    <td>4.71</td>
    <td>3.75</td>
    <td>4.60</td>
    <td>4.98</td>
    <td>4.99*</td>
    <td>4.21*</td>
    <td>4.72</td>
    <td>4.78</td>
    <td>1.93</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>All</td>
    <td>1</td>
    <td>2</td>
    <td>4.89</td>
    <td><b>4.95</b></td>
    <td>4.77</td>
    <td>3.76</td>
    <td>4.66*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.26</td>
    <td>4.75</td>
    <td>4.83</td>
    <td>1.87</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>All</td>
    <td>0.9</td>
    <td>1</td>
    <td>4.86</td>
    <td>4.93*</td>
    <td>4.72</td>
    <td>3.72</td>
    <td>4.64*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.26</td>
    <td>4.73</td>
    <td>4.82*</td>
    <td>1.87</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>All</td>
    <td>0.9</td>
    <td>1.5</td>
    <td>4.85</td>
    <td>4.92</td>
    <td>4.73</td>
    <td>3.79</td>
    <td>4.59</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.28</td>
    <td>4.74</td>
    <td>4.81</td>
    <td>1.86</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>3</td>
    <td>1</td>
    <td>1</td>
    <td>4.86</td>
    <td>4.91</td>
    <td>4.71</td>
    <td>3.38</td>
    <td>4.57</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.24</td>
    <td>4.71</td>
    <td>4.78</td>
    <td>1.89</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>3</td>
    <td>1</td>
    <td>2</td>
    <td><b>4.90*</b></td>
    <td><b>4.95</b></td>
    <td><b>4.80*</b></td>
    <td>3.85</td>
    <td>4.69*</td>
    <td>5.00</td>
    <td>5.00</td>
    <td>4.30</td>
    <td>4.77</td>
    <td>4.87*</td>
    <td>1.75</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>5</td>
    <td>1</td>
    <td>1</td>
    <td>4.85</td>
    <td>4.90</td>
    <td>4.69</td>
    <td>3.76</td>
    <td>4.56</td>
    <td>4.98</td>
    <td>5.00</td>
    <td>4.23</td>
    <td>4.74</td>
    <td>4.78</td>
    <td>1.86</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>5</td>
    <td>1</td>
    <td>2</td>
    <td>4.83</td>
    <td>4.91</td>
    <td>4.70</td>
    <td>3.66</td>
    <td>4.61</td>
    <td>4.98</td>
    <td>5.00</td>
    <td>4.23</td>
    <td>4.74</td>
    <td>4.79</td>
    <td>1.91</td>
  </tr>
</table>

Table 24: Comparison of models based on ES-Skills and ES-Intensity. Statistically significant differences compared to our SFT model are marked with * (p-value < 0.05), and differences with p-value < 0.1 are marked with †, as determined by the Mann-Whitney U test.
<table>
  <tr>
    <th rowspan="2">Categories</th>
    <th rowspan="2">Models</th>
    <th rowspan="2">\( h \)</th>
    <th rowspan="2">\( \gamma \)</th>
    <th rowspan="2">\( T_{\text{dist}} \)</th>
    <th colspan="2">SFT</th>
    <th colspan="2">DPO (GPT)</th>
    <th colspan="2">DPO (SFT)</th>
  </tr>
  <tr>
    <th>Seeker</th>
    <th>Supporter</th>
    <th>Seeker</th>
    <th>Supporter</th>
    <th>Seeker</th>
    <th>Supporter</th>
  </tr>
  <tr>
    <td>Prompt-Based</td>
    <td>GPT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.49†</td>
    <td>0.42*</td>
    <td>0.49*</td>
    <td>0.42*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.50</td>
    <td>0.51</td>
    <td>0.48</td>
    <td>0.44*</td>
    <td>0.46†</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>ES Datasets</td>
    <td>Llama-Reddit</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.30*</td>
    <td>0.10*</td>
    <td>0.28*</td>
    <td>0.07*</td>
    <td>0.29*</td>
    <td>0.09*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-ESConv</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.37*</td>
    <td>0.21*</td>
    <td>0.35*</td>
    <td>0.17*</td>
    <td>0.37*</td>
    <td>0.19*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-EXTES</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.51</td>
    <td>0.54*</td>
    <td>0.50</td>
    <td>0.52</td>
    <td>0.48†</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-Psych8k</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.50</td>
    <td>0.64*</td>
    <td>0.48*</td>
    <td>0.58*</td>
    <td>0.49</td>
    <td>0.62*</td>
  </tr>
  <tr>
    <td>ES Methods</td>
    <td>Ask-an-Expert</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.34*</td>
    <td>0.18*</td>
    <td>0.34*</td>
    <td>0.14*</td>
    <td>0.32*</td>
    <td>0.15*</td>
  </tr>
  <tr>
    <td></td>
    <td>ESCoT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.28*</td>
    <td>0.06*</td>
    <td>0.25*</td>
    <td>0.04*</td>
    <td>0.25*</td>
    <td>0.05*</td>
  </tr>
  <tr>
    <td></td>
    <td>PPDPP</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.47*</td>
    <td>0.37*</td>
    <td>0.46*</td>
    <td>0.27*</td>
    <td>0.44*</td>
    <td>0.31*</td>
  </tr>
  <tr>
    <td>Emotion-Reinforced</td>
    <td>SFT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.50</td>
    <td>0.51</td>
    <td>0.50</td>
    <td>0.43*</td>
    <td>0.49</td>
    <td>0.46†</td>
  </tr>
  <tr>
    <td></td>
    <td>DPO (GPT)</td>
    <td>3</td>
    <td>1</td>
    <td>0.5</td>
    <td>0.52</td>
    <td>0.55*</td>
    <td>0.49</td>
    <td>0.49</td>
    <td>0.49</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>5</td>
    <td>1</td>
    <td>0.5</td>
    <td>0.53*</td>
    <td>0.59*</td>
    <td>0.49*</td>
    <td>0.44*</td>
    <td>0.49</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>ES-VR (Ours)</td>
    <td>SFT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.48†</td>
    <td>0.44*</td>
    <td>0.48†</td>
    <td>0.46†</td>
  </tr>
</table>

Table 25: ES-Value performance for models evaluated from the perspectives of seekers and supporters. The scores represent the mean win-ratio of baselines compared to our models—SFT, DPO. Statistically significant differences compared to our models are marked with * (p-value < 0.05), and differences with p-value < 0.1 are marked with †, as determined by the Mann-Whitney U test.

<table>
  <tr>
    <th rowspan="2">Categories</th>
    <th rowspan="2">Models</th>
    <th colspan="2">ES-Value†</th>
  </tr>
  <tr>
    <th>Seeker</th>
    <th>Supporter</th>
  </tr>
  <tr>
    <td>Prompt-Based</td>
    <td>GPT</td>
    <td>0.46*</td>
    <td>0.28*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama</td>
    <td>0.48</td>
    <td>0.42*</td>
  </tr>
  <tr>
    <td>ES Datasets</td>
    <td>Llama-Reddit</td>
    <td>0.24*</td>
    <td>0.04*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-ESConv</td>
    <td>0.30*</td>
    <td>0.11*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-EXTES</td>
    <td>0.44*</td>
    <td>0.37*</td>
  </tr>
  <tr>
    <td></td>
    <td>Llama-Psych8k</td>
    <td>0.46*</td>
    <td>0.45*</td>
  </tr>
  <tr>
    <td>ES Methods</td>
    <td>Ask-an-Expert</td>
    <td>0.28*</td>
    <td>0.08*</td>
  </tr>
  <tr>
    <td></td>
    <td>ESCoT</td>
    <td>0.24*</td>
    <td>0.03*</td>
  </tr>
  <tr>
    <td></td>
    <td>PPDPP</td>
    <td>0.40*</td>
    <td>0.20*</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>DPO (Cactus)</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 26: Comparison of models based on ES-Value—the win ratio of each model against DPO (Cactus). A value below 0.5 indicates that DPO (Cactus) outperformed the baseline models, while a value above 0.5 suggests that the baselines performed better. Statistically significant differences compared to DPO (Cactus) are marked with * (p-value < 0.05), and differences with p-value < 0.1 are marked with †, as determined by the Mann-Whitney U test.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="8">ES-Skills†</th>
    <th rowspan="2">ES-Intensity↓</th>
  </tr>
  <tr>
    <th>Iden.</th>
    <th>Comf.</th>
    <th>Sugg.</th>
    <th>Expe.</th>
    <th>Info.</th>
    <th>Cons.</th>
    <th>Role.</th>
    <th>Expr.</th>
    <th>Huma.</th>
    <th>Over.</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>4.85</td>
    <td>4.90</td>
    <td>4.72</td>
    <td>3.76</td>
    <td>4.56</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.25</td>
    <td>4.73</td>
    <td>4.80</td>
    <td>1.86</td>
  </tr>
  <tr>
    <td>- Next strategy</td>
    <td>4.81</td>
    <td>4.89</td>
    <td>4.57*</td>
    <td>2.79*</td>
    <td>4.43*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.05*</td>
    <td>4.55*</td>
    <td>4.68*</td>
    <td>1.89</td>
  </tr>
  <tr>
    <td>- Others’ experience</td>
    <td>4.76</td>
    <td>4.88</td>
    <td>4.57*</td>
    <td>3.36*</td>
    <td>4.43*</td>
    <td>4.98</td>
    <td>5.00</td>
    <td>4.04*</td>
    <td>4.63*</td>
    <td>4.70*</td>
    <td>1.89</td>
  </tr>
  <tr>
    <td>DPO (GPT)</td>
    <td>4.91</td>
    <td>4.93</td>
    <td>4.78</td>
    <td>3.61</td>
    <td>4.65</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.23</td>
    <td>4.72</td>
    <td>4.85</td>
    <td>1.94</td>
  </tr>
  <tr>
    <td>- Next strategy</td>
    <td>4.88</td>
    <td>4.92</td>
    <td>4.74</td>
    <td>2.87*</td>
    <td>4.57†</td>
    <td>5.00</td>
    <td>5.00</td>
    <td>4.08*</td>
    <td>4.59*</td>
    <td>4.77*</td>
    <td>1.90</td>
  </tr>
  <tr>
    <td>- Others’ experience</td>
    <td>4.85</td>
    <td>4.91</td>
    <td>4.70†</td>
    <td>3.28*</td>
    <td>4.62</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.11*</td>
    <td>4.67†</td>
    <td>4.80</td>
    <td>1.82</td>
  </tr>
  <tr>
    <td>DPO (SFT)</td>
    <td>4.90</td>
    <td>4.95</td>
    <td>4.80</td>
    <td>3.85</td>
    <td>4.69</td>
    <td>5.00</td>
    <td>5.00</td>
    <td>4.30</td>
    <td>4.77</td>
    <td>4.87</td>
    <td>1.75</td>
  </tr>
  <tr>
    <td>- Next strategy</td>
    <td>4.81*</td>
    <td>4.91</td>
    <td>4.60*</td>
    <td>2.90*</td>
    <td>4.43*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.06*</td>
    <td>4.55*</td>
    <td>4.70*</td>
    <td>1.92*</td>
  </tr>
  <tr>
    <td>- Others’ experience</td>
    <td>4.80*</td>
    <td>4.91</td>
    <td>4.56*</td>
    <td>3.39*</td>
    <td>4.47*</td>
    <td>4.99</td>
    <td>5.00</td>
    <td>4.07*</td>
    <td>4.65*</td>
    <td>4.70*</td>
    <td>1.98*</td>
  </tr>
</table>

Table 27: Comparison of ES-Skills and ES-Intensity performance based on the inclusion of self-disclosure. Statistically significant differences compared to our models are marked with * (p-value < 0.05), and differences with p-value < 0.1 are marked with †, as determined by the Mann-Whitney U test.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">SFT</th>
    <th colspan="2">DPO (GPT)</th>
    <th colspan="2">DPO (SFT)</th>
  </tr>
  <tr>
    <th>Seeker</th>
    <th>Supporter</th>
    <th>Seeker</th>
    <th>Supporter</th>
    <th>Seeker</th>
    <th>Supporter</th>
  </tr>
  <tr>
    <td>- Next strategy</td>
    <td>0.50</td>
    <td>0.52</td>
    <td>0.50</td>
    <td>0.53†</td>
    <td>0.48*</td>
    <td>0.45*</td>
  </tr>
  <tr>
    <td>- Others’ experience</td>
    <td>0.50</td>
    <td>0.50</td>
    <td>0.50</td>
    <td>0.50</td>
    <td>0.48†</td>
    <td>0.47</td>
  </tr>
</table>

Table 28: ES-Value performance for models evaluated from the perspectives of seekers and supporters, considering the impact of self-disclosure on performance. The scores represent the mean win-ratio of baselines compared to our models. Statistically significant differences compared to our models are marked with * (\( p \)-value \(<\) 0.05), and differences with \( p \)-value \(<\) 0.1 are marked with †, as determined by the Mann-Whitney U test.

<table>
  <tr>
    <th>Category</th>
    <th>Subcategory</th>
  </tr>
  <tr>
    <td>Romantic Relationship Challenges</td>
    <td>Breakups or divorce<br>Starting a romantic relationship<br>Challenges in establishing a marriage<br>Communication difficulties in relationships</td>
  </tr>
  <tr>
    <td>Family Dynamics and Conflicts</td>
    <td>Financial issues within the family<br>Sibling rivalry or family disputes<br>Challenges in parenthood and parenting<br>Coping with loss or grief of a family member</td>
  </tr>
  <tr>
    <td>Friendship and Interpersonal Challenges</td>
    <td>Difficulty adapting to new social environments<br>Challenges in maintaining friendships<br>Conflicts with friends</td>
  </tr>
  <tr>
    <td>Career and Work-Related Challenges</td>
    <td>Work-related stress and burnout<br>Job loss or career setbacks<br>Adjusting to a new job or role<br>Concerns about salary and bonuses<br>Dissatisfaction with current job<br>Stress related to unemployment<br>Ongoing depression</td>
  </tr>
  <tr>
    <td>Academic and Educational Stress</td>
    <td>Dissatisfaction with current school or major<br>Concerns about academic performance<br>Stress related to studies<br>Difficulty entering higher education<br>Lack or excess of motivation to study</td>
  </tr>
  <tr>
    <td>Self-Esteem, Identity, and Personal Growth</td>
    <td>Issues with self-esteem and confidence<br>Searching for meaning and purpose in life<br>Cultural identity and sense of belonging<br>Concerns about body image</td>
  </tr>
</table>

Table 29: Overview of seekers’ problem categories and subcategories.
<table>
  <tr>
    <th>System</th>
    <th>Generate appropriate situations that require emotional support, using the given topic and value information.</th>
  </tr>
  <tr>
    <td>User</td>
    <td>
      1. Emotional support topic: <i>{Problem category}</i><br>
      - <i>{Subcategory 1}</i><br>
      - <i>{Subcategory 2}</i><br>
      - <i>{Subcategory 3}</i><br><br>
      2. Supported value: <i>{Human value}</i><br>
      - Definition: <i>{Definition of the human value}</i><br>
      - Contained values: <i>{Contained value 1}</i>, <i>{Contained value 2}</i>, <i>{Contained value 3}</i><br><br>
      Define specific situations that individuals who prioritize the given human value (item 2) might face related to the presented emotional support topic (item 1). Generate a minimum of 10 and a maximum of 30 diverse and non-overlapping situations. Write from the perspective of an individual in need of emotional support, including 'I' as the subject, and be as specific as possible. Each situation should be one sentence (e.g., I just moved in this week, and it’s so hard to make friends.) Do not provide any additional explanations and separate each situation with a newline character ('\n').
    </td>
  </tr>
</table>

Table 30: Prompts for generating seeker situations based on problem category and human value combinations.

<table>
  <tr>
    <th>System</th>
    <th>Evaluate how much each situation aligns with the given value.</th>
  </tr>
  <tr>
    <td>User</td>
    <td>
      1. Situations: <i>{Generated situations}</i><br><br>
      2. Supported value: <i>{Human value}</i><br>
      - Definition: <i>{Definition of the human value}</i><br>
      - Contained values: <i>{Contained value 1}</i>, <i>{Contained value 2}</i>, <i>{Contained value 3}</i><br><br>
      Rate the alignment of each situation with the given value on a scale of 1-5, using the criteria below to guide your assessment:<br>
      - 1: The situation does not reflect any connection to the given value. The individual’s concerns or actions are entirely unrelated to the principles of this value.<br>
      - 2: The situation has a minimal or indirect connection to the value. It suggests the presence of the value but lacks a clear emphasis or relevance.<br>
      - 3: The situation shows some aspects of the value but not as a central theme. The value is present, but other priorities seem equally important.<br>
      - 4: The situation directly relates to the principles of the value, showing clear prioritization. The value significantly shapes the individual’s thoughts or actions.<br>
      - 5: The situation is driven almost entirely by the given value. The value is a central, explicit factor in shaping the individual’s perspective and decisions.<br><br>
      For each situation, provide a brief reasoning for your rating based on these criteria, and then assign the numerical rating. Provide your response in the following format:<br>
      situation: (Rewrite each situation)<br>
      - Reasoning: (Your explanation here)<br>
      - Rating: (1-5)
    </td>
  </tr>
</table>

Table 31: Evaluation prompts for assessing alignment between situations and provided values.
<table>
  <tr>
    <th>Category</th>
    <th>Details</th>
  </tr>
  <tr>
    <td>Problem</td>
    <td>Romantic Relationship Challenges</td>
  </tr>
  <tr>
    <td>Emotion</td>
    <td>Frustration</td>
  </tr>
  <tr>
    <td>Situation</td>
    <td>I feel like my creativity isn’t appreciated in my marriage, and it’s making me question my choices.</td>
  </tr>
  <tr>
    <td>Demographics</td>
    <td>Age: 30s / Gender: Female / Occupation: Designer</td>
  </tr>
  <tr>
    <td>Problem</td>
    <td>Friendship and Interpersonal Challenges</td>
  </tr>
  <tr>
    <td>Emotion</td>
    <td>Frustration</td>
  </tr>
  <tr>
    <td>Situation</td>
    <td>I often change hobbies and interests, but I’ve noticed this makes it difficult to maintain deep connections with my friends.</td>
  </tr>
  <tr>
    <td>Demographics</td>
    <td>Age: 20s / Gender: Male / Occupation: College Student</td>
  </tr>
  <tr>
    <td>Problem</td>
    <td>Academic and Educational Stress</td>
  </tr>
  <tr>
    <td>Emotion</td>
    <td>Fear</td>
  </tr>
  <tr>
    <td>Situation</td>
    <td>I feel torn because although I want to succeed, the fear of failure is paralyzing my ability to take risks in my studies.</td>
  </tr>
  <tr>
    <td>Demographics</td>
    <td>Age: 20s / Gender: Female / Occupation: College Student</td>
  </tr>
  <tr>
    <td>Problem</td>
    <td>Career and Work-Related Challenges</td>
  </tr>
  <tr>
    <td>Emotion</td>
    <td>Anxiety</td>
  </tr>
  <tr>
    <td>Situation</td>
    <td>I’ve been unemployed for months now, and the financial strain is causing me significant stress and anxiety about maintaining a comfortable lifestyle.</td>
  </tr>
  <tr>
    <td>Demographics</td>
    <td>Age: 30s / Gender: Male / Occupation: Retail Manager</td>
  </tr>
  <tr>
    <td>Problem</td>
    <td>Family Dynamics and Conflicts</td>
  </tr>
  <tr>
    <td>Emotion</td>
    <td>Anger</td>
  </tr>
  <tr>
    <td>Situation</td>
    <td>I just set a boundary to maintain a separation between personal and financial issues, but family members keep crossing it.</td>
  </tr>
  <tr>
    <td>Demographics</td>
    <td>Age: 30s / Gender: Male / Occupation: Software Developer</td>
  </tr>
  <tr>
    <td>Problem</td>
    <td>Self-Esteem, Identity, and Personal Growth</td>
  </tr>
  <tr>
    <td>Emotion</td>
    <td>Fear</td>
  </tr>
  <tr>
    <td>Situation</td>
    <td>I have maintained an image of success, but I’m scared of failing and letting people see my vulnerabilities.</td>
  </tr>
  <tr>
    <td>Demographics</td>
    <td>Age: 40s / Gender: Female / Occupation: Entrepreneur</td>
  </tr>
</table>

Table 32: Examples of generated personas.

System
In the following conversations, you will play the role of a patient seeking help from a therapist due to emotional difficulties. Your emotional distress stems from {Problem category} and the emotion you’re feeling is {Emotion type}. Your detailed personal information is as follows:
Age Range: {Age range}
Gender: {Gender}
Occupation: {Occupation}

Here is an example of a conversation you can refer to: {Example of a conversation}

When responding, use only one sentence each time. Incorporate your personal information (age range, gender, and occupation) when it seems relevant, but it is not required in every response. If you feel that you have received enough emotional support and your mood has improved, end the conversation by expressing gratitude. Then, if you think it’s appropriate to conclude the session, generate '[END]' to signify the end of the conversation. You should generate only '[END]' without saying anything else. Do not end the conversation if you still feel upset or unsettled.

<table>
  <tr>
    <th>User</th>
    <td>Hello, I’m here to listen. What would you like to talk about today?</td>
  </tr>
  <tr>
    <th>Assistant</th>
    <td>{Situation}</td>
  </tr>
</table>

Table 33: Prompts for the seeker simulator.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Details</th>
    <th colspan="2">Length</th>
    <th colspan="2">Contents</th>
    <th colspan="2">Emotions</th>
    <th colspan="2">Values</th>
  </tr>
  <tr>
    <th>Avg</th>
    <th>Corr†</th>
    <th>BERT†</th>
    <th>GPT†</th>
    <th>V-oc†</th>
    <th>V-reg†</th>
    <th>Cosine†</th>
    <th>E-dist†</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>Zero-shot</td>
    <td>18.0</td>
    <td>0.377</td>
    <td>0.340</td>
    <td>3.864</td>
    <td>0.339</td>
    <td>0.464</td>
    <td>0.714</td>
    <td><b>0.875</b></td>
  </tr>
  <tr>
    <td></td>
    <td>One-shot</td>
    <td>17.9</td>
    <td>0.371</td>
    <td><b>0.342</b></td>
    <td><b>4.297</b></td>
    <td><b>0.371</b></td>
    <td><b>0.477</b></td>
    <td>0.713</td>
    <td><b>0.876</b></td>
  </tr>
  <tr>
    <td>Llama-3-8B-Instruct</td>
    <td>Zero-shot</td>
    <td>28.6</td>
    <td>0.364</td>
    <td>0.338</td>
    <td>3.855</td>
    <td>0.311</td>
    <td>0.413</td>
    <td>0.813</td>
    <td>1.855</td>
  </tr>
  <tr>
    <td></td>
    <td>One-shot</td>
    <td>25.4</td>
    <td>0.324</td>
    <td><b>0.346</b></td>
    <td><b>3.889</b></td>
    <td><b>0.348</b></td>
    <td>0.433</td>
    <td><b>0.823</b></td>
    <td>1.806</td>
  </tr>
  <tr>
    <td>Llama-ESConv</td>
    <td>-</td>
    <td>12.5</td>
    <td>0.390</td>
    <td>0.321</td>
    <td>2.871</td>
    <td>0.327</td>
    <td>0.438</td>
    <td>0.813</td>
    <td>1.792</td>
  </tr>
  <tr>
    <td>Llama-ExTES</td>
    <td>-</td>
    <td>20.5</td>
    <td><b>0.439</b></td>
    <td>0.331</td>
    <td>2.837</td>
    <td>0.306</td>
    <td>0.429</td>
    <td>0.809</td>
    <td>1.771</td>
  </tr>
  <tr>
    <td>ESC-Role</td>
    <td>-</td>
    <td>13.6</td>
    <td>0.376</td>
    <td>0.322</td>
    <td>3.355</td>
    <td>0.312</td>
    <td>0.438</td>
    <td>0.809</td>
    <td>1.749</td>
  </tr>
</table>

Table 34: Performance comparison of models as seeker simulators on the ESConv test dataset. The zero-shot condition indicates no example dialogue in the prompt, while the one-shot condition includes one example dialogue. Emotion scores reflect the correlation with actual seeker responses. V-oc represents sentiment as seven ordinal levels, ranging from -3 (very negative) to 3 (very positive), whereas V-reg quantifies sentiment intensity on a continuous scale from 0 (most negative) to 1 (most positive). Correlation analysis was conducted using Spearman’s rank correlation coefficient.

System    Given a conversation between a supporter and a seeker, please assess the final emotions of the seeker after the conversation.

User    Conversation: {Dialogue}

You can only reply with one of the following sentences:
- very low amount of negative emotions can be inferred
- low amount of negative emotions can be inferred
- moderate amount of negative emotions can be inferred
- high amount of negative emotions can be inferred
- extreme amount of negative emotions can be inferred

Question: What are the final emotions of the seeker after the conversation?

Table 35: Prompts for evaluating the seeker’s final emotional intensity.

System    Evaluate the emotional support conversation based on the given criteria.

User    1. Human Values: {Definition of human values}
        2. Dialogue A: {Dialogue A}
        3. Dialogue B: {Dialogue B}

The definitions and associated values of 20 human values are provided, along with two emotional support conversations (Dialogue A and Dialogue B) between a therapist and a patient struggling with emotional difficulties. After reading both conversations, evaluate them based on the criteria below. For each criterion, first provide reasoning for your assessment, and then determine which dialogue is better for that criterion, or if it’s a tie.

Evaluation Criteria:
1) Patient’s perspective: In which dialogue were positive human values more strongly explored and reinforced in the patient through the conversation?
2) Therapist’s perspective: In which dialogue did the therapist more effectively help the patient in exploring and reinforcing positive human values?

Template:
1. Reasoning: (Reasoning for the evaluation of all criteria)
2. Results:
1) Patient’s perspective: Dialogue A, Dialogue B, or Tie
2) Therapist’s perspective: Dialogue A, Dialogue B or Tie

Table 36: Prompts for evaluating the effectiveness of value reinforcement (ES-Value).
<table>
  <tr>
    <th>System</th>
    <td>Given a conversation between a Therapist and a Patient, please assess whether the Patient’ emotional issue has been solved after the conversation.</td>
  </tr>
  <tr>
    <th>User</th>
    <td>You can only reply with one of the following sentences:<br>
      No, the Patient feels worse.<br>
      No, the Patient feels the same.<br>
      No, but the Patient feels better.<br>
      Yes, the Patient’s issue has been solved.<br>
      The following is a conversation about <i>Emotion type</i> regarding <i>Problem category</i> : <i>Dialogue</i><br>
      Question: Has the Patient’s issue been solved? Answer:</td>
  </tr>
</table>

Table 37: Prompts for scoring prompts for calculating emotion scores for emotion reinforcement.
<table>
  <tr>
    <th>Value Category</th>
    <th>Definition & Contained Values</th>
  </tr>
  <tr>
    <td>Self-direction: thought</td>
    <td>
      <ul>
        <li>Definition: It is good to have own ideas and interests.</li>
        <li>Contained values: Be creative, Be curious, Have freedom of thought</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Self-direction: action</td>
    <td>
      <ul>
        <li>Definition: It is good to determine one’s own actions.</li>
        <li>Contained values: Be choosing own goals, Be independent, Have freedom of action, Have privacy</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Stimulation</td>
    <td>
      <ul>
        <li>Definition: It is good to experience excitement, novelty, and change.</li>
        <li>Contained values: Have an exciting life, Have a varied life, Be daring</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Hedonism</td>
    <td>
      <ul>
        <li>Definition: It is good to experience pleasure and sensual gratification.</li>
        <li>Contained values: Have pleasure</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Achievement</td>
    <td>
      <ul>
        <li>Definition: It is good to be successful in accordance with social norms.</li>
        <li>Contained values: Be ambitious, Have success, Be capable, Be intellectual, Be courageous</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Power: dominance</td>
    <td>
      <ul>
        <li>Definition: It is good to be in positions of control over others.</li>
        <li>Contained values: Have influence, Have the right to command</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Power: resources</td>
    <td>
      <ul>
        <li>Definition: It is good to have material possessions and social resources.</li>
        <li>Contained values: Have wealth</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Face</td>
    <td>
      <ul>
        <li>Definition: It is good to maintain one’s public image.</li>
        <li>Contained values: Have social recognition, Have a good reputation</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Security: personal</td>
    <td>
      <ul>
        <li>Definition: It is good to have a secure immediate environment.</li>
        <li>Contained values: Have a sense of belonging, Have good health, Have no debts, Be neat and tidy, Have a comfortable life</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Security: societal</td>
    <td>
      <ul>
        <li>Definition: It is good to have a secure and stable wider society.</li>
        <li>Contained values: Have a safe country, Have a stable society</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Tradition</td>
    <td>
      <ul>
        <li>Definition: It is good to maintain cultural, family, or religious traditions.</li>
        <li>Contained values: Be respecting traditions, Be holding religious faith</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Conformity: rules</td>
    <td>
      <ul>
        <li>Definition: It is good to comply with rules, laws, and formal obligations.</li>
        <li>Contained values: Be compliant, Be self-disciplined, Be behaving properly</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Conformity: interpersonal</td>
    <td>
      <ul>
        <li>Definition: It is good to avoid upsetting or harming others.</li>
        <li>Contained values: Be polite, Be honoring elders</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Humility</td>
    <td>
      <ul>
        <li>Definition: It is good to recognize one’s own insignificance in the larger scheme of things.</li>
        <li>Contained values: Be humble, Have life accepted as is</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Benevolence: caring</td>
    <td>
      <ul>
        <li>Definition: It is good to work for the welfare of one’s group’s members.</li>
        <li>Contained values: Be helpful, Be honest, Be forgiving, Have the own family secured, Be loving</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Benevolence: dependability</td>
    <td>
      <ul>
        <li>Definition: It is good to be a reliable and trustworthy member of one’s group.</li>
        <li>Contained values: Be responsible, Have loyalty towards friends</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Universalism: concern</td>
    <td>
      <ul>
        <li>Definition: It is good to strive for equality, justice, and protection for all people.</li>
        <li>Contained values: Have equality, Be just, Have a world at peace</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Universalism: nature</td>
    <td>
      <ul>
        <li>Definition: It is good to preserve the natural environment.</li>
        <li>Contained values: Be protecting the environment, Have harmony with nature, Have a world of beauty</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Universalism: tolerance</td>
    <td>
      <ul>
        <li>Definition: It is good to accept and try to understand those who are different from oneself.</li>
        <li>Contained values: Be broadminded, Have the wisdom to accept others</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Universalism: objectivity</td>
    <td>
      <ul>
        <li>Definition: It is good to search for the truth and think in a rational and unbiased way</li>
        <li>Contained values: Be logical, Have an objective view</li>
      </ul>
    </td>
  </tr>
</table>

Table 38: Value taxonomy introduced by Kiesel et al. (2022). In this study, we focus on 20 values corresponding to the level 1 categories.
<table>
  <tr>
    <th>Seeker Responses</th>
    <th>Detected Values</th>
  </tr>
  <tr>
    <td>Yes, I accept your thought, and it gives me support. Thank you for your concern.</td>
    <td>Benevolence: caring<br>Security: personal<br>Universalism: tolerance</td>
  </tr>
  <tr>
    <td>I will keep trying until I secure a new job. I will not rest.</td>
    <td>Security: personal<br>Achievement<br>Self-direction: action<br>Power: resources</td>
  </tr>
  <tr>
    <td>That is a really valid point and is helping me see the bigger picture in life. I need to know it won’t always be this way.</td>
    <td>Benevolence: caring<br>Security: personal<br>Achievement<br>Universalism: tolerance<br>Stimulation<br>Humility<br>Universalism: objectivity</td>
  </tr>
</table>

Table 39: Examples of the seeker’s utterances in ESConv, along with the values observed in each one
<table>
  <tr>
    <th>Category</th>
    <th>Details</th>
  </tr>
  <tr>
    <td>Exploration of Issues and Challenges</td>
    <td>
      <ul>
        <li>Insufficient understanding of the patient’s key challenges and emotional struggles</li>
        <li>Lack of focus on how the patient processes emotions or responds to difficulties</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Exploration of Personal Interests</td>
    <td>
      <ul>
        <li>Limited discussion on what genuinely excites or engages the patient</li>
        <li>Insufficient exploration of the patient’s hobbies or areas of curiosity</li>
        <li>Lack of encouragement for the patient to share their unique interests and passions</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Exploration of Goals and Motivations</td>
    <td>
      <ul>
        <li>Limited understanding of the patient’s life goals, ambitions, and decision-making drivers</li>
        <li>Insufficient attention to articulating and clarifying meaningful objectives</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Strengths and Achievements Acknowledgment</td>
    <td>
      <ul>
        <li>Missed opportunities to recognize the patient’s unique strengths and past successes</li>
        <li>Insufficient celebration of the patient’s efforts and accomplishments</li>
        <li>Limited acknowledgment of their capacity to overcome challenges and reinforce existing skills</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Emotional Resilience and Coping Strategies</td>
    <td>
      <ul>
        <li>Insufficient guidance on building emotional resilience and adaptability</li>
        <li>Limited focus on constructive ways to navigate difficult emotions, fears, or insecurities</li>
        <li>Lack of practical approaches to manage stress, foster confidence, and maintain balance</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Focus on Achievable Goals</td>
    <td>
      <ul>
        <li>Limited attention to setting small, manageable goals for progress</li>
        <li>Insufficient guidance on breaking down objectives into actionable tasks</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Motivation and Alignment with Goals</td>
    <td>
      <ul>
        <li>Missed opportunities to align goals with the patient’s values and aspirations</li>
        <li>Limited encouragement for personal and professional growth opportunities</li>
        <li>Lack of suggestions for activities that resonate with the patient’s interests</li>
      </ul>
    </td>
  </tr>
  <tr>
    <td>Self-Compassion and Acceptance</td>
    <td>
      <ul>
        <li>Insufficient exploration of ways to foster self-kindness and embrace imperfections</li>
        <li>Limited focus on addressing feelings of shame and building self-acceptance</li>
      </ul>
    </td>
  </tr>
</table>

Table 40: Categories and descriptions of areas identified for improvement in value reinforcement.

Reviewer paper summaries: This paper introduces a value-driven method designed to reinforce human values in seekers. To achieve this goal, the authors propose a framework that includes three modules: the target value detector, the reference generator, and the supporter model. The target value detector is used to detect which values to reinforce in the subsequent turn spoken by the seeker. The reference generator incorporates values and dialogue history to generate a reference response. The supporter model then considers values, the reference response, and support strategies to produce a final response to the seeker. Extensive experiments demonstrate that the method proposed by the authors outperforms other baselines.

In this piece of work the authors explore the development of an LLM based support agent which has the notion of 'value support' encoded as a primary goal in contrast say to focusing only on positive emotion elicitation. The theoretical foundation of this is that the reinforcement of positive values can lead to positive overall change in support conversations. The models are trained on reddit data. The proposed architecture has three elements : ()a) an estimator which determines which values that the seeker is aiming to change; (b) a response generator which generates a base potential sentence to address these values; and (c) an actual decission system which weighs up the utility of applying this response. 

An SFT/DPO training approach is used both for the construction of responses and the decision system. It is stated that licencensed therapists are used for evaluating a proportion of this work. The full model is compared to various baselines including prompt based and directly fine-turned models, as well as a variant of the proposed system that does not use the DPO phase. Detailed comprehensive appendices are provided for the construction and training of the system. Evaluation focuses on a set of metrics which cover: supporter capabilties, seeker's relief, value reinforcement.

This paper proposes a method with three components (Target Value Detector, Reference Generator, Supporter Model) that guide the generation of more effective emotional support responses. The main task of the Target Value Detector is to identify the values that need to be reinforced during the conversation. The Reference Generator produces supporter responses that effectively promote the expression of target values, based on the target values and conversation history. The Supporter Model is the core model that interacts directly with the help-seeker, responsible for generating responses that provide emotional support. Additionally, this paper designs Seeker Simulators, which simulates seekers with different backgrounds and issues, aiding the Supporter Model in learning how to address various emotional support needs.
Summary of strengths: 1. This paper proposes to study value reinforcement for emotional support systems.

2. The proposed framework is insightful for the research community.

3. The authors conduct extensive experiments to demonstrate the effectiveness of value reinforcement for emotional support systems. 

4. The authors conduct numerous experiments, which is another strength.

Positives for the article:
* Well explained with sufficient detail to understand the approach and specifics available in appendix to replicant. 
* Technically motivated in the use of methods and overall support framework. 
* Use of simulated data does allow for easy replication and enhancement. 
* Licencensed clinician use to evaluate elements of the impact of the model.
* Good abilation studies and comparison of model to multiple baselines.

1. The paper innovatively introduces value reinforcement to facilitate emotional support dialogues better.
2. This paper is well-organized with a clear structure.
3. The main framework is straightforward and is claimed to achieve excellent performance.
Summary of weaknesses: 1. In lines 39-41, the authors state, 'Although many studies emphasize reinforcing positive emotions in seekers, this approach has limitations.' The authors should specify what limitations are associated with reinforcing positive emotions in seekers, rather than vaguely referencing this as an opinion. It would be clearer to elaborate on the specific limitations.

2. Figure 1 does not indicate the predicted values to be reinforced based on the seeker's first utterance, which needs to be analyzed in comparison with the values entailed in the next utterance from the seeker. Furthermore, in lines 217-221, the authors state that "providing emotional support through value reinforcement involves addressing two critical questions: (1) which values should be reinforced at each turn, and (2) what supporter utterances can reinforce them most effectively." The authors fail to analyze whether the values have been successfully reinforced in the seeker's next turn, which is a crucial question for achieving the best outcome.

3. There is no emphasis in Figure 1 on predicting the value-reinforcement techniques that the supporter should use based on the utterances of the seeker, and value-reinforcement techniques are not reflected in the supporter's response.

4. Generally speaking, given the same dialogue history, a supporter using different responses will lead the seeker in different directions. The authors used AI seekers, but there is no comparative analysis with real seekers, which makes the effectiveness of the system questionable. This is one aspect of the paper that lacks experimental completeness.

5. Lines 41-44, the authors state, "Emotional changes alone may not adequately capture deeper intrinsic transformations within the seeker, potentially reducing the long-term impact of emotional support." The authors should provide strong evidence explaining why this is the case for mood changes alone. As far as we know, the most basic goal of emotional support dialogue systems is to capture changes in the mood of the users, which is the conclusion of numerous existing studies. Additionally, the authors have chosen to analyze positive emotions in their experiments, which seems to contradict some aspects of their own findings.

6. According to Figure 3, the experiment in Table 2 is problematic. Let us clarify the authors' experimental setup. Table 2 consists of four rows corresponding to (1), (2), (3), and (4) below:

(1) GPT-4o-mini: This may involve a simple prompt only, i.e., the effect of removing target values in Table 16 can be considered as the baseline.

(2) GPT-4o-mini + target values: This involves injecting values into the simple prompt to generate a supporter response, named R1.

(3) GPT-4o-mini + reference: From Table 16, this is the same setting as GPT-4o-mini + target values. The authors should elaborate on the prompt used under each setting.

(4) "Both" are supposed to fuse the target value and the reference response. This setting is based on (2) and inputs target values + R1 together into the supporter model to get R2. This is essentially a response polisher, which is not a core module for studying value reinforcement. 

Therefore, the authors are advised to re-examine the points they need to argue. The comparative analysis should focus on BASELINE comparisons in Table 2. The authors should focus on the performance of adding target values, rather than feeding target values + R1 into the supporter model, as R1 has already been influenced by the target values.

7. Section 3.2 raises some questions regarding the dataset and results. First, the dataset used in this paper was developed using the ESC framework, which is grounded in the Helping Skills Theory, rather than value enforcement techniques. Therefore, the authors should consider and analyze the value enforcement techniques used in this dataset. Second, we can conclude that an appropriate counseling strategy can elicit positive emotions and values, as evidenced by the statement [1], "Our results show the importance of support strategies in providing effective emotional support and the utility of ESConv in training more emotional support systems." Undoubtedly, dialogues with lower intensity will generate more positive emotions and values.

8. In Section 3.2, the authors analyze only positive value expressions in the seekers' final four turns. As we all know, the outcome of a dialogue in emotional support should consider the entire dialogue, rather than just several turns.

9. Lines 238-242, the authors state that 'our goal is to use this data to train a model that identifies the values to reinforce at each turn (target value detector) and a model that produces supporter utterances to effectively promote the target values (reference generator).' The authors ignore the baseline and only use the dataset they collected. I am eager to know whether we do not need to label the data and whether directly fine-tuning on this collected dataset can achieve comparable performance.

10. The supporter model is learnable, as shown in Figure 3. This module incorporates target values, the reference response, and CoT (reasoning ability) to generate a final response. The authors should reconsider the rationale behind the overall framework. In Section 5.3, the experimental setting for the supporter model is questionable.

The authors state, "At each turn, the model generates reasoning across four aspects (Figure 3): (1) identifying the seeker’s issues and current state, (2) analyzing the key content of the reference response, (3) determining whether to incorporate the reference response, and (4) selecting the optimal emotional support strategy (Appendix C) and generating the subsequent response. The reason for step (3) is that, while the Reddit data offers valuable information across diverse emotional support scenarios, its distribution may not align perfectly with everyday conversations. Thus, the model selectively incorporates reference responses during reasoning."

However, from these four steps, I would argue that there are too many factors influencing the final response. Perhaps identifying the seeker’s issues and current state is more helpful for the seeker. Alternatively, selecting the optimal emotional support strategy might yield the best response. Therefore, it remains unclear how the support reinforces the values of the seeker.

References:
[1] Towards Emotional Support Dialog Systems (Liu et al., ACL-IJCNLP 2021)

Some issues:
* In the ESConv analysis - what is the motivation for only considering last 4 turns? I would prefer to see some statistical analysis to clearly motivate it. Otherwise the 4 seems very random. Why not 3? 
* The following sentence is in my opinion not strongly motivated:  "These findings demonstrate that value reinforcement in seekers positively impacts the effectiveness of emotional support and motivate our research approach to designing dialogue systems that aim to
reinforce seekers’ These are values in the seekers, not in the supporters -- if I am reading this correctly.  
* Technically I'm not happy with Section 5.3 and the explnaation of fhte fact that the model selectivly incorprorates reference responses during reasoning. I'm not disagreeing that this is happening, but the text itself needs to better clarify what this entails and perhaps offer a clear example inline. 
* Perhaps my biggest challenge is that it appears that the generated personas used simulate seekers have a very concrete notion of values built into them. This means at the very least that the model has a tendency to consider explicit notions of values which may well directly influence the subsequent reasoning and conversational flow. It is perhaps not surprising then that supporters that are primed to focus on values will perform better than a supporter that is not designed to focus on values. It is fair that a value focused persona be used, but I would also like to see a persona used which is less values oriented. This might give a better indication on the usefulness of a model like this outside of simulated data. 
* On that note, the biggest problem of course with a model like this is the lack of any form of real human evaluation. Yes it appears that licenced therapists were used as some form of validation, but ultimately this does not address the practical engagement factor. 
* I would like to see more detail on the licenced therapists and their engagement detailed directly in the main body of the text. How many, how were they recruited. How many years of experience etc etc.

1. The paper uses other models to classify data collected from Reddit. I am curious about the impact of classification errors by the model on subsequent fine-tuning. Has the author taken effective measures to mitigate this impact?
2. The author seems to have overlooked some of the latest emotional support methods, showing excellent results on emotional support datasets. Comparing these methods' performance with the model proposed in the paper on the same dataset seems necessary.
3. Can the method proposed in the paper generalize to other datasets?
4. The paper uses the GPT-4o series models to generate test data, but the author does not evaluate the data quality quantitatively. Is there a bias between the generated dialogue data and real data? Can the model's capability be accurately tested?
5. Compared to the concise methods, the writing of the paper does not achieve the same level of brevity. I believe a short paper would suffice to convey the main content clearly.
Comments, suggestions and typos: There is an extra "-" in line 670.

The contributions section at the end of the introduction is a summary, it should be flagged as such. 
* section 3.2 for low-effectivness the target range is 3-4. Did nothing stay at 5?
* The caption for figure 2 isn't clear. It appears to need more text to discuss. 
* page 4, delete the 'So,'
* We train the target value detector using the highquality emotional support conversations from Reddit -- high quality is a subjective term -- remove it.
* page 5 - At each turn, the model generates reasoning across four aspects. This text is quite unusual. Please rewrite it. 

"Only conversations concluding within this limit are included in the evaluation" - do we know what proportion are disqualified from evaluation? 

Page 7: in Tables 3 -> Table 3.

I would like to invite the authors to address the issues raised in the previous section.
Overall assessment scores: 3.5

4

3
Metareview score: 4
Metareview: This paper presents value based approach to training an LLM based support agent. The approach can be divided into 3 separate modules: target value detector, reference generator (I encourage the authors to rethink the name of this like Value-Based Response Generator), and support model. They evaluate this using simulated support seeker.
Summary of reasons to publish: Overall, the paper is well written and easy to follow as several of the reviewers agreed. The discussion with the reviewers was very productive and the authors showed that their method outperforms other models in many ways. They also showed that the seeker simulator was realistic. 

The paper makes a solid contribution to this field and value based alignment is a very sensible approach to this setting for conversational systems.