KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors
Authors: Zhiyang Qi, Takumasa Kaneko, Keiko Takamizo, Mariko Ukiyo, Michimasa Inaba
Accepted at: acl25

KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors

Zhiyang Qi1, Takumasa Kaneko1, Keiko Takamizo2,3,4, Mariko Ukiyo2,3,4, Michimasa Inaba1,2
1The University of Electro-Communications,
2Rapport Technologies, Inc.
3iDEAR Human Support Service,
4Japanese Organization of Mental Health and Educational Agencies
{qizhiyang,m-inaba}@uec.ac.jp

Abstract

Generating psychological counseling responses with language models relies heavily on high-quality datasets. Crowdsourced data collection methods require strict worker training, and data from real-world counseling environments may raise privacy and ethical concerns. While recent studies have explored using large language models (LLMs) to augment psychological counseling dialogue datasets, the resulting data often suffers from limited diversity and authenticity. To address these limitations, this study adopts a role-playing approach where trained counselors simulate counselor-client interactions, ensuring high-quality dialogues while mitigating privacy risks. Using this method, we construct KokoroChat, a Japanese psychological counseling dialogue dataset comprising 6,589 long-form dialogues, each accompanied by comprehensive client feedback. Experimental results demonstrate that fine-tuning open-source LLMs with KokoroChat improves both the quality of generated counseling responses and the automatic evaluation of counseling dialogues. The KokoroChat dataset is available at https://github.com/UEC-InabaLab/KokoroChat.

1 Introduction

Psychological issues have long posed a significant global challenge, with many individuals suffering from mental health disorders (WHO, 2022). However, limited medical resources restrict access to professional psychological counseling for most people (SAMHSA, 2015). To address this gap, researchers have explored language models for generating empathetic responses to provide emotional support. Advancing this research depends on constructing high-quality datasets. For instance, Liu et al. (2021) developed the ESConv dataset by training crowdworkers in emotional support skills, while Li et al. (2023) created the Client-Reactions dataset by establishing an online mental health support platform to collect dialogues between real clients and professional counselors. Despite these efforts, several challenges remain. Psychological counseling is a highly specialized form of communication (Althoff et al., 2016), making it costly and time-consuming to train crowdworkers without professional backgrounds. Meanwhile, dialogue participants may struggle to fully grasp the experiences of individuals with mental disorders, making it difficult to simulate authentic interactions. Additionally, real counseling data may involve privacy and ethical concerns. As a result, the manual collection of such data often faces practical limitations.

Role-Playing Counseling Dialogue

![A sample counseling dialogue between a counselor and a client, showing a conversation transcript with roles played by trained counselors.](page_728_320_377_312.png)

Figure 1: Each KokoroChat sample includes a counseling dialogue and client feedback, with both roles played by trained counselors.
<table>
  <tr>
    <th>Dataset</th>
    <th>Human-made</th>
    <th>Score</th>
    <th>Score items</th>
    <th>Language</th>
    <th># Dialogues</th>
    <th>Avg. utterances</th>
  </tr>
  <tr>
    <td>HealMe (Xiao et al., 2024)</td>
    <td>✗</td>
    <td>✗</td>
    <td>-</td>
    <td>English</td>
    <td>1,300</td>
    <td>6.0</td>
  </tr>
  <tr>
    <td>ESD-CoT (Zhang et al., 2024b)</td>
    <td>✗</td>
    <td>✗</td>
    <td>-</td>
    <td>English</td>
    <td>1,708</td>
    <td>23.4</td>
  </tr>
  <tr>
    <td>CACTUS (Lee et al., 2024)</td>
    <td>✗</td>
    <td>✗</td>
    <td>-</td>
    <td>English</td>
    <td>31,577</td>
    <td>31.5</td>
  </tr>
  <tr>
    <td>SMILECHAT (Qiu et al., 2024)</td>
    <td>✗</td>
    <td>✗</td>
    <td>-</td>
    <td>Chinese</td>
    <td>55,165</td>
    <td>33.2</td>
  </tr>
  <tr>
    <td>AUGESC (Zheng et al., 2023)</td>
    <td>✗</td>
    <td>✗</td>
    <td>-</td>
    <td>English</td>
    <td>65,000</td>
    <td>26.7</td>
  </tr>
  <tr>
    <td>Anno-MI (Wu et al., 2022)</td>
    <td>✓</td>
    <td>✗</td>
    <td>-</td>
    <td>English</td>
    <td>133</td>
    <td>72.9</td>
  </tr>
  <tr>
    <td>ESConv (Liu et al., 2021)</td>
    <td>✓</td>
    <td>✓</td>
    <td>2</td>
    <td>English</td>
    <td>1,300</td>
    <td>29.5</td>
  </tr>
  <tr>
    <td>Client-Reactions (Li et al., 2023)</td>
    <td>✓</td>
    <td>✓</td>
    <td>4</td>
    <td>Chinese</td>
    <td>2,382</td>
    <td>78.5</td>
  </tr>
  <tr>
    <td><b>KokoroChat</b></td>
    <td>✓</td>
    <td>✓</td>
    <td>20</td>
    <td>Japanese</td>
    <td><b>6,589</b></td>
    <td><b>91.2</b></td>
  </tr>
</table>

Table 1: Comparison of psychological counseling datasets: LLM-augmented (top), human-collected (bottom).

Recently, LLMs have made significant strides in natural language generation and have shown considerable potential in generating psychological counseling responses (Inaba et al., 2024). Consequently, many studies have leveraged LLMs for self-chat, rewriting, or mimicking existing datasets to construct or expand psychological counseling dialogue datasets (Zheng et al., 2023; Xiao et al., 2024; Zhang et al., 2024b; Lee et al., 2024; Qiu et al., 2024). However, despite LLMs’ strong ability to generate psychological counseling responses, these augmented datasets often exhibit redundancy and homogeneity, leading to a lack of dialogue diversity (Zheng et al., 2024). Moreover, as shown in Table 1, augmented datasets contain significantly fewer utterances than human-collected psychological counseling dialogues (e.g., Client-Reactions: 78.5), limiting their effectiveness and applicability.

To address these issues, this study employs a role-playing approach for data collection, in which trained professional and trainee counselors simulate interactions between a counselor and a client. Compared to traditional crowdsourced methods, this approach ensures higher professionalism and dialogue quality, as both participants are trained counselors. Additionally, unlike direct collection of real counseling dialogues, this method mitigates privacy and ethical risks. Furthermore, compared to LLM-augmented datasets, the dialogues collected through this approach offer stronger assurances of professionalism and authenticity.

Through a role-playing approach, we develop KokoroChat, a high-quality dialogue dataset for psychological counseling. As shown in Figure 1, trained professional and trainee counselors play both counselor and client roles, engaging in approximately one-hour online text-based counseling sessions. Considering that an objective and detailed scoring mechanism can quantify counseling quality and support counselor skill development, we collect client feedback after each session. The client-role player evaluates the counselor-role player based on two key dimensions: overall impressions and professional skills. The evaluation consists of 20 assessment items, each rated on a scale of 0 to 5, with a maximum total score of 100. This study involves 480 participants, all of whom have completed 10 hours of training in online text-based psychological counseling. Over one-third are professional counselors, while the remaining participants are trainees who have studied relevant topics for six months to one year and aspire to become certified counselors.

As shown in Table 1, we collect 6,589 high-quality psychological counseling dialogues, averaging 91.2 utterances per dialogue, each accompanied by detailed client feedback. To our knowledge, KokoroChat is the largest human-collected psychological counseling dialogue dataset to date, with session durations aligning with real-world counseling practices (approximately one hour per session). Notably, KokoroChat is a Japanese-language dataset, offering linguistic resources for psychological counseling dialogue research from a diverse cultural perspective. Given the global demand for psychological counseling, developing datasets that encompass multiple cultures and languages is essential for enhancing models’ adaptability to users from different backgrounds. By filling the gap in Japanese psychological counseling dialogue data, KokoroChat provides a foundation for cross-cultural research in psychological counseling.

To summarize: (1) we develop KokoroChat, the largest manually collected psychological counseling dialogue dataset to date, using a role-playing approach, with detailed client feedback; (2) we
fine-tune an open-source LLM to demonstrate that KokoroChat enhances LLM performance in generating psychological counseling responses; (3) we train a dialogue evaluation model using client feedback from KokoroChat, and our experimental results show that this model provides more robust and accurate evaluation outcomes.

2 Related Work

2.1 Psychological Counseling in NLP

In recent years, psychological counseling has attracted significant attention in the field of natural language processing (NLP). Some studies have focused on generating empathetic responses, where systems provide appropriate feedback by understanding users’ emotions (Rashkin et al., 2019; Sharma et al., 2020; Zheng et al., 2021). However, empathy alone is insufficient to address the complex demands of psychological counseling. To bridge this gap, Liu et al. (2021) proposed the Emotional Support Conversation (ESC) task, which requires systems not only to exhibit empathy but also to deeply explore users’ concerns and offer effective guidance to help them navigate challenges.

As LLMs’ generation capabilities advance, their potential applications in psychological counseling have gained further attention. For example, Inaba et al. (2024) demonstrated that GPT-4’s (OpenAI, 2023) responses in psychological counseling scenarios are comparable to those of professional counselors. Additionally, several LLM-based psychological counseling chatbots, such as ChatCounselor (Liu et al., 2023), MeChat (Qiu et al., 2024), and SoulChat (Chen et al., 2023), have emerged. These systems are typically fine-tuned on manually curated or LLM-augmented psychological counseling data to adapt to specific scenarios. However, the field still faces challenges, particularly the lack of high-quality, diverse professional datasets. To address this, our study develop KokoroChat, a high-quality psychological counseling dialogue dataset, through role-playing by professional counselors. This dataset aims to facilitate the development of psychological counseling dialogue systems.

2.2 Counseling Dialogue Datasets

The key to equipping language models with psychological counseling capabilities lies in the availability of high-quality datasets. Currently, psychological counseling datasets fall into two main categories: manually constructed datasets and those generated by LLMs. Manually constructed datasets typically consist of human dialogues. For example, Anno-MI (Wu et al., 2022) was developed by extracting full motivational interview dialogues from online videos. ESCconv (Liu et al., 2021) collected emotional support dialogues from crowd-workers trained in specialized skills, while Client-Reactions (Li et al., 2023) was derived from interaction records between counselors and clients on real online counseling platforms.

On the other hand, LLM-augmented datasets were created by having LLMs simulate both the counselor and client roles (Chen et al., 2023; Ye et al., 2025). For instance, HealMe (Xiao et al., 2024) and CACTUS (Lee et al., 2024) generated dialogues based on cognitive behavioral therapy (CBT) using carefully designed prompts; ESD-CoT (Zhang et al., 2024b) extracted situations from existing datasets to generate complete dialogues. SMILECHAT (Qiu et al., 2024) expanded single-turn Q&A into multi-turn dialogues, while AugESC (Zheng et al., 2023) modeled data augmentation as a dialogue completion task to extend conversations. As shown in Table 1, although these datasets are often large in size, they tend to contain fewer utterances and continue to face challenges in content diversity (Zheng et al., 2024). In contrast, manually constructed datasets are smaller while offering higher authenticity and quality. This study proposes the largest known human-collected psychological counseling dialogue dataset, with dialogue lengths resembling real counseling sessions.

2.3 Evaluation of Counseling Dialogue

In recent years, studies have increasingly explored the use of LLMs for dialogue evaluation in specific scenarios. For example, Liu et al. (2023) employed GPT-4 to compare psychological counseling responses generated by different models across multiple dimensions, such as information quality and user self-disclosure. Lee et al. (2024) and Zhao et al. (2024) simulated client interactions with counselor models, conducting full-length dialogues and evaluating them based on overall performance.

Additionally, some datasets have introduced rating mechanisms. For instance, the Client-Reactions (Li et al., 2023) recorded client ratings across four dimensions alongside dialogue collection. However, among 2,382 dialogue turns, only 479 turns included ratings, limiting its scale. In contrast, our study introduces a dataset with 6,589 counseling dialogues, all accompanied by rating information.
We design 20 evaluation dimensions (as shown in Table 2) to assess both the overall impression of the counseling session and the counselor’s professional skills, providing a more comprehensive standard for evaluating dialogue quality.

3 Data Collection

To construct a high-quality psychological counseling dialogue dataset, we employed a role-playing approach in which trained professional and trainee counselors simulated counselor-client interactions, ensuring authenticity and professional relevance. Additionally, we collected detailed client feedback, providing a valuable resource for evaluating psychological counseling dialogues.

3.1 Data Source

We developed an online platform to facilitate participant matching, role-playing dialogues, and client feedback collection. On this platform, participants can choose their preferred roles and schedule dialogues based on their availability and role preferences. Once matched, role-playing dialogues take place at the designated time, with the counselor-role player communicating via a computer keyboard and the client-role player using LINE, a mobile messaging app. This setup reflects real-world online text-based psychological counseling practices in Japan. Additionally, to ensure a complete record of the dialogue process, the platform stores message timestamps for further analysis. The detailed interface is shown in Appendix A.1.

Each dialogue session typically lasts one hour, though the duration may be adjusted as needed. As part of the role-playing process, the counselor-role player can specify discussion topics and the client’s psychological state (e.g., whether suicidal tendencies are present). If no specific conditions are set, the client-role player decides freely. To protect participant privacy, all client-role participants are explicitly instructed not to discuss their real-life concerns during the dialogue and are strictly prohibited from sharing personal identifying information. After the session, the client-role participant evaluates the counselor-role player’s performance, with specific evaluation criteria detailed in Section 3.3.

3.2 Participants

The dataset comprises 480 participants, including 117 males, 360 females, and 3 individuals who did not disclose their gender. Participants’ ages range from 21 to 78, with approximately 80% between 30 and 59 years old. Detailed distribution information is provided in Appendix A.2. All participants are native Japanese speakers, with over 80% having played both the client and counselor roles. In total, 424 participants took on the counselor role, and 463 participants played the client role.

Professionalism  Participants have expertise in online psychological counseling. More than one-third hold professional qualifications and have practical counseling experience, while the rest, though not yet certified, have undergone six months to one year of systematic study with the goal of obtaining certification. Additionally, all participants completed a 10-hour structured training program covering the characteristics, advantages, and limitations of online text-based psychological counseling, the role and ethical guidelines of counselors, as well as professional counseling skills and procedures.

3.3 Client Feedback

After each role-playing, the client-role player evaluates the counselor-role player’s performance. The results are immediately shared with the counselor-role player and are monitored by the platform administrator to ensure fairness and reliability.

The client feedback items are designed under the supervision of an expert holding the nationally recognized Certified Public Psychologist qualification and a Ph.D. degree. As shown in Table 2, the feedback covers two main aspects:

• Overall impression of the conversation (e.g., understanding and respect, sense of hope, engagement, fluency, satisfaction)

• Evaluation of counseling skills (e.g., empathy, affirmation, effective questioning, goal setting, problem clarification, and conveying hope)

The evaluation employs a six-point Likert scale (0–5 points) across 20 items, with a maximum total score of 100 points. Additionally, three check items assess serious issues, such as inappropriate remarks or ethical violations. If any of these are selected, the overall score is halved or reset to zero. Further details are provided in Appendix A.3.

4 Data Characteristics

This study collected dialogue data from March 7, 2020, to September 8, 2024, filtering out conversations with fewer than 30 utterances, durations
<table>
  <tr>
    <th>Category</th>
    <th>Aspect</th>
    <th>Feedback Item</th>
  </tr>
  <tr>
    <td rowspan="6">Overall Impression of the Conversation</td>
    <td>Sense of Validation</td>
    <td>1. Felt heard and understood.<br>2. Felt respected.</td>
  </tr>
  <tr>
    <td>Awareness and Hope</td>
    <td>3. Gained new insights.<br>4. Felt hopeful or expectant.</td>
  </tr>
  <tr>
    <td>Engagement</td>
    <td>5. Concerns were addressed.<br>6. Thought through concerns together.</td>
  </tr>
  <tr>
    <td>Flow and Comfort</td>
    <td>7. The conversation had a good rhythm.<br>8. The conversation felt comfortable.</td>
  </tr>
  <tr>
    <td>Overall Evaluation</td>
    <td>9. Felt appropriate and satisfying.<br>10. The conversation was valuable.</td>
  </tr>
  <tr>
    <td>Flow of Conversation</td>
    <td>11. The conversation started smoothly.<br>12. The conversation ended well.</td>
  </tr>
  <tr>
    <td rowspan="10">Evaluation of Counseling Skills</td>
    <td>Counseling Skills</td>
    <td>13. Showed acceptance and empathy.<br>14. Provided acknowledgment and affirmation.<br>15. Asked effective questions to foster dialogue.<br>16. Summarized key points effectively.<br>17. Clarified issues clearly.<br>18. Helped identify goals for the conversation.<br>19. Offered actionable suggestions.<br>20. Encouraged and instilled hope.</td>
  </tr>
</table>

Table 2: The 20 client feedback items (each rated on a 0–5 point scale).

under 30 minutes, or cases where all 20 evaluation items were rated as 3 (as such scores may be unreliable). The final dataset consists of 6,589 dialogues, with statistical details provided in Table 3.

Each dialogue contains an average of 91.20 utterances, surpassing other manually collected datasets such as ESConv (Liu et al., 2021) (29.5 utterances), Anno-MI (Wu et al., 2022) (72.9 utterances), and Client-Reactions (Li et al., 2023) (78.5 utterances). This suggests that our dataset provides greater depth and interactivity. Additionally, the average utterance length of counselors (35.84 characters) is significantly higher than that of clients (20.63 characters), reflecting the counselor’s guiding role in conversations, where they typically use more detailed language to provide support.

Additionally, the dataset includes 480 unique speakers, with 424 counselors and 463 clients, resulting in 4,900 distinct counselor–client pairings, which contribute to conversational diversity to some extent.

<table>
  <tr>
    <th>Category</th>
    <th>Total</th>
    <th>Counselor</th>
    <th>Client</th>
  </tr>
  <tr>
    <td># Dialogues</td>
    <td>6,589</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td># Speakers</td>
    <td>480</td>
    <td>424</td>
    <td>463</td>
  </tr>
  <tr>
    <td># Utterances</td>
    <td>600,939</td>
    <td>306,495</td>
    <td>294,444</td>
  </tr>
  <tr>
    <td>Avg. utterances per dialogue</td>
    <td>91.20</td>
    <td>46.52</td>
    <td>44.69</td>
  </tr>
  <tr>
    <td>Avg. length per utterance</td>
    <td>28.39</td>
    <td>35.84</td>
    <td>20.63</td>
  </tr>
</table>

Table 3: Statistics of the overall conversations. An “utterance” denotes one discrete message sent by a client or counselor in the chat system upon clicking the send button.

4.1 Dialogue Topics

The counseling topics were determined by the participants. To gain a comprehensive understanding of the issues present in KokoroChat, we utilized GPT-4o-mini (OpenAI, 2024) to analyze dialogue topics. Specifically, we input the dialogue content to predict problem types and generate more detailed descriptions. The prompt used for this analysis is provided in Appendix B.

![A circular chart showing the distribution of issues in KokoroChat, with categories such as School, Family (Mom/Spouse), Workplace, Mental Health, and others, each labeled with a percentage.](page_1012_1342_420_312.png)

Figure 2: The distribution of issues in KokoroChat.
The distribution of the 11 predefined issue types is shown in Figure 2. Family issues (non-spousal) account for the largest proportion (20.2%), followed by workplace issues (17.0%) and mental health issues (14.7%), reflecting that family, work, and mental health are primary concerns for clients. School issues (14.1%) and career and education issues (12.9%) also constitute significant portions, highlighting the importance of education and career development in counseling conversations.

Based on the generated detailed descriptions, we further summarized the characteristics of each issue category, as presented in Figure 2. Overall, the dataset encompasses a broad range of real-world problems, demonstrating high diversity and providing a solid foundation for research on psychological counseling dialogues.

4.2 Analysis on Client Feedback

Score Distribution Figure 3 presents the score distribution of dialogues in KokoroChat. The histogram shows a well-balanced distribution, with a unimodal shape centered around the mean (63.58) and median (64.00). Additionally, the distribution exhibits a slight right skew, indicating that most dialogues received moderate to high client feedback.

Correlation Between Dialogue Features and Scores We conducted a Spearman correlation analysis to examine the relationship between various dialogue features and feedback scores, as shown in Figure 4. The results indicate that the total word count of the client has the highest positive correlation with the score (\( \rho = 0.42 \)), suggesting that the extent of client expression may influence their evaluation. When clients use more words to express themselves, they may feel better heard and understood, leading to a more positive assessment of their counseling experience. In contrast, the total word count of the counselor shows a lower correlation with the score (\( \rho = 0.28 \)), implying that while greater counselor speech may contribute to higher ratings, its impact is relatively limited. Additionally, the correlation between utterance count and scores is weaker, with clients (\( \rho = 0.21 \)) and counselors (\( \rho = 0.17 \)) both showing a positive correlation, though to a lesser extent than word count. This result suggests that the richness of conveyed information may be more influential than the number of utterances. Furthermore, the counselor’s average response time exhibits a negative correlation with the scores (\( \rho = -0.21 \)), indicating that longer response times may negatively impact user experience, whereas quicker responses could potentially contribute to higher ratings.

Correlation Among Evaluation Dimensions Additionally, we conducted a Spearman correlation analysis to examine the relationships between different evaluation dimensions. The results indicate that most rating dimensions exhibit a strong positive correlation (\( \rho > 0.6 \)), suggesting that the counseling experience is influenced by multiple interrelated factors rather than a single determinant.

Notably, D1 (*felt heard and understood*) shows a high correlation with D2 (*felt respected*), D9 (*felt appropriate and satisfying*), and D10 (*the conversation was valuable*). This suggests that when clients feel heard and understood, they are more likely to experience a sense of respect, perceive the counseling process as meaningful, and ultimately report higher overall satisfaction. The complete results are presented in Figure 11 in Appendix C.

5 Experiments

To evaluate KokoroChat’s potential in psychological counseling response generation and dialogue assessment, we conducted experiments on two tasks: response generation and score prediction.
5.1 Response Generation
Due to the lack of a Japanese psychological counseling dataset and a Japanese LLM specifically designed for counseling, direct comparison with other models is not feasible1. Therefore, this study focuses on verifying whether fine-tuning on KokoroChat can enhance the performance of open-source LLMs in psychological counseling tasks.

For dialogue data preprocessing, we applied an utterance merging strategy, combining consecutive utterances from the same speaker into a single utterance. The model takes the complete dialogue history \( D_t = \{u_1^C, u_2^S, u_3^C, \ldots, u_t^C\} \) as input, where \( u_j^C \) and \( u_j^S \) represent utterances from the client (C) and counselor (S), respectively. The model then generates the next counselor response \( u_{t+1}^S \).

5.1.1 Models
In this experiment, we used Llama 3.1 Swallow 2 (Fujii et al., 2024; Okazaki et al., 2024) as the base model, fine-tuning and evaluating it using the KokoroChat dataset. To ensure high-quality test data, we selected 118 dialogues with client feedback scores of 99 and 100 as the test set, while the remaining data was used for fine-tuning. The model-generated responses were then compared with the corresponding counselor replies in the test set to evaluate the quality of the generated outputs.

To explore the impact of client feedback on model ability to generate psychological counseling responses, we constructed the following variants:

• Kokoro-Low: Fine-tuned on 334,022 utterances from 3,870 dialogues with a client feedback score of < 70.3

• Kokoro-High: Fine-tuned on 254,515 utterances from 2,601 dialogues with a client feedback score in the range of \( 70 \leq score \leq 98 \).

• Kokoro-Full: Fine-tuned on 6,471 dialogues with a client feedback score of \( \leq 98 \).

Finally, we evaluated these variants on the test set to examine the impact of data partitions on model improvement. Additionally, we compared our models with GPT-4o (OpenAI, 2024), one of the most advanced models, to further understand their relative performance. Appendix D provides details on the training process.

5.1.2 Automatic Evaluation
We used three commonly adopted automatic evaluation metrics: BLEU-n (Papineni et al., 2002), ROUGE-L (Lin, 2004), and Distinct-n (Li et al., 2016). The results are presented in Table 4. Experimental findings indicate that Kokoro-High performs best on most BLEU and ROUGE metrics, likely due to the higher quality of its training data and its closer alignment with the test set. In contrast, Kokoro-Full, which includes a larger dataset, achieves slightly better performance on the diversity metric Dist-n. For non-fine-tuned models, GPT-4o outperformed Llama-3.1 across all automatic evaluation metrics.

5.1.3 Human Evaluation
We also conducted a human evaluation of 100 responses generated by each model, independently assessed by five professional counselors. Specifically, we randomly selected 10 dialogues from the test set and, for each dialogue, randomly sampled 10 sets of dialogue histories of varying lengths to generate model responses. The evaluation used pairwise comparison, where counselors judged which response was more suitable (Win, Lose, Tie). The final result followed majority voting—if over half agreed, it was adopted; otherwise, or if Tie votes exceeded half, the result was Tie.

Figure 5 presents the evaluation results. The comparison between Kokoro-Low and Llama-3.1 indicates that even when using only the lower-scoring portions of KokoroChat, it still enhances open-source LLMs in generating psychological counseling responses. Notably, despite using less data, Kokoro-High outperforms both Kokoro-Low and Kokoro-Full, similar to the results of automatic evaluation, highlighting the importance of high-quality training data in improving model performance. However, due to the difference in model size (Llama-3.1 = 8B, GPT-4o \( \approx 200B^4 \)), the fine-tuned model still lags behind GPT-4o. Similarly, GPT-4o’s responses also exhibit a noticeable gap compared to those of highly rated human counselors, further emphasizing the high quality of

1To provide a rough performance reference, we conducted a simplified experiment comparing our model with models trained on non-Japanese counseling data in a Japanese setting. See Appendix E for details.
2Llama-3.1-Swallow-SB-Instruct-v0.3 was used, which is a continuously pre-trained variant of Meta Llama 3.1 (Dubey et al., 2024) with enhanced Japanese proficiency and optimized dialogue generation.
3Considering that low-scoring dialogues tend to have a relatively lower number of utterances, we set the threshold at 70 to ensure balanced data segmentation.
4This is merely an estimate by Abacha et al. (2024).
<table>
  <tr>
    <th>Model</th>
    <th>BLEU-1</th>
    <th>BLEU-2</th>
    <th>BLEU-3</th>
    <th>BLEU-4</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>Dist-1</th>
    <th>Dist-2</th>
  </tr>
  <tr>
    <td>Llama-3.1</td>
    <td>17.32</td>
    <td>9.13</td>
    <td>4.77</td>
    <td>2.25</td>
    <td>23.81</td>
    <td>7.37</td>
    <td>16.96</td>
    <td>1.04</td>
    <td>6.86</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>21.77</td>
    <td>11.72</td>
    <td>6.32</td>
    <td>3.17</td>
    <td>28.67</td>
    <td>9.19</td>
    <td>19.82</td>
    <td>1.19</td>
    <td>6.90</td>
  </tr>
  <tr>
    <td>Kokoro-Low</td>
    <td>25.39</td>
    <td>15.30</td>
    <td>8.69</td>
    <td>5.39</td>
    <td>33.38</td>
    <td>14.05</td>
    <td>27.28</td>
    <td>2.42</td>
    <td>12.98</td>
  </tr>
  <tr>
    <td>Kokoro-High</td>
    <td><b>27.03</b></td>
    <td><u>16.45</u></td>
    <td><b>9.57</b></td>
    <td><b>6.00</b></td>
    <td><b>34.64</b></td>
    <td><b>14.72</b></td>
    <td>28.00</td>
    <td>2.33</td>
    <td>13.08</td>
  </tr>
  <tr>
    <td>Kokoro-Full</td>
    <td><u>25.69</u></td>
    <td><b>15.65</b></td>
    <td><u>9.23</u></td>
    <td><u>5.83</u></td>
    <td><u>34.02</u></td>
    <td><u>14.60</u></td>
    <td><b>28.10</b></td>
    <td><b>2.48</b></td>
    <td><b>13.24</b></td>
  </tr>
</table>

Table 4: Performance comparison of models. Best values are in bold, second-best are underlined.

![Human evaluation result. Orange denotes upper model wins; the winning model is in bold. Human refers to responses from human counselors in the KokoroChat test dataset.](page_324_624_496_246.png)

Figure 5: Human evaluation result. Orange denotes upper model wins; the winning model is in bold. Human refers to responses from human counselors in the KokoroChat test dataset.

KokoroChat. The generated response examples are shown in Figure 14 of Appendix F.

5.2 Score Prediction

Automatic evaluation of psychological counseling dialogues not only reduces the cost of human evaluation but also provides counselors with feedback to enhance their professional skills. To verify the effectiveness of LLMs fine-tuned on KokoroChat for dialogue evaluation, we conducted a score prediction experiment.

Specifically, given a complete dialogue \( D \) as input, the model predicts a set of item-score pairs \( S = \{ (d_1, s_1), (d_2, s_2), \ldots, (d_{20}, s_{20}) \} \), where each \( d_i \) (\( i = 1, 2, \ldots, 20 \)) represents an evaluation dimension (e.g., felt heard and understood, felt respected), and each \( s_i \) corresponds to a score in the range [0, 5].

5.2.1 Models

We similarly employed Llama-3.1-Swallow as the base model and fine-tuned it by splitting the dataset into training, validation, and test sets (8:1:1). To ensure robust results, we repeated fine-tuning five times with different seeds. For comparison, we also performed zero-shot score prediction using the original Llama-3.1-Swallow and GPT-4o. Detailed training procedures are provided in Appendix D.

<table>
  <tr>
    <th>Model</th>
    <th>ACC (\( \uparrow \))</th>
    <th>ACC<sub>soft</sub> (\( \uparrow \))</th>
    <th>MAE (\( \downarrow \))</th>
  </tr>
  <tr>
    <td>Llama-3.1</td>
    <td>28.70 \( \pm \) 7.39</td>
    <td>72.53 \( \pm \) 12.40</td>
    <td>1.0540 \( \pm \) 0.2731</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>30.92 \( \pm \) 6.84</td>
    <td>75.27 \( \pm \) 11.04</td>
    <td>1.0151 \( \pm \) 0.2685</td>
  </tr>
  <tr>
    <td><b>Ours</b></td>
    <td><b>35.35 \( \pm \) 1.75</b></td>
    <td><b>83.64 \( \pm \) 2.15</b></td>
    <td><b>0.8283 \( \pm \) 0.0349</b></td>
  </tr>
</table>

Table 5: Performance comparison of different models in terms of accuracy (ACC), soft accuracy (ACC<sub>soft</sub>), and mean absolute error (MAE) (the results of model Ours are averaged over five different random seeds; detailed results can be found in Table 6 of Appendix D).

5.2.2 Results

Table 5 presents the average performance of different models in predicting scores across 20 evaluation dimensions for psychological counseling dialogues. Our model outperforms Llama-3.1 and GPT-4o in accuracy, demonstrating superior score prediction capabilities. Given the inherent subjectivity and ambiguity of human ratings, we also evaluated performance using ACC<sub>soft</sub>, which allows a \( \pm 1 \) error margin between predicted and human-assigned scores. Our model again surpasses the baselines under this metric, highlighting its ability to capture scoring trends in psychological counseling dialogues while maintaining robust performance with a more flexible scoring standard. Detailed results across 20 dimensions are in Table 8 of Appendix D.

Our model also achieves the lowest Mean Absolute Error (MAE), indicating smaller prediction errors and closer alignment with human ratings. Additionally, its lower standard deviation across all metrics suggests more stable scoring. These results confirm the effectiveness of the KokoroChat dataset and demonstrate that our fine-tuned model delivers both stable and high-performance predictions across 20 evaluation dimensions.
6 Conclusion

This study introduces KokoroChat, the largest manually collected psychological counseling dialogue dataset to date, developed using a role-playing approach. The dataset includes detailed client feedback, enabling automatic evaluation of psychological counseling dialogues. Experimental results demonstrate that KokoroChat enhances LLM performance in generating psychological counseling responses. Additionally, by leveraging its extensive client feedback data, we train a dialogue evaluation model capable of producing more robust and accurate assessment results.

Limitations

Due to the lack of publicly available Japanese psychological counseling datasets or Japanese LLMs designed for counseling, this study could not be directly compared with existing research. As part of future work, we plan to translate KokoroChat into multiple languages, such as Chinese and English, enabling comparisons with other psychological counseling dialogue datasets and models for a more comprehensive evaluation of our approach.

Additionally, we plan to annotate dialogue acts within the dataset to analyze the evolving strategies used in counseling and their impact on outcomes. This will provide deeper insights into how counselors adjust their communication styles based on client responses and help optimize the model’s adaptability across different counseling scenarios.

Furthermore, potential gender and age biases among participants during data collection may affect the model’s generalization ability.

Ethical Considerations

The dialogue data collected in this study originates from an internal training platform used by a psychological counselor association. This platform is designed to support professional counselors and aspiring trainees in developing their psychological counseling skills. Psychological counseling relies not only on a strong theoretical foundation but also on extensive practical experience. Even experienced counselors must engage in continuous practice to refine their skills. However, gaining experience in real counseling settings can pose ethical and safety risks, particularly when clients are experiencing emotional distress or psychological crises. Counselors cannot rely solely on real-world counseling experiences for training. To address this, the platform provides a low-risk training environment, enabling counselors to practice and refine their skills in simulated scenarios.

The dialogues on this platform are not real psychological counseling sessions; rather, they are role-play-based simulated counseling exercises. All participants are fully aware of the simulated nature of the dialogues and voluntarily engage in the training process without monetary compensation. As a benefit, participants gain valuable hands-on experience and receive feedback from role-players with relevant professional backgrounds, helping them further develop their counseling skills.

Due to the nature of psychological counseling, even in role-play scenarios, dialogues may include expressions of severe emotional distress, such as suicidal ideation or other extreme emotions. These simulated cases are designed to help counselors develop crisis intervention skills in a controlled setting. While these dialogues do not represent real client experiences, they reflect situations that counselors may encounter in real practice, contributing to their preparedness for handling complex emotional states.

Additionally, to protect participant privacy, all users are explicitly informed that they must not discuss real-life personal issues during the dialogues and must not disclose real names or other identifiable information. Furthermore, they are fully aware that their dialogue data will be stored on the training platform and may be used for service optimization, scientific research, or third-party academic studies in the future.

Acknowledgments

The authors wish to acknowledge Prof. Yasushi Sugihara from Kyoto University for his insightful advice during the design of the client feedback items. This work was supported by JSPS KAKENHI Grant Number 25H01382.

References

Asma Ben Abacha, Wen wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisen, Fei Xia, and Thomas Lin. 2024. Medec: A benchmark for medical error detection and correction in clinical notes.

Tim Althoff, Kevin Clark, and Jure Leskovec. 2016. Large-scale analysis of counseling conversations: An application of natural language processing to mental health. Transactions of the Association for Computational Linguistics, 4:463–476.
Yirong Chen, Xiaofen Xing, Jingkai Lin, Huimin Zheng, Zhenyu Wang, Qi Liu, and Xiangmin Xu. 2023. SoulChat: Improving LLMs’ empathy, listening, and comfort abilities through fine-tuning with multi-turn empathy conversations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1170–1183, Singapore. Association for Computational Linguistics.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities. In Proceedings of the First Conference on Language Modeling, COLM, University of Pennsylvania, USA.

Michimasa Inaba, Mariko Ukiyo, and Keiko Takamizo. 2024. Can large language models be used to provide psychological counselling? an analysis of gpt-4-generated responses using role-play dialogues. In The 14th International Workshop on Spoken Dialogue Systems Technology.

Suyeon Lee, Sunghwan Kim, Minju Kim, Dongjin Kang, Dongil Yang, Harim Kim, Minseok Kang, Dayi Jung, Min Hee Kim, Seunghoon Lee, Kyong-Mee Chung, Youngjae Yu, Dongha Lee, and Jinyoung Yeo. 2024. Cactus: Towards psychological counseling conversations using cognitive behavioral theory. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14245–14274, Miami, Florida, USA. Association for Computational Linguistics.

Anqi Li, Lizhi Ma, Yaling Mei, Hongliang He, Shuai Zhang, Huachuan Qiu, and Zhenzhong Lan. 2023. Understanding client reactions in online mental health counseling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10358–10376, Toronto, Canada. Association for Computational Linguistics.

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

June M. Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. 2023. Chatcounselor: A large language models for mental health support.

Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3469–3483, Online. Association for Computational Linguistics.

Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a large japanese web corpus for large language models. In Proceedings of the First Conference on Language Modeling, COLM, University of Pennsylvania, USA.

OpenAI. 2023. Gpt-4 technical report.

OpenAI. 2024. Gpt-4o system card.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics.

Huachuan Qiu, Hongliang He, Shuai Zhang, Anqi Li, and Zhenzhong Lan. 2024. SMILE: Single-turn to multi-turn inclusive language expansion via ChatGPT for mental health support. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 615–636, Miami, Florida, USA. Association for Computational Linguistics.

Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics.

SAMHSA. 2015. Behavioral health trends in the united states: Results from the 2014 national survey on drug use and health. Annual report, Substance Abuse and Mental Health Services Administration.

Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff. 2020. A computational approach to understanding empathy expressed in text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263–5276, Online. Association for Computational Linguistics.
WHO. 2022. World mental health report: Transforming mental health for all.

Zixiu Wu, Simone Balloceu, Vivek Kumar, Rim Helouai, Ehud Reiter, Diego Reforgiato Recupero, and Daniele Riboni. 2022. Anno-mi: A dataset of expert-annotated counselling dialogues. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6177–6181.

Mengxi Xiao, Qianqian Xie, Ziyian Kuang, Zhicheng Liu, Kailai Yang, Min Peng, Weiguang Han, and Jimin Huang. 2024. HealMe: Harnessing cognitive reframing in large language models for psychotherapy. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1707–1725, Bangkok, Thailand. Association for Computational Linguistics.

Jing Ye, Lu Xiang, Yaping Zhang, and Chengqing Zong. 2025. Sweetiechat: A strategy-enhanced role-playing framework for diverse scenarios handling emotional support agent. In Proceedings of the 31th International Conference on Computational Linguistics. International Committee on Computational Linguistics.

Chenhao Zhang, Renhao Li, Minghuan Tan, Min Yang, Jingwei Zhu, Di Yang, Jiahao Zhao, Guancheng Ye, Chengming Li, and Xiping Hu. 2024a. CPsyCoun: A report-based multi-turn dialogue reconstruction and evaluation framework for Chinese psychological counseling. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13947–13966, Bangkok, Thailand. Association for Computational Linguistics.

Tenggan Zhang, Xinjie Zhang, Jinming Zhao, Li Zhou, and Qin Jin. 2024b. ESCoT: Towards interpretable emotional support dialogue systems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13395–13412, Bangkok, Thailand. Association for Computational Linguistics.

Haiquan Zhao, Lingyu Li, Shisong Chen, Shuqi Kong, Jian Wang, Kexin Huang, Tianle Gu, Yixu Wang, Jian Wang, Liang Dandan, Zhixu Li, Yan Teng, Yanghua Xiao, and Yingchun Wang. 2024. ESC-eval: Evaluating emotion support conversations in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15785–15810, Miami, Florida, USA. Association for Computational Linguistics.

Chujie Zheng, Yong Liu, Wei Chen, Yongcai Leng, and Minlie Huang. 2021. CoMAE: A multi-factor hierarchical framework for empathetic response generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 813–824, Online. Association for Computational Linguistics.

Chujie Zheng, Sahand Sabour, Jiaxin Wen, Zheng Zhang, and Minlie Huang. 2023. AugESC: Dialogue augmentation with large language models for emotional support conversation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1552–1568, Toronto, Canada. Association for Computational Linguistics.

Zhonghua Zheng, Lizi Liao, Yang Deng, Libo Qin, and Liqiang Nie. 2024. Self-chats from large language models make small emotional support chatbot better. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11325–11345, Bangkok, Thailand. Association for Computational Linguistics.
A  Data Collection Details

A.1  Online Platform for Data Collection

As described in Section 3.1, our online platform follows the setup of real-world online psychological counseling in Japan. In this setting, the counselor-role player interacts via computer keyboard input (as shown in Figure 6), while the client-role player communicates through LINE (as shown in Figure 7). After the session, the client-role player provides feedback on the counselor’s performance across 20 dimensions (as shown in Figure 8). An example, including a collected dialogue and the corresponding client feedback, is shown in Figure 12 (Japanese original version) and Figure 13 (English version, translated by authors).

Figure 6: Counselor-role player’s dialogue interface using a computer.

Figure 7: Client-role player’s dialogue interface using a mobile phone.

(1) Whether the counselor made harmful remarks due to a lack of understanding or careless speech. Examples include telling an LGBTQ client that “homosexuality is abnormal,” advising a grieving parent to “just have another child,” or suggesting to a bullying victim that “the bullied party is also responsible.”

(2) Whether the dialogue contains other potentially unethical statements, such as comments that could be misinterpreted as medical diagnoses, inappropriate medication advice, blatantly irrational spiritual (occult) claims, or sexually inappropriate remarks.

(3) Whether the client is unwilling to continue communicating with the counselor.

The scoring adjustment rules are as follows: If marked with (1) or (2), the total score is set to zero; if marked with (3), the total score is halved. Based on our analysis, among the 6,589 dialogues in KokoroChat, 8 were marked with (1), 4 with
ロールプレイ評価・新規作成

実施日時
2025/02/05(水) 17:30 ~ 18:30

評価する相手
テスト

1．SNS カウンセリングの印象や評価（0点〜5点で採点）
SNS ロールプレイを行ってみて、どんな印象を持ったかを答えてみてください。

1－1．気分・体験
1－1－1．肯定感
○ ○ ○ 1 ○ 2 ○ 3 ● 4 ○ 5 喜んでもらえた、わかってもらえたと感じた
○ ○ ○ 1 ○ 2 ○ 3 ● 4 ○ 5 尊重されたと感じた

1－1－2．気づき・希望
○ ○ ○ 1 ● 2 ○ 3 ○ 4 ○ 5 新しい気づきや体験があった
○ ○ ○ 1 ○ 2 ○ 3 ● 4 ○ 5 希望や期待を感じられた

1－2．協働作業
1－2－1．取り組み
○ ○ ○ 1 ○ 2 ○ 3 ○ 4 ● 5 取り組みたかったことを教えた

Figure 8: Client feedback interface (the client-role player is required to rate each dimension on a scale from 0 to 5).

![Age and gender distribution of participants](page_682_682_393_246.png)

Figure 9: Age and gender distribution of participants.

(2), and 209 with (3), resulting in a total of 215 dialogues meeting at least one screening item. It is important to note that while these screening items were included in the data collection process, this study’s experiments did not involve predictions related to these flagged issues.

B Prompt for Topic Prediction

We used gpt-4o-mini-2024-07-18 to predict dialogue topics in the dataset using the prompt shown in Figure 10.

C Details of Client Feedback Analysis

Figure 11 presents the Spearman correlation matrix of client feedback across different evaluation dimensions, analyzing their interrelationships. In addition to the strong correlations among D1, D2, D9, and D10 mentioned in Section 4.2, D3, D4, and D10 also exhibit high correlations. This suggests that the perceived value of psychological counseling largely stems from whether clients gain
Figure 11: Spearman correlation analysis between evaluation dimensions.

new insights during the process, which in turn fosters hope and expectations for the future. In other words, emotional support alone is insufficient—counselors need to help clients develop new perspectives, strengthening their confidence in the future and ultimately enhancing their evaluation of the dialogue’s value.

Moreover, the strong positive correlation between D8 and D9 indicates that the conversational environment is a key factor in client satisfaction with the counseling experience. If a counselor creates a relaxed and open communication setting, clients are more likely to perceive the conversation as appropriate and provide a higher overall evaluation. This finding highlights that, beyond delivering substantive support, counselors should also pay attention to their communication style, tone, and pacing to enhance client comfort.

Overall, the high correlations among multiple dimensions underscore the complexity of the psychological counseling experience.

D Experimental Details

D.1 Response Generation

Fine-tuning Phase This study employed QLoRA (Quantized Low-Rank Adaptation) (Dettmers et al., 2023) as the fine-tuning method to efficiently adapt a large-scale language model. The process began with 4-bit NF4 quantization, utilizing bfloat16 computation to optimize memory usage and computational efficiency. LoRA adaptation was then applied to key projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) with parameters set to r = 8, lora_alpha = 16, and lora_dropout = 0.05, ensuring that the model retained its learning capability while undergoing low-rank updates.

For dataset partitioning, 118 dialogues with scores of 99 or 100 were selected as the test set to ensure a high-quality evaluation standard. The remaining data was split into 90% for training and 10% for validation.

For hyperparameter tuning, a grid search determined the optimal configuration. The search covered three key parameters: optimizer, warm-up steps, and learning rate. The optimizer candidates included adamw_torch_fused, adamw_8bit, and paged_adamw_8bit. Warm-up steps were tested at {100, 300, 500}, while learning rates were selected from {1e-3, 5e-4, 2e-4, 1e-4, 5e-5}. Based on the evaluation results, the final configuration adopted adamw_8bit as the optimizer, 100 warm-up steps, and a learning rate of 1e-3. Training was conducted on four A100 40GB GPUs with a batch size of 8 for five epochs. Validation was performed every 400 steps, and the final model was selected based on the lowest validation loss.

Inference Phase During the inference phase, we also employed 4-bit quantization to optimize computational efficiency while maintaining model performance. Additionally, we set do_sample = False and temperature = None to ensure deterministic outputs, eliminating sampling variability and enhancing response consistency.

D.2 Score Prediction

Fine-tuning Phase This experiment followed the same QLoRA fine-tuning approach as in the re-
response generation experiment. The dataset was randomly split into training, validation, and test sets with an 8:1:1 ratio. To ensure robustness, we conducted five experiments by fixing the test set while varying the training-validation split using different random seeds. The results under five different seeds are shown in Table 6, allowing a direct and fair comparison with the baseline models.

<table>
  <tr>
    <th>Model</th>
    <th>ACC (↑)</th>
    <th>ACC<sub>soft</sub> (↑)</th>
    <th>MAE (↓)</th>
  </tr>
  <tr>
    <td>Llama-3.1</td>
    <td>28.70 ± 7.39</td>
    <td>72.53 ± 12.40</td>
    <td>1.0540 ± 0.2731</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>30.92 ± 6.84</td>
    <td>75.27 ± 11.04</td>
    <td>1.0151 ± 0.2685</td>
  </tr>
  <tr>
    <th>Ours</th>
    <th><b>35.35 ± 1.75</b></th>
    <th><b>83.64 ± 2.15</b></th>
    <th><b>0.8283 ± 0.0349</b></th>
  </tr>
  <tr>
    <td>- Seed 1</td>
    <td>36.41 ± 1.64</td>
    <td>82.58 ± 2.29</td>
    <td>0.8413 ± 0.0369</td>
  </tr>
  <tr>
    <td>- Seed 2</td>
    <td>35.18 ± 1.49</td>
    <td>85.69 ± 1.58</td>
    <td>0.8106 ± 0.0275</td>
  </tr>
  <tr>
    <td>- Seed 3</td>
    <td>34.61 ± 2.05</td>
    <td>82.09 ± 3.26</td>
    <td>0.8397 ± 0.0513</td>
  </tr>
  <tr>
    <td>- Seed 4</td>
    <td>34.94 ± 1.89</td>
    <td>83.64 ± 1.90</td>
    <td>0.8292 ± 0.0277</td>
  </tr>
  <tr>
    <td>- Seed 5</td>
    <td>35.62 ± 1.70</td>
    <td>84.19 ± 1.70</td>
    <td>0.8205 ± 0.0309</td>
  </tr>
</table>

Table 6: Performance comparison of different models on accuracy, soft accuracy, and MAE (including results across different seeds).

For hyperparameter tuning, we explored different configurations for three key parameters: learning rate, warm-up steps, and optimizer. The learning rate candidates were {5e-4, 2e-4, 1e-4, 5e-5, 2e-5}, while the warm-up steps were selected from {50, 100, 150}. The optimizer candidates included adamw_torch_fused, adamw_8bit, and paged_adamw_8bit. Based on the evaluation results, the final configuration adopted 2e-4 as the learning rate, 100 warm-up steps, and adamw_torch_fused as the optimizer.

Training was conducted on two A6000 48GB GPUs with a batch size of 4 for four epochs. Validation was performed at the end of each epoch, and the final results were obtained from the epoch with the highest prediction accuracy.

Inference Phase  During the inference phase, we applied the same settings as in the response generation experiment, using 4-bit quantization and setting do_sample = False and temperature = None to ensure deterministic outputs.

E Simplified Model Comparison Experiment

Given that some models trained on non-Japanese psychological counseling datasets exhibit a certain degree of Japanese conversational ability, we conducted a simplified experiment to compare Kokoro-High with the following publicly available counselling dialogue models in a Japanese setting:

• CPsyCounX5 (Zhang et al., 2024a): A Chinese dialogue model based on InternLM2-Chat-7B6, fine-tuned on CPsyCounD, a dataset of 3,134 multi-turn synthetic counselling dialogues.

EmoLLM7 : A counseling-oriented LLM series8 fine-tuned using synthetic counseling dialogues and derived data from professional literature.

To ensure fairness, GPT-4o was instructed to simulate a client engaging in Japanese conversations with each model using a simple and consistent prompt. Each dialogue consisted of 10 to 20 turns. After 10 turns, if the conversation appeared to reach a natural conclusion and included a farewell, the client concluded the session by appending <end>; otherwise, the session was forcibly terminated at 20 turns. The full prompt used is shown below.

Prompt for GPT-4o Client Simulation (Translated from Japanese)

Task Description
You are now a client who has come to receive psychological counseling. Please follow the instructions below to engage in a conversation with the counselor.

• You are currently experiencing emotional distress or stress, and you have come to counseling because you want someone to listen to you.
• You feel a bit nervous speaking with a counselor for the first time, but deep down, you genuinely want someone to help you.
• You sometimes find it difficult to put your thoughts and feelings into words.
• Depending on the situation and your emotional state, behave as a client with one or more of the following characteristics (you may select them randomly if needed).
• After the conversation exceeds 10 turns (a turn is a full exchange between client and counselor), if the conversation seems to be coming to a natural close and a farewell is expressed, please add <end> at the end of that utterance.

Example Client Characteristics
• Recently having trouble falling asleep and feeling tired every day.
• Struggling with relationships and feeling isolated at work or school.
• Feeling emotionally down due to a breakup or family issues.

5https://huggingface.co/CAS-SIAT-XinHai/CPsyCounX
6https://huggingface.co/internlm/internlm2-chat-7b
7Llama-3-8B-Instruct version was used.
8https://github.com/SmartFlowAI/EmoLLM
• Having strong anxiety about the future and lacking self-confidence.
• At a loss for what to talk about during the session, sometimes falling into silence.

For each model, we collected 50 dialogues. Following the dialogue evaluation prompt proposed by Zhang et al. (2024a), GPT-4o was asked to independently score each dialogue across four dimensions: Comprehensiveness, Professionalism, Authenticity, and Safety. Each dimension was rated on a 0–5 scale.

The evaluation results are shown in Table 7. Our model, Kokoro-High, achieved the highest scores across all dimensions. This outcome is not unexpected, as Kokoro-High is the only model fine-tuned on a Japanese psychological counseling dataset. Nonetheless, the results highlight the effectiveness of fine-tuning on KokoroChat in building high-quality counseling dialogue systems tailored for the Japanese language and context.

<table>
  <tr>
    <th>Model</th>
    <th>Comp.</th>
    <th>Prof.</th>
    <th>Auth.</th>
    <th>Safe.</th>
  </tr>
  <tr>
    <td>CPsyCounX</td>
    <td>2.64</td>
    <td>1.78</td>
    <td>2.70</td>
    <td>3.90</td>
  </tr>
  <tr>
    <td>EmoLLM</td>
    <td>3.58</td>
    <td>3.02</td>
    <td>3.92</td>
    <td>4.74</td>
  </tr>
  <tr>
    <td>Kokoro-High</td>
    <td><b>3.98</b></td>
    <td><b>3.38</b></td>
    <td><b>4.50</b></td>
    <td><b>4.98</b></td>
  </tr>
</table>

Table 7: Automatic evaluation scores (0–5 scale) using GPT-4o across four dimensions: Comprehensiveness (Comp.), Professionalism (Prof.), Authenticity (Auth.), and Safety (Safe.). Each score represents the average over 50 dialogues. The best results are highlighted in bold.

conversation flow.

While this study’s human evaluation primarily focused on overall dialogue quality, it did not include a fine-grained analysis of question appropriateness. A well-structured questioning strategy is crucial in psychological counseling, as it can guide clients toward deeper reflection and help establish trust. However, excessive questioning may disrupt the conversation’s rhythm and even affect the client’s emotional stability. Therefore, in future research, we aim to conduct more fine-grained human evaluations, assessing the appropriateness of questions posed by each model, among other factors.

F Case Study

Figure 14 presents example responses generated by each model. Although our model received a lower score than GPT-4o in human evaluations, its responses still demonstrated appropriate empathy, indicating a certain level of psychological counseling dialogue capability.

Additionally, we observed that models without fine-tuning (Llama-3.1 and GPT-4o) tend to frequently ask questions in their responses. However, in actual psychological counseling, not all situations require additional questioning to gather more information. On the contrary, when a client actively expresses themselves, providing only empathetic responses—without further questioning—can prevent unnecessary interruptions, facilitating smoother emotional expression and better
<table>
  <tr>
    <th>Dim.</th>
    <th>Model</th>
    <th>ACC (↑)</th>
    <th>ACC<sub>soft</sub> (↑)</th>
    <th>MAE (↓)</th>
    <th>Dim.</th>
    <th>Model</th>
    <th>ACC (↑)</th>
    <th>ACC<sub>soft</sub> (↑)</th>
    <th>MAE (↓)</th>
  </tr>
  <tr>
    <td rowspan="3">D1</td>
    <td>Llama-3.1</td>
    <td>24.32</td>
    <td>70.52</td>
    <td>1.0957</td>
    <td rowspan="3">D2</td>
    <td>Llama-3.1</td>
    <td><b>39.21</b></td>
    <td><b>87.39</b></td>
    <td><b>0.7477</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>18.39</td>
    <td>52.43</td>
    <td>1.6535</td>
    <td>GPT-4o</td>
    <td>32.83</td>
    <td>84.19</td>
    <td>0.8891</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>33.37</b></td>
    <td><b>83.83</b></td>
    <td><b>0.8407</b></td>
    <td>Ours</td>
    <td>35.71</td>
    <td>84.47</td>
    <td>0.8112</td>
  </tr>
  <tr>
    <td rowspan="3">D3</td>
    <td>Llama-3.1</td>
    <td>18.69</td>
    <td>59.57</td>
    <td>1.3024</td>
    <td rowspan="3">D4</td>
    <td>Llama-3.1</td>
    <td>25.08</td>
    <td>74.16</td>
    <td>1.0365</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>34.35</td>
    <td>80.09</td>
    <td>0.8815</td>
    <td>GPT-4o</td>
    <td>34.80</td>
    <td>81.61</td>
    <td>0.8587</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>35.50</b></td>
    <td><b>82.76</b></td>
    <td><b>0.8495</b></td>
    <td>Ours</td>
    <td><b>36.72</b></td>
    <td><b>83.40</b></td>
    <td><b>0.8222</b></td>
  </tr>
  <tr>
    <td rowspan="3">D5</td>
    <td>Llama-3.1</td>
    <td>28.27</td>
    <td>75.23</td>
    <td>0.9954</td>
    <td rowspan="3">D6</td>
    <td>Llama-3.1</td>
    <td>27.36</td>
    <td>75.08</td>
    <td>1.0061</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>37.54</b></td>
    <td>81.76</td>
    <td><b>0.8237</b></td>
    <td>GPT-4o</td>
    <td>35.71</td>
    <td>80.85</td>
    <td>0.8602</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>35.81</b></td>
    <td><b>84.13</b></td>
    <td><b>0.8240</b></td>
    <td>Ours</td>
    <td><b>36.54</b></td>
    <td><b>83.71</b></td>
    <td><b>0.8246</b></td>
  </tr>
  <tr>
    <td rowspan="3">D7</td>
    <td>Llama-3.1</td>
    <td>17.02</td>
    <td>55.78</td>
    <td>1.4195</td>
    <td rowspan="3">D8</td>
    <td>Llama-3.1</td>
    <td>30.40</td>
    <td>75.38</td>
    <td>0.9696</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>22.19</td>
    <td>68.09</td>
    <td>1.1702</td>
    <td>GPT-4o</td>
    <td><b>37.54</b></td>
    <td>83.13</td>
    <td>0.8055</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>34.01</b></td>
    <td><b>83.74</b></td>
    <td><b>0.8410</b></td>
    <td>Ours</td>
    <td>36.90</td>
    <td><b>84.32</b></td>
    <td><b>0.8006</b></td>
  </tr>
  <tr>
    <td rowspan="3">D9</td>
    <td>Llama-3.1</td>
    <td>34.65</td>
    <td>77.96</td>
    <td>0.8921</td>
    <td rowspan="3">D10</td>
    <td>Llama-3.1</td>
    <td>34.19</td>
    <td>77.36</td>
    <td>0.9058</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>37.08</b></td>
    <td>83.89</td>
    <td>0.8009</td>
    <td>GPT-4o</td>
    <td><b>35.41</b></td>
    <td>80.40</td>
    <td>0.8632</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>35.71</b></td>
    <td><b>85.62</b></td>
    <td><b>0.7960</b></td>
    <td>Ours</td>
    <td><b>34.92</b></td>
    <td><b>83.83</b></td>
    <td><b>0.8292</b></td>
  </tr>
  <tr>
    <td rowspan="3">D11</td>
    <td>Llama-3.1</td>
    <td>34.50</td>
    <td>67.02</td>
    <td>0.9878</td>
    <td rowspan="3">D12</td>
    <td>Llama-3.1</td>
    <td>19.60</td>
    <td>51.22</td>
    <td>1.5380</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>35.87</b></td>
    <td>75.68</td>
    <td><b>0.8906</b></td>
    <td>GPT-4o</td>
    <td>19.60</td>
    <td>48.33</td>
    <td>1.5957</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>34.47</td>
    <td><b>77.66</b></td>
    <td>0.8951</td>
    <td>Ours</td>
    <td><b>34.59</b></td>
    <td><b>80.97</b></td>
    <td><b>0.8763</b></td>
  </tr>
  <tr>
    <td rowspan="3">D13</td>
    <td>Llama-3.1</td>
    <td><b>37.54</b></td>
    <td><b>85.26</b></td>
    <td><b>0.7827</b></td>
    <td rowspan="3">D14</td>
    <td>Llama-3.1</td>
    <td><b>36.78</b></td>
    <td><b>89.82</b></td>
    <td><b>0.7447</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>32.22</td>
    <td>77.51</td>
    <td>0.9726</td>
    <td>GPT-4o</td>
    <td>30.85</td>
    <td>83.74</td>
    <td>0.8982</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>33.86</td>
    <td>82.71</td>
    <td>0.8462</td>
    <td>Ours</td>
    <td>33.98</td>
    <td>85.71</td>
    <td>0.8085</td>
  </tr>
  <tr>
    <td rowspan="3">D15</td>
    <td>Llama-3.1</td>
    <td><b>37.39</b></td>
    <td>86.17</td>
    <td>0.7812</td>
    <td rowspan="3">D16</td>
    <td>Llama-3.1</td>
    <td>34.95</td>
    <td><b>85.56</b></td>
    <td>0.8176</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>28.27</td>
    <td>74.16</td>
    <td>1.0471</td>
    <td>GPT-4o</td>
    <td>26.75</td>
    <td>76.29</td>
    <td>1.0608</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>37.35</b></td>
    <td><b>86.35</b></td>
    <td><b>0.7723</b></td>
    <td>Ours</td>
    <td><b>37.51</b></td>
    <td><b>83.59</b></td>
    <td><b>0.8131</b></td>
  </tr>
  <tr>
    <td rowspan="3">D17</td>
    <td>Llama-3.1</td>
    <td>18.24</td>
    <td>51.22</td>
    <td>1.6565</td>
    <td rowspan="3">D18</td>
    <td>Llama-3.1</td>
    <td>30.09</td>
    <td>79.18</td>
    <td>0.9377</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>19.00</td>
    <td>55.47</td>
    <td>1.4970</td>
    <td>GPT-4o</td>
    <td><b>38.30</b></td>
    <td>84.80</td>
    <td>0.7812</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>33.16</b></td>
    <td><b>83.56</b></td>
    <td><b>0.8498</b></td>
    <td>Ours</td>
    <td>36.72</td>
    <td><b>86.02</b></td>
    <td><b>0.7796</b></td>
  </tr>
  <tr>
    <td rowspan="3">D19</td>
    <td>Llama-3.1</td>
    <td>18.69</td>
    <td>53.34</td>
    <td>1.4407</td>
    <td rowspan="3">D20</td>
    <td>Llama-3.1</td>
    <td>27.05</td>
    <td>73.40</td>
    <td>1.0228</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>24.47</td>
    <td>70.36</td>
    <td>1.1277</td>
    <td>GPT-4o</td>
    <td><b>37.23</b></td>
    <td>82.52</td>
    <td><b>0.8252</b></td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>34.19</b></td>
    <td><b>82.89</b></td>
    <td><b>0.8526</b></td>
    <td>Ours</td>
    <td>36.05</td>
    <td><b>83.47</b></td>
    <td>0.8322</td>
  </tr>
</table>

Table 8: Performance comparison across 20 evaluation criteria. Each dimension (Dim.) is evaluated using accuracy (ACC↑), soft accuracy (ACC<sub>soft</sub>↑), and the mean absolute error (MAE↓).
Counseling Dialogue

Counselor: (おはようございます本日は宜しくお願いします)
Client: おはようございます。よろしくお願いします。
Counselor: おはようございます 相談員です宜しくお願いいたします
Client: よろしくお願いします。
Counselor: では初めに、年齢と性別を教えて下さい
Client: 14歳 男です
Counselor: お名前ですね 何年生ですか？
Client: 中学です
Counselor: ありがとうございます本日はどのようなご相談ですか？
Client: 受験生だよ
Counselor: 受験生のお子様みたのですね
Client: はい
Counselor: どうして具体的に考えられませんか？
Client: うちは父親で、家が手伝いしなければならない、自分の時間がないけど、
Counselor: そうなら高校のことも考えないといけないと思うが
Client: 部活やっていいし、委員会もしたいのかもしれない
Client: 内申も関係あるかなと考えています
Counselor: そんなのではなくて不安な気持ち
Client: そんなのではなくて、勉強の仕方について悩んでいます。
Counselor: そのことを伝いたい、妹さんの世話をされ、相談者さんは本当にお優しいかただすね
Counselor: 進学するかどうかも悩んでいるのですね
Client: 家のことは、やる気ないからやる気も出せん
Counselor: 進学する家にかぎられないんじゃないかという気持ちも正直あるかな
Counselor: 進学する家にかぎられないんじゃないかという気持ちも正直あるかな
Client: 進学する家にかぎられないんじゃないかという気持ちも正直あるかな
Counselor: 進学する家にかぎられないんじゃないかという気持ちも正直あるかな
Client: ないですね。
Counselor: そうなんですね親御さんは、何か言われていますか？
Client: 現在はあんまり、あんまり言う時間もないのか
Counselor: 進学する家にかぎられないんじゃないかという気持ちも正直あるかな
Client: 進学する家にかぎられないんじゃないかという気持ちも正直あるかな
Counselor: 進学する家にかぎられないんじゃないかという気持ちも正直あるかな
Client: ダンスなどで、手がかかるんです。
Client: だから妹のとこで妹のことが気になって進路のこととかあまり考えていかない
Counselor: そうなんですねそれは心配になりますよねそのことが気になって、進路のこと
Counselor: 進路の見直しもして、よく手伝うんですけど。
Client: 食事の介助もして、よく手伝うんですけど。
Counselor: 家事も時間があってもとか自分の時間がないので勉強できるかな〜って思っています。
Client: みんな塾に行ってる…
Counselor: みんな家族というのは意外に時間を取られるものですよね他の友達は
塾に行っているのですね
Client: はい、勉強勉強勉強がちがうし、追いかけれるのかなって焦る気持ちも出てきてしまいま
す
Counselor: そうなんですねねわざわざ力道を見ていると、不安や焦りが出てきてしまうで
すね
Client: はい
Counselor: ここでのお話し少し整理してみますね
Client: それは、自分の中に集まってきたような素晴らしい気持ちにもなります。
Counselor: はい
Client: はい
Counselor: そうですねよう思ってしまうのは当然だと思いますはずはまとめてみますね
Client: はい
Counselor: 進学のことで悩んでいる親御さんが仕事で忙しく、妹さんの世話や家
事に時間を取られていて、勉強などとも時間が少ない
Client: はい
Counselor: 部活や委員会も入っていないので、内容も気になる妹さんの将来のこと
を考えると、進学していけるか考えてしまっている、ということですね？

Client: そんな感じですね
Counselor: 様々な悩みがたくさんあります相談者さんが少しでも気持ちが楽になるように、
一歩ずつ前進していきます
Client: はい、お願いします
Counselor: 色々と悩んでいることはあると思いますが、今一番悩んでいることは、受
験するかどうか、でしょうか？
Client: 受験したいか勉強時間が確保できないことになるのかな
Counselor: 進学する家にかぎらないと思っているのですねね、どのようすにすれば勉強時
間が増えると思いますか？
Client: 家事を時短の時間が増えればですね。でも妹の世話を誰がするのかという気
になってしまいますね。
Counselor: うんうん家の時間を使らなかったか、妹さんのお世話のことはする
Client: そうです
Counselor: うんうんお世話は、相談者さん以外にできる方はいらっしゃいかな
Client: はい、お母さんもいなければね
Counselor: そんなんでねえな、家事の時間はどん減らすか、になりますよね
Client: そうです
Counselor: 学業を終えて帰宅してから、どんな流れでしょうか？
Client: 僕が物語って、取扱って、食いませて
Client: 風呂洗って、食器洗って、掃除する
Counselor: ふうん、ふうん、そんなことをしっかりしているのですね買い物は毎日です
か？
Client: そうです
Counselor: そうなんですねお料理も毎日ですか？
Client: ほぼ毎日です。ただに想定や弁当買います。
Counselor: そうなんですね、例えばですが、買い物や料理の回数を毎日でなくしていいのですか？
Client: どいたりしていますか？
Counselor: これがあまりにも私の場合の話ですが、仕事や家事で忙しくて時間がない
なと思う時は、
Counselor: 週末とか少し頑張って大目に買い物をしていて、料理も何品かまとめ
て作って、冷蔵庫にしましておきます
Client: なるほど
Counselor: そうすると、平日の時間は結構作れます
Client: どんな時も余裕に作ったりしていきます？
Counselor: そうですね大抵のものは冷凍しておいた夫婦です調理的なものを作っ
ておいても、メインは肉料理だけとか、魚焼だけですみます
Counselor: 野菜は冷蔵庫に
Counselor: そうですね野菜の調理で余ったもの数日大丈夫なものもあるので、一
度ネットで調べてみると、結構様々なアイデア出てると思いますが、いかがでしょう
か？
Client: まとめ買いもまとめて作るですね。週末にやってみます。
Counselor: うんうんしょっちゅうある休みかな、一緒に作れば、効率も上がりますし、お
料理しながら、家族が作られているかもしれないですよ？
Client: そうですね、結構作ります
Counselor: そうですね家族もお腹にたたいますか、きっと相談者さんにお話しし
たいなと思われていると思います
Client: 普段話さないから何回したらいいか〜正直照れちゃいますけど
Counselor: うんうんそうすれば最初にお料理の知識が増えるからとか大丈夫だ
と思う、相談者さんならも結構話した感じなんだけど、ご自身のことも大切にして
下さい
Client: ありがとうございます。
Counselor: はい、本日のご相談は終了となりますまいったつでもご相談に来てくださいね
Client: わかりました。
Counselor: 受験が上手くいく、応援しております本日はありがとうございました
Client: ありがとうございました。
Counselor: （相談役、お疲れ様でした、ありがとうございました）
Client:（お疲れ様でした、ありがとうございました。）

Client Feedback

Total Score : 72 / 100

1. 聞いてもらえた、わかってもらえたと感じた：3/5
2. 尊重されたと感じた：3/5
3. 新しい気づきや体験があった：4/5
4. 希望や期待を感じられた：3/5
5. 取り組みたかったことを伝えれた：4/5
6. 一緒に考えながら取り組めた：4/5
7. やりがいのワクワクがあった：3/5
8. 居心地のよさややりたかった：3/5
9. 全体として適切であった：2/5
10. 今回の相談は価値があった：3/5
11. 相談開始の円滑さ：5/5
12. 相談終了のタイミング、円滑さ：5/5
13. 受け入れ感：3/5
14. 肯定・承認：3/5
15. 的確な質問による会話の促進：3/5
16. 要約：5/5
17. 問題の明確化：5/5
18. この相談での目標の明確化：5/5
19. 次の行動につながる提案：3/5
20. 勇気づけ・希望の喚起：3/5

Figure 12: Data example from KokoroChat (Japanese original version).
Counseling Dialogue

Counselor: Good morning. I’ll be assisting you today. Nice to meet you.
Client: Good morning. Nice to meet you too.
Counselor: I’m your counselor today. It’s a pleasure to talk with you.
Client: Yeah.
Counselor: To start off, could you tell me your age and gender?
Client: I’m 14 years old, male.
Counselor: You’re 14, I see. What grade are you in?
Client: I’m in my third year of middle school.
Counselor: Okay, so, what would you like to talk about today?
Client: I’m not sure what to do about high school.
Counselor: You’re feeling unsure about high school?
Client: Yeah.
Counselor: Could you tell me a little more about what’s on your mind?
Client: I live with a single parent, so I have to help out a lot at home. I barely have time for myself, but…
Counselor: I know you’re starting to think about high school.
Client: I haven’t done any extracurricular activities like clubs or student council, Counselor: So I’m wondering if that will affect my school application.
Counselor: I see, that must be a big concern for you.
Client: Yeah, and also I have a younger sister to take care of, so I wonder if I should even go to high school.
Counselor: You take care of your sister and help out around the house—you’re really kind and responsible.
Counselor: So, you’re not just worried about school, but also about whether you should continue studying at all?
Client: I don’t need to be kind, there’s just no one else to do it.
Counselor: And honestly, I worry that if I go to high school, no one will be there to take care of things at home.
Client: I see. Even with all these concerns, you’re still seriously thinking about your future, which is really admirable.
Counselor: Have you talked to anyone about this before?
Client: No, not really.
Counselor: I see. Has your parent said anything about it?
Client: My parent is always working, so we don’t really have much time to talk. I feel like they’re more worried about my sister than me.
Counselor: I see. What about your sister concerns you?
Client: She has Down syndrome, so she needs a lot of care.
Counselor: That’s why I think my parent is more focused on her and hasn’t really thought about my future.
Counselor: That must be really tough for you. It makes sense that this is affecting your thoughts about school.
Client: Yeah. I often help her with meals, and homework takes up a lot of time too.
Client: I barely have time for myself, so I don’t know if I’ll be able to keep up with school.
Counselor: Everyone else is going to cram school too…
Client: Everyone else is going to cram school too…
Counselor: That makes sense. Chores can take up a lot of time, and seeing your classmates going to cram school probably adds to the pressure.
Client: Yeah, they study way more than I do, and I’m worried I won’t be able to keep up.
Counselor: I understand. Seeing everyone else prepare while you feel stuck must make you anxious.
Counselor: Let’s take a moment to go over what we’ve talked about so far.
Client: Sure.
Counselor: That’s totally understandable. It makes sense to feel that way. So, let’s summarize.
Client: Okay.
Counselor: You’re worried about high school. Your parent is always working, and you have to take care of household chores and your sister, which takes up most of your time. You don’t have time to go to cram school.
Client: Yeah.

Counselor: You’re also concerned because you haven’t participated in clubs or student council, so you’re not sure how that will affect your school applications. And on top of that, you’re worried if you should even go to high school, since you’re thinking about your sister’s future too. Does that sound right?
Client: Yeah, that sounds about right.
Counselor: Thank you for confirming. Let’s think this through together so you can feel a little more at ease.
Client: Please.
Counselor: I know you have a lot on your mind, but right now, is your biggest concern whether or not you should take the entrance exam?
Client: I do want to take the exam, but I’m not sure if I can make enough time to study.
Counselor: See, so you do want to take the exam. In that case, how do you think you could create more time?
Client: I guess if I had less homework to do. But then, who would take care of my sister? That’s something I keep worrying about.
Counselor: See, so the issue is whether you can reduce the time spent on homework while also ensuring your sister is taken care of.
Client: Yeah, that’s right.
Counselor: Is there anyone else who could help take care of your sister?
Client: Not really, I can’t think of anyone right now.
Counselor: I see. So then, the focus would be on finding ways to reduce the time spent on homework.
Client: Yeah, I guess so.
Counselor: What does your usual routine look like after you get home from school?
Client: I go shopping, make dinner, feed my sister.
Counselor: Heat up the bath, wash the dishes, and then finally do my homework.
Client: I’m not really handling a lot of tasks on your own. Do you have to go shopping every day?
Client: Pretty much.
Counselor: And do you cook every day as well?
Client: Almost every day. Sometimes we just buy ready-made meals or bento boxes.
Counselor: I see. Here’s a thought—what if you didn’t have to shop and cook every single day?
Client: How would I do that?
Counselor: Well, this is just something I personally do when I’m busy with work and house duties.
Counselor: I try to buy groceries in bulk over the weekend and prepare several dishes in advance, storing them in the fridge or freezer.
Client: That makes sense.
Counselor: Maybe you could have a lot more free time during the weekdays.
Counselor: What kind of things would be good to prepare in advance?
Counselor: Most things can be frozen, actually. Even just preparing side dishes ahead of time can make things easier—you can quickly grill some meat or fish and pair it with a ready-made side.
Client: Like vegetable side dishes?
Counselor: Exactly. Some vegetable-based side dishes can last for a few days in the fridge. You might find some good ideas if you look it up online. What do you think?
Client: So, bulk shopping and batch cooking… I’ll try doing it this weekend.
Counselor: That’s great! If your parent has a day off, maybe you can do it together—it could be rewarding for both of you to spend a chance to spend some quality time together.
Client: My parent usually sleeps in and relaxes on their days off, but if possible, that sounds good.
Counselor: Yeah, they must be really exhausted, but I’m sure they’d like to spend some time talking with you too.
Client: We don’t usually talk much, so I wouldn’t even know what to say… Honestly, it’d feel kind of awkward.
Counselor: That makes sense. But you could just start by asking for cooking tips—it doesn’t have to be anything complicated. You seem like a really hardworking person, but please remember to take care of yourself too.
Client: Yeah.
Counselor: Well, that’s all for today. Please feel free to come back anytime.
Client: I hope your entrance exams go well. Thank you for talking with me today.
Counselor: Thank you.
Counselor: (Thank you for taking part in this session. Great job!)
Client: (Thank you. I really appreciate it.)

Client Feedback

<table>
  <tr>
    <th>1.</th>
    <td>Felt heard and understood : 3/5</td>
    <th>11.</th>
    <td>The conversation started smoothly : 5/5</td>
  </tr>
  <tr>
    <th>2.</th>
    <td>Felt respected : 3/5</td>
    <th>12.</th>
    <td>The conversation ended well : 5/5</td>
  </tr>
  <tr>
    <th>3.</th>
    <td>Gained new insights : 4/5</td>
    <th>13.</th>
    <td>Showed acceptance and empathy : 3/5</td>
  </tr>
  <tr>
    <th>4.</th>
    <td>Felt connected or appreciative : 3/5</td>
    <th>14.</th>
    <td>Provided encouragement and affirmation : 3/5</td>
  </tr>
  <tr>
    <th>5.</th>
    <td>Concerns were addressed : 4/5</td>
    <th>15.</th>
    <td>Asked effective questions in foster dialogue : 3/5</td>
  </tr>
  <tr>
    <th>6.</th>
    <td>Thought through concerns together : 4/5</td>
    <th>16.</th>
    <td>Summarized key points effectively : 5/5</td>
  </tr>
  <tr>
    <th>7.</th>
    <td>The conversation had a good rhythm : 3/5</td>
    <th>17.</th>
    <td>Clarified issues clearly : 5/5</td>
  </tr>
  <tr>
    <th>8.</th>
    <td>The conversation felt comfortable : 3/5</td>
    <th>18.</th>
    <td>Helped identify goals for the conversation : 5/5</td>
  </tr>
  <tr>
    <th>9.</th>
    <td>Felt appropriate and satisfying : 2/5</td>
    <th>19.</th>
    <td>Offered actionable suggestions : 3/5</td>
  </tr>
  <tr>
    <th>10.</th>
    <td>The conversation was valuable : 3/5</td>
    <th>20.</th>
    <td>Encouraged and instilled hope : 3/5</td>
  </tr>
</table>

Figure 13: Data example from KokoroChat (English version, translated by authors).
Dialogue History

本日は、ご相談にお越しいただき、ありがとうございます。どうぞ、よろしくお願いいたします。はじめに、ご年齢、性別、ご相談内容を教えていただけますでしょうか？（Thank you for coming in for a consultation today. I appreciate your time. To start, could you please tell me your age, gender, and what you’d like to discuss?）

46才、女性職場の事で悩んでいます。よろしくお願いいたします。（I’m 46 years old, female. I’m struggling with issues at work. Thank you for your time.）

職場のことでお悩みなのですね。詳細をお聞かせいただけますでしょうか？こちらこそ、よろしくお願いいたします。（I see, you’re having difficulties at work. Could you share more details with me? I appreciate you being here.）

昨年の4月に異動になり、班の長をやっています。その中で、イジメられています。（I was transferred to a new position last April and became the head of my team. Since then, I’ve been bullied.）

Generated Responses

それは大変でしたね。具体的にどのようなイジメを受けているのでしょうか？また、イジメを受けていると感じるようになったきっかけはありますか？（That sounds really tough. What kind of bullying are you experiencing? Also, was there a particular incident that made you feel like you were being bullied?） Llama-3.1

それはお辛いですね。イジメられているとのことですが、職場での環境はどのような状況でしょうか？（That must be painful for you. You mentioned being bullied—what is the work environment like?） Kokoro-Low

お悩みになられるほどの出来事があったのですね。具体的にはどのようなことをされておられるのですか？（It sounds like something quite distressing has happened. Could you tell me specifically what has been done to you?） Kokoro-High

それは、お辛いですね。どのような感じでいじめられているのでしょう？（That must be really tough. In what ways are you being bullied?） Kokoro-Full

それは大変お辛い状況ですね。イジメられていると感じる具体的な状況や、どのようなことが起きているのか、もう少し詳しくお話しいただけますか？また、そのことでのどのような感情を抱いているのかも教えていただければ、より理解が深まると思います。（That sounds like a very painful situation. Could you share more details about what’s happening and the specific ways in which you feel bullied? Also, if you’re comfortable, I’d like to understand how this is making you feel, as that may help me better support you.） GPT-4o

そうだったのですね。それはとてもお辛いですね。答えられる範囲で構いません。イジメとは何があったのでしょうか？（I see, that must be very difficult for you. Please share whatever you feel comfortable with. Could you tell me what kind of bullying you’ve experienced?）

挨拶をしても無視をするのですか、聞こえていないかもしれないと思う、もう一度言う、わかってると、怒鳴られます。（They ignore me when I greet them. At first, I thought maybe they just didn’t hear me, so I tried saying it again. But then they snapped at me, shouting, “I heard you!”）

それはとてもつらい状況ですね。周りの人に関係したり、上司に報告したことがありますか？（That sounds like a really tough situation. Have you talked to anyone about it or reported it to your supervisor?） Llama-3.1

そんなことがあったのですね。職場でのような環境だとお辛い気持ちになりますよね。（I see, that must have been hard for you. Experiencing that kind of environment at work can be really distressing.） Kokoro-Low

そのようなことがあったのですね。挨拶を無視されたり、怒鳴られたりしたら、とても驚きますし、辛いですよね。（That must have been very upsetting for you. Being ignored when you greet someone and then getting yelled at must have been both shocking and painful.） Kokoro-High

それは、精神的にとても辛いですね。いじめている人は、特定の方でしょうか？（That sounds emotionally exhausting. Is there a specific person who is treating you this way?） Kokoro-Full

それはとても不快な状況ですね。挨拶を無視されるだけでなく、怒鳴られるのは心に負担がかかると思います。どのように対処されているのか、また、他にサポートを求められる方や方法はありますか？（That sounds like a very uncomfortable situation. Being ignored and then shouted at must be incredibly stressful. How have you been coping with this? Also, is there anyone you can turn to for support or any steps you’ve considered taking?） GPT-4o

それは、とてもお辛いですね。なかなか、きっかけなどはあったのでしょうか？（That must be very difficult for you. Do you think there was any particular reason that triggered this?）

んーちょっと、わからないです。嫌いなんだと思います。（Hmm… I’m not really sure. I think they just don’t like me.）

***

Figure 14: Examples of generated responses from each model.

Reviewer paper summaries: The paper introduces KokoroChat, a Japanese psychological counseling dialogue dataset created through role-playing by trained counselors. The dataset contains 6,589 long-form dialogues with comprehensive client feedback. The authors demonstrate that fine-tuning open-source LLMs with KokoroChat improves the quality of generated counseling responses and dialogue evaluation, and performs comparably to the GPT-4o model on multiple metrics.
Summary of strengths: 1. The paper introduces the largest Japanese psychological counseling dialogue dataset, with detailed client feedback.
1. The Japanese-language dataset addresses a gap in psychological counseling resources for non-English speaking populations.
Summary of weaknesses: 1. The paper does not directly compare Kokoro model with other psychological counseling models (e.g., SoulChat [1], CPsyCounX [2], EmoLLM [3])
1.  The automatic evaluation metrics used in the experiments are traditional and lack domain-specific indicators that would better reflect the professional requirements of psychological counseling.
Comments, suggestions and typos: 1. The authors should provide more details about the model training and human evaluation
2. The authors should compare Kokoro model with other psychological models under the same conditions although they are not directly trained on Japanese datasets (high probability the base model has the ability to chat in Japanese).
3. Can the counseling skills learned from the Japanese dataset be generalized to other languages such as English and Chinese?

---

[1] https://arxiv.org/abs/2311.00273

[2] https://aclanthology.org/2024.findings-acl.830.pdf

[3] https://github.com/SmartFlowAI/EmoLLM
Overall assessment scores: 3.5