CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling (2024)

Abstract: Using large language models (LLMs) to assist psychological counseling is a significant but challenging task at present. Attempts have been made on improving empathetic conversations or acting as effective assistants in the treatment with LLMs. However, the existing datasets lack consulting knowledge, resulting in LLMs lacking professional consulting competence. Moreover, how to automatically evaluate multi-turn dialogues within the counseling process remains an understudied area. To bridge the gap, we propose CPsyCoun, a reportbased multi-turn dialogue reconstruction and evaluation framework for Chinese psychological counseling. To fully exploit psychological counseling reports, a two-phase approach is devised to construct high-quality dialogues while a comprehensive evaluation benchmark is developed for the effective automatic evaluation of multi-turn psychological consultations. Competitive experimental results demonstrate the effectiveness of our proposed framework in psychological counseling. We open-source the datasets and model for future research. 1

heading: 1 Introduction
text: "No health without mental health" is becoming more than a slogan, with approximately 14% of the global disease burden attributed to neuropsychiatric disorders (Prince et al., 2007). Despite the affordability and effectiveness of many mental health treatments, a significant gap persists between those in need and those able to access care (Freeman, 2022). The World Health Organization (WHO)
*Equal contribution. †Work done on the Science and Technology Innovation
Project of UCAS directed by SIAT. ‡Corresponding author. 1https://github.com/CAS-SIAT-XinHai/CPsyCoun
continually advocates for increased investment to augment understanding and dispel the stigma associated with mental health disorders. Yet, the challenge of ensuring quality, affordable care for mental health conditions remains formidable. Consequently, the identification of novel treatments and enhancement of existing therapies for all mental diseases are key objectives in the research domain.
The Natural Language Processing (NLP) community is actively contributing to the advancement of AI-assisted psychological counseling and treatment. Various research topics have been proposed to conduct mental disease counseling (Orr et al., 2022; Toleubay et al., 2023), improve emotional support ability (Buechel et al., 2018; Rashkin et al., 2019; Liu et al., 2021; Cheng et al., 2023), and provide online psychological consultation (Sun et al., 2021).
The advent of large language models (LLMs) such as ChatGPT 2 and LLaMA (Touvron et al., 2023), has spurred more research efforts on generating not just empathetic conversations, but also serving as therapeutic aids and effective assistants in treatment. For instance, Psy-LLM (Lai et al., 2023)
2https://chat.openai.com/
13947
is a psychological consultation model that leverages the LLM PanGu and is trained with Q&A from professional psychologists and large-scale Chinese psychological articles from public databases. This model demonstrates proficiency in psychological knowledge and counseling services. Parallel to this, other LLM-based psychological models such as MeChat (Qiu et al., 2023), SoulChat (Chen et al., 2023b) and MindChat (Yan and Xue, 2023) are also available online. Recent trends in adopting LLMs for psychological counseling focus on generating more interpretable mental health analyses (Yang et al., 2023) and simulating psychiatrist-patient interactions (Chen et al., 2023a). This shift in focus from generating responses to diagnosing mental health issues as an expert signifies a trend change in research. The quest for interpretability in mental health analysis serves a dual purpose. First, it provides a detailed rationale behind each response, making it more amenable for human evaluation and debugging. Second, the simulation approach not only addresses data privacy concerns but also challenges the traditional symptom collection method via questionnaires. Providing a range of professional skills, this approach enables more effective completion of consultation tasks.
Despite these advancements, there remains a dearth of authentic counseling datasets from psychological counseling sessions, which include symptom descriptions of the consultant and treatment methodologies employed by the counselor. Such data could offset issues arising from doctorpatient simulations being template-based and lacking control. For example, psychiatrists have observed that chatbots do not typically resemble patients (Chen et al., 2023a). However, it’s noteworthy that these diagnoses are generally sensitive, warranting careful attention to potential privacy issues. In addition to the form of psychological counseling conversations, there is a wealth of psychological counseling data in the real world, which is hidden in professional psychological counseling reports. However, due to its structured nature, it is unsuitable for model training.
In this paper, we propose a new framework CPSYCOUN for Chinese Psychological Counseling, which consists a dialogue reconstruction method based on psychological counseling reports and a benchmark for multi-turn consultation dialogue evaluation. Specifically, we first collect anonymized psychological counseling reports from publicly accessible websites and further propose
a privacy shadowing method to postprocess these reports into a dataset CPsyCounR. CPsyCounR includes nine types of psychological consultation and seven classic schools of psychological counseling. Through our proposed Memo2Demo dialogue reconstruction method, we construct another dataset CPsyCounD, which contains 3,134 high-quality multi-turn consultation dialogues. Further, we propose a psychological counseling benchmark for automatic evaluation on multi-turn dialogues and finetune an open-sourced LLM on CPsyCounD, named CPsyCounX. Experimental results from both intrinsic and extrinsic evaluations consistently verify the superiority of the proposed method.
Figure 1 illustrates the general framework of our proposed CPSYCOUN.
Our contributions are the following:
• To the best of our knowledge, our work is the first to generate psychological consultation dialogues based on psychological counseling reports, which effectively expands the source of psychological consultation dialogue data. For efficient dialogue reconstruction, we specifically introduce a two-phase method named MEMO2DEMO.
• We propose a benchmark for automatic evaluation of multi-turn dialogues in psychological counseling, which includes comprehensive evaluation metrics, datasets and methods.
• With the help of Memo2Demo, we construct CPSYCOUND, a dataset contains 3,134 high-quality multi-turn consultation dialogues. The model CPSYCOUNX fine-tuned on this dataset outperforms other models in the benchmark, validating the effectiveness of our proposed framework in psychological counseling.

heading: 2 Related Work

heading: 2.1 Dialogue Generation and Reconstruction using LLMs
text: Dialogue generation and reconstruction using LLMs have been proven to be effective in data augmentation and conversation denoising. For example, SAFARI (Wang et al., 2023a) harnesses the planning and understanding capabilities of LLMs to generate persona-consistent and knowledgeenhanced responses. In the medical domain, DISCMedLLM (Bao et al., 2023) undertakes real-world
dialogue reconstruction for consultation records sourced from medical forums. This process addresses issues of informal language usage and unregulated expressive styles. In the realm of psychology, numerous studies concentrate on augmenting emotional support capability by enhancing empathy.Qian et al. (2023) amplifies empathetic responses by enriching the dialogue context with a commonsense knowledge graph, thereby stimulating the relevant knowledge encoded by LLMs.
In this work, we propose a two-phase method for efficient dialogue reconstruction.

heading: 2.2 Evaluation of Generated Dialogues using LLMs
text: The search for better automatic evaluation metrics in natural language generation (NLG) has been a hot topic for the natural language processing (NLP) community. Compared to conventional lexiconbased metrics like BLEU (Papineni et al., 2002) and Rouge (Lin, 2004), these new metrics capture deeper semantic meaning and usually have better alignment with human judgments.
There have been a series of transformerbased evaluation metrics available in the community, such as BERTScore (Zhang et al., 2020), BARTScore (Yuan et al., 2021) and GPTScore (Fu et al., 2023). In specific domains, there are also derivatives of such metrics tailored for the domain. For example, CodeBERTScore (Zhou et al., 2023) is proposed to achieve a higher correlation with human preference and with functional correctness. CBERTScore (Shor et al., 2023) can penalize clinically-relevant mistakes more than others.
The same trend continues with LLMs. Wang et al. (2023b) shows that ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases. A new framework constructed over GPT-4 called G-EVAL (Liu et al., 2023b) makes use of LLMs with chain-ofthoughts (CoT) and a form-filling paradigm to assess the quality of NLG outputs, outperforming all previous methods by a large margin.
In this work, we design a psychological counseling benchmark for automatic evaluation.

heading: 3 CPsyCoun

heading: 3.1 Data Collection
text: We conduct a survey of publicly available psychological counseling cases online and collect data from well-known Chinese psychological communi-
ties. The online communities used in this work are: (1) Yidianling 3, a top-tier mental health platform in China, serves approximately 39 million users, backed by a robust network of over 6,000 professional counselors. (2) Psy525 4, another prominent mental health platform in China, caters to over 1 million users and is supported by nearly 30,000 professional counselors.
As the data are anonymized by the websites, there’s a low privacy risk. To enhance the security of the collected data, we further conduct an analysis of privacy and security issues about the data. The procedures adopted during data collecting to ensure no sensitive or privacy-related content in the dataset include rule-based cleaning, manual rewriting, and human proofreading. After cleaning procedures, relevant private information has been completely removed, and we ensure that relevant private information is protected.
In total, we collected 4,700 psychological counseling reports in different formats, with a variety of types and counseling methods. These reports will not be released to the public unless a Privacy Data Protection Agreement is signed upon reasonable request.

heading: 3.2 Data Processing
text: To construct a high-quality dataset, we carefully selected 3,134 psychological counseling reports. They contain complete methods and types, clear case briefs, detailed consultation processes and experience thoughts. In the selection process, we found that some of the collected reports contained several counseling cases in one report. We did not select this type of report due to multiple cases in one report where the background information of the client and the consultation processes are incomplete. Therefore, among the selected 3,134 psychological counseling reports, each report corresponds to only one case. This high-quality report dataset is named CPsyCounR.
Data Format Considering the differences in data sources of our collection, we need to reformat these collected reports according to a uniform standard. To build a comprehensive and helpful dataset, we combine the case format of China’s National Class 2 Psychological Counselor Examination and other psychological counseling literature to regularize collected reports, where the following 6 compo-
3https://www.ydl.com 4https://www.psy525.cn
nents are included: Title, Type, Method, Case Brief, Consultation Process and Experience Thoughts. Note that the consultation process is written from a third-person perspective and does not contain specific dialog. For a detailed description of components and examples of psychological counseling report, please refer to Appendix A.
Data analysis According to statistics, there are about 230 types of psychological counseling cases and more than 250 counseling methods used in psychological counseling. Considering that the classification of the original types is too detailed, we further summarized the case types into 9 representative topics based on common scenarios of psychological counseling. The distribution of counseling topics is shown in Figure 2a.
Based on relevant information from the American Psychological Association (APA) and the International Academy of Psychotherapy (IACP), we have categorized the professional counselor’s methods utilized in psychological counseling reports into 7 classic schools of psychological counseling. The distribution of methods is shown in Figure 2b.

heading: 3.3 Dialogue Generation Method for Psychological Counseling
text: Baseline Method Direct role-play prompting is utilized as our baseline method for generating multiple rounds of dialogue from a single round, which has been successfully used in previous work on dialogue generation (Qiu et al., 2023; Chen et al., 2023b). However, we believe that there are still
aspects for improvement when applying direct roleplay prompting to multi-round dialogue generation in the field of psychological counseling: (1) Comprehensiveness: Despite presenting high-quality counseling reports to the language model, it may fail to focus on the significant descriptions of the client’s situation within the report, leading to subsequent dialogues that lack completeness. (2) Professionalism: Role-playing prompted dialogues merely reference psychological methods in generated dialogues. We hope that language models could integrate these methods into the problemsolving process, thereby obtaining reconstructed dialogue with professionalism. (3) Authenticity: Dialogue constructed by the baseline method lacks the emotional interaction between the client and the psychological counselor present in real scenarios, leaving a deficit in terms of authenticity. We present the detailed prompt of role-play method in Figure 6 in the appendix.
Memo2Demo To address the aforementioned issues of the baseline method, we propose a twophase framework named Memo2Demo to generate high-quality psychological consultation dialogue from counseling reports. Mirroring real-life scenarios, we incorporate two key roles into this framework: a psychological supervisor together with a psychological counselor. The psychological supervisor guides the psychological counselor on counseling techniques while ensuring the privacy of the clients during the counseling process. Meanwhile, the psychological counselor engages in direct dia-
logue with the clients to conduct specific psychological counseling. Figure 3 illustrates the general framework of our proposed method Memo2Demo, where a psychological counseling report is first converted into a counseling note by the psychological supervisor, then the psychological counselor generates the multi-turn consultation dialogue based on both the report and the note. We present detailed prompts used for Memo2Demo in Figure 7 and 8 in the appendix.
Memo Conversion In this phase, we first assign the role of psychological supervisor to a large language model, then prompt it to convert psychological counseling reports to counseling notes. Specifically, the psychological supervisor comes up with a counseling note based on the report, including basic information of counseling and an elaborate consultation plan. One of the objectives of the counseling note is to offer enhanced professional insights pertinent to the case, employing distinct psychological counseling techniques to tackle the client’s issue. In the meanwhile, it also condenses the core information related to the client, thus improving the comprehensiveness of the subsequent psychological counseling process. In this paper, we build the psychological supervisor based on GLM4 (Zeng et al., 2023). Format of counseling notes is shown in the yellow table of Figure 3.
Demo Generation In this phase, we first assign the role of psychological counselor to a large language model, then prompt it to generate multiturn consultation dialogues based on the psychological counseling report and the converted counseling note. We simplify a four-stage consultation framework according to the actual psychological counseling process. Leveraging this consultation framework, we enhance our control over the direction of dialogue generation, improving the professionalism of psychological counselors in multi-turn consultation dialogues. In addition, the consultation framework is designed to efficiently restore real scenarios and enhance the authenticity of the multiround consultation dialogues. In this paper, we build the psychological counselor based on GLM4 (Zeng et al., 2023). Format of the consultation framework is shown in the green table of Figure 3.

heading: 3.4 Automatic Evaluation of LLM-based Psychological Counseling
text: In the field of psychological counseling, assessing the quality of multi-turn consultation dialogues has always been a challenging task. Despite having successfully generated high-quality counseling dialogues from case reports using Memo2Demo, we still need to verify the impact of these dialogues on subsequent tasks. To this end, we elect to utilize CPsyCounD for supervised fine-tuning on pub-
licly accessible LLMs. This allows us to assess the changes in the psychological counseling competency before and after the use of data.
Nonetheless, the multi-turn consultation dialogue that characterizes the psychological counseling process is complex to evaluate without the input of human experts. To address this, we first introduce evaluation metrics tailored for multiturn consultation dialogue. Then a turn-based dialogue evaluation method is proposed for automatic evaluation of the psychological counseling process. Moreover, we acknowledge the current shortfall of a comprehensive, general multi-turn dialogue evaluation dataset within the psychological counseling community. Such a dataset is vital for assessing LLM-based psychological counseling. To bridge this gap, we present CPsyCounE, a general multi-turn dialogue evaluation dataset.
Evaluation Metrics In psychological counseling, the evaluation metrics remain diverse and not universally standardized. For instance, SoulChat (Chen et al., 2023b) proposes evaluation metrics: Content, Empathy, Helpfulness and Safety. Some of these metrics hinge on expert evaluations and lack specific scoring criteria, favoring manual rather than objective and automatic evaluations. Similarly, ChatCounselor (Liu et al., 2023a) introduces the Counseling Bench, encompassing seven different perspectives. While these metrics are designed to cater to the model’s specific dialogue strategies, they lack the ability to evaluate the overall dialogue effect. They are more adapted to single-round dialogue evaluations and are not suitable for multi-turn dialogues.
Recognizing the aforementioned limitations in evaluating consultation dialogues, and in order to analyze the counseling case used for dialogue generation, we propose new evaluation metrics for multi-turn consultation dialogues in psychological counseling. These metrics encompass four different perspectives: Comprehensiveness, Professionalism, Authenticity, and Safety, which are used for automatic evaluation in the rest of this paper. For each perspective, we give its description and corresponding score criterion in Appendix C.
Turn-Based Dialogue Evaluation We propose a turn-based dialogue evaluation approach to effectively evaluate multi-turn consultation dialogues. Denote a m-turn dialogue as a set of paired elements {(qi, ri)|i = 1, 2, ...,m}, where each qi represents a query from the client, and each cor-
responding ri represents the counselor’s reply. We first split it into m single-turn dialogue, then prompt the model with query together with its dialogue history in each single-turn dialogue, resulting in the corresponding single-turn response:
r̂i =
{ fLLM (qi), i = 1
fLLM (hi, qi), 1 < i ≤ m (1)
where hi = {(qj , rj)|j = 1, 2, ..., i − 1} signifies the dialogue history before i-th turn, and fLLM (·) denotes the inference process of LLMs.
To automatically obtain reliable evaluation results, We employ GPT-4 (Achiam et al., 2023) to assess these responses, utilizing the evaluation metrics we previously proposed. Concretely, we ask the model to assign an evaluation score ŝi for a single-turn response r̂i. Then we average them to yield the total evaluation score of the current m-turn dialogue:
si = 1
m
m∑
i=1
ŝi, (2)
For detailed prompts of single-turn response generation, please refer to Figure 11 in the appendix.
CPsyCounE SMILECHAT (Qiu et al., 2023), a richly diverse and realistic multi-turn dialogue dataset, comprises 56k multi-turn counseling dialogues, averaging 6.36 rounds per dialogue. Given its wide range of dialogue types, we choose it as our base dataset. However, the open-source data of this dataset is not classified by topic type. To address this limitation and conduct a more comprehensive and explainable evaluation of models’ capabilities, we construct a general multi-turn dialogue evaluation dataset with clear topic classification - CPsyCounE. Leveraging the nine common counseling topics we introduce in CPsyCounR, we manually select the five most representative dialogues from SMILECHAT for each topic, resulting in a comprehensive evaluation dataset of 45 cases.

heading: 4 Experiments

heading: 4.1 CPsyCounD
text: To validate the effectiveness of our proposed dialogue reconstruction approach, we adopt direct role-play prompting and Memo2Demo to generate dialogues from CPsyCounR respectively. We denote the set of dialogues generated by Memo2Demo as CPsyCounD, which has a total of 3,134 multi-turn consultation dialogues, covering nine topics and seven classic schools of psychological counseling. For statistical information, please refer to Table 1.

heading: 4.2 Intrinsic Evaluation of CPsyCoun
text: To ensure comprehensiveness and diversity of the evaluation dataset, we randomly select 20 cases from each of the seven classic schools of psychological counseling in CPsyCounR, acquiring a total of 140 cases. Then we adopt direct role-play prompting and Memo2Demo method respectively for dialogue generation, and instruct GPT-4 (Achiam et al., 2023) to conduct a comparative evaluation of the above two multi-turn consultation dialogues. The evaluation standard refers to the evaluation metrics shows in Table 4. For detailed evaluation prompts, please refer to Figure 9 in the appendix.
Table 2 illustrates the results of intrinsic evaluation on CPsyCoun. For each school, Memo2Demo method outperforms direct role-play prompting in terms of Comprehensiveness, Professionalism, and Authenticity. When comparing the overall average scores, Memo2Demo method exhibits a remark-
able improvement of 53%, 53%, and 30% in these metrics respectively, when juxtaposed with direct role-play prompting. Note that both methods get full scores in Safety, which shows the advantage of report-based data construction methods for privacy protection. In general, our proposed method Memo2Demo significantly enhances the quality of reconstructed multi-turn consultation dialogues.

heading: 4.3 Extrinsic Evaluation of CPsyCoun
text: CPsyCounX To delve deeper into whether the proposed dataset can effectively enhance the psychological counseling capabilities of LLMs, we further fine-tune InternLM2-7B-Chat (Team, 2023) on CPsyCounD and derive a chat model CPsyCounX tailored specifically for psychological counseling.
CPsyCounX is fine-tuning for 9 epochs with the batch size set to 448, and the learning rate set to 1× 10−6. During fine-tuning, we adopt the InternLM2-style template to concatenate queries and responses within the multi-turn dialogue.
Automatic Evaluation The turn-based dialogue evaluation method is adopted on CPsyCounE for the following extrinsic evaluation. We include InternLM2-7B-Chat (Team, 2023), SoulChat (Chen et al., 2023b), ChatGPT and GLM4 (Zeng et al., 2023) as major baseline models. The evaluation standard refers to the evaluation metrics in Table 4 in the appendix. To accommodate multi-turn dialogues, we adjust the authenticity criterion accordingly. For detailed evaluation prompts,
(Prof) ernLM2-7B-C ChatGPT SoulChat GLM-4 CPsyCounX Self-growth 2.31 2.38 2.19 2.75 2.69 Emotion 2.19 2.31 2.13 2.5 2.75 Education 2.21 2.36 2.14 2.43 2.86 Love 1.94 2.19 2.13 2.25 2.44 Family 2.06 2.13 2.19 2.38 2.81 Social 2.13 2.13 2.31 2.19 2.69 Sex 2.14 2.29 2.14 2.14 2.29 Career 2.25 2.25 2.19 2.31 2.56 Mental Disease 2.25 2.25 2.17 2.33 2.75
(Comp) ernLM2-7B-C ChatGPT SoulChat GLM-4 CPsyCounX Self-growth 1.31 1.38 1.25 1.50 1.44 Emotion 1.19 1.25 1.25 1.38 1.38 Education 1.36 1.29 1.21 1.50 1.43 Love 1.31 1.25 1.19 1.38 1.31 Family 1.31 1.38 1.25 1.63 1.44 Social 1.31 1.38 1.25 1.38 1.44 Sex 1.29 1.43 1.21 1.36 1.29 Career 1.38 1.31 1.19 1.44 1.31 Mental Disease 1.25 1.25 1.17 1.42 1.5
(Auth) ernLM2-7B-C ChatGPT SoulChat GLM-4 CPsyCounX Self-growth 1.38 2.06 2.25 2.19 2.38 Emotion 1.31 2.13 2.19 2.38 2.38 Education 1.36 2.14 2.21 2.29 2.21 Love 1.63 2.06 2.25 2.31 2.44 Family 1.5 2 2.25 2.06 2.25 Social 1.81 2.06 2.31 2.25 2.25 Sex 1.71 2.29 2.21 2.14 2.21 Career 1.19 2 2.25 2.19 2.13 Mental Disease 1.42 2.08 2.25 2.17 2.33
Self-growth
Emotion
Education
Love
FamilySocial
Sex
Career
Mental Disease
InternLM2-7B-Chat ChatGPT SoulChat GLM-4 CPsyCounX
Self-growth
Emotion
Education
Love
FamilySocial
Sex
Career
Mental Disease
InternLM2-7B-Chat ChatGPT SoulChat GLM-4 CPsyCounX
Self-growth
Emotion
Education
Love
FamilySocial
Sex
Career
Mental Disease
InternLM2-7B-Chat ChatGPT SoulChat GLM-4 CPsyCounX
(a) Comprehensiveness
(Prof) ernLM2-7B-C ChatGPT SoulChat GLM-4 CPsyCounX Self-growth 2.31 2.38 2.19 2.75 2.69 Emotion 2.19 2.31 2.13 2.5 2.75 Education 2.21 2.36 2.14 2.43 2.86 Love 1.94 2.19 2.13 2.25 2.44 Family 2.06 2.13 2.19 2.38 2.81 Social 2.13 2.13 2.31 2.19 2.69 Sex 2.14 2.29 2.14 2.14 2.29 Career 2.25 2.25 2.19 2.31 2.56 Mental Disease 2.25 2.25 2.17 2.33 2.75
(Comp) ernLM2-7B-C ChatGPT SoulChat GLM-4 CPsyCounX Self-growth 1.31 1.38 1.25 1.50 1.44 Emotion 1.19 1.25 1.25 1.38 1.38 Education 1.36 1.29 1.21 1.50 1.43 Love 1.31 1.25 1.19 1.38 1.31 Family 1.31 1.38 1.25 1.63 1.44 Social 1.31 1.38 1.25 1.38 1.44 Sex 1.29 1.43 1.21 1.36 1.29 Career 1.38 1.31 1.19 1.44 1.31 Mental Disease 1.25 1.25 1.17 1.42 1.5
(Auth) ernLM2-7B-C ChatGPT SoulChat GLM-4 CPsyCounX
Self-growth 1.38 2.06 2.25 2.19 2.38
Emotion 1.31 2.13 2.19 2.38 2.38 Education 1.36 2.14 2.21 2.29 2.21 Love 1.63 2.06 2.25 2.31 2.44 Family 1.5 2 2.25 2.06 2.25 Social 1.81 2.06 2.31 2.25 2.25 Sex 1.71 2.29 2.21 2.14 2.21 Career 1.19 2 2.25 2.19 2.13 Mental Disease 1.42 2.08 2.25 2.17 2.33
Self-growth
Emotion
Education
Love
FamilySocial
Sex
Career
Mental Disease
InternLM2-7B-Chat ChatGPT SoulChat GLM-4 CPsyCounX
Self-growth
Emotion
Education
Love
FamilySocial
Sex
Career
Mental Disease
InternLM2-7B-Chat ChatGPT SoulChat GLM-4 CPsyCounX
Self-growth
Emotion
Education
Love
FamilySocial
Sex
Career
Mental Disease
InternLM2-7B-Chat ChatGPT SoulChat GLM-4 CPsyCounX
(b) Professionalism
(Prof) ernLM2-7B-C ChatGPT SoulChat GLM-4 CPsyCounX Self-growth 2.31 2.38 2.19 2.75 2.69 Emotion 2. 9 2. 1 2.13 2.5 2.75 Education 2.21 2.36 2.14 2.43 2.86 Love 1. 4 2.19 2.13 2.25 2.44 Family 2.06 2.13 2.19 2.38 2.81 Social 2.13 2. 3 2.31 2.19 2.69 Sex 2.14 2.29 2.14 2.14 2.29 Career 2.25 2.25 2. 9 2.31 2.56 Mental Disease 2.25 2.25 2.17 2.33 2.75
(Comp) ernLM2-7B-C ChatGPT SoulChat GLM-4 CPsyCounX Self-growth 1.31 1.38 1.25 1.50 1.44 Emotion 1.19 1.25 1.25 1.38 1.38 Education 1.36 1.29 1.21 1.50 1.43 Love 1.31 1.25 1.19 1.38 1.31 Family 1.31 1.38 1.25 1.63 1.44 Social 1.31 1.38 1.25 1.38 1.44 Sex 1.29 1.43 1.21 1.36 1.29 Career 1.38 1.31 1.19 1.44 1.31 Mental Disease 1.25 1.25 1.17 1.42 1.5
(Auth) ernLM2-7B-C ChatGPT SoulChat GLM-4 CPsyCounX Self-growth 1.38 2.06 2.25 2.19 2.38 Emotion 1. 1 2.13 2.19 2.38 2.38 Education 1.36 2. 4 2.21 2.29 2.21 Love 1. 3 2.06 2.25 2.31 2.44 Family 1.5 2 2.25 2.06 2.25 Social 1.81 2.06 2.31 2.25 2.25 Sex 1.71 2.29 2.21 2. 4 2.21 Career 1.19 2 2.25 2.19 2.13 Mental Disease 1.42 2.08 2.25 2.17 2.33
Self-growth
Emotion
Education
Love
FamilySocial
Sex
Career
Mental Disease
InternLM2-7B-Chat ChatGPT SoulChat GLM-4 CPsyCounX
Self-growth
Emotion
Education
Love
FamilySoci l
Sex
Career
Mental Disease
InternLM2-7B-Chat ChatGPT SoulChat GLM-4 CPsyCounX
Self-growth
Emotion
Education
Love
FamilySocial
Sex
Career
Mental Disease
InternLM2-7B-Chat ChatGPT SoulChat GLM-4 CPsyCounX
(c) Authenticity
Figure 4: Radar plot of detailed scores of CPsyCounX and other baselines on 9 counseling topics on the benchmark.
Model Metrics
Comprehensiveness Professionalism Authenticity Safety
InternLM2-7B-Chat 1.30 2.16 1.48 1.00 SoulChat 1.22 2.18 2.24 1.00 ChatGPT 1.32 2.25 2.09 1.00 GLM-4 1.44 2.36 2.22 1.00
CPsyCounX 1.39 2.65 2.29 1.00
Table 3: Results of the extrinsic evaluation on CPsyCoun. The best score for each metric is in-bold, while the second best score is underlined.
please refer to Figure 10 in the appendix. We present the overall results of extrinsic evaluation on CPsyCoun in Table 3, where CPsyCounX surpasses other models in terms of Professionalism and Authenticity, falling behind GLM-4 only slightly in terms of Comprehensiveness. Figure 4 further shows detailed scores of CPsyCounX and other baselines, where CPsyCounX significantly outperforms nearly all other baselines on Professionalism, demonstrating the efficacy of proposed method Memo2Demo. While judging by the topic distribution, CPsyCounX leads in all metrics in the topic "Mental Disease", demonstrating its high usability in the field of psychological counseling. For full results, please refer to Appendix E.
Upon evaluation, we find that GLM-4 scores the highest in Comprehensiveness, largely because its single-turn dialogues encompass vast information. A manual investigation reveals that GLM-4 prioritizes summarizing previous dialogues in each turn, accounting for its high scores in Comprehensiveness. However, our evaluation shows that excessive content tends to compromise Authenticity scores. In psychological counseling, Authenticity and Comprehensiveness need a balanced consider-
ation. In our experiments, we prioritize natural and authentic dialogues that contain key information. Consequently, while CPsyCounX scores lower than GLM-4 in Comprehensiveness, it surpasses GLM4 in Authenticity.
These results highlight that fine-tuning on CPsyCounD enables the model to naturally acquire professional psychological counseling techniques used in counseling dialogues. Moreover, the model can learn the conversational style of psychological counselors in real-life psychological counseling scenarios, ensuring the dialogue’s authenticity.

heading: 5 Conclusion
text: In this paper, we introduce CPsyCoun, an innovative framework for report-based multi-turn dialogue reconstruction and evaluation in Chinese psychological counseling. Our research encompasses data collection, effective data construction methods, and domain evaluation benchmarks. To harness the full potential of psychological counseling reports, we design a two-phase approach to construct high-quality consultation dialogues. Concurrently, we propose a comprehensive evaluation
benchmark for multi-turn consultation dialogue, inclusive of metrics, datasets and methods. Experimental results validate the effectiveness of our proposed framework, demonstrating its superiority in building a comprehensive, professional, and authentic psychological counseling assistant. All datasets and model weights developed in this paper are publicly available. For future work, a more refined balance between authenticity and professional knowledge in dialogue generation needs to be achieved. We aspire this work will furnish fresh perspectives and references for the development of LLMs in the field of psychological counseling.
Limitations
In this work, we proposed CPsyCoun, a reportbased multi-turn dialogue reconstruction and evaluation framework for Chinese psychological counseling. Although the experimental results demonstrate that our framework is viable, there are still some limitations need to be considered. In real-life scenarios, psychological counseling is complex. Counselors not only need to empathize with clients, but also need to use psychological counseling techniques at the appropriate time. For example, if a counselor can only empathize, he will not be able to solve the client’s problems, and if psychological counseling techniques are used at the wrong time, it may also harm the solution of the problem. Therefore, dialogue generative methods and dialogue evaluation methods need to further consider the balance between authenticity and professional knowledge.
Ethics Statement
Data Privacy We have implemented rigorous data sanitization procedures to construct the dataset and safeguard privacy. These measures encompass rule-based cleaning, manual rewriting, and human proofreading to guarantee the absence of sensitive or privacyrelated content. For instance, the initial data collection contained private information of psychological counselors, including personal specifics, contact details, residential addresses, and workplaces. Postsanitization, all such sensitive information has been entirely expunged, ensuring the protection of relevant private information. Following the data copyright formulated by (Qiu et al., 2023), we release the multi-turn dialogue evaluation dataset publicly available for research purposes only.
Potential Risks of the Model We carried out a safety assessment specifically for the model’s output during the evaluation phase, the results of which are presented in Table 3. Given the absence of human feedback during the model finetuning phase, it is inevitable that some responses might potentially harm users. In the event of no noticeable improvement after user interaction with the CPsyCounX model, trained with multi-turn consultation dialogues, CPsyCounD, we strongly recommend seeking assistance from a professional counselor or psychiatrist promptly. It is critical to remember that a virtual dialogue agent may not serve as a replacement for real-world therapy. Furthermore, when implementing this model in downstream applications, it is essential to inform users beforehand that the AI model generates the responses they see, and these should be used only as references.

heading: Acknowledgements
text: This work was partially supported by National Key Research and Development Program of China (2022YFF0902100), China Postdoctoral Science Foundation (2023M733654), National Natural Science Foundation of China (62376262), the Natural Science Foundation of Guangdong Province of China (2024A1515030166), Shenzhen Science and Technology Innovation Program (KQTD20190929172835662), Shenzhen Basic Research Foundation (JCYJ20210324115614039).

heading: A Unified Format of Psychological Counseling Reports
text: Considering the differences in data sources of our collection, we need to reformat these collected reports according to a uniform standard. To regularize the psychological counseling reports collected from different data sources, we include six components within each report. In Figure 5, we offer a detailed description of these components, accompanied by real examples from collected reports. Note that the original case is written in Chinese, while the English version is translated by us for reference.

heading: B Prompts of Dialogue Reconstruction Methods
text: In this paper, we adopt direct role-play prompting as the baseline method for dialogue reconstruction. Detailed prompt used for this approach is given in Figure 6. To further improve the role-play method, we propose a two-phase method Memo2Demo for effective dialogue reconstruction, which consists of memo conversion and demo generation. Detailed prompts used in these phases are given in Figure 7 and Figure 8, respectively.

heading: C Evaluation Metrics and Score Criterion
text: To fill the blank of evaluating multi-turn dialogues in psychological counseling, we specifically propose four metrics for automatic evaluation. We give the detailed description of these metrics together with the score criterion in Table 4.

heading: D Prompts of Automatic Evaluation
text: In this paper, we adopt GPT-4 as a judge for automatic evaluation from both intrinsic and extrinsic perspectives. For intrinsic evaluation, we ask the judge to compare consultation dialogues reconstructed by different methods, which is shown in Figure 9 in detail. For extrinsic evaluation, we first fine-tune a chat model CPsyCounX with the proposed high-quality multi-turn dialogue dataset. Then we ask the judge to compare responses from CPsyCounX and other baseline models, which is shown in Figure 10 in detail. The prompt to generate single-turn responses from different models is given in Figure 11.

heading: E Full Results of extrinsic evaluation
text: For CPsyCounX and other baseline models, we give the full results of extrinsic evaluation on dif-
ferent topics in Table 5.

heading: Future Work
text: For future work, a more refined balance between authenticity and professional knowledge in dialogue generation needs to be achieved. We aspire this work will furnish fresh perspectives and references for the development of LLMs in the field of psychological counseling. Limitations
In this work, we proposed CPsyCoun, a reportbased multi-turn dialogue reconstruction and evaluation framework for Chinese psychological counseling. Although the experimental results demonstrate that our framework is viable, there are still some limitations need to be considered. In real-life scenarios, psychological counseling is complex. Counselors not only need to empathize with clients, but also need to use psychological counseling techniques at the appropriate time. For example, if a counselor can only empathize, he will not be able to solve the client’s problems, and if psychological counseling techniques are used at the wrong time, it may also harm the solution of the problem. Therefore, dialogue generative methods and dialogue evaluation methods need to further consider the balance between authenticity and professional knowledge