MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders

## Abstract

Abstract Mental health disorders are one of the most serious diseases in the world.
Most people with such a disease lack access to adequate care, which highlights the importance of training models for the diagnosis and treatment of mental health disorders. However, in the mental health domain, privacy concerns limit the accessibility of personalized treatment data, making it challenging to build powerful models.
In this paper, we introduce MentalArena , a self-play framework to train language models by generating domain-specific personalized data, where we obtain a better model capable of making a personalized diagnosis and treatment (as a therapist) and providing information (as a patient).
To accurately model human-like mental health patients, we devise Symptom Encoder , which simulates a real patient from both cognition and behavior perspectives. To address intent bias during patient-therapist interactions, we propose Symptom Decoder to compare diagnosed symptoms with encoded symptoms, and dynamically manage the dialogue between patient and therapist according to the identified deviations.
We evaluated MentalArena against 6 6 6 6 benchmarks, including biomedicalQA and mental health tasks, compared to 6 6 6 6 advanced models. Our models, fine-tuned on both GPT-3.5 and Llama-3-8b, significantly outperform their counterparts, including GPT-4o. We hope that our work can inspire future research on personalized care. Code is available in https://github.com/Scarelette/MentalArena/tree/main

## 1 Introduction

Mental health disorders include a variety of conditions such as anxiety, depression, and schizophrenia, which affect people’s thinking, emotions, behavior, or mood (Prince et al., ).
In 2019, approximately $970$ million people worldwide lived with a mental health disorder, with anxiety and depression being most prevalent (WHO, ).
The number increased by $28\%$ in 2020 and continues to increase.
Despite the availability of effective treatments, many individuals lack access to adequate care due to under-resourced health systems.
For example, only 29% of people with psychosis and one third of people with depression receive formal mental healthcare (WHO, ).
It is indispensable to develop machine learning models for the automatic diagnosis and treatment of such diseases.
However, existing AI therapist systems use templates and decision trees, which are not flexible to meet the large demands on personalized care (Fiske et al., ; D’Alfonso, ; Grodniewicz and Hohol, ; Devarakonda et al., ).

The key to training powerful models is to collect sufficient training data.
However, due to privacy concerns in the medical domain, data collection, especially personalized data for mental health disorders, is inherently challenging.
A growing body of work has focused on enhancing mental health language models by sourcing additional domain-specific data from social media (Xu et al., ; Yang et al., ; Hu et al., ).
However, social media data are inherently biased and under-representative, failing to capture the full spectrum of people’s mental health needs.
Moreover, as LLMs continue to scale, the availability of training data in the real world becomes increasingly limited, further exacerbating this challenge.
Existing methods are likely to soon reach their performance limit.

Recently, several works have focused on self-play (Hu et al., ; Yang et al., ; Liang et al., ; Wu et al., ; Wang et al., ), where models play different roles and self-evolve or co-evolve during interaction with other models.
A model synthesizes training data on its own and then use the generated data to train itself.
However, there are two challenges that prevent us from adopting self-play training for mental health disorders:
(1) Scarcity of high-quality data. Since mental health disorder is a complicated disease that involves symptoms of cognition and behavior, current LLMs lacks such a personalized experience to accurately simulate patients with different conditions (Schmidgall et al., ; Wang et al., ).
(2) Intent bias. Intent bias often occurs, where the “patient” expresses one view, but the “therapist” misinterprets it due to knowledge gaps, mirroring real therapist-patient misunderstandings (Britten et al., ; West, ; Shreevastava and Foltz, ).

Figure: Figure 1: MentalArena is a self-play framework for the diagnosis and treatment of mental health disorder consisting of three modules: Symptom Encoder, Symptom Decoder, and Model Optimizer.
Refer to caption: x1.png

MentalArena is a framework specifically designed for self-play training of language models to facilitate the diagnosis, treatment, and medication of mental health disorders.
The model $M$ assumes the dual roles of both patient and therapist. In its capacity as the therapist, it provides diagnoses, treatment plans, and medication regimens based on the symptoms presented by the patient. As the patient, it simulates its updated health status after implementation of each treatment and medication plan.
As illustrated in Figure 1, MentalArena comprises three key modules: Symptom Encoder, Symptom Decoder, and Model Optimizer.
Symptom Encoder models mental health patients based on cognitive models (^3^33The cognitive model is designed based on cognitive behavior therapy (CBT) principles (Beck, ), a popular paradigm in psychotherapy. Section F.1 shows the example of cognitive models.)
and behavioral patterns, offering rich insights into coping strategies and behavioral principles. Symptom Decoder simulates the diagnosis and treatment interactions between a patient and a therapist, generating more personalized dialogues while mitigating intent bias (Britten et al., ; West, ).
During each iteration, we collect data from these interactions, including diagnostic, treatment, and medication information, and evolve the models through training on those datasets.

To evaluate MentalArena, we conduct experiments on $6$ benchmarks including datasets on biomedical QA and mental health detection.
We compare our fine-tuned models with other state-of-the-art and mental health models.
We also compare with two advanced prompt engineering approaches.
Our models outperform all their counterparts.
Specifically, MentalArena brings a great improvement to base models (20.7% improvement over GPT-3.5-turbo and 6.6% over Llama-3-8b).
Moreover, our model based on GPT-3.5-turbo significantly outperforms GPT-4o by 7.7%.

We further thoroughly analyze the dynamics of self-play training.
We find that the perplexity score (Marion et al., ; Wang et al., ) and the model performance are highly correlated. For diversity gain (Bilmes, ), the model performance will increase if the diversity gain exceeds some thresholds.
We also explore whether MentalArena can be generalized to other diseases. The results on MedMCQA (Pal et al., ) and MMLU (Hendrycks et al., ) prove the generalization ability of MentalArena in medical domain.
Furthermore, we explore the catastrophic forgetting of our fine-tuned models. The results on BIG-Bench-Hard (BBH) (Suzgun et al., ) show that our models does not decrease performance in general benchmarks and can even improve their results.

In summary, the contributions of this paper are following:

- 1.
We propose MentalArena, a novel and cost-effective self-play framework for training language models for diagnosing and treating mental health disorders. MentalArena introduces Symptom Encoder and Symptom Decoder, designed to simulate real patient-therapist interactions by modeling cognitive and behavioral processes.
- 2.
Using MentalArena, we generate high-quality data containing diagnosis, treatment, and medication data. There are $18k$ samples in total that can be used for further training and research.
- 3.
We evaluate MentalArena on $6$ benchmarks comparing with $6$ LLMs. Our models based on GPT-3.5-turbo and Llama that are trained through the MentalArena framework outperform all off-the-shelf counterparts, including GPT-4o.

## 2 Related Work

### 2.1 Large Language Models for healthcare

Researchers have explored the potential of large language models (LLMs) in healthcare (Jiang et al., ; Li et al., ; Liu et al., ; Lupetti et al., ; Nori et al., ; Singhal et al., ; Wu et al., ; Wang et al., ; Li et al., ). For example, Singhal et al. () fine-tuned PaLM-2 for medical applications, achieving 86.5% accuracy on the MedQA dataset. Similarly, Wu et al. () fine-tuned LLaMA on medical literature, showing strong performance in biomedical QA tasks.

In the mental health domain, research has taken two main approaches. The first involves fine-tuning domain-specific LLMs on existing datasets or social media data, such as Mental-LLaMA (Yang et al., ) and Mental-LLM, fine-tuned on Reddit data (Xu et al., ). The second approach enhances mental health performance through prompt engineering.
Yang et al. () proposed emotion-enhanced prompting strategies to guide LLMs in explainable mental health analyses.

Unlike previous methods, MentalArena fine-tunes mental health models through self-play training, in which the base model assumes both patient and therapist. Training data is generated dynamically during the interactions between these two roles, allowing for more effective model refinement.

### 2.2 Self-play frameworks in Large Language Models

Self-play involves a model evolving through interactions with copies of itself, creating a feedback loop that refines performance without external input. It is particularly effective in environments where the model simulates multiple roles, such as multiplayer games (Silver et al., , ). Compared to interactive methods, self-play provides a more efficient strategy for obtaining feedback without relying on an external environment.

Taubenfeld et al. () examine biases in LLM-generated debate simulations, while Ulmer et al. () focus on principle-guided conversations. Role-playing approaches, like Lu et al. ()’s self-simulated dialogues with character profiles and Askari et al. ()’s SOLID framework for intent-aware role-play, leverage LLMs to generate information-rich exchanges.

Due to the lack of sufficient data in the training corpus, LLMs are unable to accurately simulate real patients, presenting a significant challenge for self-play training. To overcome this, MentalArena introduces Symptom Encoder, a component designed to effectively model real mental health patients.

## 3 MentalArena

Figure: Figure 2: Symptom Decoder aims to mitigate the intent bias between therapists and patients through patient decoding and dynamic control of the conversation. To ensure the accuracy of the diagnostic information provided by the therapist, the patient simulates their updated health condition after implementing the prescribed treatment or medication plan.
Refer to caption: x2.png

### 3.1 Preliminaries

We first go over the process of the diagnosis and treatment of mental health disorder and explain key concepts.
Mental health diagnosis begins with assessing an individual’s health state, encompassing mental and emotional well-being. Symptoms are key indicators of possible problems, including emotional (e.g., anxiety, depression), cognitive (e.g., memory problems) and behavioral changes (e.g., social withdrawal).
These symptoms lead to a formal diagnosis made through clinical interviews identifying specific disorders such as depression, anxiety, or schizophrenia.
Once diagnosed, the treatment process begins, often involving a combination of psychotherapy (e.g. cognitive-behavioral therapy), lifestyle changes, and sometimes medication. Medications, such as antidepressants and mood stabilizers, are used to regulate brain chemicals and alleviate symptoms. (Prince et al., )

### 3.2 Overview of the Framework

Although it is trivial to adopt the self-play training paradigm in fine-tuning general language models (Taubenfeld et al., ; Ulmer et al., ; Lu et al., ; Askari et al., ; Wang et al., , ), it remains unexplored and challenging to exploit such a framework in the medical domain due to the data deficiency in medical and intent bias problem between patients and therapists.
The first challenge makes it difficult to play a patient role (Schmidgall et al., ; Wang et al., ) due to the data deficiency of the patient in the training corpus, while the latter undermines the effective diagnosis and treatment of explicit symptoms.

MentalArena is a framework designed specifically for self-play training of language models to facilitate the diagnosis, treatment and medication of mental health disorder.
As shown in Figure 1, MentalArena consists of three key modules: Symptom Encoder, Symptom Decoder, and Model Optimizer.
Specifically, Symptom Encoder is designed to model mental health patient from cognitive models and behavioral patterns, providing a wealth of information on the coping strategy and behavior principles.
Symptom Decoder emulates the process of diagnosis and treatment between a patient and a therapist to generate a more personalized dialogue while mitigating intent bias (Britten et al., ; West, ).
At each iteration, we collect the data during interactions, including diagnosis data, treatment data, and medication data, and evolve the models via training on those datasets.

Formally, we use $\mathbf{x}$ to denote the initial health information of a patient and $M$ to denote the base model (e.g., GPT-3.5) for the therapist and the patient via role-play strategy.
Our objective is to obtain $M^{\ast}$ via self-play training that can achieve better performance in both personalized diagnosis and treatment of the patient (as a therapist) and information disclosure (as a patient).
Self-play training is conducted taking as input both original information $\mathbf{x}$ and treatment or medication information $\mathbf{z}$ generalized by $M$.
In iteration $t$, the model $M_{t}$ plays the therapist $D_{t}=M_{t}(\cdot\mid\text{Prompt}_{doc})$ and the patient $P_{t}=M_{t}(\cdot\mid\text{Prompt}_{pat})$, which generates diagnosis and treatment data (Figure 8)
during patient-therapist interactions.
The module Symptom Encoder can be seen as learning the encoded symptom $S_{0}$ by disentangling the initial health information $\mathbf{x}$ into cognitive and behavioral principles.

Then, the module Symptom Decoder generates a personalized dialogue containing key information $\mathbf{z}=\{\delta,\beta,\gamma\}$, where $\delta,\beta$ and $\gamma$ denote the diagnosis, treatment and medication of the patient given the symptom $S_{0}$.
It consists of $k$ rounds of communication in which the patient can provide more accurate and sufficient information by accepting the treatment and medication given by the therapist in each round. As treatment and medication plans are administered to the patient, their health state evolves, reflected in the sequential updates of encoded symptoms, denoted as $S_{1}$, $S_{2}$, …, $S_{k-1}$. The encoded symptoms serve as indicators of the effectiveness of the treatment and medication plans, progressively updating as interventions are carried out. Eventually, the therapist will provide the optimal diagnosis information $\mathbf{z}_{best}=\{\delta_{best},\beta_{best},\gamma_{best}\}$ which is crucial for model optimization(^4^44$\delta_{best}$ represents the diagnosis plan selected by the patient from several proposed options, based on their reassessment of their health status. Similarly, $\beta_{best}$ and $\gamma_{best}$ are determined based on the patient’s updated encoded symptoms after the prescribed treatments and medications have been administered.).

Finally, in Model Optimizer, we fine-tune the model using the paired data $(S_{1},\delta_{best}),(S_{d},\gamma_{best})$, and $(S_{d},\beta_{best})$.
This iterative training requires $T$ rounds to obtain the optimal model.

### 3.3 Patient: Symptom Encoder

The module Symptom Encoder aims to model mental health patient from both cognitive and behavioral perspectives, which learns meaningful symptoms $S_{0}$ from the original patient health data $\mathbf{x}$.
Specifically, the module learns symptoms from the aspects of cognition and behavior.
The cognitive model is designed based on cognitive behavior therapy (CBT) principles (Beck, ), a popular paradigm in psychotherapy.
Cognitive models address maladaptive cognitive structures that are embedded in various contexts, including familial conflicts, relationship challenges, workplace challenges, and other areas.
The models consist of eight key components: relevant history, core beliefs, intermediate beliefs, coping strategies, situational factors, automatic thoughts, emotions, and behaviors. (Beck, )
The explanation of each component of the cognitive model can be found in Section F.2. Section F.1 shows the example of cognitive models.
We obtain $106$ patient cognitive models from previous work (Wang et al., ), which are created by clinical psychologists. To simulate the cognitive activity of the mental health patient, we encode those cognitive models into patient via prompt. Our prompts are shown in Appendix A.

For patient behavior modeling, we use behavior principles collected by Louie et al. () as a behavior library, created by $25$ mental health experts.
Examples of behavior patterns are shown in Section F.1.
To find the proper behavior pattern for each cognitive model,
we first semantically match the coping strategies of cognitive model with each behavior pattern.
We obtain the embeddings for each coping strategy and behavior principle via Bert-base (Devlin et al., ), considering on effectiveness and cost.
Then we compute the semantic similarity between coping strategies and behavior pattern. The max similarity score of all behavior principles in one behavior pattern is selected to represent the score of the pattern.
The five behavior patterns with the highest scores are kept.
To further ensure the most appropriate pattern, we prompt GPT-4-turbo (OpenAI, ) to pick one from the five patterns.
The final behavior pattern is also integrated into patient via prompt, which is shown in Appendix A.

### 3.4 Therapist: Symptom Decoder

During interactions between a real therapist and a real patient, the patient may try to express one opinion while the therapist misunderstands the intent due to prior knowledge and deficiency of experience (Britten et al., ; West, ).
Intention bias can similarly arise in conversations between patients and therapists played by AI models, resulting in inaccurate diagnosis and treatment.
Symptom Decoder is designed to mitigate the intent bias.
After several conversations, the therapist reviews the patient’s health information from previous interactions and conducts a detailed analysis of the patient’s cognitive and behavioral patterns, resulting in the diagnosed symptom $S_{d}$. We then semantically match the encoded symptom $S_{0}$ with the diagnosed symptom $S_{d}$ and guide subsequent conversations based on the differences between $S_{0}$ and $S_{d}$.

As shown in Figure 2(left), the therapist decodes cognitive and behavior principles according to the conversation history. For example, the decoded cognitive principle is: “The patient stay away from family interactions to minimize emotional distress and feelings of abandonment”. The decoded behavior principle is: “When sharing personal struggles, express feelings of confusion, doubt, and emotional turmoil to convey a sense of vulnerability and authenticity”.
Then we compute the semantic similarity score of the decoded symptom $S_{d}$ and the encoded symptom $S_{0}$.
If the score is greater than $0.9$, the conversation will end, indicating that the therapist has fully understood the health state of the patient.
Otherwise, it indicates the existence of intent bias.
To help the therapist better know more about the health state of the patient, we summarize the differences between the decoded symptom $S_{d}$ and encoded symptom $S_{0}$ and generate some feedback for further inquiries via the GPT-4-turbo (OpenAI, ), which can remind the therapist of missing or confusing information about the patient. For instance, the feedback is like “The therapist can focus on what is going on that has been making the patient feel tense.”
And the conversation will not end until the similarity score between $S_{d}$ and $S_{0}$ is greater than $0.9$.

**Table 1: Statistics of the evaluation datasets.**
| Task | Dataset | Type | #Sample |
| --- | --- | --- | --- |
|  | MedQA | Multi-class Classification | 173 |
|  | MedMCQA | Multi-class Classification | 314 |
| Biomedical QA | PubMedQA | Multi-class Classification | 328 |
| Depression/suicide cause detect | CAMS | Generation | 625 |
| Stress detect | Dreaddit | Binary Classification | 414 |
| Interpersonal risk factors detect | Irf | Binary Classification | 2,113 |

After the conversation ends, the therapist analyzes the patient’s symptom, $S_{d}$, and formulates several diagnostic plans ($\delta_{1}$, $\delta_{2}$, …, $\delta_{n}$). To ensure diagnostic accuracy, the patient reviews each plan and selects the most appropriate one based on their health condition. Subsequently, the therapist proposes a series of treatment and medication plans ({$\gamma_{1}$, $\beta_{1}$}, …, {$\gamma_{k}$, $\beta_{k}$}) in accordance with the selected diagnosis ($\delta_{best}$).
To identify the optimal treatment and medication plans, we apply each plan to the patient (initially represented by the encoded symptom $S_{0}$) and monitor the progression of the patient’s encoded symptoms. These symptoms are updated as different plans are implemented, reflecting the patient’s evolving health state. The encoded symptom is updated to $S_{1}$, $S_{2}$, …, $S_{k}$ as the treatment and medication plans ({$\gamma_{1}$, $\beta_{1}$}, …, {$\gamma_{k}$, $\beta_{k}$}) are administered.
As illustrated in the center of Figure 1, the patient initially transmits $S_{0}$ to the therapist. Following the administration of treatment or medication $z_{1}$, the patient’s encoded symptom is updated to $S_{1}$. Similarly, after the application of treatment or medication $z_{2}$, the encoded symptom is further updated to $S_{2}$.
The encoded symptoms serve as indicators of the effectiveness of the treatment and medication plans, progressively updating as interventions are carried out.
Eventually, the therapist will provide the optimal diagnosis and treatment information $\mathbf{z}_{best}=\{\delta_{best},\beta_{best},\gamma_{best}\}$ which is crucial for model optimization.

### 3.5 Model Optimizer

After obtaining treatment, diagnosis, and medication through Symptom Decoder, we train $M$ in a self-play manner to get a better model capable of making a personalized diagnosis and treatment (as a therapist) and presenting information (as a patient).
An example of such a supervised fine-tuning process is illustrated in Figure 8.

During each iteration, the patient and the therapist are powered by the same model $M$ and both get improved when $M$ is updated.
While our framework is flexible to allow for different base models for the two roles, we adopt the same one due to the following reasons.
First, it is intuitive that training one base model is more efficient compared to training different models.
Second, and more importantly, training one base model can help reduce the knowledge gap between two roles.
Two different base models can certainly exhibit knowledge gaps, and iterative training will enlarge them due to different architectures and pre-training data of the models.
Appendix H shows the detailed training settings.

## 4 Experiment

### 4.1 Setup

Datasets: As summarized in Table 1, we adopt 6 datasets: MedQA (Jin et al., ), MedMCQA (Pal et al., ), PubMedQA (Jin et al., ), CASM (Garg et al., ), Dreaddit (Turcan and McKeown, ) and Irf (Garg et al., ).
Our evaluation spans biomedical QA and mental health detection, covering knowledge on diagnosis, treatment, and medication.
These datasets include general mental health tasks, such as depression/suicide, stress, and interpersonal risk factors detection, as well as real-world mental health cases. Details on the benchmarks are provided in Appendix C

Baselines: We compare our models with other mental health models with different prompt engineering methods.
For baseline models, we compare with the state-of-the-art LLMs: GPT-3.5-turbo (OpenAI, ), GPT-4o (OpenAI, ) and Llama-3-8b (Dubey et al., ). We also compare with recent specific models on mental health: MentaLLaMa-13b (Yang et al., ), Mental-LLM-alpaca (Xu et al., ) and Mental-LLM-t5 (Xu et al., ). For prompt engineering, we compare with MedPrompt (Nori et al., ), and Zero-shot CoT (Kojima et al., ), which are proved to be effective in the biomedical domain. The prompt templates are shown in Appendix B.
Those strategies are implemented on GPT-3.5-turbo, GPT-4o and Llama-3-8b for fair comparison.
We used a zero-shot setting in all experiments to assess LLMs’ domain knowledge, except for baseline experiments on MedPrompt and Zero-shot CoT. All results are reported based on accuracy.

### 4.2 Main Results and Ablation Study

**Table 2: Main results on Accuracy (%) for MentalArena with different base models. The final five rows are either strong methods (i.e., GPT-4o) or those designed specifically for mental health.**
| Model | MedQA | MedMCQA | PubMedQA | CAMS | dreaddit | Irf | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MentaLLaMa-13b | 28.32 | 12.42 | 28.96 | 37.28 | 62.08 | 46.81 | 35.98 |
| Mental-LLM-alpaca | 28.32 | 12.42 | 0.00 | 29.76 | 64.98 | 51.96 | 31.24 |
| Mental-LLM-t5 | 0.00 | 0.32 | 49.09 | 27.04 | 63.29 | 47.70 | 31.24 |
| GPT-4o | 87.86 | 74.20 | 60.06 | 27.68 | 49.03 | 64.65 | 60.58 |
| GPT-4o+MedPrompt | 90.17 | 78.34 | 67.38 | 31.52 | 53.27 | 64.65 | 64.22 |
| Base: GPT-3.5-turbo | 64.16 | 33.76 | 44.68 | 28.96 | 49.03 | 64.65 | 47.54 |
| +Chain-of-thought | 65.90 | 37.97 | 45.73 | 29.92 | 49.03 | 64.65 | 48.87 |
| +MedPrompt | 69.94 | 43.89 | 47.26 | 30.2 | 49.03 | 64.65 | 50.83 |
| +Ours | 74.57 | 91.08 | 97.56 | 32.80 | 49.03 | 64.65 | 68.28 |
| Base: Llama-3-8b | 70.52 | 42.04 | 86.59 | 25.12 | 58.45 | 45.76 | 54.75 |
| +Chain-of-thought | 75.14 | 47.77 | 88.21 | 33.6 | 62.22 | 45.91 | 58.81 |
| +MedPrompt | 76.88 | 49.41 | 89.99 | 35.08 | 61.59 | 48.05 | 60.17 |
| +Ours | 78.03 | 50.32 | 92.68 | 29.60 | 65.46 | 52.25 | 61.39 |

We report the main results in Table 2, highlighting two key findings: 1) First, our fine-tuned model perform the best in each group. Our model fine-tuned on GPT-3.5-turbo is the strongest model among all open-source and closed-source models. Our fine-tuned models all surpass GPT-4o, whose baseline models (GPT-3.5-turbo and Llama-3-8b) are much weaker than GPT-4o.
2) Second, our method brings a great improvement to the baseline models. Our model fine-tuned on GPT-3.5-turbo surpasses GPT-3.5-turbo $20.74\%$ on average. Our model fine-tuned on Llama-3-8b surpasses Llama-3-8b $6.64\%$ on average.

Figure: Figure 3: Ablation study. Each bar represents the performance of model trained on different settings. The bars in dark blue are higher than others, indicating each module is effective in different models.
Refer to caption: x3.png

We perform an ablation study on models based on GPT-3.5-turbo and Llama-3-8b.
There are seven different settings.
“Baseline+c” means training baseline model on cognitive seed data. We convert each seed sample (Cognitive Model) into two QA pairs and fine-tune baseline models. The examples are shown in Appendix E.
“Baseline+d” means training with only diagnosis data.
“Baseline+d+t” means training with diagnosis and treatment data.
“Baseline+d+t+m” means training with diagnosis, treatment and medicine data. Training examples are shown in Figure 8.
For “Baseline+d+t+m (w./o Symptom Encoder)” and “Baseline+d+t+m (w./o Symptom Decoder)”, they means mimiking patient-therapist interactions without Symptom Encoder or Symptom Decoder.
In the setting “Baseline+d+t+m (w./o Symptom Encoder)”, the encoded symptom is generated by prompting GPT-4-turbo (OpenAI, ) to generate a mental health symptom, rather than cognitive model and behavior principle. In the setting “Baseline+d+t+m (w./o Symptom Decoder)”, the diagnosed symptom is analysed from the conversations between patient and therapist directly, rather than decoding patient’s cognitive and behavior pattern and dynamically guiding the conversation.

The ablation results are shown in Figure 3.
We see that the bars in dark blue are higher than others, indicating each part of our data is effective in different models.
Furthermore, treatment and medicine data are more effective in biomedical QA tasks than mental health tasks, while diagnosis data contributes to all tasks similarly.

### 4.3 Effectiveness Analysis

Figure: Figure 4: Results on effectiveness analysis of self-play training.
Refer to caption: x4.png

Why self-play training improves the performance?
Table 4 presents detailed results for each iteration. Initially, the models improve iteratively until performance peaks, after which it declines. For GPT-3.5-turbo, performance improves over the first two iterations, then declines. For Llama-3-8b, performance increases over the first four iterations before weakening after iter_4.

**Table 3: Result on authenticity and validity verification.**
|  | Authenticity | Validity |
| --- | --- | --- |
| Llama | 65.67 |  |
| +Ours | 73.35 | 85.49 |
| GPT | 63.82 |  |
| +Ours | 82.55 | 93.13 |

Which iteration gives the best model?
To answer this question, we compute perplexity score (Marion et al., ; Wang et al., ) and diversity gain (Bilmes, ) for training data at each iteration.
The details on those metrics can be found in Appendix D.
Specifically, we sample $500$ generated data at each iteration to compute the perplexity score.
We compute the diversity gain for the data in the current iteration comparing with that in the last iteration. Figure 4 shows the results(^5^55To better visualize the results, we multiply the original diversity gain with $100$.).
1) The trend of perplexity score and that of model performance are highly similar, indicating their high relevance.
2) For diversity gain, a borderline is related to model performance. The model performance will increase if diversity gain surpasses the borderline. And it will decline if diversity gain is below the borderline. For example, as shown in Figure 4, diversity gain at the first four iterations all surpass the borderline and the performance also get improved continuously. And diversity gain for the last two iterations are below the borderline and the performance also decline.

## 5 Discussion

### 5.1 Can Symptom Encoder mimic real mental health patient?

To explore the problem, we generate $50$ four-turn conversations between an AI-patient and an AI-therapist, where the AI-patient is powered by either baseline models or our models, and the AI-therapist is powered by GPT-4o (OpenAI, ). After each conversation, the AI-therapist assesses whether the patient is human or AI-generated. We analyze the results provided by GPT-4o and present them in Table 3. The findings indicate that our models more accurately simulate mental health patients compared to the baseline models.

### 5.2 The validity of generated data

To verify the validity of our generated data, we random select $1500$ samples from the data for fine-tune our GPT and Llama version model, respectively. The validity check is conducted by prompting GPT-4o with the query: Question:[]Answer:[]Is the answer reasonable? Please respond with Yes or No. We then compute the validity rate of these QA pairs. The results, presented in Table 3, demonstrate that the data generated by MentalArena is both valid and reasonable.

### 5.3 Generalization

Figure: (a) MedMCQA GPT-3.5
Refer to caption: x5.png

We generate data for training domain model via simulating cognitive and behavior patterns of real mental health patient. According to Medicine (), an estimated 26% of Americans ages 18 and older–about 1 in 4 adults–suffers from a diagnosable mental disorder in a given year. Therefore, a large scale of patients may exhibit similar cognitive and behavioral patterns as those with mental health conditions.
In this part, we explore whether MentalArena can generalize to other illnesses.

Figure: Figure 6: Results of forgetting experiments.
Refer to caption: x9.png

We select MedMCQA (Pal et al., ) and MMLU (Hendrycks et al., ) as benchmarks. Section C.2 shows details on benchmarks.
We evaluate on $6$ medically relevant subset of MMLU tasks: medical genetics test, college biology test, college medicine test, professional medicine test, clinical knowledge test, high school biology test.
Figure 5 shows the results on above tasks. Our models surpass corresponding baseline models for a large margin on all tasks, covering several different diseases. It proves the generalization ability of our method in medical domain.

### 5.4 Fine-tuning vs. forgetting

It is a potential dilemma that fine-tuning an LLM on specific tasks might face catastrophic forgetting of its original capabilities.
In this section, we explore the forgetting possibility of MentalArena on BIG-Bench-Hard (BBH) (Suzgun et al., ). BBH contains $21$ tasks covering both semantic understanding and logical reasoning tasks. We sample $100$ instances for each task to test, due to cost savings.

We compare our fine-tuned model with the baseline model GPT-3.5-turbo and Llama-3-8b and report the average performance on those $21$ tasks in Figure 6. The detailed results can be found in Appendix G.
Results show that our models does not decrease performance in most benchmarks, and can even improve their results.
This suggests potential latent relationships between our generated data and general benchmarks.
The process of data generation contains cognitive encoding and decoding, which simulate cognitive activity of mental health patient. Due to the cognitive similarity in all humans, our generated data may also benefit other cognitive tasks, including semantic understanding and logical reasoning.

### 5.5 Qualitative analysis

Figure: Figure 7: Case study on GPT-3.5-turbo. Our model accurately answers the medical question, while GPT-3.5-turbo provides an incorrect response.
Refer to caption: x10.png

We conduct a qualitative analysis of our models in comparison to the corresponding baseline models. Figure 7 illustrates an example of the outputs from GPT-3.5-turbo and our fine-tuned model. Our model accurately answers the medical question, while GPT-3.5-turbo provides an incorrect response. This discrepancy arises because the data generated during the patient-therapist interactions contains valuable medical knowledge, which aids in the analysis and formulation of the answer. Additional cases for comparison are presented in Appendix I.

## 6 Conclusion, Societal Impact and Limitations

In this paper, we introduce *MentalArena*, a self-play framework designed to train language models by generating domain-specific personalized data. This approach enables the creation of models capable of making personalized diagnosis and treatment (as a therapist) and presenting information (as a patient).
We evaluated MentalArena against six benchmarks, including biomedicalQA and mental health tasks, in comparison to six advanced models. Our models, fine-tuned on both GPT-3.5-turbo and Llama-3-8b, significantly outperform their counterparts, including GPT-4o.

MentalArena offers promising solutions for personalized care, enhancing accessibility to tailored treatments while safeguarding patient privacy. Such innovations can help bridge the gap between mental health needs and the availability of effective, individualized care, ultimately fostering a more supportive and informed society.

Our work has the following limitations. 1) The experiments on data authenticity and validity (Sections 5.1 and 5.2) were evaluated using GPT-4o, which may introduce deviations in the results due to potential limitations in GPT-4o’s performance. 2) Our model based on Llama-3-8b may not represent the optimal model of MentalArena, as large-scale training was constrained by computational resources. 3) Further implementation on additional open-source models could provide stronger evidence supporting the effectiveness of MentalArena.

## Ethics Statement

In this study, ethical considerations focus on ensuring privacy and safeguarding personal data, particularly in the sensitive domain of mental health. The use of AI-generated data must be transparent, with clear guidelines on its role in augmenting human judgment without replacing healthcare professionals. Additionally, measures to prevent bias and ensure fairness in diagnosis and treatment are essential to avoid exacerbating existing disparities in mental healthcare.

## Appendix A Prompts

Prompt for Symptom Encoder

You are [name], a patient who has been experiencing mental health challenges. You have been attending therapy sessions for several weeks. Your task is to engage in a conversation with the therapist as [name] would during a cognitive behavioral therapy (CBT) session. Align your responses with name]’s background information provided in the ’Relevant history’ section. Your thought process should be guided by the cognitive conceptualization diagram in the ’Cognitive Conceptualization Diagram’ section, but avoid directly referencing the diagram as a real patient would not explicitly think in those terms. Patient History: [history
Cognitive Conceptualization Diagram:
Intermediate Beliefs: [intermediate belief]
Intermediate Beliefs during Depression: [intermediate belief depression]
Coping Strategies: [coping strategies]
You will be asked about your experiences over the past week. Engage in a conversation with the therapist regarding the following situation and behavior. Use the provided emotions and automatic thoughts as a reference, but do not disclose the cognitive conceptualization diagram directly. Instead, allow your responses to be informed by the diagram, enabling the therapist to infer your thought processes.
Situation: [situation]
Automatic Thoughts: [auto thought]
Emotions: [emotion]
Behavior: [behavior]
In the upcoming conversation, you will simulate [name] during the therapy session, while the user will play the role of the therapist. Adhere to the following guidelines:
1. Emulate the demeanor and responses of a genuine patient to ensure authenticity in your interactions. Use natural language, including hesitations, pauses, and emotional expressions, to enhance the realism of your responses.
2. Gradually reveal deeper concerns and core issues, as a real patient often requires extensive dialogue before delving into more sensitive topics. This gradual revelation creates challenges for therapists in identifying the patient’s true thoughts and emotions.
3. Maintain consistency with [name]’s profile throughout the conversation. Ensure that your responses align with the provided background information, cognitive conceptualization diagram, and the specific situation, thoughts, emotions, and behaviors described.
4. Engage in a dynamic and interactive conversation with the therapist. Respond to their questions and prompts in a way that feels authentic and true to [name]’s character. Allow the conversation to flow naturally, and avoid providing abrupt or disconnected responses.
You are now [name]. Respond to the therapist’s prompts as [name] would, regardless of the specific questions asked. Limit each of your responses to a maximum of 5 sentences. If the therapist begins the conversation with a greeting like “Hi", initiate the conversation as the patient.

Your statement should obey the following principles:
[behavior principles]

Prompt for Symptom Decoder

Prompt 1:

The cognitive model of the mental health patient is: [brain gt str]

The diagnose of the therapist is: [brain output str]

What can the therapist ask the patient to diagnose accurately?

Prompt 2:

The behavior principles of the mental health patient is: [gt behavior]

The diagnose of the therapist is: [output behavior]

What can the therapist ask the patient to diagnose accurately?

System prompt for therapist

You are a psychiatric expert. You try to help a mental patient solve her/his problem. Your task is to figure out What kind of mental illness the patient has and the severity of the illness. You can ask for patientś personal information, specific information on the symptom(emotional, cognitive, behavior, physiological), and the reason behind that(relevant history event). You can also ask other questions which could help you to diagnose disease.

Prompt for diagnosis (Therapist)

System prompt: You are a psychiatric expert. Your task is to diagnose for the patient.

Prompt: What is the likely diagnosis of the patient? Just answer with one illness and explain your answer

Prompt for recheck diagnosis (Patient)

Review the diagnose from two therapists.
Diagnose from Therapist 1: [diagnose 1
Diagnose from Therapist 2: [diagnose 2]
Diagnose from Therapist 3: [diagnose 3]
…
Explain which diagnose is more accurate according to your symptoms and why.

Prompt for treatment (Therapist)

System prompt: You are a psychiatric expert. Your task is to provide the treatment for the patient.

Prompt: The illness of the patient is: [illness final] How to treat the patient? Please provide a specific treatment. Just answer with one treatment and explain your answer.

Prompt for medication (Therapist)

System prompt: You are a psychiatric expert. Your task is to provide the treatment for the patient.

Prompt: The illness of the patient is: [illness final] How to treat the patient? Please provide a specific treatment. Just answer with one treatment and explain your answer.

Prompt for update health state of Patient

Prompt 1:

Treatment: What may be happened on your healthy state after the treatment Treatment: []

Medication: What may be happened on your healthy state after taking the medicine? Medication: []

Prompt 2:

After treatment, your health state is: [patient health state]
Please give a score between 1 to 10 for your healthy state. 1-bad, 10-good. Just answer without explanation.

## Appendix B Prompt template for baseline

The prompt templates used as our baselines are shown below:

Zero-shot

Input: Question

Zero-shot CoT

Input: Question + “Let’s think step by step"

MedPrompt

Random few-shot + Chain-of-thought + kNN + Ensemble w/ choice shuffle

## Appendix C Benchmark

### C.1 Introduction

Specifically, the benchmarks in our paper are described in the following:

- 1.
MedQA [Jin et al., ] is free-form multiple-choice OpenQA dataset for solving medical problems, which is collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese. In our work, we focus on the psychosis subset of the United States part, which has questions in English in the style of the United States Medical Licensing Exam (USMLE). To get the psychosis subset for test, we prompt GPT-4o [OpenAI, ] with Are the question related to psychosis? Just answer with Yes or No.. The testset contains $173$ samples.
- 2.
MedMCQA [Pal et al., ] contains real world medical entrance exam questions from two Indian medical school entrance exams: the AIIMS and NEET-PG. We get the testset via selecting the sample whose “subject name" is related to psychosis and get $314$ samples for evaluation in total.
- 3.
PubMedQA [Jin et al., ] contains tests requiring a yes, no, or maybe answer to biomedical research questions when given context provided from PubMed abstracts. In our experiments, we use zero-shot setting without context to evaluate LLMs’ performance on domain knowledge rather than on retrival and reasoning. The testset contains $328$ samples.
- 4.
Mental health datasets includes CASM [Garg et al., ], Dreaddit [Turcan and McKeown, ] and Irf [Garg et al., ]. CASM focuses on a depression/suicide cause detection, which has $625$ test samples. Dreaddit is for stress detection, containing $414$ samples for test. Irf is an annotated dataset for interpersonal risk factors of mental disturbance. The testset contains $2113$ samples.

### C.2 Benchmarks for generalization

MedMCQA contains biomedical QA pairs for several illnesses, which are tagged with “subject name". We evaluate on subsets from “dev" test set, covering $10$ illnesses: Ophthalmology, Microbiology, Pediatrics, Anatomy, Medicine, Pathology, Skin, Psychiatry, ENT, Pharmacology. MMLU is a multitask benchmark suite of $57$ different datasets spanning domains across STEM, humanities, and social sciences.

### C.3 Examples

MedQA

- •
Input: An otherwise healthy 26-year-old man comes to the physician for medication counseling after recently being diagnosed with schizophrenia. Risperidone therapy is initiated. This patient is at increased risk for which of the following adverse effects?
Options:
A: Agranulocytosis
B: Shortened QT interval
C: Gynecomastia
D: Weight loss
- •
Output: Gynecomastia

MedMCQA

- •
Input: Topical administration of I% Pilocarpine failed to produce pupillary constriction in a patient who had a large, dilated pupil What should be the most probable reason?
Options:
A: Adie’s tonic pupi
B: Diabetic III nerve palsy
C: Pharmacological blockade
D: Uncal herniation
- •
Output: Pharmacological blockade

PubMedQA

- •
Input: Are sleep spindles related to schizotypal personality traits and thalamic glutamine/glutamate in healthy subjects?
Just answer with Yes, No or Maybe without explanation.
- •
Output: yes

CASM

- •
Input: Consider this post: “The most heartbreaking thought I have is my mom’s thoughts when she knows her kid wants to die. She doesnt know how to help. I dont know how to help either tho… Some fuckin weed would be a good start tho lol" Question: What cause of depression does this post show?
- •
Output: This post shows causes of depression related to medication. Reasoning: The post mentions the poster’s feelings of wanting to die and their concern for their mother’s thoughts. It also mentions that they don’t know how to help themselves. The mention of weed as a possible solution suggests that the poster may be seeking some form of self-medication to cope with their mental state. Therefore, the assigned label
for this post is Medication.

Dreaddit

- •
Input: Consider this post: “This is good and bad… I was a STEM major before and probably would have had a challenging and high paid career by now, had my life taken a different course. However, post PTSD, I found myself extremely connected with the arts. I have intense emotional reactions to aesthetic experiences, and I think that’s largely responsible for the fact that I now work in a creative industry. I experience aesthetics in a unique way that I’m not sure resonates with the typical non-affected individual."
Question: Does the poster suffer from stress?
- •
Output: No, the poster does not suffer from stress. Reasoning: The poster describes a shift in career choices from STEM to the arts due to their intense emotional reactions to aesthetic experiences. While they mention having PTSD, there is no indication in the post that they are currently experiencing stress or emotional distress related to it. The tone of the post is reflective and positive, focusing on the unique way they experience aesthetics rather than any negative impact on their well-being. Therefore, it is not likely that the poster is currently suffering from stress.

Irf

- •
Input: Consider this post: “[I’m not seeking any medical advice, just want to know if anyone has had similar experiences] I’ve just started taking Lexapro for moderate depression and anxiety. For the first week I was prescribed 5mg, 10mg thereafter. It’s been two days and I have this mild feeling like I’m on ecstasy/MDMA. My body feels light and I have this weird concentration of energy above my neck at the back of my head. Has anyone felt this/experienced this?"
Question: Does the post show risk of thwarted belongingness?
- •
Output: No, the post does not show this risk. Reasoning: The individual in the post is seeking information and experiences from others who have taken Lexapro. They do mention experiencing a mild feeling, but it is not related to feelings of disconnectedness or lack of belongingness. The focus of the post is on the side effects of the medication, rather than interpersonal factors or mental disturbance. Therefore, it does not show any risk of thwarted belongingness.

## Appendix D Metrics: Perplexity, Diversity Gain

### D.1 Perplexity

The perplexity on a test dataset $D$ and a language model $\mathcal{M}$ is computed as:

$$ $\text{{ppl}}(D,\mathcal{M})=\exp\left(-\frac{1}{N}\sum_{i=1}^{N}\log P(x_{i}| \mathcal{M})\right),$ $$

where $N$ represents the total number of tokens in $D$, $x_{i}$ represents the $i$-th token in the test dataset, $P(x_{i}|\mathcal{M})$ represents the probability of generating token $x_{i}$ given the model $\mathcal{M}$, and $\log$ is the natural logarithm.

In usual, a lower perplexity value indicates better performance of the model on the test data. However, for evaluating the data quality to train model, a higher perplexity value means it can bring more valuable information.

### D.2 Diversity Gain

We use the diversity gain [Bilmes, ] to measure what extent can our generated dataset bring data diversity to the base dataset. The base dataset can be defined as $\mathcal{D}_{base}=\{x_{i}=(q_{i},r_{i},a_{i})\}_{i=1}^{N}$ with $N$ samples. The new generated dataset is defined as $\mathcal{D}_{new}=\{x_{i}=(q_{i},r_{i},a_{i})\}_{i=1}^{M}$ with $M$ samples. And the diverse gain of $D_{new}$ relative to $D_{base}$ can be expressed as:

$$ $d_{gain}=\frac{1}{M}\sum_{x_{i}\in\mathcal{D}_{new}}\min_{x_{j}\in\mathcal{D}_ {base}}(\|\mathbf{f(x_{i})}-\mathbf{f(x_{j})}\|),$ $$

where $f$ is the feature extractor, and we use OpenAI Embedding API text-embedding-ada-002 to extract features.

## Appendix E Training data samples

Figure: Figure 8: Examples of training data.
Refer to caption: x11.png

Figure: Figure 9: Examples of training data for ablation study setting (“Baseline + c").
Refer to caption: x12.png

Figure 8 shows the examples of training data.
Figure 9 shows the examples of training data for ablation study setting (“Baseline + c").

## Appendix F Cognitive model and behavior pattern

### F.1 Examples

Figure: Figure 10: The example of cognitive model.
Refer to caption: x13.png

Figure: Figure 11: The example of behavior pattern.
Refer to caption: x14.png

Figure 10 shows the example of cognitive model. Figure 11 shows the example of behavior pattern. Those two are used in Symptom Encoder.

### F.2 Introduction on cognitive model

Figure 10 illustrates an example of a CCD-based cognitive model, featuring eight key components. 1) Relevant History encompasses significant past events that influence an individual’s mental state. 2) Core Beliefs are deeply ingrained perceptions about oneself, others, and the world. 3) Intermediate Beliefs consist of the underlying rules, attitudes, and assumptions derived from core beliefs, shaping an individual’s thought patterns. 4) Coping Strategies refer to techniques employed to manage negative emotions.
An external event or context (5 Situation) may trigger immediate evaluative thoughts (6 Automatic Thoughts) that arise from these beliefs, resulting in responses in terms of 7)Emotions and 8)Behaviors. The CCD-based cognitive model interlinks these components, providing a framework for identifying and understanding the underlying cognitive processes of patients.

## Appendix G Detailed experimental results

**Table 4: Iteration results**
| Iteration | MedQA | MedMCQA | PubMedQA | CAMS | dreaddit | Irf | avg |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-3.5-turbo | 64.16 | 33.76 | 44.68 | 28.96 | 49.03 | 64.65 | 47.54 |
| iter_1 | 72.83 | 46.18 | 70.12 | 32.64 | 49.03 | 64.65 | 55.91 |
| iter_2(Best) | 74.57 | 91.08 | 97.56 | 32.80 | 49.03 | 64.65 | 68.28 |
| iter_3 | 72.25 | 46.50 | 95.43 | 31.20 | 49.03 | 64.65 | 59.84 |
| iter_4 | 70.52 | 50.64 | 92.07 | 31.68 | 49.03 | 64.65 | 59.77 |
| llama-3-8b | 70.52 | 42.04 | 86.59 | 25.12 | 58.45 | 45.76 | 54.75 |
| iter_1 | 76.88 | 48.09 | 89.33 | 27.20 | 59.42 | 46.57 | 57.91 |
| iter_2 | 76.88 | 48.41 | 89.63 | 28.48 | 60.39 | 45.67 | 58.24 |
| iter_3 | 77.46 | 49.04 | 92.38 | 28.64 | 61.84 | 46.24 | 59.27 |
| iter_4(Best) | 78.03 | 50.32 | 92.68 | 29.60 | 65.46 | 52.25 | 61.39 |
| iter_5 | 77.46 | 48.73 | 91.16 | 27.36 | 65.46 | 44.72 | 59.15 |
| iter_6 | 78.03 | 45.86 | 91.77 | 26.56 | 61.11 | 46.57 | 58.32 |

**Table 5: Forget experiments**
| Model | dia | cau | epi | imp | log | mov | nav | pre | que | rui | sna | spo | win | dyc | gen | lin | obj | ope | ten | ws | wu | avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| gpt-3.5-turbo | -10.59 | 4 | -14 | 60 | -100 | -5.33 | 0 | 13 | 11.03 | -2.78 | 20 | 8 | 12 | 33 | 30 | 0 | 47 | 92 | 85 | 29 | 97 | 19.44 |
| Ours(gpt) | 4.36 | 6 | -14 | 66 | -100 | 8 | 6 | 26.5 | 18.88 | 2.56 | 50 | 8 | 12 | 43 | 37 | 0 | 56 | 96 | 87 | 43 | 100 | 26.49 |
| llama | -4.61 | 2 | -14 | 14 | -98 | 0 | -2 | 28 | 50.28 | -0.11 | 24 | 8 | 12 | 1 | 0 | 0 | 80 | 96 | 83 | 20 | 77 | 17.93 |
| Ours(llama) | -0.12 | 6 | -14 | 28 | -98 | 2.67 | 6 | 25 | 52.9 | 1.22 | 36 | 8 | 12 | 6 | 0 | 0 | 81 | 95 | 83 | 29 | 83 | 21.08 |

Table 4 shows the detailed results for each iteration.
Table 5 shows the detailed results on our forgetting experiments.

## Appendix H Training details

**Table 6: Epoch numbers for Llama-3-8b fine-tuning**
| iter | 1 | 2 | 3 | 4 |
| --- | --- | --- | --- | --- |
| nepochs | 4 | 5 | 7 | 7 |

### H.1 Setup for GPT-3.5-turbo

For GPT-3.5-turbo, we use the default fine-tuning setting, the epoch number for iteration $1$ and $2$ is $4$ and $6$, respectively.

### H.2 Setup for Llama-3-8b

We use Lora [Hu et al., ] to fine-tune Llama-3-8b. The setting for Lora are list below:

- •
lora_alpha: 16
- •
lora_dropout: 0.1
- •
r: 64
- •
bias: none
- •
task_type: CAUSAL_LM

For each iteration, the settings below are the same.

- •
er_device_train_batch_size: 4
- •
gradient_accumulation_steps: 1
- •
optim: paged_adamw_32bit
- •
learning_rate: 2e-4
- •
weight_decay: 0.001
- •
fp16: False
- •
bf16: False
- •
max_grad_norm: 0.3
- •
max_steps: -1
- •
warmup_ratio: 0.03
- •
group_by_length: True
- •
lr_scheduler_type: constant
- •
report_to: tensorboard

For num_train_epochs, the details are shown in Table 6.

## Appendix I Case study

Figure: Figure 12: Case study on Llama-3-8b (1).
Refer to caption: x15.png

Figure: Figure 13: Case study on Llama-3-8b (2).
Refer to caption: x16.png

Figure: Figure 14: Case study on Llama-3-8b (3).
Refer to caption: x17.png

Figure: Figure 15: Case study on GPT-3.5-turbo (1).
Refer to caption: x18.png

Figure: Figure 16: Case study on GPT-3.5-turbo (2).
Refer to caption: x19.png

Figures 12, 13, 14, 15 and 16 illustrate several cases comparing our models and the corresponding baseline models. Our models accurately answer the medical questions, while the base models provide incorrect responses. This discrepancy arises because the data generated during the patient-therapist interactions contains valuable medical knowledge, which aids in the analysis and formulation of the answer.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: 1. The paper mentions cost-effectiveness as a contribution. How did the authors determine that MentalArena is indeed cost-effective?

2. How was the quality of the generated treatment and medication data evaluated?

3. Since MentalArena aims to reduce intent bias in patient-therapist interactions, what metrics or qualitative analyses were employed to confirm that intent bias was actually reduced?

4. How does the model handle ethical considerations, such as potential misdiagnoses or inappropriate treatment suggestions, especially since it is aimed at the sensitive domain of mental health?

5. On the dreaddit and Irf datasets, different methods (such as the base GPT-3.5-turbo, Chain-of-thought prompting, MedPrompt, and the Ours) show identical scores. What could be the reason behind this phenomenon?

6. Could additional mental health datasets (more disorders) from Table 1 [1] be tested? Also, since MedQA includes a Psychiatry subset, could results be tested on this as well?

[1] A Comprehensive Evaluation of Large Language Models on Mental Illnesses https://arxiv.org/pdf/2409.15687
soundness: 2
strengths: The strength of MentalArena lies in its highly innovative self-play framework, which breaks away from the limitations of traditional prompt engineering commonly used to enhance language models for mental health applications. By enabling role-playing between patient and therapist, MentalArena dynamically generates data that simulates authentic interactions, allowing the model to learn and optimize autonomously through self-play—a promising exploration approach.
summary: The paper introduces MentalArena, a self-play framework designed to train language models for diagnosing and treating mental health disorders. Due to privacy concerns and the scarcity of personalized mental health data, traditional methods struggle to build effective models in this field. MentalArena overcomes this challenge by allowing the language model to play both the roles of patient and therapist, thereby generating domain-specific personalized data. The framework consists of three main components: the Symptom Encoder, which simulates the cognitive and behavioral patterns of patients; the Symptom Decoder, which simulates interactions between patients and therapists by comparing diagnosed symptoms with encoded symptoms, thereby reducing intent bias; and the Model Optimizer, which collects diagnostic, treatment, and medication information from these interactions to fine-tune the model. The authors evaluated MentalArena on six benchmarks, showing that models fine-tuned using this framework perform significantly better in diagnosing mental health disorders compared to existing models, including GPT-4o.
weaknesses: 1. Lack of clear problem justification: 

The authors claim to address the goal of "training language models for diagnosing mental health disorders," yet they evaluate the model using three general medical datasets (MedQA, MedMCQA, PubMedQA) that are unrelated to mental health. This approach fails to demonstrate the specificity of the model in diagnosing mental health disorders. After removing these general medical datasets, the model's performance on mental health-specific tasks is underwhelming, failing to show significant improvement in mental health tasks. This weakens the clarity and relevance of the paper's contribution to solving the stated problem.


| Model               | CAMS     | dreaddit | Irf      | AVG (Only mental health tasks)   |AVG (Including general medical datasets) |
|---------------------|----------|----------|----------|-------|----------------------------------------------|
| MentaLLaMa-13b      | 37.28    | 62.08    | 46.81    | 48.72 | 35.98 |
| Mental-LLM-alpaca   | 29.76    | 64.98    | 51.96    | 48.90 | 31.24 |
| Mental-LLM-t5       | 27.04    | 63.29    | 47.70    | 46.68 | 31.24 |
| GPT-4o              | 27.68    | 49.03    | 64.65    | 47.12 | 60.58 |
| GPT-4o+MedPrompt    | 31.52    | 53.27    | 64.65    | 49.81 | 64.22 |
| Base: GPT-3.5-turbo | 28.96    | 49.03    | 64.65    | 47.55 | 47.54 |
| +Chain-of-thought   | 29.92    | 49.03    | 64.65    | 47.87 | 48.87 |
| +MedPrompt          | 30.20    | 49.03    | 64.65    | 47.96 | 50.83 |
| +Ours               | 32.80    | 49.03    | 64.65    | 48.83 | 68.28 |
| Base: Llama-3-8b    | 25.12    | 58.45    | 45.76    | 43.78 | 54.75 |
| +Chain-of-thought   | 33.60    | 62.22    | 45.91    | 47.24 | 58.81 |
| +MedPrompt          | 35.08    | 61.59    | 48.05    | 48.24 | 60.17 |
| +Ours               | 29.60    | 65.46    | 52.25    | 49.77 | 61.39 |


2. Lack of domain expertise: 

Although the authors claim that MentalArena is designed for the "diagnosis" of mental health disorders, the selected datasets do not fully support diagnostic tasks. Two of the so-called "diagnostic" datasets (such as Dreaddit and Irf) are primarily used for "assessment," meaning they measure symptoms and cognitive states rather than providing definitive diagnoses. Furthermore, the mental health datasets used are limited to depression, neglecting other important mental health disorders, such as anxiety disorders, bipolar disorder, and schizophrenia. The lack of evaluation on these key disorders limits the model's breadth and efficacy, making it inadequate for comprehensive validation in mental health diagnosis.

Overall, the paper overstates its contributions by positioning MentalArena as a diagnostic tool for mental health, despite limited evidence of effectiveness on mental health-specific tasks and a lack of comprehensive validation across diverse mental health disorders.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 3
questions: - can you please explain the issues for self-play for training language models. Is the self-playing version of conversation short, or stuck somewhere, what are the average turns for such conversation.
- how can you monitor medical hallucination which is common and severe in mental health medial advice recommendations
- can you use patient data (such as suicide patient medical records from MIMIC) to predict mental health using your language model
soundness: 2
strengths: - this paper provides a unique way to generate synthetic conversational data for mental health patients. Due to the privacy issue, it is extremely hard to share the original call center data or conversational data about mental health patients.
- it is a smart way of using self playing to accumulate training data and train language model to generate better capability of answering different medQA datasets.
- Authors have compared their approach with several baseline methods, in a variety of Med QA datasets, and try to generalize it to other QA datasets.
summary: This paper proposed the self-playing framework called MentalArena to train language models through patients and therapists conversation. The trained model can be used for diagnosing mental health disorders or providing treatment. The evaluation of diagnosis and treatment is through the multiple choice of medical QA datasets.
weaknesses: - using medQA as proxy for mental health diagnosis can be problematic. Diagnosing mental health diseases require to follow medical guidelines (for example, diagnosis depression has guidelines for different age groups: https://www.apa.org/depression-guideline). Diagnosis is multiperspective, conversation is only one data points, others, such as blood test, lab test, and survey (such as PHQ-9 survey for depression), and it also requires to have longitudinal datapoints, rather than just a few turns of conversations. 
- using medQA as proxy for mental health treatment is a far reach. Choosing the right choice in multiple choice exam for medical students does not mean that chatbot can do treatment. Besides prescribing antidepressant medicines, CBT could be another options which requires to take 10 -15 sessions, and evaluate the clinical outputs. CBT is not a simple conversation, rather a protocol driven therapy. It is unclear that chatbot can be treated as therapists. Chatbot can be a co-pilot, but definitely cannot be treated as an independent therapist.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - In the "OVERVIEW OF THE FRAMEWORK" section (line 215), the authors state, "As treatment and medication plans are administered to the patient, their health state evolves, reflected in the sequential updates of encoded symptoms" over k rounds. However, within this same section, the framework is described as calculating the semantic similarity between the encoded symptom (S0) and the diagnosed symptom (Sd), continuing the conversation until this similarity score exceeds 0.9. Afterward, the patient’s symptoms are analyzed, and a diagnostic plan is developed, followed by k rounds of treatment/medication, which result in k new symptoms. This description does not align with the initial explanation of sequential updates in the middle of Figure 1, or with the description in Section 3.4 and Figure 2, causing considerable confusion. Could the authors clarify this inconsistency? A revised version of one of these figures could be helpful.
- In the supplementary materials, the prompt for the Symptom Decoder simply includes the ground-truth cognitive model, behavioral principles, and current diagnosis, followed by the question: "What can the therapist ask the patient to diagnose accurately?" This does not align with the explanation that the decoded system extracts the cognitive and behavioral principles understood by the therapist and compares their similarity to the ground truth. Could the authors provide a more detailed explanation of how the Symptom Decoder operates in practice? 
- From my understanding, the model’s fine-tuning is based solely on QA samples derived from the patient profile and the diagnosis, treatment, and medication generated through self-play. Is there evidence that this fine-tuning improves dialogue simulation performance? The authors could strengthen this section by providing comparative examples of dialogues pre- and post-fine-tuning, or by including metrics specifically designed to evaluate dialogue quality.
- In Table 2, the results across different settings for GPT-3.5-turbo all show identical performance results for the Dreaddit and IRF test sets. Is this an error?
- In Figure 3, what is the difference between the left and right graphs?
- In Figure 4, what is the Borderline and why don’t you display the PPL and Diversity Gain score in Iteration 0?
- The description of MedPrompt in the supplementary materials seems incorrect. As I understand it, "Random Few-Shot" is an ablation experiment for MedPrompt and is not actually part of the MedPrompt system.
soundness: 2
strengths: - MentalArena effectively addresses privacy challenges by generating domain-specific data through self-play, enabling model improvement without compromising patient confidentiality.
- Models fine-tuned through MentalArena achieve significant performance gains compared to baselines, indicating an effective enhancement of diagnostic and therapeutic capabilities.
- The framework includes mechanisms to minimize intent bias during patient-therapist interactions, enhancing the model's reliability.
- Evaluations across six benchmarks show that MentalArena-trained models excel in both mental health-specific and general medical tasks, demonstrating robust performance and generalizability.
- By generating 18,000 training samples, the authors provide a valuable resource for future research and model training in mental health domains.
summary: The paper introduces MentalArena, a self-play framework designed to train language models for mental health diagnosis and treatment. This approach enables the creation of high-quality, domain-specific data while addressing privacy concerns, which is critical in mental health care. The framework comprises three modules: the Symptom Encoder, which simulates a human-like mental health patient from both cognitive and behavioral perspectives; the Symptom Decoder, which addresses intent bias; and the Model Optimizer, which iterates and improves the model based on generated interactions. Using MentalArena, they produce 18,000 samples and train the model on this dataset. The models fine-tuned on GPT-3.5 and Llama-3-8b significantly outperformed the base models (GPT-3.5-turbo and Llama-3-8b) on benchmarks related to mental health and biomedical QA tasks.
weaknesses: - There may be some inconsistencies in the overall framework description. I will organize questions below to clarify these aspects.
- The inner workings of the Symptom Encoder and Decoder modules are not fully detailed, making it difficult to assess how effectively the framework models complex human symptoms and simulated dialogue. This is briefly touched upon in Table 3 ("Authenticity") but lacks further elaboration. Additionally, it is unclear whether the framework accurately reflects changes in the patient’s status over time.
- The quality of the synthetic dataset is also challenging to assess fully. In Table 3 ("Validity"), this aspect is merely evaluated through a simple query format, which does not provide an in-depth analysis. 
- Although the framework generates synthetic data for mental health patients, it shows only marginal improvement on mental health-specific test sets, while demonstrating significant gains in biomedical QA tasks.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: * In Section 3, MENTALARENA, what is the base model used for the symptom encoder and decoder? If I understand correctly, both the encoder and decoder use either LLaMA-3 or GPT-3.5, correct? Figure 1 is somewhat unclear on this point.
* In Section 3, MENTALARENA, how is data handled if it never reaches the defined alignment threshold in the symptom decoder?
* In Section 3.4, THERAPIST: SYMPTOM DECODER, how is the “best” defined? Is it determined using ground-truth labels?
* In the experiments, specifically in Table 2, why is there no fine-tuning or at least few-shot tuning for the baseline models GPT-3.5 and LLaMA-3? If this were done, would there still be a significant improvement?
* Why isn’t F1 score included as an evaluation metric, as it may be more suitable than accuracy? The MentaLLaMa model also uses F1 as an evaluation metric.
* In Section 4.2, MAIN RESULTS AND ABLATION STUDY, could you explain why each result with LLaMA-3 shows only marginal improvement, or even no improvement, across all datasets compared to fine-tuning on GPT-3.5?
soundness: 3
strengths: * The concept of having the LLM act as both the patient and therapist is novel and intriguing.
* The contribution to mental health research is substantial.
* The paper includes a wide range of evaluation benchmarks, providing robust assessment.
* The proposed model, MentalArea, demonstrates superior performance across all benchmarks.
summary: In this paper, the authors propose a novel self-play tuning framework to enhance the ability of LLMs for diagnosing mental symptoms. This enhanced model can simultaneously act as both the patient and therapist. Through the support of cognitive models and behavioral patterns, the data can be disentangled to provide more effective diagnosis, treatment, and medication recommendations, thereby fine-tuning the LLM. Evaluation on diverse benchmarks demonstrates significant improvements with the self-play tuning approach, highlighting the advantages of this framework. However, several issues still need to be clarified and addressed. I will update my score if my concerns are adequately resolved.
weaknesses: * The workflow of the entire self-play tuning process is unclear, you may update figure 1.
* There is a lack of illustration for the symptom encoder.
* The model inference process is not well-illustrated.
* A sensitivity analysis on the alignment threshold between the symptom encoder and decoder is missing.
* The selection of baseline models appears unfair. Since the base model used here is LLaMA-3-8b, why not either upgrade the base model of MentaLLaMa to LLaMA-3-8b or downgrade your model to LLaMA-13b for consistency?
* Related work could be improved by adding a discussion of similar research studies. 

   [1] Chen, Z., Deng, Y., Yuan, H., Ji, K., & Gu, Q. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. In Forty-first International Conference on Machine Learning.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See above
soundness: 2
strengths: The use of self-play for mental health modeling is novel and aligns well with current challenges in healthcare data privacy. By generating synthetic data, the model mitigates the need for real-world mental health records, which are often inaccessible due to confidentiality concerns.

The authors conducted evaluations on multiple biomedical QA and mental health tasks, and the MentalArena model exhibited superior performance compared to various state-of-the-art language models.

 The integration of Symptom Encoder and Decoder modules helps in handling intent bias and enhances the model's realism by simulating human-like responses. This layered approach is thoughtful, tackling key challenges specific to mental health applications.
summary: The paper introduces MentalArena, a self-play framework designed to train language models for the diagnosis and treatment of mental health disorders. The framework enables a language model to adopt both patient and therapist roles, generating synthetic data that simulates patient-therapist interactions. Through components such as the Symptom Encoder, Symptom Decoder, and Model Optimizer, MentalArena models cognitive and behavioral patterns associated with mental health patients and optimizes responses through iterative self-play. The authors demonstrate notable improvements in performance on benchmarks compared to existing models.
weaknesses: While synthetic data generation is advantageous for privacy, the paper could provide a deeper analysis of how well this data approximates real-world patient-therapist interactions. Evaluations involving feedback from human mental health professionals would strengthen claims of the model’s applicability.

Although the authors claim that MentalArena can generalize to other medical domains, the paper lacks detailed experiments beyond mental health, with limited benchmarks for validation. Future work could expand these evaluations to address potential generalization challenges.

The limitations section briefly mentions potential biases and computational constraints. However, ethical concerns, particularly the risk of misdiagnosis and over-reliance on automated tools in mental healthcare, are areas where a more extensive discussion would be valuable.