MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance

Authors: Jia Xu, Tianyi Wei, Bojian Hou, Patryk Orzechowski, Shu Yang, Ruochen Jin, Rachael Paulbeck, Joost Wagenaar, George Demiris, Li Shen
Primary area: datasets and benchmarks
Keywords: Mental Health, Large Language Model, Behavioral Health, Question and Answering

Abstract:
We introduce MentalChat16K, an English benchmark dataset combining a synthetic mental health counseling dataset and a dataset of anonymized transcripts from interventions between Behavioral Health Coaches and Caregivers of patients in palliative or hospice care. Covering a diverse range of conditions like depression, anxiety, and grief, this curated dataset is designed to facilitate the development and evaluation of large language models for conversational mental health assistance. By providing a high-quality resource tailored to this critical domain, MentalChat16K aims to advance research on empathetic, personalized AI solutions to improve access to mental health support services. The dataset prioritizes patient privacy, ethical considerations, and responsible data usage. MentalChat16K presents a valuable opportunity for the research community to innovate AI technologies that can positively impact mental well-being.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730590019548

thread_type:
official_review

<thread_object>
creator: reviewer_a2qn

<content>
summary:
This paper introduces a new dataset, MentalChat16k, as a potential benchmark dataset for evaluating the use of LLM in mental health counseling. The dataset hybridizes both (a) real interview transcripts between Behavioral Healthcare Coaches and caregivers of hospice patients and  (b) synthetic mental healthcare counseling question-answer pairs generated by GPT-3.5 Turbo and encapsulates over 16k question-answer pairs. The authors fine-tuned a wide range of families of small-scale (7B) LLMs on the datasets and conducted extensive qualitative evaluation on the performance of the fine-tuned models on a hold-out evaluation dataset of 200 questions collected from real-world discussion of mental health issues on public forums. The evaluation metrics differ from conventional quantitative metrics and focuses more on the empathetic aspects of the model generated responses to the healthcare questions, and both automatic evaluation done by LLMs (GPT-4 Turbo and Gemini Pro 1.0) and manual judgement done by domain experts have shown significant improvement in the empathy demonstrated in the generated responses of models fine-tuned on MentalChat16k, indicating the effectiveness of the dataset in helping models tackle the nuance and sensitive nature of assisting in the mental health domain.

strengths:
Originality: Excellent

Most existing datasets on depression and mental health issues focus on detection, thereby datasets are usually structured for classification, not generation. This dataset is a nice addition to this research space of NLG in healthcare.

Also, the seven-factor evaluation metrics is a novel contribution to the conventional evaluation metrics in NLG. The proposed metrics suite focuses on empathy more than accuracy, aligning better with the nuanced emotional requirements of real-world application of LLMs in sensitive areas such as mental health counseling and terminal care. However, the hastily introduction of new metrics can also contribute to reproducibility and generalizability problem with the research, which I will address in the weakness section.

Quality: Good
Paper presentation and language are generally excellent, with a few editing typos that need to be addressed.

Clarity: Excellent
The structure of the paper and the writing makes it very easy for the reviewer to follow.

Significance: Good
This paper aims at a focused group of researchers and rather than contributing to the general theoretical understanding of deep neural networks. The integration of deep learning and healthcare is becoming ever more relevant from both a research and a realistic perspective, and almost all areas of research in this space suffer from lack of quality data, so contribution of large-scale, high-quality healthcare data is always welcomed.

weaknesses:
Clarity:
1. A few details on the data collection procedure could be clarified. For example: 
  - Is the demographic information the only type of meta data collected? 
  - What about the 93 caregivers without demographic information? Will the question-answer pairs from them be missing the meta data? 
  - How good (in terms of diversity, topic coverage, etc.) are the synthetic question-answer pairs? 
  - It would be great if the authors could provide concrete example of the actual data, both from interviews and from the synthetic data, 
  - A brief explanation (or examples) that lays out the data structure (.json format? plain txt? what meta data is available? any data splits?) would be helpful, either in the main body or in the Appendix.
2. Basic statistics about the dataset are lacking in the paper, I would recommend the following statistics be added and organized in a table:
  - Exact dataset size and breakdown by interview data and synthetic data. Both are only mentioned now in the main text.
  - Length distribution (word count or character count) of the QA pairs.
  - For interview data portion, distribution of number of QA pairs across interview transcripts; or at least provide the average number of QA pairs obtained from each visit. 
  - For synthetic data, number of QA pairs distribution across each topic.

Quality:
1. It is unclear why the authors would break a complete interview, which the reviewer assumes to be a coherent series of questions and answers between healthcare coaches and caregivers, into individual QA pairs. It would be constructive to include discussion or reasoning on this particular choice for data collection. Specifically,
  - Please explain how the loss of contextual information, when breaking conversation into QA pairs, would affect the data quality.
  - Is the paraphrasing conducted to address partly this concern? So that each QA pair becomes somewhat independent of each other?

2. The introduction of the seven-factor evaluation metrics on empathy is interesting and novel to the reviewer, but because this is the only evaluation done by the authors, it also tends to conflate the focus of the paper into a paper of both a new dataset and a new, untested metrics. A few related points that could strengthen the paper, but lacking in the current version, are:
  - Why are the scoring scale different between LLM-based and human evaluation? 
  - The reviewer appreciates the detailed explanation of each score to be used in the prompt for LLM-based evaluation, but since this is such a detailed and nuanced range of scores, would the LLM really be capable of differentiating between different scores? Showing a distribution of LLM-based evaluation score one or few factors would be helpful. For example, it would point to potential issues if the resulting scores across 200 examples concentrate around "typical good" values, like 7 or 8.
  - Are the seven-factor metrics employed only in LLM-based evaluation, or do human experts follow along the same line of logic when evaluating the quality of the response? 
  - Since human evaluation is involved, and multiple experts have worked on the same question during evaluation, it would be more robust to include inter-rater agreement between the experts (or at least the variance in human ratings in addition to simple mean).

3. The findings in Table 3 and Table 8 also seem to point to questions left unanswered about the dataset. As can be seen in both of the tables:
    1. It is not true that more data generally leads to better performance
    2. It seems consistent that combining interview data with synthetic data actually HURTS model performance across all seven factors and model families
    3. It seems human evaluation consistently prefers models fine-tuned on synth data only

**3.3** alone could put into question the value of real data based on interviews if evaluation shows synthetic data seems to help model more. But the reviewer believes this may be attributed to misalignment between the interview data and the evaluation set. A couple of suggestions on analysis that could be included to pinpoint potential causes:
   1. Since the evaluation set contains data (InFamousCoder) originally designed for depression classification. Use the models (base and fine-tuned) to run classification on the question; the model performance could then point out the generalizability of the dataset (if fine-tuning benefits the model) or domain shift (if fine-tuning hurts the model performance) between MentalChat16k and the evaluation set.
   2. Conduct similarity-based analysis between questions in the synthetic data, in the interview data, and in the evaluation set. Reddit is a public data source that could be used in the training of GPT models, so this could reveal potential data leakage issue in the synthetic data. This is **not** an invalidation of the synthetic data, but more revealing the lack of efficacy in the evaluation approach. And it could also help reveal if there's an obvious domain shift between interview data and the synthetic data.
   3. Both 1 and 2 are trying to identify the cause behind the performance degradation when both sources of data are combined in model fine-tuning. But regardless of the findings, table 3 and 8 do point to the need of a finer-categorization of the dataset. It may be helpful to separate the dataset into two parts and caution the users to combine them naively together. 
   4. Another evidence supporting the separation of interview data and synthetic data is the contrast between the context; The interview data was created based on a very focused group of caregivers offering care to a specific group of patients, this is in stark contrast to the unbounded target profile used in generating the synthetic data. 
   5. If the goal of the dataset is to help improve **empathetic responses** from LLMs in the setting of mental healthcare counseling, then the reviewer suggests expanding the dataset by covering a wide range of responses different in level of empathy; that is, intentionally including non-empathetic responses, to better cover the range of the seven-factor metrics. Given the novelty of the metrics, such a dataset could also be ideal to further evaluate the robustness of  the metrics and LLM's capability of automated evaluation. This could be an extension of future work.

I'd like to emphasize that these weaknesses do NOT diminish the value of this dataset, but it does require more elaborated discussion, and probably a more thorough statement in the **Limitations** section.

questions:
1. Do you intend to publish the dataset? It is unclear from the writing whether this is intended for public use. I apologize if I missed a statement or url on this.
2. Clarifying question: what is the hypothesis testing used in calculating P-value in Table 3 and Table 8? I assume it's two sample t-test, but it would be nice to clarify it in writing.
3. Are the 200 questions crawled from Reddit and Mental Health Forum to be included as part of the dataset, or are they mostly for evaluation purpose for this paper? The depression data from Reddit, judging by the dataset from InFamousCoder, are typically not in the form of questions (more like posters explaining the hardship they've been through), were there any filtering or cleaning involved in selecting questions from the data?

Miscellaneous:
1. In Table 3 and 8, would it be better to rearrange the models in each family by the number of examples used in training? That is, order them by Base Model, Base Model + Interview data (\~7K), Base Model + Synth Data (\~10K), Base Model + Both (16K). It could help visualize the effect of dataset size on the finetuning (which is not significant, as would be obvious in the proposed order).
2. Typos: page 2, "toaddress"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647532221

thread_type:
official_review

<thread_object>
creator: reviewer_tuxe

<content>
summary:
This paper introduces MentalChat16K, a benchmark dataset for conversational mental health assistance. The dataset contains 16,000 question-answer pairs, including synthetic counseling conversations generated by GPT-3.5 Turbo and anonymized transcripts of interventions between caregivers of palliative or hospice patients and behavioral health coaches. Researchers utilized the QLoRA technique to fine-tune multiple state-of-the-art lightweight language models with 7 billion parameters to meet the needs of mental health counseling. They defined seven evaluation metrics and used GPT-4 Turbo, Gemini Pro 1.0, and human experts as evaluators to assess the models’ performance.

strengths:
This paper addresses the critical gap in real-world data for conversational mental health assistance. The lack of authentic psychological data has long posed a challenge in this field, and by collecting anonymized transcripts of intervention dialogues between behavioral health coaches and caregivers of palliative or hospice care patients, this paper provides researchers with a valuable data source.

weaknesses:
Although the paper demonstrates commendable efforts in creating the MentalChat16K dataset and fine-tuning large language models (LLMs) for mental health counseling, weaknesses in the evaluation methodology limit its impact and generalizability.

1. Reliability of LLM-Based Evaluation: The study primarily uses GPT-4 Turbo and Gemini Pro as automated evaluators to assess the performance of fine-tuned models. This approach presumes that these LLMs can objectively and accurately evaluate counseling responses across seven custom-defined mental health metrics. However, due to inherent biases and limitations, LLMs may not be entirely reliable as evaluators, especially in specialized fields such as mental health counseling.

2. Inconsistency in Evaluation Scores: As shown in Table 3, there is a notable inconsistency between evaluations from GPT-4, Gemini Pro, and human experts. Almost all scores from Gemini are higher than those from GPT-4, and Gemini’s scores diverge significantly from Human Rankings. This discrepancy raises concerns about the validity and robustness of using LLMs as evaluators, as they might not effectively capture the nuances involved in mental health counseling.

3. Outdated Evaluation Techniques: The paper employs simple prompt-based evaluation methods but does not leverage advanced frameworks in NLG evaluation. State-of-the-art methods such as G-Eval [1] and Prometheus [2] provide more sophisticated and reliable assessments of LLM outputs, which could yield a more accurate and nuanced evaluation of the models’ effectiveness.

[1] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment https://arxiv.org/abs/2303.16634

[2] Prometheus: Inducing Fine-grained Evaluation Capability in Language Models https://arxiv.org/abs/2310.08491

questions:
1. How were human evaluators instructed to rank model responses, and were there specific criteria or guidelines provided to ensure consistency and objectivity? Additionally, was any inter-rater reliability measure (e.g., Cohen’s Kappa) applied to validate the consistency of human rankings across evaluators?

2. Would the paper benefit from including side-by-side examples of model responses versus human counselor and baseline model responses? Could such examples better illustrate the model’s performance on key metrics like empathy, clarity, and safety?

3. Were participants explicitly informed about the potential use of their anonymized data for AI model development and research purposes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730735601762

thread_type:
official_review

<thread_object>
creator: reviewer_vt66

<content>
summary:
The authors develop MentalChat16k, a dataset consisting of two parts -- synthetic counseling conversations and paraphrased and anonymised real life interventions between behavioral health coaches and caregivers of patients in hospice or palliative care. They benchmark their dataset using 7 LLM models and 2 other baseline models. Finally, they use the finetuned and baseline methods to answer 200 mental health related questions, from Reddit, to see the performance difference. The authors found that the models finetuned on the MentalChat16k outperformed the other methods in the QA task, using their developed evaluation framework.

strengths:
- The authors curate an interesting dataset consisting of two type of important communication in the domain of mental health.
- They provide an empirical validation of the use of dataset by a thorough comparison of different LLMs and other baselines.
- The authors provide valid use cases for the curated data and verify it experimentally too.

weaknesses:
- The paper should have had a table summarising the dataset statistics for easy access.
- Examples of generated counseling transcripts in the paper would be useful to evaluate them.
- It would be interesting to observe if the paraphrasing step, as illustrated in Table 1, results in any hallucinations to complete the given "requirements" in the prompt.
- Inter annotation agreement for human evaluation would enhance the human evaluation.

questions:
See weaknesses please.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730788236568

thread_type:
official_review

<thread_object>
creator: reviewer_yjs7

<content>
summary:
This paper proposes a new benchmark mental health counseling dataset, named MentalChat16K. The dataset comprises question-answer pairs derived from transcripts of interactions with patients in palliative or hospice care. Additionally, the authors create a synthetic counseling dataset using large language models (LLMs). Experimental results show that fine-tuning LLMs with MentalChat16K significantly improves performance over non-fine-tuned baselines across seven metrics, as evaluated by both LLMs and human annotators.

strengths:
The primary strength of this paper is its introduction of the MentalChat16K dataset, a valuable mental health benchmark. Generating counseling responses is challenging due to the sensitive nature of the data, making dataset creation and public availability rare. MentalChat16K, being both sizable and relevant, is likely to prove useful in broader mental health and conversation tasks. The paper is generally well-written, though some parts would benefit from further elaboration.

weaknesses:
I have a few unresolved questions that would clarify the study's contributions and could enhance the paper.

questions:
- Who annotated MentalChat16K (referred to as human experts in lines 182 and 188)? Are these the same annotators who evaluated model performance (referred to as human experts in line 275)?
- MentalChat16K contains a total of 6,338 question-answer pairs. How many conversations does the dataset comprise? Are the pairs presented as individual exchanges, or do they maintain conversational continuity?
- What are the scores of the ground-truth responses in MentalChat16K when evaluated by the same metrics used to assess LLM performance? Since the evaluation method is reference-free, it would be useful to know how the dataset responses score according to the proposed metrics, and whether these scores exceed those of the fine-tuned LLMs.
- What test dataset was used in the experiments? Was an 80/20 split applied for training and testing, or was a different setup used? The reported p-values suggest multiple experimental runs, but Section 4.4 lacks details on the experimental setup.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732379763210

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 3/3

comment:
**Synthetic Data**

**Question**: Social situations terrify me. I have extreme social anxiety that makes it difficult for me to interact with others. I fear judgment and rejection, so I tend to isolate myself instead. This loneliness only worsens my depression and feelings of worthlessness. I want to overcome this fear and develop genuine connections with people.

**Answer**: I can understand how challenging social situations can be for you, and it's really brave of you to reach out and express your feelings. It takes a lot of strength to acknowledge these fears and desires for genuine connections.

One step that might help is gradually exposing yourself to social situations in a way that feels manageable for you. Start by setting small goals, like attending a social gathering for a short period of time or initiating a conversation with someone you feel comfortable around. Remember, progress doesn't have to happen overnight, so take things at your own pace.

It could also be helpful to challenge the negative thoughts and beliefs you may have about judgment and rejection. Remind yourself that everyone has their own insecurities and that people are often more understanding than we give them credit for. Building self-confidence and self-compassion can go a long way in overcoming these fears.

Additionally, finding supportive communities or groups where you share common interests can provide a safe space to practice socializing. This can help alleviate some anxiety as you'll be interacting with others who may have similar experiences or passions.
Engaging in activities that bring you joy and boost your self-esteem can also contribute to building connections. Pursue hobbies or join clubs where you can meet like-minded individuals who appreciate and value your presence.

Lastly, remember that developing genuine connections takes time and effort from both sides. Be patient with yourself and others, as relationships naturally evolve over time. Celebrate every small victory along the way and don't hesitate to seek support from trusted friends or family members who can offer encouragement during this journey.

You're not alone in wanting to overcome social anxiety and build meaningful connections. Keep taking those small steps forward, and I believe you will find the connection and sense of worthiness you desire.

**Interview Data**

**Question**: I've been having trouble sleeping for the past few months. I've tried various methods, like melatonin, meditation, and even changing my bedtime routine, but nothing seems to be working. I've also been feeling increasingly anxious and irritable, which I suspect is contributing to my sleep issues.

**Answer**: It's clear that your sleep disturbances have been a persistent issue for you, and it's concerning that you've tried various methods without much success. The fact that you've also been feeling increasingly anxious and irritable suggests that there may be underlying emotional triggers that are contributing to both your sleep disturbances and your emotional state.

To begin, we'll explore stress management techniques that can help you better cope with the emotional and mental demands that may be contributing to your sleep disturbances. This may include time management strategies, prioritizing tasks, setting realistic goals, and practicing relaxation techniques such as deep breathing exercises, progressive muscle relaxation, and mindfulness meditation.

Additionally, we'll delve deeper into the emotional aspects of your sleep issues. It's possible that there are underlying emotional triggers that are contributing to your sleep disturbances and your emotional state. We'll work together to identify these emotional triggers and develop strategies to process and resolve them, which may include cognitive-behavioral techniques, psychotherapy, or even mindfulness practices.

It's important to remember that improving sleep patterns and managing emotional distress takes time and consistent effort. However, with a collaborative and compassionate approach, we can work together to help you develop the skills and strategies necessary to achieve more restful and consistent sleep and to better manage the emotional and mental demands that may be contributing to your sleep disturbances.

---

**For other reviews, please refer to the detailed response for each reviewer. We believe these responses and revisions will address the reviewers' concerns effectively and strengthen our manuscript significantly. All these revisions have been reflected in the updated manuscript PDF marked in blue. Thank you once again for your valuable feedback, and we’re happy to address any further questions.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732380235055

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 2/3

comment:
>**A1: Inter-Rater Agreement among Human Evaluators**

Reviewer VT66, TuXE, and A2qn highlighted the need for assessing the inter-rater agreement among our human evaluators to enhance the robustness of our human evaluation. We have calculated the Cohen’s Kappa score to assess the consistency among the evaluators. Specifically, we randomly selected 30 questions from the 200 evaluation questions and had responses generated by 7 models (**3 global baselines** including Samantha-1.11, Samantha-1.2 and ChatPsychiatrist, **1 randomly selected base model before fine-tuning** such as Zephyr-Alpha, Vicuna-7B-V1.5, LLaMA2-7B, Mistral-7B-V0.1, Mistral-7B-Instruct-V0.2, Mixtral-8x7B-V0.1, Mixtral-8x7B-Instruct-V0.1, and its corresponding **3 fine-tuned models** fine-tuned on synthetic, interview and both data respectively). Four human evaluators ranked these responses from 1 (best) to 7 (worst). By treating each rank as the prediction target, we make the task become a 7-class classification problem and each human evaluator will generate a list of predictions for the 30 questions. We calculate the Cohen’s Kappa score among all the human evaluators’ lists of predictions. The resulting agreement is 0.441, which is larger than the acceptable threshold 0.4 as indicated by [1]. The details are also discussed in Section 3.3 from Lines 312 to 323 in the revised manuscript. 

[1] Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.


>**A2: Comprehensive Dataset Statistics**

Reviewer VT66 and A2qn suggested providing a table to summarize the statistics of our dataset. We hereby summarize comprehensive dataset statistics in the following table including the dataset size, average input word count, number of sessions, number of topics, etc. We have also added this information in Table 1 in Section 3.1 of our revised manuscript marked in blue.
 
| **Category**                     | **Interview Data** | **Synthetic Data** |
|----------------------------------|---------------|---------------|
| **Dataset Size (Rows)**          | 9775          | 6338          |
| **Columns**                      | instruction, input, output | instruction, input, output |
| **Average Input Word Count**     | 69.94         | 111.24        |
| **Average Output Word Count**    | 235.85        | 363.94        |
| **Number of Sessions**           | 378           | -             |
| **Average #QA Pairs per Session**| 16.8          | -             |
| **Number of Topics**             | -             | 33            |

We also provide a detailed summary for the topic distribution of the synthetic data as follows. We have also added the illustration of this information in Figure 3 in Appendix A.3.2 of our revised manuscript marked in blue.

| **Topic**                | **%** | **Topic**              | **%** | **Topic**              | **%** |
|--------------------------|-------|------------------------|-------|------------------------|-------|
| Relationships            | 15.0  | LGBTQ                 | 2.1   | Workplace Relationships | 1.0  |
| Anxiety                  | 9.4   | Substance Abuse       | 2.0   | Sleep Improvement       | 0.8  |
| Depression               | 8.1   | Anger Management      | 1.7   | Domestic Violence       | 0.7  |
| Intimacy                 | 7.8   | Addiction             | 1.6   | Grief and Loss          | 0.7  |
| Family Conflict          | 7.2   | Professional Ethics   | 1.5   | Career Counseling       | 0.7  |
| Marriage                 | 5.5   | Spirituality          | 1.3   | Diagnosis               | 0.4  |
| Social Relationships     | 4.2   | Legal & Regulatory    | 1.1   | Children & Adolescents  | 0.4  |
| Parenting                | 4.1   | Behavioral Change     | 2.8   | Military Issues         | 0.4  |
| Self-esteem              | 3.9   | Relationship Dissolution | 2.6 | Self-harm               | 0.4  |
| Human Sexuality          | 3.6   | Stress                | 2.2   | Eating Disorders        | 0.4  |
| Counseling Fundamentals  | 3.5   | Trauma                | 2.9   | Alzheimer's             | 0.1  |

>**A3: Example of Training Data**

Reviewer VT66, TuXE, and A2qn suggested providing a concrete example of our data. We agree that including some examples from the dataset could enhance the completeness of the paper and provide readers with a more intuitive understanding of the content. To address this, we incorporated a few representative examples of the synthetic data and the interview data in our revision (See Appendix A.11 marked in blue). We will also release our data in Huggingface to let everyone have access to it once the paper is published. We attached the same examples here (see **General Response 3/3**):
</content>
</thread_object>

</thread>
<thread>
creation date:
1732380324994

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 1/3

comment:
We sincerely thank all the reviewers for their thoughtful and detailed feedback, which has greatly helped us refine and strengthen our submission. We are pleased that the reviewers recognized the significance of our work in developing MentalChat16K as a valuable benchmark for advancing conversational AI in mental health counseling. Below, we summarize the key points raised by the reviewers and outline how we have addressed their concerns.

**Reviewer YJS7** commended the novelty and relevance of MentalChat16K and requested clarifications regarding annotators, dataset structure, evaluation methodology, and participant consent for AI applications. 

`We have clarified all these aspects accordingly.`

**Reviewer VT66** highlighted the novelty of MentalChat16K and its value to the mental health research community. Reviewer also recommended including inter-rater agreement metrics, a summary of dataset statistics, representative examples, and an analysis of potential hallucinations in the paraphrased data. 

`We have adopted all the suggestions including conducting the inter-rater agreement test, adding a comprehensive dataset statistics table, illustrating representative examples, and analyzing the hallucination issues.`

**Reviewer TuXE** appreciated our efforts in developing and fine-tuning LLMs for mental health counseling and raised important points about the reliability of LLM-based evaluations, the consistency of scores between GPT-4, Gemini Pro, and human evaluators, and the inclusion of state-of-the-art evaluation frameworks. 

`We have addressed these concerns by demonstrating the reliability and consistency of our evaluation strategy. We have also cited and discussed the suggested state-of-the-art evaluation framework.`

**Reviewer A2qn** acknowledged the significance of our contribution and the innovation in our evaluation metrics while suggesting improvements in demographic representation, evaluation robustness, and statistical analysis. 

`We have addressed these concerns by explicitly highlighting the demographic and contextual distinctions in the Section 7 Limitations, performing additional statistical analysis to validate the performance improvements, and recommending separate handling of the datasets to ensure effective use and mitigate potential performance degradation.`

Below (see **General Response 2/3**), we address the common concerns or questions raised by all (or most of) the reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733195507908

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript Submission – Thank You for Your Valuable Feedback

comment:
Dear Reviewers,

We wanted to express our sincere gratitude for the time and effort you’ve invested in reviewing our manuscript. Your insightful comments and constructive suggestions have been incredibly valuable, and we’ve made significant revisions to the paper based on your feedback.
We’ve updated the manuscript to address the concerns raised, including adding more detailed analyses, clarifying our methodologies, and enhancing the discussion of our evaluation metrics. We believe these changes have substantially strengthened the paper.

We kindly invite you to review the revised manuscript at your convenience. Your feedback is crucial to us, and we hope that the revisions meet your expectations and address your concerns.

Thank you again for your thoughtful contributions.

Best regards,

_The Authors_
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces MentalChat16K, a benchmark dataset designed for conversational mental health counseling assistance. It comprises 16,000 question-answer pairs, blending real transcripts that are anonymized and paraphrased interactions between behavioral health coaches and caregivers of hospice/palliative care patients and synthetic data comprising counseling conversations generated by GPT-3.5 Turbo. The authors fine-tuned lightweight 7B parameter LLMs using the QLoRA technique on the MentalChat16K dataset. Benchmarked performance of fine-tuned models against seven evaluation metrics (e.g., empathy, nuance)  using LLM-based evaluation (GPT-4 Turbo and Gemini Pro 1.0) and human evaluation.   

**Strengths identified**
1. The MentalChat16K dataset is a significant contribution to the field of mental health counseling and conversational AI. Its size and relevance make it a valuable resource for broader mental health and natural language generation (NLG) tasks.

2. Thorough empirical evaluation has been conducted comparing fine-tuned models with baselines demonstrates the dataset’s ability to improve LLM performance in mental health tasks.

**Weaknesses that need to be addressed**:
1. There seem to be concerns over the validity and reliability of the evaluation methodology, including missing inter-annotator agreement and reliance on LLM-based evaluations. The use of GPT-4 Turbo and Gemini Pro as evaluators assumes their reliability, which is not substantiated, especially in nuanced mental health tasks. Inconsistencies between LLM-based and human evaluations raise concerns about the validity of the evaluation approach. 

2. Insufficient experimental setup details and inadequate explanation of performance degradation when combining real and synthetic data. Tables (e.g., Table 3, Table 8) indicate that combining real and synthetic data often leads to performance degradation, which is not adequately analyzed or explained. 

3. Potential loss of context from breaking conversations into QA pairs and the impact of domain mismatch between synthetic and real data. 

4. Limited validation of the proposed evaluation metrics and missed opportunities to expand dataset coverage. No justification for differences in scoring scales between LLM-based and human evaluations.