MentalGPT: Harnessing AI for Compassionate Mental Health Support
Authors: Jia Xu, Tianyi Wei, Bojian Hou, Patryk Orzechowski, Shu Yang, George Demiris, Li Shen


Reviewer paper summaries: The paper describes a set of LLMs finetuned on the mental health domain. The models are trained on actual and synthetically generated conversation transcripts. The paper compares a variety of model families like Llama, mistral, vicuna, and zephyr for mental health generations. These generations are automatically evaluated with GPT 4 and Gemini Pro. All the base models have different tuning techniques like instruction tuning and qlora.
Summary of strengths: - The paper covers a good number of models for finetuning.
- The paper is well written.
- The prompts are explicitly given.
Summary of weaknesses: There are many drawbacks of this paper:
1. The key results focus on the fact that models do well in evaluating few categories of generations. These signify the strengths of GPT-X/ Gemini. There are no judgements on how these compare to open source models or other models. If the results focus on evaluative abilities, then more models should exist for comparison.
2. The evaluation is cyclic, LLMs evaluate LLM generations. There is no validation/ human-model agreement scores.
3. The paper describes models for finetuning but lacks coherences in how these models differ. Furthermore, the impact of quantization is also not ablated on. 
4. The paper also talks about significant impact on Mental Health + AI research space without referring to lot of key works in literature.
Comments, suggestions and typos: NA
Overall assessment scores: 2