Severity Prediction in Mental Health: LLM-based Creation, Analysis, Evaluation of a Novel Multilingual Dataset
Authors: Konstantinos Skianis, John Pavlopoulos, A. Seza Doğruöz


Reviewer paper summaries: The paper presents a multilingual investigation of the capabilities of large language models (GPT-3.5-turbo per se) for depression and suicidal thoughts identification. The authors translated two existing mental health corpora (Dep-Red and SUI-TWI) into six languages (Greek, Turkish, French, Portuguese, German, and Finnish) and employed carefully crafted prompts to extract mental health labels for each language. The study is intriguing, highlighting the limitations of GPT-3.5-turbo’s ability to comprehend and address mental health issues across diverse linguistic and cultural contexts. However, the paper lacks completeness and evaluation across different LLMs, has significant technical gaps, and requires writing refinements. I strongly encourage the authors to review the weakness and comments section in detail and consider incorporating the provided suggestions to bolster the paper’s overall strength.
Summary of strengths: 1. This paper appears to be the first study, to analyze various mental health disorders across multiple languages using LLMs.
2. The authors clearly articulate their prompting strategy for translation and identifying mental health disorders, providing detailed prompts as well.
3. The authors present results across all languages in terms of accuracy and F1 scores.
4. The authors also include a thorough error analysis and quantitative evaluations.
Summary of weaknesses: 1. The paper would benefit from improved writing quality, as it currently contains numerous grammatical mistakes and errors.
2. The paper lacks sufficient coverage of relevant literature in the related works. Mental health in NLP is a vibrant research area with numerous pertinent studies published in recent years. I have listed some missing references in the next section.
3. The authors limit their experimentation to one LLM, GPT-3.5-turbo, which provides a myopic perspective on the broader landscape of LLMs. Other potent general purpose LLMs such as Llama, Claude, and the Gemini series, as well as domain specific open-sourced models like Mental-T5, Mental-Alpaca, and Mental-Llama should have been considered for this study.
4. The results section is overly simplistic. While accuracy and F1 scores are reported, they do not provide sufficient insight into the classification capabilities of the LLMs. It is crucial to include precision and recall metrics, which indicate how aggressive or conservative a model is in predicting specific labels.
5. The authors do not to conduct extensive experiments using different seed and temperature values to assess the robustness of the model's predictions.
6. Translation existing corpora using LLM might induce additional errors. The authors do not provide details of whether and how they performed a quality check of the generated translations. Crawling actual data from social media might be a better alternative.
Comments, suggestions and typos: **Typos and Suggestions:**
1. Line 56 - The citations of Guntuku et al., 2019 and  Chancellor et al., 2019 do not discuss the ability of LLMs to identify linguistic markers associated with mental health conditions. Perhaps remove them from the introduction and elaborate more in the related works section.
2. Line 71 — The sentence is wrongly constructed.
3. Line 152 — Zhang et al., 2020 does not talk about cultural challenges in multilingual translation as assumed by the authors.

**Missing Reference:**

1. Adrian Benton, Glen Coppersmith, and Mark Dredze. 2017. Ethical research protocols for social media health research. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 94–102, Valencia, Spain. Association for Computational Linguistics
2. Glen Coppersmith, Mark Dredze, and Craig Harman. 2014a. Quantifying mental health signals in Twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 51–60, Baltimore, Maryland, USA. 
3. Association for Computational Linguistics. Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. 2015. CLPsych 2015 shared task: Depression and PTSD on Twitter. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 31– 39, Denver, Colorado. Association for Computational Linguistics.
4. Munmun De Choudhury, Scott Counts, and Eric Horvitz. 2013a. Social media as a measurement tool of depression in populations. In Proceedings of the 5th Annual ACM Web Science Conference, WebSci ’13, page 47–56, New York, NY, USA. Association for Computing Machinery. 
5. Munmun De Choudhury and Sushovan De. 2014. Mental health discourse on reddit: Self-disclosure, social support, and anonymity. Proceedings of the International AAAI Conference on Web and Social Media, 8(1):71–80.
6. Munmun De Choudhury, Sachin R Pendse, and Neha Kumar. 2023. Benefits and harms of large language models in digital mental health. arXiv preprint arXiv:2311.14693.
7. Shaoxiong Ji, Tianlin Zhang, Kailai Yang, Sophia Ananiadou, Erik Cambria, and Jörg Tiedemann. 2023. Domain-specific continued pretraining of language models for capturing long context in mental health. arXiv preprint arXiv:2304.10447.
8. Keith Harrigian, Carlos Aguirre, and Mark Dredze. 2021. On the State of Social Media Data for Mental Health Research. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 15–24, Online. Association for Computational Linguistics. 
9. Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghassemi, Anind K. Dey, and Dakuo Wang. 2024. Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8, 1, Article 31 (March 2024), 32 pages. https://doi.org/10.1145/3643540 
10. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, and Sophia Ananiadou. 2023a. On the evaluations of chatgpt and emotion-enhanced prompting for mental health analysis. arXiv preprint arXiv:2304.03347
11. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. Towards Interpretable Mental Health Analysis with Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6056–6077, Singapore. Association for Computational Linguistics.
Overall assessment scores: 1.5