Consistent Client Simulation for Mental Health Counseling
Authors: Yizhe Yang, Palakorn Achananuparp, Heyan Huang, Jing Jiang, Ee-Peng Lim


Reviewer paper summaries: The paper proposes a detailed framework for simulating client responses in mental counseling conversations. The framework operates by modeling mental states of the clients that are initialized with profile values such as behavioral problem, motivation, and personas. Actual modeling of transitions and client behavior is executed by LLM prompting. The authors conduct automatic and human evaluations to validate their methodology and claim that their framework leads to more realistic simulation of clients.
Summary of strengths: The paper proposes a much-needed study and implementation of client simulation in mental health conversations. In the domain of NLP & counseling, much of work has focused on evaluating and simulating care provider behavior. Having a well-implemented client simulator that can be easily controlled and mimic real client behavior well can be used for multiple ends such as training and evaluating counselors (real and automated) or generating datasets.

The framework proposed to implement client behavior is well motivated in terms of previous mental health & MI literature.

Automated evaluation focuses on includes profile consistency, which provides a good comparison of how different frameworks fare in this regards. Same goes for distribution of turn cont.
Summary of weaknesses: Since the simulation of conversation is fully automated (as opposed to last-turn generation), the contribution of counselor simulation should be discussed further. Although I understand that counselor simulation is fixed across comparisons, a small comparison or discussion of how this might affect the evaluation will help persuade the reviewers on the clarity of the evaluation.

Who were the human annotators who gave the human-like score? This is key information because I believe experts (in MI or mental counselors) will have a much clearer view on how clients act in practice. I couldn't find this information. Please correct me if this information was included in the manuscript and I missed it, then I will retract this point. Otherwise, this is a key limitation.

This point is not limited to the current work and I commend the authors for their work, but this work is based on LLM prompt engineering, and lacks interesting scientific or engineering contribution. This is not to say that this work is without value. However, I feel that this work will benefit strongly from stronger human evaluation or a case study based on the deployment of this system.
Comments, suggestions and typos: Full simulation of conversation is provided as experiment, but I am curious about the last response generation-based simulation of client response. That is, what if you provide real counseling conversation history and generate only the last counseling response? That will complement full simulation result and also show how well the proposed framework can mimic real client behavior. You can compute text similarity / overlap measure such as BERTScore, BLEU, ROUGE, etc.

Also AnnoMI is not a collection of "real" MI dataset. I believe it consists "staged" conversations between actors, collected for educational purposes.
Overall assessment scores: 3