CBT-Bench: Evaluating Large Language Models on Assisting Cognitive
  Behavior Therapy
Authors: name: Mian Zhang; name: Xianjun Yang; name: Xinlu Zhang; name: Travis Labrum; name: Jamie C. Chiu; name: Shaun M. Eack; name: Fei Fang; name: William Yang Wang; name: Zhiyu Zoey Chen
Published: 2024-10-17 04:52:57

Abstract: There is a significant gap between patient needs and available mental health
support today. In this paper, we aim to thoroughly examine the potential of
using Large Language Models (LLMs) to assist professional psychotherapy. To
this end, we propose a new benchmark, CBT-BENCH, for the systematic evaluation
of cognitive behavioral therapy (CBT) assistance. We include three levels of
tasks in CBT-BENCH: I: Basic CBT knowledge acquisition, with the task of
multiple-choice questions; II: Cognitive model understanding, with the tasks of
cognitive distortion classification, primary core belief classification, and
fine-grained core belief classification; III: Therapeutic response generation,
with the task of generating responses to patient speech in CBT therapy
sessions. These tasks encompass key aspects of CBT that could potentially be
enhanced through AI assistance, while also outlining a hierarchy of capability
requirements, ranging from basic knowledge recitation to engaging in real
therapeutic conversations. We evaluated representative LLMs on our benchmark.
Experimental results indicate that while LLMs perform well in reciting CBT
knowledge, they fall short in complex real-world scenarios requiring deep
analysis of patients' cognitive structures and generating effective responses,
suggesting potential future work.