Venue: NAACL 2025

# Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoning

Subin Kim1, Hoonrae Kim*1, Heejin Do1, and Gary Geunbae Lee1,2

Footnote 1: Equal Contribution

###### Abstract

Previous research has revealed the potential of large language models (LLMs) to support cognitive reframing therapy; however, their focus was primarily on text-based methods, often overlooking the importance of non-verbal evidence crucial in real-life therapy. To alleviate this gap, we extend the textual cognitive reframing to multimodality, incorporating visual clues. Specifically, we present a new dataset called **M**ulti **I**odal-**C**ognitive **S**upport **C**onversation (M2CoSC), which pairs each GPT-4-generated dialogue with an image that reflects the virtual client's facial expressions. To better mirror real psychotherapy, where facial expressions lead to interpreting implicit emotional evidence, we propose a multi-hop psychotherapeutic reasoning approach that explicitly identifies and incorporates subtle evidence. Our comprehensive experiments with both LLMs and vision-language models (VLMs) demonstrate that the VLMs' performance as psychotherapy is significantly improved with the M2CoSC dataset. Furthermore, the multi-hop psychotherapeutic reasoning method enables VLMs to provide more thoughtful and empathetic suggestions, outperforming standard prompting methods.

1Graduate School of Artificial Intelligence, POSTECH, South Korea

2Department of Computer Science and Engineering, POSTECH, South Korea

{kimsubin, hoonrae, heejindo, gblee}@postech.ac.kr

## 1 Introduction

As a crucial part of cognitive behavioral therapy (CBT), _cognitive reframing_ addresses lots of mental health issues rooted in deeplygrained negative and distorted thought patterns Beck (1970); Powles (1974); Beck (1987); Walen et al. (1992); Halamandaris and Power (1997); DiTomasso et al. (2000); Hofmann et al. (2012). Recently, studies attempted to utilize large language models (LLMs) in this task, highlighting their growing potential in the field of psychotherapy Ziems et al. (2022); Maddela et al. (2023); Sharma et al. (2023); Qu et al. (2023); Yang et al. (2023, 2024); Xiao et al. (2024). Conventionally, cognitive reframing has been explored with text-based sentence rewriting methods aimed at shifting negative viewpoints to positive ones Ziems et al. (2022); Maddela et al. (2023); Sharma et al. (2023). Concerned that sentence-based cognitive reframing can feel unnaturally imposed, Xiao et al. (2024) suggest a three-stage conversational approach with LLMs encouraging clients to engage more actively and form self-positive viewpoints.

Despite the promising results of LLMs in previous systems, non-verbal aspects of psychootherapeutic theory are often overlooked, creating a significant gap between real face-to-face therapy and the systems. In actual psychotherapy situations, recognizing non-verbal emotions is essential for effective communication and is a critical skill closely linked to the therapist's ability to provide effective therapy Hutchison and Gerstein (2012); Dollinger et al. (2021).

In this study, we propose to extend the concept of cognitive reframing into multimodality, integrating visual clues into the therapy process. Our aim is to provide a natural and effective cognitive reframing

Figure 1: Illustration of a multimodal conversational cognitive reframing. The therapist uses both verbal and non-verbal information to assess the client’s states and then provides appropriate interventions.1

framework within a multimodal context, incorporating clients' non-verbal clues in the conversation (Figure 1). To this end, we create a novel synthetic benchmark dataset, **M**ulti **M**odal-**C**ognitive **S**upport **C**onversation (M2CoSC), which pairs each synthetic dialogue with an image of the client's facial expression.

To create M2CoSC, we utilize the powerful roleplaying capabilities of LLMs. We leverage GPT-4 Vision and GPT-4 [8] to take on the roles of psychotherapist and client, respectively, simulating therapy sessions as shown in Figure 2. Inspired by counseling theory's [35, 60, 79]_Initial Disclosure_ stage among the five stages, we add the Introduction phase into our multimodal psychotherapy framework, extending the three-stage model proposed by Xiao et al. [7]: Introduction, Problem Exploration, Brainstorming, and Suggestion. In addition, to provide explicit guidance based on the client's state, we introduce a multi-hop psychotherapeutic reasoning method. By exploring the implicit evidence necessary for the therapy and then generating responses based on this evidence, the AI therapist can offer adequate guidance after identifying the client's state.

We evaluate our approach by conducting extensive experiments with two test scenarios, dialogue-level evaluation and stage-level evaluation, using both LLMs and Vision-Language Models (VLMs). The results show that training with our M2CoSC significantly enhances the counseling capabilities of VLMs, surpassing those of existing LLMs. Moreover, the multi-hop psychotherapeutic reasoning method allows VLMs to offer more rational and empathetic suggestions, outperforming standard prompting methods. Human evaluations further suggest that capturing the client's facial expressions in the system, as practical therapists do, can remarkably assist counseling.

To sum up, our contributions are as follows: (1) We propose multimodal cognitive reframing therapy using non-verbal information, creating the M2CoSC dataset that pairs dialogues with client facial expressions. (2) We establish a baseline for the M2CoSC dataset and propose a multi-hop psychotherapeutic reasoning approach to improve the capabilities of VLMs in delivering rational therapeutic interventions.

This work is a first step toward bringing multimodal cognitive reframing into AI-enhanced psychotherapy. By introducing a benchmark dataset

Figure 2: An example illustrating the construction of the M2CoSC dataset. Left: the prompt given to GPT-4 in the **client** role; Right: the prompt given to GPT-4 Vision in the **t****herapist** role. GPT-4 Vision is given a client’s face image. [dialogue history] denotes a history of conversations accumulated during role play.

 

and a structured reasoning approach, we hope to inspire future research on leveraging non-verbal cues for more effective therapeutic conversations.

## 2 Problem Definition and Goals

In cognitive reframing therapy, a therapist must understand the client's states, which include their problematic situations, distorted thoughts, and thinking traps. The therapist then encourages the client to consider alternative possibilities. Building rapport with the client by expressing empathy is also crucial Horvath and Luborsky (1993); Lambert and Barley (2001). In the real-world psychotherapy procedure, these stages involve both verbal and non-verbal information, where the therapist has sufficient ability to understand the client's states.

Here, our goal is to enhance the abilities of an AI psychotherapist by leveraging non-verbal information, guiding it to focus on facial expressions and to comprehend the client's states. Given the client's facial image and dialogue history, we aim to provide empathetic responses while maintaining a consistent focus on the client's issues throughout the procedure, offering rational interventions free from logical errors or contradictions.

To achieve our goals, we created the M2CoSC dataset founded on three key values--empathy, logical coherence, and guidance--which serve as evaluation criteria in a prior study Xiao et al. (2024).

* Empathy reflects the therapist's ability to understand and connect with the client's emotions, assisting in building trust, connection, and emotional support, all critical to a strong therapeutic relationship.
* Logical coherence denotes the therapist's ability to organize thoughts and provide well-structured insights, enhancing the quality of the conversation.
* Guidance indicates the therapist's capacity to offer practical advice, solutions, and direction,iding the client to navigate challenges, make informed decisions, and achieve positive outcomes.

We also utilize overall scores encompassing all three items. (see Appendix A for details.)

## 3 M2 Cognitive Support Conversation

### Dataset Construction

Actual therapy conversations are rarely accessible due to the sensitive nature of mental health therapy; thus, we have created a synthetic dataset that can be shared with the research community2. To construct multimodal conversational cognitive reframing dataset, we utilize two publicly available sources: the Facial Expression Recognition (FER) dataset called AffectNet Mollahosseini et al. (2019), and the cognitive reframing dataset from Sharma et al. (2023). To address potential privacy concerns associated with using images of real people from AffectNet, we obtained agreement for all research participants, ensuring full compliance with AffectNet's policies.

Footnote 2: Under AffectNet’s license, M2CoSC dataset is partially available at [https://github.com/nobel-postech/M2C05C](https://github.com/nobel-postech/M2C05C).

For construction, we set up role-play scenarios with two agents: GPT-4 in the client role and GPT-4 Vision in the therapist role.

As shown in Figure 2, we prompt GPT-4 in the client role and GPT-4 Vision in the therapist role using a set of four inputs: _image, facial expression, thinking traps_, and _thought_. 3 The _image_ represents the client's facial image, the _facial expression_ denotes the client's facial expression, the _thought_ reflects the client's thoughts, and the _thinking traps_ capture cognitive distortions present in the _thought_.

Footnote 3: We used version qpt-4–0613 of the GPT-4 API and version gpt-4-1106-vision-preview of the GPT-4 Vision API.

For facial expressions and images, We employ AffectNet, containing publicly accessible images from the internet collected for research under non-commercial use. For thinking traps and thoughts, we utilize the well-designed open-sourced dataset from Sharma et al. (2023), which was collected following ethical guidelines, including informed consent and participant privacy safeguards. To the best of our knowledge, this work is the first to combine multiple datasets to create a multimodal conversation specifically designed for the mental health domain.

Each dialogue consists of four turns, which correspond to different stages of a psychotherapeutic conversation. In this context, a "turn" is the same as a "stage." The action expected from the client is to follow the psychometricist's instructions, and the actions required for the psychotherapist at each stage are as follows.

 

1. **Introduction**: The AI psychotherapist expresses empathy and encourages the client to explore their problems.
2. **Exploration**: The AI psychotherapist guides the client to distinguish their thoughts from their situation.
3. **Brainstorming**: The AI psychotherapist discusses other possibilities for the client's interpretation. This involves asking about the basis for the client's thoughts or considering the possibility of alternative interpretations.
4. **Suggestion**: The AI psychotherapist first recognizes the client's effort to explore other possibilities and presents specific and rational suggestions for the client.

Considering the characteristics of cognitive re-framing counseling, which often involves addressing negative emotions, we excluded the "happy" expression from the 8 facial expressions in AffectNet. The matching between (_image, facial expression_) and (_thinking traps, client's thought_) was performed randomly with uniform distribution.

The statistics for the M2CoSC dataset are summarized in Table 1. The M2CoSC dataset contains a total of 429 conversations, each consisting of exactly four turns.

### Dataset Cleansing

To ensure the quality of the M2CoSC dataset, we conducted manual data cleansing with the three native speakers, focusing on four aspects: _Clientarity_, _Client-role_, _Therapist-role_, and _Image-Dialogue Consistency_ (see Appendix B for detailed criteria). The _Image-Dialogue Consistency_ is a criterion that evaluates whether the client's visual information and dialogue are consistent. If any of the four criteria received a score of 0, the corresponding data was deleted. Table 2 indicates a considerable correlation between the client's facial expressions and their verbal responses in our M2CoSC dataset.

We hired three native English speakers through Upwork4, a crowdsourcing platform, to support this cleansing process.

Footnote 4: [https://www.upwork.com/](https://www.upwork.com/)

### Dataset Quality Validation

To further validate the quality of the M2CoSC, we evaluate the test set of the M2CoSC dataset based on three criteria: empathy, logical coherence, and guidance, along with an overall score. Each criterion was rated on a scale from 0 to 3 following the manual provided by Xiao et al. (2024). The test set evaluation was conducted using both human and GPT-4 assessments. We hired an English-fluent psychotherapist for this evaluation and engaged them to manually evaluate the test set according to the detailed guidelines. We also employed GPT-4 for evaluation, feeding it the scorecard criteria and the dialogues from the test set.

The results in Table 3 indicate that both the human evaluator and GPT-4 provided high scores, showing similar scoring tendencies except for "Guidance." GPT-4 tended to give lower scores in this aspect, possibly attributable to its higher degree of expectation for problem-solving than human evaluators Chiu et al. (2024). Despite this discrepancy, the overall consistency in the other criteria supports the dataset's reliability.

\begin{table}
\begin{tabular}{l c|c c|c} \hline \hline  & \multicolumn{2}{c|}{**Avg. Tokens**} & \\  & **\# of Dialogue** & **Client** & **Therapist** & **Rounds** \\ \hline Train & 329 & 24.93 & 63.64 & 4 \\ Test & 100 & 24.01 & 62.81 & 4 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dataset statistics for M2CoSC. _# of Dialogue_ indicates the total number of dialogues in the subset. _Avg. Tokens_ represents the average number of tokens per utterance from the _Client_ and the _Therapist_. _Rounds_ denotes the number of turns per dialogue in the subset.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **Empathy** & \begin{tabular}{c} **Logical** \\ **Coherence** \\ (0-3) \\ \end{tabular} & \begin{tabular}{c} **Guidance** \\ (0-3) \\ \end{tabular} & 
\begin{tabular}{c} **Overall** \\ (0-3) \\ \end{tabular} \\ \hline Human & 2.929 & 2.980 & 2.879 & 2.929 \\ GPT-4 & 2.920 & 2.930 & 2.400 & 2.420 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Human and GPT-4 evaluation results on M2CoSC test set.

 



### Multi-hop Psychotherapeutic Reasoning

To ensure that the interventions are tailored to the client's needs, professional psychotherapists typically first understand the client's state and then provide interventions grounded on that (Greenberg and Safran, 1989; Rice and Elliott, 1996). To mimic the real therapy process, we introduce multi-hop psychometric reasoning. This approach identifies implicit evidence crucial for cognitive reframing and incorporates it into step-by-step instructions. Initially, the AI therapist detects the client's state and then generates a response based on it, as illustrated in Figure 3. In this work, we focus on three major aspects of the client's states--_facial expression_, _thought_, and _thinking traps_--which accumulate over the stage of the conversation. Each piece of evidence is identified at the appropriate stage. The detected evidence is included in the client's states, and the states are fed to the AI therapist as the prompt for the next evidence detection.

## 4 Experiments

### Settings

Baseline models.We utilize two representative models for our experiments: LLaMA2-chat-7b (Touvron et al., 2023)5, which is widely used in text generation tasks, and LLaVA-v1.5-7b6, renowned for vision-related tasks. For simplicity, we refer to LLaMA2-chat-7b as LLaMA2 and LLaVA-v1.5-7b as LLaVA in this work. In addition, we denote the versions of LLaMA2 and LLaVA that were trained on the M2CoSC dataset as CS-LLaMA2 and CS-LLaVA, respectively. When multi-hop psychotherapeutic reasoning is applied, we add MH to their names.

Footnote 5: [https://huggingface.co/meta-llama/Llana-2-7b-chat-hf](https://huggingface.co/meta-llama/Llana-2-7b-chat-hf)

Hyper-parameters.Both LLaMA2 and LLaVA were fine-tuned with LoRA (Hu et al., 2022) on the M2CoSC dataset using default settings, except for the number of epochs. For LLaMA2, we used the official open-source models from Hugging Face, and for LLaVA, we followed the official code defaults7. We split the M2CoSC train set into 80:20 training and validation subsets to select the optimal epoch based on validation performance. All models were trained with 4 \(\times\) A100-80GB GPUs,

Figure 3: Comparison of standard prompting and multi-hop psychotherapeutic reasoning. The multi-hop approach integrates the client’s emotional and cognitive state (facial expressions, thoughts, and cognitive distortions) at each step of the intervention. The conversation on the left shows the therapist’s replies, which correspond to the four stages—Introduction, Guidance, Brainstorming, and Suggestion—outlined on the right.



using a batch size of 32 per GPU.

### Evaluators

Gpt-4Recent research has shown that the evaluation of natural language generation (NLG) models using GPT-4 closely aligns with human evaluations. Therefore, GPT-4 is increasingly used as a judge for NLG tasks across various domains, including common applications, medical fields, and mathematics Liu et al. (2023); Sottana et al. (2023); Hsu et al. (2023); Khodaker et al. (2023); Xiao et al. (2024). Also, in conversation models, Zheng et al. (2023) showed that GPT-4 achieves high agreement with human judgment in evaluations, releasing the corresponding judging prompt and the used codes8. Building on this research, we evaluated the AI therapists using GPT-4 (API version)9 for evaluation in two ways:

Footnote 8: We utilize prompts from [https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge)

Footnote 9: We used the gpt-4-0613 version of the GPT-4 API.

* **Score assessment**: We adopt a three-dimensional scoring system for AI therapists, evaluating them on empathy, logical coherence, and guidance.
* **Pairwise comparison**: We compared the interventions of therapists to determine whether Model A is better than Model B and vice versa or if it's a tie for all possible pairs.10 Footnote 10: To ensure fairness and prevent position bias, we tested each case twice, swapping the positions each time.

Human.To enhance the reliability of the intervention evaluation, we conducted human evaluations with domain experts. We hired two fluent English-speaking psychotherapists via Upwork. The experts performed a pairwise comparison between our CS-LLaVA with multi-hop psychometric reasoning and others (see Appendix C).

## 5 Results and Discussions

For reliable comparison, we compared the performance of both LLMs and VLMs with two test scenarios: dialogue-level evaluation and stage-level evaluation. The dialogue-level testbed, which has been used in prior research, allows us to observe how interventions are carried out throughout conversations. However, relying solely on this testbed makes it difficult to accurately compare the AI therapists' abilities due to the variability of the AI client. To better assess interventions in terms of empathy, logical coherence, and rationality, we also conducted a stage-level evaluation. This approach enabled us to compare therapists' interventions more precisely by analyzing turn-level performance on the M2CoSC test set, using consistent contextual input across the models.

### Dialogue-level Evaluation

In this scenario, we employ ChatGPT (API version)11 as an AI client to test our approach in AI-to

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline  & & **Empathy** & 
\begin{tabular}{c} **Logical** \\ **Coherence** \\ \end{tabular} & **Guidance** & **Overall** & **Avg.** \\ \hline \multirow{2}{*}{Baselines} & LLaMA2 & 2.665\({}^{*}\) & 2.390\({}^{*}\) & 1.600\({}^{*}\) & 1.540\({}^{*}\) & 2.218\({}^{*}\) \\  & LLaVA & 2.640\({}^{*}\) & 2.570\({}^{*}\) & 1.790\({}^{*}\) & 1.740\({}^{*}\) & 2.333\({}^{*}\) \\ \hline \multirow{3}{*}{Ours} & CS-LLaMA2 & 2.690\({}^{*}\) & 2.410\({}^{*}\) & 1.640\({}^{*}\) & 1.580\({}^{*}\) & 2.247\({}^{*}\) \\  & CS-LLaVA & 2.915\({}^{*}\) & 2.890 & 2.380 & 2.400 & 2.728 \\ \cline{1-1}  & CS-LLaVA w/ MH & **2.980** & **2.960** & **2.510** & **2.490** & **2.817** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Dialogue-level assessment results evaluated by GPT-4 using a role-playing approach with an AI client. Stars (*) next to values indicate a significant difference compared to CS-LLaVA w/ MH , with a p-value ¡ 0.05 determined by the paired t-test.

Figure 4: Dialogue-level win rates assessed by GPT-4. Detailed numerical results are provided in Appendix D.1.

 

AI scenarios. For prompting to AI client, we leverage 100 resources which are used as base resources from the test set, originally sourced from Sharma et al. (2023) and Mollahosseini et al. (2019). The AI client's role aligns with our data construction method, and we use the same prompts throughout the process. To evaluate the performance difference between using only the text modality and incorporating image information, we also compared the results of CS-LLaMA2. For CS-LLaMA2, only the text modality was used without incorporating image information.

Table 4 shows the dialogue-level assessment results evaluated by GPT-4. Our M2CoSC dataset with the LLaVA family of models led to significant improvements across all aspects. By integrating multi-hop psychotherapeutic reasoning with three implicit evidences--_facial expressions_, _thoughts_, and _thinking traps_--the models achieved enhancements across all evaluation aspects, with a particularly remarkable improvement in empathy. These results support our hypothesis that understanding the client's emotional state before responding leads to more tailored and compassionate interactions.

LLaMA2, in contrast, shows minimal improvement when trained on the M2CoSC dataset, primarily due to the absence of visual information during the Introduction stage, which hampers effective training and results in subtle change. These findings validate our hypothesis that models integrating multimodal information in counseling conversations possess superior therapeutic qualities compared to models relying solely on text. Consequently, we shifted our focus to the LLaVA family for further analysis instead of assessing CS-LLaMA2 in stage-level evaluations, except during human evaluation settings.

Similar trends were observed in the pairwise comparison (Figure 4). Despite showing the lowest performance, LLaVA exhibited a significant improvement when training on the M2CoSC and applying our multi-hop psychotherapeutic reasoning method, achieving the highest performance in CS-LLaVA w/ MH. No significant performance difference was noted between LLaMA2 and CS-LLaMA2 due to the absence of visual information, further highlighting the impact of multimodal integration in counseling conversations.

\begin{table}
\begin{tabular}{c l|l c c|c c c|c c|c c|c c} \hline \hline  & & \multicolumn{3}{c|}{**Introduction**} & \multicolumn{3}{c|}{**Exploration**} & \multicolumn{3}{c|}{**Brainstorming**} & \multicolumn{3}{c}{**Suggestion**} \\  & & **Emp.** & **Coh.** & **Gui.** & **Emp.** & **Coh.** & **Gui.** & **Emp.** & **Coh.** & **Gui.** & **Emp.** & **Coh.** & **Gui.** \\ \hline \multirow{2}{*}{Baselines} & LLaMA2 & 1.58\({}^{*}\) & 1.79\({}^{*}\) & 0.80\({}^{*}\) & 2.16 & 2.20\({}^{*}\) & 1.03\({}^{*}\) & 2.10\({}^{*}\) & 2.18\({}^{*}\) & 1.44\({}^{*}\) & 2.17\({}^{*}\) & 2.06\({}^{*}\) & 0.97\({}^{*}\) \\  & LLaVA & 0.64\({}^{*}\) & 0.98\({}^{*}\) & 0.05\({}^{*}\) & 1.94\({}^{*}\) & 1.19\({}^{*}\) & 1.12\({}^{*}\) & 1.86\({}^{*}\) & 1.99\({}^{*}\) & 1.39\({}^{*}\) & 2.21\({}^{*}\) & 2.37\({}^{*}\) & 1.50\({}^{*}\) \\ \hline \multirow{2}{*}{Ours} & CS-LLaVA & 1.87\({}^{*}\) & 1.99 & 0.92\({}^{*}\) & 2.15 & 2.24\({}^{*}\) & **1.64** & 2.11\({}^{*}\) & 2.25\({}^{*}\) & 1.68 & 2.54 & 2.61 & 1.71 \\  & CS-LLaVA w/ MH & **2.11** & **2.16** & **1.02** & **2.23** & **2.39** & 1.60 & **2.27** & **2.39** & **1.79** & **2.59** & **2.67** & **1.80** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Stage-wise assessment results as evaluated by GPT-4 at each stage on the M2CoSC testset. **Emp., Coh.**, and **Gui.** represent Empathy, Logical Coherence, and Guidance, respectively. Stars (*) next to values indicate a significant difference compared to CS-LLaVA w/ MH , with a p-value ¡ 0.05 as determined by the paired t-test.

Figure 6: Dialogue-level score assessment ratio for CS-LLaVA w/ MH evaluated by GPT-4



Error analysisExamining the aspect-wise results of our CS-LLaVA w/ MH, we found that the guidance received lower scores than the other aspects (Figure 6). Hence, we further conducted a detailed error analysis, where the primary (six out of nine) cases were the AI therapist's failure to offer forward-looking strategies. Specifically, the AI therapist could not provide future-oriented strategies, hindering its ability to assist clients in preventing similar possible distortions. Detailed examples and additional qualitative analysis are described in Appendix E and F.

### Stage-level Evaluation

In this testbed, each AI therapist responds to the same dialogue history, allowing us to directly compare their interventions. To ensure reliability, we conducted evaluations using both GPT-4 and two human psychotherapists, and the evaluation was carried out at the turn level for each stage. As in the dialogue-level evaluation, we report score assessment and win rate results.

Throughout both assessment results and pairwise comparisons, the M2CoSC dataset had a noticeable impact across all stages. Multi-hop psychometric reasoning outperformed the standard prompting method across most stages; however, in the Exploration stage, the guidance performance was slightly lower. This is likely because this stage emphasizes a deeper exploration of the client's situation rather than offering direct suggestions.

Stage-wise assessment results evaluated by GPT-4 (Table 5) reveal that the score distribution is lower than that of the dialogue-level evaluation, as it assesses intervention at the turn level rather than the entire conversation. One key finding is the substantial performance improvement of the multi-hop psychotherapeutic reasoning process observed in the Introduction stage. The results demonstrate that our attempt to initially detect and incorporate the client's emotions led to more empathetic and coherent interactions. Figure 5 illustrates the stage-wise pairwise comparison results among the four models, as judged by GPT-4. Note that LLaVA, without our method, had the lowest win rate during the Introduction stage. This is attributable to its difficulty in expressing empathy using the client's non-verbal evidence, as it had not yet learned to effectively integrate multimodal information into conversations. The results highlight the importance of teaching models to recognize and utilize such information in counseling phases.

To further strengthen the reliability of the human evaluation results, we derived the win rate by comparing the proposed methodology with other approaches. Specifically, we compared CS-LLaVA with multi-hop psychotherapeutic reasoning to CS-LLaVA with standard prompting and other baselines, as evaluated by two domain experts. All stages aggregated results in Figure 7 exhibit that CS-LLaVA w/ MH achieved the highest wins and significantly fewer losses. Stage-wise results in Table 6 show that CS-LLaVA w/ MH outperforms across all stages. However, in the Exploration stage, CS-LLaVA performed similarly to CS-LLaVA w/ MH, likely due to the nature of the stage, which focuses more on exploring the client's situation than providing suggestions.

Overall results strengthen the essence of integrating multi-hop psychotherapeutic reasoning, particularly in stages where understanding and responding to emotional and cognitive states is critical.

\begin{table}
\begin{tabular}{c l c c c c} \hline \hline  & & \multicolumn{3}{c}{**Win Rate (\%)**} \\  & & **Intro.** & **Explo.** & **Brain.** & **Sugg.** \\ \hline \multirow{2}{*}{Baselines} & LLaMA2 & 90.0 & 75.5 & 68.0 & 90.0 \\  & LLaVA & 97.5 & 98.0 & 85.5 & 70.0 \\ \hline \multirow{2}{*}{Ours} & CS-LLaMA2 & 88.0 & 77.5 & 70.5 & 88.0 \\  & CS-LLaVA & 71.0 & _46.0_ & 61.0 & 57.0 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Stage-wise win rates of CS-LLaVA w/ MH against other models, as evaluated by two psychotherapists at each stage of the M2CoSC testset.

Figure 7: All stages aggregated win rates of CS-LLaVA w/ MH against other models, as evaluated by two psychotherapists on the M2CoSC test set. The domain experts are asked to choose which intervention is better at assessing the given intervention.

 

Conclusion

In this paper, we explore cognitive reframing therapy within a multimodal context. Recognizing the gap between real face-to-face cognitive reframing therapy and prior research, as well as the potential benefits of AI in psychotherapy, we take an initial step toward enhancing the therapeutic capabilities of AI therapists by incorporating non-verbal cues, particularly facial expressions, into the intervention process. To address the challenge of restricted access to real client data in the field of psychology, which hinders research efforts, we synthetically create a novel multimodal conversational cognitive reframing dataset called M2CoSC. Our experiments across two test scenarios, dialogue- and stage-level evaluations, exhibit significant improvements in the therapeutic capabilities of VLMs when using M2CoSC. The proposed multi-hop psychotherapeutic reasoning strategy, which integrates facial expressions, thoughts, and thinking traps, demonstrates superior performance in providing clients with empathetic, logically coherent, and specific rational suggestions.

### Limitations

We expanded the concept of cognitive reframing into multimodality, demonstrating that incorporating multimodal evidence and multi-hop psychometric reasoning significantly enhances the therapist's abilities. However, these results were limited to virtual clients whose facial images and dialogues were consistent. This controlled setting may not fully capture the complexities of real-world interactions.

While we used benchmark images for facial expression recognition, capturing real clients' facial expressions can be challenging and may influence the consultation's content. Moreover, our study only utilized facial images as the source of non-verbal information, which presents a limitation compared to actual face-to-face cognitive reframing therapy. Real-life therapy involves a broader spectrum of non-verbal cues, such as body language, tone of voice, and other contextual factors, which were not accounted for in our research.

Another important consideration is that facial expression recognition can vary across cultural contexts. Different cultures express and interpret emotions in distinct ways, which can affect the accuracy and fairness of emotion recognition models. Such cultural differences may introduce biases in how virtual clients' emotions are understood and addressed, meaning our findings might not generalize well across diverse populations. Future research should work on mitigating these biases and adapting emotion recognition models to better account for cultural diversity.

Moving forward, we plan to expand the modalities to include a wider range of non-verbal information. By incorporating diverse non-verbal cues, we aim to further enhance the model's ability to mimic real-life therapy scenarios. This will help bridge the gap between virtual and actual consultations, ultimately enabling the model to learn how to effectively utilize non-verbal information in a more realistic setting.

### Ethics Statement

This study explores multimodal cognitive reframing therapy constructing a synthetic dataset, M2CoSC. Importantly, no real client data was used in this work; all information in the dataset was generated synthetically.

To construct the dataset, we conducted a thorough data cleansing process with the assistance of three native English speakers, compensating them $0.13 per data entry. For human evaluation, we engaged three professional psychotherapists. One was responsible for evaluating the quality of the M2CoSC test set, while the other two conducted pairwise comparisons. We compensated $0.80 per conversation for the dataset evaluation and $0.0625 per entry for the pairwise comparisons.

Additionally, to adhere to the AffectNet license, images attached in this paper were not sourced from the AffectNet dataset; instead, all images were created using DALL-E 3 (Betker et al., 2023). Furthermore, we obtained consent for all research participants, including annotators and evaluators, to ensure adherence to the license.

To address privacy concerns while complying with the license, we only partially release our M2CoSC dataset, with full access requiring an AffectNet license. Specifically, we provide synthetic dialogues paired with image IDs.

## Acknowledgements

This work was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP)-ITRC (Information Technology Research Center) grant funded by the Korea government (MSIT) (IITP-2025-RS 

2024-00437866, 47.5%) and Smart HealthCare Program funded by the Korean National Police Agency(KNPA) (No. RS-2022-PT000186, 47.5%), and Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2019-II191906, Artificial Intelligence Graduate School Program (POSTECH), 5%).

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Beck (1970) Aaron T. Beck. 1970. Cognitive therapy: Nature and relation to behavior therapy. _Behavior Therapy_, 1(2):184-200.

* Beck (1988) Aaron T. Beck. 1988. Love is never enough.
* Betker et al. (2023) James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023. Improving image generation with better captions. _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_, 2(3):8.
* Chiu et al. (2024) Yu Ying Chiu, Ashish Sharma, Inna Wanyin Lin, and Tim Althoff. 2024. A computational framework for behavioral assessment of llm therapists. _arXiv preprint arXiv:2401.00820_.
* Counseling (2024) Claibourne Counseling. 2024. What are the five stages of counseling? Accessed: 2024-10-08.
* DiTomasso et al. (2000) Robert A DiTomasso, Donna M Martin, and Karel D Kovnat. 2000. Medical patients in crisis.
* Dollinger et al. (2021) Lillian Dollinger, Petri Lauka, Lennart Bjorn Hogman, Tanja Banziger, Irena Makower, Hakan Fischer, and Stephan Hau. 2021. Training emotion recognition accuracy: Results for multimodal expressions and facial micro expressions. _Front. Psychol._, 12:708867.
* Greenberg and Safran (1989) Leslie S Greenberg and Jeremy D Safran. 1989. Emotion in psychotherapy. _American psychologist_, 44(1):19.
* Halamardaris and Power (1997) Kallirroy-Fereniki Halamardaris and KG Power. 1997. Individual differences, dysfunctional attitudes, and social support: A study of the psychosocial adjustment to university life of home students. _Personality and individual differences_, 22(1):93-104.
* Hofmann et al. (2012) Stefan G Hofmann, Anu Asnaani, Imke JV Vonk, Alice T Sawyer, and Angela Fang. 2012. The efficacy of cognitive behavioral therapy: A review of meta-analyses. _Cognitive therapy and research_, 36:427-440.
* Horvath and Luborsky (1993) Adam O Horvath and Lester Luborsky. 1993. The role of the therapeutic alliance in psychotherapy. _Journal of consulting and clinical psychology_, 61(4):561.
* Hsu et al. (2023) Ting-Yao Hsu, Chieh-Yang Huang, Ryan Rossi, Sungchul Kim, C. Giles, and Ting-Hao Huang. 2023. GPT-4 as an effective zero-shot evaluator for scientific figure captions. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 5464-5474, Singapore. Association for Computational Linguistics.
* Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzh Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Hutchison and Gerstein (2012) A. N. Hutchison and L. H. Gerstein. 2012. What's in a face? counseling trainees' ability to read emotions. _Training and Education in Professional Psychology_, 6:100-112.
* Khondaker et al. (2023) Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdullageed. 2023. GPTArEval: A comprehensive evaluation of ChatGPt on Arabic NLP. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 220-247, Singapore. Association for Computational Linguistics.
* Krishnan (2015) Sunil Krishnan. 2015. The counselling process: Stages of the counselling process. Technical report, Slideshare. Retrieved: 2016-07-09.
* Lambert and Barley (2001) Michael J Lambert and Dean E Barley. 2001. Research summary on the therapeutic relationship and psychotherapy outcome. _Psychotherapy: Theory, research, practice, training_, 38(4):357.
* Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chengquang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 2511-2522, Singapore. Association for Computational Linguistics.
* Maddela et al. (2023) Mounica Maddela, Megan Ung, Jing Xu, Andrea Madotto, Heather Foran, and Y-Lan Boureau. 2023. Training models to generate, recognize, and reframe unhelpful thoughts. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13641-13660, Toronto, Canada. Association for Computational Linguistics.
* Mollahosseini et al. (2019) A. Mollahosseini, B. Hasani, and M. H. Mahoor. 2019. Affectnet: A database for facial expression, valence, and arousal computing in the wild. _IEEE Transactions on Affective Computing_, 10(01):18-31.
* Powles (1974) William E. Powles. 1974. Beck, aaron t. depression: Causes and treatment. philadelphia: University of pennsylvania press, 1972. pp. 370. SS4.45. _American Journal of Clinical Hypons_, 16:281-282 

The Psylog. 2024. Five stages of the counseling process. Accessed: 2024-10-08.
* Qu et al. (2023) Renyi Qu, Lyle Ungar, and Joao Sedoc. 2023. Conditioning on dialog acts improves empathy style transfer. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
* Rice and Elliott (1996) Laura N Rice and Robert Elliott. 1996. _Facilitating emotional change: The moment-by-moment process_.
* Sharma et al. (2023) Ashish Sharma, Kevin Rushton, Inna Lin, David Wadden, Khendra Lucas, Adam Miner, Theresa Nguyen, and Tim Althoff. 2023. Cognitive reframing of negative thoughts through human-language model interaction. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 9977-10000, Toronto, Canada. Association for Computational Linguistics.
* Sottana et al. (2023) Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023. Evaluation metrics in the era of GPT-4: Reliably evaluating large language models on sequence to sequence tasks. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 8776-8788, Singapore. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhahiri, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Walen et al. (1992) Susan R Walen, Raymond DiGiuseppe, and Windy Dryden. 1992. _A practitioner's guide to rational-emotive therapy_. Oxford University Press.
* Xiao et al. (2024) Mengxi Xiao, Qianqian Xie, Ziyan Kuang, Zhicheng Liu, Kailai Yang, Min Peng, Weiguang Han, and Jimin Huang. 2024. HealMe: Harnessing cognitive referring in large language models for psychotherapy. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1707-1725, Bangkok, Thailand. Association for Computational Linguistics.
* Yang et al. (2023) Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with large language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6056-6077.
* Yang et al. (2024) Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. Mentallama: Interpretable mental health analysis on social media with large language models. In _Proceedings of the ACM on Web Conference 2024_, WWW '24, page 4489-4500, New York, NY, USA. Association for Computing Machinery.
* Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Ziems et al. (2022) Caleb Ziems, Minzhi Li, Anthony Zhang, and Diyi Yang. 2022. Inducing positive perspectives with text reframing. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3682-3700, Dublin, Ireland. Association for Computational Linguistics.

 

Overall Trait

We introduced the Overall score as a metric to assess the therapist's overall ability, with the calculation method illustrated in Algorithm 1. \(e\), \(c\), \(g\) stand for empathy, logical coherence, and guidance.

```
functiongetOverallScore(_e_, \(c\), _g_) if\(e\leq 1\) or \(c\leq 1\)then return\(0\) endif if\((e\geq 2\)and \(c\geq 2)\)and \(g\leq 1\)then return\(1\) endif if\((e\geq 2\)and \(c\geq 2\)and \(g==2)\)then return\(2\) endif if\(e\geq 2\)and \(c\geq 2\)and \(g==3\)then return\(3\) endif endfunction
```

**Algorithm 1** Overall Score Calculation

### Dialogue-level Pairwise Comparison

In Table 8, we have provided the numeric results as additional data for Figure 4. When comparing the comparison results for each model pair, CS-LLaVA w/ MH showed a win rate exceeding 50%, outperforming all models.

### Stage-level Pairwise Comparison

In Table 9 and Table 10, we have provided the numeric results as additional data for Figure 5. When comparing each model pair, CS-LLaVA w/ MH showed a win rate that surpassed other models, similar to the dialogue-level testbed results.

## Appendix E Case Study

We conducted additional analysis on test cases to compare our approaches with the LLaMA2.

The comparisons are illustrated in three figures: Figures 8, 9, and 10 show the full conversations between LLaMA2, CS-LLaVA, and CS-LLaVA w/ MH with an AI client, respectively. All three conversations were generated using the same source from the M2CoSC test set, where the client exhibits cognitive distortions, specifically overgeneralization.

In the Introduction stage, LLaMA2 primarily offers unconditional consolation, as it lacks the ability to draw on client-specific information. In contrast, both CS-LLaVA and CS-LLaVA w/ MH demonstrate a more targeted empathy by tailoring their facial expression.

When it comes to suggestions, LLaMA2 tends to focus on generic advice, like "having an open conversation with a friend," without utilizing specific cognitive reframing techniques. On the other hand, CS-LLaVA and CS-LLaVA w/ MH encourage the client to consider alternative viewpoints. Additionally, CS-LLaVA w/ MH goes even further by prompting the client to reflect on past instances where they may have made similar cognitive errors.

## Appendix F Error Analysis

Based on the dialogue-level evaluation results (see Figure 6), we analyzed the cases where guidance received lower scores, specifically those rated as I.

 

For instance, in Figure 11, the therapist successfully reframed the client's feelings about oversharing but neglected to suggest how to handle similar situations moving forward.

In such cases, the therapist could have recommended practical strategies, such as discussing personal issues openly with colleagues to gauge their comfort levels. Providing forward-looking strategies is essential for helping clients develop resilience and effective coping mechanisms for recurring issues.

### Inadequate Crisis Management

In situations where clients are experiencing severe emotional distress, our CS-LLaVA w/ MH struggled to respond flexibly to the crisis, such as by recommending immediate crisis intervention. For example, in Figure 12, a client expressed suicidal thoughts, but the therapist failed to suggest professional help or provide resources for dealing with a crisis. This represents a serious limitation in ensuring safe guidance, especially in high-risk situations.

The staged approach we used to enhance logical consistency in the LLM's counseling conversations mandates that the therapist adhere to a predefined role at each session stage. However, this rigidity impedes the model's ability to adapt during crises. Furthermore, this staged conversation assumes that clients are willing to engage openly with the therapist; if clients display resistance or strong defense mechanisms, the model may fail to deliver effective support.

Future research should aim to improve the AI's flexibility in crisis situations while preserving the logical coherence central to its design. Addressing these limitations is crucial for enhancing the AI therapist's capacity to provide meaningful, practical, and safe guidance, especially during critical moments.

 



\begin{table}
\begin{tabular}{l|c c c c|c} \hline \hline  & **LLaMA2** & **LLAVA-7b** & **CS-LLAMA2** & **CS-LLAAV** & **CS-LLAAV w/ MH** & **Win Rate** \\ \hline
* & 54.096 & 27.562 & 22.877 & 34.676 \\
* & 25.710 & 21.676 & 31.179 \\ \hline
* & 47.067 & 64.848 \\
* & **69.677** \\ \hline \end{tabular}
\end{table}
Table 10: Numerical results of stage-level pairwise comparison of four models on the M2CoSC testset, evaluated using GPT-4.

\begin{table}
\begin{tabular}{p{113.8pt}|p{113.8pt}|p{113.8pt}} \hline \hline
**Metric** & **Description** & **Scale** \\ \hline _Client-clarity_ & The client expresses his or her situation clearly in the conversation & 1/0 \\ \hline _Client-role_ & The client adheres to the role of the client throughout the conversation. & 1/0 \\ \hline _Therapist-role_ & The therapist adheres to the role of the therapist in all conversations. & 1/0 \\ \hline _Image-Dialogue Consistency_ & There is no consistency between the client's facial image and the client’s utterances or situation. The client’s facial image is relevant to neither the client’s utterances nor the client’s situation. & 0 \\  & There is acceptable consistency between the client’s facial image and the client’s utterances or situation. & 1 \\  & There is strong consistency between the client’s facial image and the client’s utterances or situation. & 2 \\ \hline \end{tabular}
\end{table}
Table 7: Guideline for data cleansing in M2CoSC. _Client-clarity_, _Client-role_, and _Therapist-role_ are assigned 1 if they match the description, and 0 otherwise.

\begin{table}
\begin{tabular}{l|c c c c|c} \hline \hline  & **LLaMA2** & **LLAVA-7b** & **CS-LLAMA2** & **CS-LLAMA** & **CS-LLAAV w/ MH** & **Win Rate** \\ \hline
* & 52.551 & 49.495 & 9.694 & 9.794 & 30.485 \\
* & 49.485 & 6.566 & 7.071 & 27.481 \\ \hline
* & 10.309 & 7.071 & 29.592 \\
* & 47.959 & 80.357 \\
* & **82.061** \\ \hline \hline \end{tabular}
\end{table}
Table 8: Numerical results of dialogue-level pairwise comparison of five models, evaluated using GPT-4.

\begin{table}
\begin{tabular}{l|c c c c|c} \hline \hline  & **LLaMA2** & **LLAVA-7b** & **CS-LLAMA2** & **CS-LLAMA** & **CS-LLAAV w/ MH** & **Win Rate** \\ \hline
* & 52.551 & 49.495 & 9.694 & 9.794 & 30.485 \\
* & 49.485 & 6.566 & 7.071 & 27.481 \\ \hline
* & 10.309 & 7.071 & 29.592 \\
* & 47.959 & 80.357 \\
* & **82.061** \\ \hline \hline \end{tabular}
\end{table}
Table 9: Stage-wise win rates of four models on the M2CoSC testset, evaluated using GPT-4.


Figure 8: A case study between an AI Client (ChatGPT) and LLaMA2. The red shading highlights the client’s distorted thoughts, while the yellow shading emphasizes the key components of the therapist's intervention.

Figure 9: A case study between an AI Client (ChatGPT) and CS-LLaVA.

 

* [16] A. K. M. C. Davis, and R. J. S. Thornejst. The impact of the \({}^{3}\)He-\({}_{2}\)O reaction on the production of a heavy ion in the nucleus. In _Proceedings of the 7th International Conference on High Energy Physics_, pages 459-480, 2000.
* [17] D. W. Anderson, P. B. Anderson, and G. E. Hinton. The energy loss of the \({}^{3}\)He-\({}_{2}\)O reaction at high energies. _Phys. Rev. Lett._**106**, 106001 (2009).
* [18] T. H. Lee, Y. Li, and L. Zhang. The role of the \({}^{3}\)He-\({}_{2}\)O reaction on the production of a heavy ion in the nucleus. _Nature_**466**, 467-478, 2000.

[MISSING_PAGE_POST]

 

In order to make the first of our attention, we need to know that the second author is interested in a _good_ and _bad_. The third author uses the _bad_ and _bad_. The third author uses the _bad_ and _bad_. The third author uses the _bad_ and _bad_. The fourth author uses the _bad_ and _bad_.

 


#### Overview

The basic task is to build an crowd flat platform cognitive estimating for the specific example and evaluate multiple or models that perform cognitive defining communities. The rationale must be used for research purposes.

This work aims to use Open 10G mobile (location), i.e. resource in R, job check on her site with the same location, if she to contact between a second of the previous stages helps (Each N would proceed according to its private displays history).

An aspect, given these efforts such of the two approaches is nearly (i.e., R, or Tie)

A high description of how can we identify (R, R, or Tie)

Web Description

The communication focuses on CRT, with the therapist working to enhance the clients distributed training.

We will employ 1000 parts of A model responses.

If you set at the Google Downstation, each city has questiona_id, response **A**, and response **B**, respectively.

There is a one off the corresponding to ensemble, (i) a dialog'2 family and (ii) a single-arm, self-add, and the off course the content of the previous conversation on the client before image.

These select the response that best fits us the conversation context and the clients itself expression consumed in the pdf (i.e., **A**, or Tie)

Evaluation traits

Thus to create evaluation trait, but please mainly observe everyday, logical discourse, and intensity.

Figure 14: Instruction for human pairwise comparison.

Figure 13: Instruction for human dataset evaluation.