Response Generation for Cognitive Behavioral
Therapy with Large Language Models:Comparative Study with Socratic Questioning
Kenta Izumi, Hiroki Tanaka, Kazuhiro Shidara, Hiroyoshi Adachi, Daisuke
Kanayama, Takashi Kudo, and Satoshi Nakamura

Abstract Dialogue systems controlled by predefined or rule-based scenarios derived
from counseling techniques, such as cognitive behavioral therapy (CBT), play animportant role in mental health apps. Despite the need for responsible responses, it isconceivable that using the newly emerging LLMs to generate contextually relevantutterances will enhance these apps. In this study, we construct dialogue modules
based on a CBT scenario focused on conventional Socratic questioning using two
kinds of LLMs: a Transformer-based dialogue model further trained with a social
media empathetic counseling dataset, provided by Osaka Prefecture (OsakaED), and
GPT-4, a state-of-the art LLM created by OpenAI. By comparing systems that use
LLM-generated responses with those that do not, we investigate the impact of generated responses on subjective evaluations such as mood change, cognitive change,and dialogue quality (e.g., empathy). As a result, no notable improvements are observed when using the OsakaED model. When using GPT-4, the amount of moodchange, empathy, and other dialogue qualities improve significantly. Results suggestthat GPT-4 possesses a high counseling ability. However, they also indicate that evenwhen using a dialogue model trained with a human counseling dataset, it does notnecessarily yield better outcomes compared to scenario-based dialogues.

1 Introduction
The number of people with mental disorders worldwide is estimated to be around
970 million in 2019, having increased by 25% since 2000 with the growth of theworld population [12]. In particular, anxiety disorders and depression accounted for
Kenta Izumi, Hiroki Tanaka, Kazuhiro Shidara and Satoshi Nakamura
Nara Institute of Science and Technology, Takayama-cho 8916-5, Ikoma-shi, Nara, Japan.
e-mail: {izumi.kenta.if6, hiroki-tan, shidara.kazuhiro.sc5, s-nakamura }@is.naist.jp
Hiroyoshi Adachi, Daisuke Kanayama, and Takashi Kudo
Health and Counseling Center, Osaka University, Yamadaoka 1-1, Suita-shi, Osaka, Japan.

1

2

Kenta Izumi et al.

approximately 60% of the prevalence of mental disorders in 2019 [13]. Under thesecircumstances, qualified psychiatrists and other therapists remain in short supply,
and thus the growing number of patients who are unable to receive the care they needhas become a major concern [21, 22]. On the other hand, with the spread of smartphones, various mobile health apps have been developed, and it’s expected that they
will help to provide immediate care to people who have difficulty accessing needed
healthcare [4, 19]. Among other solutions, dialogue systems such as Wysa [15],which encourages users to build cognitive resilience through empathetic responses
based on counseling techniques such as cognitive behavioral therapy (CBT) [5], areexpected to be useful in the form of mental healthcare apps. Toward this approach,a variety of dialogue systems have been explored so far [29, 1, 26, 3].
A review of the literature up to 2019 reported that most of these systems generate responses based on pre-defined rules, and thus system utterances are commonlydescribed as inadequate, e.g., “conversations are shallow, confusing or too short.”
Accordingly, the adoption of natural language processing technology is expected
to improve the quality of dialogue [2]. Recently, natural language understanding
and generation technology for large language models (LLMs), as exemplified by
the 2022 release of OpenAI’s ChatGPT [23], has developed rapidly, and CBT dialogue systems that use high-performance LLMs to generate responses have also
been researched and developed [8, 18]. Most previous apps had been controlled bypredefined or rule-based scenarios, mainly due to technological limitations. Nevertheless, they had the advantage of providing responsible responses, compared to the
LLM-generated responses that the developers cannot fully control and predict.
To the best of our knowledge, no study has examined how the use of LLMbased response generation changes the effectiveness of CBT and user experiencescompared to the conventionally adopted simple scenario or rule-based systems. We
examined whether the user’s subjective evaluation of the system was improved byreplacing or combining the Socratic questions in a scenario with responses generated by LLMs.

2 Methods
In this study, dialogue scenarios were created with reference to the CBT dialogue
scenarios centered on Socratic questions, considered important in CBT, as used in
Shidara et al. [26]. After the Socratic questions in a scenario were replaced or combined with responses generated by an LLM, we conducted an experiment to investigate the effect on the user’s mood and cognitive change [24, 25], which are indicesfor measuring the effectiveness of CBT.

Response Generation for CBT with LLMs

3

2.1 Dialogue scenarios
In this study, scenarios consisting of 15 system utterances were created for the CBTbased dialogue scenario [26] (Table 1). In CBT, when a certain mood arises, thethoughts that occur automatically, such as assumptions and interpretations of thesituation, are called automatic thoughts. Patients are encouraged to modify these
into more balanced and adaptive thoughts [6]. In the scenario, the system first asks
the user about the situation that is troubling him/her, then asks about the moodthat arises when she/he thinks about it and its intensity, and finally asks about thethoughts that occur at that time and puts the strongest thoughts on the agenda as
automatic thoughts (utterance numbers 1-7). This part can be considered questions
asking about ABC: A) Activating event, B) Beliefs, and C) Consequences, which
is a basic model in CBT [10]. Then, six turns of Socratic questions are asked toencourage the user to objectively view and reconsider his/her automatic thoughts(utterance numbers 8-13). In addition, the user is again asked about the intensityof her/his current mood and the scenario ends with a parting expression (utterance
numbers 14 and 15).
Socratic questioning is a questioning technique that allows the therapist to explore the client’s thoughts and beliefs and to help the client think critically about
them, broaden their perspective, and arrive at a more realistic and adaptive perspective. This technique has been considered an important component of CBT for
depression [30].
In this study, we implemented the following three types of dialogue systems,
where utterances 8-13 of the scenarios were varied according to the system: 1) asking Socratic questions as in the scenario, 2) uttering the responses generated by the
LLMs, and 3) combining LLM-generated responses with Socratic questions. We focus on how each dialogue system can help people rethink their automatic thought,
arrive at a more adaptive and balanced thought, and thus improve their mood after
asking ABC questions (in utterance numbers 1-7). In particular, the dialogue flowof system 2, where the system asks ABC questions and utters generated responses
after that, is roughly similar to that suggested by Mousavi et al. [20]. We expectedthat adding Socratic questions to LLM-generated responses would help the systems
encourage the user to rethink their automatic thought and not deviate from the context of CBT. In each of the latter two systems, which generate responses by LLMs,
the following two LLMs were used to implement the dialogue systems: Osaka empathetic counseling dialogue model (OsakaED) and GPT-4. We chose these two
LLMs because they have different characteristics from each other. In this way, five
dialogue systems were implemented, each with different utterances 8-13 of the dialogue scenario. Socratic questions are represented by SQ and OsakaED, and when
Socratic questions are added to the utterances of each model, they are connected by“+” to form the system name.

4

Kenta Izumi et al.

Table 1 Dialogue scenario for the dialogue system in the SQ condition. Question items and question order are the same in all experiments. Only questions in bold differ from the LLM condition;
the conditions with LLMs use only generated sentences or combinations of a Socratic question anda generated sentence.
Question item System’s response
1 Situation

Let’s address your concerns together. Are you currently facing any issues ordifficulties? If you feel comfortable doing so, please share what’s troubling you.

2 MoodHow did you feel at that time?

Initial mood
3
score
Automatic4
thought
Automatic
5
thought
Automatic
6
thought
Automatic
7
thought
Socratic
8question 1
Socratic
9question 2
Socratic
10question 3
Socratic
11
question 4
Socratic
12question 5
Socratic
13
question 6
Mood score
14
at the end
Closing15
remarkOn a scale of 0 to 100, where 0 is no problem at all and 100 isa huge problem, how intense is that feeling?
What thoughts came to your mind at that time?
I see. Are there any others? Please share anything that comes to mind.
Please tell me the strongest thought among those that came to your mind.
The thoughts that come to mind in response to a situation are called automatic thoughts.
Today we are going to delve into your thought. Is that okay?
If the automatic thought is correct, what is the basis for it?
On the other hand, if your automatic thought is incorrect, what evidence can you think of?
Can you think of any other perspectives regarding the situation that’s troubling you,aside from your current automatic thoughts?
What would be the worst possible outcome in this situation?
What would be the best possible outcome in this situation?
What do you think is the most realistic outcome in this situation?
Now, how intense is the feeling you initially had?
Please rate it again on a scale from 0 to 100.
If your mood has changed, that’s a sign that you’ve successfully organized your thoughts.
That’s it for today. Good job!

Table 2 List of dialogue systems compared in this study.
System name

Description

SQ
OsakaED
OsakaED+SQ

Socratic questions based on the scenario are utteredResponse generated by Osaka SNS empathetic dialogue model are uttered
Socratic questions based on the scenario are added to the responses generated
by Osaka SNS empathetic dialogue model and utteredResponse generated by GPT-4 are uttered
Socratic questions based on the scenario are added to the responses generated
by GPT-4 and uttered

GPT-4
GPT-4+SQOsakaED
OsakaED was developed by further fine-tuning the Japanese Transformer EncoderDecoder dialogue model with a dataset provided by Osaka Prefecture. This model,
created by Sugiyama et al. [28], had already been fine-tuned with empathic dialogue

Response Generation for CBT with LLMs

5

data and possesses 1.6 billion parameters. We chose this model as a base model because it’s trained with text chat conversations via the internet and thought to alreadyhave empathetic response generation ability in a form similar to human speech on
SNS. The SNS counseling dataset consists of 2077 dialogues and 62,066 utterancesof counselors counseling their clients using the messenger application LINE1 . Fromthese, 29,062 pairs of context and counselor utterances were extracted so as not toinclude utterances consisting only of system utterances, which are mainly placed at
the beginning of the dialogues. In training, 90% of the data were training data and
10% were validation data, with a batch size of 4, cross-entropy as the objective function, and Adafactor as the optimizer. Here, 11 epochs were trained with a learning
rate of 1.0 × 10−5 , and the validation perplexity used the model from the 6th epochwith the lowest validation perplexity. The response generation method was basicallythe same with Sugiyama’s implementation [27], that sampled five response candidates by top-p sampling with p = 0.90 and ranked them. In the dialogues in the
pre-experiment, keywords that were included in the generated responses, which attempted to end or guide the dialogues specific to the dataset and were consideredinappropriate, were observed and set as NG words. The highest rank response notcontaining any of NG words was selected as the response. When NG words are included in all response candidates, an error message is output, and in this case, thesample is not treated as valid in the subjective evaluation. In addition, when Socratic questions were added, the response with the highest rank not containing anyof NG words and not ending with a question was selected to avoid duplicating questions. If all response candidates were questions, the response with the highest rank
among the candidates that did not contain any of the NG words was selected as the
response.

GPT-4
GPT-4 is a large-scale language model developed by OpenAI that shows humanlevel performance on tasks across a wide range of domains, including mathematics,
coding, and medicine [7]. In this study, responses were generated by inputting as
much of the dialogue history as possible into GPT-4, including all previous userand system utterances, in addition to the system prompts shown in Table 3. Kumar
et al. explored prompt design for mental health chatbots using GPT-3, focusing on
the three dimensions of identity, intent, and behavior, with two identities and threeintents and behaviors, respectively, for a total of 18 different dialogue systems, andthus 18 prompts were created and evaluated. The system prompts with “coach” as
the identity and “strong interpersonal skills” as the behavior received the highest
overall evaluation among the prompts with intent for CBT [17]. When adding a
Socratic question to the generated response, as shown in Table 3, we have “thecoach does not ask questions.” This sentence was inserted after the first sentence todiscourage the respondent from duplicating the question.
1 https://line.me/6

Kenta Izumi et al.

Table 3 System prompts for GPT-4 and GPT-4+SQ, based on Kumar et al. [17]. The Japanesetranslated prompts were used in the experiment.
SystemSystem prompt
The following is a conversation with a coach.
The coach helps the Human understand how their thoughts, feelings,
and behaviors influence each other.
GPT-4
If the Human demonstrates negative thoughts,
the coach helps the Human replace them with more realistic beliefs.
The coach has strong interpersonal skills.
The following is a conversation with a coach.
The coach does not ask questions.
The coach helps the Human understand how their thoughts, feelings,
GPT-4+SQ and behaviors influence each other.
If the Human demonstrates negative thoughts,
the coach helps the Human replace them with more realistic beliefs.
The coach has strong interpersonal skills.

Example responses from GPT-4 and OsakaED are shown below. Both examples
consist of the system utterances 8 and 9 and the user utterance to 8, in the context
when the user consults about the skin trouble and acne.
GPT-4

User
GPT-4

OsakaED
User
OsakaED

Thank you. Let’s dig deeper into the thoughts that arise
from that “I want to heal fast” feeling. How do you feel thatthought is affecting you? How does it affect your behavior
and daily life?
I check when looking in the mirror and pay attention to theskin and diet.
Your automatic thoughts strongly express your desire to“heal quickly”. This thought prompts you to take action andmake it easier for you to try to live a healthier life. You once
thought, “Why can’t I get well?” but in fact, that thought
may not be in alignment with reality. I would like to propose a new way of looking at it. You can help yourself moreby looking at it from a realistic perspective, as in, “I havecontrol over my skin condition,” and “I can improve it overtime if I take proper care of my skin problems.” What do
you think about this new view?
It’s hard to have skin problems and acne that won’t go away
for a long time...
Yes, it is. It’s hard.
Why is it that skin problems and acne don’t go away all thetime...

As shown in these examples, GPT-4 tends to help the user explore their thoughts
in a longer text, while OsakaED show empathy in a shorter text.

Response Generation for CBT with LLMs

7

2.2 Subjective evaluation
A subjective evaluation experiment was conducted using a crowdsourcing platform
2 , in which 105 crowd participants (healthy participants) were gathered to evaluate our dialogue systems. This experiment was conducted under the review and approval of the research ethics committee of the Nara Institute of Science and Technology (reference number: 2022-I-52). After reading the explanation of the experimentand agreeing to participate in the experiment, the participants were asked to provide their crowdsourcing ID, gender, age, and rating on the Kessler PsychologicalDistress Scale (K6) [14], a widely used measure of psychological distress, to be assigned to each participant at CrowdWorks. They then read a leaflet about CBT written by a psychiatrist3 . The participants reserved a time to perform a dialogue withthe system, started the dialogue at the reserved time, and ended the dialogue after
confirming the last utterance of the system (utterance number 15). The participants
completed the experiment by answering a questionnaire about cognitive change anddialogue quality, with space available for free-text description. Throughout the experiment, participants were not informed of the type of dialogue system with which
they interacted.
In CBT, the goal is to improve mood by modifying automatic thoughts into adaptive thoughts. In this experiment, following the work of Persons et al. [24], the moodchange was calculated by an equation 1 and then used as an index to evaluate the degree to which mood improved. Here, initial mood score and mood score at end represent the intensities of mood elicited by system utterances 3 and 14 in the scenario,
respectively:
Mood change = −(mood score at end) − (initial mood score)(initial mood score)(1)

The Cognitive Change immediate scale (CC-immediate) [25] for measuring immediate cognitive change with single-session CBT was translated into Japaneseand used to assess cognitive change due to interaction with the system. The CCimmediate consists of six questions, each rated on a 7-point Likert scale with 0
being “not at all” and 6 being “completely,” and the responses to all questions weresummed to form a 31-point scale from 0 to 30 to evaluate the immediate change incognition.
Additionally, a questionnaire consisting of 15 items was used to evaluate thequality of dialogue at the end of the conversation, which has two additional questions to that created by Sugiyama et al. [28], based on a survey for dialogue between
dialogue systems and users conducted by Fitrianie et al. [11]. The Sugiyama et al.
study originally included an item called “empathy,” but because this item asked
whether the user could empathize with the system’s utterances, “I could empathizewith the system’s utterances,” it was changed to an item called user’s empathy, and a
new item called “system’s empathy” was added along with the question “the system
2 https://crowdworks.jp/
3 https://www.cbtjp.net/downloads/skillup/

8

Kenta Izumi et al.

showed empathy” [28]. This is because assessing the system’s empathetic ability has
been suggested as an effective way of soothing the user’s mood and, more specifically, improving the outcome [9, 16]. The item called “initiative” with the question“I was able to lead the dialogue” was also added. It’s pointed out to be lost in scenarios and rule-based dialogues in a study by Abd-Arazaq et al., and we assess this toconfirm that whether this quality is improved by using LLMs and its importance for
CBT. Continued usability is measured by the item “re-speak,” which is measured by
the question “I would like to talk to this system again.” Each item was rated on an
11-point Likert scale, with 0 representing “completely disagree” and 10 representing “completely agree.” Mann-Whitney’s U-test was used to assess the difference inscales between the two systems according to the particular interests of comparison.

3 Results
The experiment resulted in a total of 86 valid responses. The number of users foreach system was 24 for SQ, 14 for OsakaED, 13 for OsakaED+SQ, 17 for GPT-4,
and 18 for GPT-4+SQ. The mean and standard deviation of K6 was 7.60 and 5.59,
and no significant differences were seen in any two groups’ comparisons (p > 0.27
in all comparisons). The results of subjective evaluations are summarized in Table
4.
Table 4 Subjective evaluations of each system (mean and standard deviation). Bold font represents
the highest value in each scale.
SQ
mood change
0.28 (0.24)
CC-immediate
16.6 (7.07)
naturalness
6.25 (2.97)
ease
7.25 (2.45)
enjoyment
5.13 (2.98)
user’s empathy
5.17 (2.62)
attentiveness5.79 (2.95)
trust
5.63 (2.65)
personality
3.96 (3.04)
agency4.29 (3.09)
topic
5.54 (3.20)
emotion3.42 (2.93)
consistency
8.04 (1.68)
engagement
5.63 (3.21)
re-speak
5.50 (3.12)
system’s empathy 5.42 (2.62)
initiative
4.21 (2.32)

OsakaED

OsakaED+SQ

GPT-4

GPT-4+SQ

0.22 (0.16)
14.4 (5.83)
5.93 (2.56)
7.00 (2.94)
5.36 (2.92)
5.07 (2.62)
6.14 (3.01)
5.43 (2.90)
4.79 (2.91)
4.57 (2.82)
4.21 (2.12)
4.29 (2.52)
5.79 (2.83)
5.43 (2.56)
5.93 (2.81)
6.93 (2.30)
5.79 (1.85)

0.19 (0.40)
16.9 (4.65)
6.54 (1.61)
8.31 (1.55)
6.23 (2.24)
5.85 (1.72)
7.15 (2.15)
5.77 (1.92)
5.00 (2.45)
4.85 (2.64)
5.54 (2.60)
4.00 (2.38)
6.77 (1.59)
5.46 (2.50)
6.15 (2.44)
6.85 (2.70)
5.46 (2.26)

0.44 (0.26)
19.8 (4.93)
7.76 (2.19)
8.06 (1.43)
6.53 (2.70)
7.76 (2.14)
7.59 (2.29)
7.76 (2.19)
6.65 (2.94)
7.18 (2.38)
5.94 (2.84)
6.41 (2.60)
8.47 (1.42)
7.06 (2.44)
6.65 (2.74)
7.94 (2.14)
5.53 (2.60)

0.49 (0.14)
18.8 (5.55)
6.56 (2.89)
6.83 (3.13)
6.39 (2.68)
7.06 (2.10)
7.56 (1.46)
7.44 (1.82)
6.33 (2.33)
6.22 (2.34)
5.72 (2.49)
5.50 (2.71)
8.00 (2.00)
6.39 (2.33)
6.50 (2.98)
7.44 (2.31)
4.94 (2.96)

To investigate the effect of replacing SQ with generated responses from LLMs,
the differences between SQ and OsakaED and between SQ and GPT-4 were assessed. In the comparison between SQ and OsakaED, there were significant differ-

Response Generation for CBT with LLMs

9ences in consistency and initiative: Consistency decreased (p < 0.05), while initiative increased (p < 0.05). In the comparison between SQ and GPT-4, there were significant differences in mood change and attentiveness (p < 0.05), as well as in user
empathy, trust, personality, agency, emotion, and the system’s empathy (p < 0.01).
In scenarios where SQ was replaced with generated responses from OsakaED,there was only one improvement in initiative. Considering that OsakaED performed
better in many of the items, including consistency and user’s empathy, than with thereported result of the base model (model trained with EmpatheticDialogue dataset)in the previous work [28], it’s surprising that simply asking Socratic questions in six
turns worked competitively. The increase in initiative suggests that LLM responses
may prompt users to take more control or be more proactive in the interaction. Conversely, the decrease in consistency indicates that OsakaED-generated responses
may be more variable or less predictable than SQ.
The significant differences observed when SQ was replaced with GPT-4 responses in terms of mood change and attentiveness, as well as a range of user experience dimensions, emphasize the sophisticated capabilities of GPT-4 in affectinguser perception, even though the provided prompt is not optimized for this very
well. The positive changes in mood, user and system empathy, and emotion are particularly noteworthy, since they suggest that interactions with GPT-4 can lead to an
emotionally satisfying user experience. However, in many of these improvements,
re-speak is not significantly improved (p = 0.23). For long-time use, this dimension
is believed to be important, and thus further consideration should be given to it.
Additionally, despite the prompt being designed to discourage duplication of questioning for GPT-4+SQ, it was observed that responses generated by both GPT-4 andGPT-4+SQ frequently involved Socratic Questions. An annotator assigned to analyze these responses reported that they consistently involved Socratic Questions inall dialogues. This indicates that GPT-4 not only understands Socratic Questions but
also has the capability to appropriately ask them in context.

3.1 Comparison between SQ and generated responses with SQ
To investigate the effect of adding generated responses from LLMs to SQ, the differences between SQ alone and SQ combined with OsakaED (OsakaED+SQ), as
well as between SQ alone and SQ combined with GPT-4 (GPT-4+SQ), were assessed. In the comparison between SQ and OsakaED+SQ, consistency was foundto decrease (p < 0.05). In the comparison between SQ and GPT-4+SQ, there weresignificant differences in user empathy, trust, agency, and emotion (p¡0.05), as wellas in mood, personality, and the system’s empathy (p < 0.01). When SQ was added
to LLM-generated responses, the decrease in consistency remained notable in thecase of OsakaED+SQ, suggesting that the addition of SQ does not necessarily mitigate the variability introduced by LLM responses. The findings from GPT-4+SQ
comparisons reinforce the idea that GPT-4’s responses significantly influence user
experience, potentially overshadowing the impact of the added SQ.

10Kenta Izumi et al.

3.2 Comparison between generated responses and generated
responses with SQ
To investigate the effect of adding SQ to generated responses from LLMs, the differences between OsakaED and OsakaED+SQ, as well as between GPT-4 and GPT4+SQ, were assessed. However, no significant differences were found at any point.
Surprisingly, contrary to our expectations, the addition of SQ to LLM-generated responses did not yield significant differences in any assessed dimension. This could
indicate that the presence of SQ does not substantially alter the user’s experience
once they are engaged in LLM-generated content. This might be due to the LLMs’
abilities to generate responses that are sufficiently informative or engaging, making
the inclusion of SQ redundant. Particularly in the comparison between GPT-4 andGPT-4+SQ, it’s noteworthy that both were observed to generate Socratic Questions.
This observation suggests that the combination with SQ becomes more redundant,
as GPT-4 alone already appears capable of producing such responses.

4 Discussion
In this study, we constructed dialogue systems based on a CBT scenario focusedon Socratic questioning using two kinds of LLMs. By comparing systems that used
LLM-generated responses with those that did not, we investigated the impact of thegenerated responses. The results suggest that GPT-4 is highly capable for counseling, although it does not necessarily improve outcomes compared to scenario-based
dialogue, even though the dialogue model used to generate responses was trainedwith a real counseling dataset. In particular, it’s notable that the outcome and manyof the dialogue qualities were often significantly improved when using GPT-4 generated responses, even though the user had just a six-turn interaction and the provided
prompt for this was not so highly optimized. This may provide many implications
for the field in analyzing what features of the GPT-4 generated responses achievethese improvements in mood and response qualities. Moreover, it was suggested thatunless responses generated using a sufficiently robust LLM such as GPT-4 are used,
it may not be possible to significantly improve dialogue quality or effectiveness as
CBT compared to a scenario-based or simple rule-based dialogue. When deploying mental healthcare systems using LLMs and conducting large-scale experiments,
it would be better for researchers to think about using GPT-4 or an equivalently
powerful LLM to generate responses first or examine whether a given system could
improve the outcome significantly, compared with scenario or rule-based systems.
This is because, considering system safety and responsibility, if those systems using
LLMs cannot perform significantly better than the scenario or rule-based systems,
there may be no practical reason to use LLMs for this purpose.
As a limitation, the experiment in this study is a small-scale one with a limited
number of crowdsourced subjects. In particular, CBT is a long-term, multi-session

Response Generation for CBT with LLMs

11

program, and if its effectiveness as CBT is to be accurately verified, it is necessary to conduct a long-term randomized controlled trial with a larger sample size
to compare its effectiveness. In the future, we will focus on counseling by GPT-4,
investigate optimally effective prompts for CBT, and analyze the needed counselingskills. Furthermore, we will verify the achieved counseling skills and the prompts
used for CBT.
Acknowledgements This research is supported by CREST (grant number: JPMJCR19A5). The
OsakaED dataset was provided by thePublic Health and Medical Administration Office, Department of Public Health and Medical Affairs, Osaka Prefectural Government, and the Kansai Counseling Center.

References
1. Alaa A Abd-Alrazaq, Mohannad Alajlani, Ali Abdallah Alalwan, Bridgette M Bewick, Peter
Gardner, and Mowafa Househ. An overview of the features of chatbots in mental health: A
scoping review. International Journal of Medical Informatics, 132:103978, 2019.
2. Alaa A Abd-Alrazaq, Mohannad Alajlani, Nashva Ali, Kerstin Denecke, Bridgette M Bewick,and Mowafa Househ. Perceptions and opinions of patients about mental health chatbots:scoping review. Journal of medical Internet research, 23(1):e17828, 2021.
3. Takuto Akiyoshi, Junya Nakanishi, Hiroshi Ishiguro, Hidenobu Sumioka, and Masahiro Shiomi. A robot that encourages self-disclosure to reduce anger mood. IEEE Robotics andAutomation Letters, 6(4):7925–7932, 2021.
4. E Anthes. Mental health: There’s an app for that. Nature, 532:20–23, 2016.
5. Aaron T Beck. Cognitive therapy of depression. Guilford press, 1979.
6. Judith S Beck. Cognitive behavior therapy: Basics and beyond. Guilford Publications, 2020.
7. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece
Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general
intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
8. Morena Danieli, Tommaso Ciulli, Seyed Mahed Mousavi, Giorgia Silvestri, Simone Barbato,
Lorenzo Di Natale, and Giuseppe Riccardi. Assessing the impact of conversational artificial
intelligence in the treatment of stress and anxiety in aging adults: Randomized controlled trial.
JMIR Ment Health, 9(9):e38067, Sep 2022.
9. Mauro De Gennaro, Eva G Krumhuber, and Gale Lucas. Effectiveness of an empathic chatbot
in combating adverse effects of social exclusion on mood. Frontiers in psychology, 10:3061,
2020.
10. Albert Ellis. The revised abc’s of rational-emotive therapy (ret). Journal of rational-emotive
and cognitive-behavior therapy, 9(3):139–172, 1991.
11. Siska Fitrianie, Merijn Bruijnes, Deborah Richards, Andrea Bönsch, and Willem-Paul
Brinkman. The 19 unifying questionnaire constructs of artificial social agents: An iva community analysis. IVA ’20, New York, NY, USA, 2020. Association for Computing Machinery.
12. Institute for Health Metrics and Evaluation. Gbd results tool. http://ghdx.healthdata.org/
gbd-results-tool?params=gbd-api-2019-permalink/cb9c37d9454c80df77adaed394d7fc0f, accessed 14 October 2023.
13. Institute for Health Metrics and Evaluation. Gbd results tool. http://ghdx.healthdata.org/
gbd-results-tool?params=gbd-api-2019-permalink/5066348dc958b095cb6ceb4bfd9c3e07,
accessed 14 October 2023.
14. Toshiaki A Furukawa, Ronald C Kessler, Tim Slade, and Gavin Andrews. The performanceof the k6 and k10 screening scales for psychological distress in the australian national surveyof mental health and well-being. Psychological medicine, 33(2):357–362, 2003.

12

Kenta Izumi et al.

15. Becky Inkster, Shubhankar Sarda, and Vinod Subramanian. An empathy-driven, conversational artificial intelligence agent (wysa) for digital mental well-being: real-world data evaluation mixed-methods study. JMIR mHealth and uHealth, 6(11):e12106, 2018.
16. Mercer SW. Jani BD, Blane DN. The role of empathy in therapy and the physician-patient
relationship. Forsch Komplementmed, 19(5).
17. Harsh Kumar, Ilya Musabirov, Jiakai Shi, Adele Lauzon, Kwan Kiu Choy, Ofek Gross, Dana
Kulzhabayeva, and Joseph Jay Williams. Exploring the design of prompts for applying
gpt-3 based chatbots: A mental wellbeing case study on mechanical turk. arXiv preprint
arXiv:2209.11344, 2022.
18. June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. Chatcounselor: Alarge language models for mental health support. arXiv preprint arXiv:2309.15461, 2023.
19. López-Coronado M Martı́nez-Pérez B, de la Torre-Dı́ez I. Mobile health applications for themost prevalent conditions by the world health organization: Review and analysis. Journal of
medical Internet research, 15(6):e120, 2013.
20. Seyed Mahed Mousavi, Alessandra Cervone, Morena Danieli, and Giuseppe Riccardi. Would
you like to tell me more? generating a corpus of psychotherapy dialogues. In Proceedingsof the Second Workshop on Natural Language Processing for Medical Conversations, pages
1–9, Online, June 2021. Association for Computational Linguistics.
21. Christopher JL Murray, Theo Vos, Rafael Lozano, Mohsen Naghavi, Abraham D Flaxman,
Catherine Michaud, Majid Ezzati, Kenji Shibuya, Joshua A Salomon, Safa Abdalla, et al.
Disability-adjusted life years (dalys) for 291 diseases and injuries in 21 regions, 1990–2010: asystematic analysis for the global burden of disease study 2010. The lancet, 380(9859):2197–
2223, 2012.
22. Gureje O Oladeji BD. Brain drain: a challenge to global mental health. BJPsych International,
13(3):61–63, 2016.
23. OpenAI. Introducing chatgpt, 2022. https://openai.com/blog/chatgpt, accessed 15 October
2023.
24. Jacqueline B Persons and David D Burns. Mechanisms of action of cognitive therapy: The
relative contributions of technical and interpersonal interventions. Cognitive therapy and Research, 9:539–551, 1985.
25. Iony D Schmidt, Benjamin J Pfeifer, and Daniel R Strunk. Putting the “cognitive” back in cognitive therapy: Sustained cognitive change as a mediator of in-session insights and depressive
symptom improvement. Journal of Consulting and Clinical Psychology, 87(5):446, 2019.
26. Kazuhiro Shidara, Hiroki Tanaka, Hiroyoshi Adachi, Daisuke Kanayama, Yukako Sakagami,
Takashi Kudo, and Satoshi Nakamura. Automatic thoughts and facial expressions in cognitive
restructuring with virtual agents. Frontiers in Computer Science, 4:762424, 2022.
27. Hiroaki Sugiyama, Masahiro Mizukami, Tsunehiro Arimoto, Hiromi Narimatsu, Yuya Chiba,
Hideharu Nakajima, and Toyomi Meguro. Empirical analysis of training strategies oftransformer-based japanese chit-chat systems, 2021.
28. Hiroaki Sugiyama, Masahiro Mizukami, Tsunehiro Arimoto, Hiromi Narimatsu, Yuya Chiba,
Hideharu Nakajima, and Toyomi Meguro. Empirical analysis of training strategies oftransformer-based japanese chit-chat systems. In 2022 IEEE Spoken Language Technology
Workshop (SLT), pages 685–691. IEEE, 2023.
29. Aditya Nrusimha Vaidyam, Hannah Wisniewski, John David Halamka, Matcheri S Kashavan,
and John Blake Torous. Chatbots and conversational agents in mental health: a review of thepsychiatric landscape. The Canadian Journal of Psychiatry, 64(7):456–464, 2019.
30. Lisa N Vittorio, Samuel T Murphy, Justin D Braun, and Daniel R Strunk. Using socratic
questioning to promote cognitive change and achieve depressive symptom reduction: evidenceof cognitive change as a mediator. Behaviour research and therapy, 150:104035, 2022.