Relationship between Mood Improvement and Questioning
to Evaluate Automatic Thoughts in Cognitive Restructuringwith a Virtual AgentK a z u h i r o S h i d a r a 1 , H i r o k i Ta n a k a 1 , H i r o y o s h i A d a c h i 2 ,D a i s u k e Ka n a y a m a 2 , Yu k a k o S a k a g a m i 2 , Ta k a s h i K u d o 2 ,Satoshi Nakamura 1
1 N a r a I n s t i t u t e o f S c i e n c e a n d Te c h n o l o g y ( N A I S T ) , J A PA N
2 O s a k a U n i v e r s i t y, J A PA N

Tapas Supported by, ANR-Crest

Cognitive Restructuring [Beck et al., 2011]n A mental health care method to ﬁnd helpful thoughts instead of unhelpful thoughts
n Well-established as a treatment for mental illness including depression
Composition of Cognitive Restructuring
Item

Description

Situation

No reply from my friend yet.

MoodAnxiety (80%)(Mood score: 0 to 100%)

Identifing an automatic thought He hates me.
(e.g.) I haven't received a reply for two days since I

Evaluating theMood Change

sent messages.

automatic thought(e.g.) He is getting busy these days.

Considering a balanced thought
Mood score after change(0 to 100%)2021/10/7

Automatic thought:
distorted and unhelpful
thought that automaticallycrossing the mind

He have overlooked my message or he is too busy tohandle it.

•Anxiety (80% → 30％)
Tapas Supported by, ANR-Crest

2 / 12

Judith, S, B., & Aaron, T, B. ``Cognitive
behavior therapy: Basics and beyond.”
Guilford Publication, 2011.

Related workn Virtual agents have been anticipated to promote Cognitive Restructuringn However, existing systems canʼt adaptively select questions for individual thought tendenciesl•••[Kathleen et al., 2017]l[Kimani et al., 2019]l[Shidara et al., 2020]•

Text-based dialogue•

Face-to-face dialogue•

Face-to-face dialogue•

Enter choices and text•

Enter choices•

Neural language input

Fitzpatrick, K. K., Alison, D., & Molly, V. ``Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent(Woebot): a randomized controlled trial." JMIR mental health 4.2, 2017: e19.
Kimani, E., Bickmore, T., Trinh, H., & Pedrelli, P. ``You'll be Great: Virtual Agent-based Cognitive Restructuring to Reduce Public Speaking Anxiety.” In IEEE ACII, 2019, (pp. 641-647).
Shidara, K., Tanaka, H., Adachi, H., Kanayama, D., Sakagami, Y., Kudo, T., & Nakamura, S. ``Analysis of Mood Changes and Facial Expressions during Cognitive Behavior Therapy througha Virtual Agent.” In ICMI, 2020, (pp. 477-481).

2021/10/7

Tapas Supported by, ANR-Crest

3 / 12

Objectiven Objective: Implement a system that enables dialogue with a higher mood scoren Research question: Which items of the dialogue help to improve mood?
Composition of Cognitive Restructuring
Item
Situation
Mood(Mood score: 0 to 100%)

Description
No reply from my friend yet.
Anxiety (80%)

Identifing an automatic thought He hates me.
Evaluating theMood Change

automatic thought
Considering a balanced thought
Mood score after change(0 to 100%)2021/10/7(e.g.) I haven't received a reply for two days since I
sent messages.
(e.g.) He is getting busy these days.
He have overlooked my message or he is too busy tohandle it.
Anxiety (80% → 30％)
Tapas Supported by, ANR-Crest

4 / 12

Hypothesis: this part
directly affectmood changes.

Approachn Compare two scenarios in a user survey
Which scenario makeGroup A scenario
Item
Situation
Mood(Mood score: 0 to 100%)

Mood Change larger?
Item

Description

Situation

No reply from my friend yet.

MoodAnxiety (80%)(Mood score: 0 to 100%)

automatic thought
Considering a balanced thought
Mood score after change(0 to 100%)2021/10/7

Description
No reply from my friend yet.
Anxiety (80%)

Identifing an automatic thought He hates me.

Identifing an automatic thought He hates me.
Evaluating theGroup B scenario(e.g.) I haven't received a reply for two days since I
sent messages.
(e.g.) He is getting busy these days.
He have overlooked my message or he is too busy tohandle it.
Anxiety (80% → 30％)

Considering a balanced thought
Mood score after change(0 to 100%)

Tapas Supported by, ANR-Crest

He have overlooked my message or he is too busy tohandle it.
Anxiety (80% → 30％)

5 / 12

Question to evaluate an automatic thoughtn Use existing 7 questions referenced from [Beck et al., 2011]

1
2

support this?
If your automatic thought is not true, what evidence do you have tosupport this?

3

Besides your automatic thoughts, is there an alternative explanation?

4

What's the effect of your believing automatic thought?

5

So, what's the most realistic outcome?

67
2021/10/7

If your automatic thought is true, what evidence do you have toIf your friend was in the situation and had this thought, what would
you tell him or her?
What action would you do about your automatic thoughts?
Tapas Supported by, ANR-Crest

6 / 12

Virtual agent design
n Virtual agent platform: Greta [Niewiadomski et al., 2009]l Speech natural language interfacel Turn-taking is based on the system's speech recognitionl Virtual agent's parameters of facial actions, speech, and poses were default settings

GretaInteraction

Niewiadomski Radoslaw, et al. "Greta: an interactive expressive ECA system." Proceedings of The 8th International Conference on Autonomous Agents and Multiagent
Systems-Volume 2. 2009.

2021/10/7

Tapas Supported by, ANR-Crest

7 / 12

Experimentn Participants (20 in total)l 10 people each in Group A and Group B
l All participants are graduated college students at NAISTl Screening mental health state: K6 [Kessler, 2002](Scores range from 0 to 24, with 13 or more likely to be mentally illness)- Mean K6 score of Group A was 3.90, and Group B was 4.30

n Experiment procedure
Written consent,screening mental
health state2021/10/7

Reading a leaflet
about CognitiveRestructuring

User-agent
interaction

Tapas Supported by, ANR-CrestQuestionnaire(Only Group A)

8 / 12

Analysis –Mood Change between two groupsn Compare mean Mood Change between Group A and Group B

Group A scenario
Item
Situation
Mood(Mood score: 0 to 100%)

Group B scenario

Description

Item

No reply from my friend yet.

Situation
MoodAnxiety (80%)(Mood score: 0 to 100%)

Identifing an automatic thought He hates me.
Evaluating theautomatic thought
Considering a balanced thought
Mood score after change(0 to 100%)2021/10/7

Description
No reply from my friend yet.
Anxiety (80%)

Identifing an automatic thought He hates me.

(e.g.) I haven't received a reply for two days since I
sent messages.
(e.g.) He is getting busy these days.
He have overlooked my message or he is too busy tohandle it.
Anxiety (80% → 30％)

Considering a balanced thought
Mood score after change(0 to 100%)

Tapas Supported by, ANR-Crest

He have overlooked my message or he is too busy tohandle it.
Anxiety (80% → 30％)9 / 12

Analysis –Amount of helpful question and Mood Change
Questionnaire (sample)
Questionn Only Group A was targeted

Helpfulness

What is the strongest thought that comes to your mind?
(Question to identify an Automatic thought)

n After the interaction

If your automatic thought is true, what evidence do youl Participants of Group A answeredeach question was helpful or not
n Correlation analysis

have to support this?

○If your automatic thought is not true, what evidence doyou have to support this?
Besides your automatic thoughts, is there an alternativeexplanation?

l Spearmanʼs rank correlation test

What's the effect of your believing automatic thought?

○

So, what's the most realistic outcome?
If your friend was in the situation and had this thought,
what would you tell him or her?
What action would you do about your automatic thoughts?

○

If you think of a new thought, try to connect them to your○automatic thought with "but...".
(Question to consider a balanced thought)
2021/10/7

Tapas Supported by, ANR-Crest

10 / 12Resultsn Mood change comparison=> Group A was significantly
changed than Group B

Boxplot of Mood Change of Group A and B(Mean of Group A = 0.41, Mean of Group B = 0.13, p=0.037)
2021/10/7

n Correlation analysis=> Significantly correlated

Scatterplot of the questions and Mood Score(N=10, ρ=0.81, p=0.005)

Tapas Supported by, ANR-Crest

11 / 12

Discussionn Asking questions signiﬁcantly improves mood compared to not
n Number of helpful questions signiﬁcantly correlated with Mood Change=> Evaluation of an automatic thought directly aﬀect Mood Change=> The more amount of helpful questions, the greater improvement achieved
n Future work
lBehavior analysis of recorded videos

lImplementation of the personalized question selection system

Thank you for your attention!
2021/10/7

Tapas Supported by, ANR-Crest

12 / 12

Appendix

2021/10/7

Tapas Supported by, ANR-Crest

13 / 12

Cognitive Restructuring with a virtual agent
n Virtual agentl Automatic dialogue is possiblel Dialogue interfaces embodied with CG or animationn Advantage in mental health carel Can be used anytime, anywherel Reduce mental barrier to mental health carel Response selection using facial expression andvoice characteristics of a participant is possible

2021/10/7

Tapas Supported by, ANR-Crest

14 / 12Results –Evaluation of each question
Questionnaire (sample)
Questionn Most selected questions: Q12, Q13
Q6
Q7
Q8
Q9

Helpfulness

What is the strongest thought that comes to your mind?
(Question to identify an Automatic thought)
If your automatic thought is true, what evidence do you
have to support this?

○If your automatic thought is not true, what evidence doyou have to support this?
Besides your automatic thoughts, is there an alternative
explanation?

Q10 What's the effect of your believing automatic thought?
Q11 So, what's the most realistic outcome?
Q12 If your friend was in the situation and had this thought,○

Q13 What action would you do about your automatic thoughts?

○

Q14 automatic thought with "but...".

○

what would you tell him or her?

If you think of a new thought, try to connect them to your(Question to consider a balanced thought)

2021/10/7

Tapas Supported by, ANR-Crest

15 / 12