Social skills training
with conversational agents
Satoshi Nakamura,with Hiroki Tanaka,
Kazuhiro Shidara, Kana Miyamoto, Takeshi Saga, and Kota Iwauchi
Nara Institute of Science and Technology (NAIST), Japan

1

NAIST, ANR-CREST

2023/1/16

Communication Disabilities⚫ Communication Disabilities⚫Daily conversation⚫Meetings⚫Interviews⚫Presentation

2

SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

Various Reasons⚫Mental Disorder⚫Anxiety disorder⚫Depression⚫Autism Spectrum Disorder (ASD)⚫Schizophrenia (SZ)
⚫Dementia

3

SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

Social Anxiety Disorder⚫Social Anxiety Disorder⚫Social anxiety can have an impact on socialskills in a variety of ways.
⚫People with a social anxiety disorder (SAD) areless likely to engage in social interactions,giving them less opportunity to build skills andgain confidence.
⚫SAD can also have a direct impact on social
behavior regardless of skill level.

4

SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

Social Anxiety Disorder: More than just Shyness⚫Blush, sweat, or tremble.
⚫Have a rapid heart rate.
⚫Feel their “mind going blank,” or feel sick to their stomach.
⚫Have a rigid body posture, or speak with an overly soft voice.
⚫Find it difficult to make eye contact, be around people they
don’t know, or talk to people in social situations, even when
they want to.
⚫Feel self-consciousness or fear that people will judge themnegatively.
⚫Avoid places where there are other people.
⚫ → Psychotherapy: Cognitive Behavior Therapy, Social Skills
Training, Medication
5SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

Various Reasons⚫Mental Disorder⚫Anxiety disorder⚫Depression⚫Autism Spectrum Disorder (ASD)⚫Schizophrenia (SZ)
⚫Dementia

6SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

CDC: Autism Spectrum Disorder (ASD)⚫Autism spectrum disorder (ASD) is a developmental disability caused by differences in the brain.
⚫People with ASD may behave, communicate, interact, and learn in ways that are different from mostother people.
⚫The abilities of people with ASD can vary significantly. Some people with ASD need a lot of help intheir daily lives; others can work and live with little to no support.
⚫As children with ASD become adolescents and young adults, they may have difficulties developing andmaintaining friendships, communicating with peers and adults, or understanding what behaviors are
expected in school or on the job.
⚫They may come to the attention of healthcare providers because they also have conditions such as
anxiety, depression, or attention-deficit/hyperactivity disorder, which occur more often in people with
ASD than in people without ASD.
⚫→ Social Skills Training (SST)7SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

ASD population

8SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

CDC: Schizophrenia⚫Schizophrenia is a serious mental illness that affects how a person thinks, feels, and behaves.
⚫People with schizophrenia may seem like they have lost touch with reality, which can be distressing forthem and for their family and friends.
⚫The symptoms of schizophrenia can make it difficult to participate in usual, everyday activities, buteffective treatments are available.

⚫Many people who receive treatment can engage in school or work, achieve independence, and enjoy
personal relationships.

⚫Psychotherapy: Social Skills Training

9SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

Social Skills Training⚫Social Skills Training by Alan Bellack*⚫Social skills training (SST) is a type of behavioral therapy for people with mental
disorders or developmental disabilities.
⚫SST may be used by psychiatrist, teachers, therapists, or other professionals*Bellack, A. S.: Social skills training for schizophrenia: A step-by-step guide, Guilford Press (2004)
https://www.verywellmind.com/social-skills-415721610SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

Social Skills Training⚫SST Techniques⚫Behavioral rehearsal: Role play which involves practicing new skills
during therapy in simulated situations⚫Corrective feedback: Used to help improve social skills during
practice⚫Instruction: The educational component of SST that involves themodeling of appropriate social behaviors⚫Positive reinforcement: used to reward improvements in social skills⚫Weekly homework assignments: Provide the chance to practice newsocial skills outside of therapy

11SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

Training Adapted Personalized Affective Social Skills
with Cultural Virtual Agents (TAPAS)
Problem: Social anxiety in schools and workplace
Approach: verbal/non-verbal interactive training system by
embodied conversational agent (ECA)◦ Target：General populations, Social Anxiety Disorder (SAD)
Autism Spectrum Disorder (ASD), Schizophrenia (SZ)
◦ Behavioral training：
Social skills training（SST）◦ Cognitive training：
Cognitive Behavioral Therapy（CBT）

Public speech: Feelnervous and stress forattentions by others, andbad reputation↓Face red, sweat, shaking,pulsation, breathing,stomachache
Fear for the symptom
Social anxiety, avoidanceSocial anxiety disorder

Training Adapted Personalized Affective Social Skills with Cultural Virtual Agents: ANR-CREST Project
12

NAIST, ANR-CREST

2023/1/16

Nervous,shamefulVirtual agents and robots for social skills training (SST)

Robins+1998

Better communication
with Virtual agents!

Hoque+2015

Kumazaki+2018
13

Nijman+2020
NAIST, ANR-CREST

2023/1/16

TAPAS project (2019-2024)
Nakamura G NAIST
Satoshi Nakamura，Hiroki Tanaka, Seitaro
ShinagawaHirokazu Kato，Yuichiro Fujimoto

Scenario and Situation
Children, Adults with
ASD and Social Anxiety

LISN + Nakamura G
Okazaki G + Kudo G(Image: the verge)

Theories of social interaction
one to one or one to multi-party

Okazaki G Nara Med. Univ.
Kosuke Okazaki，Yasuhiro Matsuda, Yoshihiro
Miyazaki, Hiromi Kanayama, Rio Ishida,
Mitsuhiro Uratani, Tsubasa Morimoto,
Hajime Sawai, Tsutomu Takeda, Masato
Honda, Natsuko Kashiwada

VR/AR/Display

Virtual agent platform
ISIRLISN

Speech recognition
Language processing

Agent system
Body movement
Facial expression

Greta platformAll groups
Evaluation
qualitative Anxiety state inventory
quantitative Physiological responses

14Online Measuring& Feedback

SST:Behavioral
Nakamura G training

CBT: Cognitivetherapy

NAISTOkazaki G
Nara Med. Univ.

NAIST, ANR-CREST

Kudo G
Osaka Univ.

Kudo G Osaka Univ.
Takashi Kudo，Hiroyoshi Adachi
Yukako Sakagami

Pelachaud G （France CNRS-ISIR)
Catherine Pelachaud, Donatella Simonetti
MARTIN G (France CNRS-LISN）
Jean-Claude MARTIN, Laurence BOLOT,
Céline CLAVEL

2023/1/16

Theoretical framework
Collaboration with Nara Medical University

SST scenario◦ Bellack model: social learning theory and psychosocial treatment◦ Self-introduction, motivation, modeling, role-playing, feedback, homework

Based on the above scenarios
Virtual agent adaptively change behaviors as humans do

15

NAIST, ANR-CREST

2023/1/16

Social skills and target populations
Difficult skills

Basic four skills [Bellack+2004]
Speaking

Public speaking
Listening
Turn-taking
AskingMutti-party
Declining
General populations

Autism, Schizophrenia
Severity

16

NAIST, ANR-CREST

2023/1/16

SST on Greta platform
Hello. I'm tapas. What is your name?
Hello, hundred's. How have you been recently?
Was that so, today we will learn speaking skills
together. Why do you think speaking skills areimportant?
...
Let's do role-playing.
Tell me your recent fun story.
Then please.
Yes, this is all right. Thank you very much.
We will calculate the result, so please wait.
Now for feedback.
...

17

NAIST, ANR-CREST

2023/1/16

Agents acceptability (prior studies)
Appearance, behavior, realism, task seriousness

Esposito+ 2019abRing et al+ 2014

McDonnell+ 2012
18

NAIST, ANR-CREST

2023/1/16

Question items
What is the appropriate appearance of virtual agents as SST trainers? (N=1218) [Tanaka+2022]
Acceptability as a trainer
Acceptability as a listener
Realism
Familiarity
Trustworthiness
Eeriness
Likeability of the face, eyes, perceived ages, voice, clothes, and overall◦ 5-point Likert scale (1: I don’t think so at all, 5: I think so very much)

SRS2, Toronto Alexithymia Scale

19

NAIST, ANR-CREST

2023/1/16

Ratings of images
Likable

(a)

20(b)

(c)

(d) (e) (f)Character(g)

(h)

(i)

1122334455

Familiarity(a) (b) (c) (d) (e) (f)Character

NAIST, ANR-CREST(g) (h)(i)2023/1/16

RealismSecond best

21

NAIST, ANR-CREST

2023/1/16

SST data collection to generate human-like feedback
Adults: Schizophrenia (N=16), Autism spectrum disorder (N=15), Healthy controls (N=21)
Children: Autism spectrum disorder (N=16), Healthy controls (N=17)

Assessments (ADOS, SRS2, third party trainers’ ratings etc), eye gaze

2*Kinect
2*Face1*whole22

NAIST, ANR-CREST

2023/1/16

Behavioral measurements (basic module for SST)Multimodal prediction model for SRS2 (N=27)
Audio speech, facial action units, and BERT similarity in speaking [Saga+2022]
Correlation coefficient for SRS 2 = 0.76

Pred.

Ground truth

23

NAIST, ANR-CREST

2023/1/16

Feedback generation using prediction modelCan multimodal features predict trainer’s score? [Saga+2022]
Applied the behavioral measurements model to the collected dataLabels••

Four SST tasks: ASK, DECLINE, TELL, LISTEN•••••

Ground truth evaluation: 1 to 5 by two third party trainers

RF1-5 scalefor 7 labelsin 4 tasks

Multimodal features（Praat: F0, Intensity, Openface: AUs,Openpose: Gesture
BERT Score）

24

NAIST, ANR-CREST

2023/1/16

Eye contact
Body direction anddistanceFacial expression
Vocal variation
Clarity
FluencySocialappropriatenessFeedback generation: prediction accuracy

25

NAIST, ANR-CREST

2023/1/16

SST system
Roleplay w/ CA

Prediction

Feedback
Predicted scores

Body direction
Eye contact

Voice variation

Facial ex.

Clarity

Fluency

You are good at body direction and distance. This is an
important skill that others understand your feeling.

It might be better if you speak more fluently. Although it isdifficult making it perfect, please remember to enjoy
conversations with others.

26

NAIST, ANR-CREST

2023/1/16

Social appropriatenessDemo video

27

NAIST, ANR-CREST

2023/1/16

Experiment
Using the developed system

28

NAIST, ANR-CREST

2023/1/16

Experimental evaluation
Participants: Healthy
One to one training in SST: conveying feelings, listening, making a request, declining
Training period: 1-month (once per week) = 4 trainings

Place: Visit to Lab.
Comparison◦ Group 1: system training (N=9)◦ Group 2: without training (N=9)

Training effect◦ General Self-efficacy scale, State-Trait Anxiety Inventory (STAI), Kiss-18, SRS2
Liebowitz Social Anxiety Scale◦ Role-play with human trainers, SST Role-play scale by third party raters

29

NAIST, ANR-CREST

2023/1/16

Results

0

Post - Pre

0-3-10-2-5-1

Post - Pre

1

52

10

3

Significant change in the generalized self-efficacy scale and state anxiety [Tanaka+ under review]Group 1

Group 1

Group 2

Group 2More experiments are to be done!
STAI, state anxiety presence (p=0.04)
General self-efficacy scale (p=0.02)30

NAIST, ANR-CREST

2023/1/16

Summary⚫ Communication difficulties⚫Social anxiety disorder⚫Autism spectrum disorder⚫TAPAS Project⚫ Social skills training⚫ Data collection⚫ Embodied conversational agent⚫ Skill prediction and feedback generation⚫ TAPAS system and evaluation⚫Future works⚫ Field experiments in hospitals and further improvements⚫ SST in AR environments and Training in various situations⚫ Cognitive behavior therapy

31

SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/1/16

References[Saga+2022] Takeshi Saga, Hiroki Tanaka, Hidemi Iwasaka, Satoshi Nakamura, Multimodal Prediction of Social Responsiveness Score with BERTbased Text Features, IEICE Trans. Information and Systems, 2022.
[Tanaka+2022] Hiroki Tanaka, Satoshi Nakamura, Acceptability of Virtual Characters as a Social Skills Trainer: Usability Study, JMIR HumanFactors, 2022.
[Saga+2021] Takeshi Saga, Hiroki Tanaka, Hidemi Iwasaka, Yasuhiro Matsuda, Tsubasa Morimoto, Mitsuhiro Uratani, Kosuke Okazaki, YuichiroFujimoto and Satoshi Nakamura, Multimodal Dataset of Social Skills Training in Natural Conversational Setting, ICMI 2021 Workshop: 2ndWorkshop on Social Affective Multimodal Interaction for Health, pp.395-399, Oct. 2021.
[Saga+2022] Takeshi Saga, Hiroki Tanaka, Yasuhiro Matsuda, Tsubasa Morimoto, Mitsuhiro Uratani, Kosuke Okazaki, Yuichiro Fujimoto, Satoshi
Nakamura, Analysis of Feedback Contents and Estimation of Subjective Scores in Social Skills Training, EMBC2022.
[Tanaka+2021] Hiroki Tanaka, Hidemi Iwasaka, Yasuhiro Matsuda, Kosuke Okazaki, Satoshi Nakamura, Analyzing Self-Efficacy and SummaryFeedback in Automated Social Skills Training, IEEE Open Journal of Engineering in Medicine and Biology, vol.2, pp.65-70, 2021
[Tanaka+2020] Hiroki Tanaka, Hidemi Iwasaka, Hideki Negoro, Satoshi Nakamura, Analysis of Conversational Listening Skills toward Agent-based
Social Skills Training, Journal on Multimodal User Interfaces, volume 14, issue 1, pp.73-82, 2020.
[Tanaka+2017] Hiroki Tanaka, Hideki Negoro, Hidemi Iwasaka, Satoshi Nakamura, Embodied Conversational Agents for Multimodal AutomatedSocial Skills Training in People with Autism Spectrum Disorders, PloS One, vol.12(8), pp.1-15, Aug. 2017.
[Tanaka+2016] Hiroki Tanaka, Sakriani Sakti, Graham Neubig, Tomoki Toda, Hideki Negoro, Hidemi Iwasaka, Satoshi Nakamura, Teaching Social
Communication Skills through Human-Agent Interaction, ACM Transactions on Interactive Intelligent Systems, vol.6(2), pp.1-26, 2016.

32

NAIST, ANR-CREST

Thank you for your attention

33

NAIST, ANR-CREST

2023/1/16