TAPAS: Training Adapted Personalised
Affective Social Skills with Cultural Virtual
AgentsSatoshi Nakamura, Nara Institute of Science and Technology
H iro ki Tanaka , N a ra Institute of Sc ience and Tec hnology
H idemi Iw a sa k a, N ara Me dica l U n iv ersity
Takashi K udo, O saka U niversity

1

2020/10/1
2

TAPAS SUPPORTED BY ANR-CREST

TAPAS: Background
Increase of people who are not good at social communication and social anxiety disorder in schools andworkplace (3 to 13%)
Try to solve by verbal / non-verbal interactive training system by Embodied Conversational Agent (ECA)◦ Target：general populations、Autism（ASD: Autism Spectrum Disorder) 、Schizophrenia, SAD: Social Anxiety
Disorder◦ Behavioral training：Apply Social skills training（SST）◦ Cognitive training：Apply Cognitive Behavioral Training（CBT）

Public speech: Feelnervous and stress forattentions by others, andbad reputation↓Face red, sweat, shaking,pulsation, breathing,stomachache
Fear for the symptom
Social anxiety, avoidanceSocial anxiety disorder

2TAPAS SUPPORTED BY ANR-CREST

2020/10/12

Nervous,shamefulSST skills and Target populations
OthersSSTPublic speaking

Speaking

Basic four skills

Listening
Turn take
AskingMulti-party
Declining
General populations

ASD, SCSeverity

CBTCBT**CBT is used independent to severity

3

TAPAS SUPPORTED BY ANR-CREST

Automated social skills trainer
Training tools for social communication difficulties
People with autism have good systemizing skills [Baron-Cohen et a., 2009]
Target skill: speaking skills [Tanaka, et al., 2015] and listening skills [Tanaka, et al., 2020]

4TAPAS SUPPORTED BY ANR-CREST

Role-playAnalyze behaviors of users and generate agent’s actions

Features based on [Tanaka et al., 2014]
5

TAPAS SUPPORTED BY ANR-CREST

Summary feedback
After role-play, the system displays feedback
Repetitive training until mastery

6

TAPAS SUPPORTED BY ANR-CREST

Multiple Linear regression on subjectiveratings(input: words per minute, amplitude,words over six letters, and smiling ratio)

Training effectOverall narrative skill: 17 subjects, average of two students’ ratings (1 to 7)
One-way ANOVA: (F[2,24]=4.70, p<.05) , Post hoc comparisons with Bonferroni’scorrection (p<0.05)

7

TAPAS SUPPORTED BY ANR-CREST

Children with autism spectrum disorders
Communicate to unfamiliar person (pre and post)
Overall narrative skill: clinical psychologist’s rating (1 to 7)
Pre-post comparison（p = 0.002, Cohen's d = 1.17)
7

Overall Narrative Skills

65432108(a) pre(b) post

TAPAS SUPPORTED BY ANR-CREST

Human-human and human-agent interaction

9TAPAS SUPPORTED BY ANR-CREST

Detection of early stage of dementia
10 minutes interaction: 12 early stage of dementia and healthy control[Tanaka et al., 2017]10

TAPAS SUPPORTED BY ANR-CREST

Multimodal-based classifiers
Answers to fixed questions: e.g. what is the date today?
Areas under the ROV curve◦ 0.90 (SVM), 0.88 (Logistic regression)

Features: response gap, language,speech, face etc

11

TAPAS SUPPORTED BY ANR-CREST

12TAPAS: Goal and contribution
Verbal / non-verbal interactive training system by ECA in everyday situation◦ Theoretical framework of social interaction inspired by social psychology: WP1◦ Behavioral emotion sensing, behavior measurement: WP2◦ Virtual agent that acts as a coach or member of the audience: WP3◦ Collect and model training and build dialogue based on SST and CBT: WP4◦ Scenario setting and evaluation: WP5

Expected Contributions◦ Evidence-based SST and CBT in VR environment by non-verbal / verbal integrated dialogue◦ Dialogue system research with the above functions has not been developed◦ Online / summary feedback, dynamic training according to user's skill◦ Considering verbal / non-verbal information, SST- and CBT-based online and summary feedback are new.

◦ Communication training with various difficulty levels◦ There is no research corresponding to one-to-one (coach / audience role), one-to-many, many-to-many situations, and difficulty levels◦ Comparative consideration of cultural differences between Japan and France
12Organization
Public speaking training
Teenagers, children with ASD

Situation & immersion
WP1
Theories of social interaction

Through human-agent interaction(Image: the verge)

No stress, high self-efficacy
Learn appropriate behaviors

Technical approachFranceVirtual agent platform

Nakamura G NAIST (WP2, WP3, WP4, WP5)
Satoshi Nakamura，Hiroki Tanaka, Seitaro
ShinagawaHirokazu Kato，Yuichiro Fujimoto

one to one one to multi-party VR/AR/Display

Japan

Agents integration, translation

Agent’s eye gaze, attitude (e.g. cooperative)(NAIST)

Speech recognition

WP3

Kudo G Osaka Univ. (WP1, WP4, WP5)
Takashi Kudo，Hiroyoshi Adachi
Yukako Sakagami

Language processing

Dialog modeling
Body movement(ISIR)

Chat-based

WP2

Multimodal behaviors (CNRS-LIMSI)

Cognitive
Behavioral
therapytraining(Nara Med. Univ.)(Osaka Univ.)
SSTWP5

qualitative

AttrakDiff questionnairequantitative Anxiety state inventory
Physiological responses

Culture difference

WP4

Gathering therapy data

Data collection
TAPAS SUPPORTED BY ANR-CREST

Pelachaud G （France CNRS-ISIR） (WP1, WP3,WP5)
Catherine Pelachaud
Donatella Simonetti

CBT(Image: psychology toolbox)

System improvement

New scenarios
13

WP4

Facial expression

Online measuring& feedback
Evaluation

Iwasaka G Nara Med. Univ. (WP1, WP4,WP5)
Hidemi Iwasaka，Yasuhiro Matsuda, KosukeOkazaki

MARTIN G (France CNRS-LIMSI） (WP1, WP2,WP5)
Jean-Claude MARTIN
Laurence BOLOT
Céline CLAVELWP1: Theoretical framework
Collaboration with other research groups
SST scenario◦ Bellack model◦ Self-introduction, motivation, modeling, role-playing, feedback, homework

CBT scenario◦ Beck model◦ Cognitive reconstruction

Based on the above scenarios
Virtual agent adaptively change behaviors as human do142020©HIROKI TANAKA AHC-LAB, IS, NAISTWP2: Behavior sensing on SST
Predicting Social Responsiveness Scale-2 (SRS2)
Input BERT similarity, speech and action units（n=27） [Saga et al., 2020]
Correlation coefficient for SRS2 = 0.6

Feature weights

15

TAPAS SUPPORTED BY ANR-CREST

WP2: Behavior sensing on CBTAnalysis of action units during CBT with virtual agent
AU5 is related to the mood change (n=23) [Shidara et al., 2020]

16

TAPAS SUPPORTED BY ANR-CREST

WP3: Virtual agent
Platform: Greta Ogre3D
Speech recognition: Google ASR
Speech synthesis: CereProc TTS、Lip synch
Connection: Java ActiveMQ
Character: Original （anime and real）
Head nodding

17

TAPAS SUPPORTED BY ANR-CREST

Greta details
Thrift: connect to Greta Unity 3D
Greta Unity 3D• Lower body movement• Implement to Hololens

Speech recognition
Connection

18

TAPAS SUPPORTED BY ANR-CREST

Greta Unity towards VR/AR
Presentation in head mount display [Zhou et al., 2020]

19

TAPAS SUPPORTED BY ANR-CREST

WP4: Social skills training （SST）
Medically established method for behavior training
Flow: Situation setting-> Motivation -> Modeling-> Role play-> Feedback-> Homework
1 period: about 1 hour × 10 times

Participant

Trainer

Language: words, expression,consistency, coherence, causality,etc.
Speech: pitch, speaking speed,filler, voice quality, amplitude etc.
Image: facial expression, gesture,posture, eye gaze etc.

20

TAPAS SUPPORTED BY ANR-CREST

SST data collection
Start 2020.9-、annotation, diagnosis (ASD, Schizophrenia)

2*Kinect
2*Face1*wholeDue to COVID19, need partition

21

TAPAS SUPPORTED BY ANR-CREST

Dialogue
Nara Medical Univ. and NAIST
Hello. I'm tapas. What is your name?
Hello, hundred's. How have you been recently?
Was that so, today we will learn speaking skills
together. Why do you think speaking skills areimportant?
Thank you very much. When you have a difficult
time, you tend to focus on yourself. You canincrease your sense of solidarity with others bylooking at others and the good things andcommunicating them appropriately.
...
Let's do role-playing.
Tell me your recent fun story.
Then please.
Yes, this is all right. Thank you very much.
We will calculate the result, so please wait.
Now for feedback.
...
Use the skills you learned today in your daily life.
Thank you very much. See you again.

22TAPAS SUPPORTED BY ANR-CREST

Cognitive behavioral therapy（CBT）
CBT: Cognitive Behavior Therapy◦ Medical method to change the schema(cognitive tendency, belief)◦ Psychotherapy-based instructional method◦ Cognitive Reconstruction, Behavioral Activation, Situation
Analysis, Problem Solving, Progressive Relaxation, Assertion,
Schema Modification

This study uses Cognitive reconstruction◦ Change schema depended on only minimal information◦ Situation setting-> initial mood-> automatic thinking->
disapproval-> adaptive thinking-> change in mood-> analysis(homework)◦ 1 period: 30 minutes or more (usually 4-50 minutes) x 16 times

23

TAPAS SUPPORTED BY ANR-CREST

Cognitive Therapy Record
ECA Dialogue
Pointed out my small voice in
Situation presentation, and weakness logicality Situation breakdownby the audience.
Describe emotion
Initial score Discouraged（８０）、Sad（９０）state
Negativethinking

I am not able to present in front ofExtract tendency
audience. I am not good for this job.

Evidence for My voice is small in general. I am not
Extract evidence
this thinking able to explain logically.
There are many people who give a
Provide information toAlternative good presentation in small voice. It is
guide for extracting
thinking possible to present logically if I
counter thinking.
prepare well in advance.
Changethoughts

I can give a good presentation even
Extract alternativevoice is small. I will prepare more inthinking
advancescore

Discouraged（２０）、Motivated（８０）Recognize change ofthoughts

Event and Fact
EVENT
Fact 124

TAPAS SUPPORTED BY ANR-CREST

Facts and memory
EVENT
Fact 1
Fact 2
Fact 3
Fact 4
Memories

25

TAPAS SUPPORTED BY ANR-CREST

Thoughts
EVENT
Fact 1THOUGHTS(Cognition)

Fact 2Fact 3Fact 4Memories

26

TAPAS SUPPORTED BY ANR-CREST

Feelings
EVENT
Fact 1THOUGHTS(Cognition)

Fact 2Fact 3Fact 4Memories

27

TAPAS SUPPORTED BY ANR-CREST

FEELINGS

Limited facts
EVENT
Fact 1Fact 2Fact 3
Fact 4Memories

28

TAPAS SUPPORTED BY ANR-CREST

Unhelpful thoughts
EVENT
Fact 1Unhelpful
THOUGHTS

Fact 2Fact 3Fact 4Memories

29

TAPAS SUPPORTED BY ANR-CREST

Feelings
EVENTFact 1Unhelpful
THOUGHTS

Fact 2Fact 3Fact 4Memories

30

TAPAS SUPPORTED BY ANR-CREST

Aberrant
FEELINGSProviding new facts
EVENT
Fact 1Unhelpful
THOUGHTS

Fact 2
Fact 3
Fact 4Memories

31

CBTTAPAS SUPPORTED BY ANR-CREST

Aberrant
FEELINGS

Providing new facts
EVENT
Fact 1Unhelpful
THOUGHTS

Fact 2
Fact 3
Fact 4Memories

32

CBT
TAPAS SUPPORTED BY ANR-CREST

Aberrant
FEELINGS

Providing new facts
EVENT
Fact 1Unhelpful
THOUGHTS

Fact 2
Fact 3
Fact 4Memories

33

CBTTAPAS SUPPORTED BY ANR-CREST

Aberrant
FEELINGS

Providing new facts
EVENT
Fact 1Unhelpful
THOUGHTS

Fact 2
Fact 3
Fact 4Memories

34

CBTTAPAS SUPPORTED BY ANR-CREST

Aberrant
FEELINGS

Providing new facts
EVENTFact 1Fact 2

Morerealistic &balanced
THOUGHTS

Fact 3Fact 4Memories

35

CBTTAPAS SUPPORTED BY ANR-CREST

Aberrant
FEELINGS

Providing new facts
EVENT
Fact 1
Fact 2Morerealistic &balanced
THOUGHTS

Fact 3Fact 4Memories

36

CBTTAPAS SUPPORTED BY ANR-CREST

Calm
FEELINGSBasic dialogue
Osaka Univ. and NAIST.
.
.
37

TAPAS SUPPORTED BY ANR-CREST

CBT experiment
Virtual agent（MMDAgent [Lee, et. Al., 2013]）◦ Multimodal interaction (+text-chat)Participants◦ 23 university students
Interaction with virtual agent◦ Same procedure dialogue◦ Turn change is done by pressing buttons

38

TAPAS SUPPORTED BY ANR-CREST

Mood score improvement
Human-based CBT [Persons et al., 1980] used mood change
Mood change during CBT by the virtual agent**Decrease in negative feelings (sad, etc)

39

TAPAS SUPPORTED BY ANR-CREST

Evaluation
Assessment◦ Social Responsiveness Scale（SRS2）◦ K6◦ QIDS◦ Toronto◦ Self efficacy◦ ADOS◦ Emotion Discrimination task etc

EEG: detecting incongruity and affective state
Eye tracking study

40

TAPAS SUPPORTED BY ANR-CRESTICMI workshop (SAMIH)
https://sites.google.com/view/wsamih/home

41

TAPAS SUPPORTED BY ANR-CREST

Reference
Hiroki Tanaka, Hidemi Iwasaka, Hideki Negoro, Satoshi Nakamura, Analysis of Conversational Listening Skills toward Agentbased Social Skills Training, Journal on Multimodal User Interfaces, volume 14, issue 1, pp.73-82, 2020.
Hiroki Tanaka, Hiroyoshi Adachi, Norimichi Ukita, Manabu Ikeda, Hiroaki Kazui, Takashi Kudo, Satoshi Nakamura, Detecting
Dementia through Interactive Computer Avatars, IEEE Journal of Translational Engineering in Health and Medicine, vol.5(1),pp.1-11, Dec. 2017.
Hiroki Tanaka, Hideki Negoro, Hidemi Iwasaka, Satoshi Nakamura, Embodied Conversational Agents for Multimodal AutomatedSocial Skills Training in People with Autism Spectrum Disorders, PloS One, vol.12(8), pp.1-15, Aug. 2017.
Hiroki Tanaka, Sakriani Sakti, Graham Neubig, Tomoki Toda, Hideki Negoro, Hidemi Iwasaka, Satoshi Nakamura, Teaching
Social Communication Skills through Human-Agent Interaction, ACM Transactions on Interactive Intelligent Systems, vol.6(2),pp.1-26, 2016.
Takeshi Saga, Hiroki Tanaka, Hidemi Iwasaka, Satoshi Nakamura, Objective Prediction of Social Skills Level for AutomatedSocial Skills Training Using Audio and Text Information, Adjunct of the 2020 International Conference on MultimodalInteraction (ICMI), Workshop on Social Affective Multimodal Interaction for Health, Oct. 2020.
Kazuhiro Shidara, Hiroki Tanaka, Hiroyoshi Adachi, Daisuke Kanayama, Yukako Sakagami, Takashi Kudo, Satoshi Nakamura,Analysis of Mood Changes and Facial Expressions during Cognitive Behavior Therapy through a Virtual Agent, Adjunct of the2020 International Conference on Multimodal Interaction (ICMI), Workshop on Social Affective Multimodal Interaction forHealth, Oct. 2020.

42

TAPAS SUPPORTED BY ANR-CREST

CBT data [Beck and Beck, DVD]43

TAPAS SUPPORTED BY ANR-CREST