ANR-CREST 2109
TAPAS: Training Adapted Personalised Affective
Social Skills with Cultural Virtual Agents
JP: PI Satoshi Nakamura, Ph.D (NAIST)
Hiroki Tanaka, Ph. D (NAIST)
Takashi Kudo, M.D (Osaka University)
Hidemi Iwasaka, M.D (Nara Medical University)

FR: PI Catherine Pelachaud, Ph.D (CNRS-ISR, Sorbonne U)
Jean-Claude Martin, Ph.D (CNRS-LIMSI)

Research Topics at AHC-lab, NAIST
Brain Analysis
Incongruity measurement
Cognitive Load
EEG Hyper Scanning
Speech Translation
Machine Translation
Multi-language ASR, TTS
Machine Speech Chain

Data Analytics
CaptionGeneration

Affective Computing①Social Skills Training②Detection of Early Dementia

Why don’t you
join our lab!

I’m looking
for a lab.
Spoken Dialogue
Multi-modal Dialogue
2019/12/02

Natural Language
Processing
Goal-oriented Dialog
Non goal-oriented Dialogue
Deep NeuralNetwork

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

2

1. Social Skills Training
Training tools for autism spectrum disorders
People with autism have good systemizing skills
Speaking skills [Tanaka, et al., 2015] and listening skills [Tanaka, et al., 2019]2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

3

Role-playAnalyze behaviors of users and generate avatar actions

Features based on [Tanaka et al., 2014]2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

4

Summary feedback
After role-play, the system displays feedback
Repetitive training until mastery2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

5

Training effect
One-way ANOVA: (F[2,24]=4.70, p<.05)
Post hoc comparisons with Bonferroni’s method: significantly differentbetween feedback and reading book (p<0.05)

2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

6

Human-human and human-agent interaction

2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

7

2. Detection of early dementia
10 minutes interaction: 12 early stage of dementia and healthy control[Ujiro et al., 2018]2019/12/02[Tanaka et al., 2017]

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

8Multimodal-based classification
Answers to fixed questions: e.g. what is the date today?
Areas under the ROC curve– 0.90 (SVM), 0.88 (Logistic regression)

Features: response gap, language,speech, face2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST9ANR-CREST (2019-2024)

TAPAS: Training Adapted Personalised Affective
Social Skills with Cultural Virtual Agents
JP: PI Satoshi Nakamura, Ph.D (NAIST)
Hiroki Tanaka, Ph. D (NAIST)
Takashi Kudo, M.D (Osaka University)
Hidemi Iwasaka, M.D (Nara Medical University)
FR: PI Catherine Pelachaud, Ph.D (CNRS-ISR, Sorbonne U)
Jean-Claude Martin, Ph.D (CNRS-LIMSI)

Research background and objectives
Increase of people who are not good at social

training system by Embodied Conversational Agent

Public speech: Feelnervous and stress forattentions by others, andbad reputation↓Face red, sweat, shaking,pulsation, breathing,stomachache– Target population: general population, depression,

Fear for the symptom

communication and social anxiety disorder inschools and workplace (3 to 13%)
Try to solve by verbal / non-verbal interactive

Nervous,shamefulautism spectrum disorders, schizophrenia– Behavioral training: apply Social Skills Training (SST)– Cognitive training: apply Cognitive Behavioral

Social anxiety, avoidanceSocial anxiety disorder

Therapy (CBT)

2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

11

Overall framework
Public speaking training

Nakamura G NAIST
Satoshi Nakamura，Hiroki Tanaka
Hirokazu Kato，Yuichiro Fujimoto

Situation & immersion
Teenagers, children with ASD

WP1Theories of social interaction

Through human-agent interaction(Image: the verge)

No stress, high self-efficacy
Learn appropriate behaviors

Technical approach

one to one one to multi-party VR/AR/Display

FranceJapan(NAIST)

Agents integration, translation

Virtual agent platform

Agent’s eye gaze, attitude (e.g. cooperative)

Speech recognition

WP3

Language processing

Dialog modeling
Body movement

Chat-based(ISIR)

WP4

Facial expression

Online measuring& feedback

WP2

Multimodal behaviors (CNRS-LIMSI)

Evaluation

Cognitive
Behavioral
therapytraining(Nara Med. Univ.)(Osaka Univ.)
SST

qualitative

Culture difference

WP4(Image: psychology toolbox)

Gathering therapy data

System improvement

New scenarios

Kudo G Osaka Univ.
Takashi Kudo，Hiroyoshi Adachi
Yukako Sakagami
Pelachaud G （France CNRS-ISIR）
Catherine Pelachaud
Donatella Simonetti

CBTWP5

AttrakDiff questionnaire
Anxiety state inventory
quantitative
Physiological responses

Iwasaka G Nara Med. Univ.
Hidemi Iwasaka，Yasuhiro Matsuda,
Kosuke Okazaki

MARTIN G (France CNRS-LIMSI）
Jean-Claude MARTIN

Elise Prigent
Ouriel Grynszpan

Data collection
2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

12Social skills training (SST)
Medically established method for behavior training for autism, and schizophrenia.
Flow: Situation setting -> Modeling -> Role play -> Feedback -> Homework
1 period: 30 minutes or more x 10 times [Nara Med. Univ. 2018]

ParticipantTrainer

Language: words, expression,consistency, coherence, causality,etc.
Speech: pitch, speaking speed,filler, voice quality, amplitude etc.
Image: facial expression, gesture,posture, eye gaze etc.

2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

13

Cognitive behavioral therapy (CBT)e.g. thinking note (column method)

Psychotherapy-based instructional method tochange the cognitive schema
Situation

Cognitive Reconstruction, Behavioral Activation,
Situation Analysis, Problem Solving, Progressive
Relaxation, Assertion, Schema Modification
This study uses cognitive reconstruction– Situation setting -> Initial mood -> Automatic thinking-> Disapproval -> Adaptive thinking -> Change inmood -> Analysis (homework)– 1 period: 30 minutes or more (usually 4-50 minutes) x
16 times

2019/12/02

Cognitive Therapy Record
Pointed out my small voice inpresentation, and weakness
logicality by the audience.

Initial score Discouraged（８０）、Sad（９０）Negativethinking

ECA Dialogue
Situation breakdown
Describe emotion
stateI am not able to present in frontof audience. I am not good for this Extract tendency
job.

Evidence for My voice is small in general. I am
Extract evidence
this thinking not able to explain logically.
There are many people who give Provide information
Alternative a good presentation in small voice. to guide forthinking It is possible to present logically if extracting counter
I prepare well in advance.
thinking.
Changethoughts

I can give a good presentation
Extract alternativeeven voice is small. I will prepare
thinking
more in advance

score

Discouraged（２０）、Motivated（８０）ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

Recognize change ofthoughts
14

Summary
Social Skill Training– SST for Autism

Detection of early dementia
New project– ANR-CREST: TAPAS project !

2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

15

Continue to J-C Martin!

2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

16

Children with ASD
Communicate to unfamiliar person
Pre-post comparison（p = 0.002, Cohen's d = 1.17)

2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

17

Cognitive reconstruction dataCollecting interaction data between doctors and people with depression
e.g. Beck & Beck cognitive therapy live session

2019/12/02

ANR-CREST Symposium 2019 Dec. 2nd Satoshi Nakamura, AHC, NAIST

18