Detecting Dementia and Enhancing Social Skills
through Humanand Embodied Agent Interaction
Satoshi Nakamura (NAIST)
withHiroki Tanaka (NAIST), Takashi Kudo (Osaka University),and Kosuke Okazaki (Nara Medical University)

12023/6/23

TAPAS SUPPORTED BY ANR-CREST

Two Topics
1. Automatic detection of early neurodegenerative disorder through interaction
with the embodied agent. (-2018)◦◦

Hiroki Tanaka, Hiroyoshi Adachi, Norimichi Ukita, Manabu Ikeda, Hiroaki Kazui, Takashi Kudo, Satoshi Nakamura, "Detecting
Dementia through Interactive Computer Avatars, “IEEE Journal of Translational Engineering in Health and Medicine, 5(1), 1-11,Dec. 2017
Tsuyoki Ujiro, Hiroki Tanaka, Hiroyoshi Adachi, Hiroaki Kazui, Manabu Ikeda, Takashi Kudo, Satoshi Nakamura, “Detection ofdementia from responses to atypical questions asked by embodied conversational agents,” Proc. INTERSPEECH2018

2. Social skills training for the neurodevelopmental disorder throughinteraction with the embodied agent.
◦◦◦

2

Hiroki Tanaka, Satoshi Nakamura, The Acceptability of Virtual Characters as Social Skills Trainers: Usability Study, JMIR HumanFactors, 9(1), 2022.
Hiroki Tanaka, Hidemi Iwasaka, Yasuhiro Matsuda, Kosuke Okazaki, Satoshi Nakamura, Analyzing Self-Efficacy and Summary
Feedback in Automated Social Skills Training, IEEE Open Journal of Engineering in Medicine and Biology, vol.2, pp.65-70, 2021.
Takeshi Saga, Hiroki Tanaka, Yasuhiro Matsuda, Tsubasa Morimoto, Mitsuhiro Uratani, Kosuke Okazaki, Yuichiro Fujimoto,Satoshi Nakamura, Analysis of Feedback Contents and Estimation of Subjective Scores in Social Skills Training, 44th Annual
International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), July 2022.

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Two Topics
1. Automatic detection of early neurodegenerative disorder through interaction
with the embodied agent. (-2018)◦◦

Hiroki Tanaka, Hiroyoshi Adachi, Norimichi Ukita, Manabu Ikeda, Hiroaki Kazui, Takashi Kudo, Satoshi Nakamura, "Detecting
Dementia through Interactive Computer Avatars, “IEEE Journal of Translational Engineering in Health and Medicine, 5(1), 1-11,Dec. 2017
Tsuyoki Ujiro, Hiroki Tanaka, Hiroyoshi Adachi, Hiroaki Kazui, Manabu Ikeda, Takashi Kudo, Satoshi Nakamura, “Detection ofdementia from responses to atypical questions asked by embodied conversational agents,” Proc. INTERSPEECH2018

2. Social skill training for the neurodevelopmental disorder through interaction
with the embodied agent.
◦◦◦

3

Hiroki Tanaka, Satoshi Nakamura, The Acceptability of Virtual Characters as Social Skills Trainers: Usability Study, JMIR HumanFactors, 9(1), 2022.
Hiroki Tanaka, Hidemi Iwasaka, Yasuhiro Matsuda, Kosuke Okazaki, Satoshi Nakamura, Analyzing Self-Efficacy and Summary
Feedback in Automated Social Skills Training, IEEE Open Journal of Engineering in Medicine and Biology, vol.2, pp.65-70, 2021.
Takeshi Saga, Hiroki Tanaka, Yasuhiro Matsuda, Tsubasa Morimoto, Mitsuhiro Uratani, Kosuke Okazaki, Yuichiro Fujimoto,
Satoshi Nakamura, Analysis of Feedback Contents and Estimation of Subjective Scores in Social Skills Training, 44th Annual
International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), July 2022.

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Aim

Suspected of dementia
No suspected of dementia

4

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Neuropsychological Tests
Typical question: Based on neuropsychological tests◦ Frequently used screening for dementia◦ MMSE [Folstein et al. 1975], “What’s the date today?”◦ WMS-R [Wechsler 1997]→ Experts are necessary for these tests

Related works◦ Roark et al. 2011:Analyze response during Wechsler memory recall task.
◦ Aramaki et al. 2016: Find a relationship between cognitive and language abilities during HDS-R.
◦ Tanaka et al. 2017: Analyze speech and language features during MMSE test.
◼Possibilities of detecting dementia from speech and language features◼These studies are not suitable for daily use

Atypical question:5Not based on neuropsychological test
TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Methods
10 minuets to complete all tasks[Ujiro et al., 2018]
6

TAPAS SUPPORTED BY ANR-CREST[Tanaka et al., 2017]2023/6/23

Questions sets
Fixed QA: The system asks three fixed queries
based on MMSE:

Randomly select 5 from 13 questions

1) What's the date today? (Q1)
2) Tell me something interesting about yourself.(Q2)3) How did you come here today? (Q3)

7

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Retelling◼The system reads aloud from a different
part of the Wechsler logical memory inthe WMSR [14], and users paraphrase thepassage.
◼Record the audio sound spoken by aclinical psychologist in advance. Thesystem plays the sound.

8TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Embodied Agent System
MMD system architecture

Dementia detection moduleconnection

CameraSubtitle

Audio andvideorecording(FFmpeg)

Multimodal feature extraction(speech, image, linguistic)
Machine learning (SVM, logistic reg.)https://mmdagent-ex.dev/docs/system/

9

TAPAS SUPPORTED BY ANR-CREST

Dementia

2023/6/23

Controls (Non-dementia)

Experiments: Participants

Nine Alzheimer’s disease (AD), one normal pressure hydrocephalus (NPH), one MCI, and one
AD+NPH. Per the existing criteria, the dementia group was diagnosed by the psychiatrists of theOsaka University Medical School-affiliated hospital based on DSM-IV-TR [4]. We obtained theage, MMSE score, and educational history of all participants.

10TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Features⚫ Speech Features: Snack sound tool kit⚫F0, Power, Voice quality⚫Gap features⚫Time delay from the end of the agent utterance to thebeginning of the user’s response.

Normalized F0 variation per utterance

Breathiness: 1-st harmonic amplitude -3rd formant
amplitude
Gap: the start time of the user response – the end
time of the agent’s question
Ratio of the total vocabulary to the overall words⚫Language features⚫Type Token Ratio (TTR)

Normalized for the sample size

⚫Simpson’s D value
11

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Results: AUC for Randomly selected questions

12

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Demo

13

TAPAS SUPPORTED BY ANR-CREST2023/6/23

Two Topics
1. Automatic detection of early neurodegenerative disorder through interaction
with the embodied agent.
◦◦Hiroki Tanaka, Hiroyoshi Adachi, Norimichi Ukita, Manabu Ikeda, Hiroaki Kazui, Takashi Kudo, Satoshi Nakamura, "Detecting
Dementia through Interactive Computer Avatars, “IEEE Journal of Translational Engineering in Health and Medicine, 5(1), 1-11,Dec. 2017
Tsuyoki Ujiro, Hiroki Tanaka, Hiroyoshi Adachi, Hiroaki Kazui, Manabu Ikeda, Takashi Kudo, Satoshi Nakamura, “Detection ofdementia from responses to atypical questions asked by embodied conversational agents,” Proc. INTERSPEECH2018

2. Social skills training for the neurodevelopmental disorder throughinteraction with the embodied agent.
◦◦◦

14

Hiroki Tanaka, Satoshi Nakamura, The Acceptability of Virtual Characters as Social Skills Trainers: Usability Study, JMIR HumanFactors, 9(1), 2022.
Hiroki Tanaka, Hidemi Iwasaka, Yasuhiro Matsuda, Kosuke Okazaki, Satoshi Nakamura, Analyzing Self-Efficacy and Summary
Feedback in Automated Social Skills Training, IEEE Open Journal of Engineering in Medicine and Biology, vol.2, pp.65-70, 2021.
Takeshi Saga, Hiroki Tanaka, Yasuhiro Matsuda, Tsubasa Morimoto, Mitsuhiro Uratani, Kosuke Okazaki, Yuichiro Fujimoto,
Satoshi Nakamura, Analysis of Feedback Contents and Estimation of Subjective Scores in Social Skills Training, 44th Annual
International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), July 2022.

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Training Adapted Personalized Affective Social Skills
with Cultural Virtual Agents (TAPAS)
Problem: Social anxiety in schools and workplace
Approach: verbal/non-verbal interactive training system byembodied conversational agent (ECA)◦ Target：General populations, Social Anxiety Disorder (SAD)
Autism Spectrum Disorder (ASD), Schizophrenia (SZ)
◦ Behavioral training：
Social skills training（SST）◦ Cognitive training：
Cognitive Behavioral Therapy（CBT）

Public speech: Feelnervous and stress forattentions by others, andbad reputation↓Face red, sweat, shaking,pulsation, breathing,stomachache
Fear for the symptom
Social anxiety, avoidanceSocial anxiety disorder

Training Adapted Personalized Affective Social Skills with Cultural Virtual Agents: ANR-CREST Project
15

NAIST, ANR-CREST

2023/6/23Nervous,shamefulSocial Skills Training⚫Social Skills Training by Alan Bellack*⚫Social skills training (SST) is a type of behavioral therapy for people
with mental disorders or developmental disabilities.
⚫SST may be used by psychiatrists, teachers, therapists, or otherprofessionals*Bellack, A. S.: Social skills training for schizophrenia: A step-by-step guide, Guilford Press (2004)
https://www.verywellmind.com/social-skills-4157216
16

SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/6/23

Virtual agents and robots for social skills training (SST)

Robins+1998

Better communication
with Virtual agents!

Hoque+2015

Kumazaki+2018
17

Nijman+2020
NAIST, ANR-CREST

2023/6/23

SST data collection to generate human-like feedback
Adults: Schizophrenia (N=16), Autism spectrum disorder (N=15), Healthy controls (N=21)
Children: Autism spectrum disorder (N=16), Healthy controls (N=17)

Assessments (ADOS, SRS2, trainers’ ratings etc), eye gaze

2*Kinect
2*Face1*whole18

SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/6/23

Behavioral measurements (basic module for SST)
Multimodal prediction model for Social Responsiveness Scale-2 (SRS2), N=27
Audio speech, Facial action units, and BERT similarity in speaking [Saga+2022]
Correlation coefficient for SRS2 = 0.76

1Feel comfortable to stay with others compared to stay alone

1,2,3,42

My facial expression makes misunderstanding to my real feeling.

1,2,3,4

3I am confident to communicate to others

1,2,3,4Pred.

Ground truth

19

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Feedback generation using prediction model
RQ: Can multimodal features predict trainer’s score? [Saga+, 2022]
Applied the behavioral measurements model to the collected data

Four SST tasks: ASK, DECLINE, TELL, LISTEN
Ground truth evaluation: 1 to 5 for 7 skills by two third-party trainers
Prediction results indicating correlation coefficientfor third-party evaluation scores
Skill Label20

SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/6/23

Feedback generation
The feedback is generated by template sentences according to the highest and lowest scores
Reason and advice were made as referring to human SST data and finalized by consultation with NMU
Template example
Highest score: Clarity
Your [Skill name] was good. Since [Reason], I think you can dogreat communication!
clarity

they can understand what you wantclearly

Lowest score: Eye contact
It seems if you can be more aware of [Skill name], I think you can take greatcommunication. I think you better to [Advice] for improvement. Hang in there!
watch around their nose21

eye contact

SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/6/23

SST system
Roleplay w/ agents

Prediction

Feedback
Predicted scores

Body direction
Eye contact

Voice variation

Facial ex.

Clarity

Fluency

You are good at body direction and distance. This is animportant skill that others understand your feeling.

It might be better if you speak more fluently. Although it isdifficult making it perfect, please remember to enjoy
conversations with others.

22SATOSHI NAKAMURA@ NAIST, ANR-CREST

2023/6/23

Social appropriatenessDemo video

23

NAIST, ANR-CREST

2023/6/23

Summary
1. Automatic detection of early dementia through interaction with theembodied agent.
◦◦Audio, language, image features + Gap features (AUC: 0.95 Acc: 0.93)
Atypical questions can be used for daily diagnoses of early dementia

2. Social skills training for the neurodevelopmental disorder throughinteraction with the embodied agent.
•••

24

Social skills training as a behavioral training for communication disabilities:
Anxiety disorder, Autism spectrum disorder, Schizophrenia
Social skills training by an embodied computational agent
Sensing multimodal features and automatic summary feedback

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Members
Early dementia detection:◦ Hiroki Tanaka, Tsuyoki Ujiro (NAIST)◦ Takashi Kudo, Hiroyoshi Adachi, Hiroaki Kazui, Manabu Ikeda (Osaka University)

Social Skills Training: (TAPAS project)◦ Hiroki Tanaka, Takeshi Saga, Yuichiro Fujimoto (NAIST)◦ Kosuke Okazaki, Yasuhiro Matsuda, Tsubasa Morimoto, Mitsuhiro Uratani (NaraMedical University)◦ Takashi Kudo (Osaka University)◦ Catherine Pelachaud （France CNRS-ISIR)◦ Jean Claude, Martin (France CNRS-LISN）
25

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Reference
Hiroki Tanaka, Hiroyoshi Adachi, Norimichi Ukita, Takashi Kudo, Satoshi Nakamura, Automatic Detection of Very Early Stage of Dementia through Multimodal Interaction withComputer Avatars, ACM International Conference on Multimodal Interaction (ICMI), pp.261-265, Nov. 2016.
Hiroki Tanaka, Hiroyoshi Adachi, Norimichi Ukita, Manabu Ikeda, Hiroaki Kazui, Takashi Kudo, Satoshi Nakamura, Detecting Dementia through Interactive Computer Avatars, IEEE
Journal of Translational Engineering in Health and Medicine, vol.5(1), pp.1-11, Dec. 2017.

Tsuyoki Ujiro, Hiroki Tanaka, Hiroyoshi Adachi, Hiroaki Kazui, Manabu Ikeda, Takashi Kudo, Satoshi Nakamura, Detection of dementia from responses to atypical questions asked
by embodied conversational agents, Interspeech, pp.1691-1695, Sep. 2018.
[Saga+2022] Takeshi Saga, Hiroki Tanaka, Hidemi Iwasaka, Satoshi Nakamura, Multimodal Prediction of Social Responsiveness Score with BERT-based Text Features, IEICE Trans.
Information and Systems, 2022.
[Tanaka+2022] Hiroki Tanaka, Satoshi Nakamura, Acceptability of Virtual Characters as a Social Skills Trainer: Usability Study, JMIR Human Factors, 2022.

[Saga+2021] Takeshi Saga, Hiroki Tanaka, Hidemi Iwasaka, Yasuhiro Matsuda, Tsubasa Morimoto, Mitsuhiro Uratani, Kosuke Okazaki, Yuichiro Fujimoto and Satoshi Nakamura,Multimodal Dataset of Social Skills Training in Natural Conversational Setting, ICMI 2021 Workshop: 2nd Workshop on Social Affective Multimodal Interaction for Health, pp.395399, Oct. 2021.
[Saga+2022] Takeshi Saga, Hiroki Tanaka, Yasuhiro Matsuda, Tsubasa Morimoto, Mitsuhiro Uratani, Kosuke Okazaki, Yuichiro Fujimoto, Satoshi Nakamura, Analysis of Feedback
Contents and Estimation of Subjective Scores in Social Skills Training, EMBC2022.
[Tanaka+2021] Hiroki Tanaka, Hidemi Iwasaka, Yasuhiro Matsuda, Kosuke Okazaki, Satoshi Nakamura, Analyzing Self-Efficacy and Summary Feedback in Automated Social Skills
Training, IEEE Open Journal of Engineering in Medicine and Biology, vol.2, pp.65-70, 2021
[Tanaka+2020] Hiroki Tanaka, Hidemi Iwasaka, Hideki Negoro, Satoshi Nakamura, Analysis of Conversational Listening Skills toward Agent-based Social Skills Training, Journal on
Multimodal User Interfaces, volume 14, issue 1, pp.73-82, 2020.
[Tanaka+2017] Hiroki Tanaka, Hideki Negoro, Hidemi Iwasaka, Satoshi Nakamura, Embodied Conversational Agents for Multimodal Automated Social Skills Training in People withAutism Spectrum Disorders, PloS One, vol.12(8), pp.1-15, Aug. 2017.
[Tanaka+2016] Hiroki Tanaka, Sakriani Sakti, Graham Neubig, Tomoki Toda, Hideki Negoro, Hidemi Iwasaka, Satoshi Nakamura, Teaching Social Communication Skills through
Human-Agent Interaction, ACM Transactions on Interactive Intelligent Systems, vol.6(2), pp.1-26, 2016.

26

TAPAS SUPPORTED BY ANR-CREST

2023/6/23

Thank you for listening!

27

TAPAS SUPPORTED BY ANR-CREST

2023/6/23