Multimodal Dataset of Social Skills Training
in Natural Conversational Setting
Takeshi Saga 1 , Hiroki Tanaka 1 , Hidemi Iwasaka 2 ,Yasuhiro Matsuda 2 , Tsubasa Morimoto 2 , Mitsuhiro Uratani 2 ,Kosuke Okazaki 2 , Yuichiro Fujimoto 1 , Satoshi Nakamura 1
1 N a ra I n s t i t u te o f S c i e n c e a n d Te c h n o l o g y, 2 N a ra M e d i c a l U n i ve rs i t y1

2021/10/26

TAPAS SUPPORTED BY ANR-CREST

Our research background• Research flow

• Long-term objective◦ to develop methods and tools to reduce social
stress in everyday situations (e.g. publicspeaking, social communication)

Collect & analyze SST data
Develop an automated SST system• Method◦ Social Skills Training (SST), etc.

• Target population◦ Healthy people◦ Schizophrenia◦ Autism Spectrum Disorder (ASD)

2Evaluate the system• What we’d like to know

TAPAS SUPPORTED BY ANR-CREST◦ Behavior analysis in SST◦ Disease-specific difference in SST◦ User-specific difference in SST

2021/10/26

Related work
• Voleti+2019◦ Classification AUC◦ Classification using language features◦ Semantic coherence◦ Lexical diversity◦ Lexical density◦ Syntactic complexity◦ 0.960 for Clinical vs Control classification◦ 0.826 for Sz/Sza vs Bip classification◦ Task: Social Skills Performance Assessment◦ Target disease◦ Schizophrenia, Schizoaffective disorder◦ Bipolar◦ Control

3

TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Social Skills Training (SST)A training framework used in psychiatric rehabilitation programs
Especially effective for people with difficulties related to mental illnesses or developmentaldifficulties, such as Schizophrenia(SZ) or Autism Spectrum Disorder(ASD)
Bellack method [Bellack+2004] is naturalistic style SST setting

4

TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Overview of this research
Problem
No research on behavior analysis in SST, especially for SZ and ASD
Key idea
1) Collect data to analyze behavior difference in SST
2) To maximize the training effect, disease-specific method is needed
Our contribution
1) Created human-human natural SST dataset with Control/ASD/SZ groups2) Analyzed with Control/SZ classification model using the dataset5TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Dataset
6

TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Dataset — setting
Video camera for facial expression
Azure kinect for body movement
Transparent partition to prevent Covid-19 infection
Four role-play situations for each participant◦ Express positive feeling, listen to others, ask a favor, refuse an offer

7

TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Dataset — diagnosis, assessment
Clinical Assessment◦ Social skills — FEIT, Kiss-18, Singelis◦ ASD related — SRS-2, ADOS-2◦ Schizophrenia related— BACS-J, PANSS

Note: ASD, Schizophrenia- Recruited based on Psychiatrist diagnosis• 3rd-party psychiatrist evaluation:5-point Likert scale, n=2, kappa=0.84◦ For participants — seven components◦ e.g. eye-contact, facial expression◦ For trainers — four components◦ e.g. appropriateness of positive feedbacks, suggestions for improvement

8TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Classification
BEHAVIOR DIFFERENCE BETWEEN DISORDERS IN SST

9TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Classification — methodSubjects — Schizophrenia and Control
Model — random forest
Preprocessing◦ Separate video and audio files into Role-play or Feedback segment files◦ Eliminate vocal overlap from audio◦ 48 audio segments for Schizophrenia , 46 audio segments for Control

Input features — 20 audio related and 21 face related (by Praat, OpenFace)◦ Three different condition: audio-only, visual-only, audio-visual

Training — leave-one-subject-out cross validation

10TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Classification — result & discussion (1)A : Audio feature-set
V : Visual features-set

From the result ...
◦ Audio-only feature-set achieved the best scores.
◦ By adding visual features the accuracy was dropped.

Characteristic differences in the audio features

11TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Classification — result & discussion (2)From the top-three feature importance ...
◦ All of them were audio features◦ Bandwidth of F3 (F3 BW) is clearly higher than others

Wide formant bandwidth induces a significant reduction of the vowel
identification rate [Dubno+1987]
F3 BW indicates the less vowel intelligibility in Schizophrenia group

Schizophrenia with strong positive symptom have more complex articulatory
coordination [Siriwardena+2021]

12

TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Conclusion
Summary◦ We collected natural SST dataset with SZ and ASD people◦ We experimentally trained a classification model with 88.7% F1 score

Limitation◦ Gender imbalance of trainers◦ Didn’t consider text contents

Future work
◦ Classification with ASD/Schizophrenia/Control◦ Add more modalities (text, body)◦ Try sequential model to consider features related to synclonization

13TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Supplemental
14

TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Comparison

Voleti+2019

Ours

Social Skills
DatasetAnalysis

15

TaskPerformance

Social Skills Training

Assessment
Classification target

Sz,Sza/Bip/Control

Sz,ASD,Control

featurestext

audio/visual

TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Our research project: TAPAS• Work packages• Objective◦ WP1: Situation & immersion◦ WP2: Online measuring & feedback◦ WP3: Virtual agent platform◦ WP4: Gathering therapy data
◦ WP5: Evaluation

16◦ to develop methods and tools to reduce socialstress in everyday situations (e.g. publicspeaking, social skills training)• Target population◦ public speaking◦ Social Anxiety Disorder (SAD)◦ Schizophrenia◦ Autism Spectrum Disorder (ASD)TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Dataset — clinical assessment

Perception of facial expression
General social skills scales
General social skills scales
Metric for ASD traits
Metric for ASD traits
Metric for cognitive functionality
Metric for Schizophrenia symptoms

17

TAPAS SUPPORTED BY ANR-CREST

2021/10/26

Features

18

TAPAS SUPPORTED BY ANR-CREST

2021/10/26