Automated Social Skills Training with Audiovisual Information
Hiroki Tanaka1 , Sakriani Sakti1 , Graham Neubig1 , Hideki Negoro2 , Hidemi Iwasaka2 and Satoshi Nakamura1

Abstract— People with social communication difficulties tend
to have superior skills using computers, and as a result
computer-based social skills training systems are flourishing.
Social skills training, performed by human trainers, is awell-established method to obtain appropriate skills in social
interaction. Previous works have attempted to automate one orseveral parts of social skills training through human-computerinteraction. However, while previous work on simulating social
skills training considered only acoustic and linguistic features,
human social skills trainers take into account visual features(e.g. facial expression, posture). In this paper, we create andevaluate a social skills training system that closes this gap byconsidering audiovisual features regarding ratio of smiling, yaw,
and pitch. An experimental evaluation measures the differencein effectiveness of social skill training when using audio features
and audiovisual features. Results showed that the visual features
were effective to improve users’ social skills.

I. INTRODUCTION
Many people have difficulties in social interactions suchas presentations and job interviews [1]. Persistent social
skill deficits impede those afflicted with them from forming
relationships or succeeding in social situations. Social skills
training (SST) is a general cognitive behavior therapy toobtain appropriate these social skills for people who havedifficulties in social interaction, and is widely used by teachers, therapists, and trainers [2], [3]. If the SST process could
be automated, it would become easier for those requiring
SST to receive it anywhere and anytime.
Previous works have attempted to automate one or several
parts of SST (see reviews in [4], [5]), for instance, in the context of job interviews [6], public speaking [7], or emotional
expression [8]. Specifically, Tanaka et al., [9] developed asystem named “automated social skills trainer” that entirely
follows the basic training model of SST through humanagent interaction. The system provides feedback according
to extracted audio features, and the feedback was shown to
improve users’ social skill.
While this work is a first step, there still are a large numberof gaps between human-based SST and the automated social
skills training. One of the gaps is related to modality. While
the previous work considered only acoustic and linguisticfeatures, visual information (e.g., facial expression, headpose, and posture) is also an essential feature of humanbased SST [2], [3] and public speaking aids [10]. In this*This work was supported by JSPS KAKEN 26540117
1 H. Tanaka, S. Sakti, G. Neubig, and S. Nakamura are with
Graduate School of Information Science, Nara Institute of Science and Technology, Takayama-cho 8916-5, Ikoma-shi, Nara, Japan{hiroki-tan,ssakti,neubig,s-nakamura}@is.naist.jp
2 H. Negoro, and H. Iwasaka are with Center for Special Needs Education, Nara University of Education, Takabatake-cho, Nara-shi, Nara, Japan{gorosan,hiwasaka}@nara-edu.ac.jp

paper, we use the automated social skills trainer as a baseline,
and extend the system by adding audiovisual information toachieve more effective improvement in users’ social skills.
This system can be used by not only people who havedifficulties in social interaction but also people with autism
spectrum disorder (ASD) [8], [9].
II. SST AND P REVIOUS W ORKThe basic training model of SST [3] generally follows
steps of instruction, modeling, role-playing, feedback, andhomework. We briefly describe these steps and their implementation in the automated social skills trainer as follows:
1) Instruction: Instruction includes defining target skills
and explaining their goal. After the major social problems faced by the trainee are identified, the skills tobe trained are decided based on these problems. The
automatic social skill trainer sets narrative as a target
skill, which is a task of telling stories, and a basic skill
necessary for other higher-level skills [3].
2) Modeling: Trainers act as a model, demonstrating theskill that the users are focusing on so that the users
can see what they need to do before attempting to
do it themselves. The automated social skills trainer
replicates this for narrative skills by allowing users
to watch a recorded model video of people who haverelatively good narrative skills.
3) Role-playing: Users are asked to role-play. When theuser says “start role-playing,” the system says “pleasetell me about something fun you experienced recently.”
The role-playing starts after the system’s question, andcontinues for one minute. During this time, the avatar
nods its head, and the system automatically senses andanalyzes features from the video of the user.
4) Feedback: At the end of role-playing, the systemanalyzes the features of the user’s video and displays
feedback based on this analysis. This feedback helps
users to identify their strengths and weaknesses. Because displaying a large number of features in the
feedback may confuse users, the system performs
feature selection to identify the features effective in
defining narrative skills. This process was conductedin discussion with a professional social skills trainer.
5) Homework: The system sets little homework challenges that users perform in their own time throughout
the week. The system tells users to “please tell yourstory to others throughout the week, and let me know
about it.”Fig. 2.
Fig. 1.

Extracted points using the face tracker.

System framework.

III. I MPLEMENTATION OF T HE M ULTIMODAL S YSTEM
The role-playing and feedback are the most importantelements of the system, which consist of sensing and analysis
from video, and feedback visualization (Figure 1). In this section, we describe the integration of multimodal information
into each module.
A. Sensing and analysis from video
To analyze the visual information of the video, we extracted a number of facial features by using a constrained
local model (CLM) [11] based face tracker1 as shown in
Figure 2.
From a total of 66 feature points, we calculated thefeatures of outer eye-brow height (OBH), inner eyebrow
height (IBH), outer lip height (OLH), inner lip height (ILH),
eye opening, and lip corner distance (LipCDT) following
Naim et al. [12]. Using these features, we modeled smilingfaces using The Japanese Female Facial Expression (JAFFE)
database [13]. The database contains 213 images of 7 facial
expressions (6 basic facial expressions + 1 neutral) performed by 10 Japanese female models. Each image has been
rated with regards to 6 emotion adjectives by 60 Japanese
subjects. In the database, we used 31 samples of happy
faces and 30 samples of neutral faces. We trained a modelof two types of facial expression using SVM with linear
kernel. Precision, recall, and F-measure of leave-one-out
cross validation over the database were .91, .97, and .94
respectively.
For video, we predicted whether the label belongs to thesmiling or neutral class in each frame, and the proportionof smiling frames among all frames was named the ratioof smiling. To verify that the model generalizes to otherspeakers and video, we used the NOCOA+ database [14],which contains derisive and friendly videos of four Japanesemen. We extracted the ratio of smiling from these videos andconfirmed that friendly videos have a significantly larger ratioof smiling compared to derisive videos (p=0.002, Cohen’s
d=1.89).
In addition to the smile features, we separately incorporated two head pose features (yaw and pitch), based on
1 https://github.com/kylemcdonald/FaceTracker

the corresponding elements of the global transformations
associated with rotation. Here, the yaw indicates horizontal
direction, and the pitch indicates vertical direction of headpose. These features reflect looking away and looking down
during talking [12]. We calculated the absolute value of theyaw and took the average of the entire frame to analyze
shift from the front. With regard to the pitch, both facingup (negative values) and facing down (positive values) wereconsidered to be important, and thus we calculated theaverage value of the entire video without taking the absolutevalue. Because the system did not record images below thechest, we did not take into account non-facial gestures.
Extracted features related to speech and language follow
Tanaka et al. [9] as follows: F0 variation, amplitude, voice
quality, pauses, words per minute, words of more than 6
letters, fillers.
B. Feedback
Based on the calculated features, the system displays
feedback to the users (Figure 3). We attempt to display thisfeedback in a way that is easier to understand and interpretas follows:• User video: Users can watch the recorded video andaudio in the narrative.
• Overall score: The system displays a predicted overall
score, which motivates the user to practice more andimprove their score. We predict the overall score using
the generalized linear multiple regression method on ascale of 0 to 100 [9].
With regard to features of the regression model, because
we analyzed the data and found that ratio of smilingfor models was significantly higher than others (p=0.04,
d=1.21), we added ratio of smiling as input features of
the regression model. We confirmed that the correlation coefficient between the predicted narrative skills
and subjectively evaluated skill using leave-one-userout cross validation was 0.55 (p=0.02), which indicatesa weak correlation, when using statistically significant
features (words per minute, amplitude, words of morethan 6 letters, and ratio of smiling). We also confirmedthat the correlation coefficient was 0.51 when using only
speech and language features [9] showing that ratioof smiling provides a slight gain in correlation withsubjective evaluation.

Fig. 3.

Audiovisual feedback provided by the automated social skills trainer.

Comparison with models: The system uses a radar
chart to visualize the comparison of extracted features
between the user’s current narrative and model persons’narratives in terms of z-score, which is a statisticalmeasurement of a score’s relationship to the mean ina group of scores. The users are informed that theyshould attempt to emulate the model in all aspects.
• Comments: The system generates positive commentsthat reinforce the user’s motivation based on the featuresthat have values closest to those of the models. In
addition, the system also generates comments about
points to be improved based on the feature that has
a median distance from the models. This choice ofthe median, instead of the farthest, feature was madethrough discussions with a professional social skills
trainer, who noted that it may be fundamentally difficultfor people with communication difficulties to improvetheir worst points.
•

IV. E XPERIMENT: T RAINING E FFECT
In our experiment, we examined the difference in effectiveness of social skill training when using feedback related
to audio features and feedback related to audiovisual features.
A. Procedure
We recruited a total of 18 graduate students (15 males
and 3 females) all of whom were native Japanese speakers.
Participants were given instructions by the first author andtold that their speech and video would be recorded2 . A
webcam placed on top of the laptop and headset recorded
the video and audio of participants.
We separated participants into two groups: the audio group(6 males and 3 females), and the audiovisual group (9males)3 . All participants spoke a story to a familiar person
2 Note that the Research Ethic Committee of our institution has reviewed
and approved this experiment. Written informed consent was obtained fromall participants before the experiment.
3 It should be noted that while the genders are not balanced, we also performed the forthcoming analysis without including the female participants,
and it did not affect any of the statistical differences.

(pre), used the automated social skills trainer for 50 minutes,
and spoke a story to the same familiar person (post). In
the training, participants followed the procedure of the basic
training model. Because we did not control the duration ofwatching videos or performing role-play, the participants canselect the content by themselves. The audio group received
feedback regarding speech and language features, while
the audiovisual group received feedback regarding not only
audio but also ratio of smiling, yaw, and pitch features.
We asked an experienced social skill trainer, who has
been a supervisor of young adult developmental support
performing social skills training more than three years, toevaluate the overall narrative skills according to a Likert
score rated on a scale of 1 (not good) to 7 (good). He watchedrandomly ordered pre and post videos and rated the score.
To find important features that related to narrative skills,
we calculated statistical differences of each feature between
recorded videos of the relatively low scores (low score group)and the relatively high scores (high score group). We also
calculated post−pre scores to measure the effect of training.
We present the p-values of one-tailed t-tests and Cohen’s dvalues as a measure of the effect size.
B. ResultFirst, we examined differences in audiovisual features
between groups with lower and higher levels of social skills.
Because the social skill trainer’s rating was between 3 and6 (mean: 4.5), we separated each video into the low scoregroup (3 or 4, n=16) and the high score group (5 or 6, n=20).
We found a statistical difference between the two groups inthe case of amplitude (the high score group was significantly
louder than the low score group, p=0.03, d=0.63), ratioof smiling (the high score group smiled significantly morethan the low score group, p=0.03, d=0.66), and pitch (thehigh score group looked up significantly more than the lowscore group, p=0.04, d=0.63) in Figure 4. This showed that
amplitudes, ratio of smiling, and pitch are strongly related
to narrative skills.

Fig. 4. Boxplot of audiovisual features between the low score group andthe high score group.

Next, we examine the effect of training using audio oraudiovisual features. Before training, the initial scores of thetwo groups were not significantly different (p=0.96 (twotailed t-tests), d=0.03): the audio group had a mean of 4.0 (sd:0.91) and the audiovisual group had a mean of 4.2 (sd: 0.83).
Figure 5 shows the improvement of overall narrative skills
in two groups. These results show that audiovisual feedback
significantly affected the increase in overall narrative skills(p=0.03, d=0.98). The overall gain in skills using audiovisualfeedback was 1.1, which is comparable to, or slightly greaterthan similar previous work (around a 1.0 point improvement
through interview skill training for one week using a virtual
tutoring agent [6], and around a 0.7 point improvement
through narrative skill training [9]).
The advantage of audiovisual feedback compared to audio
can likely be attributed to slight improvements in ratioof smiling (p=0.09, d=0.66), with the audiovisual group
achieving a mean improvement of 0.088 (sd: 0.19) and theaudio group seeing a loss of -0.026 (sd: 0.15).
These results also reflect knowledge of human-based SST,
which has shown the importance of smiling and facingstraight to express that the speaker is having fun [2].
V. C ONCLUSION
We extended a method for automatic social skill training
by adding audiovisual information. We extracted features
regarding ratio of smiling and head pose. The experimental
evaluation confirmed the training effect and the relationshipbetween narrative skills and extracted features.
For future directions, we examine the timing of eachfeature (e.g. smiling) in more detail. We would like to add
other target social skills as referred to by Bellack [3], andcompare with human-based SST. Furthermore, we would liketo confirm the training effect over a longer period, and recruit
special-need populations such as people with ASD who arepotential users of the system.
ACKNOWLEDGMENT
We would like to thank children of Nara autism society,and Probono Nara for their helpful comments. This study
was supported by JSPS KAKEN 26540117.

Fig. 5. The improvement of overall narrative score in two groups. Error
bars indicate standard error (*: p < .05).

R EFERENCES[1] A. American Psychiatric Association, Diagnostic and Statistical ManRual of Mental Disorders (DSM-5⃝).
American Psychiatric Pub, 2013.
[2] R. Liberman and C. Wallace, “Social and independent living skills:
Basic conversation skills module,” Camarillo, Calif: Author, 1990.
[3] A. S. Bellack, Social skills training for schizophrenia: A step-by-stepguide. Guilford Press, 2004.
[4] N. Aresti-Bartolome and B. Garcia-Zapirain, “Technologies as support
tools for persons with autistic spectrum disorder: a systematic review,”
International journal of environmental research and public health,vol. 11, no. 8, pp. 7767–7802, 2014.
[5] J. A. Kientz, M. S. Goodwin, G. R. Hayes, and G. D. Abowd,“Interactive technologies for autism,” Synthesis Lectures on Assistive,
Rehabilitative, and Health-Preserving Technologies, vol. 2, no. 2, pp.
1–177, 2013.
[6] M. E. Hoque, M. Courgeon, J.-C. Martin, B. Mutlu, and R. W. Picard,“Mach: My automated conversation coach,” in Proceedings of the2013 ACM international joint conference on Pervasive and ubiquitous
computing. ACM, 2013, pp. 697–706.
[7] M. I. Tanveer, E. Lin, and M. E. Hoque, “Rhema: A real-time insitu intelligent interface to help people with public speaking,” inProceedings of the 20th International Conference on Intelligent User
Interfaces. ACM, 2015, pp. 286–295.
[8] B. Schuller, E. Marchi, S. Baron-Cohen, H. O’Reilly, D. Pigat,
P. Robinson, and I. Daves, “The state of play of asc-inclusion: an
integrated internet-based environment for social inclusion of children
with autism spectrum conditions,” arXiv preprint arXiv:1403.5912,
2014.
[9] H. Tanaka, S. Sakti, G. Neubig, T. Toda, H. Negoro, H. Iwasaka, and
S. Nakamura, “Automated social skills trainer,” in Proceedings of the20th International Conference on Intelligent User Interfaces. ACM,
2015, pp. 17–27.
[10] M. Chollet, T. Wörtwein, L.-P. Morency, A. Shapiro, and S. Scherer,“Exploring feedback strategies to improve public speaking: An interactive virtual audience framework,” in Proceedings of the 2015
ACM International Joint Conference on Pervasive and Ubiquitous
Computing. ACM, 2015, pp. 1143–1154.
[11] J. M. Saragih, S. Lucey, and J. F. Cohn, “Face alignment through subspace constrained mean-shifts,” in IEEE 12th International Conference
on Computer Vision, 2009, pp. 1034–1041.
[12] I. Naim, M. I. Tanveer, D. Gildea et al., “Automated analysis and prediction of job interview performance,” arXiv preprint
arXiv:1504.03425, 2015.
[13] M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, “Coding facial
expressions with gabor wavelets,” in Proceedings of Third IEEE
International Conference on Automatic Face and Gesture Recognition.
IEEE, 1998, pp. 200–205.
[14] H. Tanaka, S. Sakti, G. Neubig, T. Toda, and S. Nakamura, “Nocoa+:
Multimodal computer-based training for social and communication
skills,” IEICE Transaction on Information and Systems, vol. E98-D,
no. 8, pp. 1536–1544, 2015.