Noname manuscript No.
(will be inserted by the editor)

Analysis of Conversational Listening Skills toward Agent-based
Social Skills Training
Hiroki Tanaka · Hidemi Iwasaka · Hideki Negoro · Satoshi Nakamura

Received: date / Accepted: date

Abstract Listening skills are critical for human communication. Social skills training (SST), performed byhuman trainers, is a well-established method for obtaining appropriate skills in social interaction. Previous work automated the process of social skills training
by developing a dialogue system that teaches speaking skills through interaction with a computer agent.
Even though previous work that simulated social skills
training considered speaking skills, the SST framework
incorporates other skills, such as listening, asking questions, and expressing discomfort. In this paper, we extend our automated social skills training by considering user listening skills during conversations with computer agents. We prepared two scenarios: Listening 1
and Listening 2, which respectively assume small talkand job training. A female agent spoke to the participants about a recent story and how to make a telephone
call, and the participants listened. We recorded the dataof 27 Japanese graduate students who interacted withthe agent. Two expert external raters assessed the participants’ listening skills. We manually extracted features that might be related to the eye fixation and behavioral cues of the participants and confirmed that asimple linear regression with selected features correctlypredicted listening skills with a correlation coeﬃcient
above 0.50 in both scenarios. The number of noddings
H. Tanaka
Nara Institute of Science and Technology, Japan
E-mail: hiroki-tan@is.naist.jp
H. Iwasaka
Nara Medical University / Heartland Shigisan, Japan
H. Negoro
Nara University of Education, Japan
S. Nakamura
Nara Institute of Science and Technology, Japanand backchannels within the utterances contributes tothe predictions because we found that just using these
two features predicted listening skills with a correlationcoeﬃcient above 0.43. Since these two features are easier to understand for users, we plan to integrate theminto the framework of automated social skills training.

1 Introduction
Social skills are critical factors that influence human
life. Persistent social skill deficits hamper those withsuch aﬄictions from forming relationships or succeeding in social situations. Social skills training (SST) [6],a general psychosocial treatment through which people with social diﬃculties can obtain appropriate social skills, is widely used by teachers, therapists, andtrainers. Automating the SST process will simplify theacquisition of such social skills by those who require
them.
Autism spectrum disorder (ASD) is a spectrum condition [1, 12], meaning it has a broad range of characteristics from mild to severe. Using computer agents in
SST is motivated by the fact that even though somepeople with high-functioning autism experience diﬃculty during social communication, they also show goodor even superior systemizing skills [3, 13]. Systemizingis the drive to analyze or build systems and understand and predict behavior in terms of underlying rules
and regularities. The use of systematic computer-based
training for people who need to improve their social
skills exploits the following criteria: 1) such people favor computerized environments because they are predictable, consistent, and free from social demands; 2)
they can work at their own pace and level of understanding; 3) training can be repeated until the goal is2achieved; and 4) interest and motivation can be maintained through computerized rewards. It may also beeasier for those who suﬀer from social diﬃculties to use
computer agents than to directly interact with humans[44,10]. A recent paper suggested that people with suchsocial diﬃculties as ASD feel safer and more comfortable in a virtual environment than during interaction
with an actual person [35].
Previous works conducted SST using computer agents[7,30], including in speaking and emotional regulation
contexts [45, 42,18,52]. A special issue of the Journal on
Multimodal User Interfaces on SST addressed the Theory of Mind (ToM) reasoning [50], gaze leading [36],virtual discrete trial training for teacher trainees [40],and bad news conversations [33].
Specifically, Tanaka et al., [45] proposed an automated social skills trainer that consists of the following
aspects: 1) instruction and motivation of target skills,2) modeling, 3) role-playing, 4) feedback, and 5) homework. In terms of interactive application design, theseprocesses mostly include two challenging parts: assessment of the target skills and giving feedback about
those target skills. The system follows the evidencebased training scheme called the Bellack method [5].
The Bellack method (or step-by-step SST) is a wellstructured and widely used evidence-based approach. It
is a cognitive-behavioral SST inspired by Social Learning Theory’s five core principles: modeling, shaping,
reinforcement, overlearning, and generalization [2, 39].
The Bellack method defines the SST framework and its
four basic skills: speaking, listening, asking questions,and expressing discomfort. These skills are beneficial
for all people [5]. Training should be done in the above
order; since expressing discomfort is more diﬃcult, it
should be the last. These skills are helpful and basis toobtain other skills such as how to be more assertive, andhow to maintain conversation. In this paper, we focus
on listening skills as second most important skills. The
following are the critical aspects of listening skills: 1)
looking conversation partners in the eye (eye contact),2) nodding, and 3) repeating the keywords of the conversation partner [5]. The listening skills investigatedby this study are not for hearing and understanding
speech [4,8]. Listening skills explicitly express that one
is actively listening to the partner’s speech [5, 48]. For
example, a previous study showed that people with social diﬃculties tend to avoid looking conversation partners in the eye [20].
Despite the importance of listening skills, most automated SSTs focus on speaking skills. Okada et al.
assessed interaction skills that addressed listening attitudes [34]. Many diﬀerently motivated works have been
designed to generate (model) human-like head tilting,Hiroki Tanaka et al.

backchannels, and nodding on humanoid robots or computer agents based on analyzing human behaviors [25,31, 16, 17, 22, 29, 19]. Ward et al. proposed listening-skills
training that produces immediate feedback, although
they did not use computer agents and focused only on
backchanneling behavior [51]. Another previous workargued that personality [14] is related to empathic listening skills [38].
In this paper, we hypothesized that the SST process in listening skills can be automated for interaction between humans and computer agents. First, weanalyzed part of the automatic prediction of listening
skills by collecting the listening data of the interactionof graduate students and computer agents and investigated the possibility of automatically assessing user’s
listening skills from multimodal aspects. To the best of
our knowledge, this is the first work to analyze humanconversational listening skills in human-agent interaction.
This paper is an extended version of previously published work [43]. We extend that previous work by scrutinizing correlation and making predictions with a few
features toward the SSTs of actual users.

2 Computer Agents
We used an MMD agent [24] as a computer agent andshow an example in Fig. 1. We used default parameters
for its speech such as speaking rate and voice pitch. For
example, the articulation rate, which divides the number of morphemes (measured in MeCab [21]) by voiced
seconds, is 1.85 seconds for the following utterance of
the agent: “hazime ni denwa o kake te.” The pitch wasaround 270 Hz.
Four Japanese people (two males and two females)
created the agent’s spoken sentences: one person wasa licensed psychiatrist with more than three years ofexperience with SST and another was a licensed speechtherapist. We created the following three tasks: Speaking, Listening 1, and Listening 2:
1. Speaking: The user describes a recent story/experienceto the computer agent. This module follows the sameprocedure as a previous work [45].
2. Listening 1: The user listens to the agent’s story.
Table 1 shows the sentences that we created (translated into English). This assumes a casual, smalltalk situation.
3. Listening 2: The user listens to a procedure that explains how to make a telephone call. These sentences
are shown in Table 1 (translated into English). They
are designed for more serious situations such as job
training.

Analysis of Conversational Listening Skills toward Agent-based Social Skills Training

3

for each participant. The first author explained the experiment to them and obtained their informed consent.
They completed consecutive Speaking (60 sec.), Listening 1 (60-90 sec.), and Listening 2 (60-90 sec.) sessions.
The completion time varied based on how long theparticipants spoke (about four minutes of completiontime).

3.2 Procedure

Fig. 1 Interaction with computer agents.

We provided them in the same order for all theparticipants because we assumed the order of actual
training. We aimed to develop from the basic skills
from speaking to listening and then to practical skillsin actual job-training situations. Since we are examining the relationship between autistic traits and speaking/listening skills, we placed the speaking task before
the listening tasks.
In Listening 1 and Listening 2, the agent spoke forabout one minute, including several four- or five-second
pauses between the sentences. These pause lengths were
empirically determined by two people (one of whom wasa licensed psychiatrist with over three years of experience with SST) for the naturalness of the conversation and knowledge that even elderly persons can respond to the agent’s question in an average of less thanfour seconds [41]. During the pauses, the agent nodded if the user said something and waited over threeseconds after the user’s final utterance. The collected
data had few overlap phenomena because the system
explicitly waited for three seconds after speaking sentences and waited for three more seconds after the user
utterance. Since backchannel and turn-taking generally
impact overlapped speech, we must consider such phenomena for our assessment and training in the future.
At the end, if the user answers yes to the questions, “Isthere anything you would like to ask?” and “Do you
have any more questions?”, the agent gives no answer,
waits for the end of the user’s utterance, and finallyanswers “Thank you.”

The first author explained how to use the system byplaying an example video that showed head nodding
and backchannel feedback. This was a video of an experimenter testing the system.
Data were collected in a soundproof room using
a laptop PC (IBM ThinkPad). A webcam (ELECOM
UCAM-DLY300TA) was placed on top of the laptop,
and an eye-tracker (Tobii X2-30) was set at the bottomof the laptop screen. We turned oﬀ the light in the room
to minimize external distractions (Fig. 1).
After collecting data, we gave two questionnaires(explained below) to all of the participants. The total amount of time for all the procedures was approximately 20 minutes. From the collected data, we calculated the following multimodal aspects: eye fixation,
image, and speech features. We selected these features
based on previous studies [17, 31, 22], specifically influenced by the critical aspects of listening skills from the
Bellack method’s SST [5].

3.3 Eye Fixation
We used a table-mounted eye tracker (Tobii X2-30) because it can obtain focal points with high resolution accuracy when head movements are relatively small [11].
We applied an IV-filter to the raw eye-gaze data
and manually categorized the following areas of interests (AOIs): 1) eyes, 2) mouth, 3) face, 4) other. Face
includes eyes and mouth regions. Finally, we extracted
the ratio of each AOI. These regions are represented inFig. 2.

3.4 Speech- and Image-based Features
3 Methods
3.1 Participants
We recruited 27 participants (six females and 21 males
whose mean age was 25.1, SD: 2.13) from the Nara Institute of Science and Technology. We payed 1000 yen

We manually coded head nodding (video) and speech(audio) using the ELAN tool. The following information
was coded by one male annotator: backchannel feedback(defined as the following Japanese vocalizations: un, un
un, un un un, hu un, hai, hai hai, and hai hai hai),backchannel feedback between an agent’s utterances,

4

Hiroki Tanaka et al.

Table 1 Sentences spoken by computer agents: pause denotes three seconds of silence, and long pause denotes five seconds ofsilence.
Listening1
The other day, my friends and I went to a trendy cafe in Kyoto that was advertised in a magazine. pause
I drank a caﬀe latte. The foam on top of it had these really cute animal designs.
So I took a picture and uploaded it to Instagram. pause
After that, my friend and I drove to Kiyomizu Temple.
I found a souvenir shop on a side street on the way.
After carefully considering all the choices, I bought a very delicious roll cake. pause
At Kiyomizu Temple, the autumn leaves were so beautiful.
Since I also think that cherry blossoms are beautiful, I’m hoping to return in the spring.
That’s all. Is there anything you would like to ask? long pause
Thank you.
Listening2
First, dial the number, and after someone answers, give your name and aﬃliation. pause
After that, say the name and aﬃliation of the person to whom you want to speak
and ask to be connected. pause
When you are connected, briefly state your purpose. pause
If the person in charge is unavailable, explain that you will call back, and then hang up. pause
It is important to make a phone call at a proper time.
Since calling late at night or early in the morning will probably annoy people, avoid doing so. pause
This concludes the explanation. Do you have any more questions? long pause
Thank you.

User video
Face (AOI)

Eye (AOI)

Fixation point
Mouth (AOI)

Fig. 2 Exported video, AOI (rectangles), and fixation point(red circle) examples: these are inexact areas.

backchannel feedback within an agent’s utterances, repetition of an agent’s utterance (paraphrase), questions,miscellaneous utterances, head nod (once), head nod(twice), and head nod (three or more times). We manually coded one nod diﬀerently from two or more than
three nods. In this study, we simplified how we counted
the number of nods as the total amount of head nods as
once, twice, and three or more times based on a previous
work [31]. We defined more than one second as separate
coding. This means an interval between two consecutive
nods. Here the backchannel feedback is a verbal behavior that expresses something, e.g., I’m adjusting to myconversation partners. Nodding, a non-verbal behavior,is represented by vertical head movements that express
acknowledgement and agreement, e.g.
As coding output, we extracted the following features: 1) the number of backchannel feedback instances(also the backchannels within and between utterances),2) the number of repetitions/paraphrase utterances, 3)
the number of questions, 4) the number of miscellaneous utterances, 5) the number of nods, 6) gap (thetiming between the end of an agent’s utterance and the
beginning of a user’s utterance).
3.5 Social Responsiveness Scale and Big Five
Personality Test
We collected answers to two questionnaires: the Social Responsiveness Scale (SRS) [9], which is related
to autistic traits based on the DSM-V [1], and the
Big Five Personality Test [15]. We collected answers
from these two questionnaires because we hypothesizedthat autistic traits and personality [14] are related toeach other as well as to speaking and listening skills.
The latter consists of the following sub-areas: extraversion, agreeableness, conscientiousness, neuroticism, andopenness. The relationship between these two questionnaires was previously investigated [47]. Personality isrelated to empathic listening skills [38]. We found asignificantly negative correlation between SRS and extraversion (Spearman’s ρ = −0.5, p < 0.05) (Fig. 3).
3.6 Clinical Psychologists’ Ratings of Listening Skills
Two licensed clinical psychologists with over three yearsof experience with SST rated both the listening andspeaking skills by watching videos exported from theTobii video recorder (Fig. 4). Their impressions focusedon the participants in addition to such behaviors asAnalysis of Conversational Listening Skills toward Agent-based Social Skills Training

mographic information of all participants is shown in
Table 2.

SRS
-0.5

-0.05

Opn
-0.46

Ext

0.3
0.18
0.23

0.31
-0.03
-0.3

-0.11
0
0.15

Nrt

5

Ag.

-0.47
-0.28
-0.16

Cn.

Fig. 3 Relationships among each social psychological scale.

Fig. 4 Screenshot of video recording for ratings.

eye movements, head noddings, facial expressions, andspeech (as with the usual SST). After watching multiple
videos, they evaluated the overall listening skills of theparticipants with Likert scores on a scale of 1 (not good)to 7 (good) [45].
We calculated the Kappa statistics of the two raters
using a weighted Kappa set to 1 (on the diagonal) anddecreased weights oﬀ the diagonal [23]. The following
are the weighted Kappa correlation coeﬃcients: 0.37
(Speaking), 0.47 (Listening 1), and 0.59 (Listening 2).
They show fair to moderate agreement [23]. We also
calculated the Pearson’s correlation coeﬃcients of thetwo raters: 0.44 (Speaking), 0.46 (Listening 1), and 0.66
(Listening 2) (all, p<0.05). Finally, we averaged the tworaters’ scores for further analysis. Although we tried
to separately analyze their individual scores, averaging their results led to more consistent results. For example, as reported below in section 4.2, we confirmed
larger variation and fewer mean values of correlationcoeﬃcient than the averaged results beforehand. De-

3.7 Statistical Analysis
This section presents our experimental evaluation of thecollected data. After analyzing the relationship among
each feature, we evaluated our prediction model for automatic listening-skill prediction. In this study, we used
Spearman’s correlation coeﬃcient to observe correlation. We defined good models as people who scoredabove five in both raters [45] and will be used themas good examples in SST.
First, we analyzed the relationship between eachquestion and the two listening skills. Then we normalized the extracted features using z-score normalization.
Regarding automatic assessment, we used multiple linear regression (LR), which is a very simple linear approach to predict listening skills. Leave-one-person-out
cross validation evaluated the generalizability. We automatically selected features based on Akaike’s information criterion (AIC) in a stepwise algorithm (forward)
on the training set.
Finally, after confirming normality by the KolmogorovSmirnov test, we calculated the Pearson’s correlationcoeﬃcient between the actual and predicted values aswell as the root mean square error (RMSE). We also
evaluated the statistical significance using the Wilcox
rank sum test and the Cohen’s D value between goodmodels and the others in terms of SRS scores and multimodal behaviors.

4 Results
4.1 Statistical Analysis
The following are the correlation coeﬃcients: Listening 1 and Speaking were 0.31 (p=0.10), Listening 2
and Speaking were 0.41 (p=0.03), and Listening 1 and
Listening 2 were 0.54 (p=0.003). Eight people were selected as good models for Speaking. Seven were selected
as good models for Listening 1. Five were chosen for Listening 2. Specifically, participants ID5 and ID12 weregood for all three tests (speaking task and two listening tasks). Fig. 5 represents the SRS score distribution
between the good models and the others. Good models
have lower SRS scores than the others. Specifically, we
found a significant diﬀerence for Listening 2.
Regarding eye fixation, we found mean values of
40% for the eyes, 7% for the mouth, 88% for the face,and 5% for the others within all of the fixation points in6

Hiroki Tanaka et al.

Table 2 Participant demographics: ID, gender, age, SRS scores (min: 0, max: 195), extraversion, agreeableness, conscientiousness, neuroticism, and openness (min: 2, max: 14), averaged scores of speaking skills, listening skills 1, and listening skills 2(min: 1, max: 7). Bold fonts indicate good models.
ID
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

Gender
m
m
f
m
f
m
m
m
f
m
m
m
m
m
m
m
m
m
m
m
m
m
m
m
f
f
f

Age
23
23
24
23
29
25
24
28
24
29
27
23
26
26
23
29
24
23
24
26
28
25
28
24
24
23
24

SRS
80
34
78
61
70
53
105
101
37
63
87
54
59
77
75
59
57
67
73
39
19
80
52
75
34
97
52

Extra.
5
4
5
6
6
11
6
6
11
13
4
11
4
4
3
14
5
5
3
6
14
2
7
9
11
5
6

Agree.
9
13
9
8
7
8
11
11
9
11
12
12
12
12
14
13
11
12
9
12
14
11
8
9
10
12
12

Consc.
7
4
2
7
5
11
7
3
8
5
5
5
4
3
5
2
4
3
6
9
10
10
7
6
6
7
10

Table 3 Top five features: Brackets denote correlation coefficient (**: p < 0.01, *: p < 0.05, †: p < 0.10).
Rank
1
2
3
4
5

Listening 1
Nods (0.51**)
Questions (0.42*)
Back. w/ (0.36†)
Repetitions (0.25)
Miscellaneous (0.22)

Listening 2
Back. (0.55**)
Back. w/ (0.48*)
Nods (0.42*)
Back. b/ (0.25)
Repetitions (0.23)

every participant. We did not find any significant correlations in terms of eye fixation. The averaged # ofnods was 4.89 in Listening 1 and 4.85 in Listening 2.
The averaged # of backchannels was 5.26 in Listening
1 and 5.44 in Listening 2. We did not find any eﬀects
of elapsed time; nods and backchannels were not more
frequent at the beginning or at the end of the video.
Table 4 indicates the top five features that are correlated to listening skills. The # of nods was significantly
related to listening skills as was the # of backchannels. The backchannels between an agent’s utterances
are more important than those within an agent’s utterances. Fig. 6 shows the examples of the actual timingsof the head nodding and backchannel feedback in Listening 1 and Listening 2. We confirmed that persons
with low listening scores tended to nod only during the

Neuro.
12
5
12
14
12
3
14
10
12
8
4
6
8
5
7
8
12
9
6
4
2
2
13
5
13
9
8

Open.
7
8
5
7
8
13
9
8
5
9
8
8
8
6
9
14
5
13
8
5
14
12
8
8
5
8
10

Speaking
6
5
3.5
3.5
7
5
3
6
3
4
3.5
6.5
5
4.5
4
6.5
6.5
4.5
4.5
4
4.5
3
3.5
5
5
3
5

Listening 1
5.5
6
5.5
2.5
7
3
4
4
1
4
2
6.5
4
2
5
2.5
3.5
3
2
4
5.5
4.5
5
4
5.5
2.5
3.5

Listening 2
4
3.5
3
2.5
6
1.5
4
3
1
4.5
3
6
3.5
2
5.5
5
5.5
3.5
5
4
5.5
4
4
4
4.5
2
2

agent’s pauses. In contrast, those with high listening
scores nodded and uttered at other times. They tended
to respond at the positions of specific keywords, commas, and periods of the agent’s transcripts within thesentence based on the pitch of the agent’s speech. It
was also discussed: backchannel occurs everywhere in
Japanese, e.g., in near sentence-final particles and interjection particles [28]. Further study needs to investigate
these eﬀects.

4.2 Predicting Listening Skills
4.2.1 Scoring based on regression models
For linear regression, we identified the following correlation coeﬃcients between the predicted and actual
values: Listening 1 was 0.50 (p < 0.01), and Listening 2
was 0.51 (p < 0.01). Their RMSEs were 1.52 (Listening
1) and 1.26 (Listening 2).
Decreasing the number of features is crucial to reduce user’s cognitive complexity when an automatedsystem provides feedback in an actual SST. To do so, we
extracted two important features found by previous results. We chose the number of noddings and backchannels within agent utterances to predict the listeningAnalysis of Conversational Listening Skills toward Agent-based Social Skills Training
Table 4 Prediction results of listening skills (**: p < 0.01,*: p < 0.05).
Model
LR with AIC (Lis 1)
LR with AIC (Lis 2)
LR with two features (Lis 1)
LR with two features (Lis 2)

Correlation
0.504**
0.511**
0.481**
0.438*

RMSE
1.52
1.26
1.34
1.23

Listening 2
p=0.14, D=0.54

Good models Others

Table 5 Mean and SD values of good models and others:
Wilcoxon rank sum test with p-values.
Listening 1
back /w
nod
Listening 2
back /w
nod

Model

OthersWp-value

3.6 (2.5)
7.0 (4.0)

0.68 (1.2)
5.09 (3.8)

91.5
106.5

0.13
0.02

5.0 (4.8)
9.85 (6.8)

1.85 (1.9)
3.65 (3.1)

91
72.5

0.007
0.14

80

100

In Listening 1, the total amount of time that agents
spoke was 45 seconds (total interaction time was around
60 seconds). In Listening 2, the total amount of timethat agents spoke was 38 seconds (the total interaction
time was around 56 seconds). Thus, good models nodded once every 8-9 seconds in Listening 1, and in Listening 2 they gave backchannel feedback when the agent
spoke every 7-8 seconds. This seems relevant to humanhuman interaction in Japanese and demonstrates thatbackchannels occur on average every 7-8 seconds [28].

20

40

60

80
60
40
20

20

40

60

80

100

Listening 1
p=0.03, D=0.82

100

Speaking
p=0.27, D=0.217Good models Others

Good models Others

Fig. 5 Good models and SRS scores: We represent p-valuesof Wilcox rank sum test and Cohen’s D values.

skills with these two features and obtained a 0.48 correlation coeﬃcient in Listening 1 (p < 0.01) and a 0.43
correlation coeﬃcient in Listening 2 (p < 0.05). Even
though this slightly lowered the prediction values, theyremain significant.
4.2.2 Good models and others
This subsection represents the diﬀerences between goodmodels and the others in terms of the number of nodding and backchannels within an agent’s utterances. We
also confirmed a relationship between good models and
SRS scores in Figure 5.
As shown in Table 5, we found a significant difference of nodding (Wilcox rank sum test (one-tailed),
p=0.02). Good models nod with a mean of 7.0 in Listening 1. In contrast, the others had a mean of 5.09.
The number of backchannel feedbacks within the utterances was not significantly diﬀerent between the twogroups in Listening 1. We did find a significant difference of backchannel feedback within an agent’s utterances (Wilcox rank sum test (one-tailed), p=0.007).
Good models gave backchannels with a mean of 5.0 in
Listening 1; the mean of the others was 1.95. The number of nodding was not significantly diﬀerent between
the two groups in Listening 2.

5 Discussion
This paper analyzed listening skills from conversational
behaviors by interaction with computer agents. We prepared two scenarios: Listening 1 and Listening 2, which
respectively are related to small talk and job training. We collected data from three types of settings inhuman-computer interaction. Several multimodal behavioral features were coded and extracted based on
previous work [17, 31, 22, 5], and a linear regression model
achieved predictions of 0.50 in Listening 1 and 0.51
in Listening 2 of correlation coeﬃcients. We confirmedthat the correlation coeﬃcients of the two raters were0.46 (Listening 1) and 0.66 (Listening 2), and our prediction model achieved similar or slightly better predictions in Listening 1 and a previous work on speaking
skills [45]. For Listening 2, the human raters agreed
more than our prediction model. However, our study
did not consider such behavioral eﬀects as loud/clear
speech, smiling, or posture. We need to investigate andextract such additional information to improve our model[22]. We also found that the amount of backchannel
feedback was more important in Listening 2 than in Listening 1, probably because the former denotes a more
serious type of interaction that requires explicit cues toindicate that one is listening. For fully automated listening skills analysis, we need to evaluate each behavior
and our prediction model using backchannel recognitionand nod detection [26].
This study investigated evidence-based findings thatsuggest that the following aspects are critical for listening skills: 1) looking conversation partners in the eye,8

Hiroki Tanaka et al.

!""#$%"#&'(

)*+$,

)*+$2

)*+$1

)*+$31-./&0(

0456$!! "#$%&&'$(!! "#)*+,+-*$$./"#'$#0*1*//./
Elapsed time [s]"#*(.$234#40..+80!""#$%"#&'(

)*+$,
)*+$2

)*+$,3
)*+$2

)*+$1
)*+$4

)*+$35)*+$31-./&0(

0678$!! "#$%&&'$(!! "#)*+,+-*$$./"#'$#0*1*//./
Elapsed time [s]"#*(.$234#40..+80

Fig. 6 (a) Timing is displayed in Listening 1; (b) Timing is displayed in Listening 2. User nodding and backchannels, andagent’s speech: Graphs are lined horizontally. Rectangles show that nodding and backchannel occur in parallel.

2) nodding, and 3) repeating the keywords of conversation partners [5]. Our results confirmed that the second
and third points are important. Although the number ofrepetitions is not significantly diﬀerent in our study, we
identified greater values of repetition in people who canbe described as good models. However, the first point
was not found in this study. This diﬀerence might reflect interaction types. We did not find any significant
correlations in terms of eye fixation; the mean values
for every participant were 40% for the eyes, 7% for themouth, and 88% for the face. Most participants madeeye contact in the context of human-agent interactionthat was unrelated to listening skills.
This study did not control the training’s order effects. Since the participants were all trained by speaking and listening 1 before listening 2, listening 2 might
have been easier for them in the sense that they were already experienced. During the speaking task, the agent
expressed backchannels. This is important since agent
behavior might induce delayed imitation by the participants in the listening tasks. We will consider such ordereﬀects in future work.

Analysis of Conversational Listening Skills toward Agent-based Social Skills Training

Also, this study did not investigate the eﬀects ofhuman-agent interaction or human-human interaction
as well as cultural aspect. A previous work suggested
that people treat computers as real people and exhibit
politeness to them [37]. In contrast, another recent work
found that the dynamics of facial expressions diﬀer forusers interacting with a human and a virtual agent[32]. After the recording, one participant (ID 11) commented that human-agent interaction was safer thanhuman-human interaction because people are complex
and the feelings of conversational partners should be estimated in real time. As explained in the introduction,
people with social diﬃculties favor computerized environments because they are predictable and consistent(not complex). The major diﬀerence between Japaneseand American speakers was in the frequency and thediscourse contexts in which backchannels occurred [27].
A previous study also showed that American speakers
provided backchannel every 19-20 seconds. We need toconsider such cultural aspects in the future.
The present study was conducted with neurotypicalparticipants (without ASD). Thus training with multiple modalities was simplified. However, training participants with ASD with multisensory information is more
complicated because they might get overwhelmed by
the quantity and the multisensory nature of the stimuli/motor skills [42]. One possibility is to train each
modality individually to enable ASD users to iteratively
build upon the learned modalities one after the other.
We will integrate our listening-skills assessment into
an automation feedback framework [26, 45] and test it
on people with ASD to measure the eﬀects of the training on physiological attributes [49,46].
Acknowledgements This is a post-peer-review, pre-copyedit
version of an article published in the Journal on Multimodal
User Interfaces. The final authenticated version is available
online at: https://doi.org/10.1007/s12193-019-00313-y
Funding was provided by Core Research for Evolutional
Science and Technology (Grant No. JPMJCR19A5) and Japan
Society for the Promotion of Science (Grant Nos. JP17H06101
and JP18K11437).

References
1. Association,
A.P.:
Diagnostic
andStatistical
Manual of Mental Disorders: Dsm-5.
Diagnostic and Statistical Manual of Mental Disorders.
Amer Psychiatric Pub Incorporated (2013).
URL
https://books.google.co.jp/books?id=EIbMlwEACAAJ
2. Bandura, A.: Social learning theory of aggression. Journal of communication 28(3), 12–29 (1978)
3. Baron-Cohen, S., Richler, J., Bisarya, D., Gurunathan,N., Wheelwright, S.: The systemizing quotient: an investigation of adults with Asperger syndrome or highfunctioning autism, and normal sex diﬀerences. Philos.

9Trans. R. Soc. Lond., B, Biol. Sci. 358(1430), 361–374(2003)
4. Barry, J.G., Tomlin, D., Moore, D.R., Dillon, H.: Useof Questionnaire-Based Measures in the Assessment of
Listening Diﬃculties in School-Aged Children. Ear Hear
36(6), 300–313 (2015)
5. Bellack, A., Mueser, K., Gingerich, S., Agresta, J.: Social
Skills Training for Schizophrenia, Second Edition: A
Step-by-Step Guide. Guilford Publications (2013). URL
https://books.google.co.jp/books?id=TSMxAAAAQBAJ
6. Bohlander, A.J., Orlich, F., Varley, C.K.: Social skills
training for children with autism. Pediatric Clinics
of North America 59(1), 165 – 174 (2012). DOI
https://doi.org/10.1016/j.pcl.2011.10.001. Autism Spectrum Disorders: Practical Overview for Pediatricians
7. Cassell,
J.:
Embodied
conversational
agents:Representation and intelligence in user interfaces.
AI Mag. 22(4), 67–83 (2001).
URL
http://dl.acm.org/citation.cfm?id=567363.567368
8. Cigerci, F., Gultekin, M.: Use of digital stories to develop
listening comprehension skills 27, 252–268 (2017)
9. Constantino, J.N., Davis, S.A., Todd, R.D., Schindler,
M.K., Gross, M.M., Brophy, S.L., Metzger, L.M.,
Shoushtari, C.S., Splinter, R., Reich, W.: Validation ofa brief quantitative measure of autistic traits: comparison of the social responsiveness scale with the autism
diagnostic interview-revised. Journal of autism and developmental disorders 33(4), 427–433 (2003)
10. DeVault, D., Artstein, R., Benn, G., Dey, T., Fast,
E., Gainer, A., Georgila, K., Gratch, J., Hartholt,
A., Lhommet, M., Lucas, G., Marsella, S., Morbini,F., Nazarian, A., Scherer, S., Stratou, G., Suri, A.,
Traum, D., Wood, R., Xu, Y., Rizzo, A., Morency,
L.P.: Simsensei kiosk: A virtual human interviewer for
healthcare decision support. In: Proceedings of the2014 International Conference on Autonomous Agentsand Multi-agent Systems, AAMAS ’14, pp. 1061–
1068. International Foundation for Autonomous Agentsand Multiagent Systems, Richland, SC (2014). URL
http://dl.acm.org/citation.cfm?id=2617388.2617415
11. Duchowski, A.T.: Eye Tracking Methodology: Theory
and Practice. Springer-Verlag New York, Inc., Secaucus,
NJ, USA (2007)
12. Frith, U., Happe, F.: Autism spectrum disorder. Curr.
Biol. 15(19), R786–790 (2005)
13. Golan, O., Baron-Cohen, S.: Systemizing empathy: teaching adults with Asperger syndrome or high-functioning
autism to recognize complex emotions using interactive
multimedia. Dev. Psychopathol. 18(2), 591–617 (2006)
14. Gosling, S.D., Rentfrow, P.J., Swann, W.B.: A very brief
measure of the big-five personality domains. Journal of
Research in Personality 37, 504–528 (2003)
15. Gosling, S.D., Rentfrow, P.J., Swann Jr, W.B.: A very
brief measure of the big-five personality domains. Journalof Research in personality 37(6), 504–528 (2003)
16. Gratch, J., Wang, N., Gerten, J., Fast, E., Duﬀy, R.: Creating Rapport with Virtual Agents. In: Lecture Notes
in Artificial Intelligence; Proceedings of the 7th International Conference on Intelligent Virtual Agents (IVA),
vol. 4722, pp. 125–128. Paris, France (2007)
17. Heylen, D.: Listening heads.
In: Proceedings ofthe Embodied Communication in Humans andMachines, 2Nd ZiF Research Group International
ConferenceonModeling
Communication
withRobots and Virtual Humans, ZiF’06, pp. 241–259.
Springer-Verlag, Berlin, Heidelberg (2008).
URL
http://dl.acm.org/citation.cfm?id=1794517.1794530

10
18. Hoque, M.E., Courgeon, M., Martin, J.C., Mutlu, B., Picard, R.W.: Mach: My automated conversation coach.
In: Proceedings of the 2013 ACM International Joint
Conference on Pervasive and Ubiquitous Computing,
UbiComp ’13, pp. 697–706. ACM, New York, NY,
USA (2013).
DOI 10.1145/2493432.2493502.
URL
http://doi.acm.org/10.1145/2493432.2493502
19. Huang, L., Morency, L.P., Gratch, J.: Learning backchannel prediction model from parasocial consensus sampling: A subjective evaluation. In: J. Allbeck, N. Badler,
T. Bickmore, C. Pelachaud, A. Safonova (eds.) Intelligent
Virtual Agents, pp. 159–172. Springer Berlin Heidelberg,
Berlin, Heidelberg (2010)
20. Klin, A., Jones, W., Schultz, R., Volkmar, F., Cohen,
D.: Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence inindividuals with autism. Arch. Gen. Psychiatry 59(9),809–816 (2002)
21. KUDO, T.: Mecab : Yet another part-of-speech andmorphological analyzer. http://mecab.sourceforge.net/(2005). URL https://ci.nii.ac.jp/naid/10019716933/
22. Lala, D., Milhorat, P., Inoue, K., Ishida, M., Takanashi,K., Kawahara, T.: Attentive listening system withbackchanneling, response generation and flexible turntaking. In: Proceedings of the 18th Annual SIGdial
Meeting on Discourse and Dialogue, pp. 127–136. Association for Computational Linguistics (2017). URL
http://aclweb.org/anthology/W17-5516
23. Landis, J.R., Koch, G.G.: The measurement of observer
agreement for categorical data. Biometrics 33(1) (1977)
24. Lee, A., Oura, K., Tokuda, K.: Mmdagent - a fully opensource toolkit for voice interaction systems. In: ICASSP(2013)
25. Liu, C., Ishi, C.T., Ishiguro, H., Hagita, N.: Generationof nodding, head tilting and eye gazing for human-robot
dialogue interaction. In: ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 285–292(2012)
26. Liu, F., Surendran, D., Xu, Y.: Classification of statement
and question intonations in mandarin. Proc. 3rd Speech
Prosody pp. 603–606 (2006)
27. Maynard, S.K.: Conversation management in contrast:
Listener response in japanese and american english. Journal of Pragmatics 14(3), 397–412 (1990)
28. Maynard, S.K.: Kaiwa bunseki (discourse analysis) [written in japanese] (1993)
29. McKeown, G., Valstar, M., Cowie, R., Pantic, M.,
Schroder, M.: The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans.
Aﬀect. Comput. 3(1), 5–17 (2012). DOI 10.1109/TAFFC.2011.20.
URL http://dx.doi.org/10.1109/TAFFC.2011.20
30. Milne, M., Raghavendra, P., Leibbrandt, R., Powers,
D.M.W.: Personalisation and automation in a virtual
conversation skills tutor for children with autism. Journal
on Multimodal User Interfaces 12, 257–269 (2018)
31. Nori, F., Lipi, A.A., Nakano, Y.: Cultural diﬀerence in nonverbal behaviors in negotiation conversations: Towards a model for culture-adapted conversational agents.
In: Proceedings of the 6th International Conference on Universal Access in Humancomputer Interaction: Design for All and eInclusion - Volume Part I, UAHCI’11, pp. 410–419.
Springer-Verlag, Berlin, Heidelberg (2011).
URL
http://dl.acm.org/citation.cfm?id=2022591.2022639

Hiroki Tanaka et al.
32. Ochs, M., Libermann, N., Boidin, A., Chaminade, T.:Do you speak to a human or a virtual agent? automatic analysis of user’s social cues during mediated communication.
In: Proceedings of the 19th
ACM International Conference on Multimodal Interaction, ICMI 2017, pp. 197–205. ACM, New York, NY,
USA (2017).
DOI 10.1145/3136755.3136807.
URL
http://doi.acm.org/10.1145/3136755.3136807
33. Ochs, M., Mestre, D., de Montcheuil, G., Pergandi, J.M.,
Saubesty, J., Lombardo, E., Francon, D., Blache, P.:
Training doctors social skills to break bad news: evaluation of the impact of virtual environment displays on thesense of presence. Journal on Multimodal User Interfaces
13, 41–51 (2019)34. Okada, S., Ohtake, Y., Nakano, Y.I., Hayashi, Y., Huang,
H.H., Takase, Y., Nitta, K.: Estimating communication skills using dialogue acts and nonverbal featuresin multiple discussion datasets. In: Proceedings of the
18th ACM International Conference on Multimodal Interaction, ICMI 2016, pp. 169–176. ACM, New York,NY, USA (2016). DOI 10.1145/2993148.2993154. URL
http://doi.acm.org/10.1145/2993148.2993154
35. Poyade, M., Morris, G., Taylor, I., Portela, V.: Using mobile virtual reality to empower people with hidden disabilities to overcome their barriers. In: Proceedings ofthe 19th ACM International Conference on MultimodalInteraction, pp. 504–505. ACM, New York, NY, USA(2017). DOI 10.1145/3136755.3143025
36. Recht, S., Grynszpan, O.: The sense of social agency ingaze leading. Journal on Multimodal User Interfaces 13,
19–30 (2019)37. Reeves, B., Nass, C.I.: The media equation: How people treat computers, television, and new media like real
people and places. Cambridge university press (1996)
38. Sims, C.M.: Do the big-five personality traits predict empathic listening and assertive communication?
International Journal of Listening 31(3), 163–188(2017). DOI 10.1080/10904018.2016.1202770. URL
https://doi.org/10.1080/10904018.2016.1202770
39. Skinner, B.: Science And Human Behavior. Free Press
paperback. Psychology. New York, Macmillan (1953)
40. Sveinbjornsdottir, B., Johannsson, S.H., Oddsdottir, J.,
Siguroardottir, T.P., Valdimarsson, G.I., Vilhjalmsson,
H.H.: Virtual discrete trial training for teacher trainees.
Journal on Multimodal User Interfaces 13, 31–40 (2019)41. Tanaka, H., Adachi, H., Ukita, N., Ikeda, M., Kazui, H.,
Kudo, T., Nakamura, S.: Detecting dementia throughinteractive computer avatars. IEEE Journal of Translational Engineering in Health and Medicine 5, 1–11
(2017). DOI 10.1109/JTEHM.2017.2752152
42. Tanaka, H., Negoro, H., Iwasaka, H., Nakamura,S.: Embodied conversational agents for multimodal
automated social skills training in people with
autism spectrum disorders.
PLOS ONE 12(8), 1–
15 (2017). DOI 10.1371/journal.pone.0182151. URL
https://doi.org/10.1371/journal.pone.0182151
43. Tanaka, H., Negoro, H., Iwasaka, H., Nakamura,S.: Listening skills assessment through computer
agents.
In: Proceedings of the 20th ACM International Conference on Multimodal Interaction,
ICMI ’18, pp. 492–496. ACM, New York, NY,
USA (2018).
DOI 10.1145/3242969.3242970.
URL
http://doi.acm.org/10.1145/3242969.3242970
44. Tanaka, H., Sakriani, S., Neubig, G., Toda, T., Negoro, H., Iwasaka, H., Nakamura, S.: Teaching social communication skills through human-agent interaction.
ACM Trans. Interact. Intell. Syst. 6(2),Analysis of Conversational Listening Skills toward Agent-based Social Skills Training
18:1–18:26 (2016).
DOI 10.1145/2937757.
URL
http://doi.acm.org/10.1145/2937757
45. Tanaka, H., Sakti, S., Neubig, G., Toda, T., Negoro, H., Iwasaka, H., Nakamura, S.: Automated social skills trainer.
In: Proceedings of the 20th
International Conference on Intelligent User Interfaces, IUI ’15, pp. 17–27. ACM, New York, NY,
USA (2015).
DOI 10.1145/2678025.2701368.
URL
http://doi.acm.org/10.1145/2678025.2701368
46. Tanaka, H., Watanabe, H., Maki, H., Sakriani, S., Nakamura, S.: Electroencephalogram-Based Single-Trial Detection of Language Expectation Violations in Listening
to Speech. Front Comput Neurosci 13, 15 (2019)47. Tsai, M.N., Wu, C.L., Tseng, L.P., An, C.P., Chen, H.C.:
Extraversion Is a Mediator of Gelotophobia: A Study ofAutism Spectrum Disorder and the Big Five. Front Psychol 9, 150 (2018)
48. Tyagi, B.: Listening: An important skill and its various
aspects. The Criterion An International Journal in English 12, 1–8 (2013)
49. Van Hecke, A.V., Stevens, S., Carson, A.M., Karst, J.S.,
Dolan, B., Schohl, K., McKindles, R.J., Remmel, R.,
Brockman, S.: Measuring the plasticity of social approach: a randomized controlled trial of the eﬀects of the
PEERS intervention on EEG asymmetry in adolescentswith autism spectrum disorders. J Autism Dev Disord
45(2), 316–335 (2015)
50. Veltman, K., de Weerd, H., Verbrugge, R.: Training theuse of theory of mind using artificial agents. Journal on
Multimodal User Interfaces 13, 3–18 (2019)
51. Ward, N.G., Escalante, R., Bayyari, Y.A., Solorio,
T.: Learning to show you’re listening.
Computer Assisted Language Learning 20(4), 385–407(2007).
DOI 10.1080/09588220701745825.
URL
https://doi.org/10.1080/09588220701745825
52. Zhao, R., Li, V., Barbosa, H., Ghoshal, G., Hoque,
M.E.: Semi-automated 8 collaborative online training
module for improving communication skills.
Proc.
ACM Interact. Mob. Wearable Ubiquitous Technol.
1(2), 32:1–32:20 (2017). DOI 10.1145/3090097. URL
http://doi.acm.org/10.1145/3090097

11