Proceedings of APSIPA Annual Summit and Conference 201516-19 December 2015

Synchrony in Prosodic and Linguistic Features
between Backchannels and Preceding Utterances in
Attentive Listening
Tatsuya Kawahara, Takashi Yamaguchi, Miki Uesato, Koichiro Yoshino, Katsuya Takanashi
School of Informatics, Kyoto University,
Sakyo-ku, Kyoto 606-8501, Japan
Abstract—In human-human dialogue, especially in attentive
listening such as counseling, backchannels play an importantrole. Appropriately coordinated backchannels will not only makesmooth communication but also help establish rapport. Bycollecting counseling dialogue, we investigate whether and how
synchrony is expressed by prosodic and linguistic features ofbackchannels with respect to the preceding speaker’s utterances.
First, we find out correlation patterns according to the typeof backchannels and prosodic features; a larger correlation isobserved for reactive tokens than acknowledging tokens andfor the power features than the pitch features. Next, we investigate the relationship between the morphological complexityof backchannels and the syntactic complexity of the precedingclause/sentence unit. The result can be useful for generating avariety of backchannels adaptive to the speaker’s utterances.

Index Terms: dialogue, backchannel, prosody
I. I NTRODUCTION
In recent years, a number of spoken dialogue systems havebeen deployed in smart phones and car navigation systems to
conduct simple tasks and information retrieval. These systems
basically assume that a user makes one utterance per one
turn, which is responded by the system, and the utterances
are not supposed to overlap each other. This “half duplex”
communication mode is much different from that of humanhuman dialogue, in which one person often has a long speech
turn consisting of many utterances while the listener gives anoccasional feedback within the turn. Feedback behaviors play
an important role in smooth communication [1]. In speechcommunication or spoken dialogue, verbal backchannels, suchas “okay” and “right” in English, convey feedback. Backchannels are used to express the listener’s feedback to what isuttered while suggesting that the current speaker can keep
the dialogue turn. Specifically, backchannels suggest that thelistener is listening, understanding, and agreeing to the speaker.
In addition to the effect of individual backchannels,
backchannels make a “rhythm” of the dialogue as a whole.
By making “synchrony”, dialogue partners feel comfortablein keeping the dialogue. The phenomenon is regarded as one
aspect of entrainment [2]. In counseling, it is crucial for acounselor to keep the client talking on his/her matter by establishing rapport. To that end, counselors make effective use of
backchannels to express empathy and make synchrony in the
dialogue [3]. The work presented in this paper focuses on thesynchrony effect of backchannels rather than their individual

978-988-14768-0-7©2015 APSIPA

role and meaning. Specifically, we first investigate whether
prosodic synchrony is observed in generating backchannels
in counseling dialogue by analyzing the relationship between
prosodic features of the backchannels and those of preceding utterances by the speaker. Next, we investigate anothersynchrony effect by analyzing the relationship between themorphological structure of backchannels and the syntactic
structure of the preceding utterances.
The finding of this work would be useful for designinga new kind of spoken dialogue systems or conversational
agents, which conduct attentive listening to particular user
populations such as elderly and ill persons. The current spoken
dialogue systems usually generate the same or limited patternsof backchannels in terms of lexical choice and prosodic
parameters. However, in order to make smooth communication by establishing rapport, it is very important to generate
appropriate backchannels adaptive to the speaker’s utterances.
Note that speech recognition and understanding may not benecessary to realize this function.
II. A NALYSIS AND G ENERATION OF BACKCHANNELS
A verbal backchannel is a short response generated by thelistener during the dialogue, usually at the end of utterances,
without taking a turn; instead, backchannels suggest that thelistener does not take a turn. By this definition, backchannels
are distinguished from acknowledgment and fillers, which areused to take or keep a turn in the dialogue.
In generating or analyzing backchannels, we need to determine or identify the following three factors: timing (when),prosody (how) and lexical entry (what).
A. Timing
Timing of backchannels is usually constrained at the endof the current speaker’s utterances, but whether to make abackchannel is determined by a number of factors.
There are a number of previous studies that investigated thecues of backchannels, or when to make a backchannel. As
early work, Ward et al. [4], [5] pointed out the low pitch as amajor prosodic cue of backchannels. Koiso et al. [6] introduced
a decision tree to derive rules from prosodic and morphologicalpatterns.
There are also several studies which actually implemented
a dialogue system to generate backchannels using a decision

392

APSIPA ASC 2015

Proceedings of APSIPA Annual Summit and Conference 2015

tree [7]. Recently, more elaborate discriminative modeling andan efficient learning mechanism using the wisdom of crowds
have been introduced [8], [9].
Although timing is an important issue to generate backchannels, it is not the focus of this work.

16-19 December 2015

apparently difficult to distinguish “un” from “hun” and also“u:n” from “hu:n”, we deal with them collectively. They are
clustered based on whether they are prolonged (perceptually)
and the number of the repetitions, and represented, for example, by “un x2” or “(un)+”.
IV. A NALYSIS ON P ROSODIC S YNCHRONY BETWEENBACKCHANNELS AND P RECEDING U TTERANCES

B. Lexical entry
The lexicon of backchannels is language-dependent. In
general, morphological patterns of backchannels are classified
into two categories. One is lexical tokens, and they are usuallysame entries as acknowledging tokens such as “okay” and“right” in English and “hai” and “un” in Japanese. In Japanese,
their lexical patterns are limited, but can be repeated, forexample, “un un un”. The other category is reactive tokens,which are often non-lexical, such as “wow” in English and“he:” in Japanese. The acknowledging tokens are more frequently used and they indicate that the listener is listening andunderstanding, while the reactive tokens are specially used toindicate the listener’s strong reaction and assessment to what
is uttered.
Although the role of individual tokens has been discussed
in relation with the listener’s state of mind, there is almost no
previous work on the choice of lexical patterns according to
the context of the preceding utterance by the speaker.
C. Prosody
Prosody of backchannels is important especially for expressing assessment, and we have identified particular patterns to
express interest and surprise in conversations [10], [11]. On theother hand, the general prosodic patterns of backchannels havenot been carefully investigated, compared to the prosodic cuesof backchannels. Actually, almost all systems that generate
backchannels mentioned above use the same recorded orsynthesized backchannel pattern “okay” or “hai”, which givesa monotonic impression.
III. C ORPUS OF C OUNSELING D IALOGUE
In order to conduct an analysis of attentive listening anddevelop a prototype system of such function, we have recorded
sessions of counseling dialogue. These are not real counseling,in that the subjects were asked to come to the session for
dialogue data collection, not for counseling. But they wereasked to talk about their real personal troubles, for example,
human relationship and career path, to a counselor. The
subjects are eight college students of 20 to 25 years old. We
had two counselors (one male of 7-year counseling experienceand one female of 4-year experience), and each took part infour sessions. All participants are Japanese native.
The dialogue started with some chatting and the following
counseling session lasted around 20-30 minutes. It is observedthat the counselors make backchannels every 5 to 7 seconds,
that is at almost every end of the speaker’s utterances.
The statistics by the lexical entries show that a large
majority of backchannels are acknowledging tokens such as“hun”, “u:n”, “un”, “hu:n”, “hun hun”, “un un un”, “un un”
in the descending order of the occurrence count. Since it is978-988-14768-0-7©2015 APSIPAFirst, we investigate synchrony in prosodic patterns expressed by the listener’s backchannels with respect to thepreceding speaker’s utterances.
A. Prosodic Features
We focus on the prosodic features of the speaker’s utterances
preceding the backchannels of the counselor. There are many
overlapping cases between them, but each segment of 500
msec from the end of the utterance was analyzed using the
speech data captured by the head-set microphone.
Fundamental frequency (F0) was computed with a frame
shift of 10 msec using wavesurfer 1.8, 1 then it was converted
to the logarithm scale and normalized with the mean and thestandard deviation computed per person for the entire session.
The final value is referred to as z-score. Power (in dB) was
also computed using wavesurfer 1.8 and normalized in thesame manner.
B. General Synchrony in Prosodic Features
We first investigate the general tendency of synchrony,which was reported by Heldner et al. [12]. The synchrony
is measured by comparing with normal turn-taking. To thisend, we also computed the prosodic features (mean log F0
and power) for the beginning segment of 500 msec when
the counselor takes a turn to ask some questions or make acomment.
We used 952 samples of backchannels and 279 samplesof normal turn switches in this experiment. Since counselors
are mostly engaged in attentive listening, their turn-taking is
not frequent. We measured the distance between the prosodic
feature of these segments and that of the preceding utterances(i.e. backchannel vs. preceding utterance & turn-taking vs.
preceding utterance), as shown in Figure 1.
Heldner et al. [12] reported that the distance of F0 between
backchannels and their preceding utterances is significantly
smaller than that of normal turn switches, that is, pitch of
backchannels is closer to that of the preceding utterances
compared with normal turn-taking. However, this phenomenon
is not confirmed in our corpus. There is not a significant
difference in F0 between these two cases (left graph of
Figure 1). Instead, we observe a significant difference (pvalue< 0.05) in the power feature; power of backchannels
is much closer to that of the preceding utterances comparedwith normal turn-taking (right graph of Figure 1).
Although we do not have a clear explanation for the results,
the difference in general prosodic patterns between English
1 http://www.speech.kth.se/wavesurfer

393

APSIPA ASC 2015

Proceedings of APSIPA Annual Summit and Conference 2015

Fig. 1.
Difference in prosodic features from the preceding utterances(comparison of backchannels and normal turn-taking)16-19 December 2015

“un un”, while a small correlation is found in the F0 featureof short backchannels of “un” and “u:n”. It suggests thatthe listener can easily control the power parameter in long
backchannels.
Large correlation patterns are observed for the reactive
tokens of “a:” and “ha:” although there are a small number ofthese samples. It is natural as they are used to express strong
reaction to the speaker.
In summary, power is adjusted to make synchrony inrepeated tokens, while pitch plays some role in short tokens,and both are used in reactive tokens.

TABLE IC ORRELATION OF PROSODIC FEATURES BETWEEN BACKCHANNELS AND

V. A NALYSIS ON M ORPHOLOGICAL PATTERNS OFBACKCHANNELS AND S YNTACTIC F EATURES OF
P RECEDING U TTERANCESTHE PRECEDING UTTERANCES

morph. pattern
u:n

count
225

log F0max (0.14)

un

361

max (0.22)
mean (0.12)

un x2
un un
un x3
un un un
un x4
un un un un
a:

146

ha:

27

mean (0.20)
169
117

power
max (0.14)
mean (0.18)
max (0.23)
max (0.34)
mean (0.35)
max (0.32)
mean (0.30)
max (0.24)
mean (0.22)

28
mean (0.22)
mean (0.23)

mean (0.25)
max (0.47)
mean (0.29)

and Japanese and also the difference in segmental and prosodic
patterns of backchannels may be attributed.
C. Correlation of Prosodic Features between Backchannels
and Preceding Utterances
Next, we investigate the correlation of the prosodic featuresof backchannels and those of the preceding utterances. This
will reveal more precisely whether and how synchrony isrealized in generating backchannels. The analysis is conducted
for each category (acknowledging tokens and reactive tokens)and for each clustered morphological pattern, but those withfewer occurrence counts (less than 25) are not used.
We compute F0 and power in the same manner as in
the previous sub-section, and parameterize them with theirmean, maximum (max) and range within the segment. Then, acorrelation coefficient is computed between the parameter ofbackchannels and that of the preceding utterances.
The results of the correlation analysis are presented inTable I. Here we list those with significant correlation (pvalue< 0.05) for the acknowledging tokens (upper part ofTable I) and those larger than 0.20 for the reactive tokens(lower part of Table I) since the latter does not have a large
number of samples. We can see more correlation patterns withregard to the power feature of backchannels. This confirms
the result of the previous sub-section: synchrony is observedfor power rather than pitch. Larger correlation patterns areobserved in the power feature of repetition patterns such as

978-988-14768-0-7©2015 APSIPA

Next, we investigate the relationship between morphologicalpatterns of backchannels and syntactic features of precedingutterances. Here, we hypothesize that the lexical choice ofbackchannels is affected by the syntactic patterns of thepreceding utterances, and that, especially in Japanese, the repetition patterns are affected by the linguistic complexity of thepreceding utterances; more complex morphological patterns
are used when the preceding utterance is long, complex or at
the end of complete sentences. This realizes a rhythmic effectwith respect to the linguistic features.
The utterances (IPUs) are chunked into clause and sentence
units according to the guideline of the CSJ [13]. The boundaries are annotated with three types: sentence boundary, strongclause boundary, and weak clause boundary. While strongclause boundaries usually appear between parallel clauses,
weak clause boundaries are defined before/after the clausesthat depend on other clauses, such as those starting with“because”.
A. Relationship between Boundary Type and Morphological
Patterns of Backchannels
The frequency distribution of the backchannel clusters according to the boundary type is plotted in Figure 2. Here, “unx4” is merged into “un x3”.
A clearly different tendency is observed between the sentence boundary and the two clause boundary types. While
simple morphological patterns of backchannels, mostly one or
two repetitions of acknowledging tokens, “un” and “un un”,
are more often used in the clause boundaries, the sentenceboundaries are more likely to be followed by reactive tokens.
Simple patterns are preferred to encourage the speaker tokeep talking, and reaction with prominent prosodic patterns,
as discussed in the previous section, are expressed after the
speaker completes a sentence. This suggests that stronger
reactions are made in stronger boundaries. However, there isno clear distinction between the weak clause boundary andthe strong clause boundary. It is also infeasible to control thenumber of repetitions with this feature.

394

APSIPA ASC 2015

Proceedings of APSIPA Annual Summit and Conference 2015

Fig. 2.
Frequency distribution of backchannel clusters according to theboundary type of the preceding utterances.
TABLE IIS TATISTICS OF SYNTACTIC FEATURES IN THE PRECEDING
CLAUSE / SENTENCE UNIT

no. of phrases
depth of parse tree
width of parse tree

un
4.73
2.18
1.88

un x2
5.52
2.57
2.00

un x3 x4
5.42
2.56
1.89

reactive
5.15
2.54
1.75

B. Relationship between Syntactic Complexity and Morphological Patterns of Backchannels
We also conduct a morphological and syntactic analysis
on the preceding clause/sentence unit and investigate therelationship between the linguistic features and the morphological patterns of the backchannels. Specifically, we count the
number of bunsetsu phrase units and the depth and the widthof the parse tree generated by a Japanese syntactic parser KNP.
2
The width means how many phrases depend on the verb, andthe depth means how many dependencies at maximum exist
before the verb. These measures show the complexity of theclause/sentence unit.
The statistics according to the backchannel clusters areshown in Table II. The difference in the first two measures
between “un” and “un x2”, shown in bold fonts, are statistically significant (p-value< 0.05). Interestingly, these two
short acknowledging tokens cannot be distinguished in theanalysis in the previous subsection, but it is reasonable that
repeated tokens are used when the number of phrases is larger
or the depth of the parse tree is deeper, which involves morecomponents in the unit. This is regarded as a synchrony withregard to the linguistic patterns.
Overall, the hypothesis on the relationship between thecomplexity of the preceding utterances and the complexity ofthe morphological patterns of the backchannels is verified tosome extent.
VI. C ONCLUSIONS
We have investigated whether and how synchrony is expressed by prosodic and linguistic features of backchannels
2 http://nlp.ist.i.kyoto-u.ac.jp/index.php?KNP

978-988-14768-0-7©2015 APSIPA

16-19 December 2015

with respect to the preceding speaker’s utterances. To thisobjective, we recorded counseling sessions, in which a counselor conducts attentive listening by generating backchannels
frequently and carefully.
The major findings in this work are summarized as follows:• There is a different tendency between acknowledging
tokens and reactive tokens. The reactive tokens are more
likely to have synchrony.
• In addition to F0, the power feature plays an importantrole. Specifically, the power feature tend to have morecorrelation patterns for repeated tokens and reactive tokens.
• The morphological complexity of backchannels is also
related with the syntactic complexity of the precedingclause/sentence unit.
The findings will be useful for appropriately determining
the lexical choice and the prosodic parameters of backchannels
generated by a conversational system to make it more naturaland friendly to users. For example, when the speaker utters alonger clause, a backchannel should be repeated, and when asentence is completed, a reactive token should be made, withpower adjusted to the speaker’s voice. We plan to develop thiskind of system and evaluate with human subjects.
Acknowledgment: This work was supported by JST ERATO
Ishiguro Symbiotic Human-Robot Interaction program.
R EFERENCES[1] N.Ward, D.Novick, L.P.Morency, T.Kawahara, D.Heylen, and J.Edlund,
editors. Proc. Interdisciplinary Workshop on Feedback Behaviors in
Dialog, 2012.
[2] R.Levitan and J.Hirschberg. Measuring acoustic-prosodic entrainment
with respect to multiple levels and dimensions. In Proc. InterSpeech,
pages 3081–3085, 2011.
[3] B.Xiao, P.G.Georgiou, Z.E.Imel, D.Atkins, and S.Narayanan. Modelingtherapist empathy and vocal entrainment in drug addiction counseling.
In Proc. InterSpeech, pages 2861–2864, 2013.
[4] N.Ward. Using prosodic clues to decide when to produce back-channelutterances. In Proc. ICSLP, pages 1728–1731, 1996.
[5] N.Ward and W.Tsukahara. Prosodic features which cue back-channel
responses in English and Japanese. J. Pragmatics, 32(8):1177–1207,
2000.
[6] H.Koiso, Y.Horiuchi, S.Tutiya, A.Ichikawa, and Y.Den. An analysis ofturn-taking and backchannels based on prosodic and syntactic featuresin Japanese Map Task dialogs. Language & Speech, 41(3-4):295–321,
1998.
[7] N.Kitaoka, M.Takeuchi, R.Nishimura, and S.Nakagawa. Response
timing detection using prosodic and linguistic information for humanfriendly spoken dialog systems. J. Japanese Society for Artificial
Intelligence, 20(3):220–228, 2005.
[8] Y.Kamiya, T.Ohno, and S.Matsubara. Coherent back-channel feedback
tagging of in-car spoken dialogue corpus. In Proc. SIGdial, 2010.
[9] D.Ozkan and L.-P.Morency. Modeling wisdom of crowds using latentmixture of discriminative experts. In Proc. ACL/HLT, 2011.
[10] T.Kawahara, Z.Q.Chang, and K.Takanashi. Analysis on prosodic features of Japanese reactive tokens in poster conversations. In Proc. Int’l
Conf. Speech Prosody, 2010.
[11] T.Kawahara, S.Hayashi, and K.Takanashi. Estimation of interest andcomprehension level of audience through multi-modal behaviors inposter conversations. In Proc. InterSpeech, pages 1882–1885, 2013.
[12] M.Heldner, J.Edlund, and J.Hirschberg. Pitch similarity in the vicinityof backchannels. In Proc. InterSpeech, pages 3054–3057, 2010.
[13] K.Maekawa. Corpus of Spontaneous Japanese: Its design and evaluation.
In Proc. ISCA & IEEE Workshop on Spontaneous Speech Processing andRecognition, pages 7–12, 2003.

395

APSIPA ASC 2015