Reducing Speech Collisions by Using an Artiﬁcial Subtle
Expression in a Decelerated Spoken Dialogue– Should communication robots respond quickly? –Kotaro Funakoshi 1 and Kazuki Kobayashi 2 and Mikio Nakano 3
and Takanori Komatsu 4 and Seiji Yamada 5
Abstract. We argue that spoken dialogue systems or communication robots do not need to quickly respond verbally as long as theyquickly respond non-verbally by showing their internal states by using an artiﬁcial subtle expression. This paper describes an experiment
whose results support this point. In this experiment, 48 participants
engaged in reservation tasks with a spoken dialogue system coupledwith an interface robot using a blinking light expression. The expression is designed as an artiﬁcial subtle expression to intuitively notify
a user about a robot’s internal states (such as processing) for the sakeof reducing speech collisions as consequences of turn-taking failures due to end-of-turn misdetection. Speech collisions harm smoothspeech communication and degrade system usability. Two experimental factors were setup: the blinking light factor (with or withouta blinking light) and the reply speed factor (moderate or slow reply
speed), resulting in four experimental conditions. The results suggest that the blinking light expression can reduce speech collisions
and improve user impression, and surprisingly that users do not careabout slow replies.

1

Introduction

An important issue in spoken dialogue systems is the managementof turn-taking. Failures of turn-taking due to systems’ end-of-turn
misdetection cause undesired speech collisions, which harm smoothcommunication and degrade system usability. Such speech collisions
lead users to stop speaking and bring about troubles in spoken dialogue systems [11] because interrupted speech is hard to recognize
automatically and dialogue states after collisions are unclear. Deterioration in user impressions is also a problem.
There are two approaches to reducing speech collisions due to endof-turn misdetection. The ﬁrst approach is using machine learningtechniques to integrate information from multiple sources for accurate end-of-turn detection in early timing. The second approach is tomake a long interval after the user’s speech signal ends and before
the system replies simply because a longer interval means no continued speech comes. As far as the authors know, all the past worktakes the ﬁrst approach (e.g., [2, 8, 12, 14]) because the second approach deteriorates responsiveness of dialogue systems. This choice
1 Honda Research Institute Japan Co., Ltd., 8-1 Honcho Wako, Saitama 351-

0188, Japan, email: funakoshi@jp.honda-ri.com
2 Shinshu University, Japan
3 Honda Research Institute Japan Co., Ltd., Japan
4 Shinshu University, Japan
5 National Institute of Informatics / The Graduate University for Advanced

Studies, Japanis based on the presumption that users prefer a responsive system
to less responsive systems. The presumption is true in most cases if
the system’s performance is at human level. However, if the system’s
performance is below human level, high responsiveness might not bevital or even be harmful. For instance, a user study reported that thefamiliarity of a spoken dialogue system with back-channel feedback
was inferior to that without feedback due to a small portion of errors
even though the overall timing and frequency of feedback were fairly
good (but did not come up to human operators) [8]. Technologies areadvancing but they are still below the human level. We challenge tothe past work that took the ﬁrst approach.
The second approach is simple and stable against user differences and environmental changes. Moreover, it can afford to employ
more powerful but computationally expensive speech processing or
to build systems on small devices with limited resources. A concern
with this approach is debasement of user experience due to poor responsiveness as stated above. Another issue is speech collisions due
to users’ following-up utterances such as repetitions because systems’ late responses tend to induce such utterances, which are notdesired from the viewpoint of systems.
Taking the second approach, we showed the possibility that nonspeech feedback using a blinking light based on the concept of artiﬁcial subtle expressions (see the next section) can suppress undesiredutterances (repetitions) from users [4]. However, our experiment was
not with an automatic spoken dialogue system but with a humanoperated Wizard-of-Oz system. We showed that our method reduced
repetitions (potential causes of collisions) but did not show that our
method reduced collisions. We also showed that the blinking lightfeedback improved user impression but did not show whether the
improvement in user impression was enough to compensate for thedebasement of user experience due to slow replies.
This paper shows the results of the experiment in which participants engaged in hotel reservation tasks with a spoken dialogue system equipped with an artiﬁcial subtle expression-based method proposed in [4], which intuitively notiﬁed a user about the system’s internal states (such as processing or busy). The results suggest thatthe method can reduce speech collisions and provide users with a
comfortable impression and a modesty impression. The comparisonsof user evaluations between systems with a slow reply speed and amoderate reply speed suggest that users of spoken dialogue systems
do not care about slow replies. These results indicate that taking thesecond approach, decelerating spoken dialogues, is not a bad idea.
Section 2 explains artiﬁcial subtle expressions. The experimentand results are described in Section 3 and Section 4 respectively. Sec-tion 5 concludes this paper.

2

Blinking Light as Artiﬁcial Subtle Expression

Although human communication is explicitly achieved through verbal utterances, non-verbal facial expressions, gaze, gestures, etc.,
also play an important role [6, 7]. Such non-verbal communication
often inﬂuences the accuracy of utterance understanding [15].
Furthermore, researchers have reported that very small changes(called subtle expressions) in facial expressions and gestures might
inﬂuence human communication. We believe that we can utilize such
subtle expressions to make humans easily understand a robot’s internal state because humans can intuitively understand subtle expressions. Some studies have been done on applying subtle expressions
to human-agent interaction [1, 3, 13]. However, since they tried toenable subtle expressions on real faces and with real arms, their implementations were considerably expensive.
Equipping a robot to express its turn-taking intention by using
body/eye movements as humans do may reduce speech collisions.
Though, such an approach is technically difﬁcult and uneconomical
as mentioned above. Using spoken back-channel feedback (such as“well” and “uh”) is another option, however, it is not an easy matterbecause such buck-channel feedback expressions are not arbitrarily
used but require appropriate timing and situations to be used [18, 19].
Moreover, such approaches strongly restricts characters of robots,
therefore, their applicabilities inevitably go low. Cultural differences
also affect them.
In contrast with such human-like approaches, subtle expressions
have been studied for artifacts like a robot or PC. Komatsu and Yamada [9] reported that an agent’s subtle expression of simple beeping
sounds with decreasing/increasing frequency enabled humans to interpret the agent’s positive/negative states. Their work indicated theeffectiveness of subtle expressions such as varying beeping sounds
for a robot or agent. They named such subtle expressions “artiﬁcial
subtle expressions (ASEs)” and deﬁned them as expressions fulﬁlling the following four requirements [10].
• Simple: ASEs should be implemented on a single modality. It is
expected that implementation cost also should be lower.
• Complementary: ASEs should only have a complementary rolein communication and should not interfere with communication’s
main protocol. This means that the ASEs themselves do not haveany meaning without any communication context.
• Intuitive: ASEs should be understood by humans who do notknow about the ASEs beforehand.
• Accurate: ASEs should convey the speciﬁc meanings accurately.
Speciﬁcally, ASEs should convey the internal states of the artifact
like subtle expressions are doing.
Following their work, we proposed the use of a blinking light as ameans of artiﬁcial subtle expression to intuitively notify a user abouta robot’s internal states (such as processing or busy) and showedthat the blinking light expression potentially can reduce speech collisions, however, indirectly by showing the reduction of users’ speech
repetitions in a last-and-ﬁrst game dialogue [4]. This paper veriﬁes
the effectiveness of the blinking light expression in a more practicaltask-oriented dialogue with an automatic spoken dialogue system todirectly show this approach reduces collisions.
The use of a blinking light may not be the best way in terms of
reducing speech collisions in comparison with other human-like approaches explained above. However, it will be easy and cheap to im-

B

l

i

n

k

i

n

g

L

E

D

Figure 1. The LCD monitor and robot with an embedded LED

plement, and applicable to wider conversational agents/devices not
limited to robots.

3 Experiment
We conducted an experiment in which 48 participants engaged in hotel reservation tasks with a conversational robot. The dialogue system, robot, blinking light, experimental conditions and method are
explained below.

3.1 Spoken Dialogue System
A spoken dialogue system (not a voice command system) that canhandle user requests in a hotel reservation domain was built. The system was equipped with an LCD monitor to show reservation information and an interface robot with an LED attached to its chest (seeFigure 1). Participants’ utterances were recognized by a free automatic speech recognizer Julius6 , and interpreted by using a domainontology-centered language understanding method [5]. The robot’sutterances were voiced by a commercial speech synthesizer (NTTIT FineVoice [17]). The LCD monitor was used to reduce the time inwhich the system just read up reservation details and to reduce participants’ cognitive loads to catch the system’s lengthy speech by showing reservation details on the monitor only when the system asked for
conﬁrmation.
The system was tuned to uplift its speech understanding performance as much as possible. For example, the language model usedby Julius was built using the data collected in the same task domain
in advance (4223 utterances / 140-minute speech of 47 users).
Default values were used for the parameters in Julius. Julius output a recognition result (a sequence of recognized words) to the dialogue system at 400 msec after an input speech signal ended, but the
dialogue system awaited the next input for a ﬁxed interval (we callthis interval wait interval, whose length is given as an experimental
factor). If the system received an additional input, it awaited the next
input for the same interval again. Otherwise, the system concatenatedrecognition results, made an interpretation, and replied.

3.2 Robot and Blinking Light Expression
We adopted the same robot (WowWee RS-Media) and the same red
LED (diameter: 4 mm) as [4]. The blinking pattern was also the same
6 http://julius.sourceforge.jp/

detected speech onset
user speech VAD tail margin
short pausesαdetected end-of-turnwait interval
processing delaysystem speech

USER
SYSTEM

Y

X

blinking LED
Figure 2. Behavior of the dialogue system along a timeline(1/30 sec even-intervals). The LED started blinking when a speech
signal was detected and stopped when the system started replying.
The basic function of the blinking light expression is similar tohourglass icons used in GUIs. A big difference is that basically GUIs
can ignore any input while they are showing those icons, but dialoguesystems must accept successive speech while it is blinking an LED.
What we intend to do is to suppress only collision-eliciting speechsuch as repetitions and rephrasings which are negligible but difﬁcult
to automatically distinguish from barge-ins when speech collisions
happen (we call them follow-ups). Barge-ins are users’ intentional
interruptions to systems’ speech and dialogue systems must accept
them.

3.3 Experimental Conditions and ParticipantsTwo experimental factors each having two levels were setup, that is,the blinking light factor (with or without a blinking light) and the
reply speed factor (moderate or slow reply speed), resulting in four
experimental conditions A, B, C, and D as below.
Condition A: slow reply speed, with a blinking light,
Condition B: slow reply speed, without a blinking light,
Condition C: moderate reply speed, with a blinking light,
Condition D: moderate reply speed, without a blinking light
We prepared a dialogue system for each group, four systems in
total and randomly assigned 48 Japanese participants of 18 to 55
years old (mean age 30.9, SD = 10.2) to one of the four conditions, while genders and generations were controlled to be equallydistributed among the conditions.
A reply speed depends on a wait interval for which the dialoguesystem awaits the next input. A user study [16] showed that the best
reply speed for a conversational robot was one second. Thus wechose 800 msec as the wait interval for the moderate reply speed
because an actual reply speed was the accumulation of the wait interval and a delay for processing a user request, and 800 msec is simply
twice the default length (the VAD tail margin) by which the Julius
speech recognizer recognizes the end of a speech. For the slow reply speed, we chose 4000 msec as the wait interval. Wait intervals
include the VAD tail margin.
Figure 2 shows how the system and the LED works along with user
speech. In this ﬁgure, a user utters a continuous speech with a rather
long pause that is longer than the VAD tail margin but shorter than thewait interval. If the system detects the end of the user’s turn and starts
speaking within the interval marked with an ‘X’, a speech collision
would occur. If the user utters a follow-up within the interval marked
with a ‘Y’, a speech collision would occur, too. We try to suppress
the former speech collision by decelerating dialogues and the latterby using a blinking light.

3.4 Experimental Method
The experiment was conducted in a room for one participant at one
time. Participants entered the room and sat on a chair in front of adesk as shown in Figure 1. They were asked to wear headphones toseparate system voice and user voice. All dialogues were videotaped.
The experimenter gave the participants instructions so as to reserve
hotel rooms ﬁve times by talking with the robot in front of them. Allof them were given the same ﬁve tasks which require them to reserve
several rooms (one to three) at the same time. The meaning of theblinking light expression was not explained to them. After giving the
instructions, the experimenter left the participants, and they begantasks when the robot started to talk to them. Each task was limited
up to three minutes. The experimenter aborted dialogues if they could
not ﬁnish a given task in three minutes. They worked on the ﬁve tasks
without intermission in a ﬁxed order. For illustration, Figure 3 shows
the instruction of one of the tasks, and Figure 4 shows an example
dialogue observed with this task.
After ﬁnishing the tasks, the participants answered a questionnaire
about their impressions of the dialogue and robot, and other particular questions. Participants were paid for their participation.

4 Results
4.1 Observed reply speeds
All dialogues were transcribed into texts with timestamps. Averagesof observed reply speeds including the system’s delays were calculated from those timestamps. They were 4.53 sec (SD = .75) for theslow conditions (Aand B) and 1.42 sec (SD = .43) for the moderateconditions (Cand D).

4.2 Task completion
The average numbers of completed tasks in the four conditions areshown in Table 1. A two-factor two-level ANOVA did not ﬁnd any

Hotel Reservation Task 3
Reserve rooms as below
Stay
As specified with the orange-colored frameon the calendar
Room
Twin, 1 room, non-smoking
Double, 1 room, non-smoking

Figure 3. An example of task instructions

S: Welcome to Hotel Wakamatsu-Kawada. May I help you?
U: I want to stay from March 10th to 11th.
S: What kind of room would you like?
U: One non-smoking twin room and one non-smoking double room.
S: Are your reservation details correctly shown on the screen?
U: Yes. No problem.
S: Your reservation has been accepted. Thank you for using us.
Figure 4. An observed dialogue (translated into English)

signiﬁcant difference among the four conditions on the numbers ofcompleted tasks.

Table 3. Contingency table on the numbers of participants who hadcollisions in the slow reply speed conditions

Table 1. Numbers of completed tasks
reply speed

slow

was talking (i.e., barge-ins). These speech collisions were not thetargets of this paper, hence they were not included in the counts.
Speech collisions due to participants’ back-channel feedbacks
were not included, either. We think that it is possible to ﬁlter out
such feedback because feedback utterances are usually very shortand variations are small. On the other hand, as we mentioned above,it is not easy to automatically distinguish negligible speech such as
repetitions from barge-ins. We want to suppress only such speech
negligible but hard to distinguish from other not negligible speech.
Speech collisions were counted both at the experiment by an experiment supporting staff and after the experiment on the dialogue
transcriptions while watching videos by a research supporting staff.
The two counting results were checked and merged by one of theauthors. The numbers are shown in Table 2 for each participant.
First we performed a two-factor two-level ANOVA on the numbers of collisions. A signiﬁcant difference between the slow reply
speed (A and B) and the moderate reply speed (C and D) was found(F1,44 = 4.06, p < 0.005). This result conﬁrms that making a longinterval after the user’s speech ends and before the system replies
reduces speech collisions.
The ANOVA test did not ﬁnd a signiﬁcant difference on the blinking light factor. This was, however, reasonable because the effectof the blinking light suppressing troublesome follow-ups from users
were expected only in the slow reply speed conditions and the numbers of collisions in the slow reply speed conditions were fairly
smaller than those in the moderate reply speed conditions.
Hence, we performed a Fisher’s exact test (one-side) on the numbers of participants who had speech collisions between the two conditions of the slow reply speed (A and B). The contingency table isshown in Table 3. The test found a signiﬁcant difference (p < 0.05).
This result indicates that the blinking light can reduce speech collisions by suppressing users’ unnecessary follow-ups in decelerated
dialogues.

moderate

blinking light

w/

w/o

w/

w/o

condition label

A

B

C

D

mean
SD

4.00
1.28

3.83
1.34

3.83
1.27

4.33
0.98

condition
w/ a blinking light (A)w/o a blinking light (B)had collisions
3
8

had no collision
9
4

total
12
12

4.4 Impression on the Dialogue
4.3 Speech Collisions
As mentioned above, we use the term follow-ups to refer to all user
utterances such as so called repetitions, corrections, and additions,which are potentially inducing speech collisions. In the experiment in[4], they counted repetitions and it was not difﬁcult in their quite simple last-and-ﬁrst game dialogues where each user utterance contains
only one word. In our experiment, in contrast, it is hard to provide
follow-ups with a concrete deﬁnition, which is necessary to countthem in the dialogue data. Therefore, we do not count follow-ups but
directly focus on the number of speech collisions.
We counted speech collisions for which the dialogue system was
responsible, that is, the cases where the robot spoke while participants were talking (i.e., end-of-turn misdetections). Of course, there
were speech collisions for which participants were responsible, thatis, the cases where participants intentionally spoke while the robot

Table 4 shows the results of participants’ ratings for the dialogue.
The adjective pairs in the table are translated from Japanese words
that we used in the questionnaire. The ratings are based on a sevenpoint Likert scale (1:strong agreement with a negative adjective, 4:
neutral, 7: strong agreement with a positive adjective). The highest
mean score for each adjective pair is underlined.
We performed a factor analysis (principal factor method) underthe varimax rotation and obtained ﬁve factors from the scree plot.
Table 5 shows the factor loadings under the varimax rotation.
We interpreted the factors according to the adjective pairs. The
ﬁrst factor was named the likability factor. The second factor wasnamed the comfortability factor. The third factor was named the interest factor. The fourth and ﬁfth factors were named casualness factor and comprehensibility factor, respectively. Their factor scores byusing a regression method were calculated and compared among theconditions. Table 6 shows the factor scores for the impression of theTable 2. Numbers of collisions by each participant

reply speed

condition
blinking light

label

w/
w/o
w/
w/o

A
B
C
D

slow
moderate

1
1
2
7

2

3

1

2

participant ID in each condition
4 5 6 7 8 9 10
3

5
2

5
6

1
6
6

3

2
3
2

2
1
1

11

1
14
5

1
5

12

total

1
1
2

5
11
45
30

(All the 48 participants are different from each other. Blank slots mean zeros.)dialogue. A two-factor two-level ANOVA found a 10 % level significant main effect for the blinking light in the second factor’s score(F3,44 = 3.53, p = .07). This result suggests that the blinking lightprovides users with a comfortable impression on the dialogue.

Table 4. Rated adjective pairs for impression of the dialogue
reply speed
blinking light
condition label
positivecasualsmoothdecentexciting
relaxedeasy
warm
pleasant
leisurely
informal
light
clear
likablegood
peacefulinteresting
spiritedsettlednegative
grave
rough
indecent
dull
tensional
uneasycold
unpleasanthurriedformaldark
confusingdislikablepoor
annoying
boringdispiritedunsettled

slow

moderate

w/
A

w/o
B

w/
C

w/o
D

mean (SD)
3.42 (1.24)
3.25 (1.29)
4.42 (1.00)
4.58 (0.67)
3.25 (1.06)
3.33 (1.37)
3.25 (1.48)
4.33 (0.89)
4.75 (1.06)
3.08 (1.16)
3.42 (1.00)
4.00 (1.76)
4.42 (0.67)
3.75 (1.22)
4.33 (1.37)
4.58 (1.24)
3.42 (0.67)
4.50 (1.31)

mean (SD)3.33 (1.07)
2.58 (1.00)
4.08 (0.79)
4.17 (1.03)
3.25 (1.48)
2.42 (1.24)
3.25 (0.97)
3.67 (1.15)
4.92 (1.51)
2.67 (0.89)
3.00 (0.85)
3.58 (1.51)
3.75 (0.87)
3.17 (1.11)
3.08 (1.56)
4.25 (1.14)
2.92 (1.00)
3.50 (1.38)

mean (SD)3.50 (1.31)
3.17 (1.59)
4.08 (1.00)
3.83 (1.95)
3.42 (1.16)
2.75 (1.06)
3.67 (0.98)
3.17 (1.03)
4.83 (1.27)
2.67 (1.44)
3.17 (0.94)
4.17 (1.59)
4.08 (1.16)
3.17 (1.19)
3.83 (1.95)
4.33 (1.67)
3.00 (0.85)
3.83 (1.11)

mean (SD)4.00 (1.54)
3.00 (1.35)
4.42 (1.08)
4.17 (1.11)
3.75 (1.48)
2.58 (1.08)
3.67 (1.15)
4.25 (0.97)
4.67 (0.89)
3.17 (1.03)
3.67 (0.78)
4.50 (0.80)
4.33 (0.78)
3.58 (1.24)
3.75 (0.87)
4.67 (1.30)
3.25 (0.87)
3.58 (0.79)

Table 5. Results of factor analysis of impression of the dialogue (varimax
rotation, factor loading matrix)

decentlikablewarm
peaceful
light
informalsettledeasy
goodspiritedsmoothinteresting
exciting
pleasantcasualrelaxedclear
leisurely

1
0.80
0.80
0.58
0.56
0.54
0.49
0.35
0.02
0.13
0.32
0.07
0.31
-0.03
0.10
0.20
0.14
0.04
0.09

factor loadings
2
3
4
0.13
0.23
0.04
0.16
0.15
0.17
0.01
-0.10
0.34
0.52
0.23
-0.25
0.23
0.05
0.27
0.37
0.15
0.38
0.86
0.06
-0.07
0.68
0.05
0.28
0.62
0.37
0.42
0.57
0.36
0.31
0.49
0.33
0.26
0.03
0.88
0.14
0.22
0.59
0.06
0.27
0.43
0.15
0.08
0.14
0.68
0.22
0.11
0.63
0.26
0.13
0.03
0.00
0.17
-0.13

5
0.13
-0.23
0.16
0.15
0.28
0.04
0.06
0.15
0.14
-0.04
0.23
0.02
0.27
0.21
-0.10
-0.05
0.87
0.34

contribution (%)

16.12

16.04

7.27

factor

1

2

3
4
5

item (positive)

10.79

9.91

4.5 Impression on the Robot
We analyzed the impression on the robot with the same method asthat on the dialogue. Table 7 shows the results of participants’ ratings
for the robot.
Table 7. Rated adjective pairs for impression of the robot
reply speed
blinking light
condition label
positive
aggressive
innocent
respectfulaccessible
prettytolerantsociable
responsible
carefulmodestserious
excitedregal
decentdiscreetfriendly
activeconﬁdent
patient
kind

slow
w/
A

negativemean (SD)defensive
3.67 (0.78)
4.08 (1.08)
wickedimpudent
4.75 (0.87)
inaccessible 3.08 (1.08)
provoking4.17 (0.39)
intolerant
4.17 (0.58)
unsociable
3.75 (1.22)
irresponsible 4.42 (1.08)
careless
5.42 (1.08)
4.08 (0.51)
shamelessfrivolous
5.00 (1.28)
3.50 (0.52)
gloom
servile4.92 (1.51)
indecent
4.25 (1.06)
indiscreet
4.67 (1.23)
unfriendly
3.75 (1.29)
inactive3.83 (1.03)
unconﬁdent 4.50 (1.00)
impatient4.42 (0.79)
unkind3.67 (1.30)

moderate
w/o
B

w/
C

w/o
D

mean (SD)3.58 (0.79)
3.92 (1.31)
4.17 (1.59)
3.42 (1.31)
4.17 (0.58)
3.75 (0.75)
3.50 (1.17)
4.17 (1.34)
4.92 (0.90)
4.08 (0.51)
4.67 (1.07)
3.67 (1.07)
5.00 (1.21)
3.67 (1.23)
4.33 (1.07)
3.33 (1.50)
3.58 (0.67)
4.25 (0.87)
4.92 (1.31)
4.08 (1.44)

mean (SD)3.92 (1.16)
4.17 (0.94)
4.42 (0.90)
3.50 (1.09)
4.17 (0.83)
3.92 (0.79)
3.33 (1.15)
4.08 (1.24)
4.92 (1.24)
3.83 (0.72)
5.00 (1.35)
3.33 (0.78)
4.75 (0.97)
3.58 (0.90)
3.92 (1.38)
3.17 (1.40)
3.25 (1.48)
4.00 (0.95)
4.00 (1.04)
3.33 (1.15)

mean (SD)
4.17 (1.19)
4.33 (0.78)
5.25 (1.36)
4.00 (0.95)
4.08 (0.79)
4.08 (0.79)
3.92 (0.90)
4.50 (1.00)
4.50 (1.31)
3.83 (0.39)
4.50 (1.17)
4.00 (1.04)
4.83 (0.72)
4.17 (0.72)
4.33 (0.78)
4.42 (1.00)
4.17 (0.94)
4.67 (0.89)
4.33 (1.07)
4.00 (1.04)

Table 8 shows the factor loadings (principal factor method) under the varimax rotation. We interpreted the obtained factors by afactor analysis (principal factor method, varimax rotation) according
to the adjective pairs. The ﬁrst factor was named the civility factor.
The second factor was named the seriousness factor. The third factor
was named the friendliness factor. The fourth and ﬁfth factors werenamed aggressiveness factor and modesty factor, respectively.
Table 9 shows the factor scores by using regression method forthe impression of the robot. A two-factor two-level ANOVA among
factor scores (regression method) found a 10 % level signiﬁcant main
effect for the slow reply speed in the ﬁfth factor’s score (F3,44 =
3.39, p = .07). This result suggests that the slow reply speed makes
the robot look modest.

Table 6. Factor scores for impression of the dialogue

factorreply speed
slow

1. likability
moderateslow
2. comfortability
moderateslow
3. interest
moderateslow
4. casualness
moderateslow
5. comprehensibility
moderate

blinking light
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/

mean
-0.31
0.13
0.21
-0.03
-0.25
0.53
-0.25
-0.04
-0.03
0.12
0.05
-0.13
-0.08
-0.20
0.38
-0.10
-0.18
-0.18
0.29
0.06

SD
0.27
0.27
0.27
0.27
0.26
0.26
0.26
0.26
0.28
0.28
0.28
0.28
0.25
0.25
0.25
0.25
0.27
0.27
0.27
0.27

ANOVA
reply speed
blinking light
reply speed
×blinking light
reply speed
blinking light
reply speed
×blinking light
reply speed
blinking light
reply speed
×blinking light
reply speed
blinking light
reply speed
×blinking light
reply speed
blinking light
reply speed
×blinking light

F3,44
0.45
0.12

p
0.51
0.73

1.59

0.21

1.19
3.53

0.28
0.07

1.21

0.28

0.09
0.00

0.77
0.96

0.35

0.56

1.24
1.37

0.27
0.25

0.49

0.49

1.78
0.17

0.19
0.68

0.20

0.66

F3,44
0.38
0.32

p
0.54
0.58

1.71

0.20

0.30
0.85

0.59
0.36

0.12

0.73

0.13
1.42

0.72
0.24

2.68

0.11

1.15
1.45

0.29
0.23

0.25

0.62

3.39
0.18

0.07
0.67

0.54

0.47

Table 9. Factor scores for impression of the robot

factorreply speed
slow

1. civility
moderateslow
2. seriousness
moderate
slow
3. friendlinessmoderateslow
4. aggressivenessmoderateslow
5. modesty
moderate

blinking light
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/
w/o
w/

mean
-0.19
0.02
0.35
-0.17
-0.10
0.26
-0.16
0.00
-0.11
0.01
0.42
-0.32
-0.39
0.09
0.05
0.25
0.39
0.09
-0.28
-0.20

SD
0.28
0.28
0.28
0.28
0.28
0.28
0.28
0.28
0.26
0.26
0.26
0.26
0.28
0.28
0.28
0.28
0.26
0.26
0.26
0.26

ANOVA
reply speed
blinking light
reply speed
× blinking light
reply speed
blinking light
reply speed
× blinking light
reply speed
blinking light
reply speed
× blinking light
reply speed
blinking light
reply speed
× blinking light
reply speed
blinking light
reply speed
× blinking lightTable 8. Results of factor analysis of impression of the robot (varimax
rotation, factor loading matrix)

respectful
innocentsociabledecentaccessible
prettykind
excitedtolerant
seriouscareful
responsible
discreetregalfriendly
activeconﬁdent
patient
aggressivemodest

1
0.85
0.79
0.72
0.71
0.69
0.65
0.65
0.63
0.62
-0.15
-0.11
-0.14
0.41
0.17
0.54
0.26
0.05
0.16
0.34
0.10

factor loadings
2
3
4
0.12
-0.02
-0.10
0.14
0.07
0.27
0.02
0.08
-0.18
0.02
0.32
0.29
-0.16
0.37
0.02
-0.23
0.10
0.23
0.24
0.31
-0.02
-0.43
0.33
0.00
-0.01
0.05
0.19
0.86
-0.08
-0.18
0.65
0.04
-0.01
0.61
0.52
-0.10
0.54
0.21
-0.05
0.47
0.07
-0.03
-0.19
0.71
-0.04
0.20
0.43
-0.01
0.09
0.41
0.27
0.32
0.15
0.78
-0.07
0.15
0.71
-0.03
-0.06
-0.03

5
-0.05
0.23
-0.02
0.30
0.22
0.03
0.20
-0.01
0.01
-0.27
0.24
0.19
0.45
-0.12
0.14
-0.07
-0.22
-0.43
0.15
0.56

contribution (%)

26.15

13.02

6.08

factor

1

2

3
4
5

item (positive)

8.55

7.60

4.6 System Evaluations
The participants evaluated the dialogue system in two measures ona scale from 1 to 7, that is, the convenience of the system and their
willingness to use the system. The greater the evaluation value is, thehigher the degree of convenience or willingness.
The average convenience scores of the four conditions in Table 10.
The average willingness scores of the four conditions are shown in
Table 11. The largest mean values among the four conditions areshown in bold letters. ANOVAs did not ﬁnd any signiﬁcant difference
among the four conditions both for the two measures.

4.7 Discussion on User Preference
The analysis of the questionnaire suggests that the blinking light expression gives users a comfortable impression on the dialogue. In
addition, comparing mean scores of adjective pairs and system evaluations between the conditions A and B, a tendency giving higherscores to A is observed. These support the effectiveness of the blinking light expression. However, comparing scores between C and D,
a tendency giving higher scores to D is observed. This intimates thatthe blinking light expression includes some negative effects, too. We
are planning to investigate the blinking pattern from this aspect infuture work.
The analysis also suggests that the slow reply speed gives users
a modest impression on the interface robot. Meanwhile, no negative
impression with a statistical signiﬁcance is found on the slow reply
speed. Although no statistically signiﬁcant difference is found between the four conditions (that is, differences are small), numbersof completed tasks shown in Table 1 and convenience scores shownin Table 10 strongly correlate. However, users’ willingness to use thesystems, which is the most important measure for systems, is inverted
between condition A and D as shown in Table 11. Convenience will
be primarily dominated by to what degree a user’s purpose (reserving rooms) is achieved, thus, it is reasonable that convenience scores
correlate with numbers of completed tasks. On the other hand, willingness will be dominated by not only practical usefulness but also
overall usability. Therefore, we can interpret that the improvementsof impressions and reduction of aversive speech collisions let condition A have the highest score for willingness. These results indicatethat decelerating spoken dialogues is not a bad idea in contradiction
to the common design policy in human computer interfaces (HCIs),
and they suggest to exploit merits brought about by decelerating interactions rather than pursuing quickly responding human-like systems.
Our ﬁnding contradicts not only the common design policy in
HCIs but also the design policy in human robot interaction found by
Shiwa et al. [16], that is, the best response timing of a communication
robot is at one second. We think this contradiction is superﬁcial andis ascribable to the following four major differences between theirstudy and our study.

Table 10. Convenience scores
reply speed

slow

moderate

blinking light

w/

w/o

w/

w/o

condition label

A

B

C

D

mean
SD

3.50
2.02

3.17
1.53

3.17
1.47

3.92
1.62

Table 11.

Willingness scores

reply speed

slow• They adopted a within-subjects experimental design while weadopted a between-subjects design. A within-subjects design
makes subjects do relative evaluations and tends to emphasis differences.
• Their question was speciﬁc in terms of response timing, that is,”Answer your preference about the timing of the system response.”
Our questions were overall ratings of the system such as convenience.
• They assumed a perfect machine (Wizard-of-Oz experiment). Our
system was elaborately crafted but still far from perfect.
• Our system quickly returns non-verbal responses even if verbal
responses are delayed.

moderate

blinking light

w/

w/o

w/

w/o

condition label

A

B

C

D

mean
SD

3.58
1.73

2.58
1.31

2.83
1.34

3.42
1.56

From these differences, we hypothesize that response timing has
no signiﬁcant impact on the usability of dialogue systems in an absolute and holistic context at least in the current state of the art spoken dialogue technology, even though users prefer a system which
responds quickly to a system which responds slowly when they compare them with each other directly, given an explicit comparison metric of response timing appropriateness with perfect machines.

5Conclusion and Future work

This paper showed the results of the experiment in which forty-eightparticipants engaged in hotel reservation tasks with a spoken dialogue system coupled with an interface robot equipped with an LED.
The experiment aimed to investigate the inﬂuences of decelerating
dialogues and the effects of a blinking light expression devised as an
artiﬁcial subtle expression (ASE) on reducing speech collisions andcompensating the deterioration of responsiveness due to decelerating
dialogues. Decelerating dialogues is the simplest way to build stablespoken dialogue systems with speech collisions reduced, and affords
to employ more powerful but computationally expensive speech processing or to build systems on small devices with limited resources.
The blinking light expression as an ASE is quite simple, thus it iseasy and cheap to implement, and applicable to wider conversational
agents/devices not limited to robots.
The two experimental factors used in the experiment were theblinking light factor (with or without a blinking light) and the reply
speed factor (moderate or slow reply speed). The analysis showedthat speech collisions were reduced by slowing the reply speed, andthey were reduced further by using the blinking light expression withstatistical signiﬁcances. The analysis of a questionnaire suggested
that the blinking light expression gave participants a comfortable impression, and surprisingly that users did not care about slow replies.
In addition, although a statistical signiﬁcance was not found, the system with the slow reply speed and a blinking light obtained the highest score for users’ willingness to use the system, which is the mostimportant measure for systems.
While our method using an LED can apply to any other interfaces on wearable/handheld devices, vehicles, whatever, it is difﬁcult
to directly apply it to call-centers (i.e., telephone interfaces), which
occupy a big portion of the deployed spoken dialogue systems pie.
However, the underlying framework, that is, “decelerating spoken dialog with an artiﬁcial subtle expression”, will be applicable even to
telephone interfaces by using an auditory artiﬁcial subtle expression
which is to be explored in future work.
So far, our conclusion is that spoken dialogue systems or communication robots do not need to quickly respond verbally as long as
they quickly respond non-verbally by showing their internal statesby using an artiﬁcial subtle expression, while many researchers try
to make them verbally respond as fast as possible. Decelerating dialogue has many practical advantages as stated above. However,through the experiment, we suspect that this conclusion is not valid
in some speciﬁc cases. That is, we think in some situations users
are troubled by slow verbal responses primordially, and those situations are such as when users simply reply to systems’ yes/no questions or greetings. Our hypothesis is that users expect quick verbal
responses (and hate slow verbal responses) only when users expectthat it is not difﬁcult for systems to understand their responses or todecide next actions. If this hypothesis is valid, and if users’ expectations can be easily estimated not from unreliable information such as
speech recognition results, intonations, facial motions, etc but fromsolid information such as the speech act type of a system’s preceding utterance, we will be able to immediately achieve better speechcommunication systems which respond quickly only when needed.
We will seek this direction in future work.

REFERENCES[1] C. Bartneck and J. Reichenbach, ‘Subtle emotional expressions of synthetic characters’, International Journal of Human-Computer Studies,62(2), 179–192, (2005).

[2] L. Bell, J. Boye, and J. Gustafson, ‘Real-time handling of fragmented
utterances’, in Proc. NAACL-2001 workshop on Adaptation in Dialogue
Systems, (2001).
[3] S. Brave, C. Nass, and K. Hutchinson, ‘Computers that care: investigating the effects of orientation of emotion exhibited by an embodied
computer agent’, International Journal of Human-Computer Studies,62(2), 161–178, (2005).
[4] Kotaro Funakoshi, Kazuki Kobayashi, Mikio Nakano, Seiji Yamada,Yasuhiko Kitamura, and Hiroshi Tsujino, ‘Smoothing human-robot
speech interactions by using a blinking-light as subtle expression’, inProc. 2008 International Conference on Multimodal Interfaces (ICMI),
pp. 293–296, (2008).
[5] Kotaro Funakoshi, Mikio Nakano, Yuji Hasegawa, and Hiroshi Tsujino,‘Semantic interpretation supplementarily using syntactic analysis’, inProc. 2007 International Conference /Recent Advances in Natural Language Processing/ (RANLP), Borovets, Bulgaria, (Sep. 2007).
[6] A. Kendon, ‘Some functions of gaze direction in social interaction’,Acta Psychologica, 26, 1–47, (1967).
[7] A. Kendon, ‘Do gestures communicate?’, A Review. Research in Language and Social Interaction, 27(3), 175–200, (1994).
[8] Norihide Kitaoka, Masashi Takeuchi, Ryota Nishimura, and Seiichi
Nakagawa, ‘Response timing detection using prosodic and linguistic
information for human-friendly spoken dialog systems’, Journal of The
Japanese Society for Artiﬁcial Intellignece, 20(3), 220–228, (2005).
[9] T. Komatsu and S. Yamada, ‘How do robotic agents’ appearances affect people’s interpretations of the agents’ attitudes?’, in Proc. 25th
ACM International Conference on Human Factors in Computing Systems (CHI), pp. 2519–2524, (2007).
[10] Takanori Komatsu, Seiji Yamada, Kazuki Kobayashi, Kotaro Funakoshi, and Mikio Nakano, ‘Artiﬁcial subtle expressions: Intuitive notiﬁcation methodology of artifacts’, in Proc. 28th ACM International
Conference on Human Factors in Computing Systems (CHI), p. (to beappeared), (2010).
[11] Mikio Nakano, Yuka Nagano, Kotaro Funakoshi, Toshihiko Ito, Kenji
Araki, Yuji Hasegawa, and Hiroshi Tsujino, ‘Analysis of user reactions
to turn-taking failures in spoken dialogue systems’, in Proc. 8th SIGdial
Workshop on Discourse and Dialogue, (2007).
[12] Tomoko Ohsuga, Masafumi Nishida, Yasuo Horiuchi, and Akira
Ichikawa, ‘Investigation of the relationship between turn-taking andprosodic features in spontaneous dialogue’, in Proc. 9th European Conference on Speech Communication and Technology, pp. 33–36, (2005).
[13] H. Prendinger, J. Mori, and M. Ishizuka, ‘Using human physiology toevaluate subtle expressivity of a virtual quizmaster in a mathematicalgame’, International Journal of Human-Computer Studies, 62(2), 231–
245, (2005).
[14] A. Raux and M. Eskenazi, ‘Optimizing endpointing thresholds using
dialogue features in a spoken dialogue system’, in Proc. 9th SIGdial
Workshop on Discourse and Dialogue, (2008).
[15] W. Rogers, ‘The contribution of kinesic illustrators towards the comprehension of verbal behavior within utterances’, Human Communication
Research, 5, 54–62, (1978).
[16] Toshiyuki Shiwa, Takayuki Kanda, Michita Imai, Hiroshi Ishiguro,
and Norihiro Hagita, ‘How quickly should communication robots respond?’, in Proc. 3rd ACM/IEEE International Conference on HumanRobot Interaction (HRI), pp. 153–160, (2008).
[17] S. Takano, K. Tanaka, H. Mizuno, M. Abe, and S. Nakajima, ‘A
Japanese TTS system based on multi-form units and a speech modiﬁcation algorithm with harmonics reconstruction’, IEEE Transactions
on Speech and Audio Processing, 9(1), 3–10, (2001).
[18] Wataru Tsukahara and Nigel Ward, ‘Evaluating responsiveness in spoken dialog systems’, in Proc. International Conference on Spoken Language Processing, (2000).
[19] Nigel Ward, ‘On the expressive competencies needed for responsive
systems’, in Proc. the CHI2003 workshop on Subtle Expressivity for
Characters and Robots, (2003).