Smoothing Human-robot Speech Interactions by Using a
Blinking-Light as Subtle Expression
Kotaro Funakoshi

Kazuki Kobayashi

Mikio Nakano

Honda Research InstituteJapan Co., Ltd.
8-1 Honcho, Wako-shi
Saitama, Japan

Shinshu University
4-17-1 Wakasato, Nagano
Nagano, Japan

Honda Research InstituteJapan Co., Ltd.
8-1 Honcho, Wako-shi
Saitama, Japan

funakoshi@jp.hondari.com
Seiji Yamada
National Institute ofInformatics
2-1-2 Hitotsubashi, Chiyoda
Tokyo, Japanseiji@nii.ac.jp

kkobayashi@cs.shinshuu.ac.jp

nakano@jp.honda-ri.com

Yasuhiko Kitamura

Hiroshi Tsujino

Kwansei Gakuin University
2-1 Gakuen, Sanda
Hyogo, Japan

Honda Research InstituteJapan Co., Ltd.
8-1 Honcho, Wako-shi
Saitama, Japanykitamura@kwansei.ac.jp
1.

tsujino@jp.honda-ri.com

INTRODUCTION

Speech overlaps, undesired collisions of utterances between systems and users, harm smooth communication and degrade systemusability. When speech overlaps occur, users tend to stop speaking.
This prevents systems from responding adequately because interrupted speech is hard to recognize automatically [7].
Often a speech overlap arises when the system misrecognizes
a speech pause as a turn-end and starts speaking. Detection accuracy of turn-ends can be improved by using linguistic information [2] and para-linguistic information such as F0 (fundamental
frequency) [8]. However, these methods are not sufﬁcient. Linguistic information is affected by speech misrecognition. F0 is notavailable for fragmented short utterances because a reliable F0 contour requires a substantial length of speech.
Another approach to avoid turn-end misrecognitions is to make along interval after the user’s speech signal ends and before the system responds. This approach is simple and steady, but deteriorates
system responsiveness and leads to another speech overlapping situation, that is, the user repeats her/his last utterance because of thelack of response from the system.
Equipping the system to express its turn-taking intention by using body/eye movements as humans do may resolve this situation.
However, such an approach is technically difﬁcult and uneconomical. Using spoken back-channel feedback (such as “well” and “uh”)
is another option. It is, however, not an easy matter [10] and auser study found that users preferred systems using spoken backchannel feedback to systems not using them only in cases systems
could use them appropriately [4].
In contrast to the above approaches trying to make machines
resemble humans, there is another approach that utilizes expressions characteristic of artifacts [6]. Following such an approach,
we believe there is a way that we can avoid the difﬁculties in using human-like expressions but still gain the desired effect from a
simple and economical implementation.
This paper veriﬁes to what extent a method using a blinking-light
can ameliorate this situation. We devised a robot that expresses its
internal state by using a blinking-light and conducted experiments
in which users and the robot engaged in last-and-ﬁrst games (shiritori) through speech. The smoothness of the resulting dialogues

ABSTRACT
Speech overlaps, undesired collisions of utterances between systems and users, harm smooth communication and degrade the usability of systems. We propose a method to enable smooth speechinteractions between a user and a robot, which enables subtle expressions by the robot in the form of a blinking LED attached to its
chest. In concrete terms, we show that, by blinking an LED fromthe end of the user’s speech until the robot’s speech, the numberof undesirable repetitions, which are responsible for speech overlaps, decreases, while that of desirable repetitions increases. In experiments, participants played a last-and-ﬁrst game with the robot.
The experimental results suggest that the blinking-light can prevent
speech overlaps between a user and a robot, speed up dialogues,
and improve user’s impressions.

Categories and Subject Descriptors
H.5.m [Information interfaces and presentation (e.g., HCI)]: Miscellaneous

General Terms
Experimentation, Veriﬁcation

Keywords
Turn-taking, speech overlap, subtle expression, human-robot interaction

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for proﬁt or commercial advantage and that copies
bear this notice and the full citation on the ﬁrst page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior speciﬁc
permission and/or a fee.
ICMI’08, October 20–22, 2008, Chania, Crete, Greece.
Copyright 2008 ACM 978-1-60558-198-9/08/10 ...$5.00.

293was analyzed from the number of user repetitions of last utterances
and users’ answers to questionnaires. In concrete terms, we show
that, by blinking an light from the end of the user’s speech until
the robot’s speech, the number of undesirable repetitions decreases
while that of desirable repetitions increases, and participants’ impressions improve.

2. BLINKING LIGHT AS SUBTLE
EXPRESSION
Although human communication is explicitly achieved throughverbal utterances, non-verbal facial expressions, gaze, gestures, etc.,
also play an important role [3]. Such non-verbal communication
often inﬂuences the accuracy of utterance understanding [9].
Furthermore, researchers have reported that very small changes(called subtle expression) in facial expressions and gestures might
inﬂuence human communication. We believe that we can utilize
such subtle expression to make humans easily understand a robot’sinternal state because humans can intuitively understand subtle expression. Some studies have been done on applying subtle expression to human-agent interaction (e.g., [1]). However, since they
tried to enable subtle expression on real faces and with real arms,
their implementations were considerably expensive.
In contrast with such approaches, subtle expression has beenstudied for artifacts like a robot or PC. Komatsu and Yamada [6]reported that an agent’s subtle expression of simple beeping soundswith decreasing/increasing frequency enabled humans to interpretthe agent’s positive/negative states. Their work indicated the effectiveness of subtle expressions such as varying beeping sounds for arobot or agent.
In this work, we propose the use of a blinking light as a meansof subtle expression to intuitively notify a user about robot’s internal states (such as processing or busy). We implemented the subtle
expression on a robot that engaged in dialogue with a user and conducted experiments on participants to verify the effectiveness of thesubtle expression.

Figure 1: Operational Interface

Figure 2: RS Media and position of embedded LED

participant’s answer. The operator directs the robot to utter“Say it again.”
4. Order to restart: Dialogue can break down for a variety ofreasons. In such cases, the operator directs the robot to restartanother game by uttering “Let’s start again.”

3. EXPERIMENTS

5. Record the participant’s repetition: If the participant repeats
her/his last utterance, the operator records it by pressing the
ESC key.

We conducted experiments in which participants played a lastand-ﬁrst game with a robot. A last-and-ﬁrst game was an appropriate task to investigate speech overlaps because it involved a lotof turn-taking. We counted the number of times the participants
repeated their last utterances and estimated their impression of the
dialogue. The dialogue system, robot, light-blinking expression,and experimental method are explained as follows.

We controlled the experiments as follows so that the target phenomena could be clearly observed. First, the intervals between theusers’ answers and the robot’s answers were randomly varied between 0 and 15 seconds by inserting a waiting time. In actual spoken dialogue systems, these intervals correspond to delays due tointernal processing. Second, the robot neglected the participants’
answers with a probability of 1/4. In actual dialogue systems, thiscorresponds to rejecting speech as noise by mistake. Third, to resolve a standoff, the robot automatically re-uttered its last utterances after 30 seconds if the participant said nothing.

3.1 Last-and-First Dialogue System
We chose a Wizard-of-Oz method to avoid speech recognition errors. The operator listened to the participant and operated the robot
using the interface shown in Figure 1. The robot’s utterances werevoiced by a commercial speech synthesizer (NTT-IT FineVoice).
The operator performed the following operations when needed.

3.2 Robot and Blinking-Light Expression

1. Input participant’s answer: The operator inputs the participant’s answer immediately after the participant utters it. If
the answer violates rules, the robot claims a foul and restarts
another game. Otherwise, it chooses its next answer from apredeﬁned lexicon.

The participants talked to the small human-like robot “RS Media(WowWee)” shown in Figure 2. In our experiments, the robot did
not drive any actuators. We embedded a red LED (diameter: 4 mm)
in its chest. The LED started blinking when an operator began to
input a participant’s utterance and stopped blinking when the robot
began to utter. In other words, the LED blinked while the robot
processed the participant’s utterance to prepare an answer. Whenthe robot rejected utterances, the LED stayed off. The LED blinked
at an even interval of 1/30 seconds.

2. Order to re-utter: The robot’s speech is sometimes hard tolisten to. Upon a participant’s request, the operator directs
the robot to re-utter its last utterance.
3. Order to clarify: Sometimes the operator cannot catch the294

3.3 Participants
Forty seven participants of 19 to 62 years old were divided into
two experimental groups:

condition
blinking
non-blinking(1) the blinking condition(11 men, 12 women, mean age 32.0, S.D. = 11.0), and(2) the non-blinking condition(11 men, 13 women, mean age 31.0, S.D. = 10.3).

Table 1: Mean repetition rate
in-processing
rejection
1.3% (S.D. = 3.6)
55.3% (S.D. = 50.0)
5.4% (S.D. = 10.2) 24.3% (S.D. = 34.0)

Table 2: Rated adjective pairs for impression of the gameadjective pairs
positivecasualsmoothcomfortable
exciting
relaxedeasy
warm
pleasant
leisurely
informallightcomprehensible
likablegood
peacefulinterestingencouraging
settled

3.4 Experimental Method
The experiments were conducted in a small room. Participants
entered the room, sat on a chair in front of a desk. They answereda questionnaire after they had been given instructions about the experiments.
After answering the questionnaire, they were asked to play a lastand-ﬁrst game with the robot for 10 minutes and explained the rulesof the game (see [5] for the rules). They could answer as they liked
if they forgot the rules. The experimenter told them that the robot
sometimes replied quickly, sometimes leisurely, and sometimes rejected their utterances. Moreover, they were requested to continue
the game as long as they could. The meaning of the light-blinking
expression was not explained to them. After giving the instructions,the experimenter left the room, and the participants began to playa game when the robot started to talk to them. The game ﬁnishedafter ten minutes, when the robot said that it was over.
After ﬁnishing the game, the participants answered another questionnaire about their impression of the game and the robot’s behavior, and other particular questions.

4.2

negative
grave
rough
uncomfortabledull
tensional
uneasycold
unpleasanthurriedformaldark
incomprehensible
dislikablepoor
annoying
boringdiscouraging
unsettled

blinking
mean
S.D.
4.61
1.62
2.83
1.23
3.96
1.30
4.91
1.59
3.78
1.41
3.83
1.23
3.96
1.11
4.48
0.99
5.26
0.96
4.40
1.40
4.04
1.07
3.74
1.57
4.35
1.03
4.17
0.98
3.83
1.19
4.61
1.56
3.17
0.72
4.13
1.14

non-blinking
mean
S.D.
4.04
1.46
2.33
1.09
4.25
1.26
4.04
1.65
3.71
1.30
3.42
1.35
3.67
1.05
4.25
1.42
5.17
1.20
3.71
1.23
4.25
0.99
3.21
1.10
4.21
1.41
4.21
1.06
3.67
1.52
4.46
1.74
3.50
1.44
3.54
1.25

Impression of the Game and the Robot

The participants evaluated their impressions of the game andthe robot by rating adjective pairs based on a scale from 1 to 7,
where “1” equals strong agreement with a negative adjective and“7” equals strong agreement with a positive adjective. Table 2
and Table 3 show the rating results of the game and the robot respectively. Original adjective pairs in the questionnaire were in
Japanese. We performed factor analysis (principal factor method)in the same way as [5], obtained factor scores with a regression
method, and compared the two conditions by using a t-test.
No factor with a signiﬁcant difference between the conditions
was extracted with regard to the impression of the game. On theother hand, with regard to the impression of the robot, a factor witha signiﬁcant difference (t = 2.68, d.f. = 45, p = .010) was extracted, to which the two adjective pairs of “responsible/irresponsible”
and “broad-minded/narrow-minded” contribute positively for theblinking condition. This result suggests that the blinking light gives
users the sincere impression of the robot.

4. RESULTS AND DISCUSSION
4.1 Participants’ Repetitions
We classiﬁed participants’ repetitions into two cases:(1) the in-processing case (uttered while the robot was processing their utterances), and(2) the rejection case (uttered when the robot rejected their utterances, i.e., the robot failed to catch the utterances).
Table 1 shows the mean repetition rates of the in-processing andrejection cases.
The mean repetition rate of the in-processing case is the averageof the rates in which each participant repeated his/her last wordwhile the robot was recognizing the word and preparing its next
answer. In the in-processing case, a signiﬁcant tendency between
the mean repetition rates in the blinking condition and the nonblinking condition was found by using a t-test (t = 1.81, d.f. =45, p = .079). This supports that a blinking-light can suppress
users’ undesirable repetitions, and thus can prevent speech overlaps
between a user and a robot.
The mean repetition rate of the rejection case is the average ofthe rates in which each participant repeated his/her last word when
the robot failed to catch what the participant said. In the rejection
case, a signiﬁcant difference between the mean repetition rates inthe two conditions was found (t = 2.67, d.f. = 45, p = .010).
This suggests that a blinking-light can speed up dialogues because
dialogues are suspended when the robot rejected the participants’utterances unless they repeat or the robot prompts.
Indeed, the average number of the robot’s answering was significantly greater in the blinking condition than in the non-blinking
condition (avg. 25.8 vs 21.1, t = 3.06, d.f. = 43.29, p < .005).
This suggests that the dialogues in the blinking condition were
faster.

4.3

Interpretation of the Blinking-Light

As free format questions in the questionnaire, we asked the participants about distinctive behaviors of the robot, situations whenparticipants repeated their utterances and reasons they repeated,etc. From the answers to these questions, we extracted in whatmanner the participants in the blinking condition interpreted theblinking-light. Table 4 shows the classiﬁcation of their interpretations. This shows that at least 21 participants out of 23 in theblinking condition were aware of the blinking-light, and most ofthem interpreted the meaning of the blinking light as we intended.
We can expect that the rate of repetitions will change over time
if it requires a certain amount of interactions for participants to interpret the meaning of the blinking-light. Therefore we conductedt-tests between the mean repetition rates in the ﬁrst ﬁve minutes
and the last ﬁve minutes of dialogues. In the blinking condition, no
signiﬁcant difference was found. This suggests that the participants
grasped the meaning of the blinking-light in a short time.

295

Table 3: Rated adjective pairs for impression of the robot
adjective pairs
positive
aggressive
innocent
respectfulaccessible
prettybroad-mindedsociable
responsible
carefulshy
serious
excitedregal
decentdiscreetfriendly
activeconﬁdent
patient
kind

negative
defensive
wickedimpudent
inaccessible
provokingnarrow-mindedunsociable
irresponsible
carelessshamelessfrivolous
gloom
servileindecent
indiscreet
unfriendly
inactive
unconﬁdent
irritable
unkindblinking
mean
S.D.
4.22
1.31
4.17
1.27
4.00
1.48
3.96
1.19
4.26
1.14
4.22
1.09
4.13
1.46
4.52
1.08
4.78
1.00
4.04
0.56
4.35
1.15
3.65
1.19
4.96
1.15
4.22
1.13
4.52
1.04
4.09
1.53
4.22
1.17
5.00
1.21
4.65
1.23
3.52
1.34

Table 5: Reasons for repetitions

non-blinking
mean
S.D.
4.21
1.02
4.29
1.46
4.50
1.56
4.17
1.31
4.08
1.41
3.67
1.27
3.83
1.17
4.50
1.44
4.50
1.44
3.79
0.72
4.21
1.35
3.79
0.78
4.83
1.13
3.92
1.35
4.46
1.18
3.96
1.57
4.21
0.88
4.46
1.22
4.83
1.13
3.71
1.33

condition
blinking

non-blinking# of participants
14
4
3
2

#
16
5
2
10
7
5
2

(percentage)(69.6%)(21.7%)(8.7%)(41.7%)(29.2%)(20.8%)(8.3%)

their last utterance when the robot rejected them and they did not
repeat when it processed them. The analysis of the questionnairesuggested that (1) the blinking-light created sincere impressions
about the robot on users, (2) most of the participants interpreted
the blinking adequately in a short time, and (3) the blinking suppressed the negative impression of late responses. These results
supported the effectiveness of the blinking-light expression. We
think they indicate that LED-based subtle expression is helpful forsmoothing human-robot speech interaction by preventing speech
overlaps between a user and a robot, by speeding up dialogues, andby improving user’s impressions.
In future work, we would like to bring the proposed method into
task-oriented dialogues, and demonstrate the method can improvetask achievement ratio in addition to smoothness. Comparisonswith other approaches are also to be addressed.

Table 4: Interpretation of the blinking-lightinterpretation
recognizing the user’s answer
preparing the next answer
other(no answer)

reasonthe LED did not blink
the robot repeated its last word
(no repetition)
the robot repeated its last word
the robot’s response was lateother(no repetition)(percentage)(60.9%)(17.4%)(13.0%)(8.7%)

6.

On the other hand, in the non-blinking condition, signiﬁcant differences were found both in the in-processing case and in the rejection case. In the case of in-processing, the mean repetition rate of
the ﬁrst half was 5.9% and that of the last half was 3.6% (t = 2.32,
d.f. = 23, p < .05). In the case of rejection, the mean repetition
rate of the ﬁrst half was 29.7% and that of the last half was 12.9%(t = 2.11, d.f. = 23, p < .05). This might be because the participants found that their repetitions were ineffective. The participants’repetitions were effective only in the case of rejection. However, inthe non-blinking condition, the participants could not distinguish
rejection cases from in-processing cases while in-processing cases
had a higher proportion.
These results suggest that the adequate interpretation of the blinkinglight can be obtained easily and immediately.

4.4 Reasons for Repetitions
Table 5 shows the classiﬁcation of reasons for repetitions. Although about 30% of participants in the non-blinking condition
made repetitions because of the robot’s late response, no participants in the blinking condition reported such a reason. This suggests that the blinking-light dilutes the negative impression of late
responses.

5. CONCLUSION
We proposed a method to enable smooth speech interactions between a user and a robot. Our method was based on subtle expression whereby a robot blinks a small LED attached to its chest. We
performed experiments in which participants were divided into twogroups: the blinking condition and the non-blinking condition, andplayed last-and-ﬁrst games. The number of repetitions made by theparticipants and their answers to the post-experiment questionnaire
were analyzed.
The analysis of the number of participants’ repetitions showed
that, in the blinking condition, the participants adequately repeated

296

REFERENCES[1] C. Bartneck and J. Reichenbach. Subtle emotional
expressions of synthetic characters. International Journal of
Human-Computer Studies, 62(2):179–192, 2005.
[2] L. Bell, J. Boye, and J. Gustafson. Real-time handling offragmented utterances. In Proc. of NAACL-2001 workshop
on Adaptation in Dialogue Systems, 2001.
[3] A. Kendon. Do gestures communicate? A Review. Researchin Language and Social Interaction, 27(3):175–200, 1994.
[4] N. Kitaoka, M. Takeuchi, R. Nishimura, and S. Nakagawa.
Response timing detection using prosodic and linguistic
information for human-friendly spoken dialog systems.
Journal of The Japanese Society for Artiﬁcial Intellignece,20(3):220–228, 2005.
[5] K. Kobayashi, K. Funakoshi, S. Yamada, M. Nakano,
Y. Kitamura, and H. Tsujino. Smoothing human-robot
speech interaction with blinking-light expressions. In Proc.
of RO-MAN 2008, 2008.
[6] T. Komatsu and S. Yamada. How do robotic agents’appearances affect people’s interpretations of the agents’attitudes? In Proc. of CHI-2007, pages 2519–2524, 2007.
[7] M. Nakano, Y. Nagano, K. Funakoshi, T. Ito, K. Araki,
Y. Hasegawa, and H. Tsujino. Analysis of user reactions toturn-taking failures in spoken dialogue systems. In Proc. of
SIGdial-2007, 2007.
[8] T. Ohsuga, M. Nishida, Y. Horiuchi, and A. Ichikawa.
Investigation of the relationship between turn-taking andprosodic features in spontaneous dialogue. In Proc. ofEuropean Conference on Speech Communication andTechnology, pages 33–36, 2005.
[9] W. Rogers. The contribution of kinesic illustrators towards
the comprehension of verbal behavior within utterances.
Human Communication Research, 5:54–62, 1978.
[10] N. Ward. On the expressive competencies needed for
responsive systems. In Proc. of the CHI2003 workshop on
Subtle Expressivity for Characters and Robots, 2003.