670

 特集論文

人工知能学会論文誌

25 巻 6 号 SP-A（2010 年）

 HAI (Human-Agent Interaction)

Detecting Robot-Directed Speech by Situated Understanding in Physical Interaction
Xiang Zuo

Advanced Telecommunication Research Labs and Kyoto Institute of Technology

Naoto Iwahashi

Advanced Telecommunication Research Labs and National Institute of Information and Communications Technology

d8821502(at)edu.kit.ac.jp

naoto.iwahashi(at)nict.go.jp

Kotaro Funakoshi

Honda Research Institute Japan Co., Ltd.
funakoshi(at)jp.honda-ri.com

Mikio Nakano(affiliation as previous author)
nakano(at)jp.honda-ri.com

Ryo Taguchi

Advanced Telecommunication Research Labs and Nagoya Institute of Technology
taguchi.ryo(at)nitech.ac.jp

Shigeki Matsuda

National Institute of Information and Communications Technology

Komei Sugiura(affiliation as previous author)

Natsuki Oka

Kyoto Institute of Technology

shigeki.matsuda(at)nict.go.jp
komei.sugiura(at)nict.go.jp
nat(at)kit.ac.jp

keywords: robot-directed speech detection, multimodal semantic confidence, human-robot interaction
Summary
In this paper, we propose a novel method for a robot to detect robot-directed speech: to distinguish speech thatusers speak to a robot from speech that users speak to other people or to themselves. The originality of this workis the introduction of a multimodal semantic confidence (MSC) measure, which is used for domain classification of
input speech based on the decision on whether the speech can be interpreted as a feasible action under the current
physical situation in an object manipulation task. This measure is calculated by integrating speech, object, andmotion confidence with weightings that are optimized by logistic regression. Then we integrate this measure withgaze tracking and conduct experiments under conditions of natural human-robot interactions. Experimental results
show that the proposed method achieves a high performance of 94% and 96% in average recall and precision rates,
respectively, for robot-directed speech detection.

1. INTRODUCTION
Robots are now being designed to be a part of the everyday lives of ordinary people in social and home environments. One of the key issues for practical use ofsuch robots is the development of user-friendly interfaces.
Speech recognition is one of our most effective communication tools for use in a human-robot interface. In recent
works, many systems using speech-based human-robot interfaces have been implemented, such as [Asoh 99, Ishi
06]. For such an interface, the functional capability ofdetecting robot-directed (RD) speech is crucial. For example, a user’s speech directed to another human listenershould not be recognized as commands directed to a robot.
To resolve this issue, many works have used humanphysical behaviors to estimate the target of the user’s speech.
Lang et al. [Lang 03] proposed a method for a robot
to detect the direction of a person’s attention based onface recognition, sound source localization, and leg detection. Mutlu et al. [Mutlu 09] conducted experiments
under conditions of human-robot conversation, and theystudied how a robot could establish the participant rolesof its conversational partners using gaze cues. Yonezawa
et al. [Yonezawa 09] proposed an interface for a robot tocommunicate with users based on detecting the gaze direction during their speech. However, this kind of method
raises the possibility that users may say something irrelevant to the robot while they are looking at it. Consider asituation where users A and B are talking while looking at

671

Detecting Robot-Directed Speech by Situated Understanding in Physical Interaction

the robot in front of them (Figure 1).
A: Cool robot! What can it do?
B: It can understand your command, like “Bring me thered box.”Note that the speech here is referential, not directed to therobot. Moreover, even if user B makes speech that sounds
like RD speech (“Bring me the red box”), she does notreally want to give such an order because no red box exists
in the current situation. How can we build a robot that
responds appropriately in this situation?
To settle such an issue, the proposed method is based
not only on gaze tracking but also on domain classification of the input speech into RD speech and out-of-domain(OOD) speech. Domain classification for robots in previous works were based mainly on using linguistic andprosodic features. As an example, a method based on keyword spotting has been proposed by [Kawahara 98]. However, in using such a method it is difficult to distinguish
RD speech from explanations of system usage (as in theexample of Figure 1). It becomes a problem when bothtypes of speech contain the same “keywords.” To settle
this problem, a previous work [Takiguchi 08] showed thatthe difference in prosodic features between RD speech andother speech usually appears at the head and the tail of the
speech, and they proposed a method to detect RD speechby using such features. However, their method also raised
the issue of requiring users to adjust their prosody to fit
the system, which causes them an additional burden.
In this work, the robot executed an object manipulation task in which it manipulates objects according to auser’s speech. An example of this task in a home environment is a user telling a robot to “Put the dish in the
cupboard.” Solving this task is fundamental for assistive
robots. In this task, we assume that a user orders therobot to execute an action that is feasible in the current
situation. Therefore, the word sequences and the object
manipulation obtained as a result of the process of understanding RD speech, should be possible and meaningful
in the given situation. In contrast, word sequences andthe object manipulation obtained by the process of understanding OOD speech would not be feasible. Therefore,
we can distinguish between RD and OOD speech by using the feasibility for the corresponding word sequenceand the object manipulation obtained from a speech understanding process as a measure. Based on this concept,
we developed a multimodal semantic confidence (MSC)
measure. A key feature of MSC is that it is not basedon using prosodic features of input speech as with themethod described above; rather, it is based on semantic
features that determine whether the speech can be inter-

It can understand yourcommand, like “Bring me
the red box.”

Cool robot!
What can it do?

Fig. 1 People talking while looking at a robot.

preted as a feasible action under the current physical situation. On the other hand, for an object manipulation task
robots should deal with speech and image signals and tocarry out a motion according to the speech. Therefore,
the MSC measure is calculated by integrating information
obtained from speech, object images and robot motion.
The rest of this paper is organized as follows. Section 2
gives the details of the object manipulation task. Section
3 describes the proposed RD speech detection method.
The experimental methodology and results are presented
in Section 4, and Section 5 gives a discussion. Finally,
Section 6 concludes the paper.

2. Object Manipulation Task
In this work, humans use a robot to perform an object
manipulation task. Figure 2 and Figure 3 show the robot
used in this task. It consists of a manipulator with 7 degrees of freedom (DOFs), a 4-DOF multi-fingered grasper,a SANKEN CS-3e directional microphone for audio signal input, a Point Grey Research Bumblebee 2 stereo vision camera for video signal input, a MESA Swiss Ranger
SR4000 infrared sensor for 3-dimensional distance measurement, a Logicool Qcam Pro 9000 camera for humangaze tracking, and a head unit for robot gaze expression.
In the object manipulation task, users sit in front of therobot and command the robot by speech to manipulate objects on a table located between the robot and the user.
Figure 4 shows an example of this task. In this figure,the robot is told to place Object 1 (Kermit) on Object 2(big box) by the command speech “Place-on Kermit bigbox”∗1 , and the robot executes an action according to thisspeech. The solid line in Figure 4 shows the trajectoryof the moving object manipulated by the robot. Figure 5
shows the sequential photographs of the robot executing
an action according to command speech “Place-on Bar∗1 Commands made in Japanese have been translated into English
in this paper.

672

人工知能学会論文誌

Fig. 2 Robot used in the object manipulation task.

Fig. 425 巻 6 号 SP-A（2010 年）

Example of object manipulation tasks.

3. Proposed RD Speech Detection Method

Fig. 3 Cameras, microphone, sensor and head unit of the robot.

bazoo∗2 red box”.
Commands used in this task are represented by a sequence of phrases, each of which refers to a motion, an
object to be manipulated (“trajector”), or a reference object for the motion (“landmark”). In the case shown in Figure 4, the phrases for the motion, trajector, and landmark
are “Place-on,” “Kermit,” and “big box,” respectively. Moreover, fragmental commands without a trajector phrase ora landmark phrase, such as “Place-on big box” or just“Place-on,” are also acceptable.
To execute a correct action according to such a command, the robot must understand the meaning of each wordin it, which is grounded by the physical situation. The
robot must also have a belief about the context information
to estimate the corresponding objects for the fragmental
commands. In this work, we used the speech understanding method proposed by [Iwahashi 07] to interpret the input speech as a possible action for the robot under the current physical situation. However, for an object manipulation task in a real-world environment, there may exist
OOD speech such as chatting, soliloquies, or noise. Consequently, an RD speech detection method should be used.
∗2 Kermit and Barbazoo are the stuffed toy’s names used in our experiment.

The proposed RD speech detection method is based on
integrating gaze tracking and the MSC measure. A flowchart is given in Figure 6. First, a Gaussian mixture model
based voice activity detection method (GMM-based VAD)
[Lee 04] is carried out to detect speech from the continuous audio signal, and gaze tracking is performed to estimated the gaze direction from the camera images∗3 . If the
proportion of the user’s gaze at the robot during her/his
speech is higher than a certain threshold η, the robot judgesthat the user was looking at it while speaking. The speechduring the periods when the user is not looking at the
robot is rejected. Then, for the speech detected while
the user was looking at the robot, speech understanding
is performed to output the indices of a trajector object anda landmark object, a motion trajectory, and corresponding phrases, each of which consists of recognized words.
Then, three confidence measures, i.e., for speech (CS ), object image (CO ) and motion (CM ), are calculated to evaluate the feasibilities of the outputted word sequence, thetrajector and landmark, and the motion, respectively. The
weighted sum of these confidence measures with a bias is
inputted to a logistic function. The bias and the weightings {θ0 , θ1 , θ2 , θ3 }, are optimized by logistic regression[Hosmer 09]. Here, the MSC measure is defined as theoutput of the logistic function, and it represents the probability that the speech is RD speech. If the MSC measureis higher than a threshold δ, the robot judges that the input
speech is RD speech and executes an action according to
it. In the rest of this section, we give details of the speech
understanding process and the MSC measure.

∗3 In this work, gaze direction was identified by human face angle.
We used faceAPI (http://www.seeingmachines.com) to extract human face angles from images captured by a camera.

673

Detecting Robot-Directed Speech by Situated Understanding in Physical Interaction

Fig. 5 Sequential photographs of the robot executing an action according to utterance “Place-on Barbazoo red box”.

Camera images

Audio signal

Gaze Tracking

GMM based VAD

Is human user looking at
the robot during his speaking?
Physical situations

NO

YES
Speech Understanding
Object Confidence
Measure CO

Speech Confidence
Measure CS

θ1

Motion Confidence
Measure CM

θ2

θ3

θ0
MSC measure CM S (s, O, q)
CM S (s, O, q) > δ?

NOYESRD speech

OOD speechFig. 6 Flowchart of the proposed RD speech detection method.

3 ·1 Speech Understanding
Given input speech s and a current physical situationconsisting of object information O and behavioral context
q, speech understanding selects the optimal action a basedon a multimodal integrated user model. O is represented
as O = {(o1,f , o1,p), (o2,f , o2,p) . . . (om,f , om,p )}, which includes the visual features oi,f and positions oi,p of all objects in the current situation, where m denotes the numberof objects and i denotes the index of each object that isdynamically given in the situation. q includes information on which objects were a trajector and a landmark in
the previous action and on which object the user is now
holding. a is defined as a = (t, ξ), where t and ξ denotethe index of trajector and a trajectory of motion, respectively. A user model integrating the five belief modules –(1) speech, (2) object image, (3) motion, (4) motion-object
relationship, and (5) behavioral context – is called an integrated belief. Each belief module and the integrated belief
are learned by the interaction between a user and the robot
in a real-world environment.

§ 1 Lexicon and Grammar
The robot has basic linguistic knowledge, including alexicon L and a grammar Gr . L consists of pairs of a word
and a concept, each of which represents an object image
or a motion. The words are represented by the sequencesof phonemes, each of which is represented by HMM using mel-scale cepstrum coefficients and their delta parameters (25-dimensional) as features. The concepts of object
images are represented by Gaussian functions in a multidimensional visual feature space (size, color (L∗ , a∗ , b∗ ),and shape). The concepts of motions are represented by
HMMs using the sequence of three-dimensional positions
and their delta parameters as features.
The word sequence of speech s is interpreted as a conceptual structure z = [(α1 , wα1 ), (α2 , wα2 ), (α3 , wα3 )],where αi represents the attribute of a phrase and has avalue among {M, T, L}. wM , wT and wL represent thephrases describing a motion, a trajector, and a landmark,respectively. For example, the user’s utterance “Place-on
Kermit big box” is interpreted as follows: [(M , Place-on),(T , Kermit), (L, big box)]. The grammar Gr is a statisti-

674

人工知能学会論文誌

cal language model that is represented by a set of occurrence probabilities for the possible orders of attributes in
the conceptual structure.
§ 2 Belief modules and Integrated Belief
Each of the five belief modules in the integrated belief
is defined as follows.
Speech BS : This module is represented as the log probability of speech s conditioned by z, under grammar Gr .
Object image BO : This module is represented as thelog likelihood of wT and wL given the trajector’s and thelandmark’s visual features ot,f and ol,f .
Motion BM : This module is represented as the log
likelihood of wM given the trajector’s initial position ot,p ,the landmark’s position ol,p , and trajectory ξ.
Motion-object relationship BR : This module represents the belief that in the motion corresponding to wM ,features ot,f and ol,f are typical for a trajector and a landmark, respectively. This belief is represented by a multivariate Gaussian distribution of vector [ot,f , ot,f − ol,f , ol,f ]T .
Behavioral context BH : This module represents thebelief that the current speech refers to object o, given behavioral context q.

Given weighting parameter set Γ= γ1 ..., γ5 , the degree of correspondence between speech s and action a isrepresented by integrated belief function Ψ, written asΨ(s,a, O, q, Γ) =
max γ1 log P (s|z)P (z; Gr )z,l+γ2 log P (ot,f |wT ) + log P (ol,f |wL)

[BS ]
[BO ]

+γ3 log P (ξ|ot,p, ol,p , wM )[BM ]+γ4 log P (ot,f , ol,f |wM )B+γ5 H (ot , q) + BH (ol , q) ,

[BR ]
[BH ]

(1)
where l denotes the index of landmark, ot and ol denotethe trajector and landmark, respectively. Conceptual structure z and landmark ol are selected to maximize the valueof Ψ. Then, as the meaning of speech s, corresponding
action â is determined by maximizing Ψ:â = (t̂, ξ̂) = argmax Ψ(s, a, O, q, Γ).
a(2)Finally, action â = (t̂, ξ̂), index of selected landmark ˆl,and conceptual structure (recognized word sequence) ẑ are
outputted from the speech understanding process.
§ 3 Learning the Parameters
In the speech understanding, each belief module and theweighting parameters Γ in the integrated belief are learned
online through human-robot interaction in a natural way in

25 巻 6 号 SP-A（2010 年）

an environment in which the robot is used [Iwahashi 07].
For example, a user shows an object to the robot whileuttering a word describing the object to make the robot
learn the phoneme sequence of the spoken word whichrefers to the object and the Gaussian parameters representing the object image concept based on Bayesian learning.
In addition, the user orders the robot to move an object bymaking an utterance and a gesture, and the robot acts in
response. If the robot responds incorrectly, the user slaps
the robot’s hand, and the robot acts in a different way inresponse. The weighting parameters Γ are learned incrementally, online with minimum classification error learning [Katagiri 98], through such interaction. This learning
process can be conducted easily by a non-expert user. In
contrast, other speech understanding methods need an expert to manually adjust the parameters in the methods, andthe operation is not practical for ordinary users. Therefore,in comparison with other methods, the speech understanding method used in this work has an advantage in that it
adapts to different environments, depending on the user.
3 ·2 MSC Measure
Next, we describe the proposed MSC measure. MSC
measure CM S is calculated based on the outputs of speech
understanding and represents an RD speech probability.
For input speech s and current physical situation (O, q),
speech understanding is performed first, and then CM S iscalculated by the logistic regression as
CM S (s, O, q) = P (domain = RD|s, O, q)=

1
1 + e−(θ0 +θ1 CS +θ2 CO +θ3 CM ).

(3)

Logistic regression is a type of predictive model that canbe used when the target variable is a categorical variablewith two categories, which is quite suitable for the domainclassification problem in this work. In addition, the outputof the logistic function has a value in the range from 0.0 to
1.0, which can be used directly to represent an RD speechprobability.
Finally, given a threshold δ, speech s with an MSC measure higher than δ is treated as RD speech. The BS , BO ,and BM are also used for calculating CS , CO , and CM ,each of which is described as follows.
§ 1 Speech Confidence Measure
Speech confidence measure CS is used to evaluate thereliability of the recognized word sequence ẑ. It is calculated by dividing the likelihood of ẑ by the likelihood ofa maximum likelihood phoneme sequence with phoneme

675

Detecting Robot-Directed Speech by Situated Understanding in Physical Interaction

network Gp, and it is written as
CS (s, ẑ) =1
P (s|ẑ)log,n(s)maxu∈L(Gp ) P (s|u)(4)

where n(s) denotes the analysis frame length of the input
speech, P (s|ẑ) denotes the likelihood of ẑ for input speechs and is given by a part of BS , u denotes a phoneme sequence, and L(Gp ) denotes a set of possible phoneme sequences accepted by phoneme network Gp . For speechthat matches robot command grammar Gr , CS has a greater
value than speech that does not match Gr .
The speech confidence measure is conventionally used
as a confidence measure for speech recognition [Jiang 05].
The basic idea is that it treats the likelihood of the mosttypical (maximum-likelihood) phoneme sequences for the
input speech as a baseline. Based on this idea, the object
and motion confidence measures are defined as follows.
§ 2 Object Confidence Measure
Object confidence measure CO is used to evaluate thereliability that the outputted trajector ot̂ and landmark ol̂
are referred to by ŵT and ŵL. It is calculated by dividing
the likelihood of visual features ot̂,f and ol̂,f by a baseline
obtained by the likelihood of the most typical visual features for the object models of ŵT and ŵL . In this work,
the maximum probability densities of Gaussian functions
are treated as these baselines. Then, the object confidence
measure CO is written as
CO (ot̂,f ,ol̂,f , ŵT , ŵL) =log

P (ot̂,f |ŵT )P (ol̂,f |ŵL)maxof P (of |ŵT ) maxof P (of |ŵL)(5),

where P (ot̂,f | ŵT ) and P (ol̂,f | ŵL) denote the likelihoodof ot̂,f and ol̂,f and are given by BO , and maxof P (of |ŵT ) and maxof P (of | ŵL ) denote the maximum probability densities of Gaussian functions, and of denotes thevisual features in object models.
For example, Figure 7(a) describes a physical situation
under which a low object confidence measure was obtained for input OOD speech “There is a red box.” Theexamples in Figure 7 are selected from the raw data of theexperimental results. Here, by the speech understanding
process, the input speech was recognized as a word sequence “Raise red box.” Then, an action of the robot raising object 1 was outputted (solid line) because the “redbox” did not exist and thus object 1 with the same color
was selected as a trajector. However, the visual feature ofobject 1 was very different from “red box,” resulting in a
low value of CO .
§ 3 Motion Confidence Measure
The confidence measure of motion CM is used to evaluate the reliability that the outputted trajectory ξ̂ is referred

to by ŵM . It is calculated by dividing the likelihood of ξˆby a baseline that is obtained by the likelihood of the mosttypical trajectory ξ̃ for the motion model of ŵM . In thiswork, ξ˜ is written asξ̃ = argmax P (ξ|otraj, ol̂,p , ŵM ),
p
traj

(6)

ξ,op

where otraj
denotes the initial position of the trajector. ξ˜pis obtained by treating otraj
as a variable. The likelihoodpof ξ̃ is the maximum output probability of HMMs. In thiswork, we used the method proposed by [Tokuda 95] toobtain this probability. Different from ξ̂, the trajector’s
initial position of ξ̃ is unconstrained, and the likelihood ofξ̃ has a greater value than ξ̂. Then, the motion confidence
measure CM is written as
CM (ξ̂, ŵM ) = log

P (ξ̂|ot̂,p , ol̂,p , ŵM )maxξ,otraj
P (ξ|otraj, ol̂,p , ŵM )
p
p

,

(7)
where P (ξ̂|ot̂,p , ol̂,p ,ŵM ) denotes the likelihood of ξ̂ andis given by BM .
For example, Figure 7(b) describes a physical situation
under which a low motion confidence measure was obtained for input OOD speech “Bring me that Chutotoro.”
Here, by the speech understanding process, the input speech
was recognized as a word sequence “Move-away Chutotoro.” Then, an action of the robot moving away object 1
from object 2 was outputted (solid line). However, the typical trajectory of “move-away” is for one object to moveaway from another object that is close to it (dotted line).
Here, the trajectory of outputted action was very different
from the typical trajectory, resulting in a low value of CM .
§ 4 Optimization of Weights
We now consider the problem of estimating weight Θ.
The ith training sample is given as the pair of input signal(si , O i , q i ) and teaching signal di . Thus, the training set
TN contains N samples:
TN = {(si , O i , q i , di )|i = 1, ..., N },(8)

where di is 0 or 1, which represents OOD speech or RD
speech, respectively. The likelihood function is written as
P (d|Θ) =N
ii(CM S (si , O i , q i ))d (1 − CM S (si , O i , q i ))1−d ,i=1(9)where d= (d , . . ., d ). Θ is optimized by the maximumlikelihood estimation of Eq. (9) using Fisher’s scoring algorithm [Kurita 92].
1

N

676

人工知能学会論文誌

25 巻 6 号 SP-A（2010 年）

Input speech: “There is a red box.”
Recognized as: [Raise red box.]

Input speech: “Bring me that Chutotoro.”
Recognized as: [Move-away Chutotoro.](a) Case for object confidence measure(b) Case for motion confidence measure

Fig. 7 Example cases where object and motion confidence measures are low. These examples are selected from the raw
data of the experimental results.

4. Experiments
4 ·1 Experimental Setting
We first evaluated the performance of MSC. This evaluation was performed by an off-line experiment by simulation where gaze tracking is not used, and speech is extracted manually without the GMM based VAD to avoid
its detection errors. The weighting set Θ and the threshold δ were also optimized in this experiment. Then we
performed an on-line experiment with the robot to evaluate the whole system.
The robot lexicon L used in both experiments has 50
words, including 31 nouns and adjectives representing 40
objects and 19 verbs representing ten kinds of motions.
Figure 8 shows some of the objects used in the experiments. Figure 9 shows the examples for each motion. The
solid line in each example represents the motion trajectory. L also includes five Japanese postpositions. Different from other words in L, each of the postpositions
is not associated with a concept. By using the postpositions, users can speak a command in a more natural way.
The parameter set Γ in Eq. (1) was γ1 = 1.00, γ2 = 0.75,
γ3 = 1.03, γ4 = 0.56, and γ5 = 1.88.
The speech detection algorithm was run on a Dell Precision 690 workstation, with an Intel Xeon 2.66GHz CPU
and 4GB memory for speech understanding and the calculation of MSC measure. In the on-line experiment, weadded another Dell Precision T7400 workstation with anIntel Xeon 3.2GHz CPU and a 4GB memory for the image
processing and gaze tracking.
4 ·2 Off-line Experiment by Simulation§ 1 Setting
The off-line experiment was conducted under both clean
and noisy conditions using a set of pairs of speech s andscene information (O, q). Figure 7(a) shows an example

Fig. 8 Some of the objects used in the experiments.

of scene information. The yellow box on object 3 represents the behavioral context q, which means object 3
was manipulated most recently. We prepared 160 different
such scene files, each of which included three objects on
average. We also prepared 160 different speech samples(80 RD speech and 80 OOD speech) and paired them withthe scene files. The RD speech samples included words
that represent 40 kinds of objects and ten kinds of motions, which were learned beforehand in lexicon L. EachRD and OOD speech sample included 2.8 and 4.1 words
on average, respectively. Table 1 shows examples of the
speech spoken in the experiment. In addition, a correct
motion phrase, correct trajectory, and landmark objects are
given for each RD speech-scene pair. We then recorded
the speech samples under both clean and noisy conditions
as follows.
• Clean condition: We recorded the speech in a soundproof room without noise. A subject sat on a chair
one meter from the SANKEN CS-3e directional microphone and read out a text in Japanese.
• Noisy condition: We added dining hall noise whoselevel was from 50 to 52 dBA to each speech record
gathered under a clean condition.
We gathered the speech records from 16 subjects, including eight males and eight females. All subjects were
native Japanese speakers. All subjects were instructed to677

Detecting Robot-Directed Speech by Situated Understanding in Physical Interaction

raise

put-down∗

move-closer∗

move-away∗

rotateplace-on∗jump-overplace-on the middle

place-on the left sideplace-on the right side

Fig. 9 Examples for each of the 10 kinds of motions used in the experiments. “ ∗ ” means that synonymous verbs are
given in the lexicon for this motion.

speak naturally as if they were speaking to another human listener. As a result, 16 sets of speech-scene pairs
were obtained, each of which included 320 pairs (160 for
clean and 160 for noisy conditions). These pairs were
inputted into the system. For each pair, speech understanding was first performed, and then the MSC measure
was calculated. During speech understanding, a Gaussian mixture model based noise suppression method [Fujimoto 06] was performed, and ATRASR [Nakamura 06]
was used for phoneme and word sequence recognition.
With ATRASR, accuracies of 83% and 67% in phoneme
recognition were obtained under the clean and noisy conditions, respectively.
The evaluation under the clean condition was performedby leave-one-out cross-validation: 15 subjects’ data wereused as a training set to learn the weighting Θ in Eq. (3),and the remaining one subject’s data were used as a test
set and repeated 16 times. By cross-validation, the generalization performance for different speakers was evaluated. The average values of the weighting Θ̂ learned by
the training set in cross-validation were used for the evaluation under the noisy condition, where all noisy speechscene pairs collected from 16 subjects were treated as atest set.
System performances was evaluated by recall and precision rates, which were defined as follows:
Recall =N cor,
N total

(10)

N cor
,
(11)
N det
where N cor denotes the number of RD speech correctly
detected, N total denotes the total number of existing RD
speech, N det denotes the total number of speech detected
as RD speech by the MSC measure.
Finally, for comparison, four cases were evaluated for
RD speech detection by using: (1) the speech confidence
measure only, (2) the speech and object confidence measures, (3) the speech and motion confidence measures and,P recision =Table 1 Examples of the speech spoken in the experiments.

RD speech

OOD speechMove-away Grover.
Place-on Kermit small box.
Rotate Chutotoro.
Raise red Elmo.

Good morning.
How about lunch?
There is a big Barbazoo.
Let’s do an experiment.

(4) the MSC measure.
We also evaluated the speech understanding using the
RD speech-scene pairs. Differences between the output
motion phrase, trajectory, and landmark objects and thegiven ones were treated as an error in speech understanding.
§ 2 Results
The average precision-recall curves for RD speech detection over 16 subjects under clean and noisy conditions
are shown in Figure 10. The performances of each of fourcases are shown by “Speech,” “Speech + Object,” “Speech+ Motion,” and “MSC.” From the figures, we found that(1) the MSC outperforms all others for both clean andnoisy conditions and, (2) both object and motion confidence measures helped to improve performance. The average maximum F-measures under clean and noisy conditions are shown in Figure 11. By comparing it with the
speech confidence measure only, MSC achieved an absolute increase of 5% and 12% for clean and noisy conditions, respectively, indicating that MSC was particularly
effective under the noisy condition. We also performed thepaired t-test. Under the clean condition, there were statistical differences between (1) Speech and Speech + Object(p < 0.01), (2) Speech and Speech + Motion (p < 0.05),
and (3) Speech and MSC (p < 0.01). Under the noisy
condition, there were statistical differences (p < 0.01) between Speech and all other cases.
Examples of the raw data of the experimental results
are shown in Figure 7 and Figure 12. The examples in
Figure 7 are for OOD speech and have been explained in
Sections 3.2.2 and 3.2.3. The examples in Figure 12 arefor RD speech “Place-on Elmo big box” and “Jump-over678

人工知能学会論文誌

SpeechSpeech + Object
Speech + Motion
MSC (Speech + Object + Motion)(a) Under clean condition

25 巻 6 号 SP-A（2010 年）

SpeechSpeech + Object
Speech + Motion
MSC(b) Under noisy condition

Fig. 10 Average precision-recall curves obtained in the off-line experiment.

(a) Under clean condition(b) Under noisy condition

Fig. 11 Average maximum F-measures obtained in the off-line experiment.

(a) “Place-on Elmo big box”(b) “Jump-over Barbazoo Totoro”

Fig. 12 Examples selected from the raw data of the experiment.

679

Detecting Robot-Directed Speech by Situated Understanding in Physical Interaction

Table 2 Means (m) and variances (v) of weighted confidence measures
for all RD and OOD speech obtained under noisy conditions.

m
v

CS
−0.71
1.1

RD
CO
−0.88
0.55

CM
−0.30
0.72

CS
−3.8
6.4

OOD
CO
−6.0
130

CM
−3.3
23

Table 3 Accuracy of RD speech understanding.

Clean
Noisy

Total
99.8%
96.3%

Detected
100%
98.9%

Barbazoo Totoro”. These utterances were successfully detected by the MSC measure. The processing times (seconds) spent on the speech understanding process and the
MSC-based domain classification was 1.09 and 1.36 forthe examples shown in Figure 6(a) and (b), respectively,
1.39 and 1.36 for the examples shown in Figure 11(a) and(b), respectively. These times indicated that our method
could respond quickly in practical human-robot interactions in real time. Table 2 shows the means and variancesof the weighted confidence measures for all RD and OOD
speech obtained under the noisy condition. Notice thatthe variances of CO and CM have large values for OOD
speech, which means it is difficult to perform RD speechdetection using CO or CM only.
In the experiment, weight Θ and threshold δ were optimized under the clean condition. The optimized Θ̂ were:θ̂0 = 5.9, θ̂1 = 0.00011, θ̂2 = 0.053, and θ̂3 = 0.74. The
optimized δ̂ was set to 0.79, which maximized the average F-measure. This means that a piece of speech with
an MSC measure of more than 0.79 will be treated as RD
speech and the robot will execute an action according tothis speech. The above Θ̂ and δ̂ were used in the on-line
experiment.
Finally, the accuracies of speech understanding using
all RD speech and RD speech detected with the proposed
method are shown in Table 3, where “Total” and “Detected” represent all RD speech and the detected RD speech,
respectively, and “Clean” and “Noisy” represent clean andnoisy conditions, respectively.
4 ·3 On-line Experiment Using the Robot§ 1 Setting
In the on-line experiment, the whole system was evaluated by using the robot. In each session of the experiment,
two subjects, an “operator” and a “ministrant,” sat in frontof the robot at a distance of about one meter from the microphone. The operator ordered the robot to manipulate
objects in Japanese. He was also allowed to chat freely
with the ministrant. Figure 13 shows an example of thisFig. 13 Example of on-line experiment.

experiment. The threshold η of gaze tracking was set to0.5, which means that if the proportion of operator’s gazeat the robot during input speech was higher than 50%, therobot judged that the speech was made while the operator
was looking at it.
We conducted a total of 4 sessions of this experimentusing 4 pairs of subjects, and each session lasted for about
50 minutes. All subjects were adult males. As with theoff-line experiment, the subjects were instructed to speak
to the robot as if they were speaking to another human listener. There was constant surrounding noise of about 48
dBA from the robot’s power module in all sessions. For
comparison, five cases were evaluated for RD speech detection by using (1) gaze only, (2) gaze and speech confidence measure, (3) gaze and speech and object confidence measures, (4) gaze and speech and motion confidence measures and, (5) gaze and MSC measure.
§ 2 Results
During the experiment, a total of 983 pieces of speech
were made, each of which was manually labeled as either RD or OOD. The numbers of them are shown in Table 4. “w/ gaze and w/o gaze” show the numbers of speech
productions that were made while the operator was looking/not looking at the robot. “RD/OOD ” shows the numbers of RD/OOD speech productions that were manuallylabeled after the experiment. Aside from the RD speech,
there was also a lot of OOD speech made while the subjects were looking at the robot (see “w/ gaze” in Table 4).
The accuracies of speech understanding were 97.6% and98.1% for all RD speech and the detected RD speech,
respectively. The average recall and precision rates for
RD speech detection are shown in Figure 14. The performances of each of five cases are shown by “Gaze,” “Gaze+ Speech,” “Gaze + Speech + Object,” “Gaze + Speech+ Motion,” and “Gaze + MSC,” respectively. By using
gaze only, an average recall rate of 94% was obtained (see“Gaze” column in Figure 14(a)), which means that almost
all of the RD speech was made while the operator was
looking at the robot. The recall rate dropped to 90% by680

人工知能学会論文誌

Table 4 Numbers of speech productions in the on-line experiment.

RD
OOD
Total

w/ gaze
155
553
708

w/o gaze
10
265
275

Total
165
818
983

integrating gaze with speech confidence measure, whichmeans some RD speech was rejected by the speech confidence measure by mistake. However, by integrating gazewith MSC, the recall rate returned to 94% because themis-rejected RD speech was correctly detected by MSC.
In Figure 14(b), the average precision rate by using gaze
only was 22%. However, by using MSC, these instancesof OOD speech were correctly rejected, resulting in a highprecision rate of 96%, which means the proposed method
is particularly effective under situations where users makea lot of OOD speech while looking at a robot.

5. Discussion
5 ·1 Using in a Real World Environment
Although the proposed method was evaluated in our laboratory, we consider that our method could be used forreal world environments because the used speech understanding method is adaptable to different environments. In
some cases, however, physical conditions can dynamicallychange. For example, lighting conditions may change suddenly due to sunlight. The development of a method thatworks robustly in such variable conditions is future work.
5 ·2 Extended Applications
This work can be extended in many kinds of ways, andwe mention some of them. Here, we evaluated the MSC
measure under situations where users usually order the
robot while looking at it. However, users possibly order a robot without looking at it under some situations.
For example, in such an object manipulation task wherea robot manipulates objects together with a user, the user
may make an order while looking at the object which heis manipulating instead of looking at the robot itself. For
such tasks, the MSC measure should be used separately
without integrating it with gaze. Therefore, a method that
automatically decides whether to use the gaze informationaccording to the task and user situation should be implemented.
Moreover, aside from the object manipulation task, theMSC measure can also be extended to the multi-task dialog including both the physically grounded and ungroundedtasks. In the physically ungrounded tasks, users’ utterances express no immediate physical objects or motions.

25 巻 6 号 SP-A（2010 年）

For such dialog, a method that automatically switches between the speech confidence and MSC measures shouldbe implemented. In the future works, we will evaluate the
MSC measure for various dialog tasks.
In addition, we can use the MSC to develop an advanced
interface for human-robot interaction. The RD speech probability represented by MSC can be used to provide feedback such as the utterance “Did you speak to me?”, andthis feedback should be made under situations where the
MSC measure has an ambiguous value. Moreover, eachof the object and motion confidence measures can be used
separately. For example, if the object confidence measures
for all objects in a robot’s vision were particularly low,an active exploration should be executed by the robot tosearch for a feasible object in its surroundings, or an utterance such as “I cannot do that” should be made for situations where the motion confidence measure is particularly
low.
Finally, in this work, we evaluated the MSC measureobtained by integrating speech, object and motion confidence measures. In addition, we can consider the use of
the confidence measure obtained from the object-motion
relationship. In the future, we will evaluate the effect ofusing this confidence measure.

6. Conclusion
This paper described an RD speech detection methodthat enables a robot to distinguish the speech to which it
should respond in an object manipulation task by combining speech, visual, and behavioral context with humangaze. The remarkable feature of the method is the introduction of the MSC measure. The MSC measure evaluates the feasibility of the action which the robot is going
to execute according to the users’ speech under the current physical situation. The experimental results clearly
showed that the method is very effective and provides anessential function for natural and safe human-robot interaction. Finally, we would emphasize that the basic idea
adopted in the method is applicable to a broad range ofhuman-robot dialog tasks.

♦ References ♦[Asoh 99] Asoh, H., Matsui, T., Fry, J., Asano, F., and Hayamizu, S.:
A spoken dialog system for a mobile robot, in Proc. Eurospeech, pp.
1139–1142 (1999)[Fujimoto 06] Fujimoto, M. and Nakamura, S.: Sequential NonStationary Noise Tracking Using Particle Filtering with Switching
Dynamical System, in Proc. IEEE Int. Conf. on Acoustics, Speechand Signal Processing, Vol. 2, pp. 769–772 (2006)[Hosmer 09] Hosmer, D. W. and Lemeshow, S.: Applied Logistic Regression, Wiley-Interscience (2009)

681

Detecting Robot-Directed Speech by Situated Understanding in Physical Interaction(a) Recall rates(b) Precision rates

Fig. 14 Average recall and precision rates obtained in the on-line experiment.

[Ishi 06] Ishi, C. T., Matsuda, S., Kanda, T., Jitsuhiro, T., Ishiguro, H.,
Nakamura, S., and Hagita, N.: Robust speech recognition system forcommunication robots in real environments, in Proc. IEEE-RAS Int.
Conf. on Humanoid Robots, pp. 340–345 (2006)[Iwahashi 07] Iwahashi, N.: Robots That Learn Language: A Developmental Approach to Situated Human-Robot Conversations,
Human-Robot Interaction, pp. 95–118 (2007)[Jiang 05] Jiang, H.: Confidence measures for speech recognition: A
survey, Speech Communication, Vol. 45, pp. 455–470 (2005)[Katagiri 98] Katagiri, S., Juangs, B. H., and Lee, C. H.: Patternrecognition using a family of design algorithms based upon the generalized probabilistic descent method, in Proceedings of the IEEE,
Vol. 86, pp. 2345–2373 (1998)[Kawahara 98] Kawahara, T., Ishizuka, K., Doshita, S., and Lee, C.H.: Speaking-Style Dependent Lexicalized Filler Model for KeyPhrase Detection and Verification, in Proc. IEEE Int. Conf. on Spoken
Language Processing, pp. 3253–3259 (1998)[Kurita 92] Kurita, T.: Iterative weighted least squares algorithms forneural networks classifiers, in Proc. Workshop on Algorithmic Learning Theory (1992)[Lang 03] Lang, S., Kleinehagenbrock, M., Hohenner, S., Fritsch, J.,
Fink, G. A., and Sagerer, G.: Providing the Basis for Human-RobotInteraction: A Multi-Modal Attention System for a Mobile Robot, in
Proc. ACM Int. Conf. on Multimodal Interfaces, pp. 28–35 (2003)[Lee 04] Lee, A., Nakamura, K., Nishimura, R., Saruwatari, H., andShikano, K.: Noise robust real world spoken dialogue system using
GMM based rejection of unintended inputs, in Proc. Interspeech, pp.
173–176 (2004)[Mutlu 09] Mutlu, B., Shiwa, T., Kanda, T., Ishiguro, H., andHagita, N.: Footing in human-robot conversations: how robots mightshape participant roles using gaze cues, in Proc. ACM/IEEE Int. Conf.
on Human-Robot Interaction, pp. 61–68 (2009)[Nakamura 06] Nakamura, S., Markov, K., Nakaiwa, H., Kikui, G.,
Kawai, H., Jitsuhiro, T., Zhang, J., Yamamoto, H., Sumita, E., andYamamoto, S.: The ATR multilingual speech-to-speech translation
system, IEEE Trans. on Audio, Speech, and Language Processing,Vol. 14, No. 2, pp. 365–376 (2006)[Takiguchi 08] Takiguchi, T., Sako, A., Yamagata, T., and Ariki, Y.:
System Request Utterance Detection Based on Acoustic and Linguistic Features, Speech Recognition, Technologies and Applications, pp.
539–550 (2008)[Tokuda 95] Tokuda, K., Kobayashi, T., and Imai, S.: Speech parameter generation from HMM using dynamic features, in Proc. Int. Conf.
on Acoustics, Speech, and Signal Processing, pp. 660–663 (1995)[Yonezawa 09] Yonezawa, T., Yamazoe, H., Utsumi, A., and Abe, S.:
Evaluating Crossmodal Awareness of Daily-partner Robot to User’s
Behaviors with Gaze and Utterance Detection, in Proc. ACM Int.
Workshop on Context-Awareness for Self-Managing Systems, pp. 1–8(2009)〔担当委員：稲邑 哲也〕

Received January 30, 2010.

682

人工知能学会論文誌

Author’s Profile
Xiang Zuo (Student Member)is a Ph.D. student at graduate school of Engineering Design,
Kyoto Institute of Technology. He received his B.E. degree
in Mechanical Engineering, and M.S. degree in Information
Science from University of Science and Technology Beijing
and Kyoto Institute of Technology, respectively in 2002 and2007. Since 2009, he has been an intern researcher at Advanced Telecommunications Research labs. His research interests include robot language acquisition and human-robot
interaction. He is a member of the Robotics, InformationProcessing and AcousticalSociety of Japan.

25 巻 6 号 SP-A（2010 年）

Komei Sugiura (Member)is an expert researcher at Knowledge Creating Communication Research Center, National Institute of Information
and Communications Technology (NICT). He received his
B.E. degree in electrical and electronic engineering, M.S.
and Ph.D. degrees in informatics from Kyoto University in
2002, 2004, and 2007, respectively. From 2006 to 2008, he
was a research fellow, Japan Society for the Promotion of
Science, and he has been with NICT since 2008. His research interests include robot language acquisition, spoken dialogue systems, machine learning, and sensor evolution. He is a member of the Society of Instrument
and Control Engineers, and the Robotics Society of Japan.

Natsuki Oka (Member)
Naoto Iwahashi (Member)
Naoto Iwahashi received the B.E. degree in engineering from
Keio University in 1985, Yokohama, Japan. He received
Ph.D. degree in engineering from Tokyo Institute of Technology in 2001. In 1985, he joined Sony Corporation,Tokyo,
Japan. From 1990 to 1993, he stayed at ATR Interpreting
TelephonyResearch Laboratories, Kyoto, Japan. From 1998
to 2003, he was with Sony Computer Science Laboratories
Inc., Tokyo, Japan. In 2003, he joined Advanced Telecommunication Research Laboratories International. In 2006, he Joined National Institute of Information and Communications Technology. His research areas include
interaction speech system, language acquisition, human-robot interaction. He is amember of IEICE, SOFT, JCSS, and RSJ.

Kotaro Funakoshi (Member)
He is with Honda Research Institute Co., Ltd. since 2006.
He received the B.S. degree in 2000 from Tokyo Instituteof Technology, the M.S. and the Dr. Eng. degrees from
Tokyo Institute of Technology in 2002 and 2005, respectively. His research interests are natural language understanding/generation, spoken dialogue systems and conversational robots. He is a member of AAAI, ACM SIGCHI,
IPSJ, and NLP.

Mikio Nakano (Member)
Mikio Nakano, Sc.D. is a Principal Researcher at Honda Research Institute Japan Co., Ltd. (HRI-JP). He received his
M.S. degree in Coordinated Sciences and Sc.D. degree inInformation Science from the University of Tokyo, respectively in 1990 and 1998. From 1990 to 2004, he worked for
Nippon Telegraph and Telephone Corporation. In 2004, hejoined HRI-JP, where he engages in research on speech communication. He is a member of ACM, ACL, IEEE, ISCA,
AAAI, IPSJ, RSJ, IEICE, and ANLP.

Ryo Taguchi (Member)He received the B.E., M.E. and Ph.D. degrees in engineering from ToyohashiUniversity of Technology, in 2002, 2004
and 2008, respectively. Currently, he is working at Nagoya
Institute of Technology. He is a member of the Robotics
Society of Japan, Japanese Cognitive Science Society, Information Processing Society of Japan, and the AcousticalSociety of Japan.

Shigeki Matsuda
received his B.S. degree from the Department of Information Science, Teikyo University, in 1997, completed his doctoral program at the Japan Advanced Institute of Scienceand Technology in 2003. From 2003, he was a researcher at
ATR Spoken Language Communication Laboratories, andin 2008 became a senior research scientist. He is an expert
researcher at National Institute of Information and Communications Tehnology (NICT). He holds a doctoral degree in
information science. He is engaged in research on speech recognition, and is amember of the institute of Electronics, Information and Communication Engineers,
the Acoustic Society of Japan, and Information Processing Society of Japan.

Dr. Oka received the B.E. degree from the University ofTokyo in 1979. After working at Shimadzu Corporation,
the University of Tokyo, Institute for New Generation Computer Technology (ICOT), and Panasonic Corporation, hejoined Kyoto Institute of Technology as a professor in 2003.
His recent concern has been the understanding of and themodeling of cognitive development. He won FIT2007 Best
Paper Award.