Bayesian Learning of Conﬁdence Measure Function for Generation of
Utterances and Motions in Object Manipulation Dialogue Task
Komei Sugiura1 , Naoto Iwahashi12 , Hideki Kashioka1 , Satoshi Nakamura12
1 National Institute of Information and Communications Technology
2 Advanced Telecommunications Research Institute International{komei.sugiura, naoto.iwahashi, hideki.kashioka, satoshi.nakamura}@nict.go.jp

Abstract

1. Bayesian logistic regression (BLR) is used for learning a
conﬁdence measure of multimodal utterance understanding so that we can estimate the utility of the robot’s responses.

This paper proposes a method that generates motions and utterances in an object manipulation dialogue task. The proposed method integrates belief modules for speech, vision, andmotions into a probabilistic framework so that a user’s utterances can be understood based on multimodal information. Responses to the utterances are optimized based on an integrated
conﬁdence measure function for the integrated belief modules.
Bayesian logistic regression is used for the learning of the conﬁdence measure function. The experimental results revealed thatthe proposed method reduced the failure rate from 12% down
to 2.6% while the rejection rate was less than 24%.
Index Terms: multimodal spoken dialogue system, robot language acquisition, conﬁdence, Bayesian logistic regression

2. The estimated utility is then used for decision-making on
the responses as motions or conﬁrmation utterances, andfor generating conﬁrmation utterances.
BLR presents several advantages over other methods (e.g. [4])such as (1) predicting the probability of success as a posterior
probability density function, and (2) sample efﬁciency.

2. The LCore Method
The LCore method [3] selects the optimal action based on an
integrated user model trained by multimodal information whena user’s utterance is input. A user model corresponding to each
modality (speech, vision, etc.) is called a belief module. The
user model integrating the ﬁve belief modules – (1) speech, (2)motion, (3) vision, (4) motion-object relationship, and (5) behavioral context– is called the shared belief Ψ.

1. Introduction
The needs of an aging society have raised the hope of robots
supporting humans in daily environments. For such assistive
robots, the functional capability of natural communication withusers is crucial. However, the state-of-the-art techniques havesafety concerns since a user’s utterances are processed with
non-grounded knowledge. Neither the situation nor previous
experiences are taken into account when a robot processes anutterance, so there is a possibility that it will execute motions
that the user had not imagined.
The goal of this study is to decrease the risk. The target taskof this study is called an object manipulation dialogue task inwhich a robot manipulates objects according to a user’s utterances. An example of object manipulation dialogue tasks in ahome environment happens when a user tells a robot to “Put thedish in the cupboard.” Solving this task is fundamental for assistive robots, but it is difﬁcult to program beforehand. This is because many candidate objects exist in the home and the desired
motion depends on elements speciﬁc to each home. Therefore,in object manipulation dialogue tasks two kinds of disambiguations are necessary: (1) the disambiguation of object referenceand (2) the disambiguation of motion reference.
Much work has been done to solve the disambiguation
problems (e.g. [1,2]), however no previous research has realized
the disambiguation of both (1) and (2). On the other hand, we
have proposed the LCore framework that enables robots to learn
the capability of linguistic communication from scratch [3].
In this study, we extend LCore with a scheme of dialogue
management method based upon an adaptive conﬁdence measure. The proposed method called LCore-DEC, which generates motions and utterances in an object manipulation dialoguetask, is presented. LCore-DEC has two key features:

Copyright © 2009 ISCA

2.1. Object Manipulation Dialogue Task
Figure 1 shows an example of an object manipulation dialoguetask. The ﬁgure depicts a camera image in which the robot istold to place Object 1 (Barbabright) on Object 2 (red box). The
solid line shows the trajectory intended by the user. The relative
trajectory between the trajector (moved object) and the reference object is modeled with a hidden Markov model(HMM) [5].
The reference object can be the trajector itself or a landmark
characterizing the trajectory of the trajector. In the case shownin Figure 1, the trajector, reference object, and reference point
are Object 1, Object 2, and Object 2’s center of gravity, respectively.

Figure 1: An example of object manipulation dialogue tasks.
2.2. Utterance Understanding in LCore
An utterance s is interpreted as a conceptual structure z =(WT ,WL ,WM ), where WT , WL , and WM represent the segments
describing the trajector, landmark, and motion, respectively.

2483

6 - 10 September, Brighton UKference in the Ψ values between ak and the action maximizing
Ψ, a j ( j 6= k) :d(s, ak , O, q) = Ψ(s, ak , O, q) − max Ψ(s, a j , O, q)(4)

For example, the user’s utterance, “Place-on Barbabright redbox,” is interpreted as follows:
WT : [ Barbabright ], WL : [ red , box ], WM : [ place-on ]The LCore method does not deal with function words such as
prepositions and articles, i.e. the user is not supposed to use
words such as “on” and “the”.
Suppose that an utterance s is given under a scene O. O
represents the visual features and positions of all objects in thescene. The set of possible actions A under O is deﬁned as follows:h ji
A = {(it , ir ,CV ) | it = 1, ..., ON , ir = 1, ..., RN , j = 1, ...,VN }, {ak | k = 1, 2, ..., |A|},(1)
where it denotes the index of a trajector, ir denotes the index ofa reference object, ON denotes the number of objects in O, RN
denotes the number of possible reference objects for the verb
h ji
CV , and VN denotes the total number of CV in the lexicon.
Each belief module is deﬁned as follows. First, the belief
module of speech, BS , is represented as the log probability of s
conditioned by z. Here, word/segment orders is learned by using bigrams/trigrams. Next, the belief module of motion, BM , isdeﬁned as the log likelihood of a probabilistic model given the
maximum likelihood trajectory Ŷk for ak . The belief moduleof vision, BI , is represented as the log likelihood of WT given
hi i
Object it ’s visual features xI t . Similar to BI , the belief module of motion-object relationship, BR , is represented as the log
likelihood of a probabilistic model given the visual features of
Objects it and ir . The belief module of behavioral context, BH ,represents the adequateness of Object i as the referent under thecontext qhii such as “Object i is being grasped”.
The shared belief function Ψ is deﬁned as the weighted sumof each belief module:{
Ψ(s, ak , O, qhit i ) = max
γ1 log P(s|z)
z

hi i

hi i

j6=k

Let al be an action that gives the second maximum Ψ value.
When the margin for the optimal action âk is almost zero, theshared belief values of âk and al is nearly equal; this means thatthe utterance s is a likely expression for both âk and al . In contrast, a large margin means that s is an unambiguous expression
for âk . Therefore, the margin function can be used as a measureof the utterance’s ambiguity.
Now we deﬁne the integrated conﬁdence measure (ICM)
function by using a sigmoid function as follows:1f (d; w) =
,
(5)
−(w
1 + exp 1 d+w0 )where d is the value of the margin function for an action, andw = (w0 , w1 ) is the parameter vector. The ICM function is used
for modeling the probability of success.
3.2. Learning ICM Function
We now consider the problem of estimating the parameters w of
the ICM function based on logistic regression. The ith training
sample is given as the pair of the margin di and teaching signal
ui , {(di , ui )|i = 1, ..., N}, where ui is 0 or 1.
BLR [7] is used for obtaining the MAP estimate of w. A
univariate Gaussian prior with mean 0 and variance τi (i = 0, 1)
on each parameter wi is used:
−w2i
1
exp
(6)
P(wi |τi ) = N (0, τi ) = √2τi
2πτi3.3. Decision-Making on Multimodal Responses Based on
Expected Utility
Let a∗ be the action that the user intended to indicate by utterings. For safety reasons, it is undesirable for the robot to execute
an incorrect action ak (6= a∗ ). A conﬁrmation request to the user
before the execution of an action can prevent the robot from executing an incorrect action. The ICM function can be used asa criterion for making a decision about whether a conﬁrmation
request is needed prior to executing the optimal action âk . In
the proposed method, the ICM function is also used when conﬁrmation request utterances are generated.
Now we consider the problem of making optimal decisions
on responses to the user’s utterances. We assume that the response is either the execution or conﬁrmation of an action. The
optimal response is selected based upon the expected utility.
Let b1 be a response as a motion and b2 be a response asa conﬁrmation utterance. The ICM function f (d) models theprobability that the utterance is correctly recognized under themargin d. The expected utility E[Ri ] for a response bi (i = 1, 2)
is estimated as follows:
E[Ri ] = ri1 f (d) + ri2 (1 − f (d)),(7)
where ri1 and ri2 denote the utility for bi in the cases of âk = a∗and âk 6= a∗ , respectively.
The equation E[R1 ] = E[R2 ] has the solution f (d) =θ0 (0 < θ0 < 1) under the condition r12 < r22 < r21 < r11 .
Therefore, we can use θ0 as the threshold for selecting the optimal response b̂ = argmaxi E[Ri ].

h ji+γ2 log P(Ŷk |x p t , x p r ,CV )
(
)
hi i
hi i
+γ3 log P(xI t |WT ) + log P(xI r |WL )
hi i

hi i

h ji

+γ4 log P(xI t , xI r |CV )()}+γ5 BH (it , qhit i ) + BH (ir , qhir i ) ,

(2)

hi i
where x p r denotes the position of Object i, and γ = (γ1 , ..., γ5 )denotes the weights of the belief modules. The Minimum Classiﬁcation Error (MCE) learning is used for the learning of γ.
Inappropriate speech recognition results are re-ranked
lower by using Ψ. There are several methods for re-ranking
an utterance hypothesis (e.g. [6]). In contrast, information on
physical properties such as vision and motion is used in Ψ since
object manipulation needs physical interactions.

3. Learning Integrated Conﬁdence Measure
for Generation of Utterances and Motions
3.1. Modeling Conﬁdence for Utterance Understanding
The proposed method quantiﬁes ambiguities in a user’s utterances, and generates motions or utterances as responses by
maximizing a utility function. In this subsection, we ﬁrst explain the ambiguity criterion used in this study.
Given a context q, a scene O, and an utterance s, the optimal
action âk is obtained by maximizing the shared belief function.
âk = argmax Ψ(s, ak , O, q)(3)

3.4. Generation of Conﬁrmation Utterancesak ∈AThe proposed method paraphrases object descriptions to makethem more appropriate for the user and situation. Therefore, a

We deﬁne the margin function d for the action ak ∈ A as the dif-2484

conﬁrmation utterance by the proposed method is not a mere
speech recognition result. To paraphrase the user’s utterances,words are inserted into the segments WT and/or WL . The words
are selected from the lexicon L based on the maximization of
the margin d as follows.
Let ψ(s, ak , O, qhit i , z) be the weighted sum of belief modules. The differences between ψ and Ψ are such that ψ does
not contain acoustic likelihood, and ψ is not maximized withrespect to z (cf. Equation (2)). We deﬁne dz as the margin givenz:dz (s, a j , O, q, z) = ψ(s, a j , O, q, z) − max ψ(s, ak , O, q, z)

multiﬁngered grasper, a microphone/speaker, a stereo vision
camera, and a gaze-expression unit. The visual features and positions of objects were extracted from image streams obtained
from the stereo vision camera. The visual features had six dimensions: three for color (L*a*b*) and three for shapes.
The lexicon used in the experiments contained 23 words (8nouns, 8 adjectives, and 7 verbs). The user taught the names
or properties of objects in Japanese by showing the objects to
the robot. Unsupervised learning was used for obtaining thephoneme sequences of the words [3]. Those words had been
grounded to the physical properties of objects and motions in
the learning phase of the lexicon [3, 5].
To evaluate the proposed method, we conducted two kindsof experiments: (1) learning of the ICM function, and (2) generation of utterances and motions. The objective of Experiment(1) is to investigate the number of samples necessary for convergence of the learning. Experiment (2) was aimed at evaluating
the decrease in the failure rate for the motions.
In Experiment (1), we obtained the training and test data
as follows. A subject was told to sit across the table from therobot, as shown in Figure 1, and make utterances in Japanese1
to make the robot manipulate objects. This ﬂow which starts
from the subject’s utterance and ends with the robot’s manipulation is called an episode. Thus, 100 pairs of utterance s andscene O were obtained. Each pair was labeled with the action
intended by the subject, a∗ . The average chance performancefor all of the data was 2.34%, and the average number of words
contained in each utterance was 2.54. Half of the data was used
as a training set and the other half was used as a test set. The
hyper-parameter τi (i = 0, 1) was set to 100.
In Experiment (2), a subject had object manipulation dialogues with the robot. The training and test set were obtainedin the same manner as Experiment (1). The parameters of the
ICM function were trained by the training set (50 samples) and
ﬁxed in Experiment (2). The dialogue was conducted as follows. First, a sample was drawn from the test set (50 samples),and the scene O was reconstructed. Then, the recorded utterance s for the sample was input to the system, and a response
was selected using the proposed method. If the response wasa conﬁrmation utterance, the user made a positive or negative
response. An executed motion ak was compared with a∗ to determine whether it was correct. An episode ended if the robot
executed a motion or the utterance was rejected. In Experiment(2), θ0 was set to 0.7.

k6= jSuppose that the word set c0 = {c0m | m = 1, ..., M} is inserted into the segment W (WT or WL ). Here W is a sequence ofwords: W , c1 c2 · · · c|W | , where |W | represents the length of W .
The optimal word set ĉ0 = {ĉ0m | m = 1, ..., M} and insertionposition set p̂ = { p̂m | m = 1, ..., M} are obtained as follows:(ĉ0 , p̂) = argmax dz (s, a j , O, q, z)(8)
c0m 6∈W, p

Thus, we obtain the following W 0 after the insertion.
W 0 = c1 · · · c p̂1 −1 ĉ01 c p̂1 · · · c p̂2 −1 ĉ02 c p̂2 · · · c|W |(9)These operations are performed for WT and/or WL , and ﬁnally
we obtain an updated conceptual structure z0 :z0 = (WT0 ,WL0 ,WM ).
(10)
Based on the above, the LCore-DEC algorithm is summarized as follows:
Input Let hO, q, si be an input set; a scene, behavioral context,
and user’s utterance.
1. Generate trajectories for all items in the action candidate
set A (see Equation (1)), and obtain the shared belief values Ψ(s, ak , O, q) for every ak .
2. Obtain the optimal action âk according to Equation (3).
If f (d(s, âk , O, q)) ≥ θ0 holds, then execute âk and terminate. Otherwise go to 3.
3. Initialize the conﬁrmation target set A0 as A0 = A.
4. Let the target action a j = argmax f (d(s, a j , O, q)). Inia j ∈A0tialize the number of inserted words, M = 0.
5. Increment M : M ← M + 1, and generate z0 according toEquation (10).
6. If the updated margin d 0 satisﬁes f (d 0 ) ≥ θ0 , go to 7.
Otherwise go to 6(a).

4.2. Results (1): Learning ICM Function

6(a) If there exists any word that can be added to z0 ,then go to 5. Otherwise go to 9.

The qualitative results for the learning of the ICM function are
shown in the left-hand side of Figure 2.
The right-hand side of Figure 2 shows a quantitative evaluation of the logistic regression. In this ﬁgure, the log likelihood Lof the ICM function given a test set is plotted against the number of training samples. The line shows the average log likelihood, where ten different combinations of a training and test
set were used. The left- and right-hand side ﬁgures reveal that
L converged after 20 training samples. Thus, Figure 2 clearly
indicates that the proposed method could give an appropriate
estimation of the probability.

7. Make a conﬁrmation utterance on a j . A speech is synthesized according to z0 . If WT0 or WL0 has no change fromthe original WT or WL , it is not included in the utterance.
8. If the user’s response is positive, execute a j and terminate. Otherwise remove a j from A0 and go to 8(a).
8(a) If A0 is empty, go to 9. Otherwise, go to 4.
9. Reject s by uttering “Sorry, I cannot understand.”, andthen terminate.

4.3. Results (2): Generation of Motion and Utterances

4. Experiments

First, we address the qualitative results. Figure 3 shows an example dialogue between the subject (U) and the robot (R). The

4.1. Experimental Setup
We conducted experiments using a platform consisting of a manipulator with seven degrees of freedom (DOFs), a four-DOF

1 In this paper, the utterances are translated into English.

2485

¨

Figure 2: Left: The forms of f (d; w) trained by (a) 10, (b) 20,(c) 30, and (d) 50 samples. Right: Average test-set log likelihood of the ICM function.

¥[Situation: Object 2 was manipulated most recently]
U: Move-closer box Elmo.
R: Move-closer green box?
U: No.
R: Move-closer blue box?
U: Yes.
R: (The robot moves Object 3 closer to Object 1.)

ICM value is displayed in the circle at the top right.
In Figure 3, the ICM value of the optimal action âk was
small: f (d) = 0.478 < θ0 . Hence, a conﬁrmation utterance was
the optimal response. Therefore, the robot ﬁrst asked whether“green box” was the trajector. Here, the word “green” was used
to describe the major difference between Object 2 (the greenbox) and Object 3 (the blue box). In the second conﬁrmationutterance, the word “blue” was inserted into the segment WT ,since this gave the maximum margin. In contrast, the landmark
was not mentioned in either generated utterance since no word
insertion to WL had a large inﬂuence on the ICM values.
Table 1 summarizes the quantitative results of decisionmaking based upon the ICM values. In the table, Pf , Pr , Pc ,and Tc represent the incorrect motion execution rate, rejection
rate, conﬁrmation utterance rate, and average number of conﬁrmation requests, respectively:
Pf = N f /(Ns + N f ),Pr = Nr /Na ,Pc = Nc /Na ,where Na , Ns , N f , Nc , and Nr denote the number of all episodes,
episodes in which correct motions were executed, episodes inwhich incorrect motions were executed, episodes in which conﬁrmation utterances were generated, and episodes in which thesubject’s utterances were rejected (i.e. no motions were executed), respectively. Here, Na = Ns + N f + Nr . Tc means thelength of interactions; for example, there were two conﬁrmation requests in Figure 3.
Under the condition θ0 = 0, the robot always executes a
motion as a response to the subject’s utterance. We can regard
this condition as the baseline in which the proposed method
was not used. Pf was 12% (6/50) under this condition. Table 1
shows that Pf was less than 12% under other conditions, where
the proposed method was used. From Table 1, we can see thatPf decreased with an increase in θ0 . Speciﬁcally, we obtained
Pf = 6.5% (3/46) when θ0 = 0.9 and Pf = 2.6% (1/38) whenθ0 = 0.999. Table 1 reveals that conﬁrmation utterances weregenerated in at most half of the scenes since Pc was less than
50% in all cases. Table 1 shows that Tc was approximately 1.2
under all conditions other than those where θ0 = 0.
Finally, we investigate the rejection rate. Table 1 exhibitsthat Pr increased with an increase in θ0 . The episodes that ended
with rejection can be categorized into two groups: (1) utterances giving f (d) ≥ θ0 could not be generated for the scenes,and (2) the subject could not understand the generated utterances. An example of (1) was a scene in which no combinationof the learned words could identify the trajector and/or landmark. Speciﬁcally, one of identical green boxes could not be
identiﬁed since words for spacial relationships such as “right”
or “below” was not learned in the experiments. An example of(2) occurred when the generated utterance included the name ofan object that did not exist in the scene due to uncertainties inimage processing.

§¦

Figure 3: Dialogue example: Motion execution with conﬁrmation utterances. The correct action is to move Object 3 (bluebox) closer to Object 1 (Elmo).
Table 1: Evaluation of decision-making based on the ICM value
θ0
Pf [%]
Pr [%]
Pc [%]
Tc

0
12.0
0
0
-

0.7
10.4
4.0
12.0
1.17

0.9
6.5
8.0
22.0
1.27

0.99
7.1
16.0
28.0
1.21

0.999
2.6
24.0
48.0
1.25

5. Conclusion
In this paper, we proposed LCore-DEC that generates motions
and utterances in an object manipulation dialogue task to decrease the risk of incorrect motion executions by robots. Oneof the contributions of this study is that we integrated learning techniques studied in different research ﬁelds within a probabilistic framework; the learning of motions has been mainly
studied in the robotics community, while the learning of objects
has been studied in the computer vision and artiﬁcial intelligence communities. Another contribution is the introduction ofutility-based dialogue management using Bayesian logistic regression into a multimodal spoken dialogue system.
Acknowledgements
This research was partially supported by the Ministry of Education,
Science, Sports and Culture, Grant-in-Aid for Scientiﬁc Research (C),20500186, 2008, and Tateishi Science and Technology Foundation.

6. References[1] D. Roy, “Learning visually grounded words and syntax for a scene
description task,” Computer Speech and Language, vol. 16, no. 3,
pp. 353–385, 2002.
[2] Y. Yamakata et al., “Belief network based disambiguation of objectreference in spoken dialogue system for robot,” in Proc. of the 7th
ICSLP, 2002.
[3] N. Iwahashi, “Robots that learn language: Developmental approachto human-machine conversations,” in Human-Robot Interaction,
2007, pp. 95–118.
[4] D. Bohus et al., “Online supervised learning of non-understanding
recovery policies,” in Proc. of the IEEE/ACL Workshop on Spoken
Language Technology, 2006, pp. 170–173.
[5] K. Sugiura et al., “Learning object-manipulation verbs for humanrobot communication,” in Proc. of IWMISI, 2007, pp. 32–38.
[6] O. Lemon and I. Konstas, “User simulations for context-sensitive
speech recognition in spoken dialogue systems,” in Proc. of EACL,
2009, pp. 505–513.
[7] A. Genkin et al., “Large-scale bayesian logistic regression for text
categorization,” Technometrics, vol. 49, no. 3, pp. 291–304, 2007.

2486