Wizard of Oz Evaluation of Listening-oriented
Dialogue Control using POMDP
Toyomi Meguro† , Yasuhiro Minami† , Ryuichiro Higashinaka‡ , and Kohji Dohsaka† ,† NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237 Japan‡ NTT Cyber Space Laboratories, NTT Corporation
1-1 Hikarinooka, Yokosuka, Kanagawa, 239-0847 Japan{meguro.toyomi,minami.yasuhiro,higashinaka.ryuichiro,dohsaka.kohji}@lab.ntt.co.jp
Abstract—We have been working on dialogue control for
listening agents. In our previous study [1], we proposed a dialogue
control method that maximizes user satisfaction using partiallyobservable Markov decision processes (POMDPs) and evaluated
it by a dialogue simulation. We found that it significantly
outperforms other stochastic dialogue control methods. However,this result does not necessarily mean that our method works
as well in real dialogues with human users. Therefore, in thispaper, we evaluate our dialogue control method by a Wizardof Oz (WoZ) experiment. The experimental results show that
our POMDP-based method achieves significantly higher user
satisfaction than other stochastic models, confirming the validityof our approach. This paper is the first to show the usefulness
of POMDP-based dialogue control using human users when thetarget function is to maximize user satisfaction.

Utterance
S: Good evening.
L: Good evening.
S: Our dialogue topic is “food.”
I had curry and rice for dinner.
Do you like curry and rice?
L: Nice to meet you.
What kind of curry?
S: Homemade.
L: I love curry!
S: No secret ingredients whatsoever.
L: No secret ingredients!

Dialogue act
GREETING
GREETING
GREETING
S - DISC (sub: fact)
QUESTION (sub: fact)
GREETING
QUESTION (sub: fact)
S - DISC (sub: fact)
S - DISC (sub: fact)
S - DISC (sub: fact)
REPEAT

Fig. 1.
Excerpt of a typical listening-oriented dialogue. Dialogue acts
corresponding to utterances are shown in parentheses (See Table 1 for their
meanings). S - DISC stands for SELF - DISCLOSURE, S for speaker, and L forlistener. The dialogue was translated from Japanese by the authors.

I. I NTRODUCTION
We have been working on listening-oriented dialogues(LoD) in which one conversational participant attentively
listens to the other (See Fig. 1 for a typical LoD). Our aim is tobuild listening agents that can implement a human-like listening process so that users can satisfy their desire to speak andto be heard. As a first step for building such listening agents,
we are focusing on a dialogue control module that chooses
the best system dialogue act (i.e., meaning representation ofan utterance) at every dialogue point using the dialogue history
and the user dialogue act as input. We focus on this module
because it is the most important one in a listening agent, since
it decides the dialogue flow and because our analysis of LoDs
showed that dialogue flow is what characterizes LoDs [2].
Since it is difficult to completely handcode a listening
process, it is desirable that we train a dialogue control modulefrom data. For this purpose, we have been adopting partiallyobservable Markov decision processes (POMDPs), which canlearn an optimal dialogue control policy from dialogue data.
Our current objective is to train, by using POMDPs, a dialogue control policy that can satisfy users by attentively
listening to them. Note that our target function is different
from conventional POMDP-based work [3] that maximizes
task-completion; little work has focused on maximizing user
satisfaction because defining a target function is difficult. In
our previous work [1], we introduced two kinds of rewards(user satisfaction and smoothness) for the target function of
a POMDP, trained a dialogue control module, and verified its

978-1-4673-0367-5/11/$26.00 ©2011 IEEE

318

superiority to other stochastic methods by dialogue simulation;that is, we used a user simulator to create sets of simulated
dialogues, which were then evaluated by human raters. Human
raters were needed at the final evaluation stage because automatically estimating user satisfaction with precision remains
difficult [4].
Although we successfully showed that our module performs
well in simulations, this does not necessarily mean that it
works as well in real dialogues with human users; whenhumans use a system, they greatly change their behavior
based on system actions, which is not true in simulations.
Simulations are also fraught with the fundamental difficulty ofperfectly tracing human behavior, especially when there is lesstask-restriction as in LoDs. Since we are aiming at building
listening agents that can talk to real users and satisfy them, it
is crucial that we verify the validity of our dialogue control
method with human users.
Therefore, in this paper, we evaluate our dialogue control
method by a Wizard of Oz (WoZ) experiment. Since we only
focus on the dialogue control performance in this work, we introduce WoZ experimenters for understanding user utterancesand generating system utterances in natural language fromsystem dialogue acts. Our contribution lies in being the firstto examine the usefulness of POMDP-based dialogue control
using real human users when the target function is to maximize
user satisfaction.

ASRU 2011

II. R ELATED WORKWith increased attention on social dialogues and senior peer
counseling, work continues to emerge on listening-oriented
dialogues, ranging from the analysis of LoDs to system
implementations.
Regarding analysis, in a WoZ experiment, Maatman et al.
showed that virtual agents can provide users a sense of being
heard using such gestures as nodding and head shaking [5].
Our previous work [2] analyzed the characteristics of LoDs,
in which LoDs were compared with casual conversations,revealing that the two types of dialogues have significantly
different flows and that listeners actively question with frequently inserted self-disclosures. The speaker utterances were
mostly concerned with self-disclosure.
Shitaoka et al. [6] implemented listening agents and focusedon their response generation components. Their system considers the confidence scores of speech recognition and changes
the system response accordingly; it repeats the user utterance
or makes an empathic utterance for high-confidence user
utterances and provides backchanneling when the confidence
is low. The system’s empathic utterances include “I’m happy”
or “That’s too bad,” depending on whether a positive or
negative expression is found in the user utterances. Theirsystem’s response generation only uses the speech recognition
confidence and the polarity of user utterances as cues tochoose its actions. Currently, it does not consider the utterance
content such as dialogue acts. For listening agents to achieve
high smoothness, a switching mechanism between the “active
listening mode,” in which the system is a listener, and the“topic presenting mode,” in which the system is a speaker,
has been proposed [7], [8]. Here, the objective of their system
is to maintain high user interest and to keep the system in an
active listening mode.
All of these implementations use hand-crafted rules for
dialogue control, which we considered needed automationand led to our previous work using POMDPs [1]. POMDPs
were first introduced for robot action control to learn suitable
movements for completing a certain task [9]. Only recently
have they been actively applied to dialogue systems. Williams
et al. [3] successfully used POMDPs for dialogue control in aticket-buying domain to fix the departure and arrival places fortickets. Recent work on POMDPs has indicated that dialogue
control modules can be trained in task-oriented dialogues, but
it is not obvious whether they can be used for less task-oriented
dialogues such as LoDs. We verified their effectiveness for
LoDs by simulation, but further verification by experiments
using human users is necessary.
III. POMDP- BASED D IALOGUE C ONTROL
Before the WoZ experiment, we made two enhancements toour proposed dialogue control that were necessary for naturalinteractions between the WoZ experimenters and the humanusers. We first describe our POMDP-based dialogue control
and then our enhancements.

319

TABLE ID EFINITIONS OF DIALOGUE ACTS WITH SAMPLE GENERATION TEMPLATESTHAT WERE USED BY A W O Z EXPERIMENTER TO GENERATE UTTERANCESFOR DIALOGUE ACTS AND ADAPTED FOR THE “ FOOD ” TOPIC . E XCEPTFOR REPEAT , PARAPHRASE , AND OTHER , EACH DIALOGUE ACT HAS
SEVERAL GENERATION TEMPLATES SHOWN IN BRACKETS . ( TIME ) CAN BE
REPLACED WITH LUNCH , BREAKFAST, OR DINNER , AND ( DAY ) WITHTOMORROW, TODAY, ETC . ( FOOD ) IS FILLED WITH ARBITRARY ITEMS BY A
W O Z EXPERIMENTER BASED ON THE DIALOGUE CONTEXT.
Dialogue act
G REETING

Definition and sample generation templates
Greeting and confirmation of a dialogue theme. e.g.,[Hello.], [The dialogue theme is XX.]
Delivery of objective information. [(place-name) is
I NFORMATION
famous for (food).]
S ELF - DISCLOSURE Disclosure of preferences and feelings.
sub: fact[I ate (food) for (time) (day).][I have eaten (food).]sub: experience[I always go out to eat.]sub: habit
sub: preference[I like (food).][ (food) is delicious.](positive)[I don’t like (food).]sub: preference(negative)[(food) is not delicious.][(food) is (adjective: either positive
sub: preference(neutral)or negative).
I want to eat (food) for (time) (day).
sub: desire
I will eat (food) for (time) (day).
sub: plan
ACKNOWLEDGMENT Encourages conversational partner to speak. [Well.][Aha.]
Utterances that expect answers.
Q UESTION
sub: information[Please tell me about (food).][What did you eat for (time) (day)?]
sub: fact[Have you eaten (food) before?]
sub: experience
sub: habit[Do you usually go out to eat?][Do you like (food)?]sub: preference[What do you want to eat for (time) (day)?]
sub: desire
sub: plan[What are you going to have for (time) (day)?]
Sympathetic utterances and praises. [Me, too]S YMPATHY
N ON - SYMPATHY
Negative utterances. [Me neither.]Confirm what conversation partner said. [Really?]
C ONFIRMATION
Encourage partner to act. [Please try (food).]
P ROPOSAL
R EPEAT
Repeat partner’s previous utterance.
Paraphrase partner’s previous utterance.
PARAPHRASE
Bring up or show goodwill toward partner. [AbsoA PPROVAL
lutely!]Express thanks [Thank you.]
T HANKS
Express regret [I’m sorry.]
A POLOGYF ILLER
Filler between utterances. [Uh.], [Let me see.]
Express affection. [A-ha-ha.]
A DMIRATIONOTHEROther utterances.

A. Training a Dialogue Policy for LoDs
In a POMDP, we train a policy that uses the sum of two
rewards: user satisfaction (i.e., how satisfied one is with thedialogue flow) and smoothness (i.e., how probable/natural the
dialogue flow is) [1]. These two rewards are necessary because
we found that if either is missing, unnatural or unsatisfactory
dialogues result [10].
To obtain our POMDP structure, we first trained the dynamic Bayesian network (DBN) structure shown in Fig. 2 (left)from collected LoDs with their user satisfaction evaluation
scores (obtained by questionnaires). Here, so and sa are thedialogue and action states, o is a speaker observation, a is alistener action, and d is a random variable for an evaluation
score. Then the DBN structure is converted into the POMDP
structure (Fig.2, right), where r is the reward for taking action
a in so , which is defined as the weighted sum of r1 (usersatisfaction) and r2 (smoothness), where r1 is the averaged
evaluation score for (so , a) and r2 is the reward for the

d

so

so

so

o

o

'

o

o

sa

sa

sa

sa

a

a
EƐƚƌƵĐƚƵƌĞ
'

a

'

so

acts generated until the termination of the estimation.
Here, the system basically chooses the first dialogue act
that has the largest reward at each step. However, if
the first dialogue act is “skip,” the system chooses thesecond one, because if the system chooses “skip” for theuser’s estimated “skip”, this could result in uncooperative
dialogue.
By these processes, the system can treat multiple dialogue acts.

r͚͚Fig. 2.

͚'͚a
WKDWƐƚƌƵĐƚƵƌĞ

C. Enhancement 2: Feedback from System Action

DBN and POMDP structures

predictive probability of a.
We can calculate the average reward that will be gained in
the future at time t by:
Vt =

∞

τ =0

γτ



bτ +t ((so , sa ))r((so , sa ), aτ +t ),swhere bt is the belief state (probability distribution overstates) and τ is a discount factor. Here, r((so , sa ), aτ +t ) is
the weighted sum of the two rewards. Finally, a policy islearned by value iteration so that the action can be chosenthat maximizes Vt .
B. Enhancement 1: Dealing with Multiple Dialogue Acts
In our POMDP structure (Fig. 2), a system dialogue act
and a user observation have to occur alternately. However,as the example dialogue (Fig. 1) illustrates, humans do notnecessarily produce a single dialogue act in one turn. Since
the primary purpose of LoD is to let users speak freely andallow them to feel that they are being listened to, we should
avoid limiting the length of user utterances. A system that only
produces one dialogue act at all turns would also be unnatural.
Therefore, we modified our method to allow multiple dialogue acts per user/system turn by introducing a “skip” action.
For training a policy that has “skip,” we first re-annotate thedialogue acts for each utterance in the collected LoDs. Since
there can be one or more dialogue acts in one utterance, we
insert “skip” between the dialogue acts, meaning that a “skip”
operation was done by the conversational partner. Finally,using the skip-inserted data, we learn a policy, as describedin Section III-A. By having “skip,” multiple dialogue acts aretreated as follows:• When the user input has multiple dialogue acts in one
utterance, the system assumes that the system’s “skip”
has been inserted between the dialogue acts and calculates
the output and transition probabilities.
• When the system has the turn, the next user dialogue
act is estimated by a user simulator. If the estimation
result is “skip,” the system remembers that the user act
was “skip” and calculates the next system action. At
this step, the system still does not output its action.
Only when the simulator estimate is not “skip” does thisestimation process stop. The system outputs all dialogue

320

For a system to generate more appropriate system dialogue
acts, we modified the states for system action sa in Fig. 2.
Here, sa is a hidden state and the system has the distributionof sa at each dialogue step. In our proposed model, eachstate of sa has a one-to-one correspondence to each system
dialogue act (See [1] for details, but we did this to facilitate
the calculation of the predictive probability of sa ).
In an on-going dialogue, when the system generates thebest dialogue act, it can immediately be aware of the system’s
dialogue act. With this information as feedback, the system
can expect better distributions for the next step. To be more
concrete, after the system’s generation of its dialogue acts, sa
can be set as non-distribution and the states can be clearly
decided; that is, we set the state’s probability of the best
dialogue act to one and the probabilities of all other states
to zero.
IV. E XPERIMENT
A. Experimental Setup
We evaluated our system by a WoZ experiment. Here, oursystem means our proposed system in [1] with the enhancements explained in the previous section. In a dialogue of theexperiment, one WoZ experimenter acted as a listener and
one evaluator as the speaker. They chatted using a text-chatinterface in separate rooms. The use of facial expressions was
not allowed. Our system was compared to six other systems(described in Section IV-B). For each system including ours,a dialogue took place in the following steps:
1) The evaluator as a speaker inputs a natural language
utterance in the chat interface to talk to the system.
The evaluator does not know that a human is behindthe system.
2) The WoZ experimenter sees the evaluator’s utterance andconverts the natural language utterance into a dialogue
act sequence based on the definition of dialogue acts inTable I. Then he/she inputs the converted dialogue acts
as the speaker’s utterance to the system.
3) The system receives the dialogue act sequence andgenerates the system’s next dialogue act sequence.
4) The WoZ experimenter sees the dialogue act sequenceand generates the system’s natural language utterance
from the sequence. Since each dialogue act has several
possible generation templates, as shown in Table I, theWoZ experimenter chooses an appropriate one matching
the dialogue flow. Some templates have blanks to befreely filled with well-suited words by the experimenter
based on the context.
5) The WoZ experimenter inputs the generated utterance
into the chat interface and waits for the next user
utterance. (Return to Step 1).
We recruited 14 participants (seven males; seven females)
as evaluators and two participants (one female; one male)as WoZ experimenters. The participants did not include theauthors. The WoZ experimenters and the evaluators could notsee each other and were strictly kept in different rooms during
the experiments, including breaks between the dialogues. Eachevaluator talked with two WoZ experimenters and talked
seven times with each WoZ experimenter, each time WoZ
experimenter using one of the seven different systems (theproposed system plus six systems for comparison). Therefore,
each evaluator participated in 14 dialogues. Each dialoguelasted approximately 15 minutes. All dialogues were about“food,” and each had a more specific theme: “yesterday’s
meal,” “favorite food,” “most impressive food,” “specialty food(If you do not have a specialty, your family, partner, friends’
are possible),” “food on trips,” “dining out (recommendationsof restaurants, convenience stores, etc.),” “famous food in yourhometown.” A theme was chosen randomly for each dialogue.
After each dialogue, the evaluators filled in questionnaires
to rate their satisfaction level by answering the following
question on a 7-point Likert scale: “Did you feel that you werelistened to?” At the same time, the WoZ experimenters filled insimilar questionnaires and answered, “Were the dialogue acts
generated by the system good in terms of listening throughthe entire dialogue?”.
B. Systems
In addition to our proposed system, we prepared six dialogue control systems for comparison. For training, thePOMDP systems (Sections IV-B1 and IV-B2) and HMM(Section IV-B5) used the dialogue-act annotated LoDs between
humans that we collected in our previous study (hereafter, LoD
corpus) [1]. The LoD corpus was annotated using the dialogue
act tag set in Table I. The corpus contains 1259 LoDs. In
what follows, we describe the systems we used in the WoZ
experiment one by one.
1) Proposed POMDP: This is our proposed system. The
number of states and weight coefficients of the two rewards(user satisfaction and smoothness) is same as in our previous
study [1].
2) POMDP-only SA: To clarify whether both user satisfaction and smoothness rewards in our method are effective, we
prepared two systems, each of which used one of the rewards.
This system’s structure and learning method are exactly thesame as our proposed system, but the reward is only calculatedusing user satisfaction. We call it the POMDP system with
only user satisfaction POMDP-only SA.
3) POMDP-only Prob: This system, which is the counterpart of POMDP-only SA, uses only the smoothness reward andis called POMDP-only Prob because it only uses the predictive
probability of actions.

3214) Rule: This system (hereafter, Rule system) uses dialogue control rules written by the authors. We conceived thembased on our analysis of LoDs [2] and other counseling-related
research. The system only sees the user’s last user dialogue
act and generates up to four dialogue acts per turn (number of
dialogue acts is determined randomly). The rules were verified
to work well by a separate WoZ experiment [11].
5) HMM: This system (hereafter, HMM system) performs
dialogue control using an HMM trained from LoDs. We havethis system because an HMM is the most typical stochastic
model in previous dialogue research [12]. It chooses the mostprobable dialogue act at each step and predicts whether itshould continue its utterance or pass its turn to the user. If themost probable action at the next step is that of the user, thesystem stops generating dialogue acts. If the most probabledialogue act is that of the system, the system continues togenerate dialogue acts by greedily choosing the most probabledialogue act at each step. The number of states for the HMM
was 18.
6) Human Dialogue Control: When using this system, theWoZ experimenters choose the system dialogue acts (both thedialogue acts and their number) at their own discretion. We
regard this system as an upper bound and call it the Humansystem.
7) Random Dialogue Control: This system randomly
chooses the system dialogue acts. The number of dialogue
acts is also randomly decided from one to four. We call thisthe Random system.
C. Experimental Results1) Subjective Evaluation: Figure 3 shows the averaged
user satisfaction ratings for the seven systems and the resultof the WoZ experimenters’ subjective evaluations. The user
satisfaction rating of our proposed system is the same as theRule system and superior to all of the following: the Random
system, the HMM system, the POMDPs only Prob and the
POMDPs only SA. Except for the Rule system, there are
significant differences (p<0.05) by a statistical test (t-test)
between our proposed system and the other systems.
Our method is superior to the other two POMDP systems with one reward, which means that both rewards arenecessary. It also outperformed the HMM system, which is
one representative of stochastic models, which means that thegeneration of dialogue acts based only on probability is not
sufficient for dialogue control. The user satisfaction ratingsof the Rule system and our proposed method are almost thesame. However, since the rules took several months to write,
our proposed method’s performance is surprising because it
was only trained from data.
The Random system is superior to the HMM system and thetwo POMDP systems with one reward, probably because of theissue of the number of dialogue acts per utterance. In the LoD
corpus (training data for the POMDP-based and the HMM
systems), the average number of dialogue acts per utterance is
1.6, and consequently, the POMDP-based and HMM systems
generated around one to two dialogue acts per utterance. This

Ğƌ
ŽĐ
^
ŶŽ
ŝƚ
ĂƵ
ůĂ
ǀ
Ě
ĞŐ
Ăƌ
Ğǀ


,ƵŵĂŶ

ZƵůĞ

ĞĐ
Ŷ
ĞŐ
ƌĞ
ǀŝ
Ě
ƌĞ
ů
ď
ŝĞ
>ʹ
ŬĐ
Ă
ď
ůů
Ƶ
<

WKDW
WƌŽƉŽƐĞĚ
ŽŶůǇ WƌŽďWKDWZĂŶĚŽŵWKDWŽŶůǇ ^ ,DD

Fig. 3.

Averaged user satisfaction ratings

number of dialogue acts was probably affected by the chatinterface we used to collect the data; entering long (twoor more) sentences was difficult. However, we used anotherinterface with better stability for this WoZ experiment, andlonger utterances were more easily entered. For example, inthe Human system, the WoZ experimenters entered an averageof 2.3 dialogue acts per utterance. In the Random system,
they averaged 2.4 dialogue acts. Note that the Random systemgenerates one to four dialogue acts per utterance. Since thenumber of dialogue acts for the Random system was closer
to that of the Human system than others, it probably induced
good subjective evaluations. Another possible reason is that
the Random system’s generation was made more natural by
the fact that it outputs more than two dialogue acts on average.
Since the probability of generating self-disclosure or questions
is almost half (Table I; there are 15 dialogue acts related toself-disclosure and questions based on their subcategories),
each utterance of the Random system has at least one selfdisclosure or question. This must have made it easier for
the WoZ experimenters to recover from problems caused byrandom dialogue acts that do not fit the context. Even though
the Random system benefited from such long dialogue acts,
our proposed system outperformed it with a smaller number of
dialogue acts (1.4), which indicates how accurately appropriate
dialogue acts were selected by our method.
Figure 3 also shows the WoZ experimenters’ evaluation
scores (See Section IV-A for the questionnaire item). Except
for the Rule and Random systems, we found significant
differences between our proposed system and other systems(p<0.05) by a statistical test (t-test). The orders of the systemsof the user satisfaction ratings and the WoZ experimenters’
evaluation scores were almost the same except that the orderof our proposed method and the Rule system was inverted;
our proposed method outperformed the Rule system. In the
WoZ experimenters’ evaluations, the system’s dialogue acts
were perceived to be better than the Rule system, indicting
the effectiveness of our proposed method. We also find theasymmetry of the results (proposed POMDP vs. Rule system)
interesting; probably, the evaluators preferred rules becauseof its predictability whereas the experimenters liked some
unpredictability to make conversations more engaging.

322

WKDW
ŽŶůǇ WƌŽď
WKDW ZĂŶĚŽŵ
ŽŶůǇ ^,DDWƌŽƉŽƐĞĚ
WKDW ZƵůĞ,ƵŵĂŶ

Fig. 4.
Kullback-Leibler divergence of output distributions of listener’s
dialogue acts when compared with LoDs used for LoD corpus and Humansystems2) Objective Evaluation: To further investigate the dialogue
quality, for the dialogues collected by each system, we calculated the Kullback-Leibler (KL) divergence of the output distribution of the listener’s dialogue acts using the LoD corpus
and the dialogues of the WoZ experimenters manipulating the
Human system; the more closely the distribution resembles
that of the distribution for the LoD corpus (human-humandialogues) or of the distribution of the WoZ experimenters
using the Human system, the better the system should be.
Figure 4 shows the KL divergence results for each system.
Low scores mean that the output distribution of the dialogue
acts are similar. Our proposed method’s KL divergence for
the Human system and the LoD corpus are smaller than theother systems. Therefore, we can confirm that it successfully
generated dialogue acts that are more similar to human’s
dialogue control than the others. The KL divergence between
the Random system and the LoD corpus is small. This meansthat if we ignore the dialogue flow (NB. KL does not take intotransition probabilities), humans generate dialogue acts rather
evenly. However, the user satisfaction of the Random system
is lower than the Human system, suggesting that dialogue flow
is important for user satisfaction. The HMM system’s output
distribution is far from that of the Human system and the LoD
corpus, probably because it generated only highly frequentdialogue acts found in the LoD corpus.
To investigate the quality of the dialogue flow, we constructed language models using dialogue-act trigrams of the
dialogues of each system and measured the perplexity when
the dialogue act sequences of the Human system and those of
the LoD corpus were input into each model. The results areshown in Fig. 5.
If the scores are low, the probability of the N-grams issimilar between the Human system/LoD corpus and eachsystem. The probability of the dialogue act sequences of our
proposed system is as good as the Rule system and superior
to the other systems, which is the same tendency we found
for the user satisfaction ratings. This means that our proposed
method and the Rule system generated dialogue act sequences
similar to humans. The HMM system is just behind the RuleZĂŶĚŽŵ

Ǉƚ
ŝǆ
Ğů
Ɖ
ƌĞ
W

Utterance and dialogue acts (5-best for system dialogue acts)
L: Nice to talk to you. [GREETING:26.0 Q (sub:desire):24.9 AP PROVAL :20.7 SKIP :20.2 S - DISC (sub:pref-):20.1)]
S: Nice to talk to you, too. [GREETING]
L: Are you interested in trying green caviar? [QUESTION (sub: de-

WKDWŽŶůǇ ^sire):24.0 SKIP:21.9 GREETING:21.1 APPROVAL:20.1 S - DISC (sub:pref):20.1]

WKDW
ŽŶůǇ
WƌŽď
WƌŽƉŽƐĞĚ
ZƵůĞ WKDW ,DD

Fig. 5. Perplexity of each system when compared with the models of the
dialogues of the Human system and the LoD corpus.

S: Yes, I am. [SYMPATHY]
I’ve never eaten it. [S - DISC (sub: experience)]
L: I like to eat it in Okinawa. [SKIP:25.6 S - DISC (sub:pref+):21.0
CONFIRMATION :20.7 S - DISC (sub:desire):20.7 INFO :20.4]
S: Let me talk about this dialogue’s theme: “food on trips.” [S - DISC
(sub: fact)]
I have some memories of Okinawa. [S - DISC (sub: pref+)]
L: Oh, really? [SKIP:24.1 CONFIRMATION:20.5 S - DISC (sub: pref+):20.5
INFO :20.0 QUESTION (sub: fact):20.0]
S: I ate rafute (braised pork belly) and goya chample (sauteed
vegetables) there. [S - DISC(sub: fact)]

system and our proposed method, which is reasonable because
it generates dialogue acts using the transition probabilities of
dialogue acts in data.

Fig. 6.
Excerpt of dialogue between a WoZ experimenter (using our
proposed method) and an evaluator. Theme is “food on trips.” INFO stands
for INFORMATION and pref+ for positive preference. S for the evaluator; andL is the WoZ (listener).

,ƵŵĂŶ

D. Obtained Dialogues
Figures 6 and 7 show the dialogues with our proposed
method (Proposed POMDP) and with the Human System. In
Fig. 6, natural language utterances and their dialogue acts areshown in brackets. For the dialogue of Proposed POMDP, wealso show the 5-best dialogue acts with their reward scores at
each step. Since the proposed system generated timely selfdisclosures and questions, the tendency of which is seen when
the Human system is employed as in Fig. 7, we consider
that our proposed method successfully learned human dialogue
control from data.

Utterance
L: Nice to talk to you.
Hello.
S: Hello.
Nice to talk to you, too.
Let’s talk about ”Dining out.”
L: Sure.
Let’s start.
S: I rarely go out to eat.
L: Really?
Do you usually eat at home?
S: Yes, I do.
I usually eat at home.
Do you frequently go out to eat?

Dialogue act
GREETING
GREETING
GREETING
GREETING
GREETINGSYMPATHYGREETING
S - DISC (sub: fact)
CONFIRMATIONQUESTION (sub: habit)
SYMPATHYSYMPATHYQUESTION (sub: habit)

Fig. 7. Excerpt of dialogue between a WoZ experimenter (using Humansystem) and an evaluator. Theme is “dining out.”

V. C ONCLUSION AND F UTURE WORKIn this paper, we evaluated our POMDP-based dialogue control method by a Wizard of Oz (WoZ) experiment and foundthat our POMDP-based method achieves significantly higher
user satisfaction than other stochastic models, confirming thevalidity of our approach. This paper is the first to show theusefulness of POMDP-based dialogue control using humanusers when the target function maximizes user satisfaction
instead of task-completion.
Future work will implement our natural language understanding and generation components since currently, our proposed method can only treat dialogue acts. In addition, wewant to consider other tasks than LoDs, in which we need toconsider different rewards. Although we believe our approachis language-independent, we must also evaluate it in otherlanguages such as English. In addition, we want to pursue
methods to efficiently collect dialogue data for training.
R EFERENCES[1] T. Meguro, R. Higashinaka, Y. Minami, and K. Dohsaka, “Controlling
listening-oriented dialogue using partially observable Markov decision
processes,” in Proc. COLING, 2010, pp. 761–769.
[2] T. Meguro, R. Higashinaka, K. Dohsaka, Y. Minami, and H. Isozaki,“Analysis of listening-oriented dialogue for building listening agents,”
in Proc. SIGDIAL, 2009, pp. 124–127.

323[3] J. Williams and S. Young, “Partially observable Markov decision processes for spoken dialog systems,” Computer Speech & Language,
vol. 21, no. 2, pp. 393–422, 2007.
[4] R. Higashinaka, Y. Minami, K. Dohsaka, and T. Meguro, “Issues in predicting user satisfaction transitions in dialogues: Individual differences,evaluation criteria, and prediction models,” in Proc. IWSDS, 2010, pp.
48–60.
[5] R. M. Maatman, J. Gratch, and S. Marsella, “Natural behavior of alistening agent,” Lecture Notes in Computer Science, vol. 3661, pp. 25–
36, 2005.
[6] K. Shitaoka, R. Tokuhisa, T. Yoshimura, H. Hoshino, and N. Watanabe,“Active listening system for dialogue robot,” in Proc. SIG-SLUD, JSAI,
vol. 58, 2010, pp. 61–66, (in Japanese).
[7] S. Yokoyama, D. Yamamoto, Y. Kobayashi, and M. Doi, “Development
of dialogue interface for elderly people—switching the topic presenting
mode and the attentive listening mode to keep chatting—,” in IPSJ SIG
Technical Report, vol. 2010-SLP-80, no. 4, 2010, pp. 1–6, (in Japanese).
[8] Y. Kobayashi, D. Yamamoto, T. Koga, S. Yokoyama, and M. Doi,
“Design targeting voice interface robot capable of active listening,” inProc. HRI, 2010, pp. 161–162.
[9] J. Pineau, G. Gordon, and S. Thrun, “Point-based value iteration: An
anytime algorithm for POMDPs,” in Proc. IJCAI, 2003, pp. 1025–1032.
[10] Y. Minami, R. Higashinaka, K. Dohsaka, T. Meguro, and E. Maeda,“Trigram dialogue control using POMDPs,” in Proc. SLT, 2010, pp.
336–341.
[11] T. Meguro, R. Higashinaka, Y. Minami, and K. Dohsaka, “Evaluationof listening-oriented dialogue control rules based on the analysis ofHMMs,” in Proc. Interspeech, 2011, (to appear).
[12] K. Shirai, “Modeling of spoken dialogue with and without visual
information,” in Proc. ICSLP, vol. 1, 1996, pp. 188–191.