Speech Communication 84 (2016) 83–96

Contents lists available at ScienceDirect

Speech Communication
journal homepage: www.elsevier.com/locate/specom

Learning cooperative persuasive dialogue policies using framing
Takuya Hiraoka a,∗, Graham Neubig a, Sakriani Sakti a, Tomoki Toda b, Satoshi Nakamura aabGraduate School of Information Science, Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara 603-0192, Japan
Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464–8601, Japana r t i c l e

i n f oArticle history:
Received 2 December 2014
Revised 24 August 2016
Accepted 12 September 2016
Available online 13 September 2016
Keywords:
Cooperative persuasive dialogue
Framing
Reinforcement learning
Dialogue modeling
Dialogue systema b s t r a c t
In this paper, we propose a new framework of cooperative persuasive dialogue, where a dialogue system simultaneously attempts to achieve user satisfaction while persuading the user to take some actionthat achieves a pre-deﬁned system goal. Within this framework, we describe a method for reinforcement
learning of cooperative persuasive dialogue policies by deﬁning a reward function that reﬂects both thesystem and user goal, and using framing, the use of emotionally charged statements common in persuasive dialogue between humans. In order to construct the various components necessary for reinforcement
learning, we ﬁrst describe a corpus of persuasive dialogues between human interlocutors, then propose
a method to construct user simulators and reward functions speciﬁcally tailored to persuasive dialogue
based on this corpus. Then, we implement a fully automatic text-based dialogue system for evaluating
the learned policies. Using the implemented dialogue system, we evaluate the learned policy and theeffect of framing through experiments both with a user simulator and with real users. The experimental evaluation indicates that the proposed method is effective for construction of cooperative persuasive
dialogue systems.
© 2016 Elsevier B.V. All rights reserved.

1. Introduction
With the basic technology supporting dialogue systems maturing, there has been more interest in recent years about dialogue systems that move beyond the traditional task-based or chatter bot frameworks. In particular there has been increasing interest in dialogue systems that engage in persuasion or negotiation(Georgila, 2013; Georgila and Traum, 2011; Guerini et al., 2003;
Heeman, 2009; Mazzotta and de Rosis, 2006; Mazzotta et al., 2007;
Nguyen et al., 2007; Paruchuri et al., 2009). In this paper, we propose a method for learning cooperative persuasive dialogue systems, in which we place a focus not just on the success of persuasion (the system goal) but also user satisfaction (the user goal). Thisvariety of dialogue system has the potential to be useful in situations where the user and system have different, but not mutually
exclusive goals. An example of this is a sales situation where theuser wants to ﬁnd a product that matches their taste, and the system wants to successfully sell a product, ideally one with a higher
proﬁt margin.
Creating a system that both has persuasive power and is able
to ensure that the user is satisﬁed is not an easy task. In order∗

Corresponding author.
E-mail addresses: takuya-h@is.naist.jp, hiraoka.et.al@gmail.com (T. Hiraoka),
neubig@is.naist.jp (G. Neubig), ssakti@is.naist.jp (S. Sakti), tomoki@icts.nagoyau.ac.jp (T. Toda), s-nakamura@is.naist.jp (S. Nakamura).
http://dx.doi.org/10.1016/j.specom.2016.09.002
0167-6393/© 2016 Elsevier B.V. All rights reserved.

to tackle this problem with the help of recent advances in statistical dialogue modeling, we build our system upon the framework of reinforcement learning and speciﬁcally partially observable
Markov decision processes (POMDP) (Levin et al., 20 0 0; Williams
and Young, 20 07; 20 07), which we describe in detail in Section 2.
In the POMDP framework, it is mainly necessary to deﬁne a reward representing the degree of success of the dialogue, the set ofactions that the system can use, and a belief state to keep track of
the system beliefs about its current environment. Once these aredeﬁned, reinforcement learning enables the system to learn a policy maximizing the reward.
In this paper, in order to enable the learning of policies for cooperative persuasive dialogue systems, we tailor each of these elements to the task at hand (Section 4):
Reward: We present a method for deﬁning the reward as a
combination of the user goal (user satisfaction), the systemgoal (persuasive success), and naturalness of the dialogue.
This is in contrast to research in reinforcement learningfor slot-ﬁlling dialogue, where the system aims to achieve
only the user goal (Levin et al., 20 0 0; Williams and Young,
20 07; 20 07), or for persuasion and negotiation dialogues,
where the system receives a reward corresponding to only
the system goal (Georgila, 2013; Georgila and Traum, 2011;
Heeman, 2009; Paruchuri et al., 2009). We use a human-tohuman persuasive dialogue corpus (Section 3, Hiraoka et al.,

84

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

2014a) to train predictive models for achievement of ahuman persuadee’s and a human persuader’s goals, andintroduce these models to reward calculation to enable thesystem to learn a policy reﬂecting knowledge of humanpersuasion.
System Action: We introduce framing (Irwin et al., 2013), which
is known to be important for persuasion, as a system action(i.e., system dialogue act). Framing uses emotionally charged
words (positive or negative) to explain particular alternatives. In the context of research that applies reinforcement
learning to persuasive (or negotiation) dialogue, this is theﬁrst work that considers framing in this way. In this paper
the system controls the polarity (positive or negative) andthe target alternative of framing (see Table 3 for an example
of framing).
Belief State: As the belief state, we use the dialogue features used in calculating the reward function. For example,
whether the persuadee has been informed that a particular
option matches their preference was shown in human dialogue to be correlated with persuasive success, which is oneof the reward factors. Some of the dialogue features reward
calculation can not be observed directly by the system, andthus we incorporate them into the belief state.
Based on this framework, we construct the ﬁrst fully automatedtext-based cooperative persuasive dialogue system (Section 5). To
construct the system, in addition to the policy module, natural language understanding (NLU), and natural language generation (NLG)
are required. We construct an NLU module using the human persuasive dialogue corpus and a statistical classiﬁer. In addition, weconstruct an NLG module based on example-based dialogue, using
a dialogue database created from the human persuasive dialogue
corpus.
Using this system, we evaluate the learned policy and the utility of framing (Section 6). To our knowledge, in context of the research for persuasive and negotiation dialogue, it is ﬁrst time thata learnt policy is evaluated with fully automated dialogue system.
The evaluation is done both using a user simulator and real users.
This paper comprehensively integrates our work in
Hiraoka et al. (2014b) and Hiraoka et al. (2015), with a more
complete explanation and additional experiments. Speciﬁcallyregarding the additional experimental results, in this paper we additionally perform 1) experimental evaluation using a reward function which exactly matches the learning phase (Section 6.1.1, 6.2),
and 2) an evaluation of the effect of NLU error rate (Section 6.1.2).
2. Reinforcement learning
In reinforcement learning, policies are updated based on exploration in order to maximize a reward. In this section, we brieﬂy
describe reinforcement learning in the context of dialogue. In dialogue, the policy is a mapping function from a dialogue state toa particular system action. In reinforcement learning, the policy
is learned to maximize the reward function, which in traditional
task-based dialogue system is user satisfaction or task completion(Walker et al., 1997). Reinforcement learning is often applied tomodels based on the frameworks of Markov decision processes(MDP) or partially observable Markov decision processes (POMDP).
In this paper, we follow a POMDP-based approach. A POMDP isdeﬁned as a tuple S, A, P, R, O, Z, γ , b0  where S is the set ofstates (representing different contexts) which the system may bein (the system’s world), A is the set of actions of the system, P: S
× A → P(S, A) is the set of transition probabilities between states
after taking an action, R: S × A →  is the reward function, O isa set of observations that the system can receive about the world,
Z is a set of observation probabilities Z: S × A → Z(S, A), and γ

a discount factor weighting longterm rewards. At any given time
step i the world is in some unobserved state si ∈ S. Because si is
not known exactly, we keep a hypothesis over states called a belief
state b.1 When the system performs an action α i ∈ A based on b,following a policy π : b → A, it receives a reward ri (si , α i ) ∈  andtransitions to state si+1 according to P (si+1 |si , αi ) ∈ P . The systemthen receives an observation oi+1 according to P (oi+1 |si+1 , αi ). The
quality of the policy π followed by the agent is measured by the
expected future reward, also called the Q-function, Qπ : b × A →.
In this framework, we use Neural ﬁtted Q Iteration(Riedmiller, 2005) for learning the system policy. Neural ﬁtted Q Iteration is an oﬄine value-based method, and optimizes
the parameters to approximate the Q-function. Neural ﬁtted Q
Iteration repeatedly performs 1) sampling training experience
using a POMDP through interaction and 2) training a Q-function
approximator using training experience. Neural ﬁtted Q Iteration
uses a multi-layered perceptron as the Q-function approximator.
Thus, even if the Q-function is complex, Neural ﬁtted Q Iteration
can approximate the Q-function better than using a linear approximation function. In a preliminary experiment, we conﬁrmedthat this is true in our domain as well. Once the Q-function islearned, the system creates the policy based on the Q-function.
In our research, we use the  -greedy policy. Namely, the system
randomly selects an action with a probability of  , otherwise selects the action which maximizes the Q-function given the current
state.
As Porta et al. noted, (discrete-state) POMDPs can be seen as
MDPs with continuous state space that has one dimension perstate, which represents the probability of each state in original
POMDP (Porta et al., 2006). More concretely, assuming the state
space of POMDPs is the discrete set S = {s1 , . . . ., sn , . . . ., sN }, thestate si in corresponding MDPs at time step i can be represented
as follows:

si = (bi (s1 ), . . . , bi (sn ) . . . ., bi (sN )),where bi represents belief state at turn i. In our paper, we followthat discrete-state POMDPs, and treat it as MDPs with continuous
state space. So neural ﬁtted Q iteration should be an appropriate
method to solve this problem.
3. Cooperative persuasive dialogue corpus
In this section, we give a brief overview of cooperative persuasive dialogue, and a human dialogue corpus that we use to construct the dialogue models and dialogue system described in later
sections. Based on the persuasive dialogue corpus (Section 3.1),
we deﬁne and quantify the actions of the cooperative persuader(Section 3.2). In addition, we annotate persuasive dialogue acts ofthe persuader from the point of view of framing (Section 3.3).
3.1. Outline of persuasive dialogue corpus
The cooperative persuasive dialogue corpus (Hiraoka et al.,2014a) consists of dialogues between a salesperson (persuader)and customer (persuadee) as a typical example of persuasive dialogue. The salesperson attempts to convince the customer to purchase a particular product (decision) from a number of alternatives(decision candidates). We deﬁne this type of dialogue as “sales dialogue.” More concretely, the corpus assumes a situation where the1Note that, in this paper we use “belief state” to refer to both 1) known information about a part of the dialogue state (e.g., the most recent system action), and2) a distribution over all possible hypotheses regarding a part of the dialogue state(e.g., the most recent users’ dialogue act). We explain about how we deﬁne thisbelief state in our domain in Section 4.2.3.

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96
Table 2
Sytem and user GPF tags

Table 1The beginning of a dialogue from the corpus (translated from Japanese)
Speaker

Transcription

GPF Tag

Cust

Well, I am looking for a camera,
do you have camera B?
(, .)
(B ?)Yes, we have camera B.
(, B )Did you already take a look at it somewhere?
?)(,  
Yes. On the Internet.
(,  !.)It is very nice. Don’t you think?
!?)Yes, that’s right, yes.
(" ,  !, " )

PropQ

Sales
Sales
Cust
Sales
Cust

Answer
PropQ
Answer
PropQ
Informcustomer is in an appliance store looking for a camera, and thecustomer must decide which camera to purchase from 5 alternatives. The customer can close the dialogue whenever they want,
and choose to buy a camera, not buy a camera, or reserve theirdecision for a later date.
Prior to recording, the salesperson is given the description of
the 5 cameras and instructed to try to convince the customer topurchase a speciﬁc camera (the persuasive target). In this corpus,
the persuasive target is camera A, and this persuasive target is invariant over all subjects. The customer is also instructed to selectone preferred camera from the catalog of the cameras2 , and choose
one aspect of the camera that is particularly important in making
their decision (the determinant). During recording, the customer
and the salesperson converse and refer to the information in thecamera catalog as support for their utterances.
The corpus includes a role-playing dialogue with participantsconsisting of 3 salespeople from 30 to 40 years of age and 19 customers from 20 to 40 years of age. All salespeople have experience
working in an appliance store. The total number of dialogues is 34,
and the total time is about 340 minutes. Table 1 show an example
transcript of the beginning of one dialogue. A further example isshown in Tables 14 and 15 in the appendix.
3.2. Annotation of persuader and persuadee goals
We deﬁne the cooperative persuader as a persuader who
achieves both the persuader and persuadee goals, and cooperative
persuasive dialogue as a dialogue where both the persuader andpersuadee goals have been achieved. To measure the salesperson’s
success as a cooperative persuader, we annotate each dialogue
with scores corresponding to the achievement of the two participants’ goals. As the persuader’s goal, we use persuasive successmeasured by whether the persuadee’s ﬁnal decision (purchased
camera) is the persuasive target or not. As the persuadee’s goal,
we use the persuadee’s subjective satisfaction as measured by results of a questionnaire ﬁlled out by the persuadee at the end of
the dialogue: “Evaluate how satisﬁed you were with the clerk (#$ %5  & )” (1: Not satisﬁed
3: Neutral 5: Satisﬁed).
3.3. Annotated dialogue acts
Each utterance is annotated with two varieties of tags, the ﬁrst
covering dialogue acts in general, and the rest covering framing.
As a tag set to represent traditional dialogue acts, we use general purpose functions (GPF) deﬁned by the ISO international stan2

85

The salesperson is not told this information about customer preferences.

InformAnswer

Question

PropQ

SetQ

Commisive

Directive

dard for dialogue act annotation (ISO24617-2, 2010). All annotated
GPF tags are deﬁned to be one of the tags in this set (Table 2).
More relevant to this work is the framing annotation. Framingis the use of emotionally charged words to explain particular alternatives, and is known as an effective way of increasing persuasive power. The corpus contains tags of all instances of negative/positive framing (Irwin et al., 2013; Mazzotta and de Rosis,2006), with negative framing using negative words and positive
framing using positive words.
The framing tags are deﬁned as a tuple a, p, r where a represents the target alternative, p takes value neg if the framing is
negative, and pos if the framing is positive, and r is a binary variable indicating whether or not the framing contains a reference tothe determinant that the persuadee indicated was most important(for example, the performance or price of a camera). The user’s
preferred determinant is annotated based on the results of the predialogue questionnaire.
Table 3 shows examples of framing. The example shows positive
framing (p=pos) about the performance of Camera A (a=a). In thisexample, the customer answered that his preference is the priceof camera, and this utterance does not contain any description ofprice. Thus, r=no is annotated. An example of negative framing
about Camera B is also shown below.
The annotation is performed by three human workers:
1 The ﬁrst worker segments speaker utterances so that one utterance unit is tagged by only one GPF. After that the ﬁrst worker
annotates framing tags for each utterance.
2 The remaining two workers annotate framing and GPF tags
without looking at the annotation of each other, and modify
segmentation if there are utterances tagged by multiple tags.
For this paper, we re-performed annotation of the framing tags
and evaluate inter-annotator agreement, which is slightly improved
from Hiraoka et al. (2014a). Two annotators are given the description and examples of tags (e.g. what a positive word is), and practice with these manuscripts prior to annotation. In corpus annotation, at ﬁrst, each annotator independently chooses the framing
sentences. Then, framing tags are independently assigned to all utterances chosen by the two annotators. The inter-annotator agreement of target alternative (a) is 91% (kappa=0.813) , framing polarity (p) is 96.9% (kappa=0.903), reference to alternative (r) is 82%(kappa=0.623)
4. Cooperative persuasive dialogue modelingIn this section, we describe a statistical dialogue model for cooperative persuasive dialogue. The proposed cooperative persuasive
dialogue model consists of a user-side dialogue model (Section 4.1)
and a system-side model (Section 4.2).
4.1. User simulator
The dialogue model for the user (customer in Section 3) is used
to simulate the system’s conversational partner in applying reinforcement learning. The user simulator estimates two aspects ofthe conversation:
1. The user’s general dialogue act.
2. Whether the preferred determinant has been conveyed to theuser (conveyed preferred determinant; CPD).

86

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96
Table 3
An example of positive framing ai = A, pi = pos, ri = no (above), and negative framing ai = B, pi =neg, ri = no (below). In these examples, the customer has indicated price as the preferred determinant.
(Camera A is) able to achieve performance of comparable single-lens cameras
and can ﬁt in your pocket, this is a point.
((A")'()*#+,& -.  $+ /,)(.'01*2
)
But, considering the long term usage, you might care about picture quality.
You might change your mind if you only buy a small camera (Camera B).
( , 3% +4.#25+.)(6 , &2 (Camera B) $, ..
525+ .)Fig. 1. Dynamic Bayesian network of the user simulator. Each node represents avariable, and each edge represents a probabilistic dependency. The system cannot
observe the shaded variables.

The users’ general dialogue act is represented using GPF. For
example, in Table 1, PropQ, Answer, and Inform appear as theuser’s dialogue act. In our research, the user simulator chooses one
GPF described in Table 2 or None representing no response at eachturn. CPD represents that the user has been convinced that thedeterminant in the persuader’s framing satisﬁes the user’s preference. For example, in Table 3, the “performance” is contained in
the salesperson’s positive framing for camera A. If the persuadee isconvinced that the decision candidate satisﬁes his/her preference
based on this framing, we say that CPD has occurred (r=yes)3 . In
our research, the user simulator models CPD for each of the 5 cameras. This information is required to calculate reward described in
the following Section 4.2.1. Speciﬁcally, GPF and CPD are used for
calculating naturalness and persuasive success, which are part of
the reward function.
The user simulator is based on an order one Markov chain, andFig. 1 shows its dynamic Bayesian network. The user’s GPF Gt+1
user
t+1
and CPD Calt
at turn t + 1 are calculated by the following probabilities:t
P (Gt+1
user |D , Ueval )

(1)

t+1 t
t
P (Calt|Calt , Fsys
, Gtsys , Ueval ).

(2)

Dt

represents a dialogue act of the speaker who is taking a turnat t. If the user is taking a turn, then Dt represents Gtuser . In addit , Gt }.
tion, if the system is taking a turn, then Dt represents {Fsyssyst represents the
Gtsys represents the system GPF at turn t, and Fsys
system framing at t. These variables correspond to system actions,and are explained in Section 4.2.2. Gtuser represents the user’s GPFt
at t, Calt
represents the CPD at t, and Ueval represents the user’s
original evaluation of the alternatives4 . In our research, this is thedecision candidate that the user selected as a preferred decision
candidate at the beginning of the dialogue5 . Note that a “turn”
3
Note that the persuader does not necessarily know if r=yes because the persuader is not certain of the user’s preferred determinants.
4
Values of these variables are set at the beginning of dialogue, and invariant overthe dialogue.
5
Preliminary experiments indicated that the user behaved differently dependingon the original selection of the decision candidate, thus we introduce this variable
to the user simulator.

means one segment of sentences corresponding to one GPF (except
for “ReleaseTurn”). More concretely, a row in Table 14 corresponds
to one turn. In order to perform mutual turn taking (i.e., the turnbelongs to either the user or system), the GPF of the user simulator is ignored in calculation of the simulator’s next decision during
the system’s turn. An example of an application of Eqs. (1 and 2)
in simulated dialogue is shown in Table 4 We use the persuasive
dialogue corpus described in Section 3 for training the user simulator, considering the customer in the corpus as the user and thesalesperson in the corpus as the system.
We use logistic regression for learning Eqs. (1 and 2). As features, we use a binary vector whose elements correspond to values
t , F t , Gt , U
of Gtuser , Calteval . We performed an experiment evalusys
sys
ating the quality of the user simulator using leave-one-out crossvalidation. In this experiment, we evaluate the simulator in termsof its GPF (Eq. 1) and CPD (Eq. 2) estimation accuracy. Note that,concerning CPD, we evaluate only CPD about camera B because
other factors (such CPD about C) do not affect the actual reward
calculation.6 The result (accuracy, precision, recall, F-measure, andperplexity)7 are described in Table 5. These results are similar tolearned user simulators in other work (Schatzmann et al., 2005),
we hypothesize that the quality of our simulator is acceptable touse.8
4.2. Dialogue modeling: learning cooperative persuasive policies
Now that we have introduced the user model, we describe thesystem’s dialogue management model. In particular, we describe
the reward, system action, and belief state, which are required forreinforcement learning.
4.2.1. Reward
We deﬁne a reward function according to three factors: user
satisfaction, system persuasive success, and naturalness. As the cooperative persuasive dialogue systems must perform dialogue toachieve both the system and user goals, we deﬁne three elementsof the reward function as follows:
Satisfaction (Sat) The user’s goal is represented by subjective
user satisfaction. The reason why we use satisfaction is thatthe user’s goal is not necessarily clear for the system (andsystem creator) in persuasive dialogue. For example, some
6Note that the system goal is persuading the user to purchase camera A. Our
preliminary analysis indicates that informing the user about alternatives (i.e, camera B) other than camera A that match the user’s preference increases the system’s
persuasive power (Hiraoka et al., 2014a).
7
We use accuracy, precision, recall, and F-measure as evaluation criteria in order to follow Schatzmann et al. (2005). However, generally there are situations inwhich there are multiple GPFs that are equally appropriate, and accuracy is an evaluation metric that considers a single GPF only. Therefore, we additionally consider
perplexity as an evaluation metric in this evaluation. In perplexity, a distribution of
possible GPFs is considered.
8
The task and experimental conditions (such domain of the dialogue system) in
previous work is quite different from those of our work, and thus it is diﬃcult tomake a precise comparison.

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

87

Table 4
An example of an application of Equation (1), (2) in simulated dialogue between the system and the user simulator (only the ﬁrst 5 turns are shown).
t

Speaker(taking turn)

F

G

Equation (1), (2)1System1
Fsys= Pos BG1sys = Inform

1
P (G2user |Fsys, G1sys , Ueval = {CameraB} )2

System2= Pos AFsys

G2sys = Inform2
P (G3user |Fsys, G2sys , Ueval = {CameraB} )3

System3= Pos AFsys

G3sys = Inform3
P (G4user |Fsys, G3sys , Ueval = {CameraB} )System

RTurn

4

User

G4user = Other5(simulator)
System2
1
1
P (Cuser|Calt= { }, Fsys
, G1sys , Ueval = {CameraB} )32
2
P (Cuser= {CameraB}, Fsys
|Calt
, G2sys , Ueval = {CameraB} )43
3
P (Cuser|Calt, G3sys , Ueval = {CameraB} )= {CameraB}, Fsys

P (G5user |G4user , Ueval = {CameraB} )G5sys = Question

5
4
P (Cuser|Calt= {CameraB}, Ueval = {CameraB} )P (G6user |G5sys , Ueval = {CameraB} )

6
5
|Calt
= {CameraB}, G5sys , Ueval = {CameraB} )P (Cuser

Table 5
Quality of the user simulator. The row labeled with “GPF” shows the result of the classiﬁcation problem for 6 classes (GPFs shown in Table 2). In addition, the row labeledwith “CPD (about Camera B)” shows the result of the binary-classiﬁcation problem. Scores in brackets are those of the baseline. We use simulators that always output
majority class in training data as baseline in evaluating accuracy, precision, recall, and F-measure. In addition, we use the simulator that follows the distribution of classes
in training data as a baseline in evaluating perplexity. “∗ ” means a signiﬁcant improvement from the baseline (∗ : p < 0.05, ∗∗ : p < 0.01) according to the t-test.

GPF
CDP (about Camera B)

Accuracy

Precision

RecallF-measure

Perplexity

0.410∗∗ (0.370)
0.746 (0.690)

0.301 (0.14)
0.739∗ (0.488)

0.410 (0.374)
0.746∗ (0.698)

0.301 (0.204)
0.742∗ (0.574)

4.516∗∗ (4.815)
1.773 (1.873)

users may want the system to recommend appropriate alternatives, while some users may want the system not torecommend, but only give information upon the user’s request. As the goal is different for each user, we use abstract
satisfaction as a measure, and leave it to each user how toevaluate achievement of the goal.
Persuasive success (PS) The system goal is represented by persuasive success. Persuasive success represents whether the
persuadee ﬁnally chooses the persuasive target at the endof the dialogue. Persuasive success takes the value successwhen the customer decides to purchase the persuasive target at the end of dialogue, and failure otherwise.
Naturalness (N) In addition, we use naturalness as one of the
rewards. This factor is known to enhance the learned policy
performance for real users (Meguro et al., 2011).
t
Satuser

We deﬁne each of these variables formally as follows.
represents a 5 level score of the user’s subjective satisfaction (1:Not satisﬁed, 3: Neutral, 5: Satisﬁed) at turn t scaled into the range
between 0 and 1. P Stsys represents persuasive success (1: success, 0:
failure) at turn t. Nt represents bi-gram likelihood of the dialogue
between system and user at turn t as follows:
t
t−1
t−1
Nt = P (Fsys, Gtsys , Gtuser |Fsys, Gt−1sys , Guser ).

(3)

Next, it is necessary to combine these three factors into a single
reward function. The importance of each goal will vary depending on the use case of the system. For example, a selﬁsh system
could be rewarded with an emphasis on mostly achievement of
the system goal, and a cooperative system could be rewarded withan equal emphasis on achievement of both of the goals. However,in the current phase of our research, we have no evidence that
one of these factors is more important than the other for cooperative persuasive dialogue, and thus would like to treat them as
equally important. Unfortunately, the scale (i.e. the standard deviation) of each factor is different, and thus factors with a larger scale
are considered as relatively important, and other factors are considered as relatively unimportant. For example, in our previous research (Hiraoka et al., 2014b), the scale of naturalness N is smaller
than other factors, and as a result is largely ignored in the learning. Thus, to ensure that all the factors have an equal inﬂuence, wenormalize the factors with the z-score.

Table 6
Features for calculating reward. These features are also used as the system belief
state.
Satuser

PSsys

NFrequency of system commisives
Frequency of system question
Total time
Calt (for all 6 cameras)
Ueval (for all 6 cameras)
System and user current GPF
System and user previous GPF
System framing

These 4 normalized factors are then combined into a single reward as follows:

rt =

t
t
P Stsys − P Ssys
Satuser− Satuser
Nt − N++,
Stddev(Satuser )Stddev(P Ssys )Stddev(N )(4)where variables with a bar represent the mean of variables without a bar, and the Stddev function represents standard deviation of
the argument.
To evaluate these values automatically, Sat and PS are calculatedwith a predictive model constructed from the human persuasive
dialogue corpus described in Section 3 (Hiraoka et al., 2014a). In
constructing these predictive models, the persuasion results (i.e.
persuasive success and persuadee’s satisfaction) at the end of dialogue are given as the supervisory signal, and the dialogue features in Table 6 are given as the input. In the reward calculation,
the dialogue features used by the predictive model are calculated
by information generated from the dialogue of the user simulator
and the system. Table 6 shows all features used for reward calculation at each turn9 .
Statistics (i.e. mean and standard deviation of each factor) arecalculated from simulated dialogue with the dialogue model proposed in this section. Note that in this simulated dialogue, the system obeys a random policy (i.e. randomly selecting the next system action described in Section 4.2.2). We sampled the reward

9
Originally, there are more dialogue features for the predictive model. However
as in previous research (Hiraoka et al., 2014a), we choose a subset dialogue featuresby step-wise feature selection (Terrell and Mutlu, 2012).

88

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

Table 7
System framing. Pos represents positive framing and Neg represents negative framing. A, B, C, D, E represent camera names.
Pos A
Neg A

Pos B
Neg B

Pos C
Neg C

Pos D
Neg D

Pos E
Neg E

None

Table 8System action.
<None, ReleaseTurn><Pos A, Inform><Neg A, Inform><Pos B, Answer><None, Inform><None, Question><None, Directive><None, CloseDialogue><Pos A, Answer><Pos B, Inform><Pos E, Inform><None, Answer><None, Commissive>
Fig. 2. Structure of our dialogue system. Rectangles represent information, andcylinders represent a system module.

factor for 60,0 0 0 turns of simulated dialogue (about 60 0 0 dialogues) to calculate the statistics of each factor.

HGt+1 represents the NLU result (described in Section 5.1) at t.

4.2.2. Action
The system’s action Fsys , Gsys  is a framing/GPF a, p10 pair.
These pairs represent the dialogue act of the salesperson, and are
required for reward calculation (Section 4.2.1). There are 11 typesof framing (Table 7), and 9 types of GPF which are expanded byadding ReleaseTurn and CloseDialogue to the original GPF sets
(Table 2). The number of all possible GPF/framing pairs is 99, andsome pairs have not appeared in the original corpus. Therefore, we
reduce the number of actions by ﬁltering. We construct a unigram
model of the salesperson’s dialogue acts P(Fsales , Gsales ) from theoriginal corpus, then exclude pairs for which the likelihood is below 0.00511 . As a result, the 13 pairs shown in Table 8 remained12 .
We use these pairs as the system actions.

confusion matrix, in Section 5.1, we perform an evaluation of NLU
and use the confusion matrix from this evaluation for the estimattion of Eq. (6). P (Gt+1
user |Guser ) is constructed with the persuasive dialogue corpus described in Section 3.1.
The system uses these estimated distributions over the above
information (i.e., Calt and Guser ) in order to determine its next action. Note that other features, which are also described in Table 6,
are not estimated as a distribution.

4.2.3. Belief state
The current system belief state is represented by the features
used for reward calculation (Table 6) and the reward calculated at
the previous turn. Namely, the features for the reward calculationand calculated reward are also used as the next input of the system
policy.
Note that the system cannot directly observe Calt , thus the system estimates it through the dialogue by using the following probability.
t+1
ˆ |F t , Gt , U ) =P (Calt
eval
sys
sys



t+1
ˆ |Cˆt , F t , Gt , U )P (Cˆt ). (5)P (Caltevalsys
alt sys
alt

ˆt
Calt

ˆ represents the estimated CPD at t + 1, and Ctˆ repret+1
where Calt
alt
sents the estimated CPD at t. The other variables are the same as
those in Eq. (2).
In addition, the system also cannot directly observe Guser , thus
the system estimates it through the dialogue by using the following equation.


P(

|

Gt+1
user HGt+1
user

)= 

t+1
t
t
(HGt+1|Gt+1user )P (Guser |Guser )P (Guser )
user
.
t+1
t
t
|Gt+1
user )P (Guser |Guser )P (Guser )Gtuser P (HGt+1
user

Gtuser P



Gt+1
user

(6)
10
Note that the r is not included in the framing of system action. We assume thesystem can not control r because the system is not certain of the user’s preferreddeterminants.
11
We chose this threshold by trying values from 0.001 to 0.01 with incrementation of 0.001. We select the threshold that resulted in the number of actions closest
to previous work (Georgila, 2013).
12
Cameras C and D are not popular, and don’t appear frequently in the humanpersuasive dialogue corpus, and are therefore excluded in ﬁltering.

user

Other variables are the same as those in Eq. (1) and Eq. (2).
P (HGt+1 |Gt+1
user ) represents the confusion matrix. To construct theuser

5. Text-based cooperative persuasive dialogue system
To evaluate the policy learned with the dialogue model described in Section 4, we construct a fully automated text-based
cooperative persuasive dialogue system. The structure of the system is shown in Fig. 2. Especially, in this section, we describe theconstruction of NLU (Section 5.1) and NLG (Section 5.2) modules
that act as an interface between the policy module and the humanuser, and are necessary for fully automatic dialogue.
5.1. Natural language understanding
The NLU module detects the GPF in the user’s text input uuserusing a statistical classiﬁer. In this paper, we use bagging, using
decision trees as the weak classiﬁer (Breiman, 1996). We require
the NLU to 1) be simple and 2) output the estimated classes withprobability, and bagging with decision trees satisﬁes these requirements. The NLU uses many features (i.e. word frequency), and decision trees can select a small number of effective features, making a simple classiﬁer. In addition, by using bagging, the conﬁdence probability, which is determined by the voting rate of decision trees, can be attached to the classiﬁcation result. We utilize(Weka, 2009) for constructing the bagging classiﬁer.
As input to the classiﬁer, we use features calculated from uuser
and the history of system outputs (usys , Gsys , Fsys ). Features aremainly categorized into 4 types:
Uni: Unigram word frequency in the user’s input.
Bi: Bigram word frequency in the user’s input.
DAcl: The previous action of the system (i.e. GPF/framing pairs
Gsys , Fsys ).
Unicl: Unigram word frequency in the previous system utterance.
As we use Japanese as our target language, we perform morphological analysis using Mecab (Kudo et al., 2004), and use information about the normal form of the word and part of speech toidentify the word.

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

89

Table 9Distribution of the GPF labels in the training data.
Other

Question

SetQuestion

PropositionalQuestion

InformAnswerDirective

Commissive

46

4

12

156

260

117

36

63

Fig. 3. Accuracy of the NLU module. The vertical axis represents accuracy and the horizontal axis represents the NLU feature set. Chance rate is an NLU module that always
outputs Inform.
Table 10The confusion matrix. Each row represents the distribution of the true GPF label. Each column represents the distribution of the NLU classiﬁcation result.
Other

Commissive

PropQ

Directive

AnswerInformSetQQuestion

Classiﬁed as/True label

43
6
0
2
0
1
0
0

0
31
1
2
3
12
0
0

0
2
112
6
5
4
10
3

0
4
3
13
0
4
0
0

0
0
0
0
53
9
0
0

3
20
40
13
56
230
2
1

0
0
0
0
0
0
0
0

0
0
0
0
0
0
0
0

Other
Commssive
PropQ
Directive
AnswerInformSetQQuestionAs the NLU result HGuser , 8 types of GPF are output with membership probabilities. We use 694 customer utterances in the camera sales corpus (Section 3) as training data. In this training data,
8 types of GPF labels are distributed as shown in Table 9.
We evaluate the performance of the NLU module using the features shown above. We prepare 4 patterns of feature sets (Uni,
Uni+DAcl, Uni+CAcl+Unicl and Uni+CAcl+Bi), and evaluate the NLU
module with respect to recognition accuracy of GPF labels in thecustomer’s utterances. The evaluation is performed based on 15fold cross-validation with 694 customer utterances.
From the experimental result (Fig. 3), we can see that NLU with
Uni+CAcl+Bi achieves the highest accuracy, and thus we decided
to use Uni+CAcl+Bi for NLU of the dialogue system in Section 6.
Focusing on the details of the misclassiﬁed GPFs, we show theconfusion matrix for classiﬁcation results of the NLU module with
Uni+CAcl+Bi in Table 10. From this matrix, we can see that Answer is misclassiﬁed to Inform, and that SetQ and Question are
misclassiﬁed into PropositionalQ. This result indicates that thismodule has diﬃculty in distinguishing dialogue acts in a hypernym/hyponym or sibling relationship.
5.2. Natural language generation
Our natural language generation module produces a system utterance utilizing surface information of the previous system utterance or user utterance. Note that our dialogue model Section 4)
and natural language understanding module (Section 5.1) consider
the illocutionary force aspect of utterances (such as “Inform”, “Answer”, and “Question” in GPF), but do not consider semantic content (such as topics of “Question” in GPF, and attibutes of camerain framing) explicitly. Instead, in the natural language generation
module (Section 5.2), the system utterances are generated considering approximated’ semantic content (of both user and system ut-

terance) in order to achieve semantically coherent dialogue. Specifically, it utilizes n-gram and other surface features of speakers utterances in order to approximate semantic content.
The NLG module outputs a system response usys based on theuser’s input uuser , the system’s previous utterance usys and the system action Gsys , Fsys . Though the dialogue assumed in this paper is focusing on a restricted situation, it is still not trivial to create system responses for various inputs. In order to avoid the large
amount of engineering required for template-based NLG and allow
for rapid prototyping, we decide to use the framework of examplebased dialogue management (Lee et al., 2009).
We construct an example database D = {d1 , d2 , . . . , dM } with M
utterances by modifying the human persuasive dialogue corpus of
Section 3. In the example database, the ith datum di = s, u, g, f, p
consists of the speaker s, utterance u, GPF g, framing ﬂag f, andprevious datum p. In modifying the human persuasive dialogue
corpus, we manually make the following corrections:
Deletion of redundant words and sentences (e.g. ﬁllers and restatements).
• Insertion of omitted words (e.g. subjects or objects) and sentences.
•

Our example database consists of 2022 utterances (695 systemutterances and 1327 user example utterances). An example of thedatabase is shown in Table 11.
The NLG module determines the system response usys from D,considering uuser , usys , and Gsys , Fsys . More concretely, our NLG
module performs the following procedure:
1. We deﬁne the response candidate set R, which is a subset ofD, according to whether uuser is null ∅ or not. If uuser = ∅ (i.e.,
the user spoke to the system most recently), then we deﬁne R
as the set of utterances r for which the previous utterance is

90

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

Table 11
Part of the example database. The sentences surrounded by <> are inserted in correction.
Speaker

Utterance

Sys.

What was the good point of camera A?
(A../
?)Well, I like its shape, like a Monolith.
(  !7/.89:;- 22$ )The <main> difference between camera A<and other cameras> is the sensor.
(A.<..,2> "<1= )
It is said that sensors are essential for a digital
camera.
(>?@A"<1= B )The sensor of camera A is the same as that asa single-lens cameras.
(A..<1="3<1=%
+
!7)
In addition, the size of A is similar to othercameras.
(2C4A..,&".
7)
That’s great.
( "
!)

User

Sys.

Sys.

User

GPF

Framing

Question

6.1. Policy learning and evaluation using the user simulator
In this section, we perform two types of evaluation. At ﬁrst, weevaluate the effectiveness of framing and learning policies with theuser simulator (Section 6.1.1). We also perform an evaluation of
how NLU performance affects the learning of the dialogue policy(Section 6.1.2).

Inform

6.1.1. Evaluation for the learned policy and framing
For evaluating the effectiveness of framing and learning the policy through the user simulator, we prepare the following 3 policies.

Pos A

InformInformGuser(i.e., the system spoke to the user most recently, and is con
tinuing speaking), then we deﬁne R so r.p.s = Sys, r.p.g = Gsys ,uinput =



usys
uuser

Random: A baseline where the action is randomly output fromall possible actions.
NoFraming: A baseline where the action is output based on thepolicy which is learned using only GPFs. For constructing theactions, we remove actions whose framing is not None fromthe actions described in Section 4.2.2. The policy is a greedy
policy, and selects the action with the highest Q-value.
Framing: The proposed method where the action is output
based on the policy learned with all actions described in
Section 4.2.2 including framing. The policy is also a greedy
policy.

and r.p. f = Fsys , where Gsys represents GPF and Fsys represents
framing in the previous system action.13
2. Response candidates R are scored based on the following similarity scorecos(r.p.u, uinput ) =In this section, we describe the evaluation of the proposed
method for learning cooperative persuasive dialogue policies. Especially, we focus on examining how the learned policy with framingis effective for persuasive dialogue. The evaluation is done both using a user simulator (Section 6.1) and real users (Section 6.2).

Answera user utterance (r.p.s = User) and annotated with the GPF estimated by NLU (r.p.g = arg max HGuser ). Conversely, if uuser = ∅

6. Experimental evaluation

words(r.p.u ) · words(uinput )| words(r.p.u ) | · | words(uinput ) |(7)(uuser = ∅ )(uuser = ∅ ).

The cosine similarity cos between the previous utterance of the
response sentence candidate r.p.u (r ∈ R) and input sentence
uinput is used for the scoring. uinput is set as usys or uuser depending on uuser . The words function returns the frequency vector of
the content words (i.e. nouns, verbs, and adjectives) weighted
according to tf-idf.
3. The r∗ .u that has the highest score is selected as the output of
the NLG module usysr ∗ = arg max cos(r.p.u, uinput )(8)usys = r ∗ .u.

(9)

r∈R

Note that the language generation is used for generating theactual system utterance corresponding to system action Fsys , Gsys ,and that the decision whether a system should speak more or wait
is determined by the system policy described in Section 4.2. If thesystem selects “< None, ReleaseTurn> ” in Table 8, the corresponding system utterance “How about it? ( D?)” is generated as the system utterance, and then the system waits for theuser response. Otherwise the system keeps speaking.
13
In this paper, we use “.” for representing the membership relation between
variables. For example, Var1.Var2 means that Var2 is a member variable of Var1.

For learning the policy, we use Neural ﬁtted Q Iteration(Section 2) using the Pybrain library (Schaul et al., 2010). We set
the discount factor γ to 0.9, and the number of nodes in the hidden layer of the neural network for approximating the Q-function
to the sum of number of belief states and actions (i.e. Framing:
53, NoFraming: 47). The policy in learning is the  -greedy policy( = 0.3). These conditions follow the default Pybrain settings. We
consider 20 0 0 dialogues as one epoch, and update the parametersof the neural network at each epoch. Learning is ﬁnished when
number of epochs reaches 20 (40 0 0 0 dialogues), and the policy
with the highest average reward is used for evaluation.
We evaluate the system on the basis of average reward per dialogue with the user simulator. For calculating average reward, 10 0 0
dialogues are performed with each policy14 .
Experimental results (Fig. 4) indicate that 1) performance isgreatly improved by learning and 2) framing is somewhat effective for the user simulator. Learned policies (Framing, NoFraming)
get a higher reward than Random. Particularly, both of the learned
policies achieve better user satisfaction and naturalness than Random. In addition, reward of Framing is higher than NoFraming,
speciﬁcally because framing is effective for persuasive success. On
the other hand, user satisfaction of Framing is lower than that of
NoFraming, indicating that there is some tradeoff between user
satisfaction and other factors.
6.1.2. Evaluation of the effect of NLU performance
To elucidate how the performance of NLU affects the learning of
the policy, we prepare 4 Framing policies with different NLU error
14
We also optimized the policy in the case where the reward (Equation (4)) is
given only when dialogue is closed. However, the learning did not converge well,and thus we use the reward (Equation (4)) instead. The use of rewards that are
given incrementally, like Equation (4), to improve learning speed and convergenceis called “Reward shaping” in reinforcement learning literature (see Asri et al., 2013;
Ng et al., 1999 for the detail). The convergence of the learning in each reward condition is shown in Figs. 9 in the appendix.

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

91

average entropy (Fig. 6), the average entropy of the estimated GPF
reaches the highest value at Err75%, and its value gradually decreases as the system error rates decrease from Err75%. These results indicate that there is a correlation between the performanceof NLU and overall evaluation (reward) of the system in our persuasive dialogue model. Note that, in experimental evaluations inother sections, we use NLU constructed in the Section 5.1. This NLU
error rate is about 30%, and we can expect that the average rewardof this system will be close to that of Err25%.
6.2. Complete system evaluation with real users
Fig. 4. Average value of reward (on z-scaled scale described in Section 4.2.1) for
dialogue with the simulator. Error bars represents 95% conﬁdence intervals. Rew
represents the reward, Sat represents the user satisfaction, PS represents persuasive
success, and Nat represents naturalness.

Fig. 5. Average reward for dialogue with the user simulator. Error bars represents
95% conﬁdence intervals.

To test whether the gains shown on the user simulator will
carry over to an actual dialogue system, we perform an experimentwith real human users. In this section, we describe the results ofour user study evaluating fully automated cooperative persuasive
dialogue systems. The system follows the structure proposed in
Section 5. The purpose of this is experimental comparison of policies learnt over simulated dialogue (Section 6.1.1) and an actual
human policy. We hypothesize that Framing in Section 6.1.1 is thebest policy among the learnt policies, and comparable to Humanin terms of rewards deﬁned by Eq. (4).
For evaluation, in addition to the policies described in
Section 6.1.1, we add the following policy.
Human An oracle where the action is output based on humanselection. In this research, the ﬁrst author (who has no formal sales experience, but experience of about 1 year in analysis of camera sales dialogue) selects the action.
We evaluate policies on the basis of average reward and correct response rate of dialogues with real users. The deﬁnition of
the reward is described in Section 4.2.1. In addition, the correct
response rate is the ratio of correct system responses to all system responses. In the experiment, the dialogue system proposed in
Section 5 plays the salesperson, and the user plays the customer.
At the end of the dialogue, to calculate the reward, the user answers the following questionnaire:

Fig. 6. Average entropy of the GPF distribution. Error bars represents 95% conﬁdence intervals.

rates (Err0%, Err25%, Err50%, Err75%). These policies are basicallythe same as the Framing policy in the previous section. However,
outputs of the NLU module in these policies contains errors basedon a confusion matrix with an overall error rate corresponding totheir name (e.g the NLU error rate in Err25% is 25%). These confusion matrices are randomly created at the beginning of each dialogue. We use Neural ﬁtted Q Iteration, whose learning parameters(i.e γ and  ), and number of epochs and dialogues are the sameas the previous section.
We evaluate the system on the basis of average reward per dialogue with the user simulator. For calculating average reward, 10 0 0
dialogues are performed with 20 learned policies at each error
level. In addition, we investigate the informativeness of the estimated GPF distribution by calculating entropy (i.e Eq. (6)).
Experimental results (Figs. 5, 6) indicate that average rewards
reach the minimum value with the policy where the estimated GPF
reaches the highest average entropy. Focusing on the average reward of each system (Fig. 5), the average reward of Err75% is smallest of policies, and the average reward gradually decreases as theerror rate of policies approaches 75%. In addition, focusing on theSatisfaction: The user’s subjective satisfaction deﬁned as a 5
level score of customer satisfaction (1: Not satisﬁed, 3: Neutral, 5: Satisﬁed).
Final decision: The camera that the user ﬁnally wants to buy.
In addition, to calculate the correct response rate, we havethe user annotate information regarding whether each system response is correct or not. When we instructed each user about the
annotation, we simply ask them to “Mark system responses thatseem incorrect to you (2#,  52 E;FG.# %H  5$ & .)”. An example of correct/incorrect system responses is shown in Table 12. 13 users perform one dialogue withthe system obeying each policy (a total of 4 dialogues per users).
Experimental results for the reward are shown in Fig. 7. These
results indicate that learning a policy with framing is effective inthe text-based cooperative dialogue system15 . We can see that the
reward of Framing is higher than that of NoFraming and Random,and not statistically different from Human. The naturalness of Human is less than that of Framing. One of the reasons for this isthat naturalness is automatically evaluated by Equation (3), andFraming is learnt considering this objective naturalness because it
is included in the reward (i.e., equation (4)). In contrast to Framing,
15
Note that scores in Fig. 7 are normalized into z-score (see Equation (4) in
Section 4.2.1). The mean of user satisfaction in the original scale is 3.18 (close to“Neutral”), and thus the value zero in user satisfaction in Fig. 7 is equal to 3.18 in
the original scale.

92

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

Table 12
Part of a dialogue between Framing and an evaluator (translated from Japanese). Note that user’s GPF is estimated. In this example, the user prefers camera E at the beginningof the dialogue. The system utterance with bold font was regarded as an incorrect response by the user.
Speaker

Transcription

SystemShutter-speed of camera A is faster than that of camera B.
Have you ever taken a picture and had it turn out different from what you want?
For example, when you take a picture of a face, and the face in the picture looks quite different.
If the shutter-speed is high, the camera can capture the object in a moment.
The main difference between camera A and other cameras is the sensor.
It is said that sensors are essential for a digital camera.
The sensor of camera A is the same as that of a single-lens cameras.
In addition, the size of A is similar to other cameras.
(Camera A is) able to achieve performance of comparable single-lens cameras
and can ﬁt in your pocket, this is a point.
Camera A is most popular in Kakaku.com now.
Camera E is lightweight.
But, camera A is very popular now.
The resolution of (Camera) A is 2090 million pixels, and zoom is 3.6 times.
(Camera) A uses a very big lens, and processes the picture faster and with high resolution.
How about it?
Is (camera) A more heavy than (camera) E?
Which pictures do you want to take? Far or near?
Considering the long term usage, you might care about picture quality.
Camera A will satisfy such your need.
There are basically no cameras similar in performance to camera A.
I hope that, at ﬁrst, you look up all of the cameras, and then decide which camera you want.
But, Camera A is deﬁnitely most popular.
How about it?
Which one is more reasonably priced? A or B?
If you buy the cheap camera at ﬁrst and re-buy new one, the total price is expensive. Right?
The general price of a single-lens is about 100 or 200 thousand yen.
Considering these prices, camera A is a good choice.
How about it?
(Camera) A seems to be good.
Dialogue is closed by the user 

SystemSystemSystemSystemSystemSystemSystemSystemUser
SystemSystemSystemSystemUser
SystemSystemSystemUser

Fig. 7. Average value of reward (on z-scaled scale described in Section 4.2.1) for
dialogue with real users. Error bars represent 95% conﬁdence intervals. Rew represents the reward, Sat represents the user satisfaction, PS represents persuasive
success, and Nat represents naturalness.

Fig. 8. Correct response rate of the system utterances.

Human does not carefully focus on increasing this objective naturalness given by Equation (3), resulting in it scoring lower objective naturalness than Framing. Experimental results for the correct
response rate (Fig. 8) indicate that our cooperative persuasive dia-

Fra

GPF

Pos A

InformPos A
None

InformInform

Pos A
Pos A
Pos E
Pos A

InformInformInformInform

Pos A
None
None
None

Inform
RTurn
PropQ
PropQ

Pos A

Inform

Pos A
None
None
None

InformRTurn
PropQ
Question

Pos A
None
None

InformRTurnInformlogue system somewhat correctly responds to the user’s input. The
scores of all policies are higher than 70%, and the score of Framing
is about 77%. In addition, even the Random policy achieves a scoreof about 70%. One of the reasons for this is that NLG method usedby our system (Section 5.2) is based on examples, and thus is able
to return natural responses that will only be judged as incorrect ifthey do not match the context.
We can see that some features in human persuasive dialogue
appear in the dialogue between users and the system obeying
the Framing policy. An example of a typical dialogue of Framingis shown in Table 12 (original Japanese transcription is shown in
Table 13). The ﬁrst feature is that most of the framing that thesystem performs in the dialogue is positive framing for camera A.
Even when the user asks about other topics (e.g. camera B∼E anddeterminants) which are not camera A, the system tries to perform framing for camera A. This feature commonly appeared in
the human persuasive dialogue. The second feature is that the system checks or asks about user’s proﬁle and their thoughts before
performing framing. This feature is often found in human dialogue
when the user satisfaction is high. In contrast to these features,
there are some feature which do not appear in human dialogue.
One of the features is that the system talks much more than theuser. In the dialogue, most of the dialogue is occupied with system
dialogue, and the number of user utterances is very small (on average of about 3 or 4 utterance). One of the reasons for this is thatthe reward for the system is determined according to estimated rewards on a human corpus, which use the features in Table 6. It canbe noted from this table that there is no feature other than total
time preventing the system from being overly verbose, largely due
to the fact that none of the human persuaders used in the training
data showed this kind of behavior. This indicates that we might
potentially get further improvements in the system by using data
not only from human-human interactions, but also from humancomputer interactions in the calculation of the reward function.

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

Considering the evaluation result of Section 6.1.1, we can seethat trend of reward and its factors differs somewhat between theuser simulator and the real users. While the naturalness and reward of Framing are identical in Figs. 4 and 7, the systems are
given excessively high Sat in simulation. In addition, systems are
given underestimated PS in simulation. One of the reasons for thisis that the property of dialogue features for the predictive model
for reward differs from previous research (Hiraoka et al., 2014a).
In this paper, dialogue features for the predictive model are calculated at each turn. In addition, persuasive success and user satisfaction are successively calculated at each turn. In contrast, in previous research, the predictive model was constructed with dialogue
features calculated at end of the dialogue. Therefore, it is not guaranteed that the predictive model estimates appropriate persuasive
success and user satisfaction at each turn16 . Another reason is thatthe simulator is not suﬃciently accurate to use for reﬂecting realuser’s behavior. Compared to other works (Meguro et al., 2010;
Misu et al., 2012), we are using a relatively small sized corpus fortraining the user simulator. Therefore, improving the user simulator is an important challenge for future work.
In this Section (and the previous section), we performed evaluation of each policy, where we compare NoFraming and Framing
to elucidate the effect of framing. In the current set of these evaluations, Framing has two advantages over NoFraming: ﬁrst, it has
an awareness of the framing feature of actions, and second, it has
more actions to choose from. It is possible that some of the gain
observed is from the fact that the framing actions are better, independent of the framing feature. Detailed evaluation of the effect ofuse of framing is a potential avenue for future work.
7. Related work
There are a number of related works that apply reinforcement
learning to persuasion and negotiation dialogue. (Georgila andTraum, 2011) apply reinforcement learning to negotiation dialogue
using user simulators divided into three types representing individualist, collectivist, and altruist. Dialogue between a ﬂorist anda grocer is used as an example of negotiation dialogue. In addition, Georgila (2013) also applies reinforcement learning to twoissue negotiation dialogue where participants have a party, anddecide both the date and food type. A handcrafted user simulator
is used for learning the policy of each participant. Heeman, 2009)
models negotiation dialogue, assuming a furniture layout task, andParuchuri et al. (2009) model negotiation dialogue, assuming the
dialogue between a seller and buyer.
Our research differs from these in three major ways. The ﬁrst
is that we use framing, positive or negative statements about theparticular item, which is known to be important for persuasion(Irwin et al., 2013). By considering framing, the system has the potential to be more persuasive. While there is one previous example of persuasive dialogue using framing (Mazzotta et al., 2007),
this system does not use an automatically learned policy, relying
on handcrafted rules. In contrast, in our research, we apply reinforcement learning to learn the system policy automatically.
In addition, in these previous works, rewards and belief states
are deﬁned with heuristics. In contrast, in our research, reward isdeﬁned on the basis of knowledge of human persuasive dialogue.
In particular, we calculate the reward and belief state using thepredictive model of Hiraoka et al. (2014a) for estimating persuasive success and user satisfaction using dialogue features. In thereal world, it is unclear what factors are important for achieving
16
However, the learning of the policy converges better than in the condition
where the reward is given only at the end of dialogue (Figs. 9). Therefore, we believe giving reward at each turn is an effective way to ensure the policy is learned
robustly.

93

the dialogue goal in many persuasive situations. By considering
these predictions as knowledge of human persuasion, the system
can identify the important factors in human persuasion and cantrack the achievement of the goal based on these.
Finally, these works do not evaluate the learned policy, or evaluate only in simulation. In contrast, we evaluate the learned policy
with real users.

8. Conclusion
In this paper, we applied reinforcement learning for learning
cooperative persuasive dialogue system policies using framing, andevaluated the learned policies with a fully automated dialogue system. In order to apply reinforcement learning, a user simulator
and reward function were constructed based on a human persuasive dialogue corpus. Then, we implemented a fully automatic dialogue system for evaluating the learned policies. We evaluated thelearned policy and effect of framing using the constructed dialoguesystem, a user simulator and real users. Experimental evaluation
indicates that applying reinforcement learning is effective for construction of cooperative persuasive dialogue systems that use framing.
In the future, we plan to evaluate the system policies in more
realistic situations, that move beyond role-playing to real sales situations over more broad domains. In this research, corpus collection and evaluation are performed in a role-playing situation.
Therefore, we plan to evaluate the system policies in a real sales
scenario such as in a store with actual customers.
Further, we plan to collect additional corpora in several domains and conditions allowing us to broaden the domains to which
the proposed method can be applied. Perhaps the most importantavenue of future work is improving our current dialogue model(especially, framing in the system action) to be more general andportable. In our current dialogue model, the system can not persuade the user to make decisions (e.g, purchase a camera) that
did not appear in the training data. One way for dealing withthis problem is considering features of decision candidates (e.g.,
price of camera) instead of the decision candidate itself in systemframing. To do so, we plan to consider an argumentation framework that uses features of decision candidate (Bench-Capon, 2003;
Carenini and Moore, 2006) into the current framing framework.
We also plan to consider multimodal information (Nouri et al.,2013) for estimating persuasive success and user satisfaction. We
plan on collecting a multimodal corpus that includes such nonverbal information, and expand our dialogue model to consider
this information.
In addition, there is an open problem about the best method
to measure for achievement of the user’s goal. In this paper, we
use users’ (persuadees’) subjective satisfaction (i.e., how much theusers were satisﬁed with the systems (persuaders)) to quantify
achievement of user goal (following previous work Walker et al.,1997), but there are many other alternatives. One alternative is directly evaluating user goal satisfaction (i.e., whether the user could
purchase a camera he/she likes or not). This “user goal satisfaction”
has the potential to measure achievement of the user’s goal moreaccurately in our task. However determining which measurement
is the best for practical use is diﬃcult, and still an open problem.

Acknowledgement
Part of this research was supported by JSPS KAKENHI Grant
Number 24240032 and the Commissioned Research of National
Institute of Information and Communications Technology (NICT),Japan.

94

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

Appendix

Table 13
Original Japanese transcription of Table 12.
Speaker

Transcription

SystemA."4B.4EI)@.;JK
7
L4/.%.#4.+ 24 7
 4EI)@;JK . L4&M#+/47
A...,2 "<1= 7>?@A"<1= B 7
A..<1="3<1=%+!7
2C4A..,&". 7
A."'()*#+,& -.% 7
A."NG . 
E."7
 4A.7
A" 2  90  OG 3.6  7
A."#,2<1=%. 4P#%57
D?
A"E
?
 /Q "4R%5?
 43% +42 525+ 7
A."  /2  7
A..
+."4222
7
4S  $ 4TC
4
A.42
!7

D?
AER?
T 4  4 .C!?
UV"4 10  20 4 7/.%+4A."45W 7

D?
A
 7
Dialogue is closed by the user 

SystemSystemSystemSystemSystemSystemSystemSystemUser
SystemSystemSystemSystemUser
SystemSystemSystem
User

Fra

GPF

Pos A

Inform

Pos A
None
Pos A
Pos A
Pos E
Pos A

InformInformInformInformInformInform

Pos A
None
None
None

Inform
RTurn
PropQ
PropQ

Pos A

Inform

Pos A
None
None
None

InformRTurn
PropQ
Question

Pos A
None
None

InformRTurnInformTable 14The summary of one dialogue in the corpus (translated from Japanese)
Speaker

Transcription

GPF Tag

Customer
Customer
Salesperson
Customer

Hello.
I’m looking for a camera for traveling. Do you have any recommendations?
What kind of pictures do you want to take?
Well, I’m the member of a tennis club,and want to take a picture of landscapes or tennis.
O.K. You want a camera which can take both far and near. Don’t you?
Well, have you used a camera before?
I have used a digital camera. But the camera was cheap and low resolution.
I see. I see. Camera A is a high resolution camera.
A has extremely good resolution compared with other cameras.
Although this camera does not have a strong zoom,
its sensor is is almost the same as a single-lens camera.
I see.
For a single lens camera,
buying only the lens can cost 100 thousand yen.
Compared to this, this camera is a bargain.
Ah, I see.
But, it’s a little expensive. right?
Well, I think, camera B is good at price.
Hahaha, yes, camera B is reasonably priced.
But its performance is low compared with camera A.
If I use the two cameras will I be able to tell the difference?
Once you compare the pictures taken by these cameras,
you will understand the difference immediately.
The picture itself is very high quality.
But, camera B and E are lower resolution,and the picture is a little bit lower quality.
Is there also difference in normal size pictures?
Yes, whether the picture is small or large, there is a difference
Considering A has single-lens level performance, it is surely reasonable.
I think so too.
The general price of a single-lens is about 100 or 200 thousand yen.
Considering these prices, camera A is a good choice.
Certainly, I’m interested in this camera.
Considering its performance, it is a bargain.
I think I’ll go home, compare the pictures, and think a little more.
I see. Thank you.

InformPropQ
SetQ

salesperson
Salesperson
Customer
Salesperson

Customer
Salesperson

Customer
Customer
Customer
Salesperson
Salesperson
Customer
Salesperson

Customer
Salesperson
Customer
Salesperson
Salesperson
Customer
Salesperson
Customer
Salesperson

Answer
PropQ
PropQ
AnswerInformInformInformInformPropQ
InformAnswerInformPropQ

Answer
PropQ
AnswerInformInformInformInformInformCommisive
Directive

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

95

Table 15
Original Japanese transcription of Table 14.
Speaker

Transcription

GPF Tag

Customer
Customer
Salesperson
Customer44/ #R"
#%RD #
4 !
# .%
=XA FY;%P .# .PZ,..$FY;%P+,.
%Q2+ /Q+ 4 /Q+ 3 !
 2 +

 ">?%
RDT.%2  52 2 .2 M,452 2RD.% 2   RD
2+ 2+   !
 [RD&  /.A.+!, /R. 2T.#2C7
..#RD% .T2. ,.$ /QRD# " +
U1O. &+<1=/!RD.. 4 !/.# ".UV32 7
 2

" .RDUV5!,PZU1O$  10 \ +/ ]0?+,.U1O%/4/."2#"T#& +. ,.U1O \L/. #2 /Q!
 2
!7
RD
!
L.B./#".2

# "RDT.#2!
""""  !#!B.A#  !PZRD.R+.
/2 ."3+!
 !!/.
.2
4
.^G_?. &+, %!S-,42"5, #!.
P". ].PZ,
PZ/.BE#25+!R+. , A#+PZRD
RD 2 /Q
/.. ."#2=0O.# , ,!
/ ."+"PZ!.
RR`T.,5,L   .%, $ # 
#UV.++
!
2
!
PZUV]! 10  20 ].2. ,%!"+PZ/"//CT/ !
# .+.#"
C#"C2. 4 $ "2 $ 
# - -  
B ."  

InformPropQ
SetQ

Salesperson
Salesperson
Customer

Salesperson

Customer
Salesperson

Customer
Customer
Customer
Salesperson
Salesperson
Customer
Salesperson

Customer
Salesperson

Customer
Salesperson
Salesperson
Customer
Salesperson
Customer
Salesperson

Answer
PropQ
PropQ

AnswerInformInformInformInformPropQ
InformAnswerInformPropQ

Answer
PropQ

AnswerInformInformInformInformInformCommisive
Directive

Table 16
Average number of turns of each policies in dialogue with simulators (Section 6.1.1), and with real users(Section 6.2). Note that, in the simulator case, dialogue is closed if the total number of turns reaches 50.

Average number of turns in dialogue with simulators
Average number of turns in dialogue with real users

RandomNoFraming

Framing

Human

8.2
5.2

50.0
38.8

50.0
27.3

18.8

Fig. 9. Number of epochs (i.e. number of parameter updates) v.s. average reward of 20 learnt policies. Policies in (a) are the case where the reward (Equation (4)) is given
only when dialogue is closed, and policies in (b) are the case where the reward is given at every turn. Error bars represents 95% conﬁdence intervals.

96

T. Hiraoka et al. / Speech Communication 84 (2016) 83–96

References
Asri, L.E., Laroche, R., Pietquin, O., 2013. Reward shaping for statistical optimisation of dialogue management. In: Proceedings of the International Conferenceon Statistical Language and Speech Processing, pp. 93–101.
Bench-Capon, T.J.M., 2003. Persuasion in practical argument using value-based argumentation frameworks. J. Logic Comput. 13 (3), 429–448.
Breiman, L., 1996. Bagging predictors. Mach. Learn. 24 (2), 123–140.
Carenini, G., Moore, J.D., 2006. Generating and evaluating evaluative arguments. Artif. Intell. 170 (11), 925–952.
Georgila, K., 2013. Reinforcement learning of two-issue negotiation dialogue policies. In: Proc. SIGDIAL, pp. 112–116.
Georgila, K., Traum, D., 2011. Reinforcement learning of argumentation dialogue
policies in negotiation. In: Proc. INTERSPEECH, pp. 2073–2076.
Guerini, M., Stock, O., Zancanaro, M., 2003. Persuasion model for intelligent interfaces. In: Proc. CMNA.
Heeman, P.A., 2009. Representing the reinforcement learning state in a negotiation
dialogue. In: Proc. ASRU, pp. 450–455.
Hiraoka, T., Neubig, G., Sakti, S., Toda, T., Nakamura, S., 2014a. Construction andanalysis of a persuasive dialogue corpus. In: Proc. IWSDS, pp. 125–138.
Hiraoka, T., Neubig, G., Sakti, S., Toda, T., Nakamura, S., 2014b. Reinforcement learning of cooperative persuasive dialogue policies using framing. In: Proc. COLING,
pp. 1706–1717.
Hiraoka, T., Neubig, G., Sakti, S., Toda, T., Nakamura, S., 2015. Evaluation ofa fully automatic cooperative persuasive dialogue system. In: Proc. IWSDS,
pp. 153–167.
Irwin, L., Schneider, S.L., Gaeth, G.J., 2013. All frames are not created equal: A typology and critical analysis of framing effects. Organ. Behav. Hum. Dec. Proces. 76(2), 149–188.
ISO24617-2, 2010. Language resource management-Semantic annotation frame work
(SemAF), Part 2: Dialogue acts. ISO.
Kudo, T., Yamamoto, K., Matsumoto, Y., 2004. Applying conditional random ﬁelds toJapanese morphological analysis. Proc. EMNLP 4, 230–237.
Lee, C., Jung, S., Kim, S., Lee, G.G., 2009. Example-based dialog modeling for practicalmulti-domain dialog system. Speech Commun. 51 (5), 466–484.
Levin, E., Pieraccini, R., Eckert, W., 20 0 0. A stochastic model of human-machine interaction for learning dialog strategies. IEEE Trans. Speech Audio Process. 8 (1),
11–23.

Mazzotta, I., de Rosis, F., 2006. Artiﬁces for persuading to improve eating
habits. AAAI Spring Symposium: Argumentation for Consumers of Healthcare,
pp. 76–85.
Mazzotta, I., de Rosis, F., Caroﬁglio, V., 2007. PORTIA: a user-adapted persuasion system in the healthy-eating domain. Intell. Syst. 22 (6), 42–51.
Meguro, T., Higashinaka, R., Minami, Y., Dohsaka, K., 2010. Controlling listening-oriented dialogue using partially observable Markov decision processes. In: Proc.
COLING, pp. 761–769.
Meguro, T., Minami, Y., Higashinaka, R., Dohsaka, K., 2011. Wizard of Oz evaluationof listening-oriented dialogue control using POMDP. In: Proc. ASRU, pp. 318–323.
Misu, T., Georgila, K., Leuski, A., Traum, D., 2012. Reinforcement learning of question-answering dialogue policies for virtual museum guides. In: Proceedings ofthe 13th Annual Meeting of SigDial, pp. 84–93.
Ng, A.Y., Harada, D., Russel, S., 1999. Policy invariance under reward transformations: theory and application to reward shaping. Proc. ICML.
Nguyen, H., Masthoff, J., Edwards, P., 2007. Persuasive effects of embodied conversational agent teams. In: Proc. HCI, pp. 176–185.
Nouri, E., Park, S., Scherer, S., Gratch, J., Carnevale, P., Morency, L.-P., Traum, D., 2013.
Prediction of strategy and outcome as negotiation unfolds by using basic verbal
and behavioral features. In: Proc. INTERSPEECH, pp. 1458–1461.
Paruchuri, P., Chakraborty, N., Zivan, R., Sycara, K., Dudik, M., Gordon, G., 2009.
POMDP based negotiation modeling. Proc. MICON.
Porta, J.M., Vlassis, N., Spaan, M.T., Poupart, P., 2006. Point-based value iteration forcontinuous POMDPs. J. Mach. Learn. Res. 7, 2329–2367.
Riedmiller, M., 2005. Neural ﬁtted q iteration - ﬁrst experiences with a data eﬃcientneural reinforcement learning method. Mach. Learn. ECML 317–328.
Schatzmann, J., Georgila, K., Young, S., 2005. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: Proceedings of SIGDIAL,
pp. 45–54.
Schaul, T., Bayer, J., Wierstra, D., Sun, Y., Felder, M., Sehnke, F., Rückstieß, T., Schmidhuber, J., 2010. Pybrain. J. Mach. Learn. Res. 11, 743–746.
Terrell, A., Mutlu, B., 2012. A regression-based approach to modeling addressee
backchannels. In: Proc. SIGDIAL, pp. 280–289.
Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A., 1997. PARADISE: a framework forevaluating spoken dialogue agents. In: Proc. EACL.
Weka 3: Data Mining Software in Java, 2009. http://www.cs.waikato.ac.nz/ml/weka/.
Williams, J.D., Young, S., 2007. Scaling POMDPs for spoken dialog management. IEEE
Trans. Audio Speech Lang. Process. 15 (7), 2116–2129.