Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS.2017.DOI

Policy reuse for dialog management
using action-relation probability
TUNG. T. NGUYEN1 , KOICHIRO YOSHINO1,2 , (MEMBER, IEEE), SAKRIANI SAKTI1,3 ,(MEMBER, IEEE), AND SATOSHI NAKAMURA 1,3 , (Fellow, IEEE)
1
2
3

Nara Institute of Science and Technology, Ikoma, Nara, Japan 630-0192 Japan
Japan Science and Technology Agency, JapanRIKEN, Center for Advanced Intelligence Project AIP, Japan

Corresponding author: Tung T. Nguyen (e-mail: nguyen.tung.np5@ is.naist.jp).
Parts of this work are supported by JSPS KAKENHI Grant Number 20H05567 and JP17H06101.

ABSTRACT We study the problem of policy adaptation for reinforcement-learning-based dialog management. Policy adaptation is a commonly used technique to alleviate the problem of data sparsity whentraining a goal-oriented dialog system for a new task (the target task) by using knowledge when learning
policies in an existing task. The methods used by current works in dialog policy adaptation need muchtime and effort for adapting because they use reinforcement learning algorithms to train a new policy forthe target task from scratch. In this paper, we show that a dialog policy can be learned without training
by reinforcement learning in the target task. In contrast to existing works, our proposed method learns therelation in the form of probability distribution between the action sets of the source and the target tasks.
Thus, we can immediately derive a policy for the target task, which significantly reduces the adaptation
time. Our experiments show that the proposed method learns a new policy for the target task much more
quickly. In addition, the learned policy achieves higher performance than policies created by fine-tuning
when the amount of available data on the target task is limited.
INDEX TERMS dialog management, reinforcement learning, transfer learning, policy adaptation, mixture
density network

I. INTRODUCTION

EINFORCEMENT learning (RL) is a widely usedframework for modeling the decision-making process
in such tasks as the dialog management of conversational
systems [1]–[3]. In general, training a policy with reinforcement learning helps the system to learn a more robust
behavior and its performance will not be upper-bounded by
the human performance in the training dialog samples, which
is sometimes not optimal. Nonetheless, the training processof reinforcement learning is usually tedious and needs alarge number of samples to train an optimal policy. Policy
adaptation, or policy transfer, is a very useful technique that
can tackle this problem in reinforcement learning.
Policy adaptation refers to the process of reusing knowledge, i.e., a policy that is learned in one or multiple source
tasks to a new target task. Various literatures which hasstudied about policy adaptation in RL proposed different
techniques that show such promising results as the acceleration of the convergence rate and the reduction of data volume
requirements [4], [5].

R

VOLUME 4, 2016

Within the scope of reinforcement learning-based dialog
management, the application of policy adaptation remains
very limited. Current works in dialog policy adaptation [6]–[8] follow the weight initialization strategy, which contains
two steps: pre-training and fine-tuning. Pre-training refers to
the process that trains a policy in the source task, where thepolicy is usually represented by a neural network. Some ofthe source policy’s weight parameters are used to initialize
the neural network’s weights to train a policy in the target
task. Next, we fine-tune the network weights by training witha reinforcement learning algorithm.
However, when we do not have enough data on the targettask, this strategy does not work well because it barely uses
the knowledge from the source task’s policy, forcing thetarget task’s policy to be basically trained from scratch. Consider a situation with a dialog policy that handles restaurant
reservations and we want to adapt to manage the task ofbooking hotel rooms. Obviously, we expect that this adaptation can be performed with minor adjustment to the restaurant
booking task’s policy. Motivated by this observation, we
1Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

proposed a novel method to adapt dialog policy called the
Dialog Policy Reuse Algorithm (DPRA). Unlike previous
works, our proposed method learns the relation between
the action sets of the source and the target tasks using a
mixture density network [9], and then we can quickly infer
the policy for the target task without RL training. In otherwords, DPRA allows us to “reuse” the source task policy foraction decision-making in the target task. The following are
the primary contributions of our work:
We propose a novel method for the policy adaptationof reinforcement-learning-based dialog management. In
particular, the source task’s policy can be used for theaction selection procedure in the target task througha special mapping called action-relation probability.
We propose a mixture density network to model this
distribution.
• Since our proposed method can learn a policy for thetarget task without RL training, it reduces the effort toconstruct dialog policies in the target task.
• Experimental results show that, when only a small
amount of data is available for training, our proposed
method achieves higher performance than the baseline
adaptation method, which is based on fine-tuning, andalso requires much less time for training.

•

The remainder of this paper is organized as follows. Section
II provides background knowledge in reinforcement learning and dialog management. In Section III, we explain our
proposed method, the DPRA algorithm. Section IV describes
our experiment setting, its results, and analyses. Section V
explains current works in dialog policy adaptation and theirdrawbacks. Finally, we summarize our work and discuss
future directions in Section VI.
II. REINFORCEMENT LEARNING AND DIALOG
MANAGEMENT
A. REINFORCEMENT LEARNING

Reinforcement learning is a popular framework for learning
autonomous behavior. We consider the standard reinforcement learning setting where an agent interacts with an environment that follows a Markov decision process (MDP) overa number of discrete time steps [10]. At each time step t, thestate, action, and reward are respectively denoted by st ∈ S,
at ∈ A, and rt ∈ R. The dynamics of the task (the environment) are characterized by two random variables: statea0transition probabilities Pss
0 = P (st+1 = s |st = s, at = a),aand the expected
Preward, given by Rs = E[rt+1 |st =s, at = a] =rt+1 rt+1 P (rt+1 |st = s, at = a). The
agent’s procedure for selecting action a given state s is theagent’s policy, denoted by π(s, a) = P (a|s). We define thereturn, which
the total rewards that received by the agent,PTis−k
as Rt =
k=1 rt+k where T is the final time step. The
agent’s objective is to maximize expected return E[Rt |st , π]at each time step t when following policy π. If the agentenvironment interactions do not stop, T goes to ∞. We
describe our task as continuing. If the interactions eventually2

end when we reach certain terminal states, then our task
is called episodic. In this setting, the interactions from the
beginning until the agent reaches a terminal state is called
an episode. Appropriate setting are chosen depends on the
problem we want to solve using reinforcement learning.
Given policyPπ, the state-action value is defined as:
T −k
Qπ (s, a) = E[ k=1 rt+k |st = s, at = a, π], which is the
expectation of the return if action a is chosen at time step t.
πSimilarly,
PT −k we define the state value of policy π: V (s) =E[ k=1 rt+k |st = s, π]. Note that the above definitionsof state-action, and state value fall under the undiscounted
setting.
Reinforcement learning algorithms can be divided into
two classes: value-based and policy-based. In value-based
reinforcement learning methods, we estimate action-stateQπ (s, a) or the state value V π (s) by using a function approximator, such as neural networks or simple value tables.
Classic Q-learning [11] or deep Q-network [12] are examples of this class of algorithms. In contrast to value-based
methods, policy-based RL algorithms parameterize policy πby parameters θ, which we update by performing (typicallyapproximate) gradient ascent to maximize E[Rt |π]. There
are various policy-based RL studies, especially with policy
gradient methods, such as Actor-Critic [13], [14] or REINFORCE [15], which we are using in our evaluation.
B. DIALOG MANAGEMENT USING REINFORCEMENTLEARNING

A dialog can be divided into multiple turns, where each turn
contains a user utterance and a system response. We canformulate the problem of dialog management as an MDP andapply any RL algorithm to solve it. We can define a set ofactions for the system to interact with the user and define areward function based on the system’s goal. Since all dialogs
only have a finite number of turn (or time steps) we need touse the episodic and undiscounted setting as a formulation of
dialog management problem.
At each time step t, the required information for the action
selection procedure is defined as an observation, denotedby ot . The type of information included in an observation
depends entirely on the task that is being solved. For example, in dialog management, the observation may consist ofrecognition results of slot information [6], [7], [16], user’s
dialog actions [17], [18], or such high-level multimodal
information as user’s deception [19]. Recall that the policy
is a conditional probability for selecting action a given states, P (a|s); thus, the state must include critical information
for making the decision. A natural approach is to represent
the state by the vector of observation or the concatenation ofobservations from multiple time steps. We call such state representation as explicit state representation. In modular-based
dialog systems [1], [2], [17]–[19], the state is represented bythis explicit representation.
In dialog management procedures, the system needs toconsider the dialog history, which contains the user’s utterances and the system’s actions from the beginning. WithVOLUME 4, 2016

Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

such long-term tracking requirements, explicit representation
becomes unsuitable if the observations are high-dimensionalor the conversations are lengthy. A common solution is usinga Recurrent Neural Network (RNN) to learn state embedding,which resembles a latent state representation that stores the
dialog history and current observation [16], [20]. Such state
representation is used in end-to-end dialog modeling [20].
Fig. 1 shows an example of this approach. The RNN plays
the role of dialog state tracker (DST) and its output, the latentstate representation, is used by the dialog policy module formanagement. With an end-to-end approach, we are free fromdesigning a complicated explicit representation of the dialog
states.
Dialog policy

Dialog state
st-1

at

Dialog Policy

st

LSTM

at-1

st

olt

st+1

LSTM

at-1

olt

at+1

Many types of knowledge can be used in transfer learning
in RL, such as samples, representation, or parameters [4].
Policy adaptation, or policy transfer, refers to transfer learning methods that use knowledge of the policy from the sourcetask for the transferring process. Policy adaptation methods
are subclass of transfer learning solutions in reinforcement
learning.
We usually face the problem of data sparsity when training policies for dialog tasks, because they do not require
additional knowledge between the source and the target
task. In this situation, the policy adaptation approach is apromising solution. Multiple studies investigate its application in reinforcement-learning-based dialog management[6]–[8]. All of these methods improve learning speed, reduce
data requirements, or performance.

st+1

D. PROBLEM STATEMENT OF POLICY ADAPTATIONNote:
ol: linguistic featuress: dialog state
a: system’s action

FIGURE 1. Dialog modeling with end-to-end approach.

RL training requires a huge amount of agent-environment
interactions to learn a good policy. In dialog management,
it is impossible to collect enough dialog samples to fulfill this requirement. Scheffler and Young proposed a user
simulator as replacement for actual human users to train
the policy [1]. This approach has become the standard fortraining dialog policies with RL. The user simulator is built
from dialog samples by maximum likelihood or supervised
learning methods, to imitate the human user’s behavior. The
user simulator can also be viewed as an approximation of theadialog management task’s true state transition Pss
0 , which isprovided by the actual human user.
C. POLICY ADAPTATION

Humans are capable of learning a task better and fasterby transferring the knowledge retained from solving similartasks. This observation motivates the idea of transferringknowledge across different but related tasks to improve
the performance of machine learning (ML) algorithms. The
techniques that allow such knowledge transferring is called
transfer learning.
Application of transfer learning to RL algorithms started
to gain attention of the machine learning community from the
middle of the 2000s. In reinforcement learning, the numberof samples needed to learn an optimal solution is usuallyprohibitive, especially for dialog management, where data
sparsity is a huge challenge [21]. Transfer learning can build
prior information from the knowledge collected to solve a set
VOLUME 4, 2016of source tasks and be used for learning a policy in the target
task.

Current policy adaptation studies in reinforcement-learningbased dialog management follow the weight initialization
approach [6]–[8]. As explained in Section I, this strategy
requires us to train the target policy from scratch with an
RL algorithm, which is a tedious process that requires great
effort, especially for complex tasks. In addition, the construction of dialog policies usually involves user simulations.
When we only have a small amount of data in the target
dataset, the user simulator does not represent the behavior
from the actual human users very well. Therefore, a policythat we train can have high performance against the simulator
even though fails to work well versus actual human users.
Problem statement. Given a source task with the state
space SA and action set A, assume that we have trained
policy π(s, a) for the source task. The target task’s state space
and action set are denoted as SB and B, our goal is to derive
policy π(s, b) for the target task from π(s, a) without RL
training.
III. POLICY REUSE BASED ON ACTION-RELATIONPROBABILITY

In this section, we show that a policy can be adapted for thetarget task without RL training from the scratch. Instead oftraining a policy by interactions with a user simulator, weestablish a connection between the policies of the source
and target tasks through a special mapping distribution called
action-relation probability. Our proposed adaptation method,
DPRA, learns this distribution from dialog samples in bothtasks, and can immediately derive a policy for the target
task. Thus, DPRA can remarkbly reduce learning time. Since
our proposed method does not use the user simulator, we
can avoid the problem of low performance due to errors inconstructing the user simulator.

3

Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

A. POLICY ADAPTATION BY ACTION-RELATIONPROBABILITY

We consider policy adaptation from a source task to a target
task. First, we make the following assumptions, which allow
the derivation of relation between the policies of the source
and the target tasks :Assumption 1: The source and the target tasks share identical state space S. This assumption is actually not restrictive.
Union space S = SA ∪ SB is the state space that satisfies theassumption.
Assumption 2: The source and the target tasks have identical state representation.
We define observation as the information that is necessary
for the agent’s action selection in each time step. An example
is the features extracted from the user’s utterance at each turn.
Obviously, in a policy adaptation setting, the source and thetarget tasks are different and they require distinctive sets offeatures. Thus, the state representation of the source task isalso not identical to the target task’s. However, we can define
a unified set of features for those tasks and have the sameobservation and state representation across the source and thetarget tasks.
We establish a connection between the source and thetarget task’s policy as follows. Denote the source task’s
policy as π(s, a) and the target task’s policy as π(s, b), wherea ∈ A, b ∈ B are action sets in the source and target tasks.
X
π(s, b) =P (b|a, s)π(s, a).
(1)a∈A

Equation (1) argues that with any conditional distribution of
P (b|a, s); from source task’s policy π(s, a), we can directly
infer a policy for target task π(s, b). We call this P (b|a, s)
action-relation probability. Our proposed policy adaptation
method models this distribution instead of performing RL
training.

network. Given multivariate random variables x, y, MDN
models conditional probability density p(y|x):MXp(y|x) =2
wm (x) · N (y; µm (x), σm
(x)).

(3)

m=1

M is the number of components, and wm (x), µm (x), and2σm(x) are the component weight, mean, and standard deviation for component m. We assume that these mixture
variables are functions of input x that are approximated by
w
µ
σ
w
µ
σ
neural networks fm
, fm
, fm
, parameterized by θm
, θm
, θm
.
With some slight abuse of notation, we have:ww
exp(fm(x; θm))wm (x) ≈ PMw
w
l=1 exp(fl (x; θl ))µµµm (x) ≈ fm (x; θm )

(4a)
(4b)

σ
σ
σ(x) ≈ exp(fm(x; θm))(4c)

With MDN, we assume that all the components in multivariate random variable y are mutually independent, and thus
the covariance matrix is diagonal and can be representedµ(x). Denoteby a vector with the same dimension as fm(i)(i)
the dataset as D = {(x , y )}, i = 1..N , where x, y
are the observed data for random variables x and y. The
parameters are optimized using gradient descent with thefollowing negative log-likelihood:L=− log(

N
Y
i=1

p(

M
X

2
wm (x(i) ) · N (y(i) ; µm (x(i) ), σm(x(i) ))))

m=1

(5)
By replacing the probabilities with probability density
functions, the mixture model in (2) is now given:p(s0 |ai , s) = pi (s0 |s)(ai ∈ A)|B|

B. ACTION-RELATION MODELING WITH MIXTURE
DENSITY NETWORKThis section explains the modeling of action-relation probability distribution based on a mixture model. First, denote thestate and action at time step t as s = st and a = at , where thestate at the next time step is s0 = st+1 . The state transition in
the target task is given:
P (s0 |a, s) =X

P (s0 , b|a, s) (law of total probability)

b∈B

=

X

P (s0 |b, a, s)P (b|a, s).

b∈B

(2)
The state transition of the source task has the form ofa mixture model with the action-relation probability as thecomponent weights. Mixture density network (MDN) [9], isa suitable approach for modeling state transition P (s0 |a, s).
In principle, a mixture density network is a type of Gaussian mixture model (GMM) that utilizes an artificial neural4=X

Pij (s) · pij (s0 |s)
(6)

j=1
|B|

=

X

2
wij (s) · N (s0 ; µij (s), σij
(s)).

j=1

An illustration of these mixture models is given in
Fig. 2. In principle, the density of the state transition
caused by ai is a mixture model with each componentpij (s0 |s) = p(s0 |ai , bj , s) follows Gaussian distribution2
N (s0 ; µij (s), σij(s)).
For each action ai in the source task, we can train its
corresponding MDN with just the samples (s0 , ai , s). Particularly, s is observed variable x and s0 is the latent variable
y in (5), ai is the indicator to which Gaussian component
corresponds to sample s, s0 . The action-relation probabilityw
P (b = bj |a = ai , s) is approximated by f w (x; θij)), asshown in (4a). However, in that case, we cannot guaran2
tee that N (s0 ; µij (s), σij(s)) truly models state transition0p(s |ai , bj , s) since the source task samples do not contain
VOLUME 4, 2016

Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

𝑝 𝑠′ 𝑏1 , 𝑠

𝑝 𝑠′ 𝑏𝐿 , 𝑠

𝑝 𝑠′ 𝑏𝑗 , 𝑠

𝐴 = 𝐾; 𝐵 = 𝐿
𝑏1

𝑏𝑗

…

P (s0 |b, s) =

𝑏𝐿

…

X

P (s0 , a|b, s) (law of total probability)a∈A

𝑎1
𝑝 𝑠′ 𝑎1 , 𝑠…
2
𝑁 𝑠 ′ ; 𝜇11 𝑠 , 𝜎11𝑠2
𝑁 𝑠 ′ ; 𝜇11 𝑠 , 𝜎11
𝑠

=

…
2
𝑁 𝑠 ′ ; 𝜇1𝐿 𝑠 , 𝜎1𝐿𝑠…
𝑎𝑖𝑝 𝑠′ 𝑎𝑖 , 𝑠2
𝑁 𝑠 ′ ; 𝜇𝑖1 𝑠 , 𝜎𝑖1𝑠…

𝑁 𝑠 ′ ; 𝜇𝑖𝑗 𝑠 , 𝜎𝑖𝑗2 𝑠…

2
𝑁 𝑠 ′ ; 𝜇𝑖𝐿 𝑠 , 𝜎𝑖𝐿
𝑠

…
…

𝑎𝐾
𝑝 𝑠′ 𝑎𝐾 , 𝑠2
𝑁 𝑠 ′ ; 𝜇𝑖1 𝑠 , 𝜎𝑖1𝑠…
2
𝑁 𝑠 ′ ; 𝜇11 𝑠 , 𝜎11𝑠2
𝑁 𝑠 ′ ; 𝜇𝐾𝐿 𝑠 , 𝜎𝐾𝐿𝑠X

P (s0 |b, a, s)P (a|b, s).

a∈A

(8)
Equation (8) argues that transition distribution P (s0 |b, s)also has the form of a mixture model that can be represented by MDN. The mixture in (8) has the same component P (s0 |a, b, s) as in (2), but different component weights
P (a|b, s). We define the network’s parameters that approximates the component weights as θ̂w .

FIGURE 2. State transition modeling by mixture density network. Components
in each column correspond to an action in the target task.

Algorithm 2 Action-relation probability modeling by MDN
with component matching by MDN

any information of bj . A natural solution is to additionallytrain the components using the samples (s0 , bj , s) by component matching process, which actually “matches” the distri2
bution of component N (s0 ; µij (s), σij(s)) to the transition
0of p(s |bj , s).
In this work, we propose two methods of component
matching. In the first, we assume that p(s0 |ai , bj , s) =p(s0 |bj , s)∀ai ∈ A, bj ∈ B. With this assumption, we
can perform component matching by simply optimizing the
networks’ parameters using the negative log-likelihood of thetarget task’s samples:

Randomly initialize the network weights θw , θ̂w , θµ , θσInitialize MDN gradient dθM DN ← 0
Initialize the gradient for component matching dθCM ← 0
Initialize the iteration counter t ← 0
repeatfor all ai such that ai ∈ A dofor all bj such that bj ∈ B do
Calculate gradient dθCM by (9)Update parameters θ̂w , θµ , θσ with dθCM
end forCalculate gradient dθM DN by (5)Update parameters θw , θµ , θσ with dθM DN
end fort←t+1
until t > tmax

L = −log(

N
Y

N (s0

(i)

; µ(s(i) ), σ 2 (s(i) ))).

(7)

i=1

Since the training of this component matching method resembles the training process of a regression model, we define
it component matching by regression. Algorithm 1 shows thepseudo code for the training process of the mixture model in
6 using component matching by regression.
Algorithm 1 Action-relation probability modeling by MDN
with component matching by regression
Randomly initialize the network weights θw , θµ , θσInitialize MDN gradient dθM DN ← 0
Initialize the gradient for component matching dθCM ← 0
Initialize iteration counter t ← 0
repeatfor all ai such that ai ∈ A dofor all bj such that bj ∈ B do
Calculate gradient dθCM by (7)Update parameters θµ , θσ with dθCM
end forCalculate gradient dθM DN by (5)Update parameters θw , θµ , θσ with dθM DN
end fort←t+1
until t > tmax
The second component matching method stems from thefollowing derivation:
VOLUME 4, 2016

In principle, the components in column of bj in Fig. 2 forma mixture model for the density of state transition p(s0 |bj , s).
Similarly, we can train this mixture with a loss function that
resembles (5) using the target task samples (s0 , bj , s):L=− log(

N
Y
i=1

p(

M
X

2
ŵm (x(i) ) · N (y(i) ; µm (x(i) ), σm(x(i) )))).

m=1

(9)
We call this method component matching by MDN. The
pseudo code for action-relation probability modeling using
MDN component matching is shown in Algorithm 2.
Finally, the procedure of our proposed method, the dialog
policy reuse algorithm, is shown in Algorithm 3:
Algorithm 3 Dialog policy reuse algorithm: DPRA
Step 1: Train policy π(s, a) for source task
Step 2: Model action-relation probability P (b|s, a) using
either Algorithms 1 or 2
Step 3: Create a policy for target task π(s, b) , by using 1
with the action-relation probability learned in Step 2.
5

Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

We summarize this section by showing that resultant policy π(s, b) of DPRA is proper:X
π(s, b) = 1
(10)
b∈B

Intuitively, DPRA works as follows. Given policy π(s, a)in the source task, assume that an agent with policy π selects
action a given current state s. DPRA finds action b in thetarget task that makes similar transition s → s0 to action a: inother words, P (s0 |s, a) ' P (s0 |s, b). Instead of making a deterministic mapping, we learn a distribution that connects a toall the available actions in the target task, which is P (b|a, s).
This is why we use the term action-relation probability todefine this special distribution. With the condition that thesource and target tasks are also similar in the reward dynamic,
if learned policy π(s, a) in the source task is optimal, thenpolicy π(s, b) learned by DPRA is nearly optimal as well.

Source task:
State space: S
Action set: A
Policy: 𝜋 𝑠, 𝑎

DPRA
Model the distribution
𝑃(𝑏|𝑎, 𝑠)Target task:
State space: S
Action set: B
Policy: 𝜋 𝑠, 𝑏

FIGURE 3. Working procedure of the proposed method.

An illustration of DPRA is shown in Fig. 3. DPRA requires
identical state space and state representation of the source
and target tasks (Assumption 1 and 2). Recall that all policy
adaptation methods only work in the cases where the source
and target tasks are similar. In this situation, even if the twotasks’ state spaces and state representations are not completely identical, we expect that they still share considerable
similarities, and thus, the proposed method can still learn agood policy for the target task without the ideal conditions inAssumption 1 and 2.
IV. EVALUATION
A. SETTING

We experimentally assessed the following hypotheses:
1) Our proposed adaptation algorithm, DPRA, requires
much less training time than conventional fine-tuning
methods.
2) DPRA learns policies that achieves equivalent or
higher performance than those learned with current
methods when limited data available in the target task.
For comparison with our proposed methods, we used
policy adaptation by action embedding [8]. This method
uses the same network as our end-to-end dialog modelingand changes its last layer of the network to connect to a
new action space in the target task. This simple model does
not require any prior information such as relations between
actions. Since our proposed algorithm also does not require
such prior information either, we selected this method as thebaseline.
6

In our evaluation, we performed policy adaptation for a
multimodal goal-oriented dialog system with an end-to-end
approach. We chose a multimodal dialog setting because theavailable corpora for such conversations are mostly smallscale [21] and thus suitable to assess our second hypothesis.
In particular, we augment the original end-to-end dialog
model(Fig. 1) with a multimodal fusion component that uses
the Hierarchical Tensor Fusion Network [22]. This component’s role is to efficiently combine features from multiple
modalities: linguistic, visual, and acoustic. Since it is fully
connected to the dialog state tracker (the LSTM layer in Fig.
1), our dialog model still adheres to the end-to-end paradigm.
We formulated the problem of dialog management with
an episodic and undiscounted reward setting and trained
the dialog policies using REINFORCE [15], which updates
policy parameters θ with the following gradient:#" T
X ∂logπ θ (st , at )(11)π∂(θ) = ERt∂θ
t=0As in (11), the “vanilla” version of REINFORCE has highvariance and slow convergence. To combat this problem, abaseline technique was introduced [23], [24]. [25] described
how the most natural and effective baseline is the state-valuefunction. Thus, we use V π (s, a) as the baseline and thegradient is given by:" T#
X∂logπ θ (st , at )π∂(θ) = E(Rt − V (st , at ))π (12)∂θ
t=0
The baseline’s role in (12) is to reduce the variance of thegradient estimation and smoothen the training. In principle,
since any RL algorithm can be used for training the policies,
we chose REINFORCE due to its simplicity and good performance.
The output layers of the multimodal fusion component andthe DST both have 128 units. Therefore, the dialog state isrepresented by vector s ∈ R128 . The neural network that
represents the dialog policy has one single hidden layer with
128 units and is fully connected to the input and output
layers. We used the Adam optimizer for optimization of
the networks’ parameters and initialized the learning rate at
1e − 3. We trained the policies for the source and target tasks
with 20,000 and 10,000 episodes, of which can be seen asa simulated dialog with the user simulator. The learning rate
decreased by 10% every 1000 episodes.
The neural networks that approximate the mixture variables in Algorithms 1 and 2 have one hidden layer with
256 units. We also used the Adam optimizer for parameter
optimization. The learning rate is fixed at 1e − 4, and thenumber of training epochs is 10. Note that in Algorithms 1
and 2, the network parameters are updated sequentially withtwo different gradients. Thus, the training process has large
oscillation and converges slowly. To avoid this problem, we
adopted the p : q training scheme. Every epoch, we perform
component matching for p = 2 times and trained the mixture
model for q = 2 times.
VOLUME 4, 2016

Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

Source task
datasetTarget task
datasetUser simulator

User simulator

1. RL
training

3.1 Evaluation
with simulator

2. Policy
adaptation

Policy𝜋 𝑠, 𝑎Policy𝜋 𝑠, 𝑏

3.2 Evaluation
by DA
selection

FIGURE 4. Experiment procedure.

extracted values. We extracted acoustic features from theaudio using the OpenSMILE toolkit [27] with the Interspeech2009 (IS09) emotion challenge standard feature-set as our
feature template [28]. We used these extracted features tocreate acoustic observation oa .
The set of system’s actions in the source task is identical as [19], which includes {Offer, Framing, End}. Wechanged the system’s action set of the target task into {Offer_New, Offer_Change, Framing_Argue, Framing_Answer,
End_Dialog}. The source policy never sees these actions
during training.
For the RL training, we created a user simulator thatgenerates labels of user action u and deception information dwith the following intention and deception models:intention model = P (ut+1 |ut , dt , dt+1 , at )B. PERFORMANCE METRICS

We measured the training time of all the methods in seconds.
Only the time spent in adaptation is measured; the learning
times of the source task’s policy and for creating the simulator is not included. We run the adaptation algorithms on
identical hardware 1 to ensure a fair comparison among theevaluated methods.
To assess out second hypothesis, we used two performance
metrics: average reward per episode and the system’s action
selection accuracy. With the first metric, we used a user
simulator created from the target dataset. We ran the learned
policies against this simulator for 1000 episodes and measured the average total reward per episode. For the second
metric, several human experts read the dialog transcript andselected the most appropriate system action for each turn.
These actions are used as groundtruth for measuring theaction selection accuracy of the policies. This metric shows
the appropriateness of the learned policy behaviors on eachturn.
C. DIALOG TASKIn this evaluation setting, both the source and the target tasks
are negotiation task in healthcare domain [19]. The systemtries to convince the users that their current living style isunhealthy and suggests that they adopt its proposed living
habits.
We used the previously proposed healthcare consultation
dataset [19], which contains conversations on six topics:
sleeping, eating, working, exercising, social media usage,and leisure activities. The conversations of the first four
topics (51 dialogs) are used for training the policies ofthe source task. The remaining 24 dialogs in two topics
are used for training the target task’s policy. In each turn,we split the recorded video of the users into 30 segments
and randomly sampled one frame from each segment to be
used for extracting visual features. We extracted 14 faceaction unit (AU) regressions and 6 AU classification values
for each frame with the OpenFace toolkit [26]. The visualobservation at each turn ov is a vector that contains these
1 CPU: Intel Xeon CPU E5-2630 v4, GPU: GTX Titan X.
VOLUME 4, 2016

deception model = P (dt+1 |ut , dt , at )(13)

Recall that in each dialog turn, the system takes input features (observations) from three modalities: linguistic, visual,and acoustic. We employed the user action u as linguistic
observation ol . The visual and acoustic observations, ov andoa , are sampled uniformly from the dialog corpus using u andd.
D. EXPERIMENT RESULTS

We conduct the following experiments to assess the hypotheses raised in the beginning of this section.
1) Policy adaptation time
TABLE 1. Comparison of training time required for different policy adaptation
methods.

Model
Policy adaptation by action embedding [8]DPRA - regression component matching
DPRA - MDN component matching

Trainingtime∼ 350s∼ 40s∼ 45s

Table 1 shows the training times required for all the policy
adaptation methods. The numbers reported for DPRA are
from a case in which all 24 dialogs of the target task areavailable for training. Cases with less data will obviously takeless time for training with DPRA. With policy adaptationby action embedding [8], we chose the number of episodes(interactions with simulated users) for training the target
policy based on the average rewards received per episode.
As seen from Table 1, since DPRA requires significantly lesstime for training, our first hypothesis stands.
2) Dialog policy performance comparison

We recreated scenarios in test sets where the amount of data
available for training in the target task is limited to assess
the second hypothesis. In particular, we sampled k dialogs
from the target task dataset, k ∈ {1, 2, 4, 8, 16}. We used
these k dialog samples to create a user simulator for training
7

Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSTABLE 2. Average reward per episode of learned policies with different amounts of available data. Numbers in brackets indicate 95% confidence intervals.

# available
dialogs
Model
NoAdapt - 2k
NoAdapt - 10k
ActEmb - 2k
ActEmb - 10k
DPRA - MDN
DPRA - regression

1 dialog

2 dialogs

4 dialogs

8 dialogs

16 dialogs-52.93(±16.59)-36.95(±15.21)-33.56(±17.03)-29.98(±13.12)-25.43 (±16.16)-12.96 (±11.05)-23.42(±13.93)-17.72(±16.27)-11.48(±10.33)-18.07(±11.60)-10.33 (±8.94)-0.01(±10.24)-26.61(±17.98)-9.02(±13.92)
0.01(±11.45)-13.42(±12.22)-2.18 (±10.71)
6.11(±6.03)-19.15(±18.29)-7.75(±14.40)-8.56(±13.00)
0.64(±11.99)
14.93 (±4.11)
14.65(±6.27)-11.77(±19.80)
10.55(±19.95)
12.86(±11.05)
2.04(±13.72)
17.21 (±4.2)
19.41(±4.49)

TABLE 3. Dialog act selection accuracy of learned policies with different amounts of available data. Numbers in brackets indicate 95% confidence intervals.

# available
dialogs
Model
NoAdapt - 2k
NoAdapt - 10k
ActEmb - 2k
ActEmb - 10k
DPRA - MDN
DPRA - regression

1 dialog

2 dialogs

4 dialogs

8 dialogs

16 dialogs

33.80% (±7.54%)
37.39% (±6.91%)
33.83% (±4.88%)
28.04% (±5.24%)
40.96% (±7.79%)
45.41% (±5.99%)

35.86% (±7.06%)
44.50% (±8.01%)
33.25% (±4.73%)
24.79% (±5.04%)
39.86% (±6.81%)
52.79% (±5.18%)

35.62% (±7.06%)
47.86% (±7.61%)
41.41% (±5.92%)
26.11% (±6.78%)
56.16% (±7.82%)
60.48% (±6.02%)

34.80% (±7.17%)
44.41% (±6.31%)
39.41% (±5.75%)
25.62% (±6.06%)
62.05% (±4.94%)
62.98% (±5.08%)

43.14% (±7.50%)
52.16% (±7.76%)
43.64% (±5.41%)
29.30% (±5.04%)
62.05% (±4.64%)
69.30% (±3.22%)

the target policy in the ActEmb adaptation method and formodeling the action-relation probability in DPRA. For eachvalue of k, we sampled k dialogs five times, thus making
five different datasets. With each dataset, we perform policy
adaptation ten times for each method and conduct 50 runs of
the policy adaptation experiment with each value of k.
Performance in terms of received rewards. Table 2
shows the performance of the dialog policies in terms of
the average reward per episode. The details of the reward
function for both the source and target tasks are provided
in Appendix A. DPRA-MDN and DPRA-regression respectively refer to the proposed policy adaptation method withcomponent matching by MDN and regression. ActEmb-2k
and ActEmb-10k refer to policy adaptation by action embedding method, where the number of episodes for training in
the target task is 2,000 and 10,000. Finally, NoAdapt refers
to a policy that is trained on the source task without adaptation, where the notation for number of training episodes is
identical as ActEmb. As seen in the table, the performance
generally increases when more data are available. In Table
2, bold numbers indicate the policy with the highest average reward per episode in each scenario where the numberof available dialogs k. Policies adapted by DPRA shows
significantly higher performance than those from ActEmb
and NoAdapt (p < 0.05) with all k, and the differenceis bigger when k is small. ActEmb-2k and ActEmb-10k
perform similarly; on the other hand, the performance of
NoAdapt-10k is remarkably higher than NoAdapt-2k.
Performance in terms of DA selection. The performancein terms of dialog act selection accuracy for all the policies
is shown in Table 3. Similarly, the dialog policies learned by
DPRA outperformed those of ActEmb and NoAdapt with alarge margin (p < 0.05) for all values of k. Surprisingly,
when more data available the performance gap between
DPRA and the other methods increases. In fact, there is only asubtle increase in the action selection accuracy of the policies
8

learned by ActEmb and NoAdapt when k increases from 1 to
16. Recall that the results in Table 2 were reported under asetting where the policies were run against a simulator that iscreated from all 24 dialogs in the target dataset. Therefore, if
we train a policy using a simulator created from 16 dialogs,
the performance in terms of average reward per episode will
be much higher than using a simulator from just one dialog.
However, because ActEmbed and NoAdapt does not use
full knowledge from the source task policy, 16 dialogs are
insufficient to train a policy with high action selection accuracy. Thus, the gain is modest when increasing the amountof available data in Table 3. In contrast, DPRA can retain
knowledge from the source task policy and effectively adapt
it to the target task, thus achieving high performance in termsof action selection accuracy, especially when more data are
available.
V. RELATED WORKS

Many studies have addressed policy adaptation forreinforcement-learning-based goal-oriented dialog management.
Chen et al. [6] proposed a policy adaptation method using
a multi-agent dialog policy. They used an explicit representation of the dialog state, which contains of multiple slot
information. For each slot, they built an “agent” that learns
how to choose actions corresponding to this slot. The dialog
policy is an ensemble of these agents. In the target task,for each new slot information, the network weights of its
corresponding agent are initialized using the weights of theagents that have been trained in the source task. Although thisadaptation method does not require the state representation
to be similar, as in DPRA, it has a restriction: the state
representation of both tasks must be explicit, such as slots orvalues. In addition, this method requires identical action setsfor each agent to perform weight initialization. This makes
Chen et al.’s method [6] less flexible than DPRA in terms of
VOLUME 4, 2016

Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

action space restriction.
Ilievski et al. [7] introduced a new policy adaptation
method by using weight bootstrapping and also used slot
information for the state representation. Their method shares
the slots and actions across the source and target tasks andconstructs a neural network where the input layer ’s numberof neurons equals to the number of unique slots in both tasks.
Similarly, the output layer has the number of neurons thatis equal to the number of total grouped actions. First, theytrained a policy on a source task by reinforcement learning.
The network weights are then fine-tuned by training on thetarget task. This adaptation method has the same restrictionof state representation as in a previous work [6]. Furthermore,[7] also requires overlapping of the action sets of the source
and target tasks. Our proposed method is more flexible andlacks this limitation.
Mendez et al. [8] proposed an adaptation of dialog policy
using action embeddings. They argued that there is a set ofaction embeddings can be shared across the source and thetarget tasks. In practice, action embedding is represented bya hidden layer that is fully connected to both the input andoutput layers. After training the policies from all the source
tasks, the weight parameters that connect the input and thehidden layers are used to initialize the corresponding weights
in the target policy’s neural network. We used this method
as our baseline because this adaptation method requires noadditional knowledge or relations between the source andtarget tasks and is comparable to our proposed method interms of flexibility.

APPENDIX A REWARD FUNCTIONS IN THE
EXPERIMENT

Table 4 shows the reward in the source task that received by
the agent when selecting an action given the user dialog act(u) and the deception label (d), which are generated by theuser simulator.
TABLE 4. Reward definition for the source task.

Dialog state
User DA (u) d0
Accept1
0
Reject1
Question0
Hesitate0

Rewards
Framing
–10
+10
+10
+10
+10
+10

Offer
–10
–10
+10
–10
–10
+10

End
+100
-100
–100
–100
–100
–100

The reward definition for the target task in Section IV isshown in Table 5. Note that Offer_New gives +10 reward only
if it is selected in the first turn.
APPENDIX B PROOF OF EQUATION 1 AND EQUATION
10First, we show the proof for (1):π(s, b) = P (b|s)
X=
P (b, a|s) (law of total probability)
a∈A

=

X

P (b|a, s)P (a|s)

a∈A

VI. CONCLUSIONSWe propose a novel method for policy adaptation in dialog management called the dialog policy reuse algorithm –DPRA. Our proposed method uses action-relation probability
for adaptation, which allows the source task policy to bereused for the action selection of the target task. DPRA learns
the action-relation probability from dialog samples in bothtasks using a mixture density network, and can immediately
derive a policy for the target task. Thus, DPRA learns thetarget task’s policy much more quickly than conventional
methods that require RL training and user simulation. Since
our proposed method does not employ the user simulator, it
can avoid the problem of low performance due to errors inconstructing the user simulator.
Future work will conduct a deeper scrutiny of DPRA todetermine in which adaptation setting it works and what kindof performance we should expect from it. In addition, DPRA
currently does not take the changes in reward dynamics
into consideration. We believe that if we can incorporate an
estimation of such changes in reward functions of the source
and target tasks, we can further improve the performanceof the policies learned by DPRA. Finally, we also want toinvestigate the applications of our method to other settings,
such as adaptation for autonomous control tasks.
.
VOLUME 4, 2016=

X

P (b|a, s)π(s, a) Q.E.Da∈A

The following is full proof for (9):XX
π(s, b) =P (bj |s)
bj ∈B

b∈B

=

X X

P (ai , bj |s)bj ∈B ai ∈A=X X

P (ai |s)P (bj |ai , s)bj ∈B ai ∈A

=

X
ai ∈A

=

X
ai ∈A

=

X

P (ai |s)(X

P (bj |ai , s))bj ∈BP (ai |s)(Xwij (s))bj ∈BP (ai |s) · 1

ai ∈A(sum of component weights is 1)
X=
P (a|s) = 1 Q.E.Da∈A

9Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSTABLE 5. Reward definition for the target task. Note: Offer_New gives +10 reward only if it is selected in the first turn.

Dialog state
User DA (u)d0
Accept1
0
Reject1
Question0
Hesitate
0

Offer_Change
–10
–10
+10
–10
–10
+10

Offer_New
–10
–10
–10
–10
–10
–10

Rewards
Framing_Answer
–10
–10
–10
–10
+10
–10

Framing_Argue
–10
+10
–10
+10
–10
+10

End
+100
-100
–100
–100
–100
–100

TABLE 6. Experiment environment details.

Operating system
CPU
GPU
Programming language
Frameworks

Description
Ubuntu 16.04.6 LTS – “xenial”
Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
NVIDIA Corporation GM200 [GeForce GTX TITAN X]
Python 2.7.12
PyTorch 1.4.0, CUDA 9.0, NumPy 1.16.6

APPENDIX C DETAILS ON EXPERIMENTENVIRONMENT

Table 6 shows the details of the environment that was used
for our experiments described in Section IV.
REFERENCES[1] K. Scheffler, & S. Young, “Automatic learning of dialogue strategy using
dialogue simulation and reinforcement learning,” in Proc. HLT, San Diego,
CA, USA, 2002, pp. 12–19.
[2] J.D. Williams, P. Poupart, and S. Young, “Partially Observable Markov
Decision Processes with Continuous Observations for Dialogue Management,” in Proc.6th SigDial, Lisbon, Portugal, 2005.
[3] J.D. Williams and S. Young (2005, November). Scaling up POMDPs for
Dialog Management: The“Summary POMDP”Method. Presented at IEEE
Workshop on Automatic Speech Recognition and Understanding.
[4] A. Lazaric,“Transfer in reinforcement learning: a framework and a survey,” in Reinforcement Learning, vol. 12, Springer, Berlin, Heidelberg, 2012,
pp. 143–173.
[5] R. A. C. Bianchi, L. A. Celiberto Jr, P. E. Santos, J. P. Matsuura, , &R.L. de Mantaras, “Transferring knowledge as heuristics in reinforcement
learning: A case-based approach,” in Artificial Intelligence, vol. 226,
Elsevier, pp. 102–121, 2015.
[6] L. Chen, C. Chang, Z. Chen, B. Tan, M. Gašić, & K. Yu (2018, April).
“Policy adaptation for deep reinforcement learning-based dialogue management,” in Proc. ICASSP, Calgary, AB, Canada, 2018, pp. 6074–6078.
[7] V. Ilievski, C. Musat, A. Hossmann, & M. Baeriswyl, “Goal-oriented
chatbot dialog management bootstrapping with transfer learning,” in Proc.
IJCAI, Stockholm, Sweden, 2018, pp. 4115–4121.
[8] J.A. Mendez, A. Geramifard, M. Ghavamzadeh, and B. Liu. (2019, December). Reinforcement Learning of Multi-Domain Dialog Policies Via
Action Embeddings. Presented at the 3rd Workshop on Conversational AI:
Today’s Practice Tomorrow’s Potential, NeurIPS. Available: http://alborzgeramifard.com/workshops/neurips19-Conversational-AI/Papers/33.pdf[9] C.M. Bishop, “Mixture Density Networks," Aston University. Birmingham, UK. February, 1994.
[10] R. Bellman, “A Markovian decision process,” Journal ofmathematics and mechanics, vol. 6, no. 5, pp 679–684, 2957.
www.jstor.org/stable/24900506. Accessed on: June, 8, 2020.
[11] C.J.C.H Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8,
no.3–4, pp. 279–292, 1992.
[12] V. Mnih, K. Kavukcuoglu, D. Silver et al., “Human-level control throughdeep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533,
Feb. 2015. DOI: https://doi.org/10.1038/nature14236[13] R.H. Crites and A.G. Barto, (1995, November). “An actor/critic algorithm
that is equivalent to Q-learning,” in Proc. Advances in Neural Information
Processing Systems, Denver, CO, USA, 1995, pp. 401–408.
10[14] V.R. Konda and J.N. Tsitsiklis, “Actor-critic algorithms,” in Proc. Advances in neural information processing systems, Denver, CO, USA, 2000,
pp. 1008–1014.
[15] R.J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no.3-4, pp.
229–256, 1992.
[16] B. Dhingra, L. Li, X. Li, J. Gao, Y.N. Chen, F. Ahmad, & L. Deng, (2017,
July), “Towards End-to-End Reinforcement Learning of Dialogue Agentsfor Information Access, ” in Proc. ACL, Vancouver, BC, Canada, 2017, pp.
484–495.
[17] T. Hiraoka, G. Neubig,S. Sakti, T. Toda, and S. Nakamura. (2014, August).
Reinforcement learning of cooperative persuasive dialogue policies using
framing. Presented at COLING 2014, the 25th International Conference
on Computational. Available: https://www.aclweb.org/anthology/C141161.pdf[18] K. Yoshino, and T. Kawahara, “Conversational system for information
navigation based on POMDP with user focus tracking,” in Computer
Speech & Language vol. 34, no. 1, pp. 275–291, 2015.
[19] T.T. Nguyen, K. Yoshino, S. Sakti, and S. Nakamura, “Dialog Managementof Healthcare Consulting System by Utilizing Deceptive Information, ” inTransactions of the Japanese Society for Artificial Intelligence, vol. 35, no.
1, 2020.
[20] T. Zhao and M. Eskenazi, “Towards End-to-End Learning for Dialog State
Tracking and Management using Deep Reinforcement Learning,” in Proc.
SIGDIAL, Los Angeles, CA, USA, 2016, pp. 1–10.
[21] J.V. Serban et al, “A survey of available corpora for building data-driven
dialogue systems: The journal version,” in Dialogue & Discourse, vol. 9,
no. 1, pp. 1–49, 2018.
[22] T.T. Nguyen, K. Yoshino, S. Sakti, and S. Nakamura,. (2019, December).
Hierarchical Tensor Fusion Network for Deception Handling Negotiation
Dialog Model. Presented at the 3rd Workshop on Conversational AI:
Today’s Practice Tomorrow’s Potential, NeurIPS. Available: http://alborzgeramifard.com/workshops/neurips19-Conversational-AI/Papers/10.pdf[23] J.R. Wilson “Variance reduction techniques for digital simulation,” American Journal of Mathematical and Management Sciences, vol.4, no. 3-4,
pp. 277-–312, 1984.
[24] P. L’Ecuyer, “Efficiency improvement and variance reduction,” in Proc.
WSC, pp. 122–132, 1994.
[25] R.S. Sutton, D.A. McAllester, S.P. Singh, and Y. Mansour, “Policy gradientmethods for reinforcement learning with function approximation,” in Proc
NeuRIPS, Denver, CO, USA, pp. 1057–1063, 2000.
[26] T. Baltrušaitis, P. Robinson, and L.P. Morency, “Openface: an open source
facial behavior analysis toolkit,” in Proc. WACV, Lake Placid, NY, USA,2016, pp. 1-10. doi: 10.1109/WACV.2016.7477553.
[27] F. Eyben, M. Wöllmer, and B. Schuller. (2010, October). Opensmile:
the munich versatile and fast open-source audio feature extractor. Presented at ACM international conference on Multimedia. Available:
https://dl.acm.org/doi/pdf/10.1145/1873951.1874246
VOLUME 4, 2016

Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS[28] B. Schuller, S. Steidl, and A. Batliner. (2009). The interspeech2009 emotion challenge. Presented at Tenth Annual Conferenceof the International Speech Communication Association. Available:
https://mediatum.ub.tum.de/doc/980035/file.pdf

TUNG T. NGUYEN received his Bachelor degreein Computer Science from University of Engineering and Technology, Vietnam National University,
in 2014 and the M.Eng degree in Information
Science from Nara Institute of Science and Technology (NAIST), in 2017.
He is currently a doctoral candidate at
Augmented Human Communication Laboratory,
NAIST. His research area includes multimodal
processing, reinforcement learning, and spoken

SAKRIANI SAKTI received her B.E. degree inInformatics (cum laude) from Bandung Instituteof Technology, Indonesia, in 1999. In 2000, she
received DAAD-Siemens Program Asia 21st Century Award to study in Communication Technology, University of Ulm, Germany, and received her
MSc degree in 2002. During her thesis work, she
worked with the Speech Understanding Department, Daimler Chrysler Research Center, Ulm,
Germany. Between 2003-2009, she worked as a
researcher at ATR SLC Labs, Japan, and during 2006-2011, she worked as
an expert researcher at NICT SLC Groups, Japan. While working with ATRNICT, Japan, she continued her study (2005-2008) with Dialog Systems
Group University of Ulm, Germany, and received her Ph.D. degree in2008. She actively involved in collaboration activities such as Asian PacificTelecommunity Project (2003-2007), A-STAR, and U-STAR (2006-2011).
In 2009-2011, she served as a visiting professor of the Computer Science
Department, University of Indonesia (UI), Indonesia. In 2011-2017, she wasan assistant professor at the Augmented Human Communication Laboratory,
NAIST, Japan. She also served as a visiting scientific researcher of INRIA
Paris-Rocquencourt, France, in 2015-2016, under “JSPS Strategic Young
Researcher Overseas Visits Program for Accelerating Brain Circulation”.
Currently, she is a research associate professor at NAIST, as well as a
research scientist at RIKEN, the Center of for Advanced Intelligent Project
AIP, Japan. She is a member of JNS, SFN, ASJ, ISCA, IEICE, and IEEE. She
is also the officer of ELRA/ISCA Special Interest Group on Under-resourced
Languages (SIGUL) and a board Member of Spoken Language Technologies
for Under-Resourced Languages (SLTU). Her research interests include
statistical pattern recognition, graphical modeling framework, deep learning,
multilingual speech recognition & synthesis, spoken language translation,affective dialog system, and cognitive communication.

dialog system.
Mr. Nguyen is a recipient of the scholarship from the Japanese Ministryof Education, Culture, Sport, Science, and Technology (MEXT) from 2017
to 2020. He is a member of the Japan Association for Natural Language
Processing.

KOICHIRO YOSHINO received his B.A. degreein 2009 from Keio University, M.S. degree in informatics in 2011, and a Ph.D. degree in informatics in 2014 from Kyoto University, respectively.
From 2014 to 2015, he was a research fellow(PD) of Japan Society for Promotion of Science.
From 2015 to 2016, he was a research assistantprofessor of the Graduate School of Information
Science, Nara Institute of Science and Technology(NAIST). Currently, he is an assistant professor of
NAIST. He is also a researcher of PRESTO, JST, concurrently. He is working
on areas of spoken and natural language processing, especially on spoken
dialogue systems. Dr. Koichiro Yoshino received the JSAI SIG-researchaward in 2013. He is a member of IEEE, ISCA, IPSJ, and ANLP.
VOLUME 4, 201611

Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSSATOSHI NAKAMURA is Professor of Graduate
School of Science and Technology, Nara Institute of Science and Technology, Japan, Project
Leader of Tourism Information Analytics Team of
RIKEN, Center for Advanced Intelligence Project
AIP, Honorarprofessor of Karlsruhe Institute ofTechnology, Germany, and ATR Fellow. He received his B.S. from Kyoto Institute of Technology
in 1981 and Ph.D. from Kyoto University in 1992.
He was Associate Professor of Graduate School ofInformation Science at Nara Institute of Science and Technology in 19942000. He was Director of ATR Spoken Language Communication ResearchLaboratories in 2000-2008 and Vice president of ATR in 2007-2008. He
was Director General of Keihanna Research Laboratories and the Executive
Director of Knowledge Creating Communication Research Center, National
Institute of Information and Communications Technology, Japan in 20092010. He is currently Director of Augmented Human Communication laboratory and a full professor of Graduate School of Information Science at
Nara Institute of Science and Technology. He is interested in modeling andsystems of speech-to-speech translation and speech recognition. He is one of
the leaders of speech-to-speech translation research and has been serving forvarious speech-to-speech translation research projects in the world including
C-STAR, IWSLT, and A-STAR. He received Yamashita Research Award,
Kiyasu Award from the Information Processing Society of Japan, Telecom
System Award, AAMT Nagao Award, Docomo Mobile Science Awardin 2007, ASJ Award for Distinguished Achievements in Acoustics. He
received the Commendation for Science and Technology by the Ministerof Education, Science and Technology, and the Commendation for Scienceand Technology by the Minister of Internal Affair and Communications. He
also received the LREC Antonio Zampolli Award 2012. He has been Elected
Board Member of International Speech Communication Association, ISCA,
2011-2019, IEEE Signal Processing Magazine Editorial Board Member
since April 2012, IEEE SPS Speech and Language Technical Committee
Member since 2013-2016, IEEE Fellow since 2016 and ISCA Fellow since
2020.

12

VOLUME 4, 2016