This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access

Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier xx.xxxx/ACCESS.2020.DOI

Entrainable Neural Conversation Model
Based on Reinforcement Learning
SEIYA KAWANO1 , MASAHIRO MIZUKAMI2 , KOICHIRO YOSHINO1 (Member, IEEE), SATOSHINAKAMURA1 (Fellow, IEEE)
1
2

Graduate School of Science and Technology, Nara Institute of Science and Technology, Nara, Japan.
NTT Communication Science Laboratories, Kyoto, Japan.

Corresponding author: Seiya Kawano (e-mail: kawano.seiya.kj0@is.naist.jp).
This research and development work were supported by JST PRESTO (JPMJPR165B) and JST CREST (JPMJCR1513).

ABSTRACT The synchronization of words in conversation, called entrainment, is generally observed inhuman-human conversations. Entrainment has a high correlation with dialogue success, naturalness, andengagement. In this paper, we define entrainment scores based on the word similarities in semantic space toevaluate the entrainment of system generation. We optimized a neural conversation model to the entrainmentscores using reinforcement learning so that the system can control the degree of entrainment of the system
response. Experimental results showed that the proposed entrainable neural conversation model generated
comparable or more natural responses than conventional models and satisfactorily controlled the degree ofentrainment of the generated responses.
INDEX TERMS Neural Conversation Model, Conditional Response Generation, Entrainment, Dialogue
Evaluation, Reinforcement Learning

I. INTRODUCTION

Ntrainment is a well-known conversational phenomenon in which dialogue participants mutually synchronize with regards to various aspects: lexical choice [1],syntax [2], style [3], acoustic prosody [4], [5], turn-taking [6],[7], and dialogue acts [8]. Entrainment has a high correlation
with dialogue success, naturalness, and engagement [9]–[11].
Some existing works evaluated the dialogue quality and theperformance of dialogue systems through entrainment analysis [12]. Although phenomena related to entrainment suggest that the quality of human-human and human-machinedialogues can be improved, it remains challenging to build a
dialogue system that can explicitly consider the entrainment
phenomena in the framework of a neural conversation model,
which has been actively studied in recent years [13], [14].

E

In this paper, we incorporate entrainment phenomena intoa neural conversation model for building a more natural anduser-satisfied dialogue system. We construct a neural conversation model that can control the degree of entrainment ofgenerated responses based on a framework of reinforcement
learning (RL) [15]. We define the automatic entrainmentscores based on the local interpersonal distance [11], which
focuses on lexical entrainment. We use this score to optimize
a neural conversation model by RL.
VOLUME x, 2020

In Section III, we describe our task of entrainable conversation modeling (Section III-A), a conditional generation
model based on conventional architecture (Section III-B),
and our proposed model optimized to entrainment scores
by RL (Section III-C2). In experiments, we performed apreliminary analysis using the defined entrainment scores
to clarify the relationship between user assessment and entrainment phenomena in a chit-chat dialogue domain (Section IV). Experimental results showed that our entrainmentscores correlated with human assessment in human-humanand human-machine dialogues in the chit-chat domain (Section V). As a model evaluation, we conducted subjective
and objective evaluations (Section VI). Our proposed model
generated comparable or more natural responses comparedwith general neural conversation models, which optimized byword prediction based on cross-entropy loss, and controlled
well the degree of entrainment of the generated responses(Section VII). We discuss the challenges for the advancementof entrained response generation in neural conversation models by analyzing our experimental results (Section VIII).
II. RELATED WORKS

Many studies have analyzed entrainment in dialogues andshown that we can observe the phenomena in dialogues
1This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

from various aspects: lexical choice [1], syntax [2], style [3],acoustic prosody [4], turn-taking [6], and dialogue acts [8].
Furthermore, automatic entrainment scores have been proposed that focuses on these aspects. These entrainment scores
were highly correlated with dialogue success, naturalness,and engagement [9]–[11].
Some studies used the knowledge obtained by analyzing
the entrainment to the dialogue system. One work [6] predicted the user’s turn-taking behavior by considering entrainment. Another work [16] modeled a dialogue strategy tointentionally increase the accuracy of the automatic speech
recognition using entrainment, and another [17] unified theseworks. Although these studies were conducted on modelingand predicting the entrainment of the user’s behaviors, itremains challenging problem to build a dialogue systemthat can make entrainment to users to improve the dialoguesystem’s response quality. In other words, insufficient studies
have positively affected users through entrainment by thesystem.
On the other hand, recent neural conversation models focus
on the efficient use and encoding of dialogue history [14],[18]. However, they do not directly handle entrainment phenomena because they are achieved by minimizing the crossentropy loss of word prediction in decoder networks. Boththe model networks that consider the dialogue context andthe objective function of the model itself must be improvedto achieve entrainable response generation.
In this paper, we introduce a reinforcement learning (RL)
framework [15], [19] to optimize a neural conversation model
for automatic entrainment scores. Entrainment scores are
given as RL rewards that enable neural conversation models to generate appropriately entrained responses for theirdialogue contexts. Existing studies have already described
the correlation between human assessments and automatic
entrainment scores. We performed a follow-up analysis using
chit-chat dialogue corpora to confirm that we can use thescores as an objective function. By optimizing the model tomaximizing these scores, we expect that our neural conversation model can generate more natural responses.
III. ENTRAINABLE NEURAL CONVERSATION MODEL

In this paper, we focus on lexical entrainment, which isrelated to lexical choice in dialogues. We introduce an entrainment score based on the similarity in semantic spaces
in word-distributed representation [11], [20] to capture local
entrainment trends for each turn in the dialogue. We optimize
the neural conversation model using RL to maximize theentrainment scores of the generated responses. We formulate
the problem as conditional neural conversation modeling,which uses the degree of entrainment of the response as acondition, because generating a highly entrained response is
not always appropriate even though the model has to control
the degree of entrainment based on the dialogue contexts.
In this section, we first describe an overview of the response generation task tackled in this paper (Section III-A).
Then we describe the architecture of a conditional neural2

conversation model given the degree of entrainment (Section III-B). Finally, we describe a method that optimizes theconditional neural conversation model using RL to fit thegiven degree of entrainment (Section III-C).
A. TASK DEFINITION

We formally define the task of entrainable conversation modeling as a response generation task given a dialogue context
and a degree of entrainment to the dialogue context. Define
generated response word sequence R = {w1 , w2 , · · · , wT },given dialogue context H = {H1 , H2 , . . . , HN } and degreeof entrainment of target response rtarget ∈ R. N is the dialogue length, and T is the number of words in an utterance.

FIGURE 1: Task of entrainable conversation generation

In this setting, response R is required to satisfy not only theappropriateness to the dialogue context but also the degree ofentrainment to the dialogue context (Fig. 1). In other words,
the neural conversation model enforces entrainment degree
rgenerated ∈ R of the actual generated response to be closer
to indicated entrainment degree rtarget . This optimization isachieved by minimizing the relative error of both entrainment
degrees:minimize relative_error(rtarget , rgenerated ).
rgenerated ∈R(1)

As an approach to building such neural conversation models controllable by a given condition, such as the entrainment
degree, vector concatenation is widely used between a word
vector and the vectorized condition to feed a decoder input[21], [22]. Some other works proposed to extend models [23],[24] for conditional generation according to given emotion
labels in the task of the emotional dialogue generation.
B. NEURAL CONVERSATION MODEL BASED ON
ENTRAINMENT DEGREE

We introduce a conditional neural conversation model basedon a hierarchical encoder-decoder model [14] with a context
attention mechanism, which explicitly gives an embedded
vector of entrainment degree to the decoder (Fig. 2). We
apply the vector concatenation as a widely used method for
conditioning the decoder.
The encoder network has a hierarchical structure thatconsists of utterance and context encoders. The utterance
VOLUME x, 2020

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS[25] for each hidden state and obtain context vector h̄t :exp(score(cj , h0t ))αj = PN,0j̃=1 exp(score(cj̃ , ht )
h̄t =

N
X

αj̃ · cj̃ .

(6)

(7)

j̃=1

The output words in step t are predicted using computed
context vector h̄t :FIGURE 2: Neural conversation model with entrainment degree as a conditionĥt = tanh(Linearattn ([h̄t ; h0t ])),

(8)pt = softmax(Linearproj (ĥt )).

(9)

In general, training the neural conversation model is basedon minimizing the cross-entropy:
LCE = −

encoder receives a word at each time step using forward
RNNs to encode an utterance into a fixed-length vector:
hi,t = RNNutterance (hi,t−1 , Embed(wi,t )).

(2)

Here i is the number of turns in the dialogue context, and hi,tis the hidden vector obtained by inputting each word wi,t inutterance Hi . Each word wi,t , which is encoded to a fixedlength vector using an embedding layer, is used as input.
In the context encoder, utterance vectors are input toencode the dialogue history:
ci = RNNcontext (ci−1 , hi,T ).

h0t = RNNdec (h0t−1 , [Embed(wi,t−1 ); Linearent (rtarget )]),
(4)
0

(5)

Here Linearproj is a projection layer, which maps h0t to avector of vocabulary size |V|. Linearent is a linear transformation layer that embeds target entrainment degree rtarget into
a fixed-length vector. wi,t is sampled from pt and used as apart of the input for the next step. In this decoding architecture, we expect the decoder to generate a response with anappropriate degree of entrainment for the dialogue history byalso inputting entrainment degree rtarget in addition to alreadygenerated words. Note that we used teacher-forcing in the
training process [13].
We also introduce a simple attention mechanism to theabove decoder for efficiently handling the information fromthe encoded dialogue context. Specifically, let c1:N be asequence of vectors obtained by the context encoder, andlet h0t be the hidden states of the decoder in the t step. We
compute the alignment weights based on general-attention
VOLUME x, 2020

exp(xt,e ).
log P|V|t=1
k exp(xt,k )(10)Here |V| denotes the vocabulary size, xt ∈ R|V| denotes
the output of the projection layer in the decoding steps, andxt,e ∈ R|V| denotes the e-th element that correspond to target
word wt .
However, perhaps models based on minimizing crossentropy loss do not efficiently use the information in the
dialogue context [26]. Furthermore, since cross-entropy loss
is not designed to handle entrainment phenomena, we haveto define a new objective function for building a neuralconversation model that is optimized to entrainment scores.

(3)

Here hi,T is a hidden vector obtained at the last step in theencoding for each utterance. Resultant vector ci is fed into
the decoder to generate a response sentence as initial hidden
state h00 . In the decoder, hidden state h0t of the decoder andthe output probability of word pt are calculated:pt = softmax(Linearproj (ht )).

T
X

C. MODEL OPTIMIZATION TO ENTRAINMENT DEGREE
BASED ON REINFORCEMENT LEARNING

Our final goal is to build an entrainable neural conversationalmodel based on the given entrainment degree. However,model optimization based on existing cross-entropy loss does
not satisfactorily control the generation because the optimization is calculated word-by-word. In contrast, optimization
based on reinforcement learning has the potential to train
such a controllable response generation model [27]. Thus,
we introduce the REINFORCE algorithm, which is based on
reinforcement learning [15], [19].
In this section, we describe the objective function andits optimization method using the REINFORCE algorithm(Section III-C1). We introduce a reward function using an
automatic entrainment score to optimize the model (Section III-C2) and scrutinize the training procedure for ourneural conversation model based on reinforcement learning(Section III-C3).
1) REINFORCE Algorithm

The generation process in the neural conversation model isformalized as a Markov decision process (MDP) and optimized with reinforcement learning (RL) [28]. The problem
of response generation in the neural conversation model isgenerating response word sequence R = {w1 , w2 , · · · , wT }that corresponds to given dialogue history H and target
3

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

entrainment degree rtarget . Formally the generation process isdefined as choosing an action to generate word wt given astate, already generated words {w1 , w2 , · · · , wt−1 }, in timestep t [29]. Such a word selection process in the generation
is defined as an action sequence, which is generated by anactual policy in MDP.
We define a reward function based on entrainment scoresto encourage the model to generate entrained responses. The
entrainment’s evaluation score is fed as a reward to update
the generator’s policy in the RL. We use a policy gradient(REINFORCE algorithm) [15], [19] to train the policy. The
objective function and its gradient are defined:X
JRL (θ) =Gθ (wt |w1:t−1 ) · QGθ (w1:t−1 , wt ), (11)
w1:T
T

∇JRL (θ) '1 X X GθQ (w1:t−1 , wt )
T t=1
wt ∈V

· ∇θ Gθ (wt |w1:t−1 )
=

T
1X

T t=1

(12)

Ewt ∼Gθ [QGθ (w1:t−1 , wt )· ∇θ log p(wt |w1:t−1 )].

(13)

Here θ is a parameter of the policy. V is a vocabulary,
w1:t−1 indicates the already generated word sequence (state
in MDP), and p(wt |w1:t−1 ) = Gθ (wt |w1:t−1 ) is the generative probability of word wt ∈ V (action in MDP) in the
decoder. QGθ (w1:t−1 , wt ) is an action-value function thatgives an expected future reward when the system generates
word wt given the state: already generated word sequence
w1:t−1 .
The action-value function for each step is calculated usinga Monte Carlo tree search (MCTS) [29], [30] under thecurrent policy and its parameter θ:
QGθ (R1:t−1 , wt ) =(14)
 1 PNnrefn=1 r(H, R1:t , R , rtarget ) for t < T ,Nr(H, R1:t , Rref , rtarget )for t = T .
Here r(·) is a reward function that evaluates the entrainmentndegree of response R1:T = {w1 , w2 , · · · , wT }. R1:t
isthe generated response using a roll-out [29] from partialgenerated response R1:t using parameter θ. Rref is a reference
response, and n is the number of roll-outs1 . This reward
function calculates the reward based on the relative error
between a given entrainment degree and the entrainment
degree of the actual generated response to allow it to control
the entrainment degree of the generated response. Note that
we can use an arbitrary score in this formulation. We use
a reliable entrainment score based on the similarity of thesemantic space of the words to feed the entrainment degree(score) as the reward.
1 We set the number of roll-outs to 5. However, since the computation
cost of MCTS is high when training the large model, we can also adopt an
approach for speeding up the training, such as REGS [28], instead of MCTS.

42) Reward Function for Evaluating Entrainment Degree

We construct a reward function based on the idea of a local
interpersonal distance (LID), which is a previously proposed
turn-level entrainment score [11]. LID uses a predefined
number of turns (context lengths) in response to the utterances of the primary speaker (anchor). The anchor utterance
and response pair that has a minimum distance is chosen to
calculate LID. This calculation is based on local entrainment,
which is not always observed in the immediate responseto the primary speaker’s turn. It might be sustained andexhibited after a few turns [31]. In this paper, unlike the LID’s
original definition [11], we calculate the similarity between
the anchor utterance and each past contextual utterance byanother speaker and choose an anchor and contextual utterance pair with minimum distance. However, note that there isno difference in the nature of both scores.
To calculate the distance between two utterances, we use
Word Mover’s Distance (WMD) [20], which is calculated
from the distributed vector representations of words in adocument. WMD targets both semantic and syntactic information to get a distance between text documents. WMD
calculates the Earth Mover’s Distance [32] between setsof word vectors that are contained in the target sentences(documents). This calculation is based on the minimum travel
distance. Specifically, let ei ∈ Rd represents i-th word, as
defined by word-embedding E ∈ Rd×n for vocabulary of nwords. We also define a and b are n-dimensional normalized
vectors, which consist of bag-of-words of two sentences.
ai indicates the count of the word i in the sentence2 . The
WMD introduces an transport matrix T ∈ Rn×n , such that
Ti,j indicates how much of ai should be transported to bj .
Formally, the WMD learns T to minimize:
WMD(a, b) = min

n
n X
X

T ≥0

subject toand

Tij ||ei − ej ||i=1 j=1
n
X
j=1
n
X

Tij = ai∀i,

Tij = bj

∀j.

(15)

i=1

To solve this minimization problem, we used the efficient
implementations [20], [33], [34]3 .
We define target entrainment degree rtarget based on theidea of LID :2

sim(x, y) = e−WMD(bow(x),bow(y)) ,rtarget = rtarget (H, Rref ) =max

Hjother ∈H other ⊂H(16)

sim(Hjother , Rref ).
(17)

Here the sim(·) function normalizes the calculation resultsof WMD as the similarities between utterances (x and y). e
2 Note that a is normalized over all words in a.
i
3 https://github.com/RaRe-Technologies/gensim
VOLUME x, 2020

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

is a natural logarithm, and bow(·) is a function to convert
the given sentence to bag-of-words representation. Rref is thereference response corresponding to dialogue history H, andH other ⊂ H is a set consisting of the most recent k utterancesfrom the dialogue history H, excluding the primary speaker’s
utterances. rtarget is assumed to be a similarity that takes
values from 0 to 1.
Next we define entrainment degree rgenerated of the actual
generated response:
utarget = utarget (H, Rref ) =arg maxsim(Hjother , Rref ),Hjother ∈H other ⊂H

(18)
ref

rgenerated = rgenerated (H, R , R) = sim(R, utarget ).
(19)
Here utarget is a target utterance to make entrainment bysystem generation, which has the maximum similarity of
every pair formed by the anchor utterance of the reference
and a context utterance. Thus, rgenerated is calculated as thesimilarity between the generated response and the target.
The reward given to the generated response is calculated
from the relative error between rtarget and rgenerated :r = r(H, R, Rref , rtarget ) = 1 −|rtarget − rgenerated |.
max(rtarget , 1 − rtarget )(20)This reward function gives more reward when the relative error between the entrainment degree of the generated response
and the indicated entrainment degree is small. In other words,
it gives penalty if the generated utterance is over or underentrained compared with the reference.
We used different functions for rtaget and rgenerated because
using the same function will lead to learning a lazy policythat always refers to the previous utterance.
3) Model Training based on REINFORCE Algorithm

The training procedure of a neural conversation model with
RL is shown in Algorithm 1.
Algorithm 1 Training Procedure
Require: generator policy Gθ ; roll-out policy Gθ
1: Initialize Gθ with random weights θ
2: Pretrain Gθ to minimize LCE. (10)
3: for number of iterations do
4:
G0θ ← Gθ
5:
for number of steps do
6:
sample (H, Rref , rtarget ) from training data
7:
generate response R using G0θ on H and rtarget
8:
compute QGθ for (H, R, rtarget ) using G0θ . (14)9:update Gθ based on JRL (θ). (11)First, we pre-train the neural conversation model by minimizing cross-entropy loss LCE . Then we train it to maximize
objective function JRL (θ) using reinforcement learning andadd LCE to the loss to stabilize the training. This approach
VOLUME x, 2020

works as a teacher forcing and prevents the collapse ofpolicies due to the model’s inability to access the reference
response [28]. The policy used to calculate QGθ is updated
every 20 steps. We use the model with the highest reward sum
for the generated response and the deterioration of perplexity
within 1.0 points in the validation set for the evaluation.
IV. ENTRAINMENT ANALYSIS SETTING

LID, which is used as a reward in this paper, probably
correlates with human assessment (therapeutic outcomes andaffective behaviors) in the dialogues of clinical psychology
and psychotherapy [11]. On the other hand, no examination
has focused on chit-chat dialogues, which are the main focusof this paper. Therefore, we performed a preliminary analysis to clarify the relationship between user assessment andentrainment in chit-chat dialogues.
We used Spearman’s rank correlation coefficient to evaluate the relationship between the Conversational Linguistic
Distance (CLiD) [11], a dialogue session-level entrainmentscore calculated by the mean of the LIDs, and the user
assessment assigned to each dialogue. CLiD is defined:P
ref(H,Rref )∈D rtarget (H, R ).
(21)
CLiD(D) =|D|
We applied (17) to each turn of the dialogues and averaged
the results as the dialogue level entrainment scores. Here,(H, Rref ) ∈ D is a context and response pair for each turnin the dialogue. Note that the definition changes from theoriginal CLiD to fit our problem.
For the entrainment analysis, we used the following chitchat dialogue corpora:
ConvAI2-wild-evaluation: Dialogues between a human and a system that participated in the ConvAI2
competition4 . Each dialogue was evaluated by humanparticipants on a five-point scale.
• NTT-chit-chat : Dialogues between human participantsthat covered as wide range of topics [35]. Participantsin each dialogue evaluated it from the following three
viewpoints on a seven-point Likert scale: 1) strongly
disagree; 2) disagree; 3) Slightly disagree; 4) neither;5) slightly agree; 6) agree; 7) strongly agree.
– Q1: “I am satisfied with the current dialogue. I’dlike to have such a dialogue again.”– Q2: “I found myself interested in the topic of thecurrent dialogue.”– Q3: “In the current dialogue, I spoke positively on
my own.”•TABLE 1: Number of dialogues/utterances in each corpus
CorpusConvAI2-wild-evaluation
NTT-chit-chatDialogues
2,483
3,483

Utterances
41,41556,5664 http://convai.io/data5

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

V. ENTRAINMENT ANALYSIS RESULTS

We performed a correlation analysis of ConvAI2-wildevaluation, as shown in Table 2. Here ρ is the correlation
calculated by Spearman’s rank correlation analysis, and thep-value is the probability of the null hypothesis. We calculated the CLiD for two types based on their attributes because
the speaker has distinctly different attributes: “Human →System” shows the human responses to the system, and“System → Human” shows the system responses to thehumans.
TABLE 2: Entrainment analysis results for ConvAI2-wildevaluation
Types
Human→System
System→Human

ρ
0.19
0.22

k=1
p-value
6.25 × 10−22
9.22 × 10−31

ρ
0.24
0.23

k=2
p-value
2.83 × 10−33
5.51 × 10−31

As shown in Table 2, we confirmed that CLiD has apositive correlation with human scores, regardless of any
setting used to calculate it. This result indicates that theentrainment degree is critical for improving the quality ofhuman-machine dialogues. Since many systems based on aneural network are not able to handle the dialogue context[26], this result might be deeply related not only to entrainment but also to whether the system can generate a contextrelevant response. To compare cases using different values ofk, we confirmed a stronger correlation in the case of k = 2.
In fact, humans often respond with an awareness of both theprevious utterances but also deeper utterances from the past
in a dialogue history [31].
We also performed a correlation analysis between averageassessments by two participants and CLiD in an NTT-chitchat (Table 3).
TABLE 3: Entrainment analysis results for NTT-chit-chat
Question
Q1
Q2
Q3

ρ
0.13
0.11
0.09

k=1
p-value
2.45 × 10−15
6.30 × 10−12
4.19 × 10−8

ρ
0.10
0.08
0.07

k=2
p-value
7.13 × 10−10
4.11 × 10−7
2.66 × 10−5

Table 3 shows a moderately positive correlation between
the user assessments corresponding to these questions andthe CLiD. This result suggests that using entrainment in
dialogues is an effective strategy to improve user satisfaction
in chit-chat dialogues. As stronger correlation is observed in
k = 1 than in k = 2. This is probably because NTT-chitchat has multiple utterances per turn. In other words, when
we apply CLiD to dialogues that contain a lot of informationin one turn, it is difficult to find strong correlations between
CLiD and human scores, because LIDs, which are CLiD
components, will be biased by the number of words in theutterances.
These results indicate that CLiD, which is calculated by
LID averages, is a useful and strict score to evaluate dialogues in the chit-chat domain, if their utterance length is6

limited. In other words, these results support our hypothesis:
maximizing the LIDs in dialogues increases dialogue quality.
VI. EVALUATION SETTING FOR RESPONSE
GENERATION

We confirmed that improving LID scores is important in
human-machine dialogue setting as well, which is a basic
hypothesis of our entrainable dialogue modeling. In thissection, we describe the experimental settings to confirm theeffect of our proposed entrainable neural conversation model.
1) Dataset

We used the dataset provided at the ConvAI2 competition,
which was also used to train the system in the ConvAI2-wildevaluation described in Section IV. This dataset was divided
into train, validation, and test sets (Table 4). We divided theoriginal development data into validation and test sets5 .
TABLE 4: Number of dialogues/utterances in ConvAI2
datasetTrain
Validation
Test

Dialogues
17,876
498
499

Utterances
262,862
7,798
7,788

2) Competing Models

We compared the following different types of neural conversation models in our evaluations:• ASEQ2SEQ: a standard neural conversation model that
encodes a previous utterance as a query for decoding a response with an attention mechanism (generalattention) [25].
• HED: a hierarchical encoder-decoder model [14] without an attention mechanism and conditioning to a decoder.
• AHED: a model that combines an attention mechanismwith the HED model.
• C-ASEQ2SEQ: a model with conditioning based on
ASEQ2SEQ. We gave the condition (degree of entrainment) as described in Section III-B.
• C-HED: a HED model with a conditioning mechanism.
We gave the condition (degree of entrainment) as described in Section III-B.
• C-AHED: an AHED model with conditioning. We gave
the condition (degree of entrainment) as described in
Section III-B.
We trained these neural conversation models using conventional cross-entropy loss (+CE) and our proposed optimization based on reinforcement learning (+RL). We usedthe entrainment scores as a condition given to the decoderand explored the case with different k ∈ {1, 2} for score
calculation. When k = 1, the entrainment score is calculatedusing only the previous utterance; when k = 2, it is calculated using the two most recent utterances by a non-primary
5 Note that the original test set in ConvAI2 dataset are private.
VOLUME x, 2020

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

speaker. Since the dialogue is performed alternately by two
speakers, the neural conversation model needs to handle atmost four utterances in a dialogue history when k = 2.
3) Hyper-parameter Settings

We used the same hyper-parameter settings in these models.
The vocabulary size was 15000, the word embedding size
was 300, the entrainment embedding size was 50, the hiddensize was 500. We used a two-layer Gated Recurrent Unit
(GRU) [36] as an RNN. In the training, we used a minibatch size of 128, and an Adam optimizer [37] with a learning
rate of 1e-4. For the WMD calculation, we used pre-trained,
word-distributed vectors6 , which normalized the norm to 1
for each. We set the maximum length of the dialogue history
to 4.
4) Automatic Evaluations

We automatically evaluated the generation results using references in the test set. We used a beam search (a beam widthof 5) for generating examples to be evaluated. For automatic
evaluation, we used the following five different metrics:• Perplexity (PPL) is a general metric for evaluating alanguage model performance. The model Likelihoodsof the reference responses are calculated. Note that the
perplexity scores do not directly reflect the quality ofgeneration; for example, dull responses also have good
perplexity scores.
• BLEU, which is the most popular automatic evaluation
metric of language generation tasks, calculates the similarity between references and generated responses [38]based on n-gram precision. We used BLEU2, which
considers uni-grams and bi-grams because matches inhigher-order n-grams are rarely observed response generation tasks.
• WMD is the average similarity between the references
and the generated responses for each case in the test set.
The similarity is calculated based on (16). The score ismultiplied by 100 and displayed in a range of 0 to 100.
• r is an average reward calculated from (20) to eachgeneration. When this score is high, the entrainment
degree of the generated response shows a similar degree
to the reference. In other words, it shows that the neuralconversation model controlled the response content wellbased on the entrainment degree. We sorted the test
sets by the entertainment scores of the references anddivided them into four parts to calculate r for each(r0∼25% , r25∼50% , r50∼75% , r75∼100% ). For example,
r0∼25% shows the average reward of examples that haveless entrainment scores in the references. These scoresare multiplied by 100 and displayed in a range of 0 to100.
• Entropy (Ent) is a diversity metric [39] thatreflects the evenness of the empirical n-gram
6 For English data: http://nlp.stanford.edu/projects/glove/and for Japanese data: http://www.worksap.co.jp/nlp-activity/word-vector/
VOLUME x, 2020

distribution Pfor the given responses: Ent=
1P
P C(w) 0 , where, V is
C(w)
log
w∈V
w∈V C(w)w0 ∈V C(w )the set of all n-grams in the given responses, and C(w)denotes the frequency of n-gram w. We set the uni-gramfor evaluation.
5) Human Subjective Evaluations

Automatic evaluation scores still have a problem since they
do not have high correlation with human subjective evaluation results [40]. Thus, we also examined models with ahuman subjective evaluation to confirm the naturalness ofthe generated responses. In the evaluation of naturalness, weused a 3-point scale in accordance with an existing work [41].
240 generated responses were randomly selected from thetest set, and human annotators added their evaluation scoresfor each sample by looking at the dialogue contexts. Detailed
descriptions follow.
• +2: This response is not only relevant and natural, butalso informative and interesting.
• +1: This response can be used as a response to thecontext, although it is universal, like “Yes, I see,” “Me
too,” or “I don’t know.”• 0: This response cannot be used as a response to thiscontext. It is either semantically irrelevant or dis-fluent.
Three annotators evaluated each sample, and the finalscore was decided by a simple majority. If the evaluation wascompletely disagreed (0, +1, and +2), the example was scoredas 1.
VII. EXPERIMENTAL RESULTS ON ENTRAINEDRESPONSE GENERATION
A. AUTOMATIC EVALUATION RESULTS

We show the automatic evaluation results of response generation models in Table 5. Our proposed models using the target
entrainment degree as a condition showed improvement onr from the baseline models and achieved comparable performance on other metrics. Our proposed models controlled
the generation at a high level, based on the indicated entrainment degree. In particular, we confirmed a remarkable
improvement in models that applied reinforcement learning(C-HED+RL and C-AHED+RL). C-AHED+RL showed thebest performance in k = 2, indicating that the attention
mechanism for context works well when the model uses
longer contexts. However, we still have a problem with generation performance r75∼100% , which gives very high entrainment scores as a condition. In other words, generating highly
entrained responses is more challenging. Furthermore, ourproposed models based on reinforcement learning showed aconsistent improvement of WMD and Ent, and BLEU wascomparable to the baseline models.
Then we traced the changes in the generation performance
when we gave different fixed examples of rtarget as a condition for the generation models instead of the oracle. The
results are shown in Fig. 3. Here the vertical dashed lineis the median of oracle rtarget . Our proposed models, which
7

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSTABLE 5: Automatic evaluation results for each neural conversation model

Models (k = 1)
ASEQ2SEQ+CE
HED+CEAHED+CEC-ASEQ2SEQ+CE
C-HED+CE
C-AHED+CE
C-ASEQ2SEQ+RL
C-HED+RL
C-AHED+RL
Models (k = 2)
HED+CEAHED+CE
C-HED+CE
C-AHED+CE
C-HED+RL
C-AHED+RL

PPL
40.80
37.53
39.11
40.05
37.12
39.91
41.41
37.54
40.01
PPL
37.53
39.11
37.01
38.76
37.24
40.79

BLEU2
7.82
6.62
6.71
3.69
6.56
6.64
7.28
6.69
7.42
BLEU2
6.62
6.71
7.28
5.17
6.39
7.30

WMD
38.77
36.40
38.16
38.77
39.17
38.84
39.59
40.57
39.96
WMD
36.40
38.16
39.89
38.43
40.67
40.48

r0∼25%
91.22
90.72
91.04
90.90
91.41
91.58
91.60
91.72
91.84
r0∼25%
87.99
88.11
90.79
89.49
90.25
90.93

r25∼50%
88.24
89.21
88.44
88.08
88.52
89.25
89.36
90.51
90.53
r25∼50%
86.69
86.21
88.84
87.64
89.49
91.24

r50∼75%
82.80
83.79
82.40
84.19
84.25
85.00
86.12
88.01
86.36
r50∼75%
82.45
81.14
84.77
84.44
86.91
87.14

r75∼100%
62.35
63.06
62.11
68.82
70.04
70.02
72.65
74.53
72.75
r75∼100%
62.74
61.86
70.15
67.54
74.18
72.63

r
81.15
81.69
81.00
83.01
83.55
83.96
84.93
86.19
85.37
r
79.97
79.33
83.64
82.27
85.21
85.49

Ent
5.03
5.21
5.11
5.33
5.20
5.11
5.59
5.97
5.52
Ent
5.21
5.11
5.17
5.18
5.62
5.87

are optimized by reinforcement learning, showed consistentimprovement compared with the other models. For r, we
confirmed the highest performance around the median oforacle rtarget . Lower scores on high rtarget were probably
caused by a lack of training samples of high rtarget . For WMD
and Ent, we confirmed increasing trends in both scores when
we give a high rtarget . On the other hand, both scores are low
in the range of low rtarget . This result was caused by dull
responses, which have small diversity and little relevance tothe references. In some cases, our models outperformed theresults of giving the oracle conditions. This indicates that ourmodels are robust even if the condition given to the model isdifferent from the oracle.

entrainment scores. Note that C-HEDs and C-AHEDs are including both results of cases in +CE and +RL. We calculated
the polyserial correlation ρ [42] between human scores andentrainment scores instead of Spearman’s rank correlation
since there is only a 3-point scale for human scores. We
identified significant positive correlations between humanscores and entrainment scores for all of the groups regardlessof the k settings. This result is consistent with the resultsof the preliminary analysis in Section V. Note that we can
not compare the magnitude of the correlations in each group.
This is due to the correlations will be small for groups witha low frequency of score 0 since the nature of the evaluation
score based on the 3-point scale.

B. HUMAN SUBJECTIVE EVALUATION RESULTSTABLE 6: Frequency of each subjective evaluation score.
Weighted-Avg is a weighted average by frequency of scores.

Table 6 shows the human evaluation results for the naturalness of the generated responses in each model. We used
the oracle entrainment degrees as the given conditions. Our
proposed models, based on reinforcement learning, generated
a more acceptable response to the dialogue context than thebaseline models under the oracle condition. C-AHED+RL(k = 2) showed the highest performance. However, CHED+RL (k = 2) did not improve the score compared with
C-HED+RL (k = 1). This indicates that C-HED model,
which has no attention mechanism, can not take enoughadvantage of reward signals that considering the more past
context.
We also evaluated the relationship between human evaluation scores and entrainment scores based on LID7 . Fig. 4
shows the box-plots for human evaluation scores and entrainment scores of three models (no-conditioned models and
C-HEDs, C-AHEDs). Here the horizontal axis indicates thehuman evaluation scores and the vertical axis indicates the
7 The LID was calculated based on (17).

8ModelsHED+CEAHED+CE
C-HED+CE (k = 1)
C-HED+CE (k = 2)
C-HED+RL (k = 1)
C-HED+RL (k = 2)
C-AHED+CE (k = 1)
C-AHED+CE (k = 2)
C-AHED+RL (k = 1)
C-AHED+RL (k = 2)

2
44
33
43
40
49
45
45
44
38
47

1
85
119
88
103
105
106
105
110
123
126

0
111
88
109
97
86
89
90
86
79
67

Weighted-Avg
0.72
0.77
0.73
0.76
0.85
0.82
0.81
0.83
0.83
0.92

Table 7 shows the generation examples of the comparedmodels based on AHED in k = 2. Their naturalness seems atleast comparable; even our proposed models generated more
entrained responses. Note that it is difficult to find using thesame words because our proposed method is based on WMD
that calculates the semantic similarity in semantic spaces.
VIII. CONCLUSION

We proposed a neural conversation model that can control
the entrainment degree of generated responses according
VOLUME x, 2020

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS(a) Performance comparison when k = 1(b) Performance comparison when k = 2

FIGURE 3: Changes in generation performance when given a fixed rtarget(a) Box-plots for entrainment scores and human scores when k = 1(b) Box-plots for entrainment scores and human scores when k = 2

FIGURE 4: Relationships between human scores and entrainment scores

VOLUME x, 20209

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSTABLE 7: Generated responses from models given a dialogue history and a target entrainment degree: A and B denote system
and user. One utterance of each speaker is given as a history in the example. Generation results of each model are shown after
the context. Before generation results, given target entrainment degrees and rewards bestowed on generated responses areindicated by brackets.
A
B: AHED+CE
B: C-AHED+CE
B: C-AHED+RL
B: REFERENCE
A
B
A
B
A
B
A: AHED+CE
A: C-AHED+CE
A: C-AHED+RL
A: REFERENCE
A
B
A
B
A
B
A: AHED+CE
A: C-AHED+CE
A: C-AHED+RL
A: REFERENCE
A
B
A
B
A
B
A: AHED+CE
A: C-AHED+CE
A: C-AHED+RL
A: REFERENCE
A
B
A
B
A
B
A: AHED+CE
A: C-AHED+CE
A: C-AHED+RL
A: REFERENCE

hi there ! how is your day ?
i am doing well . how are you ?
(rtarget = 0.51, r = 0.79) i am doing well . just got back from the gym . how are you ?.
(rtarget = 0.51, r = 0.98) hello , i am good thanks . how are you ?
(rtarget = 0.51, r = 1.0) i am doing great out here
have you thought about finding another job ?
i want to leave the job but my parents make me staymaybe if you found a better one before leaving .
yeah but i’ll move out my parents soon .
very cool ! i loved it when my kids left .
i’ll love it when i leave then i can do as i like
i am sorry to hear that . do you have any pets ?
(rtarget = 0.62, r = 0.72) i am sorry to hear that . do you have any pets ?
(rtarget = 0.62, r = 0.83) i hope so too ! what else do you like to do in your spare time ?
(rtarget = 0.62, r = 1.0) it does not always work out that way
uh , i guess that is cool , so you like food ?
yup love it . i also survived cancer though .
bet that was pretty rough’yes it was hard . i love to run . how about you ?
i spend a lot of time trying to cook , i’m not super good .
that is fine . i also like preparing for marathons .
i love to cook , but i do not have a job , but i do not like it(rtarget = 0.73, r = 0.45 ) i am sorry to hear that . do you have any pets ?
(rtarget = 0.73, r = 0.55) that is awesome , what kind of music do you like ?
(rtarget = 0.73, r = 1.0) is that super hard work ?
i used to paint and would need to stop taking my meds to do a big job .
i understand . i like the city .
i am currently a cleaning lady and do alot of work in the city .
that sounds fun and interesting .
it is . what do you do ?
i work on construction right now .
that sounds like a lot of fun . do you have any hobbies ?
(rtarget = 0.23, r = 0.95) that sounds like a lot of fun . do you have any hobbies ?
(rtarget = 0.23, r = 0.84) that is cool . i am a professional person .
(rtarget = 0.23, r = 1.0) do you have any hobbies ?
that’s great , i am a grad student
do you have any hobbies ? i really love sky diving , rollercoasters and cooking shows !
kayaking and reading psychology books
i’d go kayaking but i’m really not good at swimming so i’d be scared
i love it , i’m an awesome swimmer
do you like hummus ? it is a great alternative as a vegan , i love it !
i do not have any pets , i do not have any pets
(rtarget = 0.28, r = 0.93) i like to go to the beach , do you have any hobbies ?
(rtarget = 0.28, r = 0.95) i like to go to the park and listen to music(rtarget = 0.28, r = 1.0) not really a fan , i am a meats kind of guy

to the given entrainment degree. We applied reinforcement
learning to optimize our model for automatic entrainmentscores that incorporate local interpersonal distance as a reward. Experimental results showed the entrainment scores
correlated with human assessments in both human-humanand human-machine dialogues in a chit-chat domain. Our
proposed model also generated comparable or more natural
responses than conventional models based on the minimization of cross-entropy loss, while the degree of entrainment of
the generated responses is well controlled.
Although our method outperformed the existing method
based on cross-entropy loss, the entrainment degree of generated response can still be improved. This is because ourmethod does not have any mechanism to explicitly accessthe vocabulary used in the dialogue context on its decoding
process. Hierarchical attention [43] or a copying mechanism[44] may explicitly solve this problem based on the word
10

information in dialogue contexts. Incorporating such different aspects of entrainment as dialogue act choice is also
important [8], [27].
A phenomenon where synchronization with each other ina dialogue, such as entrainment, is strongly related to theattributes of the dialogue participants and the relationships
between them [45]. Therefore, we will focus on individuality
or personality to control entrainment [21].
REFERENCES[1] S. E. Brennan and H. H. Clark, “Conceptual pacts and lexical choice inconversation.” Journal of Experimental Psychology: Learning, Memory,
and Cognition, vol. 22, no. 6, p. 1482, 1996.
[2] D. Reitter and J. D. Moore, “Predicting success in dialogue,” 2007.
[3] K. G. Niederhoffer and J. W. Pennebaker, “Linguistic style matching insocial interaction,” Journal of Language and Social Psychology, vol. 21,
no. 4, pp. 337–360, 2002.
[4] M. Natale, “Convergence of mean vocal intensity in dyadic communicaVOLUME x, 2020

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access
Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

tion as a function of social desirability.” Journal of Personality and Social
Psychology, vol. 32, no. 5, p. 790, 1975.
[5] A. Ward and D. Litman, “Measuring convergence and priming in tutorial
dialog,” University of Pittsburgh, 2007.
[6] N. Campbell and S. Scherer, “Comparing measures of synchrony andalignment in dialogue speech timing with respect to turn-taking activity,”in Eleventh Annual Conference of the International Speech Communication Association, 2010.
[7] v. Beňuš, A. Gravano, R. Levitan, S. I. Levitan, L. Willson, and
J. Hirschberg, “Entrainment, dominance and alliance in supreme court
hearings,” Know.-Based Syst., vol. 71, no. 1, pp. 3–14, Nov. 2014.
[Online]. Available: https://doi.org/10.1016/j.knosys.2014.05.020[8] M. Mizukami, K. Yoshino, G. Neubig, D. Traum, and S. Nakamura,“Analyzing the effect of entrainment on dialogue acts,” in Proceedings ofthe 17th Annual Meeting of the Special Interest Group on Discourse andDialogue, Sep. 2016.
[9] A. Nenkova, A. Gravano, and J. Hirschberg, “High frequency word entrainment in spoken dialogue,” in Proceedings of the 46th annual meetingof the association for computational linguistics on human language technologies: Short papers. Association for Computational Linguistics, 2008,
pp. 169–172.
[10] R. Levitan, S. Benus, A. Gravano, and J. Hirschberg, “Entrainment andturn-taking in human-human dialogue,” in 2015 AAAI spring symposium
series, 2015.
[11] M. Nasir, S. N. Chakravarthula, B. R. Baucom, D. C. Atkins,
P. Georgiou, and S. Narayanan, “Modeling Interpersonal Linguistic
Coordination in Conversations Using Word Mover’s Distance,” inProc. Interspeech 2019, 2019, pp. 1423–1427. [Online]. Available:
http://dx.doi.org/10.21437/Interspeech.2019-1900[12] B. Weiss, “Talker quality in interactive scenarios,” in Talker Quality in
Human and Machine Interaction. Springer, 2020, pp. 67–106.
[13] O. Vinyals and Q. Le, “A neural conversational model,” arXiv preprint
arXiv:1506.05869, 2015.
[14] I. V. Serban, A. Sordoni, Y. Bengio, A. C. Courville, and J. Pineau,“Building end-to-end dialogue systems using generative hierarchical neural network models.” in AAAI, 2016, pp. 3776–3784.
[15] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 3-4, pp.
229–256, 1992.
[16] A. Fandrianto and M. Eskenazi, “Prosodic entrainment in an informationdriven dialog system,” in Thirteenth Annual Conference of the International Speech Communication Association, 2012.
[17] R. Levitan, “Entrainment in spoken dialogue systems: Adopting, predicting and influencing user behavior,” in Proceedings of the 2013 NAACLHLT Student Research Workshop, 2013, pp. 84–90.
[18] Z. Tian, R. Yan, L. Mou, Y. Song, Y. Feng, and D. Zhao, “How tomake context more useful? an empirical study on context-aware neuralconversational models,” in Proc. of ACL, vol. 2, 2017, pp. 231–236.
[19] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba, “Sequence level training
with recurrent neural networks,” arXiv preprint arXiv:1511.06732, 2015.
[20] M. Kusner, Y. Sun, N. Kolkin, and K. Weinberger, “From word embeddings to document distances,” in International conference on machinelearning, 2015, pp. 957–966.
[21] J. Li, M. Galley, C. Brockett, G. P. Spithourakis, J. Gao, and W. B.
Dolan, “A persona-based neural conversation model,” CoRR, vol.
abs/1603.06155, 2016.
[22] C. Huang, O. Zaïane, A. Trabelsi, and N. Dziri, “Automatic dialogue
generation with expressed emotions,” in Proceedings of the 2018
Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 2(Short Papers). New Orleans, Louisiana: Association for Computational
Linguistics, Jun. 2018, pp. 49–54. [Online]. Available: https://www.
aclweb.org/anthology/N18-2008[23] Y. Peng, Y. Fang, Z. Xie, and G. Zhou, “Topic-enhanced emotional conversation generation with attention mechanism,” Knowledge-Based Systems,
vol. 163, pp. 429–437, 2019.
[24] G. Zhou, Y. Fang, Y. Peng, and J. Lu, “Neural conversation generation
with auxiliary emotional supervised models,” ACM Transactions on Asianand Low-Resource Language Information Processing (TALLIP), vol. 19,
no. 2, pp. 1–17, 2019.
[25] M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches
to attention-based neural machine translation,” arXiv preprint
arXiv:1508.04025, 2015.
VOLUME x, 2020[26] C. Sankar, S. Subramanian, C. Pal, S. Chandar, and Y. Bengio, “Do neuraldialog systems use the conversation history effectively? an empirical
study,” in Proceedings of the 57th Annual Meeting of the Association for
Computational Linguistics, 2019, pp. 32–37.
[27] S. Kawano, K. Yoshino, and S. Nakamura, “Neural conversation model
controllable by given dialogue act based on adversarial learning and labelaware objective,” in Proceedings of the 12th International Conference on
Natural Language Generation, Oct.–Nov. 2019.
[28] J. Li, W. Monroe, T. Shi, S. Jean, A. Ritter, and D. Jurafsky, “Adversarial
learning for neural dialogue generation,” in Proc. of EMNLP, 2017, pp.
2157–2169.
[29] L. Yu, W. Zhang, J. Wang, and Y. Yu, “Seqgan: Sequence generative
adversarial nets with policy gradient.” in AAAI, 2017, pp. 2852–2858.
[30] C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling,
P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton,“A survey of monte carlo tree search methods,” IEEE Transactions on
Computational Intelligence and AI in games, vol. 4, no. 1, pp. 1–43, 2012.
[31] M. J. Pickering and S. Garrod, “Alignment as the basis for successfulcommunication,” Research on Language and Computation, vol. 4, no. 2-3,
pp. 203–228, 2006.
[32] Y. Rubner, C. Tomasi, and L. J. Guibas, “A metric for distributions withapplications to image databases,” in Sixth International Conference on
Computer Vision (IEEE Cat. No. 98CH36271). IEEE, 1998, pp. 59–66.
[33] O. Pele and M. Werman, “A linear time histogram metric for improved sift
matching,” in European conference on computer vision. Springer, 2008,
pp. 495–508.
[34] O. Pele and M.Werman, “Fast and robust earth mover’s distances,” inProceedings of the 12th International Conference on Computer Vision.
IEEE, 2009, pp. 460–467.
[35] T. Arimoto, H. Sugiyama, M. Mizukami, H. Narimatsu, and R. Higashinaka, “Analysis of conversation topic between first-met speakers overmultiple text-chats,” SIG-SLUD, vol. 5, no. 03, pp. 66–71, 2019.
[36] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares,
H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn
encoder–decoder for statistical machine translation,” in Proc. of EMNLP,
2014, pp. 1724–1734.
[37] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[38] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method forautomatic evaluation of machine translation,” in Proceedings of the 40th
annual meeting on association for computational linguistics. Association
for Computational Linguistics, 2002, pp. 311–318.
[39] Y. Zhang, M. Galley, J. Gao, Z. Gan, X. Li, C. Brockett, and B. Dolan,“Generating informative and diverse conversational responses via adversarial information maximization,” in Advances in Neural Information
Processing Systems, 2018, pp. 1810–1820.
[40] C.-W. Liu, R. Lowe, I. Serban, M. Noseworthy, L. Charlin, and J. Pineau,“How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation,” in Proc. of
EMNLP, 2016, pp. 2122–2132.
[41] C. Xing, W. Wu, Y. Wu, J. Liu, Y. Huang, M. Zhou, and W.-Y. Ma, “Topic
aware neural response generation,” in Thirty-First AAAI Conference on
Artificial Intelligence, 2017.
[42] U. Olsson, F. Drasgow, and N. J. Dorans, “The polyserial correlation
coefficient,” Psychometrika, vol. 47, no. 3, pp. 337–347, 1982.
[43] C. Xing, Y. Wu, W. Wu, Y. Huang, and M. Zhou, “Hierarchical recurrent
attention network for response generation,” in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[44] J. Gu, Z. Lu, H. Li, and V. O. Li, “Incorporating copying mechanism
in sequence-to-sequence learning,” in Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), 2016, pp. 1631–1640.
[45] H. Giles, A. Mulac, J. J. Bradac, and P. Johnson, “Speech accommodation
theory: The first decade and beyond,” Annals of the International Communication Association, vol. 10, no. 1, pp. 13–48, 1987.

11

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.3027099, IEEE Access

SEIYA KAWANO received the B.E. in 2016 fromthe National Institution for Academic Degrees andQuality Enhancement of Higher Education and
the M.E. degree in 2018 from Nara Institute of
Science and Technology (NAIST), Japan. He iscurrently a Ph.D candidate at NAIST. He is also a
research fellow (DC2) of Japan Society for Promotion of Science. He is working in an area of natural
language processing, especially dialogue system.

MASAHIRO MIZUKAMI received his B.A. degree in 2012 from Doshisha University, M.S. degree in engineering in 2014, and Ph.D. degree inengineering in 2017 from Nara Institute of Scienceand Technology (NAIST), respectively. Currently,
he is a researcher of NTT Communication Science Labs. He is working on areas of spoken andnatural language processing, especially on spoken
dialogue systems.

SATOSHI NAKAMURA is Director of Data Science Center and Professor at the Graduate Schoolof Science and Technology, Nara Institute of Science and Technology, Japan, Team Leader ofRiken AIP Tourism Information Analytics Team,and Honorarprofessor of Karlsruhe Institute ofTechnology, Germany. He received his B.S. from
Kyoto Institute of Technology in 1981 and Ph.D.
from Kyoto University in 1992. He was an Associate Professor of the Graduate School of Information Science at Nara Institute of Science and Technology in 19942000. He was Director of ATR Spoken Language Communication ResearchLaboratories in 2000-2008 and Vice president of ATR in 2007-2008. He
was Director-General of Keihanna Research Laboratories and the Executive
Director of Knowledge-Creating Communication Research Center, National
Institute of Information and Communications Technology, Japan in 20092010. He is currently Director of Augmented Human Communication laboratory and a full professor Nara Institute of Science and Technology, Japan.
He is interested in modeling and systems of speech-to-speech translation andspeech recognition. He is one of the leaders of speech-to-speech translation
research and has been serving various speech-to-speech translation research
projects in the world including C-STAR, IWSLT, and A-STAR. He received
the Yamashita Research Award, Kiyasu Award from the Information Processing Society of Japan, Telecom System Award, AAMT Nagao Award,
Docomo Mobile Science Award in 2007, ASJ Award for DistinguishedAchievements in Acoustics. He received the Commendation for Scienceand Technology by the Minister of Education, Science and Technology,
and the Commendation for Science and Technology by the Minister ofInternal Affairs and Communications. He also received the LREC Antonio
Zampolli Award 2012. He has been elected Board Member of International
Speech Communication Association, ISCA, in 2011-2019, IEEE Signal
Processing Magazine Editorial Board Member since April 2012-2015, IEEESPS Speech and Language Technical Committee Member since 2013-2016.
He is ATR Fellow, IPSJ Fellow, ISCA Fellow, and IEEE Fellow.

KOIHIRO YOSHINO received his B.A. degreein 2009 from Keio University, M.S. degree in
informatics in 2011, and Ph.D. degree in informatics in 2014 from Kyoto University, respectively.
From 2014 to 2015, he was a research fellow(PD) of Japan Society for Promotion of Science.
From 2015 to 2016, he was a research assistantprofessor of the Graduate School of Information
Science, Nara Institute of Science and Technology(NAIST). Currently, he is an assistant professor of
NAIST. He is also a researcher of PRESTO, JST, concurrently. He is working
on areas of spoken and natural language processing, especially on spoken
dialogue systems. He received the JSAI SIG-research award in 2013. He isa member of IEEE, ISCA, IPSJ, and ANLP.
12

VOLUME x, 2020