Neural Conversation Model Controllable by Given Dialogue Act Based on
Adversarial Learning and Label-aware Objective
Seiya Kawano, Koichiro Yoshino and Satoshi Nakamura
Devision of Information Science, Graduate School of Science and Technology
Nara Institute of Science and Technology, Nara 630-0192, Japan{kawano.seiya.kj0, koichiro, s-nakamura}@is.naist.jp

Abstract

NCMs by using actual labels; however, these models still had some limitations (Wen et al., 2015; Liet al., 2016; Sun et al., 2017; Zhao et al., 2017;
Huang et al., 2018; Zhou et al., 2018). One crucial
issue was that they do not have any explicit training objectives to guarantee that a generation has adiscriminability for a given condition.
We extend a framework of the generative adversarial network for sequential generation (Yu et al.,2017) for improving the controllability of NCMs
under the constraint of a given dialogue act condition. We propose an adversarial learning framework that alternatively trains between conditioned
generator and a conditioned discriminator. The
discriminator has a multi-class objective that explicitly classifies a generated response into an appropriate dialogue act class. This improves thediscriminability of generation.
In this paper, we first describe the task of conditional response generation given a dialogue act
label and its existing approaches (Section 3). Second, we introduce an adversarial learning framework and extend its architecture and objective to
fit the problem of conditional generation (Section 4). In experiments, we use metrics to evaluate the controllability and naturalness of responses(Section 5). The experimental results show thatour proposed model achieved the best controllability score in both automatic and human subjective
evaluations even if it achieves better or comparable naturalness to existing methods (Section 6).

Building a controllable neural conversation
model (NCM) is an important task. In thispaper, we focus on controlling the responsesof NCMs by using dialogue act labels of responses as conditions. We introduce an adversarial learning framework for the task ofgenerating conditional responses with a new
objective to a discriminator, which explicitly
distinguishes sentences by using labels. Thischange strongly encourages the generation oflabel-conditioned sentences. We compared the
proposed method with some existing methods for generating conditional responses. The
experimental results show that our proposed
method has higher controllability for dialogue
acts even though it has higher or comparable
naturalness to existing methods.

1

Introduction

A dialogue act is defined as the intention or thefunction of an utterance in dialogues. Dialogue act
labels are defined as unique classes to distinguishbetween kinds of dialogue acts (Boyer et al., 2010;
Bunt et al., 2012). Some existing studies have exploited the dialogue act as a component in modeling the dialogue strategy of dialogue systems (Meguro et al., 2010; Yoshino and Kawahara, 2015;
Shibata et al., 2016; Keizer and Rieser, 2017).
Neural conversation models (NCMs), whichlearn a direct mapping between a dialogue history
and a response utterance, are widely researched asa scalable approach to building non-task oriented
dialogue systems (Vinyals and Le, 2015; Serban
et al., 2016). However, it is difficult to control their
responses on the basis of actual constraints such as
dialogue act classes. Some existing studies havetackled this problem to control responses from

2 Related Work
Dialogue systems that have dialogue management
modules determine a dialogue act or dialogue state
198

Proceedings of The 12th International Conference on Natural Language Generation, pages 198–207,
Tokyo, Japan, 28 Oct - 1 Nov, 2019. c 2019 Association for Computational Linguistics

of a system response by using statistical methods such as reinforcement learning (Young et al.,2010; Meguro et al., 2010; Yoshino and Kawahara,
2015; Keizer and Rieser, 2017). Response generation modules generate responses according tothese dialogue acts or dialogue states on the basis of the rules, templates, agendas or other statistical models (Oh and Rudnicky, 2000; Xu andRudnicky, 2000). Recently, neural network based
generation modules have been widely used.

Figure 1: Task of response generation conditioned bydialogue act labels.

framework of the generative adversarial network(Yu et al., 2017; Li et al., 2017a; Tuan and Lee,2019). This framework makes it possible to consider the total quality of generated sequences unlike SCE-loss, which is optimized for each token.
We extend adversarial networks to generate qualified and controlled sentences given a condition,
especially dialogue act labels.

Wen et al. (2015) proposed a conditional
language model (Semantically Conditioned
Long Short-Term Memory; SC-LSTM) for
task-oriented systems, which generates utteranceson the basis of any dialogue acts and frames inthe domain of restaurant navigation dialogue by
using gating mechanism. However, the training
framework of SC-LSTM requires state framesthat express the function and the contents of target
utterances entirely. Thus, it is not realistic to apply
this method to building an open-domain dialoguesystem. Zhao et al. (2017) proposed an NCM
based on a variation of the conditional variational
autoencoder (CVAE), which generates responses
that have high diversity in discourse level byusing latent variables as dialogue acts. However,this model has no mechanism to guarantee forgenerating discriminable responses for given
dialogue acts.

3 Response Generation Conditioned by
Dialogue Act Label
3.1 Task Settings
The task we focus on is building a controllable NCM with a given condition, typically dialogue act labels. The problem is defined as generating the ith response word sequence Ri ={w1 , w2 , · · · , wT } given a dialogue history M ={Mi−1 , Mi−2 , . . . , Mi−n } and dialogue act label
di . Here, n is the length of a dialogue, and T is
the number of words in an utterance. As shown in
Figure 1, a response Ri is required to satisfy not
only the behavioral characteristic of a given dialogue act but also the appropriateness in the dialogue context (=history).
One of the simplest approaches to building sucha conditional generation system given a class label
in NCM is adding the class label to the input ofa decoder (Li et al., 2016; Zhao et al., 2017; Sunet al., 2017). We describe this baseline in the following section.

There is another research trend in controlling
NCMs with a given condition, such as speaker or
emotion labels (Li et al., 2016; Sun et al., 2017;
Huang et al., 2018; Zhou et al., 2018). These
NCMs are optimized by softmax cross-entropy
loss (SCE-loss), which calculates a loss wordby-word. However, such existing training objectives do not necessarily guarantee that a generated response has high discriminability to for agiven class label. In other words, SCE-loss is not
an appropriate objective that explicitly evaluates
whether a generated response reflects the propertyof the given class label or not. Therefore, the generated response will be biased by majority classlabels.

3.2 Conditional NCM with Dialogue Acts
We introduce a general conditional NCM that isconditioned by dialogue act labels as a baseline.
We built an NCM on the basis of a hierarchicalencoder-decoder model that explicitly gives labels
to the decoder at any of the steps of decoding as

To prevent these problems, we introduce the199

Here, |V| indicates the vocabulary size, x ∈ R|V|
indicates the output of the projection layer in the
decoding steps, and xk ∈ R|V| indicates the kthelement of x. xc is the target word. SCE-loss optimizes the prediction of words at each decoding
step. However, it does not use the information of agiven dialogue act label in the loss calculation during training. Thus, the resultant model often generates a response that does not consider a given dialogue act label or a biased response by using the
majority dialogue act labels in the training data.
We tackle this problem by introducing an explicit
training objective to generate a conditioned word
sequence in adversarial learning.

Figure 2: Conditional-NCM with dialogue acts.

shown in Figure 2.
Recurrent neural networks (RNNs) such as longshort-term memory (LSTM) are generally used
to model a sequential generation of responses
in NCMs (Hochreiter and Schmidhuber, 1997;
Vinyals and Le, 2015; Serban et al., 2016). The
encoder receives a word at each time step by using
forward RNNs to encode an utterance into a fixed
length vector (utterance encoder). Utterance vectors of a dialogue are input to another fixed length
vector according to their time sequence to encode
the dialogue context (dialogue encoder). The resultant vector is fed into the decoder to generate a
response sentence (word sequence). We used thesame encoder architecture as Tian et al. (2017).
In the decoding steps of the NCMs, the decoder
receives a previous hidden vector ht−1 , memory
cell ct−1 , and generated word wt−1 to generate aword wt . Here, t is an actual time-step of generation. The model has been changed to receive not
only the previous word wt−1 but also the dialogue
act label d in conditional generation. The vector
representations of d and wt are concatenated andused as the input of the decoder at time-step t.1
The decoder itself also predicts the vector representation of words. This architecture is also thesame as those of the models proposed by Zhao
et al. (2017).
Softmax cross-entropy loss (SCE-loss) is
widely used to train the model.
exp(xc )losssce = − log ∑|V|.
k exp(xk )4 Conditional Response Generation
Based on Adversarial Learning
We introduce sequential generative adversarial
networks (SeqGANs) (Yu et al., 2017; Li et al.,2017a; Tuan and Lee, 2019) to improve the controllability and quality of conditional response
generation. SeqGAN is a prospective approachto preventing the problems caused by SCE-loss
based training because it can evaluate not only theword prediction of each decoding step but also thewhole quality of a generated sequence. In thissection, we first describe the architecture of SeqGAN (Section 4.1) and then propose our extension of SeqGAN to realize conditional response
generation by using given dialogue act labels (Section 4.2).
4.1 SeqGAN for Response Generation
The generation process in SeqGAN is formalized
as a Markov decision process (MDP) and optimized with reinforcement learning (RL) (Li et al.,2017a; Tuan and Lee, 2019). The problem of response generation in NCMs is generating response
word sequence R = {w1 , · · · , wT } given a dialogue context M . Such a word selection process
in the generation is defined as an action sequence,
which is generated by an actual policy in MDP. In
SeqGAN, the generator generates a sentence according to the current policy. The discriminatorgives an evaluation score to the generated sentence
after the generation. The evaluation score is fed as(1)

1
wt and d are converted into vector representation andthen concatenated.

200

a reward to update the policy of the generator in
RL. We use a policy gradient (Williams, 1992) totrain the policy. The objective function and its gradient of the policy gradient are defined as follows2 .
J(θ) =∑

under specified dialogue acts, and a discriminator
network to distinguish between generation (fake)and training data (real) that reflect their conditions.
As the generator network, we applied theconditional-NCM described in Section 3.2. Equations (2) - (4) are changed as follows.

GGθ (wt |w1:t−1 , M )・QDθϕ ((w1:t−1 , M ), wt )

w1:T

(2)
J(θ) =T∑∑ G
1∇J(θ) ≃Q θ ((w1:t−1 , M ), wt )T t=1 w ∈V Dϕ∑GGθ (wt |w1:t−1 , M, d) · QDθϕ ((w1:t−1 , M, d), wt )

w1:T

(6)

t

· ∇θ Gθ (wt |w1:t−1 , M )
=

T
1 ∑

T t=1

(3)

∇J(θ) ≃

GEwt ∼Gθ [QDθϕ ((w1:t−1 , M ), wt )· ∇θ log p(wt |w1:t−1 , M )]G

QDθϕ ((w1:t−1 , M, d), wt )

t

(4)
=

(7)

T
∑

1
G
Ewt ∼Gθ [QDθϕ ((w1:t−1 , M, d), wt )T t=1
· ∇θ log p(wt |w1:t−1 , M, d)](8)

As the discriminator network, we incorporated
dialogue act labels in the classification model (Figure 3). In the discriminator model, the utterance encoder converts dialogue contexts into fixed
length vectors and uses them as features of discrimination. We propose to use two discriminatorsfor incorporating dialogue act label informationin the discriminator: implicit and explicit. Eachmethod is described below in respective sections.
4.2.1 Binary Objective;
Implicit-Discriminator
We built a simple extension for the discriminatorthat incorporates dialogue acts in the feature vectors of the discriminator. We call this architecture“implicit.” This discriminator is defined as,
min −ER∼pdata (·|M,d) [log Dϕ (R, M, d)]min −ER∼pdata (·|M ) [log Dϕ (R, M )]ϕϕ

−ER∼Gθ (·|M,d) [log(1 − Dϕ (R, M, d))].

(5)

(9)

We expect that the implicit discriminator canuse the information of dialogue acts as a feature and discriminate generated results as fakes ifthey do not follow a given dialogue act (Figure 3,
lower-right). There are some works that have
similar approaches in emotional response generation (Sun et al., 2018; Kong et al., 2019). However, this discriminator is still a simple extensionof the standard discriminator, which classifies response in two classes. In other words, the objective is not changed; thus it probably has difficultyin distinguishing the class (dialogue act label) of

The generator and the discriminator are trainedalternatively to train their network adversarially.
4.2 SeqGAN for Conditional Response
Generation with Dialogue Acts
The generator and the discriminator in SeqGAN
are extended to produce responses according togiven dialogue acts. The adversarial framework is
extended for jointly optimizing both networks: agenerator network to produce response utterances2

T t=1 w ∈V· ∇θ Gθ (wt |w1:t−1 , M, d)Here, θ is a parameter of the policy. w1:t−1 indicates a word sequence, V is a vocabulary, andp is the generative probability of word wt ∈ V.
θQG
Dϕ ((w1:t−1 , M ), wt ) is an action-value functionthat gives an expected future reward when thesystem takes the action of generating word wt
given the state: already generated word sequence
w1:t−1 and dialogue context M . ϕ is a parameter of the discriminator. The discriminator only
outputs the reward after the whole generation of
the sequence. Thus, the value of the action-valueθfunction QG
Dϕ ((w1:t−1 , M ), wt ) for each step iscalculated by using a Monte Carlo tree search(MCTS) under the current policy and its parameter θ (Browne et al., 2012).
The discriminator is trained to classify a generated sentence (fake) and sentence in training data
(real). Its training objective is defined as,−ER∼Gθ (·|M ) [log(1 − Dϕ (R, M ))].

T
1 ∑ ∑

Detailed derivation is shown in (Yu et al., 2017).

201

responses. We propose another discriminator tosolve this problem in the next section.

GANs (Tran et al., 2017). However, the training
speed of LSTM is much slower than other typesof networks, although LSTM has dominant performance (Lei et al., 2017). This characteristic is
critical for adversarial learning, which requires alarge number of iterations.
We used the policy gradient in this research to
update the parameters of the generator, which isbased on expected reward calculation by MCTS.
However, MCTS requires enormous calculation
costs because it requires scanning the discriminator and generator r × w times per one update ofgenerator, where r is the number of rollouts and w
is the number of words in a response each time
step. Thus, we propose to use a simple recurrent unit (SRU) (Lei et al., 2018) in our generator and discriminator. SRU is known as an extension of RNN, which has comparable performanceto LSTM even if it works at significantly higher
speed. SRU is defined as follows.

Figure 3: Implicit & Explicit-Discriminator.

4.2.2 Multi-class Objective;
Explicit-Discriminator
We propose an approach extending the classification problem of the discriminator from the binaryclassification of fake/real to multi-class classification to distinguish target dialogue act classes (Figure 3 upper-right). This discriminator has a multiclass objective for N +1 class classification. Here,N is the number of unique dialogue act classes;
another one is a fake class for categorizing the
responses as generated. We call this architecture“explicit.” Its objective function is defined as,
min −
ϕ

N
∑

ṽ t = W v t
f t = σ(W f v t + bf )r t = σ(W r v t + br )ct = f t ⊙ ct−1 + (1 − f t ) ⊙ ṽ t
ht = r t ⊙ g(ct ) + (1 − r t ) ⊙ v t

Here, vt is the input vector at time-step t, ft is
the forgetting gate, rt is the input gate, ct is thememory cell, and ht is the hidden vector. The key
idea of SRU is minimizing the number of vectors
and gates affected by previous states. Under thisdefinition, only ct is affected by the previous statect−1 . Furthermore, ct and ht are calculated onlyby the element-wise production and summation ofa vector for easy speed up. It was reported that forward and backward propagations in SRU are 1016 times faster than LSTM (Lei et al., 2018). SRU
leads to computational advantage compared withanother type of RNNs including GRU as well. We
expect to have a significant improvement in speeding up adversarial learning by using SRU insteadof LSTM. However, applying SRU to NCM has
no track record; thus, we introduce SRU to boththe existing methods and our proposed adversarial
network; however, we also perform a comparison
with another baseline implemented by LSTM.

ER∼pdata (·|M,d) [log Dϕ (di |R, M )]i=1

−ER∼Gθ (·|M,d) [log Dϕ (df ake |R, M )].

(11)
(12)
(13)
(14)
(15)

(10)

We used the posterior probability Dϕ (d|R, M ) estimated by the discriminator as the reward of thegenerator. We expect that this discriminator willencourage training the generator to generate discriminative sentences with dialogue acts because
generations that follow different dialogue act manners are penalized even if they are natural. Odena(2016) proposed a similar idea to use multi-classobjective in GAN for the image generation task.
4.3 Speeding Up Adversarial Learning Using
Simple Recurrent Unit
Using LSTM or GRU as an encoder and decoder isa general method for building NCMs (Vinyals andLe, 2015; Serban et al., 2016). LSTM is also often used for classification problems to encode a hierarchical structure, such as the discriminators of202

5Experimental Settings

applied alternatively to train them. In G-steps, agenerated response RˆG is sampled by using a dialogue history MG and dialogue act dG , and thenthe reward rRˆG for the generation is calculated by
the discriminator D. By using the calculated reward rRˆG , parameters of G are updated. In Dsteps, a real response RD , given a dialogue history
MD and a dialogue act label dD , is sampled fromthe training data. A fake response RˆD is generated
from the generator G by using the dialogue history
MD and the dialogue act label dD . Parameters ofthe discriminator D is updated by using the real
sample and the fake sample.

5.1 Dataset
We used the DailyDialog corpus that covers ten
categories from a wide variety of topics (Li et al.,2017b). The corpus contains 13,118 dialogues,
with a total of 102,979 utterances annotated with
dialogue act labels: inform (46,532 utterances),questions (29,428 utterances), directives (17,295
utterances) and commissive (9,724 utterances).
We divided the corpus into training/validation/test
sets with 11,118/1,000/1,000 dialogues according
to the work of Li et al. (2017b). In all experiments,
the vocabulary size was set to 25,000, and all the
OOV words were replaced by “UNK” symbol.

We set the number of G-steps to 4 and D-steps
to 20. In the generator, we used two-layers SRUs
in both the encoder and the decoder as 1024 hidden units. We used the Adam optimizer with alearning rate of 1e-5. For the discriminator, weused a one-layer SRU, 1024 hidden units, and the
SGD optimizer with a learning rate of 1e-3. We
set the number of rollout to 5 in MCTS.

5.2 Training Settings
We used the same setting for embedding: words,
256, dialogue acts, 100. The mini-batch size was
32. In the training for conditional-NCM, we set
two-layers RNNs in both the encoder and the decoder and used the Adam optimizer with a learning rate of 1e-5.
In the proposed adversarial learning, we followed the training procedure proposed by Li et al.
(2017a). The training algorithm that we used isshown as follows.

5.3 Automatic Evaluation Metrics
We automatically evaluated generation results bycomparing with references in the test-set. As theautomatic evaluation, we used three different typesof metrics: perplexity, relevance scores, and controllability.

Algorithm 1 Training procedure
1: for number of iterations do
2:

G′ ← G

3:

for number of G-steps do

4:

sample (MG , RG , dG ) from training data

5:

generate response RˆG by using G′ on (MG , dG )6:compute reward rRG for (MG , RˆG , dG ) by using D

7:

update G on (MG , RˆG , dG ) using rRG

8:

Perplexity is a metric for evaluating a language
model performance. Likelihoods of models forreference responses are calculated as perplexities.
Note that the perplexity score does not directly reflect the quality of generation; dull responses also
have good perplexity scores.
Relevance scores are similarities between references and generated results. We used NIST, a variation of BLEU, which focuses on content words
more than BLEU (Doddington, 2002). However, using count-based metrics such as BLEU
and NIST are not appropriate, because they havesmall correlations with human judgment score in
response generation tasks (Liu et al., 2016). Thus,
we also used three different relevance scores proposed by Liu et al. (2016): embedding average (“Average”), greedy matching (“Greedy”) andfor number of D-steps do

9:sample (MD , RD , dD ) from training data

10:

generate response RˆD by using G on (MD , dD )11:

update D using (MD , RˆD , dD ) and (MD , RD , dD )In the training of SeqGAN for conditional response generation based on dialogue acts, we prepare well pre-trained conditional generator and adiscriminator in advance. After initializing parameters by pre-trained models, G-steps for the generator G and D-steps for the discriminator D are203

vector extrema (“Extrema”)3 . “Average” calculates a cosine similarity between the reference sentence vector and the generated sentence vector.
Each sentence vector is calculated by an averageof word embedding vectors in the sentence. “Extrema” also calculates a cosine similarity between
sentence vectors; however, the sentence vector isconstructed in a different way. Each dimensionof the sentence vector is selected from the same
dimension of a word embedding vector, which
has the highest absolute value in the sentence.
“Greedy” calculates cosine similarities of wordpairs in the reference and the generated sentence,
which is paired by alignment, and then averages
these similarities.

• 1: The response can be used as a reply, but it
is too generic like I don’t know.
• 0: The response cannot be used as a reply to
the given dialogue history. It is either semantically irrelevant or disfluent.
Each sample was evaluated by three annotators,
and the final score was decided by majority voting.
If the evaluation was completely separated (0, 1
and 2), the example was evaluated as 1.
In the controllability evaluation, we requested
one annotator to annotate dialogue acts for generated responses, who had two years experience indialogue act annotation. The annotator was trainedby using training data of DailyDialog corpus before the evaluation.

The last automatic metric we used is “controllability”, which is given by the classification resultof the pre-trained dialogue act classifier by using
the training set. We connected our encoder forconditional-NCM (Figure 2 left side) to a multiclass softmax layer to build the classifier.4 Any
generated sentences are labeled by the classifier
and then compared with the given condition label
to calculate the label accuracy.

6 Experimental Results
6.1 Results of Automatic Evaluation
Table 1 shows the results of the automatic
objective evaluation. We compared our proposed SeqGAN based on explicit-discriminator(Adversarial-Explicit) with the following baselines. “Vanilla-NCM” shows performances ofvanilla LSTM, which has no mechanism to receive
condition labels. These scores indicate a general performance of systems in DailyDialog corpus. “Conditional-NCMs” show performances ofNCMs that receives condition labels on its decoder
as proposed by Zhao et al. (2017). We compared
two variations of “Conditional-NCMs”, LSTM
and SRU, to check the performance of SRU compared with LSTM. “Adversarial-Implicit” shows
performances of SeqGAN that has implicit discriminator, which is proposed by Sun et al. (2018)
and Kong et al. (2019). “Adversarial-Explicit” indicates the proposed model that has a multi-classdiscriminator on its SeqGAN. The table shows
both results of beam search (width=5) and random
sampling in the decoding process.

5.4 Human Subjective Evaluation Metrics
The automatic evaluation scores still have a problem in that they do not have high correlations withhuman subjective evaluation results (Liu et al.,2016). Thus, we also evaluated systems with ahuman subjective evaluation to confirm the naturalness and controllability of responses.
In the evaluation of naturalness, we used a 3point scale score in accordance with the existing
work (Li et al., 2019). Thirty generated responses
were randomly selected from each dialog act (120in total) and human annotators selected an evaluation for the sample in following instructions.
• 2: The response can be used as a reply and
it is informative and interesting; the response
is natural and can make the conversation continue.

6.1.1 Speeding up Using SRU
The comparison between “Conditional-NCM w/
LSTM” and “Conditional-NCM w/ SRU” indicates that the speeding up using SRU works well;
SRU achieves higher relevance scores to LSTM.
SRU used 53,539K parameters, whereas LSTM

3We used fastText embeddings trained by wikipediadump data. The size of vectors was 300.
4
The accuracy of the classifier in the test-set was 0.8303.

204

DecodingBeam search

Sampling

ModelsVanilla-NCM w/ LSTM
Conditional-NCM w/ LSTM
Conditional-NCM w/ SRU
Adversarial-Implicit
Adversarial-Explicit
Vanilla-NCM w/ LSTM
Conditional-NCM w/ LSTM
Conditional-NCM w/ SRU
Adversarial-Implicit
Adversarial-Explicit

Perplexity
33.9076
33.273936.661439.2864
39.6993
33.9076
33.273936.661439.2864
39.6993

NIST
0.03860.0502
0.06100.0596
0.06130.02650.03420.03650.0401**
0.0411**

Relevance
Average Greedy
0.76030.57630.7804
0.59800.78500.59810.77730.6041**
0.78240.6034**0.7746
0.5326
0.78670.55400.79170.55900.78530.5641**0.79070.5687**Extreme0.50790.53190.53800.5417**0.5401*0.46620.48490.49050.5018**0.5036**

Controllability
0.86510.87020.86150.8946**0.80150.81760.80810.8531**

Table 1: Results in automatic evaluation: note that results of random sampling are an average of five decoding
runs, each decoding is initialized with random seed. *: p < 0.05 and **: p < 0.001 indicate results of significancetests (vs. conditional-NCM w/ SRU).

required 82,924K parameters. These results support our experiments using SRU instead of LSTM.
We will mainly focus on comparisons of SRU
models in the following sections.

Dialogue acts
Inform (46%)
Questions (30%)
Directives (16%)
Commissive (8%)
Weighted avg

6.1.2 Qualities of Generated Responses

Table 2: Result for beam search decoding, controllability of each dialogue act (adversarial-explicit). The
score in round brackets indicates the improvement fromconditional-NCM w/ SRU.

The “Vanilla-NCM w/ LSTM” scores show thedifficulty of conversation modeling in DailyDialog corpus.
By comparing these scoreswith scores of other conditional generation models, “Conditional-NCM” and “Adversarial”, conditional generation models improved relevance
scores even if they have controllability. This isprobably because the dialogue act condition canbe a training constrain to prevent dull responses.
Generation methods using adversarial learning improved relevance scores than “Conditional-NCM
w/ SRU”.

Dialogue acts
Inform (46%)
Questions (30%)
Directives (16%)
Commissive (8%)
Weighted avg

Prec.
0.8881
0.96930.93720.62850.8980Prec.
0.86840.92430.76280.63380.8477

Recall
0.9291
0.99820.71640.69210.8946

Recall
0.8973
0.97380.6652
0.55790.8531F1 (improv.)
0.9081 (+0.0167)
0.9836 (+0.0060)
0.8121 (+0.0971)
0.6588 (+0.0240)
0.8935 (+0.0192)

F1 (improv.)
0.8826 (+0.0164)
0.9484 (+0.0281)
0.7107 (+0.1152)
0.5934 (+0.0382)
0.8494 (+0.0257)

Table 3: Result for random sampling, controllability of each dialogue act (adversarial-explicit). The
score in round brackets indicates the improvement fromconditional-NCM w/ SRU.

tags. Our model generated more discriminative
sentences even if these classes have similar attribute.

6.1.3 Controllability of each Dialogue Act
By comparing “Controllability” scores, the proposed “Adversarial-Explicit” achieved best scoresin both beam-search and sampling decoding. For adetailed analysis, we show a controllability scoreof “Adversarial-Explicit” for each dialogue act label in Table 2 and Table 3 (beam-search and sampling). Tables show precision, recall and theirharmonic mean (F1) for each dialogue act, andthe improvement from the score of “ConditionalNCM w/ SRU”, which achieved the best controllability in baselines (improv.). The proposed“Adversarial-Explicit” achieved improvements forany classes, but in particular, it achieved large
improvements on “Directives” and “Commissive”

6.2 Results of Human-Subjective Evaluation
Table 4 and Table 5 show human evaluation results for naturalness and controllability, respectively. We used beam search (beam width of 5)
for generating examples to be evaluated. Regarding the naturalness of responses (Table 4), models used adversarial learning produced a more acceptable response to the dialogue context. Regarding the controllability of response generation (Table 5), the adversarial-explicit model achieved thebest performance among the compared models.
In summary, the proposed model based on205Models
Conditional-NCM w/ SRU
Adversarial-Implicit
Adversarial-Explicit

+2
0.06
0.04
0.06

+1
0.43
0.53
0.51

+0
0.51
0.43
0.42

Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp
Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE
Transactions on Computational Intelligence and AI
in games, 4(1):1–43.

Table 4: Response quality of each model. This tableshow the distribution of scores.
ModelsConditional-NCM w/ SRU
Adversarial-Implicit
Adversarial-Explicit

Accuracy
0.74320.70580.7971

Harry Bunt, Jan Alexandersson, Jae-Woong Choe,
Alex Chengyu Fang, Koiti Hasida, Volha Petukhova,
Andrei Popescu-Belis, and David R Traum. 2012.
Iso 24617-2: A semantically-based standard for dialogue annotation. In Proc. LREC, pages 430–437.

Weighted-F1
0.75830.68100.7868

Table 5: Controllability of response generation. Table shows filtered results of contradicted responses.
Weighted-F1 is a weighted average of F1 score of eachdialogue act.

George Doddington. 2002.
Automatic evaluationof machine translation quality using n-gram cooccurrence statistics. In Proceedings of the second
international conference on Human Language Technology Research, pages 138–145. Morgan Kaufmann Publishers Inc.

adversarial learning with multi-class objective
achieved the best controllability, the main focusof this paper, even if it realized a comparable naturalness to existing methods.

7

Sepp Hochreiter and Jürgen Schmidhuber. 1997.
Long short-term memory. Neural computation,
9(8):1735–1780.
Chenyang Huang, Osmar Zaiane, Amine Trabelsi, andNouha Dziri. 2018. Automatic dialogue generation
with expressed emotions. In Proc. of NAACL-HLT,
volume 2, pages 49–54.

Conclusion

In this paper, we introduced an extended framework of the generative adversarial network that is
optimized by both conditioned generation and discrimination of dialogue act classes. Experimental
results showed that our conditional response generation model improved both the response quality
and controllability in neural conversation generation. In future works, we will examine the possibility of incorporating our adversarial frameworkin various generation approaches (Serban et al.,2017; Shen et al., 2017; Zhou and Wang, 2018)
to build a more generalized conditional response
generation model. We will also focus on different
types of labels to be used as conditions.

Simon Keizer and Verena Rieser. 2017. Towards learning transferable conversational skills using multidimensional dialogue modelling. In Proc. of SEMDIAL, pages 158–159.
Xiang Kong, Bohan Li, Graham Neubig, Eduard Hovy,and Yiming Yang. 2019. An adversarial approach tohigh-quality, sentiment-controlled neural dialogue
generation. In AAAI 2019 Workshop on Reasoning and Learning for Human-Machine Dialogues(DEEP-DIAL 2019), Honolulu, Hawaii.
Tao Lei, Yu Zhang, and Yoav Artzi. 2017. Training rnns as fast as cnns.
arXiv preprint
arXiv:1709.02755.
Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav
Artzi. 2018. Simple recurrent units for highly parallelizable recurrence. In Proc. of EMNLP, pages
4470–4481.

Acknowledgements

Jiwei Li, Michel Galley, Chris Brockett, Georgios P.
Spithourakis, Jianfeng Gao, and William B. Dolan.
2016. A persona-based neural conversation model.
CoRR, abs/1603.06155.

This research and development work was supported by the JST PRESTO (JPMJPR165B) andJST CREST (JPMJCR1513).

Jiwei Li, Will Monroe, Tianlin Shi, Sėbastien Jean,
Alan Ritter, and Dan Jurafsky. 2017a. Adversarial
learning for neural dialogue generation. In Proc. of
EMNLP, pages 2157–2169.

References
Kristy Elizabeth Boyer, Eun Young Ha, Robert
Phillips, Michael D Wallis, Mladen A Vouk, andJames C Lester. 2010. Dialogue act modeling in
a complex task-oriented domain. In Proc. of SIGDIAL, pages 297–305.

Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang
Cao, and Shuzi Niu. 2017b. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proc. of
IJCNLP, volume 1, pages 986–995.

206

Ziming Li, Julia Kiseleva, and Maarten de Rijke. 2019.
Dialogue generation: From imitation learning to inverse reinforcement learning. In Proc. of AAAI.

Quan Hung Tran, Ingrid Zukerman, and Gholamreza
Haffari. 2017. A hierarchical neural model for learning sequences of dialogue acts. In Proc. of ACL,
volume 1, pages 428–437.

Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016.
How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for
dialogue response generation. In Proc. of EMNLP,
pages 2122–2132.

Yi-Lin Tuan and Hung-Yi Lee. 2019. Improving conditional sequence generative adversarial networks by
stepwise evaluation. IEEE/ACM Transactions on
Audio, Speech, and Language Processing.
Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. CoRR, abs/1506.05869.

Toyomi Meguro, Ryuichiro Higashinaka, Yasuhiro
Minami, and Kohji Dohsaka. 2010. Controlling
listening-oriented dialogue using partially observable markov decision processes. In Proc. of ACL,
pages 761–769.

Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, PeiHao Su, David Vandyke, and Steve Young. 2015.
Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In
Proc. of EMNLP, pages 1711–1721.

Augustus Odena. 2016. Semi-supervised learning withgenerative adversarial networks. arXiv preprint
arXiv:1606.01583.

Ronald J Williams. 1992. Simple statistical gradientfollowing algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256.

Alice H Oh and Alexander I Rudnicky. 2000. Stochastic language generation for spoken dialogue systems. In ANLP-NAACL 2000 Workshop: Conversational Systems.

Wei Xu and Alexander I Rudnicky. 2000. Task-based
dialog management using an agenda. In ANLPNAACL 2000 Workshop: Conversational Systems.

Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016.
Building end-to-end dialogue systems using generative hierarchical neural network models. In Proc.
of AAAI, pages 3776–3784.

Koichiro Yoshino and Tatsuya Kawahara. 2015. Conversational system for information navigation basedon pomdp with user focus tracking. Computer
Speech & Language, 34(1):275–291.

Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe,
Laurent Charlin, Joelle Pineau, Aaron Courville, andYoshua Bengio. 2017. A hierarchical latent variable
encoder-decoder model for generating dialogues. In
Thirty-First AAAI Conference on Artificial Intelligence.

Steve Young, Milica Gašić, Simon Keizer, François
Mairesse, Jost Schatzmann, Blaise Thomson, andKai Yu. 2010. The hidden information state model:
A practical framework for pomdp-based spoken dialogue management. Computer Speech & Language,24(2):150–174.

Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi
Niu, Yang Zhao, Akiko Aizawa, and Guoping Long.
2017. A conditional variational framework for dialog generation. In Proc. of ACL, volume 2, pages
504–509.

Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu.
2017. Seqgan: Sequence generative adversarial nets
with policy gradient. In Proc. of AAAI, pages 2852–
2858.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi.
2017. Learning discourse-level diversity for neuraldialog models using conditional variational autoencoders. In Proc. of ACL, volume 1, pages 654–664.

Tomohide Shibata, Yusuke Egashira, and Sadao Kurohashi. 2016. Chat-like conversational system basedon selection of reply generating module with reinforcement learning. In Situated Dialog in SpeechBased Human-Computer Interaction, pages 63–69.
Springer.

Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan
Zhu, and Bing Liu. 2018. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Proc. of AAAI.

Xiao Sun, Xinmiao Chen, Zhengmeng Pei, and Fuji
Ren. 2018. Emotional human machine conversation
generation based on seqgan. In Proc. of ACII Asia,
pages 1–6. IEEE.

Xianda Zhou and William Yang Wang. 2018. Mojitalk:Generating emotional responses at scale. In Proc.
ACL, volume 1, pages 1128–1137.

Xiao Sun, Xiaoqi Peng, and Shuai Ding. 2017.
Emotional human-machine conversation generation
based on long short-term memory. Cognitive Computation, 10:389–397.
Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, and Dongyan Zhao. 2017. How to makecontext more useful? an empirical study on contextaware neural conversational models. In Proc. of
ACL, volume 2, pages 231–236.

207