DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder

1 INTRODUCTION . Neural response generation has been a long interest of natural language research . Most of the recent approaches to data-driven conversation modeling primarily build upon sequence-to-sequence learning ( Cho et al. , 2014 ; Sutskever et al. , 2014 ) . Previous research has demonstrated that sequenceto-sequence conversation models often suffer from the safe response problem and fail to generate meaningful , diverse on-topic responses ( Li et al. , 2015 ; Sato et al. , 2017 ) . Conditional variational autoencoders ( CVAE ) have shown promising results in addressing the safe response issue ( Zhao et al. , 2017 ; Shen et al. , 2018 ) . CVAE generates the response conditioned on a latent variable - representing topics , tones and situations of the response - and approximate the posterior distribution over latent variables using a neural network . The latent variable captures variabilities in the dialogue and thus generates more diverse responses . However , previous studies have shown that VAE models tend to suffer from the posterior collapse problem , where the decoder learns to ignore the latent variable and degrades to a vanilla RNN ( Shen et al. , 2018 ; Park et al. , 2018 ; Bowman et al. , 2015 ) . Furthermore , they match the approximate posterior distribution over the latent variables to a simple prior such as standard normal distribution , thereby restricting the generated responses to a relatively simple ( e.g. , unimodal ) scope ( Goyal et al. , 2017 ) . A number of studies have sought GAN-based approaches ( Goodfellow et al. , 2014 ; Li et al. , 2017a ; Xu et al. , 2017 ) which directly model the distribution of the responses . However , adversarial training over discrete tokens has been known to be difficult due to the non-differentiability . Li et al . ( 2017a ) proposed a hybrid model of GAN and reinforcement learning ( RL ) where the score predicted by a discriminator is used as a reward to train the generator . However , training with REINFORCE has been observed to be unstable due to the high variance of the gradient estimate ( Shen et al. , 2017 ) . Xu et al . ( 2017 ) make the GAN model differentiable with an approximate word embedding layer . However , their model only injects variability at the word level , thus limited to represent high-level response variabilities such as topics and situations . In this paper , we propose DialogWAE , a novel variant of GAN for neural conversation modeling . Unlike VAE conversation models that impose a simple distribution over latent variables , DialogWAE models the data distribution by training a GAN within the latent variable space . Specifically , it samples from the prior and posterior distributions over the latent variables by transforming contextdependent random noise with neural networks , and minimizes the Wasserstein distance ( Arjovsky et al. , 2017 ) between the prior and the approximate posterior distributions . Furthermore , our model takes into account a multimodal1 nature of responses by using a Gaussian mixture prior network . Adversarial training with the Gaussian mixture prior network enables DialogWAE to capture a richer latent space , yielding more coherent , informative and diverse responses . Our main contributions are two-fold : ( 1 ) A novel GAN-based model for neural dialogue modeling , which employs GAN to generate samples of latent variables . ( 2 ) A Gaussian mixture prior network to sample random noise from a multimodal prior distribution . To the best of our knowledge , the proposed DialogWAE is the first GAN conversation model that exploits multimodal latent structures . We evaluate our model on two benchmark datasets , SwitchBoard ( Godfrey and Holliman , 1997 ) and DailyDialog ( Li et al. , 2017b ) . The results demonstrate that our model substantially outperforms the state-of-the-art methods in terms of BLEU , word embedding similarity , and distinct . Furthermore , we highlight how the GAN architecture with a Gaussian mixture prior network facilitates the generation of more diverse and informative responses . 2 RELATED WORK . Encoder-decoder variants To address the “ safe response ” problem of the naive encoder-decoder conversation model , a number of variants have been proposed . Li et al . ( 2015 ) proposed a diversitypromoting objective function to encourage more various responses . Sato et al . ( 2017 ) propose to incorporate various types of situations behind conversations when encoding utterances and decoding their responses , respectively . Xing et al . ( 2017 ) incorporate topic information into the sequence-tosequence framework to generate informative and interesting responses . Our work is different from the aforementioned studies , as it does not rely on extra information such as situations and topics . VAE conversation models The variational autoencoder ( VAE ) ( Kingma and Welling , 2014 ) is among the most popular frameworks for dialogue modeling ( Zhao et al. , 2017 ; Shen et al. , 2018 ; Park et al. , 2018 ) . Serban et al . ( 2017 ) propose VHRED , a hierarchical latent variable sequenceto-sequence model that explicitly models multiple levels of variability in the responses . A main challenge for the VAE conversation models is the so-called “ posterior collapse ” . To alleviate the problem , Zhao et al . ( 2017 ) introduce an auxiliary bag-of-words loss to the decoder . They further incorporate extra dialogue information such as dialogue acts and speaker profiles . Shen et al . ( 2018 ) propose a collaborative CVAE model which samples the latent variable by transforming a Gaussian noise using neural networks and matches the prior and posterior distributions of the Gaussian noise with KL divergence . Park et al . ( 2018 ) propose a variational hierarchical conversation RNN ( VHCR ) which incorporates a hierarchical structure to latent variables . DialogWAE addresses the limitation of VAE conversation models by using a GAN architecture in the latent space . GAN conversation models Although GAN/CGAN has shown great success in image generation , adapting it to natural dialog generators is a non-trivial task . This is due to the non-differentiable nature of natural language tokens ( Shen et al. , 2017 ; Xu et al. , 2017 ) . Li et al . ( 2017a ) address this problem by combining GAN with Reinforcement Learning ( RL ) where the discriminator predicts a reward to optimize the generator . However , training with REINFORCE can be unstable due to the high variance of the sampled gradient ( Shen et al. , 2017 ) . Xu et al . ( 2017 ) make the sequenceto-sequence GAN differentiable by directly multiplying the word probabilities obtained from the decoder to the corresponding word vectors , yielding an approximately vectorized representation of the target sequence . However , their approach injects diversity in the word level rather than the level of the whole responses . DialogWAE differs from exiting GAN conversation models in that it shapes the distribution of responses in a high level latent space rather than direct tokens and does not rely on RL where the gradient variances are large . 1A multimodal distribution is a continuous probability distribution with two or more modes . 3 PROPOSED APPROACH . 3.1 PROBLEM STATEMENT . Let d= [ u1 , ... , uk ] denote a dialogue of k utterances where ui= [ w1 , ... , w|ui| ] represents an utterance andwn denotes the n-th word in ui . Let c= [ u1 , ... , uk−1 ] denote a dialogue context , the k-1 historical utterances , and x=uk be a response which means the next utterance . Our goal is to estimate the conditional distribution pθ ( x|c ) . As x and c are sequences of discrete tokens , it is non-trivial to find a direct coupling between them . Instead , we introduce a continuous latent variable z that represents the high-level representation of the response . The response generation can be viewed as a two-step procedure , where a latent variable z is sampled from a distribution pθ ( z|c ) on a latent space Z , and then the response x is decoded from z with pθ ( x|z , c ) . Under this model , the likelihood of a response is pθ ( x|c ) = ∫ z p ( x|c , z ) p ( z|c ) dz . ( 1 ) The exact log-probability is difficult to compute since it is intractable to marginalize out z . Therefore , we approximate the posterior distribution of z as qφ ( z|x , c ) which can be computed by a neural network named recognition network . Using this approximate posterior , we can instead compute the evidence lower bound ( ELBO ) : log pθ ( x|c ) = log ∫ z p ( x|c , z ) p ( z|c ) dz ≥ ` ( x , c ) = Ez∼qφ ( z|x , c ) [ log pψ ( x|c , z ) ] −KL ( qφ ( z|x , c ) ||p ( z|c ) ) , ( 2 ) where p ( z|c ) represents the prior distribution of z given c and can be modeled with a neural network named prior network . 3.2 CONDITIONAL WASSERSTEIN AUTO-ENCODERS FOR DIALOGUE MODELING . The conventional VAE conversation models assume that the latent variable z follows a simple prior distribution such as the normal distribution . However , the latent space of real responses is more complicated and difficult to be estimated with such a simple distribution . This often leads to the posterior collapse problem ( Shen et al. , 2018 ) . Inspired by GAN and the adversarial auto-encoder ( AAE ) ( Makhzani et al. , 2015 ; Tolstikhin et al. , 2017 ; Zhao et al. , 2018 ) , we model the distribution of z by training a GAN within the latent space . We sample from the prior and posterior over the latent variables by transforming random noise using neural networks . Specifically , the prior sample z̃∼pθ ( z|c ) is generated by a generator G from context-dependent random noise ̃ , while the approximate posterior sample z∼qφ ( z|c , x ) is generated by a generator Q from context-dependent random noise . Both ̃ and are drawn from a normal distribution whose mean and covariance matrix ( assumed diagonal ) are computed from c with feed-forward neural networks , prior network and recognition network , respectively : z̃ = Gθ ( ̃ ) , ̃ ∼ N ( ; µ̃ , σ̃2I ) , [ µ̃ log σ̃2 ] = W̃fθ ( c ) + b̃ ( 3 ) z = Qφ ( ) , ∼ N ( ; µ , σ2I ) , [ µ log σ2 ] =Wgφ ( [ x c ] ) + b , ( 4 ) where fθ ( · ) and gφ ( · ) are feed-forward neural networks . Our goal is to minimize the divergence between pθ ( z|c ) and qφ ( z|x , c ) while maximizing the log-probability of a reconstructed response from z . We thus solve the following problem : min θ , φ , ψ −Eqφ ( z|x , c ) log pψ ( x|z , c ) +W ( qφ ( z|x , c ) ||pθ ( z|c ) ) , ( 5 ) where pθ ( z|c ) and qφ ( z|x , c ) are neural networks implementing Equations 3 and 4 , respectively . pψ ( x|z , c ) is a decoder . W ( ·||· ) represents the Wasserstein distance between these two distributions ( Arjovsky et al. , 2017 ) . We choose the Wasserstein distance as the divergence since the WGAN has been shown to produce good results in text generation ( Zhao et al. , 2018 ) . Figure 1 illustrates an overview of our model . The utterance encoder ( RNN ) transforms each utterance ( including the response x ) in the dialogue into a real-valued vector . For the i-th utterance in the context , the context encoder ( RNN ) takes as input the concatenation of its encoding vector and the conversation floor ( 1 if the utterance is from the speaker of the response , otherwise 0 ) and computes its hidden state hctxi . The final hidden state of the context encoder is used as the context representation . At generation time , the model draws a random noise ̃ from the prior network ( PriNet ) which transforms c through a feed-forward network followed by two matrix multiplications which result in the mean and diagonal covariance , respectively . Then , the generator G generates a sample of latent variable z̃ from the noise through a feed-forward network . The decoder RNN decodes the generated z̃ into a response . At training time , the model infers the posterior distribution of the latent variable conditioned on the context c and the response x . The recognition network ( RecNet ) takes as input the concatenation of both x and c and transforms them through a feed-forward network followed by two matrix multiplications which define the normal mean and diagonal covariance , respectively . A Gaussian noise is drawn from the recognition network with the re-parametrization trick . Then , the generator Q transforms the Gaussian noise into a sample of latent variable z through a feed-forward network . The response decoder ( RNN ) computes the reconstruction loss : Lrec = −Ez=Q ( ) , ∼RecNet ( x , c ) log pψ ( x|c , z ) ( 6 ) We match the approximate posterior with the prior distributions of z by introducing an adversarial discriminator D which tells apart the prior samples from posterior samples . D is implemented as a feed-forward neural network which takes as input the concatenation of z and c and outputs a real value . We train D by minimizing the discriminator loss : Ldisc = E ∼RecNet ( x , c ) [ D ( Q ( ) , c ) ] − Ẽ∼PriNet ( c ) [ D ( G ( ̃ ) , c ) ] ( 7 )

Summary: This paper uses Wasserstein GAN in conditional modeling of the dialog response generation. The main goal is to learn to use two network architecture to approximate the posterior distribution of the prior network. Instead of a KL divergence, like in VAE training, they use adversarial training and instead of using softmax output from the discriminator, they use Wasserstein distance. They also introduce a multi-modal distribution, GMM, while sampling from a the posterior during training, prior during the test time. The multi-modal sampling is based on gumbel-softmax over K possible G-distributions. They experiment on Daily Dialog and Switchborad datasets and show promising improvements on qualitative measures like BLEU and BOW embedding similarities, as well as qualitative measures including human evaluations comparing againsts substantial amount of baselines.