LocalGAN: Modeling Local Distributions for Adversarial Response Generation

1 INTRODUCTION . End-to-End generative conversational agents ( a.k.a. , generative Chat-bots ) are believed to be practicable on the basis of the Sequence-to-Sequence ( Seq2Seq ) architecture ( Sutskever et al. , 2014 ) trained with large amounts of human-generated conversation sessions ( Shang et al. , 2015a ; Sordoni et al. , 2015 ) , and this task is named as Neural Response Generation ( NRG ) . Similar to the Neural Machine Translation ( NMT ) approaches ( Bahdanau et al. , 2014 ; Wu et al. , 2016 ) , the deep Seq2Seq models are expected to directly generate appropriate and meaningful responses according to the input query . Compared to the success of NMT systems , the application progress of NRG models is not satisfying at present due to the “ safe response ” problem ( Li et al. , 2016 ) . That is , most of the generated responses are boring and meaningless , which blocks the continuation of conversations . Indeed , eliminating “ safe responses ” is the essential task of NRG models . Thus , various methods have been considered to address this problem ( Li et al. , 2016 ; Xu et al. , 2017 ; Li et al. , 2017 ; Xing et al. , 2017 ; Pandey et al. , 2018 ; Zhang et al. , 2018a ; Du et al. , 2018 ) . More recently , Generative Adversarial Nets ( GAN ) ( Goodfellow et al. , 2014 ) have been introduced to eliminate “ safe responses ” ( Li et al. , 2017 ; Xu et al. , 2017 ) . Basically , this methodology is reasonable since the GAN framework involves an adversarial discriminator that helps NRG models leap out of the shortsighted state of minimizing the empirical risk on word distribution , by providing feedback on real samples from the model generated ones . Despite the improvement on the diversity , the adversarial training process of GAN based response generation models is generally unstable and sensitive to the training strategy ( Yu et al. , 2017 ) . The unstable convergence problem is largely ascribed to the complicated data distribution in practical scenarios ( Arora et al. , 2017 ; Arora & Zhang , 2017 ) . For the response generation oriented GAN models , in particular , the data distribution appears to be much more complicated . Fundamentally , an essential characteristic of conversation data is that , for each given query , there always exists a group of semantically-diverse responses , rather than the semantically-unified ones . Furthermore , the response groups of two different queries tend to keep great divergences in the semantic space . In this scenario , the discriminator needs to consider the distributions of the generated result in the semantic space , rather than simply examining whether one single sample comes from the generator or the original dataset , so as to make the generator sense the distribution of conversation dataset in the adversarial training procedure . This paper aims at presenting a specific adversarial training schema for neural response generation . Beginning with the investigation on the reason for the unsatisfying performance of GANs on the NRG task , we find the upper-bound of the current GAN learning strategies taking query-response pairs as the independent training samples . On this basis , we claim that the training schema , including the actual adversarial strategy and the overall loss function , should be re-defined to agree with the distribution of NRG training samples in the semantic space , rather than roughly adopting the GAN framework designed for generating images . Consequently , we describe the distributional state of the given query and the corresponding responses with the free energy defined on the basis of Deep Boltzmann Machines ( DBM ) ( Salakhutdinov & Hinton , 2009 ) . In this way , we can quantify the formation process of generating the response set with the topic restriction of the given query . From the perspective of free energy , this paper proposes a new cost function to measure the expansion degree of the responses in the local area of the real-valued semantic space . Cooperating with the traditional implicit density discriminating loss of GAN , the proposed cost actually provides an explicit density approximation for the local distribution of each response cluster . Thus , the adversarial learning procedure can be expected to be more stable with better response generation results obtained1 . 2 THE LIMITATION OF GENERAL GAN IN THE NRG SCENARIO . According to ( Goodfellow et al. , 2014 ) , the standard GAN framework contains a generator G and a discriminator D , which are trained by an iterative adversarial learning procedure based on the following objective function : J ( D ) = Ex∼pd [ logD ( x ) ] + Ez∼pz [ log ( 1−D ( G ( z ) ) ) ] ( 1 ) J ( G ) = Ez∼pz [ logD ( G ( z ) ) ] ( 2 ) where pz denotes the prior on input noise variables , and pd is the true data distribution . It should be noted that the GAN tries to learn the manifold of a given dataset ( Khayatkhoei et al. , 2018 ; Kumar et al. , 2017 ) , and the discriminator D actually provides a metric for judging whether the results generated according to z fits the expected manifold or not . In the NRG scenario , a naive Seq2Seq model without the guidance signal from D can not capture the data manifold of the real query-response corpus , which is one of the major facts the safe-response problem can be ascribed to . Assuming that there exists an oracle discriminator with the ability of distinguishing the generated fake samples from the ground-truth ones , by mapping each query-response pair ( q , r ) 2 to a confidence score s , and meanwhile , it can be assume that any practically existing discriminator of GAN gives the confidence s̃ to ( q , r ) . If the practical discriminator can make s̃ → s , the generator will obtain more meaningful guidance for the better generation . That is , to improve the capability of GAN-NRG , it is wise to construct more powerful discriminators for more reasonable J ( G ) . Now let ’ s pay attention to the actual change of NRG models brought by GAN . In the generative conversation agent scenario , G ( z ) is corresponding to a generated response r̃ to a given query q . Thus , the objective of the generator in the GAN based NRG model can be simply formulated as : J ( G ) = E ( q , r̃ ) ∼pg [ logD ( q , r̃ ) ] ( 3 ) The training of the generator is actually the procedure to maximize J ( G ) toward E ( q , r ) ∼pd [ logD ( q , r ) ] , so as to generate realistic responses according to given queries . Thus , in 1The code of our proposed model LocalGAN can be found in https : //github.com/Kramgasse49/ local_gan_4generation 2Here q and r represent the vectorized query and its response . We take the simple embedding-averaging based method to transform texts into vectors . Besides , due to our adopted text vectorizing method , the pretraining phase of the Deep Boltzmann Machine takes some specified trick detailed in Appendix A. the context of adversarial learning , J ( G ) should satisfy the following inequality : J ( G ) 6 E ( q , r ) ∼pd [ logD ( q , r ) ] ( 4 ) However , it is well known that a conversational dataset should not be simply taken as a collection { ( q , r ) } composed of independent query-response pairs . Instead , to each given query q , there exists a finite set of corresponding responses Rq = { ri } . In this case , it is of great necessity to consider the whole training dataset as a collection of Rq , which takes the form of a number of clusters with their own local distributions in the semantic space . And we can rewrite the joint distribution pd in ( 4 ) as p ( q ) p ( r|q ) and assume every corresponding response to a query follows equal-probability distribution , which means that p ( r|q ) = 1|Rq| . Thus , in the real NRG scenario , on the basis of Equation 3 and Inequation 4 , J ( G ) follows the inequality as below : J ( G ) 6 ∑ q E ( q , r ) ∼q , Rq [ logD ( q , r ) ] = ∑ q ∑ r∈Rq p ( q ) 1 |Rq| [ logD ( q , r ) ] 6 ∑ q p ( q ) log 1 |Rq| ∑ r∈Rq D ( q , r ) ( 5 ) where p ( q ) denotes the probability of the query q , and Rq is defined above . According to Equation 5 , the upper bound of J ( G ) is obtained , in which the log [ 1 |Rq| ∑ r∈Rq D ( q , r ) ] part is the essence . Apparently , the expression 1|Rq| ∑ r∈Rq D ( q , r ) indicates the mean value of the confidence scores given by the discriminator to each member of the response set Rq to a given query q . Moreover , it should be noted that current studies tend to utilize semantic relevance oriented models to build the discriminators of GAN ( Xu et al. , 2017 ; Li et al. , 2017 ) . Consequently , D ( q , r ) can be actually considered as the spatial relationship of q and r in the semantic space . Thus , 1|Rq| ∑ r∈Rq D ( q , r ) stands for the spatial center of all the responses within Rq . That is , the optimization process of adversarial learning upon conversational datasets will make the generated responses approach to the center of each local distribution of Rq corresponding to each given dependent query . The practical value of this change lies in that , intuitively , the GAN architecture forces the generation to pay attention to the local distributions of the individual response clusters , rather than taking the ( q , r ) -pairs as an entirety . According to the thorough studies on the safe responses of NRG models ( Li et al. , 2016 ; Xu et al. , 2017 ; Zhang et al. , 2018a ; Pandey et al. , 2018 ) , it can be inferred that the general Seq2Seq will fall into the divergence state of generating the patterns with the maximum probabilities taking account of the entire dataset , ignoring the individual-difference of each query . By introducing the implicit loss focusing on the response clusters , GAN makes the divergence of the generator much closer to the ‘ local patterns ’ rather than the general patterns , and thus the higher diversity can be expected and observed ( Xu et al. , 2017 ; Li et al. , 2017 ) . The problem turns to : Is the upper bound in Equation 5 powerful enough ? Apparently , there exists an obvious gap between the ‘ local patterns ’ and the vivid and interesting generated responses . The upper bound only focuses on the mean of each response set , but the local distribution ( or the actual ‘ shape ’ ) of each cluster has not been taken into account . This situation does not change when the cost function of adversarial training is defined by Wasserstein GANs ( WGAN ) ( Arjovsky et al. , 2017 ) with 1-Lipschitz function f : J ( G ) W = E ( q , r̃ ) ∼pg [ f ( q , r̃ ) ] ( 6 ) Intuitively , it is of paramount importance to estimate both the ‘ location ’ and the ‘ shape ’ of the response set Rq in the semantic space ( indeed , 1|Rq| ∑ q , rD ( q , r ) is only relative with ‘ location ’ ) , so as to determine the optimization objective of adversarial training . Consequently , we have two critical problems to be discussed and addressed in the following sections : • How to describe the state of the response setRq with a given query q in the semantic space ? • Taking account of the reasonable state modeling of ( q , Rq ) , what is the loss function for adversarial training to generate responses ?

Summary: The main contribution of this paper lies in the proposed LocalGAN for neural response generation. The key observation is that for a given query, there always exists a group of diverse responses that are reasonable, rather than a single ground-truth response. Therefore, the local semantic distribution of responses given a query should be modeled. Besides the original GAN loss, the proposed LocalGAN adds an additional local-distribution-oriented objective, resulting in a hybrid loss for training, which claims to achieve better performance on response generation datasets.