Year: 2019

Abstract:
This paper presents a new methodology for modeling the local semantic distribution of responses to a given query in the human-conversation corpus, and on this basis, explores a specified adversarial learning mechanism for training Neural Response Generation (NRG) models to build conversational agents. The proposed mechanism aims to address the training instability problem and improve the quality of generated results of Generative Adversarial Nets (GAN) in their utilizations in the response generation scenario. Our investigation begins with the thorough discussions upon the objective function brought by general GAN architectures to NRG models, and the training instability problem is proved to be ascribed to the special local distributions of conversational corpora. Consequently, an energy function is employed to estimate the status of a local area restricted by the query and its responses in the semantic space, and the mathematical approximation of this energy-based distribution is finally found. Building on this foundation, a local distribution oriented objective is proposed and combined with the original objective, working as a hybrid loss for the adversarial training of response generation models, named as LocalGAN. Our experimental results demonstrate that the reasonable local distribution modeling of the query-response corpus is of great importance to adversarial NRG, and our proposed LocalGAN is promising for improving both the training stability and the quality of generated results.

heading: 1 INTRODUCTION
text: End-to-End generative conversational agents (a.k.a., generative Chat-bots) are believed to be practicable on the basis of the Sequence-to-Sequence (Seq2Seq) architecture (Sutskever et al., 2014) trained with large amounts of human-generated conversation sessions (Shang et al., 2015a; Sordoni et al., 2015), and this task is named as Neural Response Generation (NRG). Similar to the Neural Machine Translation (NMT) approaches (Bahdanau et al., 2014; Wu et al., 2016), the deep Seq2Seq models are expected to directly generate appropriate and meaningful responses according to the input query. Compared to the success of NMT systems, the application progress of NRG models is not satisfying at present due to the “safe response” problem (Li et al., 2016). That is, most of the generated responses are boring and meaningless, which blocks the continuation of conversations. Indeed, eliminating “safe responses” is the essential task of NRG models. Thus, various methods have been considered to address this problem (Li et al., 2016; Xu et al., 2017; Li et al., 2017; Xing et al., 2017; Pandey et al., 2018; Zhang et al., 2018a; Du et al., 2018).
More recently, Generative Adversarial Nets (GAN) (Goodfellow et al., 2014) have been introduced to eliminate “safe responses” (Li et al., 2017; Xu et al., 2017). Basically, this methodology is reasonable since the GAN framework involves an adversarial discriminator that helps NRG models leap out of the shortsighted state of minimizing the empirical risk on word distribution, by providing feedback on real samples from the model generated ones. Despite the improvement on the diversity, the adversarial training process of GAN based response generation models is generally unstable and sensitive to the training strategy (Yu et al., 2017).
The unstable convergence problem is largely ascribed to the complicated data distribution in practical scenarios (Arora et al., 2017; Arora & Zhang, 2017). For the response generation oriented GAN models, in particular, the data distribution appears to be much more complicated. Fundamentally, an essential characteristic of conversation data is that, for each given query, there always exists a
group of semantically-diverse responses, rather than the semantically-unified ones. Furthermore, the response groups of two different queries tend to keep great divergences in the semantic space. In this scenario, the discriminator needs to consider the distributions of the generated result in the semantic space, rather than simply examining whether one single sample comes from the generator or the original dataset, so as to make the generator sense the distribution of conversation dataset in the adversarial training procedure.
This paper aims at presenting a specific adversarial training schema for neural response generation. Beginning with the investigation on the reason for the unsatisfying performance of GANs on the NRG task, we find the upper-bound of the current GAN learning strategies taking query-response pairs as the independent training samples. On this basis, we claim that the training schema, including the actual adversarial strategy and the overall loss function, should be re-defined to agree with the distribution of NRG training samples in the semantic space, rather than roughly adopting the GAN framework designed for generating images.
Consequently, we describe the distributional state of the given query and the corresponding responses with the free energy defined on the basis of Deep Boltzmann Machines (DBM) (Salakhutdinov & Hinton, 2009). In this way, we can quantify the formation process of generating the response set with the topic restriction of the given query. From the perspective of free energy, this paper proposes a new cost function to measure the expansion degree of the responses in the local area of the real-valued semantic space. Cooperating with the traditional implicit density discriminating loss of GAN, the proposed cost actually provides an explicit density approximation for the local distribution of each response cluster. Thus, the adversarial learning procedure can be expected to be more stable with better response generation results obtained1.

heading: 2 THE LIMITATION OF GENERAL GAN IN THE NRG SCENARIO
text: According to (Goodfellow et al., 2014), the standard GAN framework contains a generator G and a discriminator D, which are trained by an iterative adversarial learning procedure based on the following objective function:
J (D) = Ex∼pd [logD(x)] + Ez∼pz [log(1−D(G(z)))] (1)
J (G) = Ez∼pz [logD(G(z))] (2) where pz denotes the prior on input noise variables, and pd is the true data distribution.
It should be noted that the GAN tries to learn the manifold of a given dataset (Khayatkhoei et al., 2018; Kumar et al., 2017), and the discriminator D actually provides a metric for judging whether the results generated according to z fits the expected manifold or not. In the NRG scenario, a naive Seq2Seq model without the guidance signal from D can not capture the data manifold of the real query-response corpus, which is one of the major facts the safe-response problem can be ascribed to. Assuming that there exists an oracle discriminator with the ability of distinguishing the generated fake samples from the ground-truth ones, by mapping each query-response pair (q, r)2 to a confidence score s, and meanwhile, it can be assume that any practically existing discriminator of GAN gives the confidence s̃ to (q, r). If the practical discriminator can make s̃ → s, the generator will obtain more meaningful guidance for the better generation. That is, to improve the capability of GAN-NRG, it is wise to construct more powerful discriminators for more reasonable J (G).
Now let’s pay attention to the actual change of NRG models brought by GAN. In the generative conversation agent scenario, G(z) is corresponding to a generated response r̃ to a given query q. Thus, the objective of the generator in the GAN based NRG model can be simply formulated as:
J (G) = E(q,r̃)∼pg [logD(q, r̃)] (3)
The training of the generator is actually the procedure to maximize J (G) toward E(q,r)∼pd [logD(q, r)], so as to generate realistic responses according to given queries. Thus, in
1The code of our proposed model LocalGAN can be found in https://github.com/Kramgasse49/ local_gan_4generation
2Here q and r represent the vectorized query and its response. We take the simple embedding-averaging based method to transform texts into vectors. Besides, due to our adopted text vectorizing method, the pretraining phase of the Deep Boltzmann Machine takes some specified trick detailed in Appendix A.
the context of adversarial learning, J (G) should satisfy the following inequality:
J (G) 6 E(q,r)∼pd [logD(q, r)] (4)
However, it is well known that a conversational dataset should not be simply taken as a collection {(q, r)} composed of independent query-response pairs. Instead, to each given query q, there exists a finite set of corresponding responses Rq = {ri}. In this case, it is of great necessity to consider the whole training dataset as a collection of Rq , which takes the form of a number of clusters with their own local distributions in the semantic space. And we can rewrite the joint distribution pd in (4) as p(q)p(r|q) and assume every corresponding response to a query follows equal-probability distribution, which means that p(r|q) = 1|Rq| . Thus, in the real NRG scenario, on the basis of Equation 3 and Inequation 4, J (G) follows the inequality as below:
J (G) 6 ∑ q E(q,r)∼q,Rq [logD(q, r)]
= ∑ q ∑ r∈Rq p(q) 1 |Rq| [logD(q, r)] 6 ∑ q p(q) log  1 |Rq| ∑ r∈Rq D(q, r)  (5) where p(q) denotes the probability of the query q, and Rq is defined above. According to Equation 5, the upper bound of J (G) is obtained, in which the log [
1 |Rq| ∑ r∈Rq D(q, r) ] part is the essence. Apparently, the expression 1|Rq| ∑ r∈Rq D(q, r)
indicates the mean value of the confidence scores given by the discriminator to each member of the response set Rq to a given query q. Moreover, it should be noted that current studies tend to utilize semantic relevance oriented models to build the discriminators of GAN (Xu et al., 2017; Li et al., 2017). Consequently, D(q, r) can be actually considered as the spatial relationship of q and r in the semantic space. Thus, 1|Rq| ∑ r∈Rq D(q, r) stands for the spatial center of all the responses within Rq . That is, the optimization process of adversarial learning upon conversational datasets will make the generated responses approach to the center of each local distribution of Rq corresponding to each given dependent query.
The practical value of this change lies in that, intuitively, the GAN architecture forces the generation to pay attention to the local distributions of the individual response clusters, rather than taking the (q, r)-pairs as an entirety. According to the thorough studies on the safe responses of NRG models (Li et al., 2016; Xu et al., 2017; Zhang et al., 2018a; Pandey et al., 2018), it can be inferred that the general Seq2Seq will fall into the divergence state of generating the patterns with the maximum probabilities taking account of the entire dataset, ignoring the individual-difference of each query. By introducing the implicit loss focusing on the response clusters, GAN makes the divergence of the generator much closer to the ‘local patterns’ rather than the general patterns, and thus the higher diversity can be expected and observed (Xu et al., 2017; Li et al., 2017).
The problem turns to: Is the upper bound in Equation 5 powerful enough? Apparently, there exists an obvious gap between the ‘local patterns’ and the vivid and interesting generated responses. The upper bound only focuses on the mean of each response set, but the local distribution (or the actual ‘shape’) of each cluster has not been taken into account. This situation does not change when the cost function of adversarial training is defined by Wasserstein GANs (WGAN) (Arjovsky et al., 2017) with 1-Lipschitz function f :
J (G) W = E(q,r̃)∼pg [f(q, r̃)] (6)
Intuitively, it is of paramount importance to estimate both the ‘location’ and the ‘shape’ of the response set Rq in the semantic space (indeed, 1|Rq| ∑ q,rD(q, r) is only relative with ‘location’), so as to determine the optimization objective of adversarial training. Consequently, we have two critical problems to be discussed and addressed in the following sections:
• How to describe the state of the response setRq with a given query q in the semantic space? • Taking account of the reasonable state modeling of (q,Rq), what is the loss function for
adversarial training to generate responses?

heading: 3 MODELING THE STATE OF THE LOCAL DISTRIBUTION FOR RESPONSES
text: As mentioned above, the semantic one-to-many relationship between queries and responses makes it necessary to model the local distribution of the response cluster Rq corresponding to each query in the space, and it is paramount to turn to the fitting of each local distribution in the adversarial learning procedure, rather than considering each (q, r)-pair as an independent sample. Basically, this issue equals to the task of reasonably modeling the state of (q,Rq) in the semantic space, by considering each (q,Rq) as a systematic entirety and assigning the state of the entirety with probabilistic distribution. The additional major challenge of this task is, indeed, we have to infer the state of a local area in the semantic space from a group of finite samples, since it is impossible to sample all the possible responses to a query from the given corpus, regardless of the corpus size.

heading: 3.1 REPRESENTING LOCAL DISTRIBUTIONS WITH QUERY-RESPONSE ORIENTED FREE ENERGY
text: In this part, we typically take an energy based statistical model, the Average Free Energy (Hinton & Zemel, 1994; Friston et al., 2006; Ngiam et al., 2011; Friston et al., 2012), to describe the state of the local distribution of (q,Rq) in the semantic space, for the reasons that: a) energy based models are considered as a promising avenue towards learning explicit generative models (LeCun et al., 2006; Le Roux & Bengio, 2008), by representing data distributions without any prior assumptions; and b) energy based models can be trained in the unsupervised way, and the energy functions of such models have the potential to estimate the state of generative models (Zhao et al., 2016).
At first, the free energy of a given query-response pair (q, r) can be defined as following:
F (q, r) = − log [∑ H exp(E(q, r,H)) ] (7)
where E(q, r,H) stands for the energy function defined according to the relationship of the query q and its response r via the hidden variable H .
We employ the Deep Boltzmann Machine (DBM) (Salakhutdinov & Larochelle, 2010; Smolensky, 1986; Hinton & Salakhutdinov, 2006) to implement E(q, r,H), as illustrated by Figure 1. The reason for this choice lies in that, from the view of conversational agents, the meaningful query and the corresponding response are generally considered to maintain strong semantic relevance. Thus, the query and response can be mutually transformed into each other, which is supported by the considerable amount of studies on response generation (Shang et al., 2015b; Shao et al., 2017; Zhang et al., 2018a; Baheti et al., 2018) and question generation (Du et al., 2017; Zhao et al., 2018; Sun et al., 2018). Without loss of generality, the pairwise semantic relationship of the query q and the corresponding response r can be
modeled by a two-layer DBM. The bottom layer is actually an abstract version of Seq2Seq models, in which a response r can be generated based on a hidden variable hq , and hq depends on the given query q theoretically. In the top layer, hq conditionally depends on a hyper hidden variable h. Figure 1 illustrates the Deep Boltzmann Machine for modeling the semantic relationship of a query and its responses.
Following (Salakhutdinov & Hinton, 2009), on the basis of the DBM in Figure 1, the energy of the state {(q, r), H} is defined as:
E(q, r,H) = E (q, r, hq, h) = −hTq Wqrr − hTq Wqqq − hTWqhhq (8)
where q denotes the query, r stands for the response and H = {hq, h} represents the hidden units. Wqr, Wqq and Wqh stand for the weights on the corresponding connections of the query, response and the hidden variables respectively in the graph model shown by Figure 1.
Consequently, we can define the average free energy of the query q and its response set Rq as follows:
F (q,Rq) = 1 |Rq| ∑
ri∈Rq
F (q, ri) (9)
For better conducting the following discussion, we further define the energy difference between response ri and rj as:
Definition 3.1 (Scaled Energy Difference).
∆q,ri,rj = F (q, ri)− F (q, rj)
F (q,Rq) (10)
Meanwhile, it is necessary to assign a spatial intuition to Rq in the semantic space by defining:
Definition 3.2 (Response Cluster). In the semantic space, the meaningful responses to the given query q lie in a restricted region (e.g., a hyper sphere), which can be named as the Response Cluster, in which F (q, r) can be taken as the distance from a response r to the cluster center.
3.2 ESTIMATION OF F (q,Rq)
Basically, the DBM in Figure 1 provides the definition of the energy function E(q, r,H) of the free energy given in Equation 8. Thus, the local distribution state of the responses Rq with the given query q can be mathematically described by F (q,Rq) based on Equation 7 - 9. In practice, however, this procedure is not operable yet because the computation of F (q,Rq) requires all the response in Rq , and it is impractical to perform the exhaustive enumeration over all the possible responses of the given query, regardless of the amount of the training query-response pairs. Consequently, it is highly necessary to approximate F (q,Rq) under some reasonable assumptions.
Considering the response cluster defined in Definition 3.2, we can naturally assume that the response random variable r to a given query q follows multivariate normal distribution with mean rc and covariance matrix Σ, where rc and Σ is only determined by query q. Afterwards, the observed Rq can be considered as a realization of the |Rq|-variate random variable (r1, · · · , r|Rq|) for ri, 1 ≤ i ≤ |Rq|, i.i.d random variables drawn from the distribution N(rc,Σ). Consequently, an executable approximation of F (q,Rq) can be obtained based on the following Lemmas and Theorem3.
Lemma 1. If E[F (q, r)] < ∞, F (q,Rq) converges almost surely to the expected value E[F (q, r)], as |Rq| goes to infinity. That is,
P (
lim |Rq|→∞
F (q,Rq) = E[F (q, r)] ) = 1. (11)
Lemma 1 indicates that for any > 0 there exists N( ) such that if |Rq| > N( ) then the inequality |F (q,Rq) − E[F (q, r)]| < holds. In other words, F (q,Rq) can be very close to E[F (q, r)] if the responses are sufficient.
Lemma 2. lim
E‖r−rc‖2→0 |E[f(r)]− f (rc) | = 0 (12)
Practically, the expected Euclidean distance between random variable r and rc can not be zero. Thus, the fact conveyed by Lemma 2 is that, actually, if the expected Euclidean distance is small enough, the difference between E[F (q, r)] and F (q, rc) can be controllable (or even close to zero).
Based on the lemmas we have:
Theorem 1. Given E[F (q, r)] <∞, we have
|F (q,Rq)− F (q, r̂c)| a.s.−→ 0 when |Rq| → ∞ and E‖r − rc‖2 → 0 (13)
where r̂c is the estimation of rc based on the well-trained DBM.
3The proof of Lemma 1, Lemma 2 and Theorem 1 is given in Appendix B.
proof of theorem 1.
|F (q,Rq)− F (q, r̂c)| ≤ |F (q,Rq)− E[F (q, r)]|+|E[F (q, r)]− F (q, rc)|+|F (q, rc)− F (q, r̂c)| (14)
Following lemma 1, the first term converges almost surely to zero when |Rq| → ∞. And following lemma 2, the second term goes to zero when E‖r − rc‖2 → 0. Finally, given the well-trained DBM with the capability of mapping a query q to r̂c (Wang et al., 2010; Srivastava & Salakhutdinov, 2012), r̂c can be taken as the estimation of rc. Meanwhile, since the function F (q, r) is the composition and combination of simple continuous functions, we have |F (q, rc)− F (q, r̂c)| → 0.

heading: 4 THE HYBRID LOSS OF ADVERSARIAL RESPONSE GENERATION
text: As discussed in Section 2, the ability of the response generator in the general GAN architecture is limited to learning the dense distribution around 1|Rq| ∑ r∈Rq D(q, r) (see Equation 5), which is composed of the most frequent patterns in the semantic space. By contrast, it is difficult for the general architecture to sense the remaining sparse space containing high-quality diverse responses. Therefore, reasonably describing the local distribution of the responses to a given query is highly necessary. According to the analysis in Section 3, the average free energy can be taken to model the state of the local area of the responses to a query, and such energy can be reasonably approximated via the DBM defined on the query-response pairs. On the basis of the previous sections, this section will finally propose the new hybrid loss function to force the generator to produce responses with better diversity through the more stable adversarial training process.

heading: 4.1 THE RADIAL DISTRIBUTION FUNCTION OF THE RESPONSE
text: The analysis in Section 3 have shown that the local distribution state of the responsesRq to the given query q can be modeled by the average free energy F (q,Rq). On this basis, it is possible to propose the description of the spatial state of a single response r in the semantic space, and consequently, we can give a new adversarial loss indicating the cost of simulating the local distribution of Rq .
According to Definition 3.1 and 3.2, in each response cluster Rq , the distance from a response r to the cluster center rc is actually equivalent to the scaled energy difference between them, that is,
∆q,r,rc = F (q, r)− F (q, rc)
F (q,Rq) (15)
Meanwhile, on the basis of Theorem 1, F (q,Rq) can be approximated by F (q, r̂c), and r̂c is modeled from training data, and thus we have:
∆q,r,rc ≈ F (q, r)− F (q, r̂c)
F (q, r̂c) = α(q,r) (16)
Here we approximate ∆q,r,rc with α(q,r), and formally call α(q,r) as the Radial Distribution Function (RDF), indicating the relative cost ratio to F (q, rc) for obtaining r from a given q (also the distinctiveness of r, actually).

heading: 4.2 THE HYBRID OBJECTIVE FUNCTION
text: Based on the previous discussions, for the adversarial response generation methodology, the essence is to reasonably describe the state of the local distribution of the response cluster given by Definition 3.2, and further more, to take this important element into account in the final optimization.
Especially, in Subsection 4.1, we have defined the Radial Distribution Function in Equation 16 to quantify the distinctiveness of a response, the very basis of which is the description of the local state F (q,Rq) in the semantic space. Thus, we can further build a mechanism to quantify the difference between the generated response and the golden response as follows:
δα = α(q,r) − α(q,r̃) (17)
where r̃ is the generated response given by the generator and r comes from the original data. If δα moves toward zero, α(q,r̃) would be close to α(q,r) sharing the same F (q, rc).
Consequently, a new expectation comes out. That is, the generator needs to provide results that can minimize δα, so as to fit the local distribution of the existing responses to a given query. Thus, a hybrid objective of the generator can be finally defined as:
min JG = −E [logD(q, r)] +ReLU(δα) (18)
A hinge loss, conducted by the ReLU function ReLU(δα) = max(0, δα), is especially introduced to reform δα. The primary reason of this operation is that the ReLU function has positive output only if δα > 0, according to the definition of ReLU(δα). Apparently, δα < 0 indicates that the generated response r̃ is too far from the center of the response cluster in the semantic space, so that its relevance may be highly questionable. Meanwhile, minimizing a negative variable is against the optimization direction. After the ReLU transformation, there remains valid loss only when δα > 0, and thus both the diversity and the relevance of generated results are taken into account.

heading: 4.3 THE PHASE-WISE OPTIMIZATION
text: According to the analysis in Section 2, the trivial adversarial training directed by −E [logD(q, r)] can only determine the form of general responses to a given query. From the spatial perspective in the semantic space, the original adversarial objective is helpful to roughly locate the response cluster to be generated. However, the local distribution can not be captured by this procedure.
By contrast, according to the discussions above, the proposed hybrid objective actually provides a way to force the generated responses, originally gathering around the general form, to expand into the expected local shape described by the golden truth, by conducting a phase-wise optimizing operation. This mechanism can be detailed in an intuitive way:
Foundation: Once the DBM in Figure 1 is well-trained with the query-response corpus, the semantic center rc of a Response Cluster can be determined by the given query q.
Phase-1: In the early stage of the adversarial training, a generated response r̃ is not semantically relevant to the query q. Thus, it can be inferred that r̃ is radially farther from the cluster center rc than the golden response r. In this situation, according to Equation 16 and Equation 17, we can claim that δα ≤ 0. In this phase, the hyper objective goes back to the general adversarial objective due to the ReLU function. Thus, the model is trying to force the generated samples to approach the center of each cluster, ignoring local distributions.
Phase-2: During the adversarial training in Phase-1, the generated result r̃ will go approaching to the cluster center rc, which means α(q,r̃) → 0 . It should be noted that, for any meaningful existing training sample r, α(q,r) > 0. Therefore, at some point, it turns to δα > 0 and the right part of the hybrid objective in Equation 18 takes effect. Consequently, for each given query, the distribution of the generated results will expand to fit the local distribution of the golden samples.

heading: 5 EXPERIMENTS

heading: 5.1 EXPERIMENTAL SETUPS
text: Datasets. Our experiments are conducted on two main stream open-access conversation corpora: The Opensubtitles corpus and the Sina Weibo corpus. The OpenSubtitles dataset contains 5,200,000 movie dialogues, where we extract query-response pairs following (Xu et al., 2018; Li et al., 2016). The Sina Weibo Corpus (Shang et al., 2015a) contains 2,500,000 single-turn Chinese dialogues, in which the length of the query and response ranges from 4 to 30. We sample 100,000, and 2,000 unique query-response pairs as validation and testing dataset respectively from both of the corpora4.
Baselines. For meaningful comparison, we introduce the following models as the baselines:
(1) Seq2Seq: a sequence-to-sequence model trained with maximum likelihood estimation (MLE); 4Both the English and the Chinese datasets used in our experiments are uploaded to https://www. dropbox.com/sh/k8i079gd2111lsb/AACLLtlNAzile543Da8Qs9tFa?dl=0.
(2) Seq2Seq-MMI: the NRG model with a Maximum Mutual Information criterion (Li et al., 2016);
(3) Adver-REGS: the NRG model trained using adversarial framework, in which the policy gradient was employed to transfer the reward of the discriminator to the generator (Li et al., 2017);
(4) GAN-AEL: an adversarial framework with an approximate embedding layer for connecting the generator with the discriminator directly (Xu et al., 2017).
(5) AIM / DAIM: the adversarial training strategy allowing distributional matching of synthetic and real responses and explicitly optimizing a variational lower bound on pairwise mutual information between the query and response, so as to improve the informativeness and diversity of generated responses (Zhang et al., 2018b)5.
Evaluation Metrics. To evaluate diversity of results, we adopt three widely-applied metrics: Distinct-1 (Dist-1), Distinct-2 (Dist-2), and Entropy (Ent4) (Li et al., 2016; Zhang et al., 2018b; Jost, 2006). Besides, the relevance (Rel.) is measured by summing three embedding-based similarities (greedy, average, extreme) (Liu et al., 2016) upon the ground-truth and generated responses.
Training Details. The vocabulary size of both datasets is 40,000. The embedding layer of OpenSubtitles and Sina Weibo is initialized using 200-dimensional Glove vectors (Pennington et al., 2014) and 300-dimensional Weibo vectors (Li et al., 2018) respectively. All the models are first pre-trained by MLE, and then the models including Adver-REGS, GAN-AEL, AIM, DAIM and LocalGAN are trained with adversarial learning. The discriminator of Adver-REGS and GAN-AEL are based on CNN following (Yu et al., 2017; Xu et al., 2017), in which the filter sizes are set to (1,2,3,4) and the filter number is 128, while that of LocalGAN adopts DBM with (2×embedding size, 128, 128) to represent the semantic of queries and responses. The hidden size of the generator is set to 256 and 512 in GAN-based models and Seq2Seq respectively. To guarantee the performance consistency of AIM and DAIM, we adopt the recommended parameter settings given by Zhang et al. (2018b). The experiments are conducted on the Tesla K80 GPU.

heading: 5.2 RESULTS & ANALYSIS
text: Table 1 lists quantitative results on the diversity and relevance of generated responses on both datasets. As shown by the results, compared to Seq2Seq and Seq2Seq-MMI, the GAN-based methods give better results on the diversity oriented metrics, including Dist-1, Dist-2 and Ent4. This observation indicates that adversarial learning does provide the meaningful guidance to NRG models to avoid some of the safe-responses.
It can be observed that LocalGAN outperforms the baselines with adversarial learning architecture (Adver-REGS, GAN-AEL, AIM and DAIM) on both the diversity metrics and the relevance metrics. Generally, a notable improvement on diversity may lead to some negative influence on relevance, and thus promoting the diversity of generated response while maintaining their relevance is essentially desired for any methodologies, which has been achieved by our LocalGAN. The performances of LocalGAN can be attributed to the fact that LocalGAN has taken the local distribution of responses to a given query into account. By adopting the hybrid objective function, the proposed adversarial
5We have taken the codes of AIM and DAIM from https://github.com/dreasysnail/ converse_GAN implemented by the authors of this work for comparisons.
model gets to capture the spatial characteristics of response clusters, and the generation process is consequently forced to fit the semantic distributions of response clusters.
The training stability is a tough issue to be addressed for adversarial learning (Yu et al., 2017), and as discussed in the previous sections, one of the motivations of our LocalGAN is to make adversarial learning more stable. To verify this aspect, we track the changing of the Entropy (Ent4) of results given by GAN-AEL, Adver-REGS, AIM, DAIM and LocalGAN, as shown in Figure 2.
It can be observed that the training of LocalGAN and Adver-REGS is stable. By contrast, there exist obvious fluctuations on the curves of AIM and DAIM, and GAN-AEL rapidly gets out of control after 1000 batch, which makes it rather difficult to grasp the models with the best status. This group of results indicates the necessity of introducing additional restrictions into adversarial learning processes. For this purpose, Adver-REGS introduces a teacher-forcing loss (Li et al., 2017), while AIM and DAIM have taken the informativeness oriented constraints to partially control the stability (Zhang et al., 2018b). However, GAN-AEL only takes the Wasserstein distance as the objective (Xu et al., 2017), and thus the entropy goes down rapidly. Compared to the Adver-REGS, our LocalGAN achieves better diversity with even a more smooth entropy curve. The training of LocalGAN benefits from the phase-wise optimization driven by the hybrid loss, and its stability also indicates the meaningfulness of modeling and utilizing local distributions of responses.
5.3 HUMAN EVALUATION
To further conduct intuitive comparisons among the NRG models, we perform human evaluations on 500 testing samples. Five annotators are asked to judge whether a response is relevant to the given query and whether the response is informative or not respectively. Both human metrics “Relevance” and “Informativeness” solely have two labels 0 and 1 (0: irrelevant or non-informative; 1: relevant or informative) for simplicity. The human evaluation results are listed in Table 2, and the results basically match the observations in Table 1. The
Kappa of relevance and informativeness annotations are 0.734 and 0.561 respectively.

heading: 5.4 CASE STUDY
text: The effectiveness of the NRG models can be illustrated by the responses generated by them intuitively. Table 3 lists the sampled results given by all the baselines and our proposed LocalGAN, trained on the OpenSubtitle corpus (The samples based on the Sina Weibo dataset are given in Appendix C). It can be observed that, compared to the basic Seq2Seq and Seq2Seq-MMI, the models incorporating adversarial learning have generated results with varying degrees of semantic diversity,
as expected. Among the baselines, DAIM produces more diverse responses, which matches the quantified evaluation results in Table 1, The samples from LocalGAN have shown better semantic relevance, and meanwhile the diversity and informativeness are also guaranteed.

heading: 6 CONCLUSIONS
text: This paper has given the theoretical proof of the upper bound of the adversarial training leveraged models on the Seq2Seq-based neural response generation task. The proof indicates that, due to the local distribution nature of query-response corpora, the GAN based NRG models will converge to the states mostly generating specialized patterns corresponding to given queries. To address this issue, we proposed to model the local distribution of queries and their response in the semantic space by adopting energy-based function, and found the approximation of this function. According to this approximated distribution representation, a new loss function describing the local expansion cost in the fitting of response distribution is presented and finally combined with the traditional GAN loss to form a hybrid training objective for the GAN based NRG model. This paper provides a reasonable explanation to the unstable training process and unsatisfying results of GAN based NRG approaches, and meanwhile gives a different perspective to leverage the local data distribution to enhance classic GAN approaches.

heading: A THE GRAPHICAL MODEL FOR RESPONSE DISTRIBUTION MODELING
text: Different from the computer vision related scenario, training a DBM on a set of text vectors is not trivial, since the training procedure is difficult to converge due to the value scale of text vectors is much larger than image vectors. Moreover, the simple scaling methods are not effective enough for this issue. For this purpose, this paper adopts standard-scaler6 to remove the mean and scale to unit variance. To valid the effectiveness of standard-scaler, We conduct experiments on the queryresponse matching task using normalized vectors. The experiment result show that the matching performance based normalized vectors is similar to that of CNN based architecture (Kim, 2014).

heading: B DETAILED PROOF OF LEMMAS
text: Proof of Lemma 1. As mentioned before, ri is assumed to be independent and identically distributed (i.i.d) normal variable, thus F (q, ri) can be seen as i.i.d random variable for a fixed query q. Based on the strong law of large numbers,
F (q,Rq) = 1 |Rq| ∑
ri∈Rq
F (q, ri) a.s.−→ E[F (q, r)] when |Rq| → ∞
Proof of Lemma 2. For a fixed query, F (q, r) can be seen as the the scalar function of vector r. For simplicity, we denote F (q, r) as f(r).
Taylor expansions for the first moment of function of random variables are as follows.
E[f(r)]] = E [f (rc + (r − rc))] = E [ f (rc) + (r − rc)TDf(rc) + 1
2 (r − rc)>
{ D2f(rc) } (r − rc) +Rrc,2(r − rc) ] where Df(rc) is the gradient of f evaluated at rc, D2f(rc) is the Hessian matrix and Rrc,2(r− rc) is the Lagrange remainder. Since Er = rc, the second term (r − rc)TDf(rc) disappears. Then we will try to find the upper bound of the third term and the remainder term. The relevant theorems used in the proof are listed as follows.
• According to (Petersen & Pedersen, 2012), assume A is symmetric, c = E[x] and Σ = Var[x], then E [ xT Ax ] = Tr(AΣ) + cT Ac.
• (Mirsky, 1975) states following theorem: If A,B are complex n×n matrices with singular values α1 ≥ α2 ≥ · · · ≥ αn and β1 ≥ β2 ≥ · · · ≥ βn respectively, then
|Tr(AB)| ≤ n∑
i=1
αiβi
Firstly, since r−rc ∼ N(0,Σ) and Σ is positive semi-definite matrix, the third term can be simplified as follows.
|E[(r − rc)> { D2f(rc) } (r − rc)]| = |Tr({D2f(rc)}Σ) + 0>D2f(rc)0|
≤ n∑
i=1
αiβi
≤ α1Tr(Σ) = α1E [ ‖r − rc]‖22 ] where α1 ≥ α2 ≥ · · · ≥ αn and β1 ≥ β2 ≥ · · · ≥ βn denote the singular value of matrix {D2f(rc)} and Σ respectively.
6https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html
Meanwhile, according to the Corollary 1 in (Folland, 2005), we have that
|Rrc,2(r − rc)| ≤ M̃
3! ‖r − rc‖31
where M̃ is the upper bound for absolute value of third-order partial derivatives of f .
Next, we show that the α1 and third-oder partial derivatives can be bounded by M , where M is max
hq,1≤i≤n |b(hq, i)| (hq follows multinomial distribution) and n is the dimension of r.
Substituting the definition of E(q, r,H) into F (q, r), we have following equation:
f(r) = F (q, r) = − log ∑ hq,h exp(hTq Wqrr + h T q Wqqq + h TWqhhq).
After that, its first-order, second-order and third-order partial derivative are calculated as follows:
∂f(r)
∂ri = −
∑ hq,h exp(hTq Wqrr + h T q Wqqq + h
TWqhhq)× (hTq Wqr)i∑ hq,h exp(hTq Wqrr + h T q Wqqq + h TWqhhq)
∂2f(r) ∂ri∂rj = ∑ hq,h a(hq, h)× b(hq, i)× [∑ h̃q,h̃ a(h̃q, h̃)× b(h̃q, j)− b(hq, j) ]
∂3f(r) ∂ri∂rj∂rk = ∑ hq,h c(hq, h, k)× b(hq, i)× [∑ h̃q,h̃ a(h̃q, h̃)× b(h̃q, j)− b(hq, j) ]
+ ∑ hq,h a(hq, h)× b(hq, i)× [∑ h̃q,h̃ c(h̃q, h̃, k)× b(h̃q, j) ]
where
a(hq, h) = exp(hTq Wqrr + h T q Wqqq + h TWqhhq)∑̃ hqh̃ exp(h̃Tq Wqrr + h̃ T q Wqqq + h̃ TWqhh̃q)
c(hq, h, k) = ∂a(hq, h)
∂rk = a(hq, h)b(hq, k)− a(hq, h) ∑ h̃q,h̃ a(h̃q, h̃)b(h̃q, k)
and b(hq, i) = (hTq Wqr)i representing the i-th element of the vector h T q Wqr. According to the definition of a(hq, h), it is obvious that a(hq, h) > 0 and ∑
hq,h a(hq, h) = 1.
The upper bounds for the second-order and third-order partial derivative are shown as follows.∣∣∣∣ ∂2f∂ri∂rj ∣∣∣∣ ≤∑
hq,h |a(hq, h)| × |b(hq, i)| × [∑ h̃q,h̃ ∣∣∣a(h̃q, h̃)× b(h̃q, j)∣∣∣+ |b(hq, j)| ] ≤ 2M2
|c(hq, h, k)| ≤ |a(hq, h)| × |b(hq, k)|+ |a(hq, h)| × ∑ h̃q,h̃ |a(h̃q, h̃)| × |b(h̃q, k)| ≤ 2M |a(hq, h)|
| ∂ 3f(r)
∂ri∂rj∂rk | ≤ ∑ hq,h |c(hq, h, k)| ×M × [M +M ] +M2 ∑ h̃q,h̃ |c(h̃q, h̃, k)| ≤ 6M3
Based on the upper bound above, we can see that M̃ can be 6M3 and the upper bounds for α1 and |Rrc,2(r − rc)| are as follows.
α1 = σmax({D2f(rc)}) ≤ ‖{D2f(rc)}‖F = √√√√ n∑ i=1 n∑ j=1 | ∂ 2f ∂ri∂rj |2 ≤ 2nM2
|Rrc,2(r − rc)| ≤ M̃
3! ‖r − rc‖31 = M3‖r − rc‖31 ≤ (
√ nM‖r − rc‖2)3
Hence, we have
|E[f(r)]− f (rc) | = | 1 2 E [ (r − rc)> { D2f(rc) } (r − rc) ] + E [ Rrc,2(r − rc) ] |
≤ 1 2 α1E‖r − rc]‖22 + E|Rrc,2(r − rc)| ≤ E( √ nM‖r − rc]‖2)2 + E( √ nM‖r − rc‖2)3
Therefore, lim
E‖r−rc‖2→0 |E[f(r)]− f (rc) | = 0

heading: C ADDITIONAL CASES

Review summaries:
The main contribution of this paper lies in the proposed LocalGAN for neural response generation. The key observation is that for a given query, there always exists a group of diverse responses that are reasonable, rather than a single ground-truth response. Therefore, the local semantic distribution of responses given a query should be modeled. Besides the original GAN loss, the proposed LocalGAN adds an additional local-distribution-oriented objective, resulting in a hybrid loss for training, which claims to achieve better performance on response generation datasets.