Title: LOCALGAN: MODELING LOCAL DISTRIBUTIONS FOR ADVERSARIAL RESPONSE GENERATION

Abstract: This paper presents a new methodology for modeling the local semantic distribution of responses to a given query in the human-conversation corpus, and on this basis, explores a specified adversarial learning mechanism for training Neural Response Generation (NRG) models to build conversational agents. The proposed mechanism aims to address the training instability problem and improve the quality of generated results of Generative Adversarial Nets (GAN) in their utilizations in the response generation scenario. Our investigation begins with the thorough discussions upon the objective function brought by general GAN architectures to NRG models, and the training instability problem is proved to be ascribed to the special local distributions of conversational corpora. Consequently, an energy function is employed to estimate the status of a local area restricted by the query and its responses in the semantic space, and the mathematical approximation of this energy-based distribution is finally found. Building on this foundation, a local distribution oriented objective is proposed and combined with the original objective, working as a hybrid loss for the adversarial training of response generation models, named as LocalGAN. Our experimental results demonstrate that the reasonable local distribution modeling of the query-response corpus is of great importance to adversarial NRG, and our proposed LocalGAN is promising for improving both the training stability and the quality of generated results.

1. INTRODUCTION:

End-to-End generative conversational agents (a.k.a., generative Chat-bots) are believed to be practicable on the basis of the Sequence-to-Sequence (Seq2Seq) architecture (Sutskever et al., 2014) trained with large amounts of human-generated conversation sessions (Shang et al., 2015a; Sordoni et al., 2015) , and this task is named as Neural Response Generation (NRG). Similar to the Neural Machine Translation (NMT) approaches (Bahdanau et al., 2014; Wu et al., 2016) , the deep Seq2Seq models are expected to directly generate appropriate and meaningful responses according to the input query. Compared to the success of NMT systems, the application progress of NRG models is not satisfying at present due to the "safe response" problem (Li et al., 2016) . That is, most of the generated responses are boring and meaningless, which blocks the continuation of conversations. Indeed, eliminating "safe responses" is the essential task of NRG models. Thus, various methods have been considered to address this problem (Li et al., 2016; Xu et al., 2017; Li et al., 2017; Xing et al., 2017; Pandey et al., 2018; Zhang et al., 2018a; Du et al., 2018) .

More recently, Generative Adversarial Nets (GAN) (Goodfellow et al., 2014) have been introduced to eliminate "safe responses" (Li et al., 2017; Xu et al., 2017) . Basically, this methodology is reasonable since the GAN framework involves an adversarial discriminator that helps NRG models leap out of the shortsighted state of minimizing the empirical risk on word distribution, by providing feedback on real samples from the model generated ones. Despite the improvement on the diversity, the adversarial training process of GAN based response generation models is generally unstable and sensitive to the training strategy (Yu et al., 2017) .

The unstable convergence problem is largely ascribed to the complicated data distribution in practical scenarios (Arora et al., 2017; Arora & Zhang, 2017) . For the response generation oriented GAN models, in particular, the data distribution appears to be much more complicated. Fundamentally, an essential characteristic of conversation data is that, for each given query, there always exists a group of semantically-diverse responses, rather than the semantically-unified ones. Furthermore, the response groups of two different queries tend to keep great divergences in the semantic space. In this scenario, the discriminator needs to consider the distributions of the generated result in the semantic space, rather than simply examining whether one single sample comes from the generator or the original dataset, so as to make the generator sense the distribution of conversation dataset in the adversarial training procedure.

This paper aims at presenting a specific adversarial training schema for neural response generation. Beginning with the investigation on the reason for the unsatisfying performance of GANs on the NRG task, we find the upper-bound of the current GAN learning strategies taking query-response pairs as the independent training samples. On this basis, we claim that the training schema, including the actual adversarial strategy and the overall loss function, should be re-defined to agree with the distribution of NRG training samples in the semantic space, rather than roughly adopting the GAN framework designed for generating images.

Consequently, we describe the distributional state of the given query and the corresponding responses with the free energy defined on the basis of Deep Boltzmann Machines (DBM) (Salakhutdinov & Hinton, 2009) . In this way, we can quantify the formation process of generating the response set with the topic restriction of the given query. From the perspective of free energy, this paper proposes a new cost function to measure the expansion degree of the responses in the local area of the real-valued semantic space. Cooperating with the traditional implicit density discriminating loss of GAN, the proposed cost actually provides an explicit density approximation for the local distribution of each response cluster. Thus, the adversarial learning procedure can be expected to be more stable with better response generation results obtained1 .

2. THE LIMITATION OF GENERAL GAN IN THE NRG SCENARIO:

According to (Goodfellow et al., 2014) , the standard GAN framework contains a generator G and a discriminator D, which are trained by an iterative adversarial learning procedure based on the following objective function:

J (D) = E x∼p d [log D(x)] + E z∼pz [log(1 -D(G(z)))]

(1)

J (G) = E z∼pz [log D(G(z))]

(2) where p z denotes the prior on input noise variables, and p d is the true data distribution.

It should be noted that the GAN tries to learn the manifold of a given dataset (Khayatkhoei et al., 2018; Kumar et al., 2017) , and the discriminator D actually provides a metric for judging whether the results generated according to z fits the expected manifold or not. In the NRG scenario, a naive Seq2Seq model without the guidance signal from D can not capture the data manifold of the real query-response corpus, which is one of the major facts the safe-response problem can be ascribed to. Assuming that there exists an oracle discriminator with the ability of distinguishing the generated fake samples from the ground-truth ones, by mapping each query-response pair (q, r)2 to a confidence score s, and meanwhile, it can be assume that any practically existing discriminator of GAN gives the confidence s to (q, r). If the practical discriminator can make s → s, the generator will obtain more meaningful guidance for the better generation. That is, to improve the capability of GAN-NRG, it is wise to construct more powerful discriminators for more reasonable J (G) . Now let's pay attention to the actual change of NRG models brought by GAN. In the generative conversation agent scenario, G(z) is corresponding to a generated response r to a given query q. Thus, the objective of the generator in the GAN based NRG model can be simply formulated as:

J (G) = E (q,r)∼pg [log D(q, r)]

(3)

The training of the generator is actually the procedure to maximize J (G) toward E (q,r)∼p d [log D(q, r)], so as to generate realistic responses according to given queries. Thus, in the context of adversarial learning, J (G) should satisfy the following inequality:

EQUATION

However, it is well known that a conversational dataset should not be simply taken as a collection {(q, r)} composed of independent query-response pairs. Instead, to each given query q, there exists a finite set of corresponding responses R q = {r i }. In this case, it is of great necessity to consider the whole training dataset as a collection of R q , which takes the form of a number of clusters with their own local distributions in the semantic space. And we can rewrite the joint distribution p d in (4) as p(q)p(r|q) and assume every corresponding response to a query follows equal-probability distribution, which means that p(r|q) = 1 |Rq| . Thus, in the real NRG scenario, on the basis of Equation 3 and Inequation 4, J (G) follows the inequality as below:

EQUATION

where p(q) denotes the probability of the query q, and R q is defined above.

According to Equation 5, the upper bound of J (G) is obtained, in which the log 1 |Rq| r∈Rq D(q, r) part is the essence. Apparently, the expression 1 |Rq| r∈Rq D(q, r) indicates the mean value of the confidence scores given by the discriminator to each member of the response set R q to a given query q. Moreover, it should be noted that current studies tend to utilize semantic relevance oriented models to build the discriminators of GAN (Xu et al., 2017; Li et al., 2017) . Consequently, D(q, r) can be actually considered as the spatial relationship of q and r in the semantic space. Thus, 1 |Rq| r∈Rq D(q, r) stands for the spatial center of all the responses within R q . That is, the optimization process of adversarial learning upon conversational datasets will make the generated responses approach to the center of each local distribution of R q corresponding to each given dependent query.

The practical value of this change lies in that, intuitively, the GAN architecture forces the generation to pay attention to the local distributions of the individual response clusters, rather than taking the (q, r)-pairs as an entirety. According to the thorough studies on the safe responses of NRG models (Li et al., 2016; Xu et al., 2017; Zhang et al., 2018a; Pandey et al., 2018) , it can be inferred that the general Seq2Seq will fall into the divergence state of generating the patterns with the maximum probabilities taking account of the entire dataset, ignoring the individual-difference of each query. By introducing the implicit loss focusing on the response clusters, GAN makes the divergence of the generator much closer to the 'local patterns' rather than the general patterns, and thus the higher diversity can be expected and observed (Xu et al., 2017; Li et al., 2017) .

The problem turns to: Is the upper bound in Equation 5 powerful enough? Apparently, there exists an obvious gap between the 'local patterns' and the vivid and interesting generated responses. The upper bound only focuses on the mean of each response set, but the local distribution (or the actual 'shape') of each cluster has not been taken into account. This situation does not change when the cost function of adversarial training is defined by Wasserstein GANs (WGAN) (Arjovsky et al., 2017) with 1-Lipschitz function f :

J (G) W = E (q,r)∼pg [f (q, r)] (6)

Intuitively, it is of paramount importance to estimate both the 'location' and the 'shape' of the response set R q in the semantic space (indeed, 1 |Rq| q,r D(q, r) is only relative with 'location'), so as to determine the optimization objective of adversarial training. Consequently, we have two critical problems to be discussed and addressed in the following sections:

• How to describe the state of the response set R q with a given query q in the semantic space?

• Taking account of the reasonable state modeling of (q, R q ), what is the loss function for adversarial training to generate responses?

As mentioned above, the semantic one-to-many relationship between queries and responses makes it necessary to model the local distribution of the response cluster R q corresponding to each query in the space, and it is paramount to turn to the fitting of each local distribution in the adversarial learning procedure, rather than considering each (q, r)-pair as an independent sample. Basically, this issue equals to the task of reasonably modeling the state of (q, R q ) in the semantic space, by considering each (q, R q ) as a systematic entirety and assigning the state of the entirety with probabilistic distribution. The additional major challenge of this task is, indeed, we have to infer the state of a local area in the semantic space from a group of finite samples, since it is impossible to sample all the possible responses to a query from the given corpus, regardless of the corpus size.

3.1. REPRESENTING LOCAL DISTRIBUTIONS WITH QUERY-RESPONSE ORIENTED FREE ENERGY:

In this part, we typically take an energy based statistical model, the Average Free Energy (Hinton & Zemel, 1994; Friston et al., 2006; Ngiam et al., 2011; Friston et al., 2012) , to describe the state of the local distribution of (q, R q ) in the semantic space, for the reasons that: a) energy based models are considered as a promising avenue towards learning explicit generative models (LeCun et al., 2006; Le Roux & Bengio, 2008) , by representing data distributions without any prior assumptions; and b) energy based models can be trained in the unsupervised way, and the energy functions of such models have the potential to estimate the state of generative models (Zhao et al., 2016) .

At first, the free energy of a given query-response pair (q, r) can be defined as following:

EQUATION

where E(q, r, H) stands for the energy function defined according to the relationship of the query q and its response r via the hidden variable H. We employ the Deep Boltzmann Machine (DBM) (Salakhutdinov & Larochelle, 2010; Smolensky, 1986; Hinton & Salakhutdinov, 2006) to implement E(q, r, H), as illustrated by Figure 1 . The reason for this choice lies in that, from the view of conversational agents, the meaningful query and the corresponding response are generally considered to maintain strong semantic relevance. Thus, the query and response can be mutually transformed into each other, which is supported by the considerable amount of studies on response generation (Shang et al., 2015b; Shao et al., 2017; Zhang et al., 2018a; Baheti et al., 2018) and question generation (Du et al., 2017; Zhao et al., 2018; Sun et al., 2018) . Without loss of generality, the pairwise semantic relationship of the query q and the corresponding response r can be modeled by a two-layer DBM. The bottom layer is actually an abstract version of Seq2Seq models, in which a response r can be generated based on a hidden variable h q , and h q depends on the given query q theoretically. In the top layer, h q conditionally depends on a hyper hidden variable h. Figure 1 illustrates the Deep Boltzmann Machine for modeling the semantic relationship of a query and its responses.

ℎ " ℎ 𝑞 𝑟 𝑤 "" 𝑤 "& 𝑤 "'

Following (Salakhutdinov & Hinton, 2009) , on the basis of the DBM in Figure 1 , the energy of the state {(q, r), H} is defined as:

E(q, r, H) = E (q, r, h q , h) = -h T q W qr r -h T q W qq q -h T W qh h q (8)

where q denotes the query, r stands for the response and H = {h q , h} represents the hidden units. W qr , W qq and W qh stand for the weights on the corresponding connections of the query, response and the hidden variables respectively in the graph model shown by Figure 1 .

Consequently, we can define the average free energy of the query q and its response set R q as follows:

F (q, R q ) = 1 |R q | ri∈Rq F (q, r i ) (9)

For better conducting the following discussion, we further define the energy difference between response r i and r j as:

Definition 3.1 (Scaled Energy Difference). ∆ q,ri,rj = F (q, r i ) -F (q, r j ) F (q, R q ) (10)

Meanwhile, it is necessary to assign a spatial intuition to R q in the semantic space by defining: Definition 3.2 (Response Cluster). In the semantic space, the meaningful responses to the given query q lie in a restricted region (e.g., a hyper sphere), which can be named as the Response Cluster, in which F (q, r) can be taken as the distance from a response r to the cluster center.

3.2 ESTIMATION OF F (q, R q )

Basically, the DBM in Figure 1 provides the definition of the energy function E(q, r, H) of the free energy given in Equation 8. Thus, the local distribution state of the responses R q with the given query q can be mathematically described by F (q, R q ) based on Equation 7-9. In practice, however, this procedure is not operable yet because the computation of F (q, R q ) requires all the response in R q , and it is impractical to perform the exhaustive enumeration over all the possible responses of the given query, regardless of the amount of the training query-response pairs. Consequently, it is highly necessary to approximate F (q, R q ) under some reasonable assumptions.

Considering the response cluster defined in Definition 3.2, we can naturally assume that the response random variable r to a given query q follows multivariate normal distribution with mean r c and covariance matrix Σ, where r c and Σ is only determined by query q. Afterwards, the observed R q can be considered as a realization of the |R q |-variate random variable (r 1 , • • • , r |Rq| ) for r i , 1 ≤ i ≤ |R q |, i.i.d random variables drawn from the distribution N (r c , Σ). Consequently, an executable approximation of F (q, R q ) can be obtained based on the following Lemmas and Theorem3 .

Lemma 1. If E[F (q, r)] < ∞, F (q, R q ) converges almost surely to the expected value E[F (q, r)],

as |R q | goes to infinity. That is,

EQUATION

Lemma 1 indicates that for any > 0 there exists N ( )

such that if |R q | > N ( ) then the inequality |F (q, R q ) -E[F (q, r)]| < holds.

In other words, F (q, R q ) can be very close to E[F (q, r)] if the responses are sufficient.

Lemma 2. lim E r-rc 2→0 |E[f (r)] -f (r c ) | = 0 (12)

Practically, the expected Euclidean distance between random variable r and r c can not be zero. Thus, the fact conveyed by Lemma 2 is that, actually, if the expected Euclidean distance is small enough, the difference between E[F (q, r)] and F (q, r c ) can be controllable (or even close to zero).

Based on the lemmas we have:

EQUATION

)

where rc is the estimation of r c based on the well-trained DBM.

proof of theorem 1.

|F (q, R q ) -F (q, rc )| ≤ |F (q, R q ) -E[F (q, r)]|+|E[F (q, r)] -F (q, r c )|+|F (q, r c ) -F (q, rc )| (14)

Following lemma 1, the first term converges almost surely to zero when |R q | → ∞.

And following lemma 2, the second term goes to zero when E r -r c 2 → 0.

Finally, given the well-trained DBM with the capability of mapping a query q to rc (Wang et al., 2010; Srivastava & Salakhutdinov, 2012) , rc can be taken as the estimation of r c . Meanwhile, since the function F (q, r) is the composition and combination of simple continuous functions, we have |F (q, r c ) -F (q, rc )| → 0.

4. THE HYBRID LOSS OF ADVERSARIAL RESPONSE GENERATION:

As discussed in Section 2, the ability of the response generator in the general GAN architecture is limited to learning the dense distribution around 1 |Rq| r∈Rq D(q, r) (see Equation 5), which is composed of the most frequent patterns in the semantic space. By contrast, it is difficult for the general architecture to sense the remaining sparse space containing high-quality diverse responses. Therefore, reasonably describing the local distribution of the responses to a given query is highly necessary. According to the analysis in Section 3, the average free energy can be taken to model the state of the local area of the responses to a query, and such energy can be reasonably approximated via the DBM defined on the query-response pairs. On the basis of the previous sections, this section will finally propose the new hybrid loss function to force the generator to produce responses with better diversity through the more stable adversarial training process.

4.1. THE RADIAL DISTRIBUTION FUNCTION OF THE RESPONSE:

The analysis in Section 3 have shown that the local distribution state of the responses R q to the given query q can be modeled by the average free energy F (q, R q ). On this basis, it is possible to propose the description of the spatial state of a single response r in the semantic space, and consequently, we can give a new adversarial loss indicating the cost of simulating the local distribution of R q .

According to Definition 3.1 and 3.2, in each response cluster R q , the distance from a response r to the cluster center r c is actually equivalent to the scaled energy difference between them, that is,

∆ q,r,rc = F (q, r) -F (q, r c ) F (q, R q ) (15)

Meanwhile, on the basis of Theorem 1, F (q, R q ) can be approximated by F (q, rc ), and rc is modeled from training data, and thus we have:

EQUATION

Here we approximate ∆ q,r,rc with α (q,r) , and formally call α (q,r) as the Radial Distribution Function (RDF), indicating the relative cost ratio to F (q, r c ) for obtaining r from a given q (also the distinctiveness of r, actually).

4.2. THE HYBRID OBJECTIVE FUNCTION:

Based on the previous discussions, for the adversarial response generation methodology, the essence is to reasonably describe the state of the local distribution of the response cluster given by Definition 3.2, and further more, to take this important element into account in the final optimization.

Especially, in Subsection 4.1, we have defined the Radial Distribution Function in Equation 16to quantify the distinctiveness of a response, the very basis of which is the description of the local state F (q, R q ) in the semantic space. Thus, we can further build a mechanism to quantify the difference between the generated response and the golden response as follows:

EQUATION

)

where r is the generated response given by the generator and r comes from the original data. If δα moves toward zero, α (q,r) would be close to α (q,r) sharing the same F (q, r c ).

Consequently, a new expectation comes out. That is, the generator needs to provide results that can minimize δα, so as to fit the local distribution of the existing responses to a given query. Thus, a hybrid objective of the generator can be finally defined as:

EQUATION

A hinge loss, conducted by the ReLU function ReLU (δα) = max(0, δα), is especially introduced to reform δα. The primary reason of this operation is that the ReLU function has positive output only if δα 0, according to the definition of ReLU (δα). Apparently, δα < 0 indicates that the generated response r is too far from the center of the response cluster in the semantic space, so that its relevance may be highly questionable. Meanwhile, minimizing a negative variable is against the optimization direction. After the ReLU transformation, there remains valid loss only when δα 0, and thus both the diversity and the relevance of generated results are taken into account.

4.3. THE PHASE-WISE OPTIMIZATION:

According to the analysis in Section 2, the trivial adversarial training directed by -E [log D(q, r)] can only determine the form of general responses to a given query. From the spatial perspective in the semantic space, the original adversarial objective is helpful to roughly locate the response cluster to be generated. However, the local distribution can not be captured by this procedure.

By contrast, according to the discussions above, the proposed hybrid objective actually provides a way to force the generated responses, originally gathering around the general form, to expand into the expected local shape described by the golden truth, by conducting a phase-wise optimizing operation. This mechanism can be detailed in an intuitive way:

Foundation: Once the DBM in Figure 1 is well-trained with the query-response corpus, the semantic center r c of a Response Cluster can be determined by the given query q.

Phase-1: In the early stage of the adversarial training, a generated response r is not semantically relevant to the query q. Thus, it can be inferred that r is radially farther from the cluster center r c than the golden response r. In this situation, according to Equation 16and Equation 17, we can claim that δα ≤ 0. In this phase, the hyper objective goes back to the general adversarial objective due to the ReLU function. Thus, the model is trying to force the generated samples to approach the center of each cluster, ignoring local distributions.

Phase-2: During the adversarial training in Phase-1, the generated result r will go approaching to the cluster center r c , which means α (q,r) → 0 . It should be noted that, for any meaningful existing training sample r, α (q,r) > 0. Therefore, at some point, it turns to δα > 0 and the right part of the hybrid objective in Equation 18 takes effect. Consequently, for each given query, the distribution of the generated results will expand to fit the local distribution of the golden samples.

5. EXPERIMENTS:



5.1. EXPERIMENTAL SETUPS:



Datasets. Our experiments are conducted on two main stream open-access conversation corpora::

The Opensubtitles corpus and the Sina Weibo corpus. The OpenSubtitles dataset contains 5,200,000 movie dialogues, where we extract query-response pairs following (Xu et al., 2018; Li et al., 2016) . The Sina Weibo Corpus (Shang et al., 2015a ) contains 2,500,000 single-turn Chinese dialogues, in which the length of the query and response ranges from 4 to 30. We sample 100,000, and 2,000 unique query-response pairs as validation and testing dataset respectively from both of the corpora4 .

Baselines. For meaningful comparison, we introduce the following models as the baselines:

(1) Seq2Seq: a sequence-to-sequence model trained with maximum likelihood estimation (MLE); (2) Seq2Seq-MMI: the NRG model with a Maximum Mutual Information criterion (Li et al., 2016) ;

(3) Adver-REGS: the NRG model trained using adversarial framework, in which the policy gradient was employed to transfer the reward of the discriminator to the generator (Li et al., 2017) ;

(4) GAN-AEL: an adversarial framework with an approximate embedding layer for connecting the generator with the discriminator directly (Xu et al., 2017) .

(5) AIM / DAIM: the adversarial training strategy allowing distributional matching of synthetic and real responses and explicitly optimizing a variational lower bound on pairwise mutual information between the query and response, so as to improve the informativeness and diversity of generated responses (Zhang et al., 2018b) 

Evaluation Metrics. To evaluate diversity of results, we adopt three widely-applied metrics: Distinct-1 (Dist-1), Distinct-2 (Dist-2), and Entropy (Ent4) (Li et al., 2016; Zhang et al., 2018b; Jost, 2006) . Besides, the relevance (Rel.) is measured by summing three embedding-based similarities (greedy, average, extreme) (Liu et al., 2016) upon the ground-truth and generated responses.

Training Details. The vocabulary size of both datasets is 40,000. The embedding layer of OpenSubtitles and Sina Weibo is initialized using 200-dimensional Glove vectors (Pennington et al., 2014) and 300-dimensional Weibo vectors (Li et al., 2018) respectively. All the models are first pre-trained by MLE, and then the models including Adver-REGS, GAN-AEL, AIM, DAIM and LocalGAN are trained with adversarial learning. The discriminator of Adver-REGS and GAN-AEL are based on CNN following (Yu et al., 2017; Xu et al., 2017) , in which the filter sizes are set to (1,2,3,4) and the filter number is 128, while that of LocalGAN adopts DBM with (2×embedding size, 128, 128) to represent the semantic of queries and responses. The hidden size of the generator is set to 256 and 512 in GAN-based models and Seq2Seq respectively. To guarantee the performance consistency of AIM and DAIM, we adopt the recommended parameter settings given by Zhang et al. (2018b) . The experiments are conducted on the Tesla K80 GPU.

5.2. RESULTS & ANALYSIS:

Table 1 lists quantitative results on the diversity and relevance of generated responses on both datasets. As shown by the results, compared to Seq2Seq and Seq2Seq-MMI, the GAN-based methods give better results on the diversity oriented metrics, including Dist-1, Dist-2 and Ent4. This observation indicates that adversarial learning does provide the meaningful guidance to NRG models to avoid some of the safe-responses.

It can be observed that LocalGAN outperforms the baselines with adversarial learning architecture (Adver-REGS, GAN-AEL, AIM and DAIM) on both the diversity metrics and the relevance metrics.

Generally, a notable improvement on diversity may lead to some negative influence on relevance, and thus promoting the diversity of generated response while maintaining their relevance is essentially desired for any methodologies, which has been achieved by our LocalGAN. The performances of LocalGAN can be attributed to the fact that LocalGAN has taken the local distribution of responses to a given query into account. By adopting the hybrid objective function, the proposed adversarial model gets to capture the spatial characteristics of response clusters, and the generation process is consequently forced to fit the semantic distributions of response clusters.

The training stability is a tough issue to be addressed for adversarial learning (Yu et al., 2017) , and as discussed in the previous sections, one of the motivations of our LocalGAN is to make adversarial learning more stable. To verify this aspect, we track the changing of the Entropy (Ent4) of results given by GAN-AEL, Adver-REGS, AIM, DAIM and LocalGAN, as shown in Figure 2 .

It can be observed that the training of LocalGAN and Adver-REGS is stable. By contrast, there exist obvious fluctuations on the curves of AIM and DAIM, and GAN-AEL rapidly gets out of control after 1000 batch, which makes it rather difficult to grasp the models with the best status. This group of results indicates the necessity of introducing additional restrictions into adversarial learning processes. For this purpose, Adver-REGS introduces a teacher-forcing loss (Li et al., 2017) , while AIM and DAIM have taken the informativeness oriented constraints to partially control the stability (Zhang et al., 2018b) . However, GAN-AEL only takes the Wasserstein distance as the objective (Xu et al., 2017) , and thus the entropy goes down rapidly. Compared to the Adver-REGS, our LocalGAN achieves better diversity with even a more smooth entropy curve. The training of LocalGAN benefits from the phase-wise optimization driven by the hybrid loss, and its stability also indicates the meaningfulness of modeling and utilizing local distributions of responses. To further conduct intuitive comparisons among the NRG models, we perform human evaluations on 500 testing samples. Five annotators are asked to judge whether a response is relevant to the given query and whether the response is informative or not respectively. Both human metrics "Relevance" and "Informativeness" solely have two labels 0 and 1 (0: irrelevant or non-informative; 1: relevant or informative) for simplicity. The human evaluation results are listed in Table 2 , and the results basically match the observations in Table 1 . The Kappa of relevance and informativeness annotations are 0.734 and 0.561 respectively.

5.3. HUMAN EVALUATION:



5.4. CASE STUDY:

The effectiveness of the NRG models can be illustrated by the responses generated by them intuitively. For this purpose, this paper adopts standard-scaler6 to remove the mean and scale to unit variance. To valid the effectiveness of standard-scaler, We conduct experiments on the queryresponse matching task using normalized vectors. The experiment result show that the matching performance based normalized vectors is similar to that of CNN based architecture (Kim, 2014) .

B DETAILED PROOF OF LEMMAS:

Proof of Lemma 1. As mentioned before, r i is assumed to be independent and identically distributed (i.i.d) normal variable, thus F (q, r i ) can be seen as i.i.d random variable for a fixed query q. Based on the strong law of large numbers,

F (q, R q ) = 1 |R q | ri∈Rq F (q, r i ) a.s. -→ E[F (q, r)] when |R q | → ∞

Proof of Lemma 2. For a fixed query, F (q, r) can be seen as the the scalar function of vector r. For simplicity, we denote F (q, r) as f (r).

Taylor expansions for the first moment of function of random variables are as follows.

E[f (r)]] = E [f (r c + (r -r c ))] = E f (r c ) + (r -r c ) T Df (r c ) + 1 2 (r -r c ) D 2 f (r c ) (r -r c ) + R rc,2 (r -r c )

where Df (r c ) is the gradient of f evaluated at r c , D 2 f (r c ) is the Hessian matrix and R rc,2 (r -r c ) is the Lagrange remainder. Since Er = r c , the second term (r -r c ) T Df (r c ) disappears.

Then we will try to find the upper bound of the third term and the remainder term. The relevant theorems used in the proof are listed as follows.

• According to (Petersen & Pedersen, 2012) , assume A is symmetric, c = E[x] and Σ = Var[x], then E x T Ax = Tr(AΣ) + c T Ac.

• (Mirsky, 1975) Meanwhile, according to the Corollary 1 in (Folland, 2005) , we have that

|R rc,2 (r -r c )| ≤ M 3! r -r c 3 1

where M is the upper bound for absolute value of third-order partial derivatives of f .

Next, we show that the α 1 and third-oder partial derivatives can be bounded by M , where M is max hq,1≤i≤n |b(h q , i)| (h q follows multinomial distribution) and n is the dimension of r.

Substituting the definition of E(q, r, H) into F (q, r), we have following equation:

f (r) = F (q, r) = -log hq,h

exp(h T q W qr r + h T q W qq q + h T W qh h q ).

After that, its first-order, second-order and third-order partial derivative are calculated as follows:

∂f (r) ∂r i

= -hq,h exp(h T q W qr r + h T q W qq q + h T W qh h q ) × (h T q W qr ) i hq,h exp(h T q W qr r + h T q W qq q + h T W qh h q )

∂ 2 f (r) ∂r i ∂r j = hq,h a(h q , h) × b(h q , i) × hq, h a( hq , h) × b( hq , j) -b(h q , j)

∂ 3 f (r) ∂r i ∂r j ∂r k = hq,h c(h q , h, k) × b(h q , i) × hq, h a( hq , h) × b( hq , j) -b(h q , j)

+ hq,h a(h q , h) × b(h q , i) × hq, h c( hq , h, k) × b( hq , j)

where a(h q , h) = exp(h T q W qr r + h T q W qq q + h T W qh h q ) hq h exp( hT q W qr r + hT q W qq q + hT W qh hq ) c(h q , h, k) = ∂a(h q , h) ∂r k

= a(h q , h)b(h q , k) -a(h q , h) hq, h a( hq , h)b( hq , k)

and b(h q , i) = (h T q W qr ) i representing the i-th element of the vector h T q W qr . According to the definition of a(h q , h), it is obvious that a(h q , h) > 0 and hq,h a(h q , h) = 1.

The upper bounds for the second-order and third-order partial derivative are shown as follows. 

∂ 2 f ∂r i ∂r j ≤ hq,h |a(h q , h)| × |b(h q , i)| × hq, h a( hq , h) × b( hq , j) + |b(h q , j)| ≤ 2M 2 |c(h q , h, k)| ≤ |a(h q , h)| × |b(h q , k)| + |a(h q , h)| ×

:

The code of our proposed model LocalGAN can be found in https://github.com/Kramgasse49/ local_gan_4generation

Here q and r represent the vectorized query and its response. We take the simple embedding-averaging based method to transform texts into vectors. Besides, due to our adopted text vectorizing method, the pretraining phase of the Deep Boltzmann Machine takes some specified trick detailed in Appendix A.

The proof of Lemma 1, Lemma 2 and Theorem 1 is given in Appendix B.

Both the English and the Chinese datasets used in our experiments are uploaded to https://www. dropbox.com/sh/k8i079gd2111lsb/AACLLtlNAzile543Da8Qs9tFa?dl=0.

We have taken the codes of AIM and DAIM from https://github.com/dreasysnail/ converse_GAN implemented by the authors of this work for comparisons.

https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html

Figures and Tables:

Figure 1: The DBM for modeling the semantic relationship of Query-Response pairs.

Figure 2: The Entropy Trend of adversarial learning based models in the Training Process.

states following theorem: If A, B are complex n × n matrices with singular valuesα 1 ≥ α 2 ≥ • • • ≥ α n and β 1 ≥ β 2 ≥ • • • ≥ β n respectively, then | Tr(AB)| ≤ n i=1 α i β iFirstly, since r-r c ∼ N (0, Σ) and Σ is positive semi-definite matrix, the third term can be simplified as follows.|E[(r -r c ) D 2 f (r c ) (r -r c )]| = | Tr({D 2 f (r c )}Σ) + 0 D 2 f (r c )0| ≤ n i=1 α i β i ≤ α 1 Tr(Σ) = α 1 E r -r c ] 2 2 where α 1 ≥ α 2 ≥ • • • ≥ α n and β 1 ≥ β 2 ≥ • • • ≥ β n denotethe singular value of matrix {D 2 f (r c )} and Σ respectively.

hq , h)| × |b( hq , k)| ≤ 2M |a(h q , h)| | ∂ 3 f (r) ∂r i ∂r j ∂r k | ≤ hq,h |c(h q , h, k)| × M × [M + M ] + M 2 hq, h |c( hq , h, k)| ≤ 6M 3Based on the upper bound above, we can see that M can be 6M 3 and the upper bounds for α 1 and |R rc,2 (r -r c )| are as follows.α 1 = σ max ({D 2 f (r c )}) ≤ {D 2 f (r c )} F = ∂r i ∂r j | 2 ≤ 2nM 2 |R rc,2 (r -r c )| ≤ (r)] -f (r c ) | = | 1 2 E (r -r c ) D 2 f (r c ) (r -r c ) + E R rc,2 (r -r c ) | ≤ 1 2 α 1 E r -r c ] 2 2 + E|R rc,2 (r -r c )| ≤ E( √ nM r -r c ] 2 ) 2 + E( √ nM r -r c 2 ) 3 Therefore, lim E r-rc 2→0 |E[f (r)] -f (r c ) | = 0

Performances of LocalGAN and Baselines on the Opensubtitles and Weibo Datasets.

Table content: <table><tr><td/><td/><td colspan="2">Opensubtitle</td><td/><td/><td>Weibo</td></tr><tr><td>Model</td><td>Dist-1</td><td colspan="2">Dist-2 Ent4</td><td>Rel.</td><td colspan="2">Dist-1 Dist-2 Ent4</td><td>Rel.</td></tr><tr><td>Seq2Seq</td><td>0.025</td><td>0.081</td><td colspan="3">5.650 1.090 0.055</td><td>0.153</td><td>6.400 0.315</td></tr><tr><td colspan="2">Seq2Seq-MMI 0.027</td><td>0.086</td><td colspan="3">5.698 1.067 0.059</td><td>0.172</td><td>6.860 0.309</td></tr><tr><td>Adver-REGS</td><td colspan="2">0.0296 0.098</td><td colspan="3">5.701 1.113 0.061</td><td>0.181</td><td>7.658 0.320</td></tr><tr><td>GAN-AEL</td><td>0.030</td><td>0.100</td><td colspan="3">5.733 1.106 0.062</td><td>0.183</td><td>7.765 0.318</td></tr><tr><td>AIM</td><td colspan="2">0.0292 0.095</td><td colspan="3">5.783 1.120 0.064</td><td>0.189</td><td>7.833 0.321</td></tr><tr><td>DAIM</td><td>0.031</td><td>0.103</td><td colspan="3">5.873 1.098 0.067</td><td>0.195</td><td>8.042 0.316</td></tr><tr><td>LocalGAN</td><td>0.036</td><td>0.110</td><td colspan="3">6.073 1.132 0.071</td><td>0.212</td><td>8.561 0.327</td></tr></table>

Human Evaluation Results of Models.

Table content: <table/>

Table3lists the sampled results given by all the baselines and our proposed LocalGAN, trained on the OpenSubtitle corpus (The samples based on the Sina Weibo dataset are given in Appendix C). It can be observed that, compared to the basic Seq2Seq and Seq2Seq-MMI, the models incorporating adversarial learning have generated results with varying degrees of semantic diversity,A THE GRAPHICAL MODEL FOR RESPONSE DISTRIBUTION MODELINGDifferent from the computer vision related scenario, training a DBM on a set of text vectors is not trivial, since the training procedure is difficult to converge due to the value scale of text vectors is much larger than image vectors. Moreover, the simple scaling methods are not effective enough for this issue.

Table content: <table/>

Peer review (iclr2020):
In this paper, the author proposed a model to address the training instability of a GAN model on the NRG task. The authors take advantage of an energy based model to measure the distance between a predicted response and the center of all qualified responses. The training process becomes a hybrid one with the original loss function in the beginning followed by the loss that pulls the response to the center of the response cluster later. In general the paper is well written, with experiments clearly showcased improved training stabilities. However, one major flaw in the experiments in that the authors almost only compared diversity measures such as Dist-1, Dist-2 and Ent4. These measures did not take into consideration the matches between the predicted one and the ground truth. The only relevance measure used in this paper is the Rel., which the authors defined as the average of embedding distances. Such a measure is by no means an objective measure and can*t really demonstrate the effectiveness of the model in terms of generating responses that are close to the ground truth. The authors would need to submit results on one of the widely adopted benchmark metrics (e.g., BLEU, ROGUE) or their equivalents in order to demonstrate the quality of the generated response. And this is the main reason of my rating recommendation.