LocalGAN: Modeling Local Distributions for Adversarial Response Generation
Authors: Zhen Xu
Baoxun Wang
Huan Zhang
Kexin Qiu
Deyuan Zhang
Chengjie Sun
Venue:  (2020)
Keywords: neural response generation
adversarial learning
local distribution
energy-based distribution modeling

Abstract: This paper presents a new methodology for modeling the local semantic distribution of responses to a given query in the human-conversation corpus,  and on this basis, explores a specified adversarial learning mechanism for training Neural Response Generation (NRG) models to build conversational agents. The proposed mechanism aims to address the training instability problem and improve the quality of generated results of Generative Adversarial Nets (GAN) in their utilizations in the response generation scenario.  Our investigation begins with the thorough discussions upon the objective function brought by general GAN architectures to NRG models, and the training instability problem is proved to be ascribed to the special local distributions of conversational corpora.  Consequently, an energy function is employed to estimate the status of a local area restricted by the query and its responses in the semantic space,  and the mathematical approximation of this energy-based distribution is finally found.  Building on this foundation, a local distribution oriented objective is proposed and combined with the original objective, working as a hybrid loss for the adversarial training of response generation models, named as LocalGAN.  Our experimental results demonstrate that the reasonable local distribution modeling of the query-response corpus is of great importance to adversarial NRG, and our proposed LocalGAN is promising for improving both the training stability and the quality of generated results.


Under review as a conference paper at ICLR 2020
LOCALGAN: MODELING LOCAL DISTRIBUTIONS FOR ADVERSARIAL RESPONSE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents a new methodology for modeling the local semantic distribu- tion of responses to a given query in the human- conversstoonnni corpus, and on this basis, explores specified adversarial learning mechanism for training Neural Re- sponse Generation (NRG) models to build conversational agents. The proposed mechanism aims address the training instability problem and improve the qual- ity of generated results Generative Adversarial Nets (GAN) in their utilizations in the response generation scenario. Our investigation begins with the thorough discussions upon the objective function brought by general GAN architectures NRG models, and the training instability problem is proved to be ascribed the special local distributions of conversational corpora. Consequently, an en- ergy function employed estimate the status of local area restricted by the query and its responses in the semantic space, and the mathematical approxima- tion of this energy-based distribution is finally found. Building on this foundation. a local distribution oriented objective is proposed and combined with the original objective, working as a hybrid loss for the adversarial training of response genera- tion models, named LocalGAN. Our experimental results demonstrate that the reasonable local distribution modeling of the query-response corpus is of great importance to adversarial NRG, and our proposed LocalGAN is promising for improving both the training stability and the quality of generated results.
INTRODUCTION
End-to-End generative conversational agents (a.k.a., generative Chat-bots) are believed to be prac- ticable on the basis of the Sequence-to-Sequeeee (Seq2Seq) architecture (Sutskever et al.) 2014) trained with large amounts of auman-generated conversation sessions (Shang et al. 2015a 5al Sordoni
ce al. 2015), and this task is named Neural Response Generation (NRG). Similar ar the Neural Machine Translation (NMT) approaches (Bahdanau et al. 2014 Wu et al. 2016), the deep Seq2Seq models expected to directly generate appropriate and meaningful responses according to the in- put query. Compared to the success of NMT systems, the application progress of NRG models is not satisfying present due the "safe response" problem (Li et al. 2016). That is, most of the generated responses are boring and meaningless. which blocks the continuation of conversations. Indeed, eliminating "safe responses" is the essential task of NRG models. Thus, various methods
have been considered to address this problem (Li et al. 2016 Xu et al. 2017 Li ct al., 2017 Xing
et al. 2017 Pandey et al. 2018 Zhang et al., 2018a at. 2018).
More recently, Generative Adversarial Nets (GAN) (Goodfellow et al. 2014) have been introduced eliminate "safe responses" (Li et al., 2017 Xu al.
(017). Basically, this methodology is reasonable since the GAN framework involves an adversarial discriminator that helps NRG models leap out of the shortsighted state of minimizing the empirical risk on word distribution, by providing feedback real samples from the model generated ones. Despite the improvement on the diversity, the adversarial training process GAN based response generation models is generally unstable and sensitive to the training strategy YYu et al. 2017)
The unstable convergence problem is largely ascribed to the complicated data distribution in practi- cal scenarios (Arora et al., 2017 Arora & Zhang 2017). For the response generation oriented GAN models, in particular, the data distribution appears to be much more complicated. Fundamentally, essential characteristic of conversation data is that, for each given query, there always exists

---

Under review as a conference paper at ICLR 2020
group of semantically-divee responses, rather than the semantically- uniiied ones. Furthermore, the response groups of two different queries tend to keep great divergences in the semantic space. In this scenario, the discriminator needs to consider the distributions of the generated result in the semantic space, rather than simply examining whether one single sample comes from the generator the original dataset, so to make the generator sense the distribution of conversation dataset in the adversarial training procedure.
This paper aims at presenting a specific adversarial training schema for neural response generation. Beginning with the investigation on the reason for the unsatisfying performance of GANs on the NRG task, we find the upper-bound of the current GAN learning strategies taking query-respons' pairs as the independent training samples. On this basis, we claim that the training schema, including the actual adversarial strategy and the overall loss function, should be re-defined to agree with the distribution of NRG training samples in the semantic space, rather than roughly adopting the GAN framework designed for generating images.
Consequently, we describe the distributional state of the given query and the corresponding sponses with the free energy defined on the basis of Deep Boltzmann Machines (DBM) (Salakhutdi- nov & Hinton] 2009). In this way, we can quantify the formation process of generating the response set with the topic restriction of the given query. From the perspective of free energy, this paper proposes new cost function to measure the expansion degree of the responses in the local area of the real-valued semantic space. Cooperating with the traditional implicit density discriminating loss of GAN, the proposed cost actually provides explicit density approximation for the local distribution of each response cluster. Thus, the adversarial learning procedure can be expected to be more stable with better response generation results obtained
2 THE LIMITATION OF GENERAL. GAN IN THE NRG SCENARIO
According to (Goodfellow et al.) 2014), the standard GAN framework contains generator G and a discriminator D, which are trained by an iterative adversarial learning procedure based on the following objective function:
j(D) = Exnpa [log D(x)] + Ezmp, [log(1 D(G(=)))
( = Ez-p, [log D(G(z))]
where P= denotes the prior on input noise variables, and is the true data distribution.
should noted that the GAN tries learn the manifold of a given dataset (Khayatkhoei et al, 2018 Kumar et al. (017)) and the discriminator D actually provides a metric for judging whether the results generated according to z fits the expected manifold or not. In the NRG scenario, naive Seq2Seq model without the guidance signal from D can not capture the data manifold of the real query-response corpus, which one of the major facts the safe-response problem can be ascribed to. Assuming that there exists oracle discriminator with the ability of distinguishing the generated fake samples from the ground-truth ones, by mapping each query-response pair (q. r confidence score 8, and meanwhile, it can be assume that any practically existing discriminator of GAN gives the confidence & to (g.r). the practical discriminator can make 3 + 8, the generator will obtain more meaningful guidance for the better generation. That is, to improve the capability of GAN-NRG, it is wise to construct more powerful discriminators for more reasonable (
Now let's pay attention to the actual change of NRG models brought by GAN. In the generative conversation agent scenario, G(z) is corresponding to generated response i to given query q. Thus, the objective of the generator in the GAN based NRG model can be simply formulated as:
j( B(g,r)<,, (log D(q,
The training of the generator is actually the procedure to maximize J(C) toward B(grr>y [log ((g, r)]. so as to generate realistic responses according to given queries. Thus, in
'The code of proposed model LocalGAN can be found found intttttst:////ttttttt com. wtt.:///aaaaaae9//// local Baraal__nn_____ sa__AAgnneatiion
'Here q and represent the vectorized query and its response. We take the simple embedding--vvrrggng based method to transform texts into vectors. Besides, due to our adopted text vectorizing method, the pre- training phase of the Deep Boltzmann Machine takes some specified trick detailed in Appendix [
2

---

Under review as a conference paper at ICLR 2020
the context of adversarial learning, J(C) should satisfy the following inequality:
j[G) & B(g.))ppa [log (gg,r))
(4)
However, it well known that a conversational dataset should not be simply taken as a collection rg) composed of independent query-response pairs. Instead, to each given query q. there exists a finite set of corresponding responses Rg = {ra}. J. In this case, is of great necessity to consider the whole training dataset as collection of Rg- which takes the form of a number of clusters with their own local distributions in the semantic space. And we can rewrite the joint distribution Pa in (4) as p(g)p(rlq) and assume every corresponding response to a query follows equal-probability distribution, which means that p(rlq) =
R.. Thus, in the real NRG scenario, on the basis of
Equation and Inequation 4 j(G) follows the inequality as below:
j(G]
Blar DogD(g.)))
(5)
1 D(a [Ral rER,
Σ
p(a) [log D(g,r)] & Er(e) log
RRg
FER,
where p(4) denotes the probability of the query q, and R2 is defined above.
According Equation 5 the upper bound of J(G) is obtained, in which the log [RT Eren, D(g,r) part is the essence. Apparently, the expression RR Eren, P(g.r) indicates the mean value of the confidence scores given by the discriminator to each member of the response set Rg to given query q. Moreover, it should be noted that current studies tend to utilize semantic relevance oriented models to build the discriminators of GAN (Xu et al.) 2017 Li et al.] 2017). Consequently, D(g,r) r) can be actually considered the spatial relationship of q and in the semantic space. Thus, [Rty] Ere R2 D(g, r) stands for the spatial center of all the responses within Rg That is, the optimization process of adversarial learning upon conversational datasets will make the generated responses approach to the center of each local distribution of R2 corresponding to each given dependent query.
The practical value of this change lies in that, intuitively, the GAN architecture forces the generation to pay attention to the local distributions of the individual response clusters, rather than taking the (q, r)-pairs as an entirety. According to the thorough studies on the safe responses NRG mod- els (Li al., al.][2116 (2016; Xu et al. [017] ZZhng et al., 2018a) 8a Pandey et al.) 2018), it can be inferred that the general Seq2Seq will fall into the divergence state of generating the patterns with the maximum probabilities taking account of the entire dataset, ignoring the andividua-ddiffernndd of each query. By introducing the implicit loss focusing on the response clusters, GAN makes the divergence of the generator much closer to the 'local patterns' rather than the general patterns, and thus the higher diversity can be expected and observed (Xu et al. 2017: Li et al. 2017).
The problem turns to: Is the upper bound in Equation powerful enough? Apparently, there exists an obvious gap between the 'local patterns' and the vivid and interesting generated responses. The upper bound only focuses on the mean of each response set, but the local distribution (or the actual 'shape') of each cluster has not been taken into account. This situation does not change when the cost function of adversarial training is defined by Wasserstein GANs (WGAN) (Arjovsky et al. 2017) with I-Lipschitz function f:
J E(g.))p,, (f(q.
(6)
Intuitively, it is of paramount importance to estimate both the 'location' and the 'shape' of the response set Rg in the semantic space (indeed, [2,] Egrr D(g.r) is only relative with 'location") )) so as to determine the optimization objective of adversarial training. Consequently, we have two critical problems to be discussed and addressed in the following sections:
How to describe the state of the response set R, with given query in the semantic space?
Taking account of the reasonable state modeling of (q, R.y), what is the loss function for adversarial training to generate responses?
3

---

Under review as a conference paper at ICLR 2020
3 MODELING THE STATE OF THE LOCAL DISTRIBUTION FOR RESPONSES
As mentioned above, the semantic one-to-many relationship between queries and responses makes necessary to model the local distribution of the response cluster R. corresponding to each query in the space, and it is paramount to turn to the fitting of each local distribution in the adversarial learning procedure, rather than considering each (g. -aprrr as an independent sample. Basically, this issue equals to the task of reasonably modeling the state of (g. Ry) in the semantic space, by considering each (4, R,)) as a systematic entirety and assigning the state of the entirety with probabilistic distribution. The additional major challenge of this task is, indeed, we have infer the state of a local area in the semantic space from a group of finite samples, since it is impossible to sample all the possible responses to a query from the given corpus, regardless of the corpus size.
3.1 REPRESENTING LOCAL DISTRIBUTIONS WITH QUERY-RESPONSE ORIENTED FREE ENERGY
In this part, we typically take an energy based statistical model, the Average Free Energy (Hinton &
Zemel 1994; Friston et al. 2006; Ngiam et al. 2011 Friston et al. 2012). to describe the state of the local distribution of (q. Rgg) in the semantic space, for the reasons that: a) energy based models are considered as a promising avenue towards learning explicit generative models (LeCun et al.) 2006 Le Roux & Bengio, 2008), by representing data distributions without any prior assumptions: and b) energy based models can be trained in the unsupervised way, and the energy functions of such models have the potential estimate the state of generative models (Zhao et al. 2016).
At first, the free energy of a given query-response pair (4, r) can be defined as following:
F(g. r) loog Σ Exp((F(g,r, H))
where E(g,r, r, H) stands for the energy function defined according to the relationship of the query and its response via the hidden variable H.
We employ the Deep Boltzmann Machine (DBM) (Salakhutdinov & Larochelle, 2010; Smolensky
1986 Hinton & Salakhutdinov] 2006) to implement
h
E(g,r, H). as illustrated by Figure ! The reason for this choice lies in that, from the view of conversational agents, the meaningful query and the corresponding response are generally considered to maintain strong semantic relevance. Thus, the query and response can mutually transformed into each other, which is supported by the considerable amount of studies on response gener- ation (Shang et al. 22015b Shao et al. 2017 Zhang et al. 2018a Baheti et al. 2018) and question generation (Du et 2017 Zhao et al. 2018 Sun et al. 2018). Without loss of generality, the pairwise semantic relationship of the query q and the corresponding response can be modeled by a two-layer DBM. The bottom layer is actually an abstract version of Seq2Seq models, in which a response can be generated based on hidden variable hyg and h2 depends on the given query g theoretically. In the top layer, hy conditionally depends on hyper hidden variable h. Figure [ illustratee the Deep Boltzmann Machine for modeling the semantic relationship of query and its responses.
Wah
hq
Weg
War
q
r
The DBM for modeling the semantic relationship of Query- Response pairs.
Figure 1:
Following (Salakhutdinv & Hinton 2009), on the basis of the DBM in Figure I the energy of the state ((g, r), HJ is defined as:
Elg.r, r, H) = E (g,r, h) = -h g h W Wogg h? Wyhhh
(8)
where q denotes the query, stands for the response and H (h., h} represents the hidden units. WT W ag and WS, stand for the weights on the corresponding connections of the query, response and the hidden variables respectively in the graph model shown by Figure
4

---

Under review as a conference paper at ICLR 2020
Consequently, we define the average free energy of the query q and its response set Rg as follows:
F(g, R₂j = Σ
F(g, ri)
Rg
HER
For better conducting the following discussion, we further define the energy difference between response Ty and rj as: Definition 3.1 (Scaled Energy Difference).
piriiss F(q. F(g,r;) F(g, R₂y
(10)
Meanwhile, is necessary assign spatial intuition to Rg in the semantic space by defining:
Definition 3.2 (Response Cluster). In the semantic space, the meaningful responses to the given query q lie in restricted region (e.g.. hyper sphere), which can be named as the Response Cluster, which F(g, r) can taken as the distance from a response r to the cluster center.
3.2 ESTIMATION OF Fgg, Rg)
Basically, the DBM in Figure 1 provides the definition of the energy function Elg.r, r, H) of the free energy given in Equation 8 Thus, the local distribution state of the responses R2 with the given query can be mathematically described by F(g. R2y based on Equation 7 7-2 In practice, however, this procedure is not operable yet because the computation of Fg.. R2y) requires all the response in Rg. and it is impractical to perform the exhaustive enumeration over all the possible responses of the given query, regardless of the amount of the training query-response pairs. Consequently, highly necessary to approximate F(q, R₂) under some reasonable assumptions.
Considering the response cluster defined in Definition 3.2 we can naturally assume that the response random variable to a given query follows multivariate normal distribution with mean re and covariance matrix Σ, where Te and Σ is only determined by query g. Afterwards, the observed Rg can be considered i < [R i.i.d random variables drawn from the distribution N (rr,, Σ). Consequently, executable approximation of Fqq, R,)) can be obtained based on the following Lemmas and Theoren |
realization of the | -variate random variable (r1, .(iz))) for <
Lemma 1. If E[F(F(g, r]] < oc, Flq. F(g. converges almost surely to the expected value [[F as RRyl goes infinity. That is,
P( lim 1
F(g, R₂) ((F(g,r))) = 1.
(11)
Lemma Iindicats that for any e there exists N (e) such that if R2y N(e) then the inequality [g. RG (F. r)] < holds. In other words, F(g, R2)) can be very close to ((. if the responses are sufficient.
Lemma 2.
lim
-f
(12)
B Fella +0
Practically, the expected Euclidean distance between random variable r and r. can not be zero. Thus, the fact conveyed by Lemma [ that, actually, if the expected Euclidean distance is small enough, the difference between [E[ r)] and F(g, re) can be controllable (or even close to zero).
Based the lemmas have:
Theorem 1. Given E[F(q,r)] < 20, have
F(gg. fc). R.4
when [Ryl and E|ir rella - 0
(13)
where f.. is the estimation of based on the well-trained DBM.
"hh proof of Lemma
Lemma 2 and Theorem given in Appendix

---

Under review as a conference paper at ICLR 2020
proof of theorem 1
- F(), & [F R4) [ -F Flq, fo) (14)
Following lemma]] 1 the first term converges almost surely to zero when [Ry + 20.
And following lemma the second term goes to zero when E|r rell2 0.
Finally, given the well-trained DBM with the capability of mapping query to f. (Wang et al, 2010 Srivastava & Salakhutdinov 2012), Pe can be taken the estimation of r.. Meanwhile, since the function Fgg, r) is the composition and combination of simple continuous functions, we have (F(g,r)) F(g, Fe) 0.
THE HYBRID LOsS OF ADVERSARIAL. RESPONSE GENERATION
As discussed in Section 2 the ability of the response generator in the general GAN architecture is limited to learning the dense distribution around Rgl ErER, D r) (see Equation 5 which is composed of the most frequent patterns in the semantic space. By contrast, it is difficult for the general architecture to sense the remaining sparse space containing high-quality diverse responses. Therefore, reasonably describing the local distribution of the responses to given query is highly necessary. According to the analysis in Section 3 the average free energy can be taken to model the state of the local area of the responses via the DBM defined on the query-response pairs. On the basis of the previous sections, this section will finally propose the new hybrid loss function force the generator produce responses with better diversity through the more stable adversarial training process.
query, and such energy can reasonably approximated
4.1 THE RADIAL DISTRIBUTION FUNCTION OF THE RESPONSE
The analysis in Section have shown that the local distribution state of the responses Rg to the given query q can be modeled by the average free energy F(g, R2). On this basis, it is possible propose the description the spatial state of a single response r in the semantic space, and consequently, we
give new adversarial loss indicating the cost of simulating the local distribution of Rg.
According to Definition 3.1 and 3.2 in each response cluster R.g. the distance from a response r to the cluster center v. is actually equivalent to the scaled energy difference between them, that
A qurire F(q.r) -F(gTTe)
(15)
F(q, Rgy
Mcanwhile, the basis of Theorem I F(g, Rgy can be approximated by F(g, and f. is mod- eled from training data, and thus we have:
A A F(g.r) F(g. fe)
(g(g.))
(16)
F()
Here approximate Agrar with ((gr)) and formally call (((a.)) as the Radial Distribution Func- tion (RDF), indicating the relative cost ratio to F(g.re) for obtaining from given 9 (also the distinctiveness actually).
4.2 THE HYBRID OBJECTIVE FUNCTION
Based the previous discussions, for the adversarial response generation methodology, the essence is to reasonably describe the state of the local distribution of the response cluster given by Defini- tion 3.2] and further more, take this important element into account in the final optimization.
Especially, in Subsection 4.1 we have defined the Radial Distribution Function in Equation I6 to response, the very basis of which is the description of the local state
quantify the distinctiveness F(q, Rg) in the semantic space. Thus, we can further build mechanism to quantify the difference between the generated response and the golden response as follows:
5a (g.r)) Ok(g.F)
(17)

---

expectation comes out. That is, the generator needs to provide results that can Under review as a conference paper at ICLR 2020
where F is the generated response given by the generator and r comes from the original data. If da moves toward zero, (().F) would be close to ((g.)) sharing the same Fe).
Consequently, minimize da, so as to the local distribution of the existing responses to a given query. Thus, a hybrid objective of the generator can be finally defined
min J -E llog D(g, + ReLU(
(18)
A hinge loss, conducted by the ReLU function (ōx) max(0, da), is especially introduced to reform 8a. The primary reason of this operation is that the ReLU function has positive output only if ga > 0, according to the definition of ReLU( Apparently, da < 0 indicates that the generated response f is too far from the center of the response cluster in the semantic space, so that its relevance may be highly questionable. Meanwhile, minimizing negative variable is against the optimization direction. After the ReLU transformation, there remains valid loss only when da > 0, and thus both the diversity and the relevance of generated results are taken into account.
4.3 THE PHASE-WISH OPTIMIZATION
According to the analysis in Section the trivial adversarial training directed by -E [log (gg r)] can only determine the form of general responses to a given query. From the spatial perspective in the semantic space, the original adversarial objective is helpful to roughly locate the response cluster to be generated. However, the local distribution can not be captured by this procedure.
By contrast, according to the discussions above, the proposed hybrid objective actually provides a way to force the generated responses, originally gathering around the general form, to expand into the expected local shape described by the golden truth, by conducting phase-wise optimizing operation. This mechanism can be detailed in an intuitive way:
Foundation:
Once the DBM in Figure 1 is well-trained with the query-responte corpus, the semantic center T. of a Response Cluster can be determined by the given query q.
Phase-1: In the early stage of the adversarial training, generated response F is not semantically relevant to the query Thus, it can be inferred that i is radially farther from the cluster center Ta than the golden response r. In this situation, according to Equation [6 and Equation [7] we can claim that da < 0. In this phase, the hyper objective goes back to the general adversarial objective due to the ReLU function. Thus, the model is trying to force the generated samples to approach the center of each cluster, ignoring local distributions.
During the adversarial training Phase-l, the generated result 7 will go approaching to the cluster center T., which means (gg,)) + 0 It should be noted that, for any meaningful existing training sample r, (K(g.r) >> O. Therefore, at some point, turns to da > 0 and the right part of the hybrid objective in Equation T8 takes effect. Consequently, for each given query, the distribution of the generated results will expand to fit the local distribution of the golden samples.
Phase-2:
EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Datasets. Our experiments conducted two main stream open-access conversation corpora:
The Opensubtitles corpus and the Sina Weibo corpus. The OpenSubtitles dataset contains 5,200,000
movie dialogues, where we extract query-response pairs following (Xu al., 2018 Li et al. 2016). The Sina Weibo Corpus (Shang et al. 2015a) contains 2,500,000 single-turn Chinese dialogues, in which the length of the query and response ranges from 30. We sample 100,000, and 2,000
unique query-responss pairs as validation and testing dataset respectively from both of the corpor:
Baselines. For meaningful comparison, we introduce the following models as the basclines:
(1) Seq2Seq: a sequence-to-sequee
model trained with maximum likelihood estimation (MLE);
'otth the English and the Chinese datasets used in experiments uploaded to https: /www. www.eeiiii.1111111113333333 NAzii e5443 90s
dropbox. ccm/ www.khh09gg222111
7

---

Under review as a conference paper at ICLR 2020
Table 1: Performances of LocalGAN and Baselines on the Opensubtitles and Weibo Datasets.
Opensubtitlc
Weibo
Model
Dist-1
Dist-2
Ent4
Rel.
Dist-1
Dist-2
Ent4
Rel.
Seq2Seq
0.025
0.081
5.650
1.090
0.055
0.153
6,400
0.315
Seq2Seq-MMr
0.027
0.086
5.698
1.067
0.059
0.172
6.860
0,309
Adver-REGS
0.0296
0.098
5.701
1.113
0.061
0.181
7.658
0.320
GAN-AEL
0.030
0.100
5.733
1.106
0.062
0.183
7.765
0.318
AIM
0.0292
0.095
5,783
1.120
0,064
0.189
7.833
0.321
DAIM
0,031
0.103
5,873
1.098
0,067
0.195
8.042
0.316
LocalGAN
0.036
0.110
6.073
1.132
0.071
0.212
8.561
0.327
(2) Seq2Seq-MMI: the NRG model with Maximum Mutual Information criterion (Li ct al.
2016);
(3) Adver-REGS: the NRG model trained using adversarial framework, in which the policy gradient was employed to transfer the reward of the discriminator to the generator (Li et al.) 2017);
(4) GAN-AEL: an adversarial framework with an approximate embedding layer for connecting the generator with the discriminator directly (Xu et al. 2017).
(5) AIM DAIM: the adversarial training strategy allowing distributional matching of synthetic and real responses and explicitly optimizing a variational lower bound on pairwise mutual information between the query and response, so as to improve the informativeness and diversity of generated responses ZZang et al. 2018b
DAIM:
Evaluation Metrics. To evaluate diversity of results, we adopt three widely-applied metrics: D (Dist-1), 2 (Dist-2), and Entropy (Ent4) (Li et al. 2016 Zhang et al., 20188 Jost 2006). Besides, the relevance (Rel.) is measured by summing three embedding-based similar-
ities (greedy, average, extreme) (Liu et al. 2016) upon the ground-truth and generated responses.
Training Details. The vocabulary size of both datasets is 40,000. The embedding layer of OpenSub- titles and Sina Weibo initialized using 200-dimensional Glove vectors (Pennington et al.) 2014) and 300-dimensional Weibo vectors (Li al. 2018) respectively, All the models are first pre-trained by MLE, and then the models including Adve- GAN-AEL, AIM, DAIM and LocalGAN are trained with adversarial learning. The discriminator of Adver-REGS and GAN-AEL are based on CNN following YYu et al. 2017; Xu et al. (017), which the filter sizes are set to (1,2,3,4) and the filter number is 128, while that of LocalGAN adopts DBM with (2xembedding _size, 128, 128) to represent the semantic of queries and responses. The hidden size of the generator is set to 256 and 512 in GAN- models and Seq2Seq respectively. To guarantee the performance consistency of AIM and DAIM, we adopt the recommended parameter settings given by Zhang et al. (2018b). The experiments are conducted on the Tesla K80 GPU.
5.2 RESULTS & ANALYSIS
Table lists quantitative results on the diversity and relevance of generated responses on both datasets. As shown by the results, compared to Seq2Seq and Seq2Seq-MMI, the GAN-based meth- ods give better results on the diversity oriented metries, including Dist-1, Dist-2 and Ent4 This observation indicates that adversarial learning does provide the meaningful guidance to NRG mod- els to avoid some of the safe-responses.
It can be observed that LocalGAN outperforms the baselines with adversarial learning architecture (Adver-REGS, GAN-AEL, AIM and DAIM) on both the diversity metrics and the relevance metrics. Generally, a notable improvement on diversity may lead to some negative influence on relevance, and thus promoting the diversity of generated response while maintaining their relevance is essentially desired for any methodologies, which has been achieved by our LocalGAN. The performances of LocalGAN can be attributed to the fact that LocalGAN has taken the local distribution of responses to given query into account. By adopting the hybrid objective function, the proposed adversarial
'We have taken the codes of AIM and DAIM from https: / wtw..tt.....cc..eeeeeeeetoeeo cttbb//////reaassaaa111 converse_GAN impllmeete by the authors of this work for comparisons.
8

---

Under review as a conference paper at ICLR 2020
The Entropy Trend Adversarial Training
Opensubtitie
6.0
8.5
8.0
5.5
Entrapy
5.0
GM--EL
GAN-AEL
AIM
6.0
AIM
DAIM
DAIM
4.0
5.5
2000
Training Batch Number
Training Batch Number
Figure 2: The Entropy Trend of adversarial learning based models in the Training Process.
model gets to capture the spatial characteristics response clusters, and the generation process is consequently foreed fit the semantic distributions of response clusters.
The training stability a tough issue to be addressed for adversarial learning (Yu et al. 2017), and discussed in the previous sections, of the motivations of our LocalGAN is to make adversarial learning more stable. To verify this aspect, we track the changing of the Entropy (Ent4) of results given by GAN-AEL, Adver-REGS, AIM, DAIM and LocalGAN, shown in Figure
can be observed that the training of LocalGAN and Adver-REGS is stable. By contrast, there exist obvious fluctuations the curves of AIM and DAIM, and GAN-AEL rapidly gets of control after 1000 batch, which makes it rather difficult to grasp the models with the best status. This group of results indicates the necessity of introducing additional restrictions into adversarial learning processes. For this purpose, Adver-REs AdGerr-REGS introduces fracher-forcing loss (Li et al., (017)) while AIM and DAIM have taken the informativeness oriented constraints to partially control the stability (Zhang et al. 2018b). However, GAN-AEL only takes the Wasserstein distance as the objective (Xu et al. 2017), and thus the entropy goes down rapidly. Compared to the Adver-REGS, our LocalGAN achieves better diversity with even more smooth entropy curve. The training of LocalGAN benefits from the phase-wise optimization driven by the hybrid loss, and its stability also indicates the meaningfulness of modeling and utilizing local distributions of responses.
5.3 HUMAN EVALUATION
To further conduct intuitive comparisons among the NRG models, we perform human evaluations on 500 testing samples. Five anno- tators are asked to judge whether response is relevant to the given query and whether the re- sponse is informative not respectively. Both human metrics "Relevance" and "Informative- ness" solely have two labels and 1 (0: irrel- evant or non-informative; 1: relevant or infor- mative) for simplicity. The human evaluation results are listed in Ta and the results ba- sically match the observations in Table [ The Kappa of relevance and informativeness annotations are 0.734 and 0.561 respectively.
Method
Relevance
Informativeness
Seq2Seq
0.738
0.25
Seq2Seq-MMI
0.67
0.336
Adver-REGS
0.702
0.398
GAN-AEL
0.696
0.41
AIM
0.746
0.294
DAIM
0.768
0.45
LocalGAN
0.784
0.536
Table 2: Human Evaluation Results of Models.
5.4
CASE STUDY
The effectiveness of the NRG models can be illustrated by the responses generated by them intu- itively. Table 3 lists the sampled results given by all the baselines and our proposed LocalGAN, trained the OpenSubtitle corpus (The samples based the Sina Weibo dataset given in Ap- pendix C). It can be observed that, compared to the basic Seq2Seq and Seq2Seq MMI, the models incorporating adversarial learning have generated results with varying degrees of semantic diversity,

---

Under review as a conference paper at ICLR 2020
expected. Among the baselines, DAIM produces more diverse responses, which matches the quantified evaluation results in [ The samples from LocalGAN have shown better semantic relevance, and meanwhile the diversity and informativeness are also guaranteed.
Table 3; Sample results given by different models trained on the OpenSubtitle corpus.
Query: you know you're 're strong woman and i admire that.
Query: you are a perfect girlfriend.
Ground-truth: i'm sorry for what i'm about to
Ground-truth: some times wish that you would
do.
go out more.
Seq2Seq: you know what mean.
Seq2Seq: what are you talking about? S you have no idea what you're
Seq2Seq-MMI: you know what you't re doing.
talking about.
Adver-REGS: don't know how to do that.
adver-REGSS do you have a girlfriend?
GAN-AEL: what are you talking about?
GAN-AEL: what you doing here?
AIM: what you doing here?
AIM: you the only one.
DAIM: had plan for you.
DAIM: you are a good person.
LocalGAN: thought it was your idea.
LocalGAN: you part of me.
Query: maybe cause you keep marrying for money. Ground-truth: did not marry for money.
Query: you 'e just my roommate eric.
Ground-truth: millie is much much more than
that.
Seq2Seq: what do you mean?
Seq2Seq:
i don't know what you 'e talking about. Seq2Seq-MMIS need to talk to you.
Seq2Seq-MMI: what do you want me to do?
Adver-REGS: what do you want say?
Adver-REGS: want
you about this.
GAN-AEL: what is the difference?
GAN-AEL: can you help me please?
AIM: what are you talking about?
AIM: what are you talking about.
DAIM: what is the problem?
DAIM: its a pleasure to meet you.
LocalGAN: i don't i have a choice.
LocalGAN: no you are my friend.
Query: he says some kind of sea creatures
Query: would you excuse us for just minute?
washed up on his beach.
Ground-truth; how could it be sea creature?
Ground-truth: ted i'm gonna be honest.
Seq2Seq: don't know what to do.
Seq2Seq: why are you doing this?
Seq what are we supposed to do?
Seq2Seq-MMII don't 't want to you again.
Adver-REGS: he wants to go to the sandglass.
Adver-REGS, what are you going to do?
GAN-AEL: he told us to come back.
GAN-AEL: have to go to the bathroom.
AIM: thought what was going happen.
AIM: what are you supposed do?
DAIM: he said it was gift.
DAIM: why are you calling me?
LocalGAN: thought looks like a fish.
LocalGAN: please give me your reason ab-
sence.
CONCLUSIONS
This paper has given the theoretical proof of the upper bound of the adversarial training leveraged models on the Seq2Seq-based neural response generation task. The proof indicates that, due to the local distribution nature of query-responss corpora, the GAN based NRG models will converge to the states mostly generating specialized patterns corresponding to given queries. To address this issue, we proposed to model the local distribution of queries and their response in the semantic space by adopting energy-based function, and found the approximation of this function. According to this approximated distribution representation, a new loss function describing the local expansion cost in the fitting of response distribution is presented and finally combined with the traditional GAN loss to form a hybrid training objective for the GAN based NRG model. This paper provides a reasonable explanation to the unstable training process and unsatisfying results of GAN based NRG approaches, and meanwhile gives a different perspective to leverage the local data distribution to enhance classic GAN approaches.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223, 2017.
Sanjeev Arora and Yi Zhang. Do gans actually learn the distribution? an empirical study. arXiv preprint arxiv: 02200008222.... 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilib- rium in generative adversarial nets (GANs). In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 224-232. PMLR, 06-11 Aug 2017.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv e-prints, abs/1409.0473, September 2014. URL https:
//arxiv. oww.or////aa.... abs 0473
Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. Generating more interesting responses in neural conversation models with distributional constraints. In Proceedings of the 2018 Confer- on Empirical Methods in Natural Language Processing, pp. 3970-3980, Brussels, Belgium, October-November 2018, Association for Computational Linguistics.
Jiachen Du, Wenjie Li, Yulan He, Ruifeng Xu, Lidong Bing, and Xuan Wang. Variational autore- gressive decoder for neural response generation. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pp. 3154-3163, Brussels, Belgium, October- November 2018. Association for Computational Linguistics.
Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading comprehension. arXiv preprint arXiv: 2017.
G. B. Folland. Higher-order derivatives and taylors formula in several variables. https: 7/sites. math. httpl//www.......ooo/ooaaaaaaa/a cessed Nov 8, 2019.
2005. Ac-
Karl Friston, James Kilner, and Lee Harrison. A free energy principle for the brain. Journal of Physiology-Paris 100(1-3):70-87 2006.
Karl Friston, Christopher Thornton, and Andy Clark. Free-energy minimization and the dark-room problem. Frontiers in psychology, 3:130, 2012.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- 2014.
mation processing systems 27, pp.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimeoality of data with neural net- works. science, 313(5786):504-577, 2006,
Geoffrey E Hinton and Richard S Zemel. Autoencoders, minimum description length and helmholtz free energy. In Advances in neural information processing systems, pp. 3-10, 1994.
Lou Jost. Entropy and diversity. Oikos, 113(2):363-375. 2006.
Mahyar Khayatkhoei, Maneesh Singh, and Ahmed Elgammal. Disconnected manifold learning for generative adversarial networks. In Advances in Neural Information Processing Systems, pp- 7343-7353, 2018.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746-1751, 2014.
Abhishek Kumar, Prasanna Sattigeri, and Tom Fletcher. Semi-supervise learning with gans: Mani- fold invariance with improved inference. In Advances in Neural Information Processing Systems, pp. 5534-5544, 2017.

---

Under review as a conference paper at ICLR 2020
Nicolas Le Roux and Yoshua Bengio. Representational power of restricted boltzmann machines and deep belief networks. Neural computation, 2016):1631-169, 2008.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In Proceedings of NAACL-HLT, pp. 110-119, 2016.
Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2157-2169, 2017.
Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du Analogical reasoning on chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 138-143, 2018.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2122-2132, 2016.
L. Mirsky, A trace inequality of john von neumann. Monatshefte für Mathematik, -944):303-306 Dec 1975. ISSN 1436-5081. doi: 10.1007/BF016473331 URL https: ott//./..........////////
I0C7/BP0164733J
Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng. Learning deep energy models. In Proceedings of the 28th international conference on machine learning (ICML-I1), pp- 1105-
1112, 2011.
Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. Exemplar encoder-decoder for neural conversation generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1329-1338, Melbourne, Australia, July 2018. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532-1543 2014.
K. B. Petersen and M. S. Pedersen. The matrix cookbook, nov 2012. URL http::/www2 immm dtu. wtw.ppppppppppppcccp ?3274 Version 20121115.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. International Confer-
ence on Artificial Intelligence and Statistics, pp. 448-4444 2009.
Ruslan Salakhutdinov and Hugo Larochelle. Efficient learning of deep boltzmann machines. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp- 693-700, 2010.
Lifeng Shang, Zhengdong Lu, and Hang Li. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pp- 1577-1586, 2015a.
Lifeng Shang, Zhengdong Lu, and Hang Li. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pp. 1577-1586, 2015b.
Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. Gen- erating high-quality and informative conversation responses with sequence- toreeetttsssee to-seueence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2210-2219 2219, Copenhagen, Denmark, September 2017. Association for Computational Lin- guistics. doi: (0.18653/v1/D17-111
12

---

Under review as a conference paper at ICLR 2020
P. Smolensky. Parallel Distributed Processing: Foundations, volume 1, pp. 194-281. MIT Press, Cambridge, 1986.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Yun Nie, Jianfeng Gao, and Bill Dolan. A neural network approach to context-sensitly gen- eration of conversational responses. In Proceedings of the 14th Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)) pp. 196-205, 2015.
Nitish Srivastava and Ruslan R Salakhutdinov. Multimodal learning with deep boltzmann machines. In Advances neural information processing systems, pp.
2012.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 Conference on Empir- ical Methods Natural Language Processing, pp. 3930-3939, Brussels, Belgium, October- November 2018. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pp. 3104-3112, 2014.
Baoxun Wang, Xiaolong Wang, Chengjie Sun, Bingquan Liu, and Lin Sun. Modeling semantic relevance for question-answer pairs in web social communities. In Proceedings of the 48th An- nual Meeting of the Association for Computational Linguistics, pp. 1230-1238. Association for Computational Linguistics, 2010.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, ukasz Kaiser, Stephan Gouws. Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens. George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/ 44, 2016. URL http:http//a/aaa org/ abs 1609. 08144
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. Topic aware neural response generation. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. prommtii gan: A cross-entropy based generative adversarial network for diversified text generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3940-3900999 (900033999,999 2018.
Zhen Xu, Bingquan Liu, Baoxun Wang, SUN Chengjie, Xiaolong Wang, Zhuoran Wang. and Chao Qi. Neural response generation via gan with an approximate embedding layer. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 628-637, 2017.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, pp. 2852-2858, 2017.
Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Jun Xu, and Xueqi Cheng. Learning to control the specificity in neural response generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1108-16 11117, Melbourne, Australia, July 2018a. Association for Computational Linguistics.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maxi- mization. In Advances in Neural Information Processing Systems, pp. 1810-1820, 2018b.
Junbo Zhao, Michael Mathicu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXiv:1609.03120 2016.
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. Paragraph-level neural question genera- tion with maxout pointer and gated self-attention networks. In Proceedings of the 2018 Confer- ence on Empirical Methody in Natural Language Processing, pp. 3901-3910, Brussels, Belgium, Octoher-Novemher 2018 Association for Computational Linguistics.
13

---

Under review as a conference paper at ICLR 2020
A
THE GRAPHICAL MODEL FOR RESPONSE DISTRIBUTION MODELING
Different from the computer vision related scenario, training a DBM on a set of text vectors is not trivial since the training procedure is difficult converge due the value scale of text vectors is much larger than image vectors. Moreover, the simple scaling methods are not effective enough for this issue. For this purpose, this paper adopts standard-scalee scaler to remove the mean and scale to unit variance. To valid the effectiveness of standadd-scale, We conduct experiments on the query- response matching task using normalized vectors. The experiment result show that the matching performance based normalized vectors is similar to that of CNN based architecture (Kim 2014).
B DETAILED PROOF OF LEMMAS
Proof of Lemmal) 7 As mentioned before, is assumed to be independent and identically dis- tributed (i.i.d) normal variable, thus F(g, rt) can be seen i.idd random variable for a fixed query Based on the strong law of large numbers,
Flg, R₂y
Σ F(grr)) 4. B[F(grr) when [Ryl - oc nER,
|Ry
Proof of Lemma 2 For a fixed query, F(g,r) can be seen as the the scalar function of vector r. For simplicity, denote F F(q,7) r) f(r).
Taylor expansions for the first moment function random variables are as follows.
E[f(r)]] E [f (re + (r))
E ( (rr)+ (r- _ -r)r ( (r - -r) + Rr re)
where Df(re) is the gradient of f evaluated at Te. f(r) the Hessian matrix and 2 re) the Lagrange remainder. Since Er To. the second term (r - rZ)" D disappears.
Then we will try to find the upper bound of the third term and the remainder term. The relevant theorems used in the proof are listed as follows.
According to (Petersen Pedersen 2012), assume is symmetric, E[x] and Σ Var [r], then
E x Ax] Tr(AE) + c Ac.
Mirsky 1975) states following theorem: If A, are complex n n matrices with singular an and 31 > 32 > Bn respectively, then
values 01 a2
TT(AB) S
ag
Firstly, r-re ~ NO, Σ) and is positive semi-definite matrix, the third term can be simplified as follows.
(T)) rT) ((r))r)))(( (r (r- - ra)l = [())) (r.)r
< ag
< ai Tr(E) = a ||| - -00-rrr]]] r.]113
where 01 2 02 2 ... 2 On and 31 2 82 2 ... > 8, denote the singular value of matrix (D²f(r.)) and Σ respectively.
14

---

Under review as a conference paper at ICLR 2020
Mcanwhile, according to the Corollary 1 in (Folland 2005), have that
M relli
2 re) - |r
3
where M the upper bound for absolute value of third-order partial derivatives of f-
Next, show that the aj and third-oder partial derivatives can be bounded by M, where M is max 16(hg i)l (hg follows multinomial distribution) and n is the dimension of r. 1iisi
Substituting the definition of E(g,r, r, H) into Flg. r), we have following equation:
f(r) = F(g,r) - log Σ exp(hi
W Wrr + N2 W Wag + 1t2 h/W,an Wghhhg)
hegh
After that, its first-order, second-order and third-order partial derivative are calculated as follows:
Qf(r)
Engwh exp(h? T W ar + h2 W 949 + h² W Wyhhh) (
Ör:
Engn expp hi W grr hig W a + 12t W Wghhhg)
S²)(r) Ör,Or;
Σ
a(hgg h) x b(hg. 1)
Σ
alhg: h) x b(hg.:) b( b(hq:
h
hg
0 f(r( Ör,Or,0rk
Σ
c(hgg h,, x b(11g) x
Σ
altigg hi) x b(hgg.) b(hg:))
hggh
high
Σ hggh
a(hq. h) b(hg. 1) x
Σ
olhyg H k) b(hg
h
where
exp(hi W gr + W q0 + 1eth WaTWhh
a(hg;
exp(hT W oor h 4ad hT Wigh))
hih da(hy: h)) a(hy: h)b(hy R)) ahhg h) Ork
c(hyg h,k)
Σ ahg ()b((hgkk) k) hg
and b(hg,*) i) = (he" War): representing the i-th element of the vector W.r,W- According to the definition of a(hg, h), is obvious that a(hg, h) > and Σn a(hy, h) 1.
The upper hounds for the second-order and third-order partial derivative are shown as follows. f Or:Or,
la(hg hi)l x 16(hg i)1
Σ hig.t
|ath.. io) x b(hr. (b(( 2M2
lo(h, h k,, 1 la(h.. hil x lb(hg k)] + la(h, h)1 x Σ
la(h, Ri)l x 16(h, k)) S 2M h)l
hight
S3)( Ör,ūr,ürk
M M
M
Σ
Jo(hy, h, k)] S 6M3
hggh
Based on the upper bound above, we can see that M can be 6.1 and the upper bounds for ai and ( re)l are as follows.
82 < 2nM2
Q1 = Tmax ((D2 m
or,ūr,
15

---

Under review as a conference paper at ICLR 2020
M 3!-
-
-
|| relli 1 = (VRM)lr- - rell2)²
Hence, we have
2 (())) F re) ( s(re)) (r re)] + 2r. - rc]
[B[(()]
$ 2 all BlIr - r.l113 EI re)
< [(VVMM| ) + ELV/MM//Vr---- - rell2)3
Therefore,
lim
-rel-r
16

---

Under review as a conference paper at ICLR 2020
C ADDITIONAL CASES
Query: J. M. KK K. A. F.%% You don't need to make things soo complicated, and it's
Query: RIBA 1E/C0FF (1099/70055 7U0
juss Like something you are holding, if you use too much power. it will brock and hurt you.
The globelish looks so cuse when it gets angry.
Ground-truth: B2 24 2
Ground-truth: RiEELAAt i 1-
feel very helpless. Why am so exneitive? Seq2Seq: t0 8837!
I thoughh it had swallowed up ball.
Seq2Sog: Rell Rell.2233 41099
That was spoken well.
So 1.
SequSeq-MMTT 38 16
Seq2SSgg-MMa 16
That was spoiae quite well.
So thatt's it!
Adver-REGS: 10
Adver- REGS: RU 1124909
Is quite make senee
allo have like this.
GAN-AEL: 1 ft A.? What does that meon?
GAN-AEL: Riaaatre
also have the same feeling,
AIM: (070
AIM: I2 #1//::2490
These words speak the touth.
Yeal,,
DAIM: CI That's what want.
DAIM: $197:2
It looks cate!
LocalGAN: RCF WCF, 24100
LocalGAN: A0 +AAAE??
go and free younslf.
legendary globefish?
Query: 8h8r/ 190 A. E A terrible story to all the fattien: the summer comning.
Query: A1 MY 1 19277
You can feel peaceful only you are far from cities.
Ground-truth: Witrow RRS RRA -st. This fatty car looks so funny.
Ground-truth: fe Mi At369, MAGET A AIRREIE..
Agree with that, but think few people can seally do at
Seq2Seq: 2 22 bo B
Seq2Seq: R
that like this sentemce. ScQ & F. 12 11 just warnn any what do moun?
1 1just
I aalo think
Seq2Seq-MMMI REA88EE
hove the same feeling well.
Adver-REGS: RR 8-280 EF
Adver-RECS: i2. U2?
only say that provd you.
What daes this monn?
GAN-ABL: 43
GAN-AEL: X 1692--AA088110
May have
city for any single person.
ATM: RUURR, 4f
ATM: 3210 #1
only say so I.
Laleo
action.
DAIM: Rele
DAIM: 72
You my oaly, skinny.
life!
LocalGAN: 20 M.
LocalGAN: 0-B 1-09 TRRSSTI
This must come from the heart skinny guy.
If you don't get laner poace, you will the boring noise.
Query: W W8888 5 [Ltf fL Hidy $1
Query: 402444 A
Just lee me know
also love be holding
Never give up as long ne you are alive.
fingrr the tiny hand of baby!
Ground-truthe 4125-11 7
Ground-truth: B #E (f S882
Bus there always exist something repestable.
Jove be called the sister auntle with baby voice.
Seq2Seq: Niltl KEN!!!
Seq2Seq: Ri FA2
Cheer and am always supporting you.
also this
8cQ2Seq-MMII 12413.
Seq28eq-MMMI R
Every the best. day.
felling
Adver-REGS: 3520 *R. Jihh!
Adver-REGS: & W381
A new day, theer up!
The one in my home
well!
GAN-AEL: 86M -. Ai
GAN-AEL: it 1-
New day, new beginning!
you
AIM: Intii, Iaiiina
AIM: 35 413888
Cheer up.
feeling so much.
DAIM: 11
DAIM: Ri V. 433
Eveyything will pass.
want boyfriend
LocalG AN: 8E00 S1 keb.
LocalGAN: 97 R1 M9W..
Be calm, and everything will be OK
So cute, also want baby
Figure 3: Sample Results given by different models trained on the Sina Weibo corpus.
17


Reviews:
[{"review_id": "r1gblS6iFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\n\nThe main contribution of this paper lies in the proposed LocalGAN for neural response generation. The key observation is that for a given query, there always exists a group of diverse responses that are reasonable, rather than a single ground-truth response. Therefore, the local semantic distribution of responses given a query should be modeled. Besides the original GAN loss, the proposed LocalGAN adds an additional local-distribution-oriented objective, resulting in a hybrid loss for training, which claims to achieve better performance on response generation datasets.  \n\nStrengths:\n\nI think the proposed model contains some good intuitions, that is, the generated responses should be modeled as a local distribution, rather than a single ground-truth output during training. The motivation of this paper is therefore clear. Experimental results in Table 1 seems encouraging. \n\nHowever, I would have to say that the current draft is poorly presented. There are a lot of unclear parts that should be more carefully clarified, with details below.\n\nWeaknesses:\n\n(1) Writing: I think the language in this paper is repetitive, and can be much more precise and concise. Also, there are typos here and there throughout the whole draft. I would suggest the authors doing a careful proofreading before next submission. \n\nMinor: in the line before Eqn. (4), change \"SIMPLY\" to \"simply\". \n\n(2) Clarity: Overall, the presented method is unclear. \n\na) It is not entirely clear what the authors mean by saying \"this paper has given the theoretical proof of the upper-bound of the adversarial training ...\". I am not sure whether Eqn. (6) is totally correct, or at least how useful it is.\nb) The notations throughout the paper is a little bit confusing. The authors should normalize all the notations to be consistent. \nc) It is not clear what Eqn. (3) truly means. What is the value for s? The KL divergence should take two distributions as input, but here, the input are two triplets. \nd) In the line below Eqn. (6), what is \\tilde{R}_q? This is not defined. \ne) The proposed method relies on the use of R_q. However, how to define, or learn R_q is not clear. In the dataset, given a given query q, how do we find R_q?\nf) It is not clear why Deep Boltzmann Machines are needed here. I'd like the authors to more clearly clarify this design. Further, since DBM is used, then how the final model is trained together? Now, the models contains both adversarial learning, and contrastive-divergence-based algorithms for DBM training. This seems make the whole model training more unstable. \ng) Generally, I think Section 3 and Section 4 are hard to follow. Further, I did not see how useful Lemma 1 & 2 and Theorem 1 are. The final objective Eqn. (17) is also confusing.\n\n(3) Experiments: My biggest concern about the experiments is that human evaluation should be conducted, given the subjective nature of the task. This is lacked in the current draft. Only reporting numbers like Table 1 is not convincing. \n\n** This paper provides a link that actually links to a github repo. I am not sure whether this violates the policy of ICLR submissions or not. But at least from my point of review, this link should be anonymized. **", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJx4M15v9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the author proposed a model to address the training instability of a GAN model on the NRG task. The authors take advantage of an energy based model to measure the distance between a predicted response and the center of all qualified responses. The training process becomes a hybrid one with the original loss function in the beginning followed by the loss that pulls the response to the center of the response cluster later. In general the paper is well written, with experiments clearly showcased improved training stabilities. However, one major flaw in the experiments in that the authors almost only compared diversity measures such as Dist-1, Dist-2 and Ent4. These measures did not take into consideration the matches between the predicted one and the ground truth. The only relevance measure used in this paper is the  Rel., which the authors defined as the average of embedding distances. Such a measure is by no means an objective measure and can't really demonstrate the effectiveness of the model in terms of generating responses that are close to the ground truth. The authors would need to submit results on one of the widely adopted benchmark metrics (e.g., BLEU, ROGUE) or their equivalents in order to demonstrate the quality of the generated response. And this is the main reason of my rating recommendation. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1gRbInpFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "POST-REBUTTAL FEEDBACK\n\nI share the same concerns as that of reviewer 2 in the response to the rebuttal. Hence, my score remains unchanged.\n\n\nSUMMARY OF REVIEW\n\nThis paper motivates the need to \"contextualize\" responses based on the query to bring about stable training in NRG and consequently proposes localGAN to realize this. On the overall, I like the motivation and the proposed approach of this paper. The experimental results also look convincing.\n\nOn the flip side, the technical formulation and theoretical results are not presented rigorously and important technical details are missing, as discussed below. As a result, clarifications from the authors are needed to ensure the correctness of their formulation. The authors also need to improve the presentation and proof of the theoretical results; the correctness has to be checked again. In my opinion, these theoretical results do not improve my current assessment of the paper and can be removed to cut down to 8 pages. If the authors like to keep them, they need to revise them based on my concerns above.\n\nIt would be good to show some sample queries and corresponding \"meaningful\" responses produced by their proposed LocalGAN that are not considered safe responses which are produced by the other tested methods.\n\n\nDETAILED COMMENTS\n\nFor Lemmas 1 and 2 and Theorem 1, the authors need to present them rigorously by specifying the exact math expressions since they have not defined what it means by sufficient, approximates, small enough, and estimate properly. This will also eliminate any discrepancy in their interpretations. For example, the authors have used Taylor series expansion to approximate the expectation of F(q,r) in equation 19 (instead of bounding it). Hence, one can claim that Lemma 2 does not hold and hence Theorem 1 does not hold as well.\n\nIn Section 4.3, the described mechanism is confusing to me: \n\n(a) Are the authors saying that it is performed sequentially from foundation to phase-1, followed by phase-2? Or are the authors saying that these three phases are expected behaviors occurring during the optimization in equation 17? \n\n(b) For the foundation phase, is the DBM pre-trained, that is, prior to optimization in equation 17?\n\n(c) Are there multiple response clusters, that is, one for each q? If so, the second RELU term in the minimizing criterion in equation 17 does not seem to properly reflect this.\n\n(d) How are the response cluster centers r_c exactly determined? The authors vaguely say that they are modeled from training data. Is it one center per response cluster? Are the cluster centers also optimized in equation 17, besides the generator's weights? Can the authors provide the argument under the min operator in equation 17? It is confusing to leave out r_c from the subscript of alpha.\n\nI would have preferred that the authors specify the expression of the evaluation metrics to be self-contained.\n\nIn Fig. 2, how exactly do the authors measure stability? If the entropy rapidly increases like that of LocalGAN and Adver-REGS, are they considered stable?\n\n\nMinor issues\nPage 1: Despite of?\nPage 3: inequation?\nPage 3: Equation 4 and 5?\nEquation 6: The first summation should just be over q, unless there are multiple sets of R_q per q.\ntilde{R}_q is not used in equation 6.\nPage 4: a limited samples?\nPage 5: defined in 3.2?\nPage 5: To be consistent, mathbb should be applied to E.\nPage 9: valid this aspect?\nPage 9: from the the?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles neural response generation with Generative Adversarial Nets (GANs), and to address the training instability problem with GANs, it proposes a local distribution oriented objective. The new objective is combined with the original objective, and used as a hybrid loss for the adversarial training of response generation models, named as LocalGAN. Authors responded with concerns about reviewer 3's comments, and I agree with the authors explanation, so I am disregarding review 3, and am relying on my read through of the latest version of the paper. The other reviewers think the paper has good contributions, however they are not convinced about the clarity of the presentations and made many suggestions (even after the responses from the authors).  I suggest a reject, as the paper should include a clear presentation of the approach and technical formulation (as also suggested by the reviewers).
Decision: Reject