iWGAN: an Autoencoder WGAN for Inference
Authors: Yao Chen
Qingyi Gao
Xiao Wang
Venue:  (2020)
Keywords: Generative model
Autoencoder
Inference

Abstract: Generative Adversarial Networks (GANs) have been impactful on many problems and applications but suffer from unstable training. Wasserstein GAN (WGAN) leverages the Wasserstein distance to avoid the caveats in the minmax two-player training of GANs but has other defects such as mode collapse and lack of metric to detect the convergence. We introduce a novel inference WGAN (iWGAN) model, which is a principled framework to fuse auto-encoders and WGANs. The iWGAN jointly learns an encoder network and a generative network using an iterative primal dual optimization process. We establish the generalization error bound of iWGANs. We further provide a rigorous probabilistic interpretation of our model under the framework of maximum likelihood estimation. The iWGAN, with a clear stopping criteria, has many advantages over other autoencoder GANs. The empirical experiments show that our model greatly mitigates the symptom of mode collapse, speeds up the convergence, and is able to provide a measurement of quality check for each individual sample. We illustrate the ability of iWGANs by obtaining a competitive and stable performance with state-of-the-art for benchmark datasets.

Under review as a conference paper at ICLR 2020
IWGAN: AN AUTOENCODER WGAN FOR INFERENCE
Anonymous authors
Paper under double-blind review
ABSTRACT
Generative Adversarial Networks (GANs) have been impactful many problems and applications but suffer from unstable training. Wasserstein GAN (WGAN) leverages the Wasserstein distance avoid the caveats in the minmax two-player training of GANs but has other defects such as mode collapse and lack of metric detect the convergence. We introduce a novel inference WGAN (iWGAN) model, which is a principled framework to fuse auto-encoders and WGANs. The iWGAN jointly learns an encoder network and a generative network using an iterative primal dual optimization process. We establish the generalization error bound iWGANs. We further provide a rigorous probabilistic interpretation of our model under the framework of maximum likelihood estimation. The iWGAN, with a clear stopping criteria, has many advantages over other autoencoder GANs. The empirical experiments show that model greatly mitigates the symptom of mode collapse, speeds up the convergence, and is able to provide a measurement quality check for each individual sample. We illustrate the ability of iWGANs by obtaining competitive and stable performance with state-of-the-an for benchmark datasets.
INTRODUCTION
One of the goals of generative modeling is to match the model distribution Pa(x) with parameters i to the true data distribution Px for random variable X X. For latent variable models, the data point is generated from a latent variable Z E through a conditional distribution P(X| Here X' denotes the support for Px and Z denotes the support for Pz. There has been a surge of research on deep generative networks in recent years and the literature is too vast to summarize here (Kingma & Welling 2013 Goodfellow et al. 2014 Li al., 2015). These models have provided a powerful framework for modeling complex high dimensional datasets. We start introducing two main approaches for generative modeling. The first one is called variational auto-encoders (VAEs) (Kingma & Welling, 2013), which use variational inference to learn model by maximizing the lower bound of the likelihood function. VAEs have elegant theoretical foundations but the drawback is that they tend to produce blurry images. The second approach is called generative adversarial networks (GANs) [Goodfellow et al., 2014), which learn a model by using a powerful discriminator to distinguish between real data points and generative data points. GANs produce more visually realistic images but suffer from the unstable training and the mode collapse problem. Although there many variants of generative models trying to take advantages of both VAEs and GANs (Tolstikhin let al. 2017] Rosca et al.) 2017). to the best of our knowledge, the model which provides a unifying framework combining the best of VAES and GANs in principled way is yet to be discovered.
1.1 RELATED WORK
The generative model is to learn mapping, denoted by G, from Z to X' to approximate the conditional distribution Po. (X|2) of the data point X E X' given latent code Z. We consider the deterministic mapping G in this work. Both GANs and Wasserstein GANs (WGANs) AAjovssy et al., 2017) can be viewed as minimizing certain divergence between the data distribution Px and the generative model distribution Pcizy. For example, the Lensen-Shannor (JS) divergence is implicitly used in GANs (Goodfellow et al. 2014). The 1-Wasserstein distance between Px and Po(z), denoted by WT (Px, Po(z)). employed in WGANs. Empirical experiments suggest that the Wasserstein distance is a more sensible measure to differentiate probability measures supported in low-dimensional manifold, In terms of training, it turns that it is hard or even impossible to
GANs and WGANs:

---

Under review as a conference paper at ICLR 2020
compute these standard divergences in probability, especially when Px is unknown and Po(z) is parameterized by deep neural networks (DNNs). The training of GANs is converted into playing a game between two competing networks: the generator and the discriminator. The generator is to fool the discriminator and the discriminator is to distinguish between true data samples and generated samples. Instead, the training of WGANs is to study its dual problem because of the elegant form of SnuttorovichRRubinstte
duality (Villani) 2008). Analogous to GANs, the discriminator is now a real-valued 1-Lipschitz function. Many techniques such weight clipping (Arjovsky et al.) 2017) and gradient penalty (Gulrajani et al., 2017) are used to enforce the Lipschitz constraint. More
discussions about WGANs will be presented in Section 2
Autoencoder GANs: Larsen et al. 2016) first introduced the VAE-GAN, which is a hybrid of VAEs and GANs. The VAE-GAN uses a GAN discriminator to replace a VAF's decoder to learn the loss function. The motivation behind this modification is that VAFs tend to produce blurry outputs during the reconstruction phase. More recent VAE-GAN variants, such as Adversarial Generator Encoders (AGE) [yyanov et al. 2018) and Auto-encoding GANs (a-GAN) (Rosca et al. 2017), use separate encoder to stabilize GAN training. The main difference with standard GANs is that, besides the generator G, there is encoder Q X -> Z which maps the data points into the latent space. This encoder is to approximate the conditional distribution Q [ X)) of the latent variable Z given the data point X. Other encoder-decoder GANs are introduced in Adversarially Learned Inference (AL.) (Dumoulin et al. (016) and Bidirectional Generative Adversarial Networks (BiGAN) (Donahue et al.) 2016). The objective of both ALI and BiGAN is to match two joint distributions under the framework of vanilla GANs, the joint distribution of (X, ((X)) and the joint distribution of (G(Z), 2). When the algorithm achieves equilibrium, these two joint distributions roughly match. We are able obtain more meaningful latent codes by (XX., and this should improve the quality of the generator as well. Adversarial Variational Bayes (AVB) (Mescheder et al. 2017) presented a more flexible latent distribution to train Variational Autoencoders. Hu et al. (2017) provided new interpretations of GANs and VAEs and revealed strong connections between them which are linked by the classic wake-sleep algorithm.
Duality in GANs: Regarding the optimization perspectives of GANs, (Chen et al. 2018 8 Zhao et al.
2018) studied duality-based methods for improving algorithm performance for training. Primal-dual Wasserstein GANs (PD-GANs) are introduced in (Gemici et al. 2018), which proposed a new penalty term whose evaluation samples are obtained from the encoder Q. Farnia & Tse (2018) developed convex duality framework to address the case when the discriminator is constrained into a smaller class. Grnarova al. (2018) developed an evaluation metric detect the non-convergence behavior of vanilla GANs, which is the duality gap defined as the difference between the primal and the dual objective functions. (Husain et al. (2019) investigated the close relationship between WAE (Tolstikhin) et al. 2017) and f-GANs (Nowozin et al. 2016), and proved generalization results for autoencoder models.
1.2 OUR CONTRIBUTIONS
Although there are many interesting works on autoencoder GANs, remains unclear what the principles are underlying the fusion of auto-encoders and GANs, For example, do there even exist these two mappings, the encoder Q and the decoder G. for any high-dimensional random variable X, such that Q(X) has the same distribution as Z and GZZ) has the same distribution as X?? Is there any probabilistic interpretation such as the maximum likelihood principle on autoencoder GANs? We introduce inference Wasserstein GANs (iWGANs), which provide satisfying answers for these questions. We focus on the 1- Wasserstein distance, instead of the Kullback-Leibler divergence. We borrow the strength from both the primal and the dual problems and demonstrate the synergistic effect between these two optimizations. The encoder component tends out to be a natural consequence from our algorithm. Furthermore, the iWGAN has rigorous probabilistic interpretation under the maximum likelihood principle, and our learning algorithm is equivalent to the maximum likelihood estimation when our model is defined an energy-based model based on an autoencoder. Our main contributions are listed as below:
1. We propose a novel framework, called IWGAN, to learn both an encoder and a decoder simultaneously. We prove the existence of meaningful encoder and decoder, establish an equivalence between WGAN and iWGAN, and develop a generalization error bound for iWGAN.
2

---

Under review as a conference paper at ICLR 2020
2. We establish a rigorous probability interpretation for iWGANs and our training process is exactly the same as the maximum likelihood estimation. As a byproduct, this interpretation allows to perform the quality check at the individual sample level.
3. We demonstrate the natural use of the duality gap as a measure of convergence for iWGANs, and show its effectiveness for various numerical settings. Our experiments do not experience any mode collapse problem.
2 IWGAN
The autoencoder generative model consists of two parts: an encoder Q and a generator G. The encoder Q maps data sample x E X' latent variable 2 E Z, and the generator G takes latent variable 2 E 2 to produce sample G(z). The autoencoder generative model should satisfy the following three conditions simultaneously: (a) The generator can generate images which have a similar distribution with observed images; (b) The encoder can produce meaningful encodings in the latent space; (c) The reconstruction errors of this model based on these meaningful encodings are small. The benefit of using autoencoder is to encourage the model to better represent all the data it is trained with, so that it discourages mode-collapse.
We first show that, for any distribution residing smooth manifold, there always exists encoder Q' which guarantees meaningful encodings and exists a generator G which generates samples with the same distribution data points by using these meaningful codes.
Theorem 2.1. Consider a continuous random variable X E X, where X is d-dimensional smooth Riemannian manifold. Then, there exist two mappings Q* : : + Rp and G* :' X, with p = max did + 5)/2, d(dd + 3)/2 + 5}, such that Q' (X) follows multivariate normal distribution with zero mean and identity covariance matrix and G Q" is an identity mapping, i.e.
X = G'(((XX)))
Learning Q" and G' from the data points is challenging task. Recall that the 1-Wasserstein distance between the data distribution Px and the generative model distribution Pa(z) defined as
(a(z))
inf =en(Px .2) B(x.2)= IIX G(Z)
where || represents the T--norm and IT(Px, Pz) is set of all joint distributions of (X, Z) with marginal measures Px and P'z, respectively. The main difficulty in (I) is to find the optimal coupling π and this constrained optimization with Px(x) f T (2,22)d for x E X' and (z(2) = J π( z)dr for Z.
Based on the Kantorovich-Rubineee Rubinstein duality, the WGAN studies the 1- Wasserstein distance || from the dual format
Wi WP
sup JEF
Px [[x)] Ex~py [[((()
(2)
where F is the set of all bounded 1-Lipschitz functions. Weight clipping (Arjovsky et al. 2017)
and gradient penalty (Gulrajani et al. 2017) have been used to satisfy the constraint of Lipschitz continuity. The experiment of (Arjovsky et al. 2017) showed that the WGAN can avoid the problem of gradient vanishment. However, the WGAN does not produce meaningful encodings and many experiments still display the problem of mode collapse (Arjovsky et al. [017]] Gulrajani et al. 2017). On the other hand, the Wasserstein Autoencoder (WAE) (Tolstikhin al. (017), after introducing an encoder Q X' + 2 to approximate the conditional distribution of Z given X, studies the reconstruction error inf Ex || G(Q(X))]| where Q is a set of encoder mappings whose elements satisfies PQ.x) Pz. The penalty, such D(PQ(x): Pz), is added to the objective to satisfy this constraint, where D is an arbitrary divergence between Po(x) and Pz- The WAE can produce meaningful encodings and controlled reconstruction error. However, the WAE defines generative model in an implicit way and does not model the generator through G(Z) with Z ~ Pz directly.
To take the advantages of both WGAN and WAE, we propose a new autoencoder GAN model, called iWGAN, whose objective is
W Paiz)) feT
inf sup Ex||X G(Q(X)) + Ex [](G(Q(X)) - Ez [[GG())) (3)
3

---

Under review as a conference paper at ICLR 2020
|| can be treated as the autoencoder reconstruction error as well as loss to match the distributions between X and G(Q(X)). We note that the -1-norm ||. || has been used for the reconstruction term by a-GAN (Rosca et al. 2017) and CycleGAN (Zhu et al., 2017). Rx [G((2) can be treated as a loss for the generator
The term || -
Another term Ex-rx (G()(())) well as loss to match the distribution between (X)) and G(Z). We emphasize that this term is different with the objective function of the WGAN in (2.. Furthermore, it is challenging for practitioners determine when to stop training GANs. Most of the GAN algorithms do not provide any explicit standard for the convergence of the model. However, the measure of convergence for iWGAN becomes very natural and we use the duality gap as the measure. The duality gap can be defined as
DualGap G. O.F) = sup LGG fer
inf L,G, GER.EE2
(4)
where L,G, Ex G(Q(X))I Ex(((G(((X)) Ez(s(G(Z))-
Theorem 2.2. The iWGAN objective (3) is equivalent to
Wi(Px, Pa(z)) = inf { (Wx Pa(e()) + Wi (Paco(()) }
(5)
QEO
Therefore, (Pxxx, Pa(Z)) S WI(xx, Pa(z)). If there exists a Q E Q such that (x) has the same distribution with Z, then (PPx.PP(z)) solution and assume that the encoder, generator, and discriminator all have enough capacities. Then ('((())))) ( + W WI(P P()2)). Moreover, if outputs the same distribution as X and Q outputs the same distribution as Z, both the duality gap and W (Px, Pa(z)) are zeros and X G()(X)) for X Px.
WI(Px, (I(Px.Pazz)) Let (Ô, G. f) be fixed
the duality gap is larger than W1(xx,
According to Theorem 2.2 the iWGAN objective is in general the upper bound of W1 (Px, Po(z)) However, this upper bound is tight. When the space Q includes a special encoder Q" such that Q" has the same distribution as Z. the iWGAN objective is the exactly same as Wi (Px, Pa(z)) Theorem 2.2 also provides appealing property from practical point of view. The values of the duality gap and WI(Px. PGzz) give us a natural criteria to justify the algorithm convergence.
3 GENERALIZATION ERROR BOUND AND THE ALGORITHM
In practice, we minimize the empirical version, W (Px, Pa(z)). of W (PX, ) to learn both the encoder and the decoder. Before present the details of the algorithm, first develop the generalization error bound for iWGANs. For discussions of generalization performance of classical GANs, see Arora al. (017)) and Jiang et al. (2018).
Theorem 3.1. Given a generator G E G, and given n samples (#1...., Inn from X = {or || S
B}, with probability at least 1 8 for any d E (0, 1), we have
2
Wi(Px, Pa(z)) Wi(Px, Pa(Z)) + (F) 3B,
log 5
(6)
where (F) E. Eig I funcction set F, in which e; is the Rademacher variable.
is the empirical Rademacher complexity of the
Theorem
3.1 indicates that the 1-Wasserstein distance between Px and Pa(z) can be domi- nantly upper bounded by the empirical W WP(Px. Pa(z) and Rademacher complexity of F. Since
W, (P(xx, Pa(z)) S Wi Pa(Q(x)) + Wi (Paio())) Pa(z)) for any Q E 2, the capacity of Q determines the value of W, (Px. (a(z)) On the other hand, there are several existing results on the empirical Rademacher complexity of neural networks. When JF is set of 1-Lipschitz neural networks, we apply the conclusion from Bartlett al. (2017) to Rt, (FF), which produces an upper bound scaling as OBB/L3/ Here denotes the depth of network f E F. Similar upper bound with an order of O(B /Ld Ldd/n) can be obtained by utilizing the results from Li et al. 2018)) where d is the width of the network.
4

---

Under review as a conference paper at ICLR 2020
in (Gulrajani et al. 2017) to enforce the I-Lipschitz constraint on f E F. In addition, we use the maximum mean discrepancy (MMD) penalty (Gretton al. 2012 denoted by MMDx(P Pz), to enforce (XX converge to Pz, where is kernel function. The details of the algorithm are presented in Algorithm
In practice, we adopt the gradient penalty defined GP [xx((XXx
Algorithm 1: The training algorithm of iWGAN
while DualGap 1 or L(G', Q* > e2 do
for t= I,,,.. Meritic do
Sample real data x² ~ Px, latent variable 2' ~ Pz and a random number ~ U(0,1]
8' + exx + (1- (x')
Calculate L' = L(",, ""G''QQ, 1" and gradient of -L
Update by Adam: f Adam(-V,l
where for f
-V V ΣE (f(G ((G
1)²)
end for
10
1, ..., Merètic do
Sample real data xt Px, latent variable z" Pz
Calculate L/ L(G', Q', fit 1x2t 2²t) and gradient of L
Update G, Q by Adam: Gi+1, Q++1 + Adam( V
where for G¹, Q'
VO.OL* Vc.q n
Σ (Q
d2 n(n - 1)
Σ 2)) + A2
Σ ((Q(z)).
2 n²
Σ
+
"nnn -1 173
end
16 end
PROBABILISTIC INTERPRETATION AND THE MLE
The iWGAN has proposed efficient framework stably and automatically estimate both the encoder and the generator. In this section, we provide a probabilistic interpretation the iWGAN under the framework of maximum likelihood estimation.
Maximum likelihood estimator (MLE) is a fundamental statistical framework for learning models from data. However, for complex models, MLE can be computationally prohibitive due to the intractable normalization constant. MCMC has been used to approximate the intractable likelihood function but do not work efficiently in practice. The iWGAN can be treated for MILE training, which not only provides computational advantages but also allows us to generate more realistic-lookin images. Furthermore, this probabilistic interpretation enables other novel applications such as image quality checking and outlier detection.
adaptive method
Let X denote the image. Define the density of X by energy-based model based on autoencoder (Zhao et al. 2016, Berthelot et al. 2017):
p(=(0) = exp ( || Go(Qo(z))) V(0)), V(0) = log
Go(Qo (e
where 0 is the unknown parameter and V(9) is the log normalization constant. The major difficulty for the likelihood inference is due to the intractable function ((0). Suppose that we have the observed n).
data {a, : = 1.....n). The log- likelihood function of # {(() = n -1 Σn log p(r:(0), whose gradient is
(7) where B.nssl- [-] denotes the empirical average on the observed data {a;} and Ea|-] denotes the expectation under model p(z.(0). The key computational obstacle lies in the approximations of the
( = - B.s. [aula -G- (Qx( + E. [a+||= G. (2))))]]
5

---

Under review as a conference paper at ICLR 2020
model expectation E- To address this problem, propose novel dual approximation for this expectation. By Theorem 5.10 of Villani 2008). there exists optimal f such that
((..v))- [f" (v) - (=xx) = llv =I1] =1
(8)
for the optimal coupling Therefore, there exists a f such that f*) - ( (o(Q)(z)) with probability one with respect to the distribution of x. Since E& in 7 taken under the current estimated f and we also require Gy to be a good generator and the distributions of Go
and Ga (Qa ()) to be close, we approximate || Ga(Qa
by ( (2))
(Go(QuQee (2)()))
We replace | Go(Qot (Q)())) ||
in the second term of ( by f (Go(z) (G yielding gradient update for of form +- 0 + V (o)( where
Voe(O) Eora [061|= (o(Q)(=))]] + E. (oof"(G(())) 3..
Here f needs to be learned and is solved by the corresponding dual problem at each iteration. We approximate f by network fy with an unknown parameter 17. yielding a gradient update for 7) of form
(10) The advantage of using expectations in ( and (TO) is that we can evaluate them by using only marginal distributions of z and æ. The above iterative updating process is exactly the same as in Algorithm[] [ Therefore, the training of iWGAN is to seek the MLE. This probabilistic interpretation provides a novel alternative method tackle problems with the intractable normalization constant in latent variable models. The MLE gradient update of p(z(0) decreases the energy of the training data and increases the dual objective. Compare with original GANs or WGANs, our method gives much faster convergence and simultaneously provides higher quality generated images.
+ Ee (G,()()G(2)) - 8, (G.[(()(()))))]
The probabilistic modeling opens door for many interesting applications. Next, present completely new approach for determining highest density region (HDR) estimate for the distribution of X. What makes HDR distinct from other statistical methods that it finds the smallest region, denoted by Ua)) in the high dimensional space with a given probability coverage P(X= U (a)) = a. We can use (aa) to assess each individual sample quality, Note that FID or the Inception score are used measure the whole sample quality, not at the individual sample level.
a, i.e.,
Let i be the MLE. The density ratio at 21 and I2
p p (r2
exp - (1|= Gg(Qg(zi))I ||- G=(()(za))
The smaller the reconstruction error is, the larger the density value is. We can define the HDR for through the HDR for the reconstruction error e, := || Gz(Qg(z)) which is simple because it is one-dimensional problem. Let Ü (a.) be the HDR for e,, Then, U(a) {a e Ü(a)). Here (a)) defines the corresponding region in the latent space, which can be used to generate better quality samples.
EXPERIMENTAL RESULTS
5.1 MIXTURE OF GAUSSIANS
We train iWGAN model on three toy datasets with an increasing difficulty shown the right:
a). RING: a mixture of 8 Gaussians,
b). SPIRAL: mixture of 20 Gaus- sians and c). GRID; a mixture of 25 Gaussians. As the true data distribu- tions are known, this setting allows for tracking of convergence and mode dropping.
Duality gap and convergence: We illustrate that as the duality gap con- verges to O, our model converges to the generated samples from the true distribution. We keep track of the generated samples using G(z) and record the duality gap at each iteration to check the corresponding generated samples. Figure 2 shows the generated samples converge to the true distribution very fast without the mode collapse
(a) RING
(b) Swiss Roll
(c) GRID

---

Under review as a conference paper at ICLR 2020
RING
Swizs Rill
GRID
DHaNy
Figure 2: Duality gap and generated samples from iWGANs on mixture of Gaussians
problem. We compare our method with WGAN-GP in Figure 2 Both methods adopt the same structure, learning rate, number of critical steps, and other hyper-parameters. The iWGAN surpasses the performance of the WGAN-GP at very early stage and avoids the appearance of mode collapse.
Latent Space: We choose the latent distribution be a five dimensional standard normal distribution Z ~ (O, Is). After training, the distribution of Q(X) is expected to be close to the distribution of Z. We plot the (X); against (XX), for all i * in Figure 3 We can tell that the joint distribution of any two dimensions of Q(X) ) is close to bivariate normal distribution.
(a) RING
Swiss Roll
(c) GRID
Figure 3: Latent Space of Mixture of Gaussians
Individual sample quality check: From the probability interpretation of iw- GANs, naturally adopt the reconstruction error ||X G(Q(())II. the qual- ity score exp (-Il GG((())III as the metric the quality of any individ- ual sample. The larger the quality score is, the better quality the sample has. Figure[4shows their quality scores for different samples. The quality scores of samples near the modes of the true distribution are close smaller as the sample draw is away from the modes. This indicates that the iWGAN does converge and learns the distribution well, and the quality score is reliable metric for the indi- vidual sample quality.
and become
Figure Quality Check
5.2 MNIST & CELEBA
We experimentally demonstrate our model's ability well-known datasets, MNIST and CelebA. The results by iWGAN MNIST and CelebA are shown in Figure 5 and Figure 6 respectively. For latent space interpolations between MNIST or CelebA validation set examples. We sample pairs of validation set examples 21 and z2 and project them into 2 and 2n by the encoder. We then linearly interpolate between 21 and 2n and pass the intermediary points through the decoder plot the input-space interpolations. Figure]
7

---

Under review as a conference paper at ICLR 2020
displays images with high and low quality scores selected from CelebA. More experimental results, including reconstructed images, interpolations, and comparison with state-of-the-ar results by WGAN-GP, are shown in Appendix.
9
9
7
9
0
0
3
4 4
9
9
3
3
7
7 7
7
9
9
9
9
3
0
3
5
3
2
4
4
4
4
4
7
7
5
5
2
2
9
7 q
7
7
O
O
O
O
0
2
5
3
1
8
3
1
2
2
0
0
g
g
3
3
3
3
4
4 6
F 4
7 4
7
7
7
7
7 4
0
1
4
3
9
1
7
/
6
/
4
9
7
7
6
6
8
8
6
6
4
4
4
9
2
7
9 3
7
0
4
5
5
4
4
2
2
6
6
6
6
6
2
2
2
2
2
(a) Generated samples
(b) Reconstructed samples
(c) Interpolations
Figure 5: iWGAN on MNIST
(a) Generated samples
(h) Reconstructed samples
(c) Interpolations
Figure 6: iWGAN on CelebA
Figure 7: Images with high (left) and low (right) quality scores by iWGAN
CONCLUSION
We have developed novel iWGAN model, which fuses auto-encoders and GANs in principle way. We have established generalization enror bound for iWGAN. We have provided solid probabilistic interpretation on iWGAN using the maximum likelihood principle. Our training algorithm with an iterative primal and dual optimization has demonstrated proposed a stopping criteria for our algorithm and a metric for individual sample quality checking. The empirical results on both synthetic and benchmark datasets ofate-of-the-art.
efficient and stable learning. We have
We now mention several future directions for research on iWGAN. First, one might be interested in applying iWGAN into image-to-iman to-image translation, the extension should be straightforward. second direction is to develop a formal hypothesis testing procedure to test whether the samples generated from iWAGN is the same the data distribution.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv: /XX1://001.08755 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 224-232. JMLR. org, 2017.
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint arXiv: 1801.01973, 2018.
Peter Bartlett, Dylan Foster, and Matus Telgarsky. Sperttally-normalizzed margin bounds for neural networks. In Advances in Neural Information Processing Systems, pp. 6240-6249, 2017.
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv: 10717, 2017.
Xu Chen, Jiang Wang, and Hao Ge. Training generative adversarial networks via primal-dual subgradient methods: lagrangian perspective gan, arXiv preprint arXiv:1802.0111 1802.0176,, 2018.
Umberto Cherubini, Elisa Luciano, and Walter Vecchiato. Copula methods in finance. John Wiley & Sons, 2004.
Jeff Donahue, Philipp Krähenbühh, and Trevor Darrell. Adversarial feature learning. arXiv preprint
arXiv: arXiv: 1605.09722 2016.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint urXiv:1606.000 2016.
Farzan Farnia and David Tse. A convex duality framework for gans. In Advances in Neural Information Processing Systems, pp. 5248-5258, 2018.
Mevlana Gemici, Zeynep Akata, and Max Welling. arXiv:/805.00 09575,99555255 2018.
Primal-dual wasserstein gan. arXiv preprint
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Farle.Farley, Sherjil Ozair. Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, pp. 2672-26800 2014.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel (wo-sample test. Journal of Machine Learning Research, 13(Ma1):723-77,, 2012.
Paulina Grnarova. Kfir Y Levy, Aurelien Lucchi, Nathanael Perraudin, Thomas Hofmann, and Andreas Krause. Evaluating gans via duality. arXiv preprint a8Xiv: 1,005//, 2018.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville, Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Matthias Günther. Isometric embeddings of riemannian manifolds. In Proceedings of the International Congress of Mathematicians, pp. 1137- 143, 1991.
Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric P Xing. On unifying deep generative models, arXiv preprint arXiv: 100.::110000000000 00550, 2017.
Hisham Husain, Richard Nock, and Robert C Williamson. Adversarial networks and autoencoders: The primal-dual relationship and generalization bounds. arXiv preprint 1902::/902.0000 00985, 2019.
Haoming Jiang, Zhehui Chen, Minshuo Chen, Feng Liu, Dingding Wang, and Tuo Zhao. On computation and generalization of generative adversarial networks under spectrum control. 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Anders Boesen I Lindbo Larsen, Søren Kaae Spnderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In International Conference on Machine Learning, pp. 1558-1166, 2016.
Xingguo Li, Junwei Lu, Zhaoran Wang, Jarvis Haupt, and Tuo Zhao. On tighter generalization bound for deep neural networks: Cnns, resnets, and beyond. arXiv preprint 1806::8806.0511 0555805560069.8 2018
Yujia Li, Kevin Swersky, and Richard Zemel. Generative moment matching networks. arXiv preprint arXiv: 150.0027). 2015.

---

Under review as a conference paper at ICLR 2020
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unitying variational autoencoders and generative adversarial networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70. pp. 19000. JMLR, org, 2017.
Mehryar Mohri, Afshin Rostamizadeh, Ameet Talwalkar. Foundations of machine learning. MIT press. 2018.
John Nash. The embedding problem for riemannina manifolds. Annals of Mathematics, pp. 20 63, 1956.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances neural information processing systems, pp. 271-279.
2016.
Mihaela Rosca, Balaji Lakshminarayanna, David Warde-Faite Farley, and Shakir Mohamed. Variational approaches for auto auto-encoding generative adversarial networks. arXiv preprint arXiv:1706. 04987, 2017.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv:1711.01550 2017.
Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky. takes (only) two: Adversarial generator-encoder networks. In Secony- AAAI Conference on Artificial Intelligence, 2018.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXiv:1609.0311 03126, 2016.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. The information autoencoding family: A lagrangian perspective on latent variable generative models. arXiv preprint arXiv:1806.063: 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
APPENDIX A: PROOF OF THEOREM 2.1
According to the Nash embedding theorem Nash 1956 Günther [991], every d-dimensional smooth Rieman- nian manifold X' possesses a smooth isometric embedding into R" with p max max{d(( (((d+ 5)/2, d(d 5}- Therefore. there exists injective mapping u X' - Iº which preserves the metric in the sense that the manifold metric on X' is equal to the pullback of the usual Euclidean metric on I² by The mapping is injective so that we can define the inverse mapping u u= u(X) X.
Let X ui u(X) R², and write X (X....., x)). Let (z P.(X. S ..... P. be the marginal cdfs. By applying the probability integral transformation to each component, the random vector
(U1, U2. .0bb....UUU))) Up) =
F2(2)), Fx(X)))
has uniformly distributed marginals. Let C: [0, 1| - [0, 1] the copula of X, which defined as joint edf of (Uz,..., Up):
Clu1, u...... up) P(U; S u1, U2 u2...., Up S up).
The copula C contains all information the dependence structure among the components of X, while the marginal cumulative distribution functions F. contain all information on the marginal distributions. Therefore, the joint edf of X is
((,,, Fp) (( F. F2(22). Fp(Ep))
Define, for i ..
Cu(,,, ,.,...,, u.) = (u1, U2 Us 1, I,...,1). 1).
The conditional distribution of Un. given UI, ... U1-1, given by Cherubini et al. (2004)
(ux/uu ... We-1) = PP (U* ux = u1. U = ux-1)
10t Cx (Ui, ......... un) Ouk-1
1 ux m 1 ... 0ux-1]
for k = .....
We construct Q' as follows. First, we obtain X E RD by X u(X). Second, we transform X into random vector with uniformly distributed marginals and
Up) the marginal edf F. Then, define U1 Ui
........ Hence, Ül...., Up are independent uniform random variables. Finally, Z, = D-'(U,) for 1,...,P. This completes the transformation Q" from X to Z = (7.....,22 Zp).
Ux = (U Uk-1):
The above process can be inverted to obtain G First, we transform Z into independent uniform random variables by Ü. = (Z.) for .p. Next, let U1 = UT. Define
U+ = (Ü
i= 2
where OF 1-1(-lu (-lu1,-... ... uz) is the inverse of Ck and be obtained by numerical finding. Finally, let X, = (U.) for i = I..... and X u (XX), where w- u(x)) X is the inverse mapping of This completes the transformation G from Z to X.
APPENDIX B: PROOF OF THEOREM 2.2
By the iWGAN objective Bh. holds. Since W, is distance between two probability measures, (Pxxx, Pa(z)) < Wi(xx, )o(z)). there exists a Q E Q such that Q"(X) has the same distribution as Pz, we have
(PP,x, Po(z)) - Wx Poxe-(x)) Wi (Pocoe-(x)) Parz)) = Wi Pa(z)).
Hence, Wxxxx, Pa(z)) = Wi(Px. Pa(z)). Observe that sup, T, Q, = Wi(Px, Paioo))) + Wi Wi(Paeexp) Po(zz). By Thcorem 2.1 have infa.Q L(G, Q < L(G",Q",
0 when & and Q have enough capacity. Therefore, the duality gap is larger than (PxPx, Pa:e(x))) (Pa Paiz)) is easy see that, G outputs the same distribution as duality gap and W (Px, Pa(z)) zeros and X G(Q(X)) for X Px.
outputs the same distribution Z, both the
11

---

Under review as a conference paper at ICLR 2020
APPENDIX C: PROOF OF THEOREM 3.1
We first consider the difference between population Wi (Pxx Pa(z)) and empirical Wi (Pxx. .Pazz) given samples S {z1y-... Fa ). Let fi and fa be their witness function respectively. Using the dual form of 1- I-Wassertein distance, we have
Wi(xx, Pa(t)) (1x,x, Pa(z)
(xxxxxff(X)) - RZ~ry 1
Σ fa(m)) Rg-Py (f2)(()))()()
<Bxxpx(fi(X) B 2 [f(G)))
Σ fx.)) B2-p2 (f
I sup (f())
Σ f(x)) a 4SS).
Given another sample set S' = (Zi..... F In }. it is clear that
[f(x.) f(x))
|| 2
2B
(S) ((S)) v sup
n
where the second inequality is obtained since f is 1-Lipschitz continuous function. Applying McDiamond's Inequality, with probability at least 5/2 for any 6 E (0, 1). we have
2
2
((S) E (S)) B log
(11)
By the standard technique of symmetrization in Mohri (018)) have
[[) = E sup Ex Exp [S(X)]
S (mxs)
< (F).
(12)
It has heen proved in Mohri et al. 2018 that with probability at least 1 - 8/2 for any E (0, 1),
2
Rr, < Rt, (F)
B
log
5
(13)
Combining Equation 11
Equation
12 and Equation
13
have
2
Wi WiPxx, s Wi(Px,Papa + 2R + 3B
log
By Theorem 2.2, we have (P & (x(Px,Pa(z)) Thus,
(i(Px, (Px,,, Po(2)) + 25) 3B
log
APPENDIX D: EXTENSION TO f-GANS
This framework can be easily extended to other types of GANs. We consider f-GAN (Nowozin et al. 2016 here. Let : RR (-o0, oo] be function with l1() 0. The f-GAN minimizes the following objective for the generator G:
GAN, (Px, Po(2) = sup
Ex [xx)] - E2 [A (f(G(Z))
feF
where ht" (x)) = sup, (a-r--((y) } the convex conjugate of h. hxx) f-GAN recovers the original GAN (Goodfellow et al. 2014). If h(x) 0 when x = and h(x) oo otherwise, we have h" (x)
log (x)=(x+++)11 1og(x+1)-221 log
With the property that J is i-Lipschit function class, f-GNN turns to be WGAN.
Assume that F is the I-Lipschitz function class. We extent the iWGAN framework to the inference f-GAN (ifGAN) framework and define the new objective function as follows:
W (xxxPx. Po(z))
inf sup Ex |X G(Q(X)) Ex [(([((())) Ez (((G))) (14)
QEa feJ
12

---

Under review as a conference paper at ICLR 2020
Following this definition, have
W (1,x PGzz) inf (wi (Pxx (o(Q,)))) (PN)
QEO
We show GAN (Pxx Po(2)) I (xxx(xx, Po(2)) This is because GPx, P sup Ex [f(X)] - Ez 1h" (f(G(Z))
12F
inf
sup Ex [f(x)] - Ex [[C((((X)) + sup Ex (2C(( TEF
QEQ
JEF
I Po(z)).
This indicates the itGAN objective is an upper bound of f-GAN objective.
APPENDIX E: ARCHITECTURES
E1: MIXTURE GUASSIANS
Encoder architecture:
1 E R² + FC1024 + RELU
+ FC01 RELU
FC256 RELU
PC126 RELU FCs
Generator architecture:
E IR3
FC512 RELU
FCa12 + RELU
+ FCs12 + RELU + FC2
Discriminator architecture:
E R + FC512 RELU
FC512 REIU
FCa1 + RELU -> FC
E2: MNIST
Encoder architecture:
E R 28x28
Conv12s RELU
+ Conviss RELU
Convb12 RELU + FCs
Generator architecture:
E R
FCAXAX512 RELU
ConceTraasss RELU
Conell ransiss RELU - ConeTraass
Discriminator architecture:
E R 28 + Conviss RELU
+ Conv2ss RELU
Couv512
RELU - FC,
!Codes used for this paper will be available at: swt.lhha.....oooe//./.ggggg.oooe #bOY Towp
13

---

Under review as a conference paper at ICLR 2020
E3: CELEBA
Encoder architecture:
2 E R 64x64x3
+ ConU12s + LeakyRELL
Comu256
InstancsNorm + LeakyRELU
Conu512 InstanceNorm LeakyREELU Conv
Generator architecture:
R
FCAX4x1024
Conw' ranssra + BN RELU
ranssra + BN - RELU
ConcTrans128 + BN - RELU + ConvTranss
Discriminator architecture:
x E R 64x64x3 + ConU12s + LeakyRELU
Comu256 InstanceNorm + LeakyRELL
Conu512 InstaneeNorm LeakyREEL Convi
14

---

Under review as a conference paper at ICLR 2020
APPENDIX F: MORE EXPERIMENTAL RESULTS ON MIXTURE OF GAUSSIAN
We investigate the mode collapse prohlem for the iWGAN. If draw two random samples in the latent space 21, 22 . (0, Is), the interpolation, G(Az) + (1 2) _ À I 1, should fall around mode to represent a reasonable sample. In Figure we select E (0, 0.05, 0. 0.10, .1.,.... 0.95, 1.0), and do interpolations on two random samples. We repeat this procedure several times on 3 datasets as demonstrated in Figure 8 No matter where the interpolations start and end, the interpolations would fall around the modes other than the locations where distribution has low density. There may still be some samples that appears the middle of modes. This may be because the generator not able to approximate a step function well.
Figure 8: Interpolation: and indicates the first and last samples in the interpolations, other colored samples are the interpolations.
APPENDIX G: MORE EXPERIMENTAL RESULTS ON MNIST
G.1: LATENT SPACE
Figure shows the latent space of MNIST, i.e. Q(X); against Q(X); for all i +
G.2: GENERATED SAMPLES
Figure shows the comparison of random generated samples between WGAN-GP and iWGAN. Figure II shows examples of interpolations of two random generated samples.
G.3: RECONSTRUCTION
Figure 12 shows, based the samples from validation dataset, the distribution of reconstruction error. Figure L3 shows examples of reconstructed samples. Figure shows the worst samples based on quality scores from the validation dataset.
15

---

Under review as a conference paper at ICLR 2020
Figure 9: Latent Space of MNIST dataset
1
5
G
9
8
2
3
8
6
9
8
2
4
6
5
0
1
B
6
9
0
7
4
e
6
9
{
&
|
(a) WGAN-GP
0
8
7
6
7
6
4
4 6
o
4 4
102/
/
6
0
5
6 5 5
7
q
2
3
2 4
3 6
0
S
9
4
8
3
3
3
/ 9
0
0
6
4
4
1
(b) IWGAN
Figure 10: Generated samples on MNIST
16

---

Under review as a conference paper at ICLR 2020
2
2
2
2
8
7
7
9
0 9
5
5
9
9
9
5
5
5
9
9
7
7
7
7
9
9
9
9
9
9
9
8
Figure 11: Interpolations MNIST
Density for reconstruction error on test dataset
0.30
0.25
0.20
0.15
0.10
0.05
0.00
4
6
10
Figure Histogram of reconstruction error on MNIST
9
9
7
7
3
3
9
9
0
0
/
/
2
2
8
6
6
6
6
7
7
6
6
2233
7
7
7
7
/
2
2
8
8
4
4
Figure 13: Reconstructions on MNIST
17

---

Under review as a conference paper at ICLR 2020
(a) Samples with high quality scores
8 8
8
8
4
5
#
2
5
4
2
0
9 5
05
r
8 8
(b) Samples with low quality scores
Figure 14: Sample quality check by iWGAN on the validation dataset of MNIST
18

---

Under review as a conference paper at ICLR 2020
APPENDIX H: MORE EXPERIMENTAL RESULTS ON CELEBA
H.1 LATENT SPACE
Figure 15 shows the first dimensions of the latent space calculated by (()) on CelebA
H.2 RANDOM GENERATED SAMPLE
Figure T6 Figure 77 Figure 18 and Figure 19 display the random generated samples from Wasserstein GAN IWGAN, Wasserstein autoencoder(WAB) and Adversarial Learning Inference (ALI), respectively. Table I shows numerical comparison among these four methods based inception scores (IS), Fréchet inception distances (FID), reconstruction errors (RE), and maximum mean discrepancy (MMD) between encodings and standard normal random variables.
gradient penalty(
The IS and FID scores are calculated based on pre-trained inception models. However, as discussed in Barratt & Sharma (2018), inception score is not reliable metric for the wellness of generated samples. This is also consistent with experiments. Although WAE delivers the best inception scores among four methods, WAE also has worst FID scores. generated samples (Figure 18) show that WAE is not the best generative model compared with other three methods.
The reconstruction error (RE) is defined as
N RE
X
11
(15)
N
X, is the reconstructed sample for X,, RE used measure if the method has generated meaningful latent encodings. Smaller reconstruction errors indicate a more meaningful latent space which can decoded into the original samples.
The maximum mean discrepancy (MMD) is defined as
MMD = N(N 1)
,((,,2)) Σ x(2),,)) N(N - 1) (A)
2 N2
(16)
where l is a Desitive-definit reproducing kernel, 2,8 are drawn from prior distribution Pz. and & = Q(x.) are the latent encodings of real samples. MMD is used to measure the difference between distribution of latent encodings and standard normal random variables. Smaller MMD indicates that the distribution of encodings is close the standard normal distribution.
From Table [ in terms of generative models, iWGAN and ALI are better models, where WGAN-GP comes after, but WAE is suffering from generating clear pictures. In terms of RE and MMD, iWGAN and WAE are better choices, where ALI cannot always reconstruct the sample to itself (Figure 21]. In general, Table [ shows that IWGAN has successfully produce both meaningful encodings and reliable generator simultaneously.
Table 1: Comparison of iWGAN, ALI, WAE, WGAN-GP
Methods
IS
FID
RE
MMD
True
1.96(0.019)
18.63
IWGAN
1.51(0.017)
51.20
13.55(2.41)
10 3
ALI
1.50(0.014)
51.12
34.49(8.23)
0.39
WAE
1.71(0.029)
77.53
9.88(1.42)
4 X 10
WGAN-GP
1.54(0.016)
61.39
Figure 20 shows the interpolation comparison between IWGAN, WAE and ALI.
H.3: RECONSTRUCTION
Figure shows the distribution of reconstruction of CelebA. Figure 21 shows the comparison between real images and reconstructed images among three methods, IWGAN, WAE and ALI. Figure 23 shows samples with high and low quality scores CelebA validation sets.
19

---

Under review as a conference paper at ICLR 2020
Figure 15: Latent Space of CelebA dataset
Figure 16: Generated samples by WGAN-GP
20

---

Under review as a conference paper at ICLR 2020
Figure 17: Generated samples by iWGAN
Figure 18: Generated samples by WAE
21

---

Under review as a conference paper at ICLR 2020
Figure 19: Generated samples by ALI
iWGAN
WAE
ALI
Figure 20: Interpolations comparison among iWGAN, WAE and ALI
22

---

Under review as a conference paper at ICLR 2020
True
iWGAN
WAE
ALI
True
iWGAN
WAE
ALI
Figure 21: Reconstructed samples comparison among iWGAN, WAE and ALI
Density for reconstruction error CelebA
0.150
0.125
0.100
0.075
0,050
0.025
0.000
Figure 22: Histogram of reconstruction errors on CelebA
23

---

Under review as a conference paper at ICLR 2020
(a) Samples with high quality scores
(b) Samples with lower quality scores
Figure 23: Sample quality check by iWGAN on CelebA
24


Reviews:
[{"review_id": "Sylubr6TtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, an inference WGAN model (iWGAN) is proposed that fuses autoencoders and WGANs. By working on both the prime and dual problems, the proposed iWGAN takes an objective that is in general the upper bound of WGAN. The generalization error bound of iWGAN is analysed and its probabilistic interpretation under maximum likelihood estimation is provided.  The proposed iWGAN model is validated on both synthetic and real (MNIST and CelebA) datasets. \n\nPros:\nThis paper not only proposed the iWGAN model, but also provided some theoretical analysis about the iWGAN, such as the generalization error bound and the probabilistic interpretation. \nThrough the experiment, it seems that the proposed iWGAN works.  Especially, on the synthetic dataset, iWGAN seems to  have less mode collapse than WGAN.\n\nCons:\nThe major concerns of this paper lie in its experiment.  \n(1)\tThere is no quantitative comparison between iWGAN and WGAN, especially on the two real image datasets.  There are only some visual examples of the results in either the main body of the text or the appendix. \n(2)\tThe proposed iWAN is only compared with WGAN.  There is no experimental comparison with other autoencoder GANs  in the literature,  although the paper stated that iWGAN has many advantages over other autoencoder GANs (in the abstract).\n\nIn addition to experiment, other concerns include:\n(1)\tThe proposed method needs to be better motivated, although it is interpreted under the framework of maximum likelihood estimation. For example, what is the advantage to optimize the proposed upper bound of WGAN over its original objective? In the experiment, it seems iWGAN has less mode collapse than WGAN based on the synthetic data. But there is no explanation or discussion about why iWGAN could enjoy this favourable property.\n(2)\tThe proposed framework is tightly based on WGAN. It is not clear whether it could be extended to other GANs, which limits the significance of the proposed work.\n\nDue to the above concerns, rating is recommended as \"3 Weak Reject.\"\n\nSuggestions:  The authors are encouraged to provide extensive comparison with other autoencoder GANs and WGAN, especially in quantitative way.  \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Ske99gOnKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an inference WGAN (iWGAN) which fully considers to reduce the difference between distributions of G(X) and Z, G(Z) and X. In this algorithm, the authors show a rigorous probabilistic interpretation under the maximum likelihood principle. This algorithm has a stable and efficient training process. The authors provided a lots of theoretical and experimental analysis to show the effectiveness of the proposed algorithm. Therefore, the innovation of this paper is very novel. The theoretical analysis is sufficient, and the technology is sound. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkgZBiELqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel variation to the WGAN, which combines WGANs with autoencoders. The paper contains several interesting ideas and theoretical results. The proposed method is also demonstrated to perform reasonable in benchmark examples. One particularly nice property is that the authors derive a duality gap that can be used as a stopping criterion. \n\nMy main concern with the paper is that I don’t think it is sufficiently well written. In particular, the motivations for IWGAN presented in section 2 are both technically involved and vague. For instance, the optimisation problem is introduced with the sentence “The primal and dual formulations motivate us to define the iWGAN objective to be”, which does not motivate what we might benefit from using the objective. I generally feel that the presented arguments in favour of IWGAN are not sufficiently clear and convincing. Unfortunately, I also have similar concerns with later sections. In fact, even though the authors argue that the simulation results are also better than state of the art algorithms, my impression is that recent GAN modules often generate images which are even more realistic than this. \n\nIn spite of my concerns, I find the presented theory interesting and with better motivations and examples, it may eventually become a solid and well-cited contribution. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new way to stabilise GAN training.

The reviews were very mixed but taken together below acceptance threshold.

Rejection is recommended with strong motivation to work on the paper for next conference. This is potentially an important contribution. 
Decision: Reject