Title: IWGAN: AN AUTOENCODER WGAN FOR INFERENCE

Abstract: Generative Adversarial Networks (GANs) have been impactful on many problems and applications but suffer from unstable training. Wasserstein GAN (WGAN) leverages the Wasserstein distance to avoid the caveats in the minmax two-player training of GANs but has other defects such as mode collapse and lack of metric to detect the convergence. We introduce a novel inference WGAN (iWGAN) model, which is a principled framework to fuse auto-encoders and WGANs. The iWGAN jointly learns an encoder network and a generative network using an iterative primal dual optimization process. We establish the generalization error bound of iWGANs. We further provide a rigorous probabilistic interpretation of our model under the framework of maximum likelihood estimation. The iWGAN, with a clear stopping criteria, has many advantages over other autoencoder GANs.
The empirical experiments show that our model greatly mitigates the symptom of mode collapse, speeds up the convergence, and is able to provide a measurement of quality check for each individual sample. We illustrate the ability of iWGANs by obtaining a competitive and stable performance with state-of-the-art for benchmark datasets.

1. INTRODUCTION:

One of the goals of generative modeling is to match the model distribution P θ (x) with parameters θ to the true data distribution P X for a random variable X ∈ X . For latent variable models, the data point X is generated from a latent variable Z ∈ Z through a conditional distribution P (X|Z).

Here X denotes the support for P X and Z denotes the support for P Z . There has been a surge of research on deep generative networks in recent years and the literature is too vast to summarize here (Kingma & Welling, 2013; Goodfellow et al., 2014; Li et al., 2015) . These models have provided a powerful framework for modeling complex high dimensional datasets. We start introducing two main approaches for generative modeling. The first one is called variational auto-encoders (VAEs) (Kingma & Welling, 2013) , which use variational inference to learn a model by maximizing the lower bound of the likelihood function. VAEs have elegant theoretical foundations but the drawback is that they tend to produce blurry images. The second approach is called generative adversarial networks (GANs) (Goodfellow et al., 2014) , which learn a model by using a powerful discriminator to distinguish between real data points and generative data points. GANs produce more visually realistic images but suffer from the unstable training and the mode collapse problem. Although there are many variants of generative models trying to take advantages of both VAEs and GANs (Tolstikhin et al., 2017; Rosca et al., 2017) , to the best of our knowledge, the model which provides a unifying framework combining the best of VAEs and GANs in a principled way is yet to be discovered.

1.1. RELATED WORK:

GANs and WGANs: The generative model is to learn a mapping, denoted by G, from Z to X to approximate the conditional distribution P G (X|Z) of the data point X ∈ X given latent code Z ∈ Z. We consider the deterministic mapping G in this work. Both GANs and Wasserstein GANs (WGANs) (Arjovsky et al., 2017) can be viewed as minimizing certain divergence between the data distribution P X and the generative model distribution P G(Z) . For example, the Jensen-Shannon (JS) divergence is implicitly used in GANs (Goodfellow et al., 2014) . The 1-Wasserstein distance between P X and P G(Z) , denoted by W 1 (P X , P G(Z) ), is employed in WGANs. Empirical experiments suggest that the Wasserstein distance is a more sensible measure to differentiate probability measures supported in low-dimensional manifold. In terms of training, it turns out that it is hard or even impossible to compute these standard divergences in probability, especially when P X is unknown and P G(Z) is parameterized by deep neural networks (DNNs). The training of GANs is converted into playing a game between two competing networks: the generator and the discriminator. The generator is to fool the discriminator and the discriminator is to distinguish between true data samples and generated samples. Instead, the training of WGANs is to study its dual problem because of the elegant form of Kantorovich-Rubinstein duality (Villani, 2008) . Analogous to GANs, the discriminator is now a real-valued 1-Lipschitz function. Many techniques such as weight clipping (Arjovsky et al., 2017) and gradient penalty (Gulrajani et al., 2017) are used to enforce the Lipschitz constraint. More discussions about WGANs will be presented in Section 2.

Autoencoder GANs: Larsen et al. ( 2016) first introduced the VAE-GAN, which is a hybrid of VAEs and GANs. The VAE-GAN uses a GAN discriminator to replace a VAE's decoder to learn the loss function. The motivation behind this modification is that VAEs tend to produce blurry outputs during the reconstruction phase. More recent VAE-GAN variants, such as Adversarial Generator Encoders (AGE) (Ulyanov et al., 2018) and Auto-encoding GANs (α-GAN) (Rosca et al., 2017) , use a separate encoder to stabilize GAN training. The main difference with standard GANs is that, besides the generator G, there is an encoder Q : X → Z which maps the data points into the latent space. This encoder is to approximate the conditional distribution Q(Z|X) of the latent variable Z given the data point X. Other encoder-decoder GANs are introduced in Adversarially Learned Inference (ALI) (Dumoulin et al., 2016) and Bidirectional Generative Adversarial Networks (BiGAN) (Donahue et al., 2016) . The objective of both ALI and BiGAN is to match two joint distributions under the framework of vanilla GANs, the joint distribution of (X, Q(X)) and the joint distribution of (G(Z), Z). When the algorithm achieves equilibrium, these two joint distributions roughly match. We are able to obtain more meaningful latent codes by Q(X), and this should improve the quality of the generator as well. Adversarial Variational Bayes (AVB) (Mescheder et al., 2017) presented a more flexible latent distribution to train Variational Autoencoders. Hu et al. (2017) provided new interpretations of GANs and VAEs and revealed strong connections between them which are linked by the classic wake-sleep algorithm.

Duality in GANs: Regarding the optimization perspectives of GANs, (Chen et al., 2018; Zhao et al., 2018) studied duality-based methods for improving algorithm performance for training. Primal-dual Wasserstein GANs (PD-GANs) are introduced in (Gemici et al., 2018) , which proposed a new penalty term whose evaluation samples are obtained from the encoder Q. Farnia & Tse (2018) developed a convex duality framework to address the case when the discriminator is constrained into a smaller class. Grnarova et al. (2018) developed an evaluation metric to detect the non-convergence behavior of vanilla GANs, which is the duality gap defined as the difference between the primal and the dual objective functions. Husain et al. (2019) investigated the close relationship between WAE (Tolstikhin et al., 2017) and f-GANs (Nowozin et al., 2016) , and proved generalization results for autoencoder models.

1.2. OUR CONTRIBUTIONS:

Although there are many interesting works on autoencoder GANs, it remains unclear what the principles are underlying the fusion of auto-encoders and GANs. For example, do there even exist these two mappings, the encoder Q and the decoder G, for any high-dimensional random variable X, such that Q(X) has the same distribution as Z and G(Z) has the same distribution as X? Is there any probabilistic interpretation such as the maximum likelihood principle on autoencoder GANs? We introduce inference Wasserstein GANs (iWGANs), which provide satisfying answers for these questions. We focus on the 1-Wasserstein distance, instead of the Kullback-Leibler divergence. We borrow the strength from both the primal and the dual problems and demonstrate the synergistic effect between these two optimizations. The encoder component tends out to be a natural consequence from our algorithm. Furthermore, the iWGAN has a rigorous probabilistic interpretation under the maximum likelihood principle, and our learning algorithm is equivalent to the maximum likelihood estimation when our model is defined as an energy-based model based on an autoencoder. Our main contributions are listed as below:

2. We establish a rigorous probability interpretation for iWGANs and our training process is exactly the same as the maximum likelihood estimation. As a byproduct, this interpretation allows us to perform the quality check at the individual sample level. 3. We demonstrate the natural use of the duality gap as a measure of convergence for iWGANs, and show its effectiveness for various numerical settings. Our experiments do not experience any mode collapse problem.

2. IWGAN:

The autoencoder generative model consists of two parts: an encoder Q and a generator G. The encoder Q maps a data sample x ∈ X to a latent variable z ∈ Z, and the generator G takes a latent variable z ∈ Z to produce a sample G(z). The autoencoder generative model should satisfy the following three conditions simultaneously: (a) The generator can generate images which have a similar distribution with observed images; (b) The encoder can produce meaningful encodings in the latent space; (c) The reconstruction errors of this model based on these meaningful encodings are small. The benefit of using an autoencoder is to encourage the model to better represent all the data it is trained with, so that it discourages mode-collapse.

We first show that, for any distribution residing on a smooth manifold, there always exists an encoder Q * which guarantees meaningful encodings and exists a generator G * which generates samples with the same distribution as data points by using these meaningful codes. Theorem 2.1. Consider a continuous random variable X ∈ X , where X is a d-dimensional smooth Riemannian manifold. Then, there exist two mappings Q * : X → R p and G * : R p → X , with p = max{d(d + 5)/2, d(d + 3)/2 + 5}, such that Q * (X) follows a multivariate normal distribution with zero mean and identity covariance matrix and

G * • Q * is an identity mapping, i.e., X = G * (Q * (X)).

Learning Q * and G * from the data points is a challenging task. Recall that the 1-Wasserstein distance between the data distribution P X and the generative model distribution P G(Z) is defined as

EQUATION

where • represents the L 2 -norm and Π(P X , P Z ) is a set of all joint distributions of (X, Z) with marginal measures P X and P Z , respectively. The main difficulty in (1) is to find the optimal coupling π and this is a constrained optimization with P X (x) = π(x, z)dz for x ∈ X and

P Z (z) = π(x, z)dx for z ∈ Z.

Based on the Kantorovich-Rubinstein duality, the WGAN studies the 1-Wasserstein distance (1) from the dual format

EQUATION

where F is the set of all bounded 1-Lipschitz functions. Weight clipping (Arjovsky et al., 2017) and gradient penalty (Gulrajani et al., 2017) have been used to satisfy the constraint of Lipschitz continuity. The experiment of (Arjovsky et al., 2017) showed that the WGAN can avoid the problem of gradient vanishment. However, the WGAN does not produce meaningful encodings and many experiments still display the problem of mode collapse (Arjovsky et al., 2017; Gulrajani et al., 2017) .

On the other hand, the Wasserstein Autoencoder (WAE) (Tolstikhin et al., 2017) , after introducing an encoder Q : X → Z to approximate the conditional distribution of Z given X, studies the reconstruction error inf Q∈Q E X X -G(Q(X)) , where Q is a set of encoder mappings whose elements satisfies P Q(X) = P Z . The penalty, such as D(P Q(X) , P Z ), is added to the objective to satisfy this constraint, where D is an arbitrary divergence between P Q(X) and P Z . The WAE can produce meaningful encodings and controlled reconstruction error. However, the WAE defines a generative model in an implicit way and does not model the generator through G(Z) with Z ∼ P Z directly.

To take the advantages of both WGAN and WAE, we propose a new autoencoder GAN model, called iWGAN, whose objective is

EQUATION

The term X -G(Q(X)) can be treated as the autoencoder reconstruction error as well as a loss to match the distributions between X and G(Q(X)). We note that the L 1 -norm • 1 has been used for the reconstruction term by α-GAN (Rosca et al., 2017) and CycleGAN (Zhu et al., 2017) .

Another term

E X∼P X f (G(Q(X))) -E Z∼P Z f (G(Z)

) can be treated as a loss for the generator as well as a loss to match the distribution between G(Q(X)) and G(Z). We emphasize that this term is different with the objective function of the WGAN in (2). Furthermore, it is challenging for practitioners to determine when to stop training GANs. Most of the GAN algorithms do not provide any explicit standard for the convergence of the model. However, the measure of convergence for iWGAN becomes very natural and we use the duality gap as the measure. The duality gap can be defined as

EQUATION

where

L(G, Q, f ) = E X X -G(Q(X)) + E X [f (G(Q(X)))] -E Z [f (G(Z))].

Theorem 2.2. The iWGAN objective (3) is equivalent to

W 1(PX , P G(Z) ) = inf Q∈Q W1(PX , P G(Q(X)) ) + W1(P G(Q(X)) , P G(Z) ) . (5) Therefore, W 1 (P X , P G(Z) ) ≤ W 1 (P X , P G(Z) ). If there exists a Q * ∈ Q such that Q * (X) has the same distribution with Z, then W 1 (P X , P G(Z) ) = W 1 (P X , P G(Z) ). Let ( Q, G, f

) be a fixed solution and assume that the encoder, generator, and discriminator all have enough capacities. Then the duality gap is larger than

W 1 (P X , P G( Q(X)) ) + W 1 (P G( Q(X)) , P G(Z)

). Moreover, if G outputs the same distribution as X and Q outputs the same distribution as Z, both the duality gap and W 1 (P X , P G(Z) ) are zeros and X = G( Q(X)) for X ∼ P X .

According to Theorem 2.2, the iWGAN objective is in general the upper bound of W 1 (P X , P G(Z) ).

However, this upper bound is tight. When the space Q includes a special encoder Q * such that Q * (X) has the same distribution as Z, the iWGAN objective is the exactly same as W 1 (P X , P G(Z) ). Theorem 2.2 also provides an appealing property from a practical point of view. The values of the duality gap and W 1 (P X , P G(Z) ) give us a natural criteria to justify the algorithm convergence.

3. GENERALIZATION ERROR BOUND AND THE ALGORITHM:

In practice, we minimize the empirical version, W 1 (P X , P G(Z) ), of W 1 (P X , P G(Z) ) to learn both the encoder and the decoder. Before we present the details of the algorithm, we first develop the generalization error bound for iWGANs. For discussions of generalization performance of classical GANs, see Arora et al. (2017) and Jiang et al. (2018) . Theorem 3.1. Given a generator G ∈ G, and given n samples (x 1 , . . . , x n ) from X = {x : x ≤ B}, with probability at least 1 -δ for any δ ∈ (0, 1), we have

EQUATION

where

R n (F) = E sup f ∈F n -1 n i=1 i f (x i )

is the empirical Rademacher complexity of the 1-Lipschitz function set F, in which i is the Rademacher variable.

Theorem 3.1 indicates that the 1-Wasserstein distance between P X and P G(Z) can be dominantly upper bounded by the empirical W 1 (P X , P G(Z) ) and Rademacher complexity of F. Since

W 1 (P X , P G(Z) ) ≤ W 1 (P X , P G(Q(X)) ) + W 1 (P G(Q(X)) , P G(Z) ) for any Q ∈ Q, the capacity of Q

determines the value of W 1 (P X , P G(Z) ). On the other hand, there are several existing results on the empirical Rademacher complexity of neural networks. When F is a set of 1-Lipschitz neural networks, we can apply the conclusion from Bartlett et al. (2017) to R n (F), which produces an upper bound scaling as O(B L 3 /n). Here L denotes the depth of network f ∈ F. Similar upper bound with an order of O(B Ld 2 /n) can be obtained by utilizing the results from Li et al. (2018) , where d is the width of the network.

In practice, we adopt the gradient penalty defined as GP(f ) = E X ( ∇ X f (X) 2 -1) 2 in (Gulrajani et al., 2017) to enforce the 1-Lipschitz constraint on f ∈ F. In addition, we use the maximum mean discrepancy (MMD) penalty (Gretton et al., 2012) , denoted by MMD k (P Q(X) , P Z ), to enforce Q(X) to converge to P Z , where k is a kernel function. The details of the algorithm are presented in Algorithm 1.

Algorithm 1: The training algorithm of iWGAN

while DualGap > 1 or L(G i , Q i , f i ) > 2 do for t = 1, ..., n critic do Sample real data x i ∼ P X , latent variable z i ∼ P Z and a random number ∼ U [0, 1] xi ← x i + (1 -)G i (z i ) Calculate L i = L(G i , Q i , f i |x i , z i ) and gradient of -L i Update f by Adam: f i+1 ← Adam(-∇ f L i ) where for f i , -∇ f L i = ∇ f 1 n n k=1 f i (G i (z i k )) -f i (G i (Q i (x i k ))) + λ1( ∇ xi f i (x i ) 2 -1) 2 end for t = 1, ..., n critic do Sample real data x i ∼ P X , latent variable z i ∼ P Z Calculate L i = L(G i , Q i , f i+1 |x i , z i ) and gradient of L i Update G, Q by Adam: G i+1 , Q i+1 ← Adam(∇ G,Q L i ) where for G i , Q i , ∇G,QL i = ∇G,Q 1 n n k=1 ( x i k -G i (Q i (x i k )) + f i+1 (G i (Q i (x i k ))) -f i+1 (G i (z i k ))) + λ2 n(n -1) l =j k(z i l , z i j ) + λ2 n(n -1) l =j k(Q(x i l ), Q(x i j )) - 2λ2 n 2 l,j k(z i l , Q(x i j ))

end end

4. PROBABILISTIC INTERPRETATION AND THE MLE:

The iWGAN has proposed an efficient framework to stably and automatically estimate both the encoder and the generator. In this section, we provide a probabilistic interpretation of the iWGAN under the framework of maximum likelihood estimation.

Maximum likelihood estimator (MLE) is a fundamental statistical framework for learning models from data. However, for complex models, MLE can be computationally prohibitive due to the intractable normalization constant. MCMC has been used to approximate the intractable likelihood function but do not work efficiently in practice. The iWGAN can be treated as an adaptive method for MLE training, which not only provides computational advantages but also allows us to generate more realistic-looking images. Furthermore, this probabilistic interpretation enables other novel applications such as image quality checking and outlier detection.

Let X denote the image. Define the density of X by an energy-based model based on an autoencoder (Zhao et al., 2016; Berthelot et al., 2017) :

p(x|θ) = exp -x -G θ (Q θ (x)) -V (θ) , V (θ) = log exp(-x -G θ (Q θ (x)) )dx,

where θ is the unknown parameter and V (θ) is the log normalization constant. The major difficulty for the likelihood inference is due to the intractable function V (θ). Suppose that we have the observed data {x i : i = 1, . . . , n}. The log-likelihood function of θ is (θ) = n -1 n i=1 log p(x i |θ), whose gradient is

EQUATION

where Êobs [•] denotes the empirical average on the observed data {x i } and E θ [•] denotes the expectation under model p(x|θ). The key computational obstacle lies in the approximations of the

model expectation E θ [•].

To address this problem, we propose a novel dual approximation for this expectation. By Theorem 5.10 of Villani (2008) , there exists an optimal f * such that

P (x,y)∼π f * (y) -f * (x) = y -x = 1 (8)

for the optimal coupling π. Therefore, there exists a f * such that f

* (x) -f * (G θ (Q θ (x))) = x - G θ (Q θ (x))

with probability one with respect to the distribution of x. Since E θ in ( 7) is taken under the current estimated θ and we also require G θ to be a good generator and the distributions of G θ (z) and G θ (Q θ (x)) to be close, we approximate

x -G θ (Q θ (x)) by f * (G θ (z)) -f * (G θ (Q θ (x))). We replace x -G θ (Q θ (x)) in the second term of (7) by f * (G θ (z)) -f * (G θ (Q θ (x))

), yielding a gradient update for θ of form θ ← θ + ∇θ (θ), where

EQUATION

)

Here f * needs to be learned and is solved by the corresponding dual problem at each iteration. We approximate f * by a network f η with an unknown parameter η, yielding a gradient update for η of form

EQUATION

The advantage of using expectations in ( 9) and ( 10) is that we can evaluate them by using only marginal distributions of z and x. The above iterative updating process is exactly the same as in Algorithm 1. Therefore, the training of iWGAN is to seek the MLE. This probabilistic interpretation provides a novel alternative method to tackle problems with the intractable normalization constant in latent variable models. The MLE gradient update of p(x|θ) decreases the energy of the training data and increases the dual objective. Compare with original GANs or WGANs, our method gives much faster convergence and simultaneously provides a higher quality generated images.

The probabilistic modeling opens a door for many interesting applications. Next, we present a completely new approach for determining a highest density region (HDR) estimate for the distribution of X. What makes HDR distinct from other statistical methods is that it finds the smallest region, denoted by U (α), in the high dimensional space with a given probability coverage 1 -α, i.e., P(X ∈ U (α)) = 1 -α. We can use U (α) to assess each individual sample quality. Note that FID or the Inception score are used to measure the whole sample quality, not at the individual sample level. Let θ be the MLE. The density ratio at x 1 and x 2 is

p(x1| θ) p(x2| θ) = exp -( x1 -G θ (Q θ (x1)) -x2 -G θ (Q θ (x2)) ) .

The smaller the reconstruction error is, the larger the density value is. We can define the HDR for x through the HDR for the reconstruction error e x := x -G θ (Q θ (x)) , which is simple because it is a one-dimensional problem. Let Ũ (α) be the HDR for e x . Then, U (α) = {x : e x ∈ Ũ (α)}. Here Q θ (U (α)) defines the corresponding region in the latent space, which can be used to generate better quality samples.

5. EXPERIMENTAL RESULTS:



5.1. MIXTURE OF GAUSSIANS:

We train our iWGAN model on three toy datasets with an increasing difficulty shown on the right: Duality gap and convergence: We illustrate that as the duality gap converges to 0, our model converges to the generated samples from the true distribution. We keep track of the generated samples using G(z) and record the duality gap at each iteration to check the corresponding generated samples. Figure 2 shows the generated samples converge to the true distribution very fast without the mode collapse Latent Space: We choose the latent distribution to be a five dimensional standard normal distribution Z ∼ N (0, I 5 ). After training, the distribution of Q(X) is expected to be close to the distribution of Z. We plot the Q(X) i against Q(X) j for all i = j in Figure 3 . We can tell that the joint distribution of any two dimensions of Q(X) is close to a bivariate normal distribution. From the probability interpretation of iW-GANs, we naturally adopt the reconstruction error X -G(Q(X)) , or the quality score exp (-X -G(Q(X)) ) as the metric of the quality of any individual sample.

The larger the quality score is, the better quality the sample has. We have developed a novel iWGAN model, which fuses auto-encoders and GANs in a principle way.

We have established a generalization error bound for iWGAN. We have provided a solid probabilistic interpretation on iWGAN using the maximum likelihood principle. Our training algorithm with an iterative primal and dual optimization has demonstrated an efficient and stable learning. We have proposed a stopping criteria for our algorithm and a metric for individual sample quality checking.

The empirical results on both synthetic and benchmark datasets are state-of-the-art.

We now mention several future directions for research on iWGAN. First, one might be interested in applying iWGAN into image-to-image translation, as the extension should be straightforward.

A second direction is to develop a formal hypothesis testing procedure to test whether the samples generated from iWAGN is the same as the data distribution.

Following this definition, we have

W 1,h (PX , P G(Z) ) = inf Q∈Q W1(PX , P G(Q(X)) ) + GAN h (P G(Q(X)) , P G(Z) ) . We show GAN h (PX , P G(Z) ) ≤ W 1,h (PX , P G(Z) ). This is because GAN h (PX , P G(Z) ) = sup f ∈F EX [f (X)] -EZ [h * (f (G(Z))] ≤ inf Q∈Q sup f ∈F EX [f (X)] -EX [f (G(Q(X)))] + sup f ∈F EX [f (G(Q(X)))] -EZ [h * (f (G(Z))] = W 1,h (PX , P G(Z)

). This indicates that the ifGAN objective ( 14) is an upper bound of the f-GAN objective.

APPENDIX E: ARCHITECTURES 1 E1: MIXTURE OF GUASSIANS Encoder architecture:

x

∈ R 2 → F C1024 → RELU → F C512 → RELU → F C256 → RELU → F C128 → RELU → F C5 Generator architecture: z ∈ R 5 → F C512 → RELU → F C512 → RELU → F C512 → RELU → F C2 Discriminator architecture: x ∈ R 2 → F C512 → RELU → F C512 → RELU → F C512 → RELU → F C1 E2: MNIST Encoder architecture: x ∈ R 28×28 → Conv128 → RELU → Conv256 → RELU → Conv512 → RELU → F C8 Generator architecture: z ∈ R 8 → F C4×4×512 → RELU → ConvT rans256 → RELU → ConvT rans128 → RELU → ConvT rans1

Discriminator architecture:

x ∈ R 28×28 → Conv128 → RELU → Conv256 → RELU → Conv512 → RELU → F C1

1 Codes used for this paper will be available at: https://drive.google.com/drive/folders/ 1-_vIrbOYwf2BH1lOrVEcEPJUxkyV5CiB?usp=sharing

APPENDIX F: MORE EXPERIMENTAL RESULTS ON MIXTURE OF GAUSSIAN:

We investigate the mode collapse problem for the iWGAN. If we draw two random samples in the latent space z1, z2 ∼ N (0, I5), the interpolation, G(λz1 + (1 -λ)z2), 0 ≤ λ ≤ 1, should fall around the mode to represent a reasonable sample. In Figure 8 , we select λ ∈ {0, 0.05, 0.10, . . . , 0.95, 1.0}, and do interpolations on two random samples. We repeat this procedure several times on 3 datasets as demonstrated in Figure 8 . No matter where the interpolations start and end, the interpolations would fall around the modes other than the locations where true distribution has a low density. There may still be some samples that appears in the middle of two modes. This may be because the generator G is not able to approximate a step function well. 9 shows the latent space of MNIST, i.e. Q(X)i against Q(X)j for all i = j.

G.2: GENERATED SAMPLES:

Figure 10 shows the comparison of random generated samples between WGAN-GP and iWGAN. Figure 11 shows examples of interpolations of two random generated samples.

G.3: RECONSTRUCTION:

Figure 12 shows, based on the samples from validation dataset, the distribution of reconstruction error. Figure 13 shows examples of reconstructed samples. Figure 14 shows the best and worst samples based on quality scores from the validation dataset. 

:

We propose a novel framework, called iWGAN, to learn both an encoder and a decoder simultaneously. We prove the existence of meaningful encoder and decoder, establish an equivalence between WGAN and iWGAN, and develop a generalization error bound for iWGAN.

Figures and Tables:

(a) RING (b) Swiss Roll (c) GRID a). RING: a mixture of 8 Gaussians, b). SPIRAL: a mixture of 20 Gaussians and c). GRID: a mixture of 25 Gaussians. As the true data distributions are known, this setting allows for tracking of convergence and mode dropping.

Figure 2: Duality gap and generated samples from iWGANs on mixture of Gaussians

Figure 3: Latent Space of Mixture of Gaussians

Figure 4: Quality Check

Figure 8: Interpolation: and indicates the first and last samples in the interpolations, other colored samples are the interpolations.

Figure9shows the latent space of MNIST, i.e. Q(X)i against Q(X)j for all i = j.

Figure 9: Latent Space of MNIST dataset

Figure 10: Generated samples on MNIST

Peer review (iclr2020):
In this paper, an inference WGAN model (iWGAN) is proposed that fuses autoencoders and WGANs. By working on both the prime and dual problems, the proposed iWGAN takes an objective that is in general the upper bound of WGAN. The generalization error bound of iWGAN is analysed and its probabilistic interpretation under maximum likelihood estimation is provided. The proposed iWGAN model is validated on both synthetic and real (MNIST and CelebA) datasets. Pros: This paper not only proposed the iWGAN model, but also provided some theoretical analysis about the iWGAN, such as the generalization error bound and the probabilistic interpretation. Through the experiment, it seems that the proposed iWGAN works. Especially, on the synthetic dataset, iWGAN seems to have less mode collapse than WGAN. Cons: The major concerns of this paper lie in its experiment. (1) There is no quantitative comparison between iWGAN and WGAN, especially on the two real image datasets. There are only some visual examples of the results in either the main body of the text or the appendix. (2) The proposed iWAN is only compared with WGAN. There is no experimental comparison with other autoencoder GANs in the literature, although the paper stated that iWGAN has many advantages over other autoencoder GANs (in the abstract). In addition to experiment, other concerns include: (1) The proposed method needs to be better motivated, although it is interpreted under the framework of maximum likelihood estimation. For example, what is the advantage to optimize the proposed upper bound of WGAN over its original objective? In the experiment, it seems iWGAN has less mode collapse than WGAN based on the synthetic data. But there is no explanation or discussion about why iWGAN could enjoy this favourable property. (2) The proposed framework is tightly based on WGAN. It is not clear whether it could be extended to other GANs, which limits the significance of the proposed work. Due to the above concerns, rating is recommended as *3 Weak Reject.* Suggestions: The authors are encouraged to provide extensive comparison with other autoencoder GANs and WGAN, especially in quantitative way.