Title: AIM: ADVERSARIAL INFERENCE BY MATCHING PRI-ORS AND CONDITIONALS

Abstract: Effective inference for a generative adversarial model remains an important and challenging problem. We propose a novel approach, Adversarial Inference by Matching priors and conditionals (AIM), which explicitly matches prior and conditional distributions in both data and code spaces, and puts a direct constraint on the dependency structure of the generative model. We derive an equivalent form of the prior and conditional matching objective that can be optimized efficiently without any parametric assumption on the data. We validate the effectiveness of AIM on the MNIST, CIFAR-10, and CelebA datasets by conducting quantitative and qualitative evaluations. Results demonstrate that AIM significantly improves both reconstruction and generation as compared to other adversarial inference models.

1. INTRODUCTION:

Deep directed generative models like variational autoencoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014) and generative adversarial network (GAN) (Goodfellow et al., 2014) have been proved to be powerful for modeling complex high-dimensional distributions. While both VAE and GAN can learn to generate realistic images, their underlying mechanisms are fundamentally different. VAE maps the data into low-dimensional codes using an encoder, and then reconstructs the original data by a decoder. This allows it to perform both generation and inference. GAN, on the other hand, trains a generator and a discriminator adversarially. The generator learns to fool the discriminator by mapping low-dimensional noise vectors to the data space; at the same time, the discriminator evolves to detect the generated fake samples from the true ones. These two methods have complementary strengths and weaknesses. VAE can learn a bidirectional mapping between data and code spaces, but relies on over-simplified parametric assumptions on the complex data distribution, thereby causing it to generate blurry images (Donahue et al., 2016; Goodfellow et al., 2014; Larsen et al., 2015) . GAN generates more realistic samples than VAE (Radford et al., 2015; Larsen et al., 2015) because the adversarial regime allows it to learn more complex distributions. However, note that GAN only learns a unidirectional mapping for data generation, and does not allow inferring the latent codes from given samples. This is limiting because the ability of inference is very crucial for several downstream applications, such as classification, clustering, similarity search, and interpretation. Furthermore, GAN also suffers from the model collapse problem (Che et al., 2016; Salimans et al., 2016) -many modes of the data distribution are not represented in the generated samples.

Therefore, one may wonder on whether it is possible to develop a generative model that enjoys the strengths of both GAN and VAE without their inherent weaknesses. Such model should be able to generate high-quality samples as good as GAN, have an inference mechanism as effective as VAE, and also avoid the mode collapse issue. Many recent efforts have been devoted to combining VAE with adversarial discriminator(s) (Brock et al., 2016; Che et al., 2016; Larsen et al., 2015; Makhzani et al., 2015; Mescheder et al., 2017) . However, VAE-GAN hybrids tend to manifest a compromise of the strengths and weaknesses of both the approaches. The main reason is that all of them retain the autoencoder structure, which requires an explicit metric to measure the data reconstruction and assumes over-simplified parametric data distributions. To overcome such limitations, adversarially learned inference (ALI) (Donahue et al., 2016; Dumoulin et al., 2016) was recently proposed, wherein the discriminator is trained on the joint distribution of data and latent codes. In this way, under a perfect discriminator, one can successfully match joint distributions of the decoder and encoder, thereby, performing inference by sampling from the encoder's conditional distribution that also matches the decoder's posterior. However, in practice the equilibrium of the jointly adversarial game is hard to approach as the dependency structure between data and codes is not explicitly specified. ALI's reconstructions of samples are thus not always faithful (Dumoulin et al., 2016; Li et al., 2017) implying that its inference mechanism is not always effective.

To overcome the aforementioned issues, in this paper, we propose a novel approach, Adversarial Inference by Matching priors and conditionals (AIM), that integrates efficient inference to GAN and overcomes the limitations of prior approaches. The approach keeps the structure simple, involving only one generator, one encoder, and one discriminator. Furthermore, AIM's objective is directly derived from our goal of matching both prior and conditional distributions of the generator and encoder, instead of a heuristic combination with l k norm regularization. Compared to regular GANs, AIM has the ability to conduct inference, and also does not suffer from the mode collapse problem. Moreover, AIM also abandons the unrealistic parametric assumption on the conditional data distribution, and does not require any reconstruction in the data space. This is fundamentally different from VAE or VAE-GAN hybrids in which the l k norm is used to measure the data reconstruction. The usage of simple data-fitting metrics on the complex data distribution leads to worse generation performance. Different from ALI, AIM decomposes the hard problem of matching the joint distributions into two sub-tasks -explicitly matching the priors on the latent codes and the conditionals on the data. As a consequence of more restrictive constraint, it achieves better generation and more faithful reconstruction than ALI. Last but not the least, note that GAN variations with inference mechanism usually achieve worse generation performance as compared to regular GANs. However, as demonstrated in the experiments, AIM not only performs best on inference, but also improves the generation performance over GAN.

2. RELATED WORK:

The most straightforward way to learn an inference mechanism is to learn the inverse mapping of GAN's generator post-hoc (Zhu et al., 2016) . However, since its training process is the same as GAN, it still suffers from mode collapse problem. AGE (Ulyanov et al., 2017) encourages encoder and generator to be reciprocal by simultaneously minimizing an l 1 reconstruction error in the data space and an l 2 error in the code space. This is closely related to the cycle-consistency criterion (Zhu et al., 2017; Kim et al., 2017; Yi et al., 2017; Li et al., 2017) . Although the pairwise reconstruction errors help reduce mode collapse, the data reconstruction is still measured by l 1 or l 2 norm, which brings the same problem of VAE and VAE-GAN hybrids. InfoGAN (Chen et al., 2016) minimizes the mutual information between a subset c of the latent code and the generated samples, and hence can only do partial inference on c.

Different from the heuristic combination of VAE and GANs, Mescheder et al. (2017) theoretically derived an adversarial game to replace the KL-divergence term in the variational lower bound (also called ELBO), and gives the new method, adversarial variational Bayes (AVB), much more flexibility in its dependence on latent z. However, the reconstruction term on x still exists and so is the parametric assumption on the conditional data distribution, leading to the blurriness in their reconstructed and generated samples. ALI (Dumoulin et al., 2016; Donahue et al., 2016) is an elegant approach to bring inference mechanism into adversarial learning without assuming parametric distribution on the data. Different from our work, it directly plays an adversarial game to match the joint distributions of the decoder and encoder. But in practice, ALI's reconstructions are not necessarily faithful because the dependency structures within the two joint distributions are not specified (Li et al., 2017) . ALICE (Li et al., 2017) tries to solve this problem by regularizing ALI using an extra conditional entropy constraint on the data. The conditional entropy is either explicitly measured by l k norm, or implicitly learned by adversarial training. However, when the data distribution becomes complicated (e.g. CIFAR-10), the l k metric may lead to blurry reconstructions and the adversarial training is hard to achieve (Li et al., 2017) . Compared with ALI and ALICE, our method is proven to minimize the KL-divergence between both priors and conditionals of generator and encoder, and can provide consistent effective inference even on complicated distribution (see Section 5.1). Srivastava et al. (2017) proposed VEEGAN to tackle the mode collapse issue of GANs by adding an implicit variatinoal learning on the latent z. To our best knowledge, this is by far the only approach that is also reconstructing z. Different from VAEs, VEEGAN autoencodes the latent variable or noise z. By doing so, it enforces the generator not to collapse the mappings of z to a single mode, because otherwise, the encoder will not be able to recover all the noise z. The details of their model can be summarized as ALI regularized by an extra reconstruction of latent z. Therefore, VEEGAN is similar to ALICE in the sense that they are both adversarial games on the joint distribution with an extra regularization on either data or latent reconstruction. Our model AIM instead only plays the adversarial game on the marginal data distribution, and reconstructs the latent z by maximizing its log-likelihood under the latent posterior distribution. We consider the generative model p θ (z)p θ (x|z), where a latent variable z (i) is first generated from the prior distribution p θ (z), and then the data x (i) is sampled from the conditional distribution p θ (x|z). The parameter θ stands for the ground truth parameter of the distribution. The prior p θ (z) is always assumed to be a simple parametric distribution (e.g. N (0, I)), but the generative conditional p θ (x|z) is always very complex and not known to us. Moreover, the posterior p θ (z|x) is intractable but stands for an important inference procedure: given a data x (i) , it allows us to infer its latent variable z (i) .

3. BACKGROUND:



4. METHODOLOGY:

In our method, we will model the generating process by a neural network called generator, and the inference process by another neural network encoder. Consider the following two distributions of the generator and encoder, and their corresponding sampling procedures:

• the generator distribution: p(z)p(x|z); z ∼ p(z), x ∼ p(x|z).

• the encoder distribution: q(x)q(z|x); x ∼ q(x), z ∼ q(z|x).

The generator's conditional p(x|z) approximates the generating distribution p θ (x|z). The encoder's conditional q(z|x) approximates the posterior distribution p θ (z|x), which is what we need for inference. The marginal distribution q(x) stands for the empirical data distribution, and the other marginal p(z) is taken to be p θ (z), which is always a known distribution like standard Gaussian.

4.1. MATCHING PRIORS AND CONDITIONALS:

The ultimate goal is to match the joint distributions, p(x, z) and q(x, z). If this is achieved, we are guaranteed that all marginals match and all conditionals match as well. In particular, the conditional q(z|x) matches the posterior p(z|x). We propose to decompose this goal into two sub-tasksmatching the priors p(z) and q(z), and matching the conditionals p(x|z) and q(x|z). There are two advantages. Firstly, we explicitly define the dependency structure z → x. Secondly, the explicit constraints on both priors and conditionals are stronger than merely one constraint on the joint distributions.

In order to match the distributions, we will minimize the KL-divergence between the priors and between the conditionals:

EQUATION

By the properties of KL-divergence, when the minimum of equation 1 is attained, we have both p(z) = q(z) and p(x|z) = q(x|z), for all x and z.

4.2. OBJECTIVE FUNCTION:

The objective equation 1 cannot be directly optimized because both q(z) and q(x|z) are impossible to sample from, as the flow in the encoder is from x to z. Therefore, we rewrite equation 1 as the combination of a KL-divergence term and a reconstruction term, so that both of them only contain distributions that can be sampled from or directly evaluated.

Firstly, for any fixed z, the KL-divergence between the conditionals, D KL (p(x|z)||q(x|z)), can be decomposed as:

EQUATION

Then by adding (log p(z) -log p(z)) -log q(z) to both sides of equation 2, we have

D KL (p(x|z)||q(x|z))+(log p(z)-log q(z))-log p(z) = D KL (p(x|z)||q(x))-E p(x|z) [log q(z|x)].

Now taking expectation with respect to p(z) on both sides, we get

E p(z) D KL (p(x|z)||q(x|z)) + D KL (p(z)||q(z)) -E p(z) log p(z) = E p(z) D KL (p(x|z)||q(x)) I + E p(z) E p(x|z) [-log q(z|x)] II .

(3)

Note that the term E p(z) log p(z) is a constant because the prior p(z) is a fixed parametric distribution. For example, when z ∼ N (0, I d ), we have E p(z) log p(z) = -d(1 + log(2π))/2. Therefore, minimizing the objective equation 1 is now transformed to minimizing the new objective equation 3.

Intuitively, term (I) measures the difference between generated and real samples, and term (II) measures the reconstruction of the latent codes. We summarize the above procedure as a proposition.

Proposition 1 The objective function

E p(z) D KL (p(x|z)||q(x)) + E p(x|z) [-log q(z|x)]

is minimized when p(z) = q(z) and p(x|z) = q(x|z) for all z, x. And hence, at the minimum, the joint distributions p(x, z) = q(x, z).

4.3. RELATION TO VAE:

The VAE (Kingma and Welling, 2013) method, using our notations in this paper, actually depends on the following identity:

EQUATION

)

Then because of the non-negativity of KL-divergence, we have

log p(x) ≥ E q(z|x) [log p(x|z)] -D KL (q(z|x)||p(x)) ELBO ,

and hence maximizing the log-likelihood of the observations can be transferred to maximizing the evidence lower bound (ELBO). But taking a closer look at equation 4 and comparing it to equation 2, we notice that equation 4 is a decomposition of the KL-divergence between two conditionals, q(z|x) and p(z|x). Therefore, we can follow the same approach after equation 2 and get the following identity:

E q(x) D KL (q(z||x)||p(z|x)) + D KL (q(x)||p(x)) -E q(x) log q(x) = E q(x) D KL (q(z|x)||p(z)) Ivae + E q(x) E q(z|x) [-log p(x|z)] IIvae .

(5)

Since the marginal q(x) stands for the empirical data distribution, the right hand side of equation 5 is the empirical expectation of the negative ELBO, which is what VAE tries to minimize. We then conclude from equation 5 that VAE performs marginal distribution matching in the data space and conditional distribution matching in the latent space. This distribution matching of VAE is also observed by Rosca et al. (2018) .

However, the marginal distributions in the data space are very complex, and the direction x → z in the conditional distributions in the latent space is actually opposite to the generating process z → x. Hence, in order to match these distributions, VAE's objective has a reconstruction term II vae on x, and a regularization term I vae on latent z. But to evaluate both terms, we need to make parametric assumptions on both conditionals q(z|x) and p(x|z). The assumption on I vae can be loosed using GANs (Makhzani et al., 2015) , but the assumption on II vae is critical and limits the performance of VAE-GAN hybrids.

Our model, AIM, instead performs marginal distribution matching in the latent space and conditional distribution matching in the data space. From equation 3, since the term I will be replaced with an adversarial game (see Section 4.4), the only assumption we need to make is on term II, that is, on the conditional q(z|x). And our model is very flexible in its dependence on z. This assumption is much weaker than that on p(x|z) and does not lead to the problems of VAE or VAE-GANs (e.g. blurriness).

4.4. AIM FRAMEWORK:

The KL-divergence part (I) can be replaced by an adversarial game using the f -divergence theory (Nowozin et al., 2016) . The reconstruction term (II) is a log-likelihood and can be simply evaluated if we assume a parametric q(z|x). Therefore, our framework only requires exactly one generator G, one discriminator D, and one encoder E. We will now discuss how to play the adversarial game and measure the reconstruction in details.

Adversarial game Because we do not want to make any parametric assumption on the distribution p(x|z), an adversarial game will be played to distinguish p(x|z) from q(x). By the theory of f -GAN (Nowozin et al., 2016) , we then construct an adversarial game with the value function

EQUATION

Under the perfect discriminator, finding the optimal generator is then equivalent to minimizing the KL-divergence. The activation function for the discriminator in equation 6 is just the identity mapping instead of the sigmoid function in the original GAN. But just like in the original GAN, the generator of equation 6 also suffers from the gradient vanishing problem (Goodfellow, 2016) . Since the original GAN is well studied and suffices for our purpose, we will instead use its value function:

EQUATION

And as suggested in Goodfellow (2016) , in order to mitigate the generator's gradient vanishing problem, we minimize another value function -E z∼p(z) [log D(G(z))] for the generator.

Reconstruction Because of the simplicity of the distribution on z, we make a reasonable parametric assumption on q(z|x) so that the log-likelihood can be explicitly calculated. In this paper we will assume z|x ∼ N (µ(x), σ 2 (x)I), and hence

EQUATION

where d is the dimension of z. In this case, the encoder network only needs to output two vectors, µ(x) and σ 2 (x), that is, E(x) = (µ(x), σ 2 (x)). Then we can compute the approximate negative posterior log-likelihood by plugging E(x) into equation 8, i.e. L(z, E(x)).

To summarize, our final optimization problem is

EQUATION

Here, if we use V (G, D) in equation 6 derived from f -GAN, then λ = 1. If we use V (G, D) in equation 7, then λ needs to be set so that two parts of equation 9 are in the same scale. We will discuss this in detail in Section 5.

4.5. TRAINING AND INFERENCE PROCEDURES:

The training procedure is summarized in Algorithm 1. Given random z (i) ∼ p(z), we first generate samples x(i) ∼ p(x|z (i) ) using the generator. Then the discriminator is updated to distinguish between generated and real samples. The encoder outputs the parameters for the distribution q(z|x), from which we calculate the log-likelihood in (II). Then the generator and encoder are updated together to minimize the reconstruction error (i.e. maximize the expected log-likelihood), while the generator has an extra goal that is to fool the discriminator. The inference procedure is in Algorithm 2. For any data x (i) , its inferred latent code is set to be the conditional mean µ(x

(i) ) = E q(z|x (i) ) [z].

Then the reconstruction of x (i) is G(µ(x (i) )). Besides the reconstruction, we can also generate more samples which are close to x (i) in the sense that they have similar latent codes. This can be done by first sampling z's from the posterior q(z|x (i) ), and then map them to the data space using the generator.

Algorithm 1 The AIM training procedure.

θg, θ d , θe ← initialize network parameters repeat z (1) , . . . , z (n) ∼ p(z) Draw n samples from the prior x(j) ← G(z (j) ), j = 1, . . . , n

Generate samples using the generator network

(µ(x (j) ), σ 2 (x (j) )) ← E(x (j) )

Calculate mean and variance of q(z|x (j) ) ρ

(i) q ← D(x (i) ), i = 1, . . . , n Compute discriminator predictions ρ (j) p ← D(x (j) ), j = 1, . . . , n L d ← -1 n n i=1 log(ρ (i) q ) -1 n n j=1 log(1 -ρ (j) p )

Compute discriminator loss

Lg ← -1 n n j=1 log(ρ (j) p ) Compute generator loss Le ← λ n n j=1 L(z (j) , µ(x (j) ), σ 2 (x (j) ))

Compute encoder loss Lrec ← Lg + Le Compute reconstruction loss

θ d ← θ d -∇ θ d L d

Gradient update on discriminator network (θg, θe) ← (θg, θe) -∇ (θg ,θe) Lrec Gradient update on generator and encoder networks until convergence Algorithm 2 The AIM inference procedure.

x (i) ∼ q(x) Data to do inference on (µ(x (i) ), σ 2 (x (i) )) ← E(x (i) )

Calculate mean and variance of q(z|x

(i) ) x(i) ← G(µ(x (i) ))

Get reconstruction of x (i) z (j) ∼ N (µ(x (i) ), σ 2 (x (i) )), j = 1, . . . , m Sample z from the posterior q(z|x (i) ) x(i,j) ← G(z (j) ), j = 1, . . . , m Get similar neighbors of x (i)

4.6. REPARAMETRIZATION:

In order to sample from p(x|z) and q(z|x), we use the reparametrization trick (Kingma and Welling, 2013; Bengio et al., 2013; 2014) . Instead of directly sampling from a complex distribution, we can reparametrize the random variable as a deterministic transformation of an auxiliary noise variable , that is, u = f (v, ). In our case, to sample from x ∼ p(x|z), we can rewrite x = G(z, ), where z ∼ p(z) and is an auxiliary variable from a simple distribution. If we do not pass in any auxiliary noise , then the generator will be deterministic. For the encoder part, since q(z|x) = N (µ(x), σ 2 (x)I), one can draw samples by computing z = µ(x) + σ(x)

, where ∼ N (0, I).

5. EXPERIMENTS:

We evaluate our proposed method, AIM, for both reconstruction and generation tasks, on the data sets MNIST (LeCun et al., 1998) , CIFAR-10 (Krizhevsky and Hinton, 2009) and CelebA (Liu et al., 2015) . To show insights behind the effectiveness of AIM, we also conduct the same 2D Gaussian mixture experiment as in Dumoulin et al. (2016) , and explore the 2D latent representation on MNIST. The architectures of our discriminator and generator are based on DCGAN (Radford et al., 2015) and slightly simpler, which can be easily replaced by more advanced state-of-the-art GANs, and we use a deterministic generator throughout the experiments. Our encoder network consists of convolutional layers followed by two separated fully connected networks, which are used to predict the mean and variance of the posterior q(z|x), respectively. The Adam optimizer (Kingma and Ba, 2014) is used and the learning rate decay strategy suggested by Kingma and Ba ( 2014) is applied. Since the input to the log-function is one-dimensional in equation 7 but d-dimensional in equation 8, we set λ to be 1/d. We also observe that the discriminator shares a similar task with the encoder: both of them need to extract higher level features from raw images. Therefore, in order to reduce the number of parameters and to stabilize the training procedure, our encoder takes the intermediate hidden representation learned by the discriminator as its own input. It is worth noting that the encoder does not update the common feature extracting layers. More details about the model architecture and optimization methods are listed in the appendices.

5.1. QUANTITATIVE RESULTS ON REAL DATASETS:

In this section, we use quantitative measures to compare the inference and generation performance of AIM, GAN, ALI and ALICE. And for fair comparison, GAN is implemented to have the identical generator and discriminator with AIM. We also include a reduced version of AIM, named AIM-, in which the conditional distribution q(z|x) of the encoder is assumed to be a Gaussian with identity covariance matrix. To evaluate the performance of inference, we measure it through reconstructing test images and calculating the mean squared error (MSE), which has been adopted in Li et al. (2017) . As for generation, we calculate the inception score (Salimans et al., 2016) on 50, 000 randomly generated images. The inception scores on MNIST are evaluated by the pre-trained classifier from Li et al. (2017) , and the inception scores on CIFAR-10 is based on the ImageNet. The results are summarized in Table 1 . On the more complicated CIFAR-10 data set, AIM decreases the MSE by 95% and 97%. In order to alleviate the non-identifiable issue of ALI, ALICE adds the conditional entropy constraint by explicitly regularizing the l k norms between the reconstructed and real images. However, as the data distribution becomes more complicated like in CIFAR-10, the l k norms become inadequate to measure the reconstruction. Consequently, ALICE's reconstruction error on CIFAR-10 increases significantly compared with that on MNIST. In contrast, the reconstruction performance of AIM is consistent on both data sets. The reason is that our model explicitly specifies the dependency structure of the generative model, and matches both prior and conditional distributions without using the simple data-fitting l k metrics in the data space. This can be further justified by the performance of AIM-which follows the same structure. Compared with AIM-, AIM further decreases the MSE significantly by 49% on CIFAR-10, which shows that the inferred conditional variance is crucial for achieving the faithful reconstructions on complicated data sets.

Generation AIM outperforms all the baseline models including GAN on inception score. This suggests that AIM can bring further improvement on generation performance instead of deteriorating it like the other baselines. The reason that both ALI and ALICE perform worse than GAN on generation is that the task of matching two complicated joint distributions, p(z, x) and q(z, x), is more difficult than the task of the regular GAN, which is to match only the marginals, p(x) and q(x).

The proposed model AIM explicitly defines the dependency structure between z and x, which is more effective compared with one step joint distribution matching. Comparison between AIM and AIMshows that the learned variance is also critical for better generation performance. We also want to highlight that AIM's generation performance can be further improved by replacing the adversarial network with more advanced state-of-the-art GANs. 

5.2. RECONSTRUCTION RESULTS:

We compare the reconstruction produced by AIM with the results of joint distribution regimes. Since ALICE is sensitive to hyper-parameter, we show its reconstruction results with different hyperparameter settings in Appendix, from which we observe either unfaithful reconstructions or blurriness.

In Figure 2 , we compare reconstruction of AIM with the results reported in ALI(Dumoulin et al., 2016) (BiGAN (Donahue et al., 2016) ). From the first column of Figure 2 , we observe that ALI provides a certain level of reconstructions. However, it fails to capture the precise style of the original digits. In contrast, AIM can achieve very sharp and faithful reconstructions. On the more complicated dataset CIFAR-10, ALI's reconstructions are less faithful and oftentimes make mistakes in capturing exact object placement, color, style, and object identity. Our model produces better reconstructions in all these aspects. For the reconstructions of CelebA in column three, AIM reproduces the similar style, color and face placement, and even achieves a high level of face identity. As stated in Dumoulin et al. ( 2016), they believe ALI's unfaithful reconstructions is caused by underfitting. This also leads us to believe that our adversarial regime (marginal and conditional distribution matching) is more efficient for inference compared to joint distribution matching regimes. To show the effectiveness of our model on mode collapse reduction, we perform the same synthetic experiment as in Dumoulin et al. (2016) . The data is a 2D Gaussian mixture of 25 components laid out on a grid as shown in Figure 3a . To quantify the degree of mode collapse, we use the two metrics used in Srivastava et al. (2017) : the number of modes captured and the percentage of high quality samples. A generated sample is counted as high quality if it is within three standard deviations of the nearest mode. Then the number of modes captured is the number of mixture components whose mean is nearest to at least one high quality sample.

5.3. MODE COLLAPSE REDUCTION:

As shown in Table 2 , the proposed model AIM outperforms the other state-of-the-art models consistently on both measures. More specifically, AIM can capture 25 modes every time and generate more than 80% of high-quality samples. This suggests that the proposed model AIM significantly alleviates the mode collapse issue of the GAN framework and hence further improves the generation performance.

To further demonstrate the insights behind our model, we show the inferred latent codes, reconstructions, and generated samples by AIM together with the results by ALI in Figure 3 . Note that the ALI's result here is produced by the best-covering model in their paper, while the average number of modes captured by ALI is 15.84 compared to ours 25. From Figure 3b , we observe that our latent codes exhibit a disk shape and lie within the three standard deviations area of the standard Gaussian prior. Furthermore, they keep the relative spacial relationship of the test samples. In Figure 3c , AIM's reconstruction is almost identical to the original data (Figure 3a ), while ALI's reconstruction is less faithful and consists of connecting points between modes. Both models have similar generating performance as shown in Figure 3d . Some generated samples lie in the regions between the true modes. This is because the true modes are separated by large low-probability regions. It does not align well with the assumption of continuous prior distribution. The 2D latent codes of MNIST test data inferred by AIM is shown in Figure 10a . In a totally unsupervised manner, AIM does a great job at clustering the similar digits. The latent codes still lie in the standard Gaussian range but now exhibit no "holes", compared to Figure 3b . This is because the data distribution of MNIST is much more continuous than the Gaussian mixture. In Figure 10c , we show images generated by linearly interpolating between -2 and 2 along each dimension of the latent The generated images are sharp and the transitions between digits are smooth. This indicates that AIM learns smooth and meaningful representations which can generalize well. Because the explicit posterior distribution q(z|x) can be learned by the proposed model AIM, we can generate the similar samples conditioned on a given sample. This is meaningful in downstream task like data augmentation. In Figure 5 , based on any image x (i) in the first column, we conditionally generate more images by sampling z from the posterior q(z|x (i) ). The conditionally generated images are shown in the same row of x (i) . We observe that they have the similar style, color, and face placement with the original image.

5.5. UNSUPERVISED CONDITIONAL GENERATION:



6. CONCLUSION:

We proposed a novel framework, AIM, which matches both prior and conditional distributions between the generator and the encoder. Adversarial inference is incorporated into this framework and there is no parametric assumption on the conditional data distribution. Therefore, the proposed approach can not only learn an efficient inference mechanism but also improve the fidelity of generated samples. Extensive experiments on real datasets validate the effectiveness of the proposed model, and some insights are revealed by experiments on synthetic data. So just like VAE-GAN hybrids, ALICE can be treated as a VAE-ALI hybrids and also suffer from similar drawbacks. Therefore, adding conditional entropy into ALI's objective function heuristically manifests a compromise of the strengths and weaknesses of ALI and VAE.

In contrast, Figure 6 shows that our proposed model AIM can overcome the limitations of ALI and VAE. AIM's reconstructions capture the identifiable features in the original images, and are also very sharp without blurriness. In fact, we observe that AIM is able to reconstruct the images' background (color, texture), which are always lost in ALICE's reconstructions for any choice of λ.

A.2 ANALYSIS OF RECONSTRUCTION ERRORS:

We present the change of mean squared errors (MSE) on the first 50 training epochs in Figure 8 .

As can be seen, on all datasets, the MSEs decrease significantly during the first 10 epochs and then stay steadily at the similar minimum levels. Although the scales of reconstruction errors on different datasets at the beginning are different because of the various complexity of data distributions, the converged values of reconstruction errors are very similar. This observation proves that our inference mechanism can efficiently and consistently infer the latent codes of data samples, even when the data distribution is complicated. 

A.3 UNSUPERVISED CONDITIONALLY GENERATED IMAGES:

In Figure 9 , we present (unsupervised) conditionally generated samples of MNIST and CelebA experiments. Each element in the first column stands for an original data x (i) . Then based on x (i) , we conditionally generate more images by sampling z from the posterior q(z|x (i) ). The conditionally generated images are shown in the same row of x (i) .

A.4 RANDOMLY GENERATED IMAGES:

In Figure 10 , we show randomly generated images by sampling z from the marginal p(z). The generation results can be easily improved by replacing DCGAN structure with more advanced GAN structures.

B HYPERPARAMETERS FOR MNIST:

The details for the networks used in MNIST experiment are presented in Table 3 . Here the column "feature maps" for the fully-connected (FC) layers represents the dimension of their output. FC (µ) and FC (σ) are two fully-connected layers built on the previous common layer to separately predict mean µ(x) and variance σ 2 (x) of the conditional distribution q(z|x).

C HYPERPARAMETERS FOR CIFAR-10 AND CELEBA

The details for the networks used in CIFAR-10 and CelebA experiments are presented in Table 4 .

In order to reuse the same architecture for both datasets, we scaled CIFAR-10 images and center cropped CelebA images so that they have the same image size 64 × 64.

:



Figures and Tables:

Figure 1: This graphical model structure is shared with GAN and VAE.

Figure 2: Reconstruction comparison between our proposed model AIM (first row) and ALI (BiGAN) (second row) on MNIST, CIFAR-10 and CelebA. In each of the figures, the odd columns are original samples from the test set and the even columns are their reconstructions.

Figure 3: Comparison of our proposed model AIM (first row) with ALI (second row). The ALI's result shown here is the selected best-covering result reported in their original paper. And to follow the same settings with ALI, the number of points shown in column (b,c,d) is 10,000 which is 10% of the points shown in column (a).

Figure 5: Conditionally generated samples on the CelebA.

Figure 6: AIM's CelebA reconstructions. The odd columns are original samples from the CelebA test set and the even columns are their reconstructions.

Figure 7: ALICE's CelebA reconstructions with different hyper-parameter λ (reported in Pu et al. (2017)). The first column is the original samples from test set, and the following columns are reconstructions corresponding to different values of λ.

Figure 8: The MSE changes with epochs.

Figure 9: Conditionally generated samples (unsupervised). The first column consists of original images.

MSE (lower is better) and Inception scores (higher is better) on MNIST and CIFAR-10. ALI and ALICE results are from the experiments inLi et al. (2017).

Table content: <table><tr><td>Method</td><td>MSE</td><td colspan="2">MNIST Inception Score</td><td>MSE</td><td>CIFAR-10 Inception Score</td></tr><tr><td>GAN</td><td>-</td><td/><td>9.464 ± 0.020</td><td>-</td><td>6.287 ± 0.061</td></tr><tr><td>ALI</td><td colspan="2">0.480 ± 0.100</td><td>8.749 ± 0.090</td><td colspan="2">0.672 ± 0.113</td><td>5.930 ± 0.044</td></tr><tr><td colspan="3">ALICE 0.080 ± 0.007</td><td>9.279 ± 0.070</td><td colspan="2">0.416 ± 0.202</td><td>6.015 ± 0.028</td></tr><tr><td>AIM-</td><td colspan="2">0.028 ± 0.018</td><td>9.331 ± 0.021</td><td colspan="2">0.037 ± 0.017</td><td>6.324 ± 0.056</td></tr><tr><td>AIM</td><td colspan="2">0.026 ± 0.018</td><td>9.483 ± 0.020</td><td colspan="2">0.019 ± 0.009</td><td>6.450 ± 0.085</td></tr></table>

Degree of mode collapse, measured by modes captured (higher is better) and % high quality samples (higher is better) on 2D grid data. The baseline results are reported inSrivastava et al. (2017).

Table content: <table><tr><td/><td>Vanilla GAN</td><td>ALI</td><td colspan="3">Unrolled GAN VEEGAN AIM(ours)</td></tr><tr><td>Modes (Max 25)</td><td>3.3</td><td>15.84</td><td>23.6</td><td>24.6</td><td>25</td></tr><tr><td>% High Quality Samples</td><td>0.5</td><td>1.6</td><td>16</td><td>40</td><td>81.1</td></tr></table>

Peer review (iclr2019):
 UPDATE (after author response): Thank you for updating the paper, the revised version looks better and the reviewers addressed some of my concerns. I increased my score. There*s one point that the reviewers didn*t clearly address: "It might be worth evaluating the usefulness of the method on higher-dimensional examples where the analytic forms of q(x|z) and q(z) are known, e.g. plot KL between true and estimated distributions as a function of the number of dimensions." Please consider adding such an experiment. The current experiments show that the method works better on low-dimensional datasets, but the method does not seem to be clearly better on more challenging higher dimensional datasets. I agree with Reviewer1 that "Perhaps more ambitious applications would really show off the power of the model and make it standout from the existing crowd." Showing that the method outperforms other methods would definitely strengthen the paper. Section 5.4: I meant error bars in the numbers in the text, e.g. 13 +/- 5. --------- The paper proposes a new loss for training deep latent variable models. The novelty seems a bit limited, and the proposed method does not consistently seem to outperform existing methods in the experiments. I*d encourage the authors to add more experiments (see below for suggestions) and resubmit to a different venue. Section 4: - q(z) seems to be undefined. Is it the aggregated posterior? - How is equation (1) related to ELBO that is used for training VAEs? Some relevant references are missing: I’d love to see a discussion of how this loss relates to other VAE-GAN hybrids. VEEGAN: Reducing mode collapse in GANs using implicit variational learning https://arxiv.org/pdf/1705.07761.pdf Distribution Matching in Variational Inference https://arxiv.org/pdf/1802.06847.pdf Section 5.1: - The quantitative comparison measures MSE in pixel space and inception score, neither of which are particularly good measures for measuring the quality of how well the conditionals match. I’d encourage the authors to consider other metrics such as log-likelihood. - It might be worth evaluating the usefulness of the method on higher-dimensional examples where the analytic forms of q(x|z) and q(z) are known, e.g. plot KL between true and estimated distributions as a function of the number of dimensions. Section 5.4: - The error bars seem quite high. Is there a reason why the method cannot reliably reduce mode collapse? Minor issues: - CIFAT-10 -> CIFAR-10