IZATION FOR VARIATIONAL AUTOENCODERS
Year: 2019

Abstract:
Variational Autoencoder (VAE), a simple and effective deep generative model, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. However, recent studies demonstrate that, when equipped with expressive generative distributions (aka. decoders), VAE suffers from learning uninformative latent representations with the observation called KL Varnishing, in which case VAE collapses into an unconditional generative model. In this work, we introduce mutual posterior-divergence regularization, a novel regularization that is able to control the geometry of the latent space to accomplish meaningful representation learning, while achieving comparable or superior capability of density estimation. Experiments on three image benchmark datasets demonstrate that, when equipped with powerful decoders, our model performs well both on density estimation and representation learning.

heading: 1 INTRODUCTION
text: Representation learning, besides data distributions estimation, is a principle component in generative models. The goal is to identify and disentangle the underlying causal factors, to tease apart the underlying dependencies of the data, so that it becomes easier to understand, to classify, or to perform other tasks (Bengio et al., 2013). Among these generative models, VAE (Kingma & Welling, 2014; Rezende et al., 2014) gains popularity for its capability of estimating densities of complex distributions, while automatically learning meaningful (low-dimensional) representations from raw data. VAE, as a member of latent variable models (LVMs), defines the joint distribution between the observed data (visible variables) and a set of latent variables by factorizing it as the product of a prior over the latent variables and a conditional distribution of the visible variables given the latent ones (detailed in §2). VAEs are usually estimated by maximizing the likelihood of the observed data by marginalizing over the latent variables, typically via optimizing the evidence lower bound (ELBO). By learning a VAE from the data with the appropriate hierarchical structure of latent variables, the hope is to uncover and untangle the causal sources of variations that we are interested in.
A notorious problem of VAEs, however, is that the marginal likelihood may not guide the model to learn the intended latent variables. It may instead focus on explaining irrelevant but common correlations in the data (Ganchev et al., 2010). Extensive previous studies (Bowman et al., 2015; Chen et al., 2017a; Yang et al., 2017) showed that optimizing the ELBO objective is often completely disconnected from the goal of learning good representations. An extreme case called KL varnishing, happens when using sufficiently expressive decoding distributions such as auto-regressive ones; the latent variables are often completely ignored and the VAE regresses to a standard auto-regressive model (Larochelle & Murray, 2011; Oord et al., 2016).
This problem has spawned significant interests in analyzing and solving it from both theoretical and practical perspectives. We can only name a few here due to space limits. Some previous work (Bowman et al., 2015; Sønderby et al., 2016b; Serban et al., 2017) attributed the KL varnishing phenomenon to “optimization challenges” of VAEs, and proposed training methods including annealing the relative weight of the KL term in ELBO (Bowman et al., 2015; Sønderby et al., 2016b) or adding free bits (Kingma et al., 2016; Chen et al., 2017a). However, Chen et al. (2017a) pointed out that this phenomenon arises not just due to the optimization challenges, and even if we find the exact solution for the optimization problems, the latent code will still be ignored at optimum. They
proposed a solution by limiting the capacity of the decoder and applied PixelCNN (Oord et al., 2016) with small local receptive fields as the decoder of VAEs to model 2D images, achieving both impressive performance for density estimation and informative latent representations. Yang et al. (2017) embraced a similar idea and applied VAE to text modeling by using dilated CNN as the decoder. Unfortunately, these approaches require manual and problem-specific design of the decoder’s architecture to learn meaningful representations. Other studies attempted to explore alternatives of ELBO. Makhzani et al. (2015) proposed Adversarial Autoencoders (AAEs) by replacing the KL-divergence between the posterior and prior distributions with Jensen-Shannon divergence on the aggregated posterior distribution. InfoVAEs (Zhao et al., 2017) generalized the Jensen-Shannon divergence in AAEs to a divergence family and linked its objective to the Mutual Information between the data and the latent variables. However, directly optimizing these objectives is intractable, requiring advanced approximate learning methods such as adversarial learning or Maximum-Mean Discrepancy (Gretton et al., 2007; Dziugaite et al., 2015; Li et al., 2015). Moreover, these models’ performance on density estimation significantly falls behind state-of-the-art models (Salimans et al., 2017; Chen et al., 2017a).
In this paper, we propose to tackle the aforementioned representation learning challenges of VAEs by adding a data-dependent regularization to the ELBO objective. Our contributions are three-fold: (1) Algorithmically, we introduce the mutual posterior-divergence regularization for VAEs, named MAEs (§3.2), to control the geometry of the latent space during learning by encouraging the learned variational posteriors to be diverse (i.e. they are favored to be mutually “different” from each other), to achieve low-redundant, interpretable representation learning. (2) Theoretically, we establish a close relation between MAE and InfoVAE, by showing that the mutual posterior-devergence regularization maximizes a symmetric version of the KL divergence involved in InfoVAE’s mutual information term (§3.3). (3) Experimentally, on three benchmark datasets for images, we demonstrate the effectiveness of MAE as a density estimator by state-of-the-art log-likelihood results on MNIST and OMNIGLOT, and comparable result on CIFAR-10. Moreover, by performing image reconstruction, unsupervised and semi-supervised classification, we show that MAE is also capable of learning meaningful latent representations, even combined with a sufficiently powerful decoder (§4).

heading: 2 VARIATIONAL AUTOENCODERS

heading: 2.1 NOTATIONS
text: Throughout we use uppercase letters for random variables, and lowercase letters for realizations of the corresponding random variables. Let X ∈ X be the randoms variables of the observed data, e.g., X is an image or a sentence for image and text generation, respectively.
Let P denote the true distribution of the data, i.e., X ∼ P , and D = {x1, . . . , xN} be our training sample, where xi, i = 1, . . . , N, are usually i.i.d. samples of X . Let P = {Pθ : θ ∈ Θ} denote a parametric statistical model indexed by parameter θ ∈ Θ, where Θ is the parameter space. p is used to denote the density of corresponding distribution P . In the literature of deep generative models, deep neural networks are the most widely used parametric models. The goal of generative models is to learn the parameter θ such that Pθ can best approximate the true distribution P .

heading: 2.2 VAES
text: In the framework of VAEs, or general LVMs, a set of latent variables Z ∈ Z are introduced to characterize the hidden patterns of X , and the model distribution Pθ(X) is defined as the marginal of the joint distribution between X and Z:
pθ(x) = ∫ Z pθ(x, z)dµ(z) = ∫ Z pθ(x|z)pθ(z)dµ(z), ∀x ∈ X , (1)
where the joint distribution pθ(x, z) is factorized as the product of a prior pθ(z) over the latent Z, and the “generative” distribution pθ(x|z). µ(z) is the base measure on the latent space Z . Typically, prior pθ(z) is modeled with a simple distribution like multivariate Gaussian, or transforming simple priors to complex ones by normalizing flows and variants (Rezende & Mohamed, 2015; Kingma et al., 2016; Sønderby et al., 2016a).
To learn parameters θ, we wish to minimizes the negative log-likelihood of the parameters:
min θ∈Θ
1
N N∑ i=1 − log pθ(xi) = min θ∈Θ EP̃ (X)[− log pθ(X)] (2)
where P̃ (X) is the empirical distribution derived from training data D. In general, this marginal likelihood is intractable to compute or differentiate directly for high-dimensional latent space Z . Variational Inference (Wainwright et al., 2008) provides a solution to optimize the evidence lower bound (ELBO) an alternative objective by introducing a parametric inference model qφ(z|x):
Lelbo(θ, φ) = Ep(X) [ −Eqφ(Z|X)[log pθ(X|Z)] + KL(qφ(Z|X)||pθ(Z)) ] = Ep(X) [− log pθ(X) + KL(qφ(Z|X)||pθ(Z|X))] ≥ Ep(X) [− log pθ(X)]
(3)
where Lelbo could be seen as an autoencoding loss with qφ(z|x) being the encoder and pθ(x|z) being the decoder, with the first term in the RHS in (3) as the reconstruction error.

heading: 2.3 AUTOENCODING PROBLEM IN VAES
text: As discussed in Chen et al. (2017a), without further assumptions, the ELBO objective Lelbo in (3) may not guide the model towards the intended role for the latent variables Z, or even learn uninformative Z with the observation that the KL term KL(qφ(Z|X)||pθ(Z)) varnishes to zero. For example, suppose we use an auto-regressive decoder, pθ(x|z) = ∏ i pθ(xi|x<i, z), which is sufficiently expressive that it can model the data distribution P (X) without the assistance of Z, i.e, pθ(xi|x<i, z) = pθ(xi|x<i). In this case, the optimal Z w.r.t. Lelbo is the one independent with X , with the inference model reducing to the prior, i.e., qφ(Z|X) = pθ(Z),∀X ∈ X . The essential reason of this problem is that, under absolutely unsupervised setting, the marginal likelihood based objectiveLelbo incorporates no (direct) supervision on the latent space to characterize the latent variable Z with preferred properties w.r.t. representation learning. The main goal of this work is to explicitly control the geometry of the latent space, in the hope that preferred latent representations would be characterized and selected.

heading: 3 MUTUAL POSTERIOR-DIVERGENCE REGULARIZATION

heading: 3.1 GEOMETRIC PROPERTIES OF MEANINGFUL LATENT SPACE
text: Motivated by the Diversity-Inducing Mutual Angular Regularization (Xie et al., 2015) which is widely used in LVMs, we propose to regularize the posteriors qφ(z|x) of different data x ∈ X , to encourage them to diversely, smoothly, and evenly spread out in the data space X . The intuition is: (1) To make posteriors mutually diverse from each other, the patterns captured by different posteriors are likely to have less redundancy and hence characterizing and interpreting different data x. (2) To make posteriors smoothly and evenly distributed in the whole space of X , the shared patterns of similar data points are likely to be captured by their posteriors, to avoid isolating each data point from others. By balancing the diversity and smoothness of the distribution of posteriors, learned representations are encouraged to maintain global structured information, discarding detailed texture of local dependencies in the data.

heading: 3.2 MAES
text: Measure of Diversity. We propose to use expectation of the mutual KL-divergence between a pair of data to measure the diversity of posteriors. Specifically, the mutual posterior diversity is defined as:
MPD = EX1,X2∼P (X)[KL(qφ(Z|X1)||qφ(Z|X2))] (4) There are two main reasons we use KL-divergence instead of others as the measure of diversity: (1) KL-divergence is transformation invariant, i.e., for an invertible smooth function f ,
KL(qφ(Z|X1)||qφ(Z|X2)) = KL(qφ(f(Z)|X1)||qφ(f(Z)|X2)) It makes the computation efficient for complex posteriors that are transformed from simple ones, such as applying normalizing flows and variants (Rezende & Mohamed, 2015; Kingma et al., 2016;
Sønderby et al., 2016a). (2) KL-divergence has a close relation with mutual information, an important information-theoretic measure of the mutual dependence between two variables, which provides us the theoretical justification of the proposed regularizer (detailed in §3.3).
In MAEs, we propose to maximize the mutual posterior diversity (MPD) in (4). A straightforward way is to add negative MPD to the objective Lelbo in (3) that VAEs attempt to minimize. There are, however, two practical issues: (1) The scale of MPD, particularly for continuous Z, is much larger than that of Lelbo. We need to choose a hyperparameter carefully to control the scale of MPD, making optimization much more challenging and unstable. (2) For multivariate Z, e.g. a Kdimensional Z = (Z1, Z2, . . . , ZK), due to the property of KL-divergence, MPD may be dominated by a small group of dimensions, leaving others close to zero. In this case, most dimensions of Z are uninformative, which is not a desired representation.
To solve the two problems, in practice, we propose to minimize a MPD-based loss instead of directly maximizing MPD itself:
Ldiverse = EX1,X2∼P (X) [ K∑ k=1 log(1 + exp(−KL(qφ(Zk|X1)||qφ(Zk|X2)))) ] (5)
Ldiverse has two important good properties: (1) Ldiverse ≥ 0. (2) Ldiverse → 0 iff KL(qφ(Zk|X1)||qφ(Zk|X2)) → ∞,∀k. The first property sets a lower bound of Ldiverse, making optimization much more stable. The second one guarantees that all the dimensions of the latent S need to be mutually diverse w.r.t minimizing Ldiverse.
Measure of Smoothness. The smoothness of the distribution of posteriors is measured by utilizing the standard deviation of the mutual KL-divergence:
Lsmooth = STDX1,X2∼P (X)[KL(qφ(Z|X1)||qφ(Z|X2))] (6) where STD stands for standard deviation of random variables.
Lsmooth encourages the posteriors to smoothly and evenly spread out to different directions. Encouraging the standard deviation to be small can prevent the phenomenon that the posteriors fall into several small groups that are isolated from each other. It is crucially important for unsupervised clustering tasks, in which we want to cluster similar data into a big group instead of splitting them into multiple separated small groups (see §4.1.2 for detailed experimental results). Figure 1 shows two sets of distributions of data points, where the mean of the pairwise distances of the first set (Figure 1 (a)) is roughly the same as the second set (Figure 1 (b)). But the standard deviation of the first set is larger.
In the framework of MAEs, the final objective to minimize is: LMAE = Lelbo + ηLdiverse + γ Lsmooth (7) where η > 0, γ > 0 are regularization constants to balance the three losses in LMAE . Even though MAE introduces two extra hyperparameters η and γ, we find them easy to tune and MAE shows robust performance with different values of η and γ.
To solve (7), we can approximate Ldiverse and Lsmooth using Monte carlo in each mini-batch:
Ldiverse ≈ 1
M ∑ x1 6=x2 K∑ k=1 log(1 + exp(−KL(qφ(Zk|x1)||qφ(Zk|x2)))) (8)
where M is the number of valid pairs of data in each mini-batch. Lsmooth is approximately computed similarly.

heading: 3.3 THEORETICAL JUSTIFICATION
text: So far our discussion has been concentrated on the motivation and mathematical formulation of the proposed regularization method for VAE. In this section, we provide theoretical justification by connecting the mutual posterior diversity (MPD) in (4) with the mutual information term defined in InfoVAE (Zhao et al., 2017). With the end goal of theoretically justifying the proposed regularizer in mind, we first review the background of the mutual information (MI) term involved in the InfoVAE objective, which is central for linking MAE and InfoVAE.
Mutual Information Maximization. InfoVAE proposed the mutual information by first defining the joint “inference distribution”:
qφ(x, z) = p(x)qφ(z|x) where p(x) is the density of the true data distribution P (X). Then they added a mutual information maximization term that prefers high mutual information between X and Z under qφ(x, z) to the standard Lelbo: LInfoV AE = Lelbo − λIqφ(x,z)(x; z) and further proved that
Iqφ(x,z)(x; z) = EP (X)[KL(qφ(z|x)||qφ(z))] (9) where qφ(z) = ∫ X qφ(x, z)dµ(x) is the marginal of qφ(x, z). Mutual information inspired objectives have been explored in GANs (Goodfellow et al., 2014; Chen et al., 2016), clustering (Hinton et al., 1995; Krause et al., 2010) and representation learning (Esmaeili et al., 2018; Hjelm et al., 2018).
Relation between MPD and MI. The following theorem states our major result that reveals the relation between MPD and MI (proof in Appendix A): Theorem 1. The mutual posterior diversity (MPD) in (4) is a symmetric version of the KL-divergence of MI in (9):
MPD = EP (X)[KL(qφ(z|x)||qφ(z)) + KL(qφ(z)||qφ(z|x))] (10)
Roughly, Theorem 1 states that maximizing MPD and MI achieve the same goal: maximizing the divergence between the posterior distribution qφ(z|x) and the marginal qφ(z). Note that the (approximate) computation of MPD, as described in (8), is much easier than MI, which is generally intractable and requires adversarial learning or Maximum-Mean Discrepancy.

heading: 4 EXPERIMENTS
text: In this paper, we choose Variational Lossy Autoencoder (VLAE) (Chen et al., 2017a), VAE with auto-regressive flow (AF) prior, and auto-regressive decoder, as the basic architecture of our MAE models. More detailed descriptions, results, and analysis of the conducted experiments are provided in Appendix B.

heading: 4.1 BINARY IMAGES
text: We evaluate MAE on two binary images that are commonly used for evaluating deep generative models: MNIST (LeCun et al., 1998) and OMNIGLOT (Lake et al., 2013; Burda et al., 2015), both with dynamically binarized version (Burda et al., 2015). VLAE networks used in binary image datasets are similar of that described in Chen et al. (2017a): ResNet (He et al., 2016) encoder same as in ResNet VAE (Kingma et al., 2016), PixelCNN (Oord et al., 2016) decoder with 6 layers of masked convolution, and 32-dimensional latent code with AF prior implemented with MADE (Germain et al., 2015). The only difference is that the PixelCNN decoder has varying filter sizes: two 7x7 layers, followed by two 5x5 layers, and finally two 3x3 layers, instead of a fixed filter size of 3x3 used in Chen et al. (2017a). Hence the decoder we use has larger receptive field, to ensure that the decoder is sufficiently expressive. The same architecture is applied to all the experiments on both the two datasets. For pair comparison, we re-implemented VLAE using the same architecture in our MAE model. “Free bits” (Kingma et al., 2016) is used to improve optimization stability of VLAE (not for MAE). For hyperparameters η and γ, we explored a few configurations: η is selected from [0.5, 1.0, 2.0], and γ from [0.1, 0.5, 1.0].

heading: 4.1.1 DENSITY ESTIMATION
text: We first evaluate MAE on density estimation performance. Table 1 provides the results of MAE with different settings of hyperparameters on MNIST, together with previous top systems for comparison. Reported marginal negative log-likelihood (NLL) is evaluated with 4096 importance samples (Burda et al., 2015). Our MAE achieves state-of-the-art performance on both the two datasets, exceeding all previous models and the re-implemented VLAE. Note that our re-implementation of VLAE obtains better performance than the original one in Chen et al. (2017a), demonstrating the effectiveness of increasing decoder expressiveness by enlarging its receptive field.

heading: 4.1.2 REPRESENTATION LEARNING
text: In order to evaluate the quality of the learned latent representations, we conduct three sets of experiments — image reconstruction and generation, unsupervised clustering, and semi-supervised classification.
Image Reconstruction and Generation. The visualization of the of image reconstruction and generation on MNIST and OMNIGLOT is shown in Figure 2 and Figure 3. For comparison, we also show the reconstructed images from VLAE. MAE achieves better reconstruction ability than VLAE, proving that the latent code from MAE encodes more information from data.
Unsupervised Clustering. As discussed above, good latent representations need to capture global structured information and disentangle the underlying causal factors, rather than just memorizing the data. From this perspective, good image reconstruction results cannot guarantee good representations. To further evaluate the quality of the learned representation from MAE, we conduct the experiments of unsupervised clustering on MNIST. We perform K-Means clustering algorithm (Hartigan & Wong, 1979) on the learned representations. The class label of each cluster is assigned by finding the closest
sample in the training data with the cluster head. Evaluation of clustering accuracy is based on the assigned cluster labels. We run three experiments with K ∈ [10, 20, 30]. Table 2 (left section) illustrates the clustering performance. To make a thorough comparison, we also re-implemented a VAE model with a factorized decoder pθ(x|z) = ∏ i pθ(xi|z) and AF prior, which has been proven to obtain remarkable reconstruction performance. The VAE model uses ResNet (He et al., 2016) as its encoder and decoder similar to Chen et al. (2017a). From Table 2 we see that MAE significantly outperforms ResNet VAE and VLAE, especially when the number of clusters K is small. Interestingly, when K keeps increasing, clustering accuracy of ResNet VAE increases rapidly, showing that in its latent space the data are split into small groups.
In addition, towards the affects of η and γ on learned representations, MAEs with larger η obtain worse performance on K = 10. The reason might be that large η encourages the posteriors to diverse from each other, by splitting the data into small groups. Meanwhile, increasing γ is effective to
prevent the phenomenon, showing that in practice considerations on the trade-off between space diversity and smoothness are needed.
Semi-supervised Classification. For semi-supervised classification, we re-implemented the M1 model as described in Kingma et al. (2014). To test quality of information encoded in the latent representations, we choose two simple classifiers with limited capacity — K-nearest neighbor (K = 10) and linear logistic regression. For each classifier, we use different numbers of labeled data — 100, 1000 and all the training data from MNIST.
From the results listed in Table 2 (right section), MAE obtains the best classification accuracy on all the settings. Moreover, the improvements of MAE over ResNet VAE and VLAE are more significant when the number of labeled training data is small, further proving the meaningful representation learned from MAE.

heading: 4.2 NATURAL IMAGES
text: In addition to binary image datasets, we also applied MAE to CIFAR-10 dataset (Krizhevsky & Hinton, 2009) of natural images. The VLAE with DenseNet (Huang et al., 2017) encoder and PixelCNN++ (Salimans et al., 2017) decoder described in Chen et al. (2017a) is used as the neural architecture of MAE. To ensure that the decoder is sufficiently expressive, the decoder PixelCNN has 5 blocks of 96 feature maps and 7x4 receptive field. Hence the PixelCNN decoder we used is both deeper and wider than that used in Chen et al. (2017a).

heading: 4.3 DENSITY ESTIMATION
text: Density estimation performance on CIFAR-10 of MAEs with different hyperparameters is provided in Table 3, compared with the top-performing likelihood-based unconditional generative models (first section) and variationally trained latent-variable models (second section). MAE models obtain improvement over the VLAE re-implemented by us, and slightly fall behind the original one in Chen et al. (2017a). Compared with PixelSNAIL (Chen et al., 2017b), the state-of-the-art auto-regressive generative model, the performance of MAE models is around 0.11 bits/dim worse. Further improving the density estimation performance of MAEs on natural images has been left to future work.

heading: 4.4 IMAGE RECONSTRUCTION AND GENERATION
text: We also investigate learning informative representations on CIFAR-10 dataset. The visualization of image reconstruction and generation is shown in Figure 4, together with VLAE for comparison. It is interesting to note that MAE tends to preserve rather detailed shape information than VLAE, whereas the color information, particularly the color for background, is partially omitted. One reasonable explanation, as discussed in Chen et al. (2017a), is that color is predictable locally. This serves as one
example showing that MAEs can capture global structured information from data, omitting common correlations. Image samples from MAE are shown in Figure 4b.

heading: 5 CONCLUSION
text: In this paper, we proposed a mutual posterior-divergence regularization for VAEs, which controls the geometry of the latent space during training. By connecting the mutual posterior diversity with the mutual information, we have formally studied the theoretical properties of the proposed MAEs. Experiments on three benchmark datasets of images show the capability of MAEs on both density estimation and representation learning, with state-of-the-art or comparable likelihood, and superior performance on image reconstruction, unsupervised clustering and semi-supervised classification against previous top-performing models.
One potential direction for future work is to extend MAE to other forms of data, in particular text on which VAEs suffer a more serious KL-varnishing problem. Another exciting direction is to formally study the properties of the standard deviation of the mutual posterior KL-divergence used to measure smoothness, hence providing further justification of the proposed regularizer, or even introducing alternatives to further improve performances.

heading: ACKNOWLEDGEMENTS
text: The authors thank Zihang Dai, Junxian He, Di Wang and Zhengzhong Liu for their helpful discussions. This research was supported in part by DARPA grant FA8750-18-2-0018 funded under the AIDA program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA.

heading: APPENDIX: MAE: MUTUAL POSTERIOR-DIVERGENCE REGULARIZATION FOR VARIATIONAL AUTOENCODERS

heading: A PROOF OF THEOREM 1
text: Proof.
MPD = EX1,X2∼P (X)[KL(qφ(Z|X1)||qφ(Z|X2))]
= EX1,X2∼P (X) [H(qφ(Z|X1), qφ(Z|X2))−H(qφ(Z|X1))]
where H(·) denotes the entropy. Then,
EX1,X2∼P (X) [H(qφ(Z|X1))] = EP (X) [H(qφ(Z|X))]
and EX1,X2∼P (X) [H(qφ(Z|X1), qφ(Z|X2))]
= EX1,X2∼P (X) [ − ∫ Z qφ(z|x1) log qφ(z|x2)dz ] = EP (X2) [ − ∫∫ p(x1)qφ(z|x1) log qφ(z|x2)dz dx1 ]
= EP (X2) [ − ∫ (∫ p(x1)qφ(z|x1)dx1 ) log qφ(z|x2)dz ]
= EP (X2) [ − ∫ qφ(z) log qφ(z|x2)dz ] = EP (X) [H(qφ(Z), qφ(Z|X))]
So we have,
MPD = EP (X) [H(qφ(Z), qφ(Z|X))−H(qφ(Z|X))]
= EP (X) [H(qφ(Z), qφ(Z|X))−H(qφ(Z)) + H(qφ(Z))−H(qφ(Z|X))]
= EP (X)[KL(qφ(z|x)||qφ(z)) + KL(qφ(z)||qφ(z|x))]

heading: B DETAILED DESCRIPTION OF EXPERIMENTS
text: B.1 EXPERIMENTS FOR BINARY IMAGES
B.1.1 NEURAL NETWORK ARCHITECTURES AND TRAINING
The neural network architectures, including most of the hyperparameters, are the same as those in Chen et al. (2017a). The only difference in network architecture is the filter size of the PixelCNN decoder, which has been described in §4. For ResNet VAE with AF, we use the same ResNet encoder but a symmetric ResNet architecture for decoder. For encoder, we only use one stochastic layer with 32 dimensions.
In term of training, we use Adam optimizer (JLB, 2015) with learning rate 0.001, instead of Adamax used in Chen et al. (2017a). 0.01 nats/data-dim free bits was used in all the experiments. In order to get a relatively accurate approximation of Ldiverse and Lsmooth, we used a much larger batch size 100 in our experiments. Polyak averaging (Polyak & Juditsky, 1992) was used to compute the final parameters, with α = 0.999.
B.1.2 DETAILED RESULTS ON DENSITY ESTIMATION
Table 4 shows the detailed results of density estimation on MNIST. We see that increasing η always achieving more informative latent Z, but the NLL not always becomes better. It illustrates the hypothesis that good representations should encode global structured information in the data, rather than local dependencies. It is interesting to see that, the effect of γ on the latent Z is inconsistent — increasing γ from 0.1 to 0.5 leads to more informative Z (larger KL) and better NLL, but too large γ
(1.0) prevent the latent Z to learn more information from data (smaller KL), resulting worse NLL. Hence, in practice considerations on the trade-off between diversity and smoothness of the latent space are needed.
B.1.3 DETAILED RESULTS ON SEMI-SUPERVISED CLASSIFICATION WITH SVMS
Table 5 provides the performance of semi-supervised classification using SVMs with two different kernels — linear and RBF. MAE achieves the best classification accuracy on all the settings. It shoulld be noted that the accuracies of SVMs with non-linear kernels fluctuate more rapidly than linear ones, particularly when the number of labeled training data is small.
B.1.4 LATENT SPACE VISUALIZATION
Figure 5 visualize the latent spaces of VAEs and MAEs with different settings on MNIST, by t-Distributed Stochastic Neighbor Embedding (t-SNE) (Maaten & Hinton, 2008). The first row displays the visualizations of ResNet VAE, VLAE without free-bits training and VLAE with free-bits training. The following three rows display visualizations of MAEs with γ ∈ [0.1, 0.5, 1.0] and η ∈ [0.5, 1.0, 2.0]. We see that large η encourages the posteriors to diverse from each other, by splitting the data into small groups. Meanwhile, increasing γ is effective to prevent the phenomenon.
B.2 EXPERIMENTS FOR CIFAR-10
Following Kingma et al. (2016) and Chen et al. (2017a), latent codes are represented by 16 feature maps of 8x8. Prior distribution is factorized Gaussian transformed by 8 auto-regressive flows, each of which is implemented by 3-layer masked CNNs (Oord et al., 2016) with 128 feature maps. Between every other auto-regressive flow, the ordering of stochastic units is reversed. PixelCNN++ (Salimans et al., 2017) with 7x3 receptive field is used as the decoder. Due to the limitation of computational
resources, we used batch size 64 in our experiments. Same as experiments on binary images, Polyak averaging (Polyak & Juditsky, 1992) was used to compute the final parameters, with α = 0.999.

heading: C GENERATED SAMPLES FROM VLAE AND MAE
text: Figure 6 provides generated images on MNIST, OMNIGLOT and CIFAR-10 from VLAE and MAE.

Review summaries:
This paper presents a new regularization technique for VAEs similar in motivation and form to the work on InfoVAE.  The basic intuition is to encourage different training samples to occupy different parts of z-space, by maximizing the expected KL divergence between pairwise posteriors, which they call Mutual Posterior-Divergence (MPD).  They show that this objective is a symmetric version (sum of the forward and reverse KL) of the Mutual Info regularization used by the InfoVAE.  In practice however, they do not actually use this objective.  They use a different regularization which is based on the MPD loss but they say is more stable because it's always greater than zero, and ensures that all latent dimensions are used.  In addition to the MPD based term, they also add another term which encouraging the pairwise KL-divergences to have a low standard-deviation, to encourge more even spreading over the z-space rather than the clumpy distribution that they observed with only the MPD based term.