Year: 2019

Abstract:
We introduce the Mutual Information Machine (MIM), an autoencoder framework for learning joint distributions over observations and latent states. The model formulation reflects two key design principles: 1) symmetry, to encourage the encoder and decoder to learn different factorizations of the same underlying distribution; and 2) mutual information, to encourage the learning of useful representations for downstream tasks. The objective comprises the Jensen-Shannon divergence between the encoding and decoding joint distributions, plus a mutual information regularizer. We show that this can be bounded by a tractable cross-entropy loss between the true model and a parameterized approximation, and relate this to maximum likelihood estimation and variational autoencoders. Experiments show that MIM is capable of learning a latent representation with high mutual information, and good unsupervised clustering, while providing NLL comparable to VAE (with a sufficiently expressive architecture).

heading: 1 INTRODUCTION
text: Mutual information is a natural indicator of the quality of a learned representation (Hjelm et al., 2019), along with other characteristics, such as the compositionality of latent factors that are expected to be useful in downstream tasks, like transfer learning (Bengio et al., 2017). Mutual information is, however, computationally difficult to estimate for continuous high-dimensional random variables. As such, it can be hard to optimize when learning latent variable models (Belghazi et al., 2018; Chen et al., 2016a).
This paper formulates a new class of probabilistic autoencoder model that is motivated by two key design principles, namely, the maximization of mutual information, and the symmetry of the encoder-decoder components. Symmetry captures our desire for both the encoder and decoder to effectively and consistently model the underlying observation and latent domains. This is particularly useful for downstream tasks in which either one or both of the encoder and decoder play a central role. These properties are formulated in terms of the symmetric Jensen-Shannon Divergence between the encoder and decoder, combined with an objective term to maximize mutual information. We refer to the resulting model as the mutual information machine, or MIM.
We contrast MIM with models trained using (approximate) maximum likelihood, the canonical example being the variational auto-encoder, or VAE (Kingma and Welling, 2013; Rezende et al., 2014). The VAE comprises a probabilistic decoder and an approximate encoder, learned via optimization of an evidence-based lower bound (ELBO) on the log marginal data distribution. In contrast to MIM it is asymmetric in its formulation, and while often producing excellent representations, VAEs sometimes produce pathological results in which the encoder, or approximate posterior, conveys relatively little information between observations and latent states. This behavior, often referred to as posterior collapse,results in low mutual information between observations and inferred latent states (Bowman et al., 2015; Chen et al., 2016b; Razavi et al., 2019; van den Oord et al., 2016; 2017).
We formulate the MIM model, and a learning algorithm that minimizes an upper bound on the desired loss. The resulting objective can be viewed as a symmetrized form of KL divergence, thereby closely related to the asymmetric KL objective of the VAE. This also enables direct comparisons to the VAE in terms of posterior collapse, mutual information, data log likelihood, and clustering. Experiments show that MIM offers favourable mutual information, better clustering in the latent representation, and similar reconstruction, at the expense of sampling quality and data log likelihood when compared to VAE with the same architecture. We also demonstrate that for a sufficiently powerful architecture, MIM can match sampling quality and log likelihood of a VAE with the same architecture.

heading: 2 VARIATIONAL AUTOENCODERS
text: VAE learning entails optimization of a variational lower bound on the log-marginal likelihood of the data, logP(x), to estimate the parameters θ of an approximate posterior qθ(z|x) over latent states z (i.e., the encoder) and a corresponding decoder, pθ(x|z) (Kingma and Welling, 2013; Rezende et al., 2014). A prior over the latent space, P(z), often assumed to be an isotropic Gaussian, serves as a prior for qθ(z|x) in the evidence-lower-bound (ELBO) on the marginal likelihood: logP(x) ≥ Ez∼qθ(z|x) [ log pθ(x|z) ] − DKL (qθ(z|x) ‖P(z)). Here, we use the notation P(x) and P(z) to emphasize that these priors are given, and that we can draw random samples from them, but not necessarily evaluate the log-likelihood of samples under them. In what follows we often refer to them as anchors to further emphasize their role.
With amortized posterior inference, we take expectation over the observation distribution, P(x), to obtain the VAE objective:
RVAE (θ) = Ex∼P(x) [ Ez∼qθ(z|x) [ log pθ(x|z) ]−DKL (qθ(z|x) ‖P(z)) ] = Ex∼P(x),z∼qθ(z|x) [ log pθ(x|z) + logP(z)− log qθ(z|x) ] , (1)
Gradients of Eqn. (1) are estimated through MC sampling from qθ(z|x) with reparameterization, yielding unbiased low-variance gradient estimates (Kingma and Welling, 2013; Rezende et al., 2014).
VAEs are normally thought of as maximizing a lower bound on the data log-likelihood, however it can also be expressed as minimizing the divergence between two joint distributions over x and z. To see this, we first subtract logP(x) from (1), which does not change the gradients of the objective with respect to θ. We then negate the result, as we will be performing minimization. This yields LVAE (θ) = −RVAE (θ) + Ex∼P(x) [ logP(x) ] = DKL (qθ(z|x)P(x) ‖ pθ(x|z)P(z)) . (2)
The VAE optimization is therefore equivalent to minimizing the KL divergence between an encoding distribution qθ(z|x)P(x) and a decoding distribution pθ(x|z)P(z).

heading: 3 SYMMETRY AND MUTUAL INFORMATION
text: Our goal is to find a consistent encoder-decoder pair, representing a joint distribution over the observation and latent domains, with high mutual information between observations and latent states. By consistent, we mean that the encoding and decoding distributions qθ(z|x)P(x) and pθ(x|z)P(z), define the same joint distribution. Effectively, we aim to estimate an undirected graphical model with two valid factorizations. We note that consistency is achievable in the VAE when the approximate posterior qθ(z|x) is capable of representing the posterior under the decoding distribution pθ(z|x). In the general case, however, consistency is not usually achieved. In contrast to the asymmetric divergence between encoding and decoding distributions in the VAE objective (2), here we consider a symmetric measure, namely, the well-known Jensen-Shannon divergence (JSD),
JSD(θ) = 1
2
( DKL (pθ(x|z)P(z) ‖MS) +DKL (qθ(z|x)P(x) ‖MS) ) , (3)
whereMS is an equally weighted mixture of the encoding and decoding distributions; i.e.,
MS = 1
2
( pθ(x|z)P(z) + qθ(z|x)P(x) ) . (4)
In addition to encoder-decoder consistency, to learn useful latent representations we also want high mutual information between x and z. Indeed, the link between mutual information and representation learning has been explored in recent work (Hjelm et al., 2019; Belghazi et al., 2018; Chen et al., 2016a). Here, to emphasize high mutual information, we add a particular regularizer of the form
RH(θ) = 1
2
( H(pθ(x|z)P(z)) +H(qθ(z|x)P(x)) ) . (5)
This is the average of the joint entropy over x and z according to the encoding and decoding distributions. This is related to mutual information by the identity H(x, z) = H(x) + H(z) − I(x; z). That is, minimizing joint entropy encourages the minimization of the marginal entropy and maximization of the mutual information. In addition to encouraging high mutual information, one can show that this particular regularizer has a deep connection to JSD and the entropy ofMS , i.e., JSD(θ) + RH(θ) = H(MS) . (6) The derivation for Eqn. (6) is given in Appendix A.1.

heading: 4 MUTUAL INFORMATION MACHINE
text: Section 3 formulates a loss function (6) that reflects our desire for model symmetry and high mutual information. This objective is difficult to optimize directly since we do not know how to evaluate logP(x) in the general case (i.e., we do not have an exact closed-form expression for P(x)). As a consequence, we introduce parameterized approximate priors, qθ(x) and pθ(z), to derive tractable bounds on the penalized Jensen-Shannon divergence. This is similar in spirit to VAEs, which introduce a parameterized approximate posterior. These parameterized priors, together with the conditional encoder and decoder, qθ(z|x) and pθ(x|z), comprise a new pair of joint distributions, qθ(x, z) ≡ qθ (z|x) qθ (x), and pθ(x, z) ≡ pθ (x|z) pθ (z). These new joint distributions allow us to formulate a new, tractable loss that bounds H(MS); i.e.,
LCE(θ) ≡ H(MS ,Mθ) = DKL (MS ‖Mθ) +H(MS) ≥ H(MS) , (7) whereMθ = 12 ( pθ (x, z) + qθ (x, z) ) , and H(MS ,Mθ) denotes the cross-entropy betweenMS andMθ. We refer to LCE as the cross-entropy loss. It aims to match the model prior distributions to the anchors, while also minimizing H(MS). A key advantage of this formulation is that the cross-entropy loss can be trained by Monte Carlo sampling from the anchor distributions with reparameterization (Kingma and Welling, 2013; Rezende et al., 2014).
At this stage it might seem odd to introduce a parametric prior for P(z). Indeed, setting it directly is certainly an option. Nevertheless, in order to achieve consistency between pθ (x, z) and qθ (x, z) it can be advantageous to allow pθ(z) to vary. Essentially, we trade-off latent prior fidelity for increased model consistency. We provide more insights about this in Appendix E.3.
One issue with LCE is that, while it will try to enforce consistency between the model and the anchored distributions, i.e., pθ(x, z) ≈ pθ(x|z)P(z) and qθ(x, z) ≈ qθ(z|x)P(x), it will not directly try to achieve model consistency: pθ(x, z) ≈ qθ(x, z). To remedy this, we bound LCE using Jensen’s inequality, i.e.,
LMIM(θ) ≡ 1
2
( H(MS , qθ (x, z)) +H(MS , pθ (x, z)) ) ≥ LCE(θ) . (8)
Equation (8) gives us the loss function for the Mutual Information Machine (MIM). It is an average of cross entropy terms between the mixture distributionMS and the model encoding and decoding distributions respectively. To see that this encourages model consistency, it can be shown that LMIM is equivalent to LCE plus a non-negative model consistency regularizer; i.e.,
LMIM(θ) = LCE(θ) + RMIM(θ) . (9)
The non-negativity of RMIM is a simple consequence of LMIM(θ) ≥ LCE(θ) in (8). One can further show (see Appendix A.2) that RMIM(θ) satisfies
RMIM(θ) = 1
2
( DKL (MS ‖ pθ (x, z)) +DKL (MS ‖ qθ (x, z)) ) −DKL (MS ‖Mθ) . (10)
One can conclude from (10) that the regularizer RMIM is zero only when the two joint model distributions, qθ (x, z) and pθ (x, z), are identical under fair samples from the joint sample distribution MS (x, z). In practice we find that encouraging model consistency also helps stabilize learning. To understand the MIM objective in greater depth, we find it helpful to express LMIM as a sum of fundamental terms that provide some intuition for its expected behavior. In particular, as derived in the Appendix A.3,
LMIM(θ) = RH(θ) + 1
4
( DKL (P(z) ‖ pθ(z)) +DKL (P(x) ‖ qθ(x)) ) + 1
4
( DKL (qθ(z|x)P(x) ‖ pθ(x, z)) +DKL (pθ(x|z)P(z) ‖ qθ(z,x)) ) (11)
The first term in Eqn. (11) encourages high mutual information between observations and latent states. The second shows that MIM directly encourages the model priors to match the anchor distributions. Indeed, the KL term between the data anchor and the model prior is the maximum likelihood objective. The third term encourages consistency between the model distributions and the anchored distributions, in effect fitting the model decoder to samples drawn from the anchored encoder (cf. VAE), and, via symmetry, fitting the model encoder to samples drawn from the anchored decoder (both with
reparameterization). As such, MIM can be seen as simultaneously training and distilling a model distribution over the data into a latent variable model. The idea of distilling density models has been used in other domains, e.g., for parallelizing auto-regressive models (Oord et al., 2017).
In summary, the MIM loss provides an upper bound on the joint entropy of the observation and latent states under the mixture distributionMS : LMIM(θ) ≥ HMS (x) +HMS (z)− IMS (x; z). Through the MIM loss and the introduction of the parameterized model distributionMθ , we are pushing down on the entropy of the anchored mixture distributionMS , which is the sum of marginal entropies minus the mutual information. Minimizing the MIM bound yields consistency of the model encoder and decoder, and high mutual information underMS between observations and latent states.

heading: 5 EXPERIMENTS
text: In what follows we examine MIM empirically, with the VAE as a baseline. We consider synthetic datasets and well-known image datasets, namely MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017a) and Omniglot (Lake et al., 2015). All models were trained using Adam optimizer Kingma and Ba (2014) with a learning rate of 10−3, and a mini-batch size of 128. Following Alemi et al. (2017), we anneal the loss to stabilize the optimization. To this end we linearly increase β from 0 to 1 in the following expression for a number of ’warm-up’ epochs:
L̂MIM ( θ; {xi, zi}Ni=1, β ) = − 1
2N N∑ i=1 log (pθ(xi|zi) qθ(zi|xi))+β (log qθ(xi) pθ(zi)) . (12)
Training continues until the loss (i.e., with β = 1) on a held-out validation set has not improved for the same number of epochs as the warm-up steps (i.e., defined per experiment). We have found the number of epochs to convergence of MIM learning to be between 2 to 5 times greater than a VAE with the same architecture. A complete description of MIM learning is provided in Appendix C. (Code is available from https://www.dropbox.com/s/idnls2layat77sj/MIM-master.zip?dl=1 ).

heading: 5.1 POSTERIOR COLLAPSE IN LOW DIMENSIONAL DATA
text: Before turning to empirical results, it is useful to briefly revisit similarities and differences between MIM and the canonical VAE formulation. To that end, one can show from Eqn. (1) that the VAE loss can be expressed in a form that bears similarity to the MIM loss in Eqn. (8). In particular, following the derivation in Appendix D,
LVAE = 1
2
( H(MVAES , qθ(z|x)P(x)) +H(MVAES , pθ(x|z)P(z)) ) −HMVAES (z,x) (13)
whereMVAES (x, z) = qθ(z|x)P(x), and the entropy in the last term is taken under the sample distributionMVAES . Like the MIM loss, the first term in Eqn. (13) is the average of two cross-entropy terms, between a sample distribution and the encoding and decoding distributions. Unlike the MIM loss, these terms are asymmetric as samples are drawn only from the encoding distribution. Further, the last term in the VAE loss (13) is equivalent to −HMVAES (x) − HMVAES (z) + IMVAES (x; z), and thus it will also encourage a reduction in the mutual information. We posit that this plays a significant role in the phenomena often referred to as posterior collapse, in which the variance of the variational posterior grows large and the latent embedding conveys relatively little information about the observations (e.g., see (Chen et al., 2016b) and others).
To empirically support the expression in Eqn. (13), we consider experiments with synthetic data comprising 2D observations x ∈ R2, with a 2D latent space, z ∈ R2. In 2D one can easily visualize the model and measure quantitative properties of interest (e.g., mutual information). Fig. 1 depicts data drawn from a Gaussian mixture model comprising five isotropic components with standard deviation 0.25 (top), and latents from an isotropic standard Normal (bottom). The encoder and decoder conditional distributions are Gaussian whose means and variances are regressed from the input using two fully connected layers and tanh activation. The parameterized data prior, qθ(x), is defined to be the marginal of the decoding distribution (see Eqn. (23) in Appendix C.1), and the model prior pθ(z) is defined to be the anchor, so the only model parameters are those of the encoder and decoder (see Appendix C for details). We can therefore learn models with the MIM and VAE objectives, but with the same architecture and parameterization. In this section we used a warm-up

heading: VAE MIM VAE MIM VAE MIM
text: scheduler (Vaswani et al., 2017) with a warm-up of 3 steps, and with each epoch comprising 10000 samples. Training and test sets are drawn independently from the GMM.
Figure 1 depicts learned VAE and MIM models, with 5, 20 and 500 hidden units (from left to right) to control model expressiveness. Mutual information and root-mean-squared test reconstruction error are inset in the top row. For the weakest architecture (a), with 5 hidden units, VAE and MIM posterior variances are similar to the prior (MIM in one dimension), a sign of posterior collapse. As the number of hidden units increases (b,c), the VAE posterior variance remains large, preferring lower mutual information while matching the aggregated posterior to the prior. The MIM encoder produces tight posteriors, and yields higher mutual information and lower reconstruction errors.
To quantify this behavior Fig. 2 shows mutual information, the average negative log-likelihood (NLL) of test points under the model qθ, the mean reconstruction error of test points, and 5-NN classification performance1 (predicting which of 5 GMM components each test point was drawn from), all as functions of the number of hidden units. Following Belghazi et al. (2018), we estimate mutual information using the KSG estimator (Kraskov et al., 2004; Gao et al., 2016), based on 5-NN neighborhoods. The auxiliary classification task provides a proxy for representation quality. Mutual information, reconstruction RMSE, and classification accuracy for test data under the MIM model are better than with VAE models, at the expense of poorer data log likelihoods for MIM. In this case MIM learning finds near-invertible mappings with a sufficiently powerful architecture.
To further investigate MIM learning in low dimensional data, we project 784D images from FashionMNIST (Xiao et al., 2017b) onto a 20D linear subspace using PCA (capturing 78.5% of total variance). The training and validation sets had 50,000 and 10,000 images respectively. Fig. 3 summarizes the results, with MIM producing high mutual information and classification accuracy, at all but very low
1We experimented with 1-NN,3-NN,5-NN,10-NN and found the results to be consistent.
latent dimensions. MIM and VAE yield similar test reconstruction errors, with VAE having better negative log likelihoods for test data.
We conclude that the VAE is prone to posterior collapse for a wide range of models’ expressiveness and latent dimensionality, with latent embeddings exhibiting low mutual information. In contrast, MIM was empirically robust to posterior collapse. In this regard, we note that several papers have described ways to mitigate posterior collapse in VAE learning, e.g., by lower bounding, or annealing the KL divergence term in the VAE objective (Alemi et al., 2017; Razavi et al., 2019), or by limiting the expressiveness of the decoder (e.g., Chen et al. (2016b)). We posit that MIM does not suffer from this problem as a consequence of the objective design principles that encourage high mutual information between observations and the latent representation.

heading: 5.2 MIM LEARNING IN HIGH DIMENSIONAL IMAGE DATA
text: We next consider MIM and VAE learning with image data (Fashion-MNIST, MNIST, Omniglot). Unfortunately, with high dimensional data we cannot reliably compute mutual information (Belghazi et al., 2018). Instead, for model assessment we focus on log-likelihood, reconstruction, and the quality of random samples. In doing so we also explore multiple architectures, including the top performing models from Tomczak and Welling (2017), namely, convHVAE (L = 2) and PixelHVAE (L = 2), with Standard (S) priors2, and VampPrior (VP) priors3. The VP pseudo-inputs are initialized with training data samples. All the experiments below use the same experimental setup as in Tomczak and Welling (2017), and the same latent dimensionality z ∈ R80. Here we also demonstrate that a powerful prior (e.g., PixelHVAE (VP)) allows MIM to learn models with competitive NLL performance.
2pθ(z) = N (z;µ = 0, σ = I), a standard Normal distribution, where I is the identity matrix. 3pθ(z) =
1 K ∑K k=1 qθ(z|uk), a mixture model of the encoder conditioned on optimized pseudo-inputs uk.
Sampling from an auto-regressive decoder (e.g., PixelHVAE) is very slow. During training, we find that we can also learn effectively with a sampling distribution comprising just the encoding distribution, i.e., P(x) qθ(z|x), rather than the mixture of the encoder and decoder. We refer to this particular MIM variant as asymmetric MIM (or A-MIM). (For details, see Sec. C.3).
We report quantitative results in Table 1a for test negative log-likelihood (NLL). One can see that VAE models result in better NLL, but with a rather small gap for the more expressive models (i.e., PixelHVAE (VP)). We also show qualitative results for the most expressive models (i.e., PixelHVAE). Fig. 4 depicts reconstruction4 and sampling for Fashion-MNIST, MNIST, and Omniglot, for the top performing model (PixelHVAE (VP)), with MIM and VAE being comparable. See Appendix F for additional results.
The poor NLL and hence poor sampling for MIM with a weak prior model can be explained by the tightly clustered latent representation (e.g., Fig. 1). A more expressive, learnable prior can capture such clusters more accurately, and as such, also produces good samples (e.g., VampPrior). In other words, while VAE opts for better NLL and sampling at the expense of lower mutual information, MIM provides higher mutual information at the expense of the NLL for a weak prior, and comparable NLL and sampling with more expressive priors.

heading: 5.3 CLUSTERING AND CLASSIFICATION
text: Finally, following Hjelm et al. (2019), we consider an auxiliary classification task as a further measure of the quality of the learned representations. We opted for K-NN classification, being a non-parametric method which relies on semantic clustering in latent space, without any additional training. Given representations learned above in Sec. 5.2, a simple 5-NN classifier5 was applied to test data to predict one of 10 classes for MNIST and Fashion-MNIST.
Table 1b shows that in all but one case, MIM yields more accurate classification results. We attribute the performance difference to higher mutual information of MIM representations. Figure 5 provides a qualitative visualization of the latent clustering, for which t-SNE (van der Maaten and Hinton, 2008)) was used to project the latent space down to 2D. One can see that MIM learning tends to cluster classes in the latent representation more tightly, while VAE clusters are more diffuse and overlapping, consistent with the results in Table 1b.

heading: 6 RELATED WORK
text: Given the vast literature on generative models, here we only touch on the major bodies of work related to MIM, beyond the strong connection to VAEs.
As mentioned above, mutual information, together with disentanglement, is considered to be a cornerstone for useful representations (Belghazi et al., 2018; Hjelm et al., 2019). Normalizing flows
4Test data in the top row of Fig. 4 are binary, while reconstructions depict the probability of each pixel being 1, following Tomczak and Welling (2017)
5We omitted results for k ∈ {1, 3, 10} as we find them similar.
(Rezende and Mohamed, 2015; Dinh et al., 2014; 2016; Kingma and Dhariwal, 2018; Ho et al., 2019) directly maximizes mutual information by restricting the architecture to be invertible and tractable. This, however, requires the latent dimension to be the same as the dimension of the observations (i.e., no bottleneck). As a consequence, normalizing flows are not well suited to learning a concise representation of high dimensional data (e.g., images). Here, MIM often yields mappings that are approximately invertible, with high mutual information and low reconstruction errors.
Bornschein et al. (2015) share some of the same design principles as MIM, i.e., symmetry and encode/decoder consistency. However, their formulation models the joint density in terms of the geometric mean between the encoder and decoder, for which one must compute an expensive partition function. Pu et al. (2017) focuses on minimizing symmetric KL, but must use an adversarial learning procedure, while MIM can be minimized directly.
GANs (Goodfellow et al., 2014), which focus mainly on decoder properties, without a proper inference model, have been shown to minimize JSD between the data anchor P(x) and the model generative process qθ(x) (i.e., the marginal of the decoding distribution in MIM terms). In particular, prior work recognizes the importance of symmetry in learning generative models with reference to symmetric discriminators on x and z (Donahue et al., 2016; Dumoulin et al., 2017; Bang and Shim, 2018). In contrast, here we target JSD between the joint encoding and decoding distributions, together with a regularizer to encourage high mutual information.

heading: 7 CONCLUSIONS
text: We introduce a new representation learning framework, named the mutual information machine (MIM), that defines a generative model which directly targets high mutual information (i.e., between the observations and the latent representation), and symmetry (i.e., consistency of encoding and decoding factorizations of the joint distribution). We derive a variational bound that enables the maximizion of mutual information in the learned representation for high dimensional continuous data, without the need to directly compute it. We then provide a possible explanation for the phenomena of posterior collapse, and demonstrate that MIM does not suffer from it. Empirical comparisons to VAEs show that MIM learning leads to higher mutual information and better clustering (and classification) in the latent representation, given the same architecture and parametrization. In addition, we show that MIM can provide reconstruction error similar to a deterministic auto-encoder, when the dimensionality of the latent representation is equal to that of the observations. Such behaviour can potentially allow approximate invertibility when the dimensionality differs, with a stochastic mapping that is defined through consistency and high mutual information.
In future work, we intend to focus on utilizing the high mutual information mapping provided by MIM, by exploiting the clustered latent representation to further improve the resulting generative model.

heading: Appendix

heading: Table of Contents

heading: A Derivations for MIM Formulation 11
text: A.1 Regularized JSD and Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 A.2 MIM Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 A.3 MIM Loss Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

heading: B MIM in terms of Symmetric KL Divergence 14

heading: C Learning 15
text: C.1 MIM Parametric Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 C.2 Gradient Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 C.3 Training Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

heading: D Posterior Collapse in VAE 18

heading: E Additional Experiments 18
text: E.1 Entropy as Mutual Information Regularizer . . . . . . . . . . . . . . . . . . . . 19 E.2 Consistency regularizer in LMIM . . . . . . . . . . . . . . . . . . . . . . . . . . 19 E.3 Parameterizing the Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 E.4 Effect of Consistency Regularizer on Optimization . . . . . . . . . . . . . . . . 21

heading: F Additional Results 22
text: F.1 Reconstruction and Samples for MIM and A-MIM . . . . . . . . . . . . . . . . 22 F.2 Latent Embeddings for MIM and A-MIM . . . . . . . . . . . . . . . . . . . . . 25
A DERIVATIONS FOR MIM FORMULATION
In what follows we provide detailed derivations of key elements of the formulation in the paper, namely, Equations (6), (9), and (11). We also consider the relation between MIM based on the Jensen-Shannon divergence and the symmetric KL divergence.

heading: A.1 REGULARIZED JSD AND ENTROPY
text: First we develop the relation in Equation (6), between Jensen-Shannon divergence of the encoder and decoder, the average joint entropy of the encoder and decoder, and the joint entropy of the mixture distributionMS . The Jensen-Shannon divergence with respect to the encoding distribution qθ(z|x)P(x) and the decoding distribution pθ(x|z)P(z) is defined as
JSD(θ) = 1
2 (DKL (pθ(x|z)P(z) ‖MS) +DKL (qθ(z|x)P(x) ‖MS))
= 1
2
( H(pθ(x|z)P(z),MS)−H(pθ(x|z)P(z))
+H(qθ(z|x)P(x),MS)−H(qθ(z|x)P(x)) )
WhereMS = 12 (pθ(x|z)P(z) + qθ(z|x)P(x)) is a mixture of the encoding and decoding distributions. Adding on the regularizer RH(θ) = 12 (H(pθ(x|z)P(z)) +H(qθ(z|x)P(x))) gives
JSD(θ) + RH(θ) = 1
2 (H(pθ(x|z)P(z),MS) +H(qθ(z|x)P(x),MS))
= H(MS)

heading: A.2 MIM CONSISTENCY
text: Here we discuss in greater detail how the learning algorithm encourages consistency between the encoder and decoder of a MIM model, beyond the fact that they are fit to the same sample distribution. To this end we expand on several properties of the model and the optimization procedure.

heading: A.2.1 MIM CONSISTENCY REGULARIZER
text: In what follows we derive the form of the MIM consistency regularizer, RMIM(θ), given in Equation (9). Recall that we defineMθ = 12 (pθ(x, z) + pθ(x, z)). We can show that LMIM is equivalent to LCE plus a regularizer by taking their difference.
RMIM(θ) = LMIM(θ)− LCE(θ)
= 1
2 (H(MS , pθ(x, z)) +H(MS , pθ(x, z)))−H(MS ,Mθ)
= 1
2 (DKL (MS ‖ pθ(x, z)) +H(MS) +DKL (MS ‖ pθ(x, z)) +H(MS)) −DKL (MS ‖Mθ)−H(MS)
= 1
2 (DKL (MS ‖ pθ(x, z)) +DKL (MS ‖ pθ(x, z)))−DKL (MS ‖Mθ)
where RMIM(θ) is non-negative, and is zero only when the encoding and decoding distributions are consistent (i.e., they represent the same joint distribution). To prove that RMIM(θ) ≥ 0 we now construct Equation (9) in terms of expectation over a joint distribution, which yields
RMIM(θ) = 1
2 (H(MS , pθ(x, z)) +H(MS , pθ(x, z)))−H(MS ,Mθ) = Ex,z∼MS [ −1
2 log pθ(x, z)−
1 2 log pθ(x, z) + log 1 2 (qθ (x, z) + pθ (x, z)) ] = Ex,z∼MS [ − log √ qθ (x, z) · pθ (x, z) + log 1
2 (qθ (x, z) + pθ (x, z)) ] = Ex,z∼MS [ − log √ qθ (x, z) · pθ (x, z)
1 2 (qθ (x, z) + pθ (x, z))
] ≥ 0
where the inequality follows Jensen’s inequality, and equality holds only when qθ (x, z) = pθ (x, z) (i.e., encoding and decoding distributions are consistent).

heading: A.2.2 SELF-CORRECTING GRADIENT
text: One important property of the optimization follows directly from the difference between the gradient of the upper bound LMIM and the gradient of the cross-entropy loss LCE. By moving the gradient operator into the expectation using reparametrization, one can express the gradient of LMIM(θ) in terms of the gradient of logMθ and the regularization term in Equation (9). That is, with some manipulation one obtains
∂
∂θ
( log qθ + log pθ
2
) = ∂
∂θ log
( qθ + pθ
2
) + 1
2
( pθ qθ − 1 ) ∂ ∂θ qθ + ( qθ pθ − 1 ) ∂ ∂θpθ
qθ + pθ , (14)
which shows that for any data point where a gap qθ > pθ exists, the gradient applied to pθ grows with the gap, while placing correspondingly less weight on the gradient applied to qθ. The opposite is true when qθ < pθ. In both case this behaviour encourages consistency between the encoder and decoder. Empirically, we find that the encoder and decoder become reasonably consistent early in the optimization process.

heading: A.2.3 NUMERICAL STABILITY
text: Instead of optimizing an upper bound LMIM, one might consider a direct optimization of LCE. Earlier we discussed the importance of the consistency regularizer in LMIM. Here we motivate the use of LMIM from a numerical perspective point of view. In order to optimize LCE directly, one must convert log qθ and log pθ to qθ and pθ. Unfortunately, this is has the potential to produce numerical errors, especially with 32-bit floating-point precision on GPUs. While various tricks can reduce numerical instability, we find that using the upper bound eliminates the problem while providing the additional benefits outlined above.

heading: A.2.4 TRACTABILITY
text: As mentioned earlier, there may be several ways to combine the encoder and decoder into a single probabilistic model. One possibility we considered, as an alternative toMθ in Equation (7), is
Mθ = 1
β
√ qθ pθ , (15)
where β = ∫ √
qθ pθ dx dz is the partition function, similar to Bornschein et al. (2015). One could then define the objective to be the cross-entropy as above with a regularizer to encourage β to be close to 1, and hence to encourage consistency between the encoder and decoder. This, however, requires a good approximation to the partition function. Our choice ofMθ avoids the need for a good value approximation by using reparameterization, which results in unbiased low-variance gradient, independent of the accuracy of the approximation of the value.

heading: A.3 MIM LOSS DECOMPOSITION
text: Here we show how to break down the LMIM into the set of intuitive components given in Equation (11). To this end, first note the definition of LMIM:
LMIM(θ) = 1
2 (H(MS , pθ(x, z)) +H(MS , pθ(x, z))) (16)
We will focus on the first half of Equation (16) for now, 1
2 H(MS , pθ(x, z)) =
1
4
( H(pθ(x|z)P(z), pθ(x, z)) +H(qθ(z|x)P(x), pθ(x, z)) ) (17)
It will be more clear to write out the first term of Equation (17), 14H(pθ(x|z)P(z), pθ(x, z)) in full 1
4 H(pθ(x|z)P(z), pθ(x, z)) = −
1
4 ∫ x,z pθ(x|z)P(z) log(pθ(x, z))dxdz
= −1 4 ∫ x,z pθ(x|z)P(z) log(pθ(x|z))dxdz
− 1 4 ∫ x,z pθ(x|z)P(z) log(pθ(z))dxdz
We then add and subtract 14H(P(z)), where
1 4 H(P(z)) = − 1 4 ∫ x,z P(z) log(P(z))dxdz = − 1 4 ∫ x,z pθ(x|z)P(z) log(P(z))dxdz
Writing out 14H(pθ(x|z)P(z), pθ(x, z)) + 1 4H(P(z)) − 1 4H(P(z)) and combining terms, we obtain
1 4 H(pθ(x|z)P(z), pθ(x, z)) = H(pθ(x|z)P(z)) +DKL (P(z) ‖ pθ(z)) (18)
The second term in Equation (17) can then be rewritten as
1 4 H(qθ(z|x)P(x), pθ(x, z)) = 1 4 DKL (qθ(z|x)P(x) ‖ pθ(x, z)) + 1 4 H(qθ(z|x)P(x)) (19)
Combining Equations (18) and (19), we get the interpretable form for Equation 17, i.e.,
1 2 H(MS , pθ(x, z)) = 1 4 (H(pθ(x|z)P(z)) +H(qθ(z|x)P(x)))
+ 1
4 DKL (P(z) ‖ pθ(z)) +
1 4 DKL (qθ(z|x)P(x) ‖ pθ(x, z))
= 1
2 RH(θ) +
1 4 DKL (P(z) ‖ pθ(z)) + 1 4 DKL (qθ(z|x)P(x) ‖ pθ(x, z))
(20)
We can use the same basic steps to derive an analogous expression for H(MS , pθ(x, z)) in Equation (16) and combine it with Equation (20) to get the final interpretable form:
LMIM(θ) = RH(θ) + 1
4
( DKL (P(z) ‖ pθ(z)) +DKL (P(x) ‖ qθ(x)) ) + 1
4
( DKL (qθ(z|x)P(x) ‖ pθ(x, z)) +DKL (pθ(x|z)P(z) ‖ qθ(z,x)) )

heading: B MIM IN TERMS OF SYMMETRIC KL DIVERGENCE
text: As discussed above, the VAE objective can be expressed as minimizing the KL divergence between the joint anchored encoding and anchored decoding distributions. Below we consider a model formulation using the symmetric KL divergence (SKL),
SKL(θ) = 1
2 (DKL (pθ(x|z)P(z) ‖ qθ(z|x)P(x)) +DKL (qθ(z|x)P(x) ‖ pθ(x|z)P(z)) ) ,
the second term of which is the VAE objective. The mutual information regularizer RH(θ) given in Equation (5) can be added to SKL to obtain a cross-entropy objective that looks similar to MIM:
1 2 SKL(θ) + RH(θ) = 1 2 (H(MS , pθ(x|z)P(z)) +H(MS , qθ(z|x)P(x)))
When the model priors are equal to the anchors, this regularized SKL and MIM are equivalent. In general, however, the MIM loss is not a bound on the regularized SKL.
In what follows we explore the relation between SKL and JSD. In Section A.1 we showed that the Jensen-Shannon divergence can be written as
JSD(θ) = 1
2
( H(pθ(x|z)P(z),MS)−H(pθ(x|z)P(z))
+H(qθ(z|x)P(x),MS)−H(qθ(z|x)P(x)) )
= 1
2 (H(pθ(x|z)P(z),MS) +H(qθ(z|x)P(x),MS))− RH(θ)
Using Jensen’s inequality, we can bound JSD(θ) from above,
JSD(θ) ≤ 1 4
( H(pθ(x|z)P(z)) +H(pθ(x|z)P(z), qθ(z|x)P(x))
+H(qθ(z|x)P(x)) +H(qθ(z|x)P(x), pθ(x|z)P(z)) ) − RH(θ) (21)
= 1
4
( H(pθ(x|z)P(z), qθ(z|x)P(x)) +H(qθ(z|x)P(x), pθ(x|z)P(z))
+ 2RH(θ) ) − RH(θ)
= 1
4
( DKL (pθ(x|z)P(z) ‖ qθ(z|x)P(x)) +DKL (qθ(z|x)P(x) ‖ pθ(x|z)P(z))
+ 4RH(θ) ) − RH(θ)
= 1
4 (DKL (pθ(x|z)P(z) ‖ qθ(z|x)P(x)) +DKL (qθ(z|x)P(x) ‖ pθ(x|z)P(z)))
= 1
2 SKL(θ)
From Equation (21), if we add the regularizer RH(θ) and combine terms, we get
1 2 SKL(θ) + RH(θ) = 1 2 (H(MS , qθ(z|x)P(x)) +H(MS , pθ(x|z)P(z)))
Interestingly, we can write this in terms of KL divergence, 1
2 SKL(θ) + RH(θ) =
1 2 (DKL (MS ‖ qθ(z|x)P(x)) +DKL (MS ‖ pθ(x|z)P(z))) +H(MS)
= 1
2 (DKL (MS ‖ qθ(z|x)P(x)) +DKL (MS ‖ pθ(x|z)P(z)))
+ JSD(θ) + RH(θ)
which gives the exact relation between JSD and SKL. 1
2 SKL(θ) =
1 2 (DKL (MS ‖ qθ(z|x)P(x)) +DKL (MS ‖ pθ(x|z)P(z))) + JSD(θ)
= 1
2 (DKL (MS ‖ qθ(z|x)P(x)) +DKL (MS ‖ pθ(x|z)P(z)))
+ 1
2 (DKL (qθ(z|x)P(x) ‖MS) +DKL (pθ(x|z)P(z) ‖MS))

heading: C LEARNING
text: Here we provide a detailed description of MIM learning, with algorithmic pseudo-code. In addition we offer practical considerations regarding the choice of priors’ parameterization , and gradient estimation. The empirical upper bound objective is expressed in terms of two cross-entropy terms (cf. (8)). Given N fair samples, {xi, zi}Ni=1 drawn from the anchored (sample) distribution,MS(x, z) in (4), the empirical loss is
L̂MIM ( θ; {xi, zi}Ni=1 ) = − 1
2N N∑ i=1 log qθ(zi|xi) qθ(xi) + log pθ(xi|zi) pθ(zi) , (22)
where samples fromMS comprises equal numbers of points from pθ(x|z)P(z) and qθ(z|x)P(x). Samples from the anchors, P(x) and ,P(z), are treated as external observations; i.e., we assume we can sample from them but not necessarily evaluate the density of points under the anchor distributions.
Algorithm 1 specifies the corresponding training procedure. The algorithm makes no assumptions on the form of the parameterized distributions (e.g., discrete, or continuous). In practice, for gradientbased optimization, we would like an unbiased gradient estimator without the need to accurately
approximate the full expectations per se (i.e., in the cross entropy terms). This is particularly important when dealing with high dimensional data (e.g., images), where it is computationally expensive to estimate the value of the expectation. We next discuss practical considerations for the continuous case and the discrete case.
Algorithm 1 MIM learning of parameters θ Require: Samples from anchors P(x),P(z)
1: while not converged do 2: Ddec ← {xi, zi ∼ pθ(x|z)P(z)}N/2i=1 3: Denc ← {xj , zj ∼ qθ(z|x)P(x)}N/2j=1 4: D ← Ddec ⋃ Denc 5: # See definition of L̂MIM in Eq. (22) 6: LMIM (θ) ≈ L̂MIM (θ;D) 7: # Minimize loss 8: ∆θ ∝ −∇θL̂MIM (θ;D) 9: end while

heading: C.1 MIM PARAMETRIC PRIORS
text: There are several effective ways to parameterize the priors. For the 1D experiments below we model pθ(z) using linear mixtures of isotropic Gaussians. With complex, high dimensional data one might also consider more powerful models (e.g., autoregressive, or flow-based priors). Unfortunately, the use of complex models typically increases the required computational resources, and the training and inference time. As an alternative we use for image data the vampprior Tomczak and Welling (2017), which models the latent prior as a mixture of posteriors, i.e., pθ(z) = ∑K k=1 qθ(z|x = uk) with learnable pseudo-inputs {uk}Kk=1. This is effective and allows one to avoid the need for additional parameters (see Tomczak and Welling (2017) for details on vampprior’s effect over gradient estimation).
Another useful model with high dimensional data, following Bornschein et al. (2015), is to define qθ(x) as the marginal of the decoding distribution; i.e.,
qθ(x) = Ez∼pθ(z) [ pθ(x|z) ] . (23) Like the vampprior, this entails no new parameters. It also helps to encourage consistency between the encoding and decoding distributions. In addition it enables direct empirical comparisons of VAE learning to MIM learning, because we can then use identical parameterizations and architectures for both. During learning, when qθ(x) is defined as the marginal (23), we evaluate log qθ(x) with a single sample and reparameterization. When z is drawn directly from the latent prior:
log qθ(x) = logEz̃∼pθ(z) [ pθ(x|z̃) ] ≈ log pθ(x|z̃) . When z is drawn from the encoder, given a sample observation, we use importance sampling:
log qθ(x) = logEz̃∼qθ(z|x) [ pθ(x|z̃) pθ(z̃)
qθ(z̃|x)
] ≈ log pθ(x|z̃) + log pθ(z̃)− log qθ(z̃|x)
Algorithm 2 provides algorithm details with the marginal prior.

heading: C.2 GRADIENT ESTIMATION
text: Optimization is performed through minibatch stochastic gradient descent. To ensure unbiased gradient estimates of L̂MIM we use the reparameterization trick Kingma and Welling (2013); Rezende et al. (2014) when taking expectation with respect to continuous encoder and decoder distributions, qθ(z|x) and pθ(x|z). Reparameterization entails sampling an auxiliary variable ∼ p( ), with known p( ), followed by a deterministic mapping from sample variates to the target random variable, that is pθ(z) = gθ( ) and qθ(z|x) = hθ( ,x) for prior and conditional distributions. In doing so we assume p( ) is independent of the parameters θ. It then follows that
∇θEz∼qθ(z|x) [fθ(z)] = ∇θE ∼p( ) [fθ(hθ( ,x))] = E ∼p( ) [∇θfθ(hθ( ,x))]
Algorithm 2 MIM learning with marginal qθ(x)
Require: Samples from anchors P(x),P(z) Require: Define qθ(x) = Ez∼pθ(z) [pθ(x|z)]
1: while not converged do 2: # Sample encoding distribution 3: Denc ← {xi, zi ∼ qθ(z|x)P(x)}Ni=1 4: # Compute objective, approximate log qθ(x) with 1 sample and importance sampling 5: log qθ(xi) ≈ log pθ(xi|zi) + log pθ(zi)− log qθ(zi|xi) 6: L̂MIM (θ;Denc)← − 1N ∑N i=1 (log pθ(xi|zi) + log pθ(zi)) 7: # Sample decoding distribution 8: Ddec ← {xi, zi ∼ pθ(x|z)P(z)}Ni=1 9: # Compute objective, approximate log qθ(x) with 1 sample
10: log qθ(xi) ≈ log pθ(xi|zi) 11: L̂MIM (θ;Ddec)← − 12N ∑N i=1 (log qθ(zi|xi) + 2 log pθ(xi|zi) + log pθ(zi)) 12: # Minimize loss 13: ∆θ ∝ −∇θ 12 ( L̂MIM (θ;Ddec) + L̂MIM (θ;Denc)
) 14: end while
where fθ(z) is the loss function with parameters θ. It is common to let p( ) be standard normal, ∼ N (0, 1), and for z|x to be Gaussian with mean µθ(x) and standard deviation σθ(x), in which case z = σθ(x) + µθ(x), A more generic exact density model can be learned by mapping a known base distribution (e.g., Gaussian) to a target distribution with normalizing flows Dinh et al. (2014; 2016); Rezende and Mohamed (2015).
In the case of discrete distributions, e.g., with discrete data, reparameterization is not readily applicable. There exist continuous relaxations that permit reparameterization (e.g., Maddison et al. (2016); Tucker et al. (2017)), but current methods are rather involved in practice, and require adaptation of the objective function or the optimization process. Here we simply use the REINFORCE algorithm Sutton et al. (1999) for unbiased gradient estimates, as follows
∇θEz∼qθ(z|x) [fθ(z)] = Ez∼qθ(z|x) [∇θfθ(z) + fθ(z)∇θ log qθ(z|x)] .
A detailed derivation follows the use of the relation below,
∇θqθ(z|x) = qθ(z|x)∇θ log qθ(z|x)
in order to provide unbiased gradient estimates as follows ∇θEz∼qθ(z|x) [fθ(z)] = ∇θ ∫ fθ(z) qθ(z|x) dz
= ∫ qθ(z|x)∇θfθ(z) dz + ∫ fθ(z)∇θqθ(z|x) dz
= ∫ qθ(z|x)∇θfθ(z) dz + ∫ fθ(z) qθ(z|x)∇θ log qθ(z|x) dz
= Ez∼qθ(z|x) [∇θfθ(z) + fθ(z)∇θ log qθ(z|x)]
which facilitate the use of samples to approximate the integral.

heading: C.3 TRAINING TIME
text: Training times of MIM models are comparable to training times for VAEs with comparable architectures. One important difference concerns the time required for sampling from the decoder during training. This is particularly significant for models like auto-regressive decoders (e.g., Kingma et al. (2016)) for which sampling is very slow. In such cases, we find that we can also learn effectively with a sampling distribution that only includes samples from the encoding distribution, i.e., P(x) qθ(z|x), rather than the mixture. We refer to this particular MIM variant as asymmetric MIM (or A-MIM). We use it in Sec. 5.2 when working with the PixelHVAE architecture Kingma et al. (2016).

heading: D POSTERIOR COLLAPSE IN VAE
text: Here we discuss a possible root cause for the observed phenomena of posterior collapse, and show that VAE learning can be viewed as an asymmetric MIM learning with a regularizer that encourages the appearance of the collapse. We further support that idea in the experiments in Section 5.1. As discussed earlier, VAE learning entails maximization of a variational lower bound (ELBO) on the log-marginal likelihood, or equivalently, given Equation (1), the VAE loss in terms of expectation over a joint distribution:
−Ex∼P(x),z∼qθ(z|x) [ log pθ(x|z) + logP(z)− log qθ(z|x) ] . (24)
To connect the loss in Equation (24) to MIM, we first add the expectation of logP(x), and scale the loss by a factor of 12 , to obtain
Ex∼P(x),z∼qθ(z|x) [ −1
2 (log(pθ(x|z)P(z)) + log(qθ(z|x)P(x))) + logP(x) + log qθ(z|x)
] (25)
where P(x) is the data distribution, which is assumed to be independent of model parameters θ and to exist almost everywhere (i.e., complementing P(z)). Importantly, because P(x) does not depend on θ, the gradients of Eqs. (24) and (25) are identical up to a multiple of 12 , so they share the same stationary points.
Combining IID samples from the data distribution, xi ∼ P(x), with samples from the corresponding variational posterior, zi ∼ qθ(z|xi), we obtain a joint sampling distribution; i.e.,
MVAES (x, z) = P(x) qθ(z|x)
whereMVAES comprises the encoding distribution inMS . With it one can then rewrite the objective in Equation (25) in terms of the cross-entropy betweenMVAES and the parametric encoding and decoding distributions; i.e.,
1
2
( H(MVAES , pθ(x|z)P(z)) +H(MVAES , qθ(z|x)P(x)) ) +
−HMVAES (x)−HMVAES (z) + IMVAES (x; z) . (26)
The sum of the last three terms in Equation (26) is the negative joint entropy −H(z,x) under the sample distributionMVAES . Equations (1) and (26), the VAE objective and VAE as regularized cross entropy objective respectively, define equivalent optimization problems, under the assumption that P(x) and samples x ∼ P(x) do not depend on the parameters θ, and that the optimization is gradient-based. Formally, the VAE objectives (1) and (26) are equivalent up to a scalar multiple of 12 and an additive constant, namely, HMVAES (x).
Equation (26) is the average of two cross-entropy objectives ( i.e., between sample distributionMVAES and the model decoding and encoding distributions, respectively), along with a joint entropy term (i.e., last three terms), which can be viewed as a regularizer that encourages a reduction in mutual information and increased entropy in z and x. We note that Equation (26) is similar to the MIM objective in Equation (8), but with a different sample distribution, where the priors are defined to be the anchors, and with an additional regularizer. In other words, Equation (26) suggests that VAE learning implicitly lowers mutual information. This runs contrary to the goal of learning useful latent representations, and we posit that it is an underlying root cause for posterior collapse, wherein the trained model show low mutual information which can be manifested as an encoder which matches the prior, and thus provides weak information about the latent state (e.g., see (Chen et al., 2016b) and others). We point the reader to Section E.1 for empirical evidence for the use of a joint entropy as a mutual information regularizer.

heading: E ADDITIONAL EXPERIMENTS
text: Here we provide additional experiments that further explore the characteristics of MIM learning.

heading: VAE+H MIM-H VAE+H MIM-H VAE+H MIM-H

heading: E.1 ENTROPY AS MUTUAL INFORMATION REGULARIZER
text: Here we examine the use of entropy as a mutual information regularizer. We repeat the experiment in Section 5.1 with added entropy regularizer. Figure 7 depicts the effects of an added Hqθ (x, z) to the VAE loss, and a subtracted HMS (x, z) from MIM . The corresponding quantitative values are presented in Figure 8. Adding the entropy regularizer leads to increased the mutual information, and subtracting it results in a strong posterior collapse, which in turn is reflected in the reconstruction quality. While such an experiment does not represent a valid probabilistic model, it supports our use of entropy as a regularizer (cf. Eq. (5)) for JSD in order to define a consistent model with high mutual information.
E.2 CONSISTENCY REGULARIZER IN LMIM
Here we explore properties of models for 1D x and z, learned with LMIM and LCE, the difference being the model consistency regularizer RMIM(θ). All model priors and conditional likelihoods (qθ(x), qθ(z|x), pθ(z), pθ(x|z)) are parameterized as 10-component Gaussian mixture models, and
optimized during training. Means and variances for the conditional distributions were regressed with 2 fully connected layers (h ∈ R10) and a swish activation function Ramachandran et al. (2018). Top and bottom rows in Fig. 9 depict distributions in observations and latent spaces respectively. Dashed black curves are anchors, P(x) on top, and P(z) below (GMMs with up to 3 components). Learned model priors, qθ(x) and pθ(z), are depicted as red (top) and blue (bottom) curves.
Green histograms in Fig. 9(a,b) depict reconstruction distributions, computed by passing fair samples from P(x) through the encoder to z and then back through the decoder to x. Similarly the yellow histograms shows samples from P(z) passed through the decoder and then back to the latent space. For both losses these reconstruction histograms match the anchor priors well. In contrast, only the priors that were learned with LCE loss approximates the anchor well, while the LMIM priors do not. To better understand that, we consider two generative procedures: sampling from the anchors, and sampling from the priors.
Anchor consistency is depicted in Fig. 9(c,d), where Green histograms are marginal distributions over x from the anchored decoder (i.e., samples from P(z)pθ(x|z) ). Yellow are marginals over z from the anchored encoders P(x)qθ(z|x). One can see that both losses results in similar quality of matching the corresponding opposite anchors.
Priors consistency is depicted in Fig. 9(e,f), where Green histograms are marginal distributions over x from the model decoder pθ(z)pθ(x|z). Yellow depicts marginals over z from the model encoder qθ(x)qθ(z|x). Importantly, with LMIM the encoder and decoder are consistent; i.e., qθ(x) (red curve) matches the decoder marginal, while pθ(z) (blue) matches the encoder marginal. The model trained with LCE (i.e., without consistency prior) fails to learn a consistent encoder-decoder pair. We note that in practice, with expressive enough priors, LMIM will be a tight bound for LCE.

heading: E.3 PARAMETERIZING THE PRIORS
text: Here we explore the effect of parameterizing the latent and observed priors. A fundamental idea in MIM is the concept of a single model,Mθ. As such, parameterizing a prior increases the global expressiveness of the modelMθ . Fig. 10 depicts the utilization of the added expressiveness in order to increase the consistency between the encoding and decoding model distribution, in addition to the consistency ofMθ withMS .

heading: E.4 EFFECT OF CONSISTENCY REGULARIZER ON OPTIMIZATION
text: Here we explore whether a learned model with consistent encoding-decoding distributions (i.e., trained with LMIM) also constitutes an optimal solution of a CE objective (i.e., trained with LCE). Results are depicted in Fig. 11. In order to distinguish between the effects of the optimization from the consistency regularizer we initialize a MIM model by pre-training it with LCE loss followed by LMIM training in Fig. 11(i), and vice verse in Fig. 11(ii). (a-b,e-f) All trained models in Fig. 11 exhibit similarly good reconstruction (green matches dashed black). (c-d,g-h) However, only models that were trained with LMIM exhibit encoding-decoding consistency (green matches red, yellow matches blue). While it is clear that the optimization plays an important role (i.e., different initialization leads to different local optimum), it is also clear that encoding-decoding consistency is not necessarily an optimum of LCE, as depicted in a non-consistent model (h) which was initialized with a consistent model (g). Not surprisingly, without the consistency regularizer training with LCE results in better fit of priors to anchors (f) as it is utilizing the expressiveness of the parametric priors in matching the sample distribution.

heading: F ADDITIONAL RESULTS
text: Here we provide additional visualization of various MIM and VAE models which were not included in the main body of the paper.

heading: F.1 RECONSTRUCTION AND SAMPLES FOR MIM AND A-MIM
text: In what follows we show samples and reconstruction for MIM (i.e., with convHVAE architecture), and A-MIM (i.e., with PixelHVAE architecture). We demonstrate, again, that a powerful enough encoder allows for generation of samples which are comparable to VAE samples.

heading: F.2 LATENT EMBEDDINGS FOR MIM AND A-MIM
text: In what follows we show additional t-SNE visualization of unsupervised clustering in the latent representation for MIM (i.e., with convHVAE architecture), and A-MIM (i.e., with PixelHVAE architecture).

Review summaries:
This paper defines a new learning objective of an autoencoder framework for learning joint distributions over observations and latent states, where the objective is the joint entropy of  M_s, an equally weighted mixture of the encoding and decoding distributions. Since it involves the marginal distribution which is intractable, it goes further to introduce an upper bound which is the cross-entropy between M_s and M_theta. To maintain consistency, it introduce an upper bound again, which is the average of cross entropy between the mixture distribution M_S and the model encoding and decoding distribution. Through comprehensive experiments, the effectiveness is validated.