MIM: Mutual Information Machine
Authors: Micha Livne
Kevin Swersky
David J. Fleet
Venue:  (2020)
Keywords: Mutual Information
Representation Learning
Generative Models
Probability Density Estimator

Abstract:     We introduce the Mutual Information Machine (MIM), an autoencoder framework
    for learning joint distributions over observations and latent states. 
    The model formulation reflects two key design principles: 1) symmetry, to encourage 
    the encoder     and decoder to learn different factorizations of the same 
    underlying distribution; and 2) mutual information, to encourage the learning 
    of useful representations for downstream tasks. 
    The objective comprises the Jensen-Shannon divergence between the encoding and 
    decoding joint distributions, plus a mutual information regularizer. 
    We show that this can be bounded by a tractable cross-entropy loss between 
    the true model and a parameterized approximation, and relate this to 
    maximum likelihood estimation and variational autoencoders.
    Experiments show that MIM is capable of learning a latent representation with high mutual information,
    and good unsupervised clustering, while providing NLL comparable to VAE 
    (with a sufficiently expressive architecture).

Under review as a conference paper at ICLR 2020
MIM: MUTUAL INFORMATION MACHINE
Anonymous authors
Paper under double-blind review
ABSTRACT
We introduce the Mutual Information Machine (MIM), autoencoder framework for learning joint distributions over observations and latent states. The model for- mulation reflects two key design principles: 1) symmetry, encourage the encoder and decoder to learn different factorizations of the same underlying distribution; encourage the learning of useful representations
and 2) mutual information, for downstream tasks. The objective comprises the Jensen-Shannon divergence between the encoding and decoding joint distributions, plus a mutual information regularizer. We show that this can be bounded by a tractable cross-entropy loss between the true model and a parameterized approximation, and relate this to max- imum likelihood estimation and variational autoencoders. Experiments show that MIM is capable of learning a latent representation with high mutual information, and good unsupervised clustering, while providing NLL comparable to VAE (with a sufficiently expressive architecture).
INTRODUCTION
Mutual information is natural indicator of the quality of learned representation (Hjelm al., 2019), along with other characteristics, such as the compositionality of latent factors that are expected be useful in downstream tasks, like transfer learning (Bengio et al., 2017). Mutual information is, however, computationally difficult to estimate for continuous high- nigh-dimmessinal random variables. As such, it can be hard to optimize when learning latent variable models (Belghazi al., 2018; Chen et al., 6a).
This paper formulates a new class of probabilistic autoencoder model that is motivated by two key design principles, namely, the maximization of mutual information, and the symmetry of the encoder-decoder components. Symmetry captures our desire for both the encoder and decoder to effectively and consistently model the underlying observation and latent domains. This is particularly useful for downstream tasks in which either one or both of the encoder and decoder play a central role. These properties are formulated in terms of the symmetric Jensen-Shanme Divergence between the encoder and decoder, combined with an objective term to maximize mutual information. We refer to the resulting model as the mutual information machine, or MIM.
We contrast MIM with models trained using (approximate) maximum likelihood, the canonical example being the variational auto-encoder, or VAE (Kingma and Welling, 2013; Rezende al., 2014). The VAE comprises probabilistic decoder and an approximate encoder, learned via optimization of an evidence-based lower bound (ELBO) on the log marginal data distribution. In contrast MIM it is asymmetric in its formulation, and while often producing excellent representations, VAES sometimes produce pathological results in which the encoder, or approximate posterior, conveys relatively little information between observations and latent states. This behavior, often referred to as posterior ressupse.results in low mutual information between observations and inferred latent states (Bowman et al., 2015; Chen et al., 201 6b; Razavi et al., 2019; van den Oord et al., 2016; 2017).
We formulate the MIM model, and learning algorithm that minimizes an upper bound on the desired loss. The resulting objective can be viewed as a symmetrized form of KI. divergence, thereby closely related to the asymmetric KL objective of the VAE. This also enables direct comparisons to the VAE in terms of posterior collapse, mutual information, data log likelihood, and clustering. Experiments show that MIM offers favourable mutual information, better clustering in the latent representation, and similar reconstruction, at the expense of sampling quality and data log likelihood when compared to VAE with the same architecture. We also demonstrate that for a sufficiently powerful architecture, MIM can match sampling quality and log likelihood of VAE with the same architecture.

---

Under review as a conference paper at ICLR 2020
2 VARIATIONAL AUTOENCODERS
VAE learning entails optimization of a variational lower bound on the log-marginal likelihood of the data, log ((:)), to estimate the parameters 0 of approximate posterior qo (o (z)) over latent states 2 ei.e. the encoder) and a corresponding decoder, po (2|22 (Kingma and Welling, 2013; Rezende al., 2014). A prior over the latent space, P(z), often assumed to be an isotropic Gaussian, serves as prior for (o(2)x)) in the evidence-lower-bound (ELBO) on the marginal likelihood: log P(x) z Ez~qp(=(=)) [log po [a(]2)] - DxL (go [z|=) P())). Here, we use the notation P(x) and P(=) emphasize that these priors are given, and that can draw random samples from them, but not necessarily evaluate the log-likelihood of samples under them. In what follows we often refer to them as anchors to further emphasize their role.
With amortized posterior inference, we take expectation over the observation distribution, P(æ), to obtain the VAE objective:
RVAE (0) = Ex~P(x) [xx [log Po [z) [x] DxL (20(2 P()))
Em ( x) + log P(=) - log go (2|<))
(1)
Gradients of Eqn. (1) estimated through MC sampling from go (z) (z)) with reparameterization,
yielding unbiased low-variance gradient estimates (Kingma and Welling, 2013; Rezende et al., 2014).
VAES are normally thought of as maximizing a lower bound on the data log-likelihood, however can also be expressed as minimizing the divergence between two joint distributions over æ and z. To see this, we first subtract log (ax) from (1), which does not change the gradients of the objective with respect to 0. We then negate the result, as we will be performing minimization. This yields
LVAB (0) = RVAE (0) + E())(()) [log P(x)]
PxL (go a Px()) || pe (z)) (2)
The VAE optimization is therefore equivalent to minimizing the KL divergence between an encoding distribution qo (mx) and decoding distribution po (ar (())
3 SYMMETRY AND MUTUAL INFORMATION
Our goal is to find a consistent encoder-decoder pair, representing a joint distribution over the observation and latent domains, with high mutual information between observations and latent states. By consistent, we mean that the encoding and decoding distributions go (æ)) P((r) and (0(2)2) P(z), define the same joint distribution. Effectively, we aim to estimate an undirected graphical model with two valid factorizations. We note that consistency is achievable in the VAE when the approximate posterior go(z is capable of representing the posterior under the decoding distribution po (z(æ). In the general case, however, consistency is not usually achieved.
In contrast to the asymmetric divergence between encoding and decoding distributions in the VAE objective (2), here we consider a symmetric measure, namely, the well-known Censen-Shamme divergence (JSD),
JSD(O) 1 [ DKi. (po())z) P(=) Mss + DkI. (1a(>|æ) P(z) P[ MI)
(3)
where Ms an equally weighted mixture of the encoding and decoding distributions; i.e.,
1 Ms 2 (o(x))))((() + (=)==)
In addition to encoder-decoder consistency, to learn useful latent representations we also want high mutual information between æ and z. Indeed, the link between mutual information and representation learning has been explored in recent work (Hjelm et al., 2019; Belghazi et al., 2018; Chen et al., 2016a). Here, to emphasize high mutual information, we add a particular regularizer of the form
Ru (0) = [ (Ho P(=)) + H (=)())))) (=()))))))
(5)
2
This the average of the joint entropy over æ and z according the encoding and decoding distributions. This is related to mutual information by the identity H (x, (()) + (z) [(x; 2). That is, minimizing joint entropy encourages the minimization of the marginal entropy and maximization of the mutual information. In addition to encouraging high mutual information, one can show that this particular regularizer has a deep connection to JSD and the entropy of Mss ie. (0) + (0) (Ms). The derivation for Eqn. (6) is given in Appendix A.1.
(6)
2

---

Under review as a conference paper at ICLR 2020
4 MUTUAL INFORMATION MACHINE
Section 3 formulates loss function (6) that reflects our desire for model symmetry and high mutual information. This objective difficult to optimize directly since we do not know how evaluate log ((x) in the general case (i.e., we do not have an exact closed-form expression for P(x)). As consequence, we introduce parameterized approximate priors, qo (z(x) and pa(z). to derive tractable bounds on the penalized Censen-Shannon divergence. This is similar in spirit to VAEs, which introduce a parameterized approximate posterior. These parameterized priors, together with the conditional encoder and decoder, qo [z|z) 2)) and (a [z)x,, comprise new pair of joint distributions, (a x) = ge (z ()) go (æ). and pe(x, z) = pe (ar 2) pe (z).
These joint distributions allow to formulate new, tractable loss that bounds H iMs)
(o) = Mo) = DxL (Ms Mo) + H(Ms) 2 H(Ms),
(7)
where MM = 1 (po (ar, 2) + to (ar, ))). and H (MMs.Ma) denotes the cross-entropy between Ms and MM. We refer to LCE the cross-entropy loss. It aims to match the model prior distributions to the anchors, while also minimizing H (Ms). A key advantage
this formulation is that the cross-entropy loss can be trained by Monte Carlo sampling from the anchor distributions with reparameterizatioo (Kingma and Welling, 2013; Rezende et al., 2014).
At this stage it might seem odd to introduce parametric prior for P(z). Indeed, setting it directly is certainly an option. Nevertheless, in order to achieve consistency between po (ar, z) and qo (æ, z) it be advantageous allow (= vary. Essentially, trade-off latent prior fidelity for increased model consistency. We provide more insights about this in Appendix F.3.
One issue with LCE is that, while it will try to enforce consistency between the model and the anchored distributions, i.e., pe z) ss po (x]=))(()) and ga(ær, z) z qo ((x)))))(() it will not directly try achieve model consistency: (3r, z) go (ax, z). To remedy this, bound Lee using Jensen's 's inequality, i.e.,
CMIM ( =
2 (H (x, H (Ms, pe (z,)))) LCE(O).
Equation (8) gives us the loss function for the Mutual Information Machine (MIM). It is an average of cross entropy terms between the mixture distribution Ms and the model encoding and decoding distributions respectively. To see that this encourages model consistency, it can be shown that LMM equivalent LCe plus a non-negative model consistency regularizer, i.e.,
(0) (Cx(0) + RMIM (0).
The non-negativity of RMIM is simple consequence of LMMM (0) 2 fCE(O) in (8). One can further show (see Appendix A.2) that (M)M(() satisfies
RMIM (0) (KL (Ms po (x. x)) + DKL (Ms go ())) DEL (Ms MM)
(10)
One can conclude from (10) that the regularizer RMIM is zero only when the two joint model distri-
butions, (o (æ, 2) and pa (æt 2), are identical under fair samples from the joint sample distribution
Ms 2). In practice we find that encouraging model consistency also helps stabilize learning.
To understand the MIM objective in greater depth, we find helpful to express LMIM as a sum of fundamental terms that provide some intuition for its expected behavior. In particular, derived in the Appendix A.3,
LMIM (0) = (O) + 1 (Dee (P(z) (P(z) (=) + DeL (P(x) (=())))
+ } PKL (g0 x) P(x) (x =)) + DRL. (=l()))) (z)x))))) (1I)
The first term in Eqn. (11) encourages high mutual information between observations and latent states. The second shows that MIM directly encourages the model priors to match the anchor distributions. Indeed, the KL term between the data anchor and the model prior the maximum likelihood objective. The third term encourages consistency between the model distributions and the anchored distributions, in effect fitting the model decoder to samples drawn from the anchored encoder (cf. VAE), and, via symmetry, fitting the model encoder samples drawn from the anchored decoder (both with
3

---

Under review as a conference paper at ICLR 2020
reparameterization). As such, MIM can be seen simultancously training and distilling model distribution over the data into a latent variable model. The idea of distilling density models has been used in other domains, c.g., for parallelizing auto-regressive models (Oord et al., 2017).
In summary, the MIM loss provides an upper bound on the joint entropy of the observation and latent states under the mixture distribution Mys LMMM (0) > HMs (a:) + HMg (2) IMs (z: z). Through the MIM loss and the introduction of the parameterized model distribution Ma. we are pushing down on the entropy of the anchored mixture distribution Ms. which is the sum of marginal entropies minus the mutual information. Minimizing the MIM bound yields consistency of the model encoder and decoder, and high mutual information under Ms between observations and latent states.
EXPERIMENTS
In what follows we examine MIM empirically, with the VAE as a baseline. We consider synthetic datasets and well-known image datasets, namely MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 201 7a) and Omniglot (Lake et al., 2015). All models were trained using Adam optimizer Kingma and Ba (2014) with learning rate of 10-3, and mini-batch size of 128. Following Alemi et al. (2017), we anneal the loss to stabilize the optimization. To this end we linearly increase /3 from 0 to I in the following expression for number of "warm-up" epochs:
N
ÊMIM (0; (a:, 2:) = 2N
log (po( (o)) + (log to (m;) po(z:)) (12)
Training continues until the loss (i.e,, with B = 1) on held-out validation set has not improved for
the same number of epochs as the warm-up steps (i.e., defined per experiment). We have found the number of epochs to convergence of MIM learning to be between 2 to 5 times greater than a VAE with the same architecture. A complete description of MIM learning is provided in Appendix C.
(Code is available from https:/ //www..////ww/////////eee wwwpboxo.xx..ooomoo ccm/ s/iddlsslayassssee.e..eaaaaaaaaa 77s wis-mmsterr.......m.......ee...m sip?di=1 )
5.1 POSTERIOR COLLAPSE IN Low DIMENSIONAL DATA
Before turning to empirical results, is useful to briefly revisit similarities and differences between MIM and the canonical VAE formulation. To that end, one can show from Eqn. (1) that the VAE loss can be expressed in a form that bears similarity to the MIM loss Eqn. (8). In particular, following the derivation in Appendix D,
1 { [(MMAA) 90(zl=) + H(M
LVAE =
Po(2))))(2))) H(MM))
Hayaa (13)
2
where MYAE(Z (z, 2) ((xx)))P(x), and the entropy in the last term is taken under the sample distribution MYAE Like the MIM loss, the first term in Eqn. (13) is the average of two cross-entropy terms, between sample distribution and the encoding and decoding distributions. Unlike the MIM loss, these terms asymmetric samples drawn only from the encoding distribution. Further, the last term in the VAE loss (13) is equivalent to -H MyyE (x) H HyMAE (z) + MMAA (æ; z). and thus it will also encourage a reduction in the mutual information. We posit that this plays a significant role in the phenomena often referred as posterior collapse, in which the variance of the variational posterior grows large and the latent embedding conveys relatively little information about the observations (e.g., see (Chen et al., 2016b) and others).
To empirically support the expression in Eqn. (13), we consider experiments with synthetic data comprising 2D observations a R23 with a 2D latent space, R.3 In 2D one can easily visualize the model and measure quantitative properties of interest (e.g.. mutual information). Fig. depicts data drawn from a Gaussian mixture model comprising five isotropic components with standard deviation 0.25 (top), and latents from an isotropic standard Normal (bottom). The encoder and decoder conditional distributions are Gaussian whose means and variances are regressed from the input using two fully connected layers and tanh activation. The parameterized data prior, co(x), is defined to be the marginal of the decoding distribution (see Eqn. (23) in Appendix C.I), and the model prior pe (2) is defined to be the anchor, so the only model parameters are those of the encoder and decoder (see Appendix C for details). We can therefore learn models with the MIM and VAE objectives, but with the same architecture and parameterization. In this section we used warm-up
4

---

Under review as a conference paper at ICLR 2020
VAE
MIM
VAE
MIM
VAE
MIM
MI 0.019
MI 2.1
MI 0.71
M4 5.6
MI 1.5
MI 5.9
RMSE 1.2
RMSE 0.73
RMSE 0.81
RMSE 0.007
RMSE 0.5
RMSE 0.013
(a) h E R
(b) E 2220
(c) h E R.500
Figure I: VAE and MIM models with 2D inputs, a 2D latent space, and 5, 20 and 500 hidden units. Top row: Black contours depict level sets of P()); red dots reconstructed test points. Bottom row: Green contours are one standard deviation ellipses of qo [=|2))) for test points. Dashed black circles depict standard deviation of (=z). The VAE predictive variance remains high, regardless of model expressiveness, an indication of various degrees of posterior collapse. while MIM produces lower predictive variance and lower reconstruction errors, consistent with high mutual information.
3.0
10
0.8
2.5
0.0
0.2
20
100
300
500
20
100
300
500
20 100
300
500
20
100
300 500
(a) MI
(b) NLL
(c) Recon. Error
(d) Classif. (5-NN)
Figure 2: Test performance MIM (blue) and VAE (red) for the 2D GMM data (ef. Fig. 1), all as
functions of the number of hidden units (on x-axis). Each plot shows the mean and standard deviation of 10 experiments.
scheduler (Vaswani al., 2017) with a warm-up of 3 steps, and with each epoch comprising 10000 samples. Training and test sets are drawn independently from the GMM.
Figure depicts learned VAE and MIM models, with 5, 20 and 500 hidden units (from left right) to control model expressiveness. Mutual information and root-mean-squarede test reconstruction error are inset in the top row. For the weakest architecture (a), with 5 hidden units, VAE and MIM posterior variances are similar to the prior (MIM in one dimension), a sign of posterior collapse. As the number of hidden units increases (b,c), the VAE posterior variance remains large, preferring lower mutual information while matching the aggregated posterior to the prior. The MIM encoder produces tight posteriors, and yields higher mutual information and lower reconstruction errors.
To quantify this behavior Fig. 2 (NLL) of test points under the model 9o- the mean reconstruction error test points, and 5-NN classification performance! (predicting which of 5 GMM components each test point from), all as functions of the number of hidden units. Following Belghazi et al. (2018), we estimate mutual information using the KSG estimator (Kraskov al., 2004; Gao al., 2016), based on 5-NN neighborhoods. The auxiliary classification task provides a proxy for representation quality. Mutual information, reconstruction RMSE, and classification accuracy for test data under the MIM model are better than with VAE models, at the expense of poorer data log likelihoods for MIM. In this case MIM learning finds near-invertible mappings with sufficiently powerful architecture.
shows mutual information, the average negative log- liglliil
drawn
To further investigate MIM learning in low dimensional data, we project 784D images from Fashion- MNIST (Xiao et al., 2017b) onto a 20D linear subspace using PCA (capturing 78.5% of total variance). The training and validation sets had 50,000 and 10,000 images respectively. Fig. 3 summarizes the results, with MIM producing high mutual information and classification accuracy, at all but very low
We experimented with 5-N,NN 10-NN and found the results be consistent.
5

---

Under review as a conference paper at ICLR 2020
15
10
120
50
0.2
12
16
20
12
16
20
12
16
20
12
16
20
(a) MI
(b) NLL
(c) Recon. Error
(d) Classif. (5-NN)
Figure 3: MIM (blue) and VAE (red) for 20D Pashion-MNIST, with latent dimension between 2 and
20. Plots depict mean and standard deviation of 10 experiments.
Table 1: High dimensional image data. Quantitative results based 10 trials per condition.
ConvHVAE (S)
convHVAE (VP)
Dataset
MIM
VAE
MIM
VAE
Fashion MNIST
272.14 0.64
225.40 + 0.05
227.61 + 0.34
224.77 77 + 0.04
MNIST
126.85 0.56
80.50 0.05
82.73 0.08
79.66 0.06
Omniglot
141.81 0.32
97.94 + 0.29
104.10 + 2.17
97.55 + 0.16
PixcIHVAE (S)
PixelHVAE (VP)
A-MIM
VAE
A-MIM
VAE
Fashion MNIST
243.95 0.47
224.65 0.07
224.94 + 0.34
224.02 0.08
MNIST
114.96 + 0.35
79.04 + 0.05
79.04 + 0.08
78.60 + 0.04
Omniglot
126.12 + 0.38
91.06 + 0.14
91 82 + 0.20
90.74 + 0.15
(a) Test NLL (in nats). With a more powerful prior, MIM and VAE yield comparable results.
convHVAE (S)
convHVAE (VP)
PixelHVAE (S)
PixelHVAE (VP)
Dataset
MIM
VAF
MTM
VAE
A-MIM
VAF
A-MIM
VAF
Fashion MNIST
0.83
0.76
0.81
0.78
0.71
0.76
0.79
0.77
MNIST
0.97
0.92
0.97
0.92
0.95
0.86
0.96
0.81
(b) Test accuracy of 5-NN classifier. Standard deviations are well less than 0.1, omitted from the table.
latent dimensions. MIM and VAE yield similar test reconstruction errors, with VAE having better negative log likelihoods for test data.
We conclude that the VAE is prone posterior collapse for wide range of models' expressiveness and latent dimensionality, with latent embeddings exhibiting low mutual information. In contrast, MIM was empirically robust to posterior collapse. this regard, we note that several papers have described ways to mitigate posterior collapse in VAE learning, e.g., by lower bounding, or annealing the KL divergence term in the VAE objective (Alemi al., 2017; Razavi al., 2019), or by limiting the expressiveness of the decoder (e.g., Chen et al. (2016b)). We posit that MIM does not suffer from this problem as a consequence of the objective design principles that encourage high mutual information between observations and the latent representation.
5.2 MIM LEARNING HIGH DIMENSIONAL IMAGE DATA
We next consider MIM and VAE learning with image data (Pashion-MNISTT, MNIST, Omniglot). Unfortunately, with high dimensional data we cannot reliably compute mutual information (Belghazi al., 2018). Instead, for model assessment we focus on log-likelihood, reconstruction, and the quality of random samples. In doing so we also explore multiple architectures, including the top performing models from Tomezak and Welling (2017), namely, convHVAE (L and PixelHVAE (L 2), with Standard (S) priors?, and VampPrior (VP) p The VP pseudo-inputs are initialized with training data samples. All the experiments below the same experimental setup in Tomezak and Welling (2017) 7), and the same latent dimensionality ε RR Here we also demonstrate that powerful prior (e.g., PixelHVAE (VP)) allows MIM learn models with competitive NLL performance.
poo(2) = N2 = 0, I), a standard Normal distribution, where is the identity matrix. (z) 1 K go ux)) mixture model of the encoder conditioned optimized pseudo-inputs u.-

---

Under review as a conference paper at ICLR 2020
f
4
4
9
0
C.bb
77
/
4
9
<
9
0
6
R
3
31
P
/
4
8
6
9
0
6
20
1.
S
0
9
2
1
3
5
X1
D
2
4
3
6
1
7
rr
F
U
2
8
3
9
2
VAE
A-MIM
VAE
A-MIM
VAE
A-MIM
Figure 4: MIM and VAE learning with the PixelHVAE (VP) architecture, applied to Fashion-MNIST MNIST, and Omniglot (left to right). The top three rows (from top bottom) test data samples, VAE reconstruction, and A-MIM reconstruction. Bottom: random samples from VAE and A-MIM.
Sampling from an auto-regressive decoder (e.g., PixelHVAE) very slow. During training, we find that we can also learn effectively with sampling distribution comprising just the encoding distribution, i.e., P(x) qo (2|æ), rather than the mixture the encoder and decoder. We refer to this particular MIM variant as asymmetric MIM (or A-MIM). (For details, see Sec. C.3).
We report quantitative results in Table la for test negative log-likelihood (NIL)). One can see that VAE models result in better NLL, but with rather small gap for the more expressive models (i.e., PixeIHVAE (VP)). We also show qualitative results for the most expressive models (i.e., PixelHVAE). Fig. 4 depicts reconstruction' and sampling for Fashion- MNIST, and Omniglot, for the top performing model (PixelHVAE (VP)), with MIM and VAE being comparable. See Appendix F for additional results.
The poor NLL and hence poor sampling for MIM with a weak prior model can be explained by the tightly clustered latent representation (e-g., Fig. 1). A more expressive, leamable prior can capture such clusters more accurately, and as such, also produces good samples (e.g.. VampPrior). In other words, while VAE opts for better NLL and sampling at the expense of lower mutual information, MIM provides higher mutual information at the expense of the NLL for a weak prior, and comparable NLL and sampling with more expressive priors.
5.3 CLUSTERING AND CLASSIFICATION
Finally, following Hjelm et al. (2019), we consider auxiliary classification task further measure of the quality of the learned representations. We opted for K-NN classification, being non-parame non-maraetriic method which relies on semantic clustering in latent space, without any additional training. Given representations learned above in Sec. 5.2, simple 5-NN classifier was applied to test data to predict one of 10 classes for MNIST and Fashion-MNIS"
Table l shows that in all but one case, MIM yields more accurate classification results. We attribute the performance difference to higher mutual information of MIM representations. Figure 5 provides a qualitative visualization of the latent clustering, for which t-SNE (van der Maaten and Hinton, 2008)) was used to project the latent space down to 2D. One can see that MIM learning tends to cluster classes in the latent representation more tightly, while VAE clusters are more diffuse and overlapping, consistent with the results in Table lb.
6 RELATED WORK
Given the vast literature on generative models, here we only touch on the major bodies of work related to MIM, beyond the strong connection to VAFs.
As mentioned above, mutual information, together with disentanglement, is considered to be cornerstone for useful representations (Belghazi et al., 2018; Hjelm al., 2019). Normalizing flows
4 Test data the top row Fig. are binary, while reconstructions depict the prohability of each pixel being
1, following Tomezak and Welling (2017)
We omitted results for & (1,3, 10) as find them similar.

---

Under review as a conference paper at ICLR 2020
VAE
MIM
VAE
A-MIM
ConvIVAE (VP)
PixelIIVAE (VP)
Figure 5: MIM and VAE embedding for Fashion MNIST (top) and MNIST (botom).
(Rezende and Mohamed, 2015; Dinh et al., 2014; 2016; Kingma and Dhariwal, 2018; Ho et al., 2019) directly maximizes mutual information by restricting the architecture to be invertible and tractable. This, however, requires the latent dimension to be the same as the dimension of the observations (f.e., bottleneck). As consequence, normalizing flows not well suited to learning concise representation of high dimensional data (e.g., images). Here, MIM often yields mappings that are approximately invertible, with high mutual information and low reconstruction errors.
Bornschein et al. (2015) share some of the same design principles as MIM, i.e., symmetry and encode/decoder consistency. However, their formulation models the joint density in terms of the geometric mean between the encoder and decoder, for which one must compute an expensive partition function. Pu et al. (2017) focuses on minimizing symmetric KL, but must use an adversarial learning procedure, while MIM can be minimized directly.
GANs (Goodfellow et al., 2014), which focus mainly decoder properties, without a proper inference model, have been shown to minimize JSD between the data anchor P(x) and the model generative process go (x)) (i.e., the marginal of the decoding distribution in MIM terms). In particular, prior work recognizes the importance of symmetry in learning generative models with reference to symmetric discriminators on æ and (Donahue al., 2016; Dumoulin al., 2017: Bang and Shim, 2018). In contrast, here we target JSD between the joint encoding and decoding distributions, together with a regularizer encourage high mutual information.
7 CONCLUSIONS
We introduce representation learning framework, named the mutual information machine (MIM), that defines a generative model which directly targets high mutual information (i.e., between the observations and the latent representation), and symmetry (i.e., consistency of encoding and decoding factorizations of the joint distribution). We derive a variational bound that enables the maximizion of mutual information in the learned representation for high dimensional continuous data, without the need to directly compute it. We then provide possible explanation for the phenomena of posterior collapse, and demonstrate that MIM does not suffer from it. Empirical comparisons to VAEs show that MIM learning leads to higher mutual information and better clustering (and classification) in the latent representation, given the same architecture and parametrization. In addition, show that MIM can provide reconstruction error similar to deterministic auto-encoder, when the dimensionality of the latent representation is equal to that of the observations. Such behaviour potentially allow approximate invertibility when the dimensionality differs, with stochastic mapping that is defined through consistency and high mutual information.
In future work, intend to focus utilizing the high mutual information mapping provided by MIM, by exploiting the clustered latent representation to further improve the resulting generative model.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexander A. Alemi, Ben Poole, lan Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy, An information-theorete analysis of deep latent-variable models. CoRR, abs/1711.00464, 2017.
Duhyeon Bang and Hyunjung Shim. High quality bidirectional generative adversarial networks. CoRR, abs/1805 10717, 2018.
Ishmael Belghazi, Sai Rajeswar, Aristide Baratin, R Devon Hjelm, and Aaron Courville. MINE: Mutual information neural estimation. In ICML, 2018.
Emmanuel Bengio, Valentin Thomas, Joelle Pineau, Doina Precup, and Yoshua Bengio. Independently controllable features. CoRR, abs/1703.07718, 2017.
Jörg Bornschein, Samira Shabanian, Asja Fischer, and Yoshua Bengio. Bidirectional helmholtz machines. CoRR, abs/ 2015.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Generating sentences from continuous space. CoRR, abs/1511.06349, 2015.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, 2016a.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. CoRR, abs/1611.02770 2016b.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components 2014.
estimation.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP.
arXiv: 1605.0080088 2016.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. CoRR, abs/1605.09782. 2016.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. ICLR, 2017.
Weihao Gao, Sewoong Oh, and Pramod Viswanath. Demystifying fixed k-nearest neighbor infor- mation estimators. CoRR, abs/1604.03006 2016. URL htttp://arxiae wwtpxiiiirrg//aaaa /abs///////// 4 03006.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pages 2672-2680, 2014.
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Karan Grewal, Phil Bachman, Adam
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. CoRR, abs/1902.00075. 2019. URL http: / /ttp//axivv/.////////eee/e/eaaa/ www.//.ii.orr.ooooooo..eooooeere abs/1902. 00275.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, art. arXiv:1412.6988 Dec 2014.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible Ixl Convolutions. In NIPS, 2018.
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In ICLR, 2013.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. In NIPS, 2016.

---

Under review as a conference paper at ICLR 2020
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys. Rev. E, 69:066138, Jun 2004, doi: 10.100//hyyReeE..6.0661.3. URL https: //ink httpss/////ink aps. org/doi/ 10. P 69. 38.138.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, (5066266):1332-83 2015.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86( 0112278-2324 1998.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. CoRR, abs/161 1.00712, 2016.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, and George van den Driessche. Parallel wavenet: Fast high-fidelity speech synthesis. arXiv preprint arXiv:1711.10433 2017.
Yuchen Pu, Weiyao Wang, Ricardo Henao, Liqun Chen, Zhe Gan, Chunyuan Li, and Lawrence Carin. Adversarial symmetric variational autoencoder. In Advances in Neural Information Processing Systems, pages
2017.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. In ICLR, 2018.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. CoRR, abs/1901.0344 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative Models. In ICMI., 2014.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS'99, pages 1057-1063, Cambridge. MA, USA, 1999. MIT Press.
Jakub M. Tomezak and Max Welling. VAE with a vampprior. CoRR, abs/1705.07100, 2017. URL 170500010.
George Tucker, Andriy Mnih, Chris J. Maddison, and Jascha Sohl-Dickstein, REBAR: low-variance, unbiased gradient estimates for discrete latent variable models. CoRR, abs/1703.07370, 2017.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. CoRR, abs/1600.00679) 2016.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. CoRR, abs/ 10s00771.,0037, 2017.
Laurens van der Maaten and Geoffrey E. Hinton. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9:2579-2605. 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 1998-6008, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist novel image dataset for benchmarking machine learning algorithms. CoRR, /tlps///0..07777,. 2017a. URL htL:ttp::// /ttppv.//////i///aeeaaeepeeaae http//axxi...ooaa/araa/ 1708. 07747.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: A novel image dataset for benchmark- ing machine learning algorithms, 2017b.
10

---

Under review as a conference paper at ICLR 2020
Appendix
Table of Contents
Derivations for MIM Formulation
A.1 Regularized JSD and Entropy
12
A.2 MIM Consistency
12
A.3 MIM Loss Decomposition
13
B MIM in terms of Symmetric KL Divergence
14
C Learning
15
C.1 MIM Parametric Priors
16
C.2 Gradient Estimation
16
C33 Training Time
Posterior Collapse in VAE
Additional Experiments
E.1 Entropy as Mutual Information .e..............................
19
E.2 Consistency regularizer in LaMM
19
E.3 Parameterizing the Priors
20
E,4
Effect of Consistency Regularizer on Optimization
21
F
Additional Results
22
F.1
Reconstruction and Samples for MIM and A-MIM
................................
22
F2
Latent Embeddings for MIM and A-MIM
25
A DERIVATIONS FOR MIM FORMULATION
2
(a)
(b)
(c)
Figure 6: MIM learning estimates two factorizations of a joint distribution: (a) encoding; (b) decoding factorizations. (c) The estimated joint distribution.
In what follows provide detailed derivations of key elements of the formulation in the paper, namely, Equations (6), (9), and (11). We also consider the relation between MIM based on the Gensen-Shannon divergence and the symmetric KI. divergence.
11

---

Under review as a conference paper at ICLR 2020
A.1 REGULARIZED JSD AND ENTROPY
First we develop the relation in Equation (6), between Jensen-Shannon divergence of the encoder and decoder, the average joint entropy of the encoder and decoder, and the joint entropy of the mixture distribution Ms.
The Jensen- -hasnnnn divergence with respect the encoding distribution go [x) P(x) and the decoding distribution (x z) P(z) is defined as
1
JSD(O)
2 (DKL. (po(z))) Mss + DxL (90(2/2)) Ms))
2 2)P) Ms) - (
+ (go P(x), Huu (1))
Where Ms = 2 (po x: x P(z) + go x) P(z)) is mixture of the encoding and decoding distri-
butions. Adding on the regularizer R4i 0 = { ( (z)
gives
1
OS + (+(O)=
H(Ms)
A.2 MIM CONSISTENCY
Here we discuss in greater detail how the learning algorithm encourages consistency between the encoder and decoder of MIM model, beyond the fact that they are fit to the same sample distribution. To this end we expand on several properties of the model and the optimization procedure.
A.2.1 MIM CONSISTENCY REGULARIZER
In what follows we derive the form of the MIM consistency regularizer, (0), given in Equation (9). Recall that we define Ma = 1/2 (Pe (z. pe pa(x, z)). We can show that LMIM is equivalent to LCE plus a regularizer by taking their difference.
RMIM(() = LMIM(0) LCE(O)
1 (H polx, =)) H (M H (Mss
1
2 (DEL (Ms p0 x)) + H(Ms) + DKL (Ms 1l (o HMMss)
DEL (Ms [Ma) HMMs)
1
2 (DLL (Ms po z)) + DKL (Ms po 2))) DKL (Ms MM)
where RMIM(() (0) is non-negative, and is zero only when the encoding and decoding distributions are consistent (i.e, they represent the same joint distribution). To prove that RMIM(() ()= 2 0 we now construct Equation (9) in terms of expectation over joint distribution, which yields
RMIM
H (Mss, po (z, 2)) + ((Ms, po( 2))) (MMs. Mo)
1
1
Es
log po (x, log logpo 2) + log (go (x, 2) po (x,, z))
1 (go 2) + po (z, x))
Mx
log /e go (x. z) pe (x. x) + log 2
v 40 (x, z) Po (x. 2) 2 (go (x, 2) + Po (x. z))
Ms
log
2
where the inequality follows Jensen's ' inequality, and equality holds only when go (x, x) = Po (x,
(i.e., encoding and decoding distributions consistent).
12

---

Under review as a conference paper at ICLR 2020
A.2.2 -OL--CORECTTNG GRADIENT
One important property of the optimization follows directly from the difference between the gradient of the upper bound LMIM and the gradient of the cross-entropy loss CCB. By moving the gradient operator into the expectation using reparametrization, one can express the gradient of LMN (0) in terms of the gradient of log Me and the regularization term in Equation (9). That is, with some manipulation one obtains
log qo + log po
a
90
+ po
Pe 90
1)
90 + [ge - 70
8
30 Pe
0
2 which shows that for any data point where a gap do > Pe exists, the gradient applied to po grows with the gap. while placing correspondingly less weight on the gradient applied to qo. The opposite true when go < pe. In both case this behaviour encourages consistency between the encoder and decoder. Empirically, we find that the encoder and decoder become reasonably consistent early in the optimization process.
ae
= ae
log
2
+
2
+ Pe
(14)
A.2.3 NUMERICAL STABILITY
Instead of optimizing upper bound LMIM. one might consider direct optimization of Len- Earlier we discussed the importance of the consistency regularizer in LMIM- Here we motivate the use of LMIM from a numerical perspective point of view. In order to optimize LCe directly, one must convert log and log pe to qe and pg. Unfortunately, this is has the potential to produce numerical errors, especially with 32-bit floating-point precision on GPUs. While various tricks can reduce numerical instability, we find that using the upper bound eliminates the problem while providing the additional benefits outlined above.
A.2.4 TRACTABILITY
As mentioned earlier, there may be several ways to combine the encoder and decoder into a single probabilistic model. One possibility we considered, as an alternative to MMg in Equation (7), is
Mo
qo Po
(15)
where B = /40 pe dæ dz is the partition function, similar to Bornschein et al. (2015). One could then define the objective to the cross-entropy as above with regularizer to encourage ,3 to be close to 1, and hence to encourage consistency between the encoder and decoder. This, however, requires a good approximation to the partition function. Our choice of Mg avoids the need for good value approximation by using reparameterization, which results in unbiased low-variance gradient, independent of the accuracy of the approximation the value.
A.3 MIM Loss DECOMPOSITION
Here show how break down the LMIM into the set of intuitive components given in Equation (11). To this end, first note the definition of LMIM:
1
LMIM (0)
2 (HMMs 2)) + H(MMspo(a 2)))
(16)
We will focus on the first half of Equation (16) for now,
1
2 HMMs, Po 2)) = 4 H( P(z), po (x, 2)) + H()) 2))
(17)
It will be more clear to write out the first term of Equation (17). 1/2 H (pe (z) P(z), pa( x)) in full
1 H(pe(x))) Pe x)) =
1
/
Pe p=( (oz ddzd
4
4
2,2
1
P pa [ (=))
4
m.z
1
P(x) log(pe (z) dxdz
4
13

---

Under review as a conference paper at ICLR 2020
We then add and subtract (P(())) where
4 H(P(z)) - 1 4// P(z) (oo) - - 12
/ pa ((2)
Writing out ()) (())) =)) H(((2)) (P(()))) and combining terms, we obtain
H (()) = ())) DEL (P(2) (18)
The second term in Equation (17) can then rewritten as
1
1 PKI (2)(z P(x) [I (o(
1 (10 (19)
II (g0 Poz), 2)) po
4
Combining Equations (18) and (19), we get the interpretable form for Equation 17, i.e.
H (.Ms,pe(x.5)) H (po( ((0(2
2
DKL (P(z) (1p 1
DxL (g0 P(x) po(
= (0) + DXL (lz))))
+ DEL (90 (()) pl x))
4
4
(20)
We the same basic steps derive analogous expression for H Ma z)) in Equation (16) and combine it with Equation (20) to get the final interpretable form:
MMIM (0) = (O) 2 DkL (P(z) (=)) + DeL (P(x)) go (x))
+ 1 DKL (90 (0 z)) DxL (po(x 2) P(z)) (z
B MIM IN TERMS OF SYMMETRIC KL DIVERGENCE
As discussed above, the VAE objective can be expressed as minimizing the KI. divergence between the joint anchored encoding and anchored decoding distributions. Below we consider model formulation using the symmetric KL divergence (SKL),
1
(KI = (DKL. (=)())))) P(z) || (o x:) P(x)) + DKL. (1o(z)=) (r))
the second term of which is the VAE objective. The mutual information regularizer Rμ (0) given in Equation (5) can be added to SKL obtain a cross-entropy objective that looks similar to MIM:
SKL(O)) + (0) (I (<) + (Ms, go P(x)))
When the model priors are equal to the anchors, this regularized SKL and MIM are equivalent. In general, however, the MIM loss is not bound the regularized SKL.
In what follows we explore the relation between SKL and JSD. In Section A.1 we showed that the divergence can be written
JSD 2 (H)p( H(po(cl))P(z)))
H((g(2|2) H(go(z|))
(()p)())))((z))44)))))) Ms) + H(go(z)x)
( P(x), Ms)) - Rn(O)
14

---

Under review as a conference paper at ICLR 2020
Using Jensen's ' inequality, bound JSD(@) from above,
JSD(0) - H(po(z))) 44000 (
+ ((() + H(go(() H(oo Ra(()
(21)
= = (oott:/aaddappotttooooooooooooop
+ +2R((0) RH(()
= DKL (po()))) (x() + DEL
+ +4Rn(()) RH(0)
(o(z))) (DKL (ol + DxL (=o(z)=) (o(()))) P(z)))
SKI((0)
From Equation (21), if we add the regularizer Rn( and combine terms, we get
SKL( + RH () = (H(Ms, (=( P(=)) + (MMss
Interestingly, we write this in terms of KI. divergence,
1
SKL(O) RO)
2 (KKL (Ms ((x=)) DKL (Ms
(Mss)
1
2 (DRI. (Ms (0(2))))))))))))))))) P(x)) + DeI. (Ms (Ms(lpo(ml)))))(
+ JSD(O) + (0)
which gives the exact relation between JSD and SKL.
SKL(()
(Dxı. (Ms (2))))))) DRI. (Ms pa pol=) ())
2
2 (DKL. (Ms (=(==)))((p))))) DKL (Ms (o(z))) (MMll
+ 2 (DeL (c0(z)x)P(z)) Ms) + D (pe( |MM)) Mss)
C LEARNING
Here provide detailed description of MIM learning, with algorithmic pseudo-code. In addition we offer practical considerations regarding the choice of priors' parameterization and gradient estimation. The empirical upper bound objective is expressed in terms of two cross-entropy terms (cf. (8)). Given N fair samples, {æ,, 2, drawn from the anchored (sample) distribution, Ms(x, in (4), the empirical loss is
ÉMIM (0: (m. 2 Mi)
log (2; go (m:) + log (0:))))))))))))))))))))))))))))) po(z:)
(22)
2N
where samples from Ms comprises equal numbers of points from pe (x)22) P(=) and (0(2)x) P(x). Samples from the anchors, P(x) and PP))), treated external observations; i.e., assume can sample from them but not necessarily evaluate the density of points under the anchor distributions.
Algorithm I specifies the corresponding training procedure. The algorithm makes assumptions the form of the parameterized distributions (e.g., discrete, or continuous). In practice, for gradient- based optimization, would like unbiased gradient estimator without the need to accurately
15

---

Under review as a conference paper at ICLR 2020
approximate the full expectations per se (t.e., in the cross entropy terms). This is particularly important when dealing with high dimensional data (e.g., images), where it is computationally expensive to estimate the value of the expectation. We next discuss practical considerations for the continuous case and the discrete case.
Algorithm MIM learning of parameters 0
Require: Samples from anchors P(x))
1: while not converged do
Doec + {Zi ži pe ()))
[/) N/2
3: Dene + {o,. 40 ( P x N/2
/=1
4:
D+ Ddoc U
5:
# See definition of LMIM in Eq. (22)
6:
CMIM (0) LAMM (0: D)
7: # Minimize loss
AO x -VoLMMM (0; D)
9: end while
C.1 MIM PARAMETRIC PRIORS
There are several effective ways to parameterize the priors, For the ID experiments below we model p=(z) using linear mixtures of isotropic Gaussians. With complex, high dimensional data one might also consider more powerful models (e.g., autoregressive, or flow-based priors). Unfortunately. the use of complex models typically increases the required computational resources, and the training and inference time. As an alternative we use for image data the vampprior Tomczak and Welling (2017), which models the latent prior as a mixture of posteriors, i.c., Po()) = Ek=1 (0 = ug) with learnable pseudo-inputs {uz K ===1- This is effective and allows one to avoid the need for additional parameters (see Tomezak and Welling (2017) for details vampprior's effect over gradient estimation).
Another useful model with high dimensional data, following Borsschei al. (2015), is define qo (z) as the marginal of the decoding distribution; i.e.,
go xx) = Ez~po(z) Pe (æ) [[)
(23)
Like the vampprior, this entails no new parameters. It also helps encourage consistency between the encoding and decoding distributions. In addition it enables direct empirical comparisons of VAE
learning to MIM learning, because we can then use identical parameterizations and architectures for both. During learning, when qe (x) is defined the marginal (23), evaluate log go(x) with single sample and reparameterization. When z is drawn directly from the latent prior:
log go (x) log Egg-po (x) [Po (x: log ž)
When is drawn from the encoder, given sample observation, use importance sampling:
(=(
log (xx log (x(a)(
po
log log po() log go (3(())
qo ( x>)
Algorithm 2 provides algorithm details with the marginal prior.
C.2 GRADIENT ESTIMATION
Optimization is performed through minibatch stochastic gradient descent. To ensure unbiased gradient estimates of LMIM we use the reparameterization trick Kingma and Welling (2013); Rezende et al. (2014) when taking expectation with respect to continuous encoder and decoder distributions, qo (2|#r and po (æ) 22). Reparameterization entails sampling an auxiliary variable ~ p(e), with known p(e). followed by deterministic mapping from sample variates to the target random variable, that is (=(z) = ge(e) and (0 (x) = he(e, x) for prior and conditional distributions. In doing so we assume p(c) is independent of the parameters 0. It then follows that
Vo (x [=(])) VolFep(() (ho
Eepp(e) [Vo.o(ho(ss
16

---

Under review as a conference paper at ICLR 2020
Algorithm 2 MIM learning with marginal (0(
Require: Samples from anchors P(x), P(z)
Require: Define to (x) = (=) [Po [x|=)]
while not converged do
2: # Sample encoding distribution
3: Done + {ar; 2; ~ {g(()=))P()))
4:
# Compute objective, approximale log qo (x) with 1 sample and importance sumpling
5:
log (0(x))) = log (z) (02;222 x.) log Po (zi) - log (z xx,)
6:
ÉMIM (0; Denc) + N Ex= (log (2; log (z;))(z))
7:
# Sample decoding distribution
8:
Daar + {xx (x( x))(z)}
9:
# Compute objective, approximate log qo (xr) with sample
10:
log goggorr (a:;) R po:
ΣMIM (0; Dace) t 2N Ein (log go log loggo(zz. (z.))(
11:
12:
# Minimize loss
13: AO x -oo (CMIM (0; Ddec) + EXIM (0; Dene))
14: end while
where f=(=) is the loss function with parameters 0. It is common to let p(e) be standard normal, < (O, 1), and for 2|æ to be Gaussian with mean (z) and standard deviation de (a)) ). in which case (x) + (æ), A more generic exact density model can be learned by mapping known base distribution (e.g., Gaussian) to target distribution with normalizing flows Dinh et (2014; 2016); Rezende and Mohamed (2015).
In the case of discrete distributions, e.g., with discrete data, reparameterization is not readily applica- ble. There exist continuous relaxations that permit reparameterization (e.g.. Maddison et al. (2016); Tucker et al. (2017)), but current methods are rather involved in practice, and require adaptation of the objective function the optimization process. Here simply the REINFORCE algorithm Sutton et al. (1999) for unbiased gradient estimates, as follows
(=)1)= (z)] (=)=)(==))) [Vefe + fo(z) log
A detailed derivation follows the use of the relation below,
Vege (z (z)
qe (e(=)az (x V log (=(xx)
order to provide unbiased gradient estimates as follows
VolEz [fo(z)]
Ve
fz(z) (z(2)))
/ qo g x) Vofe(z) dz / fo(z) (ogo
()(=)=) dz so(=) (=(=(=)) Vo log dz)
= (z)g)(z)=) (e) + fe log (2(x)))]
which facilitate the use of samples to approximate the integral.
C.3 TRAINING TIME
Training times of MIM models are comparable to training times for VAEs with comparable architec- tures. One important difference concerns the time required for sampling from the decoder during training. This is particularly significant for models like auto-regressive decoders (e.g., Kingma et al. (2016)) for which sampling is very slow. In such cases, we find that we can also learn effectively with sampling distribution that only includes samples from the encoding distribution, i.e., P(x)) ge te(2))) rather than the mixture. We refer to this particular MIM variant as asymmetric MIM (or A-MIM). We use in Sec. 5.2 when working with the PixelHVAE architecture Kingma et al. (2016).
17

---

Under review as a conference paper at ICLR 2020
D POSTERIOR COLLAPSE IN VAE
Here we discuss possible root cause for the observed phenomena of posterior collapse, and show that VAE learning can be viewed as an asymmetric MIM learning with a regularizer that encourages the appearance of the collapse. We further support that idea in the experiments in Section 5.1. As discussed earlier, VAE learning entails maximization of a variational lower bound (ELBO) on the log marginal likelihood, equivalently, given Equation (1), the VAE loss in terms of expectation over a joint distribution:
a (( (=)))) + log P(z) - log go (=|[z))
(24)
To connect the loss in Equation (24) to MIM, we first add the expectation of log P(æ), and scale the loss by a factor of z. to obtain
(g-P(x)) (7)
(log(o((z)2)P(3)) + log(g0 (ogo))))))))))))))))))
+ log P(xx) + log go [2/rr]
(25)
where (ax)) is the data distribution, which is assumed to be independent of model parameters 0 and to exist almost everywhere (i.e., complementing P()))) Importantly, because ((x)) does not depend on 0, the gradients of Eqs. (24) and (25) are identical up multiple of 1. they share the same stationary points.
Combining IID samples from the data distribution, a² P(x)), with samples from the corresponding variational posterior, z' ~ (z (,),, we obtain joint sampling distribution; i.e.,
M ( 2) = ((x)) go (2(ar)
where M VAE comprises the encoding distribution in Mss. With it one can then rewrite the objective in Equation (25) terms of the cross-entropy between MYAE and the parametric encoding and decoding distributions; i.c.,
H(MEA po(x))))(()) + H(M (0(2))) P(x)))+
HMyaa(()) - H Hagan(=) + [x;
(26)
The sum of the last three terms in Equation (26) is the negative joint entropy - -H(z, æ)) under the sample distribution VAE.
Equations (1) and (26). the VAE objective and VAE as regularized cross entropy objective respectively, define equivalent optimization problems, under the assumption that P(x)) and samples æ P(xr) do not depend on the parameters 0, and that the optimization is madient-hased. Formally, the VAE objectives (I) and (26) are equivalent up to a scalar multiple of 2 and an additive constant, namely, H. MygAE (x))
Equation (26) is the average of two cross cross-entropy objectives ( (i.e., between sample distribution M and the model decoding and encoding distributions, respectively). along with joint entropy term (i.e., last three terms), which can be viewed as regularizer that encourages a reduction in mutual information and increased entropy in and We note that Equation (26) is similar to the MIM objective in Equation (8), but with a different sample distribution, where the priors are defined to be the anchors, and with additional regularizer. In other words, Equation (26) suggests that VAE learning implicitly lowers mutual information. This runs contrary to the goal of learning useful latent representations, and we posit that an underlying root cause for posterior collapse, wherein the trained model show low mutual information which can be manifested as an encoder which matches the prior, and thus provides weak information about the latent state (e.g., see (Chen al., 2016b) and others). We point the reader to Section E. for empirical evidence for the use of joint entropy as mutual information regularizer.
E ADDITIONAL EXPERIMENTS
Here provide additional experiments that further explore the characteristics of MIM learning.
18

---

Under review as a conference paper at ICLR 2020
VAE+H
MIM-H
VAE+H
MIM-H
VAE+H
MIM-H
MI 2.1
MI -0,0059
MI 5.8
141 0.34
MI 5.9
MI -0.16
RMSE 0.83
RMSE 1.7
RMSE 0.016
RMSE 8.8.
RMSE 0.016
RMSE 4c+
(a) h E R2
(b) E 2220
(c) h E R5500
Figure 7: Effects of entropy mutual information regularizer in 2D and 2D synthetic problem. VAE and MIM models with 2D inputs, a 2D latent space, and 5, 20 and 500 hidden units. Top row: Black contours depict level sets of (a:); red dots reconstructed
points. Bottom row: Green contours are one standard deviation ellipses of go (= (=|=) for test points. Dashed black circles depict one standard deviation of P(z). Here added Hgo (x. x) to VAE loss, and subtracted HMs (a. x) from MIM loss, in order to demonstrate the effect of entropy
mutual information. Posterior collapse in VAE mitigated following the increased mutual information. MIM, on the other hand, demonstrates severe posterior collapse as result of the reduced mutual information (i.e. posterior matches prior over almost perfectly). (see inset quantities).
1.5
0.8
50
0.4
100
300
100
300
100
200
100
300
(a) MI
(b) NLL
(c) Recon. Error
(d) Classif. (5-NN)
Figure 8: Effects of entropy as mutual information regularizer in 2D æ and 2D synthetic problem. Test performance for modified MIM (blue) and modified VAE (red) for the 2D GMM data with (cf. Fig. 7), all as functions of the number of hidden units (on x-axis). Each plot shows the mean and standard deviation of 10 experiments. Adding encoding entropy regularizer to VAE loss leads high mutual information (i.e., prevent posterior collapse), low reconstruction error, and better classification accuracy. Subtracting sample entropy regularizer from MIM loss results in almost zero mutual information (severe collapse), which leads to poor reconstruction error and classification accuracy.
E. ENTROPY AS MUTUAL INFORMATION REGULARIZER
Here we examine the use of entropy as a mutual information regularizer. We repeat the experiment in Section 5.1 with added entropy regularizer. Figure 7 depicts the effects of added Hqp (x, =) to the VAE loss, and a subtracted Hs (xc 2) from MIM The corresponding quantitative values are presented in Figure 8. Adding the entropy regularizer leads to increased the mutual information, and subtracting it results in a strong posterior collapse, which in turn is reflected the reconstruction quality. While such experiment does represent valid probabilistic model, it supports our use of entropy as a regularizer (cf. Eq. (5)) for ISD in order to define a consistent model with high mutual information.
E.2 CONSISTENCY REGULARIZER IN LMIM
Here explore properties of models for ID and learned with LMIM and Les, the difference being the model consistency regularizer RMIM()) (0). All model priors and conditional likelihoods (go (z),, (2)==) po (=),, p=(=))) parameterized 10-component Gaussian mixture models, and
19

---

Under review as a conference paper at ICLR 2020
Reconstruction
Anchor Consistency
Prior Consistency
(a) CE
(b) MIM
(c) CE
(d) MIM
(e) CE
(1) MIM
Figure 9: We explore the influence of consistency regularizer Rg. CE and MIM indicate the loss, CCE or LMM (the regularized objective). respectively. Top row shows anchor P(x) (dashed), prior (x) (red), and reconstruction distribution Φ; ~ P(x) + 2; (=(=) + x ~ pe p0(2) (z||z) (green). Bottom row mirrors the top row, with anchor P(z) (dotted), prior pe (blue), and reconstruction distribution 2; ~ P(z) + x ~ po n0(2)z)) - 2 go (2) x:) (yellow). In (c-d) the reconstruction is replaced with decoding from anchors 2; ~ P(z) + x ( 2;) (green), and encoding a;; ~ P(x) + 2 ~ priors Po 2)) x Pa (zr z;) (green), and encoding a; ~ go (ax)) + z (0(=22) (yellow). While both models offers similar reconstruction (a-b), and similar consistency w.r.t. the anchors (c-d),
(yellow). In (e-f) the reconstruction replaced with decoding from
only MIM finds consistent model (e-f). See text for details.
optimized during training. Means and variances for the conditional distributions were regressed with fully connected layers (h E 10 and a swish activation function Ramachandran et al. (2018).
Top and bottom rows in Fig. 9 depict distributions in observations and latent spaces respectively. Dashed black curves are anchors, P(x:) on top, and P(z) below (GMMs with up to 3 components). Learned model priors, qo (x) and (z().. are depicted as red (top) and blue (bottom) curves.
Green histograms in Fig. ((,bb) depict reconstruction distributions, computed by passing fair samples from P(æ)) through the encoder to and then back through the decoder to æ. Similarly the yellow histograms shows samples from P(z)) passed through the decoder and then back to the latent space. For both losses these reconstruction histograms match the anchor priors well. In contrast, only the priors that were learned with LCE loss approximates the anchor well, while the ΣMIM priors do not. To better understand that, consider generative procedures: sampling from the anchors, and sampling from the priors.
Anchor consistency is depicted in Fig. 9(c,d), where Green histograms marginal distributions over x from the anchored decoder (i.e., samples from P( (x: z) ). Yellow are marginals over z from the anchored encoders P (o=:)oo( ar). One can see that both losses results in similar quality of matching the corresponding opposite anchors.
Priors consistency is depicted in Fig. 9e..f,, where Green histograms are marginal distributions over æ from the model decoder pe pe(=) (=)))()))) Yellow depicts marginals over z from the model encoder (x)x)) ()) Importantly, with LMIM the encoder and decoder are consistent; i.e., go (x (red curve) matches the decoder marginal, while (2) (blue) matches the encoder marginal. The model trained with CCF (i.e., without consistency prior) fails learn a consistent mccoer-decoded pair. We note that in practice, with expressive enough priors, LMIM will be tight bound for Ler.
E.3 PARAMETERIZING THE PRIORS
Here explore the effect of parameterizing the latent and observed priors. A fundamental idea in MIM is the concept of a single model, MM. As such, parameterizing prior increases the global expressiveness of the model Mg. Fig. 10 depicts the utilization of the added expressiveness in order to increase the consistency between the encoding and decoding model distribution, in addition to the consistency of Mo with M.s.
20

---

Under review as a conference paper at ICLR 2020
(a) go (a), p=())
(b) (())),p(())
(c) qo p(z)
(d) (()).
Figure 10: MIM prior expressiveness. In this experiment we explore the effect of learning a prior, where the priors g(ær) and p(z) are normal Gaussian distributions. Top row shows anchor P(x)) (dashed), prior go (x:) (red), and decoding distribution ~ po(z) + x¹ po(o: z,) (green). Bottom row mirrors the top row, with anchor P(z) (dotted), prior pe(z) (blue), and encoding distribution æ; ~ (æ) + ~ qo qe(= ()) (yellow). As can be seen, parameterizing priors affects all learned distributions, supporting the notion optimization of a single model MM. We point that (a) demonstrates the best consistency between the priors and corresponding generated samples, following the additional expressiveness.
E,4 EFFRCT OF CONSISTENCY REGULARIZER ON OPTIMIZATION
Here we explore whether learned model with consistent pecoding-decodingg distributions (i.e., trained with LIM)) also constitutes an optimal solution CE objective (i.e., trained with LcE). Results are depicted in Fig. 11. In order to distinguish between the effects of the optimization from the consistency regularizer we initialize MIM model by pre-tra pre-training it with Lee loss followed by LMIM training in Fig. 11(i,, and vice verse in Fig. (ii). (a-b.e-f) All trained models in Fig. 11 exhibit similarly good reconstruction (green matches dashed black). (c-d.g-h) However, only models that trained with ZλTM exhibit encoding-decoding consistency (green matches red, yellow matches blue). While it is clear that the optimization plays an important role (i.e., different initialization leads different local optimum), it also clear that macoding-decodiee optimum of LCE, as depicted in a non non-consistent model (h) which was initialized with a consistent model (g). Not surprisingly. without the consistency regularizer training with Lee results in better fit of priors to anchors (f) as it is utilizing the expressiveness of the parametric priors in matching the sample distribution.
consistency not necessarily an
21

---

Under review as a conference paper at ICLR 2020
Reconstruction
Prior Sampling
(a) Leu
(b) LMIM
(c) Leu
(d) LuIa
(i LcB LMIM
Reconstruction
Prior Sampling
M
(c) Lsuma
(f) Lee
(g) LMIM
(h) Lcu
(ii) LMM LCB
Figure 11: Effects of MIM consistency regularizer and optimization on encoding-decoding con- sistency. (i) and (ii) differ in initialization order. Odd rows: anchor P(x) (dashed), prior
(x)) (red). Even rows: anchor P(z) (dotted), prior po(z) (blue). (a-b.e-f) Reconstruction 2; P(x) + 2; (0(2)z;)) - x (=]2z, (x green, 2; yellow). (c-d.g-h) Prior decod- ing 2i p=)=) + o. (a( z2) (green), and prior encoding æ; (0(x)) - g0(2|00 (=)=)) (yellow). See text for details.
F ADDITIONAL RESULTS
Here provide additional visualization of various MIM and VAE models which not included in the main body of the paper.
F.1 RECONSTRUCTION AND SAMPLES FOR MIM AND A-MIM
In what follows show samples and reconstruction for MIM (i.e., with convHVAE architecture), and A-MIM (i.e., with PixelHVAE architecture). We demonstrate, again, that powerful enough encoder allows for generation of samples which are comparable VAE samples.
22

---

Under review as a conference paper at ICLR 2020
Standard Prior
VampPrior Prior
(a) VAE (S)
(b) A-MIM (S)
(c) VAE (VP)
(d) A-MIM (VP)
Figure 12: MIM and VAE learning with PixeIHVAE for Fashion MNIST. The top three rows (from top to bottom) are test data samples, VAE reconstruction, A-MIM reconstruction. Bottom: random samples from VAE and MIM. (c-d) We initialized all pseudo-inputs with training samples, and used the same random seed for both models. As a result the samples order is similar.
Standard Prior
VampPrior Prior
VAR (S)
(b) MIM (S)
(c) VAE (VP)
(d) MIM (VP)
Figure 13: MIM and VAE learning with convHVAE for Fashion MNIST. The top three rows (from top to bottom) are test data samples, VAE reconstruction, MIM reconstruction. Bottom: random samples from VAE and MIM.
23

---

Under review as a conference paper at ICLR 2020
Standard Prior
VampPrior Prior
7
2__
7
4
6
7
2
4
7
2
0
4
/ 4
A
6
9
O
6
7
2
/
0
4
1
4
6
9
O
6
7
2_
O
4
/
4
9
9
0
6
7
2
/
0
4
/
4
8
6
9
O
6
L 2
6
g
5
<
9
5
A 4
3
3.
77 3
22
5 | 2
0 4
9
/
-2
1
3
5
0
4
/
8 3
2
3 7 /
3
9
X
5
23
2
4
4
3
6
7
1
4 8 2
4
3
6
4
2
2
9
5
8 2
0
9
/
2
O
9 3
O
3
6
S
X
Y
8
V
J
T
3
7
8 3 8
J
8
8 6 5 P 6
O
6
6 5 4 9
6
7
X
4
&
8
44
6
/
6
0
7
6
6
9
0
6
7 8
8 6
2
&
O
1 6
PY
1
8
3
9
8
5
I
8
3
4
7
3
O
2 3
J
X
8
3
3
O
7
7
8
3
5
7
7
9
9 Q
9
9
H
<<
3
D
9 (c) VAE (VP)
4
6
O
O
9
9
(a) VAE (S)
(b) A-MIM (S)
(d) A-MIM (VP)
Figure 14: MIM and VAE learning with PixeIHVAE for MNIST. Top three rows are test data samples, followed by VAE and A-MIM reconstructions. Bottom: random samples from VAE and MIM. (c-d)
We initialized all pseudo-inpus with training samples, and used the same random seed for both models. As result the samples order is similar.
Standard Prior
VampPrior Prior
7
2
4
{
7
9
O
6
7
2_
4
f
7
9
7
2
O
4
/ 4
9
5
9
0
6
7
2
O
4
/
4
9
5
9
O
6
7
2
O
4
1
4
6
5
9
O
6
7
2
/
O
4
/
4
9
S
9
O
6
5 1 4
O
4
3
3
8 8
2
2 5
2 4
5
3 4
O 4
4
/
2
3
0
4
/
9
2
3 7
5
Y
4
9
7
5
Y
5
3
6
1 7
1
3
5
3
6
1
&
7
4
9
4
-
7
5
2
P
6
7
4
0
4
/
2
8
5
9
4
0
9 3 7
/ 8 6 S 8 O
3
5
/
5
4
9
2
9
5
J
2
9
2
8
J
2
3
3
7
7
6
8
3
0
Y
5
G
9
6
0
7
6
6
7
0
2
1
0
6
9
O
9
Z
Y
I
8
4
9
8
5
1
8
9
3
9
8 9 6
6
9
2
2
0
6
7
A
FX
F
3
3
3
6
7
K
9
8
3
3
3
0
7
7
à
/
0
/
5
A
1
O
9
/
6
O
O
7
4
4
(a) VAE (S)
(b) MIM (S)
(c) VAE (VP)
(d) MIM (VP)
Figure 15: MIM and VAE learning with ConvHVAE for MNIST. Top three rows are test data samples, followed by VAE and MIM reconstructions. Bottom: random samples from VAE and MIM.
24

---

Under review as a conference paper at ICLR 2020
Standard Prior
VampPrior Prior
A
A
C
2
3
A1
34
2
A
O
V
57
A
19
Ø2
.713
6F9
10
ME
411
1
6
A
S
20
W
3X
C
3
33
10
54
7
5
b
Qu
QAI
10
OO
II
E
IL
4C
3
2X
M
2
C
3
3
C2
2
44
A
30
3
(a) VAE (S)
(b) A-MIM (S)
(c) VAE (VP)
(d) A-MIM (VP)
Figure 16: MIM and VAE learning with PiXelHVAE for Omniglot. Top three rows are test data samples, followed by VAE and A-MIM reconstructions. Bottom: random samples from VAE and MIM.
Standard Prior
VampPrior Prior
A
>
7
A
7
4
O
7
V
7
C
CJ
A
O
J
V
A
O
V
5
P
CJ
A
O
7
3
T
23
T-
P
&
X
3
35
M
2
2
O
LT
U
VAE (S)
(b) MIM (S)
(c) VAE (VP)
(d) MIM (VP)
Figure 17: MIM and VAE learning with convHVAE for Omniglot. Top three rows are test data samples, followed by VAE and MIM reconstructions. Bottom: random samples from VAE and MIM.
F.2 LATENT EMBEDDINGS FOR MIM AND A-MIM
In what follows show additional t-SNE visualization of unsupervised clustering in the latent representation for MIM (i.e., with convHVAE architecture), and A-MIM (i.e, with PixeIHVAE architecture).
25

---

Under review as a conference paper at ICLR 2020
VAR (S)
(b) MIM (S)
(c) VAE (VP)
(d) MIM (VP)
Figure 18: MIM and VAE embedding for Fashion MNIST with convHVAE architecture.
(a) VAB (S)
(b) MIM (S)
(c) VAE (VP)
(d) MIM (VP)
Figure 19: MIM and VAE = embedding for MNIST with convHVAE architecture.
(a) VAE (S)
(b) A-MIM (S)
(c) VAE (VP)
(d) A-MIM (VP)
Figure 20: A-MIM and VAE z embedding for Fashion MNIST with PixelHVAE architecture.
VAE (S)
(b) A-MIM (S)
(c) VAE (VP)
(d) A-MIM (VP)
Figure 21: A-MIM and VAE embedding for MNIST with PiXelHVAE architecture.
26


Reviews:
[{"review_id": "rkxjPJvb5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper defines a new learning objective of an autoencoder framework for learning joint distributions over observations and latent states, where the objective is the joint entropy of  M_s, an equally weighted mixture of the encoding and decoding distributions. Since it involves the marginal distribution which is intractable, it goes further to introduce an upper bound which is the cross-entropy between M_s and M_theta. To maintain consistency, it introduce an upper bound again, which is the average of cross entropy between the mixture distribution M_S and the model encoding and decoding distribution. Through comprehensive experiments, the effectiveness is validated.\n\nI don't think Mutual Information Machine (MIM) is a proper name for this approach, since it intends to minimizing joint entropy M_s. Also the notation of cross-entropy H(p,q) is confusing, it's better to use CE(p,q) instead.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rkx0FE1VFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present the Mutual Information Machine (MIM), which is essentially a  latent variable model learned via Adversarially Learned Inference with regularizations that prefer learning representations with high mutual information. While the method is interesting, it does not seem to be well justified in representation learning, and the empirical results are not well-justified due to the use of weak baselines.\n\nPros:\n\nThe idea of formulating the L_MIM objective appears interesting, which uses the mixture of P_\\theta and Q_\\theta and minimizes an upper bound to H(x) + H(z) - I(x, z), so it is effectively minimizing a lower bound to I(x, z), under the proposed symmetric distribution.\n\nCons:\n\nSymmetry argument:\n\t- Prior work on maximizing mutual information for latent variables focus on the MI as evaluated in the \"encoder distribution\", whereas in this paper the argument is focused on symmetry, i.e. the MI estimated is based on \\mathcal{M}_s. \n\t- However, for practical purposes, it is unclear how this is superior than the \"encoder distribution\" paradigm. Eventually, we wish to use the z for some downstream tasks, which requires us to obtain z from some distribution (e.g. the encoder q(z|x)) and then run classification on this z. \n\t- Unless we sample z | x from the conditional distribution of \\mathcal{M}_s as proposed by the authors, it is unclear why we need this symmetry in representation learning; it seems slower to sample from the MIM distribution though (see motivation for A-MIM).\n\t- The authors did not demonstrate results that compare A-MIM and MIM on the same architecture.\n\nFailure to discuss certain related literature, such as Alemi et al. 2017 and Zhao et al. 2018. These are also \"strongly related to VAEs\" and related to mutual information objectives with latent variables.\n\t- Alemi et al. 2017 discussed this in the context of \"rate discussion trade-off\"\n\t- Zhao et al. 2018 proposed objectives to find specific trade-offs under constraints\nIn fact, Zhao et al. 2018 considered the MIs in both P_theta and Q_theta, and one should be able to obtain a \"symmetric\" MI by adding these two Mis uniformly (although again, I am not sure how this is useful for downstream applications on the model).\n\nUsing VAE as a baseline comparison. \n\t- It is well known that the VAE objective does not encourage maximizing mutual information, and various existing works have already been proposed to address this problem. Outperforming VAE in terms of MI is hence not very surprising.\n\t- Methods that encourage mutual information maximization under the latent variable generative modeling framework would be more suited for comparison. How does MIM compare with the methods introduced in Chen et al. 2016, Alemi et al. 2017 and Zhao et al. 2018? \n\t- In the context of ALI and BiGAN, one could also apply other approaches like MINE / InfoMAX to maximize MI. Again, there is no comparison with these approaches. \n\t- Alemi and Zhao both showed that there is a trade-off between consistency and mutual information under the same architecture (using their objectives). It seems that what MIM empirically does is also to sacrifice NLL for better MI, and not clear whether MIM gives comparable or better trade-offs.\n\nThe paper would be more convincing if the authors empirically compare with other sensible baselines and justify why we should use MIM as opposed to A-MIM in representation learning.\n\n[relevant papers]\nChen et al. 2016 Lossy VAEs\nAlemi et al. 2017 Fixing a broken ELBO\nZhao et al. 2018 A Lagrangian Perspective to Latent Variable Generative Models\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HklF6HxCtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "When modeling a population distribution Pop(x) one typically minimizes the cross entropy\n\n(1)  Theta^* = argmin_\\Theta   H(Pop,P_\\Theta)\n                      = argmin_\\Theta   E_{x \\sim Pop}  -ln P_\\Theta(x).\n\nThe ELBO used in the definition of a VAE provides an upper bound on this loss --- VAE is directly motivated by (1).\n\nThis paper defines a different loss function --- L_MIM.  However, I find the rather vaguely stated motivations for L_MIM uncompelling.  L_MIM does not bound (1) nor does it bound or estimate mutual information.  A lack of any compelling motivation could be overlooked if the empirical results where compelling.  But the empirical evaluation is very shallow with a focus on MNIST and similar data sets.\n\nA technical comment is that I do not understand how z_i is generated in equation (12).  What distribution are the pairs (x_i,z_i) being drawn from?  I should note, however, that this technicality is not my main issue with the paper and will not influence my judgement.\n\nPostscript: I have read the author's response and have not been swayed.  I still find the motivation obscure and the experiments weak.  The fact that other authors have done questionable things does not sway me.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an autoencoder framework for learning joint distributions over observations and latent states. The reviewers expressed concerns regarding the motivation for this work, the presentation with respect to prior work, and unconvincing experiments. In its current form the paper is not ready for acceptance to ICLR-2020.
Decision: Reject