Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper defines a new learning objective of an autoencoder framework for learning joint distributions over observations and latent states, where the objective is the joint entropy of  M_s, an equally weighted mixture of the encoding and decoding distributions. Since it involves the marginal distribution which is intractable, it goes further to introduce an upper bound which is the cross-entropy between M_s and M_theta. To maintain consistency, it introduce an upper bound again, which is the average of cross entropy between the mixture distribution M_S and the model encoding and decoding distribution. Through comprehensive experiments, the effectiveness is validated.

I don't think Mutual Information Machine (MIM) is a proper name for this approach, since it intends to minimizing joint entropy M_s. Also the notation of cross-entropy H(p,q) is confusing, it's better to use CE(p,q) instead.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: When modeling a population distribution Pop(x) one typically minimizes the cross entropy

(1)  Theta^* = argmin_\Theta   H(Pop,P_\Theta)
                      = argmin_\Theta   E_{x \sim Pop}  -ln P_\Theta(x).

The ELBO used in the definition of a VAE provides an upper bound on this loss --- VAE is directly motivated by (1).

This paper defines a different loss function --- L_MIM.  However, I find the rather vaguely stated motivations for L_MIM uncompelling.  L_MIM does not bound (1) nor does it bound or estimate mutual information.  A lack of any compelling motivation could be overlooked if the empirical results where compelling.  But the empirical evaluation is very shallow with a focus on MNIST and similar data sets.

A technical comment is that I do not understand how z_i is generated in equation (12).  What distribution are the pairs (x_i,z_i) being drawn from?  I should note, however, that this technicality is not my main issue with the paper and will not influence my judgement.

Postscript: I have read the author's response and have not been swayed.  I still find the motivation obscure and the experiments weak.  The fact that other authors have done questionable things does not sway me.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The authors present the Mutual Information Machine (MIM), which is essentially a  latent variable model learned via Adversarially Learned Inference with regularizations that prefer learning representations with high mutual information. While the method is interesting, it does not seem to be well justified in representation learning, and the empirical results are not well-justified due to the use of weak baselines.

Pros:

The idea of formulating the L_MIM objective appears interesting, which uses the mixture of P_\theta and Q_\theta and minimizes an upper bound to H(x) + H(z) - I(x, z), so it is effectively minimizing a lower bound to I(x, z), under the proposed symmetric distribution.

Cons:

Symmetry argument:
	- Prior work on maximizing mutual information for latent variables focus on the MI as evaluated in the "encoder distribution", whereas in this paper the argument is focused on symmetry, i.e. the MI estimated is based on \mathcal{M}_s. 
	- However, for practical purposes, it is unclear how this is superior than the "encoder distribution" paradigm. Eventually, we wish to use the z for some downstream tasks, which requires us to obtain z from some distribution (e.g. the encoder q(z|x)) and then run classification on this z. 
	- Unless we sample z | x from the conditional distribution of \mathcal{M}_s as proposed by the authors, it is unclear why we need this symmetry in representation learning; it seems slower to sample from the MIM distribution though (see motivation for A-MIM).
	- The authors did not demonstrate results that compare A-MIM and MIM on the same architecture.

Failure to discuss certain related literature, such as Alemi et al. 2017 and Zhao et al. 2018. These are also "strongly related to VAEs" and related to mutual information objectives with latent variables.
	- Alemi et al. 2017 discussed this in the context of "rate discussion trade-off"
	- Zhao et al. 2018 proposed objectives to find specific trade-offs under constraints
In fact, Zhao et al. 2018 considered the MIs in both P_theta and Q_theta, and one should be able to obtain a "symmetric" MI by adding these two Mis uniformly (although again, I am not sure how this is useful for downstream applications on the model).

Using VAE as a baseline comparison. 
	- It is well known that the VAE objective does not encourage maximizing mutual information, and various existing works have already been proposed to address this problem. Outperforming VAE in terms of MI is hence not very surprising.
	- Methods that encourage mutual information maximization under the latent variable generative modeling framework would be more suited for comparison. How does MIM compare with the methods introduced in Chen et al. 2016, Alemi et al. 2017 and Zhao et al. 2018? 
	- In the context of ALI and BiGAN, one could also apply other approaches like MINE / InfoMAX to maximize MI. Again, there is no comparison with these approaches. 
	- Alemi and Zhao both showed that there is a trade-off between consistency and mutual information under the same architecture (using their objectives). It seems that what MIM empirically does is also to sacrifice NLL for better MI, and not clear whether MIM gives comparable or better trade-offs.

The paper would be more convincing if the authors empirically compare with other sensible baselines and justify why we should use MIM as opposed to A-MIM in representation learning.

[relevant papers]
Chen et al. 2016 Lossy VAEs
Alemi et al. 2017 Fixing a broken ELBO
Zhao et al. 2018 A Lagrangian Perspective to Latent Variable Generative Models

review_title: Official Blind Review #3

Meta-review: The paper proposes an autoencoder framework for learning joint distributions over observations and latent states. The reviewers expressed concerns regarding the motivation for this work, the presentation with respect to prior work, and unconvincing experiments. In its current form the paper is not ready for acceptance to ICLR-2020.
Decision: Reject