Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper gives a nice interpretation why recent works that are based on variational lower bounds of mutual information can demonstrate promising empirical results, where they argue that the success depends on "the inductive biasin both the choice of feature extractor architectures and the parametrization of theemployed MI estimators." To support this argument, they carefully design a series convincing experiments which are stated in full in Section 3. Moreover they show some connection to metric learning. 

I have confusions thought about the writing. These should be addressed before the acceptance.

1. For many equations, if the X, Y, are random variables, they should be capitalized. For example, in equation (1), should p(x,y) be written as p(X,Y)? If I'm right, please correct it every where in the paper.

2. In equation (3), should the symbol E be there? I think it shouldn't. Since in real implement, Monte Carlo estimation is used and the mini-batch samples are selected in accordance to one positive sample and the rest negative samples, E shouldn't be there. But I see E in all-most every papers such as Poole and Oord's papers. To equation (5), the connection to metric learning, there is no E. If I'm correct, please correct in the paper including the appendix.

3. For the proof of proposition 1, it is stated X_1 <-- X --> X_2 is equivalent to X_1 --> X --> X_2, why? I don't have Cover's book on hand, so I couldn't make sure.

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper addresses a question on whether mutual information (MI) based models for representation learning succeed primarily thanks to the MI maximization. The motivation of the work comes from the fact that although MI is known to be problematic in treatment, it has been successfully applied in a number of recent works in computer vision and natural language processing. The paper conducts a series of experiments that constitute a convincing evidence for a weak connection between the InfoMax principle and these practical successes by showing that maximizing established lower bounds on MI are not predictive of the downstream performance and that contrary to the theory higher capacity instantiations of the critics of MI may result in worse downstream performance of learned representations. The paper concludes that there is a considerable inductive bias in the architectural choices inside MI models that are beneficial for downstream tasks and note that at least one of the lower bounds on MI can be interpreted as a triplet loss connecting it with a metric learning approach.
I consider this paper to be a considerable contribution to the understanding of what underlies the performance of unsupervised representation learning with MI maximization and provides a good discussion and analogous insights from parallel works and a number of possible directions to explore. Although I still have a couple of questions addressing which will help to understand the paper.
-  In experiment 3.1, when using RealNVP and maximizing the lower bound of MI may it be the case that the representations are learned to benefit the downstream task because of the form of the lower bounds? In other words, the lower bound is possibly not very tight and its maximization has a side effect of weights adjusting to yield simple representations useful for a linear classifier?
-  It would be interesting to see which factor contributes more to the performance: I_NCE being a triplet loss or an inductive bias in the design choice?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors studied the usage of the mutual information maximization principle in representation learning. They argue that the bias in the estimation of lower bound of MI may loosen the connection between InfoMax principle and representation learning. Specifically, they figure out the following phenomenon by experiments.
    1. Large MI is not predictive of downstream performance.
    2. Higher capacity critics can lead to worse downstream performance.
    3. Encoder architecture can be more important than the specific estimator.
Finally, the authors make a connection to deep metric learning.

Recently, using the principles of mutual information in deep learning has been very hot. However, a lot of papers just use the concept of mutual information without deep thinking (why use MI? why not other metrics? what is the estimation bias and variance for MI?). Also, most paper do not have theoretical guarantees. In this paper, the authors think deeper about the InfoMax principle, and point out some weakness about the connection between InfoMax principle and the quality of representation learning. I think this paper can trigger some deep and calm thinking in this area.

However, I think this paper is more critical then constructive. It is good to point out some weakness of the InfoMax principle, but what should we do next? How should we fix the InfoMax principle, or mutual information estimator, to make it more robustly useful for representation learning? Or does it mean that InfoMax principle is not a good tool for representation learning? I agree that this questions are out of the scope of this paper, but I would like to see the authors share more constructive thoughts in the discussion area, to make this paper more constructive.

Overall, I would like to weakly accept this paper. I think this paper worth a strong accept if some constructive ideas are provided in the rebuttal. 
review_title: Official Blind Review #1

Meta-review: This paper exams the role of mutual information (MI) estimation in representation learning. Through experiments, they show that the large MI is not predictive of downstream performance, and the empirical success of  methods like InfoMax may be more attributed to the inductive bias in  the choice of architectures of discriminators, rather than accurate MI estimation. The work is well appreciated by the reviewers. It forms a strong contribution and may motivate subsequent works in the field. 

Decision: Accept (Poster)