Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work proposes to decompose mutual information estimation into subtasks, based on the chain rule of MI, i.e. Eq.(6).
A novel bound for the proposed decomposition insight is achieved based on the classic InforNCE bound. Higher total information is expected to be captured with the proposed method. 


The proposed insight is understandable and reasonable to achieve the desired goal. it is
 also presented to be effective to achieve higher MI and results in better representations for classification. 
I have one question regarding this work.

1) Work [1]  also presents analysis on estimating mutual information in decomposing, or hierarchy, manner, i.e. Eq(19) in its paper. Can you discuss this work with yours? Can I understand your work as a variant of this work performed with InforNCE estimation? I appreciate your opinion here. 


[1] Gao, Shuyang, et al. "Auto-encoding total correlation explanation." The 22nd International Conference on Artificial Intelligence and Statistics. 2019.
review_title: good work with reasonable idea, needs to clarify its connection with another work. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper proposed a lower-bound on the mutual information by introducing a conditional mutual information term.
The authors claim that their conditional mutual information term can avoid capturing redundant information shared among different data samples and therefore is more efficient.


Reason for the score:

I think the authors could improve their paper by comparing with more methodologies such as the information bottleneck (IB), and various approximations such as MINE from ICLR 2018 (https://openreview.net/pdf?id=rJHOuiqaf). 



Pro:
- The authors provided experiments in vision, dialogue, and synthetic datasets on different models.

Con/Questions:
- Some experimental setups are not too clear. For example, how did the authors choose information-restricted views?
- I think the authors could compare their method with other relevant information theoretic methods such as the Information Bottleneck (IB). Many recently applications in vision, language, and molecules found IB very useful in balancing between compression and information retention.
- I think the paper is a bit difficult to follow in its current format particularly with its current choices of words. For example, the authors use both $x, y$ to denote data features instead of data features and data labels which is quite uncommon. "views" is also used frequently in this paper which I can only assume that the authors meant "data samples".

-----------------------------------------------------------------
Post Rebuttal:

Many thanks for the authors to update their original paper addressing some of my questions and concerns.
I have now updated the score.
review_title: Good attempt but more comparisons with other information theoretic objective functions will make it better.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a contrastive learning approach where one of the views, x, is converted into two subviews, x' and x'', and then separate InfoNCE style bounds constructed for each of I(x'';y) and I(x';y|x'') before being combined to form an overall training objective.  Critically, the second of these is based on the conditional MI, I(x';y|x''), distinguishing it from previous work using multiple views that just take the marginal I(x';y).  Estimating this conditional MI transpires to be somewhat trickier due to the additional intractability from p(y|x''), with approximations suggested to get around this.  Experiments are performed on both vision and NLP problems.


*Review Overview*


I believe the premise of the paper is quite sensible and the general idea potentially useful, albeit somewhat incremental.  However, I am rather unconvinced by the approximations used to try and estimate p(y|x''); this rather lets the paper down as I feel this is the main technical challenge with the other contributions being rather straightforward.  Further, I feel the experimental results are somewhat weak, both in terms of the fact that the gains shown are quite small, but also because there are doubts as to whether these gains are actually statistically significant at all or just a somewhat "engineered" win that does not accurately reflect the underlying approaches.


*Strengths*

+ As far as I'm aware, the suggested approach has not appeared in the literature before.
+ This is a highly active research area that is very much of interest to the ICLR community.
+ The high-level idea of using subviews and breaking down the MI in this way seems sensible and potentially useful.
+ The toy experiment in Figure 2 is helpful in showing that the approach can be helpful if p(y|x'') can be estimated well.
+ It is good to see experiments on both vision and NLP problems.
+ The paper is mostly quite easy to follow.


*Weaknesses (in roughly decreasing order of importance)*
- The experiments do not follow procedures that are clearly statistically sound and the gains could easily just be a result of luck and /or tweaking the approach until it works.  In particular, no error bars are provided for any of the results; different approaches are run for different problems without justification such that it feels like things are somewhat cherry-picked; additional heuristics are added to what is actually run without clear a priori motivation such that they feel rather based on trial and error and undermines whether the gains are actually from the suggested approach or just engineering the exact setup until it gives an improvement; some important details seem to be missing such as how lambda is chosen. 
- The experimental gains are rather modest.
- The approximation methods for p(y|x'') are rather unconvincing: the variational bound requires a very complex learning problem in itself to encapsulate p(y|x'') while permitting optima that do not produce approximations that are close to this target at all, while the importance sampling approach is very much just a heuristic with rather flimsy justification.  More explanation on this is given later.
- Much of the technical content, other than how one approximates p(y|x''), is very straightforward and is not enough to warrant publication on its own.  For example, the first time anything new is introduced is not until section 3.1 at the bottom of page 4, while the conditional InfoNCE bound/proposition 1 is trivial and arguably not even new to the literature (for example, equivalent bounds are already employed in conditional MI estimation in a Bayesian experimental design context, see e.g. http://proceedings.mlr.press/v108/foster20a/foster20a.pdf).
- Though not bad, the writing of the paper could definitely be improved.  In particular, the abstract and introduction could be much easier to follow and have a clearer narrative, while the initial explanations of what x, y, and the sub-views x' and x'' represent could be much clearer.
- [Minor] I'm not a fan of the title.  "Decomposing MI" suggests you are making manipulations to the standard MI to ease estimation or focus on different things, rather than introducing additional views and then introducing separate bounds for each (the approach isn't even a true decomposition as its a bound on the original MI, not an equality).


*Approximations*


As previously mentioned, one of my biggest misgivings is with the approximations employed to try and draw samples from p(y|x'').  This is at the core of the problem and arguably why most papers have steered clear of dealing with conditional MI directly.

Starting with the variational approach, I have two issues:
- Estimating p(y|x'') in this way is extremely difficult and is almost like learning a decoder in that we have to learn how to generate full data examples.  Though this is far from fatal in the context of the overall approach, it feels like if we are able to learn a good approximation for p(y|x''), we have already solved much of the challenge contrastive learning was trying to address in the first place.
- More immediately worryingly, the resulting objective introduced (Eq 10) does not appear to guarantee that an effective contrastive learning scheme will be learned as there are unwanted ways this objective can be optimized.  

To be more precise about the latter point, the paper claims about Eq 10

"This suggests that using to sample contrastive sets retains the properties of being a lower bound on the conditional MI."

This is a strange phrase and not properly quantified: my interpretation of Eq (10) would be that the KL term being positive means using a variational approximation does not produce a valid lower bound in any useful sense.  In particular, because \tau appears in both terms, it seems perfectly possible for there to be optima of Eq (10) that do not correspond to \tau(y|x'') = p(y|x'') and consequently critics E that do not behave in the desired manner.  Essentially, it seems like the approach, at least in principle, can learn a tau that allows for easy discrimination to the contrastive samples rather than which represents the actual distribution; the first term could potentially just dominate the KL divergence (such a phenomenon is well documented in VAEs) and avoid learning anything useful.

This is not the end of the world if it does actually train properly in practice, but I think this needs to be directly demonstrated and the potential pitfalls that are there discussed.

On the other hand, with the "importance sampling" approach, there are many large and not fully justified jumps being made such that the final objective used (i.e. Eq 11) is more of an importance sampling inspired heuristic than any principled importance sampling estimate.  There are a lot of approximations going on here, from the weights themselves (they are only valid weights if E=E* and its not clear they will give anything particularly sensible when E\neq E*)  to the rather unjustified jump from these to Eq 11.  In particular, deriving Eq 11 from the weights somewhat boils down to making the assumption E[1/X] = 1/E[X] which is typically a poor approximation (though interestingly might become exact in the limit K->\inf). In short, I would suggest that the approach Eq (11) requires far more careful justification for it to be convincing that this is an approach one would a priori expect to behave in the intended manner.  Some sort of ablation study into how reasonable the approximation it provides is would also be extremely helpful (this could also be done for the variational approach).


*Other Specific Comments *


1. It would be good to explain what you mean by "hard" contrastive samples in the intro.

2. The sentence 

"We note that the bound (Oord et al., 2018) corresponds InfoNCE to a particular choice for the variational distribution q."

is not quite true: the InfoNCE bound can be viewed as a lower bound on Eq (2) (not Eq (2) itself) for a particular choice of q as it still requires an extra invocation of Jensen's inequality.


*Questions etc *


a. Is a stop gradient is used with the w_k to stop E from being indirectly trained through these?

b. I would like to see the results of using the variational approach with the vision problem and the importance sampling approach with the NLP problem, or at the very least a justification for why this hasn't been done.  Without this the results feel a little cherry-picked.

c. What is the performance you get when setting lambda=0?  This was the originally proposed method and, presumably, experiments were run on this so it is important to include them in the results, both so we can see how important the extra heuristic it conveys actually is, and because it relates to the statistical interpretation of the results (tuning lambda reduces the significance of the results as its an extra degree of freedom the baselines don't have).  Moreover, how was lambda actually chosen?  This seems to be missing from the paper.

d. The experimental results are pretty meaningless without any sort of error bars.  The gains are quite small and it impossible to know at present whether they are the result of pure chance, or actual statistically significant improvements.  I would like to see repeat runs and the variability in the performance across these.

e. Are the same adjustments used for the dialogue experiment as for the vision experiment?  In general, section 4.3 is quite hard to follow and seems to lack complete details for the exact setup used for the suggested approach (section 4.2 still has a few things missing as well like setting lambda).

f. What is done with the bootstrap confidence intervals?  This was difficult to understand from the shortness of the explanation, while it is rather unsatisfactory to just quote that the results were significant at the 5% level rather than actually providing the variations, p values, and exact details of the test being done.
review_title: Unconvinced by approximations and experiments

Meta-review: The authors propose a learning approach based on mutual information maximization. By considering a view x, and two subviews, x’ and x’’, the authors provide a bound on MI by combining two InfoNCE-like bounds on I(x’’; y) and I(x’; y | x’’). The authors show that optimising this (approximate) bound leads to improvements in several tasks covering NLP and vision.

This paper is aiming to address a significant problem for the ICLR community and provide a novel solution. The manuscript is well written and the main idea is clear. The reviewers appreciated the fact that the experimental setup covers both vision and NLP. On the negative side, the reviewers raised several major issues, both with the presented theory and the experimental setup. From the theoretical point of view, the approach hinges on a good  approximation for p(y|x''), which could arguably be as hard as the original problem. The author's response is definitely a step in the right direction, but the changes to the original manuscript are quite substantial and there is no time for a thorough validation of the updated claims. I will hence recommend rejection and strongly suggest that the authors incorporate the reviewers’ feedback and submit the manuscript to a future venue.
Decision: Reject