Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new bound of mutual information (MI) based on Fenchel-Legendre Optimization (FLO).  The bound has a number of beneficial properties such as tightness (it can approach the true MI), the empirical estimator is unbiased, and it can be estimated using samples from the model but does not require evaluation of model probabilities (i.e. it can be used for implicit likelihood models).  The paper discusses properties of the FLO estimator in relation to existing variational estimators and those based on the conjugate function.  Finally , the paper provides theoretical convergence guarantees along with empirical analysis (and ablation study) on several interesting models that are relevant in the literature on MI estimation and sequential experiment design. 
strengths_and_weaknesses: **STRENGTHS**
The paper is very well-written, the problem is well-motived, and the topic is timely and important.  The authors provide a solid foundation for the reader, along with all necessary context.  The paper is a good blend of theory and practical application.  The empirical analysis provides comparison to most relevant MI estimators and sufficiently supports the stated claims.  

One strength that receives less attention due to the focus on tightness (see critique below) is that the empirical estimator lacks finite sample bias.  Much of the literature on MI estimation focuses on properties of the analytic approximations / bounds, when in reality these bounds cannot be calculated exactly and require further estimation.  In most cases there do not exist unbiased estimators, for example in the UBA bound, and estimators are only asymptotically unbiased / consistent.  By contrast, FLO admits a straightforward empirical estimator with zero finite sample bias.  Discussion of this is provided in Zheng et al. (2018) and Rainforth et al. (2018) for the Barber and Agakov bound.

**WEAKNESSES**
The empirical analysis in the main text is fairly superficial.  Results on the SIR model are particularly cursory, and are largely qualitative.  For example, the caption of Figure 6 states that FLO designs are somehow more useful because they are sampled more densely around the infection spike.  Yet the authors provide no comparison and the comment is debatable from visual inspection alone.  Additionally, the estimates provided in Figure 3 are mostly qualitative and do not seem to show a strong benefit of FLO compared to NWJ for example.

The paper leans heavily on tightness of the FLO estimator, yet this property does not hold in practice.  The discussion under Prop. 2.3 shows that tightness is achieved only under the optimal critic $g_{*}(x,y)$ which requires knowledge of the true posterior $p(x\mid y)$.  In addition, the tightness property is not unique to FLO, despite what the authors suggest in Table 1.  Many of the provided estimators are also asymptotically tight or tight given knowledge of the true posterior.  For example, most variational MI bounds become tight when the variational distribution $q(x \mid y) = p(x \mid y)$ almost everywhere.  By Prop. 2.2 the UBA bound is also tight given the same optimal critic, and indeed FLO is in general a looser bound than UBA by this proposition.   Similar properties hold for DV and NWJ estimators as well as InfoNCE (i.e. Proposition 2.1).

Some detailed comments below:
* It is not clear what point the authors intend to convey with the section on unnormalized statistical modeling and its connection to MI estimation.  Essentially any posterior inference problem contends with the lack of an analytic partition function, and so it is not particular to this setting.
* The statement in Prop. 2.3 is a bit terse.  At the very least authors should state that the estimator is tight at $(g_{*}, u_{g_{*}})$. 
* Line 266 : Change “direct out attention” to “direct our attention”

**REFERENCES**

Rainforth, Tom, et al. "On nesting monte carlo estimators." International Conference on Machine Learning. PMLR, 2018.

Zheng, Sue, Jason Pacheco, and John Fisher. "A robust approach to sequential information theoretic planning." International Conference on Machine Learning. PMLR, 2018.

questions: * Proposition 2.2 states that FLO is a lower bound on UBA with the same critic.  If so, what is the benefit of FLO over UBA (the unbiased empirical estimator is obviously one, but are there others)?  Is this bound true for every $u_\phi$ or only for an optimal one?

* Several estimators (e.g. DV) are based on a similar conjugate duality construction to FLO.  Yet, the authors do not discuss the relationship between these estimators in great detail.  How do these estimators compare, e.g. for similar choices of the surrogate function (i.e. the critic in the FLO case)?


limitations: The authors clearly state limitations.  This reviewer does not see obvious negative societal impact of the work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces FLO, a novel contrastive estimator for mutual information. FLO is inspired by a reexamined connection between mutual information and unnormalised statistical modelling, which yields a unified theoretical basis for several variational-based MI estimators. FLO is then proven to be tight, and to converge under SGD. Finally, several experiments confirm FLO's superior performance compared to previous estimators.
strengths_and_weaknesses: The paper is *very* well written, and as a consequence a pleasure to read, even for non-experts in the field of mutual estimation. Connections are explained clearly and the new insights and methods are well-motivated. The 'zoo' of estimators in Figure 1 is much appreciated.

I can find precious little weaknesses in the paper (although I'm by no means an expert in this particular field). A few things:
- some running time experiments contrasting the performance of the various estimators would be good, to verify the tractability claims
- in the experiments, some explanation of how significant the gains are would be good, given that some of the tasks are not familiar to all readers
- all told, given the amount of material in the paper (and consequently how much of it is left in the appendix), I wonder if a journal version would not do it better justice.
questions: My main suggestions for improvements would go for the experimental section:
- running time comparisons
- giving more context as to levels of performance for various tasks.
limitations: Limitations are addressed adequately
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces new type of contrastive mutual information estimator, which is tight and provides theoretical convergence. As the variance of previous estimator was big trouble for the accurate MI estimation, it provides new optimization technique with fenchel-legendre optimization. The relationship with the existing mutual information estimator has been nicely provided, which makes it easier to evaluate the introduced estimator with the fair manner.
strengths_and_weaknesses: Strengths:

Theoretical analyses : it provides the tightness and boundedness for the given estimator with nice derivation. It was easy to follow.
Experiments : FLO significantly outperformed its variational counterparts in the more challenging 263 high-MI regime, which implies the robustness from the noisy environments.

Weaknesses : Experiments for additional data-efficient learning is not provided. The author referred that efficient implementation for FLO can be provided, which makes it easier to apply FLO on data-efficient learning. However, it was not adequately managed in the manuscripts.
questions: Q1. how this framework can be applied on the supervised contrastive learning? Additional works from supervised contrastive learning stated that it is connected with conditional mutual information maximization. I am curious that the mutual information estimation or maximization given the label can be conducted with FLO.

limitations: Stated in weakness session.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a new mutual information estimator based on Fenchel-Legendre optimization (FLO). The paper describes existing related techniques, and then presents the derivation of the FLO estimator via convex duality. The FLO estimate is established as a lower bound on the true mutual information value, and it is shown that appropriate choices of the g and u functions in the variational bound retrieves the true value. The authors also provide a convergence to stationarity claim for SGD-based optimization of the FLO objective. Numerical experiments comparing the proposed approach to existing estimators is provided, showing good performance on the considered examples. Applications to Bayesian optimal experiment design and meta-learning are also presented.
strengths_and_weaknesses: The proposed estimation approach is interesting and may very well have merits, but in my opinion, the paper does not provide sufficient concrete evidence to support that. I liked the overview of existing/past techniques, which was comprehensive and informative, and think that the FLO approach was clearly described. Beyond that, however, while there is a considerable amount of discussion and heuristic arguments for why FLO is favorable to alternative methods, these lack rigorous supporting evidence. In various cases in the past, an estimator of mutual information was proposed and argued to drastically improve upon past approaches only for the next paper to provide examples on which it fails. The best approach to avoid such pitfalls is to couple a proposed estimator with formal performance guarantees (e.g., consistency, error bounds, sample complexity, etc.), but the paper comes short in that regard. I also found the paper to be quite heavy in jargon and at times sloppy in its mathematical notation, which are aspects the authors should work to improve on. Specific concerns/comments/questions are provided in the next section. 
questions: * The feasibility claims from Proposition 2.3 and Corollary 2.4 are trivial and have limited utility since the optimal functions depend on the population distribution, which is unknown in practice.  

* The abstract and intro make it sound like a statistical performance analysis for FLO will be provided, but it is nowhere to be found. The short paragraph after Corollary 2.4 is quite laconic and surely doesn't qualify as such. If there is a rigorous statement the authors can provide, it should be clearly formulated, stated, and proven. The current presentation is misleading in this regard.

* The convergence claim from Proposition 2.5 is local and should be framed as such from the get-go. The abstract and introduction make it sound like global guarantees would be provided, which is not the case and hence this presentation is misleading. The 2nd part of that statement that assumes convexity if I_{UBA} surely does not quality as a global guarantee since this convexity in the parameters is rarely available (e.g., when parametrizing the functions by a neural network, as often performed in practice).

* I find the statement in Table 1 that FLO has low bias and moderate variance quite odd. What actual evidence (beyond the said heuristics) did the authors provide to support this bold claim?

* In the `Efficient implementation of FLO' paragraph: jointly modeling u and g by a single neural network makes sense, but doesn't doing so typically require more parameters than approaches based on a single critic? If so, how can the authors say that this incurs no extra modeling cost? 

* The bi-linear critic design proposed in the next paragraph is indeed highly parallelizable, but how expressive is this model? Under what conditions can we expect that class to be able to approximate the optimal u and g functions (that depend on the joint population distribution)? Computational efficiency is important, but it is crucial to understand the tradeoffs involved.

* I found the content of Section 2.3 full of jargon and quite confusing. I am not sure what the goal of this section is. The authors may consider omitting it to make space for a more comprehensive and rigorous treatment of the SGD convergence guarantees.

* It is unclear how FLO helps mitigate challenges associated with large sample sets and high-dimensional data. The text suggests that this is the case on several occasions, but I have seen no results to that effect. In a similar vein, the authors should contrast their approach with the McAllester and Stratos formal limitations paper [44].

* Equation (11) is unclear; in particular, StopGrad is never defined.
limitations: I don't think that limitations of the approach were sufficiently discussed. The paper advocates quite strongly that FLO is superior to existing approaches but provided evidence to that effect is at best partial. As mentioned above, these claims would have been easier to digest if supporting theory and formal guarantees was provided. Clarity on the limitations of the approach and/or when alternative methods are preferable would be appreciated as well.
review_title: null

Meta-review: metareview: All reviewers agree that the paper proposes an interesting and novel bound of mutual information (MI) based on Fenchel-Legendre Optimization. Although some reviewers have some technical concerns at their first reviews, basically those have been resolved by the authors' responses. Thus, although there are some points that should be modified from the current form, I think we can expect the authors modify the paper in the camera-ready by reflecting the discussion. Based on these, I recommend acceptance for this paper.
confidence: Certain
Decision: Accept