Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors combine adaptive importance sampling (AIS) with control variates (CV) leading to a novel Monte Carlo estimator which they call AISCV. Non-asymptotic bounds were developed for the novel estimator and the experiments section demonstrated the superiority of AISCV versus a standard AIS estimate. 
strengths_and_weaknesses: # Strengths
I can not express how cool the proposed approach is. While standard CV estimators require $\beta$ to be recomputed for every target function $g$, AISCV sidesteps this completely! Instead---just like AIS---quadrature weights are computed using the particles and their corresponding importance weights, where, crucially, the quadrature weights **are not a function of** $g$! Thus, this process is done once and can be used for any $g$. Moreover, the bounds on the AISCV estimate are very impressive as well!

# Weaknesses

While the experiments section clearly demonstrated the superiority of AISCV, I wish a more practical experiment was done. AISCV was motivated by wanting to compute posterior expectations. Thus, it was disappointing that the only example done was Bayesian linear regression where the posterior is available analytically. Instead, it would have been more compelling if the AISCV estimate was compared against Bayesian linear regression, which can be sampled by using HMC for instance. 

questions: Below I will list general questions/comments

1. In line 126, I think there is a typo. The paper says that $\mathbb{E}_f[g - \beta^\top h] = 0$ but it should be $\mathbb{E}_f[g - \beta^\top h] = \mathbb{E}_f[g]$.
limitations: There is no section in the paper describing the limitations of the approach. While presumably the limitations of AISCV are similar to AIS and CV, it would be beneficial to state them in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel method for approximating intractable integrals that combines adaptive importance sampling and control variates. The authors prove a probabilistic bound on the error of their proposed estimator and demonstrate its superiority to the simple AIS estimator. 
strengths_and_weaknesses: Overall, I found this paper an enjoyable read although I confess, many of the technical details surrounding the theoretical results escaped me. 

1. In terms of originality, the primary proposed idea is not particularly inspired, in that it combines two commonly-used, existing methods. 
2. The level of analysis, both theoretical and empirical, is compelling. 
3. The paper is extremely well written and clear. 
4. The work seems likely to be significant as the task of estimating integrals is commonplace and the proposed method appears sufficiently general so as to be widely applicable. 
questions: Could it ever be the case that poorly (or even adversarially) constructed control variates result in unboundedly worse performance than AIS? Or is Assumption 2 a sufficient safeguard against this kind of behavior? 

Can the authors speak to why their AISCV methods in Figure 2 demonstrate a higher variance than AIS in this setting? It seems odd to me that control variants, a method originally proposed as a variance reduction technique, could cause this behavior. 
limitations: Related to my questions above, it seems that the use of control variates introduces a whole host of design choices that have the potential to result in poor performance. While the theoretical results speak to what requirements the control variates need to satisfy, in practice it seems like this could result in undesirable behavior if the requirements are not met.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present AISCV estimator that combined Adaptive Importance Sampling (AIS) and control variates (CV). The paper presents a theoretical proof of its convergence rates and presents practical considerations that are really helpful. 
strengths_and_weaknesses: The theoretical side of this paper is extremely well executed. Everything is carefully described and seems technically sound (I didn't carefully check details in the proof of Theorem 1, but seems valid based on what we know from AIS and CV. I did check the outline of the proof). 

I would like the authors to reflect on if AISCV is just a combination of AIS and CV, or if they think it is greater than "the sum of its parts".  

Could you Figure 1(a) and 1(b) have the same cutoff on the y-axis for easier visual comparison between input dimensions. Same where appropriate in Figure 2.

This paper has enough quality to be accepted to NeurIPS. The drawdown is, I think, that the short format of NeurIPS is making this paper poorer than it could be in a longer format. The experimental section is likely a bit sub-par to many other papers at this venue. Yet, the authors have more experiments in the appendix that I did not read carefully.

questions: Would it be worth having a baseline of just CV estimators? It is interesting to see how big the reduction is from CV -> AISCV. Especially, how would it compare on Figure 3. My thinking is the "virtually zero" variance is a CV effect rather than an AISCV effect. Is this correct?
limitations: The authors did not address this issue. I do not see any reason why the authors would need to address it either, the paper does not touch any sensitive area of society.
review_title: null

Meta-review: metareview: This paper proposes a novel method to perform Monte Carlo integration combining control variates and annealed importance sampling. All reviewers agreed the algorithm was of interest, the theoretical evidence was strong, and the experimental results were sufficiently convincing, so there was a consensus on acceptance.

(As a minor aside, I would encourage the authors to consider the font sizes in their figures.)
confidence: Certain
Decision: Accept