Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies a quasi-Bayesian approach to instrumental variable (IV) regression, based on using an empirical estimate of the ‘likelihood’ to obtain a Gibbs posterior. Such an approach has been studied in earlier literature due to the difficulty of incorporating moment conditions in IV via a likelihood. The paper pursues a Gaussian process approach and proves some theory, namely that the resulting posterior puts most of its mass on functions (almost) satisfying the moment constraint. The paper provides a computation approach based on the randomized prior trick and illustrates the method on various simulated datasets.
main_review: I think the paper is interesting and well written in parts (though a bit dense at times), and features a nice mixture of theory and methodology. The approach of using a quasi-likelihood is not new, and the paper’s main contribution in this direction is to develop a choice of quasi-likelihood for which more efficient computation can be performed. This is a reasonably novel idea, cleverly combining several different existing ideas in the instrumental variable (IV) setting. I work on Bayesian methods rather than IV methods, so it was not entirely clear to me what was previously known here. I also comment more on the theory since I am more familiar with this.

A main contribution is to prove some theory, namely (C1) and (C2) in the paper. (C1) was fine, but (C2), was not clear to me either mathematically or in words and needs to be looked at. The proofs follow from tools in Bayesian nonparametrics as have been applied in this setting by Kato (Annals of Statistics 2013). Saying that, his proofs are very technical and applying them to new settings requires effort, so I think the paper has a suitable technical level for this venue.

The method is illustrated in simulations where it shows promising empirical performance. The simulations are broadly fine, but do have a few issues I think could be improved.

Overall, I think the paper is interesting and relevant. If the theoretical results can be satisfactorily cleared up (especially Proposition 3.1) and better explained, this will make a solid contribution.

Main comments:
1.	Theorem 3.1 and its discussion need to be clarified. What does $\mathbb{E}^2$ mean? If I understand correctly, this result says that the posterior concentrates on functions for which the moment restriction (1) ‘almost holds’. If you were Bayesian, you would try to put a prior that directly enforces this condition rather than hope your posterior concentrates on it. Why can such a restriction not be enforced with quasi-posterior probability one?

2.	Theorem 3.1: what values of $\tau_n$ are provided by your theory for concrete examples? E.g. Matern prior and Holder/Sobolev smooth truth. The results are currently quite abstract and not interpretable.
3.	Proposition 3.1:

(a)	Why is $f \sim \Pi$ from the prior? Where is this used? The conclusion doesn’t involve the prior and the prior is only specified as a GP, so does this result hold for any function $f$ drawn from any GP? 

(b)	$\hat{E}_n$ is an operator taking $x$ valued functions to $z$ valued functions, so it’s not at all clear what the LHS means here. 

(c)	A general notational improvement throughout the paper would be to clarify what you are taking the expectation over. e.g. $\mathbb{E}_X$ for taking the expectation of $X$, etc. 

(d)	Make precise under what probability distribution $\to^p$ is.

(e)	This looks like a kind of ‘uniform law of large numbers’ since the $\hat{E}_n$ is random and depends on the same random variables as the sum. Empirical process theory tells us that such a result holds if the function class is Glivenko-Cantelli but not generally otherwise. However, I don’t see such a condition. Can you please clarify (maybe it’s implicit from $f \sim \Pi$)?

(f)	Remark 3.3 and the interpretation of Proposition 3.1 don’t make sense to me. Also, if you want to relate this to the log-likelihood, then wouldn’t it make sense to study the limit uniformly over $f$ rather than pointwise? Please carefully explain what is going on.

4.	Proposition 4.2:  Can you discuss this result a bit more. If I understand it correctly, this say the algorithm won’t get stuck at a local minimum (likely due to the stochastic part), but it doesn’t give any quantitative bounds on convergence, correct?
5.	Simulations:

(a)	I think it would be clearer to present your data in table form rather than as boxplots (one picture is fine). 

(b)	What are you plotting credible intervals of? The goal is to estimate the whole function $f$. Are you doing it at a point?

(c)	You should provide the average credible interval lengths, otherwise the coverage is not informative.

(d)	10 repetitions of the data is small and you should do more if possible. If this is computationally prohibitive, please say so and give some indication of the run time. 


Minor points:
-	p3 Maybe define the tensor product $\otimes$ 
-	p3 Could you please explain the equation on l92 a bit better
-	Assumption 3.2 One typically needs the assumption $m\tau_m^2 \to \infty$ for these kind of results. Has this been forgotten?
-	Assumption 3.2: mention that this is a standard condition in the GP literature. You should really cite van der Vaart & van Zanten (Annals of Statistics 2008) here since they first applied this theory to GP’s.
-	Theorem 3.1 I guess $\tau_{\sqrt{n}}$ should be $\tau_n$ everywhere (it’s indexed by an integer)?
- Theorem 3.1 is not a contraction rate, which involves the posterior putting most of its mass about the ‘true’ function, which is not what you are studying here. 
- THeorem 3.1: specify what assumptions you have put on the ‘true’ $f$ generating the data (taken over the expectation $\mathbb{E}_{D_n}$. Do you assume the condition (1) holds in reality?
- Proposition 4.2: $S_m$, $\mu_m$ should be $S$, $\mu$?
-	P19 What is $err_{n,f}$?

############################################ 

Note: scored changed from 6 to 7 after author response.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper concerns the uncertain quantification for instrumental variable (IV) regression that it is an approach for estimating causal effect from confounded observational data and, proposes a scalable quasi-Bayesian procedure. The procedure was built upon the kernelized IV models (Muandet et al (2020)). The contribution is forming a quasi-posterior by employing a Gaussian process prior and constructing a kernel conditional expectation estimator using a randomized prior trick. Theoretical properties and simulation studies are provided.

main_review: (a) This work is stretched out my area. With my limited search, I haven’t seen IV regressions using a kernelized IV model employing a Gaussian process prior. I feel the use of “quasi-Bayesian” is slightly abusing as only a prediction is inferenced according to the Bayes rule. 

(b) I found the paper was little difficult to follow. Some notations like C^{-1}_{zz,\nu}  or S^*_z in page 3 were not defined although they were key terms in their expressions. Although authors described steps of their approach, an algorithm (or a list of steps) will be more effective, and it is easier to see what estimates implied/steps were involved.

(c) There is some unclearness about the approach.

(c-i) It is not clear how the objective function, L(f) (Eqn 4) fits in the posterior (8) and how K-zz and other parameters (kernel specific,
scale parameters) are obtained. 

(c-ii) A randomize prior trick approximation will be sensitive to m and, how would you choose m and what do we expect to see when m is too small/large? How it is sensitive to z-distribution? 

(d) In the simulation study, authors compared their method with existing estimators and, showed the new method was competitive. Authors also honestly showed some kernel choice sensitivity in Appendix. In the main text, this sensitivity was not shown, and I thought including this in the main paper would be very valuable to provide correct strength and weakness of their method. Moreover, some discussion on the kernel choice (if there is any existing guidelines) would be good.

(e) The paper proposed a quasi-Bayesian procedure for IV regression and showed that it can be an efficient estimator when a kernel is carefully chosen. I didn’t find any false in their approach and empirical evidence of their claim is given. Weakness will be clarification and brief description of a randomized prior trick without investigation of its properties and impacts on the resulting inference.

limitations_and_societal_impact: Authors showed some sensitivities associated with the kernel choice and this will be one limitation.
I didn't see when their method will be poor in the text.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce a scalable quasi-Bayesian method for nonparametric instrumental variable (IV) regression that provides uncertainty quantification for the IV models. They base their approach on kernelized IV models and define a quasi-posterior as the Radon-Nikodym derivative of a generalized method of moments (GMM) objective function with respect to the standard Gaussian process prior. They go on to show that the quasi-posterior exhibits satisfies certain posterior contraction criteria that lead to asymptotically sound results, and prove a conservative posterior contraction rate under a set of typical assumptions. They then show how to achieve scalable inference for the quasi-posterior using a previously proposed “randomized prior trick” for Gaussian process regression, and provide a proof for the convergence of the inference algorithm. Finally, the authors showcase the performance of their method in a series of simulated experiments of varying complexity, by comparing it against other approaches that provide uncertainty quantification (Bayesian, bootstrap).
main_review: *Originality*: The present work is, to the best of my knowledge, novel. To arrive at this contribution, the authors have combined and built upon previously published results on kernelized IV regression and quasi-Bayesian analysis. The authors carefully put their method in the context of related work and adequately describe how the contribution differs.

*Quality*: The submission seems to be technically sound, and the theoretical claims are reasonably supported in the main paper, with the majority of proofs deferred to the supplement. The numerical experiments are also appropriate, with the authors providing at least one challenging example for the task under hand. However, one aspect that is missing in the numerical experiments’ evaluation is the computational cost. The authors do discuss the time and computational complexity of their method in more detail in the supplement, but since scalability is one of the main selling-points of the method, I would have liked to have seen a comparison in terms of computational cost and/or run time.

*Clarity*: The paper is well-organized for the most part and is relatively error-free. Many technical details are left for the supplement, but the key ideas all seem to be present.

*Significance*: Judging by the numerical results, the authors’ submission successfully addresses a very challenging task and equals, or improves upon, previous results in terms of estimation accuracy and uncertainty quantification. However, it seems to me that the major advantage of this quasi-Bayesian approach for nonparametric IV regression could lie in its scalability, as the use of such flexible methods is often computationally prohibitive. I think the authors could have stressed this point more to underline the significance of their work.

*Minor comments*: 
-	Line 13: “causal effect” -> “causal effects”
-	Line 38: It seems that a verbal phrase like “are” or “need be” is missing before “present”.
-	Line 90: $\bar{\nu}$ is first used here in defining $ \hat{C}_{zz, \bar{\nu}}^{-1}$, but I think both have yet to be defined.
-	Equation (11): $\nu$ can be inferred to be equal to $n \bar{\nu}$, but this does not seem to be explicitly mentioned in the paper.
-	Proposition 3.1: It is not mentioned how the log quasi-likelihood is “scaled”. Is it the same scaling as for the evidence in line 115?
-	Figure 1: The right-most plot (CI coverage for $\alpha = 0.5$) is hard to read. The results for some algorithms are barely visible or not visible at all. Perhaps leaving more margin below 0.00 and above 1.00 on the y-axis could help.
-	Lines 275-277: I’m not sure I agree with the conclusion that “quasi-Bayesian inference provides the most reliable uncertainty estimates” by looking at Figure 1. For $\alpha=0.05$, the results look promising, but for $\alpha = 0.5$, the CI coverage appears to be 100%, which I’d think is a rather poor uncertainty estimate.
-	Line 288: The first paragraph in Section 6.2 does not have numbered lines. This does not affect the review, but perhaps there is a formatting issue that the authors need to be made aware of.
-	Lines 406-409: Single-author references [30] and [31] have erroneous “et al.”.

limitations_and_societal_impact: The authors have discussed the potential impact of their work in a separate section, where they explain how uncertainty quantification can help in assessing the degree of error in the estimate, but warn the practitioner that its validity ultimately hinges on the chosen model.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a scalable quasi-Bayesian inference for instrumental variable regression. Specifically, the paper builds a quasi-posterior on the recent quadratic kernel IV loss function. For scalability, the paper extends the randomized prior trick based on the recent dual formulation of the quadratic loss. The approach has a theoretical guarantee that incorrect solutions are excluded while valid kept. Moreover, the convergence analysis for the randomized prior trick with the stochastic gradient descent-ascent is present. Experiments validate the approach.
main_review: 1. Originality

The quasi-posterior Bayesian approach employs the recent development of non-linear instrumental variable regression and the paper extends the randomized prior trick for scalability of the model. However, a recent method [Zhang, 2020] similarly connects the quadratic kernel loss with the Gaussian process and this work is expected to clarify the relation to it.

2. Quality

The approach is technically sound. The paper presents assumptions for theoretical analyses and detailed development for the approach. The theoretical analyses are sound and provides performance guarantees. The experiments assess the scalability and the modeling ability of the approach.

3. Clarity

Yes, the paper is clearly written.

4. Significance

The paper presents important results and will be interesting for researchers in the Bayesian community as well as the causality inference community.

Zhang, R., Imaizumi, M., Schölkopf, B., & Muandet, K. (2020). Maximum moment restriction for instrumental variable regression. arXiv preprint arXiv:2010.07684.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The paper investigates the problem of instrumental variable (IV) regression and proposes a novel quasi-Bayesian approach to it. The IV regression is an important problem in economics, social science, and epidemiology which has recently gained traction in the machine learning community as well. However, as the authors mentioned, we still lack satisfactory way of qualifying the uncertainty in this setting and Bayesian modelling is non-trivial in this case because of the inability to specify full likelihood. Due to the non-standard setting of this work, it thus provides an important step towards a full probabilistic treatment of IV regression and will certainly inspire future works in the Bayesian ML community. 

There is a consensus among the expert reviewers that the paper makes a sold technical and methodological contribution. All reviewers also feel that their concerns have been addressed adequately. Upon reading the paper, I do agree with Reviewer hAiR and Reviewer km2d that the paper is quite difficult for the non-expert readers to follow, especially because of the technical nature of this work. I encourage the authors to improve the writing and to make it more accessible to the broader audience. This is a minor point, but can increase the impact of this paper.

I thus recommend this paper for publication at NeurIPS2021.

**Remark**: I feel that the related work section can also be improved. Here are some references that might helpful in improving this section:

- [1] R. Zhang, M. Imaizumi, B. Schölkopf, and K. Muandet, “Maximum Moment Restriction for Instrumental Variable Regression,” arXiv:2010.07684 [cs], Oct. 2020, arXiv: 2010.07684.

- [2] K. Muandet, W. Jitkrittum, J. Kubler, "Kernel Conditional Moment Test via Maximum Moment Restriction", UAI2020.

- [3] A. Mastouri, Y. Zhu, L. Gultchin, A. Korba, R. Silva, M. J. Kusner, A. Gretton, K. Muandet: "Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction", ICML2021.

- [4] R. Singh, M. Sahani, and A. Gretton, “Kernel Instrumental Variable Regression”, NeurIPS2019.

For relevant discussions, see, e.g., related work section in [4], Section 6 in [1] and Section 5 & Appendix B1 in [2]. Section 3.3 in [3] is definitely relevant to the second paragraph L243-L250 of this submission.


Decision: Accept (Poster)