Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an instrument learning approach for non-linear instrumental variable regression with high dimensional instruments.
State-of-the-art IV estimators with fast convergence rates generally rely on computationally costly minimax procedures. However, assuming that the true function of interest lies in an RKHS, the kernel versions of these estimators often allow for closed form solutions at the cost of using less flexible RKHS models as adversarial/instrument functions compared to e.g. neural networks or random forests. The authors combine the simplicity of the kernel versions of these estimators with the predictive power of modern machine learning models by first learning a kernel whose features are represented by arbitrary ML models and then using this kernel with the kernel-based closed form solution of the SOTA minimax approaches. The paper provides theoretical guarantees and fast convergence rates similar to the ones of the minimax approaches.
strengths_and_weaknesses: As a disclaimer, I was assigned to this paper as an additional reviewer very shortly before the start of the rebuttal period. In the interest of not further delaying the process I was only able to spend a short amount of time on preparing the review and therefore might have overlooked some important aspects or errors. I have not checked the proofs and my review is based on the assumption that the theorems are correct.


Strengths
- The problem the paper solves is a very interesting and relevant one and to the best of my knowledge the approach is new
- While there is a growing literature on addressing conditional moment restrictions via minimax formulations, approaches like the proposed one that avoid the rather involved saddle-point optimization problems seem to be very useful and well-motivated
- While the final estimator builds on the kernel-kernel version of the minimax formulation of [19], the kernel over the instrumental variables is formed from arbitrary ML models and thus the proposed approach can be combined with (future) SOTA ML models tailored to the problem at hand
- Large parts of the paper are well and clearly written
- The appendix provides a lot of useful additional information 


Weaknesses / Possible improvements

- The clarity of presentation could be improved, from reading the text it was not obvious to me that the proposed algorithm is the one described in Algorithm 1
- Without being familiar with the work in reference [19] and their closed form solution for the kernel version, it might be difficult to understand what problem the paper tries to solve and how the final estimator is obtained. The explanation in Appendix F focusses on the more involved estimator based on [26]. I think it would be helful also to discuss the simpler implementation of [19].
- I did not understand why regressing $f(x)$ on $z$ provides denoised samples from $\mathcal{GP}(0,k_z)$, however this is a crucial part of the method, so I think it should be explained more clearly
- Connected to the previous point, it is also not clear to me why this procedure focusses on the most informative latent features
- The paper has a lot of content and might feel a bit too dense. To some extent the methods based on [19] and [26] seem to be independent and the paper might be easier to understand if one of the methods were moved to the appendix (or to a second paper) but this impression might be owed to the limited time I have spent on the paper
- The experiments show that the method performs sometimes better and sometimes on par with the minimax approach of [19]. I think it would be insightful to report e.g. the computation time, to see if the supposedly simpler computations of the proposed method actually result in faster estimation
questions: I appreciated if the authors clarified the following points for me:
- Why does regressing $f(x)$ on $z$ provide denoised samples from $\mathcal{GP}(0,k_z)$?
- Why does this procedure focus on the informative latent features?
- How restrictive is the assumption of $f_0$ being an RKHS function in practice?

limitations: I do not see any immediate potential for any negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a Gaussian-process-based IV estimation procedure. In a nonparametric IV regression (NPIV) problem, y = f(x) + e, E[e|z]=0, the traditional approach to consists of two stages: estimate x' = E[x|z] (stage 1), then regression y on x' (stage 2). The kernel IV paper [27] proposed to used two pre-specified RKHS for each stage. Here the paper proposed to estimate stage 1 with a procedure based on sampling from the Gaussian process. Concretely, an RKHS is learned for stage 1 instead of being pre-specified. The advantage of such a procedure is to avoid minimax formulation of the problem and provide an inferential procedure due to the Bayesian nature of the method.
strengths_and_weaknesses: Strength

The paper provides an extensive overview of related works and rigorous theoretical derivation. The work is original; inferential methods for the nonparametric IV problem are new in the related literature as far as I know. The proposed method is neat and enjoys good theoretical guarantees. 

Weakness

The presentation of the paper is a bit dense and could be improved by including an overview of the results in the paper.

questions: Minor technical questions:

1. In Assumption 2.2, does a smaller b correspond to a smaller (smoother) function space or the opposite?

2. In Line 5 of Algorithm 1, is the estimated $\tilde k_z$ an estimate for the latent space kernel $\bar k_z$ or $k_z$?
limitations: 
1. Citations of a few closely related works are outdated, e.g., 5, 20, 27. These works are published in neurips or other conferences and yet only arxiv versions of these works are cited.

2. Overall the presentation of the paper is very dense. It could improve the paper presentation if the authors provide a guideline for the theorems and examples presented in the revised version of the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed an instrumental variable regression algorithm, which authors claimed to allow a faster convergence and the adaptivity to the dimensionality of informative latent features. This paper also presented extensive theoretical analysis to analyze the algorithm and proposed to bridge the notable gap in the literature.
strengths_and_weaknesses: The strength of this paper is on proposing the simple yet faster converging algorithm, and it is well-supported by theoretical analysis and numerical experiments. Authors provided an extensive literature review and paved way of a comprehensive understanding towards literature for audience even not familiar with the specific sub-field. The presentation of this paper is very well organized with detailed explanations in steps on the motivation and design of the algorithm. It is also notable that the appendix is very well organized with detailed supplementary materials covering both theoretical analysis and experiments. The weakness is not obvious to me. But, I'd like to point out that I am not an expert in this sub-field. The only major concern that I have is that there might lack significant novelty of this work comparing with existing literature.
questions: I have some specific questions on the experiments:
1. Why would you choose a randomly initialized NN as one of the baseline? Could you be more specific on "linear"? 
2. Have you compared the time consumption of the proposed algorithm versus the baselines of your choice in different experiments? I am not sure if I see that being addressed in the main paper and appendix.
limitations: Authors adequately presented the assumptions of the algorithm and all related analysis, and I do not see potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed to perform the first-stage instrument learning via an adaptive regression on random draws from a GP prior. And they use kernelized IV  methods to estimate the second stage casual effect functions. They show that the algorithm has faster convergence rates and avoids expensive minimax optimization procedures. For kernel first-stage models, the algorithm can also be applied to evaluate quasi-posterior and thus provide uncertainty quantification.
strengths_and_weaknesses: The presentation of the paper is good, with clear explanations of the intuitions behind the conditions and assumptions. The algorithm itself is easy to implement and efficiently distills informative structures of the instruments. The first-stage can accommodate flexible ML methods. It also enables fast quasi-Bayesian uncertainty quantification with improve flexibility. They also provide extensive discussion on the connection to related work.

The content of the paper is very compact due to page limits. One may wish to see more on the simulation study in the main manuscript. It would be interesting see low-dimensional sparsity structure design.
questions: I didn’t check the technical details. But should the condition of $n_1$ in line 268 be $n_1^{-(b+2p)/(b+2p+1)}$?

The results of the paper hinge on the assumption that $f_0$ can be characterized by a RKHS $\mathcal H$. Can the authors elaborate more on how practical is this assumption and what are the consequences if the model is mis-specified or the assumption is violated?

limitations: The authors have included discussions on limitations and future directions in their paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Fast and provably adaptive instrument learning, through kernel learning with black-box ML models.
strengths_and_weaknesses: The paper applies newly developed "orthogonal ML" techniques.

However, the contribution seems rather incremental.
questions: Can you elaborate how this work is not just simply applying the "orthogonal ML" framework?
limitations: NA
review_title: null

Meta-review: metareview: An instrumental variable (IV) regression for high-dimensional data is a challenging problem as it involves either a cumbersome two-stage method or a complex minimax procedure. Kernel methods recently arise as promising tools that enable the solutions to be computed in closed form. Unfortunately, the convergence rate of the vanilla estimators is sub-optimal as it depends on the choice of kernel functions on the instruments (i.e., similar to how the second-stage estimation depends on the first-stage estimation). This paper contributes by showing that the convergence rate can be improved if the kernels on instruments are learned from data. The expert reviewers agree that the paper provides a novel approach to learning the instrument kernels, rigorous theoretical analyses, and convincing empirical results. The major limitation, which can be improved in future work, is that most of the theoretical analyses rely on the assumption that the true structural function $f_0$ lies in a reproducing kernel Hilbert space (RKHS). 
confidence: Certain
Decision: Accept