Fast Instrument Learning with Faster Rates
Authors: Ziyu Wang
Yuhao Zhou
Jun Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: causal inference
kernel method
gaussian process

Abstract: We investigate nonlinear instrumental variable (IV) regression given high-dimensional instruments. We propose a simple algorithm which combines kernelized IV methods and an arbitrary, adaptive regression algorithm, accessed as a black box. Our algorithm enjoys faster-rate convergence and adapts to the dimensionality of informative latent features, while avoiding an expensive minimax optimization procedure, which has been necessary to establish similar guarantees. It further brings the benefit of flexible machine learning models to quasi-Bayesian uncertainty quantification, likelihood-based model selection, and model averaging. Simulation studies demonstrate the competitive performance of our method.


Fast Instrument Learning with Faster Rates
Ziyu Wang, Yuhao Zhou, Jun Zhu"
Dept. of Comp. Sci. and Tech., BNRist Center, State Key Lab for Intell. Tech. Sys., Institute for AI, Tsinghua- Bosch Joint Center for ML, Tsinghua University {wzy196, yuhaoz. ca> agmail..com .om, dcszj@tsinghua. edu.cn
Abstract
We investigate nonlinear instrumental variable (IV) regression given high- dimensional instruments. We propose simple algorithm which combines ker- nelized IV methods and an arbitrary, adaptive regression algorithm, accessed as black box. Our algorithm enjoys faster-rate convergence and adapts the di- mensionality of informative latent features, while avoiding an expensive minimax optimization procedure, which has been necessary establish similar guarantees. It further brings the benefit of flexible machine learning models to quasi-Bayesian uncertainty quantification, likelihood-basee model selection, and model averaging. Simulation studies demonstrate the competitive performance of our method.
Introduction
Instrumental variable (IV) analysis widely used for causal inference [1-3]. Given confounded observational data, IV analysis identifies the causal effect through the use of instruments. Nonlinear IV regression typically defined by the following conditional moment restrictions (CMRs):
Bly - fo(x) z)) = 0 a.s. [P(d))]
where So is the causal effect funetion of interest, and x, y, z denote the observed treatment, response and instrument, respectively. Similar CMR problems also appear in other applications of causal statistics and machine learning [see, c.g., 4, 5, for examples].
Starting from [6], recent works have demonstrated great promise in applying flexible machine learning (ML) methods IV regression. Modern ML methods are appealing due to their adaptivity to the informative latent structure in data [7]: they may adapt to the low dimensionality of the informative latent features even if the observed input is high- high-dimensional and its signal-to-na -oisee ratio low. Sample complexity gaps have been established in such settings, between deep models based on neural networks (NNs) or trees, and linear models such as fixed-form kernels [8-10]. adaptivity will be highly desirable when the observed instruments are high-dimensional, which is prevalent in applications such as genomics [11], and may also arise from the general desire to use structured data instrument. Following previous work in the parametric setting [12, 13], refer to this problem of learning informative latents in instruments as instrument learning. generalizes the classical problem of instrument selection [14-16].
regression, such
Comparing with standard supervised learning. IV regression is more challenging, due to the need to estimate a conditional expectation operator which defines (I), Consequently, establishing adaptivity guarantees becomes more difficult. While many recent works have demonstrated promising empirical results using deep models, they are often used as heuristics [e.g., 6). or justified with crude slow-rate analyses, which establish convergence rates that saturate at Ω(n Q(n-1/4) [c.g.. 17J. This is in contrast to faster rates which approach n 11/2 as the regularity of model improves. The only exception is
IZ is the corresponding author. Code available at httpps://gitee www/meta. inf/fil.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

a minimax formulation of IV estimation [5, 14-288-), [19] establish faster rate convergence for this formulation, for models with local Rademacher complexity bounds. Though local Rademacher analysis covers many adaptive ML procedures [c.g.. 10, 211. it still does not fully explain the success of modern MI. approache,, with prominent alternatives including implicit regularization [22] and PAC-Bayesian analyses [23]. From a practical perspective, minimax optimization is computationally expensive, yet it cannot be avoided in the framework of [19], unless we instantiate their method with the less flexible kernel models. It requires additional hyperparameter tuning, which can be challenging in causal problems where validation is indirect and more difficult. It also prevents the use of such flexible models for uncertainty quantification, or inference, for which reliable methods have only been developed for linear nonparametric models [24-26].
This work bridges the gap between sharp theoretical guarantees and robust, practical implementation. We assume our prior knowledge about the causal effect function fo is characterized by a reproducing kernel Hilbert space (RKHS) H. and focus on the flexibility of conditional expectation estimation. This is often possible, because the treatment variable is determined by the problem at hand, and thus has fixed dimensionality." 2 We then present surprisingly simple algorithm, with faster rate guarantees which in many cases match the best known in literature. The algorithm defines the conditional expectation estimates using learned kernel, the basis of which is defined by applying adaptive regression algorithms to random draws from a Gaussian process (GP) prior. Given this learned first-stage kernel and H. we can estimate fo using kernelized IV methods [19, 20, 27], which have closed-form solutions and can be efficiently approximated (e.g.. with Nyström [19]). Our method allows easy hyperparameter tuning, and exhibits competitive performance in simulations. It accesses the regression algorithm as a black box, thus allowing for the use of any MI. methods and benefits from their established theoretical guarantees. It also enables fast quasi- Bayesian uncertainty quantification [24, 28, 29] with improved flexibility.
Our algorithm connects to many ideas in literature. Most notably, it can be viewed as an infinite- dimensional generalization of [12, 13], which consider a linear outcome model with fixed dimension- ality, and use ML methods to learn the optimal instruments [30, 31J. Our setting requires different analyses, for defining an infinite-dimen dimensional estimation target and quantifying errors with its intrinsic complexity. Additionally, analysis of the resultant IV estimator is complicated by the ill- ill-posedness for
of infinite dimensional IV models [32]. A more subtle distinction is in the choice of basis: while
function spaces we can pick any set of basis (i.e., features) and apply the black-box regressor separately, in our case seemingly obvious choices of basis lead to inferior results (Appendix C.5.3). Our analysis also connects the multi-task learning literature, which we make technical contributions. Section 6 discusses related work in detail.
The remaining of the paper is organized as follows: Section 2 reviews background knowledge. Section 3 introduces the instrument learning problem, and reduces it to a general kernel learning problem. We solve the latter problem in Section 4, and return to IV in Section 5 with our main results. We review related work in Section 6, and present numerical experiments in Section 7.
2 Notations and Setup
Denote the joint data distribution as P(dz x dr x dy), its marginal distributions as etc., and their support as X, Z. For functions of observed variables (e.g.. x or z), I1-112 denotes the Lg norm w.r.t. the respective marginal data distribution. ||-Iloc denotes the L.x norm. We use the notation [m] ;= {l..... m). Boldface (x. y, z) emphasizes the denotation of random variables. For any kernel k. GO k) refers to the "standard Gaussian process" [33] with zero mean, and covariance defined by S. Z, = represent (in)equalities up to constants; the hidden constants will not depend on any sample size. Ö()) denotes inequality up to logarithm factors.
Notations
Problem Setup Nonparametric IV regression (NPIV) is formulated as (1) [34, 35). Introduce the + Tog (P(dz)), + B(f(x) 2 = .), and define
conditional expectation operator F T.2 (P(dz)
(0(2) := EEy z = z). We can then express (1) as a linear inverse problem:
(2) where we observe 9o up to regression error. NPIV deviates from standard inverse problems in its need estimate both fo and E. Following conventions in the two stage least square method [1], we
f go:
"Our method can also be applied when contains nigh-dimensional exogenous covariates; see Appendix G.

---

refer to the modeling of fo as the second stage, and that of E - or equivalently, that of Ef for all f in hypothesis space as the first stage. The following assumption describes the setup full detail:
Assumption 2.1 (NPIV). (i) The data variables x, z satisfy (I), and y is bounded by B. (ii) We observe two sets of i..d. samples, with matching (marginal) distributions: D( == [(), (22,,tt, y1) i [n2]]. D( = [.) [ni]].
We impose (ii) since such additional samples are sometimes available, as discussed in e.g. [27]. If only n samples from the joint distribution are available, we can set 71 = n2 n/2.
In the main text, we assume that our prior knowledge about fo is fully characterized by an RKHS H. in the following sense.
Assumption 2.2 (second stage RKHS). (i) X a bounded subset of Rdx; the reproducing kernel k- of H is bounded and continuous. (ii) The integral operator Ta f + cigenvalues (Tx)) 2 &i- 16++1), for some b > 0. (iii) One of the following holds true:
has
(ii).a ("kernel scheme"): So E H.
(ii).b ("GP scheme"): b 1; and for all n, 3ʃ2 E s.t. 11st - Jollas tif 11 - Joll2s n STI
In the above, (i) and (ii) are common technical assumptions: (i) ensures the existence of Mercer's s representation, and (ii) is a complexity measure, with a larger value of b indicating a smaller hypothesis space. (iii) requires H is correctly specified for regression; its two cases cover the different assumptions in standard AKHSSbased estimation and GP modeling. (iii).a is intuitive. (iii).b is standard in the posterior contraction literature [36]; it roughly requires fu to be (at least) as regular as "typical" samples from GP(O, kz), ,, in the sense of [36, Theorem 2.1]. This is different from f because when H. is infinite dimensional, almost all GP samples fall out of H [37, 38]. Our algorithm applies to both settings, but analysis of the GP scheme requires additional effort. It is useful as it allows for Baass-Bayesiinn uncertainty quantification using a GP( kx) prior [26].
The RKHS assumption has been employed in a thread of recent work [27, 20, 39, 26], and generalizes the sieve method in literature [34, 40, 4J. It is most reasonable when has moderate dimensions; for example, when fo satisfies certain L--Sobolev regularity conditions, we can set ki to be a suitable Matérn kernel (Example A-1-A.2). Appendix G studies more general setting, where and include additional, high- -igh-dimensional exogenous covariates. Nonetheless, the assumption will be less reasonable when the treatment variable is nigh-dimenssioal and variable selection is needed for it.
NPIV is typically an ill-posed inverse problem [35]. We now quantify the degree of ill- ill-posedness:
Assumption 2.3. The operator E is compact, with singular values 8; (E) x i-,, where p > 0.
Such mildly ill-posed settings [41] match our polynomial cigendecay assumption for the kernel. In the severely ill-posed setting where the decay of (EE) is exponential, kernels with a similar eigendecay should be used. While the analyses of the two settings share many ideas, the Bayesian inverse problem literature typically restricts to the former for technical reasons [42, 43].
3 From Instrument to Kernel Learning
As assume H. is a correctly specified second-stage model, it remains to determine the first stage. In this section, we show that an ideal first stage model can be defined using another RKHS I, determined by H and E. Although its kernel h has unknown form, we demonstrate that we can access noisy samples from GP(O, k2), which, as Section below shows, enable efficient learning Z. This can be viewed instrument learning, I will only depend the informative features in (Example 3.1).
Let us first consider the GP scheme (Assumption 2.2 (iii).b) which roughly requires fo to be similar to typical samples from GP((O, htz) From a Bayesian perspective, an ideal prior for E.fo should match the distribution of F.f, for f ~ GP(O, k.). This distribution "almost equivalent" to another GP:
3

---

Lemma 3.1 (proof in Appendix B). Denote by [g)~ the L2 equivalence class of g. Under Assumptions 2.1, 2.2, there exists a kernel kg with integral operator T2 = FT, s.t. for GG,, k=).9 GP(O, k2)), [0)~ has the same distribution E[f)
Informally, the lemma shows that GP(O, k2) matches the distribution of Ef. It is thus intuitive that ky could be a good choice for the first stage. The following lemma further motivates its use in the kernel scheme: its (i) shows that Z fulfills the conditions in previous work [19, 27]; the restriction of E on H. has image contained in I, and is a bounded linear map to I. (ii) shows that I, a set of functions, cannot be made smaller while maintaining (i).
Lemma 3.2 (proof in Appendix B). Let I be the RKHS defined by kg. Under Assumptions 2.1, 2.2, (i) for any f E H. there exists g E I s.t. [g]~ E[ (ii) for any g E I, there exists f H satisfying the above. In both cases, we have || ll
We now demonstrate that I only depends on the informative latent features.
Example 3.1 (informative latent structure). Let + Z - 2 be feature extractor that maps the observed instruments z to latent features 2 == $(z), s.t. E(f(x) z) = E(f(x) ((z)) all Ly- integrable f. Then can apply Lemma 3.1, with E replaced by E : B(f(x)
2) E (P(d2)).
leading to a latent-space RKHS I with kernel kg. I induces the inpul-space RKHS
-== g E Σ}, I1g o =0b/=====> llöllz k=(z,z') h. (*)),, ((2)))
The above kg satisfies Lemma 3-1.3.2.* Observe that I perfectly approximates {ES f E H3. but its complexity only depends on I. In particular, k2 has the same Mercer eigenvalues as k-z (Claim B.2), the decay of which is a standard complexity measure [e.g.. 44, Ch. 7J.
While kg has ideal properties, it cannot be used directly as it involves the unknown operator E. Instead, we need to construct an approximation from data. Our main insight is that we can effectively draw noisy samples from GP k=); as we develop in Section 4, such samples enable the approximation of k.z To see how the noisy samples are obtained, consider f ~ GP(O, k.). By Lemma 3.1, Ef is L-g-equivalent to clean samples from GP((, kz); and we have f(x) = g(z) + (f(x) - (E))(Z)), where the latter term is unpredictable given z, and from this perspective can be viewed as noise. Thus, if we apply any regression algorithm to f ~ GP(O, kz), with as input, we will recover "denoised" sample from (P kz), up to regression errors.
In the informative latent feature setting, optimal regression error can only be achieved by methods that adapt to such structures [8-10]. Approximating I with such "denoised" samples can then be viewed as a knowledge distillation procedure, which results in a compact representation of the adaptive regression algorithm. This particularly beneficial in the NPIV setting: as discussed in the introduction, using learned kernel eliminates the need of minimax optimization in estimation, and allows the (indirect) use of adaptive methods for uncertainty quantification.
4 Black-Box Kernel Learning
In this section, address the problem of kernel learning given noisy GP samples. As our results apply to more general settings, we first state the assumptions with full generality.
Assumption 4.1 (RKHS). There exist a continuous function & Z + 2, and reproducing kernel R over Z, s.t. (i) the random variable ž = Φ(z) is supported on a bounded subset of Rdl; h2 is bounded. (ii) The eigenvalues of the integral operator Ts g + f (2, (E)))) Pddz) satisfy > (T=)) S (6++1) for some b > O. (iii) g GP R..) have finite sup norm with probability 1.
The above assumption applies to a latent-space kernel k.. As shown in Example 3.1, 4 and ky induce an input-space kernel kz, and RKHS I. which inherit the assumed regularity conditions. Our goal is to estimate kg. This is harder than the estimation of k2, as it also involves $
'Recall the L2 space is not a function space, and consists of equivalence classes of functions. Note that for readability, may occasionally ignore this distinction in the main text, and use (a version of) E to also denote the corresponding map between function spaces. All such denotations can be made unambiguous (Remark B. and all null set ambiguities this section can be removed under mild additional assumptions (Lemma B.5). *There may be multiple kernels satisfying Lemma 3.1, but they are equivalent up to null sets (Claim B B.2); the ambiguity can be removed under mild assumptions (Lemma B.5).
4

---

All conditions for k, are satisfied by Matérn kernels with sutiable order, see Appendix A. Ap- pendix B.1 discusses its applicability in the IV setting, where I defined as in Section 3. Briefly, (ii) always holds for b 2 max b, 2p 1}. and if H is further correctly specified in the sense of Assumption E.I, b + 2p. (i) and (ii) hold under mild technical assumptions.
We will "denoise" noisy GP(O, kz) samples using a regression oracle, which is specified below:
Assumption 4.2 (regression oracle). Let D(m)) {(Z,, g(2)) + e be ni iid replications of the rvs (z. g(z) + o), s.t. B(e z) = and g(z) + e has a I-subgaussian distribution. Then the oracle returns estimator Iu,n, s.t. Po B 1
o En for some En: + 0.
In the IV setting, have g Ef ~ GP(O, k2), and E(ee z) E(f(x) (Ef)(z) z) 0; the subgaussian condition is verified by Lemma A.4.
To provide some intuition on adaptivity, we instantiate the assumption with the DNN model in [10], and compare the resulted S, with fixed-fone form kernels:
Example 4.1 (adaptivity of DNN oracles). Let Z C CR,, $ 2 be f--Hölder regular, I be
Matérn-B2 RKHS, and $1, B2 2 1. Let the regression oracle return é approximate empirical risk
minimizer for the model [10]. Then for any > 0, holds that (see Appendix C.5.1 for derivations)
Sn = 2d1 + 71
+ Eopt) = =: ==:((ffenn + T2
2P244 + Eopt
(3)
In the above, fea.n characterizes the hardness of feature learning, i.., learning T. The second term characterizes that of kernelized regression given the optimal features: it matches the optimal regression rate if we had full knowledge about t or equivalently, I, and would be attainable by kernel ridge regression (KRR) using? I.
As long as Copt is small, (3) will match the minimax rate up to logarithms. When 8:/d. < Bafdis the minimax rate is Efean 2 n - 92/(282 meaning that the hardness of feature learning cannot be overlooked. Otherwise, the rate En nearly matches the rate given full knowledge of the unknown I, up to the infinitesimal 6 > 0; this is realistic when, e-g., d2 "> di and to is linear (31 oc).
We are interested in the high-dimensional regime where dg >> dj. In this case, fixed-form Matérn or mini 99 31.3
RBF kernels could only attain the rate of On miint// (P2 (P22 which can always be much worse than (3), regardless of the hardness of feature learning. This comparison suggests that fixed-form kernels cannot adapt to the latent feature structure to avoid the curse of dimensionality.
We now define the approximate RKHS. and Ju,ny be the respective estimate returned by the regression oracle, constructed from the shared dataset D(ni) = ((,,, g00 (2:) + ((2) E [ma].j E [m]] where e are subgaussian, mean- zero noise. Let ألوو = min (Cin(Ca
Let {9وو) : E [m]] be i.idd samples from the GP prior,
(C& x log n. g62 (-)), where Cs constant determined by I. Define Gz(2)= == (0!!) (z),..., (mm) (z)). Our approximate RKHS is defined as
I = [g(2) T Gn, (z) for some 0 R"), with norm ||9||=== /m//0/e
(4)
As Î is a finite-dimensional linear space, it is an RKHS. We can check that bo C2
lognrllg/1z
Under a given model, regression error is decomposed into approximation and estimation (i.e., generalization) errors. We first present the approximation error bound:
Theoretical Results
Theorem 4.1 (proof in Appendix C.2). Under Assumptions 4.1, 4.2, there exists universal constant Or > 0, and an event En, determined by (l....m) and D(nn)) with IP po)) En, + 1, on which for any g" L2(P(dz)), there exists g" E &I s.t.
llğ" 1 I c+ <rllPPojm/g''
(5)
llğ" 9" 112 S + Proojjn (I,,,, + (0++1)//22
+ +lgg" /lgg/
Projmng 112,
(6)
where m' [m/ and Proj.. denotes the projection onto the top m' Mercer eigenfunctions of k.-
'110] establishes formal lower bounds. Also, for small 32, we can replace d, with manifold dimensionality of Z, but can still be much larger than di.
5

---

We will use I to estimate functions on separate dataset with n2 samples. For single regression task, the estimation error can be simply bounded as ((vm/nz) [45]. However, our analysis of IV estimation will require quantifying the intrinsic complexity of I, which will also allow the use of larger m in practice. The following proposition provides one such result; it will be used in Section 5, analyze IV estimation in the kernel scheme (Assumption 2.2 (iii).a).
Proposition 4.2 (proof in Appendix C.3), Let I, Dím)) be defined as above, and ong be the critical radius of the local Rademacher complexity of the norn ball In [46, Ch. 14]. On the event defined in m (bb+1///2 Em²)
Theorem 1.1, we have dng- Ón2 (b+1)/((++))
IV estimation in the GP scheme is more delicate, and requires additional analysis of I, which is deferred to App. D. Before we proceed, however, we illustrate the results on simple regression task:
Example 4.2 (Example 4.1, cont'd), Let $ I be defined as before, and ف be estimated by the DNN oracle. Suppose Copt is not greater than the other terms. Then
L. Let [m] b/(b+)) On the event in Theorem 4.1, for any g" E I, there exists 3 E ī s.t.
|ل cr |1 Iz l1ğ" v"1l2=0(11 1(11g]]]EEm)
ii Let 9 (P(O,kg)
refined analysis, based on Corollary C.6, shows that when m [n//btl]] there exists s.I. |1 ni 1/26++), -gggggp(. - 12= (Em,))
(See Appendix C.5.2 for derivations, and another high-probability bound in the GP scheme.)
e,) Let 9% be the truncated OLS estimate using I, on dataset {(2,, +x)
[ng]} where [45,
E(e; 24) O, Var(e,) S 1, 1, 1o B. Then Ellg - 9" 121|== Thm. 11.3]. When n1 = the latter term is < Eng, and case (ii) above always matches the DNN
rate. Case (i) matches the DNN rate when feature learning becomes harder (E2,, Z ni otherwise the rate may be slightly inferior, but still approaches n, 1122 as the regularity b improves.
-b/(41)));
As discussed in Example 4.1, when d₂ di the DNN rate outperform fixed-form kernels by large margin. The above example demonstrates a similar superiority of the learned kernel.
5 Results for IV Regression
We shall use the approximate first stage 7 for IV regression, by plugging ÿ and H. to the kernelized estimators in [19, 26]; see Algorithm We analyze the resulted estimators in this section, while deferring implementation details, including hyperparameter selection, to Appendix F.
Algorithm 1 Kernelized IV with learned instruments.
Require: D(na)) DD
regression algorithm Regress; second-stage kernel kz; N
1: for j < 1 to m do
2: Sample f()) GP(0, kg)
3: a(1)) + Regress(
(i;)) : [m]]]
4: end for (the invocations of Regress may be replaced with single vector-valued regression)
5: Define (2,z²') == 1 Ex= gh] (z)g where gh := min(g (-), C' log m).
6: return KernelizedIV(DD) (m2) k k.) (See (7) below, or Appendix F for the closed-form solution)
Both [19] and the posterior mean estimator of [26] have the form
arg min En =1 arg min max- JEH
1
Σ
n2
- (7)
JEH geī n2
Their difference lies in the regularization scaling, which arises from the different assumptions about So and H (Assumption 2.2). Thus, we analyze the resulted two estimators separately, in Section 5.1 and Section 5.2 below. In the setting of [26] we are also able to justify the use of likelihood- based model comparison and (quasi- Bayesiaan model averaging (BMA).

---

5.1 Estimation in the Kernel Scheme
[19] establish faster rate convergence of the point estimator under simple assumptions. We now provide corresponding results using our learned I, by plugging in the results in Section 4.
Proposition 5.1 (proof in Appendix E. l). Assume Assumptions 1.1, 2.2 (kernel scheme), 4.1 and 4.2. Let I be defined by k2 in Algorithm 1, and faz be defined by (7), with K, X, U set as in Appendix E.I. On the event defined in Theorem 4.1, we have
1+1
|E(fma - ffu)ll2= fo)ll2-Op((Sm, + +02 2(442)
)11 +1fool22))
(8)
Let us compare the result with [19] in the setting of Example 4.1. Suppose n1 = n2. [19] establishes the rate of Oy((E,, + n2 (b+ (6++))(+++)) (1 + (l where E is comparable with first-stage regression rate established from local Rademacher analysis. For the DNN model in Example 4.1, we have & = θ(C)) = Ó fn tn 1)22((6-1) in the general case, fr S = = (0, assuming additional regularity (Remark C.3). Thus, the two rates are equivalent if f sufficiently large, meaning that the difficulty of feature learning cannot be ignored. Otherwise, [19] may be better if b < 6--6-+14 1; this is somewhat narrow range, as 6 2 max
b). Recall that our method is more appealing computationally: directly instantiating [19] with DNNs requires solving minimax problem similar to (7), while for our learned kernel (7) can be evaluated in closed form.
We can also compare (8) with kernelized IV using a fixed-form first stage. In the above setting, its best rate is also provided by [19], and is dominated by the kernel regression error in Ex. 4.1 which, as we discussed, can be much worse than É... Our improved rate has been made possible by the fact that we are approximating first-stage model with optimal adaptivity (Ex. 3.1) ), at rate that is also adaptive to the informative latent structure (Ex. 4.
In summary, our algorithm combines the best of both worlds: it maintains the sharp guarantees of adaptive models, and the simplicity of kernel methods.
5.2 Quasi-Bayesiag Estimation and Uncertainty Quantification
Quasi-Bayesian analysis enables efficient uncertainty quantification for NPIV, without introducing extra risks of model misspecification [24, 28]. [26] studies quasi-Bayesian posterior constructed from (7) and GP kz) prior. It is defined through the Radon-Nikodyn derivative w.r.t. the prior, (AT(-ID(s)) ((mt))/saaa//))(() (f) (f) For kernel first-stage models, the quasi-posterior can be evaluated closed form (App. F). For general models, however, it is entirely unclear if approximate inference can be possible, since for any parameter evaluation of e, (f) involves solving separate optimization problem. Our kemne learning algorithm enables the (indirect) use of such models.
Analysis of tion. Thus, [26] introduced additional assumptions. Our analysis is further complicated by a different assumption on I, and approximation errors in I, which necessitate further assumptions. App. E.2 discusses these assumptions in detail. For simplicity, we state the result in a "rate-optimal" case:?
procedures is more challenging, partly because of the weaker regulariza-
Theorem 5.2 (posterior contraction; proof in App. E.3). Assume Asms. 2.1, 2.2 (GP scheme), 2.3, 4.1, 4.2, D.I. D.2, E.I, E.2. Let nı be s.t. <2, log n2 + ni -(++2))(0++2ptt) S 5n5 and m x ni
1/(b+2p+1)
Let In, (- (((a))))) D(na)) be defined in (52) in appendix. Then, with D(m1) probbbbiiiyy + 1. we have
E Epign 1I, (If: [lf - foll22 MEn,) D + 0,
E Ding) MT (U |[E([ f0)ll22 Mr,,)) } (((aa)) + O,
where 6.,, Ö(n (2)t2))22(b---)))p 2pt1)) (6-20/220221))022 Eng = Ömn 20 b/2(b+2p+)))
'With some abuse of notation, we also use Ô to hide the infinitesimal deterioration of the polynomial order.
7 "Classical NPIV lower bounds continue to hold given full knowledge of E [47], so the rate ny 4/((++2++))) is minimax optimal irrespective of n1. In our setting. it is certainly desirable to improve the dependency on Ru. and our restriction is only employed to simplify proof. In simulations find the choice of n2 works well.
7

---

Theorem 5.2 immediately implies Theorems 5, 6 in [26] for our I, with the extra logarithms, as their proofs do not involve the first stage. Those results establish Sobolev norm rates, and justify uncertainty quantification by lower bounding the magnitude of posterior spread.
In the nonparametric Bayes literature, contraction results like Theorem 5.2 often lead to the justifica- tion of marginal likelihood-basse model selection and averaging. This is also the case here. The key ingredient the following marginal quasi-likelihood bound: Corollary 5.3 (proof in Appendix E.4). In the setting of ('Thorrem 5.2, for some C > O we have
P Di (C (C
log 8 n2 S -lo 11, (Dina)) S Cn. #+2p+T log" n22) +
This result allows the comparison of finite number of second-stage RKHSes. Of particular interest is the comparison between power RKHSes (Defn. A. which often have intuitive interpretations: e.g., for Matérn RKHS H and y E (2/(b- 1), 1), the power RKHS H is equivalent to lower-order Matérn RKHSes ([48); Ex. A.3). We can verify that such H fulfills the assumptions about H. Thus, provided the other assumptions continue hold, Corollary 5.3 will hold for all such H with b + 1 replaced by 76 + 1), showing the marginal likelihood has a different asymptotics. Consequently, it establishes asymptotically valid comparison between such models, and justifies the use of BMA.
Analysis of more general settings requires additional effort: NPIV is an inverse problem, and we anticipate the subtleties of model selection in nonparametric inverse problems. For example, analyses are usually restricted to the selection of > [43, 49, 50], and the > 1 case requires additional assumptions [49]. 8 In the IV setting, it should also be noted that valid model comparison requires a good approximation to E и. since otherwise the quasi-likelihond becomes less meaningful at any finite sample size. The same intuition applies to other model selection procedures [18, 20, 27] based on the estimated violation of (1). When the approximation cannot be guaranteed, it could be preferable stick to the prior knowledge and fix a conservative choice for H.
6 Related Work
Multi-Task Learning Our Example 4.2 can also be viewed as quantifying sample efficiency improvements in multi-task learning, if we view the GP prior draws as the labeling functions for handful of diverse training tasks, which share the representation 4. This general idea is not new: starting from [51, 52], line of recent work establishes similar results. Most related is [52, Sec. 5], which assumes a
linear model for g, and an adaptive & with metric entropy bounds. We assume more general models for both components, and do not require different training tasks to have separate inputs. On the flip side, [52] allows for non-iid training tasks. [52, Sec. 6] investigated infinite-dine diinte-ddmiiiinall g. but established a slow rate. We are unaware of any work that established fast-rate convergence for infiniteddimensional top-level models, or used MI models as a black box. Both aspects may be interesting for multi-task learning, and are necessary for instrument learning.
The double machine learning framework [53] also uses black-box ML models to estimate certain nuisance parameters in the model. While the operator E can be viewed as a nuisance parameter, the structure of the NPIV problem is quite different: [54, p. 8] noted that it is very unclear such a view can be helpful for NPIV estimation; consistent with their remarks, we have also been unable to cast our problem into the double ML framework. Note that double ML has been applied to semiparametric estimation and inference for IV [53, 55-57J. which are orthogonal to our goal.
Causal Statistics
It has long been known [30, 31] that under linear outcome model fo(x) 0 x, using E(x z.) as instrument leads to cnn-consittnnt estimates. Our Section 3 can be viewed as an chinite-dimensione generalization of this observation. 9 Given high-dimensional instruments and a parametric outcome model, there is a large body of literature on efficient inference; see [12] for a review. As we move to nonparametric models, focus estimation which becomes much more challenging, in the spirit of [54]. Still, we have provided qualitative characterization for uncertainty estimates in Section 5.2.
For the use of ML for nonlinear IV, [6] studied heuristic application of NNs. We discussed the minimax formulation in introduction. [5, 26] justified the use of NNs with the respective neural
$ We not cover it here for brevity, noting that it well undersood in inverse problem settings [43, 49]. As noted in [27]. when fo E H. for some RKHS H. the first stage should model [xf(x) | 2) for all f E H. opposed to merely modeling [5(x 2). Note that [27] did not study the optimal choice of the first stage.
8

---

ABMM
AGMM
JGMM-RSF
Proposed
synii
Irev
31pely
24r4l
10Q-//MEE (OQUIMBE)
100
3092
2902
Figure 1: Predictive performance: test MSE vs sample size n1 = n2 for all method, and D E {2, 100).
Full results are in App. H.2.
Table I: Runtime results for all methods in the predictive experiment, for N = 2500, = 100.
Method
AGMM Tree
AGMM-NN
AGMM- RBF
Proposed
Runtime /s
1374 418 303 + 16 6.7 + 0.1
25.9 + 5.6
tangent kernels (NTKs) which, like other fixed-form kernels, cannot adapt to the informative latent structure [8, 9]. [39, 58] investigated the combination of an NN-based second stage and a linear first stage, which could be useful in complementary scenarios. [17] considered feature learning in both stages, but only established slow rate; as the authors noted, also unclear if their algorithm reliably minimizes the empirical risk.
For model selection in the setting of Section 5.2, [59] prove the validity of bootstrap- based selection for the sieve estimator [34]. [39, 60] investigate the use of marginal likelihood for two different kernel-based IV estimators: [60] establish a crude -1/4log upper bound for the log marginal likelihood, and [39] connect it to the empirical leave-one-out validation error. Neither result fully justifies model selection as our Corollary 5.3. For kernelized IV models, [20. 27, 61] proposed validation statistics for comparing finite number of first stage models.
7 Simulation Study
Our main simulation setup is adapted from [18, 19]; Appendix H.4 presents additional experiment on the demand dataset [6. 17J. In [18, 19], the observed z, y, are generated by
Z Unif(-3,3] 1 z = MZZ), u (O, 1), x 21 + + e. y (fo(x) + u By 14)/a,
where is the confounder, e,, ey ~ (0, 0.12) are independent noise, and the constants standardize y. We consider three choices for h: (i) D 2, the identity function; this recovers the setup in previous work, and quantifies the hardness of the NPIV problem given true instruments. (ii) dim z = E {40, 100}. h is three-layer DNN; this simulates a feature learning scenario, and ensures the observation has low signal-to signal-to-noise ratio (O(1/D)). (iil) h maps ž, to MNIST [62] or CIFAR-10 [63] image with matching label; the MNIST setting also appeared previous work.
We consider two choices for fo: (i) widely used collection of functions (c.g., sin, abs) in [18]. (ii) fo (0, k.). (ii) ensures the correct specification of H. and allows us focus on the first stage.
We use DNN as the black-box learner, and a RBF kernel for H. with bandwidth determined by marginal likelihood (72). We set N1 = N₂ (500, 2500, 5000). We defer setup details and full results to Appendix H, and summarize the findings below:
Hyperparameter Selection (App. H.1) We first study hyperparameter selection in instrument learning. We set fo ~ GP, D E (2, 20, 100). We find our validation statistics (71) always correlates with the counterfactual MSE l.f.. - fo and that across a large hyperparameter space, trained DNNs always outperform first-stage models based on RBF kernels, or randomly initialized DNNs.
Predictive Performance (App. H.2) For h defined as in (i-ii), we compare our algorithm with [19, AGMMJ, instantiated with kernel, tree and NN models. As shown in [19], the baselines have

---

Table 2: Test MSE, radius and estimated coverage rate of the 90% Le credible ball (CB), and the average coverage of pointwise 90% credible interval (CD, for fo ~ GP, 100. For the CB coverage rate estimate, we report its 95% Wilson score interval [64]. Full results are in App. H.3.
Method
R1 =
Test MSE
90% CB. Rad.
90% CB. Cvg.
90% CI. Cvg.
500
.097 060
.201 #.025
.923 [888,, 948]
.915 1.123
Proposed
2500
.035 024
.074 2.008
.917 [,880, 943]
.908 #.127
5000
.021 =06
.049 .004
.920
[8844, 9946]
.905
500
431 1.192
.240
.187 1147..238)
.640
RBF
2500
.176 00089
.175 #.023
.517 .573]
.822 1:136
5000
.126 0.072
.156 2.019
.660 [,605,
.855 1:143
competitive performance on this setup; the latter two models also enjoy adaptivity guarantees. A representative subset of results are plotted in Fig. 1: our method has stable performance as we move to high dimensions, demonstrating excellent adaptivity. In contrast, fixed-form kernels fail to identify the informative features. AGMM-tree and AGMM-NN also have deteriorated performance as D increases, despite their theoretical guarantees, presumably due to the challenges in optimization. Table reports the run time of all methods in this experiment. As we can see, our method is more efficient than both adaptive baselines.
For image-based ht we compare with AGMM-NN and [39], which report the best results in the MNIST setting. Our method outperforms both baselines.
Uncertainty Quantification (App. H.3) Table 2 presents a subset of results for fo ~ GP. Com- paring with fixed-form RBF first stage, method produces sharper credible intervals, which also have better coverage. For fo specified as in [18], we experiment with BMA over a grid of RBF kernels, and present visualizations in Appendix H.3. We find that when the model is more correctly specified, BMA produces conservative uncertainty estimates which are nonetheless informative. However, when all models severely misspecified (e.g., when fo is step function), we cannot expect model-based uncertainty estimates to have ideal coverage.
Exogenous Covariates (App. H.4) We evaluate the extended algorithm in Appendix G the demand dataset [6], which is widely used simulation design with high-dimensional exogenous covariates. As shown in the appendix, our extended algorithm has competitive performance.
Acknowledgements
This work supported by the National Key Research and Development Program of China (2020AAA0106302);; NSFC Projects (Nos. 62061136001, 62076145, 62076147, U19B2034, U1811461, U19A2081, 61972224), Beijing NSF Project (No. JQ19016), BNRist (BNR2022RC01006), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J.Z. is also supported by the XPlorer Prize.
References
[1] J. D. Angrist and J.-S. Pischke, Mostly harmless econometrics. Princeton university press, 2008.
[2] M. A. Brookhart, J. A. Rassen, and S. Schneeweiss, "Instrumental variable methods in compara- tive safety and effectiveness research," Pharmacoepidemiology and drug safety, vol. 19, no. pp. 537-554, 2010.
[3] S. Burgess, D. S. Small, and S. G. Thompson, "A review of instrumental variable estimators for mendelian randomization," Statistical methods in medical research, vol. 26, no. 5, pp. 2333-2355, 2017.
[4] X. Chen and D. Pouzo, "Estimation of Nonparametric Conditional Moment Models With Possibly Nonsmooth Generalized Residuals." Econometrica, vol. 80, no. 1, pp. 277-321, 2012.
10

---

[5] L. Liao, Y.-L. Chen, Z. Yang. B. Dai, Z. Wang, and M. Kolar, "Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach," arXi::2007.01290 [cs. stat], 2020, arXiv: 2007.01290.
[6] J. Hartford, G. Lewis, K. Leyton-Brown, and M. Taddy, "Deep IV: A flexible approach for counterfactual prediction," " in International Conference on Machine Learning, 2017, pp. 1414 1423.
[7] Bach, "Breaking the curse of dimensionality with convex neural networks," " The Journal of Machine Learning Research, vol. 18, no. 1, pp. 629-681, 2017.
[8] C. Wei, J. D. Lee, Q. Liu, and T. Ma, "Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel," in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, Eds., vol. 32, 2019.
[9] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari, "Limitations of lazy training of two-layers neural networks," arXiv preprint arXiv: 1906.0899,,, 2019.
[10] Schmidt-Hieher "Nonparametric regression using deep neural networks with ReLU activation function," The Annals of Statistics, vol. 48, no. 4, 2020, arXiv: 1708.06633.
[11] W. Lin, R. Feng, and H. Li, "Regularization Methods for High-Dimensional Instrumental Variables Regression With an Application to Genetical Genomics, Journal of the American Statistical Association, vol. 1 110, no. 509, pp. 270,288, 2015.
[12] A. Singh, K. Hosanagar, and A. Gandhi, "Machine learning instrument variables for causal inference." in Proceedings of the 21st ACM Conference on Economics and Computation, 2020, pp. 835-836.
[13] Chen, D. L. Chen, and G. Lewis, "Mostly Harmless Machine Learning: Learning Optimal Instruments in Linear IV Models," arXiv:2011.06158 [cs, econ, stat), 2021, arXiv: 2011.06158.
[14] R. Okui, "Instrumental variable estimation in the presence of many moment conditions." Journal of Econometrics, vol. 165, no. 1, pp. 70-86, 2011.
[15] M. Carrasco, "A regularization approach to the many instruments problem," Journal of Econo- metrics, vol. 170, no. 2. pp. 383-398, 2012.
[16] A. Belloni, D. Chen, V. Chernozhukov, and C. Hansen, "Sparse models and methods for optimal instruments with an application to eminent domain," Econometrica, vol. 80, no. 6, pp. 2369-2429, 2012.
[17] L. Xu, Y. Chen, S. Srinivasan, N. de Freitas, A. Doucet, and A. Gretton, "Learning deep features instrumental variable regression," arXiv preprint arXiv:2010.07/ 154, 2020.
[18] A. Bennett, N. Kallus, and T. Schnabel, "Deep generalized method of moments for instrumental variable analysis," in Advances in Neural Information Processing 2astems.20199
[19] N. Dikkala, G. Lewis, L. Mackey, and V. Syrgkanis. "Minimax estimation of conditional moment models," in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 12 1-248-1222.
[20] K. Muandet, A. Mehrjou, S. K. Lee, and A. Raj, "Dual instrumental variable regression," " arXiv preprint arXiv: 12358, 2020.
[21] V. Syrgkanis and M. Zampetakis, "Estimation and Inference with Trees and Forests in High Dimensions," arXi::2007.03210 [cs, math, stat), 2020, arXiv: 2007.03210.
[22] B. Neyshabur, R. Tomioka, and N. Srebro, "In search of the real inductive bias: On the role of implicit regularization in deep learning." arXiv preprint arXiv: 1412.66112.66,, 2014.
[23] P. Alquier, "User- friendfy introduction to PAC-Bayes bounds," arXiv preprint arXiv:2110.11216, 2021.

---

[24] K. Kato, "Quasi-Bar Bayesian analysis of nonparametric instrumental variables models," The Annals of Statistics, vol. 41, no. pp. 2359-2390, 2013.
[25] X. Chen and T. M. Christensen, "Optimal sup-norm rates and uniform inference on nonlinear functionals of nonparametric IV regression," arXiv preprint arXiv:/508.0330 500::0308,3335 2015.
[26] Z. Wang. Y. Zhou, T. Ren, and J. Zhu, "Quasi-Bayesian dual instrumental variable regression." arXiv preprint arXiv:2 arXiv:2106.08750 2021.
[27] R. Singh, M. Sahani, and A. Gretton, "Kernel Instrumental variable regression," arXiv preprint 1XXiv:9066000222 2019.
[28] V. Chernozhukov and H. Hong, "An MCMC approach to classical estimation," Journal of Econometrics, vol. 115, no. 2, pp. 293-346, 2003.
[29] J.-P. Florens and A. Simoni, "Nonparametric estimation of an instrumental regression: A quasi- Bayesian approach based on regularized posterior," Journal of Econometrics, vol. 170, no. 2, pp. 0188-47,,22 2012.
[30] H. H. Kelejian, "Two-stage least squares and econometric systems linear in parameters but nonlinear in the endogenous variables," Journal of the American Statistical Association, vol. 66, no. 334, pp. 373-374, 1971.
[31] G. Chamberlain, "Asymptotic efficiency in estimation with conditional moment restrictions," Journal of Econometrics, vol. 34, no. 3. pp. 305-334, 1987.
[32] J. L. Horowitz, "Ill-Posed Inverse Problems in Economics," Annual Review of Economics, vol. no. 1, pp. 21-51, 2014.
[33] J. van Zanten and A. van der Vaart, "Reproducing kernel hilbert spaces of gaussian priors," in Pushing the limits of contemporary statistics: contributions honor of Jayanta K. Ghosh. Institute of Mathematical Statistics, 2008, pp. 200-222.
[34] W. K. Newey and J. L. Powell, "Instrumental variable estimation of nonparametric models." Econometrica, vol. 71, no. 5, pp. 1565-1578, 2003.
[35] L. Horowitz, "Applied nonparametric instrumental variables estimation," Econometrica, vol. 79, no. 2, pp. 347-394, 2011.
[36] A. W. van der Vaart and J. H. van Zanten, "Rates of contraction of posterior distributions based on Gaussian process priors," The Annals of Statistics, vol. 36, no. 3, 2008.
[37] M. F. Driscoll, "The reproducing kernel hilbert space structure of the sample paths of a gaussian process," Zeitschrift für Wallsscheinlicckkettshheoor und verwandte Gebiete, vol. 26, no. 4, pp. 309-316, 1973.
[38] A. W. van der Vaart and J. H. van Zanten, "Information rates of nonparametric Gaussian process regression," " Journal of Machine Learning Research, vol. 12, pp. 2095-2119, 2011.
[39] R. Zhang, M. Imaizumi, B. Schölkopf, and K. Muandet, "Maximum moment restriction for instrumental variable regression," arXiv preprint arXiv:20100076884, 2020.
[40] R. Blundell, X. Chen, and D. Kristensen, "Semi- -Semi-Nopparametric IV Estimation of Shape- Invariant Engel Curves," Econometrica, vol. 75, no. 6, pp. 1613-1669, 2007, publisher: [Wiley, Econometric Society].
[41] L. Cavalier, "Nonparametric statistical inverse problems," " Inverse Problems, vol. 24, no. 3, p. 034004, 2008
[42] B. Knapik, A. W. van der Vaart, and H. van Zanten, "Bayesian inverse problems with Gaussian priors," The Annals of Statistics, vol. 39, no. 5, pp. 1626-2657, 2011.
[43] B. T. Knapik, B. T. Szabó, A. W. Van Der Vaart, and J. H. van Zanten, "Bayes procedures for adaptive inference in inverse problems for the white noise model." Probability Theory and Related Fields, vol. 164, no. 3-4, pp. 771-813, 2016.
12

---

[44] I. Steinwart and A. Christmann, Support Vector Machines. Media, 2008.
Springer Science & Business
[45] L. Györfi, M. Kohler, A. Krzyzak, H. Walk et al., A distribution-free theory of nonparametric regression. Springer, 2002, vol. 1.
[46] M. J. Wainwright, High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, 2019, vol. 48.
[47] X. Chen and M. Reiss, "On Rate Optimality For Ill-posed Inverse Problems In Econometrics," Econometric Theory, no. 3. pp. 497-521, 2007.
[48] S. Fischer and I. Steinwart, "Sobolev norm learning rates for regularized least-squares algo- rithms," Journal of Machine Learning Research, vol. 21, pp. 205- 2020.
[49] B. Szabó, A. W. Van Der Vaart, and J. van Zanten, "Frequentist coverage of adaptive non- parametric Bayesian credible sets." The Anmals of Statistics, vol. 43, no. 4, pp. 1391-1428. 2015.
[50] Jia, Peng, and Gao, "Posterior contraction for empirical bayesian approach to inverse problems under non-diagonal assumption," Inverse Problems & &mmaging, vol. 15, no. 2, pp. 201-228, 2021.
[51] N. Tripuraneni, M. I. Jordan, and C. Jin, "On the Theory of Transfer Learning: The Importance of Task Diversity," arXiv:2006. 1660 [cs, stat], 2020, arXiv: 2006. 1650.
[52] S. S. Du, W. Hu, S. M. Kakade, J. Lee, and Q. Lei, "Few-Shot Learning via Learning the Representation, Provably," arXiv:2002.09453 [cs, math, stat], 2021, arXiv: 2002.09434.
[53] V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins, "Double/debiased machine learning for treatment and structural parameters," The Econometrics Journal, vol. 21, no. I, pp. C1-C68, 2018.
[54] D. J. Foster and V. Syrgkanis, "Orthogonal statistical learning," arXiv preprint 1rX0v:1901.09036. 2019.
[55] V. Syrgkamis, V. Lei, M. Opreseu, M. Hei, K. Battocchi, and G. Lewis, "Machine learning esti- mation of heterogeneous treatment effects with instruments," " arXiv preprint 1rXiv:1905.10176 2019.
[56] R. Singh, L. Xu, and A. Gretton, "Generalized kernel ridge regression for nonparametric structural functions and semiparametric treatment effects," arXiv preprint arXiv:2010.04855, 2020.
[57] Y. Jung, J. Tian, and E. Bareinboim, "Double machine learning density estimation for local treatment effects with instruments." in Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021.
[58] J. Chen, X. Chen, and E. Tamer, "Efficient Estimation in NPIV Models: A Comparison of Various Neural Networks-Based Estimators,"" 1xXiv:2110.06700 [cs, econ, math, stat], 2021, arXiv: 10.06667.
[59] X.
Chen, T. M. Christensen, and S. Kankanala, "Adaptive estimation and uniform confidence bands for nonparametric iv," Cowles Foundation for Research in Economics, Yale University, Tech. Rep., 2021.
[60] K. Hsu and F. Ramos, "Bayesian deconditional kernel mean embeddings," in International Conference on Machine Learning. PMLR, 2019, pp. 2830-2838.
[61] R. Zhang, K. Muandet, B. Schölkopf, and M. Imaizumi. "Instrument Space Selection for Kernel Maximum Moment Restriction," arXiv:2106.03: 03340 [cs], 2021, arXiv: 2106.03340.
[62] Y. LeCun, "The mnist database of handwritten digits," http://yann. lecun. com/exdb/nnist/, 1998.
13

---

[63] A. Krizhevsky, G. Hinton et al., "Learning multiple layers of features from tiny images." 2009.
[64] E. B. Wilson, "Probable inference, the law of succession, and statistical inference," Journal of the American Statistical Association, vol. 22, no. 158, pp. 209-212 1927.
[65] Steinwart and C. Scovel, "Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs," Constructive Approximation, vol. 35, no. 3, pp. 363-417, 2012.
[66] I. Steinwart, "Convergence types and rates in generic Karhunen-Loëwe expansions with applica- tions to sample path properties," Potential Analysis, vol. 51, no. 3, pp. 361-395, 2019.
[67] M. Kanagawa, Hennig, D. Sejdinovie, and B. K. Sriperumbudur, "Gaussian processes and kernel methods: A review on connections and equivalences," arXiv preprint arXiv: 1807.02582 2018.
[68] I. Steinwart, D. R. Hush, C. Scovel et al., "Optimal rates for regularized least squares regression," in COLT, 2009, pp. 79-93.
[69] R. A. Adams and J. J. Fournier, Sobolev Spaces. Elsevier, 2003.
[70] N. Eldredge, "Analysis and probability on infiniteddimensional spaces," arXiv preprint arXiv:1607.03590 2016.
[71] S. Ghosal and A. Van der Vaart, Fundamentals of Nonparametric Bayesian Inference. Cam- bridge University Press, 2017, vol. 44.
[72] R. Bhatia, Matrix analysis. Springer Science & Business Media, 2013, vol. 169.
[73] O. Kallenberg and O. Kallenberg, Foundations of modern probability. Springer, 1997, vol. 2.
[74] A. M. Stuart, "Inverse problems: Bayesian perspective," Acta Numerica, vol. 19, pp. 451-559, 2010.
[75] B. Simon, Operator theory. American Mathematical Soc., 2015, vol. 4.
[76] A. Edelman, "Eigenvalues and condition numbers random matrices," SIAM journal on matrix analysis and applications, vol. 9, no. 4, pp. 543-560, 1988.
[77] R. Vershynin, High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge university press, 2018, vol. 47.
[78] H. Wielandt, "An extremum property of sums of eigenvalues," Proceedings of the American Mathematical Society, vol. 6, no. 1, pp. 106-110, 1955.
[79] A. W. van der Vaart and J. H. van Zanten, "Adaptive Bayesian estimation using Gaussian random field with inverse Gamma bandwidth," The Annals of Statistics, vol. 37, no. 5B, 2009.
[80] A. Rudi, R. Camoriano, and L. Rosasco, "Less is more: Nyström computational regularization." NIPS, 2015, pp. 1657-1665.
[81] A. Rudi and L. Rosasco, "Generalization properties of learning with random features," arXiv 2016.
preprint
[82] W. V. Li and W. Linde, "Approximation, metric entropy and small ball estimates for Gaussian measures," The Annals of Probability, vol. 27, no. 1999.
[83] G. Pisier, "Sur les espaces de banach k-convexes," Séminaire Analyse fonctionnelle (dit" Maurey-Schwarte") pp. 1-15, 1979.
[84] A. Pajor and N. Tomczak- "Volume ratio and other n-numbers of operators related to local properties of Banach spaces," Journal of Functional Analysis, vol. 87, no. 2, pp. 273-293, 1989.
[85] J. Shawe-Tayaaeee C. Williams, N. Cristianini, J. Kandola, "On the eigenspectrum of the gram matrix and its relationship to the operator eigenspectrum,"" in Alt 2002, Lnai 2533, Springer- Verlag, 2002. pp. 23-40.
14

---

[86] A. Bhattacharya, D. Pati, and Y. Yang. "Bayesian fractional posteriors," The Annals of Statistics, vol. 47, no. 1, 2019.
[87] Hall and Horowitz, "Nonparametric methods for inference in the presence of instrumental variables," " The Anmals of Statistics, vol. 33, no. 6, 2005.
[88] J. Rousseau and B. Szabo, "Asymptotic behaviour of the empirical Bayes posteriors associated maximum marginal likelihood estimator," The Annals of Statistics, vol. 45, no. 2, 2017.
[89] S. Särkkä, "State space representation of gaussian processes," Aalto University, Espoo, Finland. Pg. pp. 22-23, 2013.
[90] J. L. Horowitz, "Specification testing in nonparametric instrumental variable estimation," " Journal of Econometrics, vol. 167, no. 2, pp. 383-396, Apr. 2012. [Online]. Available:
1002041
[91] F. Liang, Q. Li, and L. Zhou, "Baycsian neural networks for selection of drug sensitive genes." "
Journal of the American Statistical Association, vol. 113, 523, pp. 955-972, 2018, pMID: 3 31354179, [Online]. Available: https://doi.org/10 10000000012222 (2011)12114990000 444.2901777. 1409112
[92] B. Bauer and M. Kohler, "On deep learning as a remedy for the curse of dimensionality in nonparametric regression." The Annals of Statistics, vol. 47, no. 4, 2019.
[93] N. M. Davies, S. von Hinke Kessler Scholder, H. Farbmacher, S. Burgess, F. Windmeijer, and G. D. Smith, "The many weak instruments problem and Mendelian randomization," Statistics in Medicine, vol. 34, no. 3, pp. 454-468, 2015.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes], [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Discussions of limitations interwoven in the text. E.g., we discussed the limitation of the model selection result at the end of Section 5.2, and cautioned about the limitation of model-based uncertainty quantification at the end of Section 7.
(c) Did you discuss any potential negative societal impacts of your work? [No] We do not foresee any potential negative societal impacts, except possible misuses of our method in violation of its core assumptions. As we have cautioned about such issues in the text (see above), we do not believe an additional, dedicated discussion is still necessary.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] All thcoretical results have their asssumptions in the statement.
15

---

(b) Did you include complete proofs of all theoretical results? [Yes] Proofs are in appendix and always referenced in the main text.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix H.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] All tables and figures have error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix H, the end of each "experiment setup" paragraph.
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All assets have creators cited at their first mention.
(b) Did you mention the license of the assets? [Yes] In the supplemental material. URL? [Yes]
(c) Did you include any assets either in the supplemental material Our code is included in the suplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A] The datasets used are synthetic, and do not contain personal information.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] The datasets used are synthetic and do not contain such content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We did not use crowdsourcing, or conduct research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "xD8_m8WA23r", "reviewer": "Reviewer_AFbo", "summary": "The paper proposes a Gaussian-process-based IV estimation procedure. In a nonparametric IV regression (NPIV) problem, y = f(x) + e, E[e|z]=0, the traditional approach to consists of two stages: estimate x' = E[x|z] (stage 1), then regression y on x' (stage 2). The kernel IV paper [27] proposed to used two pre-specified RKHS for each stage. Here the paper proposed to estimate stage 1 with a procedure based on sampling from the Gaussian process. Concretely, an RKHS is learned for stage 1 instead of being pre-specified. The advantage of such a procedure is to avoid minimax formulation of the problem and provide an inferential procedure due to the Bayesian nature of the method.", "questions": "Minor technical questions:\n\n1. In Assumption 2.2, does a smaller b correspond to a smaller (smoother) function space or the opposite?\n\n2. In Line 5 of Algorithm 1, is the estimated $\\tilde k_z$ an estimate for the latent space kernel $\\bar k_z$ or $k_z$?", "limitations": "\n1. Citations of a few closely related works are outdated, e.g., 5, 20, 27. These works are published in neurips or other conferences and yet only arxiv versions of these works are cited.\n\n2. Overall the presentation of the paper is very dense. It could improve the paper presentation if the authors provide a guideline for the theorems and examples presented in the revised version of the paper.\n\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\nThe paper provides an extensive overview of related works and rigorous theoretical derivation. The work is original; inferential methods for the nonparametric IV problem are new in the related literature as far as I know. The proposed method is neat and enjoys good theoretical guarantees. \n\nWeakness\n\nThe presentation of the paper is a bit dense and could be improved by including an overview of the results in the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WmKaZyMOtMe", "reviewer": "Reviewer_PEsV", "summary": "The authors proposed to perform the first-stage instrument learning via an adaptive regression on random draws from a GP prior. And they use kernelized IV  methods to estimate the second stage casual effect functions. They show that the algorithm has faster convergence rates and avoids expensive minimax optimization procedures. For kernel first-stage models, the algorithm can also be applied to evaluate quasi-posterior and thus provide uncertainty quantification.", "questions": "I didn’t check the technical details. But should the condition of $n_1$ in line 268 be $n_1^{-(b+2p)/(b+2p+1)}$?\n\nThe results of the paper hinge on the assumption that $f_0$ can be characterized by a RKHS $\\mathcal H$. Can the authors elaborate more on how practical is this assumption and what are the consequences if the model is mis-specified or the assumption is violated?\n", "limitations": "The authors have included discussions on limitations and future directions in their paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The presentation of the paper is good, with clear explanations of the intuitions behind the conditions and assumptions. The algorithm itself is easy to implement and efficiently distills informative structures of the instruments. The first-stage can accommodate flexible ML methods. It also enables fast quasi-Bayesian uncertainty quantification with improve flexibility. They also provide extensive discussion on the connection to related work.\n\nThe content of the paper is very compact due to page limits. One may wish to see more on the simulation study in the main manuscript. It would be interesting see low-dimensional sparsity structure design.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HF1NjKLcAoc", "reviewer": "Reviewer_Kxn5", "summary": "This paper proposed an instrumental variable regression algorithm, which authors claimed to allow a faster convergence and the adaptivity to the dimensionality of informative latent features. This paper also presented extensive theoretical analysis to analyze the algorithm and proposed to bridge the notable gap in the literature.", "questions": "I have some specific questions on the experiments:\n1. Why would you choose a randomly initialized NN as one of the baseline? Could you be more specific on \"linear\"? \n2. Have you compared the time consumption of the proposed algorithm versus the baselines of your choice in different experiments? I am not sure if I see that being addressed in the main paper and appendix.", "limitations": "Authors adequately presented the assumptions of the algorithm and all related analysis, and I do not see potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of this paper is on proposing the simple yet faster converging algorithm, and it is well-supported by theoretical analysis and numerical experiments. Authors provided an extensive literature review and paved way of a comprehensive understanding towards literature for audience even not familiar with the specific sub-field. The presentation of this paper is very well organized with detailed explanations in steps on the motivation and design of the algorithm. It is also notable that the appendix is very well organized with detailed supplementary materials covering both theoretical analysis and experiments. The weakness is not obvious to me. But, I'd like to point out that I am not an expert in this sub-field. The only major concern that I have is that there might lack significant novelty of this work comparing with existing literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BmmyfXPGIm1", "reviewer": "Reviewer_Vtb2", "summary": "Fast and provably adaptive instrument learning, through kernel learning with black-box ML models.", "questions": "Can you elaborate how this work is not just simply applying the \"orthogonal ML\" framework?", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper applies newly developed \"orthogonal ML\" techniques.\n\nHowever, the contribution seems rather incremental.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ASqPwz2CRo7", "reviewer": "Reviewer_bZoU", "summary": "The paper proposes an instrument learning approach for non-linear instrumental variable regression with high dimensional instruments.\nState-of-the-art IV estimators with fast convergence rates generally rely on computationally costly minimax procedures. However, assuming that the true function of interest lies in an RKHS, the kernel versions of these estimators often allow for closed form solutions at the cost of using less flexible RKHS models as adversarial/instrument functions compared to e.g. neural networks or random forests. The authors combine the simplicity of the kernel versions of these estimators with the predictive power of modern machine learning models by first learning a kernel whose features are represented by arbitrary ML models and then using this kernel with the kernel-based closed form solution of the SOTA minimax approaches. The paper provides theoretical guarantees and fast convergence rates similar to the ones of the minimax approaches.", "questions": "I appreciated if the authors clarified the following points for me:\n- Why does regressing $f(x)$ on $z$ provide denoised samples from $\\mathcal{GP}(0,k_z)$?\n- Why does this procedure focus on the informative latent features?\n- How restrictive is the assumption of $f_0$ being an RKHS function in practice?\n", "limitations": "I do not see any immediate potential for any negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As a disclaimer, I was assigned to this paper as an additional reviewer very shortly before the start of the rebuttal period. In the interest of not further delaying the process I was only able to spend a short amount of time on preparing the review and therefore might have overlooked some important aspects or errors. I have not checked the proofs and my review is based on the assumption that the theorems are correct.\n\n\nStrengths\n- The problem the paper solves is a very interesting and relevant one and to the best of my knowledge the approach is new\n- While there is a growing literature on addressing conditional moment restrictions via minimax formulations, approaches like the proposed one that avoid the rather involved saddle-point optimization problems seem to be very useful and well-motivated\n- While the final estimator builds on the kernel-kernel version of the minimax formulation of [19], the kernel over the instrumental variables is formed from arbitrary ML models and thus the proposed approach can be combined with (future) SOTA ML models tailored to the problem at hand\n- Large parts of the paper are well and clearly written\n- The appendix provides a lot of useful additional information \n\n\nWeaknesses / Possible improvements\n\n- The clarity of presentation could be improved, from reading the text it was not obvious to me that the proposed algorithm is the one described in Algorithm 1\n- Without being familiar with the work in reference [19] and their closed form solution for the kernel version, it might be difficult to understand what problem the paper tries to solve and how the final estimator is obtained. The explanation in Appendix F focusses on the more involved estimator based on [26]. I think it would be helful also to discuss the simpler implementation of [19].\n- I did not understand why regressing $f(x)$ on $z$ provides denoised samples from $\\mathcal{GP}(0,k_z)$, however this is a crucial part of the method, so I think it should be explained more clearly\n- Connected to the previous point, it is also not clear to me why this procedure focusses on the most informative latent features\n- The paper has a lot of content and might feel a bit too dense. To some extent the methods based on [19] and [26] seem to be independent and the paper might be easier to understand if one of the methods were moved to the appendix (or to a second paper) but this impression might be owed to the limited time I have spent on the paper\n- The experiments show that the method performs sometimes better and sometimes on par with the minimax approach of [19]. I think it would be insightful to report e.g. the computation time, to see if the supposedly simpler computations of the proposed method actually result in faster estimation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept