Scalable Quasi-Bayesian Inference for Instrumental Variable Regression
Authors: Ziyu Wang
Yuhao Zhou
Tongzheng Ren
Jun Zhu
Venue: NeurIPS 2021 Poster (2021)
Keywords: probabilistic methods
causal inference
uncertainty quantification
instrumental variable
approximate inference

Abstract: Recent years have witnessed an upsurge of interest in employing flexible machine learning models for instrumental variable (IV) regression, but the development of uncertainty quantification methodology is still lacking.  In this work we present a scalable quasi-Bayesian procedure for IV regression, building upon the recently developed kernelized IV models.  Contrary to Bayesian modeling for IV, our approach does not require additional assumptions on the data generating process, and leads to a scalable approximate inference algorithm with time cost comparable to the corresponding point estimation methods.  Our algorithm can be further extended to work with neural network models.  We analyze the theoretical properties of the proposed quasi-posterior, and demonstrate through empirical evaluation the competitive performance of our method.

Scalable Quasi-Bayesian Inference for Instrumental Variable Regression
Ziyu Wang',, Yuhao Zhou!., Tongzheng Ren', Jun Zhu Deppt. of Comp. Sci. and Tech., BNRist Center, State Key Lab for Intell. Tech. & Sys., Institute for AI, Tsinghua- Boschh Joint Center for ML, Tsinghua University
Department of Computer Science, UT Austin {wzy196, yuhaoz ce)ammill....... com, tongzheng@utexas.cm edu, dcszjatainghua.er edu.
Abstract
Recent years have witnessed an upsurge of interest in employing flexible machine learning models for instrumental variable (IV) regression, but the development of uncertainty quantification methodology is still lacking. In this work we present a scalable quasi- Bass--Bayesian procedure for IV regression, building upon the recently developed kernelized IV models. Contrary to Bayesian modeling for IV, our ap- proach does not require additional assumptions on the data generating process, and leads to scalable approximate inference algorithm with time cost compara- ble to the corresponding point estimation methods. Our algorithm can be further extended to work with neural network models. We analyze the theoretical proper- ties the proposed quasi-posterior, and demonstrate through empirical evaluation the competitive performance of our method.
Introduction
Instrumental variable (IV) regression is standard approach for estimating causal effect from con- founded observational data. In the presence of confounding, any regression method estimating E(y x) cannot recover the causal relation ft between the outcome y and the treatment x, since the residual
fx() ( is correlated with x due to the unobserved confounders. IV regression enables identification the causal effect through the introduction of instruments, variables z that
are known to influence only through
IV regression widely used in economics M. epidemiology and clinical research 33, but mod- eling nonlinear effect in IV regression can be challenging. Recent years have seen great development in adopting modern machine learning models for IV regression 8). However, there is still lack of uncertainty quantification measures for these flexible IV models. Uncertainty quantification is es- pecially important for IV analysis, since unlike standard supervised learning scenarios, we do not have (unconfounded) validation data, from which we could deduce the error pattern of the estimated model. Moreover, the instrument of choice may weak, meaning that it only provides limited information for in such cases point estimators suffer from high variance [9] This problem is ex- acerbated in the nonparametric setting, where IV estimation is usually an ill-posed inverse problem, where instruments provide vanishing information for the higher-order nonlinear effects [1O].
The IV setting brings unique challenges for uncertainty quantification. For example, while it is natural to consider Bayesian approach, specification of the likelihood requires knowledge of the entire data generating process, which is typically unavailable in IV regression. Consequently, most, if not all, existing work on Bayesian IV (1H15) assumes the following data generating process:
'zW and YZ contribute equally. 'JZ is the corresponding author. An extended version is available http:///aaa...orr///////......0. wwt.or/g/.///..../////
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

= =((z) + u1 f(x) + Uz, where (u1, u2) are correlated and independent of z. These methods then conduct posterior inference on g. f as well as the generative model for the unobserved confounders (n1 u2). However, the additive error in x is an unnecessary assumption for most point estimation procedures, and is difficult to check in high dimensions. Furthermore, the need to model the generating process of (n, U2) introduces an extra risk of model misspecification, and Bayesian inference on the generative model is computationally expensive, especially on complex high-dimensional datasets. None of these issues present if only point estimation is needed.
For the above reasons, it is appealing to turn to an alternative quasi-Bayesian Bayesian approach [ 18 Mass--Baesiinn analysis views IV estimation as generalized dure, and defines the quasi-posterion as a Gibbs distribution constructed from chosen prior and violation of the moment constraints. It does not require full knowledge of the data generating pro- cess, and thus does not suffer from the aforementioned drawbacks of Bayesian inference. However, computation of the quasi-posterion is non-trivial, as its density contains a conditional expectation term Byy f(x 2). which itself needs to be estimated from data. This makes both approximate inference and theoretical analysis difficult. So far, quasi-Bayesian analysis for nonparametric IV is only developed on classical models such as wavelet basis [1 [8] or Nadaraya- Watson smoothing [9]. while adoption of flexible machine learning models such kernel machines or neural net- works remains an open challenge. Moreover, numerical study has been limited in previous work, so little is known about the empirical performance of the quasi-Bayesian approach.
sethoddof-momeeee (GMM) proce-
In this work, we present novel cent development in kernelized IV models 7 [0]. We employ Gaussian process (GP) prior and construct a quasi-likelihood using a kernel conditional expectation estimator. We establish theo- retical properties of the resultant quasi-posterior, proving its consistency and showing that it may quantify instrument strength. Furthermore, inspired from the minimax formulation of estimation [5 3 20, 21), we design a principled approximate inference algorithm using random feature expan- sion and a novel adaptation of the "randomized prior trick" [22]. The algorithm has the form of stochastic gradient descent-ascent and is thus scalable and easy to implement. It can also be adapted to work with flexible neural network models, in which case its behavior can be formally justified by analyzing the neural networks in the kernel regime. Empirical evaluation shows that the pro- posed method produces informative uncertainty estimates, scales to high-dimensional and complex nonlinear problems, and is especially advantageous when the instrument is weak.
procedure for IV regression, building upon the re-
The rest of the paper is organized as follows: In Section 2 we set up the problem. We then derive the quasi-posterior and analyze its theoretical properties in Section 3 and present the approximate inference algorithm in Section 4 Section 55) reviews related work, and Section (6 presents numerical studies. Finally, we discuss conclusion and future work in Section 7
Notations and Setup
Notations We use boldface (x. y, 2) to represent random variables on the space X' x x v/x x 2, reg-
ular font (z, 2) to denote deterministic values. [n] denotes the set 21, .n] { Vi, 2)
i E [n|]] indicates the training data. We use the notations X := (II...., ,n)) E X", likewise for Y, Z. For finiteddimensional vectors 0.0' E RT we
f(X) : (f(x)))..,,((
use [10||2, 10) to denote the Euclidean norm and inner product, respectively. For any operator
A : H1 H2 between Hilbert spaces H₃ and Hz, we denote its adjoint by A : H2 + Hy. When
H1 = H2 and À E R, we use the notation A := A + AI for simplicity.
IV regression Denote the treatment and response variables as x, y. the instrument as z, and the true structural function of interest ft. Consider the data generating process y =f((x) + u,
where the unobserved satisfies E( 2) 0, but may correlate with x. Then jt satisfies
Fyy (x(x 2) = 0 a.s. [P(dz)].
(1)
where P denotes the data distribution. This conditional moment restriction (CMR) formulation is the standard definition in literature [e.g.. 23 [8] and is used the recent work on machine learning models for IV. It connects to GMM as (I can be viewed as continuum of generalized moment constraints. Note that ( does not place any structural constraint on the conditional distribution p(x z), such as additive noise; hence, it does not require full knowledge of the data generating
2

---

process. Also, as discussed in, e.g. Hartford et al. [4], the setup can also be extended to incorporate observed confounders v. by including v in both x and z.
Kernelized IV and a dual view Let H, I be suitably chosen function spaces on X, 2, respec- tively. The generalized moment constraint || motivates the use of the following objective
min SEH
C(f) =-)==da) de, + XR(f),
(2)
H where En .H -+ I is an empirical approximation the conditional expectation operator (x((x) z = -). b is an estimator of Ely = -). {dn) is sequence of suitable (semi-)norm on I, and Q H + R is regularization term.
When H. I are reproducing kernel Hilbert spaces (RKHS) with corresponding kernels k2s hg, it is natural set (f(f) 2 Il and define ó the result of kemeel ridge regression on y with respect to z. In this case E,, can be defined with the empirical kernel conditional expectation operator: let C.z (k(, -) @ k<(,,,),, -)), C2= = B(k((z,) @ k(z,,)) +)). Assuming E maps all f E H E I,
we have 24
C2 - Ceal, ASEH.
This motivates the use of E. == Oz where C.2 == E k(1, 0 k(zz,) .). Czr is de- fined similarly, and E is regularization hyperparameter The choice of dn is flexible; the dual IV formulation uses d (9) 1 Σj= 2(2))2 + Ello11 (g, C22,p)) Introducing the evaluation operator S2 I R", ;5, S29 == (g(2)) ......................... (Zn)) have b = Öz S.Y m and
S/Y
C(f) 0/2/1 Ózal
(3)
2
7
I
(f(z))- v;)o(())))))))))))))))))))})))) y(2,)²
X 2 11
max gEI
(4)
where 44 holds because of the Fenchel duality fu²² sup (we -02) and the equality = E(E(u(x, y) z)g(z)); see [25
21
26
The dual formulation (4) circum-
vents the need compute É,, directly, operator between the typically infinite dimrettoadd spaces H. and I, and leads to a scalable estimation procedure based on stochastic gradient descent-ascent (SGDA). It can also be generalized to work with deep neural networks (DNNs) instead of kernel machines, by replacing the RKHS regularizer with a suitable regularizer for DNNs, although theo- retical analysis for the resulted algorithm requires separate effort [8] 26]. As we shall see, (4) will also enable the construction a scalable approximate inference algorithm.
3 Quasi-Bayesian Analysis of Dual IV
Introduce the notations Sg H. + R", S2f == n S Define x= nd, no. Now we can re-express in an equivalent form
so that Ć.. S C.s
-1000-00000000000000000000--0-00 + 1
-1 (((0)
E =())= = -( -
(5)
where L := S C. St is a linear map from R to R", and thus can be identified with n x n matrix. Since the first term above is equivalent to the log density of the multivariate normal distribution NYY) \L we can view (5) as the objective of kernel ridge regression problem, which has a data-dependent noise covariance XL- 1 The connection hetween kernel ridge regression and Gaussian process regression 28 thus motivates the use of the quasi-posterior
With an abuse of notation, & refers to the reproducing kernel of the corresponding RKHS (ki for H or kt for Z) whenever the demotation is clear.
'Here we assume the invertibility of L for brevity. Alternatively, observe that 5 defines linear inverse problem with the finite-dimensian dimensional observation operator f + (Lf(X) and noise variance X- I, and we can follow (27 Chapter 6J to derive the same
3

---

[I D( defined through the following Radon-Nikodye derivative w.r.. the standard GP prior GP(O, kz):
(6) Note that contrary to standard Bayesian modeling, do not assume Y (X) ) is part of the true data generating process, nor does the theoretical analysis below rely on it. Instead, the quasi-posterior (6) should be interpreted as Gibbs distribution which trades off between the properly scaled evidence X In (Én,f (Én
dll(- dlI(-11!"))
- -
(1)
- Th
dIT
b), which characterizes the estimated violation of the GMM constraint ( and our prior belief [I(df). This trade-off is most clear from the well-known
variational characterization of the Gibbs distribution (29).
IT(- |(()) arg min B (X nd, (F.. 8)] + KL IT..
Nonetheless, the fictitious data generating process f ~ GP(O, k), ~ A is useful for deriving the quasi- posterio, since conditional distribution Pac (fc YY coincides with (6) 27
Chapter 6]. In the probability space of this fictitious data generating process, for any finite set of test inputs w., we have
Pie Pie (J () (\ (..0....... K..
Ka. Kzz Kan
where K(,) denote the corresponding Gram matrices with subscript * denoting x. and x denoting X (so, c.g., K,z == k( X)). Thus by the Gaussian conditioning formula, we have
[If( | D( Pno(f(x.) Y) (),, where
(8)
m = K. (Al + LKzz)_LLY,
(9)
S = K.. - L(Al + Kan L)=L)-KKar
(10)
L (2 + +vi)'
(11)
In the above Kgz == k(2, Z) denotes the Gram matrix, and (IT) follows from the definition L n S ( + D1)-1S;; the Woodbury identity, and the observation that SyS; Kex-
Theoretical Analysis For the
II(- D(m)) to be useful measure of uncertainty, needs to satisfy the following informal criteria: as - oo, we expect
(CI) II. D(*)) will exclude incorrect solutions;
(C2) In cases of non-identification, IT(- D(x)) will not exclude any valid solution in the model.
In the following, formalize these criteria, and demonstrate that with appropriately chosen hyper- parameters, the quasi posstriot satisfies both criteria. We will work with the following assumptions:
Assumption 3.1. The restriction of the conditional expectation operator f + EE((x) = on H. denoted as E, has its image contained in I. E H + I is bounded.
Assumption i. intuitive, and is also slightly more general than some previous work 20 26 that require the conditional expectation operator to be bounded on the entire hypothesis space, which typically corresponds the "sample space" of the GP prior in our setting, and is much larger than H (see Appendix(A) Nonetheless, we note that Hypothesis 4 in Singh et al. [7] may be more general, although they impose extra smoothness assumptions on E.
Assumption 3.2. The true structural functiionn ft() is such that, there exists sequence [Tm : m N} satisfying Tm + O, and
m
inf f MA f..1s5m
llft 1 log [Ifll S Fm.))
(12)
where || denotes the sup norm.
Assumption 3.2 requires that ft can be well approximated in H;: this is more general than previous
work [e.g.. 20 that require f E H. but typical the GP literature 30, 31. The sequence
{Tm) is determined by the complexity of H and its ability for approximating ft, and is usually the optimal posterior contraction rate for Gaussian process regression on the unconfounded dataset {Z,, (xx)) + e.) : e [m]] (3I]

---

Assumption 3.3. X' and Z are Polish spaces. The kernels k k, are continuous, supnex k(z, x) + supsez 2) S K², and Mercer's representations (32) of ki and kig exist. The random variable f is sub-exponential.
Assumption 3.3 imposes technical conditions frequently assumed in literature [e.g.. 33
The re-
quirements on the kernels can be satisfied by e.g. continuous kernels on compact subsets of Rd.
Given the assumptions above, we characterize (QT) by showing that + oo, the posterior places vanishing mass on the region of functions that violates the GMM constraints ①I. Concretely,
Theorem 3.1 (Proof in Appendix B). There exist a constant M > 0 depending on the data distribu- tion and the kernels of choice, and sufficiently slowly growing sequence (%a) always have Inn log log n) such that when taking x n 1 i min ( (T - n n-2
(e.g., we can
7a), it has
Ip (x(( -y
MTiva D(m)) -
(13)
Remark 3.1. The above result should primarily be interpreted as consistency result, although also provides a posterior contraction rate (34 in the order of Typl in the semi-norm f H>
|E (lL.))(((dd))) where } is defined in Assumption 3.2. As an example, suppose the regular-
ity of fl is similar to the Matémn-1/2 RKHS; then for suitable kernels we have V-vai = O(n 1/*), see Remark A.3
The rate is suboptimal, and can be immediately improved if we impose further regularity assumption on kg if we assume the critical radius of the local Rademacher complexity [35] of the unit-norm ball
I is 5n x Tn n 20811, a rate of On 2isto) for the semi-norm can be established following similar arguments. This is still worse than (26). which provides the rate ((max 5, }) for what corresponds to our posterior mean estimator. Nonetheless, their result is also generally suboptimal, because the choice of 6,, x Tn is actually suboptimal when the IV regression problem ill-posed. To date, minimax optimal rates have only been established under additional assumptions on the relations between the conditional expectation operator and the model for f. Our extended version establishes such results, for both this semi-nome and more intuitive norms such (P(dx)))
Remark 3.2. The use of an increasing À = nı is a technical artifact due to our minimal assumptions on kg. is common impose extra regularization in nonparametric IV, due the need estimate E from data [19]. and, in the Basi-BByesiann setting. also due to the additional technical challenges [ However, this is not necessary for our method if additional assumptions are imposed, as we show in the extended version. In practice, our hyperparameter selection procedure does not produce with significant growth.
The following proposition characterizes (Q2)
Proposition 3.1 (Proof in Appendix B The scaled log quasi-likelihood estimate En(s) d2 [Ens 6) satisfies
([vr: vo > 0, lim Prt (lex()) n0000
-y |2) > -
6) =0}) 1.
In words, for II-almost every f. the log quasi-likelihood estimate scaled by n converges to ( (E(f(x) - y z)²). which characterizes the violation of (.. While the restriction to the probability-I subset can be concerning in the nonparametrie setting [34] the proof only depends on satisfying similar approximability conditions to ft: see (45).
Remark 3.3. To understand why the proposition characterizes (Q2). suppose there are multiple f that satisfy ! Then all of them will eventually have significantly higher log likelihood than functions violating (T) the difference is O(x-'n) If these functions have a similar level of regularity in the sense that their concentration functions, which is the right-hand side of (12), have the same asymptotics as + O, Borell's inequality [e.g., 34 Proposition 11.17] implies that the posterior mass of small -norm balls around them will also have the same asymptotics.
Finally, we provide the following (over- )simplified example, which compares the behavior of the quasi-posterior and bootstrap in the context of ((p): Example 3.1. Suppose z is completely non- -n-ioorrmtaive so that E(f(x) z) = Ex(x) for all f; and suppose the estimated conditional expectation F., is sufficiently accurate, so that we replace it
5

---

with E In this case bootstrap on (3) will always return the point estimator f = 0 due to the non- zero regularization on fl while the quasi-posterior behaves like the prior, correctly reflecting the complete lack of evidence in data.
While this example is oversimplified, and in practice the estimation error of E plays an important role, it is known that bootstrap uncertainty estimates can be problematic given weak IVs 33638]. We also observe similar failures for bootstrap in the experiments (see Section 6.1
4 Scalable Approximate Inference via a Randomized Prior Trick
We now turn to approximate inference with parametric models such random feature expansion or wide NNs. Scalable inference for the IV quasi-posterior appears difficult, since for any f. computing the quasi-likelihood involves computing E.ff which in turn requires either inverting an n x n Gram matrix, judging from (5) and (TT), or solving an optimization problem specific to f from (4) Nonetheless, we show that it is possible, by extending the "randomized prior" trick for Gaussian with an optimization formulation as (
process regression [2 to work with
Our algorithm works with random feature models. A random feature model for k2 approximates (z =') 2 ke.mm,, (2, = m xz) (2,m(=)), where &s.m takes value in Rm. Then the map P Vm )=,m() =: (p) parameterizes approximate RKHS H; and for all > 0, the random function ((:5p), where yp (O, cf), is distributed GP(O, ck.)k.m. The notations related to kn are similar and thus omitted. Now we can state the objective function:
Proposition 4.1 (Proof in Appendix C.1.. Let @u NO0,, - ( 1), ~ N((0,)), Vss N( A). Then the optima 0 of
())((2;))) ( Σ (U BERTH GE
((21))
min max
2 11b
2 10 00 (14)
i=1
parameterizes a random function which follows the quasi distribution nels are replaced by the random feature approximations.
C where the ker-
Given the above proposition, we can sample from the random feature-approximated quasi-posterii by solving (T4) with stochastic gradient descent-ascent; the approximation errors will be analyzed
in the following. The objective (14) is closely related to (4); as we show in Appendix C.I.I it is equivalent to
min max SER gEZ
(((x:))()(<=)) ())((.)) g(2,)²
V/ 11g
goll
+
A 215 soll 2
(15)
2
2
which differs from 44) only in the regularizers: instead of regularizing the norm of f and g, (5) encourages the functions to stay close to randomly sampled anchors [39]. Alternativeyy. we can view (15) as perturbing the point estimator (4), so that it has covariance matching that of the quasi-posterior. A similar relation is also observed in the original randomized prior trick, which transforms GP regression to the optimization problem min min Σn (f;;((;; 0.)² + A|Iʃ follž- In both cases, the resultant algorithm for approximate inference the same time complexity as ensemble training for point estimation.
While the algorithm can be directly applied neural network models in 22 we follow [40] and modify the objective, to account for the difference between the neural tangent kernel (NTK) 41 of wide neural network architecture and the NNGP kernel of the corresponding indtnite-width Bayesian neural network (42) Concretely, we modify (14) as
(fo(z.) - (z)) [o(z))² 2
X 2 00
min max
/ 110 20112 +
(16)
1=1
where 00(2) == g(2; x)- - g(2; Φo) + 00(2), (x) = v 3o
0g
00 (=))
Jo
'This setting could be realistic in the sample splitting setup considered in 7 Note that as long as we have finite samples for the estimation of (the "second stage"), we still need have X > O.

---

and @ denotes the initial value of o and do ~N I) is a set of randomly initialized NN parame- ters independent of øo; and fe is defined similarly.
We only give a formal justification for the modification, under the assumption that the NNs remain Og(=)
random feature model with the in the kernel regime throughout training, so that g(2; o) - g Φo) = (Ø
- Øo:
Ue 0o)2 47 Thus
for the purpose of analyzing g(:: - g( Φu), we can view
parameterization Φ (Ø, [g(=) de /.0/2. Thus by the argument in Appendix C.1.1 we can show that the weight regularizer | 2 222 is equivalent to (1g(:: 9(::00)l:;
1l00 Tollž where I is 10072. Similar arguments can be made for fo and fo. Consequently, (16) is equivalent to an instance of (15) with H2, I defined by the NTKs.
determined by the NTK (g, 2') == (g(=)) do: de
2n(=")
Implementation details for the algorithm, including hyperparameter selection, are discussed in Ap- pendix[D
Convergence analysis We now complete the analysis of the inference algorithm, by showing that for any fixed set of test points I., SGDA can approximate the marginal distribution [I.f(=)) D( arbitrarily well given a sufficient computational budget. This implies that the approximate posterior is good enough for prediction purposes.
We place several mild assumptions on the random feature model, listed in Appendix C.2 they are
satisfied by common approximations such as the random Fourier features (48). The SGDA algorithm is described in detail in Appendix C.4 Under this setup, we have
Proposition 4.2 (Proof in Appendix C.5). Fix D(u) and X, V/ >> 0. Then there exist sequence of choices of m and SGDA step-size schemes, such that for any E N, we have
sup EXI max( [pmm - Hmm
- 2a.ll || Fm 3.
In the above, Pm, Sm denote the mean and covariance of the approximate marginal posterior for (xx.). ). μ. S correspond to the true posterior, ||- denotes the Frobenius norm, and the convergence in probability is defined with respect to the sampling of random feature basis.
Related Work
Quasi-Bayesian analysis for GMM estimation problems was first developed in 49 50 16 which provided theoretical analysis for parametric models. The use of the quasi-posterior motivated from the maximum entropy principle, based on which similar ideas have been developed in the machine learning literature [51533.. Alternatively, it can also be obtained as a non-informative limit first analyzed the use of nonparametric models for quasi-Bayesian IV; no numerical study was presented. Closer to our work is 19] which constructed
of certain Bayesian posteriors [54] 17 18
a quasi-posterior using Nadaraya- Watson smoothing as É,, While similar in form to our method, the smoothing approach relies stronger assumptions, especially for high-dimensional data; 7 Appendix A.2.1] which compared the corresponding point estimators. Their approach is also harder to scale to large datasets.
Our quasi-Bayesian procedure builds upon the kernelized IV models 7
20 and the dual formula- proposes a similar method. Although it was motivated differently as a kernelized two-stage least squares (2SLS) esti- mator, its objective is asymptotically equivalent to 33); however, the slight difference in regulariza- tion prevents the use estimation procedures similar to (4). as we discuss in Appendix [C.14] An alternative kernel-basse based estimator is proposed in [55 [5]; ; in particular, [55] also includes Gaussian process construction, but for the different purpose of computing a leave-one-out validation statistics.
tion of IV regression [6] 20 21 26). The formulation (3)-44) is from 20
26 ?
Other recent work ML. models for IV include 4 57 58) It remains interesting future work to develop scalable quasi- -aysiaaa procedures for these methods, although the mean estimator derived from our quasi-posterior implementation has competitive performance.
59 studies semi-parametric IV estimation using an exponentially tilted empirical likelihood (ETEL) estimator. ETEL also connect to Bayesian methods. However, similar to the quasi-Bayesian
'The same linearization assumption has been employed in 40 of the linearization error in various settings similar to ours.
45
We refer readers to 46 21 for analysis
7

---

approach, the need to estimate the conditional moment restrictions complicates the understanding of its frequentist behavior, as well the design of scalable inference algorithms; in their case, especially unclear if the empirical likelihood can be computed in sublinear time. Other approaches for uncertainty quantification include Bayesian inference and bootstrap. We have discussed previ- ous works on Bayesian IV and their limitations in Section [ Bootstrap is typically justified in the asymptotic regime, which does not cover many scenarios where uncertainty most needed; this is different from the quasi-Bayesian approach which can always be justified through (7). Moreover, standard bootstrap inference on 2SLS known to be unreliable when the instrument strength is weak 38]; while remedies exist (e.g.. [38), they heavily rely the linearity and additive noise assumptions, and are thus difficult to generalize to the nonlinear setting we are interested in. As the kemeliied IV methods generalize 2SLS, we expect similar issues exist in our setting.
6
Experiments
Code reproduce the experiments is available httpps//ggtthub..co//rtataaa-ata
6.1 1D Simulation
We first experiment on variety of ID synthetic datasets, constructed by modifying the setup to incorporate a nonlinear first stage, in way similar to 60
== sigmoid(w), x == sigmoid aw + (1 (1-a)u') yi ~ (f 2u, 0.1),
Va
where (u, u²)) normal random variables with unit variance and a correlation of 0.5, NO independent of (u, w/), a a parameter controlling the instrument strength, and fo is constructed from the sine, step, abs or linear function. We choose N E {200, 1000} and a {0.05, 0.5).
Our baselines include BayesIV [5] a state-of-the-art Bayesian model based on B-splines and Dirichlet process mixture; also include bootstrap 2SLS with ridge regularization, either ap- plied directly to the input features (Linear), on their polynomial expansion (Poly), or on the same kernelized models (KIV) ours Hyperparameter for the kernelized IV methods are selected by cross validation based on the observable first-stage and second-stage losses as in previous work 7 20); see Appendix D.I For kernels we choose the RBF and Matérn kernels, although results for Matérn kernels are deferred to appendix for brevity. See Appendix D.3 for the detailed setup.
bayestv
linear
paly-3
rbf
qb-rbf
1.00
1.00
Norma lire MSE
Nommal lood MSE
Coverage
0.75
Coverage
0.75
10-2
0.25
0.25
200
1000
200
1000
200
1000
200
1000
0.05)
(o=00.05) 05)
Figure 1: Test MSE and CI coverage on the s sine dataset. The left two plots correspond to = 0.05, while the right two correspond to a 0.5. bs denotes bootstrap, qb denotes quasi-Bayesian
Normalized MSE and coverage rate of 95% credible intervals (CT) on the sine datasets are plotted in Figure We report results 20 independently generated datasets. As we can see, quasi-Bayesian inference provides the most reliable uncertainty estimates, and is especially advantageous in the weak IV setting (a 0.05). While its CI can be conservative, we note that it is still informative, and properly reflects the sample size and instrument strength; see Appendix D.4 for visualizations. These results connect to previous work showing (in different setting) that the radius of credible set produced by a GP posterior can have the correct order of magnitude [61]
We report the average run time in Table We method is also more scalable compared with the standard Bayesian baseline, as it avoids the costly modeling of the joint residual distribution.
that in addition to the improved flexibility,
'We do not compare with 19 since their source code is unavailable. Note that their smoothing moothing-bbssd
method does not scale to large datasets, and there is numerical study the credible interval [9]

---

Table 1: Average run-time (in seconds) in the 1D simulation, for single set of hyperparameters. N/A: does not converge after 20 minutes. For the approximate inference algorithm, we report the average time cost for parallel runs on a single accelerator; see Appendix [D.3 for details.
N
1000
5000
20000
BayesIV (CPU)
655
N/A
N/A
QB (closed-form, CPU)
2.37
49.1
1150
QB (approx. inf., GPU)
10
140
Full results on all datasets, visualizations and additional experiments are deferred Appendix D.4 As summary, (i) on the abs and linear datasets where all modeling assumptions hold, the results are qualitatively similar the sin dataset. Moreover, the over-smoothed RBF kernel appears have similar coverage comparing with the optimal kemel, and follow a similar contraction rate, as the pre- vious work on GP regression 62 suggests. (ii) On the step dataset which violates Assumption 3.2 the
still provides more coverage than the baselines. (iii) Uncertainty estimates pro- duced by the approximate inference algorithm are similar that from the exact quasi-posterior.
6.2 Airline Demand
We now turn to the more challenging demand simulation first proposed by [4]. The dataset simulates scenario where need predict the demand of airline tickets y. as a function of the price and two observed confounders: customer type 8, and time of year 1. The data generating process is
x = == ())( + 25 + u', y y fo(æ, t, + u, fo(x,t, t, s) == 100 + (10 + x)) - (t()) - 2x
where (u, v') are standard normal variables with correlation p. z N 1) independent of (u, u'), and v is nonlinear function whose shape is given in Figure 2 The variable s either varies across {0.....6) 66} (the lower-dimensional setting), or is observed as an MNIST image repre- senting the corresponding digit; the latter case represents the real-world scenario where only high- dimensional surrogates of the true confounder is observed. We only report results for p = 0.5, noting that results using other choices of p have been similar. We use {1000, 10000} for the iver-dimensional setting, and use n 50000 for the image-based setting.
prediction
tual
actuai
Clual
2
10
8 10
(a) lower-dim, N 10a
(b) lower-dim, N 104
(c) image setup
Figure 2: Approximate cousi-posteriors in the demand simulation. We plot a cross-section by fixing 8, x their mean values and varying
We compare method with bootstrap the same model, BayesIV, and bootstrap linear or polynomial models. Performance of other point estimators on this dataset has been reported in 77 20 18), compared with which our method is generally competitive. We implement the dual IV model using both an RBF kernel and DNN models. See Appendix D.5 for details.
We report the test MSE and CI coverage for N = 1000 in Table and visualize all approxi- mate quasi-posteriors for the NN models in Figure 2 As we can see, when implemented with DNNs, our method produces uncertainty estimates with excellent coverage, which also correctly reflects the information available in the dataset: the CI is wider when N smaller, or the high- dimensional experiment where estimation is harder. Bootstrap has noticeably worse performance when N = 1000. Still, it performs well in the (arguably less interesting) large-sample setting, with CI coverage similar to method; see Appendix [.6] This is because this dataset, the total instrument strength is stronger due to the presence of observed confounders, and the NN model is a good fit. Consequently, the asymptotic behavior of bootstrap
be observed when N is large.
Both methods have poorer performances when we switch to the RBF kernel, although the quasi- posterior is still more reliable. We conjecture that both H. and I are misspecified in this case, and

---

thus the credible intervals can only reflect the uncertainty within the prior model. These results show that NN models can be advantageous, which our inference algorithm supports.
The other baselines perform poorly due to their inflexibility; in particular, note that BayesIV uses additive models for both stages (e.g.. f(x f, 8) (x) + f2(() + fs(s)) which do approximate this data generating process well. Full results and visualizations are deferred to Appendix D.6
Table 2: Test normalized MSE, average CI coverage and CI width on the demand dataset with N 1000. Results averaged over 20 trials.
Method
BS-Lincar
BS-Poly
Bayes
BS-RBF
QB-RBF
BS-NN
QB-NN
NMSE
.37 1 .01
.31 + .06
.28 1 004
.17 + .01
.17 001
.06 .03
.04 1 .00
CI Cvg.
.09 1 .01
.15 1 .03
.27 1 06
.45 4 02
.77 1 02
.86 .02
.94 L 00
CI Wid.
.09 + 01
.16 004
.08 = .66
.18 002
.37 0.01
.14
.26 .04
7
Conclusion
In this work we propose scalable quasi-Bayesian procedure for IV regression. We analyze the theoretical properties of the proposed quasi-posterior, and derive a scalable algorithm for approx- imate inference. Empirical evaluations show that the proposed method scales to large and high- dimensional datasets, and can be particularly advantageous when the instrument strength weak.
Beyond IV regression, formulations like (T) also appear in various other problems in causal inference and statistics, discussed in, e.g.. (21): our method be readily applied to these problems. Future work includes extension to more general conditional moment restriction problems with nonlinear constraints 63
Acknowledgement
We thank the anonymous reviewers for their helpful feedback and references. Z.W., Y.Z. and J.Z. were supported by NSFC Projects (Nos. 61620106010, 62061 136001, 61621136008, 62076147, U19B2034, U19A2081, 1, U181 1461), Beijing NSF Project (No. JQ19016), Beijing Academy of Arti- ficial Intelligence (BAAI), Tsinghua-Huawei Joint Research Program, grant from Tsinghua Insti- tute for Guo Qiang. Tiangong Institute for Intelligent Computing, and the NVIDIA NVAIL Program with GPU/DGX Acceleration.
Broader Impact
IV analysis is widely used in social science and clinical research, where it can be highly undesirable to have a systematic error or increased prediction variance on subpopulations which are underrepre- sented in the training dataset, an important first step in detecting such issues, and the quasi-Bayesian approach we take can be preferable since it has a clear interpretation in non-asymptotie setting, and when the instrument strength is weak. However, epistemic uncertainty estimates are only as good as the model allows, and it should not create false sense of complete security. Caution must be taken when employing complex models in scenarios with potential fairness implications.
which the instrument is locally weak. Uncertainty quantification
References
[1] J. D. Angrist and J.-S. Pischke, Mostly harmless econometrics: An empiricist's companion. Princeton university press, 2008
[2] Greenland, "An introduction to instrumental variables for epidemiologists. International Journal of Epidemiology, vol. 29, no. 4, pp. 722-729, 2000.
[3] Cuzick, R. Edwards, and Segnan, "Adjusting for non-compliance and contamination in randomized clinical trials." Statistics in Medicine, vol. 16, no. 9, pp. 1017-1029, 1997.
10

---

[4] J. Hartford, G. Lewis, K. Leyton Leyton-Brown, and M. Taddy, "Deep IV: A flexible approach for counterfactual prediction," in International Conference on Machine Learning. PMLR, 2017, pp. 1414-1423.
[5] G. Lewis and V. Syrgkanis, "Adversarial generalized method of moments," arXiv preprint arXiv:1803.07164, 2018.
[6] A. Bennett, N. Kallus, and T. Schnabel, "Deep generalized method of moments for instrumen- tal variable analysis," arXiv preprint arXiv: /2495, 2019.
[7] R. Singh, M. Sahani, and A. Gretton, "Kernel Instrumental Variable Regression," arXiv:1906.00232 [cs, econ, math, stat], Jul. 2020, arXiv: 1906.00232.
[8] A. Bennett and N. Kallus, "The Variational Method of Moments," arXiv:2012.09422 [cs, econ, math, stat], Dec. 2020, arXiv: 2012.09422.
[9] J. H. Stock, J. H. Wright, and M. Yogo, "A survey of weak instruments and weak identification in generalized method of moments," Journal of Business & Economic Statistics, vol. 20, no. 4, pp. 518-529, 2002.
[10] W. K. Newey, "Nonparametric instrumental variables estimation," " American Economic Re- view, vol. 103, no. 3, pp. 550-56, 2013.
[11] F. Kleibergen and H. K. Van Dijk, "Bayesian simultaneous equations analysis using reduced rank structures," Econometric theory, pp. 701-743, 1998.
[12] F. Kleibergen and E. Zivot, "Bayesian and classical approaches to instrumental variable regres- sion," " Journal of Econometrics, vol. 114, no. 1, pp. 29-72, 2003.
[13] T. G. Conley, C. B. Hansen, R. E. McCulloch, and P. E. Rossi, "A semi-parametric Bayesian approach the instrumental variable problem," Journal of Econometrics, vol. 144, no. 1, pp. 276-305. 2008.
[14] H. F. Lopes and N. G. Polson, "Bayesian instrumental variables: priors and likelihoods," Econometric Reviews, vol. 33, no. 1-4, pp. 100-121, 2014.
[15] M. Wiesenfarth, C. M. Hisgen, T. Kneib, and C. Cadarso-Suarez. "Bayesian nonparametric instrumental variables regression based on penalized splines and dirichlet process mixtures," Journal of Business & Economic Statistics, vol. 32, no. 3, pp. 468-482, 2014.
[16] V. Cheroozhukov and H. Hong, "An MCMC approach to classical estimation," Journal of Econometrics, vol. 1 115, no. 2, pp. 293-346, Aug. 2003.
[17] Y. Liao and W. Jiang, "Posterior consistency of nonparametric conditional moment restricted models," The Annals of Statistics, vol. 39, no. 6, pp. 3003-3031, Dec. 2011.
[18] K. Kato, Balasi-Bayesian analysis of nonparametrie instrumental variables models," The An- nals of Statistics, vol. 41, no. 5, pp. 2359-22222202 2390, Oct. 2013.
[19] J.-P. Florens and A. Simoni, "Nonparametric estimation of an instrumental regression: A quasi- Bayesian approach based on regularized posterior." " Journal of Econometrics, vol. 170, no. 2, pp. 458-475, Oct. 2012.
[20] K. Muandet, A. Mehrjou, S. K. Lee, and A. Raj. "Dual Instrumental Variable Regression," 1910 12358 fes, econ, stat), Oct. 2020, arXiv: 1910.12358.
[21] L. Liao, Y.-L. Chen, Z. Yang, B. Dai, Z. Wang, and M. Kolar, "Provably efficient neural estima- tion of structural equation model: An adversarial approach," arXiv preprint arXiv:2007.01299) 2020.
[22] I. Osband, J. Aslanides, and A. Cassirer, "Randomized prior functions for deep reinforcement learning," in Proceedings of the 32nd International Conference on Neural Information Pro- cessing Systems, 2018, pp. 8626 8638.

---

[23] W. K. Newey and J. L. Powell, "Instrumental variable estimation of nonparametric models," Econometrica, vol. 71, no. pp. 1565-1578, 2003.
[24] K. Fukumizu, F. R. Bach, and A. Gretton, "Statistical Consistency of Kernel Canonical Corre- lation Analysis," " Journal of Machine Learning Research, p. 23, 2004.
[25] B. Dai, N. He, Y. Pan, B. Boots, and L.. Song, "Learning from Conditional Distributions via Dual Embeddings," arXi:: 16077045779 [cs, math, stat], Dec. 2016, arXiv: (607.04579.
[26] N. Dikkala, G. Lewis, L. Mackey, and V. Syrgkanis, "Minimax estimation of conditional mo- ment models," in Advances in Neural Information Processing Systems, H. Larochelle, M. Ran- zato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 142-
[27] A. M. Stuart, "Inverse problems: Bayesian perspective." Acta numerica, vol. 19, pp. 451- 559, 2010.
[28] M. Kanagawa, Hennig, D. Sejdinovic, and B. K. Sriperumbudur, "Gaussian processes and kernel methods: A review on connections and equivalences," arXiv preprint arXiv: 1807.02582, 2018.
[29] T. Zhang et al., "From e-entropy to KL--entropy: Analysis of minimum information complexity density estimation," " The Annals of Statistics, vol. 34, no. 5, pp. 2180-2210, 2006.
[30] A. W. van der Vaart and H. van Zanten, "Rates of contraction of posterior distributions based on Gaussian process priors," The Annals of Statistics, vol. 36, no. 3, Jun. 2008.
[31] "Information Rates of Nonparametric Gaussian Process Regression," " Journal of Machine Learning Research, vol. 12, pp. 2095-2119, 2011.
[32] I. Steinwart and C. Scovel, "Mercer's Theorem on General Domains: On the Interaction be- tween Measures, Kernels, and RKHSs," Constructive Approximation, vol. 35, no. 3, pp. 363- 417, Jun. 2012.
[33] L. H. Dicker, D. P. Foster, and D. Hsu, "Kernel ridge vs. principal component regression: Min- imax bounds and the qualification of regularization operators," Electronic Journal of Statistics, vol. 11, no. 1, Jan. 2017
[34] S. Ghosal and A. Van der Vaart, Fundamentals of nonparametrie Bayesian inference. bridge University Press, 2017, vol. 44.
Cam-
[35] M. J. Wainwright, High-dimensional statistics: A non-asymptotic viewpoint. University Press, 2019, vol. 48.
Cambridge
[36] M. Moreira, J. R. Porter, and G. A. Suarez, "Bootstrap and higher-order expansion validity when instruments may be weak," 2004.
[37] A. Flores- "Finite sample evidence of IV estimators under weak instruments," Journal of Applied Econometrics, vol. 22, no. 3, pp. 677-694, Apr. 2007.
[38] R. Davidson and G. Mackinnon, "Wild Bootstrap Tests for IV Regression," Journal of Busi- ness & Economic Statistics, vol. 28, no. 1, 128- 144, 2010, publisher: [American Statistical Association, Taylor & Francis, Ltd.].
[39] T. Pearce, F. Leibfried, and A. Brintrup, "Uncertainty in neural networks: Approximately Bayesian ensembling." " in Proceedings of the Twenty Third International Conference on Arti- ficial Intelligence and Statisties, ser. Proceedings of Machine Learning Research, S. Chiappa and R. Calandra, Eds., vol. 108. PMLR, 26-28 Aug 2020, pp. 234-244.
[40] B. He, B. Lakshminnarayanan, and Y. W. Teh, "Bayesian deep ensembles via the neural tangent kernel," arXiv preprint
2020.
12

---

[41] A. Jacot, F. Gabriel, and C. Hongler, "Neural Tangent Kernel: Convergence and Generaliza- tion Neural Networks," " in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett, Eds., vol. 31. Cur- ran Associates, Inc., 2018
[42] R. M. Neal, Bayesian learning for neural networks. 2012, vol. 118.
Springer Science Business Media,
[43] J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein, "Deep neural networks as Gaussian processes," arXiv preprint 1xXiv:711.00100 2017.
[44] A. G. d. G. Matthews, M. Rowland, J. Hron, R. E. Turner, and Z. Ghahramani, "Gaussian process behaviour in wide deep neural networks," arXiv preprint arXiv: /804. 1271, 2018.
[45] W. Hu, Z. Li, and D. Yu, "Simple and effective regularization methods for training on noisily labeled data with generalization guarantee," in International Conference Learning Repre- sentations, 2020.
[46] T. Hu, W. Wang, C. Lin, and G. Cheng. "Regularization matters: A nonparametric perspective on overparametrized neural network," in Proceedings The 24th International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, A. Baner- jee and K. Fukumizu, Eds., vol. 130. PMLR, 13-15 Apr 2021, pp.
[47] J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington, "Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent," " in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelz- imer, F. d. Alché Bucé, E. Fox, and R. Garnett, Eds., vol. 32.
Curran Associates, Inc., 2019.
[48] A. Rahimi, B. Recht et al., "Random features for large-scale kernel machines." in NIPS, vol. 3, no. 4. Citeseer, 2007, 5.
[49] A. Zellner, "Bayesian Method of Moments (BMOM) Analysis of Mean and Regression Mod- els," wrXiv:bayes--m/9511000 1001, Dec. 1995, arXiv: bayes-an/951 1001.
[50] J.-Y. Kim, "Limited information likelihood and Bayesian analysis," Journal of Econometrics, 19, 2002.
[51] T. Jaakkola, M. Meila, and T. Jebara, "Maximum entropy discrimination." in Proceedings of the 12th International Conference on Neural Information Processing Systems, 1999, pp. 470- 476.
[52] M. Dudík, J. Phillips, and R. E. Schapire, "Maximum entropy density estimation with gener- alized regularization and application species distribution modeling." Journal of Machine Learning Research, 2007.
[53] J. Zhu and E. P. Xing, "Maximum entropy discrimination Markov networks." " Journal of Ma- chine Learning Research, vol. 10, no. 11, 2009.
[54] J.PP. Florens and A. Simoni, "Gaussian processes and bayesian moment estimation," Journal of Business & Economic Statistics, vol. 39, no. 2, pp. 482-492, 2021.
[55] R. Zhang, M. Imaizumi, B. Schölkopf. and K. Muandet, "Maximum Moment Restriction for Instrumental Variable Regression," arXh::2010.076%. [cs], Oct. 2020, arXiv: 2010.07684.
[56] A. Mastouri, Y. Zhu, L. Gultchin, A. Korba, R. Silva, M. J. Kusner, A. Gretton, and K. Muan- det, "Proximal causal learning with kernels: Two-stage estimation and moment restriction," arXiv preprint arXiv:2105.045.44 2021.
[57] A. Puli and R. Ranganath, "General control functions for causal effect estimation from in- strumental variables," " Advances in Neural Information Processing Systems, vol. 33, p. 8440, 2020.
[58] I. Xu, Y. Chen, S. Srinivasan, N. de Freitas, A. Doucet, and A. Gretton, "Learning Deep Features Instrumental Variable Regression," a5Xiv:2010.07/54 [cs, stat], Oct. 2020, arXiv: 2010.07154.
13

---

[59] S. Chib. M. Shin, and A. Simoni, "Bayesian estimation and comparison of conditional moment models," " 2019.
[60] X. Chen and T. M. Christensen, "Optimal sup-norm rates and uniform inference on nonlinear functionals of nonparametric IV regression." Quantitative Economics, vol. 9, no. 1, pp. 39-84, 2018.
[61] B. T. Knapik, A. W. van der Vaart, and J. H. van Zanten, "Bayesian inverse problems with Gaussian priors," The Annals of Statistics, vol. 39, no. 5, pp. 2626-2657, Oct. 2011, arXiv: 1103.2692.
[62] A. Van der Vaart, H. Van Zanten et al., "Bayesian inference with rescaled Gaussian process priors," " Electronic Journal of Statistics, vol. I, pp. 433-448, 2007. 2007.
[63] C. Ai and X. Chen, "Efficient estimation of models with conditional moment restrictions con- taining unknown functions," Econometrica, vol. 71, no. 6, pp. 1795-1843, 2003.
[64] S. Ghosal, J. K. Ghosh, and A. W. Van Der Vaart, "Convergence rates of posterior distribu- tions," Annals of Statistics, pp. 500-531, 2000.
[65] A. Caponnetto and E. De Vito, "Optimal rates for the regularized least-squares algorithm," Foundations Computational Mathematics, vol. no. 3, pp. 331-368, 2007.
[66] K. Fukumizu, "Nonparametric Bayesian inference with kernel mean embedding," in Modern Methodology and Applications in Spatial-Temporal Modeling, Springer, 2015, pp. 1-24.
[67] F. De Vito, I.. Rosasco, A. Caponnetto, U. De Giovannini, F. Odone, and P. Bartlett, "Learning from examples as an inverse problem." Journal of Machine Learning Research, vol. 6, no. 5, 2005.
[68] R. Vershynin, High-dimensional probability: An introduction with applications in data sci- ence. Cambridge University Press, 2018, vol. 47.
14


Reviews:
[{"review_id": "o76wqAbLW3c", "reviewer": "Reviewer_6kaq", "summary": "The paper proposes a scalable quasi-Bayesian inference for instrumental variable regression. Specifically, the paper builds a quasi-posterior on the recent quadratic kernel IV loss function. For scalability, the paper extends the randomized prior trick based on the recent dual formulation of the quadratic loss. The approach has a theoretical guarantee that incorrect solutions are excluded while valid kept. Moreover, the convergence analysis for the randomized prior trick with the stochastic gradient descent-ascent is present. Experiments validate the approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality\n\nThe quasi-posterior Bayesian approach employs the recent development of non-linear instrumental variable regression and the paper extends the randomized prior trick for scalability of the model. However, a recent method [Zhang, 2020] similarly connects the quadratic kernel loss with the Gaussian process and this work is expected to clarify the relation to it.\n\n2. Quality\n\nThe approach is technically sound. The paper presents assumptions for theoretical analyses and detailed development for the approach. The theoretical analyses are sound and provides performance guarantees. The experiments assess the scalability and the modeling ability of the approach.\n\n3. Clarity\n\nYes, the paper is clearly written.\n\n4. Significance\n\nThe paper presents important results and will be interesting for researchers in the Bayesian community as well as the causality inference community.\n\nZhang, R., Imaizumi, M., Schölkopf, B., & Muandet, K. (2020). Maximum moment restriction for instrumental variable regression. arXiv preprint arXiv:2010.07684.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "NUC_ZbTUlb", "reviewer": "Reviewer_km2d", "summary": "The paper concerns the uncertain quantification for instrumental variable (IV) regression that it is an approach for estimating causal effect from confounded observational data and, proposes a scalable quasi-Bayesian procedure. The procedure was built upon the kernelized IV models (Muandet et al (2020)). The contribution is forming a quasi-posterior by employing a Gaussian process prior and constructing a kernel conditional expectation estimator using a randomized prior trick. Theoretical properties and simulation studies are provided.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(a) This work is stretched out my area. With my limited search, I haven’t seen IV regressions using a kernelized IV model employing a Gaussian process prior. I feel the use of “quasi-Bayesian” is slightly abusing as only a prediction is inferenced according to the Bayes rule. \n\n(b) I found the paper was little difficult to follow. Some notations like C^{-1}_{zz,\\nu}  or S^*_z in page 3 were not defined although they were key terms in their expressions. Although authors described steps of their approach, an algorithm (or a list of steps) will be more effective, and it is easier to see what estimates implied/steps were involved.\n\n(c) There is some unclearness about the approach.\n\n(c-i) It is not clear how the objective function, L(f) (Eqn 4) fits in the posterior (8) and how K-zz and other parameters (kernel specific,\nscale parameters) are obtained. \n\n(c-ii) A randomize prior trick approximation will be sensitive to m and, how would you choose m and what do we expect to see when m is too small/large? How it is sensitive to z-distribution? \n\n(d) In the simulation study, authors compared their method with existing estimators and, showed the new method was competitive. Authors also honestly showed some kernel choice sensitivity in Appendix. In the main text, this sensitivity was not shown, and I thought including this in the main paper would be very valuable to provide correct strength and weakness of their method. Moreover, some discussion on the kernel choice (if there is any existing guidelines) would be good.\n\n(e) The paper proposed a quasi-Bayesian procedure for IV regression and showed that it can be an efficient estimator when a kernel is carefully chosen. I didn’t find any false in their approach and empirical evidence of their claim is given. Weakness will be clarification and brief description of a randomized prior trick without investigation of its properties and impacts on the resulting inference.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors showed some sensitivities associated with the kernel choice and this will be one limitation.\nI didn't see when their method will be poor in the text.", "review_text": "", "extra_scores": {"ethical_concerns": "I didn't see any point that it will raise ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "LKKYr55zoR7", "reviewer": "Reviewer_rhpF", "summary": "The authors introduce a scalable quasi-Bayesian method for nonparametric instrumental variable (IV) regression that provides uncertainty quantification for the IV models. They base their approach on kernelized IV models and define a quasi-posterior as the Radon-Nikodym derivative of a generalized method of moments (GMM) objective function with respect to the standard Gaussian process prior. They go on to show that the quasi-posterior exhibits satisfies certain posterior contraction criteria that lead to asymptotically sound results, and prove a conservative posterior contraction rate under a set of typical assumptions. They then show how to achieve scalable inference for the quasi-posterior using a previously proposed “randomized prior trick” for Gaussian process regression, and provide a proof for the convergence of the inference algorithm. Finally, the authors showcase the performance of their method in a series of simulated experiments of varying complexity, by comparing it against other approaches that provide uncertainty quantification (Bayesian, bootstrap).", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*: The present work is, to the best of my knowledge, novel. To arrive at this contribution, the authors have combined and built upon previously published results on kernelized IV regression and quasi-Bayesian analysis. The authors carefully put their method in the context of related work and adequately describe how the contribution differs.\n\n*Quality*: The submission seems to be technically sound, and the theoretical claims are reasonably supported in the main paper, with the majority of proofs deferred to the supplement. The numerical experiments are also appropriate, with the authors providing at least one challenging example for the task under hand. However, one aspect that is missing in the numerical experiments’ evaluation is the computational cost. The authors do discuss the time and computational complexity of their method in more detail in the supplement, but since scalability is one of the main selling-points of the method, I would have liked to have seen a comparison in terms of computational cost and/or run time.\n\n*Clarity*: The paper is well-organized for the most part and is relatively error-free. Many technical details are left for the supplement, but the key ideas all seem to be present.\n\n*Significance*: Judging by the numerical results, the authors’ submission successfully addresses a very challenging task and equals, or improves upon, previous results in terms of estimation accuracy and uncertainty quantification. However, it seems to me that the major advantage of this quasi-Bayesian approach for nonparametric IV regression could lie in its scalability, as the use of such flexible methods is often computationally prohibitive. I think the authors could have stressed this point more to underline the significance of their work.\n\n*Minor comments*: \n-\tLine 13: “causal effect” -> “causal effects”\n-\tLine 38: It seems that a verbal phrase like “are” or “need be” is missing before “present”.\n-\tLine 90: $\\bar{\\nu}$ is first used here in defining $ \\hat{C}_{zz, \\bar{\\nu}}^{-1}$, but I think both have yet to be defined.\n-\tEquation (11): $\\nu$ can be inferred to be equal to $n \\bar{\\nu}$, but this does not seem to be explicitly mentioned in the paper.\n-\tProposition 3.1: It is not mentioned how the log quasi-likelihood is “scaled”. Is it the same scaling as for the evidence in line 115?\n-\tFigure 1: The right-most plot (CI coverage for $\\alpha = 0.5$) is hard to read. The results for some algorithms are barely visible or not visible at all. Perhaps leaving more margin below 0.00 and above 1.00 on the y-axis could help.\n-\tLines 275-277: I’m not sure I agree with the conclusion that “quasi-Bayesian inference provides the most reliable uncertainty estimates” by looking at Figure 1. For $\\alpha=0.05$, the results look promising, but for $\\alpha = 0.5$, the CI coverage appears to be 100%, which I’d think is a rather poor uncertainty estimate.\n-\tLine 288: The first paragraph in Section 6.2 does not have numbered lines. This does not affect the review, but perhaps there is a formatting issue that the authors need to be made aware of.\n-\tLines 406-409: Single-author references [30] and [31] have erroneous “et al.”.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the potential impact of their work in a separate section, where they explain how uncertainty quantification can help in assessing the degree of error in the estimate, but warn the practitioner that its validity ultimately hinges on the chosen model.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "five"}, "extra_text": {}}, {"review_id": "6E9uV_jGjp3", "reviewer": "Reviewer_hAiR", "summary": "The paper studies a quasi-Bayesian approach to instrumental variable (IV) regression, based on using an empirical estimate of the ‘likelihood’ to obtain a Gibbs posterior. Such an approach has been studied in earlier literature due to the difficulty of incorporating moment conditions in IV via a likelihood. The paper pursues a Gaussian process approach and proves some theory, namely that the resulting posterior puts most of its mass on functions (almost) satisfying the moment constraint. The paper provides a computation approach based on the randomized prior trick and illustrates the method on various simulated datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper is interesting and well written in parts (though a bit dense at times), and features a nice mixture of theory and methodology. The approach of using a quasi-likelihood is not new, and the paper’s main contribution in this direction is to develop a choice of quasi-likelihood for which more efficient computation can be performed. This is a reasonably novel idea, cleverly combining several different existing ideas in the instrumental variable (IV) setting. I work on Bayesian methods rather than IV methods, so it was not entirely clear to me what was previously known here. I also comment more on the theory since I am more familiar with this.\n\nA main contribution is to prove some theory, namely (C1) and (C2) in the paper. (C1) was fine, but (C2), was not clear to me either mathematically or in words and needs to be looked at. The proofs follow from tools in Bayesian nonparametrics as have been applied in this setting by Kato (Annals of Statistics 2013). Saying that, his proofs are very technical and applying them to new settings requires effort, so I think the paper has a suitable technical level for this venue.\n\nThe method is illustrated in simulations where it shows promising empirical performance. The simulations are broadly fine, but do have a few issues I think could be improved.\n\nOverall, I think the paper is interesting and relevant. If the theoretical results can be satisfactorily cleared up (especially Proposition 3.1) and better explained, this will make a solid contribution.\n\nMain comments:\n1.\tTheorem 3.1 and its discussion need to be clarified. What does $\\mathbb{E}^2$ mean? If I understand correctly, this result says that the posterior concentrates on functions for which the moment restriction (1) ‘almost holds’. If you were Bayesian, you would try to put a prior that directly enforces this condition rather than hope your posterior concentrates on it. Why can such a restriction not be enforced with quasi-posterior probability one?\n\n2.\tTheorem 3.1: what values of $\\tau_n$ are provided by your theory for concrete examples? E.g. Matern prior and Holder/Sobolev smooth truth. The results are currently quite abstract and not interpretable.\n3.\tProposition 3.1:\n\n(a)\tWhy is $f \\sim \\Pi$ from the prior? Where is this used? The conclusion doesn’t involve the prior and the prior is only specified as a GP, so does this result hold for any function $f$ drawn from any GP? \n\n(b)\t$\\hat{E}_n$ is an operator taking $x$ valued functions to $z$ valued functions, so it’s not at all clear what the LHS means here. \n\n(c)\tA general notational improvement throughout the paper would be to clarify what you are taking the expectation over. e.g. $\\mathbb{E}_X$ for taking the expectation of $X$, etc. \n\n(d)\tMake precise under what probability distribution $\\to^p$ is.\n\n(e)\tThis looks like a kind of ‘uniform law of large numbers’ since the $\\hat{E}_n$ is random and depends on the same random variables as the sum. Empirical process theory tells us that such a result holds if the function class is Glivenko-Cantelli but not generally otherwise. However, I don’t see such a condition. Can you please clarify (maybe it’s implicit from $f \\sim \\Pi$)?\n\n(f)\tRemark 3.3 and the interpretation of Proposition 3.1 don’t make sense to me. Also, if you want to relate this to the log-likelihood, then wouldn’t it make sense to study the limit uniformly over $f$ rather than pointwise? Please carefully explain what is going on.\n\n4.\tProposition 4.2:  Can you discuss this result a bit more. If I understand it correctly, this say the algorithm won’t get stuck at a local minimum (likely due to the stochastic part), but it doesn’t give any quantitative bounds on convergence, correct?\n5.\tSimulations:\n\n(a)\tI think it would be clearer to present your data in table form rather than as boxplots (one picture is fine). \n\n(b)\tWhat are you plotting credible intervals of? The goal is to estimate the whole function $f$. Are you doing it at a point?\n\n(c)\tYou should provide the average credible interval lengths, otherwise the coverage is not informative.\n\n(d)\t10 repetitions of the data is small and you should do more if possible. If this is computationally prohibitive, please say so and give some indication of the run time. \n\n\nMinor points:\n-\tp3 Maybe define the tensor product $\\otimes$ \n-\tp3 Could you please explain the equation on l92 a bit better\n-\tAssumption 3.2 One typically needs the assumption $m\\tau_m^2 \\to \\infty$ for these kind of results. Has this been forgotten?\n-\tAssumption 3.2: mention that this is a standard condition in the GP literature. You should really cite van der Vaart & van Zanten (Annals of Statistics 2008) here since they first applied this theory to GP’s.\n-\tTheorem 3.1 I guess $\\tau_{\\sqrt{n}}$ should be $\\tau_n$ everywhere (it’s indexed by an integer)?\n- Theorem 3.1 is not a contraction rate, which involves the posterior putting most of its mass about the ‘true’ function, which is not what you are studying here. \n- THeorem 3.1: specify what assumptions you have put on the ‘true’ $f$ generating the data (taken over the expectation $\\mathbb{E}_{D_n}$. Do you assume the condition (1) holds in reality?\n- Proposition 4.2: $S_m$, $\\mu_m$ should be $S$, $\\mu$?\n-\tP19 What is $err_{n,f}$?\n\n############################################ \n\nNote: scored changed from 6 to 7 after author response.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the problem of instrumental variable (IV) regression and proposes a novel quasi-Bayesian approach to it. The IV regression is an important problem in economics, social science, and epidemiology which has recently gained traction in the machine learning community as well. However, as the authors mentioned, we still lack satisfactory way of qualifying the uncertainty in this setting and Bayesian modelling is non-trivial in this case because of the inability to specify full likelihood. Due to the non-standard setting of this work, it thus provides an important step towards a full probabilistic treatment of IV regression and will certainly inspire future works in the Bayesian ML community. 

There is a consensus among the expert reviewers that the paper makes a sold technical and methodological contribution. All reviewers also feel that their concerns have been addressed adequately. Upon reading the paper, I do agree with Reviewer hAiR and Reviewer km2d that the paper is quite difficult for the non-expert readers to follow, especially because of the technical nature of this work. I encourage the authors to improve the writing and to make it more accessible to the broader audience. This is a minor point, but can increase the impact of this paper.

I thus recommend this paper for publication at NeurIPS2021.

**Remark**: I feel that the related work section can also be improved. Here are some references that might helpful in improving this section:

- [1] R. Zhang, M. Imaizumi, B. Schölkopf, and K. Muandet, “Maximum Moment Restriction for Instrumental Variable Regression,” arXiv:2010.07684 [cs], Oct. 2020, arXiv: 2010.07684.

- [2] K. Muandet, W. Jitkrittum, J. Kubler, "Kernel Conditional Moment Test via Maximum Moment Restriction", UAI2020.

- [3] A. Mastouri, Y. Zhu, L. Gultchin, A. Korba, R. Silva, M. J. Kusner, A. Gretton, K. Muandet: "Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction", ICML2021.

- [4] R. Singh, M. Sahani, and A. Gretton, “Kernel Instrumental Variable Regression”, NeurIPS2019.

For relevant discussions, see, e.g., related work section in [4], Section 6 in [1] and Section 5 & Appendix B1 in [2]. Section 3.3 in [3] is definitely relevant to the second paragraph L243-L250 of this submission.


Decision: Accept (Poster)