Nonparametric Instrumental Variable Regression through Stochastic Approximate Gradients
Authors: Yuri Fonseca
Caio Peixoto
Yuri Saporito
Venue: NeurIPS 2024 poster (2024)
Keywords: Nonparametric Instrumental Variables
Stochastic Gradients
RKHS
Binary response
Deep Learning
Causality
Primary area: optimization
TL;DR: We address the problem of nonparametric instrumental variable regression using stochastic gradient descent in a function space.

Abstract: Instrumental variables (IVs) provide a powerful strategy for identifying causal effects in the presence of unobservable confounders. Within the nonparametric setting (NPIV), recent methods have been based on nonlinear generalizations of Two-Stage Least Squares and on minimax formulations derived from moment conditions or duality. In a novel direction, we show how to formulate a functional stochastic gradient descent algorithm to tackle NPIV regression by directly minimizing the populational risk. We provide theoretical support in the form of bounds on the excess risk, and conduct numerical experiments showcasing our method's superior stability and competitive performance relative to current state-of-the-art alternatives. This algorithm enables flexible estimator choices, such as neural networks or kernel based methods, as well as non-quadratic loss functions, which may be suitable for structural equations beyond the setting of continuous outcomes and additive noise. Finally, we demonstrate this flexibility of our framework by presenting how it naturally addresses the important case of binary outcomes, which has received far less attention by recent developments in the NPIV literature.

Nonparametric Instrumental Variable Regression through Stochastic Approximate Gradients
Yuri R. Fonseca" "
Caio F. L. Peixoto" School of Applied Mathematics Getulio Vargas Foundation Rio de Janeiro, RJ, Brazil caio. peixoto@fgy. br
Decision, Risk and Operations Columbia University New York, NY yf yfonseca23@gab.com cww.obia..cmumbbaaa......e.
Yuri Saporito"
School of Applied Mathematics
Getulio Vargas Foundation Rio de Janeiro, RJ yuri.saporito@fgw br
Abstract
Instrumental variables (IVs) provide powerful strategy for identifying causal effects in the presence of unobservable confounders. Within the nonparametric setting (NPIV), recent methods have been based on nonlinear generalizations of Two-Stage Least Squares and on minimax formulations derived from moment conditions duality. novel direction, we show how formulate functional stochastic gradient descent algorithm to tackle NPIV regression by directly mini- mizing the populational risk. We provide theoretical support the excess risk, and conduct numerical experiments showcasing our method's ' superior stability and competitive performance relative to current state-of-the-art alternatives. This algorithm enables flexible estimator choices, such as neural networks or kernel based methods, as well as non-quadratic loss functions, which may be suitable for structural equations beyond the setting of continuous outcomes and additive noise. Finally, we demonstrate this flexibility of our framework by presenting how it naturally addresses the important case of binary outcomes, which has received far less attention by recent developments in the NPIV literature.
form bounds
Introduction
Causal inference from observational data presents unique challenges, primarily due the potential for confounding variables that can affect both outcomes and variables of interest. The unconfoundedness assumption, crucial in this context, posits that all confounding variables are observed and properly accounted for, allowing for an unbiased estimation of causal effects. However, in many real-world scenarios, this assumption
difficult to satisfy. When this is the case, approaches that rely on instrumental variables (IVs) quantities that are correlated with the variable of interest (relevance condition). do not affect the outcome in any other way (exclusion condition) and are independent of the unobservable confounders
offer way to identify causal effects confounders exist. As a concrete example, suppose we want to estimate the impact of years of education earnings. Most likely, there unobservable factors, such omitted ability, affecting both the decision to study and income. In this case, changes in compulsory schooling laws could be used instrument
when unobserved
Alphabetical order.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

While traditional parametric approaches to IV regression often require assumptions about the relation- ships between variables that may not hold practice, nonparametric IV (NPIV) models can adapt to the intrinsic structure of the data, allowing for more nuanced understanding of causal relationships. For this reason, there has been a recent boost of new algorithms applied to the NPIV estimation problem and its theoretical properties. The challenge is that NPIV estimation is an ill-posed inverse problem [28] s 91, and recent methods aim to incorporate developments from predictive models, e.g.. deep learning and kernel methods, while also accounting for the particular structure of the inverse problem at hand.
In this work, we deviate from previous approaches to NPIV estimation which end up minimizing empirical versions of the populational risk (28) 19 34 25], or trying to satisfy large collection of empirical moment restrictions [18] 24 6]. In our formulation, we compute an analytical functional gradient for the actual populational risk. After formulating a consistent estimator for this gradient, we apply stochastic gradient descent in a certain function space to recover the effect of X on the outcome Y.
The rest of the paper is structured as follows, We conclude Section I with a thorough discussion of the previous works on NPIV and our contributions. Next, we provide the basic setting for NPIV regression in Section 2 In Section 3 we detail the associated risk minimization problem and analytically compute its stochastic gradient. Based on these results, we introduce our method in Section [ with accompanying practical considerations and theoretical support. Numerical experiments are reported on Section [ where our algorithm compared to state-of-the-arr machine learning methods for NPIV. Finally, in Section 6 we study the case of binary outcomes, showing how our method naturally addresses this scenario under the current assumptions made in the literature. Proofs of all our results are presented in Appendix[A) [A] while Appendix [ contains formal comparisons with existing methods provides additional implementation details.
and Appendix C
1.1 Previous work
Many traditional approaches to IV estimattion, like the Two-Stage Least Squares (2SLS) method, rely on linear models for treatment estimation and counterfactual prediction functions (see (3 40 for
thorough survey of classical IV estimation). These approaches, while efficient for estimating policy effects, depend strong parametric assumptions. Nonparametric extensions of 2SLS then attempt to introduce model flexibility by utilizing linear projections onto known basis or sieve functions (as in [28 27) or kernel-based estimators (as in [17 13). However, these traditional methods face limitations in large, high-dimensional datasets as they are sensible to the particular choice of sieve functions and number of basis elements (see the discussion (19).
In order to propose a scalable method, 19 introduces DeepIV, a generalization of 2SLS which employs neural networks in each step of the two-stage procedure. Although their algorithm is more suitable for high dimensional data, the authors do not provide theoretical guarantees. Deep learning estimators have also appeared within methods focused on the Generalized Method of Moments (GMM), which leverage moment restrictions imposed by the IV to obtain an estimator. In (6), the authors propose DeepGMM, a reformulation of the optimal weighted GMM problem minimax problem. They rely on the identification assumption to demonstrate consistency results, provided their algorithm is able to nearly solve a smooth zero-sum game.
In another direction, some methods exploit developments in the RKHS literature and apply them to NPIV estimation. The KIV algorithm in (34] transforms the problem into two-stage kernel ridge regression through a kernel mean embedding of the distribution of the endogenous covariate given the instrument. In [25). the authors take insight from two-stage problems in stochastic programming and propose DualIV, an algorithm which uses Fenchel duality to transform NPIV into a convex-com concave saddle point problem, for which a closed-form RKHS algorithm is presented. Inspired by the gradient boosting literature, (4) propose an algorithm that iteratively fits base learners in a boosting fashion, but still needs post-processing step where the basis functions are held fixed and the weights are optimized.
We point out that most of the cited works have some flexibility issues. While kernel based methods impose a higher computational cost as the number of data points grows and may exhibit poor
Reeroduuing Kernel Hilbert Space

---

performance in high dimensions 35 25]. Deep Learning based algorithms can suffer from high variance and instability if the amount available data is small, as was seen [2]
Furthermore, in many applications of interest, ranging from consumer behavior 30 to epidemiology [23), the outcome is binary; for review, see [12] In these scenarios, straightforward application of the quadratic loss function would result in a misspecification of the problem, potentially leading to erroneous estimates. The above NPIV methods strongly leverage the additive structure of the problem formulation, and proper extensions would require significant effort, a detailed discussion is presented in Appendix B Traditionally, a common approach for binary outcomes using a semi- parametric specification based on control function [20]. This allows for nonparametric estimation of the distribution of the unobservables together with identification of the causal function's 's parameters 32 [4 [T]. To the best of our knowledge. the only work that addresses fully nonparametric IV for binary outcomes is [T0]. which proposes an estimation strategy based on Tikhonov regularization.
1.2 Contribution
We propose a novel algorithm for NPIV estimation, SAGD-IV, which works by minimizing the populational risk through stochastic gradient descent a function space. Under mild assumptions, we provide finite sample bounds on the excess risk of our estimator. Empirically, we demonstrate through numerical experiments that SAGD-IV achieves state-of-the-art performance and better stability.
We have the freedom employ a variety of supervised learning algorithms to form the estimator of the stochastic gradient, most notably kernel methods and neural networks. This means our estimator could be tailored to specific scenarios where particular method is more likely perform well. Furthermore, our algorithm is naturally able to handle non-quadratic loss functions, which allows us to extend both our algorithm and theoretical guarantees to the binary outcomes case.
Problem setup and notation
Following most of the recent literature, we start by presenting the problem setup under the additive noise assumption to demonstrate our methodological contribution. Then, in Section 6 we analyze the important case of binary outcomes, prototypical example of when this assumption does not hold.
Let (2, A,P)) P) be the underlying probability space, and let X be a random vector of covariates taking values X C Rdx, We assume that the response variable generated according to
Y = h (X)+
(I)
where E E (2, A, and satisfies Ele] = O. We denote by Pxx the distribution of the r.v. X and assume that the structural function h" belongs L2 B(Y), Px L. which we simply denote by L'()). This is a Hilbert space with norm and inner product given by [||||22(x) = E and (h, 9) (x)))= E(M(X) 0g( (X)]. We assume that E||| X] * O, that is, some covariates are endogenous. Finally, we assume the existence of random vector Z, taking values in 2 C Rd and satisfying
1. E 2 = O, the exclusion restriction;
2. X J. Z, i.e., Z is relevant.
This makes Z valid instrumental variable. We define P2 and (Z) analogously to Px and (x). We further consider the mild assumption that X and Z have joint density denoted by px,z- Our goal is to estimate h+ based on i.i.d. samples from the joint distribution of X, Z and Y.
As we have listed in the introduction, there are a few different approaches to estimate h²" We will conditioned
follow here the original one of (28), in which we take the expected value of Equation (
on Z to get
y 2 = (X) 2).
(2)
This motivates to define the conditional expectation operator P (X) (Z) given by
P[M[(=) E[M(X) 7 z].
We denote by B(X) the Borel o-algebra in X.
3

---

This is bounded linear operator which satisfies ||P|lop I 1 (the operator norm) and whose adjoint
P (2) +
).). ). that is also a bounded linear operator, is given by ("][](>) = [gg( X
z/. Defining r(2) EE Z], we can then rewrite Equation (2
r r= P
This a Fredholm integral equation of the first kind inverse problem.
22 and, as such, poses an ill-posed linear
In this context, a common assumption [13 made about P is compactness. We also need here, noting that compact operators with infinite dimensional range provide prototypical examples of ill-posed inverse problems [8). However, we wish phrase this assumption in a different, albeit equivalent [8). form: Assumption 2.1. Let
(x,
( 2)
(4)
Px (x(x))2(=)
denote the ratio of joint over product of marginals densities for the X and Z variables, with the convention that 0/0 = 0. We assume that this kernel has finite T.2 (Pxx s P2) norm, that is,
+ | [)(xxPP))
Φ(2, 2xx,)))x(x))pz drds
xxz
This is equivalent to assuming that the conditional expectation operator P [X) - [Z) is Hilbert-Schmidt and, hence, compact.
3 The risk and its gradient
Motivated by Equation (3, introduce pointwise loss function R:R. R -> R. and define the associated populational risk measure : - R as
R(h) ((((((2))PI)(((
(5)
The example the reader should keep in mind is the squared loss function E(y,y') = ( y²)² although, as we will see, other examples could used depending on the particular regression setting. Our goal is to solve the NPIV regression problem by solving
inf R(),,
hEW
where H is closed, convex, bounded subset of (xx such that h H. We also require H. For future reference, we state these conditions:
Assumption 3.1 (Regularity of H)). The set H. is closed, convex, bounded subset of (X),,, which contains the origin and satisfies h* E H.
The only part of this assumption which concerns the data generating process is ht E H. which essentially means that the set H is large enough F
For H satisfying Assumption 3.1 we let D - diam H < oo, so that ||A|| < D for every h E H. One possible choice for the set H is the L (X) ball contained in L2(X), that is
H = h E (x) ||A||> I A}.
(6)
where > 0 is a constant. This set is obviously convex and bounded in the (x² norm. It can be shown that is also closed, but not necessarily compact, a restriction on the search set imposed in (28). This can be seen by taking I-Ilaa-boundee orthonormal basis for (X),, if one exists. We denote by proju, the orthogonal projection onto H. In case 7. given by Equation (6) we have the explicit formula proj [h] = (h+ A A) (h- A A).
We now state all the assumptions needed on the pointwise loss f. We denote by O₂ partial derivative with respect to the second argument.
*We note that more fitting name for this object would be projected risk measure, since we are projecting h onto the instrument space before applying the loss function. Concerning this matter, we chose to follow the terminology used in the current NPIV literature, which refers to this projected risk simply as "risk".
'This part could generalized to (h* ker P) o so there exists E 7 such that R(h) R(h").
without prejudice to any the thcoretical results.
'Here use the notation A max(h, 0}, h (-h)+ and A min(a,b).

---

Assumption 3.2 (Regularity of E).
1. The function é R x R convex and C² with respect to its second argument;
2. The function é has Lipschitz first derivative with respect to the second argument, i.e., there exists L such that, for all y. y'. u, u E R we have
|02E(y, y') 02l(u, s L(ww ul 1y u'l).
Some useful facts which follow immediately from these assumptions are presented in Appendix A
3.1 Gradient computation
Here we divert from the previous literature and proceed to compute functional stochastic gradient for the risk R. similar idea has been considered by [15] the context of statistical inverse problems, however, their setup assumes that the operator posing the inverse problem is known, which greatly simplifies the analysis and cannot directly applied to the NPIV problem. We start by providing an analytical formula for (R(h):
Proposition 3.3. The risk R Fréchet differentiable and it's gradient satisfies
vR(h) P*(0=(((()) E (X)),,
where p (Z) + (X) is the adjoint of the operator P.
Both P and P unknown operators which commonly appear in NPIV estimation 8 13 with Nadaraya-
Aadaraya-Watson Watson kernels 26 (3] being a classical option for estimating them. However, the fact that they are nested in Equation (7) is undesired, and may impose extra computational costs, especially considering that 0g6 is nonlinear in the second argument for non-quadratic losses. We overcome this difficulty by leveraging the following characterization of the gradient:
Corollary 3.4. The gradient of the populational risk satisfies
(R(()(z) = (100
(8)
where & is defined in Equation
The benefit of our approach is that Equation
8 enables more computationally efficient way to estimate the gradient of the risk functional, as we explain next. From Equation (8). for given E X, the random variable ((x, Z Z) (2), (2)) is an unbiased stochastic estimator of (R())())) Our stochastic approximate gradient is then built using estimators Φ, f and P of D, and P respectively. With this notation, given sample Z. we consider
(R(h)()) क(x, (F2),
We have then substituted the estimation of the operator P by the simpler problem of estimating the ratio of densities + and computing its product with a,((F( ()), ((h) Moreover, density ratio estimation an area of active research within the machine learning community, which makes developments in this direction immediately translatable into benefits method.
Algorithm: implementation and theory
In Algorithm I we present Stochastic Approximate Gradient Descent IV (SAGD-IV method for estimating h* using the approximation given by Equation 9
"The nomenclature Stochastic Approximate Gradient Descent (SAGD) first appeared in another paper addressing an unrelated topic. (3I). where Langevin dynamics are employed to obtain approximate gradients when samples of the underlying random variable not available.

---

Algorithm SAGD-IV
JM Input: Samples {Zmm)==)
Estimators क, Fand P. Sequence of learning rates (am JM
Initial
guess Tig E H.
Output: h
for S m S M do
Set Um (1, 2m )22 (F(zm), Pp ((mm)
Set fimm = projæt A - am Um
end for Set MI Em= Trm
Remark 4.1. We note the fact that the internal loop of Algorithm only needs samples from the instrumental variable to unfold.
For our theoretical analysis, all we require of the estimators of 4, r and P is the following: Assumption 4.2 (Properties of 6,F and P).
1. . and p computed using dataset D, composed of N samples of the triplet (X, 2, Y). which are independent from the Z samples used in Algorithm
2. f (2( a.s.;
3. P: L2X() (Z) is bounded linear operator
4. |00 css sup ((x,2) 2) < co. This implies, in particular, (2)( | (x,))EXx2
< co.
4.1 On computing P, + and f
Algorithm is purposefully agnostic to how the estimators P, and F are obtained, order to provide the user with sufficient modeling flexibility. In what follows, we will explain the options we considered for accomplishing these tasks in the experiments section. A more detailed description is given in Appendix C
Estimating & is interesting problem in itself as it a ratio of densities, and it has drawn significant attention from the Machine Learning community. Here, we consider kernel and neural network methods, regarded the state of the On the other hand, not many options are available compute P. which means estimating all possible regression of functions of X over Z. Here, we chose to use kernel mean embeddings, as was developed (34). Finally, estimating r is the simplest procedure in our method as it is a regression of Y over Similarly to l, we also consider kernel and neural network methods.
Based on these options, we decided specify two variants of SAGD-IV built upon how we compute and F: Kernel SAGD-IV uses RKHS methods for both estimators, while Deep SAGD-IV employs neural network estimators in these two tasks. The method for computing kernel mean embeddings in both variants.
Under big data scenario, the dependence of our algorithm on kernel methods to estimate the P operator could be a limitation. However, are able to avoid this issue for the estimation of & and Nonetheless, since our algorithm agnostic to the specific estimator P, we could take advantage of any recent developments in the literature of operator-estimation problems.
4.2
Risk bound
Since we directly optimizing the projected populational risk measure, we able to provide guarantees for R(h) mean with respect the training data 21:25 = ( 2m). Our main result is the following:

---

Theorem 4.3. Let ho...., hu- be generated according to Algorithm 1 Assume that satisfies
Assumption 3.2 H satisfies Assumption 3.1 & satisfies Assumption 2.1 and & P, P satisfy Assump- tion 4.2 Then, if we let h M
Em=1 1m 1, the following bound holds:
E EZLM
RR() R((h]]
D²
$ M
Σ
Om + +
(10)
2MaM
where
$ 10 (C + 1211012((2) + + <= 11
3
$
@x2 12xxx
|| 111--fl2aaa + ||P -Pllsp
Flliz(z)
r = 2D max 3(C + 1 1, 1/) 21. o 21, 2
It is productive to analyze the RHS of the bound in Equation sequence (amm) to satisfy the usual assumptions
more carefully. If we choose the
M
Maa
and
am - 0
M
m=1
as M + oc, then the first terms the sum vanish as M grows. The last term appears due the fact that we do not know 2, nor P. but explicitly quantifies how the estimation errors of Φ F and P come together to determine the quality of the final estimator. Furthermore, converges to zero the chosen estimation methods consistent
Remark 4.4 (Y versus r(Z)). Since r P[h*), should compute the risk in Equation (5)
comparing (Z)](Z) with r(Z). However, most works on NPIV [28 34 25) compare P[A](()
directly with Y instead of r()). In Appendix E we discuss this difference and its relationship with the estimator F.
Remark 4.5 (Consistency for h²).. Since the NPIV regression is an ill-posed inverse problem, consis- tency of our algorithm a very challenging and interesting research question, and would require additional assumptions. For instance, under strong convexity of the populational risk measure, is well known that consistency is obtained if the excess risk converges zero. can be shown that, for the quadratic loss, strong convexity of the populational risk is equivalent to P having continuous inverse.
Numerical experiments
we compare the performance of the two variants of SAGD-IV explained in Section SAGD-IV and Deep SAGD-IV, with that of five other notable algorithms for estimating h" in the context of continuous responses: KIV, DualIV, DeepGMM. DeepIV and 2SLSS Hyperparameters and other implementation details are discussed in Appendix C
Here
4.1 Kernel
5.1
Continuous response
To study the performance of our estimator in a continuous response setting, we used the data generating process from (6), which recall below:
Y h" (X) +
X + (11)
Z (2,, /2) Uniform([33,33)
N(0,00 1), 7, (0000.1).
Thus, the confounding variable is and within this specification, four options for h* are considered:
step: h- 1xx))) = 1( 0},
abs ht (x) = |z],
linear: h=(x) x.
sin h" (x) sin(x).
'If one desires to substitute the term for extimator-specife bounds, rates for || T 112 can be
found in 36 Theorem 14.16]. while bounds for I|p Pll..p present in 34 Theorem 2). [38 Theorem 5]
[33 Proposition S5J. The term - is the risk for simple real-valued regression problem, and
various rates are available depending on the chosen regressor and the degree smoothness assumed [16)
Code for the experiments is available http:///tttubboom//Ccctttggggg/
7

---

We adjusted each estimator using 3000 random variable samples (see Remark C.1 and tested them on 1000 samples of X. For each method and response function, evaluated predictions over 20 realizations of the data. Log mean squared error (MSE) box plots and plots of each method's estimator for randomly chosen realization of the data are displayed in Figure] |
Data
Boral SAGI-W
Dep SAGDAY
KIV
DaptV
TSLS
Dac IV
00
Our comple MSEE
-10
M
Dap Racnvy
Peppcaor Denty
mna
Irie
flnd MoDDYY Dap SAnDay
Depaty
1914
free
Figure I: (Left) Log MSE for each model under different response functions. (Right) Plots each method's estimator in randomly selected realization of the data. On the left column, we have Y observations in green and the true structural function in black.
The first thing we can notice from Figure is that TSLS dominates the línear scenario, expected, but behaves poorly in all others, showing the importance of nonlinear approaches.
On second note, both variants of SAGD-IV performed competitively in all scenarios, with the KIV method being a strong competitor in step and sin. However, its performance was significantly worse for the abs response function. Additionally, it is worth noticing the competitive MSEs of the deep learning variant of SADG-IV, which performed consistently better than the other two neural-network-braee algorithms.
Furthermore, an important observation that, within kernel methods, Kernel SAGD-IV had the best tradeoff, having the smallest variance across scenarios. In contrast, DualIV exhibited the highest variance. This can be explained by considering DualIV's hyperparameter selec- tion algorithm [25] Section 4]. In order to perform cross-validation to choose the best regularization parameters, the algorithm requires another (different) regularization parameter as an input, which must be chosen without any guidance. The authors state that it was set to "small constant". This can explain the poor performance, given the importance of regularization parameters for ridge regression algorithms. We note method does not suffer from this problem.
Binary Response
Here outline how methodology can be applied to scenario where the quadratic loss is not the natural option: that of binary response models. We show that, by simply modifying the loss in the risk definition and, consequently, in the algorithm, formulated in the literature. This is in clear contrast with recent methods for NPIV estimation, which, when applicable, would require significant effort. We discuss the limitations of current methods and possible extensions in Appendix
are able to attack this problem it is currently
In the binary response model we consider, the only change need to make to the data generating process is the following:
Y == 0
where Y (x' + e. That is, we observe signal which indicates the response variable greater than the threshold zero. Although simple, this type of model captures important applications as, for instance, in economics/consumer behavior, where Y is the utility the consumer gets from buying product and only observe Y. the buy/no buy decision. The function h' captures the
8

---

impact of the variable X on the consumers' utility. As usual in the literature, h* is interpretable relative to the scale parameter of Here, X, Z and ε satisfy the same assumptions as before.
In order to use one of the available NPIV regression methods the literature, we must first identify a suitable risk functional, of the form shown in Equation (5. Define 1j = h (X) [ (X) Z
so that E 2] = 0 and
Y (X) 2 + 0}.
Denoting by Fl the conditional distribution function of 1 given 7, we find
B|y Z = P h (X) Z + 77> > 0 2 = 1 F (- (X) 2))
Assume that the distribution of 1 given Z is always symmetric around zero, so that F (-)z((y) = F (y), We then have r(Z) = 2 (P [[)]()()) Notice that any notion risk for a given E (X) would have to compare r(Z) and (Z) through 22 which is an unknown function. Hence, without making further assumptions, is not clear how to setup the risk in a useful way, which hinders current NPIV methods from tackling the problem.
know, the only assumption in the literature which allows one to be fully nonparametric in ht within the binary outcomes setting was first introduced in [10]. It posits that v is independent of Z, and has known distribution function F. Since Y E ZJ. it is already uncorrelated with any function of Z. The assumption extends this to independence, and makes available the resulting distribution function.
As far
Under this assumption, we conclude that r(2) F(P(h*]Z)) and, by the negative log- likelihood of the Bernoulli distribution, a natural candidate for é is
{(y, y') BCE(y, F(y')),
where BCE is the binary cross entropy function: BCE(p. lplog (1-p) -)) log( (0gg11 q)]. Then, the risk becomes
F(P(h) R(h) = P(P)) (2)()
(12)
Whereas it is obvious that the quadratic loss satisfies Assumption 3.2 it is not clear whether the same
is true for the loss E(y, y/) = BCE F(y)).. Nevertheless, if F()) = o (x) & 11((1 exp( zz B)),
the c.d.f. of logistic distribution with scale parameter B, one can verify that Assumption 3.2 holds.
6.1 Numerical Experiment
We mimicked the continuous response DGP making the necessary modifications:
Y = 1(E + 0},
X 1
7 = 72) Uniform((3,,)))
1 Logistic(0,)) 3) N.0,
Here, ,3 is scale parameter which was set V0.1. For response functions, analyzed the sin and linear cases, since we must compute [E[*((X) Z] analytically in order to generate the data. This computation easy for the selected scenarios, and we have
8T
T11
E[X Z = 2] = 21: E (X) Z = 2]
sin()1)
sinh
As with the continuous response setting, we present log MSE results and samples from the resulting estimators in Figure 2
We can immediately see that our kernel based estimator performed remarkably well for both response functions. The binary setup is sigmificantly more challenging, we are reducing the continuous response Y to a simple binary signal. Kernel SAGD-IV obtained results which are, on average, on par with the continuous response scenario and Deep SAGD-IV had more difficulty uncovering the true structural function.
7 Conclusion
In this work, we proposed SAGD-IV, a flexible framework for performing nonparametric instrumental variable regression. It is based novel way tackle the NPIV problem, in which we formulate

---

Linewe
Sin
Data
SAGIJIV
Doep SAGDJV
Out of scauple los 318D
Lioner
-10-
-1.-
500
Kemuel SAGIWIV Deep SAGD-TVNocia Kernel SAGn-IV Dwep SAGT-IV
Figure 2: (Left) Log MSE distribution in the binary response DGP. (Right) Plots of each method's estimator in a randomly selected realization of the data for the binary response DGP. On the left column, have samples from the binary response variable Y in green and the true structural function in black.
the minimization the populational risk as functional stochastic gradient descent. We are then able to naturally consider non non-quadratic loss function and incorporate various regression methods under one formulation. Under mild assumptions, we provided bounds on the excess risk of our estimator. Furthermore, we empirically demonstrated superior stability and state-of-the-an MSEs for continuous outcomes, as well promising performance for the binary regression setup.
References
[1] Hyungtaik Ahn, Hidehiko Ichimura, and James Powell. "Simple estimators for monotone index models". In: manuscript, Department of Economics, UC Berkeley 10 (1996).
[2] Joshua D Angrist and Alan B Krueger. "Does compulsory school attendance affect schooling and carnings?" In: The Quarterly Journal of Economics 106.4 (1991), pp. 979-1014.
[3] Joshua D Angrist and Jörn-Steffen Pischke. Mostly harmless econometrics: An empiricist's s
companion. Princeton university press, 2009.
[4] Edvard Bakhitov and Amandeep Singh. "Causal Gradient Boosting: Boosted Instrumen- tal Variable Regression". In: Proceedings of the 23rd ACM Conference Economics and Computation. EC 222 Boulder, CO, USA: Association for Computing Machinery, 2022,
pp. 604-605. ISBN: 9781 450391504. DOT: 10. 1145 3490486 3538251
URL: https
//doi.org/ 100 1145/3490486. 3538251
[5] Keith Battocchi et al. EconML: A Python Package for ML--Based Heterogeneous Trealment Effects Estimation. http::/githu..coo/yywwyyy con/py-why//oooMM....... Version 0.x. 2019.
[6] Andrew Bennett, Nathan Kallus, and Tobias Schnabel. "Deep Generalized Method Moments for Instrumental Variable Analysis". In: Advances in Neural Information Processing Systems. Ed. by H. Wallach al. Vol. 32. Curran Associates, Inc., 2019.
[7] Andrew Bennett et al. "Minimax Instrumental Variable Regression and L2 Convergence Guarantees without Identification or Closedness", In: Proceedings of Thirty Sixth Conference on Learning Theory. Ed. by Gergely Neu and Lorenzo Rosasco. Vol. 195. Proceedings of Machine Learning Research. PMLR, July 2023, pp. 2291-2211121111 1-2188.
[8] Marine Carrasco, Pien-e Florens, and Eric Renault. "Handbook of Econometrics". In: vol. 6B. Elsevier, 2007. Chap. Linear Inverse Problems in Structural Econometrics Estimation Based on Spectral Decomposition and Regularization.
[9] Laurent Cavalier. "Inverse problems statistics", In: Inverse Problems and High- Highh-Dimensioal Estimation: Stats in the Château Summer School, August 31 31-September 4, 2009. Springer, 2011, pp. 3-96.
[10] Samuele Centorrino and Jean-Pierre Florens. "Nonparametric Instrumental Variable Estimation of Binary Response Models with Continuous Endogenous Regressors". In: Econometrics and Statistics 17 (2021). pp. 35-63.
[11] Sandeep Chitla, Srikanth Jagabathula, and Ashwin Venkataraman. "Nonparametric Demand Estimation in the Presence Unobserved Factors". In: Available at SSRN (2022). [12] Paul Clarke and Frank Windmeijer. "Instrumental variable estimators for binary outcomes".
In: Journal of the American Statistical Association 107.500 (2012), pp. 1638-1652.
10

---

[13] S. Darolles et al. "Nonparametric Instrumental Regression". In: Econometrica 79.5 (2011),
pp. 1541-5165.
[14] Yingying Dong and Arthur Lewbel. "A simple estimator for binary choice models with endogenous regressors". In: Econometric Reviews 34.1-2 (2015), pp. 82-105.
[15] Yuri Fonseca and Yuri Saporito. "Statistical Learning and Inverse Problems: A Stochastic Gradient Approach". In: Advances in Neural Information Processing Systems 35 (2022), pp. 9591-9602.
[16] László Györfi et al. "A Distribution-Fren Theory of Nonparametric Regression", In: Springer Series in Statistics. 2002. URL: https // api semanti org CorpusID
43315484
[17] Peter Hall and Joel L. Horowitz. "Nonparametric methods for inference in the presence of instrumental variables". In: The Annals of Statistics 33.6 (2005). pp. 2904-2929. DOI: 10
1214/0090560500000077
URL: hRL::https://dot org. 10. 22140000036000000000000000000000
[18] Lars Peter Hansen. "Larre Sample Properties of Generalized Method of Moments Estimators" In: Econometrica 50.4 (1982), pp. 1029-1054. ISSN: 001 00129682, 14680262.
URL: http
//www.jstor. org/ stable/ 1912775 (visited on 05/1 05/18/2024).
[19] Jason Hartford et al. "Deep IV: A Flexible Approach for Counterfactual Prediction". In: Proceedings of the 34th International Conference on Machine Learning Volume 70. ICML'17. Sydney, NSW, Australia: JMI.R.org, 2017, pp. 1414-1423.
[20] Joel L Horowitz. "Applied nonparametric instrumental variables estimation". In: Econometrica 79.2 (2011), 1). pp. 347-394.
[21] Diederik P. Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization", In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015. URL:
actp://arxiv org/ abaaab///aa abbs/412..r 6980
[22] Rainer Kress. Linear Integral Equations. Applied Mathematical Sciences. Springer- Verler, 1989.
[23] Debbie A Lawlor et al. "Mendelian randomization: using genes as instruments for making causal inferences in epidemiology". In: Statistics in medicine 27.8 (2008), pp. 1133-1163,
[24] Greg Lewis and Vasilis Syrgkanis. "Adversarial generalized method of moments". In: arXiv preprint arXiv:/80l /803/003310110 07 164 (2018).
[25] Krikamol Muandet et al. "Dual Instrumental Variable Regression". In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS'20, Vancouver, BC, Canada: Curran Associates Inc., 2020. TSBN: 9781713829546.
[26] E. A. Nadaraya. "On Estimating Regression", In: Theory of Probability & its Applications 9.1
(1964), pp. 141-142.
[27] Whitney K. Newey. "Nonparametrie Instrumental Variables Estimation". " In: The American Economic Review 103.3 (2013), pp. 550-556. ISSN: 00028282. URL: http: jstor
org/ westtaeeeee3e999
(visited on 02/01/2024).
[28] Whitney K. Newey and James L. Powell. "Instrumental Variable Estimation of Nonparametric Models". In: Econometricu 71.5 (2003), pp. 1565-1578. DOI: http: //tt:///x... doi org/ 10
1111/1468 0262. 00459)
[29] Hemant Kumar Pathak. An Introduction to Nonlinear Analysis and Fixed Point Theory. Springer, 2018
[30] Amil Petrin and Kenneth Train. "A control function approach to endogeneity in consumer choice models". In: Journal of marketing research 47.1 (2010), pp. 3-13.
[31] Yixuan Qiu and Xiao Wang. "Stochastic Approximate Gradient Descent via the Langevin Algorithm",, In: AAAI Conference Artificial Intelligence 34 (2020), pp.
[32] Christoph Rothe. "Semiparametrric estimation of binary response models with endogenous regressors". In: Journal of Econometrics 153.1 (2009), pp. 51-64.
[33] R Singh, I. Xu, and A Gretton. "Kernel methods for causal functions: dose, heterogeneous and incremental response curves". In: Biometrika 11..2 (July 2023), pp. 497-516. ISSN: 1464-
3510. DOI: 10. ttwt/biomet./ asad042 eprint: https http::://cco /tt////aaamcccc////////////. oup. com homm//iime///////
article pdf 11/ 497 57467644 1093/ biomet/ bimet/asad042
asad042. a4sp///42 pdf
URL: https : doi. org / 10

---

[34] Rahul Singh, Maneesh Sahani, and Arthur Gretton. "Kernel Instrumental Variable Regression". In: Advances Neural Information Processing Systems. Ed. by H. Wallach et al. Vol. 32. Curran Associates, Inc., 2019.
[35] Ingo Steinart and Andreas Christmann. Support Vector Machines. Information Science and Statistics. Springer, 2008.
[36] Masashi Sugiyama, Tajji Suzuki, and Takafumi Kanamori. Density Ration Estimation in Machine Learning. Cambridge University Press, 2012.
[37] Zoltán Szabó al. "Learning Theory for Distribution Regression". In: Journal of Machine Learning Research 17.152 (2016), pp. 1-40. URL: http: ///rrrr/mm org/ papers v17/14
[38] Prem Talwai, Ali Shameli, and David Simchi-Levi. "Sobolev Norm Learning Rates for Condi- tional Mean Embeddings". In: Proceedings of The 25th International Conference on Artificial Intelligence and Statistics. Ed. by Gustau Camps-Valty Francisco J. R. Ruiz, and Isabel Valera. Vol. 151. Proceedings of Machine Learning Research. PMLR, 28-30 Mar 2022, pp. 10422- 10447. URL: /BLL:/ttps///prccedddigg mlr. ppwss//111/taalaaaiiai... htm.
[39] Geoffrey S. Watson. "Smooth Regression Analysis". In: Sankhya: The Indian Journal of Statistics, Series A 26.4 (1964), pp. 359-372.
[40] Jeffrey M. Wooldridge. Econometric Analysis of Cross Section and Panel Data. The MIT Press, 2001. ISBN: 9780262232197.
12

---

A Proofs
Proposition A.I. Assume that satisfies Assumption 3.2
Then:
1. Setting Co (324(0, 0)6 we have [ y Co Alwl + Iv. for all y, E R:
2. The map f 02((r(-), f()) from (Z) to (Z) is well-defined and L-Lipschitz.
3. The second derivative with respect the second argument is bounded: (8gf(y. v/) Lfo all v E R.
Proof.
1. Write Bgl(g.5/) Dglly. y 0z((0, (,,00,,)+00((000)) and apply the triangle inequality as well as Assumption 3.2 Item
2. From the previous item we know this map is well-defined. If and g belong to (2), we have
- -000:/00)0B0aaa--------2-------0
[[02(r((), 02(0).
I 1,2 [1( 9(
= Lf - e1liz(2)
3. It follows directly from Assumption 3.2 Item
Proof of Proposition 3.3
We start by computing the directional derivative of R at h in the direction
f. denoted by DR[[(():
= lim 1
RR(h + 5f) - R(F)
8-0 5
= lim
E [(( +5) - -((((),
8-0 5
= lim
P(N())
E(r(Z), P[A](Z)]
6-00
1
lim 5
E
P(J(Z))
8-20
8²
())((()) +0][()) - PLN(Z)
+ -
2
= E(0)((((2), P(N](2)) P[]((2)
8 (2 P(h + +f](2)) - PI/((()²
+ lim E
8-+0
2
(a) P (2)) P2)(
where 9 e (0,8) 3) is due to Taylor's 's formula. The last step is then due to Proposition A.1 Iem
We
can in fact expand the calculation a bit more, as follows:
DR(}[(f) E [[a)((r(Z), P(h](Z) P[)](Z)]
= P(M()) P()
= (P")0)((r(Z), (()())) f)
This shows that R is diffeuuxxddiffeetiiabbll. with Gâteaux derivative at h given by
DR(A)] = P P()]())))
By Proposition A. Item we have that DR[h] is continuous mapping from L2) to L*))), which implies that R is also Préchet-differentiable and both derivatives coincide (29 Therefore,
VR(h) P (211)())
(13)
13

---

Proof of Corollary 3.4
From Proposition 3.3 we know that
VR(h)(=) (bllrrrr,PMl)))))==-==0000000000
X = = x]
(2|x [=)0=((((())))[(())) d= //2 px,z(T, 2)
02 ()(((()) da px
Pz(z) px, 02 02e(((2))))))))) ((h(())) dz B (2) 210,
px(r)pz(z)
where we define 2(), px,z (z, Notice we may assume that px (x 0, since the set of X' such that this happens has probability 1, and (R(h), being an element of L (x)),, is only defined almost surely.
pz) 0 and (xx 2) = pz = 0.
Before presenting the proof of Theorem we need to prove two auxiliary lemmas. To lighten the notation, the symbols ||-I| and ( ). when written without subscript to specify which space they refer to, will act as the norm and inner product, respectively, of L,2 (X)
Lemma A.2. In the procedure of Algorithm furthermore,
we have um E L2 for all 1 S m S M and,
E Ez
Um 1 ((O,F,
where
= = (cB + 1.1111112112 + 1112211100)
Proof. By Assumption
4.2 we have:
|m 12x) |, (m)=2m)0r (F(zm), ), PT (1mm-1](=m)) (((m)) (1(2(x)
=Ex | 2m 02l )02 (F(zm); P plum- 15( )
Ex
< )g((F(zm)) Pl )(00)
Hence, Um (X) for all m. This computation and Proposition A. Item |
then imply
Ex ||mmel| 1l ?]]
s 2 (cO + L
[F + || hm 10112212(((2)
< 11
(CB ( 1.2
(17122(2) 11 17mm
S 3
(c + 1.2
(1
p2118190))
3 (c) + L 111 + L2 13 A P))
Lemma A.3. In the procedure of Algorithm 7 we have
1 VR(mm-1)
um k5( ()) (1 $ 11 + ||r Flliz12 +IIP PP123)
where
max {3(C)
+ (3(C 1''I'
2, 21 2
Proof. To ease the notation, we define
V. (Z) A (2((((Z), Ph (2)).
(Z) e az{(F(Z), P fT (2)).
14

---

Let's expand the definition of ||-1|:
( umm (X)) 2
E Bz..
(R(hm-1)
umlll
Ex
Bzm
27.
Ex
(VR(hm--)(()) - EEz.. [umm
(X)1) [um((X)]
27
Ex
Bz [xx 1 mZ - Ezn [o 2m mm) )L m (zm))
Ex (Bz [ 2) Vm (2) ( (2)) ")8 2)
2) (2)1)
Now we add and subtract $(X, 4) m (Z), so that
Z) W m (7) 6(X, 2) 1)) Ex (Bzz ((X,
[zz U (2) 2)) + (0) 2xx - (2(()))) 27
Ex
|6 (122)))) V -m (mllep(2) 27
Ex
s v2x
- 6(x, ((x.)))((z(2) + | 112222))) | # 9ml122221)
v2 (1Vmlliaz
3 ())((x077)))) + | |011 Vm - (mlli1z))
where
/
|| | (P
Φ(x, =) p(x(((((( dadæ
xxx
the norm with respect to the independent coupling of the distributions of and Z. By Proposition we have
A.
Ez (2)m(()))((m-))2))
(co + L (Ir(2)) + (p 1)
Ez
< (C8 L. 1 (z(
(/) + I (21Ip(mm-illiAe))
3 (C) + 3] +22
It is also clear that, by Assumption 4.2
1
Finally, by Assumption 3.2 2 we also have
(az((r(Z), Ph... p O2((F(Z), (Tpnii)),TTT,naa P( 1(())) )
I m
Ez
2T.
(Ir- Fli-z1)) + 11(P -P)(fm-1111
< 21 (Ir- FlLi(12) + D21pP- - P113))
To combine all terms, we first define
k² ④ max (3CC + _ + ,2 21
2L2
21 D 1210011 10
Then, it's easy to see that
E (R(mm)) um]ll 5 <x(Ō) (1 212 + -flr-21 + +11p -1IP-1i
}11D-11130)
as wanted to show.
15

---

We now have everything needed to show the
Proof of Theorem 4.3 We start by checking that R is convex in H: if h, H and [0, 1], then
R(Ah+ + (1 2)g) = P[M + (1 - x)g](2))]
(((((((2),xPII((22) + (1 (1-x))))
& XB(((r(2), +
XR(h)) + (1-A)R())
By Assumption 3.1
h
7
T0
By the Algorithm procedure, we have
hm h" 11
proju im- am Um h**
2
- 2
Am--1 mm - 1011
Am-- 1²112 am (umm 7m- h²) + a
u4m liml?
2
After adding and subtracting 0mm (VR(mm- 1), T) Tmm-1 h²),
we are left with
1imm h- Om (um VR(h... 1), Am--1 --1)
| - (VR 11
2
Applying the first order convexity inequality on the last term give us, in total,
11 h 11 < 2 7mm ht 11 Omm (um VR(mm--))) 71m-1-117
2
Um 12 - am (R (R(mmm-)) R(h))))
Hence, making this substitution and rearranging terms, we get R( - R(h*)= S
1
(1 - 1---hh1 11 l17m 1²1113)
20
a'm 1m 112 12- {um 1R) 1²))
+
2
Finally, summing over 1 S m S M leads to
M
Σ R(fm--1) R(h*))
(10m-11 - 10112 -liim 12112)
20mm
M
Olm
| 11
(14)
2
M
(R((m--))) - Uyy Rmm-1 h*).
M=l
The next step is to take the average of both sides with respect to 21:M, taking advantage of the independence between #1:as and DD the data used to compute क, f and P. Each summation in the RHS is then bounded separately.
The first summation admits deterministic bound. By assumption, the diameter D of H. is finite. Hence
M
M
Σ
(16m-1- h² 1112 - -m - h1" h²112)
Σ
1i - h 1=1l1
20m
2a
20m
m=1
m=2
1
Pho h²
112
1
| h11122
2a1
20
M
1
1
D22
D2
+
D2
20mm
20mm-1
2001
20
m=2
(15)
WeWe want to remind the reader that it is enough to assume that there exists h E (h" + ker P) n H. By the detinition of h, we have R(h) R.
16

---

The second summation can be bounded with the aid of Lemma A.2
M
ESEM
O'm
Um
2
1
Ex BaLAr
[?2]
am
1
p
F,
Σ
am
(16)
2
2
Finally, the third summation be bounded using Lemma with respect to 21,-... 2m- 2m.1..... 2M and notice that
A.3
Let Rz denote the expectation
Eznar
(VR(mm-1)) - um. Am-11 - n*)] Ez-m
Ezm
Umm 7im- - 11²))
Eg
{Bem
VR(hm-1) 11m TTm=1 1141)
Bx-m
|
VR(hm-1)
1m m1 - 1171
DE
||EZ
(TR(Tmmm1 1mm 11
Then, applying Lemma A.3 and setting - DE we get
E Ez
(VR(hm-
Um Rm-1- h^>
(17)
S T
$
(1: + || Flliz(2) + +1IP- - P
All that is left do is to apply equations 14 15 (66) and (I7 along with the inequality which defines convexity. Let 7 & M
Em= Am-- and E - p/2. Then:
((() R((h*)
EZ
E EzLM (R(hm)) R(h*))
M
m=1
D2
+ [ M
Σ Om
2.acM
2 ()(xx772) + +|| + fll22(z) + PP p11ep) i
+ ( (100
B Comparison with other methods
In this section, we provide some theoretical comparisons with other notable methods for NPIV estimation.
B.1
KIV
B.1.1
Risk bound
In our formulation, we substitute the task for estimating P by that of estimating the ratio of densities &. In this way, we bypass the need to evaluate operator estimator p* in every iteration, replacing this by evaluating & once and multiplying it by the derivative of the loss function (cf. Algorithm 1). Aside from the computational advantage this poses, it is interesting to investigate the differences which arise in theoretical bounds when comparing with method which ends up using an estimator of P
In KIV, the authors propose a first stage procedure which estimates F. (P, in our notation) and, in the second stage, they use the adjoint of this operator through the kernel mean embedding μu()) = E (x)) (ø is the feature map,
the notation in (34)). To estimate E", they use the adjoint of their estimator ET of E, where n is the sample size and is a regularization parameter.
In our Theorem 4.3
provided the learning rate has been appropriately chosen, the excess risk is upper bounded by 1 + + || -l + |P Pl. In fact, ||r fll is just the risk for simple regression
17

---

problem, and so we will focus our analysis on the other two terms. In KIV, the excess risk is bounded using (37] Theorem 2]. according to which it is smaller than the sum of five terms. We reproduce this bound here for a more detailed analysis:
Theorem B.1 (Proposition 32 of (34)) The excess risk of KIV's stage estimator H can be bounded by five terms:
E(HT)) - -(H,) - 5|5_1 + So + A(E) + S + S2),
(18)
where
8-1 || (T (T) g)
S || o (T+ TH Ha
S = o (T+ || THp
82 || o (T+ T) Hp Hp)l
A( = - -
In our notation, & corresponds to R, H corresponds and Hp corresponds to h". The space Hg a RKHS which contains the solution Hp. . is also important recall the definition of the terms in bold, since they depend on (z) = (ET) " ((z):
T
T (EE):
Ω (E.))
m
m
m
1 m
T
g
m
m
Where & and D; are second-stage samples. We refer the reader to the other mathematical objects. Here, we will argue that
34
for detailed description of
8 Sy is analogous to || क + PP-Pli PI.
(19)
The other terms in Equation (8) are controlled by optimally choosing how the regularizing parameter & goes to zero. This relates to the first two terms of our bound in Equation (1O) that goes to zero as the SGD iteration increases, long
properly choose the learning rate. In 34 Proposition 35),
these two terms are bounded as follows:
S-1 < IIV - T + E) -
(1(((et))
g -g1
So S T (T + 5) -112
T
T(2 H Hnn
Then, 34
Proposition 37] refers 37 Supplements 7.1.1 and 7.1.2] to bound [ 37 one can see that these bounds amount
-g and
1 g
2 < C.
(x)())) n (2) 101
T 2 < D
(2)(())) m())))
where C and D are constants. Then, KIV uses the bound on ((x)(E)) (( provided in their Corollary I, which based on the inequality
- p(2)) < <I E,1 (1))
in 37 there is 2h in the exponent instead of which is translated to 2: KIV. However, this is only matter of assuming Holder continuity of I2 with exponent v. one assumes Lipschitz continuity, the exponent becomes E. E.
18

---

Hence, with high probability and ignoring constants, we have
_ gl E Ex Epll | S EX -Epll
This would be an intermediate result Therefore, the estimation error of E when compared to impacts the risk of KIV's estimator through
Proposition 37].
two terms. These analogous to two terms | - Φ and ||p PlI, since ||
(Fx @P2)
the Hilbert-Schmidt norm the difference between P and the integral operator induced by [8 Theorem 2.42].
B.1.2 Binary outcomes
In Section 6
show that, under the binary outcomes setting, the correct formulation of the risk is
R(h) E[BCE((((2)
In KIV's notation, this would becom 14
E(H) ([BCE(Y,F(HH((2
Substituting the populational mean by the empirical one, taking into account that must estimate and applying regularization, we get (cf. [34] Section 4.2])
m Σ F(H || m
Em (H)
Has
Using the fact that Hy and H.x are isometrically isomorphic 34 Section 4.2], we can formulate the risk in an equivalent way, now defined on H.x:
2m
(h)
BCE(
1=1
Then, the KIV estimator is defined as
hg
arg min Em (h).
For the quadratic loss, KIV finds the solution in closed form, since this is a ridge regression problem. However, for this new loss, no obvious closed form solution exists. Applying the Representer Theorem, we find that the solution hm should satisfy
hm
anb((r))
for some a E R", where (bxx,) K(n,)) :) is the reproducing kernel of Hx. Then, one left with the following optimization problem:
arg min
Σ BCE(J, F(a' ((()) + Ça' X
RER"
Since (ETh - a'w(z) (see the definition of in 33 Appendix A.5.1). An approximate solution to this nt-dimensionl optimization problem must be found with numerical methods. Keep in mind that this problem must be solved several times in order to find a good value of the regularization parameter via cross-validation, which is crucial for the performance of regularized regression methods. contrast, our method does not require additional work after defining the modified risk.
We use symbol S mean that LHS smaller or equal than constant times the RHS, where constant does not depend on data.
"Since BCB is linear in the first argument, does not matter use Y or r(2) to define the risk.
19

---

B.2 Dual IV
In DualIV, the risk minimization problem is reformulated into a saddle point optimization problem. which we write in their notation:
min max V u). JEF well
where
Z)] - Eyzley (u(YY
Here, is the space which contains the solution to the NPIV estimation problem, 24 is the space of the "dual function" and Ey is the Fenchel dual of the function by = l(..)) In the binary outcomes setting, the loss is given by f(y,z') ) (CE((,,(y)) If F is the logistic function, i.c. Fly') cxp(y)) (11))/(111 + exp(y')) )) one may verify that
(u)) { H(u + v), i u + t
+00, otherwise,
where Hpp) -pplogp + p) log(1 - p)] is the entropy. The empirical minimax objective then becomes
((f u) = 1 n n Σ f(x)))()))))
1 n - Hi + u(ys, 2,)).
with the constrain that 0 S 1.+tuuuu =;) s for every Again, contrast with the case of quadratic loss, when one may choose F and / to be RKHS and obtain closed form ridge-regularized solution, this formulation is not as tractable.
B.3 GMM
In GMM based methods for continuous outcomes, the main idea is that since Elg /] = 0 and
Y h* (X) + E, we have E|Y (X) Z] 0 and, hence, E[f(Z) h- (X)) 0 for every
f (Z). . If we choose large number of functions fm. we may search for ht by finding function which minimizes
m
Σ
Va S1) where (h, f3)
n Σ (Z)))(Y) A(X.)).
(20)
This is possible because the error term can be written as Y - (X)). Under the structural equation Y 1(hh(X) + 0}. such a characterization of the error term not available, which already presents a difficulty. Nonetheless, as we show in Section under this new equation we have E Z (Z)) Therefore, moment conditions could be obtained through E{(Y [(P(h" (Z)))((Z) = 0 for every f E (Z)... However, this would create the need to introduce estimator P of P. This defeats one of the main advantages of turning GMM, which is to avoid the first stage two-stage methods, since errors in the first stage can have a large impact on the resulting estimator.
C Implementation and experiment details
In this section, we give practical guidelines for implementing SAGD-IV, as well further details regarding the baseline methods below:
KIV: Since the original implementation was only publicly available in Matlab, to facilitate reproducibility of our results we re- re-implemented this method in Python. This was done following the guidelines presented in the paper.
DuallV: Similarly to KIV, we rewrote the method in Python using the closed form solutions and the hyperparameter selection procedure present in the paper.
DeepGMM: We used the publicly available Python implementation
* DeepIV: We used the implementation available as part of the EconML package 5
20

---

2SLS: We include the classic Two Stage Least Squares procedure to show the benefits of following a nonparametric approach.
All experiments were conducted on Apple Sillicon MI Pro CPU. No GPUs were used. Remark C.1 (Experiments sample size). All four baseline methods
considering need single dataset (X, Z, Y) to fit their estimator (hyperpameter validation included), whereas our method needs one dataset of (X, Z, Y) triplets to compute $, F and τ. and one dataset of Z samples to conduct the SAGD loop. To make fair comparisons, we trained each method using the same total amount of random variable samples. For example, DeepGMM might use 800 triplets (XX Z, Y) for training and 200 triplets for validation, whereas SAGD-IV may use 600 triplets for fitting the preliminary estimators and then 1200 samples of Z for the SAGD loop, both totaling 3000 random variable samples used to adjust the model.
C.1 SAGD-IV
C.1.1 On computing P, & and F
Ratio of Densities. Density ratio estimation is well studied problem in Machine Learning, with (36] being a good resource for acquainting oneself with relevant algorithms. The approaches we follow here consist on obtaining an estimator & that minimizes the (empirical version of the) quantity We consider two options for T: Reproducing Kernel Hilbert Space (RKHS) estimator obtained through the Unconstrained Least Squares Importance Fitting (uLSIF) framework described (36] and also a neural network trained to minimize the same objective over its parameters. By changing the kernel or the activation function used, can make sure this estimator adheres to Assumption Item
2
Conditional expectation operator. This is the most complex object we must estimate, since it encompasses all possible regressions of functions of X over Z. Not many options exist for this task, with popular one [13 being the use of Nadaraya- -attoo kernels [26 39]. We, however, chose employ the estimation method used as first stage in KIV (34). which transforms the problem into vector-valued RKHS ridge regression, due to its straightforward implementation.
Conditional mean of Y given Z. This, in contrast, is the simplest unknown in our formulation, being the regression function of Y over Z. We again consider two options for this task: the first is to employ the method previously discussed in order to estimate the conditional expectation operator of Y given Z, and then apply it to the identity function. The second option is to train general purpose regressor, such as neural network.
C.1.2 Hyperparameters
Since the final estimator in Algorithm T is an average over the whole path of estimates produced by the SAGD loop, we can improve upon its quality by discarding the first K estimates ht,..., hgg where K must be chosen independently from M. In this way, it if started our algorithm from l after a warm up period.
Regarding sample splitting, our theoretical results do not specify an optimal ratio between the M SAGD iterations and the N samples of (X, Z, Y) used to compute $, F and P. We empirically determined that M 2N a good guideline to follow, and it the ratio we employed in Section [ Of course, in real data scenarios this ratio might be constrained due to limited data availability.
About hy hyperparameters:
The Algorithm learning rate was set to be am 1/VM M for 1 S S M:
The warm up time K was set 100;
The set H. was chosen in Equation (6). with A set to 10. During experiments, found that this parameter, aside from needing to be large enough for us to have h" e H, did not have any noticeable influence the resulting estimator.
For Kernel SAGD-IV:
21

---

We used Gaussian kernels, with lenghtscale parameter determined through the median heuristic;
-The regularization parameters for the three kernel ridge regressions were chosen through cross validation. Since these parameters are essential for actual learning to take place, selected the values to be tested through an iterative procedure, described in Algorithm
For Deep SAGD-IV:
For both networks in the roles of & and f, we used two hidden dense layers of sizes 64 and 32. the continuous response setting the activation function used was ReLU, while for the binary response one we used the sigmoid activation. For 3 we used dropout with rate 0.01.
Both networks were trained with a batch size 512, using the Adam 21 optimizer with learning rate of 0.01, accross 1.5 10 /N epochs. We adopted L regularization the two estimators, with regularization parameters of 3 10 for and 5 10-3 for T. We also implemented early stopping in the two training procedures.
We remind the reader that both SAGD-IV variants use kernel methods compute P.
C.2 DeepGMM
The modifications made with respect the original paper were the following:
Instead of using 50% of the samples for training and 50% for validation, adopted train validation split of 80 20.
We changed the batch size from 1024 to 256. This was done because we supplied the algorithm with 1000 training samples instead of the 2000 used in [6], and hence batch size 1024 would not produce stochastic gradients.
C.3 DeepIV
We followed all the guidelines presented in the original paper for low dimensional settings.
C.4
KIV
We employed 50/50 split between (X, Y) and (X, 2, Y) observations (see (34), Appendix
A.5.2), Regularization parameter candidates were also chosen using the selection method in Algo- rithm 2
Algorithm 2 Search for regularization parameter
Input: Loss function L :: R - R. List of initial values parameters = [AI,..., Bhy...,,, An). Initial offset Number of iterations T'
Output: X*
for < t t S T do
X < arg min (L(A) for À in parameters}
parameters + [X* + ke for E (-5, -4, 11.5,,....... -4...,,J| +- e/10
end for
C.5 DualIV
We followed the hyperparameter selection guidelines given in Section 4 of 25 This means we used a 50/50 550 split for train/validation samples and chose the regularization parameters from grid search, minimizing the squared norm of the dual function
22

---

D Small data regime
This section presents version of the experiments in Section 5.1 with half the sample size, providing a comparison between SAGD-IV and other methods within a smaller data availability regime. Figure contains the corresponding log-MSE results.
Contrasting Figures and we can see that the relative performances for each method mostly stay the same. Some methods, such Deep-IV in the abs scenario, exhibited enror distribution with larger variance, but this to be expected from the reduction in sample size, which makes any given dataset realization worse depiction of the true data distribution. These results support the conclusion that our method's performance is competitive with other state-of-the-arr approaches in more data availability conditions.
Step
Abs
-0.5
Out of sample log- MSE
-1.0
-1.5
Linear
0.0
Kand BAGD-V
Deep RAGDav
AZV DespOttt
Daeptyy
TSLA Dual Tv
Korast RASDaV Deep SAGD-IV
KTV Degolnr
Deepty
PSL.
Dual TV
Figure Log-MSE results for the experiments Section 5.1 with half the sample size.
E
Alternative risk definition
Computing the risk using r(Z) or Y are equivalent under the quadratic loss, however, one might think that considering Y instead of r(Z) in our formulation would lead to an algorithm that does not need to estimate r()). Below show that this is not the case, since the gradients of the risk measure under both formulations are the same. This implics that we would still need to estimate r for the algorithm.
instead of Equation we consider the risk defined as R(h) P()(()))
then proceeding in exactly the same way in the proof of Proposition
3.3
would arrive
DR(M
P[f[(2)
As it is, this expression is inner product in (Z) since Y is not measurable with respect Z. However, age linear with respect to its first argument, which is true for the quadratic and binary cross entropy losses, have
DŘ( E(0ge(Y, P()](Z)) PZ]]
E [EjO P())()) P[)(()) Z
E ( 2], (()) (P]]]Z)
E [a,((r(Z), (())( P[]](Z)),
which is the same expression obtained before. Hence, the gradient R. still depends on r(Z) and the resulting method still needs to estimate this guantity.
23

---

Nonetheless, one could still use the sample as an estimate of r(2:). However, this is not the best choice, because the term || (f-rr((=(2) which appears in the RHS of Equation (0)) would become E|Var[Y] Z]]. This is constant with respect to the size of the auxiliary dataset (cf. Assumption 4.2) 4.2)
and could be improved by taking f to be a better regressor of Y over Z.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We do not overstate the importance of our work and results. The content the abstract and introduction properly reflect the impacts of our theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations the algorithm are discussed in Section 4.1.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification: The set of assumptions is listed along the paper using a dedicated ETX envi- ronment and the proofs of all results in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Implementation details are presented in Section 4. and in Appendix C Following these directions, one could reproduce all results. We are also submitting an anonymized version of the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We synthetic data in all our experiments and we followed NeurlPS code guidelines to submit an anonymized version of the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
CodeSuomi ssiniiolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Implementation details are presented in Section 4.1 and in Appendix C Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Figures and present error bars for the log-MSE of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources are disclosed in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have conformed with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work discusses only theoretical and computational aspects of an estimation problem. We do not foresee any societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All methods are properly cited.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "DAPTQ3yNO7", "reviewer": "Reviewer_WKaj", "summary": "This paper introduces a novel IV method, called the SAGD-IV, which is more efficient and stable in the NPIV regression. Two different variants of the SAGD-IV are given, and a range of comparisons between these variants with the existing methods are given. Moreover, the performance of SAGD is not only shown in a continuous response case but also is extended to a discrete case.", "questions": "I wonder if the error term is not additive and but is a non-linear function of $X$ and $\\epsilon$, would the SAGD-IV methods still work?", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The theoretical results are solid, and it also considered different aspects of the limitation, such as the risk while computing the gradient, different types of $h^{*}$ and variants of SAGD-IV for different circumstances.\n\nI really enjoy looking at the comparison between SAGD-IV and existing methods with different options of $h^{*}$.", "weaknesses": "There are a lot of mathematical equations and deductions, like theories, assumptions etc, which is great and indicates that you have a pretty solid work, but it is also great to spend a bit more room for storytelling, which will bring people straight to the main point and get your amazing work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments!\n\nConcerning the weakness, we appreciate the reviewer's input on our exposition and we will improve the discussion on IV and the comparison with current approaches.\n\nRegarding your question, it is possible to use SAGD-IV for other models. However, it depends on finding an appropriate risk functional $\\mathcal{R}$, which itself depends on finding an appropriate pointwise loss $\\ell$. With the pointwise loss in hands, our method is immediately applicable after computing the derivative $\\partial_2 \\ell$.\n\nFor a general approach, if we assume that $Y = \\Lambda(h^*(X), \\varepsilon)$ for some nonlinear function $\\Lambda$, we have to find a function $F$ such that $\\mathbb{E}[ Y \\mid Z ] = \\mathbb{E}[\\Lambda(h^*(X), \\varepsilon) \\mid Z ] = F( \\mathbb{E}[h^*(X) \\mid Z] )$. Then, if $Y$ is real-valued, a possible choice for $\\ell$ is the $L^2$ loss $\\ell(y,y') = \\frac{1}{2} (y - F(y'))^2$, and, if $Y$ is discrete, one may use the binary cross entropy loss $\\ell(y, y') = BCE(y, F(y'))$, as was done in Section 6 for the binary outcome setting, where $\\Lambda(h(X), \\varepsilon) = 1 \\\\{ h(X) + \\varepsilon > 0 \\\\}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t2xx8uwsKC", "reviewer": "Reviewer_1tRc", "summary": "This paper considers the standard problem of non-parametric instrumental variable estimation (NPIV) and proposes a new approach of functional stochastic gradient descent to solve it (SAGD–IV), where the gradient estimator can be implemented and adapted using certain machine learning or deep learning techniques. Theoretical guarantee of the finite-time convergence of the algorithm is provided under suitable assumptions. Simulated experiments show the effectiveness of the proposed method to some extent.", "questions": "1. Could you please highlight the technical difficulty and the novelty in proving the main theoretical results of the convergence of SAGD-IV?\n\n2. It is not clearly stated how to exactly perform the functional gradient descent step as proposed in Algorithm 1 in the practical experiments. Could the authors elaborate on this a bit more? \n\n3. What is the performance of the proposed algorithm in the small data regime when compared with previous methods?", "limitations": "Please see the above weakness section and the question section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality and Significance:**\n\n1. The idea of using functional gradient descent to solve the NPIV problem is somehow new and interesting, which is different from most previous relevant works. \n2. The derivation of the functional gradient form in Equation (8) is important since it allows us to decouple the estimation of $\\mathcal{P}$ and $\\mathcal{P}^{\\ast}$ and gives a computationally efficient way to estimate the gradient of the risk functional.\n3. The algorithm can be easily adapted to different machine learning techniques to estimate the functional gradient, and can also be adapted to either continuous or binary outcomes as demonstrated in the experiments.\n\n**Clarity and Quality:**\n\nThe presentation is clear. The problem setup is nicely formulated and all the assumptions required are well stated. The main theoretical result is also sound and is well proved.", "weaknesses": "1. The consistency or sample-complexity of the proposed method is unknown. This type of finite sample results are presented in recent previous methods on NPIV including but not limited to [1, 2, 3]. \n\n2. The technique and idea behind the proof of the convergence of SAGD-IV is relatively standard and straightforward given that the risk function $\\mathcal{R}$ is convex in the functional space.\n\n\n**References:**\n\n[1] Singh, R., Sahani, M., & Gretton, A. (2019). Kernel instrumental variable regression. Advances in Neural Information Processing Systems, 32.\n\n[2] Bennett, A., Kallus, N., Mao, X., Newey, W., Syrgkanis, V., & Uehara, M. (2023, July). Minimax Instrumental Variable Regression and $ L_2 $ Convergence Guarantees without Identification or Closedness. In The Thirty Sixth Annual Conference on Learning Theory (pp. 2291-2318). PMLR.\n\n[3] Li, Z., Lan, H., Syrgkanis, V., Wang, M., & Uehara, M. (2024). Regularized DeepIV with Model Selection. arXiv preprint arXiv:2403.04236.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing a careful assessment of our work. In what follows, we address the weaknesses and questions in a point by point fashion.\n\n### **Weaknesses**\n\n1. We are assuming that \"consistency\" here means convergence of $||\\hat{h} - h^*||$ to $0$ as the number of data points grows to infinity. (In [1] the authors use this word to mean convergence of the *projected risk* to zero, the same type of convergence we provide in Theorem 4.3.).\n\n   Consistency for NPIV regression is, as we mention in Remark 4.5, a very challenging problem. The cited methods [2] and [3], as well as others which prove this kind of result (e.g. [5]), are based on Ridge regression and, hence,  can leverage the source condition, ubiquitous in the inverse problems literature, to obtain consistency. In the context of SGD, convergence to the global minimum could be achieved when the function being minimized is strongly convex, which, in our context, translates to restrictions on the operator $\\mathcal{P}$, as we point out in Remark 4.5. A similar assumption was also made in [4, Theorem 7, Appendix E.4].\n\n   With respect to sample-complexity, note that $|\\mathcal{D}|$ (cf. Assumption 4.2) can be seen as the *first stage sample size*, while the number $M$ in Algorithm 1 is the *second stage sample size*. The first two terms on the RHS of Equation (10) quantify the rate of convergence with respect to $M$, while the dependence on $|\\mathcal{D}|$ is in the $\\zeta$ term. Note that this term is comprised of three estimation errors, whose exact dependence on $|\\mathcal{D}|$ can only be known after deciding on specific estimation methods. We did not expand this term further since we chose to be agnostic with respect to the first stage estimators. Nonetheless, if one wants to obtain explicit bounds in $|\\mathcal{D}|$, rates for $||\\hat{\\Phi} - \\Phi||$ can be found in [6, Theorem 14.16], while rates for $||\\hat{\\mathcal{P}} - \\mathcal{P}||$ are present in [1, Theorem 2] and [7, Theorem 5]. The other term, $||\\hat{r} - r||$, is the risk for a simple real-valued regression problem, and various rates are available depending on the chosen regressor and the degree of smoothness assumed on $r$. The book [8] contains several results of this type.\n\n   We agree that this is a discussion which should be in the main text, and we will update Section 4.2 to include it.\n\n2. The novelty of our work does not come necessarily from the proof technique of Theorem 4.3, but rather from the insight that, by tackling NPIV with functional stochastic gradient descent and by carefully exploiting its structure, we are able to obtain an algorithm with improved flexibility that is also capable of addressing other loss functions. This reframing is what allows not only the development of an intuitive and competitive algorithm for the challenging NPIV problem, but also the use of standard tools from convex analysis to theoretically analyze it, which we believe to be an important contribution to the literature and to the NeurIPS community.\n\n   Moreover, while we agree with the referee that the proof technique is standard, we point out that Theorem 4.3 differs from straightforward convex analysis in the sense that the operator associated with the NPIV inverse problem is unknown. This is addressed in Lemma A.3, where we must carefully analyze a term which usually vanishes in simpler instances of statistical inverse problems.\n\n\n### **Questions**\n\n1. See weakness 2.\n\n2. Thank you for pointing out that more details are needed to parse the algorithm implemention. We will enrich the discussion in Appendix C and the code will be made available. SAGD-IV needs one dataset $\\mathcal{D}$ of samples from the triplet $(X, Z, Y)$ to compute $\\hat{\\Phi}, \\hat{\\mathcal{P}}$ and $\\hat{r}$ (cf. Assumption 4.2 and Remark C.1). Then, it needs another dataset, say, $\\mathcal{D}\\_Z$, of samples from only $Z$ to conduct the functional gradient descent in Algorithm 1. After one has the estimators $\\hat{\\Phi}, \\hat{\\mathcal{P}}$ and $\\hat{r}$ in hands, to evaluate $\\hat{h}(x)$ one must loop through the samples in $\\mathcal{D}\\_Z$, computing the stochastic gradient evaluated at $x$ and using it to perform the projected gradient descent step. There are optimizations which can speed up this process, for instance, the term $\\partial_{ 2 } \\ell \\left( \\hat{ r } ( z_{ m } ), \\hat{ \\mathcal{P} } [ \\hat{ h }\\_{ m - 1 } ] ( z\\_{ m } ) \\right)$ does not depend on $x$ and, hence, may be left pre-computed. Then, the only effort in computing $\\hat{h}(x)$ is in computing $\\hat{\\Phi}(x, z_m)$ for $1 \\leq m \\leq M$.\n\n3. We believe that our experiments already fall within a relatively small data regime. Nonetheless, we agree that it is interesting to explore this direction and we ran the experiments with half the sample size. The results can be found in the attached PDF and will be included in the Appendix. We can see that the relative performance of each method largely stays the same. However, as is common with fewer data points, the log-MSE distribution for each method generally exhibited higher variance.\n\n**References**\n\n[4]  Muandet, K., Mehrjou, A., Lee, S. K., & Raj, A. (2020). Dual instrumental variable regression. Advances in Neural Information Processing Systems, 33, 2710-2721.\n\n[5] Darolles, S., Fan, Y., Florens, J. P., & Renault, E. (2011). NONPARAMETRIC INSTRUMENTAL REGRESSION. Econometrica, 79(5), 1541–1565. http://www.jstor.org/stable/41237784\n\n[6] Sugiyama M, Suzuki T, Kanamori T. Density Ratio Estimation in Machine Learning. Cambridge University Press; 2012.\n\n[7] Talwai, P., Shameli, A., & Simchi-Levi, D. (2022, May). Sobolev norm learning rates for conditional mean embeddings. In International conference on artificial intelligence and statistics(pp. 10422-10447). PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RKZRSWA1ii", "reviewer": "Reviewer_spVa", "summary": "The authors propose an estimator for nonparametric instrumental variable regression (NPIV), with an extension to the binary outcome case. They prove that the excess risk of the projected estimator is controlled by rates of the density ratio, a regression, and the conditional expectation operator. Extensive comparative simulations are conducted. Careful analysis in the appendix also compares this approach with others.", "questions": "Are there known rates for the density ratio given in the first term of zeta? Please provide at least one reference where a rate in this norm, or a sufficient norm, is given.\n\nSimilarly, it would be good to point to rates for the third term of zeta, such as Theorem 2 of ''Kernel Instrumental Variable Regression'' by Singh et al. (2019), Theorem 5 of ''Sobolev Norm Learning Rates for Conditional Mean Embeddings'' by Talwai et al. (2022), and Proposition S5 of ''Kernel Methods for Causal Functions: Dose, Heterogeneous and Incremental Response Curves'' by Singh et al. (2024).", "limitations": "NA", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The main result is impressively agnostic about the component estimators.\n\nThe paper is very well written, and I particularly appreciated the authors’ comparison across formulations and notations in the literature.", "weaknesses": "Algorithm 1 of ''Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach’’ by Liao et al. (2020) is another stochastic approximate gradient approach for NPIV. I believe the difference in the algorithms amounts to whether the primal or dual formulation is optimized with stochastic gradients. A clarification would be welcome.\n\nIt would be good to put into words that the excess risk being analyzed is for objects projected onto the instrument space, which is common in this literature but perhaps less so in the NeurIPS community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the attentive analysis and relevant suggestions. Below, we separately address the weaknesses and questions.\n\n### **Weaknesses**\n\nThank you very much for pointing out such an interesting paper. The cited work explicitly uses NN classes as approximations to $L^2$ spaces and conducts SGD on the *weights* of these neural nets. This is different from our approach, which formulates the SGD updates *within the function space $L^2(X)$* and does not assume a parametric form for the stochastic gradients. Another difference is that our work directly addresses the primal problem and Liao et al. (2020) converts it to a saddle-point problem in order to apply a primal-dual algorithm, as you mentioned. Additionally, we do not have any regularization and we do not assume $r$ is known (page 5, paragraph 2, $b$ in their notation). We will include this discussion in the literature review.\n\nRegarding the matter of \"risk\" versus \"projected risk\", we wholly agree and will include an appropriate remark in Section 3.\n\n### **Questions**\n\nIn [1, Theorem 14.16] the authors provide a convergence rate for our proposed norm of $((\\log n) / n)^{1/(2 + \\gamma)}$, where $\\gamma$ controls the decay of the spectrum of the kernel used to estimate the density ratio.\n\nConcerning the references for the operator regression convergence rate, we thank the reviewer for bringing the second and third to our attention, which will be included in the final version of the paper.\nWe could not find Proposition S5 in the third paper but if the referee can clarify this misunderstanding, we would appreciate it.\nWe will add a paragraph in Subsection 4.2. pointing out these known rates.\n\n**References**\n\n[1] Sugiyama M, Suzuki T, Kanamori T. Density Ratio Estimation in Machine Learning. Cambridge University Press; 2012.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SLBsa0cxoB", "reviewer": "Reviewer_xcFr", "summary": "This paper studies the nonparametric instrumental variable (NPIV) regression problem. The authors present a new algorithm, SAGD-IV, which utilizes stochastic gradient descent in a function space to directly minimize the populational risk. The gradient can be computed via computing the estimating the conditional density $\\Phi$ and the conditional expectation operator $\\mathcal{P}$. This approach is distinguished by its flexibility, accommodating a variety of supervised learning algorithms, such as neural networks and kernel methods, and supporting both continuous and binary outcome scenarios. Finally, the authors also proved the consistency of the NPIV estimator under regularity assumptions.", "questions": "See weakness.", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Estimating the gradient to perform SGD in the second stage for NPIV appears to be novel.\nThe algorithms can also be applied to non-quadratic loss function is also new.", "weaknesses": "I have several concerns of the paper: \n\n1 The author claims that existing methods in estimating NPIV solutions face the problem of cannot apply to large, high dimensional datasets. Typically in the NPIV context, such a problem arises due to the estimation of the condtional density function $\\Phi$. However, in the paper, computing the gradient also requires us to compute the conditonal density fucntion, so I do not see how such limitation on high dimensional dataset can be avoided.\n\n2. The novelty of the algorithm relies on performing the sgd in the second stage after we estimate $\\Phi$ and $\\mathcal{P}$. I believe in standard two stage estimation methods, once we obtain $\\mathcal{P}$ (no estimationf of $\\Phi$ required), the conditional expectation operator, we could also apply standard gradient descent to obtain the final estimator. I do not see how the proposed algorithm is significantly different from the standard ones. However, the proposed algorithm have the additional requirement to compute the conditional density function which is significantly harder than computing $\\mathcal{P}$.\n\n3. The conducted experiments is not convicing as even in the low dimensional regime, the obtained mse is not significantly better than the standard two stage estimation method. And the experiments do not contain the high dimensional setting. So it is not clear how the method works in such settings.  A nolvety the authors seem to claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful observations. Below, we separately address each weakness:\n\n1. The referee is correct in pointing out that estimating conditional densities in high dimensional settings poses nontrivial difficulties, being a limitation which applies in our case, as well as in other approaches to the NPIV problem. In Section 1.1, we start by noting that classical NPIV methods, like those based on sieves or Nadaraya-Watson kernel estimators, exhibit difficulties when dealing with large, high dimensional datasets, while more modern approaches leverage deep learning techniques in an attempt to overcome these issues, but end up susceptible to higher prediction variance under the opposing scenario of few data points. Our goal with this discussion is to point out that, since our formulation is agnostic with respect to how the density ratio is estimated, it could be tailored to specific scenarios and leverage recent advances from other areas. For example, one could use neural networks in large data/dimension scenarios, and kernel methods for situations where there is not a lot of data/the data is low dimensional.\n\n2. We would argue that estimating $\\Phi$ is not significantly harder than estimating $\\mathcal{P}$, as the latter is an operator acting on an infinite dimensional space. This is evidenced by the very few results discussing convergence rates for estimators of this conditional expectation operator. Furthermore, according to Proposition 3.3., in our loss-agnostic framework, the gradient at $h$ is given by the application of the adjoint operator $\\mathcal{P}^*$ at the composition of $\\partial_2 \\ell$ with $\\mathcal{P}[h]$. So, any gradient-based algorithm needs to somehow tackle the estimation of both $\\mathcal{P}$ and $\\mathcal{P}^*$ as operators. For instance, in [1] they consider sieve estimators using separate basis expansions for $\\mathcal{P}$ and $\\mathcal{P}^*$. Our stochastic gradient formulation is able to overcome this problem by substituting the outer application of $\\mathcal{P}^*$ with a simple multiplication by $\\Phi$.\n\n3. To our understanding, in Figure 1, our MSE is lower and statistically better than the standard two-stage method, except for the case in which TSLS is correctly specified. Note, however, that this is expected against any method, as it falls within the realm of parametric estimation. The log-MSE shows significant improvement over the other methods. In particular, the Deep learning based methods (DeepGMM, DeepIV) show larger, more variable log-MSE than the two variants of SAGD-IV. The same is true for Dual IV. The strongest competitor is the KIV method, which demonstrates similar log-MSE for the step and linear function scenarios, lower log-MSE for sine (but with larger variance) and statistically higher log-MSE in the Abs scenario. Therefore, we argue that the numerical experiment indicates that our method is at least comparable with the state-of-the-art methodologies.\n\n   As indicated in our answer to the reviewers first point, we are not claiming that our method is the first to address the high-dimensional setting, we have just pointed out the flexibility of our framework to possibly deal with such scenarios. Given the short reviewing period and the computational resources available to us, unfortunately, we could not run the experiments again in high-dimensional settings.\n\n**References**\n\n[1] Darolles, S., Fan, Y., Florens, J. P., & Renault, E. (2011). NONPARAMETRIC INSTRUMENTAL REGRESSION. Econometrica, 79(5), 1541–1565. http://www.jstor.org/stable/41237784", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for reading our paper and providing valuable feedback which will certainly result in improvements to the final version.
We have uploaded a PDF containing experiment results which address points raised by reviewer 1tRc.

Meta-review: The majority of reviewers tend to agree that the paper provides a solid contribution to the research area of Nonparametric Instrumental Variable (NPIV) Regression. It proposes to solve NPIS using a functional stochastic gradient descent algorithm, which is novel. The theoretical results are also very thorough. I believe that people working in this area will appreciate this work.
Decision: Accept (poster)