Learning Causal Models from Conditional Moment Restrictions by Importance Weighting
Authors: Masahiro Kato
Masaaki Imaizumi
Kenichiro McAlinn
Shota Yasui
Haruo Kakehi
Venue: ICLR 2022 Spotlight (2022)
Keywords: Causal inference
Conditional moment restrictions

Abstract: We consider learning causal relationships under conditional moment restrictions. Unlike causal inference under unconditional moment restrictions, conditional moment restrictions pose serious challenges for causal inference. To address this issue, we propose a method that transforms conditional moment restrictions to unconditional moment restrictions through importance weighting using a conditional density ratio estimator. Then, using this transformation, we propose a method that successfully estimate a parametric or nonparametric functions defined under the conditional moment restrictions. We analyze the estimation error and provide a bound on the structural function, providing theoretical support for our proposed method. In experiments, we confirm the soundness of our proposed method.

Published as a conference paper at ICLR 2022
LEARNING CAUSAL MODELS FROM CONDITIONAL MOMENT RESTRICTIONS BY IMPORTANCE WEIGHT-
ING
Masahiro Kato Masaaki Imaizumi", Kenichiro McAlinn', Shota Yasui', and Haruo Kakehi
AI Lab, CyberAgent, Inc. 2The University of Tokyo Temmle University
ABSTRACT
We consider learning causal relationships under conditional moment restrictions. Unlike causal inference under unconditional moment restrictions, conditional moment restrictions pose serious challenges for causal inference, especially in high- dimensional settings. To address this issue, we propose a method that transforms conditional moment restrictions to unconditional moment restrictions through importance weighting, using a conditional density ratio estimator. Using this transformation, we successfully estimate nonparametric functions defined under conditional moment restrictions. Our proposed framework is general and can be applied to a wide range of methods, including neural networks. We analyze the estimation error, providing theoretical support for our proposed method. In experiments, confirm the soundness of our proposed method.
INTRODUCTION
Consider learning the causal relationship between airline ticket prices and demand. As one might expect, prices and demand rise and fall through the seasons, being affected by other events like vacation periods, which are called confounders and may or may not be observable. Due to confounders, naively inferring from this pattern that higher (lower) prices increase (decrease) demand would be incorrect, and potentially detrimental. Thus, controlling for confounding effects is essential. This issue frequently arises in practice, especially when learning causal (structural) relationships is essential to answer counterfactual questions regarding policy intervention and outcome (Hansen 2022).
One approach to deal with confounding effects (like the airline example above) is the instrumental variable (IV) approach (Wooldridge] 2002 Greene 2003). In the IV approach, the conditional moment restriction is defined as such that the causal model satisfies the restriction zero expected value given IVs, thus conditioning out the confounding effect. The simplest representation of this idea is the two-step least squares (2SLS) for linear models (Wooldridge) [2002] Greene 2003). However, given the complex nature of the causal effect and confounding effect (and their relation), assuming linear relation can be too strong. Thus, in this paper, we focus on nonparametric IV (NPIV) regressions, allowing for much more flexible estimation (Newey & Powell| (003).
NPIV can be viewed as an instance of a more general framework of causal inference under conditional moment restrictions. In this light, the machinery for inference under conditional moment restrictions also applies to NPIV, as well as its shortcomings. One major issue with using the conditional moment restrictions for causal inference is that one must approximate the conditional expectation, which is often difficult to do (see, e.g., Newey (193)); Donald et al. (2003) for parametric and Newey & Powell (2003) : Ai Chen (2003) for nonparametric IVs using sieves). For instance, Lewbel (2007) and Otsu (2011) estimate the conditional expectation by local kernel density estimation. However, local kernel density estimation suffers under high dimensionality. For this problem, recent methods
suggest the of machine learning methods, such neural networks (Hartford al. 2017

---

Published as a conference paper at ICLR 2022
In this paper, we propose transforming conditional moment restrictions into unconditional moment restrictions by importance weighting using the conditional density ratio, which defined as the ratio of the conditional probability density, conditioned on the IVs, to the unconditional probability density, We show that the unconditional expectation of a random variable weighted by the conditional density ratio is equal to the conditional expectation. Further, conditional density ratio with the least-squares method with a neural network. Once the conditional density ratio is estimated, the usual method of moments, such GMM, can be used straightforwardly.
show that it is possible to estimate the
The contribution of this paper is as follows: (i) we propose a novel approach to convert conditional moment restrictions to unconditional moment restrictions by importance weighting; (ii) using our proposed transformation, we develop methods for NPIV; (iii) we analyze the estimation error
2 SETUP AND NOTATION
Among various problems of learning causal relationships from conditional moment restrictions, we focus on the NPIV regression for ease of discussion. Note that our proposed method can be applied to more general settings, similar to Ai & Chen (2003).
Suppose that the observations {(Y,, X;, Z:) are i...d, where Y; E y S R is an observable scalar random variable, X; X' C R is a dx dimensional explanatory variable, Z₁ E 2 C RRd is dz dimensional random variable, called an IV, and X' and Z are compact with nonempty interior. We also assume that the probability densities of (Y; X,, Z:). (Y, XX), Z; exist and denote them by p(y, x. 2), x(,, and p(=), respectively. Let define the causal relationships between Y; and X; as
Y = f'((((() +
where /" : - y is a structural function, e; is the aub-Gaussian error term with mean zero. To learn f", suppose the IV Zi satisfies the following conditional moment restrictions:
E e 2 = 0 Vi E (1, (1,
(1)
Then, we also assume that under the conditional moment restriction, we can uniquely identify f". Our goal is to learn f from the conditional moment restrictions in (T). If Z, = X2, this problem boils down to the estimation of the conditional expectation (regression function) [X]] However, when Ele, X.] # O, E[Y; [X,] is not equivalent to f (X,);; that is, typical regression analysis, such as least squares, may not return the correct estimate of f (*.))..
3 PRELIMINARIES AND LITERATURE REVIEW
In this section, we briefly review causal inference under moment restrictions.
3.1 IV METHOD FOR LINEAR STRUCTURAL FUNCTIONS
One of the basic cases of using the IV when f* is linear model X, 8" with dx dimensional vector O" and the error term E; is correlated with the explanatory variable X,. In this case, the parameter 0* of the linear model can be estimated there are IVs of dimension dx or more that satisfy the uncon- ditional moment restrictions, [[ Oázz where O₂ is d dimensional zero vector. In the just- identified case (dx = d2), we can estimate 0* as 0yy = - T nn over-identified case (dx I dz). we can estimate it by the 2SLS. In the 2SLS, regress X, by Z;; then using a dx dimensional function ğ (Z.) obtained from the first stage regression, we estimate 8" as
E YY In the
0ALLS = ( En X ER X,Y, where Xa, = 2, ( Σÿ= 2, - 2, Xa3 and X, = (X 18.,,..., x
formulate the estimation of the linear structural function by uncondi- tional moment restrictions defined using the IV; that is, Efm X,, = Od.-. where gr E θ is parameter representing the causal effect, θ is the parameter space, and m xxRxX'x + Tedm is a d,, dimensional moment function. Then, the GMM estimator is defined as ÔAMM arg mingee ( Σi_ m(0; Y;, X,, Z.)) Wa.. EH m(0 Y, X,, Zi)), where Wa,, is dm dm, weight matrix. If Wa., 4,,, is chosen W E|m(("; Y,, Xτ, 2a) m(0;; Y,, X,, 7:)],
More generally,
2

---

Published as a conference paper at ICLR 2022
the estimator faMM is efficient under the posited moment conditions. Note that the GMM includes the 2SLS as a special case where E, has the same variance among E {l,...., n and the zero covariance.
Three methods have been proposed to obtain W twwostep GMM, iterative GMM (Hayashi 2000), and continuous updating GMM (CU estimator; CUE, Hansen et al. 1996). These estimators have the same asymptotic distribution, but different non-asymptotic properties. In particular, CUE is known as a special case of the generalized empirical likelihood (GEL) estimator (Owen) 1988) |mmthh 1997).
which plays an important role in estimation with many moment restrictions (Newey &Smith 2004).
3.2 NPIV AND LEARNING FROM CONDITIONAL MOMENT RESTRICTIONS
In linear structural functions, zero covariance between the IV and error term suffices for identification. However, in nonparametric structural functions, we require a stronger restriction: the error term has conditional expectation zero given the IVs. Then, we can characterize the solution of NPIV as that of an integral equation K (2) E| ZZ, 2| (X,, [z] = f (z)dF(() (x (2) with F denoting the conditional c.d.f of x given The identification also results in the uniqueness of the solution.
Several estimators have been proposed. Newey Powell (2003) proposes a nonparametric analogue of the 2SLS for linear structural functions. They first define the linear-in- seear-in-paac in-aarameter model as a model to approximate the nonparametric structural function f", where they use a linear approximation with basis expansion, such as sieve (series) regression. Let us denote the approximation as f*(X,) R (X,) 0", where 1i X + Rdv, , dg S n is a vector-valued function consisting of outputs of basis functions. Then, they conduct the 2SLS follows: (i) they define lincar-in parameter model as an approximation to the nouparametric model of Z₁ and estimate E[b((X |2, using that model; (ii) they regress Y, by B({)(X, [2,] to estimate 0", In contrast, Ai & Chen 2003 proposes nonparametric
innearin-paramettr
analogue of the GMM. Ai & Chen (2003) also approximate the nonparametric structural function f by sinear-in-paramere model. However, unlike Newey & Powell (2003), Ai & Chen (2003)
estimates B[[Y; - f(X.)) Z,), instead of estimating E (X) 1. by another linear-in-parameter
model. Using the estimator of [E, (Xx) )) 2, Ai & Chen 2003) transforms conditional moment
restrictions into unconditional ones and apply the GMM to estimate f*. In addition to these two typical methods, Darolles et al. (20IT) Carrasco al. (2007a) propose their own methods for NPIV.
This is how the NPIV problem is typically cast into the framework of conditional moment restrictions, where a parameter representing the causal relationship satisfies E/m(0"; Y,, X,, Z,) [z, 0d... As with NPIV, we often define the moment function as a function of the nonparametrie function f instead of the parameter 0. We can estimate f* defined under conditional moment restrictions using variants of GMM or empirical likelihood, e.g.. [Ai & Chen (2003): Domínguez & Lobato 2004); Otsu (201)
Lewbel (2007), by transforming the conditional moment restrictions to unconditional ones.
3.3 APPROACHES IN THE MACHINE LEARNING LITERATURE
2SLS with more flexible models. Hartford al. (2017) extends the two-stage approach by employ- ing a neural network density estimator, and Singh et al. (2019) does it by conditional mean embedding in RKHS. For example, in Hartford et al. (017), they first approximate K K(2) by estimating F F(x|=) with neural networks. Then, for B samples { generated from the estimator (x]2),, they train
neural networks by minimizing the empirical risk 1 n El (Y: EX f(x)))
In contrast, Xu et al. a first approximates f(X.) by neural networks and regard the last layer as (X,) in the 2SLS of NPIV. Then, they predict (X.) by another model yp (4). Because (X,) is transitive in the learning process, they alternatively train the models sp Z,) and fXX)).
Minimax approach. There has also been a recent surge in interest in minimax approaches that reformulate conditional moment restrictions as a minimax optimization problem (Bennett et al. 2019: Bennett & Kallus 2020; MMandet et al. 2020) Chernozhukov et al. 2020 [iao et 2020). For this approach. Dikkala et al. (2020) shows strong theoretical guarantee the estimation error bound.
3

---

Published as a conference paper at ICLR 2022
4 PROPOSED METHOD: NPIV BY IMPORTANCE WEIGHTING
Suppose that p(y, x)) > 0 for all (y, x) E y X. Note that we have already assumed the existence of p(y, for all (y, z) y x A' x Z by assuming the existence of the conditional moment restrictions, Define the conditional density ratio function r y x xxXX x 2 (0, C) as
r' (v.x)2) x(),
ply, x) ply. x))(2)²
for 0 < <C<< c0. We assume the existence of the conditional density ratio function.
Assumption 1. For all (y,z, y x X Z. the conditional density ratio r" (y, 2|2) exists.
Then, we transform conditional moment restrictions into unconditional moment restrictions as BI(Y, f(x))l= / (v f()) ply, x|z) ply, x)dydr IE [(Y: - ((x.))r X|/=]]
x(y,
Thus, if we know the conditional density ratio function r" (2,,, we can approximate [Y, [x for by the following sample average:
1 Σ ((x))
Based on this property, we propose the following two-stage method: (i) estimate the conditional density ratio function r*: (ii) approximate conditional moment restrictions by the sample average of (Y; ((x)) using the estimate of the conditional density ratio function r". Since we do not know the value of the conditional density ratio r' and cannot calculate the expected value, we consider replacing r" with an estimator e and approximating the expected value with the sample mean.
4.1 CONDITIONAL DENSITY RATIO ESTIMATION
First, we consider estimating r" (y, z| While it is possible estimate the probability density func- tions of the numerator and the denominator, individually, following Vapnik's 's principle (Vapnik 1998). we should avoid solving more difficult intermediate problems. (Sugiyama al. (2012) summarizes various methods to estimate the density ratio directly. Inspired by least-squares importance fitting (LSIF) of Kanamori et al. (2009), we estimate the conditional density ratio by minimizing
i = arg min Ey.x ER 2
E2 (1 (r, X.1,,) r(y, x, 2,)
where R. denotes a set of measurable functions and for a function g y x X x Z + R. Ey.x [Ez (9,.,, X,, Z,] denotes J g x, =)p(y, x))(2) (dddddzz... It is easy to confirm that = r" by taking the first derivative of the risk. Because this risk includes the unknown r", it may seem intractable objective function. However, we can obtain the risk that does not include r" as
1 2
arg min r
[EYxx
[Ez (X, X112,) 2r (Y, X, 2, (r X,12,) + ? 12)])
arg min PE
Ry. EZ (V, X (Y,, X, [2,)]| 2 Ry,x [E (v x. X1
arg min rEte
- Ey.x, IY,, XZ),) Ey.x [Ez [r² X, X,12,)]1
Here, we used Ey.x [Z [rr (Y,, X,12 X, 1 Ry.x
Ez ((r.x.i)))))
(()(..))p((2)) ( r(,, x,12))]
Ey.x,z X, For some hypothesis class R. by approximating the risk with its sample approximation, we estimate the conditional density ratio as
f = arg min FER
1
R
(Y, X.
11 2n
(Y. x1.)
For the hypothesis class R, networks. Suzuki et al. (2008) also proposes a similar formulation based on maximum likelihood estimation with constraints, which is not easy to solve with neural networks.
various models, such linear-in iinearin-parrmmtter
parameter models and neural
4

---

Published as a conference paper at ICLR 2022
4.2 NPIV REGRESSION BY IMPORTANCE WEIGHTING
If we know the conditional density ratio function r", we can obtain unconditional moment restrictions Ey,x [p(f: Y,, X, 21,r" p(f: f X, 22, Y(f:Y,, Zno r))
O,.
where P(f:Y, 2,,
(Y, - f(x,))r"(Y X,(2). By replacing r' and its expectation with its estimator and the sample average, obtain the sample vector moment restrictions, EE (pf;Y, X, ZI f) pf; Y, 22, ... p(f;Y, ())
Once obtain the sample average, tional moment restriction. For instance, for hypothesis class F, we estimate f by minimizing
apply various methods for learning f* from the uncondi-
n RR,(J,)) - Σ (Y, - ((XXAAXX
This objective function is closely related with the projected mean squared error (MSE) introduced in Dikkala al. (2020), defined
((f.r))= Ez (By,x (xf (V,
(3)
The objective function is special case of the GMM and GEL. In the GMM, estimate f by min- imizing = (- - (Y; f(X,)()))))))))) (Y, 2,)) W ( Ei= XX, a weight (Ai & Chen 2003). Thus, our framework transforming conditional moment restrictions to unconditional moment restrictions using importance weighting allows to conduct causal inference using NPIV with conditional moment restrictions using a variety of models, such linear-in- parameter models with basis functions and neural networks.
where W/ >
inparrii--arramete
Linear-in- -arammeer models. As an example, we introduce a linear-in parameter model with some basis functions. Here, let us consider using the Gaussian kernel as the basis function. For x E X, let 0²r; X":".... K(x, (z; o²)) T. where (2xxX,;;))
exp || x, 20
is
the Gaussian kernel with hyperparameter σ² > O and || lla is the L2 norm. Then, we define linear-in-parameter model as ((;; = =3" p0a;; o + Bo. where 3 E RR, and 30 R.
Neural networks. We can also use neural networks for approximating f* In this case, we need to carefully determine the network structures because cannot uniquely determine the solution owing to the overparameterization. In existing studies, it is assumed that the network modes are well defined for the complexity of the nonparametric function f" and the sample size
Advantages of our proposed method. Our method has the following three advantages: (i) our method is applicable general causal inference problems. and is not limited the NPIV problem formulated in Section 2 as well as the [A & Chen (2003), (ii) our method can deal with high dimensional variables, owing to the machine learning technique. This is in contrast to related studies Ai & Chen (203) and Orsu (201T), which use sieve and Nadaraya- Sattrayy-aatoonnnnoo Watson estimator, respectively. and hence not work in high dimension settings, and (iii) our method computationally efficient by the use of the importance weight approach. A similar study Hartford al. (2017) requires additional sampling of Y, from an estimated conditional density ply 2, 3). Another related study Dikkala et al. (2020) requires difficult algorithm to solve its minimax optimization problem. Our importance weight approach avoids such computational burden.
4.3 LEARNING WITH OVERPARAMETRIZED MODELS
In learning from moment restrictions, the structural function f* is determined by the equations that the expected value of random variables satisfy. When approximating the structure function f* using model with more parameters than the sample size, the solution cannot be uniquely determined. When minimizing the prediction emrrr directly, such overparameterizztion may not pose major problem, and, in fact, may be necessary to improve the generalization error, as a recent finding suggests in the case of linear regression (Bartlett al. 2020). However, in NPIV, the objective function is
5

---

Published as a conference paper at ICLR 2022
the set of conditional moment restrictions. Unlike in direct prediction error minimization, model trained to minimize empirically approximated conditional moment restrictions does not necessarily minimize the MSE for f (X( f (X,)) In fact, empirically confirm that methods using neural network sometimes do not work well partly because of their overfitting to empirical moment minimization, not to the empirical MSE for f". Despite these potential problems, there is a strong motivation to use neural networks, owing to the reported superiority in some applications, such computer vision (Xu et 2021a Yuan et al. 202T), natural language processing (Ash et al. 2019 Chen et al. 2020) tasks. For this reason, we introduce heuristic to avoid this problem.
As we explained, the parameters are not uniquely determined by conditional moment restrictions alone. Therefore. from the set of parameters satisfying conditional moment restrictions, we need to select set of parameters that works well in prediction. We consider training a model that has the minimum MSE with Y, while satisfying conditional moment restrictions as follows:
min E ((V; f(x.))² s.t. E [(Y; (000--(xx] Z(x = 0 vj (1,2..... ... FEF
Since it is difficult to solve constrained optimization with neural networks, we propose to solve the penalized optimization. In the case of linear combination with penalties, we train the model by
1 JEF
min
(Y, f(XI))(X,
(4)
where 7) 2 0 regularization coefficient.
The motivation of this heuristic is to select function f(x) that is the closest to E[Y x). among multiple functions satisfying the conditional moment restriction. This heuristic works well when f" (z)) takes near value of E[Y; ær] while f*(x)) * EEY, X, ær.
5 ESTIMATION ERROR ANALYSIS
We show the estimation eτror of the conditional density ratio r and structural function f". We denote the sample counterpart of [(f,r") as
2
Cn(srr)) 1 Σ j=1
n - n
n2
((.) 1
Let us denote the distributions of (Y,, X₂) and 74 by P and Q respectively, and define the 7.2 risk of function with P and Q as || (P(()) = (w))d( Let us denote the distribution of (Y,,X, ., Z,) by O, and define the 2 risk of function g with as |[al]; (w))OO We put the following assumptions on the error term &
Assumption 2. The error term is sub-Gaussian random variables. In addition, the distributions of X; and Z, have probability densities that are finite and bounded away from zero.
Note that the randomness of Y; depends on E; and X,, Define the hypothesis classes of the conditional density ratio r* and structural function f as R and F, respectively. Suppose that the hypothesis classes are & Wellner (1996)). This class include the true models, and the hypothesises are bounded.
(VC) class (for rigorous definition, see Section 2.6 in van der vaart
Assumption 3. The hypothesis class R is VC class, includes the true model, r* E R, and all r R are uniformly bounded by B >
Assumption 4. The hypothesis class F is VC class, includes the true model, f E F, and all f R are uniformly bounded by B >
We define measure of complexity of the hypothesis classes: for F. define the com- plexity CB(F) Je Vlog.N(88, F, - with a covering number N6 inf(N s.t. F FC Ff - f,ll 8'}} in terms of a sup-norm || = sup, lf(x))
In this following parts, for the conditional density ratio estimation,
derive the bound of the MSE || (xx)); for the structural function estimation, we derive the bound of the projected MSE
L(f, r"), which corresponds the upper bound of the MSE of f (']lz>(o) (Section 5.3

---

Published as a conference paper at ICLR 2022
5.1 MSE OF THE CONDITIONAL DENSITY RATIO
First, we consider the MSE of the conditional density ratio. For a multilayer perception with ReLU
activation function (Definition D. show the following bound. The proof shown in Appendix B.2
Lemma (MSE of r"), Suppose that Assumptions 3 Let [(r be a non-negative funciini on
R and [(r") < co. Define RM fr R: I(r) M satisfying R UM2i RM- Suppose that
there exist co > and 7 such that SuPgERM ||r 1r--r r"1l coM and sup
|||
FERM
r"1 M for all 8 0, and that log.N (ō, Ras, || [|~~) = (M/8)) Then,
| - Op (111/())))
(5)
5.2 PROJECTED MSE OF STRUCTURAT. FUNCTION f"
Next, we consider bounding the projected MSE of the structural function f" To bound the projected MSE, we use a technique associated with U-statistics. We first obtain the following lemma. Let (Y, be W,, and W be the i.i.d. copy of W, The proof is shown in Appendix B.3
Lemma 2. Suppose that Assumptions
4 hold. For any f F and r E R, C(frr) to,))
holds as a freely chosen hyper-parameter as n. + 0o, where
E(Srr) Bw,w", ,
((X))((W1)1))
Regarding the form, we define an empirical version of r(f,r) in an U-statistic form:
- 1 - n 1)
n Σ (
E.(f.r)
By a property of U-statistics (for example, see Arcones & Giné 1993)) (.( = (f,r))
+
Op (I/n) clearly holds. Then, we can decompose the projected MSE [(f,r") as follows:
c(frr*) -(f. - [(f.F) (.. C., (f. + (1/n). (,/n)))
(6)
E:A,
We can handle A, by the estimation error of r* as shown in Lemma Appendix B.4 We evaluate the projected MSE by combining (6) with Lemma [and Lemma 6
Theorem 1. Assume that the conditions of Lemmas 7 2 hold. Then, for any any 8 E (0, 1), there exists a constant >> 0 such that the following inequality holds with n 2 1 and with probability at least 1 for any 7 E 10,
c(f,r*) S Cn(F) + Cn(R)
+ Op max / log(1/0)
Vn
n
n 1/(27)
This result reveals the following two findings: (i) the projected MSE is affected separately by the complexity of F and R. and (ii) the overall convergence is ((1/V)) when these complexities are finite. For instance, when using neural networks with Definition T the assumptions the hypothesis classes are satisfied (Lemma
5.3 FROM PROJECTED MSE TO MSE OF f"
Following Chen & Pouzo 201 2 and of the upper bound of the MSE of from the projected MSE. Let us define the measure of ill-
Dikkala et al. (2020) Appendix C.4), we discuss the derivation
posedness (Chen & Pouzo| 2012
Dikkala al. 2020) with respect the function class F as
[ I
Then, the MSE of f is upper bounded ||ʃ 112z(0) ((
SUR
Chen & Pouzo
2012
Dikkala et al.
2020 Thus, Theorem also implies
c7 2 CB(F) + CB() OO
log(1/0)
n1((2+)) })-
- f'
max
Vn
Note that Dikkala al. 2020 derived 1/n rate on the MSE of s
7

---

Published as a conference paper at ICLR 2022
0.0
0.5
-0.5
-1.0
-1.5
-2.0
-2.
-2.
100
Kmm
DeeplV DeepGNM
DTV
IW
HW MN
IW Kmmi
15
Deeply DeegGGM
Kiv
DrIV
Figure I: The log scaled MSEs of the setting in Newey Powell (2003. The left graph shows the results using the original dataset. The right graph shows the results with additional IVs.
2.5
2.0
2.0
1.5
10
1.0
-05
0.0
-10
-0.5
-1.
Deeply
DFiv
Figure 2: The log10 scaled MSEs of the original setting in Ai & Chen 2003
The left graph shows
the result with R 0.1 and the right graph shows the result with R 0.9.
Sampine soce
KV
MM
DFV
Figure 3: The log10 scaled MSEs of the setting in Ai Chen (2003) with additional IVs. The left graph shows the result with R = 0.1 and the right graph shows the result with R = 0.9.
EXPERIMENTS
We implement the following three methods based on our proposed method: first, we use neural networks to predict f" and train the model by penalized least-squares in 4 (IW-LS); second, we use neural networks to predict /* and train model by minimizing the sum of approximated moment restrictions in (2) (IW-MM), which is the same as IW-LS except for the penalized term in the IW-LS; model with the Gaussian kernel to predict f and train the model by GMM (rnl...).. For all cases, proposed methods with four methods: DeepGMM (Bennett et al. (201)). DFIV (Xu et al.] 1))), DeepIV (Hartford al. (2017)), and KIV (Singh al. (2019)) We the datasets proposed in Newey & Powell 2003). Ai & Chen (0003), and Hartford et al. (2017). In addition, we train neural networks by simple least squares (LS), ignoring the dependency between X; and E;, as comparison. To fairly evaluate the performances. for DeepGMM. DFIV, DeepIV, and KIV, we use the code and hyperparameters used in Xu et (2021a]] For IW-LS, IW-MM. IW-Kml. and LS, also follow the model and hyperparameters of the code as possible. More details are shown in Appendix
third, we use a innear-in-aaaamee
neural networks for estimating r". We compare
6.1 EXPERIMENTS WITH DATASETS OF NEWEY & POWELL 2003) AND AI & CHEN (2003
First, we investigate the performances of the proposed methods using econometric settings of Newey Powell (2003) and & Chen 2003). These settings have simpler structures than recently proposed settings, such as in Hartford et al. (2017). When using complex and oigh-dimensional datasets, there an inherent difficulty in learning due its complexity, separate from the problem setting. Therefore. we use simple datasets to check whether the proposed method can actually learn f*
Newey & Powell (003)) generates {(Y,, Z as follows: first, they generate {(E,, U,, Z
(0.5
0.5
0 ) then, they generate
0
from the multivariate normal distribution V
0.5
1
0
0
X, = Z, + U, and Y, = fx, xx +5, where (X,) = In
1
+ 1) sgn X₁ - 1).
8

---

Published as a conference paper at ICLR 2022
4.50
4.00
4.00
3.75
8.75
3.30
3.50
3:00
2.00
2.75
2:375
230
2.50
Deeple DeepGMM
Deegiv DeepONN
Figure 4: The log10 scaled MSEs of the demand design experiments with 1,000 samples. The left graph show the results with 0.25 and the right graph shows the results with 0.75.
Lid. Ai & Chen (0033 generates {(Y, X,, 2-))== as follows: first, they generate {(e;, X1i, Vi, U:)
~ Onif 1], V4 L.d. ~ Uni[[O, 1], and Uj ~ N (0, X0, + V,?); second, they generate X2; = X1x + V R x + U, and Y, = X1=%0 + ho (X2:) + where ho (X2:) exp (X2:) /1 + exp (X2:)) and R is chosen 0.9; then, obtain X; (X1s X2.) and Z, = (X V.). Here, 6; and U, are unobservable, and l (X)) = X1:70 + ho (X2c), where the function by and To are unknown.
as ~ N (0, + V. X18 i.d.
We run each algorithm 20 times on each dataset with = 1, 000 and calculate the mean squared error (MSE). In the left graph of Figure and Figure 2 we report the MSES. In the dataset of Newey & Powelll (203). DFIV and our proposed IW-MM produce smaller MSE. The dataset only has dimennnnnionnll X,, which mitigates the identification problem of neural networks. In contrast, the dataset Ai & Chen (2003). IW-LS leads to smaller MSE. In Appendix C.4 using the experimental setting of Newey & Powell (2003), we also show additional experimental results on the empirical convergence of the MSE of the IWMM and a comparison between the IWMM and classical 2SL.S.
Because the dimensions of the IVs are low in the original settings, we add more IVs to the original settings and investigate the performances of the algorithms. The detailed settings are described in Appendix C.T. The results are shown in the right graph Figures and Figure
6.2 SIMULATION STUDIES USING DEMAND DESIGN DATASETS
To investigate the performance with more complicated and high-dimensional datasets, demand design dataset for synthetic airline ticket sales proposed by Hartford et al.] (2017). In this dataset, we observe Pt,
the
P., T,, S. C.). where Y; is sales, P. price, T: is time, S, consumer's emotion, and C is cost to use as IV. Here, Ti and S; are covariates, that is. X; = (T), S:), and the IV is Z, (C,, T., S.). The sales Y; is generated as Y, 100 + (10 + P.)S,((T;))
2P; + E,, 2) Since P. is an endogenous variable and correlated with the IV, generate P. to contain C. as P. 25 + (C, + 3)4(T,) + V. In the simulation, we assume E; N (pV,, - ,²), V; ~ (N(0, 1), C. ~ NN(0, 1). T; is sampled from the uniform distribution with the continuous support [o, 10), and S; is sampled from the uniform distribution with the discrete support (1...., 7}. Here note that Ele, CC, T,,,T,,5. S.SSS. = E[,, = 0. The extent of correlation between P: and 1/8 is controlled by E {0.25, 0.75) and the larger the p is, the more severe the correlation problem becomes.
where (()) 2 (t-5)" + exp (-4(t- 5)2) +
600
Figure shows the results with 1, 000 samples. The results under other settings are reporetd in Appendix
this setting, IW-LS and LS outperform the other methods. We consider this is because f takes large values compared to the error term. Under this situation, model trained to predict Y: may perform well because the influence of EEe, XX.] o is limited. However, the purpose of using NPIV in the first place is because the influence is large. or else effects of EE|s]| + 0 can be ignored. Therefore, this dataset often used in the existing studies, it may not be suitable.
7 CONCLUSION
This paper proposed a method for learning causal relationships from conditional moment restrictions. Our method is based importance weighting using the conditional density ratio. The proposed method showed superior performance in experiments. We point out potential problems in recently proposed methods concerning identification and empirical performances.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors would like to thank Liyuan Xu for his constructive advice.
REFERENCES
Chunrong Ai and Xiaohong Chen. Efficient estimation of models with conditional moment restrictions containing unknown functions. Econometrica, 71(6):1995-1444 2003.
Miguel A Arcones. A bernstein-type inequality for u-statistics and u-processes. Statistics &
probability letters, 22(3):239-247, 1995.
Miguel A Arcones and Evarist Giné. Limit theorems for u-processes. The Annals of Probability, pp- 1494-1542, 1993.
Elliott Ash, Daniel Chen. Xinyue Zhang, Zhe Huang, and Ruofan Wang. Deep iv in laa: Analysis of appellate impacts sentencing using high-dimensional instrumental variable, 2019.
Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48) 30070, 2020.
Andrew Bennett and Nathan Kallus. The variational method of moments, 2020.
Andrew Bennett, Nathan Kallus, and Tobias Schnabel. Deep generalized method of moments for instrumental variable analysis. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Herman J. Bierens. Consistent model specification tests. Journal of Econometrics, 20(1):105-134, 1982.
Herman J. Bierens. A consistent conditional moment test of functional form. Econometrica, 58(6): 1443-1455 1458, 1990.
Richard Blundell, Xiaohong Chen, and Dennis Kristensen. Semi-nonparamettrcc iv estimation of shape-invariant engel curves. Econometrica, 75(6):1613-1669 2007.
Marine Carrasco, Dean-Pierre Florens, and Eric Renault. Linear inverse problems in structural econometrics estimation based on spectral decomposition and regularization. In Handbook of Econometrics, volume 6B, chapter 77. Elsevier, edition, 2007a.
Marine Carrasco, Jean-Pierre Florens, and Eric Renault. Linear inverse problems in structural econometrics estimation based on spectral decomposition and regularization. In J.J. Heckman and E.E. Leamer (eds.), Handbook of Econometrics, volume 6B, chapter 77. Elsevier, edition, 2007b.
Gary Chamberlain. Efficiency bounds for semiparametric regression. Econometrica, 60(3):567-596. 1992.
Jiafeng Chen, Daniel L.. Chen, and Greg Lewis. Mostly harmless machine learning: Learning optimal instruments in linear iv models, 2020.
Xiaohong Chen and Sydney C. Ludvigso22W9Ludvigson. Land of addicts? empirical investigation of habit-based asset pricing models. Journal of Applied Econometrics, 24(7):105--1093, 2009.
Xiaohong Chen and Demian Pouzo. Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals. Econometrica. 80(1):277-325 2012.
Victor Chemozhukov and Christian Hansen, An iv model of quantile treatment effects. Econometrica, 73(1):245-261, 2005.
Victor Chernozhukov, Guido W. Imbens, and Whitney K. Newey, Instrumental variable estimation of nonseparable models. Journal of Econometrics, 139(1):4-14, 2007.
Victor Chernozhukov, Whitney Newey, Rahul Singh, and Vasilis Syrgkanis. Adversarial estimation of riesz representers, 2020.
10

---

Published as a conference paper at ICLR 2022
Serge Darolles, Yanqin Fan, Jean-Pierre Florens, and Eric Renault. Nonparametric instrumental regression. Econometrica, 7965::541-1165,, 2011.
Nishanth Dikkala, Greg Lewis, Lester Mackey, and Vasilis Syrgkanis. Minimax estimation of conditional moment models. In Advances Neural Information Processing Systems, volume 33, pp. 12248-12262. Curran Associates, Inc., 2020.
Manuel A. Domínguez and Ignacio N. Lobato. Consistent estimation of models defined by conditional moment restrictions. Econometrica, 1205):101116155 2004.
Stephen G. Donald, Guido W. Imbens, and Whitney K. Newey. Empirical likelihood estimation and consistent tests with conditional moment restrictions. Journal of Econometrics, 117(1):55-93, 2003.
Patrick Gagliardini and Olivier Scaillet. Nonparametrie instrumental variable estimation of structural quantile effects. Econometrica, 80(4):1533-1562, 2012.
Evarist Giné and Richard Nickl. fouiia of infiiite-dimensiional statistical models. Cambridge University Press, 2021.
William H. Greene. Econometric Analysis. Pearson Education, 2003.
Peter Hall and Joel L. Horowitz. Nonparametric methods for inference in the presence of instrumental variables. The Annals of Statistics, 33(6))2904 - 2929, 2005.
Bruce E. Hansen. Econometrics. 2022.
Lars Peter Hansen, John Heaton, and Amir Yaron. Finite-sample properties of some altemaatiee gmm estimators. Journal of Business & Economic Statistics, 14(3):262-280, 1996.
Jason Hartford, Greg Lewis, Kevin Deyton-Brown. and Matt Taddy. Deep IV: A flexible approach for counterfactual prediction. Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1414-1423. PMLR, 06-11 Aug 2017.
Fumio Hayashi. Econometrics. Princeton Univ. Press, Princeton, NJ [u.a.]. 2000.
Wassily Hoeffding.
Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13-30, 1963.
Joel L. Horowitz and Sokbae Lee. Nonparametric instrumental variables estimation of a quantile regression model. Econometrica, 75(4):1191-1208 2007a.
Joel L. Horowitz and Sokbae Lee. Nonparametric instrumental variables estimation a quantile regression model. Econometrica, 75(4):1191-1208 2007b.
Takafumi. Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach direct
importance estimation. Journal of Machine Learning Research, 10(Jul.):1311-M 2009.
Masahiro Kato and Takeshi Teshima. Non-negative bregman divergence minimization for deep direct density ratio estimation. Proceedings of the 38th International Conference on Machine Learning, 2021.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database, 2010.
Arthur Lewbel. A local generalized method of moments estimator. Economics Letters, 94(1):124-128, 2007.
Hong Li, Chuanbao Ren, and Luoqing Li. U U-processes and preference learning, Neural computation, 26( 12):22996-22244 2014.
Luofeng Liao, Yiu Chen, Zhuoran Yang, Bo Dai, Mladen Kolar, and Zhaoran Wang. Provably efficient neural estimation of structural equation models: An adversarial approach. In Advances in Neural Information Processing Systems, volume 33, pp. 8947-8958. Curran Associates, Inc., 2020.

---

Published as a conference paper at ICLR 2022
Krikamol Muandet, Arash Mehrjou, Si Kai Lee, and Anant Raj. Dual instrumental variable regression. In Advances in Neural Information Processing Systems, volume 33, pp. 2710-2721. Curran Associates, Inc., 2020.
Whitney K. Newey. Efficient estimation of models with conditional moment restrictions. In Econo- metrics, volume 11 of Handbook of Statistics, pp. 419-454. Elsevier, 1993.
Whitny K. Newey and James L. Powell. Instrumental variable estimation of nonparametric models. Econometrica, 7165):1555-1578 2003.
Whitney K. Newey and Richard J. Smith. Higher order properties of gmm and generalized empirical likelihood estimators. Econometrica, 72(1):219-255 255, 2004.
Taisuke Otsu. Empirical likelihood estimation of conditional moment restriction models with unknown functions. Econometric Theory, 27(1):8-46, 2011.
Art B. Owen. Empirical likelihood ratio confidence intervals for single functional. Biometrika, 75 (2):237-249, 06 1988.
Olav Reiersöl. Confluence analysis by means of instrumental sets of variables, 1945.
Andres Santos. Inference in nonparametric instrumental variables with partial identification. Econo- metrica, 80(1):213-2755 2012.
Johannes Schmidt-Hicber. Nonparametric regression using deep neural networks with ReLU activa- tion function. Annals of Statistics, 48(4):1875-1897, 2020.
Rahul Singh, Maneesh Sahani, and Arthur Gretton. Kernel instrumental variable regression. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Richard Smith. Alternative Semi- parimerra Likelihood Approaches to Generalised Method of Moments Estimation. Economic Journal, 107(441):503 519, March 1997.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, New York, NY, USA, 1st edition, 2012.
Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Takafumi Kanamori. Approximating mutual infor- mation by maximum likelihood density ratio estimation. In Proceedings of the Workshop on New Challenges for Feature Selection in Data Mining and Knowledge Discovery at ECMUPKKD 2008, volume 4, pp. 5-20, 2008
Sara van de Geer. Empirical Processes in M-Estimation, volume 6. Cambridge university press, 2000.
Aad van der vaart and Jon Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics (Springer Series in Statistics). Springer, 1996.
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
Jeffrey M. Wooldridge. Econometric analysis of cross section and panel data. MIT Press, 2002 2002.
Philip Green Wright. The Tariff on Animal and Vegetable Oils. Investigations in international commercial policies. Macmillan, 1928.
Liyuan Xu, Yutian Chen, Siddarth Srinivasan, Nando de Freitas, Arnaud Doucet, and Arthur Gretton. Learning deep features in instrumental variable regression. In International Conference on Learning Representations, 2021a.
Liyuan Xu, Heishiro Kanagawa, and Arthur Gretton. Deep proxy causal learning and its application to confounded bandit policy evaluation. In Fiiity Conference on Neural Information Processing Systems, 2021b.
Junkun Yuan, Xu Ma, Kun Kuang, Ruoxuan Xiong, Mingming Gong, and Lanfen Lin. Learning domain-invariant relationship with instrumental variable for domain generalization, 2021.
12

---

Published as a conference paper at ICLR 2022
A LITERATURE REVIEW OF IV METHODS
In the field of economics, the causal effect often referred to as the structural effect, because the causal relationship arises from some economic structure. The idea of endogeneity has a close relationship with the estimation of the structural effect. For example, in estimation problems with supply/demand models, because the supply and demand are determined simultaneously. there is simultaneous equation bias, which causes the correlation between the explanatory variable and error term. This correlation called enodogeneity. Under the enodogeneity, the OLS does not yield consistent estimator of the structural model. To obtain a consistent estimator to capture the causal effect, the method of IVs have been used Wright (1928): Reiersöl (1945).
A.1 IV METHODS FOR LINEAR STRUCTURAL MODELS
First, we consider linear structural model. For random variables Y E R. X Re endogeneity refers to a situation, such that
Y x E
where E IR is the enror term, B is the parameter of interest, and
[[X=] * O.
(8)
This regression model called a structural equation and 3 is structural parameter. Under endogeneity, the least squares method does not yield consistent estimator.
Let us consider an Z & such that
E Z 0.
For the estimation of the structural parameter under endogeneity, the IV Z plays an important.
A.2 IV METHODS FOR NONPARAMETRIC STRUCTURAL MODELS
Nonparametric estimation structural models under conditional moment restrictions an important topic in statisties and econometrics, because it allows to model economic relationships flexibly. be unknown nonparametric structural functions, where each function (-) may depend on X, and Y, In general, we can consider estimating the unknown nonparametric structural functions h"( defined
Let (-)
E [pYY Co h 2-))) =
(10)
where p(:; 0o (-))) is a vector of residuals with functional forms. Note that the conditional distribution of Y given X is not specified.
This model a generalization of semi/nonparametricc estimation with conditional moment restrictions considered in [Chmbeellin (1992) and Newey & Powell (2003). This model includes many structural models as special cases. A typical example is the target structural model of the NPIV problem defined in Section 2 (Newey & Powell 2003 Darolles et al. 2011). The NPIV problem also includes the estimation of shape-invariant system of Engel curves with an endogenous total expenditure (Blundell al. 2007). Another important special case of (T0) is the quantile instrumental variables (IV) treatment effect model of Chernozhukov & Hansen (2005), and the nomparametric quantile instrumental variables regression Chernozhukov al. (2007) and Horowitz & Lee (2007a). There
are other applications, such as asset pricing models (Chen & Ludvigson2009Luuvvisso 2009) and
reinforcement learning.
There are three approaches to the estimation of the structural models under the conditional moment restrictions: the sieve minimum distance (SMD) method, the function space Tikhonov regularized minimum distance (TMD) method, and the minimax optimization method.
SMD method. The SMD procedure minimizes a consistent estimate the minimum distance criterion
dineeniimensooall compact sieve space. Newey & Powell (2003) extends the 2SLS methods for linear structural models to the NPIV problem. The authors approximate the nonparametric structural function with sieve estimator, which is linear combination of growing
13

---

Published as a conference paper at ICLR 2022
feature basis functions. Ai & Chen (2033) and Chen & Pouzo 2012 distance sieve estimator to a more general problem defined as (10)
propose the penalized minimum
When applying the minimum distance method to the NPIV problem, we consider solving the following problem:
min Ez [ Y hx AR(M), REH
where R(hh is a regularizer. CCenn & Pouzo (2012) approximates the function class H, by linear functions in growing feature space with the sample size n. Subsequently, the authors also estimate the function m(z) = Ely h(z) z] based on another growing sieve.
TMD method. In the TMD method, we minimize consistent penalized estimate of the minimum distance criterion over the whole infinite dimensional function space H. in which the penalty function of the classical Tikhonov type (Hall & Horowitz, 2005 Horowitz Lee 2007b, Carrasco et al.
2007b Darolles et al., 2011 Gagliardini & Scaillet 2012). In the NPIV problem, this approach is equivalent to solving
min Ez By 1 ((X) +
(11)
KER
Note that this is equivalent to the minimum distance estimation, such Chen & Pouzo (2012).
Minimax optimization method. Recently, in the machine learning literature, the minimax ap- proach has garnered attention (Bennett et al., 2219)) Dikkala et al.) 2020). In this approach, we solve the following minimax optimization problem:
min max B[(Y ((X))A(2)) 11M12 fEJ KEH
(12)
Such representation of the conditional moment restrictions has long been used in statistics and cconometrics. One of the related studies the specification testing by Bierens (1982] 1990). The authors assume that a statistical model that satisfies the conditional moment restrictions is correct and propose specification testing method investigate whether the model correct. These studies transform the conditional moment restrictions to unconditional moment restrictions by using the product of any function a random variable used in the conditioning and the moment function. This idea of transformation to unconditional moment restrictions is also used in other related work, such as Santos (2012). In particular, Dikkala al. 2020) shows the minimax optimality of a method based such minimax optimization. This minimax optimization technique is known to be applicable, not only to the NPIV problem, but also a wider range of problems. It is also closely related to debiased machine learning literature (Chernozhukov et al. 2020).
B PROOFS IN SECTION 5
B.1 MATHEMATICAL TOOLS
Concentration inequalities for empirical processes Given probability distribution /' and random variable g(X), we denote the expectation of g(X) under P by gdl, Given samples X2,........ Xx, from P. the empirical distribution is denoted by P., and the empirical mean is
denoted by gdPm: that is, f gdP,, = Σ 9(X,). We also denote f gdP E 9(X,) by gd(P- -P)).
We also define
(x( = 2K² -2k] (exx//(xxxx/)) (exp(lal/K (11p(11//-11111/)))
K K>0 0.
(13)
Let 9 satisfy
sup Pk -
(14)
gE9
Then, summarize tools used in the theoretical analysis.
14

---

Published as a conference paper at ICLR 2022
Proposition 1 (Theorem 5.11 in van de Geer (2000). C is (sufficiently large) universal constant,
whereas a, Co, and C1 may be chosen, but do have to satisfy the following conditions:
s CIvnR/
a s TRRR;
Co (max { vareev G.. I1dd,, 11)du, R1))
. c 2 (² 1).
Then,
Vt gd gd (nn - -P) 2a) < Cexp (-a(ch
a²
P sup gEg
(15)
1)R2
Proposition 2 (Lemma 5.14 in van de Geer 2000). Let & c (P) be a function class and the map [g) be complexity measure of E G, where I is non-negative function functioon on R and I (go) < co for a fixed gu G. We now define GM {g E 9 [(g) M} satisfying
UM21 92s- Suppose that
there exist co > and 7 such that
sup lgg goll S c,M, 9E9M
sup
- gullo I CoMM for all ő > 0,
FEPM 119 119-00l1 (0
and that N6,,N6,,80 (6,9MM P) = (M/8)7 Then, we have
[ go)d( Pa) = O0 (1), (n o0),
sup
D(g)
sE9
where D(g) is defined by
l1g 90l 10ll|1111111 1-y/2 (2(P)
1(g 22/(2+7)
[(g)
D(g) = max
VTi
Lemma 3 (Lemma 9 of Kato & Teshima 2021 Let l (br, B,) R. be v-Lipschitz continuous
function. Let log N (6.F.I) FI (p)) denote the bracketing entropy of F with respect to distribution P. Then, for any distribution P. any y > any M 1, and any 5 O. we have
log N (8 HM. (l(a())) < (s 12 M
7
Moreover. there exists co 0 such that for any M 2 1 and any distribution P. |eor f r- r=((( S CouM,
sup torEtoHas
- - lor"ll S CouM, for all 8 > O.
toreroN
[for- (2(P)S
Concentration inequalities for U-statistics. Given probability distributions P and Q and random variable g Z). we denote the expectation of gXX, Z) under P and Q by I gdPdQ; that
is, Ex Ez (X ] = f gdPdQ. Given samples X1, X2...., X,, from P and Z1, Zz..... Zn from Q. the empirical distributions denoted by P,, and Q.. and the empirical mean is denoted by s gdP,, dQ..; that is. ʃ gdP,,d,, = + L2 i- (X; Zz). We also denote f gdPdQ- SE 1 E o(X,, Zz) by I gd(P- Px)a(Q Qn).
We also define
(g) = 2K2 / (oxp - loll/K)) AP,
-llll//
K K>0 0.
(16)
Let satisfy
sup sEg
(g S R
(17)
Then, summarize tools used in the theoretical analysis.
15

---

Published as a conference paper at ICLR 2022
Proposition 3 (Concentration inequality empirical processes with U-statistics: Theorem 5 in Arcones ( adjusted to our setting). Suppose that Si.... Sn are S-valued i.i.d. random variables and consider class of symmetric functions H. с LL x S). Also, suppose that any function in H. is uniformly bounded by > O) and define a² = SuPNEH Vars( [h(S, S')]) where S' is an i..d copy of S. If the (e, H.ll 11 (12) S (A/e)" for some A, v 0. for any 2 c with some c. O, we obtain
P
n 1/2
sup
Σ
Sv) - Es,s [K( 2t
NEH
n(n 1)
1, 1,17"
(a² 8exp( 1txp(-t2/100 12222(20 + thn tbb-11/2) + 8A (0 2thm 1/2) exp(-n(o + tbn 1/bm-1/2/2/ 1/23/2//2//
-2exp(-1²
+ lbm 1))
Proposition 4 (Bernsteins' inequality for U-statistics (Hoefiding 1963;
Arcones & Giné
1993)
Let Ig < c. gdPdQ O, and 0² I [ gdddQ. Then, for any
0:
gdPp,dn sdd,ddn>>d P (
>d) eexp
na n/2
20 + (2/3)ea/
This implies that for any 0,
( [ -
(o)(P- Q.)
a²/2
P
S exp
66
2ca V
As well as Suzuki et al. (2008), by applying this result, we can obtain the following result.
Proposition 5 (From Proof of Theorem 1 in Suzuki et al. (2008)). Let g C TP Q) be function class and the map I(g) be complexity measure of g E G. where is a non-negative function on
R and 1 (go) < oo for fixed go G. We now define GM {g- G 1(g) s M) satisfying G = UNzi1 GM- Suppose that there exist co > 0 and 0 < 7 < 2 such that
sup lgg goll S Mo
sup PERM 119-90 ollo
1g - 20 S c, for all > 0,
and that log (8, Gm, P Q) O (M/6)" Then, we have
[ go)d(P - P,)d((Q Qw) =Op (1), (n + oo),
sup
9Eg
((g)
where ((g) is defined by
[|g - 90
1-7/2
(P(Px (
[(g)
D(g) = max
nn
2/((2+))
/ [ (00 (o0o- PP)--Pp)------0 Qn) = Op (1/Vz), (n + oc).
sup g
(18)
Complexities neural networks.
Definition (ReLU neural networks; Schmidt-Hieber 2020). For L E N and p (po. .P+11)
N4-2
F(L.))==ff: I Wio Wr.-1 ... WIo, Woz:
W, E RP 187 uy E BP 0 0...,, L)},
where (v) a and a(.)= max 0} applied in an element-wise manner. Then, for I. N, and p E NL+? define
E N. 0,
= E F(L.P)
[ llo + 1lyyllo & &, F).
==0
16

---

Published as a conference paper at ICLR 2022
where ll Ill denotes the number of non-zero entries of the matrix the vector, and cloo denotes
the supremum norm. Now, fixing L. p, 8 E N as well as F 0, we define
Ind = {(L.p) L L N, L - L, PE 10]
and we consider the hypothesis class
H =
U
P(L.P.s.F)
(L.p)
H = fr H Im(r) C (br, B.)}.
Moreover, we define I Ind - and I :: [0, 00) by
(1,,,) = 2, * + 1IV?
I(r) max
||r|loc
min
ILL,p)
(aa)e PEH(T.D.a.))
where V = II I++1 (pi + 1), and define
HM E H: I(r) S M).
Lemma (Lemma 5 in Schmidt-Hieber (0200). For L E N and E N442 let V = -I b+1 (pr + +1). Then, for any d > O,
(8, H(L, oc), (1x0) (s+ log(26 (L
1)1
Definition (Derived function class and bracketing entropy). Given a real-valued va function class F, define ( F (Eo f f F). By extension, we define 1 & o - (o) by 1(l r) I(r) and & o Hm == (for:r: E Has} Note that, as a result, o HM coincides with (for E o H T(lor) M).
Notations. Let denote pair of random variables (Y, X;) by W, We denote the distribution of (W,, Z,) by R and its empirical distribution as O,, Besides, we denote the distributions of W; and Z, by P and Q, and their empirical distributions by P,, and Q,, For function g(X, Z), define the L risk over the distribution P and Q as
Ex [E2 VJJ gdpaQ.
| (0llz
B.2 PROOF LEMMA II ESTIMATION ERROR CONDITIONAL DENSITY RATIO ESTIMATION
Lemma (Decomposition of MSE).
|| - r (112200) - [w,z (W F(W[2))
Σ
((WWIZ))F(W(2z) F(W3122))
1 i=1
1 + 2
Ez Ew (w - (w(12)]]]
Σー
Σ
n
(f'2(W (OVW))))>((W/22)) P(WZZ)))
Proof of Lemma inequality
Since The estimator & is the minimizer of the empirical risk, it satisfies the
Z.W
+
11
Σ 8 (WI1Z)) <
r
Z. + 2n F= 111
11 (W,
Σ
1=1 2n
1=1
1=1
(19)
17

---

Published as a conference paper at ICLR 2022
Additionally, consider expectation of squared residuals
Ez Ew (((WW)))r-(w))) + (
2
Bz [Ew [(2(WZZ)
2 EZ [Ew [[](W])]] Ew [[(]Z)]] + マ Ew,z 'r" (WIZ))
+E [r" (W [2]] Ew,z |r' ( (W12))
Ez [Ew [F2(W] - Ew,z [P(W1Z)] 2
2 Bw (r' (WIZ)) +Ew,z (r (WI2]
Bw (WW
1 Ez Ew [r*² (W]2) - [W + Ew,z (r" (W - F(W]2)]
(20)
2
Taking sum of the both hand sides of 19) and
20
then we obtain the following by subtracting from the both side
> F(W.)
2z)
1 Σド E (W;
Ew Ew,z (F(WIZ) r (W1Z))
$ - Rz FE (w12) - [W]2)]]] + Ew [r" (WIZ) - F(W]Z))
2
+ 11
2 ト Σ
(22(W)))))) 1000-~-(w0 P(wal))))
Σ (=(W:)Z)-F(WW:: (C[W)/z)-F(WW/)) F(WslZ.))
Then, we obtain the statement.
Our remaining task is to bound the following target values:
Ez Ew [<2(W12)]] - 1 Σ (WWl2))
sup
and
(21)
rER
sup
Ew.z [r(W[z)] - n
rW, 2z)
(22)
rER
Proof of Lemma 7 Lemma to obtain
Since < we can apply Propositions _ and 5
combination with
E Z LE [P(WIZ)]] - -3 - r" O, (1),
sup rew
Di(r)
Ew.z [r(W[Z)] 1 " - r(W1)2)
sup rE
= O, (1).
D2(r)
where
1 22 1-4/2
L2(PXQ) I(r22112
T(r2)
Di(r) max
and
Vni
n2/(247)
||---* -1-r"1 16))) I(r)
D2(r) max
Vn
2/(4))
18

---

Published as a conference paper at ICLR 2022
Noting that suprem [(r) oo and suprew I(r2) < cc. Then, for any 0 < 2, we have
1 (WW)2))
Ez [Ew [W]]]]]]]
L2(PXQ)
||a - r2 1-7/2
(2(PxQ)
20p max
and
n2/(2+7)
Bw,z [[(W]Z)]
1=1
L2(PXQ)
|| -
1-7/2 (2(0)
50p
max
n2/(2++)
These inequalities imply
|/ r" (LA((XQ) (n
B.3 PROOF OF LEMMA
We start with the expected loss function for a learner f : + R and a conditional density function r: X' y 2
r(S,r) = Ez (Ew -
(23)
where W = X). For T E T, we consider T i..dd random variables W; = (X,, Yi), i = 1....,
n
generated from conditional distribution P and approximate the inner expectation term
Ew (x("(x) (()))(12()))))))))=))))))))))))) =
n1 " - ((x))) - K. (J,r),
where
Kin Kn([rr)== Ew (X**(X) ((X)))WW22) -n "
- -(X))))-(X)))((A1)))
is a residual. We substitute this form into 3 and obtain
27 n Σ - f(x)) (
C(f.r) = EE
Ez
Σ ((X)))(((()))))11111 ((x)))))1))
k,k==1
n (fx))))
2Ez
Kn (f)
Ez inn (f,r)
=1Ta
a
Ez
(F(()- -(X)) (((x))
=/T-
Ez
n
-(x))
+T + Tz.
i=1
=:To
19

---

Published as a conference paper at ICLR 2022
For To. the uniformly bounded property of f*, and implies that is To (1/n). For Ti and Tz- the VC-class property and the Glivenko-Canm Cantelli theorem (Section 2.8.1 in van der vaart & Wellner (1996)), we obtain K., (f. the boundedness of f" f and r, we obtain that Ti = (1( and T2 = (1). About T", which is an U-statistic, a convergence theorem (Theorem 3.1 in Arcones & Giné (T993)
0 in probability as T - uniformly on JF x R. Associate with
oc implies that ((X')r(W'1)) where
the U-statistic converges to Ww,w"iz (x) f(X ())(()))
W" = .X" is an i..d. copied random element of W. Hence, we obtain
C(f.r) = Ez [ =Ew,w',2 (X) (10T000-0(()00022101000000001000 o,(),,
Z)]] (1())
for any fE F r R as T oc.
B.4 ESTIMATION ERROR BOUND OF A,
We can bound A, as follows:
Lemma 6. Suppose that Assumptions 7 holds. Then,
A, = O r" (())))))))
(24)
Proof. Let us define Ej f= f We bound A by the Lipschitz continuity of C(f, r) in r. For any r,r' E R, we obtain
-(f,r)- -((,r')(r''r)))) E ((W
E ((L1))9())Y)/////WW/333-33-3
+ E (1111110(((01X/12211112212200/00
S 32B
1
1'
by the Cauchy-Schwartz Schwartz inequality and the uniformly bounded property over F and R. As result, bounded, we obtain
the conditional density ratio p(w,z)
p(")p(()
A, = O - r 112 = O(IF ((]l=-r"l r"
B.5 PROOF OF THEOREM
I ESTIMATION ERROR BOUND
We use the following lemma to show Theorem T Lemma 7.
Proof. We start with the following basis inequality following the definition of f: since it is an minimizer of the empirical risk with f, we obtain
2
(Y ((X))((A1))))
Σ 1=1
(Y, - ((.))
<
n
n1
j=1
i=1
Since Y, = f (X₂) + e, the above inequality updated as
( (25) (2)) 2
n
n
n
(a. - E ((((.))((()zz)
<
(25)
j=1
20

---

Published as a conference paper at ICLR 2022
where we define Ej = f Since the left-hand side is expanded as
(x((:))((W11zz))
(,(x))((:313))
F=
j=1
2
Σ
-2 -
Σ
112
n jJ=
1=1
1,1=11,41
r(ff)
=0,1//n)
Substituting this expansion into
25
obtain
(,,)) <
sup JEF,rER
(0,
Therefore,
sup feFrrER
P(f.r) + (L/n).
Then show Theorem
Step (i): Bound of E. (). A goal of this step is to show that the definition of f implies that for some O, with high probability,
From Lemma] 7
Op fEFRER
sup
rest of this step is to bound sup fer R (f.rr. We bound the tail-probability by the Talagrand's inequality (Theorem 3.3.9 in Giné & Nickl ) for U-statisties (Theorem in Areones (1995), Theorem in Li et al. (2014)) as the following inequality; for any > 0, with probability at least 1 - 5, we obtain
T(,r)) +0 (v 1og(1/0)
sup
T(Лr) S ED
sup
JEFRER
fERRERR
provided the uniformly bounded and finite variance properties over F x R.
To the end, we study an expectation of I(f,F) f) in terms of the dataset D = {YY, Zi) and obtain
ED
r(srr) EEFRrER
sup
s ED T("", +*]
sup f.R
[T - -+)1
50+n-12
Vlg N (6, x R, 1)..
Here, for the joint set F x R, we define distance [(f |[f]| + +||ll The last inequality follows T(f*.r") = 0 and the sub-Gur Sabb-aussiantty inequality (Corollary 2.2.8 in van der vaart Wellner (996))
associated with the sub-Gaussianity of e; and the Lipschitz continuity of T(T(r)) in (f,r).
21

---

Published as a conference paper at ICLR 2022
By these results, obtain
(f.
B VlogN(o,FF x R. 1116 +0 10)
(
(Vhest100)
/log(1/0]
S 1/2
n
1/2 /
B
log(1/0)
- 1/2
Voo.A((',, F1 11) + n211
v
R. 11) +0
71
which follows Dog F x R, II) S 1og, 11,,F111111 log (6. R, -1-
Step (ii): Bound of , To bound A, ū(f,r') rr) (f, r"), we apply the concentration
inequality on empirical processes (displayed Proposition 3 which is originally developed in
Theorem 5 in Arcones (195)). We regard h in Proposition 3 as (v. N. V. x²) + (v (x)))((
(x)))) ( A Jr ( Z;)r(w' Z'1),) and obtain
A Ew.w22 22 [(f ((X))((11))) ((X)) ((0))))(100)))2))222
n(n 1) 1FELi
log(1/5)
n
with probability at least for any 5 E (0, 1).
Step (iil): Conclusion.
Finally, by combining the above results, with probability at least 1 5 with
5 > 0,
c(f,r") = -(f,,r))(.( [f( - -I.()) -2,(
=:A,
=.A
B
(v 1og(1/0)
$ A A 11/2
1/2
Nlo.N(88FF. 2,F,RR111, Mdo +0
5011I -r r11l)
+n 1/2
[8, 1111
(og(115) (V
+ + 1/2
VlogN(6 R, 1| 1) + +0
Vlog.( F, 11. 1)d6 +n 1/2 fo B
n 11/ 1/2
VlogN R. 11)66
log log(1/5)
(nt/et (from Lemma
+ 0
max
C EXPERIMENTAL SETTINGS AND ADDITIONAL RESULTS
C. SIMULATION STUDIES USING ECONOMICS DATASETS
Here, report the details of the model and hyperparameters used in our experiments. Since the network structures used in Ai & Chen (2003) and Newey & Powell (2003) have many common features, we describe the network structure in Newey & (200).
For DeePGMM, DFIV, DeepIV, and KIV, we use the same model and hyperparameters published the following two networks for estimating density ratio by Xu et al. (2211a). For IW-LS, (1.1128)FF(122 FC1128,128)FFCCC (C()FFC(128.)) and and predicting f respectively FC(128,1). Each fully-connected layer (FC) is followed by leaky ReLU activations with leakiness
22

---

Published as a conference paper at ICLR 2022
OV- 22 -----mrlle a02 Woggo
5.0
Ou----S-mae Serp-ieeeeee soc Deggt
3.5
DeepCHM
DeepGHM
Figure 5: Demand design experiments with 5, 000 samples. The left graph show the results with p 0.25 and the right graph shows the results with p 0.75.
8.75
4.0
4.25
8.00
4.0
2.50
350
3.00
IW
Km
Decply DeepGHN
KI
DFVV
IW
HN
Kmm
Deeply DeepGHH
ki
DFIV
Figure Demand design experiments with stronger correlation between X, and &,. The sample sizes are 1, 000. The left graph show the results with p 0.25 and the right graph shows the results with p = 0.75.
0.2. A regularization coefficient is set 0.001 result of cross-validation. For LS, use the same network structure in IW-LS for predicting f*. For IW-MM, we use the same network structure as IW-LS. For IW-Krnl, the same network structure as IW-LS to estimate the density ratio and and o² are selected via cross-validation. In Ai & Chen (2003) experiment, we change only the first layer in the network structure to match the dimension of X,, and the rest of the network structure is the same.
In addition the original settings of Newey & Powell (2003 and Ai Chen 2003), we investigate cases where we add more IVs.
In Newey & Powell (200)), we generate ((Y, X;, Zi,1, Zi Z4,3, Zi ) 2 where Z,, Z,,,, Z, are additional IVs, as follows: first, they generate {(s,, U, Zi,1, Zi, 2. Zi,, Z, 4) from the mul-
0.5
0
0
0
0.5
1
0
0
0
0
0
1
0
0
tivariate normal distribution V
then, they generate X, =
0
0
1
0
0
0
0
1
0
0
0
1
Z + Zi,2 Zr.3 + Z,,4 +U, and Y, (x(X)) + Es. where (X, In([X, 1/+ sgn(X, -1).
Here, e; and U, are unobservable.
In Ai & Chen 2003),, they generate {(Y, Zi Wt,,, Wa,, W1,3) where W.,. Wt., Wa3
additional IVs, follows: first, generate {le,, X1s Vi, U,) 1-1 E, N (0, X2: V.2).
W W
1
0.3
0.3
X1
Lid. ~
Onif(O, 1). VG Lid. 1n,
i.d. ~
W
0.3
0.3
W, =
W.a/
0..
0.3
Σj=1 Wayi and U; iidd N (0, V,2 |W,]); second, generate X2: X1x V W,
RX + U, and X1.70 ho (Xzz) + Es, where hg (X2,) exp (X2) (1++ cxpp (X2:)) and R is chosen as 0.9: then, obtain X; (Xis X2.) and Z; (X V, W., W. W.,).. Here, and U, are unobservable, and (x(),) = X1+30 + ho (X2:), where the function fng and "o are unknown.
C.2 SIMULATION STUDIES USING DEMAND DESIGN DATASETS
For DeePGMM, DFIV, DeepIV, and KIV, we use the exact same model and hyperparameters in Xu et al. (2021a). For LS and IW-LS, network. 77 is 0.001 as a result volos-vali.tion.. For W-ml,l, we use the same network structure IW-LS to estimate the density ratio and select Ç and d² via cross-var validatii.....
the network structure based the economics datasets
23

---

when ignoring Published as a conference paper at ICLR 2022
5.75
4.00
4.00
3.50
2.75
Figure 7: Demand design experiments with stronger correlation between X, and &. The sample sizes are 5, 000. The left graph show the results with p = 0.25 and the right graph shows the results with p 0.75.
Figure 5 shows the results with 5000 samples. First, our proposed IW-LS outperforms the existing methods and minimizes the MSE. Second, LS, which naive nonlinear regression without IV, outperforms the other existing methods using IV. In contrast, IW-LS outperforms LS.
We consider the case where endogeneity causes a larger change outcome. Since the original demand design dataset has small effect of the bias, we slightly change V; in the price equation:
P. = 25 + (C₁ + (T, + 10V, In the original dataset, Y, is generated as Y, 100 + (10 P.) 2P, Eq. However, because the impact of E; on the price is very limited, because the variance of E is relatively small compared to that of Y, This is the reason why the LS also performs well in the previous result; that is, endogeneity (because f"(),) is close to E Y (X)). For this reason, this dataset is not appropriate for investigating the performances of the methods, although it is used in existing studies. Here, consider the different model Y, = 100 + (10 + P.).S.((..) 2P; + 100e,, in which we multiply the error term by 100. In this case, the bias has more serious impact on the values of price and outcome. Figure 0 shows the results with 1, 000 samples, and Figure 7 shows the results with 5, 000 samples. Regardless of the sample size and p. our proposed method outperforms existing methods. Even in this experiment, the LS still performs well. We consider that this is because the correlation between X, and E; is not strong. follows standard normal distribution and has limited impact on price due to the constant term 25 (P: = 25 + (C, + 3),(T,) + VJ), that is, in the variation of P. V; has a small effect compared with the other variables.
obtain good performance
the dataset, V,, which is a cause of the bias,
In this dataset, (X,, takes large values compared to the error term. Under this situation, a model trained to predict Y, may perform well because the influence of Ble * 0 is limited. However, the purpose of using NPIV in the first place is because the latter influence is large, or else effects of [X]] * can be ignored. As expected, in our experiments using Hartford et al. (011)) the least-squares method also performs well, even though the training process ignores the problem of NPIV. To investigate the performance for causal inference, we recommend using simpler datasets, before using more complicated datasets.
We also show how the MSE of the IWMM decreases the sample size grows in Appendix C.4
C.3 SIMULATION STUDIES USING MNIST DATASETS
We also investigate how our proposed method performs high-dimensional data. By using MNIST dataset (LeCun & Cortes, 2010), we convert the low dimensional IVs of economics artificial datasets used Newey & Powell (2003) and Ai & Chen (2003) high-dimensional IVs. We compare our proposed methods, IW-LS and IW-MM, with the LS and KIV.
Extension of the dataset in Newey & Powell (203)): For creating high-dimensionall IVs, we equip the IVs used in Newey & Powell| 2003) to the feature vector of the MNIST dataset. Let D, E R1000 be randomly chosen feature vector of the MNIST dataset of the number 0. We reduce the dimension to 100 by using the principle component analysis. We multiply the original IV in Newey & Powell (2003) by the feature vector to convert the low-dimensional IV the high-dimensional IV; that is, create new IVs 2, E R
this new IV Z; and estimate the structural function f*. The sample size is 2, 500. The other settings are the same as Section The experimental results are shown in the left figure of Figure
2, Z,D,. Instead of Z₁,
24

---

Published as a conference paper at ICLR 2022
-0.4
-10
-088
-0.8
-0.7
0.6
-14
-16
-11
IW
IW
TW
KIV
Figure 8: The log10 scaled MSEs using the setting in Newey & Powell (2003). The left graph shows the results using the original dataset with the MNIST dataset. The right graph shows the results with additional IVs and the MNIST dataset.
wew.s-saaaaaaee antt (eg (oman
0.0
-9.4
-0.8
IS
IW MM
LS
Kiv
LS
IN KM
L5
Kiv
Figure 9: The log10 10 scaled MSEs of dataset in Ai & Chen (2003) with the MNIST dataset. The left graph shows the result with R 0.1 and the right graph shows the result with R 0.9.
Next, add IV the original dataset in Newey & Powell (2003) {(Y, X,, Zr,1, )4
1
0.5
0
0.5
M
0 where Zı, is an additional IV. Then, we generate X; Z₁
0
0
0
Z1 + U. and Y1 = f((() + Eg, where f*XX,) In( 1| 1) sgn(X, 1). Here, E, and Uj unobservable. Let Di,1 E TR.10 be randomly chosen feature vector of the MNIST dataset of the number O, and Da.2 E 11000 be randomly chosen feature vector of the MNIST dataset of the number 1. We reduce the dimension to 100 by using the principle component analysis. We multiply the original IVs by the feature vector to convert the low-dimensional IVs high-dimensional IVs; that is, create use these new IVs Z and Z, estimate the structural function f*. The experimental results are shown in the right figure of Figure g
IV Z,, R Z,, Zi, and Z,,2 R Z Di, We
We also equip the MNIST dataset to the IVs of the dataset used Ai Chen (2003). Let D;,1 100 be randomly chosen feature vector of the MNIST dataset with the number 0, and D 1000 be randomly chosen feature vector of the MNIST dataset with the number 1. We reduce the dimension to 100 by using the principle component analysis. We transform the original IV's in Ai & Chen (2003). W W. IR, to the high-dimension IV' by multiplying them with the MNIST feature vectors: that is, we create new IV W., E IR 100 as Wi, = W;, and W E IR 100 as W,, 2i2 We use these new IVs, WE, and Wr to estimate the structural function ʃ*. The sample size is 2. 500. The other settings are the same as Section 6.1 The experimental results are shown in Figure
Extension of the dataset in Ai & Chen 2003):
C.4 SIMULATION STUDIES ON THE RATE OF THE MSE
We show experimental results on the MSEs of the NPIV and the classical 2SLS under various sample sizes. Here, we have two goals: (i) we validate the theoretical MSE derived Section [5.3 and (ii) show the relative performance of the NPIV method against the classical 2SLS in the learning problem with conditional moment restrictions.
In Figure 10 show the MSEs of the IWMM in the setting of Newey & Powell (2003) with various
sample sizes. We follow the same setting used in Section 6.1 except for the choices of sample sizes. We investigate whether the empirical MSE of IWMM converges 0 with (1/11/7) for sample size n, which theoretically provided in Section 5.3 under some assumptions. We compute the empirical MSEs with sample sizes (500, 1000, 1500, 2000, 3500, 4000, 4500, 5000) with 10 trials, then obtain
25

---

Published as a conference paper at ICLR 2022
0.05
NPIV 1//n
0.04
MSE
0.03
0.01
0.00
1000
2000
3000
4000
5000
sample size
Figure 10: MSEs of the IWMM in the setting of Newey & Powell (2003) with various sample sizes. The light blue region represents the standard deviation.
NPIV
2SLS: Bo B1X
2SLS: Bo B,X + B2X²
MSE
2SLS: Bo + BiX + B2X² + B3X²
0.02
0.00
1000
2000
3000 sample size
4000
5000
Figure 11: MSE of the IWMM the setting of Newey & Powell (2003).
its mean and standard deviation. In Figure 10 compare the empirical MSEs with the line that represents l/ As in our theoretical results, the MSES decays in (1///nn)
With the same setting, also compare the MSEs of IWMM with the classical 2SLS in Figure 11 to understand the effectiveness of the NPIV method. We consider three models for the 2SI.S:
Y 8 + X1 +
Y; 9 + B1 X; + 32X? +
Y, 8 + 8, + 3,X? 8,X? + +
The difference between these three models lies in the choice of the polynomial basis. For instance, the last model approximates the structure function by cubic function. If a model introduces an infinite number of polynomial bases, they can approximate various smooth functions by the series expansion. In other words, as the number of the polynomial basis increases, the classical 2SLS approaches the NPIV. This method of introducing basis called sieve regression econometrics, and Newey Powell (2003) proposed using to solve NPIV. The experimental result in Figure [ also shows that second and third models, with the polynomial bases, X? and X perform closer to the NPIV method than the first model using only X..
important to note that the 2SLS with polynomial basis function is difficult to implement in nigh-dimensiooal situations. In the setting of Newey & Powell (2003). because both X and Z have one dimension, polynomial approximation effective. However, the dimension increases, the series expansion of multivariate functions requires huge number of basis functions, hence approximation becomes very difficult. machine learning, for instance, (Singh et al. (2019) proposes to introduce RKHS to avoid this difficulty.
26


Reviews:
[{"review_id": "jr_A-kugNW2", "reviewer": "Reviewer_S3Qv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Instrumental variable regression is an important problem to tackle for a number of application fields. Therefore an improvement of the methodology in this area should be welcome for the conference audience. \n\nThe paper presents a method that is novel in terms of the estimand it proposes, the density ratio, in order to learn the structural function. Their method can be used with various models and objective functions in learning the causal function. The authors present error analyses for the density ratio and the causal function, as well as experiment results comparing their work to recent algorithms in the field. These constitute the strengths of the present work. \n\nOne of the weaknesses of the present work is in its presentation. Which shortcomings of the original methods are addressed by the present work, and how, is never clearly presented. The lack of description regarding this issue leads the reader to look for a clear performance increase from the present work. \n\nThis leads me to the second weakness of the paper, which is that the experiments do not demonstrate a unanimous advantage for a specific method proposed by the authors. The (potential) advantages of the paper could be more accurately depicted by extending the experiments, e.g. as in Xu et al. (2021).\n\nThough I recommend the acceptance of the paper, I think that the paper would benefit from the aforementioned issues being addressed.", "summary_of_the_paper": "The authors propose a novel method for conducting instrumental variable regression by learning the ratio of conditional and unconditional densities, and using this estimate when learning the causal relationship between cause and effect variables. The authors present error analyses for their estimation tasks, as well as empirical comparison to previous works.", "summary_of_the_review": "The authors present a novel method for instrumental variable regression that promises to be useful in various applications. However the motivation for the method and its empirical testing should be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Urwm2KbqgOm", "reviewer": "Reviewer_ThQb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Improving non-parametric instrumental variables methods is an important problem in causal effect estimation. Based on my limited knowledge, I think this paper is providing a very interesting and promising result, and I do appreciate the theoretical analysis of the estimation error. \n\nMy only concern is that it is a bit technical and hard to follow for the general ICLR community, so I would suggest the authors to try to make it a bit more accessible and readable and therefore even more impactful.\n\nIn particular, as a minor point, I think the abstract is a bit too vague and would benefit from a more precise description of what the paper is about.", "summary_of_the_paper": "The paper presents a method for estimating causal effects (or more generally non/parametric functions) under conditional moment restrictions, focusing in this case on nonparametric IVs. The main idea is casting these conditional restrictions to an unconditional version through importance resampling using a conditional density estimator (e.g. based on a least-squares method with a NN). The paper also provides a theoretical analysis of the estimation error, providing am error bound (Thm1). The method is evaluated based on three econometrics datasets from literature, showing a smaller MSE than the compared methods.", "summary_of_the_review": "The paper seems to provide a novel, interesting and applicable method that improves the accuracy of an important causal inference tool (instrumental variables). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L358ULZIRDb", "reviewer": "Reviewer_FAMR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strong points: \n- The approach of using importance weighting to overcome the difficulty of learning under conditional moment restrictions is an elegant solution to solve the presented NPIV regression problem.\n- The article is well written and the balance between motivation, theory and experiments allows to get a good understanding of the proposal.\n- The experiments are well described and motivated. I appreciate the honest assessment of the relevance of the chosen settings for the proposed method.\n\n## Issues/Points that require clarification:\n- In Section 3.1, there should be references given for the cited methods (general reference for IV methods, for two-step GMM, CU estimator, etc.)\n- The assumptions part is sometimes vague and only phrases certain assumptions without formalizing them. To list a few examples:\n  - In Section 2, do you assume i.i.d. samples? If so, please add this when introducing the available dataset and if not, please specify why you don't need i.i.d. samples.\n  - In Section 4, the definition of the conditional density ratio function $r^*$ implicitly assumes that $p(y,x)\\neq0$ for all $(x,y)\\in\\mathcal{X}\\times \\mathbb{R}$. This should be made explicit, and also be discussed w.r.t. the density ratio estimator $\\hat{r}$ and maybe also w.r.t. the motivating example. (Your Assumption 2 in Section 5 excludes this case of $p(y,x)=0$ but this should maybe be stated already at the moment where you define $r^*$.)\n  - In Section 5, you introduce Assumptions 1 and 2 with the remark \"For ease of discussion\". Does that mean that these are not necessary?\n- Since the authors suggest that their approach should be relevant for learning causal relationships in the context of computer vision and NLP tasks (Section 4.3 and Section 6.2), the claimed added value of their proposal in practice would be more convincing if they could provide an example from either of these contexts where such an IV approach would be suitable.\n- At the end of Section 4.3, the authors conjecture that their proposed heuristic works well when $f^*$ is close to $\\mathbb{E}[Y_i|X_i]$. If I understood correctly, this means that in the case where $\\mathbb{E}[\\varepsilon|X_i]\\approx 0$ the heuristic should work, but from what I understood this approach is motivated by cases where the assumption $\\mathbb{E}[\\varepsilon|X_i]= 0$ is clearly violated, motivating the use of IVs. Is that correct or did I misunderstand the heuristic?  \n- In Lemma 2, what is the role of $T$? From Appendix A.3, it is not clear what $T$ represents. In the proof of Lemma 2, it is written ``_[...] we consider $T$ i.i.d. random variables $W_i=(X_i,Y_i), i=1,\\dots,n$ [...]''. What is $T$ in relation to $n$ in this definition? Is it the same $T$ as in Section 5.3?\n- Since the authors provide a theoretical bound on the MSE of $f^*$, it would be interesting and a valuable addition to empirically assess this bound.\n- Did the authors also assess the performance of the classical linear 2SLS estimator in both experiments (Sections 6.1 and 6.2)? It would be interesting to see the impact of the nonlinear form of the structure function $f^*$ on the performance of this classical estimator to compare with the neural network based approaches.\n\n### Minor comments (that did not impact the score):\n- p.1: when learning causal (structural) relationships _are_ $>>$ when learning causal (structural) relationships _is_\n- p.1: please add a (standard) reference such as Wooldridge (2010) when you first mention the instrumental variable approach (second paragraph of the introduction). \n- p.1: we focus on _on_ nonparametric IV $>>$ we focus on nonparametric IV \n- p.2, Section 3.1: the parameter $\\theta$ of the linear model: shouldn't it rather be $\\theta^*$?\n- p.2, Section 3.1: if there are IVs of dimension $d_X$ or more that _satisfies_ $>>$ that _satisfy_\n- p.2, Section 3.1: $\\tilde{X}=...$: shouldn't it rather be $\\hat{X}$?\n- Please try to avoid alternating between past and present tense when referring to other works (e.g., p. 3 ``Newey \\& Powell (2003) proposed [...] They first define [...]''. I would suggest using only present tense.\n- p. 3, Section 3.3: approaches recently _propose_ $>>$ approaches recently _proposed_\n- p.5, Section 4.3: as a recent finding, such as Bartlett et al. (2020) in linear regression $>>$ as a recent finding suggests in the case of linear regression (Bartlett et al., 2020).\n- p. 6, Section 5: the distribution of $(Y_i, X_i, Z_i$ by $O$ $>>$ the distribution of $(Y_i, X_i, Z_i)$ by $O$\n\n##########################################################################\n### Post-rebuttal update\n\nI thank the authors for their detailed, pertinent and timely responses. The reformulated assumptions and other paragraphs, the changed title and the requested additional experimental results have addressed my concerns. I have therefore decided to increase my rating and to recommend this paper to be accepted. \n\n##########################################################################\n\n### References:\n[1] Guido W. Imbens and Donald B. Rubin. _Causal inference in statistics, social, and biomedical sciences_. Cambridge University Press, 2015.\n\n[2] Judea Pearl. _Causality_. Cambridge university press, 2009.\n\n[3] Jeffrey Wooldridge. _Econometric Analysis of Cross Section and Panel Data, 2nd edition_. MIT Press, 2010.\n", "summary_of_the_paper": "The present paper proposes an importance weighting approach to address the issue of regression under conditional moment restrictions in the context of non-parametric instrumental variables. The authors provide error bounds on the learned structural function. They show that their approach has a convergence rate of $\\mathcal{O}(1/\\sqrt{n})$.\nThe main contributions of this work are presented in Sections 4 and 5, where they first introduce their method based on re-weighting by an estimated conditional density ratio function and then provide an estimation error analysis.\nThe theoretical results are complemented with two synthetic experiments which show that the proposed method can compete with and in certain cases improves upon state of the art.", "summary_of_the_review": "In summary, I am rather convinced that the contribution of this paper is important and its theoretical as well as empirical findings are well described and motivated. However the authors should try to provide relevant applications or settings for their method that would indeed benefit from this novelty (compared to existing works). The experiments do not support the authors' claim that their method outperforms other competitors. Also, they should maybe clarify the paper's content in the title by specifying the context of IVs instead of ``learning causal relationships''. Indeed, from the title one expects to find references to the classical causal inference frameworks (either Imbens \\& Rubin (2015) or Pearl (2009)) and specification of causal estimands. \nI will read the rebuttal carefully and am willing to increase the score if the authors address the raised concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method that uses conditional moment restriction methods to estimate causal parameters in non-parametric instrumental variable settings.  This is done by converting to an unconditional moment restriction setting common in the econometrics causal inference literature.

The paper was reviewed quite favorably by reviewers, and the authors updated the manuscript to address specific issues raised by reviewers.
Decision: Accept (Spotlight)