Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regression
Authors: Juno Kim
Dimitri Meunier
Arthur Gretton
Taiji Suzuki
Zhu Li
Venue: ICLR 2025 Poster (2025)
Keywords: instrumental variable regression
DFIV
deep neural networks
minimax optimality
Primary area: learning theory

Abstract: We provide a convergence analysis of \emph{deep feature instrumental variable} (DFIV) regression (Xu et al., 2021), a nonparametric approach to IV regression using data-adaptive features learned by deep neural networks in two stages. We prove that the DFIV algorithm achieves the minimax optimal learning rate when the target structural function lies in a Besov space. This is shown under standard nonparametric IV assumptions, and an additional smoothness assumption on the regularity of the conditional distribution of the covariate given the instrument, which controls the difficulty of Stage 1. We further demonstrate that DFIV, as a data-adaptive algorithm, is superior to fixed-feature (kernel or sieve) IV methods in two ways. First, when the target function possesses low spatial homogeneity (i.e., it has both smooth and spiky/discontinuous regions), DFIV still achieves the optimal rate, while fixed-feature methods are shown to be strictly suboptimal. Second, comparing with kernel-based two-stage regression estimators, DFIV is provably more data efficient in the Stage 1 samples.

Published as a conference paper at ICLR 2025
OPTIMALITY AND ADAPTIVITY OF DEEP NEURAL FEA- TURES FOR INSTRUMENTAL VARIABLE REGRESSION
Juno Kim 1,2+ Dimitri Meunier
Arthur Gretton'
Taiji Suzuki!
Zhu Li
Departmmet of Mathematical Informatics, University of Tokyo
Center for Advanced Intelligence Project, RIKEN Gatsby Computational Neuroscience Unit, University College London
ABSTRACT
We provide convergence analysis of deep feature instrumental variable (DFIV) regression (Xu et al., 2021), a nonparametrie approach to IV regression using data-adaptive features learned by deep neural networks in two stages. We prove that the DFIV algorithm achieves the minimax optimal learning rate when the target structural function lies in Besov space. This is shown under standard nonparametric IV assumptions, and an additional smoothness assumption on the regularity of the conditional distribution of the covariate given the instrument, which controls the difficulty of Stage 1. We further demonstrate that DFIV, as a data-adaptive algorithm, superior Exxtufeeture (kernel sieve) IV methods two ways. First, when the target function possesses low spatial homogeneity (i.e., it has both smooth and spiky/disconinnous regions). DFIV still achieves the optimal rate, while fixed-feature methods are shown to be strictly suboptimal. Second, comparing with kernel-based two-stage regression estimators, DFIV provably more data efficient in the Stage samples.
INTRODUCTION
We study the nonparametric instrumental variable (NPIV) regression problem (Newey & Powell, 2003; Ai & Chen, 2003; Darolles et al., 2011). For random variables X, Y. ,and ç, we have
y= (X) & E[E|X] +
(1)
Z
X
Y
where X E X' is the endogenous variable, Y is the outcome, and & denotes unobserved confounding which affects both X and Y. The central object of interest is the structural function far: this may be estimated by utilizing an exogenous instrumental variable Z E Z. which satisfies
Figure 1: Causal graph of IV,
E [[ =0 Z X, (Z Y)Ge
(the second requirement is the relevance assumption, and the final requirement denotes independence of Z and Y when incoming edges to X are removed in the graphical model, thus excluding a hidden common cause of Z and Y; see Figure 1). The IV setting can be understood in terms of an example (Angrist & Krueger, 2001): let X denote the price of coffee, Y denote coffee consumption, and denote the level of demand, unknown the coffee vendor (e.g. increased demand due conference deadlines), which affects both X and Y. In this case, Z might be the wholesale price of coffee beans, which influences the price of coffee and is assumed be independent Ç.
Given (1) and (2), taking the conditional expectation w.r.t. Z. fatr satisfies the functional equation fs Y Z], where Px, Pz are the marginal laws of X, Z, and T T.2(Px) (P2(Pz) 2) the bounded linear operator that maps each h (PP to [[h(X)[2] E T.2(Pz)
(Darolles et al., 201 1; solve for this possibly under-determined system, where
Bennett et al., 2023b). The aim of NPIV far is uniquely identified if and only if T' is injective. This task is inherently challenging due to its ill-posed nature. For example, a small perturbation
outcome can lead to estimators that are far
"Correspondence <junokin@@erkelley.cook edu>

---

Published as a conference paper at ICLR 2025
Table 1: Summary of main results for IV regression, when the structural function lies in Besov
space _ (X"). m, denote the number of Stage I, samples, respectively. The constants 7/1 S. 70
denote the decay rates of the link and reverse link conditions and the degree of separation A
(1/p - 1/2)+. The upper bound of DFIV assumes T has maximal smoothness (see Lemma 3.2), while the projected fixed-feature lower bound assumes d: S d₂ and d' d V (dg 2(s A)).
Rates
DFIV
Optimal
Fixed-feature
2e+22
20+220
2e+2on 444
H(1-
Projected
O m 2e-201 daz + n
Ze 200
n
f n
ala-a)taite" ? n 31 21
Full
Ō Ōmm=
20-251 + n
3(1-70+71)
200-70+71) stn
a n 2x4291425
2(1-6)
from the true solution T is often unbounded (Carrasco et al., 2007). There has been growing interest in addressing this challenge using modern statistical and machine learning techniques.
NPIV estimation plays a crucial role in various fields, including causal inference (Angrist & Imbens, 1995; Newey & Powell, 2003), addressing missing data challenges (Wang et al., 2014; Miao et al., 2015), and reinforcement learning (Liao al., 2021; Uehara et al., 2021; Xu et al., 2021; Chen et al., 2022). Existing methods can be broadly categorized into two main approaches: conditional moments methods (Muandet et al., 2020; Liao al., 2020; Dikkala et al., 2020; Bennett et al., 2019; 2023a;c) and two-stage estimation techniques (Newey & Powell, 2003; Carrasco et al., 2007; Chen, 2007; Horowitz, 2011; I; Darolles et al., 201 1: Hartford al., 2017: Chen & Christensen, 2018: Singh et al., 2019; Xu et al., 2021; ;; Ren et al., 2024; Wang et al., 2022; Meunier et al., 2024). Conditional moment methods approach NPIV by constructing a min-max optimization problem of the form min gEg [(J.9) g) for certain function classes F, G. However, obtaining these estimators can be difficult in practice since the solutions are typically saddle points. Furthermore, information- theoretic lower bounds for these methods have not yet been established, it is unknown whether the proposed estimators can achieve the optimal learning rate.
In the present work, we consider two-stage methods. These decompose NPIV into the following steps: Stage learns either the conditional expectation operator or the conditional density X Z depending the algorithm. In Stage 2. the outcome Y is regressed using the estimator obtained from Stage 1. While this offers more stability compared to conditional moment methods by avoiding saddle-point optimization, key challenge remains represent the conditional distribution X Z from the first stage. One approach is to explicitly learn the conditional density X Z from data (Darolles et al., 2011; I; Hartford et al., 2017: Li et al., 2024b). Density estimation is challenging when X and Z are high dimensional, however, and convergence rates can be correspondingly slow (see e.g. Wasserman, 2006, for convergence properties of density estimates).
A second approach is to learn the conditional mean of features of X given Z, where the features of X are the input features of Stage (see e.g.. Chen & Reiss, 2011; Singh al., 2019; Chen & Christensen, 2018; Meunier et al., 2024). This approach, known as two-stage least-squares (2SLS) regression, has the advantage that the Stage problem only as difficult as needs to be in order to solve Stage 2, and does not require to address the harder problem of conditional density estimation. The question of feature dictionary choice remains a challenge, however, and the above methods employ dictionaries for classes of smooth functions (i.e., spline and RKHS classes), which are not data adaptive. More recently, two-stage IV approaches have been proposed with data-adaptive feature dictionaries represended by neural nets (Xu et al., 2021; Ren et al., 2024; Wang et al., 2022). While these have shown strong empirical performance, the corresponding theoretical guarantees remain incomplete, and minimax rates are yet to be established. Finally, while sample splitting is predominantly used in two-stage NPIV, the ratio between Stage 1 (m) and Stage 2 (n) samples is rarely studied in the literature. Singh et al. (2019) and Meunier al. (2024) recently showed that m > n is required in order to achieve the minimax optimal rate in the case of fixed-feature estimators.
Our contributions. We study the deep feature instrumentall variable (DFIV) algorithm proposed by Xu et al. 22 where learnable deep neural network (DNN) features are employed to jointly optimize both stages. As data-adaptive estimator, DFIV exhibits superior empirical performance
2

---

Published as a conference paper at ICLR 2025
compared with exed-feature methods, most notably in cases where the structural function may have both smooth and non-smooth regions (Xu et al., 2021; Chen et al., 2022). In this work, we provide minimax convergence guarantees for DFIV, and formally characterize the conditions under which deep neural features yield performance advantage over fixed-dictionary approaches. These build on existing statistical analyses of DNNs (e.g. Suzuki, 2019; Hayakawa & Suzuki, 2020), which study ordinary regression settings, by contrast with our two-stage setting where two dependent DNNs must be jointly learned. Our result
follows:
We prove an upper bound for the risk of the DFIV estimator fatr in terms of the number n of Stage and 2 samples, respectively, when far lies in Besov space pp (2²). We obtain rates with respect to both the projected pseudometric fate Taee ([aeel(2((p2)) (Theorem 3.1) and the
(Theorem full non-projected metric [|Saur /etel (Theoreem 3.5).
We obtain information-theone DFIV can achieve the minimax optimal rate (Proposition 3.3, 3.7). Moreaver, we show that DFIV can attain the optimal rate with Stage I samples m Rs n, whereas kernel IV estimators require m/n + oo (Singh et al., 2019; Meunier et al., 2024).
lower bounds for NPIV in Besov spaces and demonstrate that
Our rates recover known rates for smooth fatr. established previously (Chen & Reiss, 2011; Meunier al., 2024) with fixed-feature estimators. More importantly, we demonstrate striet separation between DFIV and any exed-feature estimator (Chen & Reiss, 201 1; Meunier et al., 2024) when the target function has both smooth and spiky/diseontinuous regions (Corollary 4.2).
detailed discussion of prior two-stage approaches, and of relevant works addressing convergence of regression with DNNs, is provided in Appendix A. A summary of our rates is given in Table 1. The rest of the paper structured as follows: In Section 2, we outline the DFIV algorithm and define target and hypothesis classes. The upper bounds for DFIV and comparison to NPIV minimax lower bounds are presented in Section 3. The separation between fixed-featur methods and DFIV is established in Section 4. All proof's are deferred to the appendix.
2
BACKGROUND AND DFIV ALGORITHM
2.1 INSTRUMENTAL VARIABLE REGRESSION
Nonparametric instrumental variable (NPIV) regression typically has a treatment E X' and outcome Y E R. related via the structural function fstr X' - R and unobserved & that affects both X, Y:
Y = [x) + B[{] = O, E[g[ * 0.
This implies that (z) A E| x] so that ordinary supervised regression methods cannot be used. Instead, we introduce an instrumental variable Z E 2 known to be uncorrelated with &, that is E[g Z] 0, and satisfying the requirements (2). For simplicity we assume that X, Z are bounded and set X' [0, 1]d, Z j0, 1J equipped with the induced probability measures Px, Pz. By defining the projection operator
T (Px) + (P22) (Tf) E[)
eq. (3) can be described as the nonparametric indirect regression (NPIR) model (Chen & Reiss, 2011)
= T (2) 1), " sar Tarr E.,
where E[n] = O, and hence we seek to solve the inverse problem fatr || The problem is generally ill ill-posed, however; the solution may ill-behaved or not unique (Nashed & Wahba, 1974; Carrasco et al., 2007). To analyze this problem, we require mild regularity of the noise.
Assumption 1. (i) There exists 01 > O such that ( = 2) is a crssuaaaassian for all 2 E 2.
(ii) (For lower bound only) There exists de > 0 such that Var(n) =) 2 où and the KL divergence between 1) = 2), p + 7/ 7 = 2) is bounded above by 257
for all z E 2, E TR.
This is satisfied for example 17 = is (0, o² (z)) distributed with oo S <())) $ o1 which is standard assumption in the literature (Bissantz et al., 2007; Chen & Reiss, 2011).
3

---

Published as a conference paper at ICLR 2025
Besov spaces. In order facilitate leaming-theoretic analysis, we suppose that the structural function fate belongs to (the unit ball of) a Besov space (2)) of smoothness s on X. Besov spaces are well-studied class of functions of generalized smoothness, and include Hölder spaces, Sobolev spaces, and classes of bounded variation. See Definition A. for details. In particular, Besoy spaces with smoothness below the threshold du/p contain functions with discontinuities which are more difficult to learn; our theory accounts for this regime. We also assume & > d1(l/p 1/2)+ so that (.pg(()))) continuously embeds into L (2))).
Assumption 2 (Structural function). fatr E (UB)i.(()) and fstel I C for some 0 < p. S oo, 8 > (1/p - 1/ and C' > O. If S ds/p. Px also has Lebesgue density bounded above.
Our analysis utilizes the B-spline system, which forms a hierarchical basis for Besov spaces: the span of B-splines up to resolution k is strictly contained in the span of B-splines at resolution k + 1. Definition 2.1 (B-spline basis). The cardinal B-spline of order r E N is iteratively defined as the repeated convolution 4, 4 * to E C(R) where to(2) = 1 10,11] (z). The tensor product B-spline
of order with resolution k E Z,2 and location E E T II (-r, 1, ,2*, is
wk,e(x) = II (2 E,).
2.2 TWO-STAGE LEAST SQUARES REGRESSION
A popular method for performing IV analysis is two-stage least squares (2SLS) regression. In 2SLSS, the structural function far is modeled as linear combination u (xx of fixed basis functions (xx). for instance Hermite polynomials (Newey & Powell, 2003) or reproducing kernel Hilbert spaces (RKHS) (Singh al., 2019; Meunier al., 2024). The weight vector u estimated by performing two successive regressions. In Stage 1, the conditional mean embedding (CME) Ex |x|z (X)] (Song al., 2009; Park Muandet, 2020; Klebanov al., 2020) is approximated by another set of basis functions V o0( (Z), where the coefficient matrix V is computed by solving the rector-valued regression (Grünewälder et al., 2012; Mollenhauer Koltai, 2020; Li et al., 2022: 2024a):
arg miny Ex, -
- -
(5)
In Stage 2, we obtain u by regressing Y against Ex12(fair ~ T Vo(Z), where the Stage 1 CME estimate is utilized:
à = arg min, By, | u (2) >2
Finally, the structural function is estimated as (x) = à (z)). Both stages can be solved in closed form by performing ridge regression, including for infinite (RKHS) feature dictionaries.
2.3 DEEP FEATURE INSTRUMENTAL VARIABLE REGRESSION
We now recall the deep feature instrumental variable (DFIV) regression algorithm (Xu et al., 2021), which extends the 2SLS framework to incorporate learnable feature representations th, (x), de, (z) in both stages. The respective hypothesis classes, equipped with the - -ormm are denoted as
Fr = (tb, X - IR: 0g E Ox), F= = {cho 2 + R 0g E Oz).
As in Singh et al. (2019): Meunier et al. (2024); Xu et al. (2021), we do not presume access to samples from the joint law (X,Y, Z), and instead are provided with m i.i.d. samples D, = {(x,. 2, km from (X, 2) for Stage 1, and i.i.d. samples D2 (w,, { from (Y, 2) for Stage 2.
Stage 1 regression: Given data Dj and fixed Stage parameter Oz, the conditional expectation Exiz [vbe, (X) is learned using the network Φe (7) by minimizing the empirical loss.
0. = Q arg min 0.50, m
1
Σ 2=1
(Wo, (2.) -oo (2i)))
The resulting estimate is also denoted as Éx z [Vo__ =- ( to emphasize that 02 is function of the current Stage 2 input feature parameters 0z.
4

---

Published as a conference paper at ICLR 2025
Stage regression: Given data D₂ and the Stage estimate Pb.- we solve the following regression where R: O,, > R22 is an optional regularizer:
_ arg min 0z
1
Ut (xxixi/(oo(((;)) AR(O.).
(8)
The final DFIV for estimate fatr is returned fatr -0g
Remark 2.2. We note that jointly optimizing (7) and (8) challenging as the optimal 0_ is itself function of the current 0n. The DFIV algorithm originally proposed in Xu et al. (2021) takes vy.,: 09. to be vector-valued mappings, and treats separately the linear maps u, V (eqs. (5) & (6) via ridge regression. This enables more effective optimization by partially backpropagating 0₂ through the analytical solutions of u, V; see Section 3 of Xu et al. (2021) ) for details. From the sample complexity viewpoint, however, it suffices consider the scalar output u Va. single neural network and learn its conditional expectation in Stage 1. more sophisticated bilevel optimization approach is described by Petrulionyte et al. (2024).
Smooth DNN class. In our paper, the 60 chosen from class of smooth DNNs, defined with depth L, width W, sparsity S, norm bound M and a C" activation (applied elementwise) as
FDNN W, S, (clip..c (W o ((0)). (W + b()) X + RR w E Wxd w E Wxw w R ( W.W ((L)) TR,
El 1l + |160 S S, / || | S M]
Here clipo.c :R + I-C is any bounded, I-Lipschitz C function equal the identity when |x|| & C: we fix any C C and omit them from the notation. The use of smooth activations and smooth clipping is in order to ensure FDNN C. (x) B B (X) (Suzuki, 2019) and thus to obtain Besov norm guarantees for the DNN estimator. Moreover, we specifically consider sigmoid activations (x) (1 -- for ease of analysis, however the results
be extended any sufficiently smooth activation e.g. SiLU, GELU, Softplus, as well as piecewise polynomial activations such ReQU; see the discussion in Appendix C. Some of our results also hold when the ReLU DNN class, defined with activation (zz) = 0 V x and clipe(x) = (-C) V (x A C), is used instead.
THEORETICAL ANALYSIS OF DFIV
We now establish the sample complexity of DFIV when fstr lives in the Besov space , (X) with smoothness s. We obtain upper and lower bounds for both the projected error (Section 3.2) and non-projected error with p 2 2 (Section 3.3), and obtain precise conditions for when the minimax optimal rate is achieved. We begin by stating our assumptions the projection operator T.
3.1 LINK AND SMOOTHNESS CONDITIONS
The operator T plays crucial role in determining the sample complexity due to the following factors.
The NPIV model (4) shows that Stage 2 essentially learns faue using data from the projected function fssr, leading loss of information. Hence the rate will worsen depending how contractive T is in an sense, which is usually quantified by a link condition (see e.g., Nair al., 2005; Chen & Reiss, 2011; Meunier et al., 2024), here given as Assumption 3.
The difficulty of Stage 1, where we estimate the conditional expectation Twi, for the output the, of Stage 2, depends on the regularity properties of T for f in certain function classes. We will control this using a novel smoothness condition (Assumption
4*).
'A minimizer of Stage always exists as we take θ to compact and (7) is continuous w.r.t. 0g. minimizer of Stage 2 is not guaranteed to exist the output ô_ of Stage may not depend continuously on 0g. but may simply take approximate minmmzzer e-close the infimum and obtain the same results e
5

---

Published as a conference paper at ICLR 2025
To obtain the final non non-projected rate Il fatell Saael1120 (P(x)) we must start with the projected rate [T - (P) and then use reverse link condition (Assumption 6) to derive the former.
We now state and discuss each assumption in full. Denote by P/ = span & 1 C Bp.g (2) the linear span of all B-splines of order up to resolution i and II,.* the projection operator to P/ Assumption 3 (link condition). There exists 7 2 0 such that for all f E (x))
II,. [T( - - - fll
(9)
That is, T' is more contractive higher resolutions. Assumption 3, and the reverse link condition Assumption 6, are equivalent to or follow from standard conditions imposed in the IV literature; for example, they follow from the link condition in Chen & Reiss (201 1) and Meunier et al. (2024) stated in terms of the Hilbert scale generated by certain operator." 2 Note that rank II, = dim p/ * x 2hd_, hence (9) corresponds to polynomial (mildly ill ill-posed) rather than exponential (severely ill ill-posed) decay w.r.t. the number of basis elements (Blundell et al., 2007).
We now discuss the issue of smoothness. Stage aims to learn the conditional expectation Téa for the data- bata-dependent predictor V0_- E Fx with the DNN class Fs.. This is a separate regression problem 2, its risk depends on the smoothness of Tva a nonsmooth predictor 1/a (possibly due overfitting) may have an irregular projection and end up incurring larger error in Stage 1. Thus we would like to understand the smoothness properties of functions residing in the projection of the Stage 2 DNN class T[F,). In particular, an estimate such as the following desirable.
Assumption 4t DNN S CT U(B o ot (2)) for each class of sigmoid or ReLU DNNs on X,
for some 0- < p', S 00, s > d1(p/p' 1 and Or > O that do not depend on L, W, S, M.
Although Assumption 4* may hold T is sufficiently mollifying (see the example below), it is quite restrictive as the class FDN is much larger than the unit ball of any Besov space* and varies depending on design choice. However, motivated by the observation that the neural network Va is approximating /str E U( p.q (2)),, we can use a much less restrictive assumption:
p.g Assumption (smoothness of T). T (X))) Cr UB (2)) for some 0 < <'p' S co,
(2))for >> (1/p 1/2)+ and Cr > 0. If S dz//', Pz also has Lebesgue density bounded above.
The above formulation is a natural condition as it quantifies how much preserves smoothness of the target Besov space. In comparison, the link conditions essentially quantify how much T contracts in an Le-sense, giving us two distinct perspectives on the action T'. Assumption also interacts with the link conditions to relate s' back to see Lemma 3.2 for details.
We give some concrete examples in which Assumption
satisfied. Suppose T smooth in the sense that the conditional density of X given Z exists and satisfies Hölder condition w.r.t
[xxiz(0\21) - - (x1=) & ( =21l for some 1) E LL'(
It follows that [T](2)) - T=(2)) < ||
22 for any bounded integrable function f. Since Ca (X) continuously embeds into the Zygmund space B (X) (Giné & Nickl, 2015, Proposition 4.3.23), Assumption holds with = at in this scenario, Assumption 4 also holds. At the opposite extreme, consider the case where the instrument is perfect, dN = d, and T restricts to quasi-isometry between Besov spaces of the same order, the simplest example being = idx. Then Assumption 4 holds with s' s, however Assumption 4* cannot be true. Thus Assumption 4 is much more general condition encompassing both high and low degrees of smoothing.
Controlling Stage 2 smoothness. In order to replace Assumption 4# with Assumption 4, we need to ensure that the Stage 2 predictor vh. lies in the U(B (2)), up to normalization. Using the
"This can be seen by taking the spectrum = & in Assumption of Chen & Reiss (2011) and restricting to the span of first N basis elements or complement in Assumptions 5 with link function ø(t) = /4, 'Even when using smooth DNN class, the Besov norm of generic DNN can scale polynomially in the specified weight norm bound, which we in turn take scale polynomially in m, a in results.

---

Published as a conference paper at ICLR 2025
smooth DNN class ensures V/2- E (X))(X)) but does not by itself guarantee a norm bound (see footnote 3). A simple way to ensure this is to restrict the domain as follows:
0 = 0g E +₂ | (1o._l S Cw)..
(10)
In practice, since the restriction cannot be applied priori, we can run the DFIV algorithm from random initialization and simply discard solutions which are ill-behaved: we show that the subset (10) still contains a rate-optimal solution. A more sophisticated method to guarantee smoothness which we also consider in our theoretical framework is to explicitly add the Besov (semi)norm as a penalty to be optimized for in Stage 2,
R(0²)= = l 19 (2) or R(0²) 1l B (x),
(11)
for any exponent 2. The seminorm can be easily estimated from its equivalent discretization Ifl mp (2) x (E wr. (f.2=
(Giné & Nickl, 2015, p.330); the L' component can
be ignored since t'o, is bounded. It is also bounded above by the Sobolev seminorm |f wg(x))
Ej D (())) due to the string of continuous embeddings (x')) + By (x) - (2)
(Giné & Nickl, 2015, Proposition 4.3.20). This can be numerically computed or backpropagated through O2 any desired accuracy independently of the data by differentiating the network output with respect to its input at mesh points; thus we assume I can be computed exactly for simplicity.
Such smoothness-basee or gradient-nomt penalties have been considered before. For example, regularizers of the form E [13]. B|] Dive_ [i (equivalent to the Besov seminorm penalty when p = q 2. 1) similar have been successfully used to improve training generalization of deep networks (Gulrajani et al., 2017; Sokolić et al., 2017; Arbel et al., 2018). Furthermore, Rosca al. (2020) argue that smoothness regularization leads to benefits in inductive capabilities, robustness, and modeling performance.
3.2 PROJECTED UPPER AND LOWER BOUNDS
We now present the upper bound for the projected mean squared error (MSE) of DFIV. The proof quite involved and is presented throughout Appendix D, with a brief sketch provided in Appendix B. We also require some new theory on rate-optimal estimation with sigmoid DNNs, which is developed in Appendix C.
Theorem 3.1 (projected upper bound for DFIV), Under Assumptions 1(),2,3 and Assumption 4 with domain restriction (10) regularization (11) with À asymptotic the rate below, Assumption 4* without regularization, by choosing Fz = FDNN ([log, dz + 1, (oly poly(m), poly (m)) and similarly F2 to be smooth DNN classes, it holds that
2x12 ED Da [ITfar - TSrr (( E m 1d log m + (m A n) 20
2d
log(m A n). (12)
See below for discussion. Rather surprisingly. the rate in Stage 2 samples n depends only on the Stage smoothness s', while the rate in Stage 1 samples m depends on the smoothness of both stages. We remark that F= (and F2 if Assumption 4* is used) can be replaced with ReL.U DNNs since smoothness of the network output is not required. In this case, the network depth must scale logarithmically in m, and the log factors in (12) replaced by log".
Projected minimax lower bound. We now demonstrate the minimax optimality of the above learning rate. As usual, we state assumptions first. The following is required lower bound the error in L 22(Px) k) by the ordinary 2 error over the domain.
Assumption 5. Px has Lebesgue density bounded below; more generally, the Rudon-Nikodym dm w.E.. Lebesgue measure mx exists and is bounded above, so (1-1lza(Px) 2 (1=)) derivative aPx
Assumption (reverse link condition). There exists 7o 2 m such that for all f E p!
2 2- 1fllaa)))) & (TT))))))))
This is equivalent to assuming a polynomial rate for the sieve measure of ill-pssedness, which depends on the operator 7' and can be estimated from the data along with y with the method in (Blundell
7

---

Published as a conference paper at ICLR 2025
et al., 2007). We are most interested in the "oo 71, which gives precise characterization of the decay of T. This assumption also allows us to connect the exponent s' appearing in (12) to the target smoothness Comparing the sizes of the standard and projected unit balls, can show: Lemma 3.2. For Assumptions 4 and to both be satisfied, it must hold that
s' /d2 S (s + (1)/ /0)/d2.
(13)
The lemma is proved in Appendix F.2, where we also show that equality can hold if e.g. d₂ = d2 and maps B-splines to (scaled) B-splines. equality is achieved in (13), we say that T has maximal smoothness. In this case, the DFIV rate in Table 1 is retrieved.
Now the minimax lower bound for any estimator for the NPIV problem can be derived as follows. This is nontrivial extension of the known lower bound in the Sobolev setting (Hall & Horowitz, 2005; Chen & Reiss, 20,1), as B-splines are neither orthogonal nor independent.
Proposition 3.3 (projected minimax lower bound). Under Assumptions 1(11),233,566, holds that
inf sup (X),
[IT -Tf ( Zn 2x4291
2+4270
(14)
NPII
where farr NPIV is taken over all measurable functions of the data (DI.D))
The proof is given in Appendix F.1 and relies on reduction the NPIR model to which the Yang- Barron method can be applied; in fact, the lower bound holds for all valid estimators of Ts The bound is independent of m, hence most informative in the limit m - co. The usual nonparametric rate is retrieved when To 71 0. Comparing with the upper bound (12), we conclude:
Corollary 3.4 (projected optimality of DFIV). 1f = 7/2 and has maximal smoothness, DFIV 264-20
attains the nearly minimax optimal rale n
2x42n0+dy logn n in the projected metric if m = (2()).
We also observe two potential factors of suboptimality. If the inequality (13) is strict, DFIV is upper bounded by the rate f > (reduced regularity) of the Stage regression. If to > ",, the link conditions are loose, so the lower bound also becomes comparatively weaker. In the latter case, it may also be beneficial to scale as 1V 2n 22,2 de 20d
2n
2x+20p 224210+ which is suboptimal due to the increased difficulty
Ω
201221
20 1d
rather than 2(n), since the rate in can become slower than in (12).
3.3 FULL UPPER AND LOWER BOUNDS
Corollary 3.4 establishes minimax optimality of the projected MSE, however since T is often unbounded in NPIV, we cannot directly deduce the full MSE from the projected rate. Therefore, more interesting result is whether DFIV can achieve minimax optimality in the non-projected rate. To this end, we prove the following non-projected upper bound in Appendix E.2.
Theorem 3.5 (full upper bound for DFIV), Let 2 2. Under Assumptions 1(1),2,3,6 and Assumption 4 with domain restriction (10) or regularization (11), by choosing as in Theorem 3.1, F, = FDNn (flog d. + 1, poly(m), poly(m), (m)) and F, to be smooth DNN classes, it holds that
ED,,D, De [llfarr
far (AIllLAA(P))
E m
26x to to1l 129
log m + (mAn) 2 2 m1
2d' 2-2
log(m (15)
The log factors can be improved to log
Stage 2 smoothness revisited. The control over the smoothness of fore now plays dual role: besides bounding the difficulty of Stage I through T f it also determines the strength of the reverse link condition that determines the final non projected MSE, since Assumption 6 is stated in terms of the B-spline basis. To be precise, the I,² risk must be bounded by taking fN, for to be the width N approximations of fars farr, respectively, and evaluating the truncation
[l - Saul-aull(((p)) & Ilfar fNll + 2 IIT -T + IISaar - SINNl (16)
Then the truncation error | f directly depends on the regularity of far. and smoothness-based restriction regularization becomes unavoidable obtain theoretical guarantees.
8

---

Published as a conference paper at ICLR 2025
Remark 3.6. We point out that Theorem 3.5 only studies the situation when p 2 2. The regime p < 2 where separation with exed-feature estimators is achieved requires special treatment, and proved under extended link condition in Section 4; see the discussion therein.
We now provide the uniform NPIV lower bound, proved in Appendix F.1.
Proposition 3.7 (full minimax lower bound). Under Assumptions 1(i)),,33 it holds that
inf NPIV
sup E,
TE [ILfar - [stillEa(2) 2 n 2.+271+ds
From this, we conclude the following result, which to our knowledge is the first result along with Corollary 3.4 to establish minimax optimal rates for NPIV with deep neural networks.
Corollary 3.8 (optimality of DFIV). If 2 2, 70 "1 and T has maximal smoothness, DFIV attains the nearly minimax optimal rate n. 2++2 tax (log n) नकक if m (2
More generally, the upper bound always scales as n have maximal smoothness, the first exponent will deteriorate due increased difficulty of Stage 1; if the link conditions are loose (7o > 71), the second exponent will deteriorate.
for large enough m. If does
Optimal splitting. We find that the sample requirement for Stage of DFIV in Corollaries 3.4, 3.8 is only = (n)). In contrast, for kernel IV, Singh et al. (2019) and Meunier al. (2024) require m > (()) to obtain minimax optimal rates, prescribing an asymmetrie splitting if the Stage 1, samples are split from single dataset. However, this leads to suboptimal scaling in the total number of samples m + n. Our results show that DFIV requires strictly less Stage data, retrieving the same optimal rate in the total number of samples.
SEPARATION WITH FIXED-FEATURE IV
It has been empirically observed that DFIV outperforms existing methods especially when the structural function is non-smooth or discontinuous; see the experiments on the conditional average treatment effect and behavior reinforcement learning tasks in Xu al. (2021). In this section, we study the efficiency of DFIV in the setting where fate lives in a Besov space with < 2. In this regime, we rigorously establish a separation between DFIV and linear estimators including sieve or kernel-based algorithms, proving the superiority of deep neural features over non-adaptive methods.
For ordinary regression d' d-dimensional Besov space, the minimax rate over all linear estimators 2(x-a) known be lower bounded by Iо 2012 where A d(1/p - 1/2) determines the degree of separation from the optimal rate (Donoho & Johnstone, 1998; Zhang et al., 2002). This occurs because functions in the regime p < are highly spatially inhomogeneous, consisting of both jagged (possibly discontinuous when < d/p) and smooth regions. Adaptive models such as DNNs have an inherent advantage over linear estimators by allocating more complexity (i.e. choosing higher-frequene basis elements as needed) in regions of low smoothness to capture spatial variability efficiently (Donoho & Johnstone, 1998; Suzuki, 2019).
A linear IV estimator is formally defined as any estimator of fstr of the following form,
fL(x) = Σ (z (2) D1) U1 Un X x (X x x2) + R.
(
We only require linearity w.r.t. the Stage 2 responses, as the lower bound will again hold for any reduction to the NPIR model (4). fixed-feature methods such 2SLS, nonparametric 2SLS (Newey & Powell, 2003), Nadaraya- Watson methods (Carrasco et al., 2007; Darolles et al., 201 1), sieve IV (Newey & Powell, 2003; Chen & Christensen, 2018), kernel IV (Singh et al., 2019; Meunier et al., 2024), and moment- noment-based methods (Zhang et al., 2023) are all examples of linear IV estimators.
We first show that all linear IV estimators ineur degree of suboptimality when the target function possesses low homogeneity. The following lower bound is proved in Appendix F.3 by adapting the approach in Zhang al. (2002) for univariate regression to IV regression.

---

Published as a conference paper at ICLR 2025
Theorem 4.1 (lower bound for linear IV estimators). Under Assumptions 1(i),,,3 and assuming Px has Lebesgue density bounded above, for = dz(1/p /p- 1/2)+, holds that
inf
sup
E [1I. Satr 122 Zn" 3(a-A)422
2(a-A)
fr. finerr far EU( (
(2))
This is not directly comparable with DFIV, however, as we assumed 2 2 in Theorem 3.5 so as to not incur any looseness when evaluating (16) with the reverse link condition. More precisely, fate: fat had to be approximated using B-splines up to fixed resolution cutoff, which lower bounds the truncation error by the best N-term linear approximation enror Kolmogorov width, which is suboptimal when p < 2 (Vybíral, 2008). Nonetheless, DNNs are capable of choosing from a much wider pool of basis elements to achieve the optimal nonlinear approximation rate N-s/d, for functions of low spatial homogeneity (Suzuki, 2019). To account for such adaptive selection, now extend Assumption 6 to varying linear subsets of a higher resolution horizon.
8-2 Assumption 7 (extended reverse link condition). Fix C* Ad, + 1 and let S any subset of
size ((2kda) of the set of B-splines up to resolution [C*k], k]. Then 2 2-200/11fllle (f S (Tfllla(P))) holds for all f E span S.
The intuition is that 7' is more contractive for functions with higher complexity (i.e. requiring many basis elements to construct), and hence the tighter link constant 2 "oo
2- holds for any subset with size similar to P.Ak, instead of 2-noC*k guaranteed for the whole space P. Such a dimensional separation is in fact easily satisfied: for example, suppose T is the projection in R." to the orthogonal complement of span(v) for unit vector u such that |ui],-..,, //pl 2 _ for some c (0, Then VD
for any size D' subset A C {1, DD and unit vector w E span(ej S
[Twll2 = 1 - ()jEn)ee)) 2 1-
Ejea Ejen - oz 2 c(1 - D'/D).
Hence as long as D' = D(D) (indeed D' D1/C" in Assumption 7). it is possible that T is 'mini- mally contractive' on all subspaces P spanned by D' basis elements (inf we Pawzol [||l// /////// R 1) even as D, D' - oo, but 'maximally contractive' on the entire space (inf wnnfoyll Tull 2 ||
T
0).
With this modification, we conclude the following strict separation in Appendix E.2. notable consequence is that the assumption of maximal smoothness of TT' can be weakened to range of s depending on the gap A.
Corollary 4.2. Under the setting of Theorem 3.5 and also Assumption 7, the upper bound (15) and Corollary 3.8 hold for < p <
Hence comparing with Theorem 4.1, DFIV is faster than any linear IV estimator if yo 7 and
(s A A)d
8 + 7/0
To 8
cf. (13).
8 A)d A(2 + dz)
d, dz
do
Remark 4.3. We also show a separation result for the projected rates in Appendix F.4. In this case, Assumption is needed since Theorem 1 is valid for all p 0, being independent of the reverse link condition. On the other hand, the lower bound requires an additional assumption that Z sufficiently 'spatially covered' by the projection operator so as to be difficult for spatially non-adaptive estimators. We then prove the projected minimax error for any linear IV estimator is
2(a 4) 2ng
at least 92 n TA A14211+ 1V11
(Theorem F.2), which is worse than the DFIV upper bound
(12) e.g. 70 = 71. T' has maximal smoothness and dg > (dz 2(s A)) V
A dz.
844700
CONCLUSION
In this paper, we developed a novel estimation error analysis for two-stage IV regression with neural features. We proved that the DFIV algorithm achieves the minimax optimal rate when the structural function lies in Besov space and further demonstrated that DFIV can outperform fixed-feature IV estimators by adaptively learning functions with low spatial homogeneity. Furthermore, we showed that balanced number of Stage and 2 samples suffices to attain optimal performance. These results provide a rigorous foundation for the advantages of DFIV in terms of both adaptivity and sample efficiency, paving the way for further exploration of neural features for causal inference.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Juno Kim was partially supported by JST CREST (IPMICR2015), Taiji Suzuki was partially supported by JSPS KAKENHI (24K02905) and JST CREST (JPMJCR2115). Dimitri Meunier, Arthur Gretton and Zhu Li were supported by the Gatsby Charitable Foundation.
REFERENCES
Chunrong Ai and Xiaohong Chen. Efficient estimation of models with conditional moment restrictions containing unknown functions. Econometrica, 71(6):195--1443 2003.
J. Angrist and A. Krueger. Instrumental variables and the search for identification: From supply and demand to natural experiments. Journal of Economic Perspectives, 15:69-85, 2001.
Joshua Angrist and Guido Imbens. Identification and estimation of local average treatment effects, 1995.
Michael Arbel, Danica Sutherland, Mikolaj Binkowski, and Arthur Gretton. On gradient regularizers for MMD GANs. Advances in Neural Information Processing Systems, 2018.
Benedikt Bauer and Michael Kohler. On deep learning as a remedy for the curse of dimensionality in nonparametrie regression. The Annals of Statistics, 47(4):2261-2288 2019.
Andrew Bennett, Nathan Kallus, and Tobias Schnabel. Deep generalized method of moments for instrumental variable analysis. In Advances in Neural Information Processing Systems, 2019.
Andrew Bennett, Nathan Kallus, Xiaojie Mao, Whitney Newey, Vasilis Syrgkanis, and Masatoshi Uchara. Inference on strongly identified functionals of weakly identified functions. In Conference on Learning Theory, 2023a.
Andrew Bennett, Nathan Kallus, Xiaojic Mao, Whitney Newey, Vasilis Syrgkanis, and Masatoshi Uehara. Minimax instrumental variable regression and 12 convergence guarantees without identifi- cation or closedness. In Conference on Learning Theory, 2023b.
Andrew Bennett, Nathan Kallus, Xiaojie Mao, Whitney Newey, Vasilis Syrgkanis, and Masatoshi Uehara. Source condition double robust inference functionals of inverse problems. arXiv preprint arXiv:2307. 13793, 2023c.
M. Z. Berkolaiko and I. Ya. Novikov. Unconditional bases in spaces of functions anisotropic smoothness. Proc. Steklov Inst. Math., 204(3):27-41, 1994.
N. Bissantz, T. Hohage, A. Munk, and F. Ruymgaart. Convergence rates of general regularization methods for statistical inverse problems and applications. SIAM Journal on Numerical Analysis, 45(6):2610-2636, 2007.
Richard Blundell, Dennis Kristensen, and Xiaohong Chen. Semi-nonparametric IV estimation of shape-invariant Engel curves. Econometrica, 75:1613-1669, 2007.
Marine Carrasco, Jean-Pierre Florens, and Eric Renault. Linear inverse problems in structural econo- metrics estimation based on spectral decomposition and regularization. Handbook of Econometrics, 6:5633-5751, 2007.
Xiaohong Chen. Large sample sieve estimation of comi-nonparametric models. Handbook of Econometrics, 6:5549-5632, 2007.
Xiaohong Chen and Timothy M. Christensen. Optimal sup-norm rates and uniform inference nonlinear functionals of nonparametric IV regression. Quantitative Economics, 9(1):39-84) 84, 2018.
Xiaohong Chen and Markus Reiss. On rate optimality for ill- posed inverse problems in econometrics. Econometric Theory, 2773):997-521, 2011.
Yutian Chen, Liyuan Xu, Caglar Gulcehre, Tom Le Paine, Arthur Gretton, Nando de Freitas, and Arnau Doucet. On instrumental variable regression for deep offline policy evaluation. Journal of Machine Learning Research, 23(302):1-4)) 2022.

---

Published as a conference paper at ICLR 2025
S. Darolles, Y. Fan, J. Florens, and E. Renault. Nonparametric instrumental regression. Economet- rica, 7965):554--1563 2011.
Tim De Ryck, Samuel Lanthaler, and Siddhartha Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 143:732-750, 2021.
Ronald A. DeVore and Vasil A. Popov. Interpolation of Besov spaces. Transactions of the American Mathematical Society, 305(1):397-414, 1988.
Nishanth Dikkala, Greg Lewis, Lester Mackey, and Vasilis Syrgkanis. Minimax estimation of conditional moment models. In Advances Neural Information Processing Systems, 2020.
David L. Donoho and lain M. Johnstone. Minimax estimation via wavelet shrinkage. The Annals of Statistics, 2673)877-921, 1998.
Dinh Düng. Optimal adaptive sampling recovery. Advances in Computational Mathematics, 34:1-41, 2011.
Dinh Düng. Continuous algorithms in adaptive sampling recovery. Journal of Approximation Theory, 166: 166:136-13,, 2013.
Evarist Giné and Richard Nickl. Mathematical foundations of infunite-dimensionall statistical models. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2015.
Steffen Grünewälder, Guy Lever, Luca Baldassarre, Sam Patterson, Arthur Gretton, and Massimilano Pontil. Conditional mean embeddings as regressors. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 1803-1810, 2012.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. Advances in Neural Information Processing Systems, 2017.
Peter Hall and Joel L. Horowitz. Nonparametric methods for inference in the presence of instrumental variables. The Annals of Statistics, 33()):2904-2929, 2005.
Jason Hartford, Greg Lewis, Kevin Leyton-Browe and Matt Taddy. Deep IV: flexible approach for counterfactual prediction. In International Conference Machine Learning, 2017.
Satoshi Hayakawa and Taiji Suzuki. On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces. Neural Networks, 123:343-361, 2020.
Joel Horowitz, Applied nonparametric instrumental variables estimation. Econometrica, 79(2): 347-394, 2011.
Ilja Klebanov, Ingmar Schuster, and Timothy John Sullivan. A rigorous theory of conditional mean embeddings. SIAM Journal on Mathematics of Data Science, 2(3):583-600) 2020.
Sophie Langer. Approximating smooth functions by deep neural networks with sigmoid activation function. Journal of Multivariate Analysis, 182, 2021.
Zhu Li, Dimitri Meunier, Mattes Mollenhauer, and Arthur Gretton. Optimal rates for regularized conditional mean embedding learning. Advances in Neural Information Processing Systems, 35: 4433 4445, 2022.
Zhu Li, Dimitri Meunier, Mattes Mollenhauer, and Arthur Gretton. Towards optimal Sobolev norm rates for the vector-valued regularized least-squares algorithm. Journal of Machine Learning Research, 25(181):1-51, 2024a.
Zihao Li, Hui Lan, Vasilis Syrgkanis, Mengdi Wang, and Masatoshi Uehara. Regularized Deepl with model selection. arXiv preprint arXiv:2403.0422 04436, 2024b.
Luofeng Liao, Yiu Chen, Zhuoran Yang, Bo Dai, Mladen Kolar, and Zhaoran Wang. Provably efficient neural estimation of structural equation models: An adversarial approach. In Advances in Neural Information Processing Systems, 2020.
12

---

Published as a conference paper at ICLR 2025
Luofeng Liao, Zuyue Fu, Zhuoran Yang. Yixin Wang, Mladen Kolar, and Zhaoran Wang. Instrumental variable value iteration for causal offline reinforcement learning, arXiv preprint arXiv:2102.09907 2021.
Dimitri Meunier, Zhu Li, Tim Christensen, and Arthur Gretton. Nonparametric instrumental regres- sion via kernel methods is minimax optimal. arXiv preprint arXiv:2411.19655 2024.
H. N. Mhaskar and Charles Micchelli. Approximation by superposition of sigmoidal and radial basis functions. Advances in Applied Mathematics, 3(3):350-373, 1992.
Wang Miao, Lan Liu, Eric Tchetgen Tchetgen, and Zhi Geng. Identification, doubly robust estimation, and semiparametric efficiency theory of nonignorable missing data with a shadow variable. arXiv preprint arXiv: (50.0005000005555 2015.
Mattes Mollenhauer and Péter Koltai. Nonparametric approximation of conditional expectation operators. arXiv preprint UrXiv:2012.12911 2020.
Krikamol Muandet, Arash Mchrjou, Si Kai Lee, and Anant Raj. Dual instrumental variable regression. In Advances in Neural Information Processing Systems, 2020.
Thamban Nair, Sergei V Pereverzev, and UIrich Tautenhahn. Regularization Hilbert scales under general smoothing conditions. Inverse Problems, (6):1185,, 2005.
M. Z. Nashed and Grace Wahba. Generalized inverses in reproducing kernel spaces: an approach to regularization of linear operator equations. SIAM Journal on Mathematical Analysis, 5(6))974-993 1974.
Whitney K. Newey and James L. Powell. Instrumental variable estimation of nonparametric models. Econometrica, 5165511555,1578, 2003.
Junhyung Park and Krikamol Muandet. measure-theoretic approach to kernel conditional mean embeddings. In Advances in Neural Information Processing Systems, 2020.
Ieva Petrulionyte, Julien Mairal, and Michael Arbel. Functional bilevel optimization for machine learning. arXiv preprint vrXiv:24033202333 2024.
Tongzheng Ren, Haotian Sun, Antoine Moulin, Arthur Gretton, and Bo Dai. Spectral representation for causal estimation with hidden confounders. arXiv preprint arXiv:2407.1044 2024.
Mihaela Rosca, Theophane Weber, Arthur Gretton, and Shakir Mohamed. case for new neural network smoothness constraints. Proceedings of Machine Learning Research, 137:21 Dec 2020.
H Schmeisser. An unconditional basis in periodic spaces with dominating mixed smoothness properties. Analysis Mathematica, 13(2):153-168, 1987.
Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activa- tion function. The Annals of Statistics, 48(4), 2020.
Rahul Singh, Maneesh Sahani, and Arthur Gretton. Kernel instrumental variable regression. In Advances in Neural Information Processing Systems, 2019.
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel Rodrigues. Robust large margin deep neural networks. Trans. Sig. Proc., 65(16):425--4280 Aug 2017.
Le Song, Jonathan Huang, Alex Smola, and Kenji Fukumizu. Hilbert space embeddings condi- tional distributions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 961-968, 2009.
Taiji Suzuki. Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse dimensionality. In International Conference on Learning Representations, 2019.
13

---

Published as a conference paper at ICLR 2025
Taiji Suzuki and Atsushi Nitanda. Deep learning is adaptive intrinsic dimensionality of model smoothness in anisotropic Besov space. In Advances in Neural Information Processing Systems, 2021.
Hans Triebel. Theory of function spaces. Monographs in mathematics. Birkhäuser Verlag, 1983.
Masatoshi Uehara, Masaaki Imaizumi, Nan Jiang, Nathan Kallus, Wen Sun, and Tengyang Xie, Finite sample analysis of minimax offline reinforcement learning: completeness, fast rates and first-order efficiency. arXiv preprint arXiv:2102.02981, 2021.
Jan Vybíral. Widths of embeddings in function spaces. Journal of Complexity, 24(4):545-570 2008.
Martin J. Wainwright. High-dimensional statistics: a non-asymptotic viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.
Sheng Wang. Jun Shao, and Jae Kwang Kim. An instrumental variable approach for identification and estimation with nonignorable nonresponse. Statistica Sinica, pp. 1097-1116, 2014.
Ziyu Wang. Yucen Luo, Yueru Li, Jun Zhu, and Bernhard Schölkopf. Spectral representation learning for conditional moment models. arXiv preprint arXiv:2210. 16525, 2022.
L. Wasserman. All of Nonparametric Statistics. Springer, 2006.
Liyuan Xu, Yutian Chen, Siddarth Srinivasan, Nando de Freitas, Arnaud Doucet, and Arthur Gretton. Learning deep features in instrumental variable regression. In International Conference Learning Representations, 2021.
Yuhong Yang and Andrew Barron. Informatiinn-theortiee determination of minimax rates of conver- gence. The Annals of Statistics, 2765):1564-159, 1999.
regression via kemel maximum moment loss. Journal Causal Inference, 11(1):1 1-12, 2023. Rui Zhang, Masaaki Imaizumi, Bernhard Schölkopf, and Krikamol Muandet. Instrumental variable
Shuanglin Zhang, Man-Y Yu Wong, and Zhongguo Zheng, Wavelet threshold estimation of a regression function with random design. Journal of Multivariate Analysis, 80(2):256-284, 2002.
14

---

Published as a conference paper at ICLR 2025
TABLE OF CONTENTS
1
Introduction
2
Background and DFIV Algorithm
2.1
Instrumental Variable Regression
2.2
Two-Stage Least Squares Regression
2.3 Deep Feature Instrumental Variable Regression
3
Theoretical Analysis of DFIV
3.1 Link and Smoothness Conditions
3.2 Projected Upper and Lower Bounds
3.3 Full Upper and Lower Bounds
4
Separation with Fixed-feature IV
Conclusion
10
Comparison with Existing Works
17
A.1 Definition of Besov Space
17
B Proof Sketch of DFIV Upper Bound
18
Deep Sigmoid Neural Networks
19
C.1 Preliminaries
19
C.2 Smooth Approximation of B-Splines
21
C,3 Error Rates Besov Space
24
D Proof of Theorem 3.1
27
D.I Reduction of Stage 2 Error
27
D.2 Constructing the Dynamic Cover
29
D.3 Reduction of Supremal Stage Error
29
D.4 Bounding Subgaussian Complexities (38), (39)
31
D.5 Bounding Supremal Deviations (37), (40)
32
D.6 Putting Things Together..
33
D.7 Error Rates for DNN Classes
35
Proof of Theorem 3.5
36
E.1 Adding Smoothness Regularization
36
E,2 Obtaining Non-projected Rates
38
Proofs of Minimax Lower Bounds
38
F.1 Proof of Propositions 3.3, 3.7
38
F2 Proof of Lemma 3.2
40
F3 Proof of Theorem 4.1
F.4 Separation in Projected Rates
43
15

---

Published as a conference paper at ICLR 2025
TABLE OF SYMBOLS
Table List of symbols, given roughly in order appearance the main text. Some important symbols from the appendix also included, while auxiliary symbols omitted.
Symbol
Description
X, YY Z
endogeneous variable, outcome, instrumental variable confounder, NPIR model error
fatr
structure function
fat
DFIV estimator
C
upper bound for fatr
duy d2 Pz
dimension X E X, / E marginal distribution of X, Z NPIV projection operator regularity constants of //
0D, 0n
B (x)
Besov space on X' with smoothness parameters 8,
(*))
Hölder space of smoothness 8 Sobolev space of order r in Io-norm unit ball of (quasi-)Banaae space cardinal B-spline of order
(F(
U()
whe
B-spline basis element with resolution k; location parameters of 2SLS algorithm
u, V
08., Vo.
Stage 1,2 DNNs with parameters in O,, Ox
Fz,
Stage 1,2 DNN function classes
D2
Stage 1,2 datasets
m,n
number of Stage 1, 2 samples (2,, 2;), (Ui. 2,)
Exizl-
Stage estimator
R
Stage regularizer with regularization strength smooth activation function
L. W, M. C,C
parameters of DNN class
thresholds of clipping operation exponents in link and reverse link condition
70,711
P.
linear span of resolution & B-splines
II
projection to P.*
A
degree of separation linear IV estimator
fL
No
6z, 8- -coerring numbers of F... F, in -ormmo covers corresponding to Nz, dynamic extended cover population extended cover smooth DNN approximation of wh,0 coefficient sequence of B-spline expansion sequence quasi-norm with smoothness parameters s, N-basis truncation of fetr smooth DNN approximation of fN Stage population error function Stage empirical error function linear minimax optimal rate
Oz.k. Oni
3x
Illlop..
IN
IN
RL
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A COMPARISON WITH EXISTING WORKS
Error analysis of NPIV estimators. Many existing analyses of NPIV study linear (kernel or sieve) methods, where the estimator is constructed by regressing against a known basis expansion in an RKHS or Sobolev ball; some are further shown attain the minimax optimal rate (Ai & Chen, 2003; Newey & Powell, 2003; Blundell et al., 2007; Chen & Reiss, 2011; I; Horowitz, 201 I; Chen & Christensen, 2018: Singh et al., 2019; Meunier et al., 2024). It is also possible to relax the restrictions on smoothness via e.g. spectral or Tikhonov regularization (Hall & Horowitz, 2005; Carrasco et al., 2007: Darolles et al., 2011) 1). However, these methods do not readily extend to DNN classes, which are highly nonlinear in their parameters and misspecified for both stages.
Theoretical guarantees for adaptive neural network- based IV estimators are at present very limited. Xu et al. (2021) give an initial analysis of DFIV with no specific learning rate, under strong assumptions on and identifiability. Hartford et al. (2017); Li et al. (2024b) study a two-stage NPIV estimator with neural networks, however there are two key differences. First, from the algorithmic side, the two works both employ a DNN to learn the conditional density f fxiz (-) in Stage 1. However, our Stage only needs to estimate the conditional mean of the relevant features for Stage 2, which can be much easier depending on the Stage network. Second, Li et al. (2024b) only derive an upper bound in terms of abstract function class complexity measures. It is not clear whether the derived rate is minimax optimal since neither a concrete evaluation for DNNs nor matching lower bound is provided.
Estimation ability of DNNs. For ordinary least-squares regression, it is known that DNNs achieve the minimax optimal learning rate for various target classes and furthermore outperform fixed- feature estimators when the target function possesses low homogeneity (Suzuki, 2019) or directional smoothness (Suzuki & Nitanda, 2021). These works build on classical separation results between adaptive and fixed-feature estimators (Donoho & Johnstone, 1998; Zhang et al., 2002; Düng, 2011 I). Our results in Section 4 extend this line of work to two-stage regression. A significant challenge of 2SLS with DNN is how can we control the smoothness of stage 2 DNN, it is the target of stage I regression. This is achieved by using DNNs with smooth (e.g. sigmoid) activations, for which we extend existing approximation results (Bauer& & Kohler, 2019; Langer, 2021; De Ryck et al., 2021) to obtain a new approximation guarantee for Besov functions which is both rate optimal in T,2 norm (see Suzuki, 2019) and also converge in Besov norm.
A.1 DEFINITION OF BESOV SPACE
Definition A.1. Let 8 > 0 and 0 < p.g S co. For I E TP(X') and r E N,r MVr 8 - 1/p, the rth modulus of smoothness of f is given as
wr.p(f. t) == Miinn s I (p)pp
())(())= = (z, Ej=o (---1)- f(x + jh).
Then the Besov space with parameters 8,P, q.p.qk is defined as the following subspace of TP((X) (g, = (f e (2() IIfflll;, := []ffl + -f < <x),
where the Besov seminorm is detined as Ifl IflB,., = (Jo0 t a- (f. 1/9 if oo and suP ",, (f, (f,t) if oc.
The unit ball of (B, p.o (2"), Il I1..1ag.)) is denoted as U(B p.g (2)). Moreover, Besov spaces on 2 are defined in the same manner. We have the following classical results (Triebel, 1983):
For > 0, s & N, the Hölder space C*(x) 00 (X'). For m N, C" (X)) + Bm (X).
For m E N, the Sobolev space W33" (X) = BT and B (X) + WT 4 Bmo (2))
If > dz/p then B p + (X))
Ifs > d1/ppp 100-1117 1/r)+ then B (2) < T.T(X))
(x) is a Banach space unless p q 1, when it becomes a quasi-Banade space.
17

---

Published as a conference paper at ICLR 2025
B PROOF SKETCH OF DFIV UPPER BOUND
In this section, we give a brief outline of our proof of the upper bound for DFIV Theorem 3.1 which is one of our key contributions. The full proof is presented throughout Appendices C. D and the non-projected MSE upper bounds (Theorem 3.5 and Corollary 4.2) are derived using this result in Appendix E.
First note that in the population limit of (7), Stage is essentially minimizing the loss
Σx,z [| (X) Φo_ (2)1l Æx,z [llve, (X) -TVe, (2)ll²) + 1| Φb_ (1))
and so factoring out the projection error, we may view ||T Exxiz/vo.oo (11(((2))))))) as the 'effective' Stage estimation error.
Denote the dz, -overing numbers of the hypothesis classes Fa, as Nº = N (|llla=(a (2)) 6x) and N2 - (F (1-1lz= (2), 62), respectively. We begin by showing the following oracle inequality from the definition of 0, as the empirical risk minimizer of (8) conditioned on DT,
ED [|T - Ex z 1 2 (p)) S inf ||T Ex1z Ex|z|Vo 11,20,,,1/?<pp)) + log + 82, 8.E9.
where the second term can be further bounded by
2 inf || Twe (122(())) + sup |Tve. - Exxiz(x,llllll 0.60,
0,E0,
With some manipulation, the projected error can be bounded as
||T - T (=rll(2((22)) S [[T Ex|z 11l [[ + 2/ 2/| [va_] S (2=(()))))))
log 8
S inf |T - Two To 1122((2))) + o, 0,50,
|Tvo. - Exiz
Thus becomes necessary to bound the expected supremum of the effective Stage estimation error, Sup//Ten Ex z [vo,]] (P(22) over the hypothesis space, which is highly nontrivial. We aim to achieve this by controlling both the supremum the corresponding empirical process
Σ (Tee, - Exiz(vo_]xx.) sup e_
m
1=1
and the supremum of their difference. Naively these quantities are evaluated by reducing to finite cover (1a,,)s of F.. This does work for DFIV, however, since the Stage estimator Exzz can be ill-behaved in general; approximating the by vhe., does not guarantee that Ex|z ( is good approximation of [v Instead, construct an extended dynamic or data-dependent cover č, which maps to a subset of the product cover for Fa x Fe, that approximates all possible combinations of dhe and Exiz[vb.]] A similar construction C* is given for the population version of Stage risk minimization. By converting to the supremum over Č, and applying classical concentration bounds, we obtain the following general bound:
S inf [['fsar Tvoo (1TA((2)) + log 0,E0,
EDD,DA [1Tfarr (12
+ 6,
m
log log.N + 6z,
+ sup inf IiTb,, de 112((2))) 0.E0, 0.E0,
in terms of the Stage 2 and Stage 1 (supremal) approximation errors and covering entropies. In particular, the Stage approximation error for each 6₂ O₂ is determined by the smoothness of Ten.. which can be bounded by Assumption 4* Assumption 4 combined with domain restriction (10). instead regularization is used, we can repeat the argument including the regularization term to also obtain smoothness bound.
remains to choose the appropriate DNN classes and explicitly evaluate the terms above. The approximation rates for DNNs equal the optimal nonlinear rate; this is known for ReLU DNNs (Suzuki, 2019), but we prove this result for sigmoid DNNs together with a stronger Besov norm convergence guarantee in Theorem C.8, which allows us make the above smoothness argument rigorous. Taking the DNN class variables to balance the approximation and entropy terms, we conclude the projected upper bound (12).
n
18

---

Published as a conference paper at ICLR 2025
C DEEP SIGMOID NEURAL NETWORKS
In this section we prove key results on the approximation of Besov functions by deep sigmoid neural networks. The results are obtained by carefully applying the Sobolev norm bounds developed in De Ryck et al. (2021) to the B-spline system. The main difference that instead of relying on local polynomial approximation, we exploit the fact that B-splines are piecewise polynomials to give sparse construction similar to Suzuki (2019). This allows us to achieve rate optimality in the sense of best approximation width (Theorem C.8) well small covering number (Lemma C.9). Moreover, the proof will serve to demonstrate how to extend the construction to any C" activation with a decaying part (e.g. either of lim (x) exists) by manipulating Taylor expansions to approximate polynomials as in Mhaskar & Micchelli (1992); De Ryck et al. (2021); Langer (2021), or by constructing piecewise polynomials exactly in the case of ReQU.
C.I PRELIMINARIES
We first present some preliminary results. Denote the dimension of the domain d = de for brevity. easy to see that the cardinal B-spline 6 is a piecewise polynomial of degree supported on [0, r + 1] and 4 W (R), im S [0, 1]. We make use of the following expansion (Mhaskar & Micchelli, 1992):
1+1
1+1
(-1) (r (1++)---- (17) j
(x) - a 5)4
Σ (2)
Σ [ay(x-11 i)' where aj aj= j=0
1=0
Note that the indicator (x) can be well approximated by the shallow sigmoid network (p(z)= (Br) - oBB(r (09(x-1)
for some B 1, whose translates form smooth partition of unity over R. Lemma C.1 (smooth decay of åg). It holds that:
(I) [ + 1) + 6p(x) + óg( 1) 11ll: (1o.1)) Se-B.
(2) For all 0 < kr with b E 2 we have [1ōB(x k)ll & Bre-B. Here, constants depending only on are hidden.
Proof. (1) The rth order derivative satisfies Lemma A.4), so that
Sr ^e=) (De Ryck et al., 2021,
[1 + 1) + (b(c) fz( - 1) 11|
= |la(B(s + 1)) - a B(x - 2)) ([21))
< (-x 1)) ((01)) + lla (B(x 2))llws
2
-B <B + 21 h
+ e
(2) For any pair k, with k- - 12 2 it holds that
[ k)ll 1 (lo(Bz) o (B(( 1))llw (1- ((-
< a (B(( - k+ 1)) + B² (1( (xx) - -
1
< It B + +1 Bº - $ B
The same bound holds when k- l 2 by symmetry.
We also require a finer control over the boundary decay of 6g:
Lemma C.2. It holds for sufficiently large B that [ιō + 1)x Il=" S B-22
Proof. We first note for all j 1,..... that
| Bu (0x1))
|| L0'1l~~(0.1// lle B2 1 B-12 -VB B-112
19

---

Published as a conference paper at ICLR 2025
for large B. We have that
- (1 - a + 1w) (10.1) + [I(1 (="))))("ll
1o(--B(g 1))x² (Iw] (10.1)) + ||a( Bxr ([..1)
Then by the general Leibniz rule and the bound for |زاا above, the first term is bounded as
( + 1))=" ll (l0.1))
r!Bi
$ x'c B(++1)
+
+
(- (-B(x : +1))z² ((.1))
1000 (0.1)
j!
5=0
r!Bi
S <e-Ba B +
x e B(x+1)
S Se-B.
j!
((0(11)))
and the second term is bounded as
||a(- Br) (l5(11,1))
$ | - Bx ((o.)) + r! (rlllo(BBx))lll Bx))l2(((o1)) + Σ 3=1
() r!B'
- -Br (0011)
j!
$ //2 (1o(( + (B-1/2 1/2 + ... B 2-/
For the remaining term above we further have
1/VB dr
dx
5 1
(o(- (I)
11/DB (I and hence ||o ||o(-Bx)) (0.1)) & B-1/22
(1 (
2VB (1 + eVBB''''
The following lemma will also be used control Sobolev norms.
Lemma C.3 (De Ryck et al. (2021), Lemma A.6 and A.7). Lerd Zzo and Ω C Rdd.
(I) For f.g E W (22 it holds that [[Slllw S 2"1///lww (2) lllllwww
(2) Let d' E N and S CR.. For C'(S,, R), g (2,2))) à holds that
|l o & 16(e²r d' y ll (wr)
max 1,00 d [1iillyg
The next two results provide the basic building blocks of our construction.
Lemma C.4 (De Ryck et al. (2021), Lemma 3.2). Let r E Zzo and M > 0. For every 0, there exists shallow sigmoid network Ç [-M, M] - I with width at most 1x 2 and weights at most r//2 poly(M) such that
[I((x) x" Iww ((.MM) Se.
Lemma C.5 (De Ryck al. (2021), Corollary 3.7). Let N. Zzo and M 0. For every O, there exists sigmoid network za [|, R with depth [logz d), width 3d and weights at most 1/2 poly(M) such that
Ta( ... .Z)) - II 2,
(C Sc.
Here, poly(-) notation ignores multiplicative constants depending only r, d.
20

---

Published as a conference paper at ICLR 2025
C.2 SMOOTH APPROXIMATION OF B B-SPLINES
Recall that the base tensor product B-spline is defined as wn,o - - (x.). This subsection is devoted proving the following smooth approximation result:
Proposition C.6. For all 0 sufficiently small, there exists sigmoid neural network is with depth [loga d + 1, width 3r ld + 4d and weights at most poly(E such that
| wb,llll;(()) + || wo.ollem(B) S.
Proof. Fix C, > O and let Ç be the network given by Lemma C.4 with error c and M replaced by 2r From the construction in De Ryck et al. (2021) we see that [[<] - E-- Also, let T2d be the network given by Lemma C.4 with error e' and M = |K|| 1- bounded as (e')-1/² poly(E-)).
(R)), (R), so that the weights of w2a are
From the decomposition (17), we aim to approximate the multivariate polynomial I[ (x; 54)
on the interval I [kt k; + 1) by the networks
* J (x) (2 ki).... ,8p(aa ka), 5(x1 ii)," ,((ra ja))
and the tensor product B-spline wa,o(a) = -I (x;);) by the network
K kid
a(x)
Σ
a aa (,)x)
Σ
Σ
Oj (22t)()))).
(18)
J.
Note that we have used the symbol above to indicate a subseript to be iterated over dimensions 1, in a consistent manner. By taking C >> the clip operation will not affect the output due to the L bound, so it is ignored.
We proceed evaluate the error ||ا (o by breaking down into several steps.
Multiplicative approximation error.
easily seen that |6 allllww (B) & 2B ||/llo/ (R) and
Isllwa (1-2r-3.r-2) S |1((z) - x' r" llw (1- 2r-3,r+2) E (2r + 3)" +r!
Moreover, |a,l < 2. Thus we can bound the error arising from the multiplication network T2a (here using the stronger W norm) as
a(x)
ΣI
5B(a
(o)(a,-,,(((;; - -ji)
k==0 ja i=
W& ((---22.)+)))
r+1 ho
d
2d
(x)( II
8 (xx k1)5(x - -Jt)
Ne k==0 J.
Wx (11--22,)+))))
1+ he
2d
2d
T2a(x1,"" (22)) - II
h. 0 j. =
1=1
W (1
x max
[11ó(( - ks)llwz (aa(xa;-k)lllaae 11( - je) llll (1-))
i=1,-.,d d
S 2d(r + 2)2d max (2 (2r+3) + trll'd
$ <B²
(19)
by applying Lemma C.3.
Monomial approximation error. Next, it holds for all 0 $ ... .id $ + that
d
II
((:; -i) -
(a; i 1=1
21

---

Published as a conference paper at ICLR 2025
8-1
d 1.) II S(ac - 12)
( j. II
S(x; - j:) - - (at
i=1
x=++1
1=1
x=t
W (1---22--))))
((ri - ja) ((ri j())
(a: 10
II ((xi ji) 6=+1
6=1
W -
d
Σ
|0" -(xx)())
(1-2- liiz (1-21 2011211-------
|| |ISllliw (1-2 S
t=1
and hence the error due to approximating monomials by Ç is bounded as
d
Σ ΣEI
aj 5 - ki)
II c(2, - j:) - II
(a; - j.)
=0
1=1
i=1
Wx (1--222))))))
R+
d II
d II (a. - 1.)
16 10lli
c(2; - ji)
0==0
Wx. W1----r2.
< B e.
(20)
Indicator approximation error. For the approximation error of the indicators, one needs to be more careful since 10, 6g is nonsmooth near boundary points. Nonetheless, the difference of the piecewise polynomials (17) at each knot k = 0, ... r + 1 is always of the form (x k)" which will smooth out the error terms. Restricting to each unit interval Je IT. (4, es + 1) inside the box [-r- 2,r r + 2]0, that is for -1 &,, la + 1, gives that
4
Σ EI aj 85(a, kx) 1i)
II
ag (2+
ji)"
N.
i=
==vi==
(W((e)
4.
a 85(1, k2) -i)"
Σ
II
aj (x, ji)"
(21)
ke
3.=0i=1
+
Σ
0 kk)(x; ja)
(22)
WW((e))
Here, have split the sum ky, ha into terms such that [k; Egl 1 for all and the remaining terms which contain at least one 'out-of-hounds' index ky with k&g- - Fpl 2 2. Also note that we are now using the W norm on Jr order to control the boundary remainder terms, although we will again upper bound by the WT. norm when necessary.
To bound (21), we rearrange the existing terms that the sum of three neighboring translates of 8 suffices to smoothly approximate the true indicator up to boundary terms. Indeed,
d ag d k.)(x; 5.)
Σ
EI
Il
Not ke 11
Σ
Σ 0z - k.) ja)"
im kelki 1
ag (6n(1; + 6n(a (,) + Ōn((= - b, -1))({((; - ja)
1= =
- - l, + +1((2 4 + a 1---1-11110
1)(x; - E4 - 13-)
22

---

d Published as a conference paper at ICLR 2025
Expanding the above product and separating all boundary terms gives
d II 0==
Σ
og ( k2) 11)
Σ
as (2; 12)
ko
4.
s jajea
==1
(ws(ee)
4 d
d aj (2, - j.
Σ
[1 - + 1) - (z) + ddb(z - -4 - 1)) -1) -1))
0=0+m
E
II (ae++18((2; l 1) 4, - 1)" ae, 0 la 1)(x; 4))
SC (1,.. .d}
W5(3))
x
aj (ōn( - l, + (.) + 5n(es - li - 1)) - 1i)"
iES
WZ(3)
(88(2 - l: + 1) 85(1; (,) 58(2; - -4
WW (Jc)
18 4, + 1)(x, e,) [["""ll(((]))) 116 - l, 1)(xx; - le - 1-)" 11""ll 1)"Ilwg((,.e
[1 - 4, + 5p(0; (,) + 6B(2;; 4 1))
x [1ō Eg + 1) + 65(1, - (j) + dz( - bj 1) (1(,,(())
Σ [1 4 + 1)(x; - Es) (ii" (la)""llwssaa Iw + [1, - l - 1)(xx - li 1--1)"ll 1)
B B-
(23)
To isolate the boundary terms, we have used that lIflllyg $ IIf (wraaa/llll (2) which is easily checked from the general Leibniz rule. The remaining terms can be bounded from above straightforward manner. Applying Lemma C.1(1) and Lemma C.2 gives the bound (23).
Finally, we use Lemma C.1 to bound (22) as
Σ
Σ
II
ag dn(;;
kz) j4)
4,
1222 Die1
(z.(Je)
r41
2d
11 (2)
ll (1 k.
sup
18 k)llwz
k==0 I Brd
(24)
Putting things together. Combining the bounds (19), (20), (23) and (24), we have shown that the construction (18) approximates Wo,0 on the box [-r- - 2, r + 2] as
|| Wo.o wg(l
booollly((--- r- 2
(x)
Σ ag (I: jir
===0i=1
Wx (1-r-2,
h. d
<
((x)
ΣI
aj 6n(z, k:)(((s -j+)
==
W (1
23

---

Published as a conference paper at ICLR 2025
Σ 0=
EI
uy 6n( - ki)
[I -j( -II (xt - 11)")
W WG((-1-2,+t11)
Σ
aj 6, - kz) 1.)"
Σ
aa (2.s _i)"
#==0+ -i
022m
WF(())
S SB B"+
B
It remains to bound the approximation error outside -- We may decompose the domain into a union sets of the form (IR M-r - 2, + 2)) x R2-1 and use Lemma C.I to bound the decay of the corresponding on component, and argument similar to (19) yields
|
wa.oll wl (R" (1---,
++
ho
a))=)((z)
h.
Wa ( (1-r-2
r+ 8
(x)()))
II 65(2; - -ka]((0s - -c))
R. 0 je
wx (B4)
5 (or: ki)((re -s)
N. =0 Je =0
1=1
Wx (Rd (---22++2)))
k.
2d
(2d(21 .. (2a)
max
{10 (wgg
0 j =0
1=1
(I-
d-1
==
S SB" [Kllw: B'e
Be "
Moreover from the weight bound in Lemma C.4 that [[]ll (c-r/2)2r= =c since has depth 2. Adding the resulting bounds finally yields
| wo,ollw; E (B* Ve + Brd B
Brddd
Hence for any " > 0 small enough, we can ensure the above error is bounded as O(e") by taking In addition, we can check that the resulting
B x (e")-2,e x (e")ard+1 and e' = (e")2r'd4 network has depth [log2 dl + 1, width är Id 4d and weights most polynomial B, c, c', thus polynomial in c".
For the _ approximation error, one can go over the proof and check that the same bounds apply. discarding the bounds for all higher-order derivatives. Finally, the results hold even if 0 < < by replacing applications of the triangle inequality by the quasi-norm inequality.
C.3 ERROR RATES IN BESOV SPACE
For sequence of coefficients (Bx.))200.E1 define the quasi-non norm
1/p
4
1/9
|1
Σ
(k/))d//) d/r)
Σ LET
1 P
with the appropriate modifications when p or 9 To approximate arbitrary function in Besov space (x)) we make use of the following adaptive recovery result based on B-spline decomposition.
24

---

Published as a conference paper at ICLR 2025
Lemma C.7 (Dūng (2011; 2013)), Suppose 0 p.q.us and A < < <s<TA(r (1--((--1111 11/p)) (r/pp// where = d(1/p- - 1/u)+. For any f E (2) and sufficiently large N there exists fN which satisfies
|| JN (2)) & /N-*/ (N
and is of the form
K IN Σ ROBE
K"
Th
B +
k= K
i=1
K where (4,) C Ik, K= [A N K* -- N +K+Lm== 2 vík K) X2.V7 with
5-A and X1, X2 chosen independently of N so that Σ + Σk E 7 S N. 1f A = 0 2A then K = K. Moreover, the sequence of coefficients can be chosen to satisfy || 2 |lf (X))
Together with Proposition C.6, this implies the following approximation result. Theorem C.8. Suppose 0 < P.g S co, d/p < s < ( 1/p). For all f E UUB p.g
(2)) with
|S| C and sufficiently large N, there exists a sigmoid neural network IN E FDNN (L, W,S, where
L [log d) +1, Wo = 1 + d, W NW. S= LWG NW: M poly(N)
such that [| - Ill (x) S N-o/d and B, (00) is bounded. If ffd(1/p - 1/2)+ < 8 - d/p. the same result holds with (x( replaced by (x)).
Proof. Consider the N-term approximation fN given by Lemma C.7. For each B-spline wi, in its sum, we construct the network Wr,e by taking the network of Proposition C.6 and scaling its input as 34 + 2 f;; note that the scaling only changes the input weights by a factor of at most 2 < 2k* S (V)y It follows that
| - www.llwwwa.......m S 2 wo,0 wo.oll S 2th and || ( Se.
Now consider the network JN obtained by laying each wx,e in parallel and scaling the output weights by Sk.e, so that (ignoring the clip)
K
m
JN
Σ
Bu Wr.
k=0
k=K
Then we have
1I fNllwy fxllw3(0) + 11 - (x)l
K
3x. -oskee-ww.ellypgg + ||ãx, - wx.elln=<3))
k= BEJa
n
3k.e: (ler. - wa.elllwsa + Wir. - wre (k.e)llm(()))
Am K
K
Σ
12
1
K
1/p
[T&l [Tx|2- 111/tee
k=D
|| bp.a
9/ld/p-))gh((1-1////
k=
2K c
25

---

Published as a conference paper at ICLR 2025
Therefore by taking (and hence all weights) polynomial in N, ensure
1 [wllwp(x)> + II (N)) S N
and thus II - (x) X N /d as desired. partieular, for sufficiently large N follows that Ilfx (x)) - C' C'+N= N s/d N < C so that including the subsequent clip operation does not affect fN- Moreover, since the B-spline expansion of f fN has coefficient zero for all wk, with resolution k < K, it follows that
1/p 9 1/9
00
1)
IJ JN JN (x)
2h(x-A/p)
Σ EELK
13eel""
+0
as N co, k= [A log N1 + 00. Hence
|| (fNwlll,,,cx) - IIffllay, (x)) S Ilff fN fwllwz
IlfN fllay,, (20) 0
and Il || (2*) is uniformly bounded.
The d-covering number MMS, 6) of metric space (S, p) is defined as the minimal number balls with radius å needed to cover S. The covering number of FDNN W,S S, M) in L -nrm can be bounded similarly to the ReLU DNN class (Suzuki, 2019, Lemma 3) with a slightly better bound. Lemma C.9 (covering number of sigmoid DNN class), If the norm bound M 2 4, it holds that log Nog.((FDNWe W. M), ||l s (T 3)8 log MW S log 6-1
Proof. Consider f, E DNN S, M) given as
f = clipao (W o + ((0)
(WO)d + +B01),
f clipgee (W σ + B(4))
(W + 8(1),
such that |W( W oc [ 8(0) 20 S d. Also denote AL+1(f) = Bi( = id and
Ae(f) = clipo,0 o (W B(4)
(WlO)id + b((),,
Be(f) σ (W((-1) ((e-1) o (w b(2).
so that - (f) (w ((09) o B.( Then Ac(S) is
with
respect to the L -nrm and [B&(())lz= 1. It follows that
| (1)2)(x))
(f) (W(Odd +b0)))(()))))))) B(ff) Aet1(f) (w 119) Be(t)
(20(())
(MW)
MMW L
(W + 1)8 s 8(W + +1) 4h-r
8 5::6'
assuming M 2 4. Thus for fixed sparsity pattern the 8' -'oovini number is bounded by dividing the range M, .] of all 5 nonzero parameters into intervals of length and counting all possible combinations,
2M
S
16M(W +1)
MW
1
S
8'
Moreover the number possible sparsity patterns is bounded as (-(w2-- Noting that 4(W + 1)2 < 16W2 < M2W and 4L < 4-, we conclude:
s 2 + W))s.
S
MW 1 s
N(FDNN (L, W. M), 11)1)
16LMW(WW + +1)²
ALM3W3
MW
< (
4
desired.
26

---

Published as a conference paper at ICLR 2025
D PROOF OF THEOREM 3.1
We first present the proof of the general upper bound for estimation risk, which contains our main techniques, over Sections D.I-D.6. We then specialize the Besov setting in Section D.7 by applying the smooth DNN analysis from Section C.3, which proves the rate under Assumption 4* or Assumption 4 with domain restriction. Finally, we show how the proof can be modified to incorporate regularization under Assumption 4 in Section E. as part of the derivation of the non-projected rates.
D.1 REDUCTION OF STAGE 2 ERROR
We begin with the following oracle inequality, which is essentially a consequence of e.g. Lemma 4 of Schmidt-Hieber (2020). This starting point is necessary to utilize the definition of 6₂ as the empirical risk minimizer of Stage 2.
Lemma D.1 (oracle incquality for NPIR). Denote the -overiing number of the Stage 2 DNN class Fs as N₂ == ( s. (Z)) 82). Then there exists a constant C₁ depending only on C such that for all 82 > with log N₂ > 1 holds conditional on D1,
ED, [atr Æx1z W1 11 [12(()))
(log.V n
S inf |T E 1x1z + + 8,60,
+ 6z
Note that even though the risk is being minimized with respect to the Stage parameter Oz, the generalization gap depends on the covering number of the Stage DNN class which contains the actual regression model I8 [x [ for 7 fatr. Also, the multiplicative factor 4 can be replaced by any constant larger than 1.
Proof. We may repeat the proof of Theorem 2.6 of Hayakawa & Suzuki (2020) while replacing the regression model = (X) + E, where was assumed to be i.i.d. Gaussian noise, with the NPIR model Y = farr () + (4) and the class of estimators by F=.. Here, we only provide the necessary modifications. For the loss
(I.).) far Exiz
and the corresponding empirical quantity*
TT(())) ED,
(far(()) - Exiz|
it can be shown in the same way that
(logN2 +
Ti((,)) m( s (1())) Cf
(25)
n
Moreover, using that [B[n] 0 and E[noo, (2)) Blde. (Z)1 0 for all de_ F.,, can be shown that
48
= f Exiz|vo,,lliee
E
mm
E
max
2)
+405
n
n
ISN
for all 0₂ E Oz. where oo.., for j s No is a 8, covering of F= and
En (on, (Z,) - T fair (5:)) Ej EE (0o.,, (2,) T Str (atr(())))) (2
'Here we are abusing notation allow for both a fixed E 22 and also an estimator 0_ which map from the data to θ For the former, it is clear that IT(,,) IT(02) always. We are interested in bounding the gap (25) for the latter.
27

---

Published as a conference paper at ICLR 2025
Here, since 1 2, is di -uubgauussian, have E||n Iz = 2,] Furthermore, each E is also d1 Di-subgaussian conditioned on the data Æ1. ... 2n as an L -poojeccioo of (m :7)), and hence
V2Toi and E[[na]] V2T01
40
E
ISN:
max +
S
max ISN
exp
407
E
exp
407
N2
(1ey) 2 201 v log u
P
du
==1
2 du 2/2N2.
N
This shows that E[max, eg] s 4af log 2v which combined with (25) concludes the desired statement.
L
Now let O2 0f denote minimizer of fatr Twe, (1) It holds that
inf 0260,
|st - Ex12(vo.]1(:=22)) S [ - Ex12(100 11
$ T - Thh; 11:=((2)+ 2 2|Tve; Ex]z|Vo:11Ba(p
Then the projected error can be bounded using Lemma D.1 as
ED, [ITTSar -Tstt
$ 2EDD, |IT Bxizlva, 112 2ED, [|Lx z|48,1 faull
< 16 - -Trbo [(2((2)) + +16||7 - Exiz 101z(too111122000 [1(2((z))
20r (logNz= +o= +2Ed, |1 zlvia,1 -Tvaa l1(=((2))
(og./2 6z
$ 16 inf 0.
|T Tole (((()e) 20g
(26)
18 sup |Tve. - Bxiz(Va. 0,
(27)
where the projected Stage 2 approximation error and covering number (26) will be explicitly evaluated later.
thus becomes necessary to uniformly control the expected supremum of the Stage 1 estimation enror over the hypothesis space F. which is highly nontrivial. This is achieved by controlling both the supremum of the corresponding empirical process:
(Tee, (z.) - Bx12)
sup ee_
(28)
1=1
and the supremum of their difference:
sup 9, e
1 m
(we_ (2i) 1(1))
i - EEizldee.llarppp
(29)
by carefully reducing to well-chosen dynamic cover of the hypothesis spaces F,, We detail this approach the following subsections.
28

---

Published as a conference paper at ICLR 2025
D.2 CONSTRUCTING THE DYNAMIC COVER
We first introduce the essential tools for our proof technique. Fix 8z 8z > O. Let the functions tv..., for I,... .N M(F,,, 2, (1)))== form a dy-cover of Fa and let de.,. for k
1,... .N. := (1)) 6.) be d--cover of F.. Naively one would attempt to bound the desired supremum over Fz, say
sup |T - Exiz 0,
with the supremum over the cover
sup [Tvo.. Exi2 JSN.
However this is not directly feasible since the Stage estimation operator Exjz can be ill-behaved; approximating the by the element the.,, satisfying I|ve, te..,, , || (X) S 6n does not guarantee that [x is good approximation of [xxzz(ve,,) even in (.2(P)) construct an extended cover é that approximates all possible combinations of the. and Exjz [Ve_. We also give similar construction for the population conditional mean approximation.
Instead, we must
Definition D.2 (dynamic extended cover). The pair of elements (Ne.. )0 is said to be joint empirical approximator tho, Fr
llvo- the.,, 11 (x) S 6 and [[x - 11. S Soz
(30)
are both satisfied, which always exists for each Oz. Similarly, (Vo.., : v=..)) is said to be joint population approximator of the if for the L'-minimizer 0 arg ming,ee. (Tve de, (1 corresponding to oz.
1ev. (1==, S 62 and |100: 6o (2) S 85
(31)
both satisfied. Moreover, the subsets C, c (1,-.. (Na) f1, .Na) are defined
Ć = (() (No.., (o.,)) is a joint empirical approximator tho for some 0₂ E { cº = (We..: ) is a joint population approximator of the for some 02 E O²)
Note that è a random subset dependent on the data D, and hence one must be careful when proving uniform bounds using Ć. In contrast, C* depends only on the operator T' and predetermined covers oz 2z.k. Also note that 1, < N₂ x N₂ by definition.
D.3 REDUCTION OF SUPREMAL STAGE 1 ERROR
We now demonstrate how to appropriately reduce the supremal empirical error (28) over the extended covers č, C defined above. For each 0g E Oz. by the definition of Ex [x/_] as the empirical risk minimizer for the Stage loss, holds for 0: = arg ming,ee, ee,
|| th. - 0e (1(((2)) that
m
doo (zx) - Ex zlv,, (2x) 2 <
(Vo, (z,) - 60: (2+))
i=1
By adding and subtracting the true conditional means Tyho, (2:) from both sides and rearranging, we obtain
Tooo (2:)- - -Bx12(00=1(3s))
m
Σ (Tuho, (2i) - -00;(2)))2 Σ
(Bxiz(vo.1(3)) - dos (2:)) (wo. (x:) - -Too, (2,))
1=1
1=1
inf 0,
[T 00.1172(P2))
(32)
29

---

Published as a conference paper at ICLR 2025
(Trbe, (2:) (000(2))) (zi)) 11Tr/o. ~bo; -00;1(22(22)
(33)
m
2
[xizlo0o,1(2z - Two (zi) (wo, (at) (2:))
(34)
2 m
+ (Tehe, (2i) - oog (2i)) (1'0 Twe, (2i))
(35)
Now that the Stage approximation error (32) has been isolated, we may reduce each of (33)-(35) to the supremum over the respective extended covers. For (34), let (woz.,, 002,e) be joint empirical approximator of Vee and define the residuals Eμ == the.,, (x.) - TThen. (2i). By the condition (30) and due the - -
ITTv=_ - Two.., (1)= S l1ve, von., (x) S dz
of T, it follows that
2
[x Exx2lu,,100 (x4) Twe (zi) (ve, (2i)
m
m
(oB (zi) -Twe.s (z:) (Vo_ (I.) Tvb,, (zi)) + 4C(6e 6z)
m
2
m
(dbe (z+) - Teee (zi) Eg + 4C(36, + 62).
1=1
Similarly for (33) and (35), letting (Vo 00 ar) be joint population approximator of tho . it is easily checked that
(Tho (2x) 20: (2:)) 1i - Φo l(2((22)
m
(Wo (z:) - Now (2i) (=))) - -1Tibo.. - 00 N + 4(20 + 1)(6, + 5.)
and
2 m
(2)) )00((2)) (2i)) (x,) Twh, (2)))
m71
2
The (zi) 20, (2i) Exr 4C((3,, 5z)
m
by taking 6z, 82 < 1. Hence we have shown that
1
Thhe (2:) - E Ex12(('e. )
sup 0 m
sup
inf 0.c0,
[Twe_ - de -o. (1:2((p))))
+ (32C + 4)(2 + 6z)
(36)
e.
+ sup (1
(TWon. - 20, (2)))
[Tee.., - -0.
(37)
m
m
sup
(00 (24) - -Twe,, (2+)) Ezit
(38)
(j.))Ed m
2
+ sup (1.))CC"
(Thons (zs) (00.((z)) Eg.i.
(39)
1=1
30

---

Published as a conference paper at ICLR 2025
In addition, repeating the above argument for the supremal difference term (29) yields the following reduction the dynamic cover C,
1
(Wee - -Exi2lvo_(zi)) - |T - _x112
sup 0
1=1
sup (j.)eed
(Teer,, (z),))))) -ITTe,. 00.x1liA((2))
(40)
1=1
+ (8C + 4) + 6z).
We now evaluate the expected value of each of the suprema (37)- by adapting standard complexity- based arguments to exploit the definitions of č, Again, the approximation error (36) will be analyzed later.
D.4 BOUNDING SUBGAUSSIAN COMPLEXITIES (38), (39)
Define the auxiliary random variables
Ej,k = Ex= (0o (2s) Two.., (z.))
N5i$N, I
E (0o (zi) Tepo..s (2)))
where εj = if the denominator zero. Note that (38), (39) are similar to classical complexity measures of function classes, but with the noise terms replaced by the function-dependent residuals En = vez. (x:) - T T' (zi). Nonetheless, conditioned on we have B[G,,/==]] = 0 and since (x:) I C, each (5 is independently C-subgaussian conditioned on (=:) It follows that Ej,A is also C-subgaussian, and repeating the tail bound argument in Lemma D.1 over all pairs (j. 1e) we obtain
N
]
exp 40
sup iSN
E
exp
j
22V2N.N..
4C
j=1
#=1
so that
ED,
sup (G,K)EC
$ EEp
sup iSNa.kSN
$ 402 log 2
Moreover for each (j,k) E é, the pair (Vir )o...) constitutes joint empirical approximator of some Wag E F2 so that
m
(00.2(2)) - Twee. (z)))
[xiz(Vog](2s) Twbey (z:) + (4C - 2)m(6 + +
Therefore, (38) is bounded in expectation
Ep. sup (GAA)E
(0o (zi) - T Two (zi)) Eis
Ep, sup (i.A)ed
Em (00.,, (2i) - T Tibo.,s (2:))
Epi
sup (G.A)EC
+ ED,
sup (iAjec
(Do (z:) - -Tho,, (zi))
2m
8
Tebo, (21) - ExIZ| 2)
log 2V22NN EED,
sup 0_EB, 2m i=1
m
+ (2C 1)(6, + 0z).
31

---

Published as a conference paper at ICLR 2025
In partieular, the second term is simply half of the supremal Stage 1 error to be bounded. Furthermore, almost the same argument for (39) with the cover C" yields the bound
2
ED,
sup (GA)EC
(Teen (2i) P0., (2:)) (0,,((2))) En
m
8C
log 2v 2N + ED,
(Two.., (2)) - 20 (z:))
sup (U,REEC*
2m
where the second term can be bounded by half the sum of (37) and
sup I|T 20.., -10.n(lppe S sup inf [T dv. ((ppp2)) + (4C + 2) + 5z); (j.)jEe"
O, 8.E8.
similarly as before (a tighter bound can be obtained with more analysis since C* is not data-dependent, but this does not affect the result).
D.5 BOUNDING SUPREMAL DEVIATIONS (37), (40)
We derive the bound for (40) first. Define the auxiliary functions
(z,((2) = (Trie.,, (2) 08., (=)) - 1|T ΦB., 1122(pz)
and set K,,a |[T,e., - 2u.,. (I.)(P2) V fig for some Ko > For each (j. kb) C comprising joint empirical approximator for some Whe F... it holds that
< S [[T $0,, 1 ((((())) n
iT Exiz|v0e + (4C 2) 52) nō.
so that
2 S sup IT Exiz/ve. + (4C + () + 5.) Ng. sup K. (G,)EC
(41)
retrieving the supremal Stage error to be bounded. In addition, it holds for all 2 that
(x)
40
Σ E gi.k
)
27
< 4.
Kij.
Kij.k
Then for the sum over (==) by Bernstein's inequality we have the tail bound
(2))
(- ( (m + u//3/ u/3ro)
u²
P
2 u
S 2cxp
Kj.k
for u > Hence the random variable
(zi)
G sup (AR)EC
Koj,
satisfies by a union bound
P(G² 2 u) < 2 exp
(-8C2(m + vu/3mo)
< ć exp
+ +2|C| 1602m
u
(xpp
3 /
16
Therefore for a cutoff un
O. we evaluate
E[G /
P(G² 2 u) du
32

---

Published as a conference paper at ICLR 2025
$ uo +
2 è exp
du +
2 Cl exe
du
1
16C
uo
ug
uo + 32 |C m
exp
16C
Ug
16C Uu
256C*
3Ro 10
+ +4/0|
+
exp
3kD
160
where we have used the fact that the antiderivative of exp( (u) is -2( + 1) exx(((-VV) We now choose Ki, ug such that
3kd 20 log éI
40
log é
uo = 160 log C
Ko
m
16
3
m
which yields
16C
E|G & 160 log C² + 32C +202 m + +4 1
< 1600mmlos čI 10). (42)
log č1
Combining (41) and (42) and substituting in the value for No. it follows that m
ED. sup L(AMEE
m Σ (z)(())
2(0 2m
ED.
G sup
Kjjk
+
ED,
sup (A.k)ed
(i,))E
800
CI+9) + 9)
sup
||Tibo
Æx1z
+ (2C 1)(5, + 82).
9m
O.E0,
Furthermore, a similar argument for (37) with the cover C* gives the bound
m Σ
ED.
sup (ix))c
Min
(2:)
m
i=1
80og(log
C² + 9)
+ sup inf 1|T 60. (()))())) (2C + 1)(6, + 82)-
9m
2 0, e 0.E0.
Remark D.3. The above uniform bounds can also be obtained up to constants via localization and chaining techniques, see c.g. Theorem 14.1 and Corollary 14.3 of Wainwright (2019). For this approach, we control deviations of empirical and population T,2 norms uniformly over the class
=e==T[Fal- F= {To.. - do. (G.A)EC)
One caveat is that the function class is usually required to be star-shaped, that is f E Fe should imply a E F for all 1]. However F is in fact only nearly star-shaped; while the output of a DNN in F Fz can generally be scaled by scaling the parameters the final layer, the clip operation (when activated) prevents this for certain functions. This can be overcome by slightly extending Fz, F2 to incorporate a scalable clipping clipaç.aC.
D.6 PUTTING THINGS TOGETHIER
Plugging in the obtained bounds for (38) and (39), we have that
Tele (2)) ExIz (oo))) 2
ED
sup 0, m
sup 0,
inf 8.60:
+ (36C + 6) + 62)
33

---

Published as a conference paper at ICLR 2025
m
ED,
sup (i.M)Cc
(Teo (2i)- - -00,, (zi))² ||Teoons -0... 112
m
16C
Tibo Ex12lvee 2
log 2v2 + ED,
sup 0.E0.
m
2m
1=1
and so, substituting in the bound for (37) as well,
1
2
Ep,
sup C
(Tho (2+) - Fxx22(00,111 ((i))
1=1
32 1og2V2NNN + (72C + 12)(6z + 62)
I 3 sup
inf 1.E9.
|[Trh, (0.1((
+
0,
m
1 m m
+ 3ED,
sup (i.k)EC
(Tvon, (2i) 20.N )0.n (2i))
(Tvo.., - - (12((P))
9
< sup
inf
||Tw0, th 11 (22)(p2))
0.E0:
1760
+ 3m
(log.Nz + log -9) (78C' (6) + 8z)
since IC"
S N Combining with the bound for (40) yields
ED,
sup
||Ty/o E Ex12(vo_]1l2=ppp
E..
m
< Ep,
sup
T Teee (2.) - -xi2((oo -11Teee. -Exz(vee.12ecpa))
m
i=1
m
Toe, (zi) - Ex(z((o 2
+ Ep,
sup
0. m
i=1
80C
C 11+99 9)
ED,
sup ||Tvo, Exiz|vo,,112:02 + (10C + 4)(6, +z)
9m
1.E0.
m
T Thee (zi) (x12(1oll(Ei) 2
+ Ep,
sup 0, m
Σ
i=1
< sup
inf 0,E0,
[Twe_ de l1(p(p2))
= ED.
sup 0.50=
[Twe. Exiz(Wo.l1(xrpp))
2
e.
68C²
+ Nlog.22 + log N₂ + 9) + (88Õ + (8, + 82),
and thus we obtain the upper bound for the Stage supremal error as
ED,
sup 1.
|Twee - -Exz(Ve.]lieeppa))
(logNN + log Ns + 6, + 6.).
< 9 sup e,
inf |Tvo, - 60. 1011122 4.E0.
+ C2
(43)
for some constant C2 depending only on Ć.
Finally combining this with (26) and (27), conclude that the projected L error is bounded Ep,,o, [ITTfar T (2
S inf [[T - Tele, 12 + sup 0.e0,
inf |Tye,_ - 0o. 1122((2)) 0.e0,
+ No log
m A + 8x + 5
(44)
34

---

Published as a conference paper at ICLR 2025
D.7 ERROR RATES FOR DNN CLASSES
Thus far, we have shown that the error can be bounded in terms of the (projected) Stage 2 approx- imation error, supremal Stage approximation error, and covering numbers for the Stage and estimator function classes. We now evaluate each of these quantities for the introduced DNN classes to prove the final result. Concretely, for sufficiently large integers N₂,, N2 we set
x = FDNN ([log2 dz + 1, (NN,, O(Na),
F3 = FDNN (logg d + 1, ((N,), O(N2), poly(Na)),
as specified in Theorem C.8. As technical note, we must also set the lower clip cutoff C. of of to be greater than the higher cutoff C of Fg. This is to ensure that the target of Stage 1, the conditional mean T The which has sup norm bounded by C, contained in the identity region of the clip for F, and thus can be properly learned.
First, the projected Stage approximation error (26) can be evaluated as follows. Let the Neer B-spline decomposition of fatr according to Lemma C.7 be
K
K' Σ
JN.
Wuke +
K= DET
k=K+1
By the adaptive recovery method given in Düng (2011), it holds that 2 x N₂, Moreover, if the residual component g = II/ (fats 10 -fN.) ) along P.' satisfies g * by redefining fN, as g + fN, (modifying the coefficients of the B-splines up to resolution K if necessary), we can ensure that 11 (faur - fN.) and the approximation error of fetr does not increase due the Pythagorean theorem. It follows from Assumption 3 that
||T(far fN.))lllnnn/.. ( 2r - SN, (((2(x))) S NN N=
//d
Here, we have bounded the (²(Px) (-norm by the (>X in the continuous regime s 2 dx/p and by the (²"X)) in the discontinuous regime < dx/p via Assumption 2.
On the other hand, by the proof of Theorem C.8, there exists a sigmoid neural network IN. such that Il fN.llw; fNallw;(x) + Il.x,, - fN. (x) & poly(N,)c
with weights at most poly(E- Thus choosing e so that this error is dominated by N (*ts)/dd possibly by increasing the norm bound of F= by (oly(N,) factor, we can ensure
[ - TTN, (2 I ||T fN. fN.))lEA(P)) + Il.xx. SNN S (=
for some JN. E Fa. Since II II,, is bounded, this construction valid even with domain restriction (10) for a suitable Cw.
Next, under Assumption 4* Assumption 4 with domain restriction and Cr O(Cw), ), it holds for all 0g 62 that Twpo, E CCr-U(B p' (Z)) by replacing + by (10) if necessary. We also have that 1/T (Z) S 11,., (x)=(x) S Ō C..
Thus by Theorem C.8 (slightly modified to account for the constant factor Cr), we are guaranteed the existence of fN, F, satisfying Il Tele [1)= N N=-6/d; /d: so that also have
sup inf IITwo, do, (1(((P2) 0.E0.
S /2
0.E0.
Furthermore, it follows from Lemma C.9 that N₂. S N₂ (og()=1Nn) and No S N₂ log(0=1N.) Hence by setting 82 x No (s+r3/dd 8g x 25 /d and substituting in (44), it follows that N₂ log No N₂ log N₂
E Nz 2.121 +N-
#
ED,,D, ||Tr
T (
+
m
A
and taking N₂ x 20 271 dx and N₂ x (m A n) म, we finally conclude:
ED,,P, [Tfar (12 S m If ReLU DNNs are used instead for F., the approximation error and covering number estimates replaced by Proposition and Lemma 3 of Suzuki (2019), respectively. In this case, the depth must scale as log(m A n) and the sparsity also incurs additional log factor. The resulting rates the same except that the log factor must be replaced by log (rather than log? suggested in the paper).
26+251
2012 log m + (m An) 2 log(m A n).
35

---

Published as a conference paper at ICLR 2025
E PROOF OF THEOREM 3.5
E. ADDING SMOOTHNESS REGULARIZATION
The Stage objective with nonnegative regularizer O2 R2z and regularization strength > reads
1 Σ ūi (xx=Exe(woo_ AR(O,))
0g = arg min
E. n 1=1
where we take R as in (11) later on. We begin by modifying the oracle inequality (Lemma D.1) to include regularization.
Lemma E.1. For N₂ == (2) 82), there exists constant C₁ such that for all 8 with log > it holds conditional on D1.
ED, |TTau - Ex|z|=,,1l + AR(O.))
Bx + AR(Ow)) + Ci log + 6z
S 4 inf 0,60_
Proof. For the quantities
I(e)) = |T 2x AR(0s),
ii(0)) DDD
Tfar
AR(O,));
7
it still follows that
logg//
(1(0)) s [TT Ex1zlWo,112 + +C
+ +6.)
72
(Log.V +
((()..) + +C¹
n
since is nonnegative. Moreover for all 0, 2z from the inequality
1
j (x12(v,, +AR(O.) S
0e (x12(vo. AR((2),
i=1
we have
2 n
i((d,) 5 (((0)) EDz
(ū: Tsar (Bxizlva_ -Exi2lva
1 (Eie))))) + + log.N2 8g
S 11(00)) EDz
2n
(logN>=
< ((
(i((.) oH
+ 5z
n
Combining the two inequalities concludes the statement.
We now complete the proof of Theorem 3.1 with regularization. By inserting additional R terms, the projected error can be bounded similarly as in Section D.1 as
Ep, [ITTfa T fa + AR(O.)
< 2Eo, [IT Exizlvz
+ AR(0.)) + +ED, [1xx| T (2(())
(oggN2 + Óz
<8 inf (IT - + AR((.)) + 20; 0.e0,
n
36

---

Published as a conference paper at ICLR 2025
+ 2ED, |[xx|z(66,] T --aaafaallla (())))
< 16tr Tvvo; (1:)( + Too Exiz(roy (onerool]l (((()))))))))))))))
+ 8AR(O2)
log.V2 8z
+ +21 - -xxiz [1li((p2) + 201
n
Note that we have not reduced the Stage error for Óz and 0% to supremum over O,. Indeed, we may retrace the arguments in Section D.2 through D.6 without reducing any of the terms to the corresponding supremum over č, C" or + but retaining their specific value (e.g. the specific joint empirical or population approximators) for 0.,, 0% until the final step. Then error bound (43) also holds with the supremal approximation error replaced by the pointwise error for the estimate.
that the Stage
E [T - Exiz (1 112)(1))
(log/G + No + 8g + 6z
S 9E
inf 0.
(T/a - Φ0. [B:(P)) + C2
m
and similarly for 0z Moreover taking F, as in Section D.6 and applying Theorem C.8 as before, the Stage 1 approximation error be bounded
inf [Tve_ Φb, (l)()))(P2) & N /d= [T Ba (2) S NN 1110 (x)) 0,E8,
for both oz. 0r Plugging this and (11) into the above, we obtain that
E far Tarr (22 + AR((.))
2 fatr Two; (1E2(p2) + AR(0%) + -N-22/d /d: (Iveoz (1Bp, (x) + [IN0 (lBg,
No lo + 8z + dz.
m
Again by Theorem C.8, there exists 0% E O2 such that [| - Tiboy (22(P2) S N₂ (o+m))/d and ||vog (x) is bounded, so that R(07) is also bounded. Hence taking N₂ x m 2 and Ng x (m n) Th. as before, it holds that
E fau Tfaulli AR(O.)
2x+27
NN 3t' N₂ log N₂
N₂ log N m n
N₂
x + N₂ da E [145 ll (2))
S m
2012 +dx
log m + + (m A n)
E
[Iva 11 (18p),
+ llogmm A n))
By further setting
log m (m A n) 2044's log(m A n))
A x m
(45)
we can guarantee that In particular, since q >>>>>>>0 2, isolating the regularizer yields by Jensen's inequality
E [[T Tfst AR(O.) S E [Wa Bag (x)] + +1)
E B((() E
[wa Poy (2)
(x))
+ < E(R((.))
for both choices in (11), and hence B[R( must be bounded above. From this and the preceding inequality, we conclude that E
[|T - T ffar 122 S X.
37

---

Published as a conference paper at ICLR 2025
E.2 OBTAINING NON-PROJECTED RATES
For a sufficiently large threshold N, let J'v be the N-term B-spline approximation of fat in Lemma C.7 such that Il fN (P)x)) S f//dd, Again, we have bounded the (2(Px) -nnorm by the 2 dz/p and by the (2 s < dz/p. Since 2 2, suffices to take K" K that JN P/ and 2kd, x N. Similarly, let fN be the N-term approximation of sarr. for which holds that [|sstr [N & N-o/d. [| BB (x) Then we have that
L nnor
fl fau faull12(Px)
$ f - fN]lLA(PX) + 220 120 T(wll + I1 [N]lLA(PX))
s 2 (1T TTNNNL + IT TSau (A(P)) +1[Tsuu T(NILL((22)
[1 - INIlLAPPE) + |l SNllLa(Px)
(2h0~>a +1) -1 fNll + Ilsar fwIlEA(Px)) + 2 | - -T ((strll(((P2)
S N to (11fatllBy + +1) + NNo//.1 [T - Tarll
by applying Assumptions 3, 6. Furthermore, [| 112.trl1lg,,,, [22g...())) (2) is bounded above with domain restriction, or bounded in expectation with regularization via the argument in the previous section by taking À as in (45). Squaring both sides and taking expectations, it follows that
ED,,D, [1Lear fa $
SNN- 2(a-20+m1)
270 +N de A.
Finally, setting N \ 2042m yields the desired rate.
Proof of Corollary 4.2. When 0 < < the approximations fN, fN adaptively constructed from B-splines up to resolution K = [C K1; nonetheless, the total number of elements used is bounded by N 2Kdz, and can also ensure compatibility with the forward link condition II [fstr - fx) = 0 as before. Hence the proof above can be repeated to obtain the same rate by applying the extended reverse link condition to the difference fN fN which is comprised of at most 2N B-splines.
Furthermore, the resulting upper bound (15) when p < 2, A >> 0 is strictly faster than the linear lower bound proved in Theorem 4. if (in the case of 70 = 7)
2s'
2(s A)
28' d2
+ 70
2(8
A) + 2711 + dr
which is equivalent to
(s (0)(s A)
(s A)d2
$ + 70 d,
d_ sdz + 2A%o
(s Add, + A(2%0 + dz)
In the same manner, a separation can obtained even if no + mi we omit the details for clarity of presentation.
F PROOFS OF MINIMAX LOWER BOUNDS
F.1 PROOF OF PROPOSITIONS 3.3, 3.7
Recall that the NPIR model corresponding to the NPIV model is given as
Y T fare (Z) + 17,
" n=
far far
f + E, Eln 2|
It can be shown that NPIV is at least difficult as NPIR (where the operator T is assumed to be known) in the minimax sense: an estimator for NPIV can be utilized to solve NPIR with the same expected risk by generating the corresponding treatments from the conditional distribution of X
38

---

Published as a conference paper at ICLR 2025
given data Z = (Chen & Reiss, 2011). This is true for any m, and thus yields a lower bound in n valid even when m - 30. The goal now is to lower bound the minimax risk over the Besov space (B
p.o (X)) for the non-prowe con-projected case or (BU (X))] for the projected case, when only samples {(2,, ]=) from the indirect model are available. We obtain this via modification of the Tang-Barron method (Yang & Barron, 1999).
First note that suppwk, covers + dyadic cubes with side length 2 in each dimension. By e.g. only considering l E l such that all components are multiples of r + 1, we can construct a subset Jg C I such that supp w contained in X' and pairwise disjoint for all E Js and
1/ (r +
Jul
2kd.
Consider the best approximation 11,0 of W1,0 w.r.t. the - on suppwi.0 in the span of the B-spline wo,0 and its integer translates. Note that p/k-1 is equal to the span of all B-splines with resolution exactly k 1. For each Wi.e with Jk. its best approximation p/ will be the correspondingly scaled and translated version T of 11,0, so that
llwk,e - ITC TT"K-Wws,ellLaCM ww,ell = lwwke - =,,ll -
kdd / T1,0122-
(46)
Now set fo = =oo= = - and define the functions
So = 2 2-2ka Ks Σ
Bu
V 1= 1, ,214
BEJ.
and
f. - fu. 11 11 Lu 22 k-
Σ EEJx
Bu,e(wnee Tk.e),
where is suitably small positive number and Br = (Bu,))es,, is an enumeration of the vertices of the hypercube {1, -131/al It follows from the sequence norm equivalence (DeVore & Popov, 1988, Theorem 5.1) that
1/p
| (x)) = delp)
Σ EEJL
12 ke Bu,elP))
Also, the coefficients of Th, (a linear combination of B-splines W resolution k 1) for each l Jk are different translates of the fixed coefficient sequence for 41,0- Denoting its ll -nom by A, it follows that the coefficient each B-spline Wk-1,t' in the sum
11 1I-k-1f fo 2-s Σ Bo,eThie
BEJe
also uniformly bounded by ks cA, and so IT. 1 11 (.) (X) S é by a similar computation at resolution k - 1. Hence
Il (ffulle,,6000 (2) S lIfelle, + || felll,, $
so we can ensure that each f, is contained the target class U(B p.g (X)) by choosing to be suitably small.
Furthermore, by the Gilbert- Varshamov bound, there exists well-separated subset V the index set [1, (1..... Je13 with log V&
= [&& 2kd such that
Σ
1B B.r.2 lJal.
u v' E Vv.
2F
REJ
which guarantees the separation
|1 - fr. 11:2(x)
2 2ks
Σ
Allware (2,
tEJe:
39

---

then the Besov norm Published as a conference paper at ICLR 2025
2 1 kdz 2
by (46). The KL divergence between the sample distributions (ū:) from the NPIR model with Lat fox fo is then bounded as
[L (1.))) = (E))
IT [Tfllla((P)))
20
11
2 2k 2. 2-211
20
$2 2k(
due to Assumption and the link condition. Here we have utilized the near-sparsity of wa,e extending the definition B-splines to all locations E zd= is easy to see that they form a partition of unity, Eregae Whet 1 and
| & 11 <
Σ
Σ
S
RETE
12(22)
RET
(x0
Hence the incquality
Σ KL(P)NPP)) log V&l x 2kd, NEV
VE
is satisfied by scaling the resolution as 2(2+22n-d ). n. Finally, applying the Yang- Yang-Barron Barron method proves that the (A)) minimax rate is lower bounded as
inf j.PIIR f.u (U(
sup
ElIl Satrl l 2 2kx x 242211
We can check that the ordinary rate n 20 is retrieved when = i and 0. while the rate becomes much worse If decays exponentially. The same rate holds in the (2(PP))nom under Assumption
proving Proposition 3.7.
For Proposition 3.3, for any estimator f. f is also valid estimator for the projected target Tfst since T is assumed to be known in the NPIR setting. Repeating the same construction given above, since fo is contained in p/ the separation the projected MSE now becomes
also derive the projected lower bound in the manner by noting that
f. - Tf (1t 2 2 2
2-2k Ilfo - for ((E 22 2202222))
due to the reverse link condition and Assumption 5. Hence conclude that 1.10 E[|[T - f 2 n 2.1211
inf f.PPIR EUK
sup
F.2 PROOF OF LEMMA 3.2
The metric entropy of Besov spaces is classical:
Theorem F.1 (Giné & Nickl (2015), Theorem 4.3.36). Ifs d(1/p-11) 1/2)+, unit ball B U(B n.o (10, 1]d)) is relatively compact in [ 1]d) and
NBB
Vg 0.
On the other hand, the construction in the previous section gives subset of ( (X))) with log p.g
cardinality log V&l x 2dd= and separation TT. Tfr 11: 2 2- 22 Assuming Pz has Lebesgue density bounded above, we may take 5 2 (-+ to satisfy
8 min |T - T (
2 Veee
40

---

Published as a conference paper at ICLR 2025
so that each 8-ball in the projected image cover most element fu. It follows that log Vvl x log ( (X)),, L3 (10, )]
S log log (2)), L' (10, 1] ),
22(++)))k
and taking the resolution of the subset is + oc, the constant factor can be eliminated, concluding that s' s//d2 S (s /o)/d2-
Finally, we verify that equality can be achieved if dg = d₂ and T acts on B-splines as Twkke = 2- wx,e- Let f be an arbitrary element of U B (B))) (2)) with B-spline decomposition
Σ
Tf =
Ta 2 20 Bk.ewk.e.
k=0 RET
Then it follows from the sequence norm equivalence that
1/07 1/4
IT B (2)
2 20+ dd./n)
Σ EE1E
Tak
Tak Bu,el
1/p 9 1/9
2k(e-de/p) /pp)
Σ EE1N
13x
x (l
implying that ( p.g (x))) S CT U (Z)) for some constant Cr. Hence the L rate of
contraction specifies the degree of which smoothness is increased under the maximal smoothness assumption.
F.3 PROOF OF THEOREM 4.1
Since the bound is equal to the overall minimax optimal rate when p 2 2, we only consider the case p 2.
Write = (21,.... 2n) for brevity and denote its joint law by Ps. We fix & 2 0 and consider the B-spline wk,t on X for E Ig. Note that (llx,ell and
|Twn,elle<(P>) $ 2 1 [wn.ell 2 1 || x k(x+d./2))
by the link condition. This implies
T2k. < (22.)) E(U2,1 < C.2 (2m
Ue ITwk, ell.Ael
for some constant Cu. By Bernstein's inequality, it follows that
(-E
n²/2
P
U0, > 2
S expp
- C. B[U?,] 22
and by union bounding.
P
sup = EET, 1=1
1
Σ
Up, Uei >
S 2kd, exp (-03.
3 2 k(27 +dol
Now assuming n 2 24(2n k - co, so that the event
for some the right-hand side converges zero as
1 3 = E n sup Σ
Ue, 5
FET, n i=1
41

---

Published as a conference paper at ICLR 2025
satisfies P(E) = (1).
Now as in Section F.I, we may reduce to the NPIR model (4) for known T and only consider estimators of the form
(x(
Σ (x
u1 Un x R.
(47)
Denote the corresponding linear minimax rate
RL
inf fuilinear fale B (2))
sup
E [1if faulllzze2)
and define the auxiliary function ((x) = Ei= le 2x( dP dP3. For all f E U( (( we have from Us
T2, + Y) that
I1.2 f112(()]
R2 2 E
> E
Tf(E))(1) f(-)
+ E
1lu 2)122000)
(2(N)
L x
dP +oo / ((x)
( f(xx)
(48)
Set 1 Is n and partition the domain into rectangles as
4 4 C
x= U
A U
2 2
BET
EETT
1=1
It follows from (48) that there exists 8" E 1 satisfying
JAce ((x) do kd. 0o R
2-do
To each A will associate the B-spline Wk.e-to where to (Ir//)....,,1/2)).. It can be that there exists a constant C, depending only on r such that
vol(x E Ao.o wo 2 C₂} 2 C...
Hence for the sets
= far E Aur wh. 2 C,), H = (o Ak,e- g(x) < 20 C,
have VolG 2 2 2- and Vol(Ak. {H) < 2 -d that Vol(G n H) 2 2 2-dd, -- -Cr. Moreover from the definition of E, for all x E H we have
Σ
Twk, Tw (2))
dP:
sup EEIn i=1
Tw,))(Z;)² Σ
(z 2) dP
1-11
4C,n C.op
< 2n Tw..
((x)= <
2 k(2z) +da) Ri.
For a moment, suppose that
4C,n
2 2- -
C² 4
(49)
C,
42

---

Published as a conference paper at ICLR 2025
We apply (48) to the scaled B-spline f ---k((-da///i www.e where llf (x2,)))) andd O(1) chosen that (((Bp.(())) It follows that
Janala Σ (00))-- wk. wo. (00) dp:dr dr
R₂ 2 2 22 - dz/
ann [V( (o
2 2 2k1s 2
1/2 2
Σ Tw (2())ue((z)
dP:
dx
1/272
2 2 2h1s 2h(s Vol(Gn
P(E)
4C,n
2 2-b(2m+
Coog
(1) 2 2k(s- 2k(--d
Comparing with (49), we have shown that either of the following must hold:
(2x1+
R 2
or RL 2 2-2
Finally taking & such that n x (2( 2)+ td,), we can verify that condition (+) is satisfied since
A, and therefore Rr Z n 22-1
F.4 SEPARATION IN PROJECTED RATES
To obtain a lower bound for the projected MSE of linear IV estimators, we require that Z to be sufficiently spatially covered by the projected class so as to be difficult to learn for non-adaptive estimators; one such sufficient condition is outlined below.
Theorem F.2. Suppose that d2 S dz. For any cube S in the dyadic partition of into 2kd, cubes of all side lengths 2 we assume there exists E Tt such that
Vol (a (1[Tw,,e()) 2 pb C1 VolS, μa = C2 .2 2--2220-d.-44
for constants C1, C2 O. Then under the conditions of Theorem 4. I, the projected linear minimax 2(1-4)+270
rate á
n (26 25 0.1912
The threshold μ4 corresponds to the natural magnitude if the mass [Tww,eli?<(z) 2 22k(2r++d is distributed uniformly on S. Since each B-spline can cover asymptotically finitely many cubes at the level Pk. the implication d, S de follows from counting argument. Hence comparing with Theorem 3.1 when p < 2, we conclude that DFIV achieves faster projected rates compared to any linear IV estimator if 7oo 71, T has maximal smoothness and d, (dz 2(s A))
A-A+20 d s+770
Proof. The proof is similar to Section F.3. We will show the lower bound for all linear estimators of fat of the form (z) >> uz(2, E)), which includes the projection Tf. = - Tu,(-,))p) of any NPIR estimator (47). Denote the desired rate by R2 and replace the auxiliary function ((x)
by q(2) Em (2, 2) dP:.. It follows for all f (BB that
2
R¹ 2 / (TAE))l2 Tf(x)) dz +ox / q(2) q(2) dz.
'The derived rate shows that it is possible for separation to he achieved even if d, < dg 2(s A) and /d, < (s //)/d, in certain smoothness regimes. We omit precise characterization.
43

---

Published as a conference paper at ICLR 2025
Then there exists cube S in the dyadic partition of Z satisfying Is g(2) d2 2 2- 000000000000000000000000 Ro² By assumption, there exists Iş such that for
G = {= E E [Twk,e(=)] us)
= (2 E ((2) - 20 Rzh
it holds that Vol G 2 2 kd. 2-= and Vol(G n H) 2 .2 2- -1 moreover
[e Σ Twk Tu
dP; -
4n
2 k(2 + Ry,
C
Then repeating the above line of reasoning gives that either of the following must hold:
R2 2kn2n td.)
or R2 2 20)
If dg > dz 2(s A), we may take n 2) +d=) so that condition (+) satisfied. d_ S dg 2(s A), we instead take n (2m=-44++) for arbitrary > The stated lower bound can be verified in both cases.
Remark F.3. The dependency of the optimal rates on dimension dy is an intrinsic property of Besov spaces; however, this can be removed by instead considering mixed (Schmeisser, 1987) or anisotropic Besov spaces (Berkolaiko & Novikov, 1994). By applying the results in Appendices C, D to the appropriate wavelet systems, existing learning-theoress analyses for these spaces (Suzuki, 2019; Suzuki & Nitanda, 2021) can also be adapted to incorporate smooth DNN classes and obtain the corresponding dimension-free bounds.
44


Reviews:
[{"review_id": "qP0BKEvL0n", "reviewer": "Reviewer_1NJE", "summary": "**Summary:**\nNonparametric two-stage approaches to instrumental variable regression can be classed as either using fixed features  features (e.g. RKHS, spline, dictionary) or data-adaptive features (e.g. deep neural network). Of the latter approaches. deep feature instrumental variable regression has been shown to empirically perform very well, but lacks theoretical analysis. The authors prove that DFIV achieves minimax optimal rates when the target structural equation belongs to a certain class (Besov space). In order to do so, the authors use standard nonparametric IV assumptions plus one more assuption on the conditioanl distribution of the covariate given the instrument. DFIV is also superior to fixed-feature methods for two other reasons: when the target function has low spatial homogeneity (it can be smooth or bumpy in places), DFIV still attains the optimal rate but fixed feature methods fail. Secondly, the two-stage regression models are more data efficient in the first stage than kernel methods.", "questions": "Please address the weakness/questions above. Referring to the details above:\n- Why can't stage 2 derivatives go through to stage 1?\n- Is Assumption 5 restrictive (in particular, does it include Gaussian?)\n- Is it assumed that stage 1 and stage 2 is solved over this neural network class? How realistic is this assumption?\n- Are the smoothness penalties/constraints actually practical, or do they only allow the theory to work? If they are not practical, then perhaps it is overselling to claim that the method of Xu et al., 2021 really achieves the minimax rate, because it is after all a practical algorithm.\n- Is it possible to show how mild these assumptions or regularisers are by doing a toy experiment?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Strengths:**\n- It is mentioned on line 450 that DFIV is shown to require strictly less data in stage 1 than in stage 2, whereas for kernel IV, an assymetric amount of data is needed in stage 1 and stage 2. \n- I think this paper provides some significant results: namely, it provides a rigorous analysis of DFIV and shows advantages in terms of adaptivity, sample efficiency and balacning in stage 1 and 2, and justifies the use of neural features.", "weaknesses": "**Questions/Weaknesses:**\n- \"The above loss generally cannot be directly optimized for θx as ˆθz cannot be backpropagated through\". Is this for practical or conceptual reasons? One can imagine either running some iterations of an iterative solver (e.g. Newton's method, SGD, ...) and then backpropagating through the iterates of the solver (often called \"unrolling\"). Alternatively, one can compute the derivatives using the implicit function theorem, by computing the solution and then plugging directly into a known form. But my guess is that either of these approaches fails to yield a useful gradient in practice?\n- Assumption 5 - $\\mathcal{P}_{\\mathcal{X}}$ has Lebesgue density bounded below. Does this exclude e.g. Gaussian densities? Is this a restrictive assumption? One way I can see this assumption satisfied is if the support is compact and the density is bounded from below (otherwise, without a compact support, I think the density may not even be integrable?)\n- As far as I understand, neural network models are used (belonging to class on line 245) as the class in stage 1 and stage 2 problems (7) and (8). It seems to be assumed that a minimum exists (and perhaps is unique?), and even more strongly, that this minimum can be obtained by the algorithm. The derived minimax rates assume that the actual minimum is found, which is not realistic. Or maybe with sufficient overparameterisation, it is realistic?\n- The smoothness of stage 2 needs to be controlled, and the authors propose two ways to do this. One is to restrict the function class, which in practice requires somehow discarding ill-behaved bumpy solutions after random intialisation. However, practically instantiating this procedure is not really discussed. Another option is to penalise the Besov norm, and I am not sure how hard it is to actually compute a Besov norm for a given neural network. Are these smoothness penalties/constraints actually practical? \n- This paper is very theoretical, containing no experiments. This is a little bit (but probably not unheard of) for ICLR. A venue like COLT might attract readers more strongly aligned with its contents. Alternatively, experiments could be done to actually demonstrate that the assumptions made are realistic. For example, it could be investigated how hard it is in practice for the smoothness of the stage 2 by discarding ill-behaved solutions, or whether a zero stage one (and/or small stage 2 loss) can be obtained with a sufficiently rich neural network. None of these experiments are hinted at.\n\n**Minor:**\n- Grammar. \"...while the rate in Stage 1 samples m is depends on the smoothness of both...\"\n- The sentence on line 368 is a bit confusing. Are you saying $\\log(\\cdot)$ is replaced by $(\\log(\\cdot))^3$?\n- Typo on line 446.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EggHcH89zq", "reviewer": "Reviewer_HafH", "summary": "The article provides a convergence analysis of the DFIV algorithm under certain assumptions and proves its minimax optimality. It compares DFIV with fixed feature methods, demonstrating its superiority in low spatial homogeneity and data efficiency.", "questions": "1.\tThe article mentions many assumptions; so what does the term \"Additional assumption\" in the abstract specifically refer to? Is it Equation (2) or others? If this assumption is crucial for the derivation, the abstract should succinctly highlight its necessity rather than merely emphasizing its existence.\n\n2.\tRegarding Equation (2), a graphical illustration could be added to provide a more intuitive understanding of this part.\n\n3.\tThe title mentions \"adaptivity.\" Is there a distinction between \"data adaptivity\" and \"data efficiency\"(in abstract)?\n\n4.\tLine 41: The functional equation is not direct, and a reference to Equation (4) in Section 2 should be added here.\n\n5.\tComparing existing statistical analyses in studying ordinary regression and here two-stage setting, what’s the difference between these two types of theoretical tools? Under the current setup, what technical challenges are brought?\n\n6.\tLine 128-143: Why is Problem/Equation (3) transformed into Problem/Equation (4)? Given that the resulting problem (4) is generally ill-posed and the solution may be ill-behaved or not unique, why is it necessary to introduce the operator \\(T\\)? Can't the original problem be directly solved? The role of \\(T\\) needs to be explained in detail here.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe convergence and optimality analysis in this paper is necessary to understand the DFIV algorithm. \n\n2.\tThe motivation is well-presented.\n\n3.\tThe theoretical analysis is solid and It provides a rigorous foundation for DFIV.", "weaknesses": "1.\tThis article is symbol-dense and hard to follow. It lacks some illustrative diagrams or symbol tables.\n\n2.\tThe rationality of some assumptions (e.g., Equation (2)), the reasons for transforming problems (e.g., from Equation (3) to (4), and the role of mapping \\(T\\) are not thoroughly explained in its introduction part. In addition, where and why is Definition 2.2 used… The text lacks explanations for these points, and the necessity of many techniques used is not clearly articulated.\n\n3.\tSee Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oPNMsYK5dh", "reviewer": "Reviewer_voVt", "summary": "This paper provides theoretical guarantees for the DFIV method, which uses adpative deep features to solve the non-parametric IV regression problem. The authors prove an upper bound for the risk of the DFIV estimator when the true structual function lies in a Besov space. Secondly, the paper presents an information-theoretic lower bounds for NPIV in Besov spaces and shows that DFIV can achieve the minimax optimal rate with a balanced number of Stage 1 and 2 samples. Lastly, the paper demonstrates that DFIV has superior empirical performances when compared with linear fixed-feature methods.", "questions": "I am curious to see a discussion on the smoothness assumptions that is used in this work and the $\\beta$-source condition that is often assumed for inverse problems (and also used to study NPIV).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Showed that DFIV can achieve the minimax optimal rates, which is positive result. \n- Included an analysis on the sample sizes required for Stage 1 and Stage 2, which can provide more insights into the algorithm.", "weaknesses": "- This paper would benefit from including a brief discussion on the technical novelty in the analysis/proofs, which is important since this paper mainly presents theoretical results. \n- Section 4 (separation with fixed-feature IV) would also be better supported with some experimental results showcasing the improvement of DFIV over linear estimators. \n- DFIV is an iterative method, but the paper did not include any results/rates of convergence, which can be an important practical issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)