It’s Hard to Be Normal: The Impact of Noise on Structure-agnostic Estimation
Authors: Jikai Jin
Lester Mackey
Vasilis Syrgkanis
Venue: NeurIPS 2025 poster ()
Keywords: structure-agnostic causal inference
minimax
causal machine learning
semiparametric inference
Neyman orthogonality
double machine learning
higher-order robustness
Primary area: theory
TL;DR: We show that the statistical optimality of estimation methods for causal inference depend in a surprising way on the distribution of the treatment noise.

Abstract: Structure-agnostic causal inference studies how well one can estimate a treatment effect given black-box machine learning estimates of nuisance functions (like the impact of confounders on treatment and outcomes). Here, we find that the answer depends in a surprising way on the distribution of the treatment noise. Focusing on the partially linear model of Robinson (1988), we first show that the widely adopted double machine learning (DML) estimator is minimax rate-optimal for Gaussian treatment noise, resolving an open problem of Mackey et al. (2018). Meanwhile, for independent non-Gaussian treatment noise, we show that DML is always suboptimal by constructing new practical procedures with higher-order robustness to nuisance errors. These ACE procedures use structure-agnostic cumulant estimators to achieve r-th order insensitivity to nuisance errors whenever the (r+1)-st treatment cumulant is non-zero. We complement these core results with novel minimax guarantees for binary treatments in the partially linear model. Finally, using synthetic demand estimation experiments, we demonstrate the practical benefits of our higher-order robust estimators.

It's Hard to Be Normal: The Impact of Noise on Structure-agnostic Estimation
Jikai Jin" Stanford University jkjin@staaffrd.edd
Lester Mackey Microsoft Research Jmackey@microsoft.. com
Vasilis Syrgkanis" Stanford University
veyrgk@stanforddeed
Abstract
duructure-agnostie causal inference studies how well one can estimate a treatment effect given black-box machine learning estimates of nuisance functions (like the impact of confounders on treatment and outcomes). Here, we find that the answer depends in surprising way the distribution of the treatment noise. Focusing on the partially linear model of Robinson [1988], we first show that the widely adopted double machine learning (DMLL estimator minimax rate-optimal for Gaussian treatment noise, resolving an open problem of Mackey et al. [2018]. Meanwhile, for independent non-Gaussian treatment noise, we show that DML is always suboptimal by constructing new practical procedures with higher-order robustness to nuisance errors. These ACE procedures structure-agnoste cu- mulant estimators to achieve r-th order insensitivity to nuisance errors whenever the (r + 1)-st treatment cumulant non-zero. We complement these core re- sults with novel minimax guarantees for binary treatments in the partially linear model. Finally, using synthetic demand estimation experiments, we demonstrate the practical benefits of our higher-order robust estimators.
Introduction
Modern machine learning (ML.) offers a rich toolbox of flexible methods for modeling complex, high-dimensional functions - ranging from regularized linear regression [Belloni al., 2014 4, Zou and Hastie, 2005] and random forests [Breiman, 2001, Biau et al., 2008, Syrgkanis and Zampetakis, 2020] to neural networks [Schmidt-Hieber 2020, Farrell et al., 2021] and hybrid combinations of these techniques [Džeroski and Zenko, 2004, Chemozzukov econometricians, is natural to ask whether ML can improve both the accuracy and the robust- ness of estimating a target parameter of interest. Recently, Balakrishnan al. [2023] introduced the paradigm of structure-agnostic estimation (SAE). SAE enables parameter inference by directly plugging in black-box MI--based estimates of the so-called nuisance functions the other regres- sion components that impact our observations while not being of primary inferential interest. This framework characterizes the best possible target estimation accuracy in terms of the error these nuisance estimates. By contrast, the classical semiparametric approach derives optimal error rates under explicit structural assumptions on the nuisance - such as smoothness or sparsity which must be exploited by the estimator and render it fragile when those assumptions fail [Stone, 1982, Chen and White, 1999, Belloni et al., 2011].
al.. 2022]. For statisticians and
In the field of causal inference, typical goal is to estimate the impact of a treatment on an observed outcome in the presence of confounders X that impact both treatment T and outcome Y in largely unknown ways. When all confounders are observed, many causal estimands admit expressions in
'ikai Jin was supported by NSF Award IIS-2337916.
Vasilis Syrgkanis was supported by NSF Award IIS 2337916.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

terms of regression functions that are themselves estimable from data, Consequently, causal param- eter estimation falls squarely within the regime where ML-based nuisance estimates may provide benefits.
Recently, double/dehiased machine learning (DML) [Chernozhukov et al., 2018, 2022] was pro- posed as an efficient way to estimate causal parameters from black-box box nuisance estimates. Specif- ically, given n independent and identically distributed (i.i.d.) observations and nuisance estimates with mean squared error e, these methods achieve O + n 1/3) error rates, improving over the O(E + 1/2) rate achieved by naively plugging in the nuisance estimates. In particular, the esti- mates are v /n-consistent even when the nuisance estimates converge slowly n-1/4, Moreover, this rate is structure-age aggntssi,,gnottcc, and recent works [Balakrishnan et al., 2023, Jin and Syrgkanis, 2024] proved that for several important causal parameters, this rate is in fact minimax rate-optimal in the SAE framework. In other words, one cannot achieve smaller estimation estimation rates without strong prior knowledge the underlying structure of the nuisance. These optimality results provide a strong justification for the popularity of doubly robust learning methods in practice.
A curious exception to this rule can be found in the work of Mackey et al. [2018]. There, the authors study the popular partially linear model of Robinson [1988]]
Y = 0o T + fo(X)+ + Big X = 0 almost surely,
T = T + 17. and Bin X| = 0 almost surely.
Here, the observation Z (X, T) comprises covariates X E X. a scalar treatment T E IR,
and a scalar outcome Y E IR. The primary objective is the estimation of the parameter 0o. which represents the average causal effect of T on Y. Within this framework, the DML estimator for 0g is derived from the Neyman-orthogone
moment condition and typically takes the form:
(7; ((X.)) V ((X.))
0DMLI
((X.)))
(2)
where 9 and q are machine learning estimators for the nuisance functions 9o = E XX] and (x(X) B[y | X]. respectively [Chernozhukov et al., 2018]. Common practice involves sample splitting, where one portion of the data used to train 9 and ) and another (independent) portion used to compute 0DML via (2). However, Mackey et al. [2018] showed that for the model (1), one can design structure-age caccurre-gnosticc estimators that more efficient than DML if the treatment noise 7) is non- Gaussian and independent of X.3 Bonhomme et al. [2024] consider parameter estimation problems in conditional likelihood setting and show that arbitrarily higher-order orthogonal estimators can be constructed if the conditional likelihood function is known up to the nuisance parameters. These works highlight gap in our understanding of how noise impacts optimal SAE rates, and this is the main question that we will address in this paper.
Given access to black-box nuisance estimates, make the following contributions:
On the one hand, we prove in Section 3.2 the existence of Gaussian treatment barrier: for Gaussian treatment, no estimators can achieve a better rate than DML, even when the variance of the treatment completely known. This implies that leveraging distributional information of the treatment noise as in Mackey et al. [2018] cannot yield a better estimate and that their restrictions to the non-Gaussian setting is not an algorithmic issue.
On the other hand, for non-Gaussian treatment, we propose a general procedure to construct higeer-order orthogonal moment functions and provide structure-agnostic guarantees in Sec- tion 4. Then, for treatment noise independent of X, we derive in Section 5 new agnostic cumulant- based estimator (ACE) that achieves r-th order insensitivity to nuisance estimate er- ror and O + n 11/2) error rates for treatment effect estimation whenever the (r + 1)-st cumulant of 1) is non-zero. To the best of our knowledge, this is the first structure-agnostic estimator that achieves arbitrarily high-order robustness.
We complement these findings with additional contributions relevant to this setting. Specifi- cally, we show in Section 3.1 that DML is minimax optimal for (1) with binary treatment, a case not covered in existing lower bounds that do not assume partially linear outcome model.
'The structure-aus aggoosicc rates are not explicitly stated in Mackey et al. [2018] but can be straightforwarddy derived from their analysis.

---

We also derive new lower and upper bounds for structure-agnosd moment and cumulant esti- mation standard non non-parametric model in Section 5.1 that might be of independent interest. Finally, in Section 6, we conduct a synthetic demand estimation experiment, highlighting the benefits of the ACE estimator compared with existing approaches.
Notation We introduce the shorthand [m] _ {l,...,,m) m for each m E N. For a function g and distribution P on a domain 2, we let | P.s - ( & (J d=))))))))())) with 8 2
represent the (P) norm. For a vector E RR', we define ||d] - maxisssi /4). For two vectors = (" ,r), 3 = (31, E R, we write a S 8 if S Br, 1 sist.
2 Structure-agnostic estimation and minimax error
To evaluate and compare method quality in this work, we adopt the minimax straeture-agnostic framework of Balakrishnan et al. [2023]. Notably, pauucture-anooste analyses make no explicit assumptions about nuisance smoothness, sparsity, or structure and instead simply assume access to black-box nuisance estimates with certain unobserved error levels.
We first define the class of all data generating distributions on our data domain Z and assign to each a target parameter (P) and a vector-valued nuisance function h 2 + R
Definition 2.1 (Data generating distributions, target parameters, and nuisance functions). Through- out, we let P be the set of candidate data generating distributions on the finite dimensional-dones
2. let H. S (I2)) 2 be the set of relevant vector-valued nuisance functions, and let 4 be determin- istic mapping from P to H. Then, for any E P, we say that (p(P) E R is the target parameter and h = (P) H is the nuisance function corresponding to the distribution P.
We will sometimes abuse notation and choose H to be subset of H S. (RR))* where X' is the domain of the covariate component of Z. Departing from prior work [Balakrishnan et al., 2023, Jin and Syrgkanis, 2024], we leave the exact choice of nuisance mapping & unspecified in Defini- tion 2.1. This allows us to study how the choice of nuisance functions affects the estimation error of the target parameter of interest.
Next, we introduce the ground-truth uncertainty sets associated with any nuisance estimate h and any target error level c. For any cector-valued function h X' + Re. distribution P E P, and _S, we define (h", ;= {h, L*(P): ||,, kpp,p, S E,, e [4].
Definition 2.2 (Uncertainty sets). For any nuisance estimate h E H. error level vector E TR4, and power $ 2 1, we define the uncertainty set Ps,c(f; (h; D) as the set of all P E Pu satisfying
lh - hillp, Sc,, for h (P) and each ie (1,2.....6 E
(3)
or, equivalently, (PP) E (B,c(h
For convenience, we will omit the dependency of P..c b)) on P. which will always be clear from context. We will sometimes write (h, when the choice of tb is obvious. Finally, for given estimator Ô of target parameter (P)), we define the worst-case Definition 2.3 (Minimax estimation error), For any of distributions P over Z, define the worst-case (1 -qqquntllle error of estimator θ Z R Rn,1-- P) 2
over uncertainty set.
supper 2p,1--(18 (l((P)) where
of 10 (Z. 7n) (o(P) when 7, lt P. We further define the minimax estimation error of P as Ma (P) A
is
inl inlg.zon_p Ri (0:P))
3 Structure-agnostic lower bounds
In this section, we establish structure-agnostic lower bounds for treatment effect estimation in the partially linear model (1).
3

---

3.1 Optimality of DML for binary treatment
We begin by establishing the minimax rate-optimality of DML when the treatment T is binary. Previous works [Balakrishnan et al., 2023, Jin and Syrgkanis, 2024] have established structure- agnostic lower bounds of similar form. However, Balakrishnan et al. [2023] consider the estimation of the expected conditional covariance defined as E[Cov(T) (T, Y X)). while Jin and Syrgkanis [2024] consider the estimation of the average treatment effect with a different set of nuisance functions. Furthermore, neither work constrains the form of TB[Y T, X while we assume partially linear structure for the outcome model. Our result implies that even with this additional assumption, is
still impossible to improve over DML. For convenience, we introduce the following definitions as the "default" choice for the class of data generating distributions and nuisance functions:
Definition 3.1 (Set of feasible distributions). We define P as the set of all distributions (X,
generated by (1), Moreover, we define the following subsets of p* as follows:
for any constants Co, CT, Oy E [0, +oo], we use (Ce, CT, C4) to denote all distribu- tions P P* that satisfy 10p) S Ca, BlTT//u S CT. and [ S Cv.
for any constants Co, Cg: Ca. 1/,, s//n (0, too,, use P (Co, Cg: Chi 1hn) to denote the set of all distributions in P E pp* that satisfy
1. 10ol & Co. (go(())) S Cg. 19o(X)] S Cg a.s. for (gu. 9x) = (E[
X], E|Y (X)
2. X and n X are the and vn -n--ub- Huub-Gaussian a.s..
for any constants Co. CT. Cy E 10, +o0]. we use Po CT, Cv) to denote all distribu- tions P PET PP* that satisfy |0n| S Co |T| S CT, and |Y| Cy.
Finally, we define oh as a mapping from P E pp* to the "default" nuisance functions ho = (go, 20)-
We are interested in the minimax suncture-agnostic estimation error induced by P = PPPs.c PP..ch,, (h. +) for some s 2 1, the sense of Definition 2.3, with being chosen as set of distributions that satisfies certain mild regularity conditions, such as the ones introduced in Definition 3.1. In other words, given black-box nuisance estimates of ho with L (P) error rates, we would like to derive the optimal worst-case estimation error for the treatment effect 0o.
Our main result in this section is lower bound for estimating On when T is Bernoulli random variable. Our lower bound, Theorem 3.1, is established under the additional assumption that X has a uniform distribution on X' [0, 1] KK; as a consequence, the minimax error rate of DML cannot be improved even when the marginal distribution of X is known.
Theorem 3.1 SStructure-agnostee lower bound for binary treatment). Fix any Co > O, cq. 8 E (0, }),
and K E N and let
P =P PolCo, 1, 10, Varp(T X) 5 and X ~ Unif( (0, 1] (4)
If Iflc|loo S 8/2, then for any estimates à = (g. () with cg S ((X)= s - Cg and (() - g(X)) 2
A 4-1 8, U.s., we have
Nr.1 1-7 (Per (h, Φz)) 2 c, [A- e1e2) + 1/2 1/2
(5)
for any y 11, where C. is universal constant that only depends T,.
When ||el|oo < 6/2, this matches the upper bound achieved by DML up to constant factors, stated in Theorem C.I for completeness. The proof of Theorem 3.1 can be found in Appendix B.
3.2 The Gaussian treatment barrier
In the previous section, we established the rate-optimality of DML for binary treatment. it possible to improve over DML if we make different distributional assumptions? In the literature, it is not uncommon to model the treatment assignment rule using a Gaussian distribution, i.e., v X ~ (O a (X)²) for some function af-) [Imai and Van Dyk, 2004, Zhao et al., 2020]. However, in Mackey et al. [2018] it shown that Gaussianality of the noise variable 7j is in fact barrier for one to construct second-order orthogonal moments, thereby preventing them from deriving better error

---

rates than DML by leveraging distributional information of However. it is unclear whether this is an issue specific to their approach or if there indeed exists fundamental, non-algorihmmic barrier for Gaussian treatment.
In this section, resolve this open question and show that the latter is true: if the treatment noise is Gaussian, then DML is already minimax rate-optimal even when is independent of X and one has exact knowledge of its distribution. Our lower bound, stated next, is proved in Appendix D.
Theorem 3.2 (The Gaussian treatment barrier). Let a, Co, Cg. C.
be known constants and P (P E Po oo, Cq) r) ~ NO o²) and [go(X)| S Cg a.s.). efe1e2 o(log 1/22 oo10g-1/2 n), then for any estimates = (g. Q) satisfying (g] S Cg. lgl S Cg and any 1 S S to0, we have
1 (Pe..(h, x*) co (
3 to too n 1/2)
(6)
for any y (1/2,1), where C, > 0 is constant that only depends on 7/-
The assumption that Lo] < Cg. lgl < Cq is natural. since for any Po E Po. the ground-truth nuisance functions 9o (X [T X] and 4o (x( = E| | XX] must satisfy 1gnl S Cg and Iqol S Cg a.s. according to our assumption. Moreover, the lower bound does not depend on the value of meaning that no improvement is possible even if the nuisance estimates have small 0 errro..
Under the same assumptions, one can show that DML attains the minimax error rate up to the factor ( (lo((/+11)) )² and matches the minimax error rate whenever 61 O(n=9) for some positive constant c. For completeness, we include the details in Theorem C.2. Notably, compared with the lower bound in Theorem 3.1, the term (cÿ)) disappears here since the variance o² is assumed to be known. Theorem 3.2 establishes the existence of method-agnostic Gaussian (realment burrier as suggested (but not proved) in Mackey et al. [2018].
The proof of Theorem 3.2 in Appendix D is based on a constrained risk inequality for testing com- posite hypothesis developed Cai and Low [2011] combined with novel constructions of the fuzzy hypotheses using moment matching techniques. For Gaussian treatment, we show that such hy- potheses can be constructed in a way such that their induced target parameters 0g are well-separated by leveraging recursive property of Hermite polynomials.
4 Structure-agnosti upper bounds
Section 3 established the rate optimality of DML for binary and Gaussian treatments but left open the possibility of improvement for other treatment distributions. To exploit this opportunity, we will first introduce new procedure that yields fast estimation rates whenever 1/ X is non-Gaussian and the cumulants of y | X are estimated accurately and next, in Section 5, show that cumulant estimation is easy when 77 is independent of X.
Our new procedure is based on the method of moments. Specifically, we will identify a moment function m satisfying Er[m (2, 0, h(X)) X] = 0, a.s. for all P in some specified distribution set P. where (P) is the ground-truth parameter of interest and h(-) is some vector-valued nuisance functions that we need to estimate from data. Moreover, we require that 0 = (b(P) is the
unique solution to the moment equation. We proceed by plug in estimates of the nuisance h derived from sample Do, and select å satisfying the empirical moment equation EiED m(Z, 9, h(X)) 0 on separate sample D. This procedure is widely adopted in the development of DML-type methods [Chernozhukov et al., 2018], and leads to efficient estimates as long as the moment function m is Neyman-orthogonal, meaning that it is insensitive, under expectation, to nuisance estimation errors. The precise definition will be presented in Theorem 4.1. The novelty of our construction lies in a specific recursive procedure that generates moment functions with arbitrarily high levels of insensitivity to nuisance estimation errors.
Consider the model (1) and let J1 (w, x) be any function of w E IR and x E X satisfying
J1 (w, x) MM an (w) and IE [11(n, X] a.s.
where each IF[a. [X) < oo and each Pa continuous. Without loss of generality, we assume that (w:) = (as one can otherwise introduce dummy summand into the expression (7) with
5

---

a11 (21 0). Let r= be series of functions defined by
(w,, Io (- d)dw'. (,) [, - z/,,
(8)
The following lemma, proved in Appendix E,4, derives the general form of J, for each h: E 24.
Lemma 4.1 (Explicit formula for J,). If M, = M1 + r 1, then
Mr
a: a;-1,--1 (2) Oir (()) for air -B|[, X) X = x]
if1 < <is Mr
(, x) = Σ
ifi= 1,
1, and Pir(w)= = Jo Pi-1--1 (w))dw',
25 i M,
(9)
In particular, for all 2 we have per (w)
We would ideally use the moment function
(10)
estimate lg However, by Lemma 4. depends the unknown data generating distribution via the functions ag-((): Fortunately, our next theorem, proved in Appendix E.3, shows that an estimated moment function (1 based on an estimate of J, yields improved treatment effect estimation rates whenever 0 is identifiable and J, is estimated sufficiently well.
Theorem 4.1 (Structure-agnostic error from estimated moments). Consider the datasets (Di, D2) ) [z)) (2//24))) with each Zi S Z. Define the estimated moment function
m, (7, (x), D1) = [V - ((x) - O(T ((x))) x(x),,
(11)
where h = 4)) and J, (w,,;; 2; D1) - E, recursively defined via (9). Fix any Co. Cg. Cqi Ve, in > 0, y (0. 1), 2 +++,, 1, and A E R.. and let P C P (Ca, Ce. WE, thn) contain all distributions satisfying, with probability 1 - 7/2 U.d. over D1 ~ P. the following four conditions simultaneously:
Mr 0gr (2; D pir ( for åir: XX 2n2 + R and Pir (-)
Ezzr Von, ( (z, 0, ho (X):
[D)]] 2 da (identifiability)
(12)
sup (Ho.
Varz (2, D());D)
6,
D < Vm (finite variance)
(13)
max AE(r.+1)
sup (ha,)
|](²) - (X), (T X;) D1 (P)
S A₂ (finite derivatives)
(14)
sup sup OSiSr
{( (17, D;) X Di]l < ( (near orthogonality)
(15)
where j.. (w, A (ww, Let h ()) be a possibly random function independent of and è be the solution of + Ei=n/241 m, Zi, 0, Di) = 0. Then there exists a constant C, > 0 that only depends on such that for all S A.
R0,1--
P.. Pa. (h, 2))) S C.
/ Vn + Ei ff 1 max max(eb+1), (ee foffledee +efe2) [V V
(16)
[r+1]] (Al(E Corln + Co)et! + re(fe)) Ar],
In particular, if e(5) (max (1, E2 r-)), then the treatment effect enror rate (16) has r-th order dependencies on the nuisance errors c; i = 1, 2 in place of the slower second-order dependencies of DML (see, c.g., Theorems C.I and C.2). One caveat of applying this bound is that dia, Vn- and X all depend on the order of orthogonality and need to be computed in a case-by-case manner. In Section 5.2, we will construct an explicit moment estimator that satisfies e ( O(max(e1,e2) } and make the dependency on r explicit.
The identifiability assumption (12) is crucial and is why the construction of Theorem 4.1 does not work for Gaussian treatments. Indeed, as show in Proposition I.2, if the treatment is Gaussian and (15) holds with (()) + 0, then did 0, so that Theorem 4. cannot yield Vi-consistency.

---

A natural choice for J1 (w, .²) satisfying (7) is J1 (1,,x) = which corresponds to selecting a11 = 0,P 1, 021 = 1, and P21 (w) = w. In this case, Lemma 4.1 implies that each (w, x) the following r-th order polynomial of w:
(w, Σ (x) where aa(x)= = (6-1)((4T-8)
E (-1) -e
II K1)](T) (17) BET
Here, II denotes the set of all partitions of [m] and (2(2) is the i-th cumulant of n, X x.
In particular, if uus (x) denotes the i-th moment of v) X = x, then (2 x) = { (22 and J3(w,z) x) = 1/4 (w2 3uz(z) pa(x)). In Section 5, we will show how to estimate these cumulant-bass J, effectively whenever 77 is non- Con-Gaussian and independent of X.
Agnostic cumulant-based estimation (ACE)
In this section, apply the general guarantee in Theorem 4.1 to derive structure estimators with better rates than DML when 1y is non-Gaussian and independent of X.
5.1 Structure-agnostic cumulant estimation
The moment function induced by the [w, x) defined in (17) requires estimating the cumulants Ky of the noise variable 1 in the treatment regression model T = go (X) + 7). In this subsection, we propose efficient structure-agnostic cumulant estimators assuming that r) is independent of X. We will also see in Remark 5.2 that our approach has potential benefits even when this assumption fails.
Our main result, stated below, indicates that an r-th order error rate can be attained for estimating the r-th cumulant of ry.
Theorem 5.1 (Efficient cumulant estimator for noise with finite moments). Let CT > be constant, 2 2 be a positive integer, and P be the set of all distributions of (X,T) gener- ated from T = (x(
n, such that E[[T X|1/" < CT holds a.s.. The target param- eter Po(P) P + is the r-th order cumulant of 1l - g under P. Let & map P E P to the muisance function 9n- Let 9 X' + R be nuisance estimate that satisfies lØ(X) S Cg and Rür {(x,, T:) ト> TR. be the r-th order cumulant of the empirical resid- ual distribution Pn. 1
n Ei=1 ŌT:-A(x,) where 8, the Dirac measure at 2. Then for any 7 E 7(00,1) and 8 2 r, 0 = k, satisfies Rm,1-- (0:Pa,c0. ()) S C. 1/2 + (2re) where
Cr.- = 1 (1 1)!.
Our next theorem derives a refined bound for Sab-Gaussian treatment noise.
Theorem 5.2 (Efficient cumulant estimator for sub-Gaussian noise). Let Cg.
th Cg.
be a constant
and P be the set of all distributions of (X, T) generated from T = go (X) + 11, such that |go (X) S Gaussian. The target parameter Oo (P) P + R. is the r-th order cumulant of = T go(X) under P. Let & map P E P to the muisance function g. Let 9 : X + R be a nuisance estimate that satisfies [g(X)] S Cg and Rr ={(X,, Ti)le + R be the r-th order cumulant of the empirical residual distribution Pn EE= 8T, -xxx(x))) where 5₂ is the Dirac measure at 2. Then for any 7 E (0. 1) and
Cz. a.s. and r mean-zero, independent of X and Uhn -ub--aaussinn
θ Ry ()) S Cn,,.n 1/2 + (2re)" where = + 12/2,-1/2/
satisfies R,,1-- (0:P..e
Remark 5.1 (Comparing the bounds in Theorem 5.1 and Theorem 5.2). Under the assumptions in
Theorem 5.2, OT in Theorem 5.1 would be OCC₂ + Vrên), so that Theorem 5.1 implies bound
which scales as (cr) 3rr33/22n111 1/2 + (2re)", while the bound in Theorem 5.2 scales as (dr)"n n 1/22 +
(2re)", which is strictly tighter.
Remark 5.2 (Relaxing the independent noise assumption). While Theorem 5.2 assumes that the noise variable n) is independent of X, the estimator Ri can also be of value when this assumption does not hold. In Appendix F.3, we consider r = and derive guarantees for this estimator when n is "nearly" independent of X.
For example, 11g ((11,2,3)), ((1,2), (3)), (11.3), (2)), (12.3), (1)) ((1).22) (2)(
7

---

The fast rates in Theorems 5.1 and 5.2 hold specifically for estimating the cumulants. When the target estimand instead a moment 17, our structure-aae agnnosii lower bound in Theorem F.1 requires Q(n-1/2 1/2 + e²) minimax error for this cubic-quadratic bottleneck implies that, unlike the cumulant-based approach espoused here, the moment-based approach of Mackey al. [2018] cannot attain arbitrarily high-order error rates.
and 1/22 + <²) minimax error for * Notably,
The proofs of Theorems 5.1 and 5.2 can be found Appendix F.2. To the best of our knowledge, these results are novel and may be of independent interest. Next, we will apply these result to construct more efficient structure-agnostic estimators of treatment effects.
5.2 Fast rates with independent treatment noise
In this subsection, we introduce agnostic cumulant-based estimation (ACE), a novel treatment effect estimator that leverages the efficient cumulant estimators of Section 5.1. Throughout, we let R be the empirical cumulant estimate defined in Theorem 5.2 and define
++1
J.(w)
Σ air --1 with āi, āar ([-]][(+--] 1=1
Σ (-1)lm
II BET
k Vi 2 (18)
(-(-) can be viewed estimate of the cumulant-based function J,(+) (17) when X is independent
of 1). The key observation that this J.(-) satisfies (15) with e(i) = Oc]), which follows from the key lemma stated below.
Lemma 5.1 (Key lemma; higher-order insensitivity condition). For all k E [r]
11hh (T - go (x)) = (.- Ereu. BB (MiB) - -
Notably, each term in the RHS in Lemma 5.1 is a product of cumulant estimation errors. Recall Theorem 5.2 we show that the estimation error R, is O(E)) when || gllp,s S él, so that
BE (k|p) Rl)) 0 (IIee e O) We additionally bound the coefficient hidden in the (()) in Lemma G.5. In view of this favorable property, we propose our estimation algorithm, ACE, in Algorithm 1.
Algorithm 1: Agnostic Cumulant-baut based Estimation (ACE)
Input : Nuisance estimates ÿ and ÿ: observations D = {2 12=(0 (X;, Y)) order r.
Output: An estimate of the treatment effect dg defined in (1).
Split the data into
D2 ((X,, Tú //) D2 {(X,, To Y.)) /
2 for 1s + 1 to r + do
L 1 - 2 9(x)))
Define the cumulant-basse function j,(n) as in (18);
5 Define the moment function m2, 0, h(X)) = - -(X) - - j. - g(X));
return& i solution of Ei m, Z,, 8, h(Xi))
The next two theorems, proved in Appendix G.3 and Appendix G.2 respectively, show that ACE can achieve higher-order error rates for treatment effect estimation when vj is non-Gaussian.
Theorem 5.3 (ACE estimation error), Let Ey and dia- Co, CT, Cy be constants and P be the set of all distributions in P2r CT, Cv) with y independent of X and |8,41| 2 8id,r. Then, if
for any 7 (1/2, 1), there exists C.2, > such that for all E1, 62 >
min aj
(19)
ag log Loglagby )
where by log(yn/100), b2 2 loog( (og(CTeT]) 1Te 1/2), and 02 = Cr. then then the r-th order ACE estimator satisfies
50 min (1, Co toar max (2,, 62,
(11)-1/24+)-1, a1 =
R. R0,1-7 (h, x))) S C,rl445, (iez + Cae + 66644477222
(20)
8

---

Remark 5.3 (Power of non- non- Gaussianity). When nj is Gaussian, its cumulant Kr+1 0 for all r, violating the assumption that Ky+1l 2 ö in Theorem 5.3. Conversely, for non- non-Gaassian тn this condition is always satisfied for some by Levy's 's Inversion Formula [Durrett, 2019, Theorem 3. allowing us to obtain higher-order error rates.
Notably, the constant OT in (19) may itself grow with For example, T go( is sub- Gaussian, we can have CT = (VF)). The theorem below makes this dependence explicit and delivers even sharper bound in the sub-Gaussian regime.
Theorem 5,4 (ACE estimation error: sub-Gaussian noise). Let oia, Co. Cg. Ca. t/ > 0 and r E Zay be constants and P be the set of all distributions in P (Co, Cg. Cqi t'g, vn) with vj independent of X and |Ky+1] u. Then, for any 7 E (1/2, 1), there exists Cj O such that Vel, 62 > 0, if
min
hi 1 log a1 ag (ggg))))))))) d1
(21)
{
at
where b, log(mn/9). b2 = 200 min 1, Co - max {eI, E2, (mm)-1/2(()))) + Covn)) ai =
2 log(6(C; vn)e and 02 = 4C2 + vnn) then the r-th order ACE estimator 0 satisfies
(0: (0: (*))
S &Crr!16' T/o 6/ + + + + + thn) + ThE + Corln) (1m) - (22)
Remark 5.4 (Scale of the leading coefficient under uniform noise). As shown in (22), the estimation error of r-th order ACE estimator depends not only on r, 61, 62, n. but also on Kr+1- This is intuitive as Kr-+1 is a measure of non- -on-Gaassianity An estimate of Kr+1 can also be used to estimate the variance of 0; see Section 6 for more details. To understand the role of Sid,y in the bound, consider the case when 1) follows a uniform distribution on [-1,1]. Then for any m E Beg, we have 822m ~ m. [Binet, 1839]. Plugging into (22), we have
4.
Rn,1--,0; (0. S 4r(Ae))) 8 Co2 (CCCC+
(23)
Hence the leading coefficient is only exponential in r, rother than super-exponential.
When r 1, ACE is identical to DML. When r 2,3 it recovers the "second-order" orthogonal estimators proposed by Mackey et al. [2018]. Interestingly, for r = 3, the rate given by Theorem 5.4 is faster than that of Mackey et al. [2018, Thm. 10), as the latter did not establish third-order thogonality. When r 2 4, to the best of our knowledge, ACE is novel, and we derive the explicit expressions for 3, in Appendix G.4.
As concrete instantiation Theorem 5.4, consider the setting of high-dimensional linear nuisance,
(2) = (oro, x²) and (z) = (Bo, x) for ao, 80 E RP,
|loollo and 82 - |1 llo (24)
where (p, 81, $2) all potentially grow with n, and the nuisance functions are estimated using Lasso regression [Hastie et al., 2015, Chap. 11]. In this setting, DML known to provide order 11/2
estimation eтror for 6g whenever the maximum sparsity level max(s) 82) o(n /log p) [Cher-
nozhukov et al., 2018, Rem. 4.3]. Remarkably, as we prove in Appendix G.5, r-th order ACE provides the same guarantee when max(s1, = /(n//(-11/ log
6 Numerical experiments
To evaluate the empirical effectiveness of our proposed estimators, we simulate a demand estima- tion scenario using purchase and pricing data. In this setting, Y represents observed demand, the treatment corresponds to an observed product price, (x) denotes a baseline product price deter- mined by covariates X that influence pricing policy, and the treatment noise 1) represents a random discount offered to customers for demand assessment. Notably, r7 is typically discrete (and thus distinctly non-Gaussian) and independent of X.
We replicate the experimental framework of Mackey et al. [2018, Section 5], where X~ N 7), U[-33, 3, 3)), and follows a discrete distribution on (0.5, 0, - -1.5, -3.5) with probabilities

---

Standard Devintion
Main Squared
64
24
Sample Size (x1if)
Sample Size (x107)
Sample Size (xIIf)
(a) ACE RMSE: V BI( 0.)
(b) ACE BIAS: B
8g.
(c) ACE SD: V Var(O).
1st order
5th onder ACE
Coverage: 94.1% (Expected: 95.0%)
0.0667,
Dondoy
Proba Droslay
Sarmel Didttaaion
3860
Dosn Nit Charna Trus Parssate? Paruzartr
Siice Estimtte
Drtmad
1266
1460
400
Eatimate Value
Eatimate Value
Par Anbotoz Value
(d) First-order ACE estimates.
(c) Fifth-order ACE estimates.
(1) Fifth-order confidence intervals.
Figure 1: Comparison of first through fifth-order ACE estimation (Algorithm 1) in the synthetic demand estimation setting of Section 6. Fourth-order ACE is omitted due substantially larger ror. All quality measures and shaded 95% confidence bands are estimated using 20000 independent replicates of the experiment.
(0.65, 0.2, 0.1, 0.05), respectively. Each nuisance function is specified as a sparse linear function in p 100 dimensions with s 40 non-zero coefficients.
We examine the r-th order ACE estimator introduced in Section 4 across different values r. For 2, this framework precisely recovers the first-order [Chernozhukov al., 2018] and second-order [Mackey et al., 2018] orthogonal estimators. First-stage nuisance function estimates obtained using Lasso regression [Tibshirani, 1996), following Corollary G.3. Complete Python code for replicating all experiments is available at https::/github. /ttpt/ttttttt/gg//// com
In view of the high-probability bounds in Theorem 5.4, we empirically assess ACE performance for orders r < across varying sample sizes. comparison the total RMSE is provided in Figure la, demonstrating that the fifth-order ACE estimator achieves optimal performance. We further decompose RMSE into bias and variance components. Figure 1b compares bias across different orders, with fifth-order ACE exhibiting the smallest bias. Moreover, Figure Ic shows that the first-order ACE estimator achieves the lowest standard deviation, followed by the fifth- order estimator. Figures 1d and le present the distribution of estimated values using first- and fifth-order ACE estimators. Both distributions are approximately Gaussian, with the first-order esti- mator exhibiting substantially larger bias. Based on Theorem 4.1, the variance of 9 is bounded by 5, (Vm/n) 1/2, m/2/, where did provides a lower bound for Ky+1 in the context of Theorem 5.4. This enables us to construct direct plug-in variance estimate Evar as Evar R Vi for Vm (x ( ((x))) - ((x.)) Lastly, following Corollary E.1, we construct the approximate 95% confidence interval _ - 1.96E1/2 e + 1.96E for du. Figure If demonstrates that approximately 95% of independent experiments yield confidence intervals that contain the true parameter value, confirming the validity of our constructed intervals.
(Y E
7 Conclusion and future directions
In this paper, we provide new insights into how distributional properties could change the statistical limit of structure-agnostic estimation. Focusing on partial linear outcome model. we show that the Gaussianity of the treatment variable creates a fundamental barrier for improving over DML, while improvements upon DML is possible for non- Gaussin treatment. Moving forward. would be of interest to exploit distributional properties to design estimators more efficient than DML for heterogeneous treatment effects.
10

---

References
Alireza Ansari. The gauss-airy functions and their properties. Annals of the University of Craiova- Mathematics and Computer Science Series, 13(2):119-127, 2016.
Sivaraman Balakrishnan, Edward Kennedy, and Larry Wasserman. The fundamental limits of structure-agnostie functional estimation. arXiv preprint arXiv:2305.04//66 2023.
Alexandre Belloni, Victor Chernozhukov, and Christian Hansen. Inference for high-dimensional sparse econometric models. arXiv preprint arXiv:1201.0220, 2011.
Alexandre Belloni, Victor Chernozhukov, and Lie Wang. Pivotal estimation via square-root lasso in nonparametric regression. The Annals of Statistics, 42(2):757, 2014.
Gérard Biau, Lue Devroye, and Gäbor Lugosi. Consistency of random forests and other averaging classifiers. Journal of Machine Learning Research, 9(9), 2008
Jacques Binet. Mémoire sur les intégrales définies eulériennes et sur leur application la théorie des suites, ainsi qu's à l'évaluation des fonctions des grands nombres. Journal de l'École Polytech- nique, 16 (27):123-344 1839.
Stéphane Bonhomme, Koen Jochmans, and Martin Weidner. A onymmaanottoooggnalizaaiioo approach to the incidental parameter problem. arXiv preprint arXiv:2412.103044 2024.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
T Tony Cai and Mark G Low. Testing composite hypotheses, hermite polynomials and optimal estimation of a nonsmooth functional. The Annals of Statistics, pages 1012-1041, 2011.
Xiaohong Chen and Halbert White. Improved rates and asymptotic normality for nonparametric neural network estimators. IEEE Transactions on Information Theory, 45(2):682-691, 1999.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural pa- rameters: Double/debiased machine learning. The Econometrics Journal, 21)1), 2018.
Victor Chernozhukov, Whitney K Newey, and Rahul Singh. Automatic debiased machine learning of causal and structural effects. Econometrica, 90(3):67-10027, 2022.
Rick Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019.
Samuel Durugo. Higher-order Airy functions of the first kind and spectral properties of the massless relativistic quartic anhurmonic oscillator. PhD thesis, Loughborough University, 2014.
Saso Džeroski and Bernard Ženko. Is combining classifiers with stacking better than selecting the best one? Machine learning, 54:255-273, 2004.
Max Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and infer- ence. Econometrica, 89(1):181-22) 2021.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical learning with sparsity. Mono- graphs on statistics and applied probability, 143(143):8, 2015.
Kosuke Imai and David A Van Dyk. Causal inference with general treatment regimes: Generalizing the propensity score. Journal of the American Statistical Association, 99(467):854-88 866, 2004.
Jikai Jin and Vasilis Syrgkanis. Structure-agnostic optimality of doubly robust learning for treatment effect estimation. arXiv preprint arXiv:2402.14266. 14264, 2024.
Lester Mackey, Vasilis Syrgkanis, and Ilias Zadik. Orthogonal machine learning: Power and limita-
tions. In International Conference Machine Learning, pages 3375-3383. PMLR, 2018.
James Robins, Eric Tchetgen Tchetgen, Lingling Li, and Aad van der Vaart. Semiparametric mini- max rates. Electronic journal of statistics, 3:1305, 2009.

---

Peter M Robinson. Noot-n-consisten semiparametric regression. Econometrica: Journal of the Econometric Society, pages 931-954, 1988.
Leonas Saulis and VA Statulevicius. Limit theorems for large deviations, volume 73. Springer
Science & Business Media, 2012.
Anselm Johannes Schmidt-Hieher Nonparametric regression using deep neural networks with relu activation function. Annals of statistics, 48(4):1875-1897, 2020.
Charles Stone. Optimal global rates of convergence for nonparametric regression. The annals of statistics, pages 1040-1053, 1982.
Vasilis Syrgkanis and Manolis Zampetakis. Estimation and inference with trees and forests in high dimensions. In Conference on learning theory, pages 3453-3454. PMLR, 2020.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.
Alexandre B Tsybakov. Introduction to nonparametric estimation. Springer Science & Business Media, 2008.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Lie Wang, Lawrence D Brown, T Tony Cai, and Michael Levine. Effect of mean on variance function estimation in nonparametric regression. The Annals of Statistics, pages 646 6666-66442 2008.
Shandong Zhao, David A van Dyk, and Kosuke Imai. Propensity score-based methods for causal inference in observational studies with non-binary treatments. Statistical methods in medical research, 29(3):709-721 2020.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Sociery Series B: Statistical Methodology, (7(2):301-320 2005.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: At the end of Section we directly point to main results.
Guidelines: See the main contributions listed at the end of Section 1.
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The main limitation is the assumed independence of noise, and this dis- cussed in Remark 5.2.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [es]]
13

---

Justification:
We explicitly point to the proof after stating each theorem.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details provided in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
The open-source code is provided in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: /ntps.ip...oo
publ c guides
des Sites/Cooo Coodeuuome ss coiiooollcyy cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cc/ cc/publiccccccccccc public// e/guidies CCode SodeSubbr Submi s sessonns conPolicy" for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These are discussed in Section and Appendix H. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars and histogram under multiple runs are provided in Section 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The experiments are of small-scale and can be run on a laptop a reasonable amount of time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics aattps://neutt
Answer: [Yes]
Justification:
We have checked the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This paper is mainly theoretical and does not have direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswl thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
that the method development in this research does involve LLMs as any important, original, or non-standard components.
The answer NA
Please refer to our LLM policy (https: //t/////uiiieeee ps wct.co/eeeen/cc/////////cco 2025/LLM) for what should or should not be described.
19

---

Appendix
Table of Contents
A Preliminaries
21
A.1 Semiparametric bounds
21
Useful properties of sub-Gaussian distributions
21
B Proof and discussion of Theorem 3.1: Structure-agnostic lower bound for binary
treatment
B.1 Some remarks on the constants
25
B.2 Discussion of the constant Cg
25
C Proofs of upper bounds for DML
27
C.I Proof Theorem C.I: Structure-agnostic rate of DML
27
C.2 Proof of Theorem C.2: Structure-agnostic upper bound with known treatment noise
29
Proof Theorem 3.2: The Gaussian treatment barrier
E General upper bounds under Neyman Orthogonality
38
E.1 Estimation error of general moment estimators
38
E.2 Proofs of Theorem E. and Corollary E.1
39
E.3 Proof of Theorem 4.1: Structure-agnostic error from estimated moments
41
E.4 Proof of Lemma 4.1: Explicit formula for Jr
42
F
Technical details in Section
F.1 Proof of Theorem F.1: Structure-agnostic limit for estimating residual moments
43
F2 Proofs of Theorem 5.1 and Theorem 5.2
44
F3 Relaxing the independent noise assumption
49
G Technical details in Section 5.2
50
G.1 Proof of Lemma 5.1: Key lemma; higher-one order insensitivity condition
50
Proof Theorem 5.4: ACE estimation emrrr: sub- Gabbsia noise
G.3 Proof of Thcorem 5.3: ACE estimation error
58
G.4 Special cases of the ACE estimator
59
G.5 ACE estimation error for high-dimensional sparse linear regression
60
H Additional experiment results and discussion
61
More results for orthogonal machine learning
62
L.1
Construction of orthogonal moment functions
1.2
Proof of Theorem I.I
63
1.3
Example: heteroscedastic nonparametric regression
65
1.4
Technical details in Appendix 1.3
66
1.5
Proof of Corollary 1.1
67
1.6
Comparison with Mackey et al. [2018]
68
1.7
More discussion on Assumption E.
68
20

---

A Preliminaries
A.1 Semiparametric bounds
Our proof' of lower bounds are based on on the method of fuzzy hypothesis. A key lemma is stated below:
Lemma A.1. (Isybakov [2008], Theorem 2.15) Let Z be observation with supp(2) 2, P
P. Pi P and a be a probability distribution on Pı, which induce the distribution (A =
SQa ('Qn(()dss (A)d AA C P. Suppose that there exists a functional T P + IK which satisfies
T(P) c. x(QQ: T(Q) 2 c+ 2s}) =
(25)
for some s > 0. If H2 (pon Q1) 5 < then:
1 5E(1 - E/44
inf 1:20- PEP
sup P f ((P)
We will use the following lemma to bound the Hellinger distance as required in Lemma A.1
Lemma A.2. (Robins et al. [2009, Theorem 2.1 see also Jin and Syrgkanis [2024, Theorem 4]) let Z = 2 be measurable partition of the sample space. Given vector X = (AI...., in
some product measurable space A A1 x ... x Am, let P and Q) be probability measures on Z such that the following statements hold:
P(2,) = Qx (2,) = Pt for every X E A, and
the probability measures P and Qx restricted to 2j depend Xj only.
Let p and qx be the densities of the measures P and Qx that are jointly measurable in the parameter and the observation x, and be a probability measure on A. Define m max; Sx, jo>=ri dy. Suppose that p (x) and that max max; Pj S A for all j for some positive constant A, then there exists a constant C that depends only on such that, for any product probability measure 11
p
/ Qtnd TH 0 S maxx; Pj Cn?b?
H pmn
A.2 Useful properties of sub-Gaussian distributions
In this subsection, we recall a few useful properties of sub-Gaussian distributions. Recall that for variable Z. its Lub-Gaussian norm is defined as
1 = inf {c 0 Eexp 7 (
(26)
Proposition A.1 (Moment bounds for sub-Gaue Gaussannn variables, see e.g. Vershynin [2018] Proposi- tion 2.5.2), Suppose that Z is mub-Gaussian random variable with Orlicz norm o = Z w. Then for every integer
equivalently where C' > O is an absolute constant; one may take C' 2
1 1/k <
E
Covi,
E|Z (Covk)t
The following bound of cumulants is due Saulis and Statulevicius [2012], Lemma 1.5. Proposition A.2 (Cumulant bounds for sub-Gaussian variables). Let Z be centred sub- Gaussian random variable with Orlicz norm o = || i.e. I6 E I exp(o222 (2/2)) /2) for all t E R. Denote by R2 (Z) its r-th cumulant, r E N. Then
J (Z) S (r (r- (1-1)
>
In particular, the sequence (2)(2)) (Z) grows at most like 20, which sharp up to the constant 2 (no smaller absolute constant works for all Sab-Gaussians),
21

---

B Proof and discussion of Theorem 3.1: Structure-agnostic lower bound for binary treatment
In this section, we present the proof of Theorem 3.1. We define the following data generating distribution:
X (f-,,+((()++))))
i= 1,2, ,M
X ~ Uniform(A (A)
(27)
T| X Bernoulli(((z))
x = ~~ Bernoulli (amooullif((()(>)))) + +0).
c and S g(x) S by assumption, it is easy to that f(x) + 0t [0, 1], {0, 1}. Hence (27) defines valid data generating distribution. We denote the joint distribution of (X, Y) as
where 0 joy and f(x) = - 1 0g(2). Since c S ((x) E -1-c
q(x)
Since 7 E ( 1), there always exists some Ç (0 such that 1- (-1 7. Let m 2 BCC
Cn² be
2
a positive integer and B,, 1, 2m be partition of the covariate space [0, 1] such that each set has a Lebesgue measure of 2m
I We also define
X; Uniform((-1,+11))
i= 1,2, ,M
X ~ Px = Uniform(2()
(28)
T X Bernoulli(gx==))
Y = +emnolli(f((=)+001)
where
8 = (1- -1-2)-
(x(x) = g(z) + é -
(29)
(x(x) = ((x) -
fx(x) = (x)) (x)
and
M/2
A()x)) = Σ (1 Caff 1 fo E B2y)).
Let PA be the joint distribution of (X,TYY induced by (28), μ be the uniform measure over X' X
T x y and pA du
dPA, From (28) we can derive the expressions px px(x, t,y) as follows:
px(x,t,0) gx(x) (1- (f2()) (1 fx(x) 0lt) 1-2
(
(30)
22

---

Specifically, we have
px(2, 1, =x(()) (fx(z) +0)
=x(()) (1x +0 (1 9x()))
- =1)20(()) (((1)()()))))(((((11((1((()()))) + (Q(=)) (()) (1
- [(x)))]
+((((2)) - -20(() + -2 -
=(x))(x)) 05(z) ((x))
(31a)
((1((2) - ē2ğ(2) + -
+ 2(1(()) v {
+
(31b)
1x(x,1,0) = 9x(x) - (x,,1111
2(z,1,0) (ā111 ((x)) + +z((() - + 2()(()) -
(31c)
0, (1 - (=)(()))
=(1- (x(x)))x(x) - 0f (x)
=qx(z) - 11,,,1,1)
= =, (ε1((z) + ()x1)))))) + - -21((z)) (())) x))
(31d)
px(x, 0,0) = - (x) px (z. 1) = 0,0) (1 ((x)) - (1-))))))) - 81 201(()) vo(z)(1 A(X))
(31e)
Crucially, all the px pss are linear functions of A(), x))
Lemma B.1 nI?-norm bounds). Let ē1 A-1/28- 1/2 1//2,,,,, A-12228-1 then if c2 $ 6 holds that
2
0 < 9A. 9A 11,
Ilax - 01l12(px) Se and 1la - -Qllze(xx) S22.
(32)
As result, Px P2, (h).
Proof: By definition, we have
llex (l)x) (X))) - = -
Similarly 11 - ill (2()) E2. Hence it directly follows that PA (h). By our assumption,
((x)(1 - ((x))= 2 A 1-15, so that ) 2 5 2 2ी and
ga (x) 2 0(x) - é / ((z) ((x)) 2 (0(z)( g(x) - -2) 2 0.
By a similar argument, one can show that gx (x) and also qu (z) E (0, 1). concluding the proof.
To apply Lemma A.1, we now use Lemma A.2 to bounding the Hellinger distance between por and f p@ndd(A)) (A). We recall the following lemma:
Lemma B.2 (Joint density lower bound). If (1 62 S we have px(x,1,8) 2 1 v(r,t,)) X' x T x V
Proof: Note that f(x) ((x) -0(z) [ 19.1-9 - and
8 (x)(z) - f(x) S + 0'e, S 7.
so we have that (x(x) E [ 1 - 3] Similarly, one can show that fa(x) + 0' E [ 4] By
Lemma B. (x) E [g. ]. so the conclusion directly follows from (30).
Lemma B.3 (Hellinger distance bound). For any Ç >> O, long M 2, it holds that
(psn, / ppmdd(()) S
23

---

Proof: : Let 2 = X' x T x v/ where T = y= +-1,+1}. We apply Lemma A.2 with A; 2j {-1,+1},
distribution on (B2 (-1, P P, Qx = Pa and π being the uniform Firstly, since X; Uniform((-1,+ +1)), we have (A, = 0 for
any fixed x E X, so that (31) implies that Ex Px = P.
By our choice of Bj, we have P(Z,) = PA(Z,) = 1 for all j. so we have that = 1. m
Notice that
(px(e, (, (())
b S m maxx p (2)) sup sup
p(x,t,u) v)
(a.tvlE2
(px(I, t, v) - ((,,t,))
sup sup
p(z,t,y)
à (2,t,))E2
where we recall that u(z;) + since μ the uniform distribution on Z. When t = = 1, we have
(px(x, 1, v. p(x, L, u))
(10))())))
((x)(((x) + 0(1 [(x))) S <01
t,
where the last step holds since ((x)) + 20(1 - ((x)) 2 0 by our choice Ū and ((x) 2 Similarly, we can deduce the same bound for (t, v) E {(0, 1), (1, (1, Hence we have that b S 0-1
We can choose A = 0-1 and n max (1, b} max, Pj S 0-1nmm- S A is satisfied. Therefore, by Lemma A.2 we can deduce that
H H (pSn)
/ P ( dm())( < Cm n2² - Cθ 2m 1m Ss.
where the last step holds since 2
Cn 82c
Now we are ready to apply Lemma A.1. We choose [Px A e [-1,+1]] be the uniform distribution on Pl, and T that maps each observation dis- tribution P generated by (1) to the corresponding 0. Then we have that
(X, P.P =
T(P) = 0, (1Q :(Q) = 0 23) =1,
where & = (0' + 2162 = - + +162) Moreover, our choice m and Lemma B.3 dr(()) (Q)) $ Ç. Therefore, Lemma A.1 implies that for any together implies that H² (p8n, .f Q dr
estimator f", it holds that
sup P - T(P) 2 > VC(1 ([]) PEP
2
Equivalently, we have
2 1 A A 1- + +112).
M (2)
(33)
We now proceed to prove the n 1/2 component of the lower bound. Define
q(z) ((x) -10 Ø -
(34)
and let P be the distribution generated by Px, T X = Bernoull(((()),
Y x, T = t Bernoulli(((2) ((x) + Ōt)
and p(z,t, t, y) be the density. Then we have that
p(a, 1, ((x)(((x) + 0(1 ((z)))
-(x)(((=)+ +
((1 g(x))]
p(x,,,1)) - V
so that
(0(x, 1,1) p(x, 1,1)111))
2 ((x) (1-)(x)
- (1
$(x,1,11
((I) + 0(1 ((x))
24

---

We also have that
pla, 1, - [(x){ ((x) - 0(1 - [(x))
((t){1 ((x) O(1 - ((z)) (z)((=((((1___//0)))))1))))) ((x)) c(11 -((x))]
p(r, 1,0) - c0(x) -1z) (2 (/(()) + v1- g(x)),
so that
(p(2, 1,0) - p(x, 1,0))
2 ((z) - 0(x)) I 4cyy 3 (()) ;(x)),
p12 1)
((x) - 0(1 9(x))
where we use 0 = c/ and (xx) 2 Co in the last step. Similarly, one show that
(f(2, 0) p(z, 0, 0))² (p(æ, 0, 1) p(x, O, 1))2
I 4c 2g=)) g(x)).
((z, 0,
p(zz 0, 1)
Combining all the inequalities above, we have
x x2 & 10c, e'E(g(
((x))] = 1
12..
1/2, By choosing 0.1Çc 1/2 8- 11/2, it holds that
H (p@r, (ea) S nH(P,, P) s nx (P, sc
so Lemma A.I directly implies that Therefore, Lemma A.I implies that for any estimator T, it holds that
vC(1 c/4) = =
sup P - 2 c/22 2
PEP
2
Equivalently, we have
- (P2c 2 e/2 = 0.05 o/ 2/228-11//// 1022
(35)
Combining (33) and (35), we obtain the desired result.
B.1 Some remarks the constants
Notice that the assumptions that we make for deriving the upper and lower bounds are not exactly the same. Here discuss some important aspects of their differences.
Remark B.1 (Assumptions on the uniform overlap). Compared with Theorem C.I, we make the additional assumption that 9(z)(1 - 9(x)) 2 A 8 which we will refer to as uniform overlap. Equivalently, this assumption states that there exists some 51 = ((A-18) such that 8n S g(x) S 1 81. This assumption is also made in previous work [Jin and Syrgkanis, 2024], albeit for different causal estimand. If 82 is treated as a universal constant, then we can choose A o and the expected overlap assumption is satisfied. In this case, our lower bound has tight dependency on both e,, = 1, 2 and ò. However, the error rate of DML does not match the lower bound in if is not uniformly extreme. The minimax optimal rale is unknown in this regime.
Remark B.2 (Assumptions on q())). Compared with Theorem C.I, another additional assumption that make is that Gg q(x) 1 Cg in Theorem 3.1. This assumption is also needed in the previous lower bound established in Balakrishnan et al. [2023] for E E(Cov(Y, T X)}. Interestingly, this assumption is not needed for upper bound. The c term in our lower bound (5) corresponds to the Cee term in our upper bound, where we recall that Ce is upper bound the ground-truth 0o. In fact, the overlap assumption on ((x) also implicitly imposes a constraint on the magnitude of the ground-truth parameter Oa: more discussions be found in Appendix B.2.
B.2 Discussion of the constant Cq
For any pair of functions (g.g) that take values in [0, 1), we define their cross ratio to be
q(x) 1 x(x) min min g(x)" I (x)
g(x)
((x)
(g. q) = max min min xEx
xeX
g(x)
g
First, note that Theorem 3.1 can be slightly strengthened follows:
25

---

Theorem B.1 (Strengthened binary lower bound). Let Cq. 8 E (0. 11) and P be the set of all possible P's generated by (1), such that the variables T.Y are binary and that the marginal distribution of P on X is Px. Lef ① be mapping that maps distribution P E P to the nuisance functions range(D). Then for any 7 (1/2, 1), there exists a constant Co > 0 such that for any 2 6/2, 1,2 and any estimates = (Ø, Q) with Epx (1) - -(X)) = 28 and
ho [go: qo) E
g(z)(1 ((z)) 2 A 8, Vx E X. we have
-m
(coch + c222),
(36)
where Y is universal constant that only depends on 7, and
Cy sup { ()) llg - (llzA(Px) S /1122, @ ( S 5e2/2).
Proof: Without loss of generality, we assume that
{ [1- g(x) q(x)
(g = min min
=EX
g(x) g(x)
For any h (g. 9) P2, P2.2/2 (h), note that
P2)/2(h) S P2,,
so that
Mm,11 y (P2c 2 Mn.11~ 2 (P2./2(th))
Now suffices to show that
Mn,1-> (P2.//(h)) 2 G. 2
6162)
(37)
Notice that this lower bound can be derived with exactly the same argument as we employed in the previous section, since the only place that we use the assumption e S ((x) I 1 Cg is that
(xx) - 20g(z), ((x) + 20(1 - g(x)) E [0,1].
Now we replace 9, q with g. q respectively, and choosing Ô = co/2 ensures that the above relation- ship holds. Therefore we obtain the desired lower bound.
The upper bound side can also be strengthened by replacing Co with
C = sup | (g. q) llg - 01lxa(*x) S E 11 (llea(Px) S {2)
The main insight is that the nuisance estimates already tells us that |0| S SC₁-
Theorem B.2 (Strengthened binary upper bound). Ler > 0 and P be the set of all distributions
P of (X, (X) generated from (1) that satisfies [ 9o( 22) 2 8 and T,Y Y are binary. Let T be mapping that maps each P E P to (9u. (o) E H = range H)). Then there exists constant
no (8) such that when 2 no- for any estimates h (g. 4) and any 7 E (0.1). the DML estimator 0DMIL derived from the moment function
m( 0o I(X)) = (Y x(X) Oo(T - (00T-9(X)110 (( h h=
(38)
satisfies
R Rn,1-, (dDML: P2.c S A, + c162) + (C + 62) + 8-112) -1/2]
for any 7 1), where A,, a constant that only depends on %
Proof: : We prove the theorem by applying Theorem C.I. First, since T and Y are binary, we can take C = 1. It remains to show that for any P E (2)c(h). the corresponding 6 is bounded by C%- Indeed, let (x)) Ep[T X x| and (z(x) Ep(T X 2:), then l|go 0llx)(xx) $ cj and
l1go (ll),(px)) I Eg. Moreover, note that
4o (x) (g)()), (x) + 0(1 go(x)) E (0, 1}, Va EX,
so we have that
Oo S (go. go) s Co,
concluding the proof.
26

---

C Proofs of upper bounds for DML
In this section, we present the formal statements and proofs of Theorem C.1 and Theorem C.2. Both results are already known in the literature, and we present here the explicit structure-agnostic rates for completeness.
Theorem C.1 (Structure-agnosti rate of DML). Let 8, Co, CT, > Po (Po E (C CT, Cy) Ep, [T 9o (X))² 8). and t Φ*. Then for any estimates h (g. ÿ) and any 7 E (0, 1) such that (g(X)) & CT a.s., the DML estimator UDML derived from the moment function (38) sattifies
(m.11-(0DMM) P2. (h)) - 48 48 (Co (ggg) C, + Cx)cg + Cyd 1y8-1/2nn-11
for any 7 E 10 and 2 15ct/2 11/2, where City a constant that only depends on 7..
The proof of this result follows the standard arguments in the DML literature and can be found in Appendix C.I. Existing theoretical guarantees largely focus on establishing sufficient conditions for achieving Onn -1/2) rate and establishing confidence intervals [Chernozhukov et al., 2018, 2022], while here we make the dependency of the error on G; 1,2 and ó explicit, which would be of interest when the rate is slower than -112 The existence a constant >> O that satisfies the the overlap assumption and is widely adopted
assumption in Theorem C. is commonly referred in the DML literature. The estimation error still be large if ó is small compared to Eg. Eq. When the assumption on is violated, i.e., 5 = so that DML is not better than the naive estimator 1 1 X₂) - 900,X,)) X.)).
the second term in the upper bound becomes (2(gg),
On the other hand, since is known, the following upper bound can be easily established for modified version of DML,, using the moment function
m (Z, Bo (X)) = (Y qo (T)) go())) 0
(39)
Theorem C.2 (Structure-agnostic upper bound with known treatment noise). Let Φb H. h be defined + + < 1, then the estimator dDML derived from the as in Theorem 3.2, and $2 > O satisfy s1 moment function (39) satisfies
Hi- R1-((ODLLL Pho(h)) S C.yy 2012 + (Coo 2a + o 10
for any y 1). where Ciy constant that only depends 7/-
Theorem C.2 can be derived in a similar way as Theorem C.I; the proof can be found in Ap- pendix C.2. Since it considers a simplified setting where the treatment variance is known in X, the of term in Theorem C.I induced by estimating the treatment variance vanishes in the current upper bound.
When the nuisance error of go small, i.e., <2 S Co. the upper bound matches the lower bound derived in Theorem 3.2 up to logarithmic factors. Moreover, if the estimation error eg of go is polynomial in n, then they differ by only constant factor, implying that DML minimax optimal.
C.1 Proof of Theorem C.I: Structure-agnostic rate of DML
Given data Ti,,, Y:) the DML estimator is defined by
0DML = 12 " (T: 9(x)))
n Σ (Y, - ((X,)))T - ((X.1)
i=1
1=1
Let 1/. T; 90 (Xi), Y, - -go(x.))
-O0Ti Ag = and A, = then
0P.I. = n Σ (Xi)) - mm)²
(A,)))) - -oo: mi))
n
i=1
For any 7 E (0, 1). by assumption there exists some constant N₂ such that for any n 2 Ny, we have
8² > 1000y-'n"
(40)
27

---

Since (A)(X)) are ii.d random variables with g = EXA, X = 0 and
(X, = (XA,
x.]] - A 4C9
have
IP 1/2
Ag (X;)ma - 2ACTT 2 21
i=
by Chebyshev's inequality, where A = 0.17 1/2, Similarly, we also have
P
12
1/2
(X.mm s 2ACTe,
2 21 PP
n2
1/2
Σ
21-A72
From
Ele,nh - 40
we deduce that
P n 1/2 Σ 4mh S 22A
2 11
Since B ([]Ag(()AA)) (X)), S EgEy by Cauchy-Schwarl inequality, Chebyshev's inequality again im-
plies that
P (x) cgCo + An A1
2 21-A-2
with similar reasoning, we have
P
,x) s + An & A1
21-A-2
Lastly, since E[n!] by assumption, we also have that
[1 1[7"
P
Σ
n - B[n7 - 2 E
1 4n
10-8CTn-118
-1822 1 1-0.447
2
so that
IP
R 2
2
Bin")
2 1 - 0.047.
1=1
Let be the event that all the above high-probability bounds hold, then
P[
N 5 0.047 - 7
Under E, we have
1 - 2ACT 11/2 > 4 Ely"),
n Σ 9 (X.) n²)²
AA,
since E[(r²) 2 8 by assumption and (40) implies that z 4ACn-112 Moreover,
Σ
- - - -, mH)
< 10 2 EgG An ", / (x)
+ -C
It is easy that (X)A,X)) 4 4C and (X), 4C As result, we can deduce that
10MLL Oll
-1
n
n1 (x,) -1:))
n "' - -;; - (x.) mm))
< <48- +C +gg5g) + 8...0 concluding the proof.
+ CY)e, + + CC0-1122nn-1/2
28

---

C.2
Proof of Theorem C.2: Strusture-agnostic upper bound with known freatment noise
Since the variance of 1) X assumed be o² where a is known constant, we have
(X,) (ox) mi)
0DML 0 0-
n
so that
lDML- 8o
nu o² ))
=o- n (x,)) -)) 7.) 90Ag(X;)m; + 0o
n A-'Ynf i=1
Note that the high-probability bounds for each term in the above expression can be obtained with similar arguments in the previous subsection, with C @(o) and 5 a². Hence it is straight- forward to deduce that
10DML Oul S Rege + AC 2g + +o to
D Proof of Theorem 3.2: The Gaussian treatment barrier
Our proof is based constrained risk inequality for testing composite hypothesis developed in Cai and Low [2011].
Lemma D.1. [Cai and Low, 2011, Corollary 1J Let be an observation with distribution E P and P 0, 1 be two subsets P satisfying Pi UPP22 = P, and μ4 be some distribution supported
on P. Define
f i ((P) uf == (T(P) - mm) (u(dP) m
(41)
to be the mean and variance of functional P R, F, be the distribution of X with prior pa and be its density with respect some common dominating measure μ.. Then for any estimator ((X) ( we have that
(Im] -mol - 1)T)
sup Ep ((P)) PEP
(1+
{
[(x)) Fo(X) - 7)
1/2
as long as [m, - -mol 1 2 0, where
the x" -distance between
Fo and Fi.
To apply this inequality, we construct the null and alternative hypotheses as mixtures of data dis- tributions with matching moments. While moment matching techniques widely adopted in tablishing minimax lower bounds, the structural nature of our causal model (1) brings additional challenges our construction. Unlike most existing works where only moments of a single vari- able need to be matched, here we need to match moments that contain two variables: we seek for distributions Vo, V1 over Pa.a (h) with corresponding mixtures Po. P, respectively, such that both
Ep |(Y Ep, XT)(T ET. XI) EA [(Y Ep DY XJ)(T EA (
and
EA [[T ER, x])*1= Ey, [[T Ep, [T') x])
hold for 1: = 1,2, kn- This would imply that (Pol small, which further implies that [ (((n)) is also small, where penn = po = 1,2 and pe is the n-fold product distribution.
To apply Lemma D.I, we need to show that there exists a sufficient gap between mg and mi which correspond to the expected value of 0 under un and respectively. Our key insight is that, for Gaussian treatment, there is no need to match E [(Y - E (T E[T X]) since this term always vanishes. This fact is due to recursive property of the Hermite polynomial Hz(x)
29

---

(z)/ (=))(()))))) (where ( is the Gaussian density); we will elaborate on this connection in Lemma D.6. As result, we can construct mixtures of distributions that are close in terms of x2- distance (Corollary D.1) but their average values of the E [(Y - E X])(T [ X]] term are well-separated. Given the structure-agnostic oracle, this separation can be as large as ()) and it further induces a separation between mi and m at the same scale, yielding the desired lower bound.
In the following, we present the full proof of this theorem.
The following lemma turns out to be a useful tool for moment matching in establishing our lower bounds.
Lemma D.2 -dsttann to univariate polynomial bases). Let qR be a linear space of polynomials Ih E a;\": where uy E 10, ,k] \ {1}, and òg be the Lixo Tiso-distance of
on [-1,1] in the form of
a(A) X to Bz. Then 0ū 213
Proof: Let b(A) E F2 be a polynomial that satisfies ||a bllz. = 64. Define r = a b, then
satisfies ||||| òg and r0())= = 1.
Since deg(r) k. the Lagrange interpolation formula implies that
2k
1ixxi( xz))
r(x) = r(xx))
II y (z; xz))
1=1
for any 2ạ E [[,,,1,,15 2k. Taking the derivative of both sides, we obtain
E1et I1 xz3) 11 (x: - xz))
r(x) = Σ rxx;)
In partieular, choose
k: + i 1sisk
k
24-
k ++1sis20 2k.
k
then it holds that As result, we have
1o 23
(k!)²
2k:2
k (k:1)
7 [k: - i)l(k + i)! S k.
jæ (2t xz))
i1
(k- i)!(k + i)!
24 S
11
1=)((0)
6%
$ 22h.0
(x; xz))
Hence 8R 2 24 as desired.
Lemma D.3 (Loc- distance to bivariate polynomial bases). Let Bt.1 be a linear space of polynomials
on [-1, 112 of the form E uu'" p": where (u,, vz) E {0, ... 1., k} x {0, \ (),, and δg, be the Tose dii-taitecee from an (>, p) = Ap to Br.1- Then 5g,1 2 1
Proof: Assume the contrary holds, i.e. δx,1 < f6.. then there exists some by (A, p) E Bx.1 such that [|b, - a1 [1 < 5k.. By definition, there exists polynomials 7ı E Bu and 81 such that b, (A, p) px1(A) + $1 (A). In particular, setting م = and p -1 implies that ||r +
All... < 2 5
and ||r: 31 -A < & The triangle inequality implies that ||r All Al < Sg. which is contradiction to the definition of 6g. Thus the conclusion follows.
Lemma D.4 (Separation of measures under matching properties). There exists two probability mea- sures Vo and V1 on [-1, 1j2 such that
I a((a, (lvo(aad))) ff a(2,, (dx)p), Va E Px.1
(42)
and
I (xpr((ssp)) - f Apen 2 41.3
(43)
30

---

Proof:: The proof is similar to that of [Cai and Low, 2011, Lemma IJ. Let C[[-1, be the space of continuous functions on [-1. 1] equipped with the Lx norm and be linear space spanned by a1 (X. p) Ap and PR.1- Define a linear functional T that maps any f == e and g E F₂) to cdg,1, where 5R,1 defined in the previous lemma. Let 91 E PR be the best apprrxxxxiaaio of d1 in Px. then ||a 9ılle = ōg, and T(aj 91) = 8k,1 so
+ g E F (where
||]|| 2 1. On the other hand, for any f = =ca + g E F. we have llflloo 2 |c|ōk,1 since otherwise the Lee distance between d1 and -c- g would be smaller than 5g,1, which is contradiction. Thus [T()) = |c|ōk,1 I | which implies that ||T'|| 1
1. By Hahn-Banach theorem, T can be extended lin- ear functional on C([-1,1) 1 with unit norm, which we still denote by T. The Riesz represen- ter theorem then implies that there exists signed measure is with unit total variation such that T(f)= J fdu, Vf E 1(3-1,13)) In particular, we have
Therefore, must have ||||
I als 0, Va Ba.1 and f Apday 6k,1
Finally, by the Hahn decomposition theorem, there exists (positive) measures vo, VI such that μ Po V1. Then it is easy to see that such and V1 satisfiy the desired properties, concluding the proof.
In the following, we provide the full proof of Theorem 3.2. For any A > 0 and E (0, 1). we define a two-piece Bernoulli distribution. denoted by (q; A), as a distribution with PDF
A- A (1
[0, A] x E (- otherwise.
p(x)
9)
It is easy to see that such a distribution has mean (q (0-1)) 2) A.
To begin with, note that we can assume that [Ø(X) S CT - e and l((()] S C. - 22/2 without
loss of generality. Indeed, since [x( S CT and [((X) S C.. there exists = (g(-),(())
satisfying ll 7 S e1/2, l اl1 -= S c2/2 and lύ(X) S CT
l/(x) < Cq E2/ Then, (h) S (h). and any lower bound for P (h) also applies to (h). implying
the desired lower bound up to constants. In the following, we will replace h with h work with the uncertainty P Ph).
Now, let = C.. Q= Cq c2/2 and G = C- c2/ Let k, > 0 be some even integer that will be specified later, and Vy. 11 be the corresponding distributions in Lemma D.4. For any X, P E RR we define the following data generating process:
X Px- Uniform(X)
N (xx(X),,²
1
(44)
B2lq: 1A).q= 4A
(2A+ 0xpT fx,(X))
if 10x + fx)(()) SA
N (0x,TT (x,(
otherwise
where
(x(x) ((x) + %oSs
(x()) = ((x) - E2P
Ox, = E1ē20 1 AAp
[x (x(x)) (x))
By choosing ěι = -0-///44 and é2 = e2/4, we can ensure that for any h2 (gx, qA) it holds that hy E (h))/2(h), Vs E [1,++00 +00|?.
31

---

We use Px.p to denote the joint distribution of (X, Y) in (44), and pa,p be its density. Then we have that
1
9x(x)
4 50
-22 + 0x + (x,(
if 10xpt [x,(x)) S A and y é [0,24]
a
1
t-
9x(x)
4,42'0
6p
(2A 0xpt fap(a))
if 1x + fx,(()) - A and y E [-2A, 0)
(2,
o
0
if 1x + fxp(a)) - A and |y| > A
t
(x(x)
4 (y - 8x fx))
if 1x + fx,((=) > A.
(45)
The following lemma derives equivalent expression for T X x. T = 1] Ox,pt + (x,(()):
Lemma D.5 (Expression for conditional mean outcome). For any x, we have
0x + (x,p(z)) = (((x) + a -- 1112pppt - ((x)) - E2 o 2 229p22
Note that the last event in (45) happens with small probability. Indeed, we can define a good event
A Q-Q-2 G -
(T|
(46)
6162 max ma,
An important property of the above definition is that the bound goes to infinity since we assumed that 61 = (1) and 62 = o(1), that it would happen with high probability.
The following result summarizes the good properties enjoyed by Eg Proposition D.1 (Properties of good events). We have
(I). & inf Ahlels1 Px EE, = 1;
(2). Ift E En then for any X, E [-1, and any x, we have 10x + [x S A.
Proof: By definition, ex Pr~N(ox(=).02) [IT S EG (1,0]] - G -G11. 11.
A-Q-2s
Since
(gx(x)) S G o(1). (1) directly follows. To prove (2), it suffices to note that
10x, + (x) li(X) + -
((X)) Epp o -27
S ε1 max(1, 3 (IT) + G a + +
where the first equation is due to Lemma D.5.
Let Hy be the k-th order Hermite polynomial and Po and P1 be the mixture of Px.p with priors Vo and respectively, and pu and 51 be their densities. Our next step would be to bound the x²- divergence between Po and Pt. To do this, we need to analyze the densities (x, t, y), i 0, 1 for two cases (xr (,,4,,) v) E E, and (,r,4 1, y) E,, separately. Our next two lemmas handle the first case.
Lemma D.6 (Taylor expansions of perturbed densities). Suppose that t Ei then we have
1
A
4.22 ((=)
(2A + ((x:) + (k- - 1)ē2p) H&(2)
ify E (0,
Mo
(x, k, =
k=
1
2k A
4.22 (2))
Mar (2A 11A-q(a) Q(x) - (k- - 1)ē2p) Hx(2) ify E (-A,0)
k=
where: 2 (-9(x)
Proof: : We only prove the statement for y [-A, 0); the other case y E [0, A] can be handled similarly. Since
(' 1-
1-gA(e)
to Hy (2) (2)
(-1)+atx
Σ
10
klak
k=
32

---

and
2A 2A-0x.pt fhpt-fxnn (oaptfx,p(())
2A ((x) E2 11--0 11-0--aa11 g())
+
by Lemma D.5, we can deduce that
+00
= 4 4 4A2 (z)
Σ
H& (z)
[2A- ((x) EEP 1 6 -((x)) + 2
klakk
+:00
A
(z)
[(2A - ((x)) - E20 - -6- (
4
klok
k=0
zk A
(2A - ((x)) - (A: - 11)Epp) Hx(:),
4
kla
k=0
where in the final step holds since Hx(z)= = 2H - (N: 1-))H=2(22) The conclusion follows.
Lemma D.7 (Bounding x² ddiitance under good event). We have
ʃ (Polz,t,)) v) (z), {t E & dxdtdy S 200g po(x, 1,
hn-1
kn
Proof: Let (c,t,y) satisfies E Eegte and y E [-A, then Lemma D.6 implies that
+oo
2 (x(:)
/
Po(a,,, - (1(1,1,))) 142)(2)) (2A
(21- - ((x))
Σ
xtd(uo - v1)
M!
[-1.1]
h=
(k- -1)E
X >pp((o - vi)
E2
(z(Z)
h!
1-111]
k=
+00
Hx(z) zt /
4,22 ((=)
(2A -x(x))
xtd(eo - -1) (47)
Rt 1
k=A
8
(k 1)21
X* Xppd(1o ( (1)) mn)
E2
Hx(2)
k!
1-1.11
k=
1 Hx(2)ca(z))
4.2 ((2)
1)!
where 1-((x) and
11) >d - v1) - -ēzēt / ||111 >*pa(uo - u1)
(x(x) = - ((x))²t
(2A
(48)
[-1,1]
On the other hand, since + - is convex, by Jensen's 's inequality we have
,,(x,,,y) / px,alt, L, tx, y,yroo
/ (022 -11))du
44
(49)
> 1
(-1 1 /
2 (-2/
(z - 2122))" p)
exp
44
1
( 1 22 (-12²
2 44 exp
2 2
8A p(2)
for sufficiently large n, since é1, 82 o(1. Hence,
2
(po(x, (,y) (x,,,,,1))
243 242 ((2)
1
Σ
(I - 1)! (late)ccco)
1
W=
33

---

Fixing 2, y and integrating both sides of (47) with respect to t, we can deduce that
(po(x,t,)) (i v))²
1 ft E dt
po 42
1
(0())
(k: 1}1 H1 (1x(2)c((z)
foz + +(xx) E E odz
2
k=
2
(0))
Hx( (2)))))aaa())
dz
2
(k: 16-1]1 1}!
k=
toc
((k 1)!)² / (x))) +2 +2
(x)
- 1)!(j - 1)! / (()))
c(x))((s)
E
2
A=
1312k+11
+00
ke
+o0
(x)
Σ
Σ
2.43
(k 1)!
A3
(k - 2)!'
k=
+1
(50)
where the last step follows from the orthogonality of Hermite polynomials:
I (x)) ks and / ((E)) = 0, A.
Moreover, (48) implies that
|cx(x)) S 6AA -2 22//2 < 8AZ
Plugging into (50), we can deduce that
/ (No(x, 1. - (1( 1 {t dt < 64 po(x,t,y)
+oc
(
kn-1
Σ
2k 100gA-1
k!
$
kn -
k=kn
For L E A), the above inequality be established in similar fashion. As result, have
{ (0ol2,11,0) ()) drdtdy & 100gA-1 polx,t,y)
(
kn
[drdy
/ dandy S 200
( EG hin 1
lin -
as desired.
Our next lemma, the other hand, develops bounds for densities outside Eg essentially shows that this part makes a negligible contribution to the overall xivviverge
Lemma D.8 (Bounding x2 2ddistance under bad event). For any t & Ex we have
(Polz,t,)) v,, v) (x,,,, 166exp (-1202 (t-(()))) (,(
Proof: For any & & and any X, define
1 2,,.y = ((,,p) E E-1,11
0x,tt + fxp(()) s A}
and
Ag.ty == [-1,1111 10
(1)
34

---

For any (,, (2, P) E [-1, 1]2 such that (A, p): (Ac, (3)) E 2,t.yr we have
A
p2, v,
(51a)
- (1 92 (x))
す [v ((x) o E1E2 [(x)) + 62 to 2 (51b)
2 20 (1 - (x))))
((x) o G'EAAit E2 + "GE
1
(61 + 2²)²
2
(()))²
(51c)
20
(1 ((r))
1
y/ - ((x)
(1 E2ps + - -0
3
4
1
4
(6) + +2)²
1/ (1 )(r))²
(51d)
20
r²
1
P (2,,, v) 24 (t g(x))
(51e)
where (51b) follows from (45), (51c) and (51d) follow from the inequality 2² 2 plu + u)²
v2 Vp E 1), and (51e) holds because é, 62 o(1) and It - ](z)1 = (1(1) by (46). With
I-p
similar reasoning, we can also deduce from (51b) that
log px,,
(t g(x))² v(A,p) stt. (A. E A
(52)
402
Define
1.5 15/ T.=JaaPAN JAPEEA
i E {0,1},
where we drop the dependency on (I, y) for convenience. Then from (51) we can deduce that
P.,1 1 max (x, s exp (- $20 (t - {(x))²)
min (A.p))A (1)
(x, t,
(A.p)EA!!).. eid.v
12o² (t- (t ((=)(()))) I,,0-
1
I <xp
Thus, combining the above inequality (52) we have
(1100 12,12
12o2 (t- -B()))))
I10 2 (I10- IT s 4exp
(53)
On the other hand, from (45) is easy to see that for any (A,p) A have
3
44 (())) px(e,tt,)
44 p(2)
faga(a) so 11.00 < lo.1 3l0.0 and
where
(Jo.0
12
< 400,0 S 4A- exp
(t (t-g(z))² ((x))))
(54)
1o.0
402
where the last inequality due to the bound 1o.0 S A- A--eexp
(-
-o (1 - ((x))
which directly
follows from its definition (45) and the argument used in (5le).
Combining (53) and (54), we can deduce that (po t, y) P1 (I, (x, (Iu,0 lo,1 + I1,0 I1,1)
(z (,y)
10.0 + 41.0
(10o0 10.1²
+ (11.0 11,12
2
Io,0
11,0
(-120p (t - (t=)(=))²)
1
< 16exp
120
as desired.
Combining the results of Lemma D.7 and Lemma D.8, obtain the following:
35

---

Corollary D.1 (Bounding the whole x² -distance between the null and alternative distributions). Let Po and Pt be the mixture distributions as defined before. Then we have that
Nn
A²
(Poo1(P)) - 200
+ 96A 022222222
(55)
kin 1
242
In particular. if E162 = o log
1/2 n) and knn 2
log then (PolP = =0(nk)))) log
and
x2 = o (k)
Proof:
By definition we have
(P
-Jenra Je
(po(x,t,u) - ()),
(Po(a,t, pI(, x)))
=
dPo +
dPo
Po(,,t, t, 2)2
Po(r, t, y)²
kn
< 200
e
+ 16
exp
12 (1 -(xx))²
1
drdidy
lin
-1
s 200a
ec
kn
+ 16
exp
1
(t - -((()))) 1 1t - ([-)(x)12
25 j dadt
A
Kin
12o2
ec
kn
1 - (s (0())a)))
A
200g
+ 16v66
exp
--(x)) z
dodt
knn -1
2 2/B0
kn
A
<
200g
+ 96A--12210 exp
knn -
2427
If é1 o (log2/2 r)) and kn is the smallest integer satisfying kin
log it easy see that
log ès
both terms in (55) are o ((n1:)) so (Po)((P) ((nk)) It follows that Po (2 t, 81 IT (Fr, tu 9²))²
x² (Pon
dz ... dr, ... ..,, ... .yy
II Do (i)
Pi (2,, t4, v:))
dx d. ... dx, ... dt, ... .yn
i (o Li mi)
P1 (22, ti, y1)2
II
dx,dt,dy;
i=1 po( to yi)
< (1 + o ( - = (k)))))
which concludes the proof.
Finally, we can apply Lemma D.1 to deduce our lower bound. We define the following functional T: for any observation distribution Php of ((x,, 4, y;)) generated from a model in (44), T(P) equals the corresponding parameter value 0x.p- Let Vo, 21 the distributions that satisfy the prop- erty in Lemma D.4 corresponding to the ks in Corollary D.I; we can also view and V as dis- tributions on the Peep Note that 0x.p = &22 by (44), we know from Lemma D.4 that the mean difference between Pi and 21 is
mi mgg = / (PR)) d(vo - v1)(A, = =0 / Axddio v1)(A, -1)( 2
E162.
4okg
On the other hand, clearly have < 20 1iēe, and Corollary D.I implies that x2 (P (( o (k=²). So for sufficiently large n, we have my - mg vol > Bork E1E2.
By Lemma D. the minimax mean-square error for any estimator Ý is at least
(1ogg
log log ē
-3
E1ē2 = -2 (-( ((log
=o(
-3
Ω (k, 3 (2)) = =
o 21c2
log En.g
36

---

In other words, we have
2 21,11 1-7 (Pa. cy
log n log é1
0 3 c
(56)
1/2 It remains to prove the 1/1 component of the lower bound. Our proof relies the following lemma that derives the >-diivergence between two Gaussian mixtures.
Lemma D.9 (x--isttacce for specific Gaussian model). Let P.ç 0, be the distribution of SK,T,Y) generated from
X Px,
T T|X X ~ ()),
V|X,T~
(57)
such that (2 - 0ol < 1. Then
PP).
[q((x) 20(x) + (0) 20)) -0000))))(()))00(())))) 900x)))² de
= [[ - -202 - -00) 1/2
exp
1 20 0b)
Proof:
is easy to see that the density of P: can be written as
t,( = 2To
Px px (x) exp
exp
(- 20 (t -=((x)))
2 (y - ((())) - (t - ==()))))))
thus
Po (I, 1, log
px (x(x))
1
=olog
+ 2²² - (2) + (t- (t) (g0(x) + (t- -o(x))0o)] y
[2( 9 (x)) (t [o(x))}]
2
12 (x) (1 9i (2) (go(x) - [x())))))
2
px (x(
log
+ y - (2(1 + (1- -1- (x))))) + (1o(x) + - ===
2no
2 91 (x))² (t - go(x))} (() (t-g1(x))0 (z)))) (g0(z) + - (o=)
20
(xx(x)
log
[v- (2(m(x)) + (1- () (x)(x) + -==
=
1 - (01 - 0o)2 (t - ((=])z))= (x) - (x()) (0, 2g1 - 90 (x))
+
202
1 - 2o (01 00)
:(tx..))
:=d(x)
where c(x) is some irrelevant function of æ. By taking integration, can deduce that
/
p pą( 1,
(P(P
po t,y) drdidy -
1
/
t,xx - b(t,x)) + d(x))d - 1
=
2na
1
px((x) exp(d(x))d (pp(-b(t,x))dt / expl-aal,,,
2
= [1 - 11 - -0,) 1/2 /
exp (d(x)) dx
as desired.
The next corollary highlights the special case of Lemma D.9 that will use in proof:
37

---

Corollary D.2 (Bounding the x2 --distance). In the setting of Lemma D.9, if = 91.90 = Q1 and
" a101 001 < 0.1, then x (II Po) < 2 (0) 00)².
Proof: By Lemma D.9, have (2 Po) - [1 20 (01 -Bu) 1/2 1, and 0 - Bul - 0.1 implies that
11-20 -
1/2 - 20 - 00)
concluding the proof. We now define
0 0, 0= 0=6 1-1 1112/12
and let P and P be distributions of (X, Y) generated from (57) with (g, q, 0) (ă. q, 0) and
(g. q. 0) respectively. Then Corollary D.2 implies that xP, P) S En 1/2 and thus
H(Pon p*n) < nH( P) < nx (P, P) 1 E/2.
Therefore, Lemma A.1 implies that for any estimator 7, holds that
sup P | - 2 221220-1 -11-in/1//// 1/2 -1/2/4 2 2 =%
VE(1 E((4
PEP
Equivalently, we have
27 (2.. 2 2//2 11 n
(58)
Combining (56) and (58), obtain the desired result.
E General upper bounds under Neyman Orthogonality
E.1
Estimation error of general moment estimators
In this section, we establish upper bounds for general orthogonal estimators beyond DML.
To state our first result, we require several assumptions, as stated below. These assumptions largely follows [Mackey et al., 2018]. We first define the Neyman orthogonality property of moment func- tions.
Definition E.1 (Orthogonality of moment function). moment function mn (2, Oo, ho(X)) R
R H + Fd is said to be (So, So)-orhhaannal for some sets S1 So Z for any E So, have Ep m Z, fo ho X)) X] 0 a.s., and for any a' E S1, we have D Z, pu, 7) a.s., where D m Z, 0o 7) Vai V Va me/mm(2 (Z, 0o, y), Vyy E R
This property is the key to constructing efficient structure-agnostic estimators. Assumption E.1 (Main assumptions). Let Si So be non-empty sets and k Z,, then the following conditions hold:
(1). The moment m (Sp, orthhogoal
(2). Ep Ep [e/mm(Z 0, ho X))] # for all t.-
(3). EEl[Vo((, 8g ho (X))] Z 5na and Varp (m (Z, 0, hx))) s Vm-
(4). Dam exists and is continuous for all ||a:]| - k + 1.
The specific choices of So, S₁ and k will be explicitly stated in all results. In Assumption E. (1) requires orthogonality of the moment function, (2) guarantees that On is the unique solution to the moment equation, (3) guarantees identifiability of 9g and lastly, (4) requires sufficient regularity of the moment function. Finally, we assume the following regularity conditions:
Assumption
E.2
(Additional
regularity
assumptions).
Define
Bio.r
(h H. max
[T [x,(X) ho, (X) sr).
E
Then there exists
such that
38

---

(1). E sup 00000-00ls- Va (Z, 0 ho(X))]l +00;
(2). For any compact set A S O it holds that SUPGEAMEB....
B 0 <
+oc and SUPeEA,hEBBa. (m 0, MX])]]] < +00;
(3). B E
8,, m Z, 8, ((x))
< +00;
(4). (00, ho) == max haalllssssuuphBB...... B | (0.0m(Z.00 (.0, < +00.
Let Z the the set of all indices 2 such that ||a]|| S k and IR,e.0 = In,t \I=-1.6. The follow- ing theorem shows that orthogonal moments as in Definition E.1 directly yields efficient structure- agnostic estimators of Pg-
Theorem E.1 (Structure-agnostic guarantee for general orthogonal estimators). Let S S S E k E E④, and p.q E [1, +oo| be such that + q = 1. Let P be a set of distributions of (X, generated from (1), & > 0, = 1,2, , and & 2 max.es Ei= 04-- Further let + be an arbitrary mapping that maps P E P to some function ho R + R in some vector space F. Consider the estimate dOML obtained by solving the moment equations
Σ
Zz (
(59)
Suppose that the moment function m RK X X RR Rd satisfies Assumption E.1 with Su, S k specified above and additional regularity conditions (stated in Assumption E.2) for all eel P. then for any 1), there exists a constant C., 0 such that
R.. (PM.L: S CCyO x (V
V
X
Σ
II 4
|alli!
aE (E(I,
(60)
with probability 2 1 - 7, where
A sup
|D m0 (2, 0u, ha ha(X)))l
PEP aE(T, (0)
Additionally, when the nuisance error rates are sufficiently fast, we have the following asymptotic normality guarantee for Ø:
Corollary E.1 (Asymptotic normality). Suppose that II = o(n 1 for all a E (Ia,e) So) U (Ie+1,00) \ Si), then vn(( - 0o) 5 (00, 8 Vm))
The proof can be found in Appendix E.3.
E.2 Proofs of Theorem E.1 and Corollary E.1
The proof is based on the standard arguments for bouding estimation errors of orthogonal estimators; e.g. [Mackey al., 2018, 8, Section A]. The only major difference is that bound is structure- agnostic while their goal is to establish (n-11/2 convergence rate under assumptions nuisance errors. For conciseness, we will not repeat the arguments that have already been covered in their paper.
To begin with, their e10 shows that
"
vā - 0n) 1fdet J(h) + 03 =
J(h) + 0) Σ
m 0o. 00 h(X.)),
Vn i=1
where J(h) = , Σ (Z,, 8, h(xx)) for some 0 = >B0 + (1 X) A [0. and J =
Elm(Z, (Z, 0, ho()))] They also show that
J 40 P - Hence
Vi - 0o) = J- Σ ml m(, 0, ((x,)) (1p
=
39

---

We then consider the decomposition of B following [Mackey et al., 2018, eq.(1))]:
B
m
Z3 to ho (X:))
Vn 1=1
=:
1
D (Z,, Bu ho (Xi)) (h ho (Xx))
n
aall
B:
Σ
D'm (Z,, Bo, hu (Xi)) (A (x..) ho (xi)
lalli!
i=1 (
(61)
Σ
1
D'mi (2,, h (Xx)) (x,)) ho (xx))
(k+ 1)!
Σ
Σ
Da (Z, (2 (k - ho (x:))
(Is + 1)1
i=l aEIetino'S
=B
where we recall that Ix,e = ta E 2200 ||all S kj. First, it is easy to see that with high probability, holds that
B1 S Var (m((, Z, Co ho(x)))
Second, by our assumption on the error h ho. we have
E BBlls
Σ
Vn
D (Z.00. ho (X)) (h(x) - ho (x)) "I]
|all!
Vn
I [x) no (x) 1o0 1/9
<
Σ
E
l (Z, 0. ho (X)
|all!
(62))
SA
Σ
0e S II
Σ
II 01
|
111
aEIx.e\St=l
where the last step follows from Holder's inequality:
8
I
hx(x) - hx)
II [x(( hoo(X)) 2=1
Px.o./a
Px.9
II 1=1
(x)
hoa(X)
a II
Px.ss
1=1
Similarly, we have
B:lBb1] <
nA
Σ aEZ
II
1)!
s=1
Finally, the arguments in [Mackey et al., 2018, Section A.2] imply that B2, Br = op()). Combining everything above, we conclude the proof of Theorem E.l.
Under the assumptions in Corollary F. it holds that B3. Bs o(n-1/2) As result, the same arguments in [Mackey al., 2018, Section A.2] would imply the desired asymptotic normality result in Corollary E.I.
40

---

E.3 Proof of Theorem 4.1: Structure-agnostic error from estimated moments
The proof follows a similar argument as the proof of the previous theorem. Consider any probability distribution P E P. We define & as the "good" event that the dataset Dj satisfies the conditions (12), (13), (14) and (15). By assumption, we known that P(ε) 2 - 7/2. Our subsequent analysis consider a fixed choice of DD that fails into &. Note that the moment function (.) is partially linear in g, so for any index a = (2), o2 2, then Dam, derivative for 02 E {0,
Now let's calculate the
When 02
O, we have
(2,0,2 D)
Y (X) 0(T (T +0. (T
so that under &, have
[ m, D1) X||
HE
J(01-1)(T
Ho
(T-2-)((--90000 xxx) $ C
for all 01 Si+1. 1. When a2 = 1, we have
(2.,(2...9.)))
gT
so that
E B (2,0,g.9) X]| = - |x| S
for all 01 S The above derivations also imply that
$ || - 0(T -0--0-00000000.llle00---0-00-0--
+ (T
- (IIY ((x)]l1x()) + (1000)=0000 -(1)))
+
X:
< 4(VE + CoC,) (0))) - Xii))
< [A(0E + CoC,)
(63)
and
( (Z, 0, (;; S -//0 S Ar,
(64)
where we use the assumed property that
max BE(r,++1
sup Deg bEBr. (ho
(T (( L=(P) SA, t00.
Let J(Di) = Ez~p m2, 8o ho (x): Di). Similar to the proof of the previous theorem, we consider the decomposition
i-00 0-00= J-
Σ
m(Z,, 0, h(x.)::)) o
i=n/2+1
By Chebyshev's inequality, with probability 2 11-7/22 we have
2
Σ
m(Z,, 0, 0p, h(x));)) - -2/m(2,0,,,,)))))) Di] s 2y 2- -vmn 1/2
i=n/2+1
which implics that
0 000l S 6:gg 6 [B]m(Z,,0,h/)) Pi] + +2y [p + op(n (/12).
41

---

Finally, by Taylor's formula and the orthogonality condition,
E mmZ lo. [i)]]
[L 0o, h(X):Di)
E
1
1 ho hox) ho(x))
1=1 OE ((1.0))(i--)))))
Σ
(10"m(Z,0,, (h(X)- hx(x))
|all1!
aE/(r+1,0),0111
(where = hu + t(h
ho), t [0, 1])
Σ
1
E (2, -x](((x)-Aoxx))
- (x))
|a//1
j=l ae ((,0),) 1.1)
Σ
(0,m(Z,0,,
|all1!
oE [(K (k)),
(1) 1
1
(0- 22- ()- 62) + (k 1)! [Al02 CCon) +refez]
11
max max(c((+1) (() +c (r 1)1
1 11و Colhh Co)eft! +refe2 Art
1= j!
(65)
where the third step uses Holder's inequality and & 2 2r + 2. This yields the desired bound. The total probability for this bound to hold is (1 7)² 2 1 27, as desired.
E.4 Proof of Lemma 4.1: Explicit formula for Jr
We prove this lemma by induction on r. When 1, the conclusion automatically holds by the assumed expression of J1 (w, x)). Now suppose that 2 2, and the conclusion holds for r 1, then by definition,
I (w,, J0- Jn- a)dw'
Σ (x) / Pir P
Σ
air (x)))+1,,
and the conclusion follows.
F Technical details in Section 5.1
In this section, we provide additional results and details that complement Section 5.1.
We first consider the problem of estimating μ4, := EErr", where 2 is some positive integer. turns out that even under the independence noise assumption, there exists fundamental bottleneck for estimating My, as stated in the following theorem. Interestingly, this statistical limit different for r 3 and all other values of r.
Theorem F.1 (Structure-agnosit limit for estimating residual moments). Let CT 0 be a constant and Po be the set of all distributions of (X, T) generated from T= 9o (X) + 7, such that [T] < CT,..s.
CT, a.s. and is some mean-zero noise variable independent of X. Let & be mapping that maps
Po E Po to the nuisance function go. and the larget parameter is defined by ((P)) E such that if = and ar 2 otherwise. Moreover, k=1 is recursively defined as 02 O, 0g
(x Then for any 7 (1/2,1) and E Z.,, there exists constant Cy,r
(P
2 Gy (Ea + n -122), where a,, t000
these rales can be attained by 0 0,, where {0 {0
U/e k0
(k: > 2), where 1/1 + (T. ((x.))
The remaining part of this section is devoted to proving Theorem F.1 and Theorem 5.2.
42

---

F11 Proof of Theorem F.1:
Structure-agnostic limit for estimating residual moments
The proof is based on the method of fuzzy hypothesis, as introduced in Lemma A.1. Let X Px be uniformly distributed on [0, 1] and 7) be random variable independent of X, with density
exp(-2*) if = S 0
pn(2) exp(-1.52 + az²) if 0,
where is chosen such that E[n] 0. Let M and À (A1 A2........ ..M) where X,ss ii.d. random variables such that A; = with probability and = -1 with probability 5. Let
B1, B2,... ... Ba be a partition of X' |0. 1] such that (x(B:) = M vi E [M]. Define Pa to be the joint distribution of (X,T) generated from
X~ T= +x)
where (x) = g(x) + 22 r), and p= J Padı (A). It is easy to see that Px E Qo-
Let ÿ and be the density of P and PA respectively, then the above definitions and Taylor's formula together imply that
pa(2, 1) (x(x) - (x)) po(( - ((x) - =,,
1
2) 1 -xx)).
2)
1=0
= = (2 - (-1)') is independent of thus E,, [ex (x, t)] only depends on x, t through (zx). As result, we can define random variable v which
For any given X = E. (A,,,
independent of X and has density pt ((x)) = Er t)]. The data generating process
X~ Px, T= - f)
thus induces a density Px (1) ((x)) = Ex [P> (z, t)] = p(xx,).
We choose P = Q2 PA and Z, = B₁ x T in Lemma A.2, the corresponding Pj = 37 For any E T, x X' and À supp( ) have
(t - ((x) c,
Pn - ((x) 6²)²
(t ](x))
Py 1 ((x) (2
I eexp (-2(t ((x) <,)² 1.5(t- (zz) + 2) alt - g(x) cg//((*)
< exp (-(t -
11lt + + -
((x) + =
rit ((=))())
since 9 is assumed to be uniformly bounded, it is easy to see that x + Sg [(t ((=-)(x))dd d is uniformly bounded well. Therefore, the setting of Lemma A.2, we have
(PA
p)²
b M maxsup
dp
J
X
2,
P
f de (ple..) 12(x)
< M maxsup
H(I,t) di
J,,
3 x
/ (((,,t)+ 1)(x,))2)
< maxsup #EX maxsup #EX A
p(z,t) dt
(p(z,t) +I(t ((x)))
+o0
is bounded by some universal constant, which we denote by b. Let C' be the constant in Lemma A.2 that corresponds to A = 1, and choose M 2 max { max 1, b), C8-1 ?n??? }. then we have that
ppnn, / p p((ddT(()) S
43

---

The final step is to verify the separation condition (25). Specifically, we choose P = Qo and define T(P) to be -Pr for any P E P. We abuse notation and use ja, to denote the value of pr corresponds to P PEER P. Then we have that
Mr (Pr(P)= = -
=== (P.)
2 + (-1)²
( / /=) + O
3(-2) 2)
Note that for S r we have
i (2)d (r - i)! ʃ x"' 'r (2)d2.
r!
In particular, consider the where i 2. If + 3 then the above equation is nonzero, implying that
Mr(P) pr (PA) + +(gg)
Therefore, Lemma A. can be applied with = (eg)). which yields the desired result.
If r = 3, then f =" (x)dd E(y] = 0, so that
pr (x( pr (P3) 0(2))
and the conclusion can be similarly derived.
F.2 Proofs of Theorem 5.1 and Theorem 5.2
In this subsection, present the proofs of Theorem 5.1 and Theorem 5.2. The proof techniques largely similar. We choose to start with the proof of Theorem 5.2, which is more vnv.
Proof of Theorem 5.2 For any P E P, let 1/ Ep((T ((X))*) then it is easy to see that | S 2t
[E[ ()( + (() - )(x)))*) - 2 (12 2 + c+) (66)
By Chebyshev's 's inequality, have
Pilpk - phl 5xl S 8 Var (T ((x)))) - 82n [T 2
2 (C2R + 1/2
(17 + 9 -(xx)) 267 [0]
82n
8.12
where the last step uses !gl. |g] S Cg and || S vn. We choose 8N 2/ (rm) (amm)-1/22) -1/22 (C +
k 1/ then it is easy to see that with probability 2 1--- 7, 1u4 - T'4 S & for all E [r] Let be the event that all these inequalities hold. The following lemma bounds the difference between and its population version (with 11 replaced by 71), defined as
1
0
0
0
0, (-1)r+1
A3
102
1
0
0
14-1
(22
Lemma E. (Moment-to- -oomelt-to-cumullnt type bounds). Let the sequences (0s J.z. (6)
futtazi
and (86 J121 satisfy the recursions
0
1/:
01
p4
(67)
1 1-
0%
ph
5 9-1 10h
(68)
j=1
k -
0, ph
(69)
'Although Theorem 5.1 consider a more general class of noise, the rate is strictly looser in the setting of Theorem 5.2, as discussed in Remark 5.
44

---

Assume there exist constants Cg. unn > 0 and Y, E (0, oo) such that for every k 2 1
1p4el - 2 C ( + k//2//))))
(70)
104e Flll S r (22)) - 1/2 22h 2
C5 2k + k w1
1/2
(71)
Then, for all k 2 1,
00s| S (8k(Cz + vin)]
(72)
0 0₂| s 3[12M(C8 + +/nn]] 8 (2)))
(73)
Proof: Throughout the argument write
A& = 2 (Og + vin) */n)////²/
D3 := /// 11/2 Ak.
(70)(71) imply that | - A and 1 - Hial S Dr. By Triangle inequality, (68) implies that
k-1
00 S A +
Σ
k 16-1 1
1)
Ac-51051-
j=1
Let 0g = Aspke then this becomes
k-1
k: 1)
Aj
Ipd|
S
Σ ==1
lppl-
(74)
j - 1
Ax
We now prove that Indeed, since
(5 1
1
A,A
k2122
<
(75)
Ak
j5/2(k: - (k-12))
n
2 (2) n
2T
< no! <
( eizn,
4n 21
e
we can deduce that
(--1)
V V2Tk(//e)* exp(1/(121) (20((()))))((1((22
k*
v 22j(i/OM
2 - j)((k= - j)/e)/
- p
Hence,
G (111) A
k 1 A,AR-) S
f2 j)(k-831/2
1//2 5o/2 - j)(2-2)
j5 ()(k-}}* j)²
k2/2
proving (75). Plugging into (74) and rearranging, we obtain
k-1
k- 1
k- k S <k k/2 +
Σ 3=1
(k ()- ( 1/2
3/2 -1200)2-2-///1/-/ S k -k/2 + Σ j - 3/2 lpsl-
1=1
lpx < 3 11)!
Vk 1.
(76)
Define
b E01/1 p.l. k 1.
Sk =
1=1
Then we have that
S < 28 + kk2
(77)
S₁
k/2 k/2 < 2($ + (k- 1)-
(k-
Moreover,
8 |pil 4(C₂ + wnn)
|
S 61
S
1 7
4
45

---

so it follows from (77) that Sy I 2 Finally, we have
02l S Aklpa| S As k2/2 S [8k(C& + tin))
We now turn to bound 0i: 8₁.. Set As := 8₂ 02.- Subtracting (68) from (69) gives k-1
(1-1)
k 1
- ph- + 1 16 رررر
4 (ike mk)
Σ
j=l
Taking absolute values and invoking the bounds already proved yields
k - 1 1) (DR-s 18,1 + (10k-,1 Dk-s) 14,1)
A S D2
+
Σ
< Di + Σ
k-1
k 16-1 1
(112(77)) -1/(()))2//Aa>>>> 22 A. + (1pk-yl + De-;) |4,]
k-1
Σ ( ( (AR-s + D
1-1
I Dx + ( A&
5)- -k- 11/2 +
5=1
5=1
k-1
k: [k-1)
< D2 +1,11 Ap +2
( j-
jml
k (k-1)
S 8821/ A + 2 Σ
i-1
A A
(78)
Let A = Az then (75) and (78) implies that
k-1
1: - 1
AR-
16xl S (r))
+
Σ j=1
10,l
1
A&
(79)
k-1
+ k/2
(K- j) -)--
j=1
Define
Ty :=
1/218/l, 1 1.
f=1
Then we have that
k-1
Ts S Tx-1 + 8/2/2(1m) 11/2 +2
>
(k: j)= (0)-0)00--01)---1111
j=1
S 8r (2 1/2 + -
which further implies
1/2 < 11/2 Tk + 4r 127)
(80)
Moreover,
Tj |51 114 - -1/1
< (12 -1/2
1
4(Cg + vnn )
so it follows from (80) that T, < 3/2 (7m) -1/2, Therefore, |A/| Ar1ool S ,T/ (1122r+111/22mm))) A,, concluding the proof.
In view of the previous lemma, we only need to bound the difference between 0, and kq. Note the following well-known property of cumulants {
400
log Ep, 1e
Ki
1!
1=1
46

---

Let f T - ((X) and Dx = (o ((X), then n= + Dx and 17, Dx are independent by our assumption. By definition, is the (-th order cumulant of T -x Hence,
+0c
1' = log E log Bic²" log B
+00
d !!
Σ
Σ
1it + di)
where di is the 1-th order cumulant of the variable Dx. From Proposition A.2 we can deduce that d, I (2re)". combining with Lemma F.1, the conclusion follows.
Proof of Theorem 5.1 For any P E P. let 14. = Er((T - (( then it is easy to see that
M S 2F + + (E B[(x())1 s C*
(81)
By Chebyshev's inequality, we have
2 C2k
Plun Mlal > 5x] S n2
Var ((T ((x)))))) S
8'n E [(T ((X))
I
8
then it is easy to see that with probability > 1 7, 1u4e Let & be the event that all these inequalities hold. The following lemma bounds the difference between 0 and its population version (with bi replaced by Ød), defined as
We choose OR =
A4&l S OR for all E [I].
a
1
0
0
0
P2
0, = (-1)r+1
Fg
F12
0p
0
F
14-1
Lemma F.2 Momentt-o-cuuullan type bounds). Let the sequences (0x J221. and (74 hk21 satisfy the recursions
0
ph:
(82)
k-1
0₂ = 1/2
Σ
1 1--1 1 1
(83)
j=1
02 = ph Σ
p 1
(84)
1 -
Assume there exist constants Cg. un > 0 and I, y,n E (0, oo) such that for every k 2
/ S (2CT)*²
(85)
1u4 - /11 (122(mn)11/((2111) (12
(86)
Then, for all k 2
0x < 2((cT)*(k==1)
(87)
00 - 0u1 < (/22 1)!.
(88)
Proof: By Triangle inequality, (68) implies that
k: (k-1)
0& < (2CT)* & +
Σ
1-1
(2CT)
Let 0g = (²CT)* ( Pk. then this becomes
k 1)
lpx|
< +
lpsl.
(89)
47

---

Since
lp1l
S
S
2CT
2CT
2
it is easy to prove by induction that
|ps|
S 2(k: - 1)!
so that
0 2(207) ( -A- 1)!.
We now turn to bound 0% - 0j.. Set A := 8 0g. Subtracting (68) from (69) gives
k (0m 10- + A
D (i44 11/)
Σ
Let D2 (1m) (/ Taking absolute values and invoking the bounds already proved yields
k 1)
A S D +
(Dx-)181+ + (14k-1l + Dx-))
j
k -
$ Di +
D D 2(2C 1)! 1)! + (1ph-1l + De-
j=l
(90)
k-1
A-1
k ( (A + +D==))4,b1
$ D2 + 2D k Σ
+ Σ
(k -))
j=1
j=l
k-1
k
S 5Dx + +2
Σ
j
j=l
Let A₂ = DRōr, then (75) and (78) implies that
k 1
|dæl 5 + 2
1 1 1851-
(91)
We also have that so by induction we can deduce that |dxl 10(k -1),, and
111 = 1i4k - 1//
1
< 1/2 (22mmm))
1(Cg + vin)
4
A S 10, 1/2 (2mm) - 1/2 (2CT) 11),
concluding the proof.
In view of the previous lemma, we only need to bound the difference between 0 and sq. Note the following well-known property of cumulants t0
d
log E..lpp/e [e
Σ
KITT
1=1
Let n T (X) and Dx = (X) 9(X), then f + Dx and n, Dx are independent by assumption. By definition, 0, is the 7-th order cumulant of T - g( X). Hence,
t
= log " = log Eie"'] + log 1Dx 1D = Σ (m + di) t !!'
0 1m1
1!
where di is the 1-th order cumulant of the variable Dx and have that di WE [Xo(XX) - (()) Del and the conclusion follows.
<
48

---

F.3 Relaxing the independent noise assumption
In this subsection, we consider a case where the noise 7j is almost independent X and show that our estimator is still better than the plug-in in estimator.
Proposition F.1 -aipite-sample accuracy of two cubic-moment estimators), Let T - go(X) + with 17 Co7 Th and assume
(i) EE [m = 0, Var(m) = of for E 10,1];
(i) m JL X (while making no restriction on the joint law of no and X);
(iii) tjo "n have finite third moments;
(iv) estimator I satisfies 5(X) -0 (X) (X) with [0]12>(PX)
(v) E [T - (x(x 20 and set M := Ei[T go (x)
Given i.idd samples {(x,, T:) 2t put Z, == T: (x)( and define
1
Pk.n
z (k=1,2,3)
-= -3,n 3/2,,n/1,,0
0n == P3,
n1
For any 0 < 8 < 1, with probability at least 5,
S - 1[1") S G + 300e0e + 4e +C*C log(6/0)
(92)
log(6/8)
18x E17'1l < 30fe + 3ogele 30 + +CM,
(93)
n
hins
where > O is an absolute constant (e.g. = 10). Consequently, if - 0 and c - while n + 0c, the plug-in estimator &n is o(e -biased and Op(n ( whereas the naive estimator in keeps a leading (c)) bias whenever of >>
Proof: With = T - (x) = - and j4k == E[Zh], we have
$. E(1"" (1a 3/22111 EE(r²)) [[03,n (s) 3122(pr, p1) 3(f2,a /2)
(94)
0, El (ma- - [(r') + (ūz,n - ma)-
(95)
The first bracket in each line is the bias, the second the sampling error.
Now write 7) = 603o + 1/, 5 = 6(X). Expanding moments and using ?h. 11.
[41 = E[6], F2 = + + 1E(8), f3 = -
- =
Hölder yields E < c, E < 2 | < c², while Cauchy-Schward Schwarz gives E[nno²? S o.e.
Substituting into (94) (95) gives the bias terms displayed in (92)-(93)
Define A = fi,m - Pr- By Bernstein's inequality for centred variables with sixth moment M
Pr Pr(
< exp 22 +2M1/3
(k: = 1,2,3).
Taking t; = R² (2 log 6/8))/n and a union bound over ensures A S tx with probability < 2 and (94),
- 8. Using |ur| S M, |u2|
Aa 3M + 3M Aa + 3/A2||A|| S 10M log(6/0)
1b. - μ3 + 3
T
An analogous bound holds for 12, p3). Combining with the bias bounds completes (93).
49

---

G Technical details in Section 5.2
In this section, we provide detailed proofs of results in Section 5.2. We let pm be the set of all possible partitions of the integer m, i.e., the set of all multisets of positive integers that sum to m (e.g. for m 4 the possible partitions are (4), (1, (), (2, 2), (1, 1,), (1, and Pm..j be the set of partitions with j terms. We define p(m) [P,m ml and p(m. = pm..jl respectively. Note that
Pm is different from II, defined in Section which the number of partitions [mu] into distinct subsets.
Proposition G.1 (Partition number bound). We have p(m) S 2nt for all m 21 1.
Proof: : Consider placing numbers 1,2, m in a row and delimiters are placed between some consecutive numbers. Clearly, the total number of ways to place the delimiters is 2 For each partition m i + .... + ij. there exists least one way of placing the delimiters, such that their induced partition of {1,2,
m) contains subsets of sizes i,, fmm- This creates an injective mapping from Pm to the set of possible delimiters, implying that p(m) < 2
G.1 Proof Lemma 5.1: Key lemma; higher-order insensitivity condition
First by definition, E (k) (T (x()))
Ei= - di where Pi-k
ErenL, II KBy is the (i - k))th moment of 1). It suffices to show that
Σ II (M|B) RIB))
Σ k=
Σ (-1)ka
II R
Σ
II K|B-) (96)
-Ell-- BEX
xEll--
BE
w'Ell- B'Ea'
To establish this, we note that the corresponding summands
each side are of the form (-1)%%, ... K. Kj ... hig. Fix ",, ;p:: "1. : such that Dx= is Ei= Jr = r k, and let i k
===l is S. Let (m, be the number ways to partition [m]] into subsets of sizes
(0 where Eee Otg. Then the coefficient of the term (-1)9k;, ... Ki, fij Rj, on the right-hand side (RHS) is N-,,,,, Ny-111-) However, the left-hand side has precisely the same coefficient, because there exist (F=2) ways to partition [r A:] into two subsets with sizes r i and i k respectively and inside each subset the number of partitions with desired subset sizes are N₁ (t.) and N, - respectively.
G.2 Proof of Theorem 5.4: ACE estimation error: sub-Gaussian noise Lemma G.1 (Log inequalities). Let a,
b 1 log we have log S b.
(1) For every 0 <
(2) Assume, in addition, that ab 2 (so log(ab) 2 1). Then for every 0 < alog(ab] have x log(ax) S
Proof: Proof of Item (1). Define g(x) aa + loo r b, 0. Because g'(x) a -/+1/x> 0, the map g is strictly increasing. Put
CD
log
(> 0).
We show g(z0) E 0. Set y P b
then
g(H0) = a
logg
- log log - = logy + log 1 logy).
Since y 1 log < y, the argument of the second logarithm is smaller than y. hence log(y a log and g(ao) 0. Because g is increasing, x S Zo implies g(x) S g(ro) S 0, i.e.
log
+ log b.
50

---

Proof of Item (2), Define h(x) - 2log(ax) - b. x > 0. We have (z) log(az) + 1, so h is
strictly increasing for 2 /a. The equation h(x) 0 has the unique positive root
a (ab)'
where W is the Lambert-W function (the solution ((bb) log(ab) (standard lower bound for W [e, oo)), hence
(2))
When ab 2 e. we have
b
(ab) a log(ab)
Thus every < b/(a log((b)) satisfies x s x and, by monotonicity of h, h(x)) S M(**) = 0: that (og(az) < b.
Lemma G.2 (Condition for bias domination). Les
A
Ki
R4 s 3[12i(Cg + ( (2ic1)
Isisr.
variance
Let a 2 2oo (6(C₂ + dnn)ett )bb 2 log(7n/9) with a, b, O. If
x
log
(97)
a
then for every 1 Siisr
3 12i(C +n))) (mm) 11/2 I (2ic1)
iee. the bias term (2ic))* dominates the variance term in Lemma G.2.
Proof:
Note that
variance bias
<1, Vie [r]
equivalent to
log (6(Cg + vi )ei 1 log! / S log(7n/9)).
can apply Lemma G.I with a = 2 log (6( + (nn)))n)) and b = log(nn/9) to obtain the desired conclusion.
Let
++
(,(w) = Σ Wirw--1
(98)
01k 1, air (i-1)((r + 1 i)!
Σ *ENI,1--
(-1)
II Kim m 2
be the exact version of j.. We first derive some bounds for the coefficients of J, and J
Lemma G.3 (Bounding polynomial coefficients). For any i [r + 1] we have
|aar| < (i 1)! (+t11
(99)
51

---

Proof: By definition, we have
1
E
II
K|B]
(100)
(i 1)!(r +
1)!
-ell++I-1 BE
1
Σ
II
B 11
(101)
(i 1)1(r+ + 1 1 i]!
TEl BE
1
(42hg) (42 )
B B1/2
(102)
(i - 1-))((r + 1 - i}!
12
r+1-i
1
(1 )rtl-
Σ
Σ
rr+ 1 -
6//2 (103)
(i-
1)!(r + 1 - i)!
11, .2/)
(i SJPP
s=1
--i
1
(48)))
11
(104)
1)!
1
(84)
(105)
where (100) follows from triangle inequality, (101) follows from the cumulant bound in Proposi-
tion A.2, (103) rearranges the summation term according to the number of subsets in the partition E 11--1-i (104) follows from
2./2 i (1+11- i)!,
r +
II
12 = (r+1 0)
i,,-..
and (105) follows from Proposition G.I.
Lemma G.4 (Bounding the estimation error of polynomial coefficients). For any i E [r + 1] we have
1 i- (i 1)1 (16m))
|air -aarl=
(106)
Proof: By definition, we have
Air dsrl = (i 1)1(1 +1 i)!
Σ
(-1)
II
K|B|
R|
-Ell,+1-4
BE
BE
Σ
KJBI
II BE
k|Bl
(1 1)1(r + 1
*Ell++1-1
BET
< (i- 1)1(r +1 -) i)!
Σ
K|B k
II max { {K11BT] |
xEII,
BET
BBE
Σ
Σ
(2)Blc1
|B|
II
22|8'++1 |3"| 1B1122201111' (8'
(1 1)1(r +1 ---
+ell++1-:
(BE
r+1
r+ i
S (1 - 1)!(r + 1 - 1)! (807)) +11-i
Σ
82
1=1 (br ()P
l+
S (1 1}! (8thn) (86i)) r-t2-1 El Σ
p(r +1 i,j)
(i 1)! (16gg)) rrtl-i 61
as desired. Here in the last but one step, use the fact that
d=/2
(++ (,++1-)) II = 1)!
io! S
(r+11-1) i)!,
and the last step follows from Proposition G.
52

---

Lemma G.5 (Key lemma: approximate orthogonality). Let
b
log lo--
(107)
a
where a 2 2og (6(C² + Vnn 2 log(>n/9) as in Lemma G.2, then under & we have E (1 (T (x() s (
Proof: For any & [r] we have that
E (T (x(x))
Σ i!
k!! dit1,
Σ
(-1)
Σ
II
KB
(r i)!(i A:j!
xEll--1
BET
w'Ell, B'Ex'
II
(MB -R
b)
=ETT, BET
(108)
By Lemma G.2, we have [K; - Ril - 2f for all i E [r]. Then for any Sisr k, we have
Σ Il
(MBB) - R1B)
1
Σ
(1110)-" ,11)
rr k
Il 2(GGe1)*
+ BET
(1 JEP Ra
==1
s 2 2 k)lle²
E
II
(1r. LyEEP- s (r - k)/p(r ki (266))
Plugging into (108), we have
E [T (T (T--)(())))
I s - (2) - S (4ec1)
Lemma G.6 (Identifiability coefficient). B[(T - (X)),, - (Too)) = 1 N
Proof: By definition, we have
E [
D0 (X) J. gxo(X)]
Σ
(-1)
1)!(r + 1 ii)
K|Bl
#Ell,tI-4
BET
(109)
Σ
II
Σ
(-1)lal II K|BI-
K
1)!(r + 1-
e'ell B'Ea'
aEll+11-i
BER
Consider any partition π E II, Without loss of generality, assume that E B1 and |Bi = k. k: $ i.
There are a total of (i-1) possible choices of B,, and the remaining sets form a partition of [i - ks.
Hence we can write
II KJB
Σ
Σ
1
x' e'lll, B'EW!
k=1
x" E'lll.-r B"Ex"
Plugging into (109), the right-hand side
r+1
Σ
Σ
(-1)l=)
Kk
K|B"|
K|
i-r
==r
x"Ell;-. B"En"
zEnl+t-i
BET
II
(K| - -(BI)
Ri
= kr+l;
k
r!
k=1
nEll+
BER
concluding the proof.
53

---

Lemma G.7 (Linear-in-G moment difference). Ler
= T= +
Eln] - 0, " I X,
| =: U/y < 00.
Assume an estimate 9 satisfies
11 - 9u (.) 51 S Uny for some 2 2,
and fix an integer Sis s/2. Then
El (T - go (x)) (T -(())) 1 i(3th Vi)
(110)
= Proof: Write 8(X) - gx ((X) so that T (X)) and T - - 8XX). Binomial expansion gives
(n) - (n) Σ n 8 8
Taking absolute values, expectations. and using independence (1) 1 X) together with Hölder,
( - (17 - 5)21 S
E/ E1ó| 8-3 v Σ
B|nl 2 si i-
(A)
j=0
Since 17 is uub-Gaussian with || = un we have E/np3- - (2 vTy² & (2 Vi - I),, so that
E(())* - (n + 5) E Σ () (2 vi TVG 1-1-1 < in
(t=1) (2 vi 1 Ty 1
(1-1)
(2
Σ 1=0
j
I Si Vi - Vi & i(3r/ Vijaa.
Lemma G.8 (Identifiability guarantee). We have
Er [ (T- ((x))) x(x)]
1
8ia 2r!
Proof: By assumption (2) in Theorem 5.4 and Lemma G.6, we know that
Ep [(T - (X)) - go (X))l ,ia: By Lemma G.3, have |aarl [1-1)] )r Moreover, by Lemma G.7, for i E [r + 1] it holds that
E (T' - () (T- < i(3e)) Vijei.
Hence, we have
Ep [[T (X) - Ep - ((x))xx) -(xx)
-
Ea. E(T -xo())) B., ((X)))
Gall-E((T - (X)) (T x(x))* + la+ .orl |B|[T ((x))
(111)
1+1
(884)) i(3 Vijler + (8th)) 22i(CE + u/)
(112)
1)!
11
< 18(C + win)''e 61
Σ
1 1+//2
1=1
< <100 + )/))) E1.
where (111) follows from triangle inequality, (112) follows from Lemma G.3, Lemma G.4, Lemma G.7 and (66). From our assumption (21) and Lemma G.I, we can deduce that this quantity is smaller than 0. concluding the proof.
D
54

---

Remark G.1. With similar reasoning, one can also deduce that
Bp [ (X 90(x))] - (Ix |3/8(C + en)l"
(113)
This inequality will be used later in the proof.
Lemma G.9 (Second-order moment bounds). The following inequalities hold:
Ep (T - (( s $2r² + v/nn 2( 2
(114)
Ep (Y -
<112(1r + Corln)) ) [16(Cg + v/)) 2r
(115)
Proof: Recall that
r+1
f(w))= = Σ
ai 1-1 lair| 5 (i- 1)! ()-1)1(80)) (80))
(116)
1=1
by Lemma G.3, and
E - < 222 + 12n)
(117)
by (66), we can deduce that
(T - ((X)) (.(T) (xx))) ((x))
Ep
((XI))) )
< Ep Σ air
(T (T
r+1
(T-((X) 27
I (88))) ((rt)))(
Σ
(118)
(i 1)1
8ehn
(8 22(++) )(rt))
Σ
(1 -8-110 1)l( - -1)! E ((T - -(())) it
1
1SU114
(1 (1+
Ce itj
< )(rt)))((+11)
2(r+1)
(8-1)) 1)l(i 1)! 22(4+))
(i (1
(119)
Un
151JS44
< (84/))) ) 2(r+1)
(3(++5)
(
(1+
C3
iti
$//23112
(120)
i!j!
in
15
Cg
itj
< 2r (8 ((++1)
23(++))
1 +
(121)
1si5544
r++1
Ce
172
Σ
8d
1+
(122)
v/n
<21218(C + vin) 2(r+1)
(123)
where (118) follows from (116), (119) follows from (117), (120) follows from ((t2)
which is a direct consequence of Jensen's 's inequality, (103) follows from the proof of (114).
z1/2 This concludes
55

---

With a similar reasoning, we can deduce that
Ep (7(T-8(x))
( (T-8(X))
1-1 )
$ <Siin) Sp
1)!
84/n
$ (811))) )år
E ((T ( - (
(+5+at)]
illlul!!
OSiJ
$ (8tb)) dr
Σ
2((++ttut))
1+
Cg in
iナj+uto
(i+j+ + w)(tt)-a-))
ijjulu!
OSiJ.uuSi
Cs
< (8tn) )4r
24(1+$+u+e)
|
12
iljtulu!
in
uSiJ,uosi
172
Cg Vn
< (8mh)) )ar
Σ 16
1
1==
s <2[16(Cg + vnn]] 4r
Since & is - sub-Gaussian, we have
Er[(Y - ((()))))-)
22 (B(0)) ((x))
< +2(+((++((++++
so that
Ep - (xx s Ep [E - ((x)))7112 Ep - x(x)) 1/2
112 + (o + wh)))²²
By definition, we have for any P P that
(-1)++1Bp Ep [P( (T go (X))
D D (, (-1)"0Ep [J so(X))
56

---

Since m m(Z, 0, h()) with h = (g. 9) can be viewed as an (1 + 2)-th order polynomial of and q.
we have
Ep mm 20 (xx))]
Ep (m( Z, 8, ((x))] - Ep (eeiemmmZZ.ep ho (x())
Ep
DD
ho (9(X) - -XX)))(XX) - -90(X))
k!
Dzm Z, 8o ho (X)) (g(X) (((x))-o((XX)))) -9o(x))*)
+
(124)
E B (m)) (0(X) (o(x)) (x(())
(k:
1}!
r+1
00l (m)] (0(X) (x)))
++
) 1 E2 Co(4ce)) (
$ (4e)" (cf + Coe
where in the last but one step we use the fact that 10ol S Co. By (114) and Chebyshev inequality, have
2
Σ (T--
(() -(x)) Ep (T- (Tx
imm/241
(125)
< (4(7m)-1/23) ((T))))(((--(()))))
< 88(m))-1/27 2 + [n)
-1/2
and
(Yr - ((x ((x.)) Ep [YY (V--(()))AA [(X))
i=n/241
(126)
$ <4(m)) (1(2) (x
< 50(mm) -00(2))11/2(bbz + Cav/n) [16(C + vn)"""
with probability 2 1 - 7. Our assumption on 1, Lemma G.8, and (125) together imply that
2
3
Σ
( - -
Ep [T - [[x0)
<
Sia
4r!
n2241
Σ
(T. - ( -BxT - -xxxX)))(((xxX(01
4r
dia
2
4r!
1
did
i=n22+1
(127)
also directly follows from (126) that
2
Σ (Y, (x-((X - o(x,))
-
< + Co%%) +w)]"
$=n/2+1
57

---

By triangle inequality,
Ep (Y - ((x))
< Ep (X - (I + - (XX]] 9(x))
x |06] Ep (I- (x))) -(X))
+ - (T ((X))
$ Ce Ep [(T (x 9x(
+ [8(Cg + [nn)] (2 + + 3/8(Cg "n))aaa
< Ce Ep [(T 90 (X)), (x) + [8(Cg wnn (3c)) E2),
where the penultimate inequality follows from Ep [J, ((x)) 3 3[8(Cg + ehn)]'es which can be shown in similar fashion as Lemma G.9, and the final inequality follows from (113). Hence can deduce that
2
Σ (Y, (--(((X)) - -
i=n/2+1
Ep (Y - (( (T
[(x)) ( (0000m)m11/()))) + Counn) [h)]
< Coli+i1 + [8(Cg + (3c) + +2)
50(mm) -/22) + Cooh) + vn)]"
(128)
< 60@kr+1,
(129)
where (128) follows from Lemma G.6 and (129) uses the constraint given in (21). Recall that i satisfies 2 n Ei=n/2+1 m (Z 0, ((x))) 0, so (127) and (129) together imply that
0 s 60e.
As result, (125) and (126) yield
Ep [m[[, 0, ((X))]] I (64(mm)--1/))))) [16(C an) Un) + tE - Coenn]
Subtracting this inequality from (124), we obtain
(80- 0)
[x 0))B 16" (c + Codt + 64(C, + v,) (1º(Cg + rh,,) + the +oo)))
(T
Therefore, we conclude that
10o - 0 < r'16" o (ffz + Coeftt + 64(C, + vn) (12(C + 2hn) + tr + (mm)
G.3 Proof of Theorem 5.3: ACE estimation error
In this section, we outline how the proof of Theorem 5.4 in the previous section can be slightly modified to obtain Theorem 5.3.
Lemma G.10 (Condition for bias domination). Suppose
A
Ral S 10(2CT)'(i 11)17 (mm) 1/2 (2ie)' for 1 1sisr, k
variance
blas
A 2oo (CTri1/2) > 0, and b A log(mm/1000) > 0. If
r < b
(130)
then every 1 S
10 1)1, 1m) < (2ic1)* '
i.e. the bias term (2ir1)* dominates the variance term in Lemma G.
58

---

Lemma G.11 (Bounding polynomial coefficients). For any i E [r + 1] we have
1 C
|
(131)
(i 1)!
Proof: The proof is the same as that of Lemma G.3.
Lemma G.12 (Bounding the estimation error of polynomial coefficients). For any E [r + huve
we
Air air - (i 1)! (2C (2C7)
(132)
Proof: The proof is the same that of Lemma G.4.
Lemma G.13 (Key lemma; approximate orthogonality). If r satisfies Lemma G.10, then under 3 (defined in the proof of Theorem 5.1 in Appendix F.2) we have
E j(k) (T gg(())) (4ec1)=-k.
The proof is the same as that Lemma G.5. Lemma G.14 (Linear in-c1 61 moment difference). Let
Proof:
== go (X) +
Ely) = 0, " I X,
|1ll =: v/ < co.
Assume an estimate 9 satisfies
1 9u (P)) 11 S Uny for some 2 2,
and fix integer sis s/2. Then
E (T (x()) - -TT ((x())) z i(2CT){e..
(133)
Proof: : The proof similar to Lemma G.7; the only difference that the moment bound of becomes E|y < Cf.
Lemma G.15 (Identifiability guarantee). We have
Ep (T -T- (
2r! 6ig.
Proof: : Similar the proof of Lemma G.8, the left-hand-side can be shown be - 3(2CT)
Combining with the constraint (19) yields the desired conclusion.
Lemma G.16 (Second-orde order moment bounds). The following inequalities hold:
Ep (T g(x)) { (2C,) 2((+1))
<
(134)
Ep (Y ((x))) - S
122C)
(135)
Equipped with the above lemmas, we can then follow the arguments in Appendix G.2 to deduce that
10. 0 < r!4 (Tc + Coc 64CF(12CT + Cx)(
G.4 Special cases of the ACE estimator
When 3, Theorem 5.4 immediately implies the following result:
Corollary G.1 (Third-order ACE estimator). Let ó,d > 0 and Co, Cg. Cq. Vn, Vs > be con-
stants and Po be the set of all distributions in PCCC, Cg. Cqi the whn such that n indepen- dent of X and [sal 2 did- Suppose that 0 is the solution to (59) with m 0 h(X)) - - (0, 1), there exists a constant C., such that
[Y - (X - 0 (T - -(())]
g(X)) (1é - 3pb1b6)- Then for any E
R Rm. (a) s C, [etez + Coei (Cg Uh + Ub + CoCT) + 24,, (mm)
59

---

The choice of the moment function in Corollary G.1 has also been proposed in Mackey et al. [2018], though their results are restricted to the high-dimensional linear regression setting. However, the rate that we derive from Corollary G.1 is faster than theirs, and as consequence, in Corollary G.3 we need a weaker sparsity assumption to achieve (/221/2) rate. The main insight for deriving this improved rate that the moment function is, in fact, third-order orthogonal. By contrast Mackey et al. [2018 8] only shows that it is second-orde orthogonal. We will revisit this setting in Section 6, where we empirically verify the effectiveness of ACE for different choices of
For 2 4, the best of our knowledge, the estimators derived from Theorem 5.4 are new. For illustration purpose, we derive the guarantee for = in the following:
Corollary G.2 (Fourth-order ACE estimator). Let big > O and Ce, Cg. Cqs Wn the 2 1 be constants and Po be the set of all distributions in P(Cg, Cg. thE 2nn) such that 1j is independent of X and [ws| 2 dis Suppose that 0 is the solution to (59) with
m(Z.0, ((X)) = = -g() -
g(x))* -6p2(T-pT---- - 9(X))²
- -4(1gg - (T -
(1/a - 6il2 4p/11ss + 12 Gyt4)]
Then for any r E (0, there exists a constant City such that
R (0: (h)) S C, [cte2 + Coci (Cg + U/ + U2 CoC + (mm)
Remark G.2 (Cumulants versus moments), Generalizing the r 3 case to r 2 4 is highly nontriv-
ial, Indeed, given the construction in Corollary G.I, one might be tempted to consider
m 7, 0, ((X)) = Y (()) 0 (T g(X)) [[T g(X)) r 9(X) (p4 rphp%-1 with h (g. q). In this case, let (xx) g(x) g(x)) then we have
E E[D(0.1) (0.1) (2, 8, ((X)) x] = -rr + 14, r
mD( -Enn' E< ((X)) X - rBin + ((X) +] X] Op(n
r(r- Bin - 1 Op(n 0p + 1121,,,
which is the same the 3 case up to constants. As result, this approach does not yield rates faster than Corollary G.I.
In Corollaries G.1 and G.2, we omit the constants in the upper bounds. As shown in Theorem 5.4, the constants for r-th order orthogonal estimators can be most (Cr)" for some constant C, that grows super-exponentially, and, as demonstrated in Remark 5.4, the growth of this constant is in some cases offset by the growth of the absolute cumulant [6,41]/
G.5 ACE estimation error for high-dimensional sparse linear regression
Corollary G.3 (ACE estimation error for high-dimensional sparse linear regression). In the setting of Theorem 5.4 with high-dimensional linear nuisance (24), suppose that the nuisance estimators (g. 9) are respectively constructed via Lasso regression of and Y onto X with an appropriately chosen regularization parameter. If max(s1, 81 r/(r+1),
82 1/(r+1) o(n//(++1) / log p), then, with 0ol S C,C,,Od, -1/2 for constants Ciyy and C, depending only probability 2 7,7, we have 10 - 0ol on y and respectively.
Proof: : As derived in Mackey et al. [2018, Sec. I], there exists some constant C² > such that on an event & with probability 2 1 11-71 7/2, the Lasso nuisance estimates simultaneously provide the bounds
l|a - aolla S (s logp, || - 30 S C, C,2(22/ logp. where we recall that 3o, ag R.. Since ~ No, Cauchy-Schward Schwarz and Khintchine's 's inequal- ity [Vershynin, 2018, Corollary 2.6.4] imply that, on this event,
(136)
61 = ||@ - 90 1l a <xx+ = [| an) 2l)ll (
à coll2 < K / + 2) /)) logg and, similarly, 22 || - 40 | 1 < K (2r + 2)( sy logp.
< K /
(137)
60

---

5xx.0 33
ooder
RMSS
4x.00"
bik uoler
2nd soker
Estimates
2.0
Metlark Tran Effect
0001
11:22
CJR.
1106 Correlation Coeffeeentt
S16
1031
0,12 1654
0.00
105
0.1
Correlation Coefficient
(a) The average MSE of ACE estimation with or- der / 2, 5 functions of Ç.
(b) The distribution of estimates for second- and
fifth-order ACE estimation, with varying &
Figure 2: The sensitivity of ACE estimators to correlation of the covariate X and the noise variable ?
2nd
Bile: 0.0014, SEd: 0,0185, RMSI: 0,0685
Blas: 0.0638, Std: 0.053l6, INUSE: 0538
Noterder
Probabiitt Demsity
Portaal Dirttbstion
Probaliiiy Denilly
Partaa Dertbatics
Nisa Gromak
1-111
Support Size
2.50
2.75
3.00
1.05
3.35
3.15
23
(a) The average MSE of ACE es- timator with 1, functions of sample size.
(b) The distribution of estimates for sccond-order ACE estimation when n 10000, 200.
(c) The distribution of estimates for fifth-order ACE estimation when 10000, 200.
Figure 3: Experiment results for ACE estimators with fixed sample size 10000 and varying sparsity.
for some universal constant K > 0. Our assumption that
max(s1, 81 82
r/(++1) 1/(++)) o(n"/(r-1)) logg)
(138)
o(2)1 1/2). Applying Theorem 5.4 with y replaced by 7/2, we can therefore conclude that the advertised bound holds with probability (1 7/2) 2 7-
further implies that, the event E, maxfe]+1 soeffftt,cc,, ,Ge2) o(n
H Additional experiment results and discussion
In view of Theorem 5.4, the error rate of r-th order ACE estimator depends on a bias term that scales as 02 + and a variance term that scales as O(n -n112,,, multiplied by a constant depending on In practice, this constant is often non-negligible. Hence, to choose an appropriate order r, one should take into consideration its effect on the final estimation error.
Varying sample size. We first investigate the performance of ACE estimators with S 6 with varying sample size. The results are reported in Figure We set r = as a baseline and only plot 1. From Figure one
the results of estimators that better than estimator performs the best, followed by the second order one. The fifth order estimator incurs large emrrr for small sample sizes 2000) but the error decreases rapidly when n grows larger.
that the fifth order
When n 2 8000, the decreasing rates of different estimators are roughly the same. This is because in this regime, variance becomes the dominating term in the total mean-squared error. From theoretical perspective, this is because for fixed s, the LASSO estimates induce errors that scale as n -1/2, so for any 2 1, the bias term becomes n (r+11)/22 we plot the distributions of the estimates produced by first and fifth order ACE estimators. One can see that the variances of these two estimators are roughly at the same level, while the first-order estimator induces significantly larger biases, especially when is small.
n/22, In Figure 1d and Figure le
61

---

In Figure If, we plot the 95% confidence intervals for each individual estimates. We observe that the actual percentage of confidence intervals covering the ground parameter 94.9%, quite close to what Corollary E.1 predicts.
Correlation between covariate and treatment noise. The theoretical benefits of ACE estimators with r 2 3 crucially relies on the assumption that X and are independent, which might be restric- tive. However, it might be the case that they weakly correlated, i.e., only a small part of r is correlated with X. We would like to understand the sensitivity of our estimators' performance with respect such correlation.
Specifically, we assume that the treatment variable is drawn from
T 90 (X) + (1 EXi)n,
(139)
where X, is the first component of X and is mean-zero random variable independent of X. We set p 100, 20000, s 40 investigate the estimation error of ACE with different r's function of the correlation coefficient Ş.
The results reported in Figure 2, where compare the top-3 estimators among 1, 6. The first- and second-order estimators have stable performance across different correlations, while the performance of the fifth-order one deteriorates rapidly when & 0.1. This suggests that it would be better to use the second-order estimator unless one has strong prior knowledge that and ? are weakly correlated. In the context of pricing experiments, the data drawn from a company's historical experimentation records, so that scientists are likely to have knowledge about the high- level design principles of such experiments.
Varying sparsity. Lastly, we investigate the relative performance of ACE estimators with different 10000. Recall that sparsity affects the first-stage nuisance
level of sparsity, for fixed p 1000, errors, which turn affects the bias of our estimates.
The results are reported in Figure 3. From Figure 3a we can see that the performance of first-order estimator deteriorates rapidly when the support size grows. By contrast, the performances of second- and fifth-order estimators are quite stable, with the fifth-order one slightly better for smaller This not surprising, since one can see from Theorem 5.4 that the bias term for the fifth-order estimator has larger bias, so it would only be smaller than the second-order counterpart when the nuisance errors are small enough. As shown in Figures 3b and 3c, when 8 = 200, the fifth-order estimator indeed incurs larger bias.
I More results for orthogonal machine learning
L1 Construction of orthogonal moment functions
Theorem I.1 (Construction of higher-order orthogonal moments). Let as and pit (w) be as defined in Lemma 4.1, then the following statements hold:
(1). E [(1 + (n)] moment function
x) < 400 holds for all S S M, and E X, then the
m, (2,0,((X),s()), (ase -0- -(0))
with nuisance functions h1 = hr = q and h442 = aik, E [Mr] is (So. oor-hoggo where So contains all a E 2 that satisfies at least one of the following conditions:
(i). Dol1| < 1: (ii).01 + a2 k; (iii). as 1. (21,02) (0,,)), (1,0),, the remaining 's equal zero;
and S {a EZ0 max { f22, Eie ai 2 2). Moreover, Dm exists and is con- tinuous for all [|a]]a S k 1
(2). Let Co, CT, Cy > be constants and P P (C Cy) Let 0 be the solution of (59) with m. My. Then under Assumption E. (2) and (3), for any E (0, 1) there exists
62

---

constant Cy > O that only depends on 7,, Co, Cr, Cy, such that
Vm kN.(E tfe2)
R ( s C,
n
(140)
+M+11d.(e + te))E (ci erez)ea
holds with probability 1 7, where é max k+ and
X -sup PEP S..
max
1' 0, ho (x))
Theorem 1.1 proven Appendix I.2. It shows that by successive integration of Jo, we can con- struct moment functions that are orthogonal with respect to (o(X and (o()) with arbitrarily high order, while being only first-order orthogonal with respect to the nuisances (x). In (3), the constants did and Vm depend on the order k; we will write them as did, and Vm,k to avoid confusion.
From another perspective, Theorem I.1 can be viewed orem 4.1, because the construction of the moment function there ensures that 1,2,...,k.. In Theorem I.1, the
special
of The-
M2 Ep EM O:j (X) X = ag a.y''s are viewed nuisance functions which allows for exact orthogonalily properties. By contrast, in Theorem 4.1 only q(-) and g(.) are treated as nuisance functions and we only ask for approximate orthogonality. While they look similar at first glance, important observation is that this result does not rely on any explicit assumptions on the estimation errors of (.)ss Indeed, it might be possible that the left-hand side of (15) is much smaller than the individual estimation errors of ā.y(-) (-)'s, because these individual errors cancel out in the summation. This observation will prove helpful in Section 5.
0.j
Recall that Theorem 3.2 we show that higher-order orthogonality impossible for the Gaussian treatment, even with known variance. In this case, since the distribution of n T (o(X) X is known, the functions ask are known as well. However, the Gaussian case, any moment function constructed from (8) when & > 2 would violate the identifiability assumption (Assumption E. (3)). We prove this in Proposition I.2.
1.2 Proof of Theorem 1.1
In this subsection, we provide straightforward instantiation of Theorem E.1 using the moment functions constructed in Lemma 1. For simplicity, we restrict ourselfs to the case where the treat- ment and outcome are both bounded.
Proof of (7). The statement follows directly from induction. By assumption, the conclusion holds for k = 0. Now assume that it holds for some k 1 O, then
Mr
I, x²) = 1 Ju-1(w', xd
Σ a (x) / Pir(w')du',
choose M, M, 1, air (1) a.-11 (x), Par (w) -Jo" Pi--1r-1 (w")dd' s
i < M, 1 and (I)) ET (x)) X 2), Pr, (1,r(w) = w. proving the result for
Proof of (2). First note that
Y y-9o(X) (x) (T - (o(X)) =e,
so that
= E E |m 2, 0, (
(x)xX)))EE X,T]] 0.
For any a E 2 2020 with |||| 1, we consider three cases,
M.+2
* = 1, then (8) implies that
E [DDmm 0o (X))) -E 900)), 0.
63

---

02 = 1, then
E m Z, 8u ho (X))
fn (T (T 9o(X), X))] 1000-1010 ((o((( - 0o(T - -0((x))) (x)) x)] ])]- =
0;-2 for some i 1, then
E E[D"m 0o No [))] = E (1 9o()))] = 0.
- (00())
Next, we consider a's of form (0,, 01 2 2 have (Z, 0g ho (X)) 0, so we only need consider the case when 01 {0, 1)-
(22, 0, ... 0) where on + a2 S r. Since m is affine in when
Similar to the arguments above,
If a1 = 02 S r 1 then
[[Dmm(Z Po ho(x))) BJ -00(),
If 01 = 0, S then
m 0o (1 (X))
- E EI - (x() Bo -o (X),
Furthermore,
Iff = 1,04 = 11, = w and the remaining a3's are all zero, then
E E[Dmm 0o (X))] -E (o
af = 1, 1, (.r(w = w and the remaining ag 's are all zero, then [ Go (o(())))) 20 go (x
This proves the orthogonality properties related Sg. Since mZZ, 0, 7) is affine n. 2 (which corresponds to the nuisance functions and dir, E [M,]). we have D = 0 as long as 2 0₁ 2 2 <> E Sj. Hence is (So, Si orrthogona as desired.
Finally, the continuity of D is obvious since m is a quadratic function in terms of the nuisance functions.
Proof (3). By Thcorem E.I,
Vm (v
(n,1L Po. S C. x
+X.
Σ oE
II (141)
|alli
1=1
As shown in part (2), NSO only contains with ag E (0,1), 01 2 o2) 23 3
and Ei>3 04 1. Thus
Σ |all!
II 5
Σ 2=2 j!
(1 + + ?+ (ei + S 3(a +22) (e) (162)
On the other hand, I,+LEd \ S1 contains with < 1 and E123 01 S 1, so that
T + 6"e2) + (ci + r- 1e2)
Jall!!
Combining the above two inequalities, the conclusion follows.
64

---

I.3 Example: heteroscedastic nonparametrie regression
In general, (-((-) may be hard to estimate since linear combination of conditional moment functions. Generally speaking, there is no guarantee that estimating these conditional moment func- tions easier than estimating the nuisance functions.
In this section, we revisit the nonparametric regression problem with heteroscedastic noise, where fast rates for estimating Oak are indeed achievable. Specifically, suppose that the treatment variable is sampled from the regression model
T= go +7,
where the noise variable = (o(X))/21))* and n² satisties In'1 S Cyp- a.s., B'n" | XX] = 0 and B X] = 1. For this problem, the following result is known from Wang et al. [2008].
Proposition 1.1. [ [Wang et al., 2008, Theorems 1,2 and Remark 3/ Assuming that X |0, 1] and go(+), Vo(-) are and 8-th order smooth respectively, then given i.i.d. data {(X,, [,,];;)==> from some distribution such that the marginal density of X exists and is bounded away from 0, there exists an estimator (-) that achieves the optimal mean mean-square enror rate [VV(X) Vo(X))l Op (n- min(2a.3((2+41))
In particular, when 3 > a, can in fact estimate Vo(-) with higher accuracy than estimating (-). In this subsection, we additionally assume that the distribution of 17 is known and let (1, Bly""| X be its r-th moment. Thus we can estimate (,) = Einn" x = x] Vo(x))/²2 with (1;)) We conjecture that by using a similar approach as Wang et al. [2008] one can directly construct higher-order moment estimates for unknown 17", so the assumption that 17" is known can be removed.
To state our main result for this setting, we assume that go and Vo are and B-th order smooth respectively:
Assumption 1.1. (M,) and V e A (Mvv for some constants M My > 0 and a. B > 0.
Let 9 be optimal estimate of go under the I -orm, that achieve the rate 61 O ( (logn//)0/(ee+t)) [Stone, 1982]. Also let V be the estimate of Vo in Wang et al. [2008]
that achieves the -ratte
6 = O( min 20/000)))))))
Consider the moment function m, defined in (10). Its nuisance functions (1x()), = 0, 1,
derive their estimates â& (x:) by functions of the conditional moments of 1) X. Then (z) The next theorem provides directly plugging in the variance estimates: A (x)
theoretical guarantee for the resulting estimate derived from this approach:
Theorem I.2 (Error rate for heteroscedastic nonparametric regression). Ler = [0. 11, Cr be a real number and P be the set of all distributions in P that satisfy Assumption 1.1 and [T| < CT. Let g, V be defined above and be some estimator of qu such that ||ğ "ii--ullaa 10l) S2. Also assume that (x)) S (2C7)r6. Then the following statements hold:
E2.
(1). Consider the nuisance functions and their estimates specified in the previous paragraph. Then for all i E [(r + 1). we have
|aarr - - 1 S rt.
6v S Aptus
i!
where A, = ==(
(2). Let doML be the solution to (59) with the moment function m = m, defined in (10). Then in the setting of Theorem 1.1, for any 7 E (0, 1), there exists constant C.,y > 0 such that
(h)) S C, Vmn + re - EfE2 Ar(EL + =e2)))) (142)
R 10.1-,,00 R P..
This is without loss of generality, since otherwise we can replace Ar with min minr() (2C)((((),,, By assumption we know that |1-1 < (2C+)*, so this would only reduce the estimation error.
65

---

Theorem 1.2 is proven in Appendix 1.4.1. Note that Theorem I.2 makes structural assumptions on Io: Vo but not on %o- As result, the assumption is stronger than the fully structure-agnostic setting, while being weaker than the Holder-smoothness setting. Even in this interpolated regime, to the best of our knowledge, there is no existing results that achieve faster rates than DMI..
It is worth noticing that the rate E, can be faster than to arbitrary order. Thus, there exists an optimal & that balances the dependency on n and the magnitude of the constants A,, ôid,r. Vm.r, A.. For 2, assuming that these constants are uniformly bounded and that g also belongs to Holder class, we can derive the following result:
Corollary I.1. In the setting of Theorem 1.2, if we additionally assume that q E (2))M)) and llğ - -00o S = ((m=221(2++1)) then the following holds:
(1). Let r 2, then Rn.1 (0OML; Pa. OO((n1122) as long minfa, 7) > and min mai 1.
(2). Letr = 3, then Rr, (0OML; (h)) = Op(00p(nn1))pp 1/2) as long as min 7 > V3-1 and min 8 3.
Corollary 1.1 implies that Op(n-1/2 rate can be achieved under weaker smoothness requirements one uses third-order orthogonal estimators. Its proof can be found in Appendix I.5.
L.4 Technical details in Appendix 1.3
L441 Proof of Theorem 1.2
Proof of part (1). By definition, we have
(x)(x) = (i (i-
Σ (-1) II (1)) xEIl,+1-1
(143)
i)!
BER
while Rt's cumulant estimates obtained by directly plugging in V(-):
1--1 i ( Pu- j=1
R4 = p4
A = 0, P En [x(X) (i 2).
By the mean value theorem, there exists some V--) between V and Vo such that (312(x)
vx/ (05/ Vo(x))) Hence
|13/22 -Vg/ 2
2 -
2
1111///2-11 - || Vollea(
2//2-111 Vollarpy:
where we recall that C, is the assumed uniform upper bound on [V((X) and [V().. We can then bound the estimation error of (-) as follows:
- 14. (X) & | 16(x)'))) (PP()) -
2ro,/2-1 CG
Via similar reasoning in Lemma F.1, we can deduce by induction that
[x)) (2jc) (x))))=x(XX))a))) ((2) + 1)Cr + +1),, S (4jc./²
66

---

which then implies an upper bound on the error of ag
air (irll22(P))
Σ
(-1)1=1-1
II BE
Min (X)
II
R
Ball
(r +
-illi!
11
BE*
1
Σ
II BE
M|Bl (X)
II (X)
(++
i)li!
+1
BET
LP(P)
Σ
Σ
||B B 11 [(()))
II max [1))) (2)
sup
(r + i)li!
ZEA BBE
+1
1
Σ
Σ
|8]
II
2(21B)C) |8)
(r + 1 i)l!!
nell,
BET
B
-+-I
1
2 CT.
rtl-s
Ev
Σ
4
Σ
r r+ h1 ,1/
II
rr 1 i)!i!
5=1
2 (20
r
Σ
+ 1 i,
i!
(8eC)
6v
where we follow the arguments employed in Lemma G.4.
Proof of part (2). This is direct consequence of Theorem I.1 and part (1),
1,5 Proof of Corollary I.1
Let = min(a,,) > 1. then we have |اا goll ll4 r0.0 Ör(n Tpं)), so that 61, Ör(n" कp+I) = (n(no1/0).
Also Proposition I.1 implies that
min (2.m(22.33/(3341)) 0(" =0 Eax
remains show that
1/22 max(c1, 62) max(cr, 1(6 +
P + min { 2a,
B
1
>
2p+1
28
+1+
2
Since 2a > 1. it remains to check that
P
8
1
+ 3p
2p+1
+
23
2
which holds by assumption. This proves (1).
To prove (2), we use similar argument except that the upper bound becomes
j 0ul Op(et + ctez + maxfer, + é).
Since >> -/-1 > $. we have max(er, Onn क O(n Onn2)/2). Moreover, the assump-
tion guarantees that
B
P
2p +1 + min
20,
>
+ min 2p, 23
> 2
23 +
2p +
since /3-1 is the positive root of the equation
+ 2p 1. This concludes the proof.
Ip+I
67

---

1.6 Comparison with Mackey et al. [2018]
In Mackey et al. [2018] the authors consider polynomial- based moment functions. These construc- tions can be derived from our Theorem 1.1 by choosing Ji (w, w pa (a:) where & is some
positive integer and us(r)- E[n | = z|. For = 2,3, we obtain the following special cases:
Example I.1. By choosing r = 2, we recover the moment function
m (2 (x())))))))))
= - 0(T - g(X)) [[T (())) -(++((X) (k. () g(X))
(144)
proposed by Mackey et al. [2018]. Thus, Theorem 1.1 implies that this moment function satisfies all conditions in Assumption E. with the orthogonality set 5 {Iaalli S 2) (),,(,, 1), (0, 1, 1)}.
Example I.2. By choosing r = 3, we obtain
m [2, 0, (X)),
(x))
= Y - ((T - g(x)) x -(x)) k22 (x+2(X) (k: + 2) g(x)))x+1(X)
- - 22'+)2))((2222)) - (1(x)))))=(())xxxxXXX)))))
(145) with nuisance functions I(X) (xx (K), (X), E R is or- thogonal with respect to S { (1,, b, 0,0,0,00 a b= 3) (a ||ll| s 2] (,),,
2 b d e
1.7 More discussion on Assumption E.1
The following result states that if the distribution of 1y X = x does not depend on and its density has certain good properties, then Assumption E.1 would not be violated with J2, unless v is Gaussian.
Proposition 1.2 (Identifiability v.s. orthogonality). The moment function m in Theorem 1.1 (2) satisfies Assumption E. if and only if
E [T go (X - 90( #0.
Moreover, the following statements hold:
(1). If m X is Gaussian for all x X, then E (TT go(X))J,,TT go)(),
0, 2.
(2). If 1 X x is non-Gaussian with twice continuously differentiable density p(-) that then there exists J1 (w,x) in the form of (7), such that B (o(X))Js(T (X)) X)] + and E[(1 + (nD)e:2()))) < +00, 1100 i M2.
does not depend
(3). Suppose that 1. X and n is non- non-Gaussian and let (J+)+t be the sequence generated from J1(w: [w, x)) = w (assuming that all of them are well-defined) Then there exists r 2 such that [T (T( 9o (X), X)] 0.
The first part of Proposition 1.2 can be directly derived from the Stein's Lemma, while the sec- ond part derived from a characterization of the solutions property of the Gauss-Airy's equation ay ay'' by" = 0. [Durugo, 2014, Ansari, 2016] Generalizing this result to k 2 3 requires char- acterizing the solution properties of higher-order Gauss-Airy's equation, which we leave for future work.
Proof: (1) is straightforward from Stein's 's lemma. To prove (2), orthogonality implies that
E [Dym 0b, ho X] -E [Ja(T (T gu(X), X) X).
(146)
and
0 E Dag 0, (o -X)= X],
(147a)
0 = E D Nog m Z2 0. ho - (0) X],
(147b)
68

---

where we slightly abuse notation and use J3, J3 to represent the partial derivatives with respect to the first argument.
Note that m is partially linear in 8, we have
m 2, no (X)) = 0 Vom( (Z, 0, ho, + (Y -X - x
so (2) and (3) are both equivalent to
E eZ, 0o. ho (x))] #0
E
(x)))
(148)
We argue that there must exist some Jz(w: æ) that is polynomial in w, such that the equations (146), (147) and (148) hold simultaneously. Since by assumption, the density p(n) of rj = T 9o (X) does not depend on X, (147a) is equivalent to
0 = f ())()) = =- x)p's (w)dw
+ f Js(us, x) = O.
and similarly, (147b) is equivalent to
f JJs(us (w, xw)"(w)d = O.
Lemma 1.1 at the end of this subsection implies that in (RR, up(w) & (pan(('(w).)2)) so by Lemma 1.2, there must exists some function Jz(w) E C (²( such that
f ((w))((w)dw = f (w(ww)"(()dd = and / (s(w))up(w)dd > 0. (149)
Moreover, we can assume WLOG that s (a(())((w))) = 0, since replacing Jg with J2 - co, Vco R does not affect the properties in (149). We define Ju(w) = ['((w and [1w) [z()). In the following, we show that J1 (w,xx = -i() satisfies all the desired properties. First, since Jg is polynomial, so is J.. Second, by (147b) we have
[ ([(TT-go(X)..)) X) x E [T - (X) x 0.
Finally, recall that p(.) is the probability density function of 17 T -o((X), so
E (T 9o(X)) 90(x))] [To(()))) =| Js(w))pp(w))) > 0,
concluding the proof (2).
Finally, under the conditions of (3), Lemma G.6 implies that E[[T (T (X) 1 fyy+1. where K; the i-th order cumulant of 7. We know from Levy's Inversion Formula [Durrett, 2019, Theorem 3.3.11] that non-Gaussian distributions must have at least one non-zero cumulant, so the conclusion immediately follows.
Lemma L11 (Solution of second-order Airy equation). Let p(-) be the probability density function of random variable that is second-order continuous differentiable, and that
rp(x) + 2ap' (x) + bp" =0
(150)
for some a, b E R, then we must have b = 0, a > 0 and thus n must be Gaussian.
Proof: : Since p(-) is a density function, the Riemann Rebesnn-Lebesgue lemma implies that its Fourier transform p(E) = Ja e = =(spp())d must vanish infinity. On the other hand, applying Fourier transform to both sides of (150) yields
(E) + 2aigg(E) bg) 0
p()) = Ce -UE
Thus we must have > O. If + [Ansari, 2016, 4.2] then implies that there exists constants e₁ C2 E I such that p(x) has the same sign as the Airy function Ai(ep + 02). However, it well-known that Ai(x) can take both positive and negative values, which is contradiction. D
69

---

Lemma 1.2 (Separability of inner products), Suppose that fi(w), 0. 0.1.2 are continuous func- tions such that fo & span(fi.fa f2). Then there exists function [[) (R) such that
J [[()) < to0, i= 0, 1,2
and
I [ww) =0, and / J(w) >
Proof: Suppose that such [ww does not exist. For any finite interval (a, and sequence of Ca functions S = {gj.j = 2, ... m supported on [a, b] (we denote the set of such functions by ([a, bl)), we define the vector u ( k=o by choosing J(w) = Eim1 X.gi(w), our assumption implies that
gr
+o0
1, Then for any X Ri
A uuf A u = 0 7
A =
Let un be the orthogonal projection of uy onto span (uf, ug) and x ug ug. then A AT A " O and A ug ||all so that À = 0 and ug (uf nz))
We consider the following cases:
1. For any finite S, rank (span(uf, ug)) < 1, then u is parallel to wf for all S. Since in +
0, it is easy to show that 24g = au1 for some E R. As a result, we have f ((w) a (w))dw = 0 for any J(-) E (]a 6)). so we must have fo afi on [a, b).
2. There exists some finite S such that rankspan u = 2, then there exists unique a, B IR such that uy auf + +Bu By considering any set S₁ SU (s.). & S. one that ug we have (w) - afi (w] (f2(w))du
show S1 axu 81 3u2 as well, so uf auf + Bug for any finite set S. As result, for any J(.) E Ca bj), so we must
have So aai B52 = 0 on (a, b).
Now we have shown that for any interval [a. b), there exists 3 E R such that fo afi 3f2 = on [a, bj. is easy derive from this fact that fo afı Bf2 E, i.e., fo E span (f1. f2). which is a contradiction. Hence the conclusion follows.
70


Reviews:
[{"review_id": "7YXBRLGrgY", "reviewer": "Reviewer_ev6C", "summary": "The statistical optimality of estimation methods for causal inference is an important research question. The authors contribute to the study of the partially linear model of Robinson (1988). The distribution of the treatment noise plays an important role: while for Gaussian noise the DML estimator achieves (minimax-) optimal estimation rates, for non-Gaussian noise better estimators can be derived. This paper makes this contrast precise by showing that no better estimator exists in the Gaussian case and that better structure-agnostic estimators can be constructed in the non-Gaussian case. Further, the authors give additional results for the binary treatment case and in the standard non-parametric model. A small set of empirical experiments are conducted to verify the proposed estimators.", "questions": "Please see the section above for questions.", "limitations": "Yes.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is clear, well-written, and to the best of my knowledge introduces new results that contribute to our understanding of optimal estimation methods in popular causal inference models.\n\n## Strengths\n- The theoretical results do address a gap in the literature and lead to novel estimators.\n- While the ACE estimators were known for l=1,2,3, higher order ones are novel and have been shown to provide an experimental gain.\n\n## Weaknesses\n- [Minor] The paper seems to study exclusively the partially linear model of Robinson. While it is relatively general it remains constrained and possibly detracts from the generality of the proposed estimators. Further it is was always clear to me which results rely on Robinson's model and which do not. For example, is Sec. 5.1 applicable to arbitrary non-parametric outcome model, i.e., for E[Y | X, T]?\n- The experimental section is a little bit light. It might be useful to supplement the experiments with additional simulations or semi-synthetic scenarios.\n- The conclusion could also be improved with a summary of the contributions and their significance.\n\n## Additional comments\n- Figure 1 (b) is a little bit difficult to parse as the boxes overlap and the means are not always easily visible.\n- Figure 1 (a) suggests that we need a minimum number of samples to realize the gain from high-order estimation methods. Could the authors comment on this observation? Is there a tradeoff between performance and sample size?\n- Since ACE estimators of different orders give slightly different results, do you have a good way of choosing between them. The numerical experiments suggest that the variance is a good indicator of the induced MSE; is this an empirical observation or is there some theoretical result to back this up?\n- What would it take to extend the binary treatment results to categorical treatments?\n\n## Minor comments\n- Define ACE in the abstract.\n- line 81, should the sentence be \"... existing estimators that do assume a partially linear outcome model.\" instead of \"... do not assume ...\"?\n- line 83, it might be useful to further comment on what the \"standard non-parametric model\" is exactly.\n- Definition 2.1: should the $\\mathcal X$ be $\\mathcal Z$?", "was_revised": true, "final_justification": "I believe the paper is of high quality, providing interesting results that are however of incremental nature since the ACE estimators were known and some of the observations regarding their statistical properties in Gaussian and non-Gaussian settings had been made before.", "rebuttal": "We thank the reviewer for the positive and constructive feedback and are delighted that the reviewer found the paper clear and well-written and our contributions novel. We respond to each point raised below.\n\n**Weakness 1. The paper seems to study exclusively the partially linear model of Robinson. While it is relatively general it remains constrained and possibly detracts from the generality of the proposed estimators. Further it was always unclear to me which results rely on Robinson's model and which do not. For example, is Sec. 5.1 applicable to arbitrary non-parametric outcome model, i.e., for E[Y | X, T]?**\n\nThe reviewer is correct that this paper focuses on the partial linear model. Hence, all upper and lower bounds established in this paper, including Section 5.1, is for this specific model.\n\nThe partial linear model is still widely adopted in the economics literature, see e.g. [1,2] for a few applications. It is true that compared with the fully non-parametric model, the partial linear model is more restrictive since the treatment effect is required to be homogeneous. Extending the results of this paper to the fully non-parametric model is an important future direction. It is worth noticing that even for the partial linear model, DML is minimax optimal in a structure-agnostic sense, as we prove in Theorem 3.1. Our cumulant-based estimator is novel, and to the best of our knowledge, no existing estimators are constructed based on similar ideas. Therefore, we believe that our approach may inspire extensions that have broader real-world applicability.\n\n**Weakness 2. The experimental section is a little bit light. It might be useful to supplement the experiments with additional simulations or semi-synthetic scenarios.**\n\nWe thank the reviewer for pointing out this limitation. Currently, our experiment section is merely a proof-of-concept to show the effectiveness of our proposed approach. We will extend the experiment section to provide (1) more insights on how the estimation error depends on the cumulant and (2) comparison with other estimation strategies.\n\n**Weakness 3. The conclusion could also be improved with a summary of the contributions and their significance.**\n\nThis is also a great suggestion. We will rewrite the conclusion part to make it a better summary of the contributions of this paper.\n\n**Comment 1. Figure 1 (b) is a little bit difficult to parse as the boxes overlap and the means are not always easily visible.**\n\nWe thank the reviewer for pointing this out. We will make the means more visible in the revised version. The main purpose of this figure is to show that the standard DML estimator incurs a much larger bias than ACE estimators. \n\n**Comment 2. Figure 1 (a) suggests that we need a minimum number of samples to realize the gain from high-order estimation methods. Could the authors comment on this observation? Is there a tradeoff between performance and sample size?**\n\nThe reviewer correctly identifies a tradeoff between performance and sample size. According to our Theorem 5.1, as $l$ increases, the bound would have a better dependency on nuisance errors $\\epsilon_i, i=1,2$, but the constant becomes worse. Therefore, it outperforms standard DML, which corresponds to the case $l=1$, only when the nuisance errors $\\epsilon_i, i=1,2$ are sufficiently small.\n\n**Comment 3. Since ACE estimators of different orders give slightly different results, do you have a good way of choosing between them? The numerical experiments suggest that the variance is a good indicator of the induced MSE; is this an empirical observation or is there some theoretical result to back this up?**\n\nAs the reviewer points out, we used variance as a good indicator of the induced MSE. That is because in our example, the variance dominates the bias for estimators with order of orthogonality $l \\geq 2$. Moreover, we do not have a good estimation of how large the bias could be. Therefore, our practical recommendation is that \n- When the sample size is small, only consider using ACE with $l=2$.\n- When the sample size is large and we believe that the nuisance function belongs to some benign function classes (e.g. linear functions with high sparsity), then theoretically, bias for $l\\geq 2$ is likely to decay faster than variance. In this case, we recommend comparing the variance for $l\\leq 5$ and pick the one with the smallest variance.\n- When the sample size is large and the nuisance functions are not so well-behaved, then bias would asymptotically dominate for small $l$. In this case, we recommend setting an upper threshold for variance (e.g. a constant multiple of the variance for $l=2$), and pick the largest possible $l\\leq L_{max}$ with the estimated variance below that threshold.\n\n**Comment 4. What would it take to extend the binary treatment results to categorical treatments?**\n\nWe suspect that, with a bit more effort, we could establish a similar lower bound for categorical treatments. The upper bound has been studied in a number of existing works. For example, [3] introduces a DML estimator with a group Lasso selection process for the propensity score. [4] studies an application in program evaluation. The main challenge in this setting lies in choosing a good model to fit the nuisances, and theoretical dependencies on nuisance errors are the same as in the binary treatment setting.\n\n**Minor Comments**\n\nWe thank the reviewers for providing these comments. In line 81, it should be “do not” rather than “do”. There we wanted to emphasize that existing lower bounds do not apply to the partial linear model, since they are established for the standard non-parametric model, i.e., $E[Y|T,X] = q(D,X)$ where $q$ does not have a parametric form.\n\n[1]. Chay, Kenneth Y., and Michael Greenstone. \"Does air quality matter? Evidence from the housing market.\" Journal of Political Economy 113.2 (2005): 376-424.\n\n[2]. Hull, Isaiah, and Anna Grodecka-Messi. \"Measuring the impact of taxes and public services on property values: a double machine learning approach.\" arXiv preprint arXiv:2203.14751 (2022).\n\n[3]. Farrell, Max H. \"Robust inference on average treatment effects with possibly more covariates than observations.\" Journal of Econometrics 189.1 (2015): 1-23.\n\n[4]. Knaus, Michael C. \"Double machine learning-based programme evaluation under unconfoundedness.\" The Econometrics Journal 25.3 (2022): 602-627."}, {"review_id": "RyrarU766G", "reviewer": "Reviewer_9YPQ", "summary": "This paper studies structure-agnostic estimation of a linear outcome model from Robinson [1988]. They explore the role of Gaussian treatment noise in this problem, first showing that double machine learning is optimal for Gaussian noise and then showing that it is (always) sub-optimal for non-Gaussian noise. They develop a new approach using cumulants that is more robust in the non-Gaussian settings.", "questions": "Line 94 and Line 107: What is \\mathcal{X} in Defn 2.1. Did you mean \\mathcal{Z}?\n\nCan you comment on the relevance of non-Gaussian treatment noise in real-world settings?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The results provide insight into the limitations of DML under non-Gaussian treatment noise. I am not 100% sure how likely non-Gaussian (and non-Binary) treatment is in real-world applications.\n\nSome might say the empirical results are limited, but I do not believe this is a limitation because the contributions are theoretical.\n\nThe paper heavily references two papers from 2023 and 2024. I personally have not read either of those papers (and did not have time to review them during the review period). The paper is not as accessible as it could be. It would be nice to have a concrete example, such as a set of structural equations and a description of what a nuisance function (relative to the treatment effect) is and what different approaches would do to estimate the treatment effect. This might also help provide some intuition for the role of normality and why you can make improvements when treatment noise is not normal.\n\nThe labels of the figures are a little small. It would be easier to read them if you made them a little larger.", "was_revised": true, "final_justification": "My only real criticism was a lack of accessibility, which the authors addressed in their rebuttal with a nice explanation. I argue for acceptance.", "rebuttal": "We thank the reviewer for providing insightful comments and for appreciating our work. Below is our response to the weaknesses and questions raised by the reviewer.\n\n**Weakness 1. The paper is not as accessible as it could be. It would be nice to have a concrete example, such as a set of structural equations and a description of what a nuisance function (relative to the treatment effect) is and what different approaches would do to estimate the treatment effect. This might also help provide some intuition for the role of normality and why you can make improvements when treatment noise is not normal.**\n\nWe thank the reviewer for this nice suggestion. In this paper, we consider a semiparametric model with partial linear outcome: $$Y = \\theta T + f(X) + \\epsilon, T = g(X) + \\eta.$$ The nuisance functions are regression functions that can be estimated from data and are used in estimating the target causal quantity. In our setting, we want to estimate $\\theta$. This can be done in several different ways. Firstly, given observational data $(X_i,T_i,Y_i), i=1,2,...,n$ and some nuisance estimate $\\hat{q}(x)$ for $q(X)=\\mathbb{E}[Y|X]$, [1, Section 1.1] derived an estimator\n$$\\hat{\\theta} = \\left(\\sum_{i=1}^n T_i^2\\right)^{-1}\\sum_{i=1}^n T_i(Y_i-\\hat{q}(X_i)) .$$\nThe error of this estimator is linear in the error of $\\hat{q}$. This error rate can be further improved by DML:\n$$\\hat{\\theta} = \\left(\\sum_{i=1}^n (T_i-\\hat{g}(X_i))^2\\right)^{-1}\\sum_{i=1}^n (T_i-\\hat{g}(X_i))(Y_i-\\hat{q}(X_i)),$$ where $\\hat{g}$ is an estimate of $g$.DML is less sensitive to the errors of the nuisance estimates. Mathematically, this insensitivity means that the first-order functional derivatives of these nuisance functions vanish, as in our Definition F.1.\n\nMore generally, if the functional derivatives up to the $k$-th order all vanish, then we can achieve a $(k+1)$-th order error rate, according to Taylor’s formula. Our work shows that this is indeed possible when the treatment noise is not normal. Our estimator is of the form \n$$\\hat{\\theta} = \\left(\\sum_{i=1}^n (T_i-\\hat{g}(X_i))J(T_i-\\hat{g}(X_i))\\right)^{-1}\\sum_{i=1}^n J(T_i-\\hat{g}(X_i))(Y_i-\\hat{q}(X_i)),$$\nwhere $J(\\cdot)$ is some fixed function. Obviously, this is a generalization of the DML estimator. Our key insight is that there exists a $k$-th order polynomial $J$ such that $(k+1)$-th order insensitivity can be achieved. However, we also require that $E[(T-g(X))J(T-g(X))]\\neq 0$, since otherwise the denominator would converge to zero. For the $J$ that we choose, the left-hand side is shown to be the $(k+1)$-th cumulant of the treatment noise $\\eta=T-g(X)$. Hence, when $\\eta$ is Gaussian, this non-zero condition can never be satisfied. On the other hand, when $\\eta$ is non-Gaussian, there must exists some $k$ to make this approach work.\n\n\n**Weakness 2. The labels of the figures are a little small. It would be easier to read them if you made them a little larger.**\n\nWe thank the reviewer for this helpful feedback. We will make them larger in a revised version.\n\n**Question 1. Line 94 and Line 107: What is \\mathcal{X} in Defn 2.1. Did you mean \\mathcal{Z}?**\n\nYes, it should be \\mathcal{Z}. We thank the reviewer for pointing out this typo.\n\n**Question 2. Can you comment on the relevance of non-Gaussian treatment noise in real-world settings?**\n\nIn real-world settings, Gaussian treatments are rare except when a controlled study is conducted, and non-Gaussian treatments are therefore much more common. As a result, we believe that assuming non-Gaussian noise is quite reasonable in practice. A canonical example (detailed in Mackey et al. and simulated in our Sec. 6 experiments) is demand estimation from purchase and pricing data. Here, Y represents observed demand, the treatment T corresponds to an observed product price, $g_0(X)$ denotes a baseline product price determined by covariates X that influence pricing policy, and the treatment noise $\\eta$ represents a random discount offered to customers for demand assessment. Notably, $\\eta$ is typically discrete (and thus distinctly non-Gaussian) and independent of X.\n\n[1]. Chernozhukov, Victor, et al. \"Double/debiased machine learning for treatment and structural parameters.\" The Econometrics Journal (2018): C1-C68."}, {"review_id": "YTcyw3kOZO", "reviewer": "Reviewer_y6th", "summary": "The authors investigate the problem of treatment effect estimation in causal inference, specifically studying a partially linear model. A popular choice in the literature is double machine learning (DML), in which case the nuisance parameters (e.g. treatment propensity and covariate effects on the outcome) are estimated via black-box ML algorithms and plugged-in. The authors show DML is optimal under Gaussian treatment noise. However, for non-Gaussian noise, the authors resort to constructing new, higher-order, cumulant-based estimators. The authors also derive minimax lower bounds for binary treatments.", "questions": "__Major__\n- I have a high-level confusion about a comment made in Section 5.1. The authors note that the fast rate of Theorem 5.1 holds specifically for estimating cumulants, and that estimation of moments runs into a bottleneck (stated in Theorem H.1). At a high-level, moments and cumulants are in 1-1 correspondence (explicit formulas given in terms of Bell polynomials). Naively, I would have the impression that if there is no bottleneck for cumulant estimation, then there ought to be no bottleneck for moment estimation. Can the authors provide high-level comments why this naive impression is incorrect? It would also be nice to elaborate in the paper.\n\n- Related to the previous comment, I see that the cumulant estimators in Theorem 5.1 are essentially just plug-in estimators (plug in $\\hat{g}$ and then compute cumulants of the empirical residual distribution). Plug-in is sensible to me since we are in a structure-agnostic setting. But why can't one do the same for moment estimation? Why does the bottleneck appear for moments but not cumulants?\n\n- In the proof of Theorem 3.1, it would be nice if the authors could provide some high-level discussion about what is the non-trivial part of constructing the set of hypotheses for using the method of fuzzy hypothesis (which they claimed right after the statement of Theorem 3.1).\n\n- The lower bound construction of Theorem 3.2 involves moment matching, which is a technique that has been successfully used in other problems (especially, functional estimation problems). In these other settings, it is very commonly the case that the optimal estimator turns out to be based on polynomial-approximation ideas (i.e. a method-of-moments estimator). In fact, there are duality arguments in some settings which immediately give a certificate of optimality (see Wu and Yang, \"Polynomial methods in statistical inference: theory and practice\"). In this paper, the authors go for a cumulant-based estimator, instead of moments. Could their moment-matching lower bound construction, paired with the duality perspective, yield a moment-based estimator? Also, since the lower bound is based on moment-matching, does that imply that the moment estimation bottleneck the authors identify is not actually fundamental, but just an artifact of analysis?\n\n- In the abstract, the authors claim DML is always suboptimal for independent non-Gaussian treatment noise, but in Section 3.1 they show optimality of DML for binary treatment?\n\n- In Section 5, the authors claim Theorem 4.1 gives better rates than DML. But I don't see that they provided lower bounds specifically for DML that are worse than the upper bounds for ACE. Can they provide these?\n\n __Minor__\n- On page 4, should \"Gaussianility\" be \"Gaussianity\"?\n- Theorem E.1 has a broken hyperlink.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The paper is quite strong in my opinion. DML is widely used, and the authors address a fundamental question about it, and obtain an interesting result which comments on a curiosity identified in the earlier work of Mackey. The paper is original, the writing and exposition is clear, and the mathematical execution is impressive. In my opinion, there are no major weaknesses.", "was_revised": true, "final_justification": "The authors addressed my questions, and I retain my score.", "rebuttal": "We thank the reviewer for the positive and constructive feedback and are elated that the reviewer found the paper “original” and “quite strong”, the writing and exposition clear, and the mathematical execution “impressive.” We respond to each point raised below.\n\n**Question 1. I have a high-level confusion about a comment made in Section 5.1. The authors note that the fast rate of Theorem 5.1 holds specifically for estimating cumulants, and that estimation of moments runs into a bottleneck (stated in Theorem H.1). At a high-level, moments and cumulants are in 1-1 correspondence (explicit formulas given in terms of Bell polynomials). Naively, I would have the impression that if there is no bottleneck for cumulant estimation, then there ought to be no bottleneck for moment estimation. Can the authors provide high-level comments why this naive impression is incorrect? It would also be nice to elaborate in the paper.**\n\nThe most direct explanation for the differing error rates between moments and cumulants lies in the fact that when cumulants are expressed in terms of moments, the leading terms of the moment estimation error cancel out, leaving only higher-order residuals.\n\nThe formal justification of this property of cumulants can be found on page 44 of the paper, starting from line 1140. The key observation is that the residual of nuisance estimates contributes additively to the cumulant generating function of the treatment noise, due to their assumed independence.   \n\n**Question 2. In the proof of Theorem 3.1, it would be nice if the authors could provide some high-level discussion about what is the non-trivial part of constructing the set of hypotheses for using the method of fuzzy hypothesis (which they claimed right after the statement of Theorem 3.1).**\n\nThe method of fuzzy hypothesis requires constructing two sets of distributions that are difficult to distinguish. In view of standard tools for applying this method (see e.g., Theorem 2.1), it is important to achieve the following simultaneously: \nThe mixtures of the two groups of distributions are the same. This property is guaranteed by appropriately scaling the “bump functions” used to perturb the nuisances. We verify this property in eq.(24).\nThese two groups yield well-separated values of the parameter of interest (which is the $\\theta$ in our setting, with separation distance being the final lower bound that we want to prove. \n\nTo the best of our knowledge, the only lower bounds for the structure-agnostic setting we consider are established in [2,3], for several different causal parameters. The constructions for these different parameters are established in a highly case-by-case manner.\n\n**Question 3. The lower bound construction of Theorem 3.2 involves moment matching, which is a technique that has been successfully used in other problems (especially, functional estimation problems). In these other settings, it is very commonly the case that the optimal estimator turns out to be based on polynomial-approximation ideas (i.e. a method-of-moments estimator). In fact, there are duality arguments in some settings which immediately give a certificate of optimality (see Wu and Yang, \"Polynomial methods in statistical inference: theory and practice\"). In this paper, the authors go for a cumulant-based estimator, instead of moments. Could their moment-matching lower bound construction, paired with the duality perspective, yield a moment-based estimator? Also, since the lower bound is based on moment-matching, does that imply that the moment estimation bottleneck the authors identify is not actually fundamental, but just an artifact of analysis?**\n\nWe thank the reviewer for connecting our results with the duality approach in the literature. Here, the lower bound that applies moment matching technique is for the case of Gaussian treatment. In this case, we show that DML is already minimax optimal up to logarithmic factors (Theorem 3.2). DML can be viewed as the simplest moment-based estimator in our setting. By contrast, cumulant-based estimators are designed only for the non-Gaussian treatment setting, because all cumulants of a Gaussian distribution are zero, which would violate the identifiability assumption.\n\n**Question 4. In the abstract, the authors claim DML is always suboptimal for independent non-Gaussian treatment noise, but in Section 3.1 they show optimality of DML for binary treatment?**\n\nWe will clarify in the revision that both of these claims hold simultaneously as the sub-optimality result concerns only distributions with independent treatment noise $\\eta$, while the binary minimax lower bound allows for the treatment noise $\\eta$ to depend on $X$.\n\n**Question 5. In Section 5, the authors claim Theorem 4.1 gives better rates than DML. But I don't see that they provided lower bounds specifically for DML that are worse than the upper bounds for ACE. Can they provide these?**\n\nThat’s a great point – indeed, although we proved lower bounds for any estimator in the binary and Gaussian treatment settings, we did not prove lower bounds for DML under the assumptions of Section 5.1, i.e., independent non-Gaussian rate. Nonetheless, it is not hard to to see that in this setting, DML still has the same rate $\\Omega(\\epsilon_1^2+\\epsilon_1\\epsilon_2+n^{-1/2})$ even under these additional assumptions. Specifically, one can start from the equation following line 778, which provides an expansion of the error induced by the DML estimator. In the worst case, both $\\Delta_g$ and $\\Delta_q$ can be arbitrary functions with $L_2$-norm bounded by $\\epsilon_1$ and $\\epsilon_2$ respectively. Thus in the worst case, the second term can be $\\Omega(\\epsilon_1^2+\\epsilon_1\\epsilon_2)$ while the first term is $\\mathbb{E}[\\eta^2]+o_P(1)$. It follows that  DML could incur an $\\Omega(\\epsilon_1^2+\\epsilon_1\\epsilon_2)$ error in the worst case.\n\nTheorem 5.1, by contrast, gives upper bounds for ACE that have strictly better dependencies on $\\epsilon_i,i=1,2$ when $l\\geq 2$. Therefore, we can conclude that in the setting of \nSection 5, ACE is strictly better than DML.\n\nWe will add these arguments into the revised version of this paper.\n\n**Other minor issues.**\n\nWe thank the reviewer for pointing them out and will fix them in the revision.\n\n[1]. Robins, James, et al. \"Semiparametric minimax rates.\" Electronic journal of statistics 3 (2009): 1305.\n\n[2]. Balakrishnan, Sivaraman, Edward H. Kennedy, and Larry Wasserman. \"The fundamental limits of structure-agnostic functional estimation.\" arXiv preprint arXiv:2305.04116 (2023).\n\n[3]. Jin, Jikai, and Vasilis Syrgkanis. \"Structure-agnostic optimality of doubly robust learning for treatment effect estimation.\" arXiv preprint arXiv:2402.14264 (2024)."}, {"review_id": "9qjf17TMWh", "reviewer": "Reviewer_7tt3", "summary": "This paper focuses on a particular estimation task: Estimating the treatment effect in the partially linear outcome model of Robinson [1988], where the treatment effect is a constant across all individuals.  In this setting, this paper introduces a few contributions: First, it introduces new lower-bounds on the minimax estimation error in the framework of structure-agnostic estimation.  It shows that in two settings (binary and Gaussian treatment) these lower-bounds match the upper bounds that already exist for DML-style estimators up to constant factors.  Second, this paper introduces \"agnostic cumulant-based estimation\" (ACE) which gives robustness to higher-order errors in non-Gaussian settings, and illustrates the use of this approach on synthetic data.", "questions": "A few points in priority order.\n\n1. Regarding weakness #1: How robust is the theoretical performance of ACE to the assumption that the data-generating process follows the PLR setting?  What can we say about the performance of ACE in the more general setting considered in [Jin & Syrgkanis 2024] when using $l > 4$?  It is not strictly necessary for the authors to answer those particular questions, as they may be out-of-scope for this paper, but I would appreciate some additional discussion on this point: Getting more clarity on this point, even if the conclusion is \"it's not clear that ACE is robust\" would be helpful here.\n\n2. Regarding weakness #2: Is ACE minimax-optimal, or can you characterize settings where it (theoretically) performs better than the DML estimator?  Some discussion on this point would be helpful...if some discussion already exists on this point in the paper (or appendix), could you point me to it?\n\n3. Regarding weakness #3: Not a question, but a suggestion---I would have liked to see an experiment comparing DML against the \"end-to-end\" ACE procedure (including data-driven selection of $l$ as suggested in Appendix J).\n\n4. Finally, (copied from above): I was a bit confused trying to understand Theorem 3.1. It seems that the lower-bound includes a term $c_q \\epsilon_1^2$ as well as the product of errors $\\epsilon_1 \\epsilon_2$.  Am I correct to read this equation as implying that, if $\\epsilon_2 = O(n^{-1/2})$ but $\\epsilon_1 = O(1)$ (i.e., if the propensity score model is miss-specified), then the minimax estimation error does not go to zero?  That would seem counter-intuitive, in light of the \"double-robustness\" of DML methods like AIPW, which is consistent if either the outcome or propensity model is consistent.", "limitations": "I think the paper could be a bit clearer when it comes to limitations, primarily in the context of my Weakness #1 above.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nIn terms of originality, the results seem interesting and novel, although I'm not sure of the significance for practitioners (see weakness #1 below).  I particularly appreciated the results regarding the minimax optimality of DML in the binary and Gaussian settings.  The inclusion of simulations was helpful for demonstrating that the proposed method out-performs DML when including higher-order terms ($l = 5$) both in terms of bias and MSE.\n\nCaveat: I did not verify the proofs.  Moreover, while I am familiar with double ML, semiparametrics, etc, I am less familiar with the more recent line of work on structure-agnostic estimation.  While I have skimmed the references in this work, it is possible that I am missing other recent context from the literature.\n\n## Weaknesses\n\nI have ordered these weaknesses in order of relative importance, and will summarize the most salient points as questions at the end of the review.\n\n(1) **Significance of PLR setting**: The results in this paper apply in particular to the setting of partially-linear regression, where the treatment effect is homogeneous (the same for all individuals).  Moreover, the novelty of the results is tailored to this setting:  As noted by the authors, and as I understand it from my skim, [Jin & Syrgkanis 2024] establish the minimax optimality of DML in the more general (and realistic) setting where there is no homogeneity constraint.  Moreover, it is unclear to me how the ACE (with $l > 1$) and DML estimators compare in the setting where this assumption is violated.  To me, the assumption of homogeneous treatment effects is more worrisome than the assumption that $\\eta$ and $X$ are independent.\n\n(2) **Unclear (theoretical) comparison to DML**: I understand that when $l=1$, then ACE is equivalent to DML, and that ACE is novel in the case where $l > 4$ (see page 8, lines 206-209).  However, I found it difficult to see in Theorem 5.2 and elsewhere how the rates compare for different settings of $l$.  It is established that DML is **not** minimax optimal with non-Gaussian noise, but I didn't see a claim that ACE is minimax optimal, or a characterization of the settings where it provides a faster rate than DML. Some more discussion on this point would strengthen the paper.\n\n(3) **Unclear (empirical) comparison to DML**: I appreciated the inclusion of simulations, and I agree that they show the method out-performs DML with $l = 5$, as noted in \"strengths\" above.  First, however, there is a lack of clarity in Section 6 of the main paper due to moving so much of the content in into the appendix [Appendix J, page 51, lines 1253-1273 in particular]. Some of these details are quite important for interpreting Figure 1.  For instance, it was not clear in the main text that Figure 1(a) **only shows those ACE procedures where the MSE is lower than DML**.  I would have liked to see an experiment comparing DML against the \"end-to-end\" ACE procedure (including data-driven selection of $l$ as suggested in Appendix J).\n\n## Minor feedback\n\nI was a bit confused trying to understand Theorem 3.1: It seems that the lower-bound includes a term $c_q \\epsilon_1^2$ as well as the product of errors $\\epsilon_1 \\epsilon_2$.  Am I correct to read this equation as implying that, if $\\epsilon_2 = O(n^{-1/2})$ but $\\epsilon_1 = O(1)$ (i.e., if the propensity score model is miss-specified), then the minimax estimation error does not go to zero?  That would seem counter-intuitive, in light of the \"double-robustness\" of DML methods like AIPW, which is consistent if either the outcome or propensity model is consistent.\n\nOther minor issues / questions:\n* Unclear references in theorem statements:  Perhaps this is a personal preference, but I like for theorem statements to be relatively self-contained.  For instance, Theorem 5.2 contains the assumption that $\\eta$ is independent of $X$, but requires the reader to remember that $\\eta$ is defined in Equation (1), 7 pages earlier.  Likewise, $\\Phi$ and $\\Phi^*$ are defined in Definitions 2.2 and 3.1 respectively.\n* In Algorithm 1, is it assumed that sample-splitting has already occurred?  Is it the idea that $\\hat{q}$ and $\\hat{g}$ were estimated on different data than $\\mathcal{D}$?  If so that might be worth clarifying.", "was_revised": true, "final_justification": "I increased my score from 3 -> 4.\n\nIn my original review, I highlighted three weaknesses, in order of priority, and discuss here how my opinion has changed:\n(1) Significance of PLR setting:  The authors have convinced me that the setting is of sufficiently general interest that this concern should not be load-bearing in accept/reject decisions.\n(2) Unclear (theoretical) comparison to DML:  The authors gave a very clear answer to this question in the rebuttal, and my concerns are addressed.\n(3) Unclear (empirical) comparison to DML:  While I would still like to see the \"end-to-end\" comparison to DML, I understand that's outside the scope of the rebuttal process, and that the main contribution of the paper is theoretical, rather than empirical.", "rebuttal": "We thank the reviewer for the positive and constructive feedback and are delighted that the reviewer found our work “interesting and novel.” We respond to each point raised below.\n\n**Weakness 1. The results in this paper apply in particular to the setting of partially-linear regression, where the treatment effect is homogeneous (the same for all individuals).**\n\nThe reviewer is correct that the results of this work applies to the partially linear outcome model. In the revision, we will highlight that while the partially linear model is more restrictive than the fully non-parametric model, it is widely employed and applies to practical problems like demand estimation from purchase and pricing data (as studied in Mackey et al. and simulated in our Sec. 6 experiments). Here, $Y$ represents observed demand, the treatment $T$ corresponds to an observed product price, $g_0(X)$ denotes a baseline product price determined by covariates X that influence pricing policy, and the treatment noise $\\eta$ represents a random discount offered to customers for demand assessment. Notably, $\\eta$ is typically discrete (and thus distinctly non-Gaussian) and independent of $X$. In addition, [1, Section V] studied a partial linear outcome model where the outcome is the median property value of a county and the treatment is the TSP, a metric for air pollution, and the more recent work [2] used the same model to study the effect of income taxes on the property values. \n\n**Question 1.  How robust is the theoretical performance of ACE to the assumption that the data-generating process follows the PLR setting? What can we say about the performance of ACE in the more general setting considered in [1] when using $l>4$?**\n\nThank you for this excellent question!  We will answer it from two different standpoints:\n- If we want to apply the same ACE estimator to data that might not satisfy the partial linearity assumption, say that the true outcome model is $$Y = g(X)+\\Delta(X)T+\\epsilon$$, and we want to estimate $\\mathbb{E}[\\Delta(X)]$, then the standard DML estimator would incur a bias $$\\frac{Cov(Var(T|X),\\Delta(X))}{\\mathbb{E}[Var(T|X)]}.$$ For the ACE estimator, the second term in the moment equation becomes $J_k(T-m(X))$. Let $I_k(X) = \\mathbb{E}[(T-m(X))J_k(T-m(X))|X]$, then the bias is instead $$\\frac{Cov(I_k(X),\\Delta(X))}{\\mathbb{E}[I_k(X)]}.$$ As a result, there are two cases when the bias (of both DML and ACE) becomes zero. The first one is the PLR setting where $\\Delta(X)$ is constant. The second is that when $\\eta=T-m(X)$ is independent of $X$, then $I_k(X)$ becomes constant, while $\\Delta(X)$ can be non-constant. \nWe would like to point out that we were unaware of the second property of ACE, which might address the reviewer’s concern about the PLR assumption. However, we find that the higher-order robustness property no longer holds in this case.\n- If we want to design ACE-type estimators for the more general setting of [1], we must make additional assumptions on the model, since [1] shows that DML is already optimal without any further assumption. One possible assumption is the independent noise assumption; as noted in 1, this guarantees that ACE is asymptotically unbiased. However, it is unclear how and whether higher-order robustness can be achieved for this general model.\n\n\n**Question 2. Is ACE minimax-optimal, or can you characterize settings where it (theoretically) performs better than the DML estimator? Some discussion on this point would be helpful...if some discussion already exists on this point in the paper (or appendix), could you point me to it?**\n\nBy theorem 5.2, ACE estimator performs better than DML in terms of the dependency on the nuisance estimation errors $\\epsilon_i, i=1,2$. Specifically, the error is $O(\\epsilon_1^l\\epsilon_2+\\epsilon_1^{l+1})$, improving upon DML’s rate $\\epsilon_1\\epsilon_2+\\epsilon_1^2$. The price to pay is a possibly larger constant, which now depends on the magnitude of $(l+1)$-th cumulant of the treatment noise as well as some super-exponential terms in $l$ (as we noted in Remark 5.2, however, the dependency can be only exponential in some cases). Therefore, as the sample size goes to infinity, $\\epsilon_i \\to 0, i=1,2$ and ACE with order $l\\geq 2$ would eventually outperform DML.\n\nThe optimal choice of $l$ is discussed in Remark 5.3 and formally stated in eq.(15). Roughly speaking, when the nuisance estimation errors $\\epsilon_i,i=1,2$ are polynomial in $n$, then one can choose the order of orthogonality $l$ to be $O(1)$ while making the bias to be $O(n^{-1/2})$. Hence, the final error rate would be $O(n^{-1/2})$. This is the well-known theoretically optimal rate given noisy observations. Therefore, the ACE estimator is minimax optimal up to constant.\n\nWe will add more explanations to make these points more transparent in an updated version.\n\n**Question 3. First, however, there is a lack of clarity in Section 6 of the main paper due to moving so much of the content into the appendix [Appendix J, page 51, lines 1253-1273 in particular]. Some of these details are quite important for interpreting Figure 1. For instance, it was not clear in the main text that Figure 1(a) only shows those ACE procedures where the MSE is lower than DML. I would have liked to see an experiment comparing DML against the \"end-to-end\" ACE procedure (including data-driven selection as suggested in Appendix J).**\n\nWe thank the reviewer for the suggestions. We will definitely make the experiment part of the main paper more self-contained in the revised version. Also, we will illustrate an end-to-end ACE procedure to compare with DML.\n\n**Question 4. I was a bit confused trying to understand Theorem 3.1. It seems that the lower-bound includes a term as well as the product of errors . Am I correct to read this equation as implying that, if $\\epsilon_2=O_P(n^{-1/2})$  but $\\epsilon_1=O(1)$  (i.e., if the propensity score model is miss-specified), then the minimax estimation error does not go to zero? That would seem counter-intuitive, in light of the \"double-robustness\" of DML methods like AIPW, which is consistent if either the outcome or propensity model is consistent.**\n\nThis is a fantastic question! The reviewer’s confusion might be caused by different parameterizations of the causal model. As pointed out in [4, Section 4.1], there exists two different debiasing schemes for the PLR setting. The first uses the moment function $$(Y-\\theta T - f(X))(T-g(X))=0$$ while the second is what we used. Here $f(X)=\\mathbb{E}[Y\\mid T=0, X]$. Although in [4], the authors did not explicitly state the rates in terms of the nuisance errors, the presence of an $\\epsilon_1^2$ term can be seen from their Assumption 4.1(b), which is in contrast with a “truly doubly robust” rate in Assumption 4.1(a) for the alternative parameterization.\n\nHere are some explanations of why seemingly different rates are obtained. A notable feature of PLR is that, given either pair of nuisances $(f,g)$ and $(q,g)$, the model is not uniquely specified. This means that these two parameterizations are not equivalent. Moreover, the relationship of these three nuisances is given by $$f(X)=q(X)-\\theta g(X)$$. Even if we can estimate $\\theta$ accurately (e.g. via DML), having $\\epsilon_1$ error for $g$ and $\\epsilon_2$ error for $q$ would result in an $O(\\epsilon_1+\\epsilon_2)$ error for $f$. Plugging into the “doubly robust” rate we obtain $$\\text{error of } g \\times \\text{error of } f = O(\\epsilon_1(\\epsilon_1+\\epsilon_2))$$ precisely matching the bound we have. So, having a doubly robust error rate does not contradict with the lower bound we establish here.\n\nFrom a technical point of view, the reason why $\\epsilon_1^2$ appears in the bound, at least for DML, is that the variance $\\mathbb{E}[(T-g(X))^2]$ (see eq.(2)) needs to be accurately estimated. If the estimation error of $g$ is too large, then this step would fail.\n\nWe hope that our response addresses all questions that the reviewer has.\n\n[1]. Jin, Jikai, and Vasilis Syrgkanis. \"Structure-agnostic optimality of doubly robust learning for treatment effect estimation.\" arXiv preprint arXiv:2402.14264 (2024).\n\n[2]. Chay, Kenneth Y., and Michael Greenstone. \"Does air quality matter? Evidence from the housing market.\" Journal of political Economy 113.2 (2005): 376-424.\n\n[3]. Hull, Isaiah, and Anna Grodecka-Messi. \"Measuring the impact of taxes and public services on property values: a double machine learning approach.\" arXiv preprint arXiv:2203.14751 (2022).\n\n[4]. Chernozhukov, Victor, et al. \"Double/debiased machine learning for treatment and structural parameters.\" The Econometrics Journal (2018): C1-C68.\n\n[5]. Robinson, Peter M. \"Root-N-consistent semiparametric regression.\" Econometrica: journal of the Econometric Society (1988): 931-954."}]

Author rebuttal:

Meta-review: This paper studies partially linear model of Robinson, and proves that DML achieves optimality under Gaussian noise assumption. When using non Gaussian noise assumption, the paper shows structure-agnostic causal estimator can achieve better results. 

Strengths:
-- The paper studies a very important problem, considering the wide usage of DML in causal inference.
-- The paper provides solid theoretical justification.
-- Empirical results are also given.

Weaknesses:
-- There were some complains about the paper is not that accessible, which has been addressed in the rebuttal,

The reasons for accept: a good theoretical paper in causal inference.

All the reviewers have actively participated in the discussion. One reviewer raised their score. The main concerns were about the readability and the reviewers are happy about the clarification made by the authors in the rebuttal.
Decision: Accept (poster)