Debiased Machine Learning without Sample-Splitting for Stable Estimators
Authors: Qizhao Chen
Vasilis Syrgkanis
Morgane Austern
Venue: NeurIPS 2022 Accept (2022)
Keywords: Double Machine Learning
Stability
Causal Inference
Treatment Effects
Debiased Machine Learning

Abstract: Estimation and inference on causal parameters is typically reduced to a generalized method of moments problem, which involves auxiliary functions that correspond to solutions to a regression or classification problem. Recent line of work on debiased machine learning shows how one can use generic machine learning estimators for these auxiliary problems, while maintaining asymptotic normality and root-$n$ consistency of the target parameter of interest, while only requiring mean-squared-error guarantees from the auxiliary estimation algorithms. The literature typically requires that these auxiliary problems are fitted on a separate sample or in a cross-fitting manner. We show that when these auxiliary estimation algorithms satisfy natural leave-one-out stability properties, then sample splitting is not required. This allows for sample re-use, which can be beneficial in moderately sized sample regimes. For instance, we show that the stability properties that we propose are satisfied for ensemble bagged estimators, built via sub-sampling without replacement, a popular technique in machine learning practice.

Debiased Machine Learning without Sample-Splitting for Stable Estimators
Qizhao Chen
Vasilis Syrgkanis Stanford University Stanford, CA 94305
Morgane Austern Harvard University Cambridge, MA 02138 morgane www.aaaaatteeenngmaaaa....
Harvard University Cambridge, MA 02138 gizhaochen&g. harvard. edu
veyrgk@stanford edu
Abstract
Estimation and inference on causal parameters is typically reduced to general- ized method of moments problem, which involves auxiliary functions that corre- spond to solutions to regression or classification problem. Recent line of work on debiased machine learning shows how one can use generic machine learning estimators for these auxiliary problems, while maintaining asymptotic normality and root-n consistency of the target parameter of interest, while only requiring meaan-qquard-ccroo guarantees from the auxiliary estimation algorithms. The lit- erature typically requires that these auxiliary problems are fitted on a separate sample in cross-fitting manner. We show that when these auxiliary estimation algorithms satisfy natural leave-one-out stability properties, then sample splitting is not required. This allows for sample re-use, which can be beneficial in moder- ately sized sample regimes. For instance, we show that the stability properties that we propose satisfied for ensemble bagged estimators, built via sub-sampling without replacement, a popular technique in machine learning practice.
Introduction
A large variety of problems in causal inference and more generally semi cemi-parametric inference can be framed as finding solution to a moment condition:
M(0,g) .g)
[ 0))
M (00: go) =
where E Z is vector random variables that, apart from the target parameter Og
1R of
interest (we assume p O(1)) also depends on unknown nuisance functions go E G. which need to be estimated a flexible manner from the data. This framework has a long history the literature on
semi- parameeettt inference 38 26
36 5T 57 12 20
58
[3
42
53
37 6) which
analyzes following two-stage estimation process with sample re-use, when having access to n iid samples (Zi,..., Zn) and grows, treating the target parameter dimension p constant:
1. Obtain an estimate ğ E G of the nuisance function gu based on all the samples.
2. Return any estimate E θ that satisfies:
M, (0.0) Op 1122)
with
M₂ (0,0) 4
RR Σ ml mZZ
Given that the estimation of function ğ is complex non-parametric problem, it will typically only satisfy slower than parametric eиror rates. Moreover, in high dimensional settings, when machine learning techniques are used to estimate ğ. then the regularization bias of Ò will propagate to the final estimate f, leading to non-regular estimates and the inability to construct confidence intervals.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The literature on efficient semi- parametric estimation provides conditions on the moment function so that the influence of the estimation error of the nuisance function 0 of second order importance and does not alter the distributional properties of the second stage [26
13
59
7 &
TO
31 32
[x]
62]. This approach dates back to the classical work as well as the more recent work on locally robust or Neyman orthogonal conditions on the moment function [8 6 [7, typically referred to double or debiased machine learning. At high-level, the earlier literature on semi-parametric inference shows that if the moment function satisfies some form of robustness to nuisance perturbations and, importantly, long the space & used in the first stage estimation is relatively simple function class, in terms of statistical complexity, typically referred to as Donsker function class, then the second stage estimate is root-n consistent and asymptotically normal. Hence, one can easily construct confidence intervals for the target parameter of interest.
46
19 4
28
29
30
on doubly robust estimation and targeted maximum likelihood 550 4
50
55 59 5
44 44
Crucially the recent literature on debiased machine learning alters the standard two-stage estimation algorithm to introduce the idea of sample-splitting [1 60). In particular, instead of estimating on all the samples, the recent work on debiased machine learning [17 [8] 22 estimates 9 on a separate sample, or for better sample efficiency invokes "cross- "cross-fitting, where we train nuisance model on half the data and evaluate it in the second stage on the other half and vice versa. Sample splitting avoids the Donsker conditions that were prevalent in the classic aami-paraettri inference literature and only requires root-mean-squaree squared--rrrrrrrrrrrr (RMSE) guarantee of the estimate ğ of op (p(n=1//) 1/4).
However, sample splitting cross-fitting still leads to poorer sample usage, we can lose half of our data when training complex non non-parametric or machine learning models, which can be problem- atic in small and moderate sample regimes. Our main result is to show that root-n consistency and asymptotic normality of the standard algorithm, without sample splitting, can be achieved without 11/2
the Donsker property, but solely if one assumes that the first stage estimation algorithm leave-one-out stable, a relatively widely studied property in the statistical machine learning and As a leading example we show that our stability conditions are satisfied by bagging estimators formed with sub-sampling without replacement.
generalization theory literature [55 64 2 251
Recent prior work of [2 also considered asymptotic normality based on stability conditions. but as we expand in the main text, their requirement on the stability property much harsher than the one we derive here. Moreover, [2]] analyzes only special case of the class of moment problems that we consider here. For instance, in our leading example of bagging estimators, the prior result of 211 would require that the bias of the base estimator decays faster than 1/n, where n is the sample size, which is typically not the case. In contrast, our stability condition does not impose any explicit assumption the bias and solely requires that the sub-sample size m is o(Vn).
To simplify the regularity assumptions required for asymptotic normality, we focus on the case where mí Z; () is linear in 6, i.e.
m mZ 9) = a(Z:g) + +v(:g)
where a (Z: E REXP is a x p matrix and gZ: E IR' is a p-vector, and we denote with: (g) = E, (Z(:))
A(g) = (2
V(g) = [x)(((Zg)))
(g) = ((Z:)))
Many of the leading examples in semi mamii-paametii problems that arise in causal inference correspond to linear moment problems. We present below a representative set of problems that are widely used in the practice of causal inference.
Example 1 (Partially Linear Treatment Effect (5I). If one assumes that the outcome of interest Y is linear in the treatment, i.e. Y = TTT + (x) + e, with Elc T, X] = 0, then estimating the treatment effect Og boils down to solving the following linear moment:
m(Z; 0, g) = (Y - x(X) - 0'(T - x(X))) (T (T-p(())
where the corresponding true values of (q. are (X) = [X] po(X)=BBTT)
Example 2 (Partially Linear IV (77). If one assumes that the outcome of interest Y is linear in the treatment, i.e. Y = T (X) + c, but the treatment is endogenous (i.e. there are unobserved confounders) and one has access
random variable Z, that is referred to as an instrument, which correlates with the trealment but is un-correlated with the residual in the outcome equation, i.e. satisfies that Ble Z, X] = O, then estimating Oo boils down to solving the following linear moment:
m mZ = (Y - x(X)= - (T - ()) (Z- - r(X))
where the true values of (g, P,T) are (X) EY IX, (x = XX) ro (X) [[
2

---

Example 3 (Average linear functionals of regression functions). Consider class of moment func- tions of the form:
m (Z;8,9) = - - m2 - )
where = (g. ut). my is a linear functional of and the corresponding true value q is regression
function qo (o x E T,X X] and po Riesz representer of the functional Elmy Z:q) (see 20
for examples), For instance, in the case of a binary treatment T. where we have that Y = q x)+ +o
and Ele T, [] = O, then the average treatment effect Bu E'q(1, q(0, X)] ] is identified by moment of the latter type, with:
m& (Z: q(,, X) (0,
(Average Treatment Effect (ATE))
While if we have a target treatment policy a X -> (0,1), and we want to estimate average value fg we can identify do with a moment of the aforementioned type, with:
my (:9) (X) -901)))9(0,,,,0)) -(0, X))
(Average Policy Effect)
For completeness, we also include in Appendix H an extension of our results to nonlinear moment problems.
2 Asymptotic Normality without Sample Splitting
We start by providing an asymptotic normality theorem for semi- marimaareettric moment estimators without sample splitting and where the moment satisfies the well-studied property Neyman or- thogonality. Our theorem requires four main conditions: i) -at---a---uurd--rrree (RMSE) rates for the nuisance function estimates of (n (1/44,, ii) Neyman orthogonality of the moment with respect to the nuisances, iii) second-order smoothness of the moment with respect to the nuisance functions and, iv) stochastic equicontinuity of the Jacobian and the off'set part of the linear moment function as the nuisance estimate 9 converges to Go- For a vector R we denote with ||a|| the l2 norm and for matrix X E Rexp we denote with || the operator norm with respect to the & norm.
Theorem 1. Suppose that the nuisance estimate 9 E G satisfies:
lا 9oll Ex [l((())
[x = Op (n-1/2).
(Consistency Rate)
Suppose that the moment satisfies the Neyman orthogonality condition: for all g
i M(O:: 90 (g -90))
M00 90) gol
(Neyman Orthogonality)
ōt
and a second-orded smoothness condition: for all g E g
+
Dgy M (fo. gg gu]
M(0. 9u t (g go)) 0 (1g-9ull - 90ll))
(Smoothness)
0t²
and that the moment m satisfy the stochastic equicontinuity conditions:
Vn A(ğ) A(go) (A. (0) An(o)))loop op(1) Vn [[V (goo (g) (an))) op()).
(Stochastic Equicontinuity)
Assume that A(go)-1 exists and that for any g E G:
AA(g) A(g'))l
opop O ( g'll2).
Moreover, assume that for any i, E [p] x (p), the random variable ar. go) has bounded variance and that [|0oll2 = =O(1). Then θ is asymptotically normal:
Vil (0-0) 0o)
m-odd N (0, A(go) 'EE [m 0g 90) ml ((; 00.900)))) T] A(go)=1).
3

---

The first three conditions are standard assumptions in the literature on debiased machine learning. The final condition (stochastic equicontinuity) is exactly where sample splitting comes very handy in the literature. To illustrate the reason why sample splitting helps with the stochastic equicontinuity condition, let us consider the first part of the condition (the reasoning analogous for the second part). It asks that the difference of two centered empirical processes, namely A, (ñ) - A(ğ) and (90) - A(g)), goes to zero faster than n n-1/2, We expect each empirical process to go down to zero at exactly n -1/2 and so this condition asks, since 9 converges to 9o is the empirical process continuous in its argument and for that reason does the difference converge to zero faster than each individual component. If the estimate I was fitted on separate sample, then conditional on 9. we have that each element = (i.j) E [p] x [p] of (n( - (g))) is an empirical average of iid random variables with mean A(g) A(go). Thus simple Bernstein inequality would show that the difference of the two empirical processes would converge to zero at the order of:
Og (V B ((((((2:))) ar (2:)0)))
(V
11 90
1 n2
+
Op
= op 11/3)
n
where we also invoked a mean- -quared-continnne property of oz (Z:g) and the fact that Il golla (1). Thus, with sample splitting, no further constraint is required from 9. other than a convergence rate on || golle- In fact, was noted in recent work of [22], in the above step it suffices to assume that [a (Z: - a. (Z; ]o = (Il - (l) for any q < oo, which is much weaker
mean-squared-conniniig assumption, and the property would still hold, since ll g0 000l222n n-1/2_ o(n-1/2), whenever || 90ll2 (1)
Without sample splitting, note that 9 is now correlated with the samples in the empirical averages and hence A,, An (go) is no longer an average of i.i.d. random variables. Typical approaches would try to prove a uniform stochastic equicontinuity property over the function space G. typically referred as a Donsker property of the function space G. In particular, if we could show that w.h.p.:
(V
(v [B[((u(220) ar (z())))))
Vg E O
-0 (o l1g golla + 52) O (62 + llg 90ll2)
n
then the above property would also hold for و. Subsequently, since we know that 1 - gu 12 = (n. by our convergence rate assumptions on 9. then it would suffice that 8% = onn 1/2).
Such localized concentration incqualities have been known hold for Donsker classes, which are typically defined via entropy integrals, and more recently it was also noted that such inequalities are satisfied with 8n being the critical radius of the space G, defined via localized Rademacher complexities.
However, the latter approach is conservative as it requires a uniform control over the function space G and does not utilize at all the properties of the estimation algorithm itself. In particular, as we will show in the next section, the main result of our work is that this stochastic equicontinuity condition follows from o(n -1/2) leave-one-out stability conditions on our estimation algorithm, which are typical in the machine learning literature and in the excess risk and generalization bounds literature.
3 Stochastic Equicontinuity via Stability
We will show that the Condition (Socchstiic Equicontinuity is satisfied, whenever the estimate 9 sat- isfies leave-one-out stability properties and the moment satisfies the weak
property of (22]. We start by some preliminary definitions required to state our stability conditions. Define ( as the data Zi,..., 2n, with the 1-th data point Z₁ replaced with an independent copy 7, Define -12) as the data Zi,..., 7n with both the 41-th and the 12-th data points Zi, Z replaced with independent copies 21, Ziz. Define similarly for 2(241.-12--l)) and so on. Let -(- be the estimator trained on z(-1) instead of Z,,..., Zn Similarly, let g(-(1.-12 be trained on (12 and so on. Moreover, we will always denote with Z a fresh random variable drawn from the distribution of the samples, but which is not part of any training sample. For any random variable X, we denote with ||X||1 == X]] with |||]| == / E 2 and with |||||p = (F(())))))/> for any p > in general.

---

Lemma 2 (Main Lemma). If the estimation algorithu satisfies the stability conditions: for all
i,j
[p]
ai (Z) - ar.j (2,, ((
- o(n 0112)
0i. (2.5 12,
1//0
1/12 = o(n 1/2)
max IE|m]
max
ME(m]
max vi(Zi, 9) vs (Zz. 0 --10)/1111 = ((2) [E(n]
(x(Z. (2, = o(n -1/2)
max [E|n]
and the moment satisfies the mean-squured--ontiniiss condition: 'g.g': [E((ar,(Z:g)) - رuا (Z: g & Lilg - 9'1
E[(v(Z:g) - vs (2: S Lilg g 11
for some 0 < 9 < oo and some L 0, then the Condition Stochastic Equicontinuity is satisfied. Remark 1. We show in section 3.7 that the stability conditions are tight. We present a counter is exactly of order n-1 and which the Stochastic example for which (z. v. (Z, 0 |
Equicontinuity condition is not satisfied.
Remark 2. We note that prior work of 12TV that established asymptotic normality without sample splitring via stability, required significantly stronger conditions than what we invoke here. In partic- ular. if we let Bn be the stability of the estimator 9 as measured by the quantities in Lemma X then the prior work of [2IV, would require that nB, llğ golla + 0. If we only know that ll - 90ll22 o(n-1/4), then the above would require 8., = o(n-3/4), which is much slower than o(n 1/3). Moreover, for bagged kernel estimators that we analyze in section 4 the prior work would require that if we use bags of size m, then the bias of the base estimator with m samples, denoted as bias(m) satisfies that mbias(m) + 0. This would rarely be satisfied and in prior work, the only concrete case that was given was forest estimators with binary variables under strong spar- sity conditions, in which case the bias decays exponentially with the sample size. For more general estimators, we expect bias(m) = 1/mm,, " some a. such settings, our work still applies and, as we show in section 7 gives results for bagged 1-nearest neighbor estimation algorithms, which do not satisfy any entropy or critical radius bound, but are stable. The key innovation that enables our improved results is a "double centering" approach that derives intuition from techniques invoked in the analysis of cross-validation via stability and the proof of the Efron-Sterr inequality This idea has already been used in the study of the cross validated risk 14 5
Proof of Main Lemma. We will show the first part of the lemma, i.e. that if for all i [p]
ar.5 9) anj (Z
||] -|)
o(n 1/2) max IE(n]
04.3 (Z. 01 (Z, gf
-()))| ||| o(n -1/2)
max /E|m]
then
Vn ||A(]) A(go)- (A, An (g0)))l - -op (1)
The analogous statement for and V. follows in an identical manner.
Since A(g) and An are p X p matrices and p = (1), it suffices to show the above property for every element (i,j) E [p] x [p]. i.c that
Ar, (G) Acz(0o) - (An,is An = (1)
For this it suffices show that:
Jn = /n A (0) - (9o) - (An.i. (0) - A o(1)
In the remainder of the proof we look at particular (i,j) and hence for simplicity we overload notation and we let a == 0i.j and A := A By triangle inequality and domination of IP norms we have
Jn =
1
(a(Z. A(g) [a(Z), A(co)]]
1
a(Zz.)) - alZi, alZi + V max /E(m
A(0) A(G
Vn
22
Σ
(a(Zi, ( a(Zi, go) - (A(g(-0) - -
+
5

---

To case notations, we denote
1
J1,a =
a(Zi; a(2i, 00
J2,n = vn max
AA()) A
(E|m]
Js.n == Vt E (a(zzzz00-1) (aa0-aaaZzza a(Z), - (A(g(-)) - A(ww))}
Now have that by triangle inequality
1 Σ a(.) - alZi, alZi, ---0),11 S vn max
JI,n S Vn
[a(Z, - = =
RE(n]
Similarly we can handle Jz
Jnn - v max a(Z,g) - a(2,g(-1) (E(n)
S Vn max (e|n
a(Z.g) - a(Z,g(-1) =(1)
We now aim to show that J3,, = o(1) Write for simplicity
k== al 21, 0) a go) - (A(G(-0) -A(go)
Now by expanding the square obtain
J3 E E ( ( k.
max E [K]] + (n - 1) [E("]
max E[[K, Kig]
1171
To bound the first term, we have by mean-squared continuity:
E[K?] = E
(a(z), (1))) - -((4,)))) - (A(g(-0)- - A((o))))
(a(z,, (1)) al alZz.90) )0)) 8
+ + (A(Ol-D) - A(gu)) - 27
< 2E
= 2E E (a(2). g (1) aaZz,, 9o))
)²
2 |2<-0) []
E (Alg) - )
(z(1)) ]
+ 2E
< 2L E
|
90 |'] + 21 E
||00--)-00
- 90 [|]]
= 4L E [l|g - 1 10100-90116 = o(1).
where we invoked the property that || 9oll2 = (1). and the second to last equality exploited the
tower law. Thus maxielm] E [K]] = o().
Double centering. We now bound the term, (n - 1) max max E[K,, Kiy). Define, for simplic- ity, for l # l
K(aa) = a(Z2, (hhr,tt)))) al Zi, (0) - (A(g - A(gn)).
Note that k((a)) does not depend on the 1y data point and only function of 222)2) Zi,. Moreover, by the definition of A, noting that Zi is independent of ( and Z₁, is independent of
z( 12),
12);
E Ki 2 E [(((1.0000) (01) - A(g 11) -7))|( x(-] - E (a(Z). A(go)) A(oo)) A
E [[(a(Zn, (-0) A(0(-0) x600 [[( A(so)) =0

---

and
K (b) -(-1
a(Z),
10 (1,))))) ( - 2(-4:--4)
+ a(Zi, 90) A(go) 2 ((-
E aa(2), ( Alg -1,, (2)) 71-111--1 (2)
+ (a 21, ,, 90) A(go)) =
For simplicity, we show that (n K K2] = (1), i.e. we show that the second term vanishes for 1 and = 2. The same exact arguments generalize arbitrary (ı, 12 We begin by writing:
(n 1) (n 1) ( x13) K2]
since by tower law:
E K 2) K2
E K2 z(-
2) .2, kP'] =E E K2 z(-2)
x10]
Similarly by conditioning on - Zz and using tower law, show that
E [[[Ks x12) _ E E K1
k
1(11)
2z k
E
[ K1 2(-1), _/]] E k Z
Zz)
R
[[ E Ki z z(-1)
E
( 2
2(-1
K 0.
Hence, we have
(n 1) (n 1)E [ xx(3)
K2] (n (n-11)B
[ x(31) (
With identical arguments, the same equality holds for any indices 11, 12. By Cauchy-Sehie Schwarz:
max(n 1)TR(K, Ki,]= = max(n 1)B [Ki, - -K((aa) (Kig (((1)) )
4
L1l12
max(n 1) Krn, x((h>// ||2 Krs k
I 1/2
Thus for maxi, #1, 1)E_K, Ki,] = o(1) it suffices that: maxi, A1a Expanding the definitions Ki, and k((a), the above simplifies to:
Ki, - x((12) onn 11/2).
Kr, x(((a) |1, |a{ | g( A()(- - -(0 (a( + -
The latter is upper bounded by triangle inequality and Jensen's 's inequality by: Kr, - x |1 [a(2,, (((b)) al Z1, (1.-1
we denote with Z a fresh random sample not part of the training sets, then since Zi, is not part of z(bb), we have:
(1)) alZi,
(( (b))
-
|a((zgg-hi) a(a, " (1),))))))
2
= [[]((2))) (
where we also used the fact that 2,, only appears in the training sets of g( and - and can simply rename it to Zi,, they are identically distributed and both independent from all the other data points. Invoking the second of our stability conditions for a, we have that:
maxi, Ki, (((b)] o(n oln-)/2). Hence, Ja,n = o(1), which completes the proof.
7

---

3.1 Tightness of Stability Condition
We present here an example that shows that, without further structural constraints (on the moment or the function space 9), the stability condition we impose is required for stochastic equicontinuity
condition to hold. Let (X,) %idd uniffo, 1] be i.i.d uniform random variables. We set Y; := I[x; S 0.5) and set Zi := (Xi, Y.). For any I [0, I], we define c( 1( x) == arg min;gn IX; x Xa) and note that the
the function that returns the index of the nearest example to x in quantity Y Yo)Zu...)) a)) is nearest neighbour estimator. Let
g (Zim)(a y) == n n'y$I(y * Yo Yo(Zama)))
(Z.g)) : 9()
We remark that v(:, 9) does not satisfy our stability conditions as -v g (2)2 [122212211 is exactly of order 1/Vn, neither does it respect the stochastic equicontinuity property.
(n, 1] be i.d uniform random variables. We set Y; == T(X; S 0.5) Lemma 3. Let (X;) "
and sei Z4 : (X,; Y.). There are constants C, e > such that
Set go(2) == O then we have []) - go( 2 +
. Si 2 [1, 9) (2, 2 f.
VnlV (go) (V)(0) (90)) fs 0.
4 Application: Bagging Estimators
We remark that if the functions and satisfy certain I L.-Liichitz conditions then the stability conditions of lemma are implied by the algorithmic stability of the estimator 9. Those conditions are only marginally stronger than the condition of wrean-squrred-ooniiii
found in lemma 2
Corollary 4. Fix any constant > 1. Suppose that there is L < oo such that the estimation algorithm satisfies the following uniform I.2" -continuity condition: for all i.j E [p) and E [n]
1/r
( az.j (Z 01) < L
IE sup ll((x) - -0 (x) (x)
2M
(1)
1/r
([(,) - (2 00 (-))) < L E sup [p(xx) - (x) 112"
(L ((*--Contnnuity)
1/r
([(a),((:)) (Z ( 1)) < L. EE sup [g(xz) g (x)
(Z()) v (Z: (1)) ) < L E sup (م(x)) gf 11 (3)
Suppose in addition that
1/2r = =n
max IS
E EZz... sup [1((x)) - (x))
(Algorithmic Stability)
and if in addition mean-squared- condition is satisfied, then the conditions of lemma are satisfied.
The uniform -ontiiniiy conditions are going to be satisfied by most moment functions m(: -)) We notably show in the appendix that example I example and example S with general moment conditions satisfy our conditions.
The condition of (Algorithmic Stability is commonly assumed condition in recent literature 4 and is satisfied by various regularized empirical risk minimization estimators and stochastic gradient descent estimators [I5 24 Notably, is satisfied by bagged estimators. We show in theorem S that under very general conditions a bagged ensemble of any machine learning estimator is stable. Let Zi...., Zn E Z be an independent and identically distributed (i.id.) sample of size n. We sample uniformly randomly without replacement from these observations repeatedly and independently. each time taking a sample of size m, for a number of B times. We denote the resulting samples as
z := {Z), ..... 2 , E [B].
8

---

Let hm 2 + +RP be base machine learning estimator trained on m observations. This base estimator can be tree. a CNN, nearest-neighhour classifier, or any other type of machine learning estimator. The corresponding bagged estimator is
B Σ bm
(())
(
(Zi:m) (-).
B
We will show that subject to B and m being sufficiently big and mild moment conditions the base estimator hm the bagged estimator satisfies condition Algorithmic Stability..
Theorem 5. Fix any constants s, 2 2r such that 1/2 +
2F Assume B, m are sufficiently large:
m m=olvn) o(vn)
B B>> m -
and ussume the base estimator h has bounded moments:
max 15n
sup
Nmm (2) (z)
SC
for some constant 0. Then
gorithmic Stability is achieved:
(0(x)) - -g
max
sup
Isa
2r
Therefore if and v satisfy the condition then the condition
is satisfied.
In particular, we can take the base machine learning estimator to be the 1-nearest neighbor estimator. This specific choice leads to bagged estimator that be proved satisfy both stability conditions and our consistency rate condition in Theorem in regression settings where covariates of small intrinsic dimension.
Let (Z, (X,, Y=) be sample of size n, drawn independently and identically distributed from Z = (X, Here X E X Rd are known covariates and Y y C R is the response variable. be B independent samples of size m drawn without replacement from observations { A bagged 1-nearest neighbor (1-NN) estimator takes the following form:
As in our bagging setting, let 7.m..... 2
B 3Σ
g(x) = B
(x)==
1
b=1
where S1(x)) is the 1-NN of x in the set Zi. The estimator is used to estimate the conditional expectation (o(r) == IE[Y X =x]] z).
Lemma (Special Case of Theorem of 35 Assume that
The marginal distribution μ. of X1 satisfies (C, -homogetttyy the ball B(x,r):
(x,( r)) S Ca ((B(r((
Ya (0,1)
for some C,r
> O. Here d is referred to as the intrinsic dimension of the distribution.
The conditional expectation BY X = is fipschitzfunctton in the coordinates x.
The response variable is bounded in L -orm:
||Y|| < so.
Then the bugged I-NN estimator 9 with 2 = satisfies the following convergence condition:
[[o(x) - (x)x 0 (m (md) +0 mP log log( log((n//n/) (n/mm) mm)
In particular, we note that when < d 2 and m = On for some 0 < e S - - we immediately have that
E [lx(x) 90 (x)
oonnnnnnnn 1144).
achieving the convergence rate assumed in Theorem Moreover, provided that we choose B to be sufficienlly large such that
B B>>
B B2n 2012
the required stability conditions can also be satisfied.

---

5 Experimental Evaluation
We consider a synthetic experimental evaluation of our main theoretical findings. We focus on the partially linear model with a scalar outcome Y E R. a scalar continuous treatment T IR and many controls X E IR"=, where: T' = (o)) + T,, T ~ (0, 1), Y = 0oT + fo() + e, e (0, 1). Our goal is the estimation of the treatment effect Oo. while estimating the nuisance (X) + +f(( in a flexible manner. We will consider estimation based on the orthogonal moment presented in Example T We considered sub-sampled 1-nearest neighbor regression (NN) and sub-sampled fully grown (with only sample minimum leaf size) random forest (RF) regression (see Appendix A for the estimation of po and qo. regressing I on X and Y on X correspondingly. The true functions Po and fo are actually linear in our data generating process, with Px(X) X6 and fx(X) = %6 and where 3o and to have only one non-zero coefficient (1-sparse), and which is the same coefficient for both So and To- In other words, only one of the nz potential confounding variables X is actually a confounder.
functions px() and (X)
We evaluate the performance of the estimate for Og for a range of values of the sample size n and the dimension of the controls n2 and with or without cross-fitting. For the cross- fesss- estimates we used 2 splits. For each specification we draw 1000 experimental samples to evaluate the distributional properties of the estimate. We considered sub-sample sizes for the nuisance regressions based on our theoretical n.44 specification and for larger specifications too. We find that the estimate without pross-fitting typically has almost equal bias and smaller (and always comparable) variance (due most probably to the smaller mean squared error of the nuisances, since they are trained on larger sizes), especially in smaller sample sizes, and has better coverage properties, when the sub-sample size m o(n'/9 Moreover, the estimate is approximately normally distributed, even without cross- cross-fitting as is verified qualitatively via quantile-quantile (Q-Q) plots (c.f. Appendix AA
We also report the mean of the estimate of the standard error across the 1000 experiments, to evaluate the bias in the estimation of the standard error. We find that the estimate of the standard error is more accurate without cross-fitting, potentially because the estimate of the standard error does not incorporate the extra variance that stems from the sample-splittng process. This inaccuracy of the standard error estimate is most probably the reason for the worst coverage properties of the confidence intervals with cross-fittee fitting..
In summary, we verify experimentally that for stable estimators, with the theoretically required level of stability, sample splitting or cross- pross-fitting is not needed to maintain asymptotic normality, small bias and nominal coverage.
bias
std
std_est
cov95
hius
std
std. _ss_
cov95
50 m.
cvel
0.021
0.151
0.140
0.917
7= n==1
cv=1
0.008
0.186
0.135
0.826
cv=2
0.022
0.173
0.139
0.866
0V=2
0.011
0.196
0.136
0.811
5=50, m2
cvel
0.042
0.149
0.143
0.920
n= n2==2
0.011
0.193
0.139
2v=2
0.046
0.170
0,142
0.863
2V=2
0,020
0.196
0.138
0.831
n=100, =1
ov=1
0.012
0.104
0.100
0.931
12 no 100,
cyml
0.001
0.132
0.098
0.838
2v=2
0.014
0.119
0,100
0.884
oV=2
0.004
0.143
0.098
0.812
n=100, n._ =2
cval 0.030
0.110
0.101
0
1=100, =2
cy=]
0.003
0.135
0.098
0.824
cv=2
0.032
0.121
0.101
0.878
0.011
0.141
0.099
0.828
n=500, =1
cv=1 0.007
0.046
0.045
0.943
n=5000 02=1
cv=1
0.002
0.056
0.045
0.881
cv=2 0.007
0.049
0.045
0.920
ev=2
0.002
0.062
0.045
n=500, R2=2
cv=l 0.015
0.045
0.045
0.927
==500, m2
cv
0.003
0.056
0.045
0.878
cv=2
0.015
0.048
0.045
0.917
cvm2
0.003
0.060
0.045
0.851
n=1000 n1,=1
cv=l
0.003
0.032
0.032
0.946
1=1000 n==1
cv=1
0.001
0.039
0.032
0.883
cvw2
0.002
0.034
0.032
0.922
cy=2
0.000
0.043
0.032
0.849
n=1000, n2==
ov=l 0.010
0.032
0.944
==1000, n.=2
cy=] 0.001
0.039
0.032
0.891
cv=2
0.010
0.033
0.032
0.930
cV=2
0.000
0.043
0.032
0.853
(a) Sub-sampled 1-NN with m = 12 0.49
(b) Sub-sampled 1-NN with m = n 10/11
Figure 1: Comparison of bias, variance and coverage properties, with (cv=2) and without (cv=1) cross-fitting (sample splitting), for the estimation of the treatment effect in the partially linear model, when a sub-sampled I-NN estimation is used for the nuisance function estimation. n is the number of samples and n. the number of controls.
10

---

References
[1] Chunrong Ai and Xiaohong Chen. Efficient estimation of models with conditional moment restrictions containing unknown functions. Econometrica, 7166):1795-1843, 2003.
[2] Chunrong Ai and Xiaohong Chen. Estimation of possibly misspecified semiparametric condi- tional moment restriction models with different conditioning variables. Journal of Economet- rics, (11(1):5 43, 2007.
[3] Chunrong Ai and Xiaohong Chen. The semiparametric efficiency bound for models of sequen- tial moment restrictions containing unknown functions. Journal of Econometrics, 170(2):442- 457, 2012.
[4] Morgane Austern and Wenda Zhou. arXiv:2001.11111, 2020.
Asymptotics of cross-validation. arXiv preprint
[5] Pierre Bayle, Alexandre Bayle, Lucas Janson, and Lester Mackey. Cross-validation confidence intervals for test error. Advances in Neural Information Processing Systems, 33:1 155116339-63355 2020.
[6] Alexandre Belloni, Victor Chernozhukov, Ivan Fernández- Val, and Christian Hansen. Program evaluation and causal inference with high-dimensional data. Econometrica, 85(1):233-295 2017.
[7] Alexandre Belloni, Victor Chemoozukov, and Christian Hansen. dimensional sparse econometric models. 0rXiv:/201.02220 2011.
Inference for high-
[8] Alexandre Belloni, Victor Chernozhukov, and Christian Hansen. Inference treatment effects after selection among high-dimensional controls. The Review of Economic Studies, 81 (2):6008 650, 2014.
[9] Alexandre Belloni, Victor Chernozhukov, and Kengo Kato. Uniform post-selection infer- ence for least absolute deviation regression and other Z-estimation problems. Biometrika, 102(1):77-94, 2014.
[10] Alexandre Belloni, Victor Chernozhukov, and Lie Wang. Pivotal estimation via square-root lasso in nonparametric regression. The Annals of Statistics, 42(2):757-788, 2014.
[11] Peter Bickel. On adaptive estimation. The Annals of Statistics, pages 647-671, 1982.
[12] Peter Bickel, Chris AJ Klaassen, Ya'acov Ritov, and Jon Wellner. Efficient and adaptive estimation for semiparametric models, volume 4. Johns Hopkins University Press, 1993.
[13] Peter Bickel and Yaacov Ritov. Estimating integrated squared density derivatives: Sharp best order of convergence estimates. Sankhyā: The Indian Journal of Statistics, Series A, pages 381-393, 1988.
[14] Stéphane Boucheron, Gábor Lugosi, and Olivier Bousquet. Concentration inequalities. In Summer school machine learning, pages 208-240. Springer, 2003.
[15] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
[16] Jelena Bradic and Mladen Kolar. Uniform inference for diigh--iensionall quantile regression: Linear functionals and regression rank scores. arXiv:1702.06209, 2017.
[17] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C6, 2018.
[18] Victor Chernozhukov, Juan Carlos Escanciano, Hidehiko Ichimura, Whitney K Newey, and James M Robins. Locally robust semiparametric estimation. arXiv preprint arXiv: 1608. 00033, 2016.
11

---

[19] Victor Chernozhukov, Christian Hansen, and Martin Spindler. Valid post-selection and post- regularization inference: An elementary, general approach. Annual Review of Economics, 7(1 71446669668 2015.
[20] Victor Chernozhukov, Whitney Newey, and Rahul Singh. Double/de- machine learn- ing of global and local parameters using regularized Riesz representers. arXiv preprint arXiv:1802.08663 2018.
[21] Victor Chernozhukov, Whimey Newey, Rahul Singh, and Vasilis Syrgkanis. Adversarial esti- mation of riesz representers. arXiv preprint S.Kv:2201.00000 2020.
[22] Victor Chernozhukov, Whitney Newey, and Rahul Singh. A simple and general debiased machine learning theorem with finite sample guarantees. arXiv preprint arXiv:2105. 15197, 2021.
[23] André Elisseeff, Massimiliano Pontil, et al. Leave-one-out error and stability of learning algo- rithms with applications. NATO science series sub series iü computer and systems sciences, 190:1 190:1,,,,30, 2003.
[24] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference machine learning, pages 1225- 1234. PMLR, 2016.
[25] Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of the 33rd International Conference Interna- tional Conference on Machine Learning Volume 48, ICML' 16, pages 1225-1234. JMLR.org. 2016.
[26] Rafail Z Hasminski and Ildar A Ibragimov. On the nonparametric estimation of functionals. In Proceedings of the Second Prague Symposium on Asymptotic Statistics, 1979.
[27] I Ibragimov and Hasâminskii. Statistical estimation, vol. of. Applications of Mathematics,
1981.
[28] Jana Jankova and Sara Van De Geer. Confidence intervals for high-dimensional inverse covari- ance estimation. Electronic Journal of Statistics, 105-:220-, 2015.
[29] Jana Jankova and Sara Van De Geer. Confidence regions for high-dimensional generalized linear models under sparsity. 1rX1::1610.01333, 2016.
[30] Jana Jankova and Sara Van De Geer. Semiparametric efficiency bounds for high-dimensional models. The Annals of Statistics, 46(5):2336-239)) 2018.
[31] Adel Javanmard and Andrea Montanari. Confidence intervals and hypothesis testing for high- dimensional regression. The Journal of Machine Learning Research, 15(1):2899-2909 2014.
[32] Adel Javanmard and Andrea Montanari. Hypothesis testing in high-dimensional regression un- der the Gaussian random design model: Asymptotic theory. IEEE Transactions on Information Theory, 60 (0110)6522--654, 2014.
[33] Adel Javanmard and Andrea Montanari. Debiasing the lasso: Optimal sample size for Gaussian designs. The Annals of Statistics, 46(6A)::593-2622 2018
[34] Satyen Kale, Ravi Kumar, and Sergei Vassilvitskii. Cross-validation and mean-square stability. In In Proceedings of the Second Symposium on Innovations in Computer Science (ICS2011, pages 487-495, 2011.
[35] Khashayar Khosravi, Greg Lewis, and Vasilis Syrgkanis. Non-parametric inference adaptive to intrinsic dimension. arXiv preprint urXiv:1901.0377 2019.
[36] Chris AJ Klaassen. Consistent estimation of the influence function of locally asymptotically linear estimators. The Annals of Statistics, pages 1548-1562, 1987.
[37] Michael Kosorok. Introduction to empirical processes and semiparametric inference.
Springer Science & Business Media, 2007.
12

---

[38] B Ya Levit. On the efficiency of class of non--onnrnaee parametric estimates. Theory of Probability & Its Applications, 20(4):723-740, 1976.
[39] Alexander R Luedtke and Mark Van Der Laan. Statistical inference for the mean outcome under possibly non-unique optimal treatment strategy. Annals of Statistics, 44(2)7113, 2016.
[40] Whitney K Newey. The asymptotic variance of semiparametric estimators. Econometrica, pages 1349-1382 1382, 1994.
[41] Whitney K Newey, Fushing Hsieh, and James M Robins. Undersmoothing and bias corrected functional estimation. Technical report, MIT Department of Economics, 1998.
[42] Whitney K Newey, Fushing Hsieh, and James M Robins. Twicing kernels and small bias property of semiparametric estimators. Econometrica, 72(3):947-962, 2004.
[43] Matey Neykov, Yang Ning, Jun S Liu, and Han Liu. A unified theory of confidence regions and testing for high-dimensional estimating equations. Statistical Science, 33(3):427-444 443, 2018.
[44] Jerzy Neyman. Optimal asymptotic tests of composite statistical hypotheses. Probability and Statistics, page 4168444, Wiley, 1959.
[45] Jerzy Neyman. C (a)) tests and their use. Sankhyā: The Indian Journal of Statistics, Series A, pages 1-21, 1979.
[46] Yang Ning and Han Liu. A general theory of hypothesis tests and confidence regions for sparse high dimensional models. The Annals of Statistics, 45(1):158-195, 2017.
[47] Johann Pfanzagl. Lecture notes in statisties. Contributions to a general asymptotic statistical theory, 13, 1982.
[48] Zhao Ren, Tingni Sun, Cun-Hui Zhang, and Harrison H Zhou. Asymptotic normality and opti- malities estimation of large Gaussian graphical models. The Annals of Statistics, 43(3)9991- 1026, 2015.
[49] James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 10(429): 1995.
[50] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Analysis of semiparametric regres- sion models for repeated outcomes in the presence of missing data. Journal of the American Statistical Association, 90(429):106-12) 1995.
[51] Peter M Robinson. Root-n-consistent semiparametric regression. Econometrica: Journal of the Econometric Society, pages 931-954, 1988.
[52] B Toth and MJ van der Laan. TMLE for marginal structural models based on an instrument. Technical report, UC Berkeley Division of Biostatistics, 2016.
[53] Anastasios Tsiatis. Semiparametric theory and missing data. Springer Science Business Media, 2007.
[54] Sara Van de Geer, Peter Bühlmann, Yaáacov Ritov, and Ruben Dezeure. On asymptotically optimal confidence regions and tests for high-ding nigh-dimensional models. The Annals of Statistics, 2014.
[55] Mark J Van der Laan and Sherri Rose. Targeted Learning: Causal Inference for Observational and Experimental Data. Springer Science & Business Media, 2011.
[56] Mark J Van Der Laan and Daniel Rubin. Targeted maximum likelihood learning. The Interna- tional Journal of Biostatistics, 2(1). 2006.
[57] Aad Van Der Vaart et al. On differentiable functionals. The Annals of Statistics, 19(1):17--204 1991.
[58] Aad W Van der Vaart. Asymptotic Statisties, volume 3, Cambridge University Press, 2000.
13

---

[59] Cun-Hui Zhang and Stephanie S Zhang. Confidence intervals for low dimensional parameters in high dimensional linear models. Journal the Royal Statistical Society: Series B (Statistical Methodology). 76(1):217-242, 2014.
[60] Wenjing Zheng and Mark J Van Der Laan. Asymptotic theory for cross-validated targeted maximum likelihood estimation. 2010.
[61] Yinchu Zhu and Jelena Bradic. Breaking the curse of dimensionality in regression. 2017.
[62] Yinchu Zhu and Jelena Bradic. Linear hypothesis testing in dense high-dimensional linear models, Journal of the American Statistical Association, (135524):15833166 2018.
14


Reviews:
[{"review_id": "aNZHWu6T3Q", "reviewer": "Reviewer_v7GP", "summary": "The paper presents a theoretical argument that debiased machine learning does not require sample splitting when the estimation algorithms are leave-one-out stable.", "questions": "1. Could you explain in greater detail the level of difficulty expected in verifying the conditions of Lemma 2? What about compared to the level of difficulty verifying the conditions of Theorem 1? What about compared to conditions previously imposed in the literature? If the algorithmic stability conditions cannot be verified, what course of action would you recommend? Also, what precisely is the relationship between the \"Algorithmic Stability\" condition of Corollary 4 and the \"Consistency Rate\" condition of Theorem 1?\n\n2. Could you explain\n- the second to last equality in the unnumbered display equation following Line 183,\n- the second to last equality in the unnumbered display equation following LIne 186, and\n- the last equality in the unnumbered display equation following Line 198?\n\n3. Could you explain in greater detail the significance of Lemma 3? In particular, how does this example illustrate that \"the stability condition [you] impose is required for stochastic equicontinuity condition to hold\" when it is later stated that \"$\\nu(\\cdot, \\hat g)$ does not satisfy our stability conditions\"?\n\n4. Would it be possible to observe experimentally the effect of algorithmic stability on asymptotic normality? For example, the size of $m$ is related to the stability of the bagged estimator. Can we observe the relationship of $m$ to the convergence to normal distribution by looking at a grid of $m$'s (either $m = an$ or $m = n^a$ for different values of $a$)?\n\n5. I feel like the paper could use more editorial judgment. This is probably just my impression, but the current version feels like it used to be a longer paper that was abruptly cut to meet the page limit. I find that individual parts are written quite well, but the paper as a whole appears to be missing some significant pieces.", "limitations": "The authors do not discuss the limitations and potential negative societal impact of their work to any noticeable extent. Still, given the topic of research, I do not consider it a defect.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper makes a highly interesting theoretical contribution by showing that the \"Stochastic Equicontinuity\" condition is implied by a stability condition on the estimation algorithm (Lemma 2). This algorithmic stability condition is shown to be weaker than the conditions previously imposed in the literature (Remark 2). The contribution is also interesting from a practical point of view because it provides an explanation of why sample splitting is or is not needed for debiased machine learning.\n\nOne criticism about the result itself is that it may be difficult to check the stability conditions for arbitrarily complicated algorithms except in some special cases, e.g., bagged estimators. If this is the case, a practitioner may feel more safe sticking with the sample splitting approach. Also, the paper does not give any experimental results to back up its claims. Since the theoretical results describe the limiting behavior, it would have been helpful to have extensive experimental results to see how the claims hold up at finite sample sizes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P0ZMACuM2eu", "reviewer": "Reviewer_Becs", "summary": "This paper deals with the following problem:\nSuppose we are interested in solving the generalized moment problem \n\n$M := \\mathbb{E}_Z m(Z,\\theta, \\hat{g}) = 0$, in $\\theta$. \n\nHere $Z$ is a random variable, $m$ the moment function, $\\theta$ a parameter, and $\\hat{g}$ an estimator, that is learned from data. \nThe goal is to find $\\hat{\\theta}$ that makes the above expectation $M$ small. \n\n\n\nSuch a situation occurs, for instance, in treatment effect estimation settings.  In that case, $Z$ consists of $Z=(Y,T,X)$, where $Y$ is the outcome, $T$ is the treatment variable, and $X$ additional features. The $\\hat{g}$ to be learned from data is an estimator of $T$ and $Y$ as functions of $X$. \n\nTypically, to obtain bounds on the estimation of $\\hat{\\theta}$, one splits the sample into two parts. Then the first part is used to estimate $\\hat{g}$ and the second part is used to estimate the empirical minimizer $\\hat{\\theta}$ of $M$, based on $\\hat{g}$.\n\nIn this paper it is shown that if $m$ is a _linear_ function of $\\theta$ (e.x.  $Y = \\theta T + \\ldots$ in example above), and the classifier is chosen by an algorithm that has the algorithmic stability property,  then one can use the full same sample to execute both estimations above, and still have the unbiasedness and asymptotic normality property for the estimator of $\\theta$.\n\nIt is also shown that bagging type estimators have algorithmic stability under some conditions on the base estimator. As an example of this, \nthe authors show that bagging ensemble of 1-nearest neighbor classifiers has algorithmic stability and thus can be used as $\\hat{g}$ in the main result. \n \n", "questions": "Please see the discussion above. ", "limitations": "Limitations and potential negative societal impact were not discussed. ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "There are three serious issues with this paper.  **(1)** The first issue is that this paper is not well written. In terms of language and focus of presentation I believe it is not suitable for general ML audience in its current form. Moreover, in terms if technical presentation, it is also very hard to read. Definitions and references are hard to find around the paper.  \n**(2)** The second issue is that the connection between bagging and stability is well known. It is not completely clear why the authors prove these results, and claim them as contributions.  **(3)** The scope of the results is limited. Only linear moment functions are considered, and the results are about just asymptotic estimates rather than finite sample bounds. \n\n\nTo discuss these points in more detail: \n\n(1) This paper will be very hard to read to people who are not familiar with the generalized method of moments, treatment effect estimation and instrumental variables. I believe most NeurIPS participants are not familiar with these notions, although they are very familiar with the underlying mathematics. Therefore I would suggest to the authors to completely rewrite the paper, carefully introduce the problems  and their importance, before discussing the paper's contribution.  \n\n(2) Connections between bagging and stability are natural and known. See for instance [a] below. Even if the specific version of the results \nin this paper is new, it should be discussed in the general context.  How do the results presented in the paper differ from standard results?\nIn addition, while 1-NN classifiers are rarely used in ML communities. Thus I would suggest providing more realistic classifier examples. Would regularized kernel regression satisfy the stability assumptions used in the paper?\n\n\n\n[a]  Elisseeff, A., Evgeniou, T., Pontil, M., & Kaelbing, L. P. (2005). Stability of Randomized Learning Algorithms. Journal of Machine Learning Research, 6(1).\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B8IjcTAym1", "reviewer": "Reviewer_ZG2K", "summary": "New conditions on nuisance function based on leave-one-out stability that warrant root-n consistency and asymptotic normality of target parameters are proposed and theoretically investigated. It is also concretely shown that these conditions are satisfied by ensemble bagged estimators.", "questions": "(1)\tI believe this paper’s contribution is substantial. That being said, I wonder to what extent $m(Z;\\theta,g)$’s linearity in $\\theta$ can be relaxed. It would be great if some discussion along this vein could be included in the paper. If $\\theta$ and $g$ are not separable, I guess the stochastic equicontinuity in line 102 will need to hold uniformly in $\\theta$ for $\\theta$ in a neighborhood of $\\theta_0$? Would stability still be sufficient for stochastic equicontinuity? If $m(Z;\\theta,g)$ is nonlinear in $\\theta$, would linearization trick like Taylor expansion to first order work under some additional conditions?\n\n(2)\tIn Theorem 5, is the norm $\\|Y\\|_s = (E[Y^s])^{1/s}$? If so, this norm is not defined.\n", "limitations": "The moment condition has to be linear in target parameters.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ New result can be applied on causal inference with machine learning method, which is timely and impactful\n+ Theoretical merits are clearly demonstrated by comparing with relevant existing works\n+ New mathematical arguments that lead to refined theoretical results, which are shown to be unimprovable\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept