Asymptotics of the Bootstrap via Stability with Applications to Inference with Model Selection
Authors: Morgane Austern
Vasilis Syrgkanis
Venue: NeurIPS 2021 Poster (2021)
Keywords: bootstrap
asymptotics
confidence intervals
model selection

Abstract: One of the most commonly used methods for forming confidence intervals is the empirical bootstrap, which is especially expedient when the limiting distribution of the estimator is unknown. However, despite its ubiquitous role in machine learning, its theoretical properties are still not well understood. Recent developments in probability have provided new tools to study the bootstrap method. However, they have been applied only to specific applications and contexts, and it is unclear whether these techniques are applicable to the understanding of the consistency of the bootstrap in machine learning pipelines. In this paper, we derive general stability conditions under which the empirical bootstrap estimator is consistent and quantify the speed of convergence. Moreover, we propose alternative ways to use the bootstrap method to build confidence intervals with coverage guarantees. Finally, we illustrate the generality and tightness of our results by examples of interest for machine learning including for two-sample kernel tests after kernel selection and the empirical risk of stacked estimators.

Asymptoties of the Bootstrap via Stability with Applications to Inference with Model Selection
Morgane Austern
Vasilis Syrgkanis Microsoft Research vasy@microsoot..cmm
Microsoft Research morgane, aesteeeeeetttteemmmm.m.... com
Abstract
One of the most commonly used methods for forming confidence intervals is the empirical bootstrap, which is especially expedient when the limiting distribution of the estimator unknown. However, despite ubiquitous role in machine learning, its theoretical properties are still not well understood. Recent developments in probability have provided new tools to study the bootstrap method. However, they have been applied only to specific applications and contexts, and it is unclear whether these techniques are applicable to the understanding of the consistency of the bootstrap in machine learning pipelines. In this paper, derive general stability conditions under which the empirical bootstrap cstimator is consistent and quantify the speed of convergence. Moreover, we propose alternative ways to use the bootstrap method to build confidence intervals with coverage guarantees. Finally, we illustrate the generality and tightness of our results by examples of interest for machine learning including for two-sample kernel tests after kernel selection and the empirical risk of stacked estimators.
Introduction
Bootstrap resampling [26], has been one the most popular techniques measuring the uncertainty of statistic, primarily due to its simple algorithmic definition and its conveniency with dealing with opaque statistical procedures that produce a test statistic. For this reason, uncertainty quantification based bootstrap resampling has been a staple in the machine learning community, starting from the early days of the field [40] and continuing into the deep learning and SVM era [39, 30, 32, 17]. Despite it's widespread use, general conditions for the consistency of the bootstrap for complex non-linear statistics generally not fully explored and hence, it is not clear when the bootstrap method will accurately capture uncertainty in machine learning pipelines, especially when model selection procedures are involved.
Our goal is to understand the distribution a large class non-lincar statistic ò, := (X X.,). the samples X, drawn from their unknown distribution. Examples of such statistics could be the out-of-sample risk machine learning predictor, or the maximum- maaximum-mean-diiccoh mcan-discrepancy of a two-sample kernel test, or the prediction of machine learning model at given point. One approach to approximating this distribution is the empirical bootstrap: sample new observations Zı,... Zn independently and uniformly from {X1,...,XX) X (with replacement), and define @buot as the value of the estimator taken at the bootstrap sample Oboot : 9n (Z,,..., Zz). This procedure can be repeated many times in order estimate the conditional distribution of @boot If this distribution approximately the same as the distribution of 6.. as the sample size grows, we say that the bootstrap method is consistent. Consistency of the bootstrap can be subsequently utilized to construct confidence intervals for the quantity of interest.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

Classical consistency proofs for the bootstrap, either require that the limiting distribution of the statistic be Gaussian, or even more stringently that the statistic is asymptotically linear. Notably, when 0, is asymptotically normal those intervals are known to be consistent under general conditions; see c.g. [33,5, 5, 18]. Moreover, classical conditions for such statements to hold are that the statistic is Hadamard differentiable and smooth with respect to the underlying distribution of the random variables. However, these properties can be violated by test statistics that are implicitly defined via machine learning data analysis pipelines and especially when the model selection is involved. Thus it crucial to provide more generally applicable sufficient conditions for the validity of the bootstrap.
Notably, recent breakthrough results in statstics by Chatterjee [13], have shown that the Lindenberg technique that has been long used to establish Central Limit Theorems, is more widely applicable and can be leveraged to show limit distributional consistency of two random variables more broadly. Notably, this intuition has already been exploited to show that the bootstrap method is consistent in particular applications in the econometric literature, such as the construction of uniform confidence bands. However, these proofs are tailored to the particular application of interest and do not provide general characterizations.
Our main contribution is to provide set of sufficient stability conditions that imply the consistency of the bootstrap and which go well beyond the existing general conditions via the use of Chatterjee's s generalized Lindenberg approach and a smart path interpolation technique. We then apply these gen- cral results to derive inference in two machine lcarning applications: i) bootstrapping the test statistic in two-sample kernel tests, after model selection on the kernel and sample re-use, ii) bootstrapping the risk of a stacked estimator with sample re-use.
Roughly, our sufficient conditions impose unilateral stability properties on the statistic, reminiscent of the stability conditions required for classical concentration inequalities, such as McDiarmid's inequal- ity. In particular, we assume that the functions (gn) are approximable by three-times differentiable functions whose first, second and third order partial derivatives, with respect to a single observation, taken at (X,,...; X.)) are of respective order o(n - 1/3), o(n - 1/2) and o(n -1). These conditions assure that the value of 9n (X.,... X₂, ) is not oversensitive to the value of a single observation (see Section for a formal exposition). Exploiting these assumptions, we exactly characterize the limiting distribution of the bootstrap estimator @hoot compare it to the distribution of the original statistic 6,, and study how fast the distribution of @boot converges. Notably, we also discover that when the mean of the obscrvations X1 is unknown, then the bootstrap method is in general not consistent and we propose corrections to bootstrap based intervals with guaranteed minimum coverage.
Related litterature The empirical bootstrap method was first introduced in a breakthrough paper by Efron [27]. Other bootstraps methods have since been proposed including the multiplier bootstrap [54], the residual bootstrap [21] or the non-remplacement bootstrap method [47]. A vast literature studies the theoretical properties of those techniques with some of the main results synthesized in the following books [33, 21, 36, 3]. Most relevant to us are studies of the asymptotics of the bootstrap method. The consistency of the bootstrap method for linear statistics, t-statisties, Von- Mises functionals and quantiles has been established in [5, 46, 48] and for U-statistics in [1, 55J. Those results, among others, have been extended to high-dimensional regression and M-estimation [6, 45, 10, 2, 24], misspecified models [49] solutions of estimating equations [11] and to robust estimators [15]. In contrast, other works established the poor performance of the bootstrap method for non smooth statistics [25, 4, 5], or for non-sparse high-dimensional regressions [28].
Several recent breakthrough papers studied the consistency of the bootstrap method, both empirical and wild, for the maximum of high-dimensional centered averages with the dimension taken to be growing exponentially fast with the sample size. Notably [18, 19] established the consistency of the bootstrap and Gaussian approximation method when respectively log(p,.) 77/8 //3 and log(p.n)7/6 = / hold. A series of works have strengthen those results: [20, 23, 41] established the consistency of the multiplier and empirical bootstrap when log(pr))$/4 55/4 = o(n-/4),
[43] established a quasi vn" rate for the wild bootstrap. [22] built slightly conservative confidence sets with guaranteed coverage under the conditions that log(p) o(n) and [16] proved that similar results hold for high-dimensional U-statistics. Those works use a combination of the Stein method, Edgeworth expansions, Lindeberg's 's method [13] and the Slepian smart interpolation path. We note that the limiting distributions of those statistics are in general not Gaussian [23]. In contrast, our results apply more broadly and are not limited to the study of maximums of centered empirical
2

---

averages. We notably apply our results to machine learning estimators that are smoothed arg-minima of an objective function. Other works have studied the accuracy of the bootstrap method for specific statistics whose distributions are known to be asympiotically not Gaussian such as: the operator norm in high dimensions [44, 34, 37J, sampled eigenvalues of random matrices in high and moderate dimensions [29] M-estimators having cube root convergence [9]. The main contrast between this series of work and ours is that, rather than studying the bootstrap method for one specific statistic or application, seck establish the asymptoties of the bootstrap method under universal conditions on the estimators (gn). Our proof builds on breakthrough method proposed by Chatterjee [13] that generalized the Lindeberg method general technique for comparing the expectations of f(X1.s and (Y) ) of a large class of functions f.
2 Problem Statement
Let (Xf)) be triangular array of independent and identically distributed (i.i.d) processes with observations X²" taking value in Rd,, Morcover, let X (X1', .x........ X,)) .x.,),,,,;;) denote its n-th row. Consider an estimator 0,, := 9n (X"), where 9n, x R", - is mcasurable function, that we will typically refer to as statistic, and let (gn) denote the sequence of measurable functions as grows. To evaluate the performance of this estimator and build confidence intervals, we need approximate its distribution. In this work, we will analyze the empirical bootstrap method.
Empirical bootstrap Bootstrap samples 2 ........ Z are sampled with replacement from the observations {XT.... X=). This implies that conditionally on X the coordinates of 2n are distributed i..d, with Z," X" unif ((.f....,..) 1)) for all E [n]]
Consistency metric and bootstrap consistency Throughout the paper we denote with Y (YT', (YT"..... ...., Y
an independent copy of X The bootstrap method is said be consistent for (gx) if conditionally on X the distribution of 9n(2") well-approximates the distribution of 9n, (Y"), as n + co. To make this statement rigorous we introduce metric on the space of probability distributions. First, we define the class of three times continuously differentiable measurable functions with bounded third-order derivatives:
{ h c (RR) sup SER
F
10 (x)) 1, visis33;
Given this, we define the distance on the space of probability measures, as the maximum mean discrepancy, where test functions range over the class F:
d)( := sup Ex~nYY> [h(X) - h(Y)). hEF
We remark that dy is metric the space of probability measures of real-valued random variables. Notably for two probability distributions fa and y d v) = O then those distributions are the same V μ.. Moreover, the topology defined by dy is finer than the weak convergence topology. Indeed, for a sequence of distributions (v,)) if we have (2,,n, (4) + 0 then (/²)) converges weakly
to μc V/. 4 p. Finally we remark that this metric is related to the classical on probability spaces [7]. Morcover, wc use the shorthand notation:
distance
dy (1,, E) sup Ex~p,Y [h(X) h(Y) E].
heF
We say that the empirical bootstrap method consistent for (gn) if:
dy (zn) x") & 0.
Centering discrepancy and centered bootstrap consistency Notably, an individual bootstrap sample Zf X has slightly different mean E [ X X == Σign X", than the one of
XT. As we will see this small difference plays a crucial role determining the consistency of the bootstrap and for this reason it will be useful to define artificially centered versions of the random variables (ZT) and (Y."). ). centered bootstrap sample
2; = -T
(X" 1E [XT])

---

is a bootstrap sample that has been re-centered to artificially have the same mean as X7. Moreover, denote with Y corrected version of Y;", artificially re-centered to have the same mean as ZT, ie:::
Y : Y?" + X E [XT]
We say that the centered bootstrap is consistent for (gn) if:
dy (In(2 (Y"") (gn(Z"))9n("))) x 2 0.
From metric consistency to confidence intervals with nominal coverage We can compare the confidence intervals of random variables X and in terms of their mutual distance dy(X,, Y) (proof in Appendix M.1).
Proposition 1 Let X and Y be two real-valued random variables and & any random event. Let 6 > 0 be a constant then for any Borel set ((B)) the following holds:
P(X E Aoc 8) 2 P(Y E A 8 - dx
e²
where we wrote A. = (x E R By As.t x yl S e). Moreover, if (a, b) is a confidence interval at level a for Y E |Y| E), conditional on E, then we have:
2d 2dz
P(X E X E E (a- 6c, be 6c] 18)22
For instance, suppose that we care about estimating 0₂ = E 19n [Y").. Then the bootstrap method, if consistent, can be used build consistent confidence intervals for 0n. Indeed since we can estimate the conditional distribution of @bootstrap == 9n (Z"), by drawing sufficiently many bootstrap
sub-samples, we can find Cha such that
P @hoolstrap - E @huolstrap
X E Ca. xx -1-
Then, if we write 0, 9n (Y"), using the consistency of the bootstrap method obtain that:
lim inf lim inf P (0,, - On E C.a.)) 21-6 a.
t.0 n-
Therefore, confidence intervals built using the bootstrap method achieve asymptotically nominal level of confidence. We note that prior works (e.g. [18, 19]). typically provide slightly stronger a, by proving anti-concentration results on the limit distribution of 0, - 8n. Such anti-concentration, allows one to argue that the mass the random variable 0,- - On contained in Ca converges to the mass contained in Ca," as + 0 and thereby, lim inf lim inf,- De P(ê. On E C.a.)) lim inf, P(θ, 0n C C) Given that these results typically require stronger conditions on the statistic and many times Gaussian limits, we omit this step in this work and note that slightly weaker, albeit still practically useful, statement on coverage is achievable a more general setup.
statement that lim i P(O, An Caaa)
From metric consistency to p-values Alternatively, suppose that we want to test if a specific null hypothesis (Hğ) holds against the alternative (HT). To do so we compute test statistic (X1... and determine rejection region R². A crucial quantity to estimate is the p-value P (XTT..).)) E R RD Ho)) The bootstrap method, if consistent, allows us upper-bound the p-value by enlarging the rejection region with an infinitesimally small quantity. Indeed according to Proposition if the bootstrap method is consistent then we have
lim lim inf P (Ta (Ta(Z E R"IX")) 2 lim P (TT(XPP.)) E RR).
<Lo n-000
x2-000
2.1 Notations and definitions
For scalar random variable X we denote with |X | the Lp-norm: |X|| := 1(P More- 1/8
over, for vector E Idd, we denote with || lp- the ep vector norm: ||=||p (Σn_ x!!)
For
simplicity, given sequence (I:), with x; E Rd and constant E Rd, shorthand
Zi.n = ( In), Zim + = (#) + c,..., + c), C22.n = (c, x2, xn) We denote the k-th coordinate of 24 E Rd as Zi,k- For function f xi_ Rd - R and a random variable X taking values in Rd., we designate f(- X) the random function: Zi.n - f + X).

---

Lindenberg path interpolation Let z and Z be the following interpolating processes between 2 and Y
z = (Yr,...,, Yn 2411....... Z
zade = n ( 8 Z
Z
Higher-order derivatives and bounds If a function f is three-times differentiable then we let:
(z1m) 0f 0, (f1m) = Ozci, f f(im))
, (m)m)) = Ōza,, Ozs,, Sanng (Ti:n)
Moreover, for potentially random function f we define the constants: Mî == [|XTxxll
DE, (f) = ME max inn
OC (Zu...8))
1|
L13
DE (f) = ME. Mc, max inn
L12
DE (f)
MR ME
Mr. max isn
max
J (Zmit.))
xc F
L12
where for any two vectors a, E Rd we denote with [a, b] their convex closure, i.c.
[a = (1- [],
Main Results
If the statistics (gn) were linear, i.. (n(11n)) - Eign Ti, then the influence of single observation
XT, on the estimate On, would depend uniquely the value of the random variable itself, i.c.
In(X") - (0Xx = -XT. This is not the case for non-linear statistics. For instance, if 9r. (Z max EiSn Fi,1, Eisn Fi.2) then the influence of observation 21 depends on the relative size of Ei> Ti,1 and Ei> T4,2- In this paper, we want study the asymptotics of the bootstrap method for such non-linear statistics, with complex influence functions. To control the degree of non-linearity, we assume that the statisties (g,.) can be approximated by three times differentiable functions.
Assumption (AAAppoxximablltt)) There exists sequence of functions (fn) with fn E C s.t.:
1. The functions (Sn) approximate the estimators (gn):
ll 9n (n")"", + f((("))) In (87))1))
0o O.
(Hl)
2. The first, second and third order derivatives respectively of size (1/3) onn 11/2). onn -1);
Rn,1 = 1/3
Σ (n)==) o
Rn,2 = vn Σ
(fx) = o(1)
Rn,3 =
Σ
D (fn) = o(1).
(H1)
To motivate Assumption 1, we present Appendix C two illustrating examples of simple estimators which fail to satisfy conditions (Il)), (II,) and for which the bootstrap method is not consistent.
Under Assumption we study the limiting distribution of the bootstrap statistic and establish that asymptotically the same as g., (Y (proof in Appendix L).

---

Theorem Let (9n R - R) be a sequence of measurable functions. Let (XT) be triangular urray of Li.d processes such that XT E L12. Under Assumption I, there a constant K independent of such that:
( - fn(( Fa(("))
1l, + =1)g(( - fn(
|d (sn(2"), 9n (F() xx)) )x))1,, S co
0.
{
n1/6 Ru,1 } Ro,3 + Ru,
+ K max
Remark We remark that the theorem also holds under slightly modified stability conditions. See Theorem in the appendix for more details. Moreover, the hypothesis that (X.) ) is an Li.d process can also be relaxed to assuming that the process (XT) is exchangeable. See Theorem 10 for more details in the appendix. Finally, note that Theorem I can also be extended to random estimators (ge), such as ones obtained by stochastic optimization methods (e.g SGD). See Theorem 10 for more details in the appendix.
Remark We note that Assumption 1 controls how stable the function 9n is to the change of one random variable for example X1. Many concentration inequalities, such as the Mediarmid inequality, impose conditions on similar quantities. To be able to derive a central limit theorem one would need to make additional assumptions regulating how much this change 9n. (X1, X Xn) In (0, X2.....
......... .x)) depends on the other random variables X2...., Xn [14]. The function hn, - max(O, inn Eisn X, - -B(())) is an example of statistic satisfying Assumption but is not asymptotically normal.
Remark 3 When the mean of the observations B(XT) is known, we propose in Appendix E an alternative bootstrap method that exploits this information, the centered centered-bootstrap method, and prove that it is consistent for 9n (Yr.).. This is useful for example for estimating p-values for hypothesis testing.
Theorem guarantees that we can use the bootstrap method to estimate the distribution of 9n. (Y"),
which implies that it can also be used build confidence intervals for gn (Ý X
Corollary Let (9n x IR R) be sequence of measurable symmettic functions. Let (X.t) be a triangular array of i.i.d processes such that X L12- Assume that (gn) and (X.) satisfy all the conditions of Theorem 1. Then there is a constant K independent of such that for en as defined Theorem 1:
| (In(Z") - Eggn((") X"), 9n (8T)) - E (pm) xz )x, 26n - 0.
The distribution we are interested that gn (Y") rather than gn (Y"). Moreover, the shape of the confidence intervals 9r, (Y") can be arbitrary compared to the ones of gn (Y"),ic i.. they are not systematically larger or smaller. This is illustrated in Appendix D by series of examples. Therefore in Appendix F we propose conditions that guarantee that the two distributions are asymptotically identical and prove in Appendix G that those conditions are tight.
When those conditions are not met we propose the use the bootstrap method to build adjusted confidence intervals that are guaranteed to have at least (but not necessarily equal to) some minimum asymptotic coverage. In Appendix H we propose an alternative method to do so by exploiting the bootstrap method for slightly shifted observations. Moreover, in Appendix I we assume that the mean (XTT belongs to a certain subset A,, and build robust confidence intervals with a guaranteed coverage level for all potential values of the mean.
According to Corollary the bootstrap method can be used build consistent confidence intervals for (g (Y*) X Therefore if we can bound the distance from E(g, (Y X*) to E(g, (Y")) we can use the bootstrap method to build confidence intervals on the latter. To do so we exploit the
fact that under mild conditions / X EEX{)] approximately normal. We assume that the function + E (Y + æ)] is a-Holder and that the moments of XT bounded. More formally,

---

suppose that there is sequence (Cn) and a constant b such that
E
In
y
I
9n )n(Y)M)
- On max [xll" kSdn
Ve E E
(Ha)
log(d,)) supksd, IXT =
min | 1l. 2 b, SSda
n1/6
Theorem 2 Lel (gn Xi
an - IR) be sequence of measurable functions satisfying Assumption 1 and (Hz)) Denote En the variance-covariance matrix of Xi and (N") to be a sequence of Gaussian vectors distributed N NO0, Let 3 > 0 be real; write 9/2 satisfying
Igin and 1/2 (X*) as quantities
P (lan(2*) El (2 (zm)
b,n 2//2(x (X
X
S 3/12
P max NPi 2 (1g12)) C, 3,122
Then the following holds:
lim sup lim sup P [E(gn(Y")) & [9n(
1812 -6, gn (2 b.n + 18/2 b.n
1//2
)+
g.n + +8]] S 3.
8.0
"
See Appendix N.6 for proof of Theorem We present in Appendix H an illustrative example.
4 P-yalue of a Two-Sample Kernel Test
In this subsection, we show how the bootstrap method can be used to obtain consistent p-values for kernel two sample tests. Given two independent i.i.d processes (X7,1 and (XI;2) taking value in Xx Rd,, the goal of two-sample tests is to determine if the two of observations (Xī11 and (XT)2) are sampled from the same distribution. For ease of notations, designate by 1r,11 and Un, respectively the distribution of the first sample (XI;;) i.d.d Nn,1 and the second sample
(XT2) itd
Pn.2; and we want test if the null hypothesis holds
(H.) Pn.1 Mn,
against the alternative
(Hf) 1r,1 # pn,
A popular method to do are non-parametric kernel two samples tests [42, 31, 51].
Let F, be class of functions from X, into IR. If the distributions are the same Pn.1 2n,2 then we have:
sup JEF,
E [Ef({T))) B(f
Moreover if F, is dense in the space of bounded continuous functions then the converse also holds. The main difficulty therefore consists of choosing the set F,, to be big enough to differentiate between the distributions 1n,11 and 2n, but structured enough that we can estimate of feF, [E(((X))))
((((()))) To do so, we choose reproducing kernel space H,, with kernel K₂ Xn, x X₂ + R and set the class of functions Fn to the unit ball of Hn. Different choices of kernels will lead various level of power for our test especially for structured high dimensional data. The goal is to choose the kernel that is the most likely to maximize the power of the test.
Let (Koa (x-))))) be a finite set of potential Kernel candidates. We write for all i S and for all l Pn.
H Ka, (XT1, XTi) (X X12) Ko. (X XT2) Ko, (XT, X(1);
and for all subsets B [n]] we denote Mo (X
में EiJEB H" The idea proposed in [42] is select the kernel that gives rise test with the highest (estimated) power. This is done by selecting

---

subset B, C [nt]] and maximizing the following quantity 0Ø- == (0 po (XB,) B,, where we have set
Ma
Po (
1011 Lieen
Eyc Hf, 10a Eisse,
H
+
where (An,) are tuning parameters. Once the kernel is chosen the test statistic computed on [[x]]] B, the remaining data: + EI Hig The fact that the kernel is chosen different sample than the test statistic is computed on, means that the conditional limiting distribution of the test statistic, under Ho, is known to be chi-square [42]. Hence one can compute a consistent estimate of the p-value. However under this approach only portion of the data is used select the kernel. This could be problematic when dealing with high-dimensional kernels.
We propose different method that does not require data splitting and uses the bootstrap method to estimate the p-value. The test statistic that we propose is a softmax:
Tx = Σ Hij we (Xmm),
ΣE>
where wx (X1.)) :=
CaPe, (x")
EN'pp Anrog (x")
SSPo
LjjSn
and where (Bn) are hyper-parameters. The bigger 3n is the more weight we give to the kernel maximizing po(X"))
We note that the distribution of Tn is unknown and depends in an intricate fashion on the set of kernels {K k S }u, as well as p.. Therefore to be able to compute the p-value we want to estimate its distribution under Hy.
For technical reasons it is convenient to apply the statistic to random vector whose coordinates are identically distributed. We achieve this by randomly permuting the two samples before passing it to the kernel test statistic (a transformation also conducted in the prior work of [42]). We remark that under the null hypothesis the distribution of X, and X are the same which implies that
the samples are interchangeable (Xi',, d (711) It is therefore natural to compare the distribution of (X()) to the corresponding randomly permuted process. This is the idea behind permutation tests [42]. In general, for an i.i.d random process (X;) taking value in I2 we define the process (X:M) obtained by randomly permuting the observations X₁, and X
X; with probability 0.5
XM = 1 (x1) with probability 0.5.
We note that this permuted process has identically distributed coordinates, i.e. XMM X As a side fact, we note that dw (X,, XM) S dw (X1,1 where dw is the Wassertein distance, but, more importantly, if the distribution of (X,) are already in Io then its distribution is left invariant by those permutations. We show that the bootstrap gives consistent and asymptotically tight bouers to the p- p-value even when Pr. grows exponentially fast (proof Appendix Q.1).
Let (X.) == (XXII;X be triangular array of i..dd processes. Let (Kol) k S Pn} be a sequence of positive definite continuous kernels. We suppose that
Proposition 2
00; On log(pn) o(n1/6);
max tr(Kon)) kSPn
X
where we shorthanded Dn == max (1) supksp.. Ko. (X D.M.MMMl...
1).
Let (Y,") be an inde-
pendent copy of (XT) and (Z be bootstrap samples of (X!). We have:
d (nIn (2M. nef., (Y-M)
x"
1 + 0.
Empirical Risk of Smooth Stacked Ensemble Estimator
ubiquitous and popular approach for model selection and ensembling in machine learning practice
known stacking [53, 8, 50]. Given a set of trained base estimators (0) ,..., 6Pm) for example

---

representing a fitted neural network, a random forest and nearest-neighhour estimator, we call the smooth-stacked estimator the linear ensemble of those estimators {0*) weighted by coefficients that are related to the out-of-sample risk of each estimator. An important question: if we use all the samples to estimate the weights of the ensemble, then can construct confidence intervals on the risk the ensemble estimator?
The most straightforward version of stacking is to put all the weight on the model with the smallest out-of-sample risk. Other approaches proposed practice are to fit a linear regression model using the outputs of each model as an input co- co-variate to the linear model and using the learned coefficients as coefficients on the ensemble [50].
In this subsection, we analyze smooth version of stacking, proposed and analyzed experimentally, for instance, [38], that adds stability to the chosen ensemble, while putting most weight on the best performing model. This ensemble can be viewed as a regularized instance of the linear regression stacking approach where an entropic regularizer is added the square loss objective. This regularization adds smoothness and stability to the chosen ensemble and allows us to show that the distribution of the ensemble's risk can be estimated with the bootstrap, even if the all the data are used to estimate the weights or fit the base models.
Let (XF') be triangular array of i.d observations taking value in RR"; and lct (m,) be an increasing sequence. Define Fn, as the space of measurable functions from = to Rd., We estimate Pn different estimators 2n == (XT.,.)) k < Pn built on the first my data-points. Each estimator 0% is training algorithm that takes as input my samples XTm, and returns model, which itself is a function from Rdd to R We denote with (XT.m,.) the returned model and with @h T.mm. (z.) E Rd the evaluation of the model a point Zu Rd,
The loss a model at a sample is measured by a common loss function Ln R2 R + R and the empirical risk of the k-th estimator is computed on all the remaining m., data points as:
n
R. (mim)
Ln (2 (x
=
n
Wh
u=
The smooth-stacked estimator is defined as the following ensemble learner
B. ((1m))
(z1m)() E ASPn
0% (1 (-).
Eksp. e BaRK (#1m))
The hyperparameter 8n controls how concentrated the stacked estimator is around the estimator(s) 0% with the lowest empirical error. We denote the empirical risk of an ensemble model E (Rda - Rd) as
(x1n)) = /n mn
1
Σ
Cn (xi, (x1, ((x:))
1=
Let (Z") a bootstrap sample of (2.))zm... We show that the bootstrap method systematically consistent if and only if 3, = o(vn m,). For simplicity we suppose that the estimators 0 return models that when evaluated at any point xa E Rd have bounded coordinates; and that the loss function Ln is smooth, and have bounded partial derivatives in its second argument.
We write the set of convex combinations of the estimators: Ω ((0p: P S pn)) =
Epsn. upOp wp 2 and Epsp. Wp = 1 } and introduce the following notations:
Tn = sup t
sup PSPn
sm.. ) (XT
V1,
1.
Tip : sup pSpn
( Ln (Xm 0g (XT. )xx))
sup esd
sup ( vSp.))
22,6 (X ((,,((X)))
V1,
Lee

---

where by (x, y) we designate oy (x. We show that if the following hypothesis (Hptacked) holds then the bootstrap method is asymptotically consistent (proof in Appendix R).
Bndd LnTT fn_ Ln
O.
(Isstccedd)
Vn mn
Proposition 3 Choose (m,), (Bx) and (pn) be increasing sequences. Let (X.) be a triangular array of i.idd observations taking value IR Set (Cn, Rd x Rd' - R) to be a sequence of smooth loss functions. Let (ZT) and (Yr") be respectively bootstrap sample and an independent
copy of (X mm+1: ) Suppose that the hypothesis (Hatacked) holds then we have:
|dz(RRg (Ro. +1m) -ED--ERR [ (Z (2m.+1:)m66.) Re, (1m.+ -D B (Y (1m) 1x=2)l1. -
where we have shorthanded Ôn :- (X²)) Therefore if we choose (X*)) be such that:
P (RRS.. (Zm.. +im) E (Ro, +1m) 1 2 la,a (X xx*) aa
then the following holds
limm P 11000
R (Y mu+lm)) E R2 (Y,,. mm) 2 (X*) @m)
If 8n grows proportionally to 3, x /n- m. then the bootstrap method is not a systematically consistent estimator of the risk of the smooth stacked estimator. We present simple example illustrating this in Appendix K and establish the asymptotic distribution of the bootstrap. However, we show that using Theorem we can still propose a corrected confidence interval with guaranteed asymptotic coverage.
References
[1] Miguel A Arcones and Evarist Giné. U-processes indexed by vapmik-éervonenkis classes of functions with applications asymptolics and bootstrap of u-statistics with estimated parameters. Stochastic Processes and their Applications, 52(1):17-38, 1994.
[2] Alexandre Belloni, Victor Chernozhukov, and Kengo Kato. Uniform post-selection inference for least absolute deviation regression and other z-estimation problems. Biometrika, 102(():77-94, 2015.
[3] Rudolf Beran and Gilles R Ducharme. Asympotic theory for bootstrap methods in statistics. 1991.
[4] Rudolf Beran, Muni S Srivastava, et al. Bootstrap tests and confidence regions for functions of covariance matrix. The Annals Statistics, 13(1):95-115, 1985.
[5] Peter J. Bickel and David A. Freedman. Some asymptotic theory for the bootstrap. Ann. Statist., 1966111911 11 1981.
[6] Peter J Bickel and David A Freedman. Bootstrapping regression models with many parameters. Festschrift for Erich L. Lehmann, pages 28-48, 1983.
[7] Patrick Billingsley. Convergence of probability measures. John Wiley Sons, 2013.
[8] Leo Breiman. Stacked regressions. Machine learning, 24(1):49-64 1996.
[9] Matias Cattanco, Michael Jansson, and Kenichi Nagasawa. Bootstrap- based inference for cube root asymptotics. Econometrica, 88(5):2203-2211 2020.
[10] Arindam Chatterjee and Soumendra Nath Lahiri. Bootstrapping lasso estimators. Journal of the American Statistical Association, 106(494)6608-623 2011.
[11] Snigdhansu Chatterjee, Arup Bose, et al. Generalized bootstrap for estimating equations. The Annals of Statistics, 33(1):414-436, 2005.
10

---

[12] Sourav Chatterjee. Superconcentration and related topics, volume 15. Springer, 2014.
[13] Sourav Chatterjee al. A generalization of the lindeberg principle. The Annals of Probability, 34(6))2061-2076, 2006.
[14] Sourav Chatterjee et al. A new method of normal approximation. The Annals of Probability, 36(4):1584-1600, 2008.
[15] Xi Chen, Wen-Xin Zhou, et al. Robust inference via multiplier bootstrap. Annals of Statistics, (8(3):1665-1611 2020.
[16] Xiaohui Chen et al. Gaussian and bootstrap approximations for high-dimensional u-statistics and their applications. The Annals of Statistics, 46(2):642-678, 2018.
[17] Yen-Chi Chen, Y. Samuel Wang, and Elena A. Erosheva. On the of bootstrap with variational inference:
Theory, interpretation, and a two-sample test example. The Annals of Applied Statistics, 12(2):846 876, 2018.
[18] Victor Cheroozuukov, Denis Chetverikov, Kengo Kato, et al. Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensione random vectors. The Annals of Statistics, (1(6))2786 2819, 2013.
[19] Victor Chernozhukov, Denis Chetverikov, Kengo Kato, al. Central limit theorems and bootstrap in high dimensions. The Annals of Probability, 45(4):2309-25 2352, 2017.
[20] Victor Chernozhukov, Denis Chetverikov, Kengo Kato, and Yuta Koike. Improved central limit theorem and bootstrap approximations high dimensions. arXiv preprint arXiv: 1912. 10529, 2019.
[21] Anthony Christopher Davison and David Victor Hinkley. Bootstrap methods and their applica- tion. Number 1. Cambridge university press, 1997.
[22] Hang Deng. Slightly conservative bootstrap for maxima of sums. 10X1v 2020.
arXiv preprint
[23] Hang Deng and Cun-Hui Zhang. Beyond gaussian approximation: Bootstrap for maxima of sums of independent random vectors. arXiv preprint 1705.92705,,9558, 2017.
[24] Ruben Dezeure, Peter Bühlmann, and Cun-Hui Zhang. High-dimensional simultaneous infer- ence with the bootstrap. Test, 26(4):685-719, 2017.
[25] Morris L Eaton and David E Tyler. On wiclandt's inequality and its application the asymptotic distribution of the cigenvalues of a random symmetric matrix. The Annals of Statistics, pages 260 271, 1991.
[26] B. Efron. Bootstrap Methods: Another Look the Jackknife. The Annals of Statistics, 7(1):1 26, 1979.
[27] Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics, pages 569-593. Springer, 1992.
[28] Noureddine El Karoui and Elizabeth Purdom. Can we trust the bootstrap high-dimensions? the case of linear models. The Journal of Machine Learning Research, 19(1):170-235, 2018.
[29] Noureddine El Karoui and Elizabeth Purdom. The non-parametric bootstrap and spectral analysis in moderate and high-dimension. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2115-2124, 2019.
[30] Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-problem. Advances in neural information processing systems, 19:513-520, 2006.
[31] Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723773,, 2012.
11

---

[32] Arthur Gretton, Kenji Fukumizu, Zaid Harchaoui, and Bharath K. Sriperumbudur. A fast, consistent kernel two-sample test. In Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
[33] Peter Hall. The bootstrup and Edgeworth expansion. Springer Science & Business Media, 2013.
[34] Fang Han, Sheng Xu, Wen-Xin Zhou, et al. On gaussian comparison inequality and its application to spectral analysis of large random matrices. Bernoulli, 24(3):187-1183,, 2018.
[35] Pawel Hitczenko. Best constants in martingale version of rosenthal's inequality. The Annals of Probability, pages 1656-1668, 1990.
[36] Roger Johnson. introduction to the bootstrap. Teaching statistics, 23(2):49-54, 2001.
[37] lain M Johnstone and Debashis Paul. Pea in high dimensions: An orientation. Proceedings of the IEEE, 106(8):1277-1222 2018.
[38] Cheng Ju, Aurélien Bibaut, and Mark van der Laan. The relative performance of ensemble methods with deep convolutional neural networks for image classification. Journal of Applied Statistics, 45((5):2800-2818 2018.
[39] Abbas Khosravi, Saeid Nahavandi, Doug Creighton, and Amir F. Atiya. Comprehensive review of neural network based prediction intervals and advances. IEEE Transactions on Neural Networks, (2(9):134111355) 2011.
[40] Ron Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Ijcai, volume 14, pages 1137-1145. Montreal, Canada, 1995.
[41] Yuta Koike. Notes on the dimension dependence in high-dimensional central limit theorems for hyperrectangles. Japanese Journal of Statistics and Data Science, pages 1-41, 2020.
[42] Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Dougal Sutherland. Learning deep kernels for non-parametric two-sample tests. arXiv preprint arXiv:2002.09) 09116,, 2020.
[43] Miles E Lopes. Central limit theorem and bootstrap approximation in high dimensions with near square root of rates. arXiv preprint arXiv:2009. 06004, 2020.
[44] Miles E Lopes, Benjamin Erichson, and Michael W Mahoney. Bootstrapping the operator norm in high dimensions: Error estimation for covariance matrices and sketching, arXiv preprint arXiv:1909.06129 2019.
[45] Enno Mammen. Asymptotics with increasing dimension for robust regression with applications the bootstrap. Annals of Statistics, 17(1):382-400, 1989.
[46] Enno Mammen. When does bootstrap work?: asymptotic results and simulations, volume 77. Springer Science & Business Media, 2012.
[47] Dimitris N Politis and Joseph P Romano. Large sample confidence regions based on subsamples under minimal assumptions. The Annals of Statistics, pages 2031-2050, 1994.
[48] Kesar Singh. On the asymptotic accuracy of efron's bootstrap. The Annals of Statistics, pages 1187-1195, 1981.
[49] Vladimir Spokoiny, Mayya Zhilova, et al. Bootstrap confidence sets under model misspecifica- tion. The Annals of Statistics, 4366):265322655 2015.
[50] van der Laan Mark, Polley Eric C, and Hubbard Alan E. Super learner. Statistical Applications in Genetics and Molecular Biology, 6(1):1-23,200)
[51] Li Wenliang, Dougal Sutherland, Heiko Strathmann, and Arthur Gretton. Learning deep kernels for exponential family densities. In International Conference on Machine Learning, pages 6737-6746, 2019.
12

---

[52] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378, 2016.
[53] David H. Wolpert. Stacked generalization. Neural Networks, 5(2):241 259, 1992.
[54] Chien-Fu Jeff Wu et al. Jackknife, bootstrap and other resampling methods in regression analysis. the Annals of Statistics, 14(4):1211-1295, 1986.
[55] Dixin Zhang. Bayesian bootstraps for u- u-processes, hypothesis tests and convergence of dirichlet u-processes. Statistica Sinica, pages 463-478, 2001.
13


Reviews:
[{"review_id": "ibNL-scP5ih", "reviewer": "Reviewer_iG7z", "summary": "The authors derive general stability conditions under which the empirical bootstrap estimator is consistent and quantify the speed of convergence. These results are illustrated for two-sample kernel tests after kernel selection and the empirical risk of stacked estimators. These two applications are relevant due to the recent interest in kernel tests with learnt deep kernels and the popularity of ensemble models in practice.", "questions": "", "limitations": "", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is highly technical and theoretical, and it still reads quite well. Although I am not very familiar with the statistical approaches and therefore not qualified to comment on its soundness, I am convinced that its originality and contribution is definitely very strong.\nAs mentioned above, the application of the general theorem to kernel tests with learnt kernels and model ensembles is very relevant to current machine learning challenges. While I am keen to see numerical results corroborating the theoretical ones, they are not a must for this paper, rather for a follow-up.\nFinally, I would have appreciate a concluding section highlighting the limitations of the current approach and a recommendation on where to focus future research.\n\nOther comments:\n- Please use a spell-checker to review the paper as there are few typos here and there.\n- Lines 82-84: p, p_n and n are used before being defined.\n- The two equations after lines 123 and 127 are identical: should they?\n- Page 4, please define C^{\\alpha, n} and C^{\\alpha, n}_\\epsilon\n- Page 8, Proposition 2: why is the result obtain using the L1-norm instead of just d_F \\to 0 in probability, as d_F is a real number anyway?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations have not been fully explained (or maybe I missed them).\nThis contribution does not have potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "This work does not reaise any ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "dOHBB4wFyCb", "reviewer": "Reviewer_XTTq", "summary": "The authors put forward a set of high-level conditions that are sufficient for bootstrap consistency of nonlinear statistics and for constructing bootstrap confidence intervals that are approximately asymptotically correct. They illustrate their framework with two examples: a) a two-sample test to test whether two distributions are identical, and b) a procedure to construct confidence intervals for the risk of smooth stacked ensemble estimators.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and clearly structured. Obviously, the authors have put a lot effort in this paper. However, I am always wary of papers that aim to develop a very general theory. Often, the level of generality of such papers leads to weak implications. Sometimes such papers also impose (hidden) assumptions, that limit the range of applicability of the supposedly general results.\n\n1) I believe that it is long clear that stability conditions such as Assumption 1 are part of the sufficient conditions for bootstrap consistency/ consistency of bootstrapped CIs. The difficulty lies in verifying these conditions for specific statistics and in deriving anti-concentration inequalities for specific statistics that are statistically meaningful. The examples provided in the supplementary materials (such as the square or p's power of the sample average) are not meaningful statistics. The two applications in the main text are essentially smoothed versions of a maximum/ supremum statistic and, therefore, it is easy to find equivalent maximum-statistics which could be analyzed via the approach developed Chetverikov, Chernozhukov and Kato in their sequence of papers from 2014-2020. What statistics other than statistics related to the maximum satisfy the stability properties? \n\n2) I credit the authors with pointing out that anti-concentration inequalities are not needed if one is willing to settle for a weaker notion of CIs.\n\n3) In the main text of the paper the authors write that their work relies on (or extends?) Chatterjee's (2006) generalized Lindeberg principle. I know that Chatterjee's results only apply to random vectors with i.i.d. or exchangeable entries. I therefore wonder how the authors modify Chatterjee's arguments to allow for arbitrary correlation between the entries of the random vectors. The proof of Theorem 1 appears to be incomplete as it references a Theorem 19 which is not in supplementary materials. Is the missing Theorem 19 is a version of Chatterjee's results for \"triangular arrays of i.i.d. processes\"? I could not locate an analogue of either Theorem 1.1 or 1.2 of Chatterjee (2006) in the supplementary materials.\n\n4) I've never seen the notion of bootstrap consistency that the authors use (lines 113 - 118). It is weaker than convergence in Wasserstein distance and it does not seem to imply consistency in Levy-Prokhorov distance. Does it imply consistency in Kolmogorov distance? Since the authors' metric does not seem to metrize weak convergence I do not think that the bootstrap consistency result in Proposition 2 is particularly useful. It doesn't seem to imply that we can compute p-values needed for hypothesis tests.\n\n5) The second main result of the paper (Theorem 2) holds only for nonlinear statistics which are also Lipschitz continuous with respect to the $\\ell_\\\\infty$-norm (Assumption $H_3$). One can easily infer from the statement of the theorem that the larger the Lipschitz constant, the larger the proposed CI. This is problematic since the Lipschitz constant might grow with the dimension. Here are some not fully rigorous calculations to illustrate this point: For simplicity assume that X_i's are i.i.d. vectors with i.i.d. entries. Take $g(X_1, \\\\ldots X_n) = \\\\|n^{-1}\\\\sum_i X_i\\\\|_2$. Then $g$ is Lipschtiz continuous w.r.t. to $\\\\ell_\\\\infty$-norm and has Lipschitz constant $d_n^{1/2}$. Hence, by Theorem 2, as the sample size $n$ (and the dimension $d_n$) grow the diameter of the CI for $\\\\mathbb{E}\\\\|n^{-1}\\\\sum_i X_i\\\\|_2$ grows roughly at a rate of $d_n^{1/2}$ (b/c $t_\\{g,n\\}$ has to grow to cancel out the Lipschitz constant otherwise the tail bound on the maximum of the Gaussian random variable $N^n$ doesn't hold). Intuitively, this CI is too wide since we know the variance of $\\\\|n^{-1/2}\\\\sum_i X_i\\\\|_2$ is roughly constant (b/c $n^{-1/2}\\\\sum_i X_i$ is approximately Gaussian and the variance of the Euclidean norm of a centered Gaussian vector with identity covariance is dimension free).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Somewhat. The authors provide several examples and counter-examples in the supplementary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "SrQlt92xFXv", "reviewer": "Reviewer_FFbw", "summary": "The paper discusses very general conditions on estimators under which the empirical bootstrap is asymptotically consistent. Explicit applications to two difficult inference problems, including two-sample tests using kernels with (approximate) kernel selection and on estimating the risk of ensemble learners.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The conditions discussed in the paper under which consistency can be achieved are impressively general, and it is clearly advantageous to consider general conditions rather than specific problems as is common. The paper is well written, and for the most part clear, although there are places in which the notation becomes slightly challenging (see later).\n\nI do not have any significant reservations about the paper, and my main comments are editorial suggestions:\n- I recommend being explicit in the abstract that the examples covered are slightly modified; especially in the case of the kernel two-sample test, where a softmax variant is substantially different from a discrete selection, which is what (at least I) expected.\n- I would recommend mentioning earlier on that the adjustment for the \"centering\" can actually be easily handled in practice. As I read through the first part of Section 3 I was bugged by the persistent presence of the centered processes, and it was only patently clear how this is handled when I got to line 214. This only bears mentioning earlier; it is not necessary that the details be given, but the ubiquity of the basic CLT as being the way around it will make less expert readers feel more at ease.\n- Am I missing something in the final statement in Theorem 2? Should it not be P(E[g_n(Y^n)] \\not \\in [interval]) \\leq \\beta?\n- The notation in Section 4 is challenging, since now the second subscript refers to the identity of the process and not the index of the component in the corresponding observation. While when the permutation test starts to be discussed, if I understand correctly, this becomes a little clearer since then the two processes (samples) are just stacked next to one another as one array of higher dimensional processes, but I think it is worth mentioning earlier on. Otherwise, the expression above line 224 is confusing if you interpret X_{1,1}^n as the first element of the first observation in X^n, and not as the entire first observation in (X^n_{i,1}).\n\nTo conclude, I have some minor comments/queries/typos, etc.\n- The word \"each\" in line 17 seems out of place\n- line 77: space between [14] and .\n- line 83: Recommend either \"A series of works has strengthened...\" or \"A series of work has strengthened...\"\n- lines 84 and 86: should not the dimension, p, explicitly depend on n as previously?\n- line 86: \"...conditions THAT log(...\"\n- line 94: \"...specific STATISTIC or...\"\n- Although not ambiguous, the metric d_{\\mathcal{F}} is defined between probability measures, but is evaluated on random variables.\n- line 124: \"same mean AS X^{n}_1\n- Aren't lines 126-127 just repeating what lies above in line 123?\n- In Proposition 1, recommend \"Let \\epsilon > 0 be constant, then...\"\n- line 164: missing subscript n in g(0X^n_{2:n})\n- Although not ambiguous, I don't think the L_{p} spaces of RVs are not explicitly defined; only the norm is.\n- this is purely cosmetic, but in the statement of assumption H_3, the parentheses in g(Y^n + x/sqrt{n}) would look better if larger\n- line 224: should not \"opposite\" better be replaced with \"converse\"?\n- What does (a) above equality mean? Should this be d?\n-  What is the metric d_W?\n- The font in Proposition 2 is inconsistent\n- Why is sometimes max()/min() and other times join/meet used?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not explicitly discussed. There is no apparent potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "5ucwjTzJfDd", "reviewer": "Reviewer_iK1w", "summary": "---------------------\n---------------------\n**Update after discussion with authors and other reviewers**\n\nThank you to the authors and other reviewers for engaging so much in the discussion period. I think the discussion has made clear that the  authors could significantly strengthen the paper by drawing connections between their proofs and others in the literature. I also think there should also be a discussion on the limitations of the example in Section 4, as the authors are not analyzing a standard kernel test (although, as they note, they are analyzing a statistic that is highly related to a common kernel test). I think this makes the results of Section 4 a little weaker. This, combined with XTTq's points about the relation to the existing literature, have led me to decrease my score somewhat. I'm still in favor of an accept because while the paper could be improved by restructuring, I think it still presents practically useful theory as-is. If the paper is accepted, I think the authors should at the very least include some discussion surrounding their discussions with XTTq.\n\n---------------------\n---------------------\n\n\nThis is a theoretical paper which aims to extend the reach of consistency results for the classical bootstrap. The authors prove some general theory about the bootstrap and then show that this theory can be specialized to prove consistency of the bootstrap for two sample kernel hypothesis tests and stacked ensemble estimators. To my knowledge, these results are novel in that previous bootstrap theory does not apply to them, and these results seem useful to the machine learning community. Besides solid results, I think this paper has a few nice strengths:\n\n1.\tThe paper is well written\n2.\tThe literature review is, to my knowledge, very thorough (which is not easy to do for a topic with as much background as the bootstrap).\n3.\tThe authors are careful to call out the flaws of their theory throughout the paper.\n4.\tThe authors try to explore the necessity of their assumptions with the use of counterexamples\n\nA downside is that all of this takes a *lot* of room – the appendix is 45 pages of additional results, examples, counterexamples, and proofs. I’ve tried to go through this as carefully as possible, but I have not been able to carefully check all the proofs. Overall, I think this is a solid paper, and my criticisms listed below are not major.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have three larger comments that I think could improve the paper, listed as (A), (B) (C) below:\n\n(A) I think it would be good to include some more discussion about what kinds of models in machine learning the paper does / does not extend the reach of bootstrap consistency proofs to. The authors give some generic theory and then show it applies to two specific examples (kernel hypothesis testing and stacked ensembled estimators). Is there some underlying theme between the proofs for these two examples? Can the authors conjecture about what kinds of other examples their proof techniques might work well for? Conversely, it seems that the authors do not cover *all* problems of interest in machine learning. My understanding is that for simple models like Poisson regression with covariates growing unbounded with n (that is, a GLM with unbounded link function with unbounded derivatives), existing common proof techniques for the consistency of the bootstrap (like linearization) are not known to apply. And it seems like the authors’ Assumption 1 would be violated in this case as well, so their theory will not apply to this case either. I don’t mean this as a criticism -- the authors definitely don’t try to sell their work as applying to everything. But it would be good to call out what’s still missing for the sake of future research.\n\n(B) The authors put a lot of discussion up front about their use of particular proof techniques backing their new results (“Chaterjee’s generalized Lindenberg approach and a smart path interpolation technique”). This sounds interesting, and I was hoping to learn more in the main text! I wonder if the authors could summarize how these techniques fuel their results in the main text. \n\n(C) I wonder if section 5, which is about bootstrapping ensembled estimators, could be reworked to include more discussion and fewer mathematical details, e.g. by making Prop 3 an informal statement and pointing to the appendix for a formal statement.  I say this because I think there are a few points in Section 5 that could use expanding on:\n1. The authors analyze the bootstrapping properties of what seems like a non-standard ensemble estimator. In particular, they consider the ensemble of estimators indexed by 1,…,p, where the ith estimator is fit on datapoints x_1, …, x_i. This seems like a very particular estimator. To be fair, the authors do note “[this specific ensemble] allows us to show that the distribution of the ensemble’s risk can be estimated with the bootstrap”. But it would be great to understand what properties of this estimator are required for the authors’ analysis vs required for the bootstrap to converge (i.e. did the authors choose a somewhat non-standard ensemble because the bootstrap doesn’t converge otherwise or because their analysis won’t go through otherwise?). \n2. What is \\beta_n doing here? Is it a hyperparameter again? It seems like \\beta_n is required to go to infinity, which in turn seems like it causes the ensembled estimator \\hat\\Theta_n to collapse to the best-performing \\hat\\theta^k_n. Is this correct? If so, why is this collapse a desirable thing?\n3. Are the base estimators (the \\hat\\theta^k_n) not required to satisfy Assumption 1? This seemed like a pretty important part of previous theory, so how is it that we are stacking Assumption 1-violating estimators to get a bootstrap-consistent estimator?\n4. Relatedly, it would be nice to have a conclusion at the very end summarizing the main results of the paper.\n\n-------- Scattered smaller comments:\nLine 263: what is d_n’? \n\nLine 156: Notation for Lindenberg path interpolation is defined, but I don’t think this was used in the main text.\n\nLine 167: Influence functions are mentioned here but not defined.\n\nThe title of Appendix N is “Further proofs from section 3”, but not all of the results discussed here seem to be from Section 3 (e.g. there is no Example 2 in Section 3) \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss societal impact in this paper, but, given its subject, I think that is appropriate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There has been a tremendous amount of discussion about this paper. Several points of criticism have been raised by Reviewer XTTq, mostly concerning the used stability condition, consistency in high dimensions and the generality of the specific kernel used and the examples in sections 4/5. On the other hand, all other reviewers had a much more positive impression of this paper, which is reflected in their high scores that they were willing to defend in the discussion. In my opinion the author rebuttal was very convincing and successfully addressed most major points of criticism in a detailed, transparent and meaningful way. So I recommend acceptance of this paper.   
Decision: Accept (Poster)