Asymptotics of the Bootstrap via Stability with Applications to Inference with Model Selection
Authors: Morgane Austern
Vasilis Syrgkanis
Venue: NeurIPS 2021 Poster (2021)
Keywords: bootstrap
asymptotics
confidence intervals
model selection

Abstract: One of the most commonly used methods for forming confidence intervals is the empirical bootstrap, which is especially expedient when the limiting distribution of the estimator is unknown. However, despite its ubiquitous role in machine learning, its theoretical properties are still not well understood. Recent developments in probability have provided new tools to study the bootstrap method. However, they have been applied only to specific applications and contexts, and it is unclear whether these techniques are applicable to the understanding of the consistency of the bootstrap in machine learning pipelines. In this paper, we derive general stability conditions under which the empirical bootstrap estimator is consistent and quantify the speed of convergence. Moreover, we propose alternative ways to use the bootstrap method to build confidence intervals with coverage guarantees. Finally, we illustrate the generality and tightness of our results by examples of interest for machine learning including for two-sample kernel tests after kernel selection and the empirical risk of stacked estimators.

Asymptoties of the Bootstrap via Stability with Applications to Inference with Model Selection
Morgane Austern
Vasilis Syrgkanis Microsoft Research vasy@microsoot..cmm
Microsoft Research morgane, aesteeeeretttteemmmm.m.... com
Abstract
One of the most commonly used methods for forming confidence intervals is the empirical bootstrap, which is especially expedient when the limiting distribution of the estimator unknown. However, despite ubiquitous role in machine learning, its theoretical properties are still not well understood. Recent developments in probability have provided new tools to study the bootstrap method. However, they have been applied only to specific applications and contexts, and it is unclear whether these techniques are applicable to the understanding of the consistency of the bootstrap in machine learning pipelines. In this paper, derive general stability conditions under which the empirical bootstrap cstimator is consistent and quantify the speed of convergence. Moreover, we propose alternative ways to use the bootstrap method to build confidence intervals with coverage guarantees. Finally, we illustrate the generality and tightness of our results by examples of interest for machine learning including for two-sample kernel tests after kernel selection and the empirical risk of stacked estimators.
Introduction
Bootstrap resampling [26], has been one the most popular techniques measuring the uncertainty of statistic, primarily due to its simple algorithmic definition and its conveniency with dealing with opaque statistical procedures that produce a test statistic. For this reason, uncertainty quantification based bootstrap resampling has been a staple in the machine learning community, starting from the early days of the field [40] and continuing into the deep learning and SVM era [39, 30, 32, 17]. Despite it's widespread use, general conditions for the consistency of the bootstrap for complex non-linear statistics generally not fully explored and hence, it is not clear when the bootstrap method will accurately capture uncertainty in machine learning pipelines, especially when model selection procedures are involved.
Our goal is to understand the distribution a large class non-lincar statistic ò, := (X X.,). the samples X, drawn from their unknown distribution. Examples of such statistics could be the out-of-sample risk machine learning predictor, or the maximum- maxximum-mean-diiccoe mcan-discrepancy of a two-sample kernel test, or the prediction of machine learning model at given point. One approach to approximating this distribution is the empirical bootstrap: sample new observations Zı,... Zn independently and uniformly from {X1,...,XX) X, (with replacement), and define @buot as the value of the estimator taken at the bootstrap sample Oboot : 9n (Z,,..., Zz). This procedure can be repeated many times in order estimate the conditional distribution of @boot If this distribution approximately the same as the distribution of 6.. as the sample size grows, we say that the bootstrap method is consistent. Consistency of the bootstrap can be subsequently utilized to construct confidence intervals for the quantity of interest.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

Classical consistency proofs for the bootstrap, either require that the limiting distribution of the statistic be Gaussian, or even more stringently that the statistic is asymptotically linear. Notably, when 0, is asymptotically normal those intervals are known to be consistent under general conditions; see c.g. [33,5, 5, 18]. Moreover, classical conditions for such statements to hold are that the statistic is Hadamard differentiable and smooth with respect to the underlying distribution of the random variables. However, these properties can be violated by test statistics that are implicitly defined via machine learning data analysis pipelines and especially when the model selection is involved. Thus it crucial to provide more generally applicable sufficient conditions for the validity of the bootstrap.
Notably, recent breakthrough results in statstics by Chatterjee [13], have shown that the Lindenberg technique that has been long used to establish Central Limit Theorems, is more widely applicable and can be leveraged to show limit distributional consistency of two random variables more broadly. Notably, this intuition has already been exploited to show that the bootstrap method is consistent in particular applications in the econometric literature, such as the construction of uniform confidence bands. However, these proofs are tailored to the particular application of interest and do not provide general characterizations.
Our main contribution is to provide set of sufficient stability conditions that imply the consistency of the bootstrap and which go well beyond the existing general conditions via the use of Chatterjee's s generalized Lindenberg approach and a smart path interpolation technique. We then apply these gen- cral results to derive inference in two machine lcarning applications: i) bootstrapping the test statistic in two-sample kernel tests, after model selection on the kernel and sample re-use, ii) bootstrapping the risk of a stacked estimator with sample re-use.
Roughly, our sufficient conditions impose unilateral stability properties on the statistic, reminiscent of the stability conditions required for classical concentration inequalities, such as McDiarmid's inequal- ity. In particular, we assume that the functions (gn) are approximable by three-times differentiable functions whose first, second and third order partial derivatives, with respect to a single observation, taken at (X,,...; X.)) are of respective order o(n - 1/3), o(n - 1/2) and o(n -1). These conditions assure that the value of 9n (X.,... X₂, ) is not oversensitive to the value of a single observation (see Section for a formal exposition). Exploiting these assumptions, we exactly characterize the limiting distribution of the bootstrap estimator @hoot compare it to the distribution of the original statistic 6,, and study how fast the distribution of @boot converges. Notably, we also discover that when the mean of the obscrvations X1 is unknown, then the bootstrap method is in general not consistent and we propose corrections to bootstrap based intervals with guaranteed minimum coverage.
Related litterature The empirical bootstrap method was first introduced in a breakthrough paper by Efron [27]. Other bootstraps methods have since been proposed including the multiplier bootstrap [54], the residual bootstrap [21] or the non-remplacement bootstrap method [47]. A vast literature studies the theoretical properties of those techniques with some of the main results synthesized in the following books [33, 21, 36, 3]. Most relevant to us are studies of the asymptotics of the bootstrap method. The consistency of the bootstrap method for linear statistics, t-statisties, Von- Mises functionals and quantiles has been established in [5, 46, 48] and for U-statistics in [1, 55J. Those results, among others, have been extended to high-dimensional regression and M-estimation [6, 45, 10, 2, 24], misspecified models [49] solutions of estimating equations [11] and to robust estimators [15]. In contrast, other works established the poor performance of the bootstrap method for non smooth statistics [25, 4, 5], or for non-sparse high-dimensional regressions [28].
Several recent breakthrough papers studied the consistency of the bootstrap method, both empirical and wild, for the maximum of high-dimensional centered averages with the dimension taken to be growing exponentially fast with the sample size. Notably [18, 19] established the consistency of the bootstrap and Gaussian approximation method when respectively log(p,.) 77/8 //3 and log(p.n)7/6 = / hold. A series of works have strengthen those results: [20, 23, 41] established the consistency of the multiplier and empirical bootstrap when log(pr))$/4 55/4 = o(n-/4),
[43] established a quasi vn" rate for the wild bootstrap. [22] built slightly conservative confidence sets with guaranteed coverage under the conditions that log(p) o(n) and [16] proved that similar results hold for high-dimensional U-statistics. Those works use a combination of the Stein method, Edgeworth expansions, Lindeberg's 's method [13] and the Slepian smart interpolation path. We note that the limiting distributions of those statistics are in general not Gaussian [23]. In contrast, our results apply more broadly and are not limited to the study of maximums of centered empirical
2

---

averages. We notably apply our results to machine learning estimators that are smoothed arg-minima of an objective function. Other works have studied the accuracy of the bootstrap method for specific statistics whose distributions are known to be asympiotically not Gaussian such as: the operator norm in high dimensions [44, 34, 37J, sampled eigenvalues of random matrices in high and moderate dimensions [29] M-estimators having cube root convergence [9]. The main contrast between this series of work and ours is that, rather than studying the bootstrap method for one specific statistic or application, seck establish the asymptoties of the bootstrap method under universal conditions on the estimators (gn). Our proof builds on breakthrough method proposed by Chatterjee [13] that generalized the Lindeberg method general technique for comparing the expectations of f(X1.s and (Y) ) of a large class of functions f.
2 Problem Statement
Let (Xf)) be triangular array of independent and identically distributed (i.i.d) processes with observations X²" taking value in Rd,, Morcover, let X (X1', .x........ X,)) .x.,),,,,;;) denote its n-th row. Consider an estimator 0,, := 9n (X"), where 9n, x R", - is mcasurable function, that we will typically refer to as statistic, and let (gn) denote the sequence of measurable functions as grows. To evaluate the performance of this estimator and build confidence intervals, we need approximate its distribution. In this work, we will analyze the empirical bootstrap method.
Empirical hootstrap Bootstrap samples z ........ Z are sampled with replacement from the observations {XT.... X=). This implies that conditionally on X the coordinates of 2n are distributed i..d, with Z," X" unif ((.f....,..) 1)) for all E [n]]
Consistency metric and bootstrap consistency Throughout the paper we denote with Y (YT', (YT"..... ...., Y
an independent copy of X The bootstrap method is said be consistent for (gx) if conditionally on X the distribution of 9n(2") well-approximates the distribution of 9n, (Y"), as n + co. To make this statement rigorous we introduce metric on the space of probability distributions. First, we define the class of three times continuously differentiable measurable functions with bounded third-order derivatives:
{ h c (RR)) sup SER
F
10 (x)) 1, visis33;
Given this, we define the distance on the space of probability measures, as the maximum mean discrepancy, where test functions range over the class F:
d)( := sup Ex~nYY> [h(X) - h(Y)). hEF
We remark that dy is metric the space of probability measures of real-valued random variables. Notably for two probability distributions fa and y d v) = O then those distributions are the same V μ.. Moreover, the topology defined by dy is finer than the weak convergence topology. Indeed, for a sequence of distributions (v,)) if we have (2,,n, (4) + 0 then (/²)) converges weakly
to μc V/. 4 p. Finally we remark that this metric is related to the classical on probability spaces [7]. Morcover, wc use the shorthand notation:
distance
dy (1,, E) sup Ex~p,Y [h(X) h(Y) E].
heF
We say that the empirical bootstrap method consistent for (gn) if:
dy (zn) x") & 0.
Centering discrepancy and centered bootstrap consistency Notably, an individual bootstrap sample Zf X has slightly different mean E [ X X == Σign X", than the one of
XT. As we will see this small difference plays a crucial role determining the consistency of the bootstrap and for this reason it will be useful to define artificially centered versions of the random variables (ZT) and (Y."). ). centered bootstrap sample
2; = -T
(X" 1E [XT])

---

is a bootstrap sample that has been re-centered to artificially have the same mean as X7. Moreover, denote with Y corrected version of Y;", artificially re-centered to have the same mean as ZT, ie:::
Y : Y?" + X E [XT]
We say that the centered bootstrap is consistent for (gn) if:
dy (gn(Z"), (Y"") (gn(Z"))9n("))) x 2 0.
From metric consistency to confidence intervals with nominal coverage We can compare the confidence intervals of random variables X and in terms of their mutual distance dy(X,, Y) (proof in Appendix M.1).
Proposition 1 Let X and Y be two real-valued random variables and & any random event. Let 6 > 0 be a constant then for any Borel set ((B)) the following holds:
P(X E Aoc 8) 2 P(Y E A 8 - dx
e²
where we wrote A. = (x E R By As.t x yl S e). Moreover, if (a, b) is a confidence interval at level for Y E |Y| E), conditional on E, then we have:
2d 2dz
P(X E X E E (a- 6c, be 6c] 18)22
For instance, suppose that we care about estimating 0₂ = E 19n [Y").. Then the bootstrap method, if consistent, can be used build consistent confidence intervals for 0n. Indeed since we can estimate the conditional distribution of @bootstrap == 9n (Z"), by drawing sufficiently many bootstrap
sub-samples, we can find Cha such that
P @hoolstrap - E @huolstrap
X E Ca. xx -1-
Then, if we write 0, 9n (Y"), using the consistency of the bootstrap method obtain that:
lim inf lim inf P (0,, - On E C.a.)) 21-68 a.
t.0 n-
Therefore, confidence intervals built using the bootstrap method achieve asymptotically nominal level of confidence. We note that prior works (e.g. [18, 19]). typically provide slightly stronger a, by proving anti-concentration results on the limit distribution of 0, - 8n. Such anti-concentration, allows one to argue that the mass the random variable 0,- - On contained in Ca converges to the mass contained in Ca," as + 0 and thereby, lim inf lim inf,- De P(ê. On E C.a.)) lim inf, P(ê, 0n C C) Given that these results typically require stronger conditions on the statistic and many times Gaussian limits, we omit this step in this work and note that slightly weaker, albeit still practically useful, statement on coverage is achievable a more general setup.
statement that lim i P(O, An Caaa)
From metric consistency to p-values Alternatively, suppose that we want to test if a specific null hypothesis (Hğ) holds against the alternative (HT). To do so we compute test statistic (X1... and determine rejection region R². A crucial quantity to estimate is the p-value P (XTT..).)) E R RD Ho)) The bootstrap method, if consistent, allows us upper-bound the p-value by enlarging the rejection region with an infinitesimally small quantity. Indeed according to Proposition if the bootstrap method is consistent then we have
lim lim inf P (Ta (Ta(Z E R"IX")) 2 lim P (TT(XPP.)) E RR).
<Lo n-000
x2-000
2.1 Notations and definitions
For scalar random variable X we denote with |X | the Lp-norm: |X|| := 1(P More-
x!!) 1/8
over, for vector E Idd, we denote with || lp- the ep vector norm: ||=||p
For
simplicity, given sequence (I:), with x; E Rd and constant E Rd, shorthand
Zi.n = ( In), Zim + = (#) + c,..., + c), C22. = (c, x2, xn) We denote the k-th coordinate of 24 E Rd as Zi,k- For function f xi_ Rd - R and a random variable X taking values in Rd., we designate f(- X) the random function: Zi.n - f + X).

---

Lindenberg path interpolation Let z and Z be the following interpolating processes between 2 and Y
z = (Yr,...,, Yn 2411....... Z
zade = n ( 8 Z
Z
Higher-order derivatives and bounds If a function f is three-times differentiable then we let:
(z1m) 0f 0, (f1m) = Ozci, f f(im))
, (m)m)) = Ōza,, Ozs,, Sanng (T1:n)
Moreover, for potentially random function f we define the constants: Mî == [|XTxxll
DE, (f) = ME max inn
OC (Zu...8))
1|
L13
DE (f) = ME. Mc, max inn
L12
DE (f)
MR ME
Mr. max isn
max
J (Zmit.))
xc Fe[
L12
where for any two vectors a, E Rd we denote with [a, b] their convex closure, i.c.
[a = (1- [],
Main Results
If the statistics (gn) were linear, i.. (n(11n)) - Eign Ti, then the influence of single observation
XT, on the estimate On, would depend uniquely the value of the random variable itself, i.c.
In(X") - (0Xx = -XT. This is not the case for non-linear statistics. For instance, if 9r. (Z max EiSn Fi,1, Eisn Fi.2) then the influence of observation 21 depends on the relative size of Ei> Ti,1 and Ei> T4,2- In this paper, we want study the asymptotics of the bootstrap method for such non-linear statistics, with complex influence functions. To control the degree of non-linearity, we assume that the statisties (g,.) can be approximated by three times differentiable functions.
Assumption (AAAppoxximablltt)) There exists sequence of functions (fn) with fn E C s.t.:
1. The functions (Sn) approximate the estimators (gn):
ll 9n (n")"", + [ In (87))1))
0o O.
(Hl)
2. The first, second and third order derivatives respectively of size (1/3n onn 11/2). onn -1);
Rn,1 = 1/3
Σ (n)==) o
Rn,2 = vn Σ
(fx) = o(1)
Rn,3 =
Σ
D (fn) = o(1).
(H1)
To motivate Assumption 1, we present Appendix C two illustrating examples of simple estimators which fail to satisfy conditions (Il)), (II,) and for which the bootstrap method is not consistent.
Under Assumption we study the limiting distribution of the bootstrap statistic and establish that asymptotically the same as g., (Y (proof in Appendix L).

---

Theorem Let (9n R - R) be a sequence of measurable functions. Let (XT) be triangular urray of Li.d processes such that XT E L12. Under Assumption I, there a constant K independent of such that:
( - fn(( Fa(("))
1l, + =1)g(( - fn(
|d (sn(2"), 9n (f() xx*) )x))1,, S co
0.
{
n1/6 Ra,1 } Ro,3 + Ru,
+ K max
Remark We remark that the theorem also holds under slightly modified stability conditions. See Theorem in the appendix for more details. Moreover, the hypothesis that (X.) ) is an Li.d process can also be relaxed to assuming that the process (XT) is exchangeable. See Theorem 10 for more details in the appendix. Finally, note that Theorem I can also be extended to random estimators (ge), such as ones obtained by stochastic optimization methods (e.g SGD). See Theorem 10 for more details in the appendix.
Remark We note that Assumption 1 controls how stable the function 9n is to the change of one random variable for example X1. Many concentration inequalities, such as the Mediarmid inequality, impose conditions on similar quantities. To be able to derive a central limit theorem one would need to make additional assumptions regulating how much this change 9n. (X1, X Xn) In (0, X2.....
......... .x)) depends on the other random variables X2...., Xn [14]. The function hn, - max(O, inn Eisn X, - -B(())) is an example of statistic satisfying Assumption but is not asymptotically normal.
Remark 3 When the mean of the observations B(XT) is known, we propose in Appendix E an alternative bootstrap method that exploits this information, the centered centered-bootstrap method, and prove that it is consistent for 9n (Yr.).. This is useful for example for estimating p-values for hypothesis testing.
Theorem guarantees that we can use the bootstrap method to estimate the distribution of 9n. (Y"),
which implies that it can also be used build confidence intervals for gn (Ý X
Corollary Let (9n x IR R) be sequence of measurable symmettic functions. Let (X.t) be a triangular array of i.i.d processes such that X L12- Assume that (gn) and (X.) satisfy all the conditions of Theorem 1. Then there is constant K independent of such that for en as defined Theorem 1:
| (In(Z") - Eggn((") X"), 9n (8T)) - E (pm) xz )x, 26n - 0.
The distribution we are interested that gn (Y") rather than gn (Y"). Moreover, the shape of the confidence intervals 9r, (Y") can be arbitrary compared to the ones of gn (Y"),ic i.. they are not systematically larger or smaller. This is illustrated in Appendix D by series of examples. Therefore in Appendix F we propose conditions that guarantee that the two distributions are asymptotically identical and prove in Appendix G that those conditions are tight.
When those conditions are not met we propose the use the bootstrap method to build adjusted confidence intervals that are guaranteed to have at least (but not necessarily equal to) some minimum asymptotic coverage. In Appendix H we propose an alternative method to do so by exploiting the bootstrap method for slightly shifted observations. Moreover, in Appendix I we assume that the mean (XTT belongs to a certain subset A,, and build robust confidence intervals with a guaranteed coverage level for all potential values of the mean.
According to Corollary the bootstrap method can be used build consistent confidence intervals for (g (Y*) X Therefore if we can bound the distance from E(g, (Y X*) to E(g, (Y")) we can use the bootstrap method to build confidence intervals on the latter. To do so we exploit the
fact that under mild conditions / X EEX{)] approximately normal. We assume that the function + E (Y + æ)] is a-Holder and that the moments of XT bounded. More formally,

---

suppose that there is sequence (Cn) and a constant b such that
E
In
y
I
9n )n(Y)M)
- On max [xll" kSdn
Ve E E
(Ha)
log(d,)) supksd, IXT =
min | 1l. 2 b, SSda
n1/6
Theorem 2 Lel (gn Xi an - IR) be sequence of measurable functions satisfying Assumption 1 and (Hz)) Denote En the variance-covariance matrix of Xi and (N") to be a sequence of Gaussian vectors distributed N NO0, Let 3 > 0 be real; write 9/2 satisfying
Igin and 1/2 (X*) as quantities
P (lan(2*) El (2 (z))
b,n 2//2(x (X
X
S 3/12
P max NPi 2 (1g11)) C, 3,122
Then the following holds:
lim sup lim sup P [E(gn(Y")) & [9n(
1812 -6, gn (2 b.n + 18/2 b.n
1//2
)+
g.n + +8]] S 3.
8.0
"
See Appendix N.6 for proof of Theorem We present in Appendix H an illustrative example.
4 P-yalue of a Two-Sample Kernel Test
In this subsection, we show how the bootstrap method can be used to obtain consistent p-values for kernel two sample tests. Given two independent i.i.d processes (X7,1 and (XI;2) taking value in Xx Rd,, the goal of two-sample tests is to determine if the two of observations (Xī11 and (XT)2) are sampled from the same distribution. For ease of notations, designate by 1r,11 and Un, respectively the distribution of the first sample (XI;;) i.d.d Nn,1 and the second sample
(XT2) itd
Pn.2; and we want test if the null hypothesis holds
(H.) Pn.1 Mn,
against the alternative
(Hf) 1r,1 # pn,
A popular method to do are non-parametric kernel two samples tests [42, 31, 51].
Let F, be class of functions from X, into IR. If the distributions are the same Pn.1 2n,2 then we have:
sup JEF
E [Ef({T))) B(f
Moreover if F, is dense in the space of bounded continuous functions then the converse also holds. The main difficulty therefore consists of choosing the set F,, to be big enough to differentiate between the distributions 1n,11 and 2n, but structured enough that we can estimate of feF, [E(((X))))
((((()))) To do so, we choose reproducing kernel space H,, with kernel K₂ Xn, x X₂ + R and set the class of functions Fn to the unit ball of Hn. Different choices of kernels will lead various level of power for our test especially for structured high dimensional data. The goal is to choose the kernel that is the most likely to maximize the power of the test.
Let (Koa (x-))))) be a finite set of potential Kernel candidates. We write for all i S and for all l Pn.
H Ka, (X71, XTi) (X X Ko. (X XT2) Ko, (XT, X(1);
and for all subsets B [n]] we denote Mo (X
में EiJEB H" The idea proposed in [42] is select the kernel that gives rise test with the highest (estimated) power. This is done by selecting

---

subset B, C [nt]] and maximizing the following quantity 00- == where we have set
po (XB,) B,,
Ma
Po (
1011 Lieen
Eyc Hf, 10a Eisse,
H
+
where (An,) are tuning parameters. Once the kernel is chosen the test statistic computed on [[x]]] B, the remaining data: + EI Hig The fact that the kernel is chosen different sample than the test statistic is computed on, means that the conditional limiting distribution of the test statistic, under Ho, is known to be chi-square [42]. Hence one can compute a consistent estimate of the p-value. However under this approach only portion of the data is used select the kernel. This could be problematic when dealing with high-dimensional kernels.
We propose different method that does not require data splitting and uses the bootstrap method to estimate the p-value. The test statistic that we propose is a softmax:
Tx = Σ Hij we (Xim),
ΣE>
where wx (X1.)) :=
CaPe, (x")
EN'pp Anrog (x")
SSPo
LjjSn
and where (Bn) are hyper-parameters. The bigger 3n is the more weight we give to the kernel maximizing po(X"))
We note that the distribution of Tn is unknown and depends in an intricate fashion on the set of kernels {K k S }u, as well as p.. Therefore to be able to compute the p-value we want to estimate its distribution under Hy.
For technical reasons it is convenient to apply the statistic to random vector whose coordinates are identically distributed. We achieve this by randomly permuting the two samples before passing it to the kernel test statistic (a transformation also conducted in the prior work of [42]). We remark that under the null hypothesis the distribution of X, and X are the same which implies that
the samples are interchangeable (X;2) d (711) It is therefore natural to compare the distribution of (X()) to the corresponding randomly permuted process. This is the idea behind permutation tests [42]. In general, for an i.i.d random process (X;) taking value in I2 we define the process (X:M) obtained by randomly permuting the observations X₁, and X
X; with probability 0.5
XM = 1 (x1) with probability 0.5.
We note that this permuted process has identically distributed coordinates, i.e. XMM X As a side fact, we note that dw (X,, XM) S dw (X1,1 where dw is the Wassertein distance, but, more importantly, if the distribution of (X,) are already in Io then its distribution is left invariant by those permutations. We show that the bootstrap gives consistent and asymptotically tight bouers to the p- p-value even when Pr. grows exponentially fast (proof Appendix Q.1).
Let (X.) == (XXII;X be triangular array of i..dd processes. Let (Kol) k S Pn} be a sequence of positive definite continuous kernels. We suppose that
Proposition 2
00; On log(pn)) o(n1/6);
max tr(Kon)) kSPn
X
where we shorthanded Dn == max (1) supksp.. Ko. (X D.M.MMMl...
1).
Let (Y,") be an inde-
pendent copy of (XT) and (Z be bootstrap samples of (X!). We have:
d (nIn (2M. nef., (Y-M)
x"
1 + 0.
Empirical Risk of Smooth Stacked Ensemble Estimator
ubiquitous and popular approach for model selection and ensembling in machine learning practice
known stacking [53, 8, 50]. Given a set of trained base estimators (0) ,..., 6Pm) for example

---

representing a fitted neural network, a random forest and nearest-neighhour estimator, we call the smooth-stacked estimator the linear ensemble of those estimators {0*) weighted by coefficients that are related to the out-of-sample risk of each estimator. An important question: if we use all the samples to estimate the weights of the ensemble, then can construct confidence intervals on the risk the ensemble estimator?
The most straightforward version of stacking is to put all the weight on the model with the smallest out-of-sample risk. Other approaches proposed practice are to fit a linear regression model using the outputs of each model as an input co- co-variate to the linear model and using the learned coefficients as coefficients on the ensemble [50].
In this subsection, we analyze smooth version of stacking, proposed and analyzed experimentally, for instance, [38], that adds stability to the chosen ensemble, while putting most weight on the best performing model. This ensemble can be viewed as a regularized instance of the linear regression stacking approach where an entropic regularizer is added the square loss objective. This regularization adds smoothness and stability to the chosen ensemble and allows us to show that the distribution of the ensemble's risk can be estimated with the bootstrap, even if the all the data are used to estimate the weights or fit the base models.
Let (XF') be triangular array of i.d observations taking value in RR"; and lct (m,) be an increasing sequence. Define Fn, as the space of measurable functions from = to Rd., We estimate Pn different estimators 2n == (XT.,.)) k < Pn built on the first my data-points. Each estimator 0% is training algorithm that takes as input my samples XTm, and returns model, which itself is a function from Rdd to R We denote with (XT.m,.) the returned model and with @h T.mm. (z.) E Rd the evaluation of the model a point Zu Rd,
The loss a model at a sample is measured by a common loss function Ln R2 R + R and the empirical risk of the k-th estimator is computed on all the remaining m., data points as:
n
R. (mim)
Ln (2 (x
=
n
Wh
u=
The smooth-stacked estimator is defined as the following ensemble learner
B. ((1m))
(z1m)() E ASPn
0% (1 (-).
Eksp. e BaRK (#1m))
The hyperparameter 8n controls how concentrated the stacked estimator is around the estimator(s) 0% with the lowest empirical error. We denote the empirical risk of an ensemble model E (Rda - Rd) as
(x1n)) = /n mn
1
Σ
Cn (xi, (x1, ((x:))
1=
Let (Z") a bootstrap sample of (2.))zm... We show that the bootstrap method systematically consistent if and only if 3, = o(vn m,). For simplicity we suppose that the estimators 0 return models that when evaluated at any point xa E Rd have bounded coordinates; and that the loss function Ln is smooth, and have bounded partial derivatives in its second argument.
We write the set of convex combinations of the estimators: Ω ((0p: P S pn)) =
Epsn. upOp wp 2 and Epsp. Wp = 1 } and introduce the following notations:
Tn = sup t
sup PSPn
sm.. ) (XT
V1,
1.
Tip : sup pSpn
( Ln (Xm 0g (XT. )xx))
sup esd
sup ( vSp.))
22,6 (X ((,,((X)))
V1,
Lee

---

where by (x, y) we designate oy (x. We show that if the following hypothesis (Hptacked) holds then the bootstrap method is asymptotically consistent (proof in Appendix R).
Bndd LnTT fn_ Ln
O.
(Isstccedd)
Vn mn
Proposition 3 Choose (m,), (Bx) and (pn) be increasing sequences. Let (X.) be a triangular array of i.idd observations taking value IR Set (Cn, Rd x Rd' - R) to be a sequence of smooth loss functions. Let (ZT) and (Yr") be respectively bootstrap sample and an independent
copy of (X mm+1:-- ) Suppose that the hypothesis (Hatacked) holds then we have:
|dz(RRg (Ro. +1m) -ED--ERR [ (Z (2m.+1:)m66.) Re, (1m.+ -D B (Y (1m) 1x=2)l1. -
where we have shorthanded Ôn :- (X²)) Therefore if we choose (X*)) be such that:
P (RRS.. (Zm.. +im) E (Ro, +1m) ) 2 la,a (X xx*) aa
then the following holds
limm P 11000
R (Y mu+lm)) E R2 (Y,,. mm) 2 (X*) @m)
If 8n grows proportionally to 3, x /n- m. then the bootstrap method is not a systematically consistent estimator of the risk of the smooth stacked estimator. We present simple example illustrating this in Appendix K and establish the asymptotic distribution of the bootstrap. However, we show that using Theorem we can still propose a corrected confidence interval with guaranteed asymptotic coverage.
References
[1] Miguel A Arcones and Evarist Giné. U-processes indexed by vapmik-éervonenkis classes of functions with applications asymptolics and bootstrap of u-statistics with estimated parameters. Stochastic Processes and their Applications, 52(1):17-38, 1994.
[2] Alexandre Belloni, Victor Chernozhukov, and Kengo Kato. Uniform post-selection inference for least absolute deviation regression and other z-estimation problems. Biometrika, 102(():77-94, 2015.
[3] Rudolf Beran and Gilles R Ducharme. Asympotic theory for bootstrap methods in statistics. 1991.
[4] Rudolf Beran, Muni S Srivastava, et al. Bootstrap tests and confidence regions for functions of covariance matrix. The Annals Statistics, 13(1):95-115, 1985.
[5] Peter J. Bickel and David A. Freedman. Some asymptotic theory for the bootstrap. Ann. Statist., 1966111911 11 1981.
[6] Peter J Bickel and David A Freedman. Bootstrapping regression models with many parameters. Festschrift for Erich L. Lehmann, pages 28-48, 1983.
[7] Patrick Billingsley. Convergence of probability measures. John Wiley Sons, 2013.
[8] Leo Breiman. Stacked regressions. Machine learning, 24(1):49-64 1996.
[9] Matias Cattanco, Michael Jansson, and Kenichi Nagasawa. Bootstrap- based inference for cube root asymptotics. Econometrica, 88(5):2203-2211 2020.
[10] Arindam Chatterjee and Soumendra Nath Lahiri. Bootstrapping lasso estimators. Journal of the American Statistical Association, 106(494)6608-623 2011.
[11] Snigdhansu Chatterjee, Arup Bose, et al. Generalized bootstrap for estimating equations. The Annals of Statistics, 33(1):414-436, 2005.
10

---

[12] Sourav Chatterjee. Superconcentration and related topics, volume 15. Springer, 2014.
[13] Sourav Chatterjee al. A generalization of the lindeberg principle. The Annals of Probability, 34(6))2061-2076, 2006.
[14] Sourav Chatterjee et al. A new method of normal approximation. The Annals of Probability, 36(4):1584-1600, 2008.
[15] Xi Chen, Wen-Xin Zhou, et al. Robust inference via multiplier bootstrap. Annals of Statistics, (8(3):1665-1611 2020.
[16] Xiaohui Chen et al. Gaussian and bootstrap approximations for high-dimensional u-statistics and their applications. The Annals of Statistics, 46(2):642-678, 2018.
[17] Yen-Chi Chen, Y. Samuel Wang, and Elena A. Erosheva. On the of bootstrap with variational inference:
Theory, interpretation, and a two-sample test example. The Annals of Applied Statistics, 12(2):846 876, 2018.
[18] Victor Cheroozuukov, Denis Chetverikov, Kengo Kato, et al. Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensione random vectors. The Annals of Statistics, (1(6))2786 2819, 2013.
[19] Victor Chernozhukov, Denis Chetverikov, Kengo Kato, al. Central limit theorems and bootstrap in high dimensions. The Annals of Probability, 45(4):2309-25 2352, 2017.
[20] Victor Chernozhukov, Denis Chetverikov, Kengo Kato, and Yuta Koike. Improved central limit theorem and bootstrap approximations high dimensions. arXiv preprint arXiv: 1912. 10529, 2019.
[21] Anthony Christopher Davison and David Victor Hinkley. Bootstrap methods and their applica- tion. Number 1. Cambridge university press, 1997.
[22] Hang Deng. Slightly conservative bootstrap for maxima of sums. 10X1v 2020.
arXiv preprint
[23] Hang Deng and Cun-Hui Zhang. Beyond gaussian approximation: Bootstrap for maxima of sums of independent random vectors. arXiv preprint 1705.92705,,9558, 2017.
[24] Ruben Dezeure, Peter Bühlmann, and Cun-Hui Zhang. High-dimensional simultaneous infer- ence with the bootstrap. Test, 26(4):685-719, 2017.
[25] Morris L Eaton and David E Tyler. On wiclandt's inequality and its application the asymptotic distribution of the cigenvalues of a random symmetric matrix. The Annals of Statistics, pages 260 271, 1991.
[26] B. Efron. Bootstrap Methods: Another Look the Jackknife. The Annals of Statistics, 7(1):1 26, 1979.
[27] Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics, pages 569-593. Springer, 1992.
[28] Noureddine El Karoui and Elizabeth Purdom. Can we trust the bootstrap high-dimensions? the case of linear models. The Journal of Machine Learning Research, 19(1):170-235, 2018.
[29] Noureddine El Karoui and Elizabeth Purdom. The non-parametric bootstrap and spectral analysis in moderate and high-dimension. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2115-2124, 2019.
[30] Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-problem. Advances in neural information processing systems, 19:513-520, 2006.
[31] Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723773,, 2012.
11

---

[32] Arthur Gretton, Kenji Fukumizu, Zaid Harchaoui, and Bharath K. Sriperumbudur. A fast, consistent kernel two-sample test. In Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
[33] Peter Hall. The bootstrup and Edgeworth expansion. Springer Science & Business Media, 2013.
[34] Fang Han, Sheng Xu, Wen-Xin Zhou, et al. On gaussian comparison inequality and its application to spectral analysis of large random matrices. Bernoulli, 24(3):187-1183,, 2018.
[35] Pawel Hitczenko. Best constants in martingale version of rosenthal's inequality. The Annals of Probability, pages 1656-1668, 1990.
[36] Roger Johnson. introduction to the bootstrap. Teaching statistics, 23(2):49-54, 2001.
[37] lain M Johnstone and Debashis Paul. Pea in high dimensions: An orientation. Proceedings of the IEEE, 106(8):1277-1222 2018.
[38] Cheng Ju, Aurélien Bibaut, and Mark van der Laan. The relative performance of ensemble methods with deep convolutional neural networks for image classification. Journal of Applied Statistics, 45((5):2800-2818 2018.
[39] Abbas Khosravi, Saeid Nahavandi, Doug Creighton, and Amir F. Atiya. Comprehensive review of neural network based prediction intervals and advances. IEEE Transactions on Neural Networks, (2(9):134111355) 2011.
[40] Ron Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Ijcai, volume 14, pages 1137-1145. Montreal, Canada, 1995.
[41] Yuta Koike. Notes on the dimension dependence in high-dimensional central limit theorems for hyperrectangles. Japanese Journal of Statistics and Data Science, pages 1-41, 2020.
[42] Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Dougal Sutherland. Learning deep kernels for non-parametric two-sample tests. arXiv preprint arXiv:2002.09) 09116,, 2020.
[43] Miles E Lopes. Central limit theorem and bootstrap approximation in high dimensions with near square root of rates. arXiv preprint arXiv:2009. 06004, 2020.
[44] Miles E Lopes, Benjamin Erichson, and Michael W Mahoney. Bootstrapping the operator norm in high dimensions: Error estimation for covariance matrices and sketching, arXiv preprint arXiv:1909.06129 2019.
[45] Enno Mammen. Asymptotics with increasing dimension for robust regression with applications the bootstrap. Annals of Statistics, 17(1):382-400, 1989.
[46] Enno Mammen. When does bootstrap work?: asymptotic results and simulations, volume 77. Springer Science & Business Media, 2012.
[47] Dimitris N Politis and Joseph P Romano. Large sample confidence regions based on subsamples under minimal assumptions. The Annals of Statistics, pages 2031-2050, 1994.
[48] Kesar Singh. On the asymptotic accuracy of efron's bootstrap. The Annals of Statistics, pages 1187-1195, 1981.
[49] Vladimir Spokoiny, Mayya Zhilova, et al. Bootstrap confidence sets under model misspecifica- tion. The Annals of Statistics, 4366):265322655 2015.
[50] van der Laan Mark, Polley Eric C, and Hubbard Alan E. Super learner. Statistical Applications in Genetics and Molecular Biology, 6(1):1-23,200)
[51] Li Wenliang, Dougal Sutherland, Heiko Strathmann, and Arthur Gretton. Learning deep kernels for exponential family densities. In International Conference on Machine Learning, pages 6737-6746, 2019.
12

---

[52] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378, 2016.
[53] David H. Wolpert. Stacked generalization. Neural Networks, 5(2):241 259, 1992.
[54] Chien-Fu Jeff Wu et al. Jackknife, bootstrap and other resampling methods in regression analysis. the Annals of Statistics, 14(4):1261-1295, 1986.
[55] Dixin Zhang. Bayesian bootstraps for u- u-processes, hypothesis tests and convergence of dirichlet u-processes. Statistica Sinica, pages 463-478, 2001.
13


Reviews:
[{"review_id": "z4OEKudf5CP", "reviewer": "Reviewer_nvUU", "summary": "This paper proposed a new metric for arguing the validity of bootstrap approximation, which leads to a new class of estimators that the bootstrap will be valid for recovering the asymptotic distribution. The authors provided a good theoretical justification of their findings. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. This paper proposed to use the maximum mean discrepancy to measure the difference between two probability measures.\n2. With this metric, the authors show that the empirical bootstrap approximates the asymptotic distribution of some estimators (e.g., of soft-max types). \n3. The key idea in the proof is a clever interpolation method (similar to the Slepian's interpolation).\n\nOverall, I found this paper novel and interesting. The new findings are pretty important and could be very useful. \nI have three comments/questions that I wish the authors to clarify:\n\nQ1. [Motivation] \nThe motivation should be clarified. The authors proved a new class of estimators that bootstrap would work. \nSo the authors could start with a class of estimator and argue that it was unclear if we can apply bootstrap to this class because all the existing theories (e.g., Berry-Esseen theorem, anti-concentration from Chernozhukov et al) do not apply.\nThen the authors can argue that in this paper, they have proved that the bootstrap also applied to this case.\n\nQ2. [Rate of approximation]\nMany results for the bootstrap will include the rate of approximation (e.g., how fast the difference in the approximation goes to 0). Do the authors have ideas on the rate based on this method?\n\nQ3. [Practical examples]\nThe only (positive) example given in this paper was in Section 5, the smooth stacked ensemble estimator. \nBut the construction seems to be a bit artificial. \nCan the authors give a more practical example? such as an estimator from boosting or other approach? \n\nQ4. [Assumption H0 and H1]\nThe authors provided counterexamples for these two assumptions in appendix. \nBut I think most people are more interested in the positive example of these two assumptions. \nCan the authors provide more examples that these two assumptions hold?\nIdeally, the examples should be estimators that will be used by practitioners.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any major weakness of the paper as a theory work. \nPerhaps the motivations should be clarified and more practical examples to be provided.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "lRpLLIjPiP", "reviewer": "Reviewer_YfRX", "summary": "The authors provide theoretical results for the bootstrap, attempting to address a common issue in machine learning: namely, that it is difficult to obtain closed-form variance estimators for many quantities of interest when machine learning is used. However, the necessary assumptions stated by the authors for the theoretical results to hold are quite strong.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is quite dense. Please consider adding some exposition to each section to aid the reader.\n- The theoretical developments build upon recent developments, and are nice. However, the assumptions are quite strong and appear unlikely to hold in many cases. Can you provide more exposition about these assumptions, why they are necessary, and in which cases they hold? Is it possible to relax the assumptions?\n- In particular, the conditions of Theorem 2 seem very strong. Can you discuss these and whether or not they can be relaxed?\n- The discussion on line 206 is important and should be given more weight.\n- The manuscript ends abruptly, and would benefit from:\n  - numerical experiments showing that the theoretical properties are achieved in finite samples\n  - a discussion of the conditions, as mentioned above\n  - a data example, showing that the procedure gives reasonable results in practice\n  - more discussion of the results in the Supplementary Material (since a lot of work went into deriving these results).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not discussed; while the paper proposes new theory, the proposal can be used widely in decision-making, which does have downstream societal impact. It would be nice to have some discussion of this point.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "I4MlIhkjRD2", "reviewer": "Reviewer_L7uL", "summary": "This paper was inspired by Chatterjee 2006's work on Lindeberg's condition. They established a new set of sufficient conditions under which some nice nonlinear statistics can be approximated using a variety of bootstrap resampling techniques. Several applications in machine learning are provided. This is a theoretically dense work that complements recent results in bootstrap and Gaussian approximations in high-dimensional statistics.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I like the theoretical results the authors obtained. But the write-up and presentation can be further polished and improved. For example, there are still quite a few typos or grammatical errors in the submission. The technical proof is quite dense, which is evident from the length of the supplementary materials. A guide for readers on how to read the supplementary materials is necessary for this paper to have more impact if they hope machine learning researchers could use their tools to perform honest statistical inference.\n\nIn terms of technical content, the authors propose a condition called $\\mathbb{C}^{3}$-approximability and use a surrogate $\\mathbb{C}^{3}$-function sequence as an intermediate step to show bootstrap consistency. This is an important assumption but the authors delay the discussion of this assumption to Appendix C. I understand due to NeurIPS's page limit there has to be a trade-off on what to be included in the main text. But I am worried that the supplementary materials will eventually be buried under most readers' desk drawers. I also hope that the authors can provide one example (if they could) that violates Assumption 1 yet the bootstrap consistency still holds.\n\nThe authors advocate that the strength of their paper provides tools to show bootstrap consistency for general nonlinear functionals beyond what people have done case by case. But it would be useful if the authors could recover several established results as special cases of their master theorem.\n\nIn addition, the authors admit that their technical results are inspired by Chatterjee's work. Then what technical challenge do they need to solve? Are the proofs, even though very long, standard or requiring some innovations? These should be made clearer.\n\nSome extra comments:\n\n(1) Lines 47-49: \"Notably, this intuition has already been exploited to show that the bootstrap method is consistent in particular applications in the econometric literature, such as the construction of uniform confidence bands.\" This sentence should be accompanied by relevant citations.\n\n(2) Lines 88-89: \"We note that the limiting distribution of those statistics are in general not Gaussian [22].\" (i) \"distribution\" should be \"distributions\"; (ii) I understand that the authors try to tell readers Gaussian limiting distributions for these statistics are established under specific conditions, e.g. $U$-statistics with dominating first-order terms in Hoeffiding's decomposition. But this and later sentences seem to be quite disconnected from previous sentences without adding more contexts. For example, Xiaohui Chen's work [15] is about Gaussian limiting distribution for high-dimensional $U$-statistics.\n\n(3) I would say 70% of NeurIPS submission does not contain such a long supplementary material filled with technical lemmas and proofs. One presentation style that I found very useful for technical papers in statistics and theoretical machine learning is to draw a diagram indicating which technical lemma was used to prove Theorem X and how different technical results are connected.\n\n(4) Lines 146-147: \"Given that these results typically require stronger conditions on the statistic and many times Gaussian limits.\" What does \"many times Gaussian limits\" mean exactly?\n\n(5) Line 231: \"the test statistics is computed on\" should be \"the test statistics are computed on\"\n\n(6) A recent paper (first appeared in 2020) by Vladimir Koltchinskii (https://arxiv.org/pdf/2011.03789.pdf) considered a relatively similar setup, but he focused on Gaussian limit and root-n parametric theory for estimating nonlinear statistics. I suggest the authors also cite this paper and discuss the connection and difference between their work and Koltchinskii's. \n\n(7) No discussion is written. The authors are recommended to write at least some sentences on future works or caveats... This is also strongly encouraged by NeurIPS.\n\nGiven above, I temporarily give the authors a score of 5 but if they could convince me with their rebuttal I am willing to change my evaluation. Also, this paper seems to fit better in a journal in statistics, JMLR or a TCS-type conference. The page limit of NeurIPS may have a negative impact on this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a theoretical study on the validity of the bootstrap approximation. All reviewers agreed that this paper is an important study and that its contributions are novel. What was discussed during the review process was the difficulty in understanding how the new theoretical results provided in this paper are positioned in relation to existing theoretical results, and the lack of discussion on the validity of the assumptions of the theoretical analysis. We received responses from the authors that both problems can be improved, but the former problem in particular requires significant revision, and we thus have concluded that this paper cannot be accepted at this time.
Decision: Accept (Poster)