Asymptotics of the Bootstrap via Stability with Applications to Inference with Model Selection

Abstract: One of the most commonly used methods for forming confidence intervals is the empirical bootstrap, which is especially expedient when the limiting distribution of the estimator is unknown. However, despite its ubiquitous role in machine learning, its theoretical properties are still not well understood. Recent developments in probability have provided new tools to study the bootstrap method. However, they have been applied only to specific applications and contexts, and it is unclear whether these techniques are applicable to the understanding of the consistency of the bootstrap in machine learning pipelines. In this paper, we derive general stability conditions under which the empirical bootstrap estimator is consistent and quantify the speed of convergence. Moreover, we propose alternative ways to use the bootstrap method to build confidence intervals with coverage guarantees. Finally, we illustrate the generality and tightness of our results by examples of interest for machine learning including for two-sample kernel tests after kernel selection and the empirical risk of stacked estimators.


Reviews and author responses:

 We want to thank the reviewer for their time and their questions.

*“In terms of the usefulness of the master theorem: it seems that recovering only one result is somewhat limited. But I am sympathetic that the paper is already quite long and dense. Maybe it will be helpful if the readers could point out several other theorems that their results might be useful to prove in a discussion section.”*

We did not mean to say that we can only recover only one known case but rather that we propose as an example only one known application which is the maximum of centered empirical averages. We are sorry for the confusion and lack of clarity of our answer. However, our theorem also applies to many other known cases such as any empirical averages or t-statistics or von Mises functional ( see e.g “
Some Asymptotic Theory for the Bootstrap” Bickel& Freedman (1981)
) or M-estimators “(see e.g “Asymptotics with Increasing Dimension for Robust Regression with Applications to the Bootstrap” Mammen (1989) )  etc...

In addition to this we also use our theorem to study two other applications that cannot be studied using previously known theorems which are two sample kernel tests where the kernel is chosen using data among exponentially many different kernels, and the empirical risks of a smoothed stack estimator (see e.g. Ju, Bibaut & Van der Laan (2017)). Both of those examples are of interest in machine learning and for both of those examples, the consistency of the bootstrap method was unknown. For the first example, this allows us to estimate the p-value and for the second example to build confidence intervals around the risk of the smooth stacked estimator.


*“In terms of Assumption 1: my question is if Assumption 1 is necessary and if not could you provide a specific example where Assumption 1 is not needed for bootstrap consistency.”*


We do not claim that it is necessary but we believe Assumption 1 to be close to tight as we provide counterexamples that “barely don’t respect” Assumption 1 and for which the bootstrap method is not consistent. 
 On the other hand, an example of statistics that is not covered by assumption 1 but for which bootstrap should be consistent are certain u-statistics such as $\sum_{i,j}\mathbb{I}(X_i\le X_j)$.




 Thank you for the responses.

In terms of Assumption 1: my question is if Assumption 1 is necessary and if not could you provide a specific example where Assumption 1 is not needed for bootstrap consistency.

In terms of the usefulness of the master theorem: it seems that recovering only one result is somewhat limited. But I am sympathetic that the paper is already quite long and dense. Maybe it will be helpful if the readers could point out several other theorems that their results might be useful to prove in a discussion section.

In terms of Chatterjee's extension: I'm ok with your response but it also seems not a "big" extension per se from his result.

Unfortunately, my original evaluation remains in light of the technicality and the applications that the authors show in the current draft.

 Thank you very much for the suggestions in part Q1 regarding the presentation of the motivation of our work. Those are great suggestions that we will incorporate.

*“Many results for the bootstrap will include the rate of approximation (e.g., how fast the difference in the approximation goes to 0). Do the authors have ideas on the rate based on this method?”*

Yes. Theorem 1 allows us to determine the rate. We are happy to add a rate for our examples.

*“The only (positive) example given in this paper was in Section 5, the smooth stacked ensemble estimator. But the construction seems to be a bit artificial. Can the authors give a more practical example? such as an estimator from boosting or other approach?”*

This is inexact. We also obtain a positive result for the two-sample kernel test in section 4, which is of interest in machine learning. We also believe that the example in section 5 is not artificial. Indeed it is known that when choosing between many different estimators, choosing the one with the lowest empirical error can lead to an estimator that is unstable. To go around this problem one can instead consider a soft argmin (which will under general conditions be stable). This is the estimator that we are studying.  Moreover, such softmax/min variants of stacking have already been proposed in the literature (see e.g. [arxiv 1704.01664](https://arxiv.org/pdf/1704.01664.pdf)); we will add more refs to the paper in this direction). Other examples could have been studied. Another example is the K-fold cross-validated risk for which deriving a confidence interval is of great interest. Our theorems apply to that context even when the limiting distribution of the K-fold cross-validated risk is unknown. We are happy to add this to the supplementary material for the camera-ready version. Another example are two-step M-estimators.



 We would like to thank the reviewer for the helpful suggestions. We answer specific concerns below.

*“However, the assumptions are quite strong and appear unlikely to hold in many cases. Can you provide more exposition about these assumptions, why they are necessary, and in which cases they hold? Is it possible to relax the assumptions?”*

In the appendix we propose two counterexamples for which Assumption 1 does not hold and for which the bootstrap method is not consistent. The conditions are close to tight (as demonstrated by our counterexamples) and we do not see how those conditions could be relaxed while keeping the theorem general and applicable to many machine learning estimators. We will provide more discussions on the assumptions for the camera-ready version. 

*“In particular, the conditions of Theorem 2 seem very strong. Can you discuss these and whether or not they can be relaxed?”*

The conditions in Theorem 2 are indeed stronger than the conditions in theorem 1. While we do not see any obvious ways to relax them we propose alternative methods to build confidence intervals in appendix H. Those methods rely on different assumptions.


 Thank you for the suggestions. The proof is indeed very technical and we will incorporate an overview of the proof for the camera-ready version. The suggestion of having a diagram to explain how each lemma is used is a very good idea that we will incorporate.
We hear the concerns of the reviewer considering the presentation. The typos and the presentation will be cleaned before the camera-ready version.


*“I also hope that the authors can provide one example (if they could) that violates Assumption 1 yet the bootstrap consistency still holds.”*

We assume that the reviewer meant an example where Assumption 1 is violated and the bootstrap is not consistent (i.e a counterexample that helps us show the tightness of our conditions). If this is the case, we do. We provide two counterexamples in the supplementary material.

*“ But it would be useful if the authors could recover several established results as special cases of their master theorem.”*

We do. In the appendix, we notably show how we recover the result by Deng &Zheng(2020) for the maximum of empirical processes.


*“In addition, the authors admit that their technical results are inspired by Chatterjee's work. Then what technical challenge do they need to solve? Are the proofs, even though very long, standard or requiring some innovations? These should be made clearer.”*

Chaterjee’s contribution is to propose a new tool that helps comparing the expectation of $f(X_1,\dots,X_n)$ and $f(Y_1,\dots,Y_n)$ when $(X_i)$ and $(Y_i)$ are  random variables  potentially with different distributions.  Using Chaterjee’s idea, while not being standard, has been done in other papers including Deng & Zheng (2020). We adapt it and expand it. As one example technical difference: instead of doing a Taylor’s expansion of $ h\circ g_n$ around each observation $X_i$, we do a Taylor expansion on $h$ around $g_n(X_0,\dots,X_n)$ and $g_n(X_0,\dots, X_{i-1},0,X_{i+1},\dots,X_n)$. 


*“Given that these results typically require stronger conditions on the statistic and many times Gaussian limits." What does "many times Gaussian limits" mean exactly?”*

We just meant that the existing results “often” require Gaussian limits. We will maybe re-phrase to: “and often require Gaussian limits”.


This paper was inspired by Chatterjee 2006's work on Lindeberg's condition. They established a new set of sufficient conditions under which some nice nonlinear statistics can be approximated using a variety of bootstrap resampling techniques. Several applications in machine learning are provided. This is a theoretically dense work that complements recent results in bootstrap and Gaussian approximations in high-dimensional statistics.  Overall I like the theoretical results the authors obtained. But the write-up and presentation can be further polished and improved. For example, there are still quite a few typos or grammatical errors in the submission. The technical proof is quite dense, which is evident from the length of the supplementary materials. A guide for readers on how to read the supplementary materials is necessary for this paper to have more impact if they hope machine learning researchers could use their tools to perform honest statistical inference.

In terms of technical content, the authors propose a condition called $\mathbb{C}^{3}$-approximability and use a surrogate $\mathbb{C}^{3}$-function sequence as an intermediate step to show bootstrap consistency. This is an important assumption but the authors delay the discussion of this assumption to Appendix C. I understand due to NeurIPS's page limit there has to be a trade-off on what to be included in the main text. But I am worried that the supplementary materials will eventually be buried under most readers' desk drawers. I also hope that the authors can provide one example (if they could) that violates Assumption 1 yet the bootstrap consistency still holds.

The authors advocate that the strength of their paper provides tools to show bootstrap consistency for general nonlinear functionals beyond what people have done case by case. But it would be useful if the authors could recover several established results as special cases of their master theorem.

In addition, the authors admit that their technical results are inspired by Chatterjee's work. Then what technical challenge do they need to solve? Are the proofs, even though very long, standard or requiring some innovations? These should be made clearer.

Some extra comments:

(1) Lines 47-49: "Notably, this intuition has already been exploited to show that the bootstrap method is consistent in particular applications in the econometric literature, such as the construction of uniform confidence bands." This sentence should be accompanied by relevant citations.

(2) Lines 88-89: "We note that the limiting distribution of those statistics are in general not Gaussian [22]." (i) "distribution" should be "distributions"; (ii) I understand that the authors try to tell readers Gaussian limiting distributions for these statistics are established under specific conditions, e.g. $U$-statistics with dominating first-order terms in Hoeffiding's decomposition. But this and later sentences seem to be quite disconnected from previous sentences without adding more contexts. For example, Xiaohui Chen's work [15] is about Gaussian limiting distribution for high-dimensional $U$-statistics.

(3) I would say 70% of NeurIPS submission does not contain such a long supplementary material filled with technical lemmas and proofs. One presentation style that I found very useful for technical papers in statistics and theoretical machine learning is to draw a diagram indicating which technical lemma was used to prove Theorem X and how different technical results are connected.

(4) Lines 146-147: "Given that these results typically require stronger conditions on the statistic and many times Gaussian limits." What does "many times Gaussian limits" mean exactly?

(5) Line 231: "the test statistics is computed on" should be "the test statistics are computed on"

(6) A recent paper (first appeared in 2020) by Vladimir Koltchinskii (https://arxiv.org/pdf/2011.03789.pdf) considered a relatively similar setup, but he focused on Gaussian limit and root-n parametric theory for estimating nonlinear statistics. I suggest the authors also cite this paper and discuss the connection and difference between their work and Koltchinskii's. 

(7) No discussion is written. The authors are recommended to write at least some sentences on future works or caveats... This is also strongly encouraged by NeurIPS.

Given above, I temporarily give the authors a score of 5 but if they could convince me with their rebuttal I am willing to change my evaluation. Also, this paper seems to fit better in a journal in statistics, JMLR or a TCS-type conference. The page limit of NeurIPS may have a negative impact on this paper. Yes.

The authors provide theoretical results for the bootstrap, attempting to address a common issue in machine learning: namely, that it is difficult to obtain closed-form variance estimators for many quantities of interest when machine learning is used. However, the necessary assumptions stated by the authors for the theoretical results to hold are quite strong.  - The paper is quite dense. Please consider adding some exposition to each section to aid the reader.
- The theoretical developments build upon recent developments, and are nice. However, the assumptions are quite strong and appear unlikely to hold in many cases. Can you provide more exposition about these assumptions, why they are necessary, and in which cases they hold? Is it possible to relax the assumptions?
- In particular, the conditions of Theorem 2 seem very strong. Can you discuss these and whether or not they can be relaxed?
- The discussion on line 206 is important and should be given more weight.
- The manuscript ends abruptly, and would benefit from:
  - numerical experiments showing that the theoretical properties are achieved in finite samples
  - a discussion of the conditions, as mentioned above
  - a data example, showing that the procedure gives reasonable results in practice
  - more discussion of the results in the Supplementary Material (since a lot of work went into deriving these results). Not discussed; while the paper proposes new theory, the proposal can be used widely in decision-making, which does have downstream societal impact. It would be nice to have some discussion of this point.

This paper proposed a new metric for arguing the validity of bootstrap approximation, which leads to a new class of estimators that the bootstrap will be valid for recovering the asymptotic distribution. The authors provided a good theoretical justification of their findings.   1. This paper proposed to use the maximum mean discrepancy to measure the difference between two probability measures.
2. With this metric, the authors show that the empirical bootstrap approximates the asymptotic distribution of some estimators (e.g., of soft-max types). 
3. The key idea in the proof is a clever interpolation method (similar to the Slepian's interpolation).

Overall, I found this paper novel and interesting. The new findings are pretty important and could be very useful. 
I have three comments/questions that I wish the authors to clarify:

Q1. [Motivation] 
The motivation should be clarified. The authors proved a new class of estimators that bootstrap would work. 
So the authors could start with a class of estimator and argue that it was unclear if we can apply bootstrap to this class because all the existing theories (e.g., Berry-Esseen theorem, anti-concentration from Chernozhukov et al) do not apply.
Then the authors can argue that in this paper, they have proved that the bootstrap also applied to this case.

Q2. [Rate of approximation]
Many results for the bootstrap will include the rate of approximation (e.g., how fast the difference in the approximation goes to 0). Do the authors have ideas on the rate based on this method?

Q3. [Practical examples]
The only (positive) example given in this paper was in Section 5, the smooth stacked ensemble estimator. 
But the construction seems to be a bit artificial. 
Can the authors give a more practical example? such as an estimator from boosting or other approach? 

Q4. [Assumption H0 and H1]
The authors provided counterexamples for these two assumptions in appendix. 
But I think most people are more interested in the positive example of these two assumptions. 
Can the authors provide more examples that these two assumptions hold?
Ideally, the examples should be estimators that will be used by practitioners.

 I do not see any major weakness of the paper as a theory work. 
Perhaps the motivations should be clarified and more practical examples to be provided.

Meta-review: This paper presents a theoretical study on the validity of the bootstrap approximation. All reviewers agreed that this paper is an important study and that its contributions are novel. What was discussed during the review process was the difficulty in understanding how the new theoretical results provided in this paper are positioned in relation to existing theoretical results, and the lack of discussion on the validity of the assumptions of the theoretical analysis. We received responses from the authors that both problems can be improved, but the former problem in particular requires significant revision, and we thus have concluded that this paper cannot be accepted at this time.

Decision: accept