Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper deals with the following problem:
Suppose we are interested in solving the generalized moment problem 

$M := \mathbb{E}_Z m(Z,\theta, \hat{g}) = 0$, in $\theta$. 

Here $Z$ is a random variable, $m$ the moment function, $\theta$ a parameter, and $\hat{g}$ an estimator, that is learned from data. 
The goal is to find $\hat{\theta}$ that makes the above expectation $M$ small. 



Such a situation occurs, for instance, in treatment effect estimation settings.  In that case, $Z$ consists of $Z=(Y,T,X)$, where $Y$ is the outcome, $T$ is the treatment variable, and $X$ additional features. The $\hat{g}$ to be learned from data is an estimator of $T$ and $Y$ as functions of $X$. 

Typically, to obtain bounds on the estimation of $\hat{\theta}$, one splits the sample into two parts. Then the first part is used to estimate $\hat{g}$ and the second part is used to estimate the empirical minimizer $\hat{\theta}$ of $M$, based on $\hat{g}$.

In this paper it is shown that if $m$ is a _linear_ function of $\theta$ (e.x.  $Y = \theta T + \ldots$ in example above), and the classifier is chosen by an algorithm that has the algorithmic stability property,  then one can use the full same sample to execute both estimations above, and still have the unbiasedness and asymptotic normality property for the estimator of $\theta$.

It is also shown that bagging type estimators have algorithmic stability under some conditions on the base estimator. As an example of this, 
the authors show that bagging ensemble of 1-nearest neighbor classifiers has algorithmic stability and thus can be used as $\hat{g}$ in the main result. 
 

strengths_and_weaknesses: There are three serious issues with this paper.  **(1)** The first issue is that this paper is not well written. In terms of language and focus of presentation I believe it is not suitable for general ML audience in its current form. Moreover, in terms if technical presentation, it is also very hard to read. Definitions and references are hard to find around the paper.  
**(2)** The second issue is that the connection between bagging and stability is well known. It is not completely clear why the authors prove these results, and claim them as contributions.  **(3)** The scope of the results is limited. Only linear moment functions are considered, and the results are about just asymptotic estimates rather than finite sample bounds. 


To discuss these points in more detail: 

(1) This paper will be very hard to read to people who are not familiar with the generalized method of moments, treatment effect estimation and instrumental variables. I believe most NeurIPS participants are not familiar with these notions, although they are very familiar with the underlying mathematics. Therefore I would suggest to the authors to completely rewrite the paper, carefully introduce the problems  and their importance, before discussing the paper's contribution.  

(2) Connections between bagging and stability are natural and known. See for instance [a] below. Even if the specific version of the results 
in this paper is new, it should be discussed in the general context.  How do the results presented in the paper differ from standard results?
In addition, while 1-NN classifiers are rarely used in ML communities. Thus I would suggest providing more realistic classifier examples. Would regularized kernel regression satisfy the stability assumptions used in the paper?



[a]  Elisseeff, A., Evgeniou, T., Pontil, M., & Kaelbing, L. P. (2005). Stability of Randomized Learning Algorithms. Journal of Machine Learning Research, 6(1).



questions: Please see the discussion above. 
limitations: Limitations and potential negative societal impact were not discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a theoretical argument that debiased machine learning does not require sample splitting when the estimation algorithms are leave-one-out stable.
strengths_and_weaknesses: The paper makes a highly interesting theoretical contribution by showing that the "Stochastic Equicontinuity" condition is implied by a stability condition on the estimation algorithm (Lemma 2). This algorithmic stability condition is shown to be weaker than the conditions previously imposed in the literature (Remark 2). The contribution is also interesting from a practical point of view because it provides an explanation of why sample splitting is or is not needed for debiased machine learning.

One criticism about the result itself is that it may be difficult to check the stability conditions for arbitrarily complicated algorithms except in some special cases, e.g., bagged estimators. If this is the case, a practitioner may feel more safe sticking with the sample splitting approach. Also, the paper does not give any experimental results to back up its claims. Since the theoretical results describe the limiting behavior, it would have been helpful to have extensive experimental results to see how the claims hold up at finite sample sizes.
questions: 1. Could you explain in greater detail the level of difficulty expected in verifying the conditions of Lemma 2? What about compared to the level of difficulty verifying the conditions of Theorem 1? What about compared to conditions previously imposed in the literature? If the algorithmic stability conditions cannot be verified, what course of action would you recommend? Also, what precisely is the relationship between the "Algorithmic Stability" condition of Corollary 4 and the "Consistency Rate" condition of Theorem 1?

2. Could you explain
- the second to last equality in the unnumbered display equation following Line 183,
- the second to last equality in the unnumbered display equation following LIne 186, and
- the last equality in the unnumbered display equation following Line 198?

3. Could you explain in greater detail the significance of Lemma 3? In particular, how does this example illustrate that "the stability condition [you] impose is required for stochastic equicontinuity condition to hold" when it is later stated that "$\nu(\cdot, \hat g)$ does not satisfy our stability conditions"?

4. Would it be possible to observe experimentally the effect of algorithmic stability on asymptotic normality? For example, the size of $m$ is related to the stability of the bagged estimator. Can we observe the relationship of $m$ to the convergence to normal distribution by looking at a grid of $m$'s (either $m = an$ or $m = n^a$ for different values of $a$)?

5. I feel like the paper could use more editorial judgment. This is probably just my impression, but the current version feels like it used to be a longer paper that was abruptly cut to meet the page limit. I find that individual parts are written quite well, but the paper as a whole appears to be missing some significant pieces.
limitations: The authors do not discuss the limitations and potential negative societal impact of their work to any noticeable extent. Still, given the topic of research, I do not consider it a defect.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: New conditions on nuisance function based on leave-one-out stability that warrant root-n consistency and asymptotic normality of target parameters are proposed and theoretically investigated. It is also concretely shown that these conditions are satisfied by ensemble bagged estimators.
strengths_and_weaknesses: Strengths:
+ New result can be applied on causal inference with machine learning method, which is timely and impactful
+ Theoretical merits are clearly demonstrated by comparing with relevant existing works
+ New mathematical arguments that lead to refined theoretical results, which are shown to be unimprovable

questions: (1)	I believe this paper’s contribution is substantial. That being said, I wonder to what extent $m(Z;\theta,g)$’s linearity in $\theta$ can be relaxed. It would be great if some discussion along this vein could be included in the paper. If $\theta$ and $g$ are not separable, I guess the stochastic equicontinuity in line 102 will need to hold uniformly in $\theta$ for $\theta$ in a neighborhood of $\theta_0$? Would stability still be sufficient for stochastic equicontinuity? If $m(Z;\theta,g)$ is nonlinear in $\theta$, would linearization trick like Taylor expansion to first order work under some additional conditions?

(2)	In Theorem 5, is the norm $\|Y\|_s = (E[Y^s])^{1/s}$? If so, this norm is not defined.

limitations: The moment condition has to be linear in target parameters.
review_title: null

Meta-review: metareview: The paper provides an analysis that avoids sample-splitting while estimating parameters with objective functions involving other estimated parameters. The results are interesting and the analysis is insightful. 

One of the reviewers (Reviewer Becs) raised some concerns about style of the writing, high-level questions on (bagging and stability) and linear moment assumptions. In my reading of the paper, I found no significant issues about the style (in fact the paper is well-written from a statistical point-of-view). The authors addressed the other two issues in the rebuttal. Hence, the score of reviewer Becs was down-weighted and the paper is recommend for acceptance at Neurips. 
confidence: Certain
Decision: Accept