Title: Which Algorithms Have Tight Generalization Bounds?
Venue: NeurIPS 2025
Accepted: yes
Keywords: learning theory
Abstract: We study which machine learning algorithms have tight generalization bounds with respect to a given collection of population distributions. Our results build on and extend the recent work of Gastpar et al. (2023). First, we present conditions that preclude the existence of tight generalization bounds. Specifically, we show that algorithms that have certain inductive biases that cause them to be unstable do not admit tight generalization bounds. Next, we show that algorithms that are sufficiently loss-stable do have tight generalization bounds.  We conclude with a simple characterization that relates the existence of tight generalization bounds to the conditional variance of the algorithm's loss.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper study the tight generalization bound under the definition 'Estimability'. And give some examples, the conditions lead to estimability and preclude estimability.\n\nSTRENGTHS AND WEAKNESSES:\n(1): $I\\ think\\ the\\ writing\\ of\\ this\\ paper\\ is\\ bad.$ The introduction is nearly 7 pages long, while the main text is hastily concluded in less than 3 pages, to the point that the experiment had to be placed in the appendix. As an article dominated by theory, I think the author has used too much space to informally present the theoretical results.  Additionally, please define the symbols before using them, such as $U(D)$.\n\n(2): $I\\ cannot\\ agree\\ with\\ the\\ author\\ on\\ the\\ significance\\ of\\ the\\ theorems\\ presented.$ \n\n$Theorem\\ 3.1\\ and\\ 3.2$: In the theorem, the reason for such a lower bound is mainly due to the choice of the distribution collection $D$, and it has nothing to do with the properties of the algorithm itself. Whether the algorithm is SGD, memorizaiton or something else, this lower bound necessarily exists due to the choice of the distribution collection $D$. The title of the article is \"Which Algorithms Have Tight Generalization Bounds?\", which clearly does not match the meaning expressed by this theorem. \n\nMoreover,  the theorem requires the restriction of the number of samples $m$, which further limits the significance of the theorem. It is clear that when the number of samples is small, the generalization performance of the algorithm on complex distributions will not be good, making the conclusion of the theorem somewhat obvious. When the number of samples is large, it is evident that this lower bound will not hold. And the author also does not further discuss the potential impact of $m$ on various algorithms.\n\nIn summary, the establishment of these two theorems primarily relies on factors external to the algorithms themselves, which does not align with the previous examples and the main theme of this paper.\n\n$Theorem\\ 4.3$:  The assumption of this theorem is too strict. The author assume the algorithm to have good stability for any distribution in theorem 4.3, then the theorem 4.3 under this assumption can be understood as: algorithms with good generalization to any distribution have a tight generalization bound, this conclusion obviously has no guiding significance in the real world, this is almost just a simple extension of the classic conclusion. We need to know which algorithms have good generalization rather than simply adding it as the assumptions.\n\nQUESTIONS:\nSee weakness. \n\nI would like the author to further explain the significance and meaning of these theorems and how these theorems can guide us in determining which algorithms have tight generalization bounds?\n\nLIMITATIONS:\nThere is no section 'conculsion and limitation' in the paper, but there is no moral issue in this article.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the question of *estimability* of learning algorithms, i.e. for which learning algorithms there may exist an estimator that is close to the population risk with high probability. It presents conditions that preclude the existence of generalization bounds for certain interpolating algorithms, and sufficient conditions for estimability, in terms of the algorithm's *stability*.\n\nSTRENGTHS AND WEAKNESSES:\n**  Strengths **\n \nThis work addresses an important challenge in understanding generalization and deriving generalization bounds, an area where progress has been slow in recent years &ndash; particularly for neural networks. The paper is well-written with the examples in Section 1.4 covering various scenarios, highlighting the nuanced differences between learnability and estimability. The theorems presented are novel, to the best of my knowledge, and have a fairly general scope.\n \n** Weaknesses **\n \nSince the authors repeatedly emphasize the case of deep learning, there should be a discussion dedicated to the hypothesis class parameterized by NNs (eg. their VC dimension) and of gradient descent (and, perhaps, other related optimization algorithms); as for the former, I would like to see Appendix A.3 in the main text. There should also be a discussion on how the theorems apply to deep learning, even if in a weak sense, as near line 517 discussing other works. Related to deep learning, I think the paragraph starting near line 162 means to discuss stability of *gradient descent* (and variants), since that is the algorithm, while the neural network parameterizes the hypothesis class.\n \nOne concern I have is that Theorems 3.1 and 3.2 are not sufficiently general, unlike Theorem 4.3. I'm not even sure what the applications of these theorems are.\n \nIt is strange that the authors couldn't think of any limitations of their work; I would urge the authors to add a paragraph on the same. Given the slow progress in this research area in recent areas, I think adding some future directions, which I am sure are plenty, would be helpful for the community.\n\nQUESTIONS:\nCan you give some intuition around Remark 1.3? It seems to make a seemingly strong claim without any justification.\n\n**Suggestions:**\n \nThe paper is reasonably self-contained, but the ordering of sections needs to be improved. Particularly, the notation in Section 2 needs to come before Section 1.1, else it becomes very hard to comprehend there onwards. Section 1.5 on related works also needs to be moved up, maybe right after Section 1.1. I am also not sure why the informal and formal versions of the theorems are both included in the main text &ndash; with Section 2 moved up, it should be sufficient to just keep the formal forms of the theorems (perhaps with informal descriptions alongside).\n \nThe abstract should include what is meant by stability of algorithms, because without that, it is hard to understand what the results are about &ndash; something like *loss-instability* would be good. The abstract also mentions that the paper is concluded with a discussion relating the existence of tight bounds with variance of algorithm's loss. That's not the case for this submission &ndash; the relevant discussion is at the end of Section 1.3. Kindly amend the abstract to accurately indicate the location of this discussion.\n \nFact C.2, as in the main text, is not at all clear, particularly what the variance is with respect to. Reading the supplementary material, my understanding is that the sampling process is $\\mathcal{D}\\sim\\mathbb{D} \\rightarrow S\\sim \\mathcal{D}^m$, and the expectation is w.r.t. marginal over $S$ and variance w.r.t. $\\mathcal{D}|S$; correct me if I'm wrong. Please clarify that.\n \nI think there are some inaccuracies in Example 1.4. Specifically, $T\\sim U(\\mathcal{X})^{m^2}$ implies sampling with replacement, as does Claim K.2, which uses a similar notation (also the reference to Birthday Paradox). However, the proof of Claim K.2 seems to assume a set of $k$ ($=m^2$ in Example 1.4) *distinct* points, i.e. sampling without replacement. This can be easily fixed by considering sampling $T$ without replacement, and fixing the notations; I don't see the example breaking down with this change. In Example 1.4, there is also a mention of $L_{\\mathcal{D}}(A(S)) = \\mathcal{D}_{\\mathcal{X}}(\\{x_1, \\ldots, x_m\\})$, which is the case under $0/1$ loss, as is also assumed in some other places, but is not mentioned anywhere before Definition 2.3.\n \nIn Example 1.6, I don't think \"$A$ always has $0$ empirical loss\", because the definition of the outputted hypothesis seems to imply that for inputs with *unique labels*, we have the hypothesis memorizing the label, but for those with non-unique labels, i.e. $(x,-1), (x,+1) \\in S$, the hypothesis predicts $-1$. For such inputs, we will have non-zero empirical risk. I don't see any distributional assumptions precluding such training sets.\n \n**Typos:**\n \nIn Fact 1.2, my understanding is that if $(A, \\mathbb{D})$ is $(\\epsilon, \\ldots)$-estimable, then the generalization bound is $2\\epsilon$-tight, *not* $\\epsilon$-tight.\n \nIn Example 1.6, first condition can omit \"$\\exists y:$\" part since that is understood.\n \nIn Example 1.7, the description of the algorithm is mentioned twice consecutively: 1. \"consider a mapping $A$ chosen uniformly from the set $\\mathcal{A}$\", and 2. \"$A(S)$ is a function that was chosen uniformly from $\\mathcal{F}$\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the conditions under which a learning algorithm A admits tight generalization bounds. It introduces the concept of \"estimability\": a learning algorithm A is estimable if there exists an estimator E such that, with high probability, \n|E(S) - L_D(A(S))| ≤ epsilon.\n\nThe paper provides several negative results showing when tight bounds cannot exist. Theorems 3.1 and 3.2 show that algorithms with \"inductive biases\" toward subsets of VC classes with the number of samples smaller than sqrt(d)/10 or toward nearly-orthogonal functions do not admit tight generalization guarantees. \n\nOn the positive side, the authors show that stable algorithms are estimable (Theorem 4.3), providing a sufficient condition for when tight bounds can exist. They also provide a simple characterization linking estimability directly to the conditional variance of the algorithm's loss, stating that an algorithm is estimable if and only if E[Var(L_D(A(S))|S)] ≤epsilon\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is actually about empirical generalization bounds, i.e., ones that can be evaluated knowing only the training data rather than relying on unknown constants/functionals of the data distribution. This makes the results more practically relevant. As a result, it seems overparameterization will rely on distributional properties. One question is that the celebrated result of Bartlett et al already shows that one needs to have a control on covariance matrix of the data distribution in order to reason about benign overfitting. I think it is indeed connected to this paper. Indeed, the assumption of loss stability is a distributional assumption. One question is how does the sufficient condition compare to other parts of the paper? Is that true that verifying E[Var(L_D(A(S))|S)] ≤ epsilon requires the distributional knowledge?\n\nOne section that is missing is open problems. I think posing questions like the required number of samples to estimate the generalization gap, whether the gap between the two negative results can be closed, or whether distribution-free sufficient conditions exist would strengthen the paper.\n\nAnother line of related work is testable learning. In this line of work, the goal is to bridge the gap between learning on particular distribution and worst-case perspective in PAC learning. It would be nice to include a discussion on this line of work.\nGollakota, Aravind, Adam R. Klivans, and Pravesh K. Kothari. \"A moment-matching approach to testable learning and a new characterization of rademacher complexity.\" Proceedings of the 55th Annual ACM Symposium on Theory of Computing. 2023.\n\nIn general, I think this paper is interesting and has some important findings.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is concerned with deriving *tight* generalization bounds for learning algorithms that are *non-uniform* w.r.t. to distribution — that is, the authors study for which pairs of algorithms and sets of distributions, the algorithm has a tight generalization bound (on the specified distributions). The core studied property to achieve this goal is that of *estimability* of an algorithm $\\mathcal{A}$, first defined by (Gastpar et al; 2024), that roughly means that there exists an $\\varepsilon$-accurate $\\delta$-confident estimator for the population loss achieved by $\\mathcal{A}$, using $m$-samples from the distribution belonging to the specified set of distributions. A simple immediate consequence of this definition is that all *estimable* algorithms that satisfy some generalization bound will satisfy it in $\\varepsilon$-tight manner with $\\delta$ confidence.\n\nThe main contribution that sets this paper apart from existing work is that it estabilshes connections between *estimability* and standard concepts in statistical learning theory, such as the variance of estimator, loss-stability, VC dimension. They also provide several simple examples that delve into what is estimability and how it is very different from learnability. The paper is a very interesting read and will be of interest to the community. I do have some remarks regarding, what is in my opinion, an unusual organization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper presents very interesting results, which significantly extend the previous results on estimability, the proofs in the appendices appear to be correct from a quick read through. \n\nWeaknesses:\nIn my subjective opinion, the presentation of the paper would benefit from a better organization of the text. Here are some of my observations that I hope might be helpful:\n\nAt the moment the majority of the paper is an introduction giving informal results (esp Sec 1.3 and 1.4), which takes 7 pages. On one hand, this makes the results easier to understand, but on the other hand, makes the reader always searching in the appendices and Sec 2 and 3 for concrete details. Most of the actual theory results are presented on 1.5 pages at the end (Sec 2 and 3). The actual Theorem 3.1 and 3.2 are not too difficult to understand in the full statement — It might be worth a though of trying to shorten the informal Sec 1 significantly and instead present more important details in Sec 2 and 3. For example, I find the presentation of “Fact C.2” very brief and I think it would deserve more explanation in the main text.\n\nI would also very strongly suggest to present “Facts” instead as “Theorems”, “Lemmas”, “Propositions”, “Corollaries”, or “Remarks” — I know some of these are simple consequences of definitions but still. Another remark I have is that there is actually quite a bit of important detail in the appendix that is not mentioned in the paper at all, for example the comparison with Gastpar (2024) et al. or the numerical experiments. There is also no concluding discussion, which might be due to lack of space, but would be very useful to have in this kind of paper to have some hints about interesting future directions etc.\n\nOverall, I really like this paper and found it a fascinating read, but the organization feels rushed. I trust the authors to know the best way to improve it (e.g. the decision about the length of informal/formal presentation of the main results)\n\nQUESTIONS:\n* Can you compare your approach to proving non-uniform generalization bounds directly? I assume this would not yield tightness, but possible could lead to new smaller bounds (that might or might not be tight).\n* In Line 85 you state: “And if the number of samples is too small to allow learning, some\ninteresting algorithms might nonetheless perform well on a certain subset of distributions and also be very estimable”. Is this technically correct statement? I thought estimability is equal to having tight bounds, shouldn’t then the sentence state that “\\ldots some interesting algorithms might nonetheless perform well on a certain subset of distributions because they are very estimable on that distribution subset.“ \n* In Line 124, I would suggest to remove the tilde as a sign of approximation and specify the number $0.16$.\n* In Line 144 you say “necessarily very weak for many distributions”. Could you please specify in what sense weak? I assume it means that no bound can be better than $1/4$-tight, correct?\n* In Line 158 you state that: ”Seeing as there are many definitions of stability in the literature, Theorem 4.3 makes a nontrivial conceptual contribution by identifying the “correct” notion of stability for under-standing estimability”. Without pointing to a later argument made in Sec 4 (which indirectly mentions the memorization Example 1.6), this reads as very strong statement. I think it would be good to back this statement here a bit more, or perhaps leave it for later.\n* In Line 216: “Hence, by Markov’s inequality, 99% of learning rules ...“ where is the 99\\% coming from? Is this in the theorem?\n* l321: “using a similiar technique” — You mean that if you use similiar technique as in Theorem 3.1 you will get 1/6 instead of 0.16? If you want to keep this statement, I would suggest specifying what is the technique based on (or perhaps just apply it and state Theorem 3.2 with the better result).\n* Definition 4.1 -> What is the notation of $S_1\\circ S_2$? I cannot find it introduced anywhere, is this the union of the sets?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}