Title: Tight Generalization Bounds for Large-Margin Halfspaces
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learning theory

Generalization Bounds

Large-Margin Halfspaces

Rademacher complexity
Abstract: We prove the first generalization bound for large-margin halfspaces that is asymptotically tight in the tradeoff between the margin, the fraction of training points with the given margin, the failure probability and the number of training points.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of learning halfspaces with large-margin and provides tight generalization bounds for this problem. More precisely, assume that we have an unknown distribution $\\mathcal{D}$ over $B^d\\times\\{0,1\\}$ where $B_d$ is the $d$-dimensional unit-ball, and assume that we have a sample $S$ of $n$ i.i.d. samples $\\mathcal{S}:=(x_i,y_i)_{1\\leq i\\leq n}\\sim\\mathcal{D}^n$. We are interested in relating the true loss $\\mathcal{L}_D(w)$ for a unit vector $w\\in\\mathcal{D}$ (which is defined as the probability under $\\mathcal{D}$ that $\\mathrm{sign}(\\langle w,x\\rangle)\\neq y$) to the $\\gamma$-margin empirical loss, which is defined as the fraction of samples $(x,y)\\in S$ satisfying $\\langle w,x\\rangle\\leq\\gamma$.\n\nThere are previously known results providing lower bound and upper bounds of $\\mathcal{L}_D(w)$ in terms of $\\mathcal{L}_S^\\gamma(w)$. While the previous upper and lower bounds are very close and almost match, there were still a gap between the two. The paper provides a tighter upper bound which asymptotically matches the previously known lower bound, and hence it is (asymptotically) tight.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper provides an asymptotically tight upper bound of the generalization loss on terms of the margin empirical loss. This closes the gap between the upper and lower bounds and settles the problem of characterizing the the generalization loss in this setting.\n- In terms of novelty and originality, I find the idea of writing the difference between the loss as an exact sum of difference terms (including Equations (12) and (13)) particularly elegant. This ultimately enabled the authors to perform a very tight analysis of these difference leveraging the powerful machinery of Rademacher complexity.\n- In terms of technical correctness, while I cannot 100% vet that all the details of the proof are correct as I have not read the appendices, I did not find major mistakes in the high-level description of the proof in the main body of the paper. Though I have not thoroughly checked all the math, the general proof strategy makes sense to me.\n- In terms of clarity and presentation, I find that the structure of how the high-level ideas are presented to be generally done well: I like how the authors motivated their work by starting with a high-level description of the proof of the tightest known upper bounds. The authors described the barriers for such approach to achieve an upper bound matching the lower bound and then explained their strategy (to overcome these barriers in order) to get a tighter upper bound matching the lower bound.\n\nWeaknesses:\n- The paper sometimes use notations without properly defining/introducing them. The reader can generally understand the intended meaning from the context in most cases, but this is not always the case. Carefully defining the non-standard notations would definitely improve the readability of the paper. Please refer to the questions below.\n- Even though the overall clarity is good, the math writing can be made slightly clearer and more formal in some cases.\n\nQUESTIONS:\n- Page 1, line 17: What is $\\mathbb{B}_2^d$? Is it the hypercube $\\{0,1\\}^d$ or is it a $d$-dimension ball?\n> - If it is the hypercube, is this restriction important?\n> - From the discussions in the text it seems to me that most likely it is a $d$-dimensional ball, but then what does the 2 subscript represent? Is it the radius?\nPlease explain and properly define the non-standard notations in the paper.\n\n- Page 6, line 195: What is $\\mathcal{X}$? I might have missed the definition but it doesn't seem to me that it was introduced anywhere. From the discussions in the paper, it seems to me that it is the set of all unit vectors satisfying $\\langle w,x\\rangle \\leq c_\\gamma$. Is this correct? If so, please explicitly say that.\n\n- Page 7, line 224: Claims 2: I think that \"with probability at least $1-\\delta$\" should be added to the claim, right?\n\n- Page 8, line 243: Shouldn't \"$p(z_i)$\" rather depend on $(Aw)_i$? Perhaps it should be written $p((Aw)_i)$ to be more formal?\n\n- Page 8, line 254: (Minor comment:) The formatting/rendering of equation (19) is a bit weird.\n\nLIMITATIONS:\nSince this is a theoretical paper, I do not see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new, asymptotically tight generalization bound for large-margin halfspace classifiers. The primary contribution is a novel upper bound (Theorem 2) that characterizes the trade-off between the true error of a classifier and its empirical margin loss on a training set. This new bound matches the existing state-of-the-art lower bound. \n\nThe work builds upon the previous state-of-the-art bound by Grønlund et al. [2020], which had a small gap compared to the established lower bound. The authors identify and overcome key barriers in the prior proof technique, which relied on random discretization and a framework from Schapire et al. [1998].\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper proves the first generalization bound for large-margin halfspaces that is asymptotically tight. This result is significant as it settles the precise generalization performance of a classic and fundamental learning model. \n2. The proof introduces several novel ideas that could be applicable to other generalization bounds. These include a more refined analysis of error terms that avoids inequalities used in prior work, a new application of Rademacher complexity to bound these refined terms, and a method using an infinite sequence of discretization grids to handle error probabilities more effectively. \n3. The paper provides a clear and detailed proof overview to build intuition before presenting the full proof. The main theorem is rigorously proven with all assumptions stated, and complex arguments are broken down into smaller, more manageable lemmas with deferred proofs in the appendix for clarity. \n\nWeakness:\n\nN/A\n\nQUESTIONS:\n1. The proof relies on a sophisticated random discretization process involving a Johnson-Lindenstrauss transform and random snapping to a grid. While this is used as a proof technique, does it offer any algorithmic insights? Could a learning algorithm based on this discretization be practical or offer any advantages?\n2. \"While one might argue that our improvement is small in magnitude, this finally pins down the exact generalization performance of a classic learning model\". Could you elaborate on the importance of closing this final gap? Does it resolve any long-standing theoretical questions or open up new avenues of inquiry that were previously blocked by the discrepancy in the bounds?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves the first generalization bound for large-margin halfspaces that is asymptotically tight in the tradeoff between margin, fraction of training points with the given margin, failure probability, and number of training points. The authors improve upon previous work by Grønlund et al. (2020) by closing the gap between upper and lower bounds. The improvement in tightness pins down the exact generalization performance for the classical halfspace classifier model. The novel techniques used in this paper may also inspire future theoretical analysis on the generalization bounds of other ML models.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n1. **Clear paper organization**: The proof is thorough and well-structured, with clear organization into lemmas and auxiliary results. The authors carefully handle all technical details.\n2.  **Well Justification**: Despite the technical complexity, the paper provides good intuition through a detailed proof overview (Section 2) that explains the key ideas and improvements over previous approaches.\n3. **Novel technical contributions**: The proof introduces several innovative techniques that collectively enable the tight bound. First, the authors develop a more refined analysis that tracks both positive and negative changes in margins during random discretization, moving beyond previous one-sided approaches that only considered margin violations in one direction. Second, they employ a sophisticated application of Rademacher complexity theory with carefully bounded Lipschitz constants, achieving sharper control over the generalization gap through an improved use of the contraction principle. Third, to handle the challenge of varying norms in the discretized hypotheses, they introduce an infinite sequence of grids with an intricate union bound argument.\n## Weaknesses:\n1.  **Lack of a formal conclusion section**: The paper ends after completing the main proof without a dedicated conclusion section that summarizes the contributions, discusses the broader implications of achieving tight bounds, or suggests future research directions. This is a minor structural issue that could be easily addressed by adding a brief concluding section.\n2.  **Lack of a related work section**: The paper lacks a related work section that positions the contribution within the broader landscape of generalization theory and margin-based learning. However, this is not a major concern, as the clear comparison with Grønlund et al. (2020) in Section 1 has already highlighted the novelty of the contribution, and this issue can be easily fixed by adding a related work section.\n\nQUESTIONS:\n## Major questions\n1. **Future direction**: In lines 65-66 on page 3, the authors claim that “Furthermore, our proof of Theorem 2 brings several novel ideas that we hope may find further applications in generalization bounds”. What are the possible future research directions of this work? For example, could the tighter bound be applied to improve other theoretical results?\n2. **Extension of the proposed method**: Do the techniques developed here have potential applications to other learning models beyond halfspaces?\n\n## Minor Issues\nHere are some minor punctuation problems and typos:\n1. Eq. (5) requires a period (“.”) at the end.\n2. Eq. (17) requires a period (“.”) at the end.\n3. All the terms in Eq. (19) are right-aligned, which is inconsistent with the formatting of other equations in the paper. I could be wrong, but it might be clearer if the second and third terms on the right-hand side of Eq. (19) could be shortened by introducing intermediate notations, allowing for a more consistent left-aligned format. \n4. There is an additional space before line 255 on page 8.\n5. It would be better if the equation between lines 284 and 285 on page 9 could be left-aligned.\n\n## Missing citations related to SVM \n[1] Yuzhou Gu, Zhao Song, Lichen Zhang. “Faster Algorithms for Structured Linear and Kernel Support Vector Machines”. ICLR 2025.\n[2] Vladimir Vapnik, Rauf Izmailov.  “Learning using privileged information: similarity control and knowledge transfer”. JMLR 2015.\n## Missing citations related to Kernel method\n[3] Timothy Chu, Josh Alman, Gary L. Miller, Shyam Narayanan, Mark Sellke, Zhao Song. “Metric Transforms and Low Rank Representations of Kernels for Fast Attention”. NeurIPS 2024.\n[4] Josh Alman, Timothy Chu, Aaron Schild, Zhao Song. “Algorithms and Hardness for Linear Algebra on Geometric Graphs”. FOCS 2020.\n[5] Corinna Cortes, Mehryar Mohri, Afshin Rostamizadeh. “Generalization Bounds for Learning Kernels”. ICML 2010. \n[6] Jan van den Brand, Binghui Peng, Zhao Song, Omri Weinstein. “Training (Overparametrized) Neural Networks in Near-Linear Time”. ITCS 2021.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the risk of large-margin halfspaces in a standard statistical learning setting.\nThe authors prove a novel bound on the generalization error $L\\_D(w) - L\\_S^{\\\\gamma}(w)$ for any $w$ in the unit $d$-dimensional sphere, where $L\\_D(w)$ is the true risk of $w$, which exactly matches the best known lower bound for every margin $\\\\gamma$, empirical $\\\\gamma$-margin loss $L\\_S^{\\\\gamma}(w)$, sample size $n$, and confidence $\\\\delta$.\nBy doing so, they close the residual gap in the generalization error upper and lower bounds from previous work.\n\nSTRENGTHS AND WEAKNESSES:\nThe main result of this paper is pretty clear: it pins down the generalization error of large-margin halfspaces up to a constant factor.\nThis optimality is achieved through a careful random discretization that first projects with Johnson-Lindenstrauss and then rounds so that the distribution of the rounded margin equals the original one, allowing a cancellation between terms that previous approaches could not exploit.\nIn their \"meet in the middle\" bound, the authors even employ a clever multi-scale chaining argument over a countable sequence of grids that removes another residual $\\\\exp(-k/c)$ factor, thus allowing to select the optimal projection dimension without compromise.\nThe combination of various different techniques in such a way shows a deep understanding of the problem and a solid technical advancement.\n\nA minor flaw of the paper is that it often omits some central definitions as they are given for granted, which makes it hard to follow the paper for readers not familiar with the topic.\nJust to give one example, the authors never define $\\\\mathbb{B}^d_2$ nor $\\\\mathcal{S}^{d-1}$, which one may assume to be the unit ball and sphere in $\\\\mathbb{R}^d$, respectively.\nI suggest the authors to clearly define all the adopted notations like these ones in the introduction or in some preliminary section, mainly to avoid any ambiguity.\nHowever, this is not a major issue as the paper is otherwise well-written.\nIn particular, the proof of the novel generalization error upper bound has a clear structure and can be followed despite its technical nature.\n\nQUESTIONS:\n- Do the authors believe some of the techniques adopted can be easily extended to other settings, or are they particularly tailored to the geometric nature of the large-margin halfspaces?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-08"}