Title: Robust learning of halfspaces under log-concave marginals
Venue: NeurIPS 2025
Accepted: yes
Keywords: adversarial robustness

linear classifiers

agnostic learning
Abstract: We say that a classifier is $\text{\emph{adversarially robust}}$ to perturbations of norm $r$ if,
with high probability over a point $x$ drawn from the input distribution, there is no point within distance $\le r$ from $x$ that is classified differently. 
The $\text{\emph{boundary volume}}$ is the probability that a point falls within distance $r$ of a point with a different label.
This work studies the task of learning a hypothesis with small boundary volume, where the input is distributed as a subgaussian isotropic log-concave distribution over $\mathbb{R}^d$.

Linear threshold functions are adversarially robust; they have boundary volume proportional to $r$.
Such concept classes are efficiently learnable by polynomial
regression, which produces a polynomial threshold function (PTF), but PTFs in general may have boundary volume $\Omega(1)$, even for $r \ll 1$.

We give an algorithm that agnostically learns linear threshold functions and returns a classfier with boundary volume $O(r+\varepsilon)$ at
radius of perturbation $r$.
The time and sample complexity of $d^{\tilde{O}(1/\varepsilon^2)}$ matches the 
complexity of polynomial regression.

Our algorithm augments the classic approach of polynomial regression with three additional steps:\
$\quad$ a) performing the $\ell_1$-error regression under $\ell_1$ noise sensitivity constraints,\
$\quad$ b) a structured partitioning and rounding step that returns a Boolean classifier with error $\mathrm{opt} + O(\varepsilon)$ and noise sensitivity $O(r+\varepsilon)$ simultaneously, and \
$\quad c)$ a local corrector that ``smooths'' a function with low noise sensitivity into a function that is adversarially robust.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel and efficient algorithm for agnostically learning linear separators (halfspaces) under the assumption that the data's marginal distribution is isotropic and log-concave. The learned hypothesis is constructed from a Polynomial Threshold Function (PTF).\n\nThe paper makes two primary contributions:\n1.  **Adversarial Robustness:** The algorithm produces a classifier that is provably robust against adversarial perturbations. This is formalized by guaranteeing a small \"boundary volume\" ($O(r+\\epsilon)$ for a perturbation radius $r$), meaning that for most inputs, small perturbations do not cause a misclassification. This robustness is achieved while matching the optimal classification error ($opt+O(\\epsilon)$) and the state-of-the-art sample complexity ($d^{\\tilde{O}(1/\\epsilon^2)}$) of the best *non-robust* learning algorithms for this problem.\n2.  **Verifiable Robustness:** Under the standard complexity assumption $P=BPP$, the paper provides an efficient, deterministic verifier. This allows a user to certify that any specific input point is robustly classified by the learned hypothesis, addressing the practical problem of trusting models from untrusted sources.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nThis paper presents a theoretical advance in robust learning, with clean technical insights and outstanding clarity.\n\n- **Significance and Originality:** The paper addresses a core challenge in robustness: designing efficient learners that are provably robust. It gives the first algorithm to efficiently and robustly learn halfspaces in the agnostic setting under general log-concave distributions, matching the error guarantees and sample complexities of non-robust learners. Establishing _verifiable robustness_ is practically important for developing certifiably safe machine learning systems.\n    \n- **Technical Quality:** The technical contribution is deep, innovative, and very well-executed. The approach combines three key components, each introducing new ideas:\n    \n    1. Constrained Polynomial Regression: The learner fits a polynomial under convex constraints that enforce low noise sensitivity and isolation probability—critical for robustness.\n        \n    2. Structured Rounding and Partitioning: The real-valued polynomial is converted to a Boolean classifier using a randomized mixture (via Carathéodory’s theorem and LP) that preserves error and robustness. This mixture induces a partition with low boundary complexity.\n        \n    3. Local Correction: A final smoothing step turns a low-sensitivity classifier into one that is certifiably robust to adversarial perturbations.\n        \n    \n- **Clarity:** The paper is exceptionally well-written. The introduction clearly motivates the problem and outlines the solution, while the technical sections provide a detailed and accessible analysis.\n    \n---\n### Weaknesses\n\nThis is a very strong submission, but a few presentation details could be improved:\n\n- The role of proper learning in the introduction could be clarified. Line 37 frames proper learning as a robustness benchmark, but it is not immediately clear that the proposed algorithm is _improper_. Since efficient proper agnostic learning remains an open problem, it would help to state explicitly and early that the paper achieves halfspace-level robustness without giving a proper learning algorithm.\n    \n- The informal versions of the main theorems (1.1 and 1.4) should reflect that the guarantees are probabilistic. Phrasing the results as holding “with high probability” would better align with the formal statements (Theorems 3.1 and 3.4). Adding forward references would also improve navigability.\n\nQUESTIONS:\n1.  Regarding verifiable robustness and the $P=BPP$ assumption:\n    - The name for Corollary 1.5, \"Deterministic robustness,\" is a bit confusing. Since the robustness guarantee for the hypothesis still holds probabilistically over the data distribution, would \"Deterministic Verifiability\" be a more accurate term? The key deterministic part seems to be the verifier itself, which can check any given point without randomness.\n    - There appears to be a minor inconsistency in the presentation of Algorithm 6 (VERIFY). The input description on line 1 of the algorithm box mentions taking a point $x$ and an error tolerance $\\epsilon$, but the function signature at the top of the box is `VERIFY(g, r, eps, data)`. Could the authors clarify the intended signature and inputs?\n    - Are consecutive runs of the verifier on different points $x_1$ and $x_2$ independent? It seems they are in the $P=BPP$ setting, since the verifier is deterministic. Does this independence still hold for the randomized verifier mentioned in Remark D.4, or does the analysis change given that the hypothesis itself is constructed with randomness?\n\n2.  The assumption $\\epsilon \\ge d^{-1/7}$ in Theorem 3.1 is a key technical condition. Could the authors add a brief remark explaining where in the proof this is necessary (it seems to be for the concentration of random projections in Claim C.6) and whether it is likely a limitation of the current analysis or potentially fundamental to the partitioning technique?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of agnostically learning halfspaces. That is given any Boolean function f over a well behaved (subgaussian and logconcave in this case) distribution D one wants to learn a hypothesis h so that Pr_{x sim D}(h(x) neq f(x)) <= OPT +eps, where OPT is the error of the best possible halfspace approximator. This can be done in d^O~(1/eps^2) samples using the L1 regression algorithm, which returns a hypothesis of a low degree PTF. On the other hand while the best-approximating halfspace is robust against adversarial examples (in the sense that for a random input x the probability that there is a nearby input y with |x-y| < r, with h(x) neq h(y) is small (O(r)), the PTF hypothesis may well not satisfy this. This paper develops techniques to learn hypothesis that both achieve the agnostic learning error guarantee but also satisfy this kind of adversarial robustness.\n\nThe basic idea seems to be the following:\nThe L1 regression algorithm uses linear programming to find a degree k (for k some O~(1/eps^2)) polynomial minimizing the empirical L1 loss E[|p(x)-f(x)|]. We would like to also add a constraint along the lines of E_x[ max_{|y-x|<r} |p(x)-p(y)| ] = O(r), which would ensure the kind of robustness that we want. Unfortunately, the maximization cannot be efficiently solved here. Instead, they replace this with a proxy E_{z ~ N(0,r*I)}[|p(x)-p(x+z)|], which can be efficiently approximated via sampling. It is not hard to show (using standard constructions) that there are polynomials that get OPT+eps error while simultaneously keeping this notion of noise sensitivity small.\n\nBy thresholding p, one obtains a boolean function with appropriate error and small \"noise sensitivity\" (i.e. probability that h(x) neq h(x+z) where x ~ D and z ~ N(0,r*I)). From here they say that a point x is isolated if Pr_z(h(x) neq h(x+z)) is large. It is then not hard to show that if two nearby points have different values of f, then at least one of them must be isolated. It can then be shown that flipping the value of h at all isolated points gives rise to a function with appropriately good adversarial robustness assuming that there are not too many points with modestly high noise sensitivity. To ensure that this doesn't flip too many points, one needs to add another constraint to the convex program to ensure that not too many points are isolated, however this is doable without affecting satisfiability.\n\nOne final issue is that the thresholding for p must be randomized and *on average* the total error is at most OPT+eps and *on average* the adversarial robustness is O(r+eps), but there is no guarantee that these are satisfied simultaneously. Fortunately, one can find only a few different thresholds that if averaged achieve this and by piecing the resulting functions together can obtain something appropriate.\n\nEssentially all of the proofs were in the appendices so I did not read them in detail (NOTE TO THE PC: THIS IS A TERRIBLE SYSTEM THAT FORCES PEOPLE TO WRITE WORSE PAPERS THAT CANNOT BE CHECKED FOR CORRECTNESS ANYWAY WITHOUT READING THE EXTRA MATERIAL), but I did read enough to convince myself that the main results could be proved along the lines suggested.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* This seems like an interesting new problem direction\n* The paper introduces some non-trivial new technical ideas\n\nWeaknesses:\n* My biggest issue here is presentational. Many of the main results are introduced *before* the terminology necessary to make sense of them. This is especially bad as they use terms like \"noise sensitivity\" in non-standard ways. This makes it very hard to follow the introduction without reading it twice. The authors should put more thought into how to organize the introduction so that it is readable.\n* I also worry some about the more general applicability of these ideas. The notion of noise sensitivity that they introduce seems to work well for functions that essentially depend on only a few directions, but I worry that for even slightly more complicated functions it will fail badly as the distribution of x and x+z are very far from each other. For example the function 1{|x| > sqrt{d}(1+delta)} will have noise sensitivity almost 1 at almost all Gaussian inputs x, suggesting that this technique has little hope of success to generalize to degree-2 threshold functions.\n\nQUESTIONS:\nThe organization of the introduction should be rethought in order to not have to rely so much on the use of terms that have not yet been properly defined.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an efficient algorithm for adversarially robust learning of halfspaces under subgaussian isotropic log-concave distributions. The algorithm achieves both low classification error and small boundary volume, ensuring robustness to adversarial perturbations. The authors build on polynomial regression techniques, augmenting them with structured rounding and local correction steps to ensure both robustness and efficiency. The results match the best known runtimes for non-robust agnostic learning and provide verifiable robustness guarantees.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nClarity: The paper is exceptionally well-written and clearly structured. The technical content is explained with clarity, and the motivation behind each algorithmic step is easy to follow. The coverage of related work is thorough and positions the contribution well in the context of existing literature.\n\nSignificance and Originality: The combination of noise-sensitivity-aware learning, randomized partitioning, and local correction for achieving robust classification is a compelling and original approach. The contribution is clearly stated and supported by theoretical analysis.\n\nWeaknesses:\nOne point that could be elaborated is the transition from the standard worst-case adversarial perturbation model (expressed via boundary volume) to average-case robustness notions like noise sensitivity. The intuitive and technical connection between the two robustness notions could benefit from a more explicit discussion, particularly early in the paper (e.g., Section 1.2). Making this bridge more transparent would improve accessibility for readers less familiar with these distinct robustness perspectives.\n\nQUESTIONS:\n1. The work focuses on subgaussian isotropic log-concave distributions. It would be helpful to understand how critical these assumptions are. Have the authors considered whether the approach or guarantees extend to broader families of distributions? Even a brief discussion on the feasibility or limitations of such relaxations would add value.\n\n2. It would be interesting to see how the proposed techniques perform beyond the theoretical setting, e.g., on synthetic or real data with more complex structure. Experiments validating the theoretical guarantees or exploring tradeoffs in practice would strengthen the work.\n\n3. Another promising direction could be applying the proposed correction and smoothing techniques to existing robust models (e.g., adversarially trained networks). For instance, could the smoothing procedure help improve or measure robustness in practical deep learning models?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
A problem of learning half spaces is considered here. The requirements on the learning algorithm are twofold, first, the learned classifier has larger error than the best halfspace classifier by at most $\epsilon$, and second, the volume of the classifier r-boundary is  $O(r)$. The classifiers r-boundary volume is defined as the probability (over the data distribution) that there is for an input x another point y such that ||x-y|| < r and h(x) and h(y) differ.

First, relaxed learner produces $1/\epsilon^2$-degree polynomial thresholding hypotheses (somewhat uniformly) robust to random (not necessarily adversarial) perturbations via convex programming. Then, a local corrector is used; roughly speaking, it computes probability that a random perturbation changes decision of the classifier. If this probability is high, the label is flipped.

The derived sample complexity then almost matches the sample complexity of the non-robust learning up to polylog factors in the exponent (the sample complexity is exponential in both cases, thus the exponential polylog factor differences are "fine").

STRENGTHS AND WEAKNESSES:
The paper is very technical but it somewhat successfully tries to be accessible. The considered problem is pretty niche without direct applications in the practice of adversarial robustness (for instance, because of the exponential sample complexity), but it solves the considered problem within a reasonable completeness. The presented techniques seem to be often borrowed from other works, but I am not able to comment if the applications are inovative or straightforward. Some technical novelty is clearly present.

QUESTIONS:
-

LIMITATIONS:
-
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an efficient algorithm for learning halfspaces that are robust against adversarial perturbations. That is, the probability that a random example is within distance $r$ of its closest point on the other side of the halfspace (called “boundary volume”) scales linearly with $r$. The method is designed for data from subgaussian, isotropic, log-concave distributions. While standard methods like polynomial regression are efficient, they can yield non-robust polynomial threshold functions (PTFs), which also constitutes an improper learner. This new algorithm overcomes the former limitation, producing a classifier with a guaranteed boundary volume of $O(r+\\epsilon)$ and a misclassification error of opt+$O(\\epsilon)$, where opt denotes the optimal misclassification error. The time and sample complexity of $d^{\\tilde{O}(1/\\epsilon^2)}$ matches that of the state-of-the-art non-robust algorithms. Technical contributions of the paper include performing $\\ell_1$-error polynomial regression under noise sensitivity constraints, a randomized partitioning and rounding method designed to find a classifier that has 3 desired properties simultaneously and finally a local computation algorithm for correction. Specifically, the latter algorithm receives classification queries for a polynomial threshold function and answers according to a robust classifier that is not too far from it. Lastly, the paper introduces the concept of verifiable robustness, showing that under the assumption P=BPP, the output model's robustness can be efficiently and deterministically checked (or with some error probability without making the assumption).\n  Minor comment: \nPage 6: footnote 3 seems incomplete.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper introduces novel techniques to obtain a robust version of a fundamental problem in machine learning, without sacrificing sample complexity or computational efficiency. The authors are still using an improper learning algorithm, which is in line with prior work, while a proper learning algorithm would be preferable in general.\n\nQUESTIONS:\nIn theorem 1.1, your guarantees rely on the input distribution being subgaussian, isotropic, and log-concave. How sensitive do you expect the algorithm's performance to be if these assumptions are slightly violated? Have you considered extending these results to more general distributions, perhaps with weaker guarantees?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}