Title: Robustly Learning Monotone Single-Index Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: agnostic learning

PAC Learning

single-index models
Abstract: We consider the basic problem of learning Single-Index Models 
with respect to the square loss under the Gaussian distribution 
in the presence of adversarial label 
noise. Our main contribution is the first computationally 
efficient  algorithm for this learning task, achieving a constant 
factor approximation,
that succeeds for the class of {\em all} monotone activations with bounded moment of order $2 + \zeta,$ for $\zeta > 0.$ This class in particular includes all monotone Lipschitz functions and even discontinuous functions like (possibly biased) halfspaces. 
Prior work for the case of unknown activation either does not attain constant factor approximation or succeeds for a substantially smaller family of activations. The main conceptual novelty of our approach lies in developing an optimization framework that steps outside the boundaries of usual gradient methods and instead identifies a useful vector field to guide the algorithm updates by directly leveraging the problem structure, properties of Gaussian spaces, and regularity of monotone functions.


Peer reviews:
comments: SUMMARY:
The paper makes progress for the problem on agnostically learning monotone single index model.

STRENGTHS AND WEAKNESSES:
The paper shows technical competency in improving on a natural problem.
However, both in terms of the proofs and in terms of the results there is a concern about how incremental the result is.

Obtaining a *constant-factor* approximation for learning *monotone* single-index model for an *unknown* activation was indeed unknown before. But removing any of the highlighted conditions either makes the results invalid or known. So the progress is for a somewhat narrow problem. Also the case where the activation has slightly stronger assumption is studied in ZWDD24.

QUESTIONS:
I feel like some ingredients in the new algorithm including the alignment part, the spectral subroutine and the approximation part have all been used for related problems. Can the authors comment on such related work or (boldly) claim these are novel.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of learning generalized linear models with Gaussian inputs $x$ where both the parameters and the activation/link function are unknown. The parameters are assumed to be bounded and $\\sigma$ is assumed to belong to the class of monotone, Lipschitz functions (or monotone functions with bounded $2 + \\delta$-moment). The goal is to jointly learn both the activation and parameters. The core contribution of this is the development and analysis of an algorithm for learning $(\\sigma, w)$. The algorithm is based on the spectrum of a matrix constructed from the data whose eigenvectors are supposed to strongly correlate with the optimal parameters $w^\\star$. The algorithm is shown to achieve a constant factor approximation of the optimum from a polynomial number of samples (with respect to the input dimension).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe main contribution of this work is the core construction of the update direction. It involves two non-trivial steps: posing the choice of update direction as a \"correlation maximization problem\" and estimating the solution of this problem from the spectrum of a matrix that can be constructed from data.\n\n\n## Weaknesses\n\nThe manuscript is difficult to read and at times informal or unclear. This makes it very difficult to parse the exact contributions of the work and assess their correctness. For instance, the main text contains several instances of notation that are either used before being defined or not defined at all. For instance, Section 1.1 refers to $T_{\\rho}$, $x^{\\perp w}$, and $\\gtrsim$ only defined later in Sec 1.2. This makes the section (which is almost 1/3 of the paper) almost impossible before having read through the full paper (with appendices). These issues are compounded with the continuous use of inline math, including to introduce proof sketches (e.g., line 190-197, which do not even point to a specific result later in the paper). This makes the section virtually unreadable. While I understands that conference papers have space constraints, I do not believe it is an excuse to jeopardize clarity.\n\nThe manuscript (and appendices) also repeatedly refer to results in the literature without explicit references, which makes them impossible to assess. This is particularly critical with respect to [ZWDD25] (which contains a plethora of results), e.g., \n    - Line 122-125 : \"Even though it is possible to control the L2 distance between $u(z)$ and $\\sigma(z)$ by $\\theta(w, w^{\\star})$ following **similar steps as in [ZWDD24]**, it is unclear how to show that $T_{cos \\theta}u'(z)$ is close to $T_{cos \\theta}$ \\sigma'(z)$ so that [...] and **the arguments of [ZWDD25] can go through**.\" (which steps/arguments?)\n    - Line 162 : \"and we **know from [ZWDD25]** that ...\" (specific proposition? theorem? proof?)\n    - Line 258 : \"and transform the regression problem to a robust halfspace learning problem, following **the same procedure** as in [ZWDD25]\" (which procedure?)\nIt is difficult to understand these references without knowing the specific results the manuscript is aluding to.\n\nThe comparisons to other works are also not very clear. For instance, this work extends [ZWDD25], which also assumes Gaussian inputs, but only mentions [GGKS23] which works with similar assumptions on the activation function, but *no distributional assumptions* (while labels are confined to [0,1], the labels in the current manuscript are also bounded given the that f is uniformly bounded). It is not straightforward, from the discussions in the paper, to see how the results are considerably weaker in this setting, seen as the sample complexity look comparable and the proportionality constants of the optimality factor are not clearly described. While [GGKS23] does depend on $W$, the radius of the space, it is not clear that this is worst than the constants for Algorithms 1-4. Particularly, since the bound on $\\|w\\|$ in [GGKS23] is replaced here by a uniform bound $B$ on the activation/link function (I believe that is in fact part of the argument in Remark 2.2 leveraging the homogeneity of $\\sigma$). Again, it is hard to adjudicate these matters from the presentations in the manuscript.\n\nOn a minor point, it is not clear to which extent the labels are per se \"adversarial.\" Though the agnostic learning setting can be seen as incorporating a form of \"adversarial label noise\" (as in, e.g., [DKMR22]), the setting of this paper is not fully agnostic since $x$ is assumed to be Gaussian. I imagine that what is meant by adversarial label noise is the fact that no assumptions are made on the conditional $y|x$, but this point should be explicitly addressed in the manuscript.\n\nWhile the paper might have relevant contributions, it requires considerable improvement in its presentation for the novelty and correctness of the contributions to be evaluated. I therefore cannot recommend the manuscript be published in its current form.\n\nQUESTIONS:\n1. Line 126 : \"Thus, to ensure $\\|T_{cos \\theta} u' - T_{cos \\theta} \\sigma'\\|$  small, it necessitates approximating the low degree coefficients of $\\sigma'$ (under adversarial noise and without the knowledge of $\\sigma$), imposing formidable technical challenges.\" Could the authors elaborate on these challenges?\n2. On line 174, the author argues that because the labels are adversarially corrupted, estimating $g^\\star_w(z)$ as $E[yx^{\\perp w} | w \\cdot x \\in (a,b)]$ is not possible. Could the authors elaborate on why this is the case?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides the first computationally efficient algorithm for robustly learning a Single-Index Model (SIM) of the form $f(x) = \\sigma(w \\cdot x)$ when the activation function $\\sigma$ is unknown but belongs to a broad class of monotone functions, the input distribution is Gaussian, and the labels are subject to adversarial noise. The algorithm achieves a constant-factor approximation to the optimal squared loss, resolving a significant open problem in the algorithmic theory of SIMs. The main technical innovation is a novel optimisation framework that moves beyond gradient-based methods, instead using a carefully constructed vector field, identified via spectral methods, to guide the search for the optimal parameter vector.\n\nSTRENGTHS AND WEAKNESSES:\n*Disclaimer: I am not an expert of the field and the proofs in the SM have not been checked.*\n\n# Strengths\n\n- It successfully provides the first polynomial-time, constant-factor approximation algorithm for agnostically learning SIMs with a very general class of monotone activations. This class includes all monotone Lipschitz functions and discontinuous ones, which covers many cases of practical and theoretical interest.\n\n- The main technical idea is novel. The authors' departure from standard gradient-based optimisation and designed a custom moment matrix $M_w$ that can be analysed with spectral methods to guide the gradient.\n\n- For a paper with high technical density, the exposition is remarkably clear.\n\n# Weaknesses\n\n- The entire analysis is contingent on the assumption that the input data $x$ follows a standard multivariate Gaussian distribution. This is a strong limitation, as the analytical tools used (e.g., properties of the Ornstein-Uhlenbeck semigroup, Gaussian integration by parts) are highly specific to this setting. The authors rightly note that generalising beyond Gaussian marginals is a major open question, even for simpler models. As it stands, the algorithm's direct applicability to non-Gaussian, real-world data is limited.\n\n- While the algorithm is proven to be \"efficient\" in the theoretical sense (i.e., runs in polynomial time), its complexity appears prohibitive for practical use. The sample complexity is $N=d^{2}poly(B,L,1/\\epsilon)$, and the overall algorithm involves multiple nested subroutines, including an initialisation procedure that grids over a parameter space and a main loop that performs a random walk, requiring the core routine to be repeated many times to ensure success. \n\n- The work is entirely theoretical and lacks any experimental validation. This absence prevents an assessment of the algorithm's finite-sample performance, the practical magnitude of the constants hidden in the theoretical bounds, and its robustness to even minor deviations from the strict Gaussian assumption.\n\nQUESTIONS:\n- Your spectral method relies on constructing a moment matrix $M_w$. To what extent are the key properties required for your analysis (specifically, the spectral gap between the top eigenvalue and the rest of the spectrum) tied to the Gaussianity of the input distribution? Could this spectral alignment framework be extended to other distributions, such as isotropic log-concave distributions?\n\n- The algorithm's final stage relies on a random walk to overcome the sign ambiguity of the identified eigenvector, requiring $poly(1/\\epsilon, B, L)$ repetitions to guarantee success with high probability. Could you comment on the practical magnitude of this number of repetitions? Furthermore, is there a potential path to \"de-randomise\" this component?\n\n- What are the primary obstacles to extending this spectral alignment approach to robustly learning a multi-index model? Would the top eigenvectors of a similarly constructed moment matrix still capture the subspace of interest, or would the interactions between different index directions fundamentally complicate the spectral picture?\n\n- Given the work is purely theoretical, what specific insights do you believe an empirical study would provide? For example, how might the constants hidden in your asymptotic analysis behave in a finite-sample setting, and what is your hypothesis on the algorithm's robustness to minor deviations from the Gaussian data assumption?\n\nLIMITATIONS:\nThe primary limitations of this work are threefold. First, its theoretical guarantees are confined to the standard Gaussian input distribution, which restricts its direct applicability. Second, the algorithm's high polynomial complexity and multi-stage, randomised nature raise questions about its practical feasibility despite its theoretical efficiency. Finally, the work is entirely theoretical, with no empirical validation provided in either the main paper or the supplementary material. The absence of experiments means there is no insight into the algorithm's finite-sample performance, the practical values of the constants in the bounds, or its robustness to even minor deviations from the strict distributional assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}