Title: Algorithms and SQ Lower Bounds for Robustly Learning Real-valued Multi-Index Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: learning theory

PAC learning

agnostic learning

multi-index models

Statistical Query model
Abstract: We study the complexity of learning real-valued Multi-Index Models (MIMs) under the Gaussian distribution. A $K$-MIM is a function $f:\mathbb{R}^d\to \mathbb{R}$ that depends only on the  projection of its input onto a $K$-dimensional subspace.  We give a general algorithm for PAC learning a broad class  of MIMs with respect to the square loss, even in the presence of adversarial label noise. Moreover, 
we establish a nearly matching Statistical Query (SQ) lower bound, providing evidence that the complexity of our 
algorithm is qualitatively optimal as a function of the dimension. Specifically, we consider the class of bounded variation MIMs with the property that degree at most $m$ distinguishing moments exist with respect to projections onto any subspace.  In the presence of adversarial label noise, the complexity of  our learning algorithm is $d^{O(m)}2^{\mathrm{poly}(K/\epsilon)}$.   For the realizable and independent noise settings,  our algorithm incurs complexity $d^{O(m)}2^{\mathrm{poly}(K)}(1/\epsilon)^{O(K)}$. To complement our upper bound, we show that if for some subspace degree-$m$ distinguishing moments do not exist, then any SQ learner for the corresponding class of MIMs  requires complexity $d^{\Omega(m)}$. As an application, we give  the first efficient learner for the class of positive-homogeneous 
$L$-Lipschitz $K$-MIMs. The resulting algorithm has complexity  $\mathrm{poly}(d) 2^{\mathrm{poly}(KL/\epsilon)}$. This gives a new PAC learning algorithm  for Lipschitz homogeneous ReLU networks with complexity   independent of the network size, removing the exponential dependence  incurred in prior work.


Peer reviews:
comments: SUMMARY:
The paper studies learning real-valued Multi-Index Models (MIMs) under Gaussian inputs, where the target depends on an unknown $K$-dimensional subspace. It gives a robust PAC regression algorithm (agnostic setting) that tolerates arbitrary label noise. Under mild regularity (“bounded variation” and “distinguishing low-degree moments”) it runs in time polynomial in $d$ (ambient dimension). A matching SQ lower bound showing any Statistical Query learner must incur roughly $d^{\Omega(m)}$ complexity when the low-degree moment condition fails. There is also a concrete application to positive-homogeneous Lipschitz MIMs (including certain ReLU networks).

STRENGTHS AND WEAKNESSES:
The paper is clearly written, and the mathematical arguments are rigorous and not hard to follow. The results are strong, providing decent results on learning MIM, and I recommend a clear acceptance provided the following concern is addressed.
I would point out one more related work (arXiv:2506.05500) that introduces a “generative leap” measure for multi-index models. It would greatly clarify the paper if the authors clarified how their parameter \$m\$ corresponds to this generative leap (which controls the overall learning complexity). 
I would be happy to raise my score further once that connection is made explicit.

QUESTIONS:
See above.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of learning real-valued multi-index models (MIMs) under Gaussian covariates in the presence of adversarial label noise. The paper proposes an iterative, subspace recovery based algorithm that PAC learns well-behaved MIMs. As a corollary, the authors derive the complexity of learning low rank Relu networks, improving existing results.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a technically solid paper. It provides a PAC learner with sample complexity $N$ that runs in $\\mathrm{poly}(N)$ time. A SQ lower bound is also given, which qualitatively matches some terms in $N$. So this is, to my knowledge, the first paper to give a partial characterization of efficient PAC learning for MIMs.\n\nMy main concern is that the paper doesnot do a great job of contextualizing the results. It says that “our understanding of the efficient learnability of MIMs is somewhat more limited,” but it is unclear what exactly this means. Are there known (possibly inefficient) PAC learners for MIMs? If so, how do the statistical and computational complexities of this paper’s algorithms compare to those?\n\n\n\nBased on Definition 1.3, it looks like the authors are working with a subclass of $C^1$. For that class, are there standard covering number-based bounds for agnostic PAC learning. How do those sample complexity bounds compare to the ones proved in this paper? A common issue with traditional sample complexity bounds is that they often rely on ERM over the class (or a discretization), which results in exponential computational complexity in $1/\\varepsilon$. But the algorithm from Theorem 1 already has exponential dependence on $1/\\varepsilon$, so it’s not clear whether this approach improves on ERM from a computational perspective. Can the authors comment on this?\n\nQUESTIONS:\n1. The moment condition in part (2) of Definition 1.3 feels quite abstract. Can the authors give a more intuitive explanation or interpretation of what this condition is actually capturing?\n\n2. Corollary 1.7 is said to improve over [CKM22] by removing the dependence on the network size $S$. But isn’t the Lipschitz constant $L$ effectively taking its place? If no prior bound on $L$ is assumed, and ReLU networks naturally have Lipschitz constants that scale with depth and width, doesn’t that reintroduce a kind of hidden dependence on $S$? Can the authors clarify whether $L$ can be controlled independently of $S$?\n\n3. Also, maybe I’m misunderstanding something, but the discussion around optimality of Theorem 1.4 focuses on the $d^m$ term, due to the SQ lower bound. But in practice, isn’t the $2^{\\mathrm{poly}(K/\\varepsilon\\sigma)}$ term a much bigger deal? For small enough $\\varepsilon$, this term should easily dominate $d^m$, no?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study the problem of PAC learning multi-index functions. They show if the multi-index function $f$ has \ndistinguishing moments of degree at most $m$, then there is an algorithm that uses at most $d^{O(m)}$ i.i.d. samples \nthat can learn this function (even in the presence of adversarial noises). As an application of this generic result, \nthey show that homogeneous Lipschitz ReLU networks can be learned using $O(d^2)$ samples. They also complement their upper \nbounds with a Statistical Query (SQ) lower bound which says that if degree $m$ distinguishing moments do not exist for \nsome subspaces, then any efficient algorithm will require a SQ oracle with tolerance smaller than $d^{-\\Omega(m)}$.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n\n* Overall, this is a well-written paper. The presentation is clear, and the proof sketches are easy-to-follow.\n* The authors provide an algorithm that learns general multi-index functions with polynomial samples (assuming $m = O(1)$) \n  under fairly general assumptions, and show the corresponding SQ lower bounds. These can potentially be used as baseline \n  results in later analysis of multi-index models with more structures. \n* The authors show that their complexity measure (the lowest degree of the distinguishing moments) is approximately \n  equivalent to the Generative Exponent in the single-index case. In some sense, this can be viewed as an evidence \n  that they are using the right/a reasonable measure. \n* The RNGCA framework introduced in the proof of the lower bounds explicitly encodes the label subspace, which not only \n  looks more natural for this type of learning tasks, but also makes the lower bounds applicable to the realizable/noiseless settings. \n\n### Weakness \n\n* The authors claim that Corollary 1.7 (Learning ReLU networks) is depth-independent. However, Corollary 1.7 requires the \n  target class to be $L$-Lipschitz and has a super-exponential dependence on $L$, and it is not clear how $L$ grows when \n  the network becomes deeper. \n* Some parts of this paper, such as Definition 1.3 and Algorithm 1 and 2, are extended/refined/continuous version of the \n  counterparts in [DIKZ25]. The authors should mention this more explicitly in the main text. \n\n\n### Typos\n* Line 45: should be $\\mathrm{err}_D(f)$.\n* Line 307: $\\| v_W \\|$. Do you mean something like $v_{W \\cap V^\\perp}$?\n\n[DIKZ25] Ilias Diakonikolas, Giannis Iakovidis, Daniel M. Kane, Nikos Zarifis. Robust Learning of Multi-index Models via Iterative Subspace Approximation. 2025.\n\nQUESTIONS:\n* Can you obtain bounds on the Lipschitz constant of ReLU networks (probably under some other conditions)?  \n  I am asking this because Corollary 1.7 depends on the Lipschitz constant of the network, which is expected to become \n  worse as the network becomes deeper. \n* Do your results cover [DIKZ25] as special cases if we treat the discrete label space as a subset of the real line?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a robust and efficient algorithm for PAC learning real-valued multi-index models, even under adversarial label noise. It matches its algorithmic sample complexity upper bounds with nearly tight Statistical Query lower bounds, showing the qualitative optimality of the proposed method in terms of dimension and noise tolerance.\n\nSTRENGTHS AND WEAKNESSES:\n### Strenghts\nThe paper is well-written, theoretically sound, and significant. It provides strong results on the learnability of multi-index models for a broader class of functions than in previous works. The exposition is clear and includes detailed algorithmic descriptions, comparisons with prior literature, and explanations of the results.\n\n### Weaknesses\nNo major weaknesses were identified. Minor points:\n1. The second condition in Definition 1.3 is quite technical and may be difficult to parse for non-experts; an intuitive explanation would improve accessibility.\n2. Line 45: is the symbol $c$ intended to be $f$?\n3. Typo on line 202: \"Specificalaly\" → \"Specifically\".\n4. The paper would benefit from a simple illustrative example to help readers build intuition around the main ideas and techniques.\n\nQUESTIONS:\n### Suggestions\n- In Section 1.4, the authors present their techniques as an generalization of [DPLB24] to MIMs. It would be relevant and helpful to extend this comparison to the recent paper [Damian et al., 2025] on MIMs, which addresses analogous questions. Moreover, the cited work [TDD+24] presents a first extension of the \"generative exponent\" concept to MIMs (in the proportional regime). This connection is worth mentioning in the same paragraph for completeness.\n- Points 1. and 4. in the Weaknesses section.\n\n\n\nDamian et al., \"The Generative Leap: Sharp Sample Complexity for Efficiently Learning Gaussian Multi-Index Models\", 2025.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning K-Multi-Index-Models (K-MIMs), which are models that depend on a K-dimensional subspace of the input.\n\nThe input distribution assumed in this paper is isotropic Gaussian, and the cases of agnostic learning, and independent noise are considered.\n\nThe paper provides:\n\n1. a new algorithm for learning a K-MIM $f$ which satisfies the following condition.\nCondition: for any subspace V,\n- either (i) a good approximation to $f$ can be found that depends only on the input projected to V\n- or, roughly speaking, (ii) if we subdivide the data space $\\mathbb{R}^d \\times \\mathbb{R}$ into fine-grained buckets depending on the projection to V and the label $y$, then with non-trivial probability a degree-$m$ polynomial applied to data from a randomly chosen bucket will be able to distinguish between the case that $V$ contains all of the directions on which $f$ mostly depends, and the case that there are more directions on which $f$ depends.\n\n2. The paper also proves matching SQ complexity bounds, showing that the dependence on the number of degree $m$ of the distinguishing polynomial is optimal.\n\nAs a corollary of the first result, the paper provides a new algorithm for learning homogeneous Lipschitz functions (which includes ReLU networks without biases), improving over the previous result of [CKM20] by removing an exponential dependence on the size of the network.\n\nSTRENGTHS AND WEAKNESSES:\nClarity: despite the technical nature of this paper, it is quite clear and easy to read. The exposition is well done, and the results are presented in logical order.\n\nOriginality: as the paper itself emphasizes, some of the techniques draw heavily from the very recent paper \"Robust Learning of Multi-index Models via Iterative Subspace Approximation\" by Diakonikolas et al. 2025. The main difference between these papers appears to be (as the paper indicates), a generalization from discrete labels to continuous labels. This necessitates new techniques, which make the technical contributions of this paper sufficiently substantial in my opinion to count as original.\n\nSignificance: understanding the complexity of learning multi-index functions is a central question in understanding feature learning in neural networks, as multi-index functions are a central toy case in that field. This paper helps clarify the picture by providing a new learning algorithm for K-MIMs, along with matching SQ bounds.\n\nQuality: the mathematics that I checked seemed correct\n\nQUESTIONS:\nNo questions -- am in favor of acceptance\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-12"}