Title: Learning single index models via harmonic decomposition
Venue: NeurIPS 2025
Accepted: yes
Keywords: single-index models

statistical and computational complexity
Abstract: We study the problem of learning single-index models, where the label $y \in \mathbb{R}$ depends on the input $\boldsymbol{x} \in \mathbb{R}^d$ only through an unknown one-dimensional projection $\langle \boldsymbol{w_*}, \boldsymbol{x} \rangle$. Prior work has shown that under Gaussian inputs, the statistical and computational complexity of recovering $\boldsymbol{w}_*$ is governed by the Hermite expansion of the link function. In this paper, we propose a new perspective: we argue that *spherical harmonics*---rather than *Hermite polynomials*---provide the natural basis for this problem, as they capture its intrinsic \textit{rotational symmetry}. Building on this insight, we characterize the complexity of learning single-index models under arbitrary spherically symmetric input distributions. We introduce two families of estimators---based on tensor-unfolding and online SGD---that respectively achieve either  optimal sample complexity or optimal runtime, and argue that estimators achieving both may not exist in general. When specialized to Gaussian inputs, our theory not only recovers and clarifies existing results but also reveals new phenomena that had previously been overlooked.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a harmonic-decomposition perspective on high-dimensional single-index models (SIMs).  SIMs have attracted intense attention in recent years, with many complementary papers already charting phase transitions, statistical-computational gaps, and learning dynamics.By expanding the likelihood in spherical harmonics rather than Hermite polynomials, the authors (i) derive tight Statistical-Query and Local-Differential-Privacy lower bounds for every harmonic level in the signal; (ii) propose two estimators—harmonic tensor unfolding (which is sample-optimal) and online projected SGD (which is runtime-optimal)—that match those bounds up to logarithmic factors; and (iii) show that the classical Gaussian SIM results emerge as a special case of their framework. The analysis clarifies why landscape-smoothing and partial-trace spectral methods succeed, highlights when vanilla SGD is provably sub-optimal, and unifies several lines of earlier work.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts: This is a technically strong and unusually original paper: by viewing single-index models through the lens of harmonic decomposition, it unifies a patchwork of open questions—tight lower bounds, sample-optimal versus runtime-optimal algorithms, and the peculiar success of spectral “partial-trace’’ tricks—within a single, coherent framework. The quality of the work is high: the Statistical-Query and Local-DP lower bounds are rigorously proved, the two complementary algorithms meet those bounds up to log factors, and the analysis cleanly recovers the classical Gaussian results as a special case, thereby subsuming several separate lines of literature\n\nWeaknesses: The exposition suffers from a notable gap in citations. The discussion of the easy/hard transition in SIMs references overlooks earlier rigorous work that first established the information-theoretic limit and computational gap; recent rigorous papers on learning dynamics are omitted as well, and the genealogy of spectral methods is incomplete. These omissions currently temper the otherwise excellent contribution.\n\nWhile the paper offers an original and compelling perspective on single-index models through harmonic decomposition, I believe several important references are missing, which would significantly improve the completeness and scholarly context of the discussion. First and foremost, the paper “Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models” (arXiv:1708.03395) deserves to be cited when discussing the GE=2 case and the statistical-to-computational gap. This work rigorously established the information-theoretic limits and the phase transition at Hermite degree 2—what is now often called the “Hermite-2 threshold”—and provided an early and formal treatment of the Parisi–Franz potential and the associated computational gap. While MM18 proposed an important spectral method to approach this threshold, the existence of the transition itself and the structure of the gap were already clearly laid out in this earlier paper. As a point of reference, this work is cited in [DPVLB24] and in several other recent contributions.\n\nIn addition, two more recent papers—arXiv:2202.00293 and arXiv:2406.02157—provide rigorous and contemporary analyses of SIM learning under Gaussian assumptions, focusing in particular on gradient-based algorithms and the implicit regularization they induce. These works are on par with other cited results (e.g., [AGJ21], [DNGL24]) and would be highly relevant to include in the broader discussion. On the spectral side, the paper also omits a key antecedent to MM18: Yue Lu’s 2017 work (arXiv:1702.06435), which provides early results on spectral thresholds in high-dimensional nonconvex estimation problems, including phase retrieval\n\nQUESTIONS:\nExtension to staircase links: Could the harmonic-decomposition framework be adapted to “staircase’’ regressions, i.e. settings where the link function is piece-wise constant rather than smooth?\n\nMulti-index generalisation: Is there a principled way to lift the analysis from single- to multi-index models that involve several latent directions?\n\nNoise robustness: How sensitive are the proposed lower bounds and algorithms to additive observation noise—especially of the type analysed in arXiv:2502.06443v1—and can the theory be modified to accommodate such perturbations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studied the problem of learning single index models under rotation-invariant distributions, where the data vector x satisfies $x=rz, r\\sim\\mu_r$ and $z\\sim\\tau_d$, and $\\tau_d = \\mathrm{Unif}(\\mathbb{S}^{d-1})$. The labels y satisfy $y | r,z \\sim \\nu_d(. | r,w^* z)$ where $w^*$ is the unit-norm hidden direction. It is assumed $\\nu_d$ is known. The authors then used spherical harmonics decomposition of the link function rather than hermite polynomials to study the computational complexity. The main contributions of this paper are three-fold:\n1. The authors showed that under the spherical model, using harmonic decomposition, there is a gap between the SQ lower bounds and LDP lower bounds, indicating a 'sample-runtime' trade-off phenomenon. In detail, SQ lower bounds indicate a runtime complexity lower bound that is higher than the sample complexity lower bound implied by LDPs. The authors proposed a characterization of the sample and runtime complexity using the degree of the Gegenbauer polynomial that has the maximum signal from the harmonic decomposition.\n2. The authors provided sample optimal and runtime optimal algorithm respectively, using tensor unfolding and online SGD. The authors conjectured that there is no algorithm that can achieve optimal sample complexity and runtime complexity at the same time.\n3. The authors applied the framework to Gaussian SIMs, i.e., they studied learning SIMs with link function $y|r,z=\\rho(w*\\cdot z)$ where $\\rho$ has generative exponent $k*$ and $r\\sim\\chi_d$ and $z\\sim \\mathrm{Unif}(\\mathbb{S}^{d-1})$. They showed that the optimal harmonic degree is either 1 or 2 (depending on the parity of k^*) and recovers the optimal sample and runtime complexity of learning Gaussian SIMs in prior literature.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. I think the observations and findings of this paper are very interesting. The paper goes beyond Gaussian distribution and studied the computational complexity of learning SIMs under more general rotational invariant distributions, and discovered a complexity measure that recovers prior complexity results on learning Gaussian SIMs. It is also very interesting to see that there is a gap between SQ and LDPs under the spherical model, indicating a sample-runtime trade-off.\n2. The technical aspect of this paper is solid and strong.\n3. The paper is mostly clearly and fluently written.\n\nWeakness\n1. I think in the introduction section where the authors compared prior works on learning Gaussian SIMs is somewhat confusing, e.g., line 40-47. [Damian et al Neurips] closes the gap in terms of the information exponent (i*) rather than the generative exponent (k*), and I don't think [Damian et al COLT] simply removes the polylog terms. Further, my understanding is that [D et al Neurips] and [D et al COLT] are essentially very different algorithms that tackle different computational barriers.\n\n2. In the comparison to prior work part, I am afraid the authors missed another intensive thread of research on learning SIMs under general or Gaussian distributions with misspecification/agnostic noise, e.g., [DGKK+20], [DKTZ22], [WZDD23],[GGKS23], [ZWDD24], [WZDD24], [GV25], [ZWDD25] and so on. It would be more well-rounded if this thread of research were mentioned in the comparison to prior works part.\n\n[DGKK+20]: Diakonikolas I, Goel S, Karmalkar S, Klivans R, Soltanolkotabi M. Approximation Schemes for ReLU Regression, in COLT 2020\n[DKTZ22]: Diakonikolas I, Kontonis V, Tzamos C, Zarifis N. Learning a single neuron with adversarial label noise via gradient descent, in COLT 2022\n[WZDD23]: Wang P, Zarifis N, Diakonikolas I, Diakonikolas J. Robustly Learning a Single Neuron via Sharpness, in ICML 2023\n[GGKS23]: Gollakota A, Gopalan P, Klivans A, Stavropoulos K. Agnostically learning single-index models using omnipredictors, in Neurips 2023\n[ZWDD24]: Zarifis N, Wang P, Diakonikolas I, Diakonikolas J. Robustly learning single-index models via alignment sharpness, in ICML  2024.\n[WZDD24]: Wang P, Zarifis N, Diakonikolas I, Diakonikolas J. Sample and computationally efficient robust learning of gaussian single-index models, in Neurips 2024.\n[GV25]: Guo A, Vijayaraghavan A. Agnostic Learning of Arbitrary ReLU Activation under Gaussian Marginals.\n[ZWDD25]: Zarifis N, Wang P, Diakonikolas I, Diakonikolas J. Robustly Learning Monotone Generalized Linear Models via Data Augmentation.\n\nQUESTIONS:\n1. In line 113-119, the authors claimed that the SGD is suboptimal, possibly due to the fact that it does not exploit the radial component $r=||x||$ and claimed that algorithms ignoring r must have runtime $d^{k*}$. Can you provide a pointer to where this claim is justified? I skimmed through section E and did not spot the supporting lemma.\n2. Furthermore, can the authors elaborate on how to exploit the radial exponent?\n\nLIMITATIONS:\nno negative limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study learning in a **single index model** of the form $x \\mapsto g(w^* \\cdot x)$, where the input distribution over $x$ is **rotation-invariant**. Their analysis leverages the **spherical harmonic decomposition** of the signal to:\n\n- **Characterize the problem difficulty**, via the spectral structure of the signal; (decomposition into the spherical harmonics)\n- **Define the optimal transformation** of the learning task to recover the planted direction $w^*$;\n- **Establish fundamental limits** through minimax **statistical complexity bounds**:\n  - **SQ lower bound** ,\n  - **LDP lower bound**\n- **Analyze the complexity** of different algorithms:\n  -  Spectral algorithm and Tensor Unfolding\n  - **Stochastic Gradient Descent (SGD)** ;\n- **Recover known results in the Gaussian case** when the *generative exponent* rules the complexity of the algorithms derived in the article.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- **Conceptual clarity**: The use of spherical harmonics provides a principled and elegant framework to analyze single index models under rotational symmetry.\n- **Theoretical depth**: The paper offers nontrivial lower bounds (SQ and LDP) that clarify the statistical hardness of the problem.\n- **Unified perspective**: The approach recovers known results in Gaussian settings and discuss the fundamental limits of SGD.\n- **Technically solid**: Proofs appear to be rigorous and well-structured, with clear connections between harmonic properties and learning performance.\n\n### Weaknesses\n\n- **Overloaded presentation**: While the authors make a commendable effort to define concepts precisely and clearly, the paper feels overly dense, with too many results introduced in rapid succession. This makes it hard to follow the central narrative, and the reader may struggle to grasp the key insights. A better organization of the material would help, although it's not immediately obvious how to restructure it. As it stands, the paper sometimes gives the impression of going in too many directions at once.\n\n- **Unclear resolution of the central question**: The paper raises the important question of why SGD-based algorithms are suboptimal, exhibiting a runtime of $d^k$ instead of the optimal $d^{k/2 + 1}$. However, the answer to this question remains hard to extract. In particular, it is unclear where it is precisely stated (or proved) that a regularity assumption on the distribution over $\\nu_{d, R}$ is necessary to halve the complexity exponent. Also, is there a clear intuition or conceptual reason behind this improvement? Making this point more explicit would strengthen the paper.\n\nQUESTIONS:\n- I find the following sentence unclear: \"The SGD algorithm of [2] is dominated by the high-frequency harmonics $V_{d,k}$, while smoothing [10] reweights the loss landscape toward low-frequency harmonics $V_{d,1}$ or $V_{d,2}$.\" It is not obvious to me how to interpret this, since the runtime complexity of SGD depends explicitly on $k^*$, whereas the spaces $V_{d,1}$ and $V_{d,2}$ do not seem to play a direct role in the complexity. Could the authors clarify what is meant here?\n\n- Could you provide some insight on Assumption 1? As stated, it appears to imply that a single harmonic component dominates the others in terms of statistical or algorithmic complexity. Is this the intended meaning?\n\n- In line 200, SGD is defined with respect to some parameter $l$, and a transformation $T_{l}$ that seems to extract the coefficient of the response in the $V_{d,l}$ harmonic. Is this interpretation correct? Also, which value of $l$ is chosen in Corollary 2? Presumably, it should be related to $k^*$; but this is not clearly stated. Could the authors make this connection more explicit?\n\nLIMITATIONS:\nNo limit, I would be nice to have clearer results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author considers the setting of learning single index models with Gaussian marginals.\nIn particular, the noise model they consider here is kind of between realizable and agnostic, \nin the sense that the distribution of y can only depends on the important direction but need not to labeled by a single target function (equation 1).\nThe authors gave a very novel and insightful observation that the hardness can be more naturally analyzed by spherical harmonics instead of Hermite polynomials (which is the typical choice of basis in previous literature).\nThis can be intuitively explained by that the spherical harmonics are the irreducible representations for the rotation symmetry.\n\nBased on this, the authors proposed the spherical single index model framework (which generalizes the regular single index model).\nWhile the authors did not provide any useful problem/application that fits into this framework but not the previous regular single index model framework.\nThey did show that by analyzing the SQ and LDP complexity of the spherical single index model, not only did they recover the previous results of generative exponents for single index models.\nBut also give some explanation about why online SGP types of algorithm failed to give the right sample complexity in SIM.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\nThe paper offers a very novel and insightful perspective on the single index model.\n\nWeakness:\nFrom what I know, there is also a line of work on agnostic single-index modes, which I think is relevant here and should be cited.\nFor example, here some that I found (I'm sure I'm missing some):\n\n1. I. Diakonikolas, D. M. Kane, and N. Zarifis. Near-optimal SQ lower bounds for agnostically learning\nhalfspaces and ReLUs under Gaussian marginals.\n2. S. Goel, A. Gollakota, and A. R. Klivans. Statistical-query lower bounds via functional gradients. \n3. I. Diakonikolas, D. M. Kane, T. Pittas, and N. Zarifis. The optimality of polynomial regression for\nagnostic learning under Gaussian marginals in the SQ model. \n4. I. Diakonikolas, D. M. Kane, and L. Ren. Near-optimal cryptographic hardness of agnostically learning\nhalfspaces and ReLU regression under Gaussian marginals.\n5. P. Manurangsi and D. Reichman. The computational complexity of training ReLU(s). \n6. I. Diakonikolas, D. Kane, P. Manurangsi, and L. Ren. Hardness of learning a single neuron with\nadversarial label noise.\n7. I. Diakonikolas, S. Goel, S. Karmalkar, A. R. Klivans, and M. Soltanolkotabi. Approximation schemes\nfor ReLU regression.\n8. I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning a single neuron with adversarial label\nnoise via gradient descent. \n9. P. Awasthi, A. Tang, and A. Vijayaraghavan. Agnostic learning of general ReLU activation using\ngradient descent.\n10. P. Wang, N. Zarifis, I. Diakonikolas, and J. Diakonikolas. Robustly learning a single neuron via\nsharpness.\n11. A. Gollakota, P. Gopalan, A. R. Klivans, and K. Stavropoulos. Agnostically learning single-index models\nusing omnipredictors.\n12. N. Zarifis, P. Wang, I. Diakonikolas, and J. Diakonikolas. Robustly learning single-index models via\nalignment sharpness.\n13. A. Guo and A. Vijayaraghavan. Agnostic learning of arbitrary ReLU activation under Gaussian\nmarginals.\n14. N. Zarifis, P. Wang, I. Diakonikolas, and J. Diakonikolas. Robustly Learning Monotone Generalized Linear Models\nvia Data Augmentation\n\nQUESTIONS:\n1. Can the authors give some further high-level comments on why LDP and SQ gives different information-computation trade-off?\nEspecially, if there is any intuition about why they predicts the right sample/time complexity for tensor unfolding and SGD respectively.\n\n2. I'm a bit confused about Lemma 1, from my understanding Lemma 1 only assume that p has generative exponent k^*.\nCombining this with the decomposition of Hermite polynomial into Gegenbauer polynomials.\nShouldn't this only implies that there exists one spherical harmonic (in this decomposition) that has large coefficients? \nInstead, Lemma states that coefficients for all spherical harmonic (in this decomposition) are this stated value (up to constant factor).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the sample complexity of learning single-index models under spherically symmetric input distributions. Unlike most existing results that characterize the sample-complexity (when restricted to a class of algorithms) for Gaussian inputs through Hermite decomposition of targets/likelihood ratio, this work provides such a characterization through a decomposition along radial components and spherical harmonics. This, in turn, subsumes the existing results on Gaussian inputs.\n\nThe main contributions are twofold:\n * Lower bounds for SQ and low-degree polynomial methods.\n* Spectral and online-SGD based algorithms achieving weak-recovery of the target direction with optimal sample-complexity and runtime, respectively. \n\nBoth the above contributions are based on the generalization of existing techniques, such as the analysis of the partial trace estimator and online-SGD from Gaussian to general spherical distributions. By providing run-time lower bounds for SQ and sample-complexity lower bounds through the low-degree polynomial framework, the paper further highlights a trade-off between achieving optimal sample-complexity vs optimal runtime.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\n* The paper is presented and organized well. The definitions are clean, and the setting is quite general. For instance, the non-zero Gegenbauer coefficients are not restricted to be $\\Theta(1)$. Appendix A provides a nice interpretable summary of the central ideas behind the lower bounds and estimators.\n* The ideas behind the extension of the results for Gaussian inputs to general spherically symmetric distributions, in particular the decomposition along irreducible representations of the symmetry group, could allow extension to other interesting distributions with symmetries such as permutationally-invariant data.\n* Certain conclusions, such as the limitations of algorithms not exploiting the radial component, and the sample-complexity/runtime tradeoffs, are insightful and could benefit future related work.\n\n# Weaknesses\n\n* Limited novelty and scope: The analysis is largely based on extensions of existing techniques from Gaussian to spherically symmetric distributions. The paper would benefit from a discussion of the novel insights gained by this extension and the novel ideas/challenges in the proofs. For instance, the proofs for Theorems 1 and the spectral/tensor-based algorithms largely follow Damian et al. 2024, while the analysis of the SGD algorithm is largely based on Zweig et al. 2023.\n\n* Inadequate discussion of literature: The paper does not cite or discuss a number of relevant works:\n\n  Learning multi-index models in polynomial time through an SGD-based algorithm:\n\n  Chen, Sitan, and Raghu Meka. \"Learning polynomials in few relevant dimensions.\" Conference on Learning Theory. PMLR, 2020.\n\n  Another missing line of work shows how the sample complexity of online-SGD can be improved to match the SQ sample-complexity through the re-use of data:\n\n  Dandi, Y., Troiani, E., Arnaboldi, L., Pesce, L., Zdeborova, L., &; Krzakala, F. The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents. In Forty-first International Conference on Machine Learning.\n\n  Lee, J. D., Oko, K., Suzuki, T., &amp; Wu, D. (2024). Neural network learns low-dimensional polynomials with sgd near the information-theoretic limit. Advances in Neural Information Processing Systems, 37, 58716-58756.\n\n  Arnaboldi, L., Dandi, Y., Krzakala, F., Pesce, L., &amp; Stephan, L. (2024). Repetita iuvant: Data repetition allows sgd to learn high-dimensional multi-index functions. arXiv preprint arXiv:2405.15459.\n\nQUESTIONS:\n* Assumption 1 doesn't appear quite transparent. Inclusion of interpretation and examples behind the assumption would strengthen the paper.\n* Definition 1 says $\\mathcal{Y}$ can be an arbitrary-measurable space. Aren't the targets restricted to be real-valued (as specified in equation 1 in the introduction)?\n* Concerning section 4 and observation 2, how does label smoothing exploit the radial components?\n* Why is the SQ-based run-time lower bound expected to hold for a general class of algorithms? The paper would benefit from a brief discussion of the evidence behind this heuristic, as well as a more transparent presentation of it as a conjecture.\n\nLIMITATIONS:\nThe work is primarily theoretical and does not pose direct societal risks. The paper would benefit from a more detailed discussion of the limitations I've listed in the review, such as the conjectural status of the run-time lower-bound prediction from the SQ framework.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-07"}