Title: Learning Orthogonal Multi-Index Models: A Fine-Grained Information Exponent Analysis
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-index model

online stochastic gradient descent

information exponent

sample complexity
Abstract: The information exponent (Ben Arous et al. [2021]) and its extensions --- which are equivalent to the lowest 
  degree in the Hermite expansion of the link function (after a potential label transform) for Gaussian single-index 
  models --- have played an important role in predicting the sample complexity of online stochastic gradient descent 
  (SGD) in various learning tasks. In this work, we demonstrate that, for multi-index models, focusing solely on the 
  lowest degree can miss key structural details of the model and result in suboptimal rates. 

  Specifically, we consider the task of learning target functions of form $f_*(x) = \sum_{k=1}^{P} \phi(v_k^* \cdot x)$, 
  where $P \le d$, the ground-truth directions $\\{ v_k^* \\}_{k=1}^P$ are orthonormal, and the information exponent of 
  $\phi$ is $L$. Based on the theory of information exponent, when $L = 2$, only the relevant subspace (not the exact 
  directions) can be recovered due to the rotational invariance of the second-order terms, and when $L > 2$, 
  recovering the directions using online SGD require $\tilde{O}(P d^{L-1})$ samples. In this work, we show that by 
  considering both second- and higher-order terms, we can first learn the relevant space using the second-order
  terms, and then the exact directions using the higher-order terms, and the overall sample and complexity of online 
  SGD is $\tilde{O}( d P^{L-1} )$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPAPER SUMMARY:\nThe information exponent (IE) of a multi-index model $f(x) = \\sum_i \\phi(v_i \\cdot x)$ is the order of the first nonzero term in the Hermite expansion of $\\phi$ around $0$.\nThe IE and its generalization, the generative exponent (GE), provide characterizations for the sample complexity of learning multi-index models via SGD. In particular, roughly $d^{IE -1}$ samples are needed if $IE \\geq 3$.\n\nWhen $IE = 2$, rotational invariance makes it impossible to recover the exact orthonormal vectors $\\{v_i\\}_i$ and only $Span(\\{v_i\\}_i)$ can be recovered. This paper investigates the statistical complexity of recovering all the orthonormal vectors $\\{v_i\\}_i$ as a function of $L$, where $IE = 2$, and $L$ is the the next non-zero term in the Hermite expansion of $\\phi$.\n\nThis work proves that $O(PdP^{L-2})$ samples are sufficient for recovering all vectors $\\{v_i\\}_i$.\nHere is the catch: even if it does not allow recovery of all the vectors $\\{v_i\\}_i$, having a non-zero second-order term is advantageous, as $d^{L-1} \\gg PdP^{L-2}$ samples would be required if the second-order term was null. In particular, the IE does not tell the full statistical complexity story when $IE = 2$.\n\nSTRENGTHS AND WEAKNESSES:\nI believe that the authors could have done a better job at articulating why this particular problem should be interesting for ML theorists. Overall, it seems like the studied phenomenon manifests itself in the particular case $IE = 2$, when we want to recover the exact vectors $\\{v_i\\}_i$, and does not add much to the bigger picture of multi-index models.\n\nThis work claims that \"focusing solely on the lowest degree can miss key structural details of the model and result in suboptimal rates\".\nIt seems to me that the main message of this work is that, for the particular case of $IE = 2$, looking at the next non-zero term is informative. This does not detract from the way more general statistical bound for $IE \\geq 3$, and does not add much depth to that line of work.\n\nQUESTIONS:\nI probably missed something, but I am not sure why you write $PdP^{L-2}$ rather than $dP^{L-1}$.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to understand the sample complexity of learning a multivariate function of the form $x \\to \\sum_{i = 1}^P \\phi(v_k \\cdot x)$. The input $x \\sim N(0,1)$ and the learning is performed via a modified version of \"SGD\" with a two-layer neural network.\n\nIn the article, the information exponent is $2$, but the aim of the paper is to show that while this first order term $h_2$ has a rotational invariance that prevent the $v_k$ to be retrieved (only the span of it), higher order terms can help having a perfect recovery, without paying any cost.\n\nThe main theorem is presented in Section 2, while the sketch of the proof is divided between the two subsequent parts : the gradient flow study and then the discretization via SGD.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- the paper seems technically very solid, with clear sketch of proofs and a concrete goal. Analyzing sample complexities in such tasks if often quite technical\n- the aim of the paper is clear and it is overall well written\n\nWeaknesses:\n- the scope of the paper is quite narrow : it deals with a specific problem in the multi-index model (a separable sum with the same link function), and the link function has a very specific structure : $h_2 + h_L + \\dots$.\n- every thing is sort of tailored for the result to hold : the neural network is adapted to the problem in the sense that the link function which could be a priori unknown is the non-linearity of the Neural Network. The algorithm is also very specific : freezing layers after the other ones. I understand these assumptions and algorithms are abundant in the literature, yet I have to signal this as a weakness.\n\nQUESTIONS:\nThe aim of the paper is to show that the network is able to retrieve the $v_k$, and not only the span of it. In the theorem, it is not clear to me that it is the case since it is stated that the population loss is small. I understand that it could be that since there are higher order terms, this implies that the $v_k$ are retrieved but this needs to be clarified\n\nI have trouble with the claim that *the information/generative exponent is insufficient* ? For me it still appears as the integer driving the complexity of the dynamics?\n\nLIMITATIONS:\nAlready discussed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the problem of learning orthogonal multi-index models under Gaussian distribution. The paper assumes the multi-index model takes the specific form of $f^\\ast(x) = \\sum_{k=1}^P \\phi(v_k^\\ast\\cdot x)$, where $\\phi$ are non-linear activations that have the form of $\\phi=he_2 +\\sum_{l\\geq L}\\hat{\\phi}_l he_l$. The authors showed that, using online sgd algorithm, using $dP^{O(L)}$ samples suffices to learn the target $f^\\ast$ to $\\epsilon$-$L_2^2$ error. The authors provided a set of frameworks and tools for the analysis of sgd dynamics on such objectives, and provided a white-box, detailed analysis of the learning algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. I think the paper conducted a thorough study of an interesting special case of multi-index models. The paper is solid in techniques. \n2. I think the characterization of the sample complexity, in general, is not surprising, as similar (qualitatively) results have been derived in prior literature like [DLS22]. However, I think it is still an interesting contribution to obtain such a result for the special case.\n\nWeakness\n1. I think the condition that $P\\geq \\log^C(d)$ is a weird condition that seemly most likely to be a side-effect of the analysis. Is it possible to remove this by refining your arguments?\n2. In [DLS22], the authors studied learning MIMs of the form $g(v^\\ast_1\\cdot x,\\dots,v^\\ast_p\\cdot x)$ where $g$ is a polynomial of bounded degree, using two layer neural nets. Though [DLS22] has sample complexity $d^2 P + d P^L$ (consider $g$ being a degree $L$ polynomial), this paper considers a much simpler form of the multi-index model. I wonder if it is possible to relax the orthogonality condition to the condition of [DLS22], i.e., $E[\\nabla^2 f^*(x)]$ has rank $P$. Otherwise, it seems the major contribution of this paper is to improve the degree of $d^2$ to $d$, with the sacrifice of model generality.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nno negative limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the sample complexity of learning multi-index models of the form $\\sum_{k=1}^P \\phi(v^\\star_k . x)$ with orthogonal $v^\\star_k$  through online-SGD on a two-layer neural network model. Unlike existing works focusing on the recovery of the target subspace and subsequent learning of the target non-linearity, the main results show that with sufficient overparameterization, the neurons of the trained model exactly recover individual target directions $v^\\star_k$ with $\\mathcal{O}(P dP^{L-2})$ sample-complexity, where $L$ denotes the information exponent. Crucially, the sample-complexity for recovering individual directions remains linear in the dimension. The exact recovery of directions proceeds through higher-order terms in the gradient dynamics, which amplify the neurons' initial difference in overlaps across different directions.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n- The paper is well-written and organized with adequate discussion of prior work.\n- The analysis is insightful. In particular, the insights into how higher-order information in the gradients leads to the exact recovery of individual directions are expected to carry over to general setups, such as while jointly training all the layers, or for deeper networks.\n- The proofs are simple and the paper provides readable proof-sketches explaining the key ideas.\n\n# Weaknesses\n- The width-complexity or overparameterization in the main result is specified as Poly(P). A precise description of this polynomial dependence would greatly strengthen the results.\n- The paper suffers from the limitations of using correlation loss (which prevents interactions between neurons). However, since this simplifying assumption is used by several related works in the literature, I consider it a justifiable starting point.\n- Another related limitation is the absence of a linear component in the target non-linearity. With correlation loss, this presumably leads to a spike from the linear components dominating the dynamics. A discussion of possible directions to relax such limitations would strengthen the work.\n\nQUESTIONS:\n* What factors does the polynomial in the Poly(P) sample-complexity depend on? Can it be well-specified?\n* Can assumption 1(c) be extended to targets with polynomially-bounded derivatives?\n* where does the diverging width requirement $ \\log d^c < P$ arise from?\n* what would happen if Assumption 1(a) is relaxed to allow a linear-component in the target non-linearity?\n\nIncluding the above clarifications would strengthen the paper.\n\nLIMITATIONS:\nThe work is primarily theoretical and does not pose direct societal risks. The paper would benefit from a more detailed discussion of the limitations of their model listed under \"weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}