Title: Optimal Spectral Transitions in High-Dimensional Multi-Index Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: spectral methods

multi-index models

approximate message passing

weak recovery

phase transitions

generalized linear model
Abstract: We consider the problem of how many samples from a Gaussian multi-index model are required to weakly reconstruct the relevant index subspace. Despite its increasing popularity as a testbed for investigating the computational complexity of neural networks, results beyond the single-index setting remain elusive. In this work, we introduce spectral algorithms based on the linearization of a message passing scheme tailored to this problem. Our main contribution is to show that the proposed methods achieve the optimal reconstruction threshold. Leveraging a high-dimensional characterization of the algorithms, we show that above the critical threshold the leading eigenvector correlates with the relevant index subspace, a phenomenon reminiscent of the Baik–Ben Arous–Peche (BBP) transition in spiked models arising in random matrix theory. Supported by numerical experiments and a rigorous theoretical framework, our work bridges critical gaps in the computational limits of weak learnability in multi-index model.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe papers studies spectral estimators for high-dimensional multi-index models. Two different spectral estimators are derived, based on linearization of an Approximate Message Passing (AMP) algorithm. These are computed via the principal eigenvector of an asymmetric and a symmetric data matrix, respectively. The weak recovery threshold for the estimators is characterized and shown to coincide with the  “optimal” threshold derived from a fixed point stability condition. Numerical results illustrate the performance of the spectral estimators.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \n\n- The problem studied is interesting and relevant. \n- The derived spectral estimators are easily implementable, and the precise characterization of the phase transition thresholds is nice.\n\nWeaknesses: \n\n- The paper is heavy on notation and tersely written. I had to read through multiple times to interpret the main results. Some of this is unavoidable due to the nature of the problem and the result, but the authors could give more high-level explanations and also be more precise with their claims. For instance, it was not clear why the reconstruction threshold was claimed to be optimal? Presumably it because it is also the value under which the trivial fixed point of state evolution is a stable one. Is this a standard definition of optimal threshold?\n\n- Related to the above, I found it hard to interpret Proposition 2.11 Please add a few sentences explaining what it means.\n\n- What is the behavior of the spectral estimators for models that do not satisfy Eq. (6). I suppose when (6) is not satisfied, the problem is easier, but it would be good to know how the estimators perform. \n\n- A recent work, presumably independent and concurrent, studies the high-dimensional asymptotics of spectral estimators (https://arxiv.org/abs/2502.01583). Please comment on how the results and the techniques compare with the ones in that paper.\n\nTypos and minor comments:\n\n- The dimensions don’t seem to match in Eq. (5).\n-  Typos: line 115 (throughout), line 118 (matrix)\n- Please explain the notation “mat” in (10) and (12). I could guess from context but should be made clear\n- In addition to [4], there are other papers in which AMP for multi-index models has been previously studied, e.g. \"Matrix Inference and Estimation in Multi-Layer Models\" and \"Mixed Regression via Approximate Message Passing\". Although these papers don't use the multi-index terminology, they study generalized linear models with matrix valued signals, which are the same models as those studied this paper.\n\nQUESTIONS:\nQuestions for the authors are listed in the comments above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of learning a Gaussian multi-index model from $n$  noisy samples. Here, the unknown $d$ dimensional function intrinsically resides on a p << d dimensional subspace. The goal is to devise algorithms in the high-dimensional regime where $n/d \\rightarrow \\alpha$ as $d,n \\rightarrow \\infty$. Based on the linearization of a message passing scheme, two spectral algorithms are proposed for this problem for recovering the unknown subspace. It is shown that both these methods achieve the optimal reconstruction threshold for weak recovery amongst efficient methods.\n\nSTRENGTHS AND WEAKNESSES:\nPros \n--------\n\n1. The results in the paper establish weak recovery results for Gaussian multi-index models which generalize existing work for single-index models. In that respect, the results do extend the state-of-art for this problem and would be of interest to researchers in high-dimensional statistics working on similar problems.\n\nCons\n----------\n1. Unfortunately, I found the exposition to be a bit hard to follow at some places, despite multiple passes of the text. The issue starts from page 3 onwards where a lot of technical details regarding GAMP is provided, but with the details deferred to the appendix. This makes it difficult to develop good intuition, even for the spectral methods introduced next, which, as the authors note, do appear to be a bit ad hoc. \n\n2.  As noted in the text, the proposed spectral methods are generalizations of existing similar methods for the single-index case. While this is fine in general of course, it does limit the novelty aspect a tad bit. Perhaps the main question is – what are the technical novelties/ difficulties in the analysis compared to the single index case? This doesn’t seem to have been discussed in detail but would make things more transparent for the reader.\n\nQUESTIONS:\n1. Is the link function $g$ assumed to be known to the learner?\n\n2. The matrix $W_{\\star}$ is assumed to have independent Gaussian columns but can something be said for any arbitrary $W_{*}$ of rank $p$?\n\n3. In the main results, the effect of $p$ is not visible in the bounds. Is it difficult to track this? Also, do the bounds depend polynomially on $p$? Some remarks on this would be helpful.\n\nLIMITATIONS:\nYes discussed towards the end", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study introduces two new spectral algorithms for weak recovery in Gaussian multi-index models. The authors review related literature, noting that while multi-index models have been thoroughly analyzed in the context of learning with neural networks, they approach the problem from the perspective of random matrix theory, where the sample complexity required to recover the relevant subspace remains an open question. Previous works conjectured a threshold but achieved it only with algorithms that relied on additional side information. The authors claim that the present work closes this gap and conjecture that the proposed spectral algorithms achieve weak recovery at the optimal sample complexity threshold. They support this claim through an analysis of the state evolution of an associated message passing scheme and extensive simulations.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- Spectral methods are important both in random matrix theory and high-dimensional statistics. The introduction of a well-performing spectral method for a topical model represents a significant contribution.\n- Although the paper does not provide a rigorous proof of the optimality of the algorithms, the analysis of the state evolutions of the message-passing schemes, together with the numerical simulations, offers solid evidence for the optimality of the introduced spectral algorithms.\n\n### Weakness\nThe work relies heavily on reference [18], which also focused on the study of a message-passing algorithm. It is not clear how closely related the algorithm introduced in the present paper is to the one discussed in [18]. This connection should be clarified further\n\nQUESTIONS:\n### Questions\n- There appears to be a mistake in Definition 1.1, as $W_* \\in \\mathbb{R}^{d \\times p}$ cannot be multiplied on the right by $v \\in \\mathbb{R}^d$. It is also unclear how to correct the expression, as there are no accompanying comments on the definition. Should $v \\in \\mathbb{R}^p$, or is an additional transpose operator missing inside equation (5)? Alternatively, perhaps the intention is to define weak recovery for subspaces of $W_*$, with $V \\subset \\mathbb{R}^p$?\n- In Figure 2, why are the orange, green, and purple lines not present for $\\alpha < \\alpha_c$? Are they overlapping with the blue line? If so, I would suggest using dashed or dotted lines to make overlapping curves visible.\n\n### Minor Remarks\n- For semipositive matrices, there is inconsistency in the notations: the + is sometimes subscripts and sometimes superscript.\n- Lines 111–112: repetition of the word \"functions.\"\n- Line 118: \"matrix,\" not \"matix.\"\n- There is inconsistency in the notation for the identity matrix; sometimes the dimension is included as a subscript and sometimes omitted. I suggest always specifying the dimension for clarity.\n\nLIMITATIONS:\nAddressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an AMP-based analysis of two spectral algorithms for Gaussian multi-index GLMs. Based on an analysis of the fixed points of GAMP with linear denoisers, two different spectral methods are provided, based on the eigenvalues (or singular values, depending on the approach) of two specially de- signed matrices that depend on observed data and the (known) likelihood for the observations given the covariates.\n\nFor both methods, a critical threshold for the sample complexity of the problem below which the spectral methods achieve asymptotically zero overlap with the ground-truth weights. Conversely, it is shown that above thresholds the proposed methods achieve weak recovery. The outputs of the two methods are related to each other in an exhaustive fashion.\nFor both approaches it is conjectured that the spectral distribution of the matrices used for the two spectral methods converges weakly to some distribution supported on a compact disk, with the exception of some outliers correlated with the weight matrices. These conjectures are supported by convincing numerical experiments.\n\nOverall, this paper is well-written and introduces novel spectral methods for multi-index models, thus generalizing previously known results about GLMs. On the technical side, relatively standard AMP techniques are used and the degree of novelty is somewhat reduced by the lack of proof for the spectral distribution of the operators described in the proposed algorithm. Nonetheless, I think this is a solid paper.\n\n\nComments:\n- The very last part of the section on numerical experiments is dedicated to providing some closed forms for the limiting overlap and critical sample complexity for specific link functions. This part is very cramped and quite unpleasant to read. I think it’d be better to place all expressions in the same appendix in which they’re derived and just have a pointer in the main text. Maybe use the extra space to expand on the description of the numerics and add the form of the two operators $F$ and $G$, which are defined in appendix but used in main text\n\n- Equation (14) and the lines immediately after use some $\\delta$ as a symbol. According to the claimed notational conventions this should be a scalar, but usage suggests otherwise, and its role is unclear. Furthermore, Kronecker deltas are used in other definitions, making it even more confusing.\n\n- Line 79: independant -> independent\n\nSTRENGTHS AND WEAKNESSES:\nsee summary.\n\nQUESTIONS:\nsee summary.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}