Title: The Generative Leap: Tight Sample Complexity for Efficiently Learning Gaussian Multi-Index Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Index Models

Low-Degree Polynomials

Computational-Statistical Gaps
Abstract: In this work we consider generic Gaussian Multi-index models, in which the labels only depend on the (Gaussian) $d$-dimensional inputs through their projection onto a low-dimensional $r = O_d(1)$ subspace, and we study efficient agnostic estimation procedures for this hidden subspace. We introduce the *generative leap* exponent, a natural extension of the generative exponent from Damian et al. 2024 to the multi-index setting. We show that a sample complexity of $n=\Theta(d^{1 \vee k^\star/2})$ is necessary in the class of algorithms captured by the Low-Degree-Polynomial framework; and also sufficient, by giving a sequential estimation procedure based on a spectral U-statistic over appropriate Hermite tensors.

The Generative Leap: Tight Sample Complexity for Efficiently Learning Gaussian Multi-Index Models

Abstract

In this work we consider generic Gaussian Multi-index models, in which the labels only depend on the (Gaussian) d -dimensional inputs through their projection onto a low-dimensional r = O d (1) subspace, and we study efficient agnostic estimation procedures for this hidden subspace. We introduce the generative leap exponent, a natural extension of the generative exponent from Damian et al. [2024] to the multiindex setting. We show that a sample complexity of n = Θ( d 1 ∨ k ⋆ / 2 ) is necessary in the class of algorithms captured by the Low-Degree-Polynomial framework; and also sufficient, by giving a sequential estimation procedure based on a spectral U-statistic over appropriate Hermite tensors.

1 Introduction

We consider learning Gaussian multi-index models:

Definition 1. We say that ( X,Y ) follows a Gaussian multi-index model with index r if X ∼ N (0 , I d ) := γ d and there exists a subspace U ⋆ ∈ G ( r, d ) such that the conditional law P [ Y | X ] only depends on the orthogonal projection P U ⋆ X .

A Gaussian multi-index model can be thus specified by choosing a basis W ⋆ of U ⋆ (ie, an element of the Stiefel manifold S ( r, d ) ), and the law P of ( Z, Y ) ∈ P ( R r × R ) 1 , where Z = ( W ⋆ ) ⊤ X . The subspace U ⋆ is referred as the index space .

Given a joint distribution P of ( Z, Y ) , a natural statistical task associated with such a model is to plant a subspace W ⋆ , uniformly drawn from the Haar measure of S ( r, d ) , and draw n iid samples from the multi-index distribution P W ⋆ , P parametrized by W ⋆ and P . Our task will be then to recover U ⋆ = span[ W ⋆ ] given these samples. We note that this task is only well-posed when the 'intrinsic' dimension of the model is r , namely that P does not admit a factorization P = γ r ′ ⊗ P S , where P S ( z S , y ) is the marginal of P over a subspace of R r × R of dimension < r +1 that includes the last coordinate. We will assume this property from now on.

We place ourselves in the setting where r = O d (1) , and consider the high-dimensional regime. Since the dimensionality of S ( r, d ) is of order rd , one expects that a brute-force estimation procedure that fits P P ,W j over a suitable ϵ -net of { W j } j ⊂ S ( r, d ) requires O ( dϵ -2 ) samples to estimate the index space up to accuracy ϵ . Our main motivation is to understand this question from the lens of computational-statistical gaps: how many samples are needed, as a function of d, P , to produce an

estimate of the planted subspace using polynomial-time algorithms, as opposed to using brute-force? This question enjoys a large literature, spanning high-dimensional statistics and learning theory, starting from the inverse regression methods from Li [1991] and beyond Xia [2008], Xia et al. [2002], Hristache et al. [2001], Cook and Li [2002], Cook [2000], Vempala [2010], Klivans et al. [2008], Mossel et al. [2003], Daniely et al. [2025] (see also Bruna and Hsu [2025] for a recent survey), where efficient algorithms have been developed for specific instances. Multi-index models are an appealing semiparametric model, and provide arguably the simplest instance of linear feature learning , in the sense that the index space provides an adapted low-dimensional representation to perform high-dimensional learning. Some notorious examples include

(noisy) Gaussian parity: Y | Z d = ξ sign[ Z 1 · Z 2 . . . Z r ] , with P [ ξ = -1] = η , P [ ξ = 1] = 1 -η independent of Z and η < 1 / 2 .

Gaussian staircase functions: Y | Z d = ϕ 1 ( Z 1 ) + ϕ 2 ( Z 1 , Z 2 ) + . . . + ϕ r ( Z 1 , . . . , Z r ) .

Intersection of r half-spaces: Y | Z d = 2 ∏ r j =1 1 ( v ⊤ j Z > α j ) -1 .

Low-rank shallow neural network: Y | Z = a ⊤ ρ ( V ⊤ Z ) + ξ for some a ∈ R M , V ∈ R r × M , ρ : R → R , additive noise ξ independent of Z .

Polynomials: Y | Z = q ( Z ) where q is a polynomial.

Focusing on the Gaussian setting, several works, starting from Dudeja and Hsu [2018], Ben Arous et al. [2021] and followed by Abbe et al. [2021, 2023], Bietti et al. [2022], Damian et al. [2022], Ba et al. [2022], Dandi et al. [2024a] have built a harmonic analysis framework to analyze a large class of algorithms, including stochastic gradient descent over NN architectures, leading to sample complexities of the form n = Θ( d k ) , where k is an explicit exponent associated with a certain harmonic expansion of P . In particular, Damian et al. [2024], focusing on Single-Index models (where r = 1 ), identified the generative exponent k ⋆ = k ⋆ ( P ) (see Section 2) as the fundamental quantity driving the sample complexity, in the sense that n = Θ( d 1 ∨ k ⋆ / 2 ) is both necessary and sufficient in the class of algorithms implemented by SQ (Statistical Queries) and Low-Degree Polynomials. In essence, the generative exponent arises from an expansion of the inverse regression of Z given Y , as put forward in the original Li [1991]. Lee et al. [2024], Arnaboldi et al. [2024], Dandi et al. [2024b] showed that SGD with reused samples can learn single index models dependent on the generative exponent, instead of the information exponent.

In this work, we extend this notion of generative exponent to the general multi-index setting. As already pointed out in the literature Abbe et al. [2023], Bietti et al. [2025], Troiani et al. [2024], Diakonikolas et al. [2025b], the general r > 1 setting gives rise to important new phenomena not present in the single-index case. In particular, gradient-based learning exhibits a sequential behavior in the form of saddle-to-saddle dynamics, where the index space is revealed incrementally along specific subspaces, with different timescales associated with each step. Moreover, such incremental alignment requires solving a semi-parametric problem, where both the subspace and the link function need to be estimated jointly. We overcome these additional challenges by identifying a suitable generalization of the generative exponent, the leap generative exponent k ⋆ (see Definition 3), arising from a 'canonical' orthogonal decomposition of the index space, the leap decomposition (see Section 2).

We first show that this exponent provides a computational lower bound of n = O ( d k ⋆ / 2 ) under the Low-degree polynomial (LDP) framework, by extending the previously established lower bound in the single-index setting Damian et al. [2024] to an appropriate detection task that is dominated by the index estimation task (Theorem 1). Next, and more importantly, we provide an algorithm that sequentially estimates the index space along the leap decomposition from the spectrum of a novel kernel U -statistic (see Eq (5)). This algorithm recovers the index space as soon as n ≳ d k ⋆ / 2 , thus matching the LDP lower bound, and, crucially, it does not require prior knowledge of the multi-index model P (Theorem 3). We complement these general results by several case studies that give novel guarantees on specific multi-index models, such as general ReLU networks or Gaussian Parities; see Section 5. Taken together, our results therefore provide the correct, sharp dimension dependence for any (Gaussian) multi-index model . In particular, as soon as k ⋆ > 2 , they provide evidence of a computational-to-statistical gap at the polynomial scale.

Related Works Chen and Meka [2020] show that any polynomial multi-index model can be learned with n ≳ d samples via an iterated filtered PCA algorithm. Chen et al. [2022] extended this to the case of multi-index models with ReLU activation with a similar algorithm. As we will show in Section 5, the generative exponent satisfies k ⋆ ≤ 2 , so our algorithm also requires only n ≳ d .

Gradient descent on two-layer networks has been extensively studied [Bietti et al., 2025, Ren and Lee, 2024, Ren et al., 2025, Damian et al., 2022, Abbe et al., 2023], these papers typically require at least n = Θ( d 1 ∨ l ⋆ -1 ) , where l ⋆ is the information exponent [Arous et al., 2021], an upper bound of the generative exponent. Abbe et al. [2023] provide a similar definition of leap exponent but tailored to the information exponent, and thus larger than the generative leap exponent. In the setting of sparse juntas, Joshi et al. [2024] showed that by changing the loss function from square loss to another loss, gradient queries learn with complexity governed by the SQ -exponent, which is analagous to the generative exponent but restricted to juntas. Troiani et al. [2024] characterize the generative leap exponent for leaps ≤ 2 . Defilippis et al. [2025], Kovaˇ cevi´ c et al. [2025] give spectral estimators for the special case when the subspace is fully identified in the first leap of generative exponent ≤ 2 . See Section 2 for further discussion.

Tensor PCA. In the context of Tensor PCA, Montanari and Richard [2014] proposed the Tensor PCA model and presented several algorithms including tensor unfolding. Zheng and Tomioka [2015] proposed a rectangular unfolding algorithm closely related to a single step of our algorithm, and showed it attain the conjectured optimal sample complexity of n = Θ( d 1 ∨ k ⋆ / 2 ) . Dudeja and Hsu [2021] provided statistical query lower bounds for the symmetric and asymmetric Tensor PCA model, and Hopkins et al. [2015, 2017] gave the corresponding lower bound in the low-degree / SOS models. Dudeja and Hsu [2024] provided a comprehensive study of communication lower bounds and efficient algorithms for Tensor PCA and the related problem of Non-Gaussian Component Analysis. Arous et al. [2024] initiated the study of stochastic gradient descent over the Stiefel manifold for the multi-spike Tensor PCA model, showing a time complexity of d 1 ∨ k ⋆ -1 where k ⋆ is the order of the tensor (analogous to the generative exponent).

Chen et al. [2020] show that deep neural networks can simulate unfolding-like algorithms and learn multi-index functions with n = Θ( d ⌈ k/ 2 ⌉ ) where k is the degree of the polynomial approximation to the groundtruth function. This method requires that the groundtruth is close to a polynomial.

Vempala [2010], Klivans et al. [2024] provides the current best known result for learning intersection of k -halfspaces in d -dimensions with n = Θ( d ) . Vempala and Xiao [2012] provide a moment-based algorithm for learning multi-index models when the first leap learns all relevant variables.

While this work was being finalized, we became aware of Diakonikolas et al. [2025b,a], which introduces a similar estimation procedure based on subspace conditioning. They define the class of m -well-behaved multi-index models. For the special case of single index models with generative exponent k ⋆ , Diakonikolas et al. [2025b][Appendix D.2] and Diakonikolas et al. [2025a][Appendix C.3.2] show m = k ⋆ ; we believe a similar equivalence holds also for multi-index models. However the proposed algorithm requires sample complexity n = d O ( k ⋆ ) even in the realizable setting, whereas the algorithm of Damian et al. [2024] and this work, require only n = Θ( d k ⋆ / 2 ) and apply when y is either continuous or discrete. On the other hand, Diakonikolas et al. [2025b,a] algorithms aim for agnostic PAC learning, not just recovery of the subspace, and thus are able to explicitly characterize the dependence in the hidden constant C ( P ) in n ≥ C ( P ) d 1 ∨ k ⋆ / 2 . By building an explicit piecewise constant discretization in the subspace, they explicitly characterize the dependence on r , ϵ , and Lipschitz parameters. We expect that our subspace recovery algorithm can be combined with a discretization algorithm to attain similar guarantees, but with improved dependence on d . We also study several examples of the leap generative exponents in Section 5 including piecewise linear functions (deep ReLU Networks with bias) and general deep neural networks with r -dimensional first hidden layer, improving upon previous results specific to multi-index polynomials and homogeneous piecewise linear functions Chen and Meka [2020], Chen et al. [2022].

Notation h k denotes the normalized k -th Hermite tensor, defined as h k ( u ) := ( -1) k √ k ! ∇ ⊗ k γ d ( u ) γ d ( u ) for u ∈ R d . S ( r, d ) is the Stiefiel manifold of r × d orthogonal matrices, and G ( r, d ) is the Grassman manifold, obtained by quotienting S ( r, d ) by r -dimensional basis transformations. For two subspaces

T ⊆ T ′ , we write T ′ \ T as the orthogonal complement of T in T ′ . For two subspaces T, T ′ we define their distance d ( T, T ′ ) to be ∥ Π T -Π T ′ ∥ op where Π T is the orthogonal projection onto T .

Paper outline. Section 2 recalls the generative exponent for single-index models and shows how to generalize the definition to multi-index models via the leap generative exponent, and discuss the relation to the leap information exponent. Section 3 gives the main computational lower bound result which shows that in the low-degree polynomial framework the multi-index model with leap generative exponent k ⋆ requires n ≥ ˜ Ω( d k ⋆ / 2 ) . Section 4 gives our main algorithm, an iterative spectral method based on a Hermite Kernel U-statistic, that recovers the subspace with the optimal sample complexity. Finally in Section 5, we study the leap generative exponents of several function classes including piecewise linear functions and general neural networks with r -dimensional first hidden layer.

2 The Leap Decomposition

Preliminaries: The Generative Exponent for Single-Index Models We start by recalling the generative exponent for single-index models Damian et al. [2024]. Given ( Z, Y ) drawn from a joint distribution P ∈ P ( R × R ) with first marginal equal to a Gaussian, and such that P = γ 1 ⊗ P y , we define for each integer k ≥ 1 , ζ k := E [ h k ( Z ) | Y ] ∈ L 2 ( P y ) , and k ⋆ = inf { k ; ∥ ζ k ∥ L 2 ( P y ) > 0 } . Equivalently, k ⋆ is the smallest integer k such that there exists a measurable function T : R → R and a mean-zero k -th degree polynomial q such that E [ T ( Y ) q ( Z )] = 0 . The main takeaway from Damian et al. [2024] is that n = Θ( d k ⋆ / 2 ∨ 1 ) is both necessary (under the SQ and the LDP frameworks) and sufficient for recovery of the planted direction 2 .

̸

Subspace Filtration and Leap exponents: We begin by generalizing the coefficients { ζ k } from [Damian et al., 2024]. The key novel ingredient is the notion of subspace filtration , capturing the sequential nature of the multi-index estimation, and which appears in several existing multi-index estimation procedures Abbe et al. [2022, 2023], Bietti et al. [2025], Diakonikolas et al. [2025b]. In essence, we now need to extend the expectations ζ k , which were conditional on the label y , to conditional expectations on an 'augmented label' that includes all the previously estimated directions of the index space. More formally, let S ∈ G ( r ′ , r ) be a subspace, and for z ∈ R r let z S ∈ S denote the orthogonal projection of z onto S . We write ¯ y S := ( z S , y ) ∈ R 1+ r ′ and ¯ z S := z S ⊥ ∈ R r -r ′ .

For any S ∈ G ( r ′ , r ) , we then define:

Intuitively, these tensors capture whether there is any information 'of order k ' that can be captured, given knowledge of the subspace S . When S = ∅ and r = 1 , these definitions reduce to those in Damian et al. [2024]. Finally, we note that these definitions only depend on the joint distribution P of ( Z, Y ) and are independent of the choice of W ⋆ .

Given a subspace S , we define the associated null distribution P S by:

Under P S , ( Y, Z S ) and ¯ Z S have the same marginals as under P , but are independent. The label transformations ζ k appear as the Hermite coefficients of the density ratio d P d P S :

Lemma 1 (Density Ratio expansion) . We have the following formal expansion in L 2 ( P S ) :

This implies the following decomposition of χ 2 ( P || P S ) , whenever this divergence exists:

Lemma 2 (Mutual Information Expansion) . If χ 2 ( P || P S ) < ∞ , χ 2 ( P || P S ) = ∑ k ≥ 1 λ 2 k ( S ) .

̸

Notice that while χ 2 ( P || P S ) may be infinite in some cases, e.g. in deterministic models where Y = σ ( Z ) , the quantities λ k ( S ) are well-defined for all k , since ζ k,S ∈ L 2 ( P ¯ Y S ) 3 . Given this expansion, we can immediately define the leap k ( S ) of a subset S :

Definition 2 (Generative Leap relative to S ) . k ( S ) is the smallest k ≥ 1 such that λ 2 k ( S ) > 0 .

̸

Note that k ( S ) < ∞ so long as P = P S .

The Leap Decomposition: We will define the flag F = {∅ = S 0 ⊊ S 1 ⊊ · · · ⊊ S L = R r } inductively as follows. Given a subspace S i , i ≥ 0 , we define k i +1 := k ( S i ) and S i +1 by:

Here, we have defined the span of a symmetric tensor T ∈ ( R r ) ⊗ k as span( T ) = span [ Mat r,r k -1 [ T ] ] where Mat r,r k -1 [ T ] denotes T reshaped as an r × r k -1 matrix.

Definition 3 (Generative Leap Exponent) . Let k i , i = 1 , . . . , L be defined as above. The generative leap exponent is defined as k ⋆ := max i k i .

We now verify that the Leap decomposition is well-defined, and give a variational representation.

Definition 4. Given two subspaces S ⊊ T , we define the relative leap k ( S, T ) of a subspace S towards T as

In words, the relative leap measures the order of the Hermite tensor needed to 'reach' the subspace T from conditional expectations over y and z S . Observe that we can relate the leaps k ( S ) and k ( S, T ) as k ( S ) = inf T ; S ⊊ T k ( S, T ) .

Proposition 1 (Variational Characterization of Leap Generative Exponent) . The leap decomposition terminates in a finite number of steps L ≤ r . Moreover, we have

Finally, k ⋆ is invariant to rotation: if ˜ P = ( U ⊗ Id) # P where U ∈ O r is any rotation of the model, we have k ⋆ ( ˜ P ) = k ⋆ ( P ) .

Relationship with Information Leap Exponent Finally, we relate the generative leap exponent to the information leap exponent, first introduced in Abbe et al. [2023] (referred to as IsoLeap in the setting of Gaussian input data); see also Bietti et al. [2025] and Dandi et al. [2024a]. Let us first recall its definition in our context. For any S ∈ G ( r ′ , r ) , we define:

By analogy with Definition 2, we define l ( S ) to be the smallest k such that ˜ λ 2 k ( S ) > 0 . Equipped with this object, the information leap exponent is recovered as follows.

Definition 5 (Information Leap Exponent, Abbe et al. [2023], Bietti et al. [2025]) . The information leap exponent of the multi-index model P is given by l ⋆ := max i l i , where l i +1 = l ( ˜ S i ) and ( ˜ S i ) i is defined recursively by ˜ S 0 = ∅ and ˜ S i +1 = ˜ S i ⊕ span[ ˜ Λ l i +1 ( ˜ S i )] .

Let us now relate the Information Leap exponent to the generative leap. We start with a direct generalization of [Damian et al., 2024, Prop 2.6]:

Proposition 2 (Generative and Information Exponents relative to subspaces) . For any subspace S ,

In particular we have k ( S ) ≤ l ( S ) for any subspace S .

In words, the generative exponent relative to a subspace S is the largest k such that E P [ T ( y, z S ) q ( z S ⊥ )] = 0 for any measurable function T and any polynomial q of degree < k . This provides a useful characterization, as illustrated in the examples of Section 5.

As expected, the generative leap is upper bounded by the information leap:

Proposition 3 (Relationship with Leap Information Exponent) . We have k ⋆ ≤ l ⋆ .

Proofs of these results are deferred to Appendix B.

3 Computational Lower Bounds in the Low-Degree Polynomial Class

Let us first establish a computational lower bound for the estimation of a multi-index model. Following Damian et al. [2024], and relying on the fact that detecting planted structure is a necessary byproduct of estimating the index space, we instantiate a hypothesis testing adapted to the leap decomposition.

Given P and its associated leap decomposition (1), we consider ¯ S the subspace of dimension r 0 associated with the generative leap, ie k ⋆ = k ( ¯ S ) . Let ¯ y = ( ¯ Sz, y ) be the effective label, with ¯ y ∈ R r 0 +1 , ¯ W = ¯ S ⊤ W ⋆ the planted subspace associated with ¯ S , and ¯ x = ¯ W ⊥ x ∈ R d -r 0 the effective input. Viewing P as the joint distribution of ( ¯ S ⊥ z, ¯ y ) , we define P ¯ y as the marginal over ¯ y . Note that r 0 < r by definition. We consider the following detection problem, conditional on ¯ W :

H 1 : there is a planted model of dimension ˜ r > r 0 using P as link function and ¯ y as label. Specifically, (¯ x, ¯ y ) ∼ E ˜ W P ˜ W , where P ˜ W (¯ y | ¯ x ) = P (¯ y | ˜ W ⊤ ¯ x ) .

H 0 : there is only planted structure up to dimension r 0 ; i.e., (¯ x, ¯ y ) ∼ P 0 := γ d -r 0 ⊗ P ¯ y .

By considering the likelihood ratio R = d H 1 d H 0 and its orthogonal projection R ≤ D in L 2 ( H 0 ) onto polynomials of degree at most D , one can assess the ability of low-degree polynomials to solve this hypothesis testing problem Bandeira et al. [2022], Hopkins [2018]. Specifically, if ∥R ≤ D ∥ L 2 ( H 0 ) = 1 + o d (1) , then no degreeD polynomial f in the input samples can weakly separate H 0 from H 1 , ie satisfy max { Var 0 [ f ] , Var 1 [ f ] } = O ( | E 0 [ f ] -E 1 [ f ] | 2 ) as d →∞ [Bandeira et al., 2022, Proposition 6.2].

Theorem 1 (Weak separation lower bound) . Consider d ≫ max( r, k ⋆ ) , D = O (log( d ) 2 ) , and n = O ( d k ⋆ / 2 -γ ) for any γ > 0 . Then ∥R ≤ D ∥ L 2 ( H 0 ) = 1 + o d (1) .

In other words, any degreeD polynomial test needs n ≥ ˜ Ω( d k ⋆ / 2 ) samples to weakly detect H 1 from H 0 . Polynomial tests of degree ω (log d ) are considered a powerful step towards ruling out all noise-tolerant polynomial-time algorithms Bandeira et al. [2022], Kunisky et al. [2019]. The proof can be found in Appendix C.

This low-degree lower bound extends the previous LDP lower-bound from the single-index setting Damian et al. [2024]. In that single-index setting, this LDP lower bound agrees with a SQ lower bound of n = Θ( d k ⋆ / 2 ) samples. While it is possible to translate our LDP lower bounds to SQ lower bounds, eg via Brennan et al. [2021], we note that there is a fundamental distinction arising in the multi-index setting, stemming from the inherent inability to perform certain spectral tasks in SQ. Dudeja and Hsu [2021] illustrated this mismatch in the setting of Tensor PCA, where asymmetric structures (such as the ones faced by multi-index model estimation) incur in additional dimension-factors. That said, some SQ lower bounds are known for the multi-index setting. Joshi et al. [2024] establishes SQ lower bounds for the number of queries of order Θ( d k ⋆ ) , and Diakonikolas et al. [2025b,a] obtains sample complexity lower bounds of order Θ( d k ⋆ / 2 ) (where the generative leap is replaced by the equivalent m in their notation), thus matching our LDP lower bounds.

4 Upper Bound via Hermite Kernel U-Statistic

We begin by describing a spectral estimator that works for a single leap. To motivate it, recall that the spectral estimator for single index models in Damian et al. [2024] began by estimating the tensor:

For a suitable label transformation T , the true expectation is proportional to ( w ⋆ ) ⊗ k , so estimating w ⋆ is similar to a single-spike tensor PCA problem. For this problem, the partial trace estimator is an effective way to estimate w ⋆ . This estimator consists in repeatedly contracting indices T ← T [ I ] until you are left with a vector whose expectation is w ⋆ or a matrix whose expectation is w ⋆ ( w ⋆ ) ⊤ . However, this trick does not work in the multi-index setting. For example, consider Gaussian k -parity: y = sign( z 1 · · · z k ) . For this problem, we can compute the population mean of an order k estimator: 4

Thus, this behaves like a symmetric multi-spike tensor PCA problem. For this problem, note that because the { w ⋆ i } are mutually orthogonal, T [ I ] = 0 so taking any partial traces of this tensor will fail to produce a consistent estimator. For standard tensor PCA, this can be solved by tensor unfolding Montanari and Richard [2014]. For example, Zheng and Tomioka [2015] showed it was sufficient to unfold T into a d × d k -1 matrix and compute the left singular vectors. Explicitly if A = Mat ( d,d k -1 ) [ T ] denotes T reshaped as a d × d k -1 matrix, then you can perform a spectral decomposition of AA ⊤ ∈ R d × d and the top eigenvectors will recover the hidden directions.

Returning to the multi-index setting, this would motivate the following estimator. Given n samples { ( x i , y i ) } n i =1 , we define the embedding ϕ , the flattened tensor Φ and the matrix estimator M n by:

Wecan then perform a spectral decomposition of M n . Note that this is exactly equivalent to estimating the tensor 1 n ∑ n i =1 T ( y i ) h k ( x i ) , unfolding it into a d × d k -1 matrix, and computing its left singular vectors. However, this strategy cannot achieve the optimal threshold of n ≳ d k 2 because the 'diagonal' terms dominate the matrix and destroy the concentration. More specifically, we can expand M n as:

̸

For this estimator, one can show that the spikes in E M n get lost in the bulk of the eigenvalues corresponding to ( I ) unless n ≳ d 1 ∨ 2 k -1 3 , which falls short of the optimal threshold d 1 ∨ k 2 . To improve this estimator, we therefore isolate the second term ( II ) :

̸

This is an order 2 matrix U -statistic which only sums over the disjoint pairs i = j . As a result the expectation is preserved: E U n = E Φ E Φ ⊤ and we prove that U n does concentrate to its expectation in operator norm with n ≳ d k/ 2 samples (Theorem 2).

̸

However, it is not true in general that a single label transformation T is enough for E U n to span the entire space when there are multiple leaps i.e. it may be necessary to use a label transformation T 1 to estimate the first direction w ⋆ 1 and T 2 to estimate w ⋆ 2 . Rather than computing the top eigenvector of this matrix U -statistic for each label transformation T i , we could simply add them together into T ( Y ) = [ T 1 ( Y ) , . . . , T m ( Y )] ∈ R m and form an aggregate matrix:

̸

Because T only enters the U -statistic through inner products, we can use the kernel trick and replace it with a general PSD kernel K :

̸

Algorithm 1: A Single Leap

̸

which reduces to the above setting by taking K ( y i , y j ) = ⟨T ( y i ) , T ( y j ) ⟩ . However, by allowing more general kernels K which correspond to 'infinite' embedding vectors T , this allows to automatically average over an 'infinite number' of label transformations. We will show that this allows us to learn the subspace corresponding to the next leap with the optimal sample complexity of n ≳ d k 2 without any knowledge of the multi-index model P . To begin, we prove the following lemma which controls the expectation of this matrix U -statistic:

Lemma 3. If K is integrally strictly positive definite, 5 there exist c ( P , K ) , C ( P , K ) > 0 independent of d such that if S := ( U ⋆ ) ⊤ span[Λ k ] denotes the subspace corresponding to the next leap then

We note that commonly used kernels like the RBF kernel automatically satisfy the assumption in Lemma 3. This implies that if we could estimate the span of E U n , we could recover the next leap. To estimate the span, we use the following theorem which bounds U n -E U n in operator norm:

Theorem 2 (Concentration of U-Statistic) . Let K be a PSD kernel with K ( y, y ) ≤ 1 for all y . Then if n ≳ k d k/ 2 /ϵ + dr k /ϵ 2 , we have that ∥ U n -E U n ∥ op ≤ ϵ with probability at least 1 -exp( -d c ) for an absolute constant c > 0 .

As a corollary, by Davis-Kahan we can recover the subspace up to error ϵ with n ≳ d k/ 2 /ϵ + d/ϵ 2 samples where the hidden constant is independent of d and depends only on the multi-index model P : Corollary 1 (Subspace Recovery) . For any multi-index model P , there exists a constant C ( P , K ) independent of d such that if n ≥ C ( P , K ) [ d k/ 2 ϵ + d ϵ 2 ] then the output S ⊂ R d of Algorithm 1 satisfies d ( S, ( U ⋆ ) T span[Λ k ]) ≤ ϵ with probability at least 1 -exp( -d c ) for an absolute constant c > 0 .

4.1 Iterating over Leaps

Once we have recovered an partial subspace S , which we hope is approximately contained in span[( U ⋆ ) ⊤ ] , we need to continue this process to take the next leap. We can consider the augmented label ¯ Y S = ( Y, Π S x ) . Then X, ¯ Y again form a multi-index model with hidden dimension at most r so we can repeat our matrix U-statistic estimator from the previous section. Note that the kernel K now maps R | S | +1 × R | S | +1 → R . We will denote the resulting kernel by U ( S ) n :

We can directly apply Corollary 1 to show that for any subspace S , we can recover the span of ( U ⋆ ) T Λ k ( S ) up to error ϵ with n ≳ d k/ 2 /ϵ + d/ϵ 2 samples. We will now control the accumulation of errors to show that we can recover the full multi-index model with n ≳ C ( P , K ) d k ⋆ / 2 /ϵ samples:

Lemma 4. If the kernel K is L -Lipschitz, then there exists a constant C ( P , K ) such that the map S → E U ( S ) n is C ( P , K ) L -Lipschitz in operator norm.

A common example of a Lipschitz kernel is the RBF kernel which is 1 /σ -Lipschitz. Therefore if we run this estimator starting with the wrong subspace ˆ S with d ( S, ˆ S ) ≤ ϵ , then the span of our estimator

Algorithm 2: Iterating over Leaps

can only change by C ( P , K ) Lϵ . By iterating this argument, Theorem 2 implies that Algorithm 2 will succeed in recovering span[ U ⋆ ⊤ ] up to error ϵ given n ≳ d k ⋆ / 2 /ϵ + d/ϵ 2 samples:

Theorem 3 (Main Result) . For any multi-index model P , there exists a constant C ( P , K ) independent of d such that if n ≥ C ( P , K ) [ d k ⋆ / 2 ϵ + d ϵ 2 ] then the output S ⊂ R d of Algorithm 2 satisfies d ( S, span [ ( U ⋆ ) ⊤ ] ) ≤ ϵ with probability at least 1 -exp( -d c ) for some c = c ( k ⋆ ) > 0 .

Remark 4. Our main upper bound, Algorithm 2, requires knowledge of the sizes of each leap { k i } and the dimension of each leap { s i } . However, these restrictions can be easily lifted, in the spirit of Dudeja and Hsu [2018]. Using the guarantee in Theorem 2, we could start with k = 1 for each leap and increase k until we detect outlier eigenvalues outside of the √ d k/ 2 /n -bulk. However, for simplicity we have written the algorithm assuming knowledge of both { k i } and { s i } .

Our Algorithm 2 is thus a streamlined version of a subspace conditioned spectral method. While it shares similarities with recent methods in the literature Chen and Meka [2020], Chen et al. [2022], Diakonikolas et al. [2025b,a], Troiani et al. [2024], it crucially relies on a U-statistic in order to reach the optimal sample complexity of d k ⋆ / 2 . An additional feature of our algorithm - that to our knowledge is novel in the literature - is the use of a generic kernel over the already discovered labels, which eliminates the need to perform successive non-parametric regressions during the subspace recovery. At the technical level, the concentration of the U-statistic is a priori challenging due to the heavy tails of the associated Hermite tensors; this is addressed using Gaussian universality results from Brailovskaya and van Handel Brailovskaya and van Handel [2024], with a dedicated analysis in the setting where k ⋆ ≤ 2 to avoid spurious log-factors.

5 Case Studies

We conclude this article by computing the generative leap exponent of representative multi-index models. For some of these models our upper and lower bounds recover known results in the literature, but some are new. For simplicity, we focus here on noiseless models where Y | Z = σ ( Z ) for a given link function σ : R r → R . Proofs for this section can be found in Appendix E.

5.1 Polynomial and Threshold Functions

We start by computing the generative leap for 'classic' multi-index classes given by parities, intersection of half-spaces and polynomials.

Proposition 4 (Generative Leaps for representative models) . We have:

(i) r -Gaussian Parity has k ⋆ = l ⋆ = r .

(ii) Staircase Parity functions have k ⋆ ≤ l ⋆ = 1 ,

(iii) Intersection of halfspaces have k ⋆ ≤ 2 ,

(iv) Polynomials have k ⋆ ≤ 2 .

For r -Gaussian parity, we thus obtain an efficient learning algorithm that requires n = Θ( d r/ 2 ) samples (which is optimal within the LDP class), and is to the best of our knowledge the first result 6

that succeeds with Θ( d r/ 2 ) samples. The sample complexity of learning intersection of half-spaces is thus linear in dimension: this was known since Vempala [2010], Diakonikolas et al. [2017], Klivans et al. [2024], and for polynomials the same conclusion was established in Chen and Meka [2020]. We emphasize that while our results do capture the correct dependency in d , they are not fine-grained enough to provide the correct dependencies in r .

5.2 Piecewise Linear Functions

Piecewise linear continuous functions, in part motivated by ReLU architectures, have been extensively studied in the context of Gaussian Multi-index models Chen et al. [2022, 2023], Diakonikolas and Kane [2024]. When σ is 1 -homogeneous, as in bias-free ReLU networks, it is not hard to see that k ⋆ ≤ 2 , by considering diverging level sets { z ; | σ ( z ) | ≥ λ } with λ →∞ . Here we extend this result to the general piece-wise linear setting, including arbitrary ReLU networks with non-zero biases.

Proposition 5 (Generative Leap for Piecewise Linear Functions) . If σ is continuous and piece-wise linear then k ⋆ ( σ ) ≤ 2 .

The proof exploits the analytic properties of Hermite functions, i.e. functions of the form f ( z ) = p ( z ) γ ( z ) . As an immediate corollary, our Algorithm from Section 4 learns arbitrary ReLU networks low-rank arbitrary ReLU networks in the proportional regime n = Θ( d ) : This improves the result of Chen et al. [2022] by allowing biases. Once the subspace is recovered, one could 'upgrade' to PAC learning the model using a standard non-parametric method, by regressing over the covariates z = S ⊤ x . This would incur in an additional sample complexity with potentially exponential dependencies in r and 1 ϵ , but, importantly, independent of d .

5.3 Generative Leap under Linear Transformations

An important feature of the generative leap exponent is that the statement ' k ⋆ ( P ) ≤ k " is an 'open' property, meaning that one should expect the leap exponent to be preserved or reduced by slightly perturbing the distribution P . We formalize this intuition in the following result which shows that for almost all weight matrices, the generative leap is ≤ 2 .

̸

Proposition 6 (Generative Leap under linear transformations) . Let σ ( z ) : R r → R ∈ L 2 ( γ r ) , σ = C , and let M r denote the set of r × r real matrices.

(i) For Θ ∈ M r , define y Θ = σ (Θ ⊤ z ) . Then ( z, y Θ ) ∼ P Θ satisfies k ⋆ ( P Θ ) ≤ 2 for every Θ , except possibly for a set of r 2 -dimensional Lebesgue measure zero,

(ii) Assume that ( z, σ ( z )) ∼ P has a single leap with generative exponent k ⋆ . Let Γ : D ⊆ R s →M r be any analytic map such that I r ∈ Im(Γ) and Γ( θ ) is invertible for all θ ∈ D . For θ ∈ D , define y θ = σ (Γ( θ ) ⊤ z ) . Then ( z, y θ ) ∼ P θ satisfies k ⋆ ( P θ ) ≤ k ⋆ for every θ , except possibly for a set of s -dimensional Lebesgue measure zero.

5.4 Shallow Neural Networks

Finally, we study two layer neural networks of the form σ ( z ) = ∑ j ρ j ( z · θ j ) . When the θ j are orthogonal, estimating the index space requires estimating each neuron, and k ⋆ ( σ ) = k ⋆ ( ρ ) :

Proposition 7 (Generative Leap for Orthogonal Weights) . Let y = ∑ r j =1 a j ρ ( z j ) . Then k ⋆ ≥ k ⋆ ( ρ ) . Moreover, if all moments of ρ ( Z ) exist, then k ⋆ = k ⋆ ( ρ ) .

We can extend this result to almost all networks with unit norm, linearly independent columns:

Corollary 2 (Non-orthogonal, invertible weights) . Let y V = ∑ r j =1 a j ρ ( v ⊤ j z ) with ∥ v j ∥ = 1 . Then k ⋆ V ≤ k ⋆ ( ρ ) for all V , except possibly for a set of r ( r -1) -dimensional measure 0 .

An interesting question left for future work is whether this uniform control of the generative exponent by k ⋆ ( ρ ) for any V could be extended to general link functions; in other words whether the exclusion of these zero-measure sets is necessary in Proposition 6.

References

Abbe, E. Boix-Adsera, M. S. Brennan, G. Bresler, and D. Nagaraj. The staircase property: How hierarchical structure can guide deep learning. Advances in Neural Information Processing Systems , 34:26989-27002, 2021.

Abbe, E. B. Adsera, and T. Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory , pages 4782-4887. PMLR, 2022.

Abbe, E. B. Adsera, and T. Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory , pages 2552-2623. PMLR, 2023.

Arnaboldi, Y. Dandi, F. Krzakala, L. Pesce, and L. Stephan. Repetita iuvant: Data repetition allows sgd to learn high-dimensional multi-index functions, 2024. URL https://arxiv.org/ abs/2405.15459 .

B. Arous, R. Gheissari, and A. Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. Journal of Machine Learning Research , 22(106):1-51, 2021.

B. Arous, C. Gerbelot, and V. Piccolo. Stochastic gradient descent in high dimensions for multi-spiked tensor pca, 2024. URL https://arxiv.org/abs/2410.18162 .

Ba, M. A. Erdogdu, T. Suzuki, Z. Wang, D. Wu, and G. Yang. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. arXiv preprint arXiv:2205.01445 , 2022.

S. Bandeira, A. El Alaoui, S. Hopkins, T. Schramm, A. S. Wein, and I. Zadik. The franzparisi criterion and computational trade-offs in high dimensional statistics. Advances in Neural Information Processing Systems , 35:33831-33844, 2022.

S. Bandeira, M. T. Boedihardjo, and R. van Handel. Matrix concentration inequalities and free probability. Inventiones mathematicae , 234(1):419-487, June 2023. ISSN 1432-1297. doi: 10.1007/s00222-023-01204-6. URL http://dx.doi.org/10.1007/s00222-023-01204-6 .

Ben Arous, R. Gheissari, and A. Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. Journal of Machine Learning Research (JMLR) , 22: 106-1, 2021.

Bietti, J. Bruna, C. Sanford, and M. J. Song. Learning single-index models with shallow neural networks. Advances in neural information processing systems , 35:9768-9783, 2022.

Bietti, J. Bruna, and L. Pillaud-Vivien. On learning gaussian multi-index models with gradient flow. Communications in Pure and Applied Mathematics , 2025.

Brailovskaya and R. van Handel. Universality and sharp matrix concentration inequalities. Geometric and Functional Analysis , 34(6):1734-1838, 2024.

Brennan, G. Bresler, S. Hopkins, J. Li, and T. Schramm. Statistical query algorithms and low-degree tests are almost equivalent. In Conference on Learning Theory , 2021.

Bruna and D. Hsu. Survey on algorithms for multi-index models. arXiv preprint arXiv:2504.05426 , 2025.

Chen, Y. Bai, J. D. Lee, T. Zhao, H. Wang, C. Xiong, and R. Socher. Towards understanding hierarchical learning: Benefits of neural representations. Advances in Neural Information Processing Systems , 2020.

Chen and R. Meka. Learning polynomials in few relevant dimensions. In Conference on Learning Theory , pages 1161-1227. PMLR, 2020.

Chen, A. R. Klivans, and R. Meka. Learning deep relu networks is fixed-parameter tractable. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 696-707. IEEE, 2022.

Chen, Z. Dou, S. Goel, A. Klivans, and R. Meka. Learning narrow one-hidden-layer relu networks. In The Thirty Sixth Annual Conference on Learning Theory , pages 5580-5614. PMLR, 2023.

D. Cook. SAVE: a method for dimension reduction and graphics in regression. Communications in Statistics - Theory and Methods , 29(9-10):2109-2121, 2000.

D. Cook and B. Li. Dimension reduction for conditional mean in regression. The Annals of Statistics , 30(2):455-474, 2002.

Damian, J. Lee, and M. Soltanolkotabi. Neural networks can learn representations with gradient descent. In Conference on Learning Theory , 2022.

Damian, E. Nichani, R. Ge, and J. D. Lee. Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models. Advances in Neural Information Processing Systems , 36:752-784, 2023.

Damian, L. Pillaud-Vivien, J. Lee, and J. Bruna. Computational-statistical gaps in gaussian singleindex models. In The Thirty Seventh Annual Conference on Learning Theory , pages 1262-1262. PMLR, 2024.

Dandi, F. Krzakala, B. Loureiro, L. Pesce, and L. Stephan. How two-layer neural networks learn, one (giant) step at a time. Journal of Machine Learning Research , 25(349):1-65, 2024a.

Dandi, E. Troiani, L. Arnaboldi, L. Pesce, L. Zdeborova, and F. Krzakala. The benefits of reusing batches for gradient descent in two-layer networks: breaking the curse of information and leap exponents. In Proceedings of the 41st International Conference on Machine Learning , pages 9991-10016, 2024b.

Daniely, I. Mehalel, and E. Mossel. Online learning of neural networks. arXiv preprint arXiv:2505.09167 , 2025.

H. de la Peña and E. Giné. Decoupling of U-Statistics and U-Processes , pages 97-152. Springer New York, New York, NY, 1999. ISBN 978-1-4612-0537-1. doi: 10.1007/978-1-4612-0537-1_3. URL https://doi.org/10.1007/978-1-4612-0537-1_3 .

Defilippis, Y. Dandi, P. Mergny, F. Krzakala, and B. Loureiro. Optimal spectral transitions in high-dimensional multi-index models. arXiv preprint arXiv:2502.02545 , 2025.

Diakonikolas and D. M. Kane. Efficiently learning one-hidden-layer relu networks via schur polynomials. In The Thirty Seventh Annual Conference on Learning Theory , pages 1364-1378. PMLR, 2024.

Diakonikolas, D. M. Kane, and A. Stewart. Learning geometric concepts with nasty noise, 2017. URL https://arxiv.org/abs/1707.01242 .

Diakonikolas, G. Iakovidis, D. M. Kane, and L. Ren. Algorithms and sq lower bounds for robustly learning real-valued multi-index models, 2025a. URL https://arxiv.org/abs/2505.21475 .

Diakonikolas, G. Iakovidis, D. M. Kane, and N. Zarifis. Robust learning of multi-index models via iterative subspace approximation, 2025b. URL https://arxiv.org/abs/2502.09525 .

Dudeja and D. Hsu. Learning single-index models in gaussian space. In Thirty-First Annual Conference on Learning Theory , 2018.

Dudeja and D. Hsu. Statistical query lower bounds for tensor pca. Journal of Machine Learning Research , 22(83):1-51, 2021.

Dudeja and D. Hsu. Statistical-computational trade-offs in tensor PCA and related problems via communication complexity. The Annals of Statistics , 52(1):131-156, 2024.

Hopkins. Statistical inference and the sum of squares method . PhD thesis, Cornell University, 2018.

B. Hopkins, J. Shi, and D. Steurer. Tensor principal component analysis via sum-of-square proofs. In Conference on Learning Theory , pages 956-1006. PMLR, 2015.

B. Hopkins, P. K. Kothari, A. Potechin, P. Raghavendra, T. Schramm, and D. Steurer. The power of sum-of-squares for detecting hidden structures. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 720-731. IEEE, 2017.

Hristache, A. Juditsky, J. Polzehl, and V. Spokoiny. Structure adaptive approach for dimension reduction. The Annals of Statistics , 29(6):1537-1566, 2001.

Joshi, T. Misiakiewicz, and N. Srebro. On the complexity of learning sparse functions with statistical and gradient queries. Advances in Neural Information Processing Systems , 37:103198103241, 2024.

Klivans, K. Stavropoulos, and A. Vasilyan. Learning intersections of halfspaces with distribution shift: Improved algorithms and sq lower bounds. In The Thirty Seventh Annual Conference on Learning Theory , pages 2944-2978. PMLR, 2024.

R. Klivans, R. O'Donnell, and R. A. Servedio. Learning geometric concepts via gaussian surface area. In 49th Annual IEEE Symposium on Foundations of Computer Science , pages 541-550, 2008.

Kovaˇ cevi´ c, Y. Zhang, and M. Mondelli. Spectral estimators for multi-index models: Precise asymptotics and optimal weak recovery. arXiv preprint arXiv:2502.01583 , 2025.

Kunisky, A. S. Wein, and A. S. Bandeira. Notes on computational hardness of hypothesis testing: Predictions using the low-degree likelihood ratio. arXiv preprint arXiv:1907.11636 , 2019.

D. Lee, K. Oko, T. Suzuki, and D. Wu. Neural network learns low-dimensional polynomials with sgd near the information-theoretic limit. Advances in Neural Information Processing Systems , 37: 58716-58756, 2024.

K.-C. Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association , 86(414):316-327, 1991.

Mityagin. The zero set of a real analytic function. arXiv preprint arXiv:1512.07276 , 2015.

Montanari and E. Richard. A statistical model for tensor pca. Advances in neural information processing systems , 27, 2014.

Mossel, R. O'Donnell, and R. P. Servedio. Learning juntas. In Proceedings of the thirty-fifth annual ACM symposium on Theory of computing , pages 206-212, 2003.

Ren and J. D. Lee. Learning orthogonal multi-index models: A fine-grained information exponent analysis. arXiv preprint arXiv:2410.09678 , 2024.

Ren, E. Nichani, D. Wu, and J. D. Lee. Emergence and scaling laws in sgd learning of shallow neural networks. arXiv preprint arXiv:2504.19983 , 2025.

K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, and G. R. G. Lanckriet. Hilbert Space Embeddings and Metrics on Probability Measures. Journal of Machine Learning Research (JMLR) , 11:1517-1561, 2010.

Troiani, Y. Dandi, L. Defilippis, L. Zdeborová, B. Loureiro, and F. Krzakala. Fundamental limits of weak learnability in high-dimensional multi-index models, 2024. URL https://arxiv.org/ abs/2405.15480 .

S. Vempala. Learning convex concepts from gaussian distributions with pca. In 51st Annual IEEE Symposium on Foundations of Computer Science , pages 124-130, 2010.

S. Vempala and Y. Xiao. Structure from local optima: Learning subspace juntas via higher order pca, 2012. URL https://arxiv.org/abs/1108.3329 .

Xia. A multiple-index model and dimension reduction. Journal of the American Statistical Association , 103(484):1631-1640, 2008.

Xia, H. Tong, W. K. Li, and L.-X. Zhu. An adaptive estimation of dimension reduction space. Journal of the Royal Statistical Society Series B: Statistical Methodology , 64(3):363-410, 2002.

Zheng and R. Tomioka. Interpolating convex and non-convex tensor decompositions via the subspace norm. Advances in Neural Information Processing Systems , 28, 2015.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We define the generative leap in Definition 3. We prove a sample complexity lower bound in Theorem 1 and we prove a matching upper bound in Theorem 3.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proof of Theorem 1 can be found in Appendix C. The proof of Theorem 3 can be found in Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper does not contain experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not provide experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper does not provide experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper does not provide experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper does not provide experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research conforms to the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is purely theoretical and is focused on understanding the fundamental limitations of learning with synthetic Gaussian data.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We are not releasing data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use any original assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not conduct any crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: This paper did not use LLMs as any important, original, or non-standard component.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Conclusions

In this work, we have extended the generative exponent k ⋆ to the general class of Gaussian multi-index models, and established a tight sample complexity n = Θ( d k ⋆ / 2 ∧ 1 ) for learning their associated index space under no prior knowledge of the link function. We provide a lower bound based on the low-degree polynomial framework, and a matching upper bound obtained with a novel spectral method that incrementally reveals directions of the index space from a kernel U-statistic. The resulting upper bound recovers and extends several dedicated estimation procedures for specific families of multi-index models, such as ReLU networks or intersection of half-spaces.

There are several avenues for future work. First, this paper focuses on the simple setting of isotropic Gaussian data. Extending both the information leap and generative leap to more complicated data distributions is left to future work. Next, we focus on deriving estimators that work with minimal information about the multi-index model P , and which succeed with the optimal sample complexity in the ambient dimension d . As a result, our sample complexity guarantees scale with constants C ( P ) which could potentially be exponentially large in the hidden dimension r . Finally, we focus primarily on subspace estimation, as it is a requirement for full end-to-end learning.

B Proofs of Section 2

Proof of Lemma 1. This is a direct generalization of [Damian et al., 2024, Lemma D.1]. The k -th Hermite expansion of the likelihood ratio, viewed as a function of the label ¯ Y S , is directly

and thus, in L 2 ( P S ) , we have

■

Proof of Lemma 2. By orthogonality of Hermite polynomials:

Proof of Proposition 1. The first statement follows immediately from the definition. To prove 3, we need to show that k ⋆ ≤ max j k ( R j , R j +1 ) for any flag F . Let ¯ S the subspace associated with the generative leap k ⋆ , and let j ′ be the largest index such that R j ′ ⊆ ¯ S .

We claim that for any k and any pair of subspaces T ⊆ T ′ , we have span (Λ k ( T )) ⊆ T ′ ⊕ span (Λ k ( T ′ )) . Indeed, writing Y ′ ∈ T ′ as Y ′ = ( Y, ˜ Y ) with Y ∈ T and ˜ Y ∈ T ′ \ T , we have ζ k,T ( Y ) = E ˜ Y ζ k,T ′ ( Y, ˜ Y ) when restricted to ( T ′ ) ⊥ (a subset of T ⊥ ). Now, suppose towards contradiction that k j ′ = k ( R j ′ , R j ′ +1 ) < k ⋆ . Since R j ′ ⊆ ¯ S , we have span (Λ k ( R j ′ )) ⊆ ¯ S ⊕ span (Λ k ( ¯ S )) for k ≤ k j ′ . But from the definition of k ⋆ we have span (Λ k ( ¯ S )) = ∅ for k ≤ k j ′ , which implies that R j ′ +1 ⊆ ¯ S , which is a contradiction.

Finally, we verify that the generative leap is invariant to rotation by observing that Λ k are covariant to rotations, and therefore their associated spans preserve the same dimensions for all k .

■

Proof of Proposition 2. The proof is an extension of [Damian et al., 2024, Prop 2.6].

To prove k ( S )[ P ] ≤ inf T ∈ L 2 ( P ¯ y S ) l ( S )[(Id z ⊗T y ) # P ] , consider k < k ( S ) and any T ∈ L 2 ( P ¯ y ) . We have

̸

since ζ k,S = 0 . To prove k ( S )[ P ] ≥ inf T ∈ L 2 ( P ¯ y ) l ( S )[(Id ¯ z ⊗T ¯ y ) # P ] , consider T = ( ζ k ( S ) ,S ) β , where β is a multiindex such that ( ζ k ( S ) ,S ) β = 0 (this β must exist by definition of k ( S ) ). We verify that

̸

which shows that ˜ ζ k ( S ) ,S = 0 for the model with label transformation T .

■

Proof of Proposition 3. Consider the flag ˜ F = {∅ , ˜ S 1 , . . . , ˜ S J = R r } associated with the leap information exponent. We claim that k ( ˜ S j , ˜ S j +1 ) ≤ l ( ˜ S j ) for all j ∈ [ J ] , or equivalently that

for k ≤ l ( ˜ S j ) . Indeed, observe that ˜ ζ k,S = E Y [ Y ζ k,S ] . As a consequence, from Proposition 1 we have that k ⋆ ≤ max j l ( ˜ S j ) = l ⋆ .

■

C Proofs of Section 3

Proof of Theorem 1. Let R ˜ W := d P ˜ W d P 0 denote the likelihood ratio conditioned on ˜ W . We begin by computing the full likelihood ratio:

Then by Lemma 1, we can expand this as

We will isolate the low degree part with respect to { ¯ x 1 , . . . , ¯ x n } , which we denote by R ≤ D . To compute this, we need to switch the product and the summation:

We note that each term on the right hand side is a polynomial in ¯ x 1 , . . . , ¯ x n of degree p which is orthogonal to all polynomials of degree less than p . Therefore R ≤ D is given by:

We can now use the orthogonality property of Hermite polynomials to compute the norms with respect to the null distribution P 0 . If if ˜ W, ˜ W ′ are independent draws from the prior on ˜ W then:

For a pair Σ , ˜ Σ of operators where Σ is PSD, observe that

thus

Now, let M = ˜ W ⊤ ˜ W ′ ∈ R r × r . We have the following control on the Hermite correlation term:

Lemma 5.

Let z be a random variable with distribution ∥ ˜ W ⊤ ˜ W ′ ∥ , where ˜ W, ˜ W ′ are drawn independently from the uniform prior on ˜ W , and let P ≤ D be the projection operator onto polynomials of degree at most D in z . Note that z is subgaussian satisfying P ( b √ 1 d ≤ z ≤ a √ 1 d ) ≤ 1 -c r b -¯ c r e -a 2 / 4 for explicit constants c r , ¯ c r ; see eg [Bietti et al., 2025, Lemma 3.14]. Then we can upper bound the above expression as:

By linearity of expectation and of the projection operator P ≤ D , we can expand this using the binomial theorem:

We can further upper bound this expression by using that λ 2 k ≤ ( r + k -1 k ) (Lemma 14). Plugging this in for k ≥ k ⋆ gives:

where the last line follows from Lemma 6. Finally, since z is Θ( √ 1 /d ) -subgaussian, we have E [ z jk ⋆ ] ≲ ( jk ⋆ /d ) jk ⋆ / 2 .

Now, if n = O ( d k ⋆ / 2 -γ ) with γ > 0 and D = O ((log d ) 2 ) , we have

■

Proof of Lemma 5. Let M ∈ R d k × d k be the matrix representation of E [ h k ( ˜ W ⊤ ¯ x ) ⊗ h k (( ˜ W ′ ) ⊤ ¯ x )] . Let H k ⊂ L 2 ( R r , γ ) be the space spanned by harmonics of degree k . Observe that for f, ˜ f ∈ H k , f = ∑ | β | = k c β h β , ˜ f = ∑ | β | = k ˜ c β h β with c β = ⟨ f, h β ⟩ , ˜ c β = ⟨ ˜ f, h β ⟩ we have

where P W f ( x ) = f ( W ⊤ x ) . We deduce that M is the 'averaging operator' A M from [Bietti et al., 2025, Definition 1.1], restricted at harmonic k . From the SVD of M = U Λ V ⊤ , we have [Bietti et al., 2025, Corollary 2.8] that

with λ β = ∏ j λ β j j . We thus conclude that ∥ M ∥ op = λ k max = ∥ M ∥ k . ■

Lemma 6. Let z = ∥ W ⊤ W ′ ∥ op , where W,W ′ are drawn iid from the Haar measure of S ( r, d ) . Then, for l, ˜ l ≤ d/ 4 , we have

Proof. The proof is adapted from [Damian et al., 2023, Lemma 26] to the r > 1 setting. From [Bietti et al., 2025, Eq (197)], the joint distribution of singular values 0 ≤ λ r ≤ λ r -1 · · · ≤ λ 1 of M is given by

with Z r,d = Γ 2 r ( r/ 2) π r 2 / 2 Γ r (( d -r ) / 2) Γ r ( d/ 2) . We have

From λ 1 ≤ 1 we have 1 -λ 2 1 ≤ 2(1 -λ 1 ) so (1 -λ 1 ) -˜ l ≤ 2 ˜ l (1 -λ 2 1 ) -˜ l , thus

where ˜ z is the largest singular value of M = W ⊤ W ′ with W,W ′ ∈ S ( r, d -2 ˜ l ) . For d ≫ 1 , we thus conclude that E ( z l (1 -z ) -˜ l ) ≲ E z [ z l ] . ■

D Proofs of Section 4

Lemma 3. If K is integrally strictly positive definite, 7 there exist c ( P , K ) , C ( P , K ) > 0 independent of d such that if S := ( U ⋆ ) ⊤ span[Λ k ] denotes the subspace corresponding to the next leap then

Proof. Let K be the kernel operator:

Using that 〈 E [ h k ( X ) | Y ] , v ⊗ k 〉 = 〈 ζ k ( Y ) , u ⊗ k 〉 where u = U ⋆ v ∈ R r , we have that for any v :

First, because K ( y, y ) ≤ 1 we have that ∥K∥ op ≤ 1 so this is upper bounded by

Therefore E M n ⪯ C ( P , K )Π S with C ( P , K ) = λ 2 k . Next, let v ∈ S with ∥ v ∥ = 1 so that ζ k ( Y )[ v ] = 0 ∈ L 2 ( P y ) . Then because K is injective we have that

Therefore by compactness, if C ( P , K ) denotes the minimum value of c ( v ) over the unit vectors in S , we have that C ( P , K ) > 0 . In addition we have that E M n ⪰ C ( P , K )Π S which completes the proof. ■

Theorem 2 (Concentration of U-Statistic) . Let K be a PSD kernel with K ( y, y ) ≤ 1 for all y . Then if n ≳ k d k/ 2 /ϵ + dr k /ϵ 2 , we have that ∥ U n -E U n ∥ op ≤ ϵ with probability at least 1 -exp( -d c ) for an absolute constant c > 0 .

Proof. The cases k = 1 , 2 are deffered to Proposition 8 so we will assume that k > 2 . Note that by the standard decoupling argument ([de la Peña and Giné, 1999, Theorem 3.4.1]), it suffices to control the tails of the decoupled U -statistic:

̸

where { ( x ′ i , y ′ i ) } n i =1 are an i.i.d. copy of { ( x i , y i ) } n i =1 . We will begin by applying Corollary 3 with respect to the randomness in { ( x i , y i ) } n i =1 , treating the replicas { ( x ′ i , y ′ i ) } n i =1 as fixed. Define

̸

where Z i = ϕ ( x i ) V ′ i ( y i ) T . Then:

so that

̸

Taking p/ 2 norms and using Lemma 9 gives for p ≥ r log r :

Next we will compute σ ∗ ( Z i ) :

by the same argument as above. Therefore applying Corollary 3 gives that if c = k -2 2( k +4) ≥ 1 14 then for p ≤ d c ,

Now let E ′ denote the expectation with respect to the replicas { ( x ′ i , y ′ i ) } n i =1 . Then by Corollary 3:

Now we decompose:

Because | K ( Y, y ′ j ) | ≤ 1 , we can use Lemma 8 and a standard symmetrization argument to show that the second term has p -norms bounded by O ( √ max( d, d k -1 ) /n ) for p < d c . For the first term we have

Combining everything and applying Markov's inequality gives that with probability at least 1 -poly( n ) e -d c ,

■

Proposition 8. If k ≤ 2 and n ≥ d , we have with probability at least 1 -2 ne -d ,

Proof. Wecan use [de la Peña and Giné, 1999, Theorem 3.4.1] to reduce the problem to concentrating:

̸

̸

where D ′ = { ( x ′ i , y ′ i ) } n i =1 are an i.i.d. copy of D = { ( x i , y i ) } n i =1 . If we define V ′ i ( y ) := 1 n -1 ∑ j = i ϕ ′ j K ( y, y ′ j ) , we can rewrite this as:

Now let R be a truncation radius to be chosen later and let ρ i := 1 ∥ V ′ i ( y i ) ∥ op ≤ R . Then define:

First, we have that for any unit vectors u, v and any p ≥ r 2 ,

Therefore the summands in u T ˜ M n v are subexponential so by Bernstein's inequality we have that with probability at least 1 -δ over the randomness in D ,

We can now union bound over a 1 / 4 -net of S d -1 to get that with probability at least 1 -δ over D ,

Taking δ = e -cd and using n ≥ d gives that with probability at least 1 -e -cd ,

Next, note that for any fixed unit vectors u, v (possibly degenerate if k = 1 ), u T ϕ ′ j vK ( y, y ′ j ) is a sub-exponential random variable so by Bernstein's inequality we have with probability at least 1 -δ over the randomness in { ( x ′ i , y ′ i ) } ,

Taking a union bound over a 1 / 4 -net of S d -1 gives that with probability at least 1 -δ over the randomness in D ′ ,

In addition, as in the proof of Theorem 2, we have that ∥ E V ′ i ( y ) ∥ op ≲ r k/ 2 . Therefore if we take R = Cr k/ 2 for a sufficiently large constant C , we have that with probability at least 1 -e -cd that ∥ V i ( Y ) ∥ op ≤ R . Therefore with probability at least 1 -ne -cd , ˜ M n = M n . Furthermore, if 1 -ne -cd > 0 so that the theorem is not vacuous,

Putting everything together gives that with probability at least 1 -2 ne -cd ,

Putting it all together we have that:

In addition, by the induction hypothesis, d ( S i , ( U ⋆ ) T S ⋆ i ) ≤ C ( P , K ) [ d k/ 2 n + r k/ 2 √ d n ] . Therefore,

and the result again follows from the Davis-Kahan inequality.

■

■

Lemma 4. If the kernel K is L -Lipschitz, then there exists a constant C ( P , K ) such that the map S → E U ( S ) n is C ( P , K ) L -Lipschitz in operator norm.

Proof. Let Z ( Y ; X ) := Mat (1 ,k ) [ ζ k ( Y ; X S ⊕ S ′ )] . Then,

where

Note that

Then by Holder's inequality,

■

Theorem 3 (Main Result) . For any multi-index model P , there exists a constant C ( P , K ) independent of d such that if n ≥ C ( P , K ) [ d k ⋆ / 2 ϵ + d ϵ 2 ] then the output S ⊂ R d of Algorithm 2 satisfies d ( S, span [ ( U ⋆ ) ⊤ ] ) ≤ ϵ with probability at least 1 -exp( -d c ) for some c = c ( k ⋆ ) > 0 .

Proof. Recall the leap decomposition F = {∅ = S ⋆ 0 ⊊ S ⋆ 1 ⊊ · · · ⊊ S ⋆ L = R r } . We will prove by induction that for any ϵ > 0 , there exists a constant C ( P , K ) such that the output S i of Algorithm 2 at step i satisfies d ( S i , ( U ⋆ ) T S ⋆ i ) with high probability whenever

Note that for i = 1 the result is implied directly by Corollary 1. Now assume the result for i > 1 . By Lemma 3

In addition we have by Theorem 2,

D.1 Auxiliary Lemmas for Concentration

We will start with this simple inequality on the Frobenius norm of a Hermite tensor: Lemma 7. ∥ h k ( X ) ∥ F ≲ k ∥ X ∥ k + d k/ 4 .

Proof. We will use the identity:

Therefore,

■

We can use this to concentrate sums of ∑ n i =1 c i ϕ ( x i ) in operator norm:

Lemma 8. There exists an absolute constant C k such that if n = d 1+ ϵ with ϵ > 0 and for any constants c i with | c i | ≤ 1 and p = d c where c = min(1 , ϵ/ 4) ,

Proof. Note that for k = 1 , 2 this follows from the standard bounds for a Gaussian covariance matrix ( k = 2) and the norm of a Gaussian vector ( k = 1 ). Therefore we will assume k > 2 . We will begin by computing σ ∗ ( ϕ ) :

Next, ∥ ϕ ∥ op ≤ ∥ ϕ ∥ F ≲ k ∥ X ∥ k + d k/ 4 . Therefore the p -norms of ∥ ϕ ∥ op are bounded by d k/ 2 for any p ≤ d . Plugging this into Lemma 12 gives that for p ≤ d ,

Plugging in p = d c gives that the second and third terms are dominated by the first which completes the proof. ■

We will also use the following simple lemma:

Lemma 9. Let ( X,Y ) follow a Gaussian multi-index model with hidden dimension r . Then for any k -tensor-valued random variable F ( Y ) ,

Proof. Without loss of generality we can assume p is even. Now ⟨ h k ( X ) , f ( Y ) ⟩ p is a polynomial of degree kp in X . Therefore by Lemma 15,

Taking p th roots gives:

Lemma 10 (Gaussian hypercontractivity) . Let f be a polynomial of degree k and let X ∼ N (0 , I d ) . Then for p ≥ 2 ,

Lemma 11. Let X,Y be random variables with ∥ Y ∥ p ≤ Bp k/ 2 for

For any mean zero random matrix Y we define:

For non-centered matrices, we define σ ( Y ) := σ ( Y -E Y ) and σ ∗ ( Y ) := σ ∗ ( Y -E Y ) .

We will rely on the following simple corollary of [Brailovskaya and van Handel, 2024, Theorem 2.6]:

Lemma 12. Let Y = ∑ n i =1 Z i where Z i are mean zero independent random matrices. Assume that for all i , P [ ∥ Z i ∥ > R ] ≤ δ . Then there exists an absolute constant C such that for any t ≥ 0 , with probability at least 1 -nδ -de -t ,

Proof. Define ˜ Z i := Z i 1 ∥ Z i ∥ 2 ≤ R and let ˜ Y := ∑ n i =1 ˜ Z i . Then,

and similarly for σ ∗ . In addition, by definition, ∥ ∥ ∥ ˜ Z i ∥ ∥ ∥ ≤ R . Therefore, by [Brailovskaya and van Handel, 2024, Theorem 2.6] and [Bandeira et al., 2023, Lemma 4.10], there exists a constant C such that for any t ≥ 0 , with probability at least 1 -de -t ,

Next, note that

Now if δ > 1 /n , then 1 -nδ < 0 so the result is trivially true. Otherwise, ∥ E Y -E ˜ Y ∥ op ≤ σ ∗ ( Y ) . Finally, as ˜ Y = Y on the event that max i ∥ Z i ∥ ≤ R , a union bound completes the proof. ■

We will use the following simple lemmas about σ, σ ∗ :

Lemma 13. For any random matrix A ∈ R d × s , σ ( A ) 2 ≤ max( d, s ) σ ∗ ( A ) 2 .

Then,

Proof. Without loss of generality we can assume E [ A ] = 0 . Expanding the definition gives:

First,

Performing the same calculation for A ⊤ in place of A gives that ∥ ∥ E [ AA ⊤ ] ∥ ∥ ≤ dσ ∗ ( A ) 2 . Combining these inequalities gives the desired result. ■

Corollary 3. Let Y = 1 n ∑ N i =1 Z i where Z i ∈ R d × d are mean zero independent random matrices. Assume that for some R,k , ∥ Z i ∥ op ≤ Rt k/ 2 with probability at least 1 -e -t for all t ≥ 0 . Then if n = d 1+ ϵ with ϵ > 0 and c = min(1 , ϵ k +4 ) , then for all p ≤ d c ,

where C is an absolute constant.

Proof. First by a union bound, we have that max i ∥ Z i ∥ 2 ≲ Rt k with probability at least 1 -ne -t . Substituting this and Lemma 13 into Lemma 12 gives that with probability at least 1 -2 ne -t ,

We can factorize this by pulling out the √ d/n and using n ≥ d 1+ ϵ :

We can convert this to an p -norm bound for p ≥ log n

Now if p = d c where c = min(1 , ϵ k +4 ) then the error terms are all less than 1 so we are done. ■

Proof. By Jensen's inequality and Gaussian hypercontractivity we have

Dividing by k ! to revert to the normalized Hermite polynomials { h k } completes the proof. ■

We will now bound the low-degree density ratio between the joint distribution of ( X,Y ) and the null distribution P 0 := P X ⊗ P Y :

Lemma 15. Let P 0 := P X ⊗ P Y be the null distribution and let P ≤ D denote the orthogonal projection onto polynomials in X of degree at most D . Then:

and

Proof. Note that the density ratio is invariant to X conditioned on Z so we can Hermite expand directly in Z :

which implies that the Hermite coefficients of d P d P 0 in Z are given by ζ k . For the second equality, we have by Lemma 14:

E Proofs of Section 5

Proof of Proposition 4. We write y = σ ( z ) to denote the deterministic link functions above.

Let S = { z ∈ R r ; σ ( z ) = +1 } . Let k < r and consider E [ h k ( z ) | z ∈ S ] = 2 E [ h k ( z ) 1 ( z ∈ S )] . Any coordinate of this tensor corresponds to a multivariate Hermite polynomial h β 1 ( z 1 ) . . . h β r ( z r ) , with β 1 + · · · + β r = k . Since k < r , there must exist a coordinate j s.t. β j = 0 . By noting that E z j 1 ( z ∈ S ) ≡ 1 and E z -j [ h β 1 ( z 1 ) . . . h β r ( z r )] = 0 , we conclude that E [ h k ( z ) 1 ( z ∈ S )] = 0 whenever k < r , and analogously for R r \ S . Finally, we easily verify that E [ σ ( z ) z 1 z 2 . . . z r ] > 0 , which shows that l ⋆ = r and hence k ⋆ = r .

This follows directly from k ⋆ ≤ l ⋆ .

Define K as the intersection of the half-spaces, determined by normals v 1 , . . . , v M . From the assumption that P is a r -dimensional multi-index model, V = [ v 1 . . . v M ] has rank r . Any unit norm vector u ∈ span ( V ) thus satisfies max i | v i · u | ≥ ϵ > 0 for some ϵ > 0 . Let Σ K = E [ zz ⊤ | z ∈ K ] -E [ z | z ∈ K ] E [ z | z ∈ K ] ⊤ be the covariance conditional on K . From [Klivans et al., 2024, Lemma B1], [Vempala, 2010, Lemma 4.7], for any u as above it holds that u ⊤ Σ K u < 1 , which implies that span (Λ 1 ) ⊕ span (Λ 2 ) = R r .

The argument appears already in Chen and Meka [2020], but we reproduce it here in our language for completeness.

We will use induction over the leaps. Suppose first S = ∅ , and consider the level sets B λ = { z ; | y | ≥ λ } . Since y = σ ( z ) is continuous and lim r →∞ | σ ( rz ) | = ∞ for any z , for any R > 0 there exists λ such that B λ does not contain the ball centered at 0 of radius R . Thus

̸

so if R 2 > r we must have E [ h 2 ( Z ) | Z ∈ B λ ] = 0 , and hence Λ 2 = 0 .

̸

Let us now iterate over leaps. Let S be the span of Λ 2 . We now consider the sets

By now viewing σ ( z ) = σ (¯ z S , z S ) as a polynomial in ¯ z S , we again argue that for any R > 0 there exists λ such that B λ,η,S does not contain a ball of radius R , and therefore we can identify another direction using the previous argument. Iterating this procedure until S spans the whole R r shows that k ⋆ ≤ 2 .

■

Proof of Proposition 5. Suppose towards contradiction that k ⋆ > 2 . Then for any g ∈ L 2 P y we have E [ g ( σ ( z )) H 2 ( z )] = 0 . Applying the coarea formula we obtain

where H k is the k -dimensional Hausdorff measure. Since this must be true for any measurable g , we conclude that

We write σ ( z ) = ∑ R ∈R ( v ⊤ R z + b R ) · 1 ( z ∈ R ) , where R are the different linear regions.

Case r = 1 : Suppose first that there exists ¯ y and ϵ > 0 such that the level sets σ -1 ( u ) contain no critical points for u ∈ (¯ y -ϵ, ¯ y + ϵ ) . The level sets σ -1 ( u ) are discrete, and we claim that we can represent them as

̸

and θ i = 1 /σ ′ ( t i ) = 0 have alternating sign. We thus have, for u ∈ (¯ y -ϵ, ¯ y + ϵ ) ,

Let us integrate this quantity twice now. Using the fact that ( h k -1 γ ) ′ = -h k γ , we have

From L ( u ) = 0 a.e. on (¯ y ± ϵ ) we have ˜ L ( u ) = 0 for all u ∈ (¯ y ± ϵ ) , leading to

Since all terms are analytic, we must have this equality for all u , which implies C = ˜ C = 0 , but this is a contradiction, since the LHS is a sum of positive terms.

Case r > 1 We can represent a piece-wise linear continuous function in terms of a simplex triangulation, and the values of the function at its vertices. Consider M = sup z { σ ( z ) } the maximum of σ , attained at a discrete set of global maxima. Now, let us start decreasing the level set until we reach another vertex, to say M ′ . We will study the family of level sets σ -1 ( y ) for y ∈ [ M ′ , M ] . We reparametrize y as y = M + u ( M ′ -M ) , so this family can now be indexed with u ∈ [0 , 1] .

For θ ∈ S r -1 and t ∈ R , let E ( θ, t ) := { z ; θ ⊤ z = t } denote a hyperplane normal to θ and passing at distance t to the origin. We can then write

where S R ( u ) = E ( v R / ∥ v R ∥ , ∥ v R ∥ u + b R ) ∩ R , and where ¯ R is the subset of linear regions crossed by these level sets. Note that by construction this family ¯ R does not depend on u .

For u ∈ ( ϵ, 1 -ϵ ) , and for each R ∈ ¯ R , we have the following homotecy representation of the level set regions:

Here, x R denotes a local maximum of σ , which is also a vertex of the corresponding simplex region R . Consider now L ( u ) := Tr { L ( u ) } . By introducing the local change of variables ˜ z = Ψ R,u ( z ) :=

x R + u ( z -x R ) for each region, we have

where θ R = ∥∇ σ ( z ) ∥ -1 for z ∈ R . Observe that L is analytic in R , since it is a linear combination of products of analytic functions. By assumption we have that L ( u ) = 0 for u ∈ (0 , 1) , which implies that L should vanish everywhere. But for u sufficiently large, observe that L ( u ) is a sum of strictly positive terms, which is a contradiction.

̸

This shows that Λ 2 = 0 . Let S = span (Λ 2 ) , and write z = ( z S , ¯ z ) , ¯ y = ( z S , y ) . We can now again suppose towards contradiction that for any g ∈ L 2 ( P ¯ y ) we have E [ g ( σ ( z ) , z S ) H 2 (¯ z )] = 0 . Defining ¯ σ -1 (¯ y ) := { ¯ z ; σ ( z S , ¯ z ) = y } , applying again the coarea formula leads to

The piece-wise linear, continuous structure is still preserved in ¯ σ , and therefore by iteratively applying the previous argument shows that all directions will be captured with generative leaps of at most k ⋆ ≤ 2 .

■

Proof of Proposition 7. Let σ ( z ) = ∑ j a j ρ ( z j ) . By definition, we have that (i) for any T : R → R and any polynomial q of degree < k ⋆ ( ρ ) , E [ T ( ρ ( z j )) q ( z j )] = 0 , and (ii) there exist a transformation ζ ( y ) such that E [ ζ ( ρ ( z j )) h k ⋆ ( ρ ) ( z j )] = 0 .

̸

̸

Let us first show that k ⋆ ≥ k ⋆ ( ρ ) . Suppose towards contradiction that we had a measurable U and multi-indices ( β 1 , . . . , β r ) with | β | < k ⋆ ( ρ ) such that E [ U ( σ ( z )) H β ( z )] = 0 . Then, denoting H β ( z ) = H β -j ( z -j ) h β j ( z j ) , we have

̸

̸

̸

̸

where we defined the label transformation ˜ T j ( y ) := E z [ U ( a j y + ∑ j ′ = j a j ′ ρ ( z j ′ ) ] . We have thus reached a contradiction. Observe that the same argument also applies if one replaces σ ( z ) by ¯ σ ( z ) = F ( ρ ( z 1 ) , . . . , ρ ( z r )) for arbitrary F .

Let us now show k ⋆ ≤ k ⋆ ( ρ ) := k ⋆ ρ . We focus on a Hermite moment along a single variable, say z 1 , given by h k ⋆ ρ ( z 1 ) . Let y = ρ ( z 1 ) and η = a -1 1 ∑ j> 1 a j ρ ( z j ) , so y and η are independent. We will find a measurable function U such that

̸

̸

We will consider a Fourier atom for U of the form U ( t ) = e i ˜ ξt for an appropriately chosen frequency ˜ ξ . For that purpose, let us first reproduce an argument from [Damian et al., 2024, Theorem 5.2]. By [Damian et al., 2024, Lemma F.2] there exists g : R → [ -1 , 1] such that E [ g ( Y ) h k ⋆ ρ ( Z )] = 0 . We consider g R ( y ) := g ( y ) 1 | y |≤ R . For R sufficiently large, we claim that E [ g R ( Y ) h k ⋆ ( Z )] = 0 . We have that

̸

̸

̸

which vanishes as R →∞ . Therefore for sufficiently large R we have E [ g R ( Y ) h k ⋆ ( Z )] = 0 . Now g R ∈ L 1 ( R ) ∩ L 2 ( R ) . Let us consider its Fourier representation g R ( y ) = ∫ ˆ g R ( ξ ) e iξy dξ . Then

̸

which shows that there must exist ξ 0 such that E P [ e iξ 0 Y h k ⋆ ( Z )] = 0 . Moreover, observe that ξ ↦→ E P [ e iξY h k ⋆ ( Z )] := ψ ( ξ ) is ∥ ρ ∥ 2 -Lipschitz, since

so we can define ϵ > 0 and δ > 0 such that | ψ ( ξ ) | ≥ δ for all ξ ∈ ( ξ 0 -ϵ, ξ 0 + ϵ ) .

Now, let us evaluate (36) with the Fourier atom. We have

where φ η ( ξ ) = E η [ e iηξ ] is the characteristic function of η . Assume, towards contradiction, that φ η ( ξ ) = 0 for all ξ ∈ ( ξ 0 -ϵ, ξ 0 + ϵ ) . By definition, we have that φ η ( ξ ) = ∏ r j =2 a 1 a j φ ( a 1 a j ξ ) , where φ ( ξ ) = E [ e iξρ ( z ) ] is the characteristic function of ρ ( z ) . We thus deduce that φ ( ξ ) must vanish on an interval ξ ∈ I ⊆ ( ξ 0 -ϵ, ξ 0 + ϵ ) . Since all the moments E [ y k ] exist by assumption, this means that

In particular, given any f ∈ L 2 ( P y ) , with expansion f ( y ) = ∑ k α k q k ( y ) , where { q k } k is an orthonormal basis of polynomials, we deduce from (43) that E [ f ( y ) e iξy ] = 0 for any f , which would mean that e iξy = 0 in L 2 ( P y ) , which is a contradiction. We have thus shown that there must exist ˜ ξ ∈ ( ξ 0 -ϵ, ξ 0 + ϵ ) where both φ η ( ˜ ξ ) and ψ ( ˜ ξ ) are non-zero, proving (36).

̸

̸

This shows that [Λ k ⋆ ] β 1 = 0 . Applying the same reasoning to z j , j ∈ [ r ] thus shows that [Λ k ⋆ ] β j = 0 . On the other hand, if we consider β with | β | = k ⋆ but β not of the form β = (0 , .., k ⋆ , 0 , . . . , 0) , applying again (33) shows that [Λ k ⋆ ] β = 0 , ie Λ k ⋆ is diagonal. We conclude that span (Λ k ⋆ ) = R r and thus that k ⋆ ≤ k ⋆ ( ρ ) .

■

Lemma 16 (Truncated and Fourier Label Transformations, [Damian et al., 2024, Theorem 5.2]) . Let P ∈ P ( R × R ) with P z = γ and let k ⋆ = k ⋆ ( P ) . Then there exists R 0 , ξ 0 , ϵ 0 > 0 and δ 0 > 0 , and label transformations T R , ˜ T ξ of the form T R ( y ) = g ( y ) 1 | y |≤ R and ˜ T ξ ( y ) = e iξy such that

for R ≥ R 0 and | ξ -ξ 0 | ≤ ϵ 0 .

As the Shallow NN becomes more overparametrised (and thus less well-conditioned), the generative leap is no longer 'related' to k ⋆ ( ρ ) . Indeed, provided ρ is not a polynomial, we can use the shallow NN to approximate any desired link function ˜ σ with prescribed generative leap exponent. Since the generative exponent is characterized as the first non-zero of an expansion, k ⋆ (˜ σ ) ≤ ℓ is an open property, meaning it is stable to small perturbations of ˜ σ . This directly leads to the following:

Proposition 9 (Generative Leap under Universal Approximation) . For any non-polynomial ρ and any integer ℓ ≥ 1 , there exists a shallow neural network of the form y = ∑ M j =1 a j ρ ( v ⊤ j z + b j ) such that ( Z, Y ) ∼ P has generative leap exponent k ⋆ ( P ) ≤ ℓ .

Proof of Proposition 9. By Proposition 7, we can reduce ourselves to the univariate case. We first verify that, given σ : R → R , there exists ϵ > 0 such that if

then k ⋆ (˜ σ ) ≤ k ⋆ ( σ ) = k ⋆ .

From Lemma 16, we can use a sinusoid label transformation ϕ ( y ) = cos( ξy ) for ξ that depends on σ such that E [ ϕ ( σ ( z )) h k ⋆ ( z )] = C = 0 .

̸

̸

It suffices to verify that E [ ϕ (˜ σ ( z )) h k ⋆ ( z )] = 0 . Let a ( z ) = σ ( z ) -˜ σ ( z ) . Indeed, since ϕ is ξ -Lipschitz, we have

with | ˜ a ( z ) | ≤ ξ | a ( z ) | . Thus

so if ϵ < C/ξ we have k ⋆ (˜ σ ) ≤ k ⋆ ( σ ) .

Finally, using a standard universal approximation theorem, e.g using the integral representation

for ν ∈ P ( R 3 ) , we can obtain ˜ σ satisfying (44) by doing a Monte-Carlo approximation.

■

Proposition 6 (Generative Leap under linear transformations) . Let σ ( z ) : R r → R ∈ L 2 ( γ r ) , σ = C , and let M r denote the set of r × r real matrices.

̸

(i) For Θ ∈ M r , define y Θ = σ (Θ ⊤ z ) . Then ( z, y Θ ) ∼ P Θ satisfies k ⋆ ( P Θ ) ≤ 2 for every Θ , except possibly for a set of r 2 -dimensional Lebesgue measure zero,

(ii) Assume that ( z, σ ( z )) ∼ P has a single leap with generative exponent k ⋆ . Let Γ : D ⊆ R s →M r be any analytic map such that I r ∈ Im(Γ) and Γ( θ ) is invertible for all θ ∈ D . For θ ∈ D , define y θ = σ (Γ( θ ) ⊤ z ) . Then ( z, y θ ) ∼ P θ satisfies k ⋆ ( P θ ) ≤ k ⋆ for every θ , except possibly for a set of s -dimensional Lebesgue measure zero.

Proof. We will again exploit analytic properties. Let us first prove (i).

Let us consider a threshold function of the form T ( y ) = 1 ( α 1 ≤ y ≤ α 2 ) , and its associated level set Ω = { z ; α 1 ≤ σ ( z ) ≤ α 2 } . Suppose first that we can pick α 1 , α 2 such that Ω is compact. Then we have Ω ⊂ B 0 ( R ) , the ball of radius R , for some R > 0 .

For M ∈ R r × r , consider the function

̸

Let us now consider Σ = Θ ⊤ Θ = V Λ 2 V ⊤ , and ˜ Z ∼ N (0 , Σ) . Observe that if Θ is invertible and ϕ (Θ -1 ) = 0 , then

̸

showing that det[Λ 2 ( P Θ )] = 0 , and thus k ⋆ ( P Θ ) ≤ 2 .

Let us now argue that ϕ is analytic in R r × r . Indeed, the determinant is analytic, and compositions preserve the analytic property, so it suffices to check that the functions M ↦→ ϕ a ( M ) =

∫ Ω e -1 2 z ⊤ M ⊤ Mz dz and M ↦→ ϕ b ( M ) = ∫ Ω z i z j e -1 2 z ⊤ M ⊤ Mz dz are analytic. Since Ω is compact, we have that

for any multi-index β , and analogously for the terms ϕ b .

̸

Now, observe that ϕ (0) = ( ∫ Ω dz ) r = 0 , which implies that ϕ cannot be identically zero. From Mityagin [2015] we then deduce that ϕ can only vanish on a set of measure 0 .

Let us now extend this argument to the setting where Ω is not compact. For any ϵ > 0 , we claim that ϕ ( M ) is analytic in S = { M ∈ R r × r ; λ min ( M ) ≥ ϵ } . Indeed, now ϕ is infinitely differentiable in S , and its components ϕ a , ϕ b satisfy

̸

and similarly for ϕ b . Since now we have ϕ (0) = ∞ , for ϵ small enough we must have ϕ ( M ) = 0 for some M ∈ S , which implies again that ϕ cannot be identically zero. It can therefore only vanish on a set of measure zero inside S , for any ϵ > 0 .

̸

Let us now prove part (ii) by adapting the previous argument. Since here we are assuming a single leap, there exists a tensor-valued label transformation T ∈ L 2 ( R , ( R r ) ⊗ k ⋆ ; P y ) such that det [ F ⊤ F ] = 0 , where

For Θ ∈ R r × r let us now define

Using again Σ = Θ ⊤ Θ = V Λ 2 V ⊤ , and ˜ Z ∼ N (0 , Σ) , we can rewrite this last expectation as

We argue again that F (Θ) is analytic in the domain { Θ;Σ ⪰ ϵI r } for any ϵ > 0 , since it is a linear combination of products of analytic functions. This implies that ψ (Θ) is also analytic in this domain. Finally, since the parametrization θ ↦→ Γ( θ ) is analytic by assumption, we deduce that ¯ ψ = ψ ◦ Γ is also analytic. We know that ¯ ψ is not identically zero, therefore we conclude that it can only vanish on a set of s -dimensional Lebesgue measure zero.

■

Peer reviews:
comments: SUMMARY:
This paper introduces the generative leap exponent, extending the generative exponent from single-index to multi-index Gaussian models. The authors establish a computational lower bound of under the Low-Degree Polynomial (LDP) framework for agnostically learning the hidden subspace. They also propose a sequential spectral estimator which achieves a matching upper bound.

STRENGTHS AND WEAKNESSES:
The paper is theoretically sound and well organized. It is significant to find the tight sample complexity for Gaussian multi-index models and the authors propose a novel algorithm to match the sample complexity. Some sections could be more accessible, especially the leap decomposition which is quite dense, but in general core claims are well-supported.

QUESTIONS:
1. L115: k-th Hermite tensor uses bold type here but not in the following texts. The authors can also emphaize that $\gamma_d$ represents Gaussian distribution.

2. L150: What does $span(\Lambda)$ mean when $\Lambda$ is a tensor?

3. It might be good to explain the meaning of generative leap exponent, as in L175-177 for generative exponent to better understand their difference. The authors can also explain the leap decomposition (L149) in more details to understand the intuition and difficulty to generalize the leap exponent from single index models to multi-index models.

4. In Proposition 4, why do we only have the upper bound of $k^*$ but not its value?

5. The authors mention that GD needs $\Theta(d^k)$ samples. Is it suboptimal?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the complexity of learning Gaussian multi-index models, where the authors generalize the notion of the generative exponent for single-index models to generative leap exponent. They prove that this exponent controls the low-degree-polynomial lower bound for sample complexity, and prove a polynomial-time algorithm with the same sample complexity as a function of $d$. They thus show that the generative leap exponent provides a tight characterization of the sample complexity for efficient learning of Gaussian multi-index models in high dimensions. The authors also establish links between the generative and information leap exponents, and show that their framework covers several examples of popular Gaussian multi-index models studied in the literature.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a technically strong paper with many novel and interesting results, and provides a complete picture of the complexity of learning Gaussian multi-index models, a problem that has received a lot of attention from the community as an example where learners can adapt to a latent low-dimensional structure in a high-dimensional ambient space and perform feature learning with non-linear predictors.\n\n The paper is generally well-written and easy to read, but there are a few points that can improve the readability. In particular, I think that the readers could have a better intuition of the constructions in Section 2 if the authors present the example of staircase functions and how it leads to concrete instantiations of Definitions 3, 4, and Proposition 1. Also the authors could explain a bit more about certain notations, discussed below.\n\nQUESTIONS:\n* What does $\\mathrm{span}(\\Gamma)$ mean for a tensor $\\Gamma \\in (S^\\perp)^{\\otimes 2k}$?\n\n* I believe the union in Equation (1) is meant to be direct sum? In particular $S_i$ live in different subspaces, I’m not sure how one can take unions of them and end up with a new subspace.\n\n* I’m a bit confused about Proposition 2. It seems that $\\mathcal{T}$ acts on $(y,z_S)$, then shouldn't the push forward map be $\\mathrm{Id}\\_{\\bar{z}\\_S} \\otimes \\mathcal{T}$ instead of $\\operatorname{Id}\\_z \\otimes \\mathcal{T}$?\n\n* From my understanding, not every subspace $S$ of the index space $U^\\*$ is seen when defining the generative leap. However, for the lower bound of Section 3, are we free to choose any subspace $\\bar{S}$ of $U^\\*$? Then would we be able to choose a subspace whose leap exponent is even larger than the one seen during Definition 3? I don’t see why we would necessarily pick the one that corresponds to the maximizer of Definition 3.\n\n* The authors mention in the paragraph starting from Line 25 that to achieve the information-theoretically optimal sample complexity of $d$, one needs to have a brute force estimator over an $\\epsilon$-net which will necessarily have an exponential computational complexity in high dimensions. However, if we assume the Gaussian multi-index model has a structured covariance, it is possible to obtain an improved computational complexity (even sub-exponential) while maintaining the same linear sample complexity (in some lower effective dimension). [1] presents an example of how gradient-based learning can achieve this computational adaptivity for multi-index models (which is absent from fixed-grid optimization over $\\epsilon$ nets). It is generally interesting to know how the picture here would change once we have a structured covariance, which is known to improve the dependence on information exponent for algorithms in the correlational statistical query class [2, 3].\n\n* The discussion in Section 4.2 can of course be extended to non-polynomial time algorithms. Without any assumptions on $P$, learning the regression function when $Y$ depends on $r$ directions could have a minimax optimal rate that looks like $(1/n)^{O(1/r)}$.\n\n* The authors may want to use $s_i$ in the 3rd line of Algorithm 1.\n\nReferences:\n\n[1] A. Mousavi-Hosseini et al. \"Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics.\" ICLR 2025.\n\n[2] J. Ba et al. \"Learning in the presence of low-dimensional structure: a spiked random matrix perspective.\" NeurIPS 2023.\n\n[3] A. Mousavi-Hosseini et al. \"Gradient-Based Feature Learning under Structured Data.\" NeurIPS 2023.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigated the computationally optimal sample complexities to learn the intrinsic features of multi-index models.\n\nThe authors first generalized the generative exponent (from single index models) to generative leap exponent $k^\\star$ with leap decomposition, which relates to the nested easy-to-hard subspace structure & leap exponent. \n\nThen they characterized the hardness of learning a multi-index models by proving that a sample complexity of $\\Omega(d^{k^\\star/2} )$ is essential for the low-degree polynomial test to have non-trivial power in detecting the planted subspace. \n\nFor a matching upper bound, the authors propose to use a U-statistics that resembles the unfolding technique in the multi-spike tensor PCA and proved that $O(d^{k^\\star/2})$ is essential for this algorithm to learn the planted direction.\n\nSTRENGTHS AND WEAKNESSES:\nPros\n1. The paper is mathematically rigorous and well organized. \n2. The content is quite complete and novel, by fully settling the low-degree detection lower bound and a matching upper bound. Several examples are provided to help readers with understanding the notion of generative leap and the hardness of the multi-index models.  \n3. The proof sketch for the upper bound is very clean, easy to follow and sufficient to cover the main idea. \n\nCons \n1. The layout, definitions and results seem to be a bit dense for readers.\n\nQUESTIONS:\n1. In the upper bound algorithm, you proposed a kernel-based algorithm that implements the label transformation of potentially infinite order and the conditions for the kernel seems to be very weak. However, in the problem of learning single-index models, batch-reusing / explicit label transformation are deployed to go beyond the CSQ queries[1,2] and certain conditions are required for general gradients to attain the optimal sample complexities [e.g. Assumpion 4.1 (b), 2]. Does this kernel trick automatically select such label transformations? Or is there a way that we can use this method in iterative algorithms to achieve optimal guarantee? \n2. Does the sub-space filtration (or flag, in (1)) from the generative leap exactly equal to the filtration defined for the information leap?  To put in other way, does the direction that is easier to learn in the notion of generative exponent necessarily easier to learn in the notion of information exponent? \n\n[1] Lee, Jason D., Kazusato Oko, Taiji Suzuki, and Denny Wu. \"Neural network learns low-dimensional polynomials with sgd near the information-theoretic limit.\" Advances in Neural Information Processing Systems 37 (2024): 58716-58756.\n\n[2] Chen, Siyu, Beining Wu, Miao Lu, Zhuoran Yang, and Tianhao Wang. \"Can neural networks achieve optimal computational-statistical tradeoff? an analysis on single-index model.\" In The Thirteenth International Conference on Learning Representations. 2025.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper considers multi-index models with Gaussian inputs in high dimensions and a finite number of indices. For this model, the authors show that one can estimate the hidden subspaces in the model with a sample complexity $n = \Theta(d^{1\lor k^*/2})$, where $k^*$, the generative leap, depends on the model. This is a necessary and sufficient condition.

STRENGTHS AND WEAKNESSES:
The paper is detailed, clear and extremely well written. The proofs are clear, and the technique is interesting, although similar in essence to the one for generative exponents. The main contribution of this paper, the description of the generative leap, is a fundamental contribution to the field and will surely have a big impact in the theory community. As the authors correctly state, most cases of practical interest have generative exponent less then or equal to 2, which were already studied in several previous works, so its fundamental limitation lies in the limited applicability of the results.

QUESTIONS:
1. In 4.2 you discussed a fine-grained analysis, in the sense of finding the minimal constant $\lim_{n,d\to\infty} n/d^{1\lor k^*/2}$ such that the hidden subspaces can be recovered. In particular you state that the dependance on the number of indices can be exponentially bad. Can you provide a more comprehensive classification?
2. In 5.3 you state that while shallow networks have a generative leap at least as large as the generative exponent, wider networks might not have this bound. Is there a practical example of this with some activations?
3. Is there some way to salvage any of this theory in the case of $r = \Theta(d)$ indices?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-30