Title: Sample and Computationally Efficient Robust Learning of Gaussian Single-Index Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: single index models

agnostic learning

adversarial label noise
Abstract: A single-index model (SIM) is a function of the form $\sigma(\mathbf{w}^{\ast} \cdot \mathbf{x})$, where
$\sigma: \mathbb{R} \to \mathbb{R}$ is a known link function and $\mathbf{w}^{\ast}$ is a hidden unit vector. 
We study the task of learning SIMs in the agnostic (a.k.a. adversarial label noise) model 
with respect to the $L^2_2$-loss under the Gaussian distribution. 
Our main result is a sample and computationally efficient agnostic proper learner 
that attains $L^2_2$-error of $O(\mathrm{OPT})+\epsilon$, where $\mathrm{OPT}$ is the optimal loss. The sample complexity of our algorithm is 
$\tilde{O}(d^{\lceil k^{\ast}/2\rceil}+d/\epsilon)$, where 
$k^{\ast}$ is the information-exponent of $\sigma$ 
corresponding to the degree of its first non-zero Hermite coefficient. 
This sample bound nearly matches known CSQ lower bounds, even in the realizable setting. 
Prior algorithmic work in this setting had focused 
on learning in the realizable case or in the presence 
of semi-random noise. Prior computationally efficient robust learners required 
significantly stronger assumptions on the link function.

Sample and Computationally Efficient Robust Learning of Gaussian Single-Index Models

Abstract

A single-index model (SIM) is a function of the form σ ( w ∗ · x ) , where σ : R → R is a known link function and w ∗ is a hidden unit vector. We study the task of learning SIMs in the agnostic (a.k.a. adversarial label noise) model with respect to the L 2 2 -loss under the Gaussian distribution. Our main result is a sample and computationally efficient agnostic proper learner that attains L 2 2 -error of O (OPT)+ ϵ , where OPT is the optimal loss. The sample complexity of our algorithm is ˜ O ( d ⌈ k ∗ / 2 ⌉ + d/ϵ ) , where k ∗ is the information-exponent of σ corresponding to the degree of its first non-zero Hermite coefficient. This sample bound nearly matches known CSQ lower bounds, even in the realizable setting. Prior algorithmic work in this setting had focused on learning in the realizable case or in the presence of semi-random noise. Prior computationally efficient robust learners required significantly stronger assumptions on the link function.

1 Introduction

Single-index models (SIMs) [Ich93, HJS01, HMS + 04, DJS08, KS09, KKSK11, DH18, DGK + 20, DKTZ22, WZDD23, DNGL23] are a classical supervised learning model characterized by hidden low-dimensional structure. The term SIM refers to any function f of the form f ( x ) = σ ( w · x ) , where σ : R → R is the link (or activation) function and w ∈ R d is the hidden vector. In most settings of interest, the link function is assumed to satisfy certain regularity properties. Indeed, for an arbitrary function, learnability is information-theoretically impossible.

The efficient learnability of SIMs has been the focus of extensive investigation for several decades. For example, the special case where σ is the sign function corresponds to Linear Threshold Functions whose study goes back to the Perceptron algorithm [Ros58]. Classical early works [KS09, KKSK11] studied the efficient learnability of SIMs for monotone and Lipschitz link functions. They showed that a gradient method efficiently learns SIMs for any distribution on the unit ball. More recently, there has been a resurgence of research on the topic with a focus on first-order methods. Indeed, the non-convex optimization landscape of SIMs exhibits rich structure and has become a useful testbed for the analysis of such methods. [Sol17] showed that SGD efficiently learns SIMs for the case that σ is the ReLU activation and the distribution is Gaussian. [CLS15, CCFM19, SQW18] showed that gradient descent succeeds for the phase retrieval problem, corresponding to σ ( t ) = t 2 or σ ( t ) = | t | .

More recently, a line of work [DH18, BAGJ21, DNGL23, DPVLB24] studied the efficient learnability of SIMs going significantly beyond the monotonicity assumption. Specifically, [BAGJ21, DNGL23] developed efficient gradient-based SIM learners for a general class of - not necessarily monotone -link functions under the Gaussian distribution. Roughly speaking, these works show that the complexity of learning SIMs is intimately related to the Hermite structure of the link function (roughly, the smallest degree non-zero Hermite coefficient). The results of the current paper are most closely related to the aforementioned works.

All the aforementioned algorithmic results succeed in the realizable model (i.e., with clean labels) or in a few cases in the presence of random label noise. The focus of this work is on learning SIMs in the challenging agnostic (or adversarial label noise) model [Hau92, KSS94]. In the agnostic model, no assumptions are made on the observed labels and the goal is to compute a hypothesis that is competitive with the best-fit function in the class. The algorithmic study of agnostically learning SIMs is not new. A recent line of work [DGK + 20, DKTZ22, ATV23, WZDD23, GGKS23, ZWDD24] has given efficient agnostic SIM learners (typically based on first-order methods) with near-optimal error guarantees under natural distributional assumptions. The key difference between prior work and the results of the current paper is in the assumptions on the link function. Specifically, prior robust learners succeed for (a subclass of) monotone and Lipschitz link functions. In contrast, this work develops robust learners in the more general setting of [BAGJ21, DNGL23].

In order to precisely describe our contributions, we require the definition of the agnostic learning problem for Gaussian SIMs. Let D be a distribution of labeled examples ( x , y ) ∈ R d × R whose x -marginal is the standard Gaussian, and let L σ 2 ( w ) := E ( x ,y ) ∼D [( σ ( w · x ) -y ) 2 ] be the L 2 2 (or squared) loss of the hypothesis h ( x ) = σ ( w · x ) with respect to D . Given i.i.d. samples from D , the goal is to compute a hypothesis with squared error competitive with OPT , where OPT is the best attainable L 2 2 -error by any function in the target class.

Problem 1.1 (Robustly Learning Gaussian SIMs) . Let D be a distribution of labeled examples ( x , y ) ∈ R d × R whose x -marginal is D x = N (0 , I d ) and y is arbitrary. We say that an algorithm is a C -approximate proper SIM learner , for some C ≥ 1 , if given ϵ > 0 and i.i.d. samples from D , the algorithm outputs a vector ̂ w ∈ S d -1 such that with high probability it holds L σ 2 ( ̂ w ) ≤ C OPT+ ϵ , where OPT := L σ 2 ( w ∗ ) and w ∗ ∈ argmin w ∈ S d -1 L σ 2 ( w ) .

First, note that Problem 1.1 does not make realizability assumptions on the distribution D . That is, the labels are allowed to be arbitrary and the goal is to be competitive against the best-fit function in the class. Second, our focus is on obtaining efficient learners that achieve a constant factor approximation to the optimum loss - independent of the dimension d . The reason we require a constant factor approximation, instead of optimal error of OPT+ ϵ , is the existence of computational hardness results ruling out this possibility. Specifically, even if the link function is the ReLU, there is strong evidence that any algorithm achieving error OPT+ ϵ in the above setting requires d poly(1 /ϵ ) time [DKZ20, GGK20, DKPZ21, DKR23].

Recent work [DGK + 20, DKTZ22, ATV23, WZDD23] gave efficient, constant-factor robust learners, for the special case of Problem 1.1 where the link function lies in a proper subclass of monotone and Lipschitz functions. In this work, we obtain a broad generalization of these results to a much more general class of link functions, defined below.

We now proceed to formalize the assumptions on the link function. Let σ : R → R be a realvalued function that admits the Hermite decomposition σ ( z ) = ∑ k ≥ 0 c k he k ( z ) , where c k = E z ∼N (0 , 1) [ σ ( z )he k ( z )] and he k is the normalized probabilist's Hermite polynomial, defined by

We make the following assumptions.

Assumption 1 (Family of Link Functions) . Suppose that σ is normalized, namely E z ∼N (0 , 1) [ σ 2 ( z )] = ∑ k ≥ 0 c 2 k = 1 . We assume the following:

(i) The first non-zero Hermite coefficient has degree k ∗ and is prominent: c k ∗ is an absolute constant that is bounded away from zero.

(ii) The fourth moment of σ ( z ) is bounded: E z ∼N (0 , 1) [ σ 4 ( z )] ≤ B 4 < ∞ .

(iii) The second moment of the derivative of σ ( z ) is bounded: E z ∼N (0 , 1) [( σ ′ ( z )) 2 ] = ∑ k ≥ k ∗ kc 2 k ≤ C k ∗ , where C k ∗ is an absolute constant whenever k ∗ is an absolute constant.

The parameter k ∗ is known as the information exponent of σ .

The information exponent k ∗ can be viewed as a complexity measure of the link function. Specifically, ReLU activations correspond to k ∗ = 1 . The same holds for the class of bounded activations considered in [DKTZ22, WZDD23]. The link functions used in phase retrieval have k ∗ = 2 .

We note that Assumption 1 is (at least) as general as those used in [BAGJ21, DNGL23] - which focused on the realizable setting. Comparing against previous constant-factor agnostic learners, Assumption 1 strongly subsumes the class of 'bounded activations' [DKTZ22, WZDD23]. In particular, it is easy to see that there exist functions satisfying Assumption 1 with constant k ∗ that are far from monotone. See Appendix A for a detailed justification.

In prior work, [DNGL23], building on [BAGJ21], gave a sample-efficient gradient method for learning SIMs under Assumption 1 in the realizable setting . The sample complexity of their method is ˜ O ( d k ∗ / 2 + d/ϵ ) . This sample upper bound essentially matches known lower bounds in the Correlational Statistical Query (CSQ) model [DLS22, AAM23].

This discussion leads to the following question, which served as the motivation for the current work:

Is there an efficient constant-factor agnostic learner for Gaussian SIMs under Assumption 1?

As our main contribution, we answer this question in the affirmative. Interestingly, our algorithm also relies on a gradient-method (Riemannian optimization over the sphere) following a non-trivial initialization step. We emphasize that this is the first polynomial-time constant-factor agnostic learner for this task under Assumption 1.

Specifically, we establish the following result (see Theorem 3.5 for a more detailed statement).

Theorem 1.2 (Main Result, Informal) . There exists an algorithm that draws n = ˜ Θ k ∗ ( d ⌈ k ∗ / 2 ⌉ + d/ϵ ) labeled samples, runs in poly( n, d ) time, and outputs a weight vector ̂ w ∈ S d -1 that with high probability satisfies L σ 2 ( ̂ w ) ≤ C OPT+ ϵ , where C = O ( C k ∗ ) .

Theorem 1.2 gives the first sample and computationally efficient robust learner for Gaussian SIMs under Assumption 1. This generalizes the algorithm of [DNGL23] to the agnostic setting and nearly matches the aforementioned CSQ lower bounds (our algorithm fits the CSQ framework). It is worth pointing out that, while more efficient (non-CSQ) algorithms have been developed for the realizable case [CM20], these algorithms provably fail in the agnostic regime. Finally, we remark that very recent work [DPVLB24] developed an efficient SIM learner and a nearly matching SQ lower bound in a model that allows for some forms of label noise. Importantly, their model does not capture the adversarial label noise studied here. More specifically, the algorithms developed in this prior work [DNGL23, DPVLB24] fail in the agnostic setting. See Appendix B for a detailed discussion.

1.1 Preliminaries

For n ∈ Z + , let [ n ] := { 1 , . . . , n } . We use lowercase bold characters to denote vectors and uppercase bold characters for matrices and tensors. For x ∈ R d and i ∈ [ d ] , x i denotes the i -th coordinate of x , and ∥ x ∥ 2 := ( ∑ d i =1 x 2 i ) 1 / 2 denotes the ℓ 2 -norm of x . We use x · y for the inner product of x , y ∈ R d and θ ( x , y ) for the angle between x , y . We slightly abuse notation and denote by e i the i th standard basis vector in R d . We use 1 {E} to denote the indicator of a statement E . We use N d to denote the d -dimensional standard Gaussian distribution, i.e., N d = N (0 , I ) . We use B d to denote the centered unit ball in R d and denote the unit sphere in R d by S d -1 . We use proj B d ( · ) to denote the projection operator that projects a vector to the unit ball.

Given M ∈ R d 1 × d 2 and a left singular vector v ∈ R d 1 of M , we denote the corresponding singular value of v by ρ ( v ) . In addition, we use ρ 1 ≥ ρ 2 ≥ · · · ≥ ρ min { d 1 ,d 2 } to denote the singular values of a matrix M ∈ R d 1 × d 2 . We use S k to denote the set of all possible permutations of k distinct objects. Given a unit vector w , we define P w ⊥ := I -ww ⊤ to be the projection matrix that maps a vector v to its component that is orthogonal to w , i.e., P w ⊥ v = v ⊥ w .

Given a vector x ∈ R d , the (normalized) Hermite multivariate tensor is defined by [Rah17]:

We use the standard O ( · ) , Θ( · ) , Ω( · ) asymptotic notation. We use ˜ O ( · ) to omit polylogarithmic factors in the argument. We write E ≳ F for two non-negative expressions E and F to denote that there exists some positive universal constant c > 0 such that E ≥ c F .

1.2 Technical Overview

Our technical approach consists of two main parts: (1) new results for tensor PCA, which allow us to obtain an initial parameter vector w 0 that is nontrivially aligned with the target w ∗ and (2) a structural 'alignment sharpness' result, which we use to argue that a variant of Riemannian minibatch stochastic gradient descent on a sphere applied to a 'truncated square loss' (defined below) converges geometrically fast. In proving these results, we review elementary tensor algebra and basic properties of Hermite polynomials, and prove several structural results for Hermite polynomials that are instructive and may be useful to non-experts entering this area.

We now highlight some of the key ideas used in our work.

Initialization via Tensor PCA For our optimization algorithm to work, a warm start ensuring nontrivial alignment between the initial vector w 0 and the target vector w ∗ , as measured by w 0 · w ∗ , is essential. In particular, a consequence of our results in Claim 3.1 and Lemma 3.2 is that w 0 · w ∗ = Ω(1) is required to deal with the highly corruptive agnostic noise. Unfortunately, if we were to select w 0 by drawing uniformly random samples from the sphere, exponentially many in d such samples would be needed to ensure that with constant probability at least one of the sampled vectors w 0 is such that w 0 · w ∗ = Ω(1) , due to standard results on concentration of measure on the (unit) sphere.

Perhaps surprisingly, we prove that the tensor PCA method developed in [RM14] when applied to our problem with O ( d ⌈ k ∗ / 2 ⌉ ) samples 1 ensures that w 0 · w ∗ ≥ 1 -min { 1 /k ∗ , 1 / 2 } . The reason that this result is surprising is that the method in [RM14] was developed to solve the following problem: given a k -tensor of the form

where A is a k -tensor with i.i.d. standard Gaussian entries and τ > 0 a 'signal strength' parameter, recover the planted (signal) vector v . This 'single-observation' model is equivalent (in law) to the following 'multi-observation' model ([BAGJ20]): given n i.i.d. copies T ( i ) = τ ′ v ⊗ k + A ( i ) with τ ′ = τ/ √ n , recover v using the empirical estimation:

In our setting, we wish to recover a vector w ∗ (up to some constant alignment error) for the k -Chow tensor C k = E ( x ,y ) ∼D [ y He k ( x )] , which can be decomposed as

The first term in this decomposition can be viewed as the 'signal' k -tensor. The second term represents noise, which, due to y being potentially arbitrary, cannot be assumed to be Gaussian. Thus, previously developed techniques for tensor PCA, which crucially rely on the 'Gaussianity' of the noise term, do not apply here.

To obtain our result, we first argue that for any k ≥ k ∗ , the top left singular vector v ∗ of the k -Chow tensor C k unfolded into a matrix of roughly equal dimensions carries a 'strong signal' about the target vector w ∗ : its associated singular value scales with c k whenever c k = Ω( √ OPT) and it has a nontrivial alignment with the vectorized version of the l -tensor w ∗⊗ l for l = ⌊ k/ 2 ⌋ (Lemma 2.2).

To prove the desired alignment result using the empirical estimate of the matrix-unfolded k -Chow tensor C k , we rely on the application of a matrix concentration inequality obtained very recently

in [BvH22]. This requires a rather technical argument utilizing Gaussian hypercontractivity of multivariate polynomials of bounded degree, which we show characterizes the different 'variancelike' quantities associated with the empirical estimate of (the matrix-unfolded) C k , for which we apply the aforementioned matrix concentration (see Lemma 2.4 and its proof).

Another intriguing aspect of our initialization result is that it is possible to use it directly to obtain an O ( √ OPT+ ϵ ) -error solution. In particular, in the realizable case studied in [DNGL23], where OPT = 0 , this result directly leads to error O ( ϵ ) in a sample and computationally efficient manner, with a rather simple algorithm and sample complexity comparable to [DNGL23].

Optimization on a Sphere The second key ingredient in our work is a structural result, stated in Lemma 3.3, which ensures that the gradient field (Riemannian gradient of a truncated loss) guiding the steps of our algorithm (which can be interpreted as a Riemmanian minibatch SGD on a sphere) negatively correlates with w ∗ to a significant extent. This property can be viewed as the considered gradient field, associated with the L 2 2 loss truncated to only contain the first nonzero term in the Hermite expansion of the activation function, containing a strong 'signal' that can 'pull' the algorithm iterates towards the target O (OPT) + ϵ solutions. We rely on this property to argue that as long as our algorithm (initialized using the tensor PCA method described above) does not have as its iterate a vector with O (OPT) + ϵ loss, the distance between the iterate vector and the target vector must contract. As a consequence, this algorithm converges in O (log(1 /ϵ )) iterations.

This argument parallels the line of work [MBM18, DGK + 20, WZDD23, ZWDD24] on addressing learning of single-index models by proving structural, optimization-theory local error bounds that bound below the growth of a loss function outside the set of target solutions. Conceptually, the local error bounds from this line of work all have an intuitive interpretation as showing existence of a strong 'signal' in the problem that can be used to guide algorithm updates towards target solutions. However, the methodology by which our structural result is obtained is entirely different, as it crucially relies on properties of Hermite polynomials, which were not considered in this past work.

2 Initialization Procedure

In this section, we show how to get an initial parameter vector w 0 such that w 0 · w ∗ = 1 -ϵ 0 for some small constant ϵ 0 . The main technique is a tensor PCA algorithm that finds the principal component of a noisy degreek -Chow tensor for any k ≥ k ∗ , as long as OPT ≲ c 2 k . Such a degreek Chow tensor is defined by C k = E ( x ,y ) ∼D [ y He k ( x )] , and we denote its noiseless counterpart by

Furthermore, let us denote the difference between C k and C ∗ k by

Note that since He k ( x ) is a symmetric tensor for any x , all C k , C ∗ k , and H k are symmetric tensors.

We use the following matrix unfolding operator that maps a k -tensor T to a matrix in R d l × d k -l : 2

for all i 1 , . . . , i l , j 1 , . . . , j k -l ∈ [ d ] . We also define the 'vectorize' operator and 'tensorize' operators, which map a vector v ∈ R d l to an l -tensor for any integer l , and vice versa. In detail,

Finally, given a vector v ∈ R d l , we can also convert this vector to a matrix of size R d × d l -1 :

Throughout this section, we take l := ⌊ k/ 2 ⌋ . We leverage the tensor unfolding algorithm proposed in [RM14], which can succinctly be described as follows. First we unfold the degreek Chow tensor

to a matrix in R d l × d k -l and find its top-left singular vector v ∈ R d l . Then, we calculate the matrix Mat (1 ,l -1) ( v ) , and output its top left singular vector u .

Our main result for initialization is that the output of Algorithm 1 significantly correlates with w ∗ .

Proposition 2.1 (Initialization) . Suppose Assumption 1 holds. Assume that OPT ≤ c 2 k ∗ / (64 k ∗ ) 2 , and let ϵ 0 = c k ∗ / (256 k ∗ ) . Then, Algorithm 1 applied to Problem 1.1 with k = k ∗ uses n = Θ(( k ∗ ) 2 e k ∗ log k ∗ ( B 4 /ϵ ) d ⌈ k ∗ / 2 ⌉ / ( c 2 k ∗ ) + 1 /ϵ ) samples, runs in polynomial time, and outputs a vector w 0 ∈ S d -1 such that w 0 · w ∗ ≥ 1 -min { 1 /k ∗ , 1 / 2 } .

We remark here that Algorithm 1 can also be used to find an approximate solution of our agnostic learning problem; however the error dependence on OPT is suboptimal , scaling with its square-root. For full details of this argument, included for completeness, see Proposition D.3 in Appendix D.

In the remainder of this section, we sketch the proof of Proposition 2.1, which relies on two main ingredients: (1) alignment of the left singular vectors v of matrix-unfolded k -Chow tensor and the target vector w ∗ , which can be interpreted as the k -Chow tensor containing a strong 'signal' about the target vector w ∗ , and (2) matrix concentration for the unfolded tensor, so that we can translate 'population' properties of the k -Chow tensor to computable empirical quantities.

Signal in the k -Chow Tensor Our first observation is that for any left singular vector v of Mat ( l,k -l ) ( C k ) , the singular value ρ ( v ) is close to the inner product between v and Vec ( w ∗⊗ l ) , where l = ⌊ k/ 2 ⌋ . Concretely, we have:

Lemma2.2. Let v be any left singular vector of Mat ( l,k -l ) ( C k ) . Then, | ρ ( v ) -c k ( Vec ( w ∗⊗ l ) · v ) | ≤ √ OPT .

Proof Sketch of Lemma 2.2. Recall that the singular value of the left singular vector v satisfies

where we used Fact C.1 (2) in ( i ) . Since C k = C ∗ k + H k , we further have

⟨ C k , Tensor ( v ) ⊗ Tensor ( r ) ⟩ = ⟨ C ∗ k , Tensor ( v ) ⊗ Tensor ( r ) ⟩ + ⟨ H k , Tensor ( v ) ⊗ Tensor ( r ) ⟩ . ∗

We bound both terms above respectively. For the first term, plugging in the definition of C k and using the orthonormality property of Hermite tensors (Fact C.3) and basic tensor algebraic calculations,

Next, for the second term, after applying Cauchy-Schwarz inequality, one can show that it holds

√

√

|⟨ H k , Tensor ( v ) ⊗ Tensor ( r ) ⟩| ≤ OPT ∥ Sym( Tensor ( v ) ⊗ Tensor ( r )) ∥ F ≤ OPT . (3) Combining Equation (2) and Equation (3), we get that the singular value of v must satisfy

where in the equation above, we used the observation that as ∥ Vec ( w ∗⊗ k -l ) ∥ 2 = ∥ w ∗⊗ k -l ∥ F = 1 , it holds max r ∈ R d k -l , ∥ r ∥ 2 =1 ( Vec ( w ∗⊗ k -l ) · r ) = ∥ Vec ( w ∗⊗ k -l ) ∥ 2 = 1 . Since Equation (3) implies ⟨ H k , Tensor ( v ) ⊗ Tensor ( r ) ⟩ ≥ -√ OPT , similarly we have ρ ( v ) ≥ c k ( Vec ( w ∗⊗ l ) · v ) -√ OPT , completing the proof of Lemma 2.2.

Lemma 2.2 implies that the top left singular vector v ∗ aligns well with Vec ( w ∗⊗ l ) . The full version of Corollary 2.3 is deferred to Corollary D.5.

Corollary 2.3. The top left singular vector v ∗ ∈ R d l of the unfolded tensor Mat ( l,k -l ) ( C k ) satisfies v ∗ · Vec ( w ∗⊗ l ) ≥ 1 -(2 √ OPT) /c k .

Concentration of the Unfolded Tensor Matrix Let us denote M ( i ) = Mat ( l,k -l ) ( y ( i ) He k ( x ( i ) )) , for i ∈ [ n ] and ̂ M = 1 n ∑ n i =1 M ( i ) , which is the empirical approximation of M = Mat ( l,k -l ) ( C k ) = Mat ( l,k -l ) ( E ( x ,y ) ∼D [ y He k ( x )]) . Though we showed in Corollary 2.3 that the top left singular vector v ∗ of the population M strongly correlates with the signal Vec ( w ∗⊗ l ) , we only have access to the empirical estimate ̂ M and its corresponding top left singular vector ̂ v ∗ . Thus, to guarantee that ̂ v ∗ correlates significantly with Vec ( w ∗⊗ l ) as well, we need to show that the angle between the v ∗ and ̂ v ∗ is sufficiently small as long as we use sufficiently many samples. To this aim, we apply Wedin's theorem (Fact D.6). Wedin's theorem states that sin( θ ( v ∗ , ̂ v ∗ )) can be bounded above by:

where ρ 1 and ρ 2 are the top 2 singular values of M . We prove in Claim D.7 that ρ 1 -ρ 2 ≥ ( c k -8 √ OPT) / 2 ≳ c k under the assumption that √ OPT ≲ c k , hence ρ 1 -ρ 2 is bounded below by a constant. Thus, our remaining goal is to bound the operator norm such that ∥ M -̂ M ∥ 2 ≤ ϵ 0 where ϵ 0 > 0 is a small constant. This can be accomplished by applying a recently obtained matrix concentration inequality from [BvH22, DPVLB24] (stated in Fact D.8), with additional technical arguments. Plugging the lower bound on the singular gap ρ 1 -ρ 2 and the upper bound on the operator norm ∥ M -̂ M ∥ 2 back into Wedin's theorem (Fact D.6), we obtain the following main technical lemma of this subsection, whose proof can be found in Appendix D:

Lemma 2.4 (Sample Complexity for Estimating the Unfolded Tensor Matrix) . Let ϵ, ϵ 0 > 0 . Consider the unfolded matrix M = Mat ( l,k -l ) ( E ( x ,y ) ∼D [ y He k ( x )]) and its empirical estimate ̂ M := (1 /n ) ∑ n i =1 Mat ( l,k -l ) ( y ( i ) He k ( x ( i ) )) , where { ( x ( i ) , y ( i ) ) } n i =1 are n = Θ( e k log k ( B 4 /ϵ ) d k/ 2 /ϵ 2 0 + 1 /ϵ ) i.i.d. samples from D . Then, with probability at least 1 -exp( -d 1 / 2 ) , ∥ ̂ M -M ∥ 2 ≤ ϵ 0 . Moreover, if ̂ v ∗ is the top left singular vector of ̂ M , then with probability at least 1 -exp( -d 1 / 2 ) ,

After getting an approximate top left singular vector ̂ v ∗ ∈ R d l of Mat ( l,k -l ) ( E ( x ,y ) ∼D [ y He k ( x )]) , the final piece of the argument is that finding the top left singular vector of the matrix Mat (1 ,l -1) ( ̂ v ∗ ) completes the task of computing a vector u that correlates strongly with w ∗ . Concretely, we have:

Lemma 2.5. Suppose that ̂ v ∗ · Vec ( w ∗⊗ l ) ≥ 1 -ϵ 1 for some ϵ 1 ∈ (0 , 1 / 16] . Then, the top left singular vector u ∈ R of Mat (1 ,l -1) ( ̂ v ∗ ) satisfies u · w ∗ ≥ 1 -2 ϵ 1 .

Proof of Proposition 2.1 Since √ OPT ≤ c k ∗ / (64 k ∗ ) ≤ c k ∗ / 64 , choosing ϵ 0 = c k ∗ / (256 k ∗ ) ≤ c k ∗ / 256 in Lemma 2.4, we obtain that using n = Θ(( k ∗ ) 2 e k ∗ log k ∗ ( B 4 /ϵ ) d ⌈ k ∗ / 2 ⌉ / ( c 2 k ∗ ) + 1 /ϵ ) , it holds with probability at least 1 -exp( -d 1 / 2 ) that

Then applying Lemma 2.5 with ϵ 1 ≤ 1 / (16 k ∗ ) ≤ 1 / 16 , we get that the output u of Algorithm 1 satisfies u · w ∗ ≥ 1 -2 ϵ 1 ≥ 1 -1 / (8 k ∗ ) ≥ 1 -min { 1 /k ∗ , 1 / 2 } , completing the proof.

3 Optimization via Riemannian Gradient Descent

After obtaining w 0 from Algorithm 1, we run Riemannian minibatch SGD Algorithm 2 on the 'truncated loss' (see definition in Equation (5)). In the rest of the section, we first present the definition of the truncated L 2 2 loss L ϕ 2 and its Riemannian gradient and then proceed to proving that Algorithm 2 converges to a constant approximate solution in O (log(1 /ϵ )) iterations. Due to space constraints, omitted proofs are provided in Appendix E.

Algorithm 2 Riemannian GD with Warm-start

1: Input: Parameters ϵ, k ∗ , c k ∗ , B 4 > 0; T, η ; Sample access to D .

2: w 0 = Initialization [ ϵ, k ∗ , c k ∗ , B 4 , ϵ 0 = c k ∗ / (256 k ∗ )] (Algorithm 1).

3: for t = 0 , . . . , T -1 do

5: ̂ g ( w t ) = 1 n ∑ n i =1 k ∗ c k ∗ y ( i ) ( I -w t ( w t ) ⊤ ) ⟨ He k ∗ ( x ( i ) ) , ( w t ) ⊗ k ∗ -1 ⟩ .

4: Draw n = Θ( C k ∗ de k ∗ log k ∗ +1 ( B 4 /ϵ ) / ( ϵδ )) samples from D

6: w t +1 = ( w t -η ̂ g ( w t )) / ∥ w t -η ̂ g ( w t ) ∥ 2 .

7:

Return:

w

T

3.1 Truncated Loss and the Sharpness property of the Riemannian Gradient

Instead of directly minimizing the L 2 2 loss L σ 2 , we work with the following truncated loss that drops all the terms higher than k ∗ in the polynomial expansion of σ :

Similarly, the noiseless surrogate loss is defined as

It can be shown (using Fact C.1 (2) ) that the gradient of the truncated L 2 2 loss equals:

while for the gradient of the noiseless L 2 2 loss we have

Recall that P w ⊥ := I -ww ⊤ . Then the Riemannian gradient of the L 2 2 loss L ϕ 2 , denoted by g ( w ) is

Similarly, the Riemannian gradient of the noiseless L 2 2 loss L ∗ ϕ 2 is defined by

We first show that g ∗ ( w ) carries information about the alignment between vectors w and w ∗ .

Let us denote the difference between the noisy and the noiseless Riemannian gradient by ξ ( w ) , i.e., ξ ( w ) := g ( w ) -g ∗ ( w ) = -2 E ( x ,y ) ∼D [( y -σ ( w ∗ · x )) P w ⊥ ∇ ϕ ( w · x )] . We next show that the norm of ξ ( w ) and the inner product between ξ ( w ) and w ∗ are both bounded. √

Lemma 3.2. Let ξ ( w ) = g ( w ) -g ∗ ( w ) as defined above. Then, ∥ ξ ( w ) ∥ 2 ≤ 2 k ∗ c k ∗ OPT and | ξ ( w ) · w ∗ | ≤ 2 k ∗ c k ∗ √ OPT ∥ ( w ∗ ) ⊥ w ∥ 2 .

We are now ready to present the main structural result of this section.

Lemma 3.3 (Sharpness) . Assume OPT ≤ c/ (4 e ) 2 for some small absolute constant c < 1 . Let w ∈ S d -1 and suppose that w · w ∗ ≥ 1 -1 /k ∗ . Let θ := θ ( w , w ∗ ) . If sin θ ≥ 4 e √ OPT , then g ( w ) · w ∗ ≤ -(1 / 2) ∥ g ∗ ( w ) ∥ 2 sin θ.

Proof. We start by noticing that by Claim 3.1, the noiseless gradient satisfies the following property:

where we used that since ∥ w ∥ 2 = ∥ w ∗ ∥ 2 = 1 , we have ∥ ( w ∗ ) ⊥ w ∥ 2 = sin θ . Furthermore, applying Lemma 3.2 we have the following sharpness property with respect to the L 2 2 loss: √

Observe that (1 -1 /t ) t -1 ≥ 1 /e for all t ≥ 1 . Therefore, when w · w ∗ ≥ 1 -1 /k ∗ , we have

Hence, when sin θ ≥ 4 e OPT and w · w ∗ ≥ 1 -1 /k ∗ , we have ∥ g ∗ ( w ) ∥ 2 ≥ 4 k ∗ c k ∗ OPT . Thus, as long as sin θ ≥ 4 e √ OPT , we have that g ( w ) · w ∗ ≤ -1 2 ∥ g ∗ ( w ) ∥ 2 sin θ.

3.2 Concentration of Gradients

Define the empirical estimate of g ( w ) as ̂ g ( w ) := (1 /n ) ∑ n i =1 k ∗ c k ∗ y ( i ) P w ⊥ ⟨ He k ∗ ( x ( i ) ) , w ⊗ k ∗ -1 ⟩ . The following lemma provides the upper bounds on the number of samples required to approximate the Riemannian gradient g ( w ) by ̂ g ( w ) .

Lemma 3.4 (Concentration of Gradients) . Let w ∗ , w ∈ S d -1 . Let ̂ g ( w ) be the empirical estimate of the Riemannian gradient. Furthermore, denote the angle between w and w ∗ by θ , and denote κ = ( k ∗ c k ∗ ) 2 e k ∗ log k ∗ ( B 4 /ϵ ) . Then, with probability at least 1 -δ , it holds ∥ ̂ g ( w ) -g ( w ) ∥ 2 ≲ √ dκ/ ( nδ ) , and ( ̂ g ( w ) -g ( w )) · w ∗ ≲ √ κ/ ( nδ ) sin 2 θ .

3.3 Proof of Main Theorem

We proceed to the main theorem of this paper. It shows that using at most ˜ Θ( d ⌈ k ∗ / 2 ⌉ + d/ϵ ) samples, Algorithm 2 (with initialization from Algorithm 1) generates a vector ̂ w such that L σ 2 ( ̂ w ) = O (OPT) + ϵ within O (log(1 /ϵ )) iterations.

Theorem 3.5. Suppose Assumption 1 holds. Choose the batch size of Algorithm 2 to be n = Θ( C k ∗ de k ∗ log k ∗ +1 ( B 4 /ϵ ) / ( ϵδ )) , and choose the step size η = 9 / (40 ek ∗ c k ∗ ) . Then, after T = O (log( C k ∗ /ϵ )) iterations, with probability at least 1 -δ , Algorithm 2 outputs w T with L σ 2 ( w T ) = O ( C k ∗ OPT) + ϵ . The total sample complexity of Algorithm 2 is N = Θ(( k ∗ /c k ∗ ) 2 e k ∗ log k ∗ ( B 4 /ϵ ) d ⌈ k ∗ / 2 ⌉ +( C k ∗ e k ∗ log k ∗ +2 ( B 4 /ϵ )) d ϵδ ) .

Proof Sketch of Theorem 3.5. Suppose first that OPT ≥ ( c k ∗ / 64 k ∗ ) 2 , then by Claim E.7 we know that any unit vector (e.g., ̂ w = e 1 ) is a constant approximate solution with L σ 2 ( ̂ w ) = O (OPT) . Now suppose that OPT ≤ ( c k ∗ / 64 k ∗ ) 2 . Consider the distance between w t and w ∗ after each update of Algorithm 2. By the non-expansive property of projection operators, we have

Let θ t = θ ( w t , w ∗ ) . For the chosen batch size, Lemma 3.4 implies

Assume for now that sin θ t ≥ 4 e OPT+ √ ϵ , hence Lemma 3.3 applies. As ̂ g ( w t ) · ( w ∗ -w t ) = ( ̂ g ( w t ) -g ( w t )) · w ∗ + g ( w t ) · w ∗ , using Lemma 3.3, we get

On the other hand, the squared norm term ∥ ̂ g ( w t ) ∥ 2 2 from Equation (12) can be bounded above by

∥ ̂ g ( w t ) ∥ 2 2 ≤ 2 ∥ ̂ g ( w t ) -g ( w t ) ∥ 2 2 +2 ∥ g ( w t ) ∥ 2 2 ≤ ( k ∗ c k ∗ ) 2 (OPT + ϵ ) + ∥ g ∗ ( w ) ∥ 2 2 . (14) Plugging Equation (13) and Equation (14) back into Equation (12), we get that w.p. at least 1 -δ ,

Let us assume first that θ t ≤ θ t -1 ≤ · · · ≤ θ 0 and sin θ t ≥ 4 e OPT + √ ϵ , then we show that it holds θ t +1 ≤ θ t . Recall in Claim 3.1 it was shown that ∥ g ∗ ( w t ) ∥ 2 = 2 k ∗ c k ∗ ( w t · w ∗ ) k ∗ -1 sin θ t . Since w 0 is the initial parameter vector that satisfies w 0 · w ∗ ≥ 1 -1 /k ∗ , by the inductive hypothesis it holds w t · w ∗ ≥ 1 -1 /k ∗ and hence 1 ≥ ( w t · w ∗ ) k ∗ -1 ≥ 1 /e . Therefore, we further obtain (2 k ∗ c k ∗ /e ) sin θ t ≤ ∥ g ∗ ( w t ) ∥ 2 ≤ 2 k ∗ c k ∗ sin θ t . Therefore, using the inductive assumption that sin θ t ≥ 4 e √ OPT + √ ϵ and further noticing that ∥ w t -w ∗ ∥ 2 = 2sin( θ t / 2) , with step size η = 9 / (40 ek ∗ c k ∗ ) we can further bound ∥ w t +1 -w ∗ ∥ 2 2 in Equation (15) as:

This shows that θ t +1 ≤ θ t , hence completing the inductive argument. Furthermore, Equation (16) implies that after at most T = O (log(1 /ϵ )) iterations it must hold that sin θ T ≤ 4 e √ OPT + √ ϵ , therefore, we can end the algorithm after at most O (log(1 /ϵ )) iterations. Applying Claim E.7 we know that this final vector w T has error bound L σ 2 ( w T ) = O ( C k ∗ (OPT + ϵ )) . Thus, choosing ϵ ′ = C k ∗ ϵ , δ ′ = δT , where T = O (log( C k ∗ /ϵ ′ )) , Algorithm 2 outputs a parameter w T such that with probability at least 1 -δ ′ , L σ 2 ( w T ) = O ( C k ∗ OPT) + ϵ ′ , with batch size n = ˜ Θ( d/ ( ϵ ′ δ ′ )) . In summary, the total number of samples required for Algorithm 2 is N = ˜ Θ( d ⌈ k ∗ / 2 ⌉ + d/ ( ϵ ′ δ ′ )) .

Acknowledgments

PW was supported in part by NSF Award DMS-2023239. NZ was supported in part by NSF Medium Award CCF-2107079. ID was supported in part by NSF Medium Award CCF-2107079 and an H.I. Romnes Faculty Fellowship. JD was supported in part by the Air Force Office of Scientific Research under award number FA9550-24-1-0076 and by the U.S. Office of Naval Research under contract number N00014-22-1-2348. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. Department of Defense.

References

[AAM23] E. Abbe, E. B. Adsera, and T. Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory , pages 2552-2623. PMLR, 2023.

[ADGM17] A. Anandkumar, Y. Deng, R. Ge, and H. Mobahi. Homotopy analysis for tensor pca. In Conference on Learning Theory , pages 79-104. PMLR, 2017.

[AS68] M. Abramowitz and I. A. Stegun. Handbook of mathematical functions with formulas, graphs, and mathematical tables , volume 55. US Government printing office, 1968.

[ATV23] P. Awasthi, A. Tang, and A. Vijayaraghavan. Agnostic learning of general ReLU activation using gradient descent. In The Eleventh International Conference on Learning Representations, ICLR , 2023.

[BAGJ20] G. Ben Arous, R. Gheissari, and A. Jagannath. Algorithmic thresholds for tensor pca. The Annals of Probability , 48(4):2052-2087, 2020.

[BAGJ21] G. Ben Arous, R. Gheissari, and A. Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. Journal of Machine Learning Research , 22(106):1-51, 2021.

[BvH22] T. Brailovskaya and R. van Handel. Universality and sharp matrix concentration inequalities. arXiv preprint arXiv:2201.05142 , 2022.

[CCFM19] Y. Chen, Y. Chi, J. Fan, and C. Ma. Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval. Mathematical Programming , 176:5-37, 2019.

[CLS15] E. J. Candes, X. Li, and M. Soltanolkotabi. Phase retrieval via wirtinger flow: Theory and algorithms. IEEE Transactions on Information Theory , 61(4):1985-2007, 2015.

[CM20] S. Chen and R. Meka. Learning polynomials in few relevant dimensions. In Conference on Learning Theory , pages 1161-1227. PMLR, 2020.

[DGK + 20] I. Diakonikolas, S. Goel, S. Karmalkar, A. R. Klivans, and M. Soltanolkotabi. Approximation schemes for ReLU regression. In Conference on Learning Theory, COLT , volume 125 of Proceedings of Machine Learning Research , pages 1452-1485. PMLR, 2020.

[DH18] R. Dudeja and D. Hsu. Learning single-index models in Gaussian space. In Conference on Learning Theory, COLT , volume 75 of Proceedings of Machine Learning Research , pages 1887-1930. PMLR, 2018.

[DH21] R. Dudeja and D. Hsu. Statistical query lower bounds for tensor pca. Journal of Machine Learning Research , 22(83):1-51, 2021.

[DJS08] A. S. Dalalyan, A. Juditsky, and V. Spokoiny. A new algorithm for estimating the effective dimension-reduction subspace. The Journal of Machine Learning Research , 9:1647-1678, 2008.

[DKPZ21] I. Diakonikolas, D. M. Kane, T. Pittas, and N. Zarifis. The optimality of polynomial regression for agnostic learning under Gaussian marginals in the SQ model. In Proceedings of The 34 th Conference on Learning Theory, COLT , 2021.

[DKR23] I. Diakonikolas, D. M. Kane, and L. Ren. Near-optimal cryptographic hardness of agnostically learning halfspaces and ReLU regression under Gaussian marginals. In ICML , 2023.

[DKTZ22] I. Diakonikolas, V. Kontonis, C. Tzamos, and N. Zarifis. Learning a single neuron with adversarial label noise via gradient descent. In Conference on Learning Theory (COLT) , pages 4313-4361, 2022.

[DKZ20] I. Diakonikolas, D. M. Kane, and N. Zarifis. Near-optimal SQ lower bounds for agnostically learning halfspaces and ReLUs under Gaussian marginals. In Advances in Neural Information Processing Systems, NeurIPS , 2020.

[DLS22] A. Damian, J. Lee, and M. Soltanolkotabi. Neural networks can learn representations with gradient descent. In Conference on Learning Theory , pages 5413-5452. PMLR, 2022.

[DNGL23] A. Damian, E. Nichani, R. Ge, and J. D. Lee. Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models. Advances in Neural Information Processing Systems , 36, 2023.

[DPVLB24] A. Damian, L. Pillaud-Vivien, J. D. Lee, and J. Bruna. The computational complexity of learning gaussian single-index models. arXiv preprint arXiv:2403.05529 , 2024.

[FCG20] S. Frei, Y. Cao, and Q. Gu. Agnostic learning of a single neuron with gradient descent. In Advances in Neural Information Processing Systems, NeurIPS , 2020.

[FFRS21] A. Fiorenza, M. R. Formica, T. G. Roskovec, and F. Soudsk` y. Detailed proof of classical gagliardo-nirenberg interpolation inequality with historical remarks. Zeitschrift für Analysis und ihre Anwendungen , 40(2):217-236, 2021.

[GGK20] S. Goel, A. Gollakota, and A. R. Klivans. Statistical-query lower bounds via functional gradients. In Advances in Neural Information Processing Systems, NeurIPS , 2020.

[GGKS23] A. Gollakota, P. Gopalan, A. R. Klivans, and K. Stavropoulos. Agnostically learning single-index models using omnipredictors. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

[Hau92] D. Haussler. Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation , 100:78-150, 1992.

[HJS01] M. Hristache, A. Juditsky, and V. Spokoiny. Direct estimation of the index coefficient in a single-index model. Annals of Statistics , pages 595-623, 2001.

[HMS + 04] W. Härdle, M. Müller, S. Sperlich, A. Werwatz, et al. Nonparametric and semiparametric models , volume 1. Springer, 2004.

[HSS15] S. B. Hopkins, J. Shi, and D. Steurer. Tensor principal component analysis via sum-ofsquare proofs. In Conference on Learning Theory , pages 956-1006. PMLR, 2015.

[HSSS16] S. B. Hopkins, T. Schramm, J. Shi, and D. Steurer. Fast spectral algorithms from sumof-squares proofs: tensor decomposition and planted sparse vectors. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 178-191, 2016.

[Ich93] H. Ichimura. Semiparametric least squares (SLS) and weighted SLS estimation of single-index models. Journal of econometrics , 58(1-2):71-120, 1993.

[KKSK11] S. M Kakade, V. Kanade, O. Shamir, and A. Kalai. Efficient learning of generalized linear and single index models with isotonic regression. Advances in Neural Information Processing Systems , 24, 2011.

[KS09] A. T. Kalai and R. Sastry. The isotron algorithm: High-dimensional isotonic regression. In COLT , 2009.

[KSS94] M. Kearns, R. Schapire, and L. Sellie. Toward efficient agnostic learning. Machine Learning , 17(2/3):115-141, 1994.

[MBM18] S. Mei, Y. Bai, and A. Montanari. The landscape of empirical risk for nonconvex losses. The Annals of Statistics , 46(6A):2747-2774, 2018.

[Rah17] S. Rahman. Wiener-hermite polynomial expansion for multivariate gaussian probability measures. Journal of Mathematical Analysis and Applications , 454(1):303-334, 2017.

[RM14] E. Richard and A. Montanari. A statistical model for tensor PCA. Advances in neural information processing systems , 27, 2014.

[Ros58] F. Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review , 65(6):386-408, 1958.

[Sol17] M. Soltanolkotabi. Learning ReLUs via gradient descent. In Advances in neural information processing systems , pages 2007-2017, 2017.

[SQW18] J. Sun, Q. Qu, and J. Wright. A geometric analysis of phase retrieval. Foundations of Computational Mathematics , 18:1131-1198, 2018.

[WZDD23] P. Wang, N. Zarifis, I. Diakonikolas, and J. Diakonikolas. Robustly learning a single neuron via sharpness. 40th International Conference on Machine Learning , 2023.

[ZWDD24] N. Zarifis, P. Wang, I. Diakonikolas, and J. Diakonikolas. Robustly learning singleindex models via alignment sharpness. 41th International Conference on Machine Learning, arXiv preprint: 2402.17756 , 2024.

Supplementary Material

Organization The appendix is organized as follows. In Appendix A, we provide a detailed discussion about our assumptions on the activation functions. In Appendix B we compare our results with the most related prior works. In Appendix C, we provide additional preliminaries on basic tensor algebra as well as Hermite polynomials and Hermite tensors. In Appendix D and Appendix E we provide full versions of Section 2 and Section 3 respectively, with omitted lemmas and proofs.

A Remarks on the Assumptions

Assumption 1( i ) appears in the same form in [BAGJ21, DNGL23]. The remaining two assumptions are implied by | σ ′ ( z ) | ≤ A | z | q + B for constants A,B,q , assumed in these works. In detail, Assumption 1 ( ii ) can be implied from | σ ′ ( z ) | ≤ A | z | q + B using the Gagliardo-Nirenberg inequality [FFRS21]; Assumption 1 ( iii ) follows from direct calculations as E z ∼N 1 [( A | z | q + B ) 2 ] is finite. Hence Assumption 1 is no stronger than the assumptions made in prior work, which considered the less challenging realizable and zero-mean label noise settings.

Some activations satisfying Assumption 1 are:

All ' ( a, b ) -well-behaved' activations [DGK + 20, DKTZ22, WZDD23] that are non-decreasing, zero at the origin, b -Lipschitz and σ ′ ( z ) ≥ a when z ∈ [0 , R ] satisfy Assumption 1 with k ∗ = 1 (see Claim A.1). This includes ReLU, σ ( z ) = max { 0 , z } and sigmoid, σ ( z ) = e z / (1 + e z ) .

Activations for phase-retrieval [CLS15, CCFM19, SQW18]: σ ( z ) = z 2 and σ ( z ) = | z | have information component k ∗ = 2 . One can verify that they satisfy Assumption 1 after normalization.

Claim A.1. Let a, b, R > 0 be some absolute constants. Let σ : R → R be an activation such that it is non-decreasing, b -Lipschitz, and satisfies σ (0) = 0 and σ ′ ( z ) ≥ a for all z ∈ [0 , R ] . Then σ satisfies Assumption 1 with information component k ∗ = 1 .

Proof. We calculate the Hermite coefficient c 1 of σ :

where in ( i ) we used the monotonicity property of σ and that σ (0) = 0 . Thus we get that all wellbehaved activations as k ∗ = 1 . In addition, the E z ∼N 1 [ σ ( z ) 2 ] ≤ E z ∼N 1 [ b 2 z 2 ] ≤ b 2 , hence after normalization, we have c 1 ≳ aR 3 exp( -R 2 ) / (2 b 2 ) , which is an absolute constant bounded away from 0. Finally, we have E z ∼N 1 [ σ ( z ) 4 ] ≤ 3 b 4 and E z ∼N 1 [( σ ′ ( z )) 2 ] ≤ b 2 since σ is b -Lipschitz. Thus, we have that all well-behaved activations satisfy Assumption 1.

B Comparison with Prior Work

B.1 Comparison with Prior Works on Agnostically Learning SIMs

A long thread of research has been focusing on agnostic learning of single index models, including [FCG20, DGK + 20, DKTZ22, WZDD23, ZWDD24]. A common assumption on the activation function σ used in these works is the so-called 'well-behaved' property, namely that σ is nondecreasing, zero at the origin ( σ (0) = 0 ), b -Lipshitz, and σ ′ ( z ) ≥ a when z ∈ [0 , R ] . In particular, ReLUs and sigmoids are well-behaved activations. For well-behaved activations, we have shown in Claim A.1 that they have k ∗ = 1 and satisfy Assumption 1. Therefore, we conclude that our assumption Assumption 1 is indeed milder compared to prior works.

We also note that Assumption 1 allows for non-monotonic activations as well, for example σ ( z ) = z 2 and σ ( z ) = | z | satisfy Assumption 1 with k ∗ = 2 .

At the level of techniques, the algorithmic approaches in the aforementioned prior works on agnostically learning SIMs [FCG20, DGK + 20, WZDD23] inherently fail in our more general activation setting. The main reason is that the underlying algorithms only exploit the information in the degree

and

1 -Chow parameters. However, under Assumption 1 with k ≥ 2 , the inner product between degree 1 -Chow vector and any unit vector v equals | E ( x ,y ) ∼D [ y x ] · v | ≤ √ OPT -which provides no information about the hidden vector w ∗ . That is, in our more general setting, considering Chow vectors of higher degree appears necessary for any optimization method to succeed.

∗

On the other hand, we remark that the algorithms in these works succed under more general distributions (including the Gaussian distribution), such as log-concave distributions.

B.2 Comparison with [DNGL23]

The work by [DNGL23] applied a smoothing operator to the L 2 2 loss inspired by [ADGM17]. For any function f : R → R , let the smoothing operator be

where z ∼ µ w is the uniform distribution on the sphere S d -2 that is orthogonal to w , and λ is the 'smoothing strength'. Applying the smoothing operator to the loss we get

The main algorithm of [DNGL23] is an online Riemannian gradient descent algorithm on the smoothed loss L λ ( w ) .

The dynamics of the online SGD algorithm in [DNGL23] can be split into three stages: In the first stage, starting from a randomly initialized vector w 0 such that w 0 · w ∗ = Θ( d -1 / 2 ) , the algorithm used a large smoothing operator with λ = d 1 / 4 . Then, after ˜ O ( d k ∗ / 2 -1 ) iterations the algorithm converges to a parameter w 1 such that w 1 · w ∗ ≥ d -1 / 4 . Then, in the second stage, with zerosmoothing λ = 0 , they run the algorithm for another ˜ O ( d k ∗ / 2 -1 ) iterations and get a parameter w 2 such that w 2 · w ∗ ≥ 1 -d -1 / 4 . In the third stage, online SGD converges to w 3 with w 3 · w ∗ ≥ 1 -ϵ in d/ϵ iterations.

However, the smoothing technique does not work in the agnostic setting. The main reason is that the agnostic noise buries the signal of the gradient. To see this, note that in Lemma 11 of [DNGL23], they showed that g λ (he k ( w · x )) = ⟨ He k ( x ) , T k ( w ) ⟩ , where

and ν ( d -1) j = E z ∼ S d -2 [ z 2 j 1 ] = Θ(( d -1) -j ) = Θ( d -j ) . Note since the smoothing operator is a linear operator, it holds that L λ ( w ) = 1 -E ( x ,y ) ∼D [ yg λ ( σ ( w · x ))] = 1 -E ( x ,y ) ∼D [ y ∑ k ≥ k ∗ c k g λ (he k ( w · x ))] = 1 -E ( x ,y ) ∼D [ y ∑ k ≥ k ∗ c k ⟨ He k ( x ) , T k ( w ) ⟩ ] . Let g λ ( w ) := P w ⊥ ∇L λ ( w ) denote the Riemannian gradient of L λ ( w ) , we have

Weshow that the noise term I 2 kills the signal term I 1 in the beginning stage when w · w ∗ = Θ(1 / √ d ) and λ = d 1 / 4 .

Claim B.1. When w · w ∗ = Θ(1 / √ d ) and λ = d 1 / 4 , it holds

Thus, in order for the signal to overcome the noise, one requires OPT ≲ d -k ∗ / 4 , which is too strict to hold in reality.

Proof of Claim B.1. Following the steps in Lemma 12 in [DNGL23], ∇ T k ( w ) ⊗ ( w ∗ ) ⊥ w equals:

Plugging the equation above back into Equation (17), we study each term |I 1 | and |I 2 | respectively in the beginning phase when w · w ∗ = O ( d -1 / 2 ) .

Using Fact C.3, and recall that in [DNGL23] λ is chosen to be λ = d 1 / 4 and ν ( d -1) j = Θ( d -j ) , ν ( d +1) j = Θ( d -j ) by definition, I 1 equals:

where in ( i ) we plugged in the value of λ and ν ( d -1) j , ν ( d +1) j ; in ( ii ) we plugged in the value of w · w ∗ = 1 / √ d ; and in ( iii ) we used the fact that ( k j ) ≤ 2 k . However, the magnitude of the signal from |I 1 | is much smaller compared to the strength of the noise, |I 2 | , as by Cauchy-Schwarz, we

have

Note that in Lemma 12 of [DNGL23], it was proved that

where in ( v ) we plugged in the value of λ and ν ( d -1) j , ν ( d +1) j . Thus, it holds that

Finally, we remark that the equality holds at ( iv ) when

B.3 Comparison with [DPVLB24]

In [DPVLB24], the authors studied a milder noise model than the agnostic setting. In their model, the joint distribution on ( x , y ) is defined as E y [ y | x ] = σ ( w ∗ · x ) + ξ ( w ∗ · x ) , where ξ : R ↦→ R is assumed to be known to the learner. Note that in this model the labels y are independent of all the directions orthogonal to w ∗ , i.e., the random vector x ⊥ w ∗ is independent of y . In comparison, in the agnostic setting, the distribution of the labels is E y [ y | x ] = σ ( w ∗ · x ) + ξ ′ ( x ) , where ξ ′ : R d ↦→ R is unknown and can depend arbitrarily on x . In particular, the aforementioned independence with the directions orthogonal to w ∗ does not hold. As a result of their milder noise model, the authors can utilize information on the joint distribution to mitigate the corruption of the noise. This assumption is significantly weaker than agnostic noise.

In addition, instead of studying the information component k ∗ defined as the first non-zero Hermite coefficient of the activation σ , [DPVLB24] considered the generative component of the label y , which is defined as ¯ k ∗ := min k ≥ 0 { λ k > 0 } , where λ k := √ E y [ ζ 2 k ] , and ζ k ( y ) := E z [he k ( z ) | y ] . The main results in [DPVLB24] are twofold. First, they proved an SQ lower bound showing that under the setting aforementioned, any polynomial time SQ algorithm requires at least O ( d ¯ k ∗ / 2 ) samples to learn the hidden direction w ∗ . Then, they provided an SQ algorithm using partial-trace operators that returns a vector ̂ w such that ( ̂ w · w ∗ ) 2 ≥ 1 -ϵ 2 with O ( d ¯ k ∗ / 2 + d/ϵ 2 ) samples, matching the SQ lower bound.

The algorithm proposed in [DPVLB24] can be described in short as follows: given joint distribution P , calculate ζ ¯ k ∗ ( y ) and transform the label y to ζ ¯ k ∗ ( y ) . Then, they applied tensor PCA on ζ ¯ k ∗ ( y ) He k ( x ) , using the partial trace operator and tensor power iteration.

We note that the partial trace method for tensor PCA fails to find an initialization vector w 0 such that w 0 · w ∗ ≥ c for some positive absolute constant c under agnostic noise. We explain this briefly below. For simplicity, let us consider k being even. Note that for a k -tensor T with even k , the partial trace operator is defined by PT ( T ) = ⟨ T , I ⊗ ( k -2) / 2 ⟩ , where I is the identity matrix in R d × d . The idea

in [DPVLB24] and [ADGM17] is to use the top eigenvector v 1 ∈ R d of PT ( E ( x ,y ) ∼D [ y He k ( x )]) as a warm-start. However, we show that under agnostic noise, this eigenvector v 1 does not contain a strong enough signal to provide information of w ∗ . Note that by definition of the partial trace operator, for any v ∈ B d we have

The first term in the equality above equals c k ⟨ w ∗⊗ k , I ⊗ ( k -2) / 2 ⊗ v ⊗ 2 ⟩ = c k ( w ∗ · v ) 2 , however, the second term can be as large as (by Cauchy-Schwarz):

As suggested by inequality ( i ) above, let v be any unit vector orthogonal to w ∗ , consider an agnostic noise model then

where ( ii ) comes from the fact that for any permutation π such that { π (1) , π (2) }∩{ 1 , 2 } ̸ = ∅ , since v ⊥ w ∗ , it holds

and on the other hand, when { π (1) , π (2) } ∩ { 1 , 2 } = ∅ , then

To see this, note that there exists a chain of identity matrices

such that π ( a 1 ) = 1 , π ( a 1 +1) = j 1 , j 1 +1 = π ( a 2 ) , π ( a 2 +1) = j 2 . . . until we have i π ( a m +1) ∈ { i 1 , i 2 } or i j m +1 ∈ { i π (1) , i π (2) } . For the latter case, it implies that

̸

Therefore, only the first case is interesting. Since π ( a 1 ) = 1 , and π is a bijection, hence π ( a m +1) = 1 therefore the only possible case would be i π ( a m +1) = i 2 , then we have

Continue the discussion for v i 2 , w ∗ i π (1) and w ∗ i π (2) , and let { j 5 , j 6 , . . . , j k -1 , j k } = { i 1 , . . . , i k } -{ i 1 , i 2 , i π (1) , i π (2) } , we get that

Similarly, for any unit vector u ⊥ v and u ⊥ w ∗ , it holds

But one can also show that

This implies that to guarantee that w ∗ is the unique top eigenvector, it has to be that OPT ≲ c 2 k d -( k -2) / 2 . Therefore, the noise term completely buries the signal ( w ∗ · v ) 2 unless OPT ≲ d -( k -2) / 2 , which is unrealistic to assume.

B.4 Remarks on Tensor PCA

We summarize some historical bits of Tensor PCA. [RM14] proposed the following 'spiked' tensor PCA problem: given a k -tensor of the form 3

where A is a k -tensor with i.i.d. standard Gaussian entries, recover the planted vector v . The 'singleobservation' model is equivalent (in law) to the following 'multi-observation' model([BAGJ20]): given n i.i.d. copies T ( i ) = τ ′ v ⊗ k + A ( i ) with τ ′ = τ/ √ n , recover v using the empirical estimation:

In [RM14], it has been shown that for model (PCA-S), it is information-theoretically impossible to recover v when τ < ( β ∗ k -o d (1)) √ d , for some real constant β ∗ k ; however, there also exists a constant β ′ k such that the information-theoretic optimal threshold for τ is τ > ( β ′ k + o d (1)) √ d (see also [DH21]). However, there is a huge statistical-computational gap for solving tensor PCA problems and it is conjectured impossible to solve (PCA-S) when τ ≲ d k/ 4 for k ≥ 3 [RM14, DH21]. For multi-observation model (PCA-M), this thresholds translates to a sample complexity of n ≳ d k/ 2 when τ ′ = O (1) .

[RM14] proposed the tensor unfolding algorithm that recovers v in (PCA-S) when τ ≳ d ⌈ k/ 2 ⌉ / 2 , i.e., for (PCA-M), the required sample complexity is Ω( d ⌈ k/ 2 ⌉ ) . However, it is conjectured in [RM14] that the tensor unfolding algorithm can actually deal with τ ≳ d k/ 4 .

Note that the unfolding algorithm requires τ ≳ d when k = 3 . Many papers are devoted to improving from τ ≳ d to τ ≳ d 3 / 4 and reducing the runtime and memory cost. To name a few, [HSS15, HSSS16] used Sum-of-Squares algorithms with partial trace operators to achieve the goal within O ( d 3 ) runtime; [ADGM17] also used partial trace operators, but instead of using SOS algorithms, they injected noise to smooth the landscape of the loss.

Perhaps an interesting aspect of our paper is that the unfolding algorithm can deal with stronger noise (compared to the Gaussian noise A , our noise is very heavy-tailed) and is more robust to the noise, as partial trace operator does not work under our agnostic noise. However, this might be attributed to the special structure of the noisy chow tensor.

C Additional Preliminaries

C.1 Elementary Tensor Algebra

We review basic definitions and elementary tensor algebra used throughout the paper. For any positive integer k , a k -tensor T is defined as a multilinear real function that maps k vectors to a real number, i.e., T : R d ×··· × R d → R . A k -tensor T can also be viewed as a multidimensional array, where each entry is associated with k indices i 1 , . . . , i k in [ d ] and equals T i 1 ,...,i k = T ( e i 1 , . . . , e i k ) .

Given a vector w ∈ R d , w = ( w 1 , . . . , w d ) ⊤ , the k -product-tensor w ⊗ k is defined by

Given a k -tensor T and an l -tensor T ′ (with l ≥ k ), the inner product (or contraction) between T and T ′ is defined by ( ⟨ T , T ′ ⟩ ) i k +1 ,...,i l := ∑ d i 1 ,i 2 ,...,i k ( T ) i 1 ,i 2 ,...,i k ( T ′ ) i 1 ,i 2 ,...,i k ,i k +1 ,...,i l . In other words, the inner product of a k -tensor and an l -tensor yields an ( l -k ) -tensor. When k = l , the inner product between T and T ′ is a real number. In particular, for vectors w , v ∈ R d , and any k ≥ 1 ,

where w · v is the standard inner product between vectors.

A tensor T is symmetric if ( T ) ...,i,...,j,... = ( T ) ...,j,...,i,... . We can symmetrize a k -tensor T by summing up all its copies with permuted indices i 1 , . . . , i k and dividing the sum by k ! , which is the total number of permutations. We define the symmetrization operator of a k -tensor T by

When k = 2 , this reduces to the symmetrization of a square matrix: Sym( T ) = (1 / 2)( T + T ⊤ ) .

Let us provide some useful observations about tensor algebra.

Fact C.1. Let w ∈ R d and let T be any k -tensor. Then,

The inner product between w ⊗ k and Sym( T ) is equal to the inner product between w ⊗ k and T :

Proof. To prove the first statement in Fact C.1, note that direct calculation yields:

If T is a symmetric tensor,

Next for the second statement, let f ( w ) = ⟨ T , w ⊗ k ⟩ where T is a k -tensor. Then, the partial derivative of f w.r.t. w j is

C.2 Hermite Polynomials and Hermite Tensors

We make use of the normalized probabilist's Hermite polynomial, defined by

We will heavily use the following properties of the normalized Hermite polynomials [AS68]:

Fact C.2. Hermite polynomials satisfy the following properties:

(Recurrence) he ′ k ( z ) = √ k he k -1 ( z ) .

Given a vector x ∈ R d , we can then define the (normalized) Hermite multivariate tensor by [Rah17]:

For Hermite tensors, we have the following facts:

Fact C.3. Let x be a d -dimensional standard Gaussian random vector.

For any k -tensor A and j -tensor B ,

For any w ∈ R d such that ∥ w ∥ 2 = 1 , he k ( w · x ) = ⟨ He ( x ) , w ⊗ k ⟩ .

C.3 Loss and Gradients

We consider the L 2 2 (or square) loss, defined by

Let w ∗ ∈ argmin w ∈ S d -1 L σ 2 ( w ) , and denote the minimum value of the L 2 2 loss by OPT := min w ∈ S d -1 L σ 2 ( w ) . Furthermore, let us define the 'noiseless' L 2 2 loss by

We observe that the L 2 2 loss is determined by the inner product between w and w ∗ , therefore, to obtain error O (OPT)+ ϵ , it suffices to minimize the angle between w and w ∗ . Concretely, we have:

Claim C.4. Let w ∈ R d be a unit vector. Then, the L 2 2 loss L σ 2 ( w ) satisfies:

Proof. Recalling that the activation σ is normalized so that E x ∼N d [ σ 2 ( w · x )] = E x ∼N d [ σ 2 ( w ∗ · x )] = 1 , we can simplify the L 2 2 loss to

The noiseless L 2 2 loss admits the following decomposition:

where in the last equality we used the orthonormality property of Hermite tensors (Fact C.3). Further, using Young's inequality and the definitions of OPT and L ∗ σ 2 ( w ) , we also have L σ 2 ( w ) ≤ 2OPT + 2 L ∗ σ 2 ( w ) , which combined with Equation (24) leads to

Remark C.5. Equation (24) suggests that even in the realizable case, some assumption on boundedness of ∑ k ≥ k ∗ kc 2 k (see Assumption 1( iii )) may be necessary to have a nontrivial bound on the L 2 2 loss. Consider an algorithm that outputs a vector w such that w · w ∗ = 1 -α for some α ∈ (0 , 1) (if α = 0 , w = w ∗ since both vectors are on the unit sphere). Since ∑ k ≥ k ∗ c 2 k = 1 , we can also write L ∗ σ 2 ( w ) = 2 ∑ k ≥ k ∗ c 2 k (1 -(1 -α ) k ) . For k = Ω(1 /α ) , 1 -(1 -α ) k ≈ kα . Thus, if we want an algorithm that works generically for any target accuracy, ∑ k ≥ k ∗ kc 2 k ought to be bounded.

Remark C.6. Even though for ∥ w ∥ 2 = 1 we have he k ( w · x ) = ⟨ He k ( x ) , w ⊗ k ⟩ , the gradients with respect to w of these two functions are different in general. For example, for k = 2 , we have

which are equal in function value, but

are different. In particular, for the derivative of the left-hand side of Equation (24) to be equal to the derivative of its right-hand side, we need to use the tensor form of Hermite polynomials, because to ensure interchangeability of differentiation and summation, the sequence needs to be uniformly convergent. Note that E x ∼N d [ ∑ k ≥ k ∗ c k ⟨ He k ( x ) , w ⊗ k ⟩ ∑ k ′ ≥ k ∗ c k ′ ⟨ He k ′ ( x ) , w ∗⊗ k ′ ⟩ ] converges to ∑ k ≥ k ∗ c 2 k ( w · w ) k uniformly for all w ∈ R d , but the sequence E x ∼N d [ ∑ k ≥ k ∗ c k he k ( w · x ) ∑ k ′ ≥ k ∗ c k ′ he k ′ ( w ∗ · x )] converges to ∑ k c 2 k ( w · w ) k only when ∥ w ∥ 2 = 1 , since it requires w · x ∼ N (0 , 1) to ensure that E x ∼N d [he k ( w · x )he j ( w ∗ · x )] = 1 { k = j } ( w · w ∗ ) k .

As observed in Remark C.6, the gradients of he k ( w · x ) and ⟨ He k ( x ) , w ⊗ k ⟩ are different in general. Throughout the paper, we will be taking the gradient with respect to the tensor form of σ ( w · x ) ; in other words, ∇ σ ( w · x ) = ∇ ( ∑ k ≥ k ∗ c k ⟨ He k ( x ) , w ⊗ k ⟩ ) .

D Full Version of Section 2

In this section, we show how to get an initial parameter vector w 0 such that w 0 · w ∗ = 1 -ϵ 0 for some small constant ϵ 0 . The main technique is a tensor PCA algorithm that finds the principal

component of a noisy degreek -Chow tensor for any k ≥ k ∗ , as long as OPT ≲ c 2 k . Such a degreek Chow tensor is defined by C k = E ( x ,y ) ∼D [ y He k ( x )] , and we denote its noiseless counterpart by

Furthermore, let us denote the difference between C k and C ∗ k by

Note that since He k ( x ) is a symmetric tensor for any x , all C k , C ∗ k and H k are symmetric tensors.

We use the following matrix unfolding operator that maps a k -tensor to a matrix in R d l × d k -l . Concretely, given a k -tensor T , we define:

For notational convenience, we also define the 'vectorize' operator and 'tensorize' operator, which map a vector v ∈ R d l to an l -tensor for any integer l , and vice versa. In detail,

and conversely, we define

Finally, given a vector v ∈ R d l , we can also convert this vector to a matrix of size R d × d l -1 :

Some simple facts on the algebra of the unfolded matrix are in order.

Fact D.1. Let T be a symmetric k -tensor, and let r ∈ R d k -l , v ∈ R d l . Then

For any index i ∈ [ d l ]

, where i ′ 1 , . . . , i ′ l ∈ [ d ] satisfies i = i ′ 1 +( i ′ 2 -1) d + · · · +( i ′ l -1) d l -1 .

For any index j ∈ [ d k -l ] ,

where j ′ 1 , . . . , j ′ k -l ∈ [ d ] satisfies j ′ 1 + · · · +( j ′ k -l -1) d k -l -1 = j .

Finally,

Proof. First we show that for a symmetric tensor T , the linear transformation Mat ( l,k -l ) ( T ) r of vector r ∈ R d k -l is equal to the tensor inner product ⟨ T , Tensor ( r ) ⟩ . This can be proved by direct calculations that for any i ∈ [ d l ] :

where i ′ 1 , . . . , i ′ j ∈ [ d ] satisfies i = i ′ 1 +( i ′ 2 -1) d + · · · +( i ′ l -1) d l -1 . Observe that the summation above further equals

Similarly, for a symmetric tensor T and any vector v ∈ R d l , and any index j ∈ [ d k -l ] , it holds

Finally, combining Equation (26) and Equation (25) we get that for any v ∈ R d l , r ∈ R d k -l , the quadratic form v ⊤ Mat ( l,k -l ) ( T ) r equals v ⊤ Mat ( l,k -l ) ( T ) r = ⟨ T , Tensor ( v ) ⊗ Tensor ( r ) ⟩ .

Throughout this section, we define l = k/ 2 when k is even, and l = ( k -1) / 2 when k is odd. In other words, l = ⌊ k/ 2 ⌋ . We leverage the tensor unfolding algorithm proposed in [RM14], which can be described in short as follows. First we unfold the degreek Chow tensor to a matrix in R d l × d k -l , and find its top-left singular vector v ∈ R d l . Then, we calculate the matrix Mat (1 ,l -1) ( v ) , and find its top left singular vector u . One can show that this eigenvector u correlates with w ∗ significantly.

Algorithm 3 k -Chow Tensor PCA

1: Input: Parameters ϵ, k, ϵ 0 , c k , B 4 > 0 ; Sample access to D

2: Let l = ⌊ k/ 2 ⌋

3: Draw n = Θ( e k log k ( B 4 /ϵ ) d k -l / ( ϵ 2 0 ) + 1 /ϵ ) samples { ( x ( i ) , y ( i ) ) } n i =1 from D

4: Construct ̂ M := (1 /n ) ∑ n i =1 Mat ( l,k -l ) ( y ( i ) He k ( x ( i ) )) ; compute its top left singular vector ̂ v ∗

5: Compute the top-left singular vector ̂ u of the matrix Mat 1 ,l -1 ( ̂ v ∗ )

̂

6: Return: u

Our main result for initialization is the following:

Proposition D.2 (Initialization) . Suppose Assumption 1 holds. Assume that OPT ≤ c 2 k ∗ / (64 k ∗ ) 2 , and let ϵ 0 = c k ∗ / (256 k ∗ ) . Then, Algorithm 1 applied to Problem 1.1 with k = k ∗ uses

samples, runs in polynomial time, and outputs a vector w 0 ∈ S d -1 such that w 0 · w ∗ ≥ 1 -min { 1 /k ∗ , 1 / 2 } .

We remark here that Algorithm 3 can also be used to find an approximate solution of our agnostic learning problem; however the dependence on the value of OPT is suboptimal , scaling with its square-root. In particular, we have the following proposition:

Proposition D.3 (Solving the Agnostic Learning Problem Using Tensor PCA) . Suppose Assumption 1 holds. Assume that OPT ≤ c 2 k ∗ / (64 k ∗ ) 2 and ϵ ≤ 1 / 64 . Let ϵ 0 = c k ∗ ϵ/ 16 . Then, Algorithm 1 applied to Problem 1.1 with k = k ∗ uses n = Θ( e k ∗ log k ∗ ( B 4 /ϵ ) d ⌈ k ∗ / 2 ⌉ / ( c 2 k ∗ ϵ 2 ) + 1 /ϵ ) samples, runs in polynomial time, and outputs a vector w 0 ∈ S d -1 such that

Furthermore, the L 2 2 error of w 0 is at most

Thus, when OPT = 0 (i.e., in the realizable cases), applying Algorithm 3 with O ( d ⌈ k ∗ / 2 ⌉ /ϵ 2 ) samples recovers the hidden vector w ∗ .

Roadmap To prove Proposition D.2 and Proposition D.3 we need three main ingredients. First, we will show (in Lemma D.4 and its corollary Corollary D.5) that the top-left singular vector v ∗ of the unfolded matrix M := Mat ( l,k -l ) ( E ( x ,y ) ∼D [ y He k ( x )]) correlates significantly with the vectorized l -product tensor, Vec ( w ∗⊗ l ) . This indicates that v ∗ contains rich information about the direction of w ∗ . However, since we only have access to ̂ M , the empirical estimation of M , we need to ensure that the top-left singular vector of ̂ M , denoted by ̂ v ∗ , is close to v ∗ . This is proved in Lemma D.13 using sophisticated matrix concentration bounds. In particular, in Equation (35) we guarantee that the angle between ̂ v ∗ and v ∗ is bounded by O ( ϵ 0 /c k ) for any small constant ϵ 0 > 0 , provided that we take ˜ Θ( d ⌈ k/ 2 ⌉ /ϵ 2 0 ) samples and assume that OPT ≲ c 2 k . The inner product between ̂ v ∗ and Vec ( w ∗⊗ l ) can then be bounded below by 1 -O (( √ OPT+ ϵ 0 ) /c k ) . Combining with Corollary D.5, this implies that ̂ v ∗ correlates with Vec ( w ∗⊗ l ) significantly. Finally, in Lemma D.17 we show that after unfolding the R d l vector ̂ v ∗ to an R d × d l -1 matrix, its top-left singular vector u correlates with w ∗ significantly; it particular, we have w ∗ · u ≳ 1 -cϵ 0 for some absolute constant c > 0 . Combining these results and choosing ϵ 0 ≈ c k ∗ /k ∗ , we get Proposition D.2, and choosing ϵ 0 ≈ c k ∗ ϵ yields Proposition D.3.

D.1 Signal in the k -Chow Tensor

Our first observation is that for any left singular vector v of Mat ( l,k -l ) ( C k ) , the singular value ρ ( v ) is close to the inner product between v and Vec ( w ∗⊗ l ) , where l = ⌈ k/ 2 ⌉ . Concretely, we have:

Lemma D.4. Let v be any left singular vector of Mat ( l,k -l ) ( C k ) . Then,

Proof. Recall that the singular value of the left singular vector v satisfies

where we used Equation (27) in ( i ) . Since C k = C ∗ k + H k , we further have

We bound both terms above respectively. For the first term, plugging in the definition of C ∗ k and using Fact C.3, we have

note that we applied Fact C.3 in equation ( i ) and Fact C.1 (1) in ( ii ) . Next, for the second term, after applying Cauchy-Schwarz inequality, it holds

Since for any k -tensor A we have ∥ Sym( A ) ∥ F ≤ ∥ A ∥ F , and in addition, observe that as ∥ v ∥ 2 = ∥ r ∥ 2 = 1 it holds

we finally have

Combining Equation (28) and Equation (30), we get that for any v ∈ R d l , r ∈ R d k -l such that ∥ v ∥ 2 = ∥ r ∥ 2 = 1 , it holds

Therefore, the singular value of v must satisfy

where in the equation above, we used the observation that as ∥ Vec ( w ∗⊗ k -l ) ∥ 2 = ∥ w ∗⊗ k -l ∥ F = 1 , it holds max r ∈ R d k -l , ∥ r ∥ 2 =1 ( Vec ( w ∗⊗ k -l ) · r ) = ∥ Vec ( w ∗⊗ k -l ) ∥ 2 = 1 .

Similarly, we have

completing the proof of Lemma D.4.

A direct application of Lemma D.4 is that the top-left singular vector v ∗ ∈ R d l of Mat ( l,k -l ) ( C k ) has singular value at least c k -√ OPT , and in addition, v ∗ aligns well with Vec ( w ∗⊗ l ) .

Corollary D.5. The top-left singular vector v ∗ ∈ R d l of the unfolded tensor Mat ( l,k -l ) ( C k ) has corresponding singular value ρ ( v ∗ ) ≥ c k -√ OPT . In addition, it holds that v ∗ · Vec ( w ∗⊗ l ) ≥ 1 -(2 √ OPT) /c k .

Proof. Plugging in v = Vec ( w ∗⊗ l ) to Lemma 2.2, we get that ρ ( Vec ( w ∗⊗ l )) ≥ c k -√ OPT . Thus, the top singular value must satisfy ρ 1 ≥ c k -√ OPT . Recall again that as proved in Lemma 2.2, it holds ρ ( v ∗ ) ≤ c k v ∗ · Vec ( w ∗⊗ l ) + √ OPT . Thus, since ρ ( Vec ( w ∗⊗ l )) ≥ c k -√ OPT we have v ∗ · Vec ( w ∗⊗ l ) ≥ 1 -(2 √ OPT) /c k .

D.2 Concentration of the Unfolded Tensor Matrix

We start with some notations. Let us denote M ( i ) = Mat ( l,k -l ) ( y ( i ) He k ( x ( i ) )) for i ∈ [ n ] and ̂ M = 1 n ∑ n i =1 M ( i ) , which is the empirical approximation of M = Mat ( l,k -l ) ( C k ) = Mat ( l,k -l ) ( E ( x ,y ) ∼D [ y He k ( x )]) . We will use Wedin's theorem to bound the distance between the top left singular vector v ∗ of M and the top singular vector ̂ v ∗ of the empirical ̂ M .

Fact D.6 (Wedin's theorem) . Let θ ( v ∗ , ̂ v ∗ ) be the angle between the top left singular vectors v ∗ ∈ R d l and ̂ v ∗ ∈ R d l of M and ̂ M respectively. Let ρ 1 and ρ 2 be the first 2 singular values of M . Then, it holds that:

We first observe that M admits a large gap between the first and second singular values.

Claim D.7 (Singular Gap of Unfolded Tensor Matrix) . Let ρ 1 , ρ 2 be the top two singular values of M = Mat ( l,k -l ) ( C k ) . Then ρ 1 -ρ 2 ≥ ( c k -8 √ OPT) / 2 .

Proof. Recall that in Corollary D.5 we showed ρ 1 = ρ ( v ∗ ) ≥ c k -√ OPT and v ∗ · Vec ( w ∗⊗ l ) ≥ 1 -(2 √ OPT) /c k . Now let v ∈ R d l be any left singular vector of M that is orthogonal to v ∗ . We can decompose Vec ( w ∗⊗ l ) into Vec ( w ∗⊗ l ) = a v ∗ + b v + v ′ where v ′ is orthogonal to both v ∗ and v , and a 2 + b 2 ≤ 1 . Then, since Vec ( w ∗⊗ l ) · v ∗ = a ≥ 1 -(2 √ OPT) /c k , we thus have Vec ( w ∗⊗ l ) · v = b ≤ √ 1 -a 2 ≤ 1 -a 2 / 2 . This implies that

and hence we get ρ 1 -ρ 2 ≥ ( c k -8 √ OPT) / 2 , completing the proof of Claim D.7.

Thus, our remaining goal is to bound the operator norm of M -̂ M . For this purpose, we use the following matrix concentration inequality from [DPVLB24] (also Theorem 2.7 in [BvH22]).

Fact D.8 (Lemma I.5 [DPVLB24]) . Let Z ( i ) , i ∈ [ n ] , be independent, mean-zero, self-adjoint matrices. Define:

Then, for any R ≥ ¯ R 1 / 2 γ 1 / 2 + √ 2 ¯ R , and any t ≥ 0 , if δ = Pr [max i ∈ [ n ] ∥ Z ( i ) ∥ 2 ≥ R ] , then with probability at least 1 -δ -de -t ,

However, note that ̂ M and M are not symmetric matrices, hence to apply matrix concentration inequalities we will be working on the symmetrization of ̂ M and M for simplicity, which we will denote by ̂ P and P :

Before we prove the main theorem of this subsection, we introduce two final pieces of tools that will be used later in the proof. The first one is Gaussian hypercontractivity.

Fact D.9 (Gaussian Hypercontractivity) . Let f ( x ) : R d → R be a multivariate polynomial of degree at most k . Let x be a standard Gaussian random variable of R d . Then, for any p ≥ 1 it holds

Gaussian hypercontractivity controls the moments of a polynomial f ( x ) . To utilize the bound on these moments, we make use of the following inequality from [DNGL23].

Fact D.10 (Lemma 23 [DNGL23]) . Let A,B be random variables such that ∥ B ∥ L p ≤ σ B p C for all p ≥ 1 and some positive real numbers σ B , C . Then,

We also make use of the following lemma that bounds the magnitude of label y without loss of generality.

Lemma D.11 (Bound on Labels) . Let P B y ( z ) : R → R be a function that truncates the value of z to the threshold B y : P B y ( z ) = z 1 {| z | ≤ B y } + B y 1 {| z | ≥ B y } . Assume that Assumption 1 holds. Then choosing B y := √ 4 B 4 /ϵ , it holds that

Therefore, it is without loss of generality to assume that | y | ≤ B y .

Proof. After truncating the label y , we have

Since E x ∼N d [ σ 4 ( w ∗ · x )] ≤ B 4 by assumption, we have by Markov's inequality that Pr [ σ ( w ∗ · x ) ≥ t ] ≤ B 4 /t 4 . Therefore, we can further bound E ( x ,y ) ∼D [( P t ( y ) -σ ( w ∗ · x )) 2 ] from above by

Thus, choosing t = √ 4 B 4 /ϵ we have

indicating that we can assume without loss of generality that | y | ≤ B y := √ 4 B 4 /ϵ , completing the proof of Lemma D.11.

After assuming that y is bounded by B y without loss of generality, we can then bound the 2 nd and 4 th moments of y . These bounds on the moments of the label y will be used when we implement Fact D.10 to get finer bounds compared to what we would get from a simple application of Cauchy-Schwarz. In particular, we use Fact D.10 to derive upper bounds on expectations like E ( x ,y ) ∼D [ y 2 f 2 ( x )] , where f ( x ) is a polynomial of x , as we have control on the p th moments of f ( x ) using Gaussian hypercontractivity Fact D.9.

Lemma D.12 (Moments of Labels) . If OPT ≤ 1 / 16 , then 1 / 2 ≤ E y [ y 2 ] ≤ 2 and E y [ y 4 ] ≤ 8 B 4 /ϵ .

Proof. We first bound the 2 nd moment of the label y . Note that since σ ( w ∗ · x ) is normalized such that E x ∼N d [( σ ( w ∗ · x )) 2 ] = 1 , we have

We used Young's inequality in ( i ) . Choosing a = 1 / 8 and since we assumed OPT ≤ 1 / 16 , it holds E y [ y 2 ] ≤ 9 / 16 + 9 / 8 ≤ 2 . In addition, using Cauchy-Schwarz inequality, we have

This yields the first statement of the lemma. For the remaining statement, notice that since y ≤ B y , we have E y [ y 4 ] ≤ B 2 y E y [ y 2 ] ≤ 2 B 2 y = 8 B 4 /ϵ .

We now proceed to bound the sample complexity of Algorithm 3, the argument for which relies on applying Fact D.8 to Z ( i ) = 1 n ( P ( i ) -P ) and is summarized in the following lemma.

Lemma D.13 (Sample Complexity for Estimating the Unfolded Tensor Matrix) . Let ϵ, ϵ 0 > 0 . Consider the unfolded matrix M = Mat ( l,k -l ) ( E ( x ,y ) ∼D [ y He k ( x )]) and its empirical estimate ̂ M := (1 /n ) ∑ n i =1 Mat ( l,k -l ) ( y ( i ) He k ( x ( i ) )) , where { ( x ( i ) , y ( i ) ) } n i =1 are n = Θ( e k log k ( B 4 /ϵ ) d k/ 2 /ϵ 2 0 + 1 /ϵ ) i.i.d. samples from D . Then, with probability at least 1 -exp( -d 1 / 2 ) ,

Moreover, if ̂ v ∗ is the top left-singular vector of ̂ M , then with probability at least 1 -exp( -d 1 / 2 ) ,

Proof. The proof hinges on applying Fact D.8, for which we need to bound above the parameters γ , γ ∗ , and ¯ R defined in the same fact. We do so in three separate claims, as follows.

Proof. By the definition of γ , we have:

Observe that v ⊤ M ( i ) ( M ( i ) ) ⊤ v = ∥ v ⊤ M ( i ) ∥ 2 2 = ∑ d k -l j =1 ( v ⊤ M ( i ) ) 2 j , and notice that by definition M ( i ) = Mat ( l,k -l ) ( y ( i ) He k ( x ( i ) )) where y ( i ) He k ( x ( i ) ) is a symmetric tensor, hence using Equation (26) we get

As ( x ( i ) , y ( i ) ) are i.i.d. copies of ( x , y ) , using the linearity of expectation, we have

Now given any indices j 1 , . . . , j k -l ∈ [ d ] , observe that f j 1 ,...,j k -l ( x ) := ⟨ He k ( x ) , Tensor ( v ) ⊗ e j 1 ⊗ . . . ⊗ e j k -l ⟩ is a polynomial of x of degree at most k , and note that

where the second line is by Fact C.3. Our goal is to apply Fact D.10 with A = y 2 and B = f j 1 ,...,j k -l ( x ) 2 . To this aim, we need to bound above the L p -norm of f j 1 ,...,j k -l ( x ) 2 , i.e., E x ∼N d [( f j 1 ,...,j k -l ( x ) 2 ) p ] 1 /p , which can be done using Fact D.9:

This implies that ∥ f j 1 ,...,j k -l ( x ) 2 ∥ L p ≤ 2 k p k . Thus, using Fact D.10 with A = y 2 and B = f j 1 ,...,j k -l ( x ) 2 , we get

Finally, using the bound on the moments of the labels as we proved in Lemma D.12, it holds that

Plugging the bound above back into Equation (33), we obtain:

We now proceed to bound above the second term E [ r ⊤ ( M ( i ) ) ⊤ M ( i ) r ] . Using Equation (25), similar calculations yield that

Thus, plugging in the value of B y = √ B 4 /ϵ from Lemma D.11, the variance γ can be bounded by γ ≲ d ( k -l ) / 2 e k log k/ 2 ( B 4 /ϵ ) √ n = √ d k -l e k log k ( B 4 /ϵ ) n .

Next, we bound the operator norm γ ∗ from above.

Proof. By the definition of γ ∗ ,

Decompose ˜ v into ˜ v ⊤ = [(˜ v (1) ) ⊤ , (˜ v (2) ) ⊤ ] , where ˜ v (1) ∈ R d l and ˜ v (2) ∈ R d k -l . Similarly, we can decompose ˜ r into ˜ r ⊤ = [(˜ r (1) ) ⊤ , (˜ r (2) ) ⊤ ] with the same structure. Then, ˜ v ⊤ P ( i ) ˜ r = y ( i ) (˜ v (1) ⊤ M ( i ) ˜ r (2) +˜ r (1) ⊤ M ( i ) ˜ v (2) ) . Thus, as ( x ( i ) , y ( i ) ) are i.i.d. samples, we can further bound γ 2 ∗ by

where in the last equality we used Equation (27).

Now for any ∥ u ∥ 2 = ∥ v ∥ 2 = 1 , define

where v ∈ R d l , r ∈ R d k -l , and f ( v , u ) is a polynomial of ( x 1 , . . . , x d ) of degree at most k . Note that the polynomial f ( v , u ) ( x ) satisfies f ( v , u ) ( x ) ≥ 0 and E x ∼N d [( f ( v , u ) ( x )) 2 ] = ∥ Tensor ( v ) ⊗ Tensor ( r ) ∥ 2 F = 1 . Similarly to the upper bound on γ , we apply Fact D.10 with A being y 2 and B being f ( v , u ) ( x ) 2 , which yields that for any ∥ v ∥ 2 = 1 , ∥ u ∥ 2 = 1 ,

Thus, plugging this inequality back into the upper bound on γ ∗ above, we obtain

Taking the square root on both sides completes the proof of Claim D.15.

To apply Fact D.8, it remains to bound ¯ R , which we do in the following claim.

Proof. By the definition of ℓ 2 norm, we have

Let us define

where v ∈ R d l , r ∈ R d k -l , and f i ( x ( i ) ) is a polynomial of ( x ( i ) 1 , . . . , x ( i ) d ) of degree at most k . Using the decomposition of ˜ v ⊤ = [(˜ v (1) ) ⊤ , (˜ v (2) ) ⊤ ] and ˜ r ⊤ = [(˜ r (1) ) ⊤ , (˜ r (2) ) ⊤ ] again, we get

Note that the polynomial f i ( x ( i ) ) satisfies f i ( x ( i ) ) ≥ 0 and E x ( i ) ∼N d [( f i ( x ( i ) )) 2 ] = ∥ Tensor ( v ) ⊗ Tensor ( r ) ∥ 2 F ≤ 1 . Note that E [max i ∈ [ n ] Z i ] ≤ ∑ n i =1 E [ Z i ] , thus using Fact D.10 we get:

Taking the square root on both sides completes the proof.

To apply Fact D.8, we need to choose the parameter R such that δ = Pr [max i ∈ [ n ] (1 /n ) ∥ P ( i ) -P ∥ 2 ≥ R ] is sufficiently small. Consider choosing R such that

To determine δ , recall that from Fact D.9, we have (using Markov's inequality):

Note that ∥ P ( i ) -P ∥ 2 = | y ( i ) | f i ( x ( i ) ) , hence

where in ( i ) we used a union bound and in ( ii ) we used Equation (34) with t = nR/B y . Now setting p k/ 2 = nR/ ( eB y ) , we get

In summary, applying Fact D.8 with the bound on δ and γ (Claim D.14), γ ∗ (Claim D.15), ¯ R (Claim D.16), and choosing t = d k/ 4 in Equation (32), we finally get that with probability at least 1 -exp( -log 2 (1 /ϵ )( ϵn ) 1 /k d 1 / 4 ) -d exp( -d k/ 4 ) , it holds

Therefore, choosing

we have ∥ M -̂ M ∥ 2 ≤ ϵ 0 , with probability at least 1 -exp( -d 1 / 2 ) .

To complete the proof of Lemma D.13, we apply Wedin's theorem (Fact D.6) and Claim D.7, which together imply that

We then decompose ̂ v ∗ into ̂ v ∗ = a v ∗ + b r , where r ∈ R d l such that r ⊥ v ∗ and ∥ r ∥ 2 = 1 , and a 2 + b 2 = 1 . Since b = sin( θ ( v ∗ , ̂ v ∗ )) , applying Corollary D.5 we have

This completes the proof of Lemma D.13.

After getting an approximate top-left singular vector of Mat ( l,k -l ) ( E ( x ,y ) ∼D [ y He k ( x )]) , ̂ v ∗ ∈ R d l , we show that finding the top-left singular vector of the matrix Mat (1 ,l -1) ( ̂ v ∗ ) completes the task of computing a vector u that correlates strongly with w ∗ .

Lemma D.17. Suppose that ̂ v ∗ · Vec ( w ∗⊗ l ) ≥ 1 -ϵ 1 for some ϵ 1 ∈ (0 , 1 / 16] . Then, the top-left singular vector u ∈ R of Mat (1 ,l -1) ( ̂ v ∗ ) satisfies u · w ∗ ≥ 1 -2 ϵ 1 .

Proof. Consider the SVD of Mat (1 ,l -1) ( ̂ v ∗ ) ∈ R d × d l -1 :

where u ( i ) ∈ R d , i ∈ [ d ] , and r ( i ) ∈ R d l -1 , i ∈ [ d ] , are two sets of orthonormal vectors. Note that { r (1) , . . . , r ( d ) } is a subset of { r (1) , . . . , r ( d l ) } , which is an orthonormal basis of R d l . Since ( w ∗ ) ⊤ Mat (1 ,l -1) ( ̂ v ∗ ) Vec ( w ∗⊗ l -1 ) = ̂ v ∗ · Vec ( w ∗⊗ l ) ≥ 1 -ϵ 1 , we have ρ 1 ≥ 1 -ϵ 1 , and thus

Since u ( i ) , i ∈ [ d ] and r ( i ) , i ∈ [ d l -1 ] form orthonormal bases of R d and R d l respectively, we can decompose w ∗ and Vec ( w ∗⊗ l -1 ) in these bases respectively:

Note that since ∥ w ∗ ∥ 2 = 1 and ∥ Vec ( w ∗⊗ l -1 ) ∥ 2 2 = ∥ w ∗⊗ l -1 ∥ 2 F = 1 , we have ∑ i a 2 i = 1 and ∑ i b 2 i = 1 . In addition, since ∥ ̂ v ∗ ∥ 2 = 1 , we have ∥ Mat (1 ,l -1) ( ̂ v ∗ ) ∥ 2 F = ∑ d i =1 ρ 2 i = 1 . Therefore, plugging the decomposition above back into Equation (36), we get

When ρ 1 ≥ 1 -ϵ 1 ≥ √ 2 / 2 , we have ρ 1 -√ 1 -ρ 2 1 ≥ 0 and then it holds that

We show that when 0 ≤ ϵ 1 ≤ 1 / 16 (which implies that 15 / 16 ≤ ρ 1 ≤ 1 ), it holds that g ( ρ 1 ) ≥ 1 -2 ϵ 1 . By the definition of g ( ρ 1 ) , it suffices to argue that

This follows by direct calculations as when ρ 1 ∈ [15 / 16 , 1] , 1 -2 ρ 1 +2 √ 1 -ρ 2 1 ≤ 0 . Therefore, when ϵ 1 ≤ 1 / 16 , it holds that a 1 b 1 ≥ 1 -2 ϵ 1 . Since 0 ≤ a 1 , b 1 ≤ 1 , it must be that a 1 ≥ 1 -2 ϵ 1 ; this further implies that w ∗ · u ≥ 1 -2 ϵ 1 .

D.3 Proof of Proposition D.2

Proof of Proposition D.2. Since √ OPT ≤ c k ∗ / (64 k ∗ ) ≤ c k ∗ / 64 , choosing ϵ 0 = c k ∗ / (256 k ∗ ) ≤ c k ∗ / 256 in Lemma D.13, we obtain that using n = Θ(( k ∗ ) 2 e k ∗ log k ∗ ( B 4 /ϵ ) d ⌈ k ∗ / 2 ⌉ / ( c 2 k ∗ ) + 1 /ϵ ) , it holds with probability at least 1 -exp( -d 1 / 2 ) that

Then applying Lemma D.17 with ϵ 1 ≤ 1 / (16 k ∗ ) ≤ 1 / 16 we get that the output u of Algorithm 3 satisfies u · w ∗ ≥ 1 -2 ϵ 1 ≥ 1 -1 / (8 k ∗ ) ≥ 1 -min { 1 /k ∗ , 1 / 2 } , completing the proof.

D.4 Proof of Proposition D.3

Proof of Proposition D.3. Since √ OPT ≤ c k ∗ / 64 and ϵ ≤ 1 / 64 , choosing ϵ 0 = c k ∗ ϵ/ 16 in Lemma D.13, we obtain that using n = Θ( e k ∗ log k ∗ ( B 4 /ϵ ) d ⌈ k ∗ / 2 ⌉ / ( c 2 k ∗ ϵ 2 ) + 1 /ϵ ) samples, it holds with probability at least 1 -exp( -d 1 / 2 ) that ̂ v ∗ · Vec ( w ∗⊗ l ) ≥ 1 -(2 /c k ∗ ) √ OPT+ ϵ/ 3( ≥ 15 / 16) . Then applying Lemma D.17 with ϵ 1 = (2 /c k ∗ ) √ OPT -ϵ/ 3 , we get that the output w 0 of Algorithm 3 satisfies w 0 · w ∗ ≥ 1 -2((2 /c k ∗ ) √ OPT+ ϵ/ 3) .

Finally, to show the upper bound on the L 2 2 loss of w 0 , we bring in the definition of the L 2 2 loss L σ 2 ( w 0 ) , which yields

Since ∑ k ≥ k ∗ kc 2 k ≤ C k ∗ by Assumption 1 ( iii ) , this completes the proof of Proposition D.3.

E Full Version of Section 3

After getting an initialized vector w 0 using Algorithm 1, we run Riemannian minibatch SGD Algorithm 2 on the 'truncated loss'. In the following sections, we will first present the definition of the truncated L 2 2 loss L ϕ 2 and its Riemannian gradient, then we will proceed to show that Algorithm 2 converges to a constant approximate solution in O (log(1 /ϵ )) iterations.

Algorithm 4 Riemannian GD with Warm-start

1: Input: Parameters ϵ, k ∗ , c k ∗ , B 4 > 0; T, η ; Sample access to D .

2: w 0 = Initialization [ ϵ, k ∗ , c k ∗ , B 4 , ϵ 0 = c k ∗ / (256 k ∗ )] .

3: for t = 0 , . . . , T -1 do

4: Draw n = Θ( C k ∗ de k ∗ log k ∗ +1 ( B 4 /ϵ ) / ( ϵδ )) samples from D and compute

5: w t +1 = ( w t -η g ( w t )) / ∥ w t -η g ( w t ) ∥ 2 .

E.1 Truncated Loss and the Sharpness property of the Riemannian Gradient

Instead of directly minimizing the L 2 2 loss L σ 2 , we work with the following truncated loss that drops all the terms higher than k ∗ in the polynomial expansion of σ :

Similarly, the noiseless surrogate loss is defined as

Using Fact C.1 (2) , the gradient of the truncated L 2 2 loss equals:

while for the gradient of the noiseless L 2 2 loss we have

Recall that P w ⊥ := I -ww ⊤ . Then the Riemannian gradient of the L 2 2 loss L ϕ 2 , denoted by g ( w ) is

Similarly, the Riemannian gradient of the noiseless L 2 2 loss L ∗ ϕ 2 is defined by

The following claim establishes that g ∗ ( w ) carries information about the alignment between vectors w and w ∗ .

Proof. Using the definition of g ∗ ( w ) from Equation (42), a direct calculation shows that

where ( i ) is by the definition of g ∗ ( w ) and σ ( w ∗ · x ) , ( ii ) is by Fact C.3, and ( iii ) is by Fact C.1 (1) , Equation (20), and ∥ w ∗ ∥ 2 = 1 . Let v ∈ R d be any unit vector that is orthogonal to ( w ∗ ) ⊥ w . Observe that v ⊥ w · w ∗ = v · ( w ∗ ) ⊥ w = 0 . Thus, we have

This implies that g ∗ ( w ) is parallel to ( w ∗ ) ⊥ w and thus g ∗ ( w ) = -2 k ∗ c k ∗ ( w · w ∗ ) k ∗ -1 ( w ∗ ) ⊥ w .

Let us denote the difference between the noisy and the noiseless Riemannian gradient by ξ ( w ) :

We next show that the norm of ξ ( w ) and the inner product between ξ ( w ) and w ∗ are both bounded: Lemma E.2. Let ξ ( w ) = g ( w ) -g ∗ ( w ) as defined above. Then,

Proof. Using the definition of ξ ( w ) and the definition of the 2-norm,

where the (only) inequality is by Cauchy-Schwarz, and the last equality is by Fact C.3. As a tensor A , it holds ∥ Sym( A ) ∥ F ≤ ∥ A ∥ F , we have

which then implies that 4

Following the same line of argument as above, we also get

This completes the proof of Lemma E.2.

As a direct corollary of Lemma E.2, we now show that the norm of the noisy gradient ∥ g ( x ) ∥ 2 is close to the norm of the noiseless gradient ∥ g ∗ ( w ) ∥ 2 .

We are now ready to present the main structural result of this section.

Lemma E.4 (Sharpness) . Assume OPT ≤ c/ (4 e ) 2 for some small absolute constant c < 1 . Let w ∈ R d such that ∥ w ∥ 2 = 1 and suppose that w · w ∗ ≥ 1 -1 /k ∗ . Let θ := θ ( w , w ∗ ) . If sin θ ≥ 4 e √ OPT , then

Proof. We start by noticing that by Claim E.1, the noiseless gradient satisfies the following property:

where we used that since ∥ w ∥ 2 = ∥ w ∗ ∥ 2 = 1 , we have ∥ ( w ∗ ) ⊥ w ∥ 2 = sin θ . Furthermore, applying Lemma E.2 we have the following sharpness property with respect to the L 2 2 loss:

Observe that (1 -1 /t ) t -1 ≥ 1 /e for all t ≥ 1 . Therefore, when w · w ∗ ≥ 1 -1 /k ∗ , the norm of the gradient vector g ∗ satisfies

therefore, when sin θ ≥ 4 e √ OPT and w · w ∗ ≥ 1 -1 /k ∗ , we have

Thus, as long as sin θ ≥ 4 e √ OPT , we have that g ( w ) · w ∗ ≤ -1 2 ∥ g ∗ ( w ) ∥ 2 sin θ.

E.2 Concentration of Gradients

For notational simplicity, define:

Then the empirical estimate of g ( w ) is

The following lemma provides the upper bounds on the number of samples required to approximate the Riemannian gradient g ( w ) by ̂ g ( w ) .

Lemma E.5 (Concentration of Gradients) . Let w ∗ , w ∈ S d -1 . Let ̂ g ( w ) be the empirical estimate of the Riemannian gradient. Furthermore, denote the angle between w and w ∗ by θ . Then, with probability at least 1 -δ it holds

Proof. By Chebyshev's inequality, we have

Let e j be the j th basis of R d , we have ∥ g ( w ; x , y ) -g ( w ) ∥ 2 2 = ∑ d j =1 ( g ( w ; x , y ) · e j -g ( w ) · e j ) 2 . Thus, it suffices to bound the expectation of each summand ( g ( w ; x , y ) · e j -g ( w ) · e j ) 2 , for j ∈ [ d ] . Note first that since g ( w ) = E ( x ,y ) ∼D [ g ( w ; x , y )] , we have

Denote f j ( x ) := ⟨ He k ∗ ( x ) , e ⊥ w j ⊗ w ⊗ k ∗ -1 ⟩ , which is a polynomial of x of degree k ∗ . In addition, note that E x ∼N d [ f j ( x ) 2 ] = ∥ e ⊥ w j ⊗ w ⊗ k ∗ -1 ∥ 2 F ≤ ∥ e ⊥ w j ∥ 2 2 ≤ 1 . Therefore, applying Fact D.9 and Fact D.10 with A = y 2 and B = f j ( x ) 2 , we get

Thus, it holds that E ( x ,y ) ∼D [( g ( w ; x , y ) · e j -g ( w ) · e j ) 2 ] ≲ ( k ∗ c k ∗ ) 2 e k ∗ log k ∗ ( B 4 /ϵ ) , which further implies that the expectation of ∥ g ( w ; x , y ) -g ( w ) ∥ 2 2 is bounded above by

Plugging this back into the Chebyshev's bound Equation (46), we obtain

Therefore, with probability at least 1 -δ , it holds

Now for the second statement of Lemma E.5, we similarly apply Chebyshev's inequality, which yields

Let f w ∗ ( x ) := ⟨ He k ∗ ( x ) , w ⊗ k ∗ -1 ⊗ ( w ∗ ) ⊥ w ⟩ . Note that E x ∼N d [ f w ∗ ( x ) 2 ] = ∥ w ⊗ k ∗ -1 ⊗ ( w ∗ ) ⊥ w ∥ 2 F = ∥ ( w ∗ ) ⊥ w ∥ 2 2 . Since w ∗ , w ∈ S d -1 , we have ∥ ( w ∗ ) ⊥ w ∥ 2 = sin θ . Thus, by Fact D.9,

Hence, applying Fact D.10 with A = y 2 , B = f 2 w ∗ ( x ) , σ B = sin 4 θ , and C = k ∗ , we obtain

Therefore, it holds that

which implies that with probability at least 1 -δ it holds

We proceed to the main theorem of this paper. It shows that using at most ˜ Θ( d ⌈ k/ 2 ⌉ + d/ϵ ) samples, Algorithm 2 (with initialization subroutineAlgorithm 1) generates a vector ̂ w such that L σ 2 ( ̂ w ) = O (OPT) + ϵ within O (log(1 /ϵ )) iterations.

E.3 Proof of Main Theorem

Theorem E.6. Suppose that Assumption 1 holds. Choose the batch size of Algorithm 4 to be n = Θ( C k ∗ de k ∗ log k ∗ +1 ( B 4 /ϵ ) / ( ϵδ )) , and choose the step size η = 9 / (40 ek ∗ c k ∗ ) . Then, after T = O (log( C k ∗ /ϵ )) iterations, with probability at least 1 -δ , Algorithm 4 generates a parameter w T that satisfies L σ 2 ( w T ) = O ( C k ∗ OPT)+ ϵ . The total number of samples required for Algorithm 4 is

Proof. Suppose first that OPT ≥ ( c k ∗ / (64 k ∗ )) 2 , i.e., OPT is of constant value. Then by Claim E.7 we know that for any unit vector ̂ w (e.g., ̂ w = e 1 ) it holds

Hence ̂ w is an approximate solution of the agnostic learning problem.

Now suppose OPT ≤ ( c k ∗ / (64 k ∗ )) 2 , then the assumption in Proposition D.2 is satisfied and Algorithm 3 can be applied. Consider the distance between w t and w ∗ after each update of Algorithm 4. By the non-expansive property of projection operators, we have

Let us denote the angle between w t and w ∗ by θ t . Furthermore, let us assume for now that θ t satisfies sin θ t ≥ 4 e √ OPT+ √ ϵ , hence the condition for Lemma E.4 is satisfied. Note by definition ̂ g ( w t ) ⊥ w t , hence using Lemma E.4 and Lemma E.5 we have that with probability at least 1 -δ , the inner product term in Equation (47) is bounded above by:

where C 1 is a sufficiently large absolute constant. On the other hand, the squared norm term ∥ ̂ g ( w t ) ∥ 2 2 from Equation (47) can be bounded above using Lemma E.5 and Corollary E.3:

for a sufficiently large absolute constant C 2 . Plugging Equation (48) and Equation (49) back into Equation (47), and denoting κ := max { C 1 , C 2 } k ∗ c k ∗ e k ∗ / 2 log k ∗ / 2 ( B 4 /ϵ ) , we get that with probability at least 1 -δ ,

Let us assume first that θ t ≤ θ t -1 ≤ · · · ≤ θ 0 and θ t satisfies sin θ t ≥ 4 e √ OPT + √ ϵ . We will argue that in this case θ t +1 ≤ θ t (in fact, that it contracts by a constant factor). Then, by an inductive argument, we immediately know that the assumption is valid and that θ t is a decreasing sequence (as long as sin θ t ≥ 4 e √ OPT+ ϵ ). To prove θ t +1 ≤ θ t , recall that in Claim E.1 it was shown that

Recall that w 0 is the initial parameter vector that satisfies w 0 · w ∗ ≥ 1 -1 /k ∗ . By the inductive hypothesis it holds θ t ≤ θ 0 , hence w t · w ∗ ≥ w 0 · w ∗ ≥ 1 -1 /k ∗ . Furthermore, as we have (1 -1 /t ) t -1 ≥ 1 /e for t ≥ 1 , it holds 1 ≥ ( w t · w ∗ ) k ∗ -1 ≥ 1 /e . Therefore, we further obtain

Now choosing n ≳ dκ/ (( k ∗ c k ∗ ) 2 ϵδ ) , and recalling that we have assumed sin θ t ≥ 4 e √ OPT+ √ ϵ by the induction hypothesis, we can further bound ∥ w t +1 -w ∗ ∥ 2 2 above as:

Observe that since θ t ≤ θ 0 and by assumption w 0 · w ∗ = cos θ 0 ≥ 1 -min { 1 /k ∗ , 1 / 2 } ≥ 1 / 2 , we have cos( θ t / 2) ≥ √ 3 / 2 and thus it further holds that ( √ 3 / 2)(2 sin( θ t / 2)) ≤ sin θ t ≤ 2 sin( θ t / 2) . Since ∥ w t -w ∗ ∥ 2 = 2sin( θ t / 2) follows from w t , w ∗ ∈ S d -1 , we finally obtain that, with probability at least 1 -δ ,

Choosing η = 9 / (40 ek ∗ c k ∗ ) yields (with probability at least 1 -δ ):

This shows that θ t +1 ≤ θ t , hence completing the inductive argument. Furthermore, Equation (51) implies that after at most T = O (log(1 /ϵ )) iterations it must hold that sin θ T ≤ 4 e √ OPT + √ ϵ , therefore, we can end the algorithm after at most O (log(1 /ϵ )) iterations. Though the contraction Equation (51) only holds when sin θ T ≥ 4 e √ OPT + √ ϵ , we can further show that if after some iteration t ∗ we have sin θ t ∗ ≤ 4 e √ OPT+ √ ϵ , then sin θ t ∗ +1 is still of order √ OPT+ √ ϵ . Concretely, if there exists some step t ∗ ≤ T such that sin( θ t ∗ ) ≤ 4 e √ OPT + √ ϵ , then at step t ∗ +1 it must hold (by Equation (50)):

In other words, for all steps t ∗ ≤ t ≤ T , it holds that sin θ t ≤ 30( √ OPT+ √ ϵ ) . Thus, in summary, choosing T = O (log(1 /ϵ )) , we get that with probability at least 1 -δT , sin θ T ≲ √ OPT + √ ϵ , and applying Claim E.7 we get:

where we set ϵ ′ = ϵ/C k ∗ ≤ ϵ/ ( ∑ k ≥ k ∗ kc 2 k ) , and used Assumption 1 ( iii ) that ∑ k ≥ k ∗ kc 2 k ≤ C k ∗ .

Thus, choosing δ ′ = δT , where T = O (log( C k ∗ /ϵ ′ )) , Algorithm 4 outputs a parameter w T such that with probability at least 1 -δ ′ , L σ 2 ( w T ) = O ( C k ∗ OPT) + ϵ ′ , with batch size

In summary, the total number of samples required for Algorithm 4 is

The final claim shows that if sin( θ ( w , w ∗ )) ≲ √ OPT+ √ ϵ , then L σ 2 ( w ) ≲ C k ∗ (OPT + ϵ ) .

Claim E.7. Let w ∈ S d and denote the angle between w and w ∗ by θ . If θ satisfies sin θ ≤ C ( √ OPT+ √ ϵ ) for some absolute constant C , then we have

Proof. Since w · w ∗ = cos θ ≥ 1 -sin 2 θ , according to Claim C.4 the L 2 2 loss L σ 2 ( w ) can be upper bounded by

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract summarizes the main contribution of our paper provided in Theorem 3.5, that is, we provide the first algorithm for agnostically learning SIMs under a broad class of activations. In the introduction, we summarize the motivation of our work, provide a detailed discussion of our results, and compare our work with prior literature.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our work are discussed in the introduction, and are clearly stated in the statements of the theorems.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our assumptions on the activations are summarized in Assumption 1. Complete proofs are provided in the main body and the appendix (Appendix D and Appendix E).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper is theoretical in nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is theoretical in nature, and we do not see any major or immediate implications for society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper is theoretical and does not use data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This work does not use any assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This work does not introduce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve any crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies efficient parameter estimation from generalized linear models with adversarial noise (a.k.a. agnostic learning), assuming a known link function. \nThe goal is to find a parameter vector that best explains the data. \nThe authors show that $d^{\\lceil k^*/2 \\rceil}$ samples suffice for this problem with $k^*$ being the information exponent. \nThe algorithm is given by certain projected gradient descent (w.r.t. loss truncated to degree $k^*$ Hermite polynomials) initialized with PCA for tensor unfolding. \nThe sample complexity matches a known CSQ lower bound $d^{k^*/2}$ for even $k^*$ in the realizable setting (i.e., random noise).\n\nSTRENGTHS:\nNA\n\nWEAKNESSES:\nNA\n\nQUESTIONS:\nMajor comments: \n\n1. The work [DPVLB24] is compared in B.3, but I'm not sure it's comparable. \nThat paper obtained the conjecturally optimal sample complexity $d^{\\bar{k}^*/2}$ with $\\bar{k}^*$ being the generative exponent which can be much smaller than $k^*$. \nThe way I read [DPVLB24] is that if preprocessing is allowed, $d^{\\bar{k}^*/2}$ can be achieved. \nOtherwise, we're stuck at $d^{{k}^*/2}$. \nIn the current paper, no preprocessing is used. \nCould the authors comment on the possibility of reducing the sample complexity in the agnostic setting with preprocessing?\nOr the difficulty of applying a suitable preprocessing that brings down the effective information exponent?\nNote that the optimal preprocessing in [DPVLB24] relies on the link function. \nWith additional work, they can make it work for misspecified models (a.k.a. unknown link function). \nBut I'm not sure if this is possible with adversarial noise. \n\n1. I would be cautious to claim that $d^{\\lceil k^*/2 \\rceil}$ **nearly** matches the CSQ lower bound $d^{k^* / 2}$ since there's a nontrivial gap in the exponent for odd $k^*$. \nThis is inconsistent with the standard definition of \"nearly\" in say TCS. \n\n1. Regarding the ceiling in the exponent, I suspect that this is a proof artifact since for tensor PCA, the ceiling in [RM14] can be removed with (much) more careful analysis https://arxiv.org/abs/2110.10210 . \n\nMinor comments: \n\n1. I didn't check all the cited papers (in the intro and appendix B) but would like to invite the authors to caution the following terminology discrepancy. \nIn classical stats, learning single-index models means learning the unknown link function. \nOTOH, for known link function with unknown parameter vector, the model is referred to as generalized linear models. \nThere're also works consider learning the parameter vector from GLMs with unknown link function (as a nuisance parameter). \nHowever, it seems in recent years, people (especially deep learning theorists) have been interchangeably using SIM and GLM to refer to the above problems. \nPlease make sure such discrepancies are taken into account when comparing prior works with the present result. \n\n1. Line 63-66, I'm not familiar with the line of cited work on hardness, why is $d^{O(1/\\epsilon)}$ regarded as hard?\nThis seems to me a legitimate polynomial running time. \nCould the authors say a few more words about under which computational model (low-deg, SQ, SoS, AMP, etc) the problem is hard, and hard in what sense?\n\n1. Line 72, what does $\\sim$ mean?\n\n1. Line 77, the condition $c_{k^*} = \\Omega(1)$ doesn't make sense to me. \nI'm not sure what's the asymptotic in $\\Omega(\\cdot)$. \nI suppose $\\sigma$ is a fixed function independent of $n,d$ (otherwise it again doesn't seem to make sense), so $c_{k^*}$ is either zero or strictly nonzero (note also that it can be negative). \n\n1. Line 103, could the authors also specify the dependence of running time on $1/\\epsilon$?\n\n1. Paragraph after Thm 1.2: there's a huge difference between CSQ and SQ. Are there more formal reasons (e.g., SQ lower bound) to believe that agnostic learning GLM is still hard under SQ (beyond the fact that known SQ algorithms for realizable learning fail)?\n\n1. Line 195, \"small constant $\\epsilon_0$\" is mildly misleading -- $1/2$ doesn't seem to be a small constant... \n\n1. End of line 200, there's a redundant 2 in the exponent. \n\n1. Line 223, is $l$ defined to be $\\lceil k/2 \\rceil$ or $\\lfloor k/2 \\rfloor$?\n\n1. Line 2 of Algorithm 2, the subroutine Initialization is undefined. I assume it refers to PCA for tensor unfolding. \n\n1. Line 557 is somewhat confusing, do the authors mean that **for any** positive constant $c$, partial trace cannot achieve $\\mathbf{w}^0 \\cdot \\mathbf{w}^* \\ge c$?\n\n1. Line 572, I'm not sure why $\\mathbf{w}^*$ is desired to be the unique top eigenvector. \nDo the authors mean that $\\mathbf{w}^*$ is an outlier top eigenvector (meaning there's an eigengap between top two eigenvalues)?\n\n1. Line 581-583, I think the exact information theoretic threshold is known for tensor PCA: https://arxiv.org/abs/1812.03403\n\n1. Line 590, I think [HSS15] studies SoS instead of tensor unfolding (correct me otherwise). \nFor tensor unfolding, the correct reference is https://arxiv.org/abs/2110.10210 which confirms the $d^{(k-2)/4}$ conjecture of [RM14]. \nI'm not sure of the relevance of the bound $d^{k/4}$. \n\n1. Line 591, I don't understand where the $O(d)$ bound for unfolding comes from. \nThe suboptimal analysis of [RM14] already gives $d^{1/2}$. \nAnd again, the above reference shows that $d^{1/4}$ suffices for tensor unfolding (which is sharp for unfolding). \nI'm not sure the relevance of $d^{3/4}$ here.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of _agnostic learning_ of single-index models, which consists of trying to learn a distribution $\\mathcal D$ on $\\mathbb R^d \\times \\mathbb R$ with an estimator of the form $y = \\sigma(\\langle w, x \\rangle)$. Since the problem is not necessarily well-posed, the goal is to reach a performance of at most $C \\cdot OPT + \\epsilon$, where $OPT$ is the best possible performance.\n\nThe authors present an algorithm that achieves this bound with a sample complexity of $d^{\\lceil k^\\star/2 \\rceil} + d/\\epsilon$, where $k^\\star$ is the _information exponent_ of the function $\\sigma$. This algorithm proceeds in two steps: the first obtains an informed initialization of the optimal vector $w^\\star$ through a tensor unfolding method, which is then passed through an SGD algorithm to obtain the final estimator.\n\nSTRENGTHS:\nI found the topic of the paper very interesting; I was mostly familiar with the realizable setting, and the adversarial version provides interesting insights and challenges. I particularly appreciated Appendix B, which compares in-depth the present work with previous papers on both the realizable and agnostic settings.\n\nThe results encompass a wide class of link functions, and make no assumption on the data distribution at all (apart from Gaussian marginals). The arguments to handle such a diverse class of problems are conceptually interesting, and (as far as I checked -- see `Weaknesses`) correct.\n\nWEAKNESSES:\nApart from a few qualms with the presentation (see the minor remarks), my main issue with the paper is simply its length. With the short review time and high review count, it is strictly impossible to certify that a proof which consists of 20-25 pages of dense math, with many computation-heavy steps, is correct. I did check that the proof outline and the main steps seem correct, but it is my opinion that such a paper is incompatible with the Neurips review format, and should be submitted to a journal instead. My recommendation for acceptance, based on the overall quality of the paper, should be taken with such a caveat.\n\nMinor remarks:\n- the vector $w^\\star$ is never rigorously defined; for the sake of exposition, the link between solving Problem 1.1 and obtaining good alignment with $w^\\star$ should be emphasized.\n- In Assumption 1, the wording is confusing: since $\\sigma$ is fixed, what is the $c_k^\\star = \\Omega(1)$ referring to ? My best guess is that the results hold uniformly over a class of link functions where $c_k^\\star$ is bounded from below, and $C_k^\\star$ and $B_4$ from above, but this should be made apparent if this is the case.\n- there should be a $w^t$ in the RHS of eq. (16).\n- You mention in the introduction that achieving $OPT+\\epsilon$ performance is likely to be hard, but it's difficult to parse why the proof wouldn't extend in this case: this seems hidden in the recursion of Theorem 3.5. Highlighting this barrier would be a nice addition to the proof.\n\nQUESTIONS:\n- In papers on the realizable setting (e.g. [DNGL23, DPVLB24]), the difficulty of the problem is not measured in terms of the link function $\\sigma$, but the distribution of $(x, y)$ (assuming that $\\sigma$ is ``nice''). Can such a measure be defined in your setting, instead of considering the worst-case scenario on $(x, y)$ ?\n- On a related note, in the aforementioned setting, a mismatch between $\\sigma$ and the ``true'' link function can lead to a value of $OPT$ of order $\\Omega(1)$, but some algorithms manage to achieve almost perfect alignment with the vector $w^\\star$. In your case, a large value of $OPT$ yields vacuous bounds on both the performance of your algorithm and its alignment with $w^\\star$; is there an explanation of this phenomenon ?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a polynomial-time algorithm reaching the optimal CSQ sample complexity (upto sub-leading factors) for a general class of single-index models under the setup of adversarial noise. Similar to existing works, the algorithm utilizes the empirical estimate of the k_th Hermite tensor of the target to obtain an initialization with a non-trivial overlap (weak-recovery) by estimating the top singular vector of an unfolded tensor. Subsequently, the algorithm utilizes mini-batch SGD on the sphere starting from non-trivial overlap to reach the optimal error upto a constant factor.\n\nSTRENGTHS:\nThe paper makes novel technical contributions towards understanding the sample-complexity of learning under the adversarial noise. Understanding the differences between the adversarial and realizable/random noise settings is crucial towards justifying applicability of CSQ and SQ lower-bounds in typical maching learning setups. The paper adequetly describes the technical challenges in matrix-concentration under adversarial noise as well as why the partial trace algorithm in Damian et al. (2024) cannot be directly applied under the adversarial noise setting (Appendix B.3).\n\nWEAKNESSES:\n* The central ideas of obtaining weak-recovery by estimating the Hermite tensor and subsequently running minibatch SGD is largely similar to existing works (Biroli et al. (2020), Damian et al. (2023), Damian et al. (2024), Chen and Meka (2020)). Therefore, the major contribution of the work are adapting the ideas and analysis in these papers to adversarial noise, which has some technical but limited conceptual novelty. Even under adversarial noise, prior works have establishing similar guarantees for smaller classes of link functions. It is unclear whether the setup of  adversarial noise is relevant for gradient based training in machine learning models. Discussion of the motivation behind studying the adversarial noise  and the novel conceptual contributions/implications of the work will improve the paper.\n\n* In the realizable and random-noise case, recent works Dandi et al. (2024) and Damian et al. (2024) show the learnability of multi/single index models under reduced SQ complexity through reuse of data (implicitly transforming the labels) and explicit transformation of labels respectively. In light of these works, it has become apparent to the community that the SQ class is more suitable towards describing the limitations of gradient-based learning rather CSQ. Therefore, it is important for the paper to justify the choice of the CSQ class and discuss the effect of data reuse, label transformation. The paper presently only discusses the limitation of some SQ algorithms such as the one in Damian et al. (2024) under adversarial noise. This alone doesn’t justify the relevance of CSQ lower-bounds under the possibility of label transformation in the adversarial noise case.\n\nMissing references of closely related works:\n- Biroli, Giulio, Chiara Cammarota, and Federico Ricci-Tersenghi. \"How to iron out rough landscapes and get optimal performances: averaged gradient descent and its application to tensor PCA.\" Journal of Physics A: Mathematical and Theoretical 53.17 (2020): 174003.\n- Dandi, Y., Krzakala, F., Loureiro, B., Pesce, L., & Stephan, L. (2023). How two-layer neural networks learn, one (giant) step at a time. arXiv preprint arXiv:2305.18270.\n- Dandi, Y., Troiani, E., Arnaboldi, L., Pesce, L., Zdeborova, L., & Krzakala, F. The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents. In Forty-first International Conference on Machine Learning.\n\nQUESTIONS:\n* Is the proposed algorithm expected to reach the SQ lower-bounds upon transformation of the labels?\n\n* Intuitively, why does unfolding fare better than the partial trace estimator under adversarial noise? \n\n* Can the memory and runtime requirements for Algorithm 1 be optimized simlar to the partial trace algorithm (see Remark 4.2 in Damian et al. 2024)?\n\n* The discussion in lines 158-161 regarding the Gaussianity of the noise term in prior works isn’t accurate since Damian et al. 2024 also consider arbitrary labels leading to general noise.\n\n* Why using the tensor PCA for a star while we know there are more efficient starting point for such problems, e.g. (https://arxiv.org/abs/1708.05932, arXiv:1811.04420,https://arxiv.org/abs/2012.04524) ? \n\n*  Correction: The references list \"G. B. Arous\" instead of \"G. Ben Arous\" (Ben Arous is the surname, not a second name). This should be corrected to accurately reflect the author's name.\n\nLIMITATIONS:\nThe work precisely describes the theoretical assumptions. One major limitation of the work is the absence of a discussion of label transformations/batch reuse (see weaknesses above). The work is primarily of a theoretical nature and has no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}