Title: On the Complexity of Learning Sparse Functions with Statistical and Gradient Queries
Venue: NeurIPS 2024
Accepted: yes
Keywords: Statistical Query Complexity

Differtiable Learning

Sparse Functions

Leap Exponent
Abstract: The goal of this paper is to investigate the complexity of gradient algorithms when learning sparse functions (juntas). We introduce a type of Statistical Queries ($\mathsf{SQ}$), which we call Differentiable Learning Queries ($\mathsf{DLQ}$), to model gradient queries on a specified loss with respect to an arbitrary model. We provide a tight characterization of the query complexity of $\mathsf{DLQ}$ for learning the support of a sparse function over generic product distributions. This complexity crucially depends on the loss function. For the squared loss, $\mathsf{DLQ}$ matches the complexity of Correlation Statistical Queries $(\mathsf{CSQ})$—potentially much worse than $\mathsf{SQ}$. But for other simple loss functions, including the $\ell_1$ loss, $\mathsf{DLQ}$ always achieves the same complexity as $\mathsf{SQ}$. We also provide evidence that $\mathsf{DLQ}$ can indeed capture learning with (stochastic) gradient descent by showing it correctly describes the complexity of learning with a two-layer neural network in the mean field regime and linear scaling.

On the Complexity of Learning Sparse Functions with Statistical and Gradient Queries

Abstract

The goal of this paper is to investigate the complexity of gradient algorithms when learning sparse functions (juntas). We introduce a type of Statistical Queries ( SQ ), which we call Differentiable Learning Queries ( DLQ ), to model gradient queries on a specified loss with respect to an arbitrary model. We provide a tight characterization of the query complexity of DLQ for learning the support of a sparse function over generic product distributions. This complexity crucially depends on the loss function. For the squared loss, DLQ matches the complexity of Correlation Statistical Queries ( CSQ ) -potentially much worse than SQ . But for other simple loss functions, including the ℓ 1 loss, DLQ always achieves the same complexity as SQ . We also provide evidence that DLQ can indeed capture learning with (stochastic) gradient descent by showing it correctly describes the complexity of learning with a two-layer neural network in the mean field regime and linear scaling.

1 Introduction

In recent years, major efforts have been devoted to understanding which distributions can be learned efficiently using gradient-type algorithms on generic models [Abbe and Sandon, 2020, Allen-Zhu and Li, 2020, Malach et al., 2021, Damian et al., 2022, Abbe et al., 2021b,a, 2022, 2023, Bietti et al., 2023, Glasgow, 2023, Dandi et al., 2023, 2024, Edelman et al., 2024, Kou et al., 2024]. In this paper, we focus on learning sparse functions (i.e. 'juntas' [Blum and Langley, 1997]), that is functions that depend only on a small number P out of a much larger set d ≫ P of input coordinates. The challenge in this setting is to identify the few relevant coordinates. For some sparse functions, such as noisy parities, learning is believed to require O ( d P ) runtime [Kearns, 1998], while others, such as linear functions, are easy to learn in ˜ O ( d ) time. Which functions are easy to learn and which are hard? What is the complexity of learning a specific sparse function? Recent works [Abbe et al., 2022, 2023] unveiled a rich 'leap' hierarchical structure and saddle-to-saddle dynamics that drives learning in this setting. The goal of the present paper is to provide a general characterization for the complexity of learning sparse functions that go beyond (i) hypercube data and Fourier analysis, and (ii) CSQ (see below) and focusing only on the squared loss.

The notion of complexity we consider is the Statistical Query ( SQ ) complexity, which studies learning by measuring expectations up to some worst-case tolerance (see [Kearns, 1998, Bshouty and Feldman, 2002, Reyzin, 2020] and Section 3). Although based on worst-case error (or almost equivalently, additive independent noise) rather than the sampling error encountered in practice, statistical query complexity has been proven to be a useful guideline for studying the complexity of learning. In

particular, gradient computations are a special case of statistical queries. In specific cases which include binary functions or gradients on the squared or cross-entropy loss, gradient queries are equivalent 1 to the restricted class of Correlation Statistical Queries ( CSQ ) which are strictly less powerful than general statistical queries. Lower bounds on the CSQ complexity have thus been seen as corresponding to the complexity of gradient-based learning 2 in these restricted cases. Part of the motivation for this paper is to emphasize that this relationship is limited to very specific loss functions and does not hold more generally. In order to study the complexity of gradient algorithms for general output and loss functions, we introduce a type of statistical query which we call Differentiable Learning Query ( DLQ ). These queries are defined with respect to a specific loss ℓ -denoted by DLQ ℓ -and are given by gradients on a loss ℓ with respect to an arbitrary model. Specifically, DLQ ℓ algorithms correspond to SQ algorithms with queries of the type

Depending on the loss ℓ and target distributions, learning with DLQ ℓ can be less, equal, or more powerful than CSQ .

For inputs on the hypercube x ∼ Unif( {± 1 } d ) and CSQ , Abbe et al. [2022, 2023] showed that the complexity of learning sparse functions is sharply captured by a leap exponent defined in terms of the non-zero Fourier coefficients of the sparse function. Informally, it states that Θ( d k ∗ ) queries are necessary and sufficient to learn with CSQ , where k ∗ is the minimum number of coordinates one need to add at once to 'leap' between non-zero Fourier coefficients. E.g., consider the following two sparse functions:

Both functions depend on an (unknown) subset of P = 4 coordinates { s ∗ (1) , s ∗ (2) , s ∗ (3) , s ∗ (4) } . For y 1 , the monomials are ordered such that we add only one new coordinate to the support at a time: the leap exponent is 1 and Θ( d ) queries are sufficient to learn y 1 . For y 2 , we need to add three new coordinates at once: the leap exponent is 3 and Θ( d 3 ) queries are required to learn y 2 .

We extend and generalize this charactarization in several significant ways:

(a) We go beyond binary input space and allow for arbitrary measurable space with product distribution. In particular, the leap complexity arises from the structure of the permutation group rather than a specific functional basis.

(b) We go beyond CSQ and squared loss, and tightly characterize the query complexity for learning sparse functions with SQ and DLQ ℓ algorithms with any loss ℓ . This complexity is in terms of a leap exponent defined analogously as above, but now over a set system C A ⊆ 2 [ P ] of 'detectable' subsets S ⊆ [ P ] , where 'detectability' depends on the type of queries A ∈ { SQ , CSQ , DLQ ℓ } . Learning with different loss functions are not necessarily comparable to CSQ or SQ and depend on the sparse function. However, we show that some losses, such as ℓ 1 or exponential loss, are 'generic', i.e. C DLQ ℓ = C SQ for any sparse function, and always match the SQ -leap exponent. This shows that differentiable learning with these losses is as powerful as learning with SQ for juntas.

(c) Finally, we introduce a cover exponent -defined on the same system of detectable sets C A -which captures learning with A ∈ { SQ , CSQ , DLQ ℓ } when the queries are chosen non-adaptively , i.e., without adapting to the responses of previous queries. This can be roughly thought of as learning with a single gradient step, versus many consecutive steps in the adaptive case. The contrast between the two helps understand to what extent learning relies on adaptivity and can help frame other related results-see discussion in Section 8.

We summarize the complexity of learning a sparse function with SQ , CSQ , DLQ ℓ and adaptive/nonadaptive queries in Table 1. To see some of these notions play out, consider again the two examples in Eq. (2):

For y 1 , DLQ ℓ will require Θ( d ) adaptive or Θ( d 4 ) non-adaptive queries to learn with squared lossequivalent to CSQ , while Θ( d ) adaptive/non-adaptive queries suffice with ℓ 1 -loss-equivalent to SQ . For y 2 , DLQ ℓ will only require Θ( d ) queries, either adaptive or non-adaptive, with ℓ 1 -loss, compared to Θ( d 3 ) queries with the squared loss. Proposition 6.2.(c) provides an example where Leap ( C SQ ) < Leap ( C DLQ ℓ ) < Leap ( C CSQ ) .

DLQ ℓ algorithms are quite different than (stochastic) gradient descent algorithms. On one hand, DLQ ℓ allows evaluating arbitrary gradients rather than following a trajectory, and is thus potentially more powerful than (S)GD on generic models 3 . On the other hand, the lower bound on DLQ ℓ algorithms in Table 1 is against worst-case noise-much more pessimistic than sampling noise encountered in SGD or GD-on-the-training-set [Abbe et al., 2021b]. Nevertheless, DLQ ℓ does provide useful guidance and we expect that it does capture important aspects of the computational complexity of GD in a number of settings. To demonstrate this, Section 7 considers learning sparse functions on the hypercube using online SGD on a two-layer neural network. We show that in this setting, the DLQ ℓ -leap exponent correctly captures learnability in the mean-field regime and linear scaling. Namely, for DLQ ℓ -leap 1 functions, SGD on loss ℓ learns in O ( d ) -steps, while for leap greater than 1 , the dynamics remains stuck in a suboptimal saddle subspace and require ω ( d ) -steps to escape.

2 Setting

Junta Recovery Problem. A sparse recovery (junta learning) problem with P relevant coordinates is defined in terms of a coordinate input space X , an output space Y 4 , a marginal distribution µ x over X and a link distribution µ y | z over Y given elements of X P . This specifies a joint distribution µ y, z over a measurable space Y ×X P , where the marginal distribution z ∼ µ P x is the product distribution, and y | z ∼ µ y | z . We further denote µ y to be the marginal distribution over Y . We further denote the junta problem by a tuple µ := ( µ x , µ y | z ) , where the support size P and the spaces X and Y are implicit. Consider some examples,

For d ≥ P , we define the junta recovery problem as learning the family of distributions

where P ([ d ] , P ) is the set of non-repeating sequences from [ d ] of length P , and the distribution D d µ,s is a distribution over Y × X d such that

In words, H d µ is the set of distributions on ( y, x 1 , . . . , x d ) where x follows the product distribution µ d x and the output y only depends on an (unknown) sequence of P coordinates. For ease of notation, we will denote D d s = D d µ,s when the junta problem µ = ( µ x , µ y | z ) is clear from context.

Success as Support Recovery. For any sequence s ∈ P ([ d ] , P ) , we additionally denote the associated unordered set of its elements by S := { s ( i ) : i ∈ [ P ] } . We consider learning as 'succeeding' if it outputs the correct index set. That is, we say a junta learning algorithm 'succeeds' in learning H d µ if for every s ∗ ∈ P ([ d ] , P ) , it outputs ˆ S ⊆ [ d ] such that ˆ S = S ∗ = { s ∗ ( i ) : i ∈ [ P ] } .

For our purpose, just the recovery of relevant coordinates is the objective as we only care about the complexity as a scaling of d . Once we recover the relevant coordinates S ∗ , learning the ordered sequence s ∗ corresponds to a problem of size P , and thus its complexity is independent of d . Note that the precise ordering s ∗ may not be identifiable if for some other s ′ ∗ with S ∗ = S ′ ∗ we have D d s ∗ = D d s ′ ∗ ; this is possible when the measure µ y | z has symmetries with respect to coordinates of z . We further emphasize that even the support S ∗ may not be identifiable, when y is independent of some coordinates of z or when only using a restricted query access to the distribution-we will return to this issue in Section 5.

Does the learner know the link µ y | z ? We always assume that the learner knows µ x . In our formulation, we also assume that the learner additionally knows the link µ y | z , and the only unknown is s ∗ . Indeed, our lower bounds hold even for this easier setting. But under mild assumptions, our upper bounds can be obtained with an algorithm that does not require knowing µ y | z (the complexity still depends on the problem µ )-see discussion in Section 5.

Well Behaved Distributions. For our lower bounds, we will consider junta problems µ = ( µ x , µ y | z ) , which are 'well-behaved' in a way that is standard in the hypothesis testing literature. Let µ y, z be the induced joint distribution on ( y, z ) where z ∼ µ P x and y | z ∼ µ y | z . For any subset U ⊆ [ P ] , let µ y, z ,U be the marginal distribution of ( y, ( z i ) i ∈ U ) and

meaning that ( y, z 1 , . . . , z P ) ∼ µ 0 y, z ,U has ( y, ( z i ) i ∈ U ) ∼ µ y, z ,U and ( z i ) i ∈ [ P ] \ U i.i.d. ∼ µ x independently of ( y, ( z i ) i ∈ U ) (we replace z i for i / ∈ U with independent draws from the marginal). The marginal distribution of z under µ 0 y, z ,U is still µ P x .

Assumption 2.1. For any U ⊆ [ P ] , we have µ y, z ≪ µ 0 y, z ,U , and the Radon-Nikodym derivative d µ y, z / d µ 0 y, z ,U is square integrable w.r.t. µ 0 y, z ,U , i.e.

This is a standard and implicit assumption in the hypothesis testing literature whenever a corresponding null distribution is considered (here all µ 0 y, z ,U , i.e., with label y depending only on a strict subset U ⊊ [ P ] ); more specifically for statistical query lower bounds [Feldman et al., 2017, Damian et al., 2024b], low-degree likelihood ratio [Hopkins, 2018, Kunisky et al., 2019], or contiguity lower bounds [Perry et al., 2018]. It always holds when X is finitely supported. We further comment on the necessity of this assumption in Appendix A.1.

3 Statistical and Differentiable Learning Queries

We will consider three classes of learning algorithms, all based on the statistical query paradigm, but differing in the type of queries allowed, as captured by a set Q of allowed queries.

For a number of queries q , tolerance τ > 0 and a set Q ⊆ R Y×X d of measurable functions Y × X d → R , a Q -restricted statistical query algorithm A ∈ Q -SQ ( q, τ ) for junta learning takes an input distribution D over Y × X d and operates in q rounds where at each round t ∈ { 1 , . . . , q } , it issues a query ϕ t ∈ Q , and receives a response v t such that

where D 0 = D y ⊗D x is the associated decoupled 'null' distribution where x and y are independent, but follow their marginals 5 . The query ϕ t can depend on the past responses v 1 , . . . , v t -1 . After

issuing q queries, the learner A outputs ˆ S ⊆ [ d ] . We say that A succeeds in learning H d µ if for any D s ∗ ∈ H d µ and any responses v t satisfying (8) for D = D s ∗ , A outputs ˆ S = S ∗ .

Above we allow the queries to be chosen adaptively, i.e., depending on past responses. We also consider Q -restricted non-adaptive statistical query algorithms , which we denote by Q -naSQ ( q, τ ) , where the query functions { ϕ t } t ∈ [ q ] are fixed in advance and do not depend on the past responses. I.e. a non-adaptive algorithm is specified by a list of queries and a mapping from the responses to an output ˆ S ⊆ [ d ] .

Statistical Queries (SQ): In regular, unrestricted Statistical Query learning, the allowed query set, denoted by Q SQ , is the set of all measurable functions. With slight overloading of notation, we refer to the class of these algorithms simply as SQ ( q, τ ) and naSQ ( q, τ ) .

Correlation Statistical Queries (CSQ): It is a special subclass of statistical queries, which require Y ⊆ R (usually we allow the input and output spaces to be abstract), and are restricted to:

We denote the class of adaptive and non-adaptive algorithms making such queries as CSQ ( q, τ ) and naCSQ ( q, τ ) respectively.

Differentiable Learning Queries (DLQ) with Loss ℓ : Let F ⊆ V , which is an open subset of some normed vector space V , be the output space of our models, e.g. usually V = F = R for models with a single output unit, but we may have V = F = R r with multiple output units. We consider a loss function ℓ : F × Y → R that is locally Lipschitz continuous in its first argument for every y ∈ Y . The loss is additionally equipped with a derivative operator ∇ ℓ : F × Y → V as a part of the loss definition such that for any u ∈ F and y ∈ Y , we have ∇ ℓ ( u , y ) ∈ ∂ 1 ℓ ( u , y ) , the set of generalized Clarke subderivatives of ℓ ( · , y ) at u ∈ F . This is a standard generalization of derivatives to non-differentiable and non-convex losses; in particular, note that ( i ) for differentiable losses, ∂ 1 ℓ ( u , y ) is a singleton with the true gradient of ℓ ( · , y ) at u ∈ F , and ( ii ) for convex losses (in the first argument), ∂ 1 ℓ ( u , y ) is the set of subderivatives of ℓ ( · , y ) at u ∈ F . Finally, let M := { f : X d × R →F | f ( x , ω ) is differentiable at ω = 0 for all x ∈ X d } be the set of allowed models 6 . Then the allowed differentiable learning query set is:

That is, at each round the algorithm chooses a parametric model f ( x , ω ) , parameterized by a single scalar ω , and the query corresponds to the derivative (with respect to the single parameter ω ) of the loss applied to the model, at ω = 0 . This captures the first gradient calculation for a single-parameter model initialized at zero. But the derivative at any other point can also be obtained by querying at a shifted model f ( ν ) ( x , ω ) = f ( x , ν + ω ) , and the gradient with respect to a model f ( x , w ) with r parameters w ∈ R r can be obtained by issuing r queries, one for each coordinate, of the form f ( w ,i ) ( x , ω ) = f ( x , w + ωe i ) , where e i is the standard basis vector. Queries of the form Q DLQ ℓ can thus be used to implement gradient calculations for any differentiable model, noting that the number of queries q is the number of gradient calculations times the number of parameters . Finally, observe that, for differentiable losses, the queries of the form (1) are equivalent to the form mentioned in (10) due to the chain rule.

We denote the class of adaptive and non-adaptive algorithms making such queries as DLQ ℓ ( q, τ ):= Q DLQ ℓ -SQ ( q, τ ) and naDLQ ℓ ( q, τ ):= Q DLQ ℓ -naSQ ( q, τ ) .

Remark 3.1. More common in the SQ literature is to restrict the query in L ∞ (or equivalently, require precision relative to L ∞ ). Precision relative to L 2 ( D 0 ) is more similar to VSTAT [Feldman et al., 2017], and is more powerful than relative to L ∞ , and our lower bounds hold against this stronger notion. In our algorithms and upper bounds we only need this additional power when

y ↦→∇ ℓ ( u , y ) is unbounded. If we further assume that these functions are bounded, e.g., the labels y are bounded and ∇ ℓ continuous, our queries have bounded L ∞ and thus operate in the more familiar SQ setting.

4 Leap and Cover Complexities

Definition 1. We define the leap and cover complexities for any system of subsets C ⊆ 2 [ P ] .

(i) For any system of subsets C ⊆ 2 [ P ] , its leap complexity is defined as

(ii) We define the cover complexity of C to be

̸

Remark 4.1. We always have Leap ( C ) ≤ Cover ( C ) . Both Leap and Cover complexities are closed under taking the union of subsets in C . Also, when supp( C ) = ∪ U ∈C U = [ P ] , then we use the convention Leap ( C ) = Cover ( C ) = ∞ . See a discussion about this convention in Appendix A.3 and in particular, the definition of the relative leap and cover complexities in Definition 3.

Here C is the system of subsets which are 'detectable', and will depend on the the query access model. Intuitively, the leap and cover complexities of C capture the exponent of d in the query complexity when recovering the support of an unknown s ∗ ∈ P ([ d ] , P ) , for adaptive and non-adaptive algorithms respectively. To discover the relevant coordinates of s ∗ , that correspond to U ∈ C , one needs to enumerate over Θ( d | U | ) . Hence, a non-adaptive algorithm, which fixes the queries in advance, requires Θ( d k i ) queries to discover i th relevant coordinate i.e. s ∗ ( i ) , where k i = min i ∈ U ∈C | U | . Therefore, non-adaptive algorithms need a total number of queries that scales as Θ( d Cover ( C ) ) to learn supp( s ∗ ) . On the other hand, adapting queries using previous answers can greatly reduce this complexity as seen in the example in (2) in Section 1. This is captured by the leap complexity, which measures the maximum number of coordinates we need to discover at once. Finally, the set system of detectable subsets will depend on the type of allowed queries.

Definition 2 (Detectable Subsets) . Let µ be a junta problem. Denote

the set of zero-mean functions. For a set of test function Ψ ⊆ L 2 ( µ y ) we say that U ⊆ [ P ] is Ψ -detectable iff

̸

We denote C Ψ ( µ ) the set of Ψ -detectable sets, i.e. those sets satisfying (13) .

The set of relevant test functions depend on the query types allowed and we define:

While the queries of the form (13), where Ψ A for A ∈ { SQ , CSQ , DLQ ℓ } is given by (14), are less general than ϕ ( y, x ) with ϕ ∈ Q A , they can be implemented by the corresponding query types Q A and are sufficient for deciding between ' S ⊆ [ d ] maps to the corresponding U ⊆ [ P ] ' or ' S ̸⊆ S ∗ '. The sets Ψ SQ , Ψ CSQ , Ψ DLQ ℓ from (14) used for detectibility arise naturally in the proof of the lower bounds of Theorem 5.1.

To ease notation, for query type A , we use the shorthand C A := C Ψ A , and

We refer to these as the A -leap exponent and A -cover exponent of the problem µ .

5 Main Result: Characterizing the Complexity of Learning Juntas

Theorem 5.1. For any junta problem µ and any loss ℓ , there exists C > c > 0 (that depend on P , µ and the loss, but not on d ), such that for query types A ∈ { SQ , CSQ , DLQ ℓ } with corresponding test function sets Ψ A as defined in (14) :

Adaptive. Let k ∗ = Leap A ( µ ) . There exists an algorithm A ∈ A ( q, τ ) that succeeds in learning H d µ with τ = c and q = Cd k ∗ . And if µ satisfies Assumption 2.1, then for any ( q, τ ) such that q/τ 2 ≤ cd k ∗ , no algorithm A ∈ A ( q, τ ) succeeds at learning H d µ .

Non-adaptive. Let k ∗ = Cover A ( µ ) . There exists an algorithm A ∈ naA ( q, τ ) that succeeds in learning H d µ with τ = c and q = Cd k ∗ . And if µ satisfies Assumption 2.1, then for any ( q, τ ) such that q/τ 2 ≤ cd k ∗ , no algorithm A ∈ naA ( q, τ ) succeeds at learning H d µ .

Remark 5.2. In the positive results in Theorem 5.1, we used all the allowed complexity to have many ( q = Θ( d k ∗ ) ) queries, and kept the tolerance constant. More generally, it is possibly to trade off between the number of queries q and tolerance τ , at a cost of a log-factor: For k ∗ = Leap A ( µ ) and k ∗ = Cover A ( µ ) , respectively, there exists algorithms A ∈ A ( q, τ ) and A ∈ naA ( q, τ ) that learn H d µ for any q ≥ C log( d ) and τ ≤ c with q/τ 2 ≥ Cd k ∗ log( d ) .

The proof of this theorem is deferred to Appendix B. Theorem 5.1 shows that the leap and cover complexities sharply capture the scaling in d of statistical query algorithms when learning H d µ .

Remark 5.3. The above upper bound uses that µ and therefore the T, T i in Definition 2 are known. In the case when µ is unknown, one can follow a similar strategy as in Damian et al. [2024b] and randomize the transformations T and T i over a sufficiently large (but finite independent of d ) linear combination of functions in Ψ SQ and L 2 0 ( µ x ) . Under some regularity assumption on µ and ℓ , one can show by anti-concentration that with constant probability, the expectation in condition (13) is bounded away from 0 by a constant independent of the dimension.

6 Relationship Between SQ , CSQ and DLQ ℓ

Obviously, CSQ ⊆ SQ , and indeed we see that C CSQ ⊆ C SQ in Definition 2 because of which Leap CSQ ≥ Leap SQ and Cover CSQ ≥ Cover SQ . For binary Y , these query models collapse, but otherwise there can be an arbitrary gap.

Proposition 6.1 ( SQ versus CSQ ) . For any µ , let C SQ := C SQ ( µ ) , and C CSQ := C CSQ ( µ ) . If |Y| = 2 (binary output), then we always have C CSQ = C SQ and Leap SQ = Leap CSQ and Cover SQ = Cover CSQ . On the other hand if |Y| > 2 , the SQ -exponents can be much smaller than the CSQ -exponents: e.g., there exist a setting with Leap SQ = Cover SQ = 1 and Leap CSQ = Cover CSQ = P .

Similarly, DLQ ℓ ⊆ SQ by definition, and thus, Leap DLQ ℓ ≥ Leap SQ and Cover DLQ ℓ ≥ Cover SQ .

Proposition 6.2 ( DLQ ℓ versus SQ and CSQ ) . Consider any µ = ( µ y , µ y | z ) .

(a) Let Y , F ⊆ R . For the squared loss ℓ : ( u, y ) ↦→ ( u -y ) 2 , we always have C DLQ ℓ = C CSQ , and thus, Leap CSQ = Leap DLQ ℓ and Cover CSQ = Cover DLQ ℓ .

(b) A sufficient condition for C SQ = C DLQ ℓ is to have span(Ψ DLQ ℓ ) dense in L 2 0 ( µ y ) . Conversely, if there exists nonzero T ∈ L 2 0 ( µ y ) bounded with T ( y ) orthogonal in L 2 ( µ y ) to any functions in Ψ DLQ ℓ , then there exists a problem µ such that Cover DLQ ℓ = Leap DLQ ℓ > Cover SQ = Leap SQ .

(c) There exists a loss ℓ and a junta problem µ such that Leap SQ ( µ ) < Leap DLQ ℓ ( µ ) < Leap CSQ ( µ ) . Similarly, we can have Leap DLQ ℓ > Leap CSQ .

The condition in Proposition 6.2.(b) can be seen as a universal approximation property of neural networks with activation y ↦→∇ ℓ ( u , y ) [Cybenko, 1989, Hornik, 1991, Sonoda and Murata, 2017]. The next lemma gives a few examples of losses with DLQ ℓ = SQ . For concreteness, we consider F = R and Y ⊆ R .

Theorem 6.3. For ℓ ∈ { ℓ 1 : ( u, y ) ↦→| u -y | , ℓ hinge : ( u, y ) ↦→ max(1 -uy, 0) } , then C DLQ ℓ ( µ ) = C SQ ( µ ) for all 7 problems µ . If we further assume Y ⊆ [ -M,M ] for M ≥ 0 , then ℓ ( u, y ) = e -uy (exponential loss) has C DLQ ℓ ( µ ) = C SQ ( µ ) for all problems µ .

The cases of ℓ 1 and Hinge loss follow directly from universal approximation of neural networks with linear threshold activation. The proofs of the above propositions and lemma can be found in Appendix C. Propositions 6.1 and 6.2 combined with Theorems 6.3 and 5.1 directly imply a number of separation results between adaptive and non-adaptive algorithms and between different loss functions. See examples (2) in the introduction, and further examples in Appendix C.

7 Gradient Descent on Neural Networks

The goal of this section is to connect the complexity of DLQ to gradient descent on standard neural networks. We focus on the simple case of x ∼ Unif( { +1 , -1 } d ) uniformly distributed on the hypercube, and Y ⊆ R and F = R . In this setting, condition (13) in Definition 2 simplifies to: there exists T ∈ Ψ A such that E µ y, z [ T ( y ) ∏ i ∈ U z i ] = E µ y, z [ T ( y ) χ U ( z )] = 0 , where χ U ( z ) := ∏ i ∈ U z i denote the standard Fourier-Walsh basis. In particular, the set C CSQ contains exactly all non-zero Fourier coefficients of h ∗ ( z ) := E [ y | z ] , and we recover the leap exponent of Abbe et al. [2023] as discussed in the introduction.

̸

We train a standard two layer neural network (see (NN1) in Appendix D) of width M , using online SGDwith a loss function ℓ : R ×Y → R ≥ 0 , i.e. SGDon E ( x ,y ) ∼D d s ∗ [ ℓ ( f NN ( x ; Θ ) , y )] for D d s ∗ ∈ H d µ . More specifically, we train the parameters Θ = { θ j : j ∈ [ M ] } using batch-SGD with loss ℓ and batch size b from initialization ( θ j ) j ∈ [ M ] i.i.d. ∼ ρ 0 specified in (69). At each step, given samples ( { ( x ki , y ki ) : i ∈ [ b ] } ) k ≥ 0 , the weights are updated using

where η is the step-size and we allow for a ℓ 2 regularization with parameter λ ∈ R + . Recall that ℓ ′ ( u, y ) ∈ ∂ 1 ℓ ( u, y ) is the defined derivative of ℓ ( · , y ) at u ∈ R . We define the test error

and further introduce the excess test error R ( f ) = R ( f ) -inf ¯ f : {± 1 } d → R R ( ¯ f ) .

Dimension-free dynamics. In the junta learning setting, when y only depends on P ≪ d coordinates, following Abbe et al. [2022, Secion 3], the SGD dynamics ( ℓ -bSGD) concentrates on an effective dimension-free (DF) dynamics as M,d →∞ and η → 0 . This equivalence holds under a certain assumption on the loss function, and other assumptions on the initialization and activation that are similar to the setup of Abbe et al. [2022] (see Appendix D for details, especially DF-PDE).

Dimension-free dynamics' alignment with the support. In the above limiting regime, Leap DLQ ℓ crisply characterizes (DF-PDE) dynamics' alignment with the support.

Theorem 7.1 (Informal version of Theorem D.6) . If Leap DLQ ℓ = 1 then for some time t , the output of the model at time t of (DF-PDE) dynamics depends on all coordinates z i . On the other hand, if Leap DLQ ℓ > 1 , then there exists a coordinate i such that for any time t ≥ 0 , the output of the model at time t of (DF-PDE) dynamics does no depend on z i .

This establishes that the ability of (DF-PDE) dynamics (comparable to ( ℓ -bSGD) in the linear scaling) to learn all relevant coordinates depends on Leap DLQ ℓ ( µ ) = 1 . That is if Leap DLQ ℓ > 1 , then (DF-PDE) remains stuck in a suboptimal saddle subspace. On the other hand, if Leap DLQ ℓ = 1 , then (DF-PDE) dynamics escapes this subspace and the weights align with the entire support.

Learning of Leap DLQ ℓ = 1 with finite SGD. Showing directly that (DF-PDE) dynamics indeed reach a near global minimizers of the test error remains challenging. Alternatively, we show that a specific layer-wise training dynamics similar to Abbe et al. [2022] achieves a vanishing excess error for Leap DLQ ℓ ( µ ) = 1 settings in the linear scaling of samples.

Roughly speaking, we train the first layer for k 1 = P steps and then the second layer weights for k 2 = O d (1) steps using batch-SGD with batch size b = O d ( d ) , both for a loss ℓ . We consider a polynomial activation σ ( x ) = (1 + x ) L of degree L ≥ 2 8 P . The most notable difference from Abbe et al. [2022] is that we further slightly perturb step-sizes for each coordinate η w i = ηκ i with κ i ∈ [1 / 2 , 3 / 2] , and denote κ = ( κ 1 , . . . , κ d ) ∈ R d for the first layer training. This perturbation is necessary to break possible coordinate symmetries; see Remark 7.3.

Theorem 7.2 (Informal version of Theorem E.1) . For a convex and analytic loss ℓ , almost surely over the perturbation κ and the initial bias ¯ c ∈ R the following holds. If Leap DLQ ℓ ( µ ) = 1 , then the above layer-wise SGD training dynamics with total sample size n = Θ d ( d ) and M = Θ d (1) achieves excess test error R ( f NN ( · ; Θ k 1 + k 2 )) = o (1) with high probability.

The formal statement and the precise training specifications can be found in Appendix D. This result generalizes Abbe et al. [2022, Theorem 9] beyond squared loss.

Remark 7.3. A slight coordinate-wise perturbation in the step-sizes for the first layer training is necessary to break the potential coordinate symmetries in the output y -see discussion in Abbe et al. [2022, Appendix A]. This can be removed by either stating the theorem for all but some measure zero set of Leap-1 functions as in Abbe et al. [2022], or by studying the dynamics for O (log d ) steps.

The query complexity of b -batch SGD on M neurons for Υ SGD-steps is T C = Θ( bM Υ d ) . The above theorems show that for Leap DLQ ℓ ( µ ) = 1 , Υ = Θ( d/b ) steps with M = Θ(1) neurons-and therefore T C = Θ( d 2 ) -suffices to learn the support and minimize the excess test error. Furthermore, for Leap DLQ ℓ ( µ ) > 1 and neural networks trained in the mean-field regime, Υ = Θ( d ) (and therefore T C = Θ( d 2 ) ) is not enough. We further comment on the general conjectural picture in Appendix A.2.

Numerical Simulation. Weconsider a function similar to y 2 in (2) with Leap CSQ = 3 but Leap SQ = 1 . Specifically, we set P = 4 and C = {{ 1 , 2 , 3 } , { 1 , 2 , 4 } , { 1 , 3 , 4 } , { 2 , 3 , 4 }} , and define y = h ∗ ( z ) where

We train with online 1-batch SGD ( ℓ -bSGD) on a two-layer net with different loss functions (without any regularization) and stepsize η ∝ 1 /d , where we consider ambiant dimensions d ∈ { 100 , 300 , 500 } . In Figure 1, we plot the test mean squared error versus η × SGD-iterations (thus also scaled with 1 /d ), over 10 trials. Additionally, we also plot the continuous time (DF-PDE) in (dashed black line) that corresponds to the limit d →∞ .

250

500

750

1000

1250

1500

1750

2000

t

=

×iterations

0.20

0.15

0.10

0.05

0.00

Test mean squared error

SGD

d

=100

SGD

d

=300

SGD

d

=500

effective

d

=

(a)

ℓ

(

u, y

) = (

u

-

y

)

2

250

500

750

1000

1250

1500

1750

2000

t

=

×iteration

0.00

0.05

0.10

0.15

0.20

Test mean squared error

SGD

d

=100

SGD

d

=300

SGD

d

=500

effective

d

=

(b)

ℓ

(

u, y

) =

|

u

-

y

|

250

500

750

1000

1250

1500

1750

2000

t

=

×iteration

0.00

0.05

0.10

0.15

0.20

Test mean squared error

SGD

d

=100

SGD

d

=300

SGD

d

=500

effective

d

=

(c)

ℓ

(

u, y

) = (

u

-

y

)

2

+

|

u

-

y

|

3

8 Conclusion and Outlook

In this paper, we considered learning juntas over general product distributions with statistical query algorithms. To capture learning with gradient evaluations over a general loss and arbitrary model, we introduced Differentiable Learning Queries ( DLQ ℓ ), which can be seen as a generalization of correlation statistical queries beyond squared loss. We then showed that the complexity of learning juntas with either SQ , CSQ , or DLQ ℓ algorithms is sharply characterized in terms of a leap exponent (adaptive queries) or a cover exponent (non-adaptive queries). These exponents are defined in terms of a minimal combination of detectable sets to cover the support, where the system of detectable sets depends on the allowed queries. In general, the leap and cover exponents for different losses are not comparable. However, we identify 'generic' losses, including ℓ 1 , where DLQ ℓ algorithms are as powerful as SQ algorithms for learning juntas. We further showed that DLQ ℓ can indeed capture the complexity of learning with SGD in the case of data on the hypercube.

Worst-case v.s. One-pass v.s. Multi-pass SGD. DLQ ℓ (like SQ ) is defined in terms of worst-case noise. It is well understood that worst-case noise is theoretically very different from estimating population gradients based on samples (either independent samples as in one-pass (S)GD, or with repeated use of samples as in full-batch or multi-pass) when highly specialized models are allowed [Abbe et al., 2021b]. However, we expect DLQ ℓ to still capture the complexity of one-pass SGD in many settings of interest-with 'regular' models-such as in Section 7. With sample reuse (e.g. multi-pass) the situation is more complex: Dandi et al. [2024] showed that two steps of full-batch gradient descent with square loss goes beyond CSQ . Heuristically, and in light of our work, two steps on the same batch can be seen as a single gradient evaluation on a modified loss, thus going beyond CSQ = DLQ ℓ sq , but remaining inside DLQ ˜ ℓ ⊆ SQ for some perturbed non-quadratic loss ˜ ℓ . Indeed, we expect generally that multi-pass SGD on a 'regular' model will remain in SQ .

Multi-index and beyond. In this paper, we focused on learning sparse functions. We hope the modular nature of our analysis framework (defining detectable sets in terms of test functions, and leap and cover complexities of set systems), our definition of DLQ , and the distinctions we emphasize between CSQ , SQ and DLQ and between adaptive and non-adaptive complexities, will be helpful in guiding and contextualizing analysis in other settings such as learning single-index or multi-index functions [e.g. Refinetti et al., 2021, Mousavi-Hosseini et al., 2022, Abbe and Boix-Adsera, 2022, Damian et al., 2022, Bietti et al., 2022, 2023, Damian et al., 2024a]. For example, the information exponent for single-index [Arous et al., 2021] can be seen as analogous to our CSQ -cover exponent, the generative exponent for single-index [Damian et al., 2024b] as analogous to our SQ -cover exponent, and the isoLeap exponent for multi-index [Abbe et al., 2023, Dandi et al., 2023, Bietti et al., 2023] as analogous to our CSQ -leap exponent. It would be interesting to obtain a unified understanding of these specialized treatments and extend our general framework to multi-index models, and also to learning under other invariances beyond permutations and rotations.

In our setup, we emphasize generic input and output spaces, without a field structure. This emphasizes that when learning juntas, polynomials or degree of input or output coordinates is irrelevant. Defining multi-index models and introducing rotational invariance necessitates a field structure and gives rise to the relevance of polynomial degrees and decomposition.

An important point is that when considering permutation (as in juntas) vs. rotational (as in multi-index models) invariance, one must consider not only the invariance structure of the target function class, but also the input distribution (i.i.d. coordinates as in our case, or more generally exchangeable vs. spherical) and learning rule. E.g., learning a parity over input coordinates requires only Θ(log d ) samples, but a rotationally equivariant algorithm effectively learns parities also over rotated axis, which requires Ω( d ) samples [Glasgow, 2023] (and thus Ω( d 2 ) runtime). This also explains the need for Θ( d ) steps and thus Θ( d 2 ) runtime to learn leap-1 functions using SGD on a rotationally invariant neural net in Section 7. In order to break the Ω( d ) -sample lower bound, we need to break the rotation-equivariance, e.g. using sparse initialization and ℓ 1 regularization, which indeed can achieve Θ(log( d )) sample complexity.

Acknowledgement. This research was done as part of the NSF-Simons Sponsored Collaboration on the Theoretical Foundations of Deep Learning and the NSF Tripod Institute on Data, Econometrics, Algorithms, and Learning (IDEAL).

References

Emmanuel Abbe and Enric Boix-Adsera. On the non-universality of deep learning: quantifying the cost of symmetry. Advances in Neural Information Processing Systems , 35:17188-17201, 2022.

Emmanuel Abbe and Colin Sandon. On the universality of deep learning. Advances in Neural Information Processing Systems , 33:20061-20072, 2020.

Emmanuel Abbe, Enric Boix-Adsera, Matthew S Brennan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hierarchical structure can guide deep learning. Advances in Neural Information Processing Systems , 34:26989-27002, 2021a.

Emmanuel Abbe, Pritish Kamath, Eran Malach, Colin Sandon, and Nathan Srebro. On the power of differentiable learning versus pac and sq learning. Advances in Neural Information Processing Systems , 34:24340-24351, 2021b.

Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory , pages 4782-4887. PMLR, 2022.

Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory , pages 2552-2623. PMLR, 2023.

Zeyuan Allen-Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep learning. arXiv e-prints , pages arXiv-2001, 2020.

Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. Journal of Machine Learning Research , 22 (106):1-51, 2021.

Alberto Bietti, Joan Bruna, Clayton Sanford, and Min Jae Song. Learning single-index models with shallow neural networks. Advances in Neural Information Processing Systems , 35:9768-9783, 2022.

Alberto Bietti, Joan Bruna, and Loucas Pillaud-Vivien. On learning gaussian multi-index models with gradient flow. arXiv preprint arXiv:2310.19793 , 2023.

Avrim L Blum and Pat Langley. Selection of relevant features and examples in machine learning. Artificial intelligence , 97(1-2):245-271, 1997.

Nader H Bshouty and Vitaly Feldman. On using extended statistical queries to avoid membership queries. Journal of Machine Learning Research , 2(Feb):359-395, 2002.

Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

Michael Celentano, Theodor Misiakiewicz, and Andrea Montanari. Minimum complexity interpolation in random features models. arXiv preprint arXiv:2103.15996 , 2021.

Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems , 31, 2018.

George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303-314, 1989.

Alex Damian, Eshaan Nichani, Rong Ge, and Jason D Lee. Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models. Advances in Neural Information Processing Systems , 36, 2024a.

Alex Damian, Loucas Pillaud-Vivien, Jason D Lee, and Joan Bruna. The computational complexity of learning gaussian single-index models. arXiv preprint arXiv:2403.05529 , 2024b.

Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. In Conference on Learning Theory , pages 5413-5452. PMLR, 2022.

Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. Learning two-layer neural networks, one (giant) step at a time. arXiv preprint arXiv:2305.18270 , 2023.

Yatin Dandi, Emanuele Troiani, Luca Arnaboldi, Luca Pesce, Lenka Zdeborová, and Florent Krzakala. The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents. arXiv preprint arXiv:2402.03220 , 2024.

Richard M Dudley. Real analysis and probability . Chapman and Hall/CRC, 2018.

Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Pareto frontiers in deep feature learning: Data, compute, width, and luck. Advances in Neural Information Processing Systems , 36, 2024.

Vitaly Feldman, Elena Grigorescu, Lev Reyzin, Santosh S Vempala, and Ying Xiao. Statistical algorithms and a lower bound for detecting planted cliques. Journal of the ACM (JACM) , 64(2): 1-37, 2017.

Margalit Glasgow. Sgd finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the xor problem. In The Twelfth International Conference on Learning Representations , 2023.

Samuel Hopkins. Statistical inference and the sum of squares method . PhD thesis, Cornell University, 2018.

Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks , 4(2): 251-257, 1991.

Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45(6):983-1006, 1998.

Yiwen Kou, Zixiang Chen, Quanquan Gu, and Sham M Kakade. Matching the statistical query lower bound for k-sparse parity problems with stochastic gradient descent. arXiv preprint arXiv:2404.12376 , 2024.

Dmitriy Kunisky, Alexander S Wein, and Afonso S Bandeira. Notes on computational hardness of hypothesis testing: Predictions using the low-degree likelihood ratio. In ISAAC Congress (International Society for Analysis, its Applications and Computation) , pages 1-50. Springer, 2019.

Eran Malach, Pritish Kamath, Emmanuel Abbe, and Nathan Srebro. Quantifying the benefit of using differentiable learning over tangent kernels. In International Conference on Machine Learning , pages 7379-7389. PMLR, 2021.

Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of twolayer neural networks. Proceedings of the National Academy of Sciences , 115(33):E7665-E7671, 2018.

Alireza Mousavi-Hosseini, Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat A Erdogdu. Neural networks efficiently learn low-dimensional representations with sgd. arXiv preprint arXiv:2209.14863 , 2022.

DJ Newman and Morton Slater. Waring's problem for the ring of polynomials. Journal of Number Theory , 11(4):477-487, 1979.

Amelia Perry, Alexander S Wein, Afonso S Bandeira, and Ankur Moitra. Optimality and suboptimality of pca i: Spiked random matrix models. The Annals of Statistics , 46(5):2416-2451, 2018.

Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdeborová. Classifying highdimensional gaussian mixtures: Where kernel methods fail and neural networks succeed. In International Conference on Machine Learning , pages 8936-8947. PMLR, 2021.

Lev Reyzin. Statistical queries and statistical algorithms: Foundations and applications. arXiv preprint arXiv:2004.00557 , 2020.

Grant M Rotskoff and Eric Vanden-Eijnden. Neural networks as interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error. stat , 1050:22, 2018.

Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications , 130(3):1820-1852, 2020.

Le Song, Santosh Vempala, John Wilmes, and Bo Xie. On the complexity of learning neural networks. Advances in neural information processing systems , 30, 2017.

Sho Sonoda and Noboru Murata. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis , 43(2):233-268, 2017.

Gregory Valiant. Finding correlations in subquadratic time, with applications to learning parities and juntas. In 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science , pages 11-20. IEEE, 2012.

Santosh Vempala and John Wilmes. Gradient descent for one-hidden-layer neural networks: Polynomial convergence and sq lower bounds. In Conference on Learning Theory , pages 3115-3117. PMLR, 2019.

A Additional discussions from the main text

A.1 Necessity of Assumption 2.1

We further elaborate on the necessity of Assumption 2.1 in our setting. Informally, we require the label to have 'enough noise'. This is necessary to obtain meaningful lower-bounds for general SQ algorithms, as noted in Valiant [2012], Song et al. [2017], Vempala and Wilmes [2019]. When learning real-valued target function classes, allowing for general measurable queries is too weak (note that arbitrary measurable functions are not practical anyway). Indeed, Vempala and Wilmes [2019] showed that any finite set H of (noiseless) functions can be learned with log |H| (bounded) queries and constant tolerance 8 . They identify three possible approaches to address this challenge: (i) Require y ↦→ ϕ ( y, x ) to be Lipschitz for every fixed x ; (ii) Insist on noisy concepts, e.g., y = f ( x ) + ζ , with ζ ∼ N (0 , σ 2 ) , which is equivalent to Lipschitz queries; (iii) Restrict the form of the queries, such as in CSQ . Our Assumption 2.1 can be viewed as a quantitative version of approach (ii). For approach (iii), if we restrict ourselves to CSQ or DLQ ℓ with ℓ piecewise analytic, we can indeed remove Assumption 2.1 by modifying the proof of Theorem 5.1.

A.2 The conjectural picture of complexity of learning ( ℓ -bSGD)

For k ∗ := Leap DLQ ℓ ( µ ) , learning the junta problem with online SGD on loss ℓ requires

Partial evidence was provided towards this conjecture for the square loss in Abbe et al. [2023], Bietti et al. [2023]: they showed that learning in this setting happens through a saddle-to-saddle dynamics, where each time the neural network needs to align to k > 1 new coordinates, a saddle appears and SGD requires ˜ Θ( d k -1 ) steps to escape.

200

400

600

800

1000

1200

×iterations

0.0

0.2

0.4

0.6

0.8

1.0

Test mean squared error

SGD

d

=100

SGD

d

=300

SGD

d

=500

effective

d

=

(a)

ℓ

(

u, y

) = (

u

-

y

)

2

200

400

600

800

1000

1200

t

=

×iteration

0.0

0.2

0.4

0.6

0.8

1.0

Test mean squared error

SGD

d

=100

SGD

d

=300

SGD

d

=500

effective

d

=

(b)

ℓ

(

u, y

) =

|

u

-

y

|

200

400

600

800

1000

1200

×iterations

0.0

0.2

0.4

0.6

0.8

1.0

Test mean squared error

SGD

d

=100

SGD

d

=300

SGD

d

=500

effective

d

=

(c)

ℓ

(

u, y

) = (

u

-

y

)

2

+

|

u

-

y

|

3

A.3 Identifiability and Relative Leap and Cover Complexities

The Leap and Cover complexity may be infinite when supp( C A ) = ∪ U ∈C A U ⊊ [ P ] , in which case Theorem 5.1 (correctly) implies that we cannot recover the relevant coordinates that correspond to [ P ] \ supp( C A ) using queries of type A (even with perfect precision). In this case, we can characterize the complexity of recovering supp( C A ) instead, and this is captured by the following 'relative' complexities:

̸

Definition 3. For any system of subsets C ⊆ 2 [ P ] the relative leap and cover complexities are:

A slight variant of Theorem 5.1 can then be shown, where relLeap A ( µ ) and relCover A ( µ ) for A ∈ { SQ , CSQ , DLQ ℓ } characterizes the complexity of recovering supp( C A ) .

For SQ , we may have supp( C SQ ) ⊊ [ P ] only when y doesn't actually depend on some of the coordinates in [ P ] (i.e. y | z = y | ( z i ) i ∈ supp( C ) ). In this case, once we recover supp( C ) (and possibly enumerating over permutations of its coordinates) we can also recover the conditional y | x . That is, relLeap SQ and relCover SQ characterize the complexity of learning the distribution D s ∗ , even if not the (unidentifiable) set S ∗ .

For DLQ ℓ , including CSQ , we may not be able to identify some coordinates even if y does depend on them. Consider for example the junta problem of size P = 2 where y | z = z 1 + N (0 , z 2 2 ) . Although y does depend on the second coordinate z 2 , it is not identifiable using CSQ queries, supp( C CSQ ) = { 1 } , and using CSQ queries we cannot recover the conditional distribution D s ∗ (even if we know the link µ y | z ). What we can say for DLQ ℓ after recovering ˆ S = supp( C DLQ ℓ ) , is that although we might not know D s ∗ , we can find the ℓ -risk minimizer f ∗ = arg min f : X→F E ( y, x ) ∼D s ∗ [ ℓ ( f ( x ) , y )] . That is, relLeap DLQ ℓ and relCover DLQ ℓ characterize the complexity of ℓ -risk minimization.

A.4 Property of SQ Leap and Cover Complexities

We show a useful property of the SQ -exponent. This characterization is similar to the definition of the generative exponent for single-index models on Gaussian data in [Damian et al., 2024b, Definition 2.4], while the detectable set Definition 2 is similar to their variational representation [Damian et al., 2024b, Proposition 2.6].

Proposition A.1 (Property of SQ -detectable sets) . U ∈ C SQ if and only if there exists T i ∈ L 2 0 ( µ x ) for all i ∈ U such that ∥ ξ U ∥ L 2 ( µ y ) > 0 where

Proof of Proposition A.1. In the forward direction, consider any U ∈ C SQ , then by Definition 2, there exists T i ∈ L 2 0 ( µ x ) and T ∈ L 2 ( µ y ) such that

̸

where we defined ξ U ( y ) := E µ y, z [∏ i ∈ U T i ( z i ) | y ] . We then have

Therefore, we conclude that ∥ ξ U ∥ L 2 ( µ y ) > 0 as desired. In the opposite direction, let us consider U ⊆ [ P ] such that there exists T i ∈ L 2 0 ( µ x ) for every i ∈ U such that ∥ ξ U ∥ L 2 ( µ y ) > 0 , where

Then let T ( y ) := ξ U ( y ) . It is straightforward to verify that

We verified T ( y ) ∈ L 2 ( µ y ) . Moreover,

Therefore, by Definition 2, we obtain U ∈ C SQ .

B Proof of Theorem 5.1

B.1 Preliminaries

Tensor basis of L 2 ( X k , µ k x ) : Since we assumed that ( X , µ x ) is a Polish probability space, L 2 ( X , µ x ) is separable [Dudley, 2018]. Consider { ψ i } i ∈I , I ⊆ N , an orthonormal basis of L 2 ( X , µ x ) such that ψ 0 = 1 without loss of generality. In particular, the space L 2 ( X k , µ k x ) admits the following tensor basis

We will denote supp( i ) = { j ∈ [ d ] : i j > 0 } .

Distributions ν σ S : For any non-repeating sequence σ ∈ P ([ d ] , P ) of length P , a subset U ⊆ [ P ] , and a system of subsets C = { U 1 , . . . , U m } ⊆ 2 [ P ] , we denote σ ( U ) := { σ ( i ) : i ∈ U } and σ ( C ) := { σ ( U 1 ) , . . . , σ ( U m ) } . The following analysis holds for fix µ and hence we will often omit the subscript µ by denoting D d µ,σ = D d σ for any σ ∈ P ([ d ] , P ) . For σ = Id , i.e. σ (1) = 1 , . . . , σ ( P ) = P , we will further omit writing Id and denote D d = D d Id . For U ⊆ [ P ] , µ 0 y, z ,U ∈ P ( Y × X P ) is the distribution of ( y U , z ) corresponding to decoupling y from ( z i ) i ̸∈ U in µ y, z following Eq. (6), i.e., ( y U , ( z i ) i ∈ U ) ∼ µ y, z ,U and ( z i ) i ̸∈ U ∼ µ P -| U | x independently.

For clarity, for every U ⊆ [ P ] and σ ∈ P ([ d ] , P ) ,

where we abuse the notation and view σ as a permutation in Π d , by extending the sequence σ ∈ P ([ d ] , P ) to an ordering of irrelevant coordinates, as notice that it yields the same ν σ U . In words, ν σ U is the distribution of ( y, x ) where we decoupled y from ( x σ ( i ) ) i ̸∈ U in D d σ . For σ = Id , we denote ν U = ν Id U . For U = [ P ] , we simply denote ν σ := D d σ . Note that ν 0 := ν ∅ = µ y ⊗ µ d x and ν σ U = ν U if σ ( i ) = i for all i ∈ U .

Basic properties under Assumption 2.1. Recall that our lower bounds in Theorem 5.1 are under Assumption 2.1. It states that the junta problem µ = ( µ x , µ y | z ) is such that Eq. (7) holds. An immediate consequence of this assumption, using Jensen's inequality is that

Indeed, this follows from

where the first inequality follows from Jensen's inequality and the second by (7).

Another consequence of this assumption is that, for any U ⊆ [ P ] and σ ∈ P ([ d ] , P ) , we have L 2 ( ν 0 ) ⊆ L 1 ( ν σ U ) . This is because for any ϕ ∈ L 2 ( ν 0 )

where we used Cauchy-Schwarz inequality and Eq. (21).

̸

Algorithms in A : Throughout the proof, we consider A ∈ { SQ , CSQ , DLQ ℓ } and C A := C A ( µ ) defined in Definition 2. First, observe that one can alternatively define Q SQ = L 2 ( ν 0 ) because for any measurable ϕ / ∈ L 2 ( ν 0 ) , the received query response can infinite by (8) and the learner does not gain any information. Therefore, it suffices to prove lower bound against the query set Q SQ = L 2 ( ν 0 ) . For the same reason, consider the allowed query sets, Q CSQ ⊆ L 2 ( ν 0 ) containing queries of the form ϕ ( y, x ) = y ˜ ϕ ( x ) , and Q DLQ ℓ ⊆ L 2 ( ν 0 ) with queries of the form

We claim that, for i ∈ I d and ψ i defined in Eq. (19) and any ϕ ∈ Q A , Definition 2 implies that if supp( i ) ̸∈ σ ( C A ) , then almost surely over ˜ x ∼ µ d x

This follows from Definition 2 of Ψ -detectibility with Eq. (14), and using the fact that ψ i is the product of ψ i j ( x j ) with ψ i j ∈ L 2 0 ( µ x ) for j ∈ supp( i ) . More specifically, since ϕ ∈ L 2 ( ν 0 ) , the function ϕ ( · , ˜ x ) ∈ L 2 ( µ y ) almost surely over ˜ x ∼ µ d x , and In particular, almost surely over ˜ x ∼ µ d x :

B.2 Theorem 5.1.(a): lower bound for adaptive queries

Definition of set S ∗ : Let k ∗ = Leap ( C A ) . Define C ∗ ⊂ C A to be the maximal subset of C A such that Leap ( C ∗ ) ≤ k ∗ -1 , and S ∗ = ∪ U ∈C ∗ U . In words, S ∗ is the collection of all coordinates in the support that can be reached by adding at most k ∗ -1 coordinates at a time. In particular, by definition of C ∗ , we have | U \ S ∗ | ≥ k ∗ for every U ∈ C A \ C ∗ . Denote r = |S ∗ | and without loss of generality, assume that S ∗ = { 1 , . . . , r } . We decompose the covariate into x = ( x S ∗ , x S c ∗ ) where S c ∗ = [ d ] \ S ∗ . Using these notations, ( y σ S ∗ , x ) ∼ ν σ S ∗ has ( y σ S ∗ , x σ ( S ∗ ) ) ∼ D y, z , S ∗ and x σ ( S c ∗ ) ∼ µ d -r x independently.

Definition of H d µ, S ∗ : We introduce P ([ d ] , P, S ∗ ) ⊆ P ([ d ] , P ) the subset of non-repeating sequence of length [ P ] from elements [ d ] such that σ ( i ) = i for all i ∈ S ∗ , and H d µ, S ∗ ⊆ H d µ the subset of hypotheses D d σ with σ ∈ P ([ d ] , P, S ∗ ) . Recall that for all σ ∈ P ([ d ] , P, S ∗ ) , we have ν σ S ∗ = ν S ∗ . To prove our lower bound, we will lower bound the complexity of learning H d µ, S ∗ which implies a lower bound on H d µ . Specifically, we will show that with q/τ 2 ≤ cd k ∗ for some constant c > 0 that only depends on P, µ (and the loss ℓ for DLQ ℓ ), statistical query algorithms in A cannot distinguish for all σ ∈ P ([ d ] , P, S ∗ ) between ν σ = D d σ and ν S ∗ .

Proof outline: For any σ ∈ P ([ d ] , P, S ∗ ) and query ϕ ∈ Q A ⊆ L 2 ( ν 0 ) , define

We will show below that there exists a constant C > 0 that only depends on µ such that

where E σ denote the expectation with respect to σ ∼ Unif( P ([ d ] , P, S ∗ )) . The intuition behind this crucial bound in (25) is that we defined the null distribution as the distribution with matching marginals ( y, x ) where the label only depends on the support of x discoverable with 'leap" k ∗ . Hence, to make any progress, any new detectable set added to the support needs to contain at least k ∗ new coordinates. In the second moment, to have a non-zero contribution, we need the permutation to map the support to a detectable subset, i.e. the permutation to map to a subset of size at least k ∗ , which has probability O ( d -k ∗ ) over permutation.

Let ϕ 1 , . . . , ϕ q be the sequence of adaptive queries with responses E ν S∗ [ ϕ t ] . Then, by Markov's inequality,

Hence, for q/τ 2 < d k ∗ /C , there exists σ ∈ P ([ d ] , P, S ∗ ) such that for every t ∈ [ q ] , we have | E ν σ [ ϕ t ] -E ν S∗ [ ϕ t ] | ≤ τ ∥ ϕ t ∥ L 2 ( ν 0 ) . Therefore Using Eq (26), we obtain that that there are two members from H d µ whose query responses are compatible with the null ν S ∗ , and hence also mutually compatible and indistinguishable from each other. This then establishes the failure of the learning algorithm in recovering the correct support.

Decomposing ∆ σ ( ϕ ) : We rewrite ∆ σ ( ϕ ) in terms of E ν S∗ using the Radon-Nikodym derivative:

Recall that ( y S ∗ , x S ∗ ) is independent of x σ ( S c ∗ ) ∼ µ d -r x under ν S ∗ , and that d ν σ / d ν S ∗ ∈ L 2 ( ν S ∗ ) by Assumption 2.1. Therefore, we have the following orthogonal decomposition in L 2 ( ν S ∗ ) using the tensor basis (19):

where

Similarly, we decompose ϕ orthogonally in L 2 ( ν S ∗ ) :

where

We deduce that we can decompose ∆ σ ( ϕ ) as

For convenience, denote m σ i the summand. From the expressions (29) and (31) and by Fubini's theorem, we can rewrite m σ i as

Let us decouple y from x S ∗ inside the query ϕ using the Radon-Nikodym derivative: conditional on ˜ x S c ∗ , we have

Similarly to Eq. (28), we have the following orthogonal decomposition in L 2 ( ν σ S c ∗ ) :

where

By Fubini's theorem,

Combining Eqs. (34), (35), and (37) into Eq. (33) yields the expression

From Eq. (32), we deduce the following simple decomposition

Bounding E σ [∆ σ ( ϕ ) 2 ] : If supp( i ) ̸∈ σ ( C A \ C ∗ ) , we have by Eq. (23) that

E

˜

x

∼

µ

d

x

ψ

i

(˜

x

)

(

y,

x

)

∼

ν

σ

Therefore we can rewrite the sum in Eq. (39) with

By squaring Eq. (41) and the previous display, we deduce that

where we used in the third and fourth line Cauchy-Schwarz inequality, and in the last line

by Assumption 2.1. Note that the above bound is uniform over all ϕ ∈ L 2 ( ν 0 ) . This concludes the proof.

[

E

[

ϕ

(

y,

˜

x

)

ψ

i

(

x

)]

]

= 0

.

(40)

Remark B.1. We remark that indeed these proofs, after alternatively defining k ∗ = relLeap ( C A ) and C ∗ to be the maximal subset of C A such that relLeap ( C ∗ ) = k ∗ -1 , shows that if q/τ 2 ≤ cd k ∗ , then mthods in A fail to recover coordinates from C A \ C ∗ .

B.3 Theorem 5.1.(b): lower bound for non-adaptive queries

The proof follows from a similar argument to the adaptive case.

Proof outline: Let k ∗ = Cover ( C A ) and l ∗ that maximizes min S ∈C A ,l ∈ S | S | . By definition of the cover complexity, | S | ≥ k ∗ for all S ∈ C A with l ∗ ∈ S . Let S ∗ := [ P ] \ { l ∗ } . We define in the non-adaptive case for all ϕ ∈ Q A and σ ∈ P ([ d ] , P ) ,

meaning that we compare the query expectation between ν σ and ν σ S ∗ where we decoupled the label y from coordinate σ ( l ∗ ) . We show again that the bound (25) holds on the second moment of ∆ σ ( ϕ ) / ∥ ϕ ∥ L 2 ( ν 0 ) . Therefore, for any set of q queries { ϕ t } t ∈ [ q ] ⊂ Q A , we get a similar bound (26) on the probability of all queries being less or equal to τ in absolute value. We deduce that for q/τ 2 ≤ cd k ∗ , the algorithm will not be able to distinguish for all σ ∈ P ([ d ] , P ) between ν σ and ν σ S ∗ , and therefore recover the coordinate σ ( l ∗ ) in the support.

Bounding E σ [∆ σ ( ϕ ) 2 ] : Following the same decomposition of ∆ σ ( ϕ ) as above, we first obtain

and then

Therefore, by Eq. (23), the summand is non zero only if supp( i ) ⊆ σ ( C A ) with σ ( l ∗ ) ∈ supp( i ) . Recall that in this case | supp( i ) | ≥ k ∗ by the choice of l ∗ . The rest of the proof follows using the same computation as in the previous section.

B.4 Upper Bounds for Adaptive and Non-Adaptive Queries

Define

with T U ∈ Ψ A and T U i ∈ L 2 0 ( µ x ) chosen to make the expectation non-zero, where without loss of generality we normalize them so that ∥ T U ∥ 2 L 2 ( µ y ) ∏ i ∈ S ∥ T U i ∥ L 2 ( µ x ) = 1 . We can then emulate the support recovery algorithm described below Definition 1 in Section 4 using for every U ∗ ∈ C A and | S | = | U ∗ | , ϕ S ( y, x ) = T U ∗ ( y ) ∏ i ∈ S T U ∗ i ( x i ) (note that here S is an ordered subset) and precision τ < β/ 2 , so that if the response | v t | > β/ 2 , then S ∈ σ ∗ ( C A ) . Note that ϕ S ( y, x ) are indeed in Q A : for SQ , CSQ this is direct, for DLQ ℓ , we can take f t ( x ; ω ) = u + ω · v ∏ i ∈ S T U ∗ i ( x i ) .

If we take τ < β/ 2 , the number of queries only needs to scale as O ( d CoverA ) or O ( d Leap A ) . We can further trade-off the precision and accuracy and get the result of several of the above queries with one query with small enough τ . For example, consider testing d subset { i } to find s ∗ (1) . Then, one can group each of this query in Θ(log( d )) groups G k following the binary representation of i . For each of these groups, consider the query

Note that this is indeed a query in Ψ A : it has L 2 ( ν 0 ) norm O (1) (indeed E [ T ( i ) ( z i ) T ( j ) ( z j )] = 0 for any i = j with at least one not in the support). With precision O (1 / √ d ) , one can test if one of those expectation is non-zero. Repeating for each of the Θ(log( d )) groups allows to recover the binary representation of { s ∗ (1) } . This idea allows to trade-off the query and precision for any q/τ 2 ≥ Cd k ∗ log( d ) .

̸

C Proofs of Technical Results

Proposition 6.1: Denote Y = { a, b } . For any U ∈ C SQ ( µ ) , by Definition 2, there exists T ∈ L 2 ( µ y ) and { T i ∈ L 2 0 ( µ x ) } i ∈ S such that

̸

̸

where we used that E [ T i ( z i )] = 0 . Hence, this expectation is non-zero for any mapping T ( b ) = T ( a ) , in particular T = Id . We deduce that U ∈ C CSQ ( µ ) too.

Consider X = R and µ x = γ the standard Gaussian measure. We assume that the label y = z 1 z 2 · · · z P . Then C CSQ only contains the entire support [ P ] and Cover CSQ ( µ ) = Leap CSQ ( µ ) = P . On the other hand, consider the mappings T ( y ) = ✶ [ | y | ≥ τ ] for some τ > 0 , and T i ( z i ) = z 2 i -1 . Then

̸

and C SQ ( µ ) contains all the singletons { i } for i ∈ [ P ] . We deduce Cover SQ ( µ ) = Leap SQ ( µ ) = 1 .

Proposition 6.2.(a): Note that ℓ ′ ( u, y ) = ( u -y ) for the squared loss. For any U ⊆ [ P ] , and any { T i ∈ L 0 2 ( µ x ) : i ∈ [ P ] } , we have

Thus,

̸

This immediately implies C DLQ ℓ = C SQ by Definition 2, and as a consequence Leap CSQ = Leap DLQ ℓ and Cover CSQ = Cover DLQ ℓ .

Proposition 6.2.(b): Consider U ∈ C SQ with

̸

Note that without loss of generality we can assume T ∈ L 2 0 ( µ y ) . If span(Ψ DLQ ℓ ) is dense in L 2 0 ( µ y ) , then for every ε > 0 , there exists M ε ∈ N and ( v j , u j ) j ∈ [ M ε ] ⊆ V × F such that

has ∥ T -˜ T M ε ∥ L 2 ( µ y ) ≤ ε . In particular,

Therefore taking ε sufficiently small, ˜ T M ε has non zero correlation with ∏ i ∈ U T i , and in particular, one of the v T j ∇ ℓ ( u j , y ) must have non-zero correlation too. We deduce that C SQ ⊆ C DLQ ℓ and we conclude using C DLQ ℓ ⊆ C SQ .

Let us assume that there exists nonzero bounded T ∈ L 2 0 ( µ y ) , which we take without loss of generality T ( y ) ⊆ [ -1 , 1] , such that E [ v T ∇ ℓ ( u , y ) T ( y )] = 0 for all u ∈ F , v ∈ V . The goal is to

define µ y, z such that C DLQ ℓ ⊊ C SQ . Specifically, we will construct a joint distribution on ( y, z i ) with { i } ∈ C SQ but { i } ̸∈ C DLQ ℓ . Let A ⊂ X with µ x ( A ) ∈ (0 , 1) and consider y that only depends on z i through ✶ [ z i ∈ A ] . Denote P ( A | y ) = P ( z i ∈ A | y ) . For any T i ∈ L 2 0 ( µ x ) , E [ T i ( z i ) | y ] = P ( A | y ) E [ T i ( z i ) | z i ∈ A ] + P ( A c | y ) E [ T i ( z i ) | z i ̸∈ A ] . (54)

Denote κ ( A ) = E [ T i ( z i ) | z i ∈ A ] . By definition E [ T i ( z i )] = 0 = µ x ( A ) κ ( A ) + µ x ( A c ) κ ( A c ) , and therefore κ ( A c ) = -µ x ( A ) κ ( A ) / (1 -µ x ( A )) . Hence,

Taking λ > (1 -µ x ( A )) /µ x ( A ) , we can set

We then define the joint distribution of ( y, z i ) as

Note that E [ P ( A | y )] = µ x ( A ) using that E [ T ( y )] = 0 , and therefore the marginals of µ y,z i are indeed µ y and µ x . For any T i ∈ L 2 0 ( µ x ) and ( u , v ) ∈ F × V , we have

and therefore { i } ̸∈ C DLQ ℓ . On the other hand T ∈ L 2 ( µ y ) and therefore { i } ∈ C SQ . We conclude that the joint distribution ( y, z i ) has Leap DLQ ℓ = Cover DLQ ℓ = ∞ while Leap SQ = Cover SQ = 1 .

Proposition 6.2.(c): We construct a simple example with X = { +1 , -1 } and µ x = Unif( X ) , i.e., uniform on the discrete hypercube. Consider P = 2 k and label

The set C CSQ ( µ ) contains all subsets [2 k ] \ { i } for all i ∈ [2 k ] , and therefore

Consider the loss function ℓ ( u, y ) = 1 2 ( y -u ) 2 + 1 4 ( y -u ) 4 . The derivative is given by

We have

For k ≥ 3 , y 3 only contains monomials over ≥ 2 k -3 ≥ 3 coordinates. Therefore C DLQ ℓ ( µ ) contains all pairs { i, j } but no singleton. Hence,

Finally, for C SQ , taking T ( y ) = y 2 k -1 , it contains all singleton { i } and therefore

C.1 Proof of Theorem 6.3

Note that C A does not change if we add or remove constant functions to Ψ A .

ℓ 1 -loss: First consider ℓ ( u, y ) = | y -u | . We have ∇ ℓ ( u, y ) := ℓ ′ ( u, y ) = sign ( u -y ) , where we can set without loss of generality sign (0) = 0 . From Hornik [1991, Theorem 5] and Hornik [1991, Theorem 1], we directly conclude that for any probability measure µ y on R , we have span(Ψ A ) dense in L 2 ( µ y ) and we conclude using Proposition 6.2.(b).

Hinge loss: Consider the Hinge loss ℓ ( u, y ) = max(1 -yu, 0) . We get ℓ ′ ( u, y ) = -y ✶ uy ≤ 1 where without loss of generality, we set ℓ ′ (1 /y, y ) = -y . For a > 0 and b < 0 , we have ℓ ′ ( a, y ) -ℓ ′ ( b, y ) = y ✶ [ y > a ] -y ✶ [ y < b ] . Taking b →-∞ , we have ℓ ′ ( a, y ) -ℓ ′ ( b, y ) that converges to y ✶ [ y > a ] in L 2 ( µ y ) (recall that we assume that the second moment of µ y is bounded). Similarly, we can construct y ✶ [ y > a ] for a ≤ 0 . Following the proof of Hornik [1991, Theorem 1], if span(Ψ A ) is not dense in L 2 ( µ y ) , there exists g ∈ L 2 ( µ y ) such that

Define the signed measure σ ( B ) = ∫ B yg ( y )d µ y . It is finite since ∫ | yg ( y ) | d µ y ≤ ∥ y ∥ L 2 ( µ y ) ∥ g ∥ L 2 ( µ y ) < ∞ by assumption. We deduce that

Exponential loss: Consider Y ⊆ [ -M,M ] and ℓ ( u, y ) = e -uy , so that ℓ ′ ( u, y ) = -ye -yu . Any Borel probability measure µ y on [ -M,M ] is regular, and in particular continuous functions and therefore polynomials are dense in L 2 ( µ y ) . To show that span { ye -yu : u ∈ R } is dense in L 2 ( µ y ) , it is sufficient to show that we can approximate any monomial y k in L 2 ( µ y ) . This is readily proven recursively, using that

(The case a = 0 is obtained by taking u → -∞ in Eq. (65).) We then use that ✶ [ ay -u > 0] is discriminatory and therefore no such finite signed measure exists.

This concludes the proof of this theorem.

D SGD, mean-field and limiting dimension-free dynamics

Fix a link distribution y | z ∼ µ y | z and consider data distribution ( y, x ) ∼ D d s ∗ ∈ H d µ with µ = ( µ x , µ y | z ) and (unknown) s ∗ ∈ P ([ d ] , P ) . We consider learning using a two-layer neural network with parameters Θ ∈ R M ( d +2)+1 and an activation σ : R → R :

For convenience, we will reparametrize the neural network for Θ = ( θ j ) j ∈ [ M ] ∈ R M ( d +3) with θ j = ( a j , w j , b j , c j ) ∈ R d +3 and

Indeed, the two neural networks (NN1) and (NN2) remain equal throughout training under the initialization c 1 = . . . = c M = c .

Also, recall the definitions of the risk and the excess risk, respectively, for a data distribution ( y, x ) ∼ D , with respect to a loss function ℓ : R ×Y → R ≥ 0 .

We start with the initialization specified by ρ 0

Batch Stochastic Gradient Descent. We train the parameters Θ using batch-SGD with loss ℓ and the batch size b . Even in more generality than stated in ( ℓ -bSGD), we allow for time-varying step size ( η k ) k ≥ 0 , where η k ∈ R d +3 can be different for different parameters (e.g., different layers), and ℓ 2 -regularization λ ∈ R d +3 . We initialize the weights ( θ j ) j ∈ [ M ] i.i.d. ∼ ρ 0 from (69), and at each step, given samples ( { ( x ki , y ki ) : i ∈ [ b ] } ) k ≥ 0 , the weights are updated using

where we introduced H k := diag( η k ) and Λ := diag( λ ) .

Mean-Field Dynamics. A rich line work [Chizat and Bach, 2018, Mei et al., 2018, Rotskoff and Vanden-Eijnden, 2018, Sirignano and Spiliopoulos, 2020] has established a crisp approximation between the dynamics of one-pass-batch-SGD ( ℓ -bSGD-g) and a continuous dynamics in the space of probability distributions in R d +3 . To any distribution ρ ∈ P ( R d +3 ) , we associate the infinite-width neural network

In particular, we recover the finite-width neural network (NN2) by taking ˆ ρ ( M ) := 1 M ∑ j ∈ [ M ] δ θ j .

Denote ˆ ρ ( M ) k the empirical distribution of the weights { θ k j } j ∈ [ M ] after k -steps of ( ℓ -bSGD-g). In our ( ℓ -bSGD-g) dynamics, we allow for a generic step-size schedule that is captured by

and that the data at each step are sampled i.i.d. from D ; the case of D empirical distribution on training data corresponds to multi-pass batch-SGD, while D population distribution corresponds to online batch-SGD. Under some reasonable assumptions (that we will list below on the activation, step-size schedule ξ , initialization ρ 0 , and the loss ℓ ) on taking M sufficiently large and η sufficiently small, setting k = t/η , ˆ ρ ( M ) is well approximated by a distribution ρ t ∈ P ( R d +3 ) that evolves according to the PDE:

where the initial distribution is ρ 0 and we defined H ( t ) = diag( ξ ( t )) . We will refer to the distribution dynamics (MF-PDE) as the mean-field (MF) dynamics. We now specify the assumptions on hyperparameters under which a non-asymptotic equivalence can be derived.

A D.1 (Activation) . Our σ : R → R is three times differentiable with ∥ σ ( k ) ∥ ∞ ≤ K , k = 0 , . . . , 3 . A D.2 (Bounded, Lipschitz step-size schedule) . t ↦→ ξ ( t ) = ( ξ i ( t )) i ∈ [ d +3] has bounded Lipschitz entries: ∥ ξ i ∥ ∞ ≤ K and ∥ ξ i ∥ Lip ≤ K .

A D.3 (Initialization) . The initial distribution ρ 0 ∈ P ( R d +3 ) is of the form:

where µ a , µ w , µ b are independent of d . We further assume that µ a supported on | a | ≤ K , | c | ≤ K , and µ w is symmetric and K 2 -sub-Gaussian.

A D.4 (Loss) . The loss ℓ : R ×Y → R ≥ 0 is twice-differentiable in its first argument for all y ∈ Y and satisfies

Assumptions D.1-D.3 are similar to Abbe et al. [2022]. Informally, Assumption D.4 states that loss has a quadratic upper bound (which is analogues to the smoothness assumption in the convex optimization literature). Later, to show the learnability of leap one junta settings, we will further require ℓ ( · , y ) to be convex and analytic. We note that together these assumptions holds for ( i ) the standard logistic loss for the classification setting when Y ∈ {± 1 } , and ( ii ) for the squared loss and any of its 'analytic perturbations' for the regression setting when Y ⊆ [ -B,B ] . In particular, we would like to mention ℓ ( u, y ) = ( u -y ) archsinh ( u -y ) , for which we have Leap DLQ ℓ = Leap SQ and these assumptions also hold, and hence our results apply. For classification, by Proposition 6.1, anyway LeapSQ = LeapCSQ = Leap DLQ ℓ for any non-degenerate loss (including the standard logistic loss for which our results hold).

Limiting Dimension-Free Dynamics. In the junta learning setting, when y only depends on P ≪ d coordinates, following Abbe et al. [2022, Secion 3], the SGD dynamics ( ℓ -bSGD) concentrates on an effective dimension-free (DF) dynamics as M,d →∞ and η → 0 . This equivalence holds under a certain assumption on the loss function, and other assumptions on the initialization and activation that are similar to the setup of Abbe et al. [2022].

Consider the isotropic parameters H k = diag( η a k , η w k I d , η b k , η c k ) and Λ = diag( λ a , λ w I d , λ b , λ c ) . Using technical ideas from Abbe et al. [2022, Secion 3], the ( ℓ -bSGD-g) concentrates on an effective dimension-free dynamics in the limit M,d →∞ and η → 0 . This limiting dynamics corresponds to the gradient flow on R ( f ) + θ T Λ θ of the following effective infinite-width neural network (recall z ∈ R P is the support)

where G ∼ N (0 , 1) and ¯ ρ t ∈ P ( R P +4 ) is the distribution over ¯ θ t = ( a t , b t , u t , c t , s t ) with u t ∈ R P .

where H ( t ) = diag( ξ a , ξ b , ξ w I P , ξ c , ξ w )( t ) and Λ = diag( λ a , λ b , λ w I P , λ w ) from initialization

where m w 2 = E W ∼ µ w [ W 2 ] 1 / 2 is the second moment of µ w . The non-asymptotic equivalence is characterized by the following theorem that can be seen as a generalization of [Abbe et al., 2022, Theorem 5] to non-squared losses.

Theorem D.5. Fix activation, step-size schedule, initialization, and the loss such that Assumptions D.1 to D.4 hold and consider any T ≥ 1 independent of d . Let (¯ ρ t ) t ≥ 0 be the solution of (DF-PDE) , and { Θ k } k ≥ 0 the trajectory of SGD ( ℓ -bSGD-g) with initialization { θ 0 j } j ∈ [ M ] i.i.d. ∼ ρ 0 . Then there exist a constant C K,T > 0 that only depends on K,T , such that for any d, M, η > 0 with M ≤ e d and 0 < η ≤ 1 / [ C K,T ( d +log( M ))] , and

for all k ≤ T/η , with probability at least 1 -1 /M .

This theorem follows by a straightforward modification of the proof of [Abbe et al., 2022, Theorem 16, Proposition 15] using propagation of chaos argument. We note that the only property about the loss that is needed is Assumption D.4.

D.1 DF-PDE alignment with the support.

In this subsection, we provide a formal statement of Theorem 7.1 and its proof. The following is an extension of [Abbe et al., 2022, Theorem 7] from square loss to general loss.

Theorem D.6 (Formal statement of Theorem 7.1) . Assume that Leap DLQ ℓ ( µ ) > 1 . Then there exists U ⊂ [ P ] , | U | ≥ 2 , such that for all t ≥ 0 , the DF dynamics solution remains independent of coordinates ( z i ) i ∈ U , i.e., f NN ( z ; ρ t ) = f NN (( z i ) i ̸∈ U ; ρ t ) , and therefore fails at recovering the support.

Conversely, if Leap DLQ ℓ ( µ ) = 1 and ℓ is analytic with respect to its first argument, then almost surely over c , there exists t > 0 such that f NN ( z ; ρ t ) depends on all [ P ] .

Proof of Theorem D.6. The proof follows similarly to the proof of [Abbe et al., 2022, Theorem 7]. Assume Leap DLQ ℓ ( µ ) > 1 . Consider C ∗ ⊂ C DLQ ℓ the maximal subset such that Leap ( C ∗ ) = 1 and denote U ∗ = ⋃ U ∈C ∗ U . In words, U ∗ contains the subset of coordinates that are reachable by doing leaps of at most 1. By the assumption that Leap DLQ ℓ ( µ ) > 1 , we must have | [ P ] \ U ∗ | ≥ 2 . Let us show that the weights u i for i ∈ [ P ] \ U ∗ remain equal to 0 throughout the dynamics.

For simplicity, we forget about the other parameters which we set to b = c = s = 0 . For convenience, denote Ω = [ P ] \ U ∗ . First, we can bound by Gronwall's inequality, for t ≤ C , | a t | ≤ KC ′ for all a t on the support of ρ t . From the proof of Theorem 5.1, conditioning on already explored coordinates does not change the leap-complexity. In particular,

Consider the time derivative of u t i for i ∈ Ω . We have

where we used Assumptions D.1 and D.4. Hence,

The first expectation is equal to 0 . Note that m t Ω := sup j ∈ Ω ,u t j ∈ supp( ρ t ) | u t j | has m 0 Ω = 0 and therefore m t Ω = 0 for all t ≥ 0 .

E Learning Leap One Juntas with SGD: Formalizing Theorem 7.2

In this section, we provide our precise layer-wise training algorithm and the proof that it succeeds in learning Leap DLQ ℓ = 1 junta problems. This section follows the similar training procedure from Abbe et al. [2022] for the layer-wise training but adapted to training with general loss function ℓ : R ×Y → R ≥ 0 to show Theorem 7.2. The key difference in the training algorithm from Abbe et al. [2022] is the use of coordinate-wise perturbed step-sizes for the first layer weights to break the symmetry between the coordinates. This allows us to learn even some degenerate cases having coordinate symmetries with leap one, in contrast to Abbe et al. [2022].

Discrete Dimension-free Dynamics. We first consider the dimension-free dynamics similar to (DF-PDE) but for discrete step-size regime (see [Abbe et al., 2022, Appendix C]). We initialize to

and (¯ ρ k ) k ∈ N are induced distribution on ( a 0 , b 0 , u 0 , c 0 , s 0 ) recursively defined as

E.1 Training Algorithm

Choose a loss ℓ : R ×Y → R ≥ 0 and the activation σ ( x ) = (1 + x ) L for L ≥ 2 8 P . We train for Θ(1) total steps in two phases and the batch size of Θ( d ) with the following choice of hyperparameters.

No regularization: set the regularization parameters λ w = λ a = λ b = λ c = 0 .

Initialization: Initialize the first layer weights and biases to deterministically 0.

The second layer weights are sampled uniformly from [ -1 , 1] , e.g. a j i.i.d. ∼ µ a ≡ Unif([ -1 , 1]) . Finally, choose c 0 ∼ δ c =¯ c for the given global bias choice ¯ c ∈ R .

For the dimension-free dynamics, this corresponds to taking

We deploy a two-phase training procedure. Given parameters η and κ ∈ [1 / 2 , 3 / 2] d :

Phase 1: For all k ∈ { 0 , . . . , k 1 -1 } , set η a k = η b k = η c k = 0 and η w k ∈ R d such that η w i k = ηκ i for all i ∈ [ d ] . For the dimension-free dynamics, train the first layer weights u k for k 1 steps, while keeping other parameters fixed, i.e a k = a 0 , b k = 0 , c k = ¯ c .

Phase 2: Set η a k = η and η w k = 0 , η b k = η c k = 0 for k ∈ { k 1 , . . . , k 1 + k 2 -1 } . In words, train the second layer weights a k for k 2 steps, while keeping the first layer weights fixed at u k = u k 1 .

We will take η > 0 to be a small constant to be specified layer and κ i ∈ Unif([1 / 2 , 3 / 2]) be the random perturbation. We also let b = Ω( d ) , hiding constants in η, ε, P, K, µ . For the first phase, it suffices to train for k 1 = P time steps. For the second phase, we train for k 2 = k 2 ( η, ε, P, K, µ ) = Θ(1) time steps to be specified later as we analyze. Note that only unspecified hyperparameter is the global bias initialization ¯ c and the results holds almost surely over this choice.

Theorem E.1 (Formal statement of Theorem 7.2) . Assume ℓ is analytic, convex and that Assumption D.4 holds. Then for any Leap DLQ ℓ ( µ ) = 1 setting, for any D d µ,s ∗ ∈ H d µ , almost surely over the initialization ¯ c ∈ R , and κ ∈ [1 / 2 , 3 / 2] d , the following holds. For any ε > 0 , with

using Ω( · ) to hide constants in ε, K, P, µ, η, κ and ¯ c , the above specified layer-wise training dynamics reaches R ( f NN ( · ; Θ k 1 + k 2 )) ≤ ε with probability at least 9 / 10 .

E.2 Proof of Theorem E.1

The proof of the theorem proceeds by first showing a non-asymptotic approximation guarantee between (d-DF-PDE) and ( ℓ -bSGD-g). To this end, we first start by noting a generalization of [Abbe et al., 2022, Theorem 23] to any loss under satisfying Assumption D.4.

Proposition E.2. Assume that Assumption D.4 on the loss holds. Then for any 0 < η ≤ K and for any Υ ∈ N , there exists a constant C ( K, Υ) such that for ( ℓ -bSGD-g) and (d-DF-PDE) of the specified layer-wise training dynamics, with probability 1 -1 /M , we have

The proof of the proposition follows the similar arguments like propagation of chaos used in Theorem D.5. A direct corollary of this proposition is that it suffices to focus only on the (d-DF-PDE) to show Theorem E.1.

Corollary E.3. Assume that Assumption D.4 holds. Then for total Υ number of ( ℓ -bSGD-g) iterations on a neural network with M ≥ C 0 ( K, Υ , ε ) and the batch-size b ≥ C 1 ( K, Υ , ε ) · d , the aforementioned layer-wise training dynamics, with probability at least 9 / 10

Proof. It is easy to see that it is possible to choose M ≥ C 0 ( K, Υ , ε ) and b ≥ C 1 ( K, Υ , ε ) d for sufficiently large constants such that the right hand side of equality in Proposition E.2 is bounded by ε/ 2 and 1 /M ≤ 1 / 10 . The corollary then follows from Proposition E.2.

In light of Corollary E.3, it suffices to simply focus on (d-DF-PDE) and show that it achieves the excess error of at most ε/ 2 within total steps Υ = O (1) hiding constants in K,µ,P,ε . This will be done by analyzing each phase separately-( i ) showing that the training of the top layer weights for steps k ∈ { k 1 , . . . , k 1 + k 2 -1 } in Phase 2 is a linear model trained with a convex loss so it succeeds in reaching a vanishing risk as long as the corresponding kernel matrix of the feature map is non-degenerate; ( ii ) indeed showing that the corresponding kernel matrix of the feature map after k 1 = P steps of Phase 1 training on the first layer weights is non-degenerate.

Simplified dynamics. We first make a simple observation that b k = 0 , s k = 0 and c k = ¯ c throughout; this allows us to analyze a simpler dynamics. To see this, for ( b k , c k ) , we use η c = η b = 0 throughout, and thus, they remain the same from the initialization. We start with s 0 = 0 and using (d-DF-PDE) (assuming s k = 0) , gives us

E.3 Phase 2 (linear training)

We first analyze a slightly simpler Phase 2 of training. For a convex loss ℓ , the second layer training just corresponds to linear dynamics with kernel K k 1 : { +1 , -1 } P ×{ +1 , -1 } P → R given by

We first show some helper lemmas, and using them, show the main lemma. We will then show the proof of the helper lemmas. We start by proving the existence of a sparse function with a small excess error.

Lemma E.4. For any ε > 0 , there exists a function f ∗ : {± 1 } d → R such that

Moreover, f ∗ ( x ) = h ∗ ( z ) for some h ∗ : {± 1 } P → R indpendent of d , where z is the support of D d s ∗ .

Approximation. The neural network output after training u for k 1 steps is given by:

Observing that for the next k 2 steps, only a is trained while keeping u k 1 fixed, the neural network output is of the following form

now start by showing that our neural network, there exists ¯ ρ ∗ such that we can exactly represent h ∗ ( z ) as long as the resultant (kernel) after 1 st layer training is non-degenerate.

Lemma E.5. If λ min ( K k 1 ) ≥ c for some constant c , then there exists ¯ ρ ∗ ∈ P ( R P +1 ) such that d¯ ρ ∗ ( a, u ) = d¯ ρ ( a ∗ ( u ))d¯ ρ k 1 ( u ) and for all z ∈ {± 1 } P

Moreover, ∥ a ∗ ∥ 2 ¯ ρ ∗ = ∫ a 2 ∗ ( u )d¯ ρ ∗ ( a ∗ ( u )) ≤ c 1 for some constant c 1 := c 1 ( λ min ( K k 1 ) , h ∗ , ¯ c ) .

Convexity and Smoothness. Finally, we will show that the risk objective in terms of ¯ ρ (of the form (75)) is convex and smooth in terms of ¯ ρ . The convexity follows from the convexity of ℓ ( · , y ) and observing that the neural network output in (75) is linear in ¯ ρ ( a ( u )) . The smoothness follows from Assumption D.4.

Lemma E.6. Consider ℓ : R × Y → R ≥ 0 such that ℓ ( · , y ) is convex for every y ∈ Y and that Assumption D.4 holds. Then

is convex and H -smooth with H = K 3 .

While we denote R (¯ ρ ) as an objective in terms of the joint measure ¯ ρ ( a, u ) , it is actually an objective in ¯ ρ ( a ( u ))) since it is of the form where d( a, u ) = d¯ ρ ( a ( u ))d¯ ρ k 1 ( u ) . Also, the convexity and smoothness is in terms of the argument ¯ ρ ( a ( u )) . Having established the convexity and smoothness of the 2 nd layer weights training, we will directly apply the convergence rate guarantees for convex and smooth objectives.

Lemma E.7. Consider any convex, differentiable objective f ( x ) that is H -smooth. Then for any step-size η ≤ 1 H , the gradient descent from initialization x 0 reaches an iterate x k after k steps such that for any ˜ x (not necessarily a minimizer), we have

This is a classical convergence rate guarantee for gradient descent on convex smooth objective [Bubeck et al., 2015, Theorem 3.3], but noting that a proof generalizes even when ˜ x is not necessarily a minimizer. We will apply this guarantees with ˜ x as a near minimizer of f , especially when the actual minimizer may not exist (for example with the logistic loss, the infimum is never achieved). We provide the proof with other lemmas for completeness.

Lemma E.8. Assume that λ min ( K k 1 ) > c > 0 for some constant c , where K k 1 is the (kernel) matrix. Then discrete dimension-free dynamics (d-DF-PDE) in Phase 2 of layer-wise training with step-size η ≤ 1 K 3 as specified before after k 2 := k 2 ( η, ε, P, K, µ, ¯ c ) steps achieves

Proof. First of all, using Lemma E.4, there exists ¯ ρ ∗ such that

Moreover, by Lemma E.6, the objective R (¯ ρ ) is convex and K 3 -smooth, and observe that (d-DF-PDE) are performing gradient descent with step-size η a k = η ≤ 1 K 3 on this objective. Thus, using Lemma E.7

Recall that a k 1 ∼ µ a ≡ Unif([ -1 , 1]) .

for some constant c 2 := c 2 ( λ min ( K k 1 ) , P, µ, ¯ c ) . Therefore, choosing k 2 ≥ 2 c 2 ηε

Combining (78) and (76)

Thus, if we can show that λ min ( K k 1 ) ≥ c > 0 for some constant c := c ( η, P, K, µ ) for a sufficiently small constant η > 0 that only depends on K,P,µ , then it immediately implies the desired result by Corollary E.3 with Υ = k 1 + k 2 . The goal of Appendix E.4 is to show that this holds after k 1 = P steps of the first layer weight training. We now return to the deferred proofs of helper lemmas.

E.3.1 Proof of helper lemmas

Proof of Lemma E.4. Note that Eq. (74) simply follows from the definition of infimum and the excess risk functional R ( · ) . To show that f ∗ ( x ) = h ∗ ( z ) , we make an observation that for any function f : {± 1 } d → R , one can define h ∗ : {± 1 } P → R that only depend on the support and achieve risk no worse than f ∗ . Define h ∗ ( z ) = E z c [ f ∗ ( x )] , where z c = x \ z (the coordinates outside the support). In other words, h ∗ is the boolean function after ignoring the monominals that do not depend on z . Then

where the only inequality follows from Jensen's inequality and convexity of ℓ ( · , y ) .

Proof of Lemma E.5. Define g ∗ ( z ) = h ∗ ( z ) -¯ c. Our goal is to show the existence of ¯ ρ ∗ such that g ∗ ( z ) = E ( a ∗ , u ) ∼ ¯ ρ ∗ [ a ∗ σ ( ⟨ u , z ⟩ )] .

Consider the following objective least square objective for the domain

While this is an infinite dimensional problem, the Representer's theorem holds and the interpolating solution exists. We refer the reader to [Celentano et al., 2021] for a detailed analysis of interpolation with the random feature model. Moreover,

k

for some constant c 1 that only depends on λ min ( K 1 ) , h ∗ , P and ¯ c .

Proof of Lemma E.6. For t ∈ [0 , 1] and any ¯ ρ (1) and ¯ ρ (2) , consider the density ¯ ρ = t ¯ ρ (1) +(1 -t )¯ ρ (2) . Then

To show smoothness, first observe that a direct consequence of Assumption D.4 (the condition | ℓ ′′ ( u, y ) | ≤ K ) is that for any u 1 , u 2 ∈ R , we have

Using this, for any ¯ ρ (1) and ¯ ρ (2)

Since ˆ f NN ( z ; ¯ ρ (1) ) -ˆ f NN ( z ; ¯ ρ (2) ) = E a (1) ,a (2) , u [ ( a (1) -a (2) ) σ ( ⟨ u , z ⟩ ) ] , further simplifying the above

Also, simplifying the third term of (80)

Finally, combining (81) and (82) with (80), we obtain

Proof of Lemma E.7. First, the classical descent lemma holds due to H -smoothness for step-size η ≤ 1 H .

By convexity, for any ˜ x (not necessarily a minimizer), we also have

Substituting this in (83)

Therefore, summing over j ∈ { 0 , . . . , k -1 } , since the right hand side is a telescopic sum

Note that the lemma statements trivially holds if f ( x k ) -f (˜ x ) ≤ 0 already. Therefore, in the case when f ( x k ) > f (˜ x ) and using the descent lemma, we finally conclude the proof.

E.4 Phase 1 (non-linear training)

We now analyze Phase 1 of the algorithm and show that λ min ( K k 1 ) ≥ c , for a constant c := c ( η, µ, P, K ) where η is also a small constant (in terms of µ, P, K ).

Writing the weight evolution with a polynomial. We will by considering a general polynomial activation σ ( x ) = ∑ L l =0 m l x l of degree L , whose coefficients are given by m = ( m 0 , . . . , m L ) .

Lemma E.9. (Training dynamics given by a polynomial) Let ξ = ( ξ S,k ) S ⊆ [ P ] , 0 ≤ k ≤ k 1 -1 ∈ R 2 P k 1 , ζ ∈ R , ρ ∈ R L +1 , γ ∈ R P be variables. For each i ∈ [ P ] , define p 0 ,i ( ζ, ξ , ρ , γ ) ≡ 0 . For each, 0 ≤ k ≤ k 1 -1 ,

Then there is a constant c := c ( k 1 , P, K ) such that for any 0 < η < c

and β = ( β S,k ) S ⊆ [ P ] , 0 ≤ k ≤ k 1 -1 is given by β S,k = -E [ ℓ ′ ( ˆ f NN ( z ; ¯ ρ k ) , y ) χ S ( z )] .

The term ˆ f NN ( · ; ¯ ρ k ) evolves non-linearly, and difficult to analyze directly. However, for sufficiently small step size η , the interaction term ˆ f NN ( · ; ¯ ρ k ) is small, and we show that it can be ignored. Formally, we define the simplified dynamics ˆ u k ( a ) for each 0 ≤ k ≤ k 1 by letting ˆ u 0 ( a ) = 0 and inductively setting for each k ∈ { 0 , . . . , k 1 -1 } ,

Using a similar argument, we may show:

Lemma E.10 (Simplified training dynamics are given by a polynomial) . There is a constant c > 0 depending only on k 1 , P, K , such that for any 0 < η < c , any i ∈ [ P ] and any 0 ≤ k ≤ k 1 , we have

where we abuse notation (since α = ( α S ) S ⊆ [ P ] otherwise) and let α = ( α S,k ) S ⊆ [ P ] , 0 ≤ k ≤ k 1 -1 be given by

Reducing to analyzing simplified dynamics We lower-bound λ min ( K k 1 ) in terms of the determinant of a certain random matrix. Let ζ = [ ζ 1 , . . . , ζ 2 P ] be a vector of 2 P variables. Define M = M ( ζ , ξ , ρ , γ ) ∈ R 2 P × 2 P to be the matrix indexed by z ∈ { +1 , -1 } P and j ∈ [2 P ] with entries

This matrix is motivated by the following fact:

Lemma E.11. There is a constant c > 0 depending only on k 1 , P, K , such that for any 0 < η < c , and any a = [ a 1 , . . . , a 2 P ] ∈ [ -1 , 1] 2 P , we have

Under this notation, using the exactly same analysis as in [Abbe et al., 2022, Lemma E.9], we can also show

Lemma E.12. There is a constant c > 0 depending on K,P such that for any 0 < η < c ,

On the other hand, we can prove a lower-bound on E [det( M ( η a , β , m , κ )) 2 ] simply by lowerbounding the sum of magnitudes of coefficients of det( M ( ζ , α , m , κ )) when viewed as a polynomial in ζ almost surely over the choice of ¯ c . This is because of (a) the fact that det( M ( ζ , α , m , κ )) and det( M ( ζ , β , m , κ )) have coefficients in ζ that are O ( η ) -close for η small, and (b) the fact that polynomials anti-concentrate over random inputs.

Analysis of simplified dynamics. The proof is now reduced to showing that det( M ( ζ , α , m , κ )) ̸≡ 0 , as a polynomial in ζ . In other words, by Lemma E.11, we are now focusing on only the simplified dynamics ˆ u k . If we can show that det( M ( ζ , α , m , κ )) ̸≡ 0 almost surely over the choice of κ ∈ [1 / 2 , 3 / 2] P , then Theorem E.1 follows. Therefore, introducing the variables γ ∈ R P , it suffices to show that

We start by following simple observation.

Claim 1. Almost surely over the choice of initialization ¯ c , the set of subsets K := { S : α S = 0 } has Leap ( K ) = 1 .

̸

̸

Proof. This basically follows from the fact that ℓ is piecewise analytic. The argument is as follows: we have that Leap DLQ ℓ = Leap ( C DLQ ℓ ) = 1 , where C DLQ ℓ = { S : ∃ u ∈ R such that E [ ℓ ′ ( u, y ) χ S ( z )] = 0 } . Since ℓ is piece-wise analytic, so is ℓ ′ ( u, y ) in the first argument. Thus, for any S ∈ C DLQ ℓ , the set { u : E [ ℓ ′ ( u, y ) χ S ( z )] = 0 } has measure 0. Thus, almost surely over the choice of ¯ c = ˆ f NN ( z ; ¯ ρ 0 ) , we have that for any S ∈ C DLQ ℓ

̸

This implies that S ∈ K as well. We finally conclude that Leap DLQ ℓ = Leap ( K ) = 1 .

Using ideas similar to Abbe et al. [2022], we now prove (84) by analyzing the recurrence relations for p k,i to show that to first-order the polynomials p k 1 ,i are distinct for all i ∈ [ P ] , but now doing smooth analysis over γ instead. Formally we show the following lemma.

Lemma E.13. Suppose that L ≥ 2 8 P and let m i = i ! ( L i ) for all 0 ≤ i ≤ L , which correspond to the activation function σ ( x ) = (1 + x ) L . Let k 1 = P . Then det( M ( ζ , α , m , γ )) ̸≡ 0 , i.e. (84) holds.

Claim 2. For any time step 0 ≤ k ≤ k 1 any j ∈ [ N ] , a learning rate η < 1 / (4 K 2 (1 + K ) Pk ) , and any a ∈ [ -1 , 1] we have

Proof. We prove this by induction on k . For k = 0 , we have u 0 = ˆ u 0 = 0 and the claim holds trivially. For the inductive step, ˆ f NN ( z ; ¯ ρ k ) ≤ E a ∼ µ a [ | a || σ ( ⟨ u k , z ⟩ ) | ] ≤ ∥ σ ∥ ∞ ≤ K , since a ∼ Unif([ -1 , 1]) . Therefore

Note that here we used ∥ κ ∥ ∞ ≤ 2 , | ℓ ′ ( ˆ f NN ( z ; ¯ ρ k ) , y ) | ≤ K (1 + | ˆ f NN ( z ; ¯ ρ k ) | ) ≤ K (1 + K ) using A 2 , and | aσ ′ ( ⟨ u k , z ⟩ ) | ≤ | a | · | σ ′ ( ⟨ u k , z ⟩ ) | ≤ K . The bound for ∥ ˆ u k ( a ) ∥ 1 follows exactly the same argument but now we use | ℓ ′ ( ˆ f NN ( z ; ¯ ρ 0 ) , y ) | ≤ K .

Proof of Lemmas E.9 and E.10. We have ζ ∈ R , ξ = ( ξ S,k ) S ⊆ [ P ] ,k ∈{ 0 ,...,k 1 -1 } , ρ ∈ R L +1 , and γ ∈ R P as variables. Let s 0 , . . . , s k 1 -1 : { +1 , -1 } P → R be s k ( z ) = ∑ S ⊆ [ P ] ξ S,k χ S ( z ) . Consider the recurrence relation ν k ∈ R P , where we initialize ν 0 = 0 and, for 0 ≤ k ≤ k 1 -1 ,

This recurrence is satisfied by ζ = ηa , ρ = m , γ = κ , and s k ( z ) = ∑ S β S,k χ S ( z ) . This is because

Finally, noting that E y | z [ -ℓ ′ ( ˆ f NN ( z ; ¯ ρ k ) , y )] = ∑ S β S,k χ S ( z ) , we showed that recurrence (85) holds with specified value of variables. Similarly, ˆ u k ( a ) with s k ( z ) = ∑ S α S χ S ( z ) for α S = -E [ ℓ ′ ( ˆ f NN ( z ; ¯ ρ 0 , y ) χ S ( z )] . This is because |⟨ u k , z ⟩| , |⟨ ˆ u k , z ⟩| ≤ 1 / 2 < 1 by Claim 2 and in the interval ( -1 , 1) , we have σ ( x ) = ∑ L r =0 m r r ! x r . It finally remains to show that

The proof is by induction on k . For k = 0 , it is true that p 0 ,i ( ζ, ξ , ρ , γ ) = 0 = ν 0 i . For the inductive step, for any r ≥ 1 and i ∈ [ d ] , we can write

and γ i E z [ s k ( z ) ⟨ ν k , z ⟩ 0 z i ] = γ i E z [ s k ( z ) z i ] = γ i ξ { i } ,k . The inductive step follows by linearity of expectation.

E.5 Proof of Lemma E.13

In Lemma E.13, we have already fixed m ∈ R L +1 to be m i = i ! ( L i ) for all i ∈ { 0 , . . . , L } . This corresponds to the activation function σ ( x ) = (1 + x ) L .

E.5.1 Reducing to minimal Leap 1 structure

To show that det( M ( ζ , α , m , γ )) ̸≡ 0 as a polynomial in ζ , γ , we first show that it suffices to consider 'minimal' Leap 1 set structure, without loss of generality.

we have

Proof. Directly substituting α S = 0 for all S ∈ K \ K ′ .

Therefore, without loss of generality (up to relabelling the indices of the variables), we assume that

Otherwise, we could remove a set from K while still having Leap ( K ) = 1 .

E.5.2 Weights to leading order

Let us define the polynomials q k,i in variables ζ , ϕ , ρ , γ where ϕ = ( ϕ S ) S ∈K . For all k ∈ { 0 , . . . , k 1 -1 } and i ∈ [ P ] ,

Therefore M ( ζ , ϕ , ρ , γ ) has entries M z ,j ( ζ , ϕ , ρ , γ ) = ∑ L r =0 ρ r r ! ( ∑ P i =1 q k,i ( ζ j , ϕ , ρ , γ ) ) r . We will explicitly compute the lowest degree term in ζ . First, we show that many terms are zero. To this end, consider the following notation: Recursively define

where the sum over an empty set is 0 by convention. Additionally, let ˜ q k,i ( ϕ, ρ , γ ) to be the coefficient of the term in q k,i ( ζ, ϕ , ρ , γ ) whose degree in ζ is o i . Furthermore, recursively define

Lemma E.14. Under the above notation, we have that q k,i ( ζ, ϕ , m , γ ) has no nonzero terms of degree less than o i in ζ . Furthermore, ˜ q k,i ( ϕ , ρ , γ ) can be decomposed as ˜ q k,i ( ϕ , ρ , γ ) = g i ( γ ) · ˆ q k,i ( ϕ , ρ ) for some ˆ q k,i ( ϕ , ρ ) .

Proof. We will prove this by induction on k . The base case for k = 0 trivially holds because q 0 ,i ≡ 0 . For the inductive step, we assume the statement holds for all k ′ ∈ { 0 , . . . , k } and we prove the statement for k +1 . By the recurrence relation of the polynomials q k,i , we have

It suffices to show that the claim holds for each one of the three terms above.

The first term, q k,i ( ζ, ϕ , ρ , γ ) , is handled by the inductive hypothesis. We can conclude that all the terms of degree less than o i in ζ are zero. Moreover, ˜ q k,i = g i ( γ )ˆ q k,i ( ρ , γ ) .

The second term is nonzero only in the case that { i } ∈ K , in which case S i = { i } ∈ K and o i = 1 , so we do not have a contradiction since the ζ o i = ζ only and we do not have terms with degree less than o i . Moreover, the coefficient of ζ is γ i ρ 1 ϕ { i } = g i ( γ ) ρ 1 ϕ { i } .

̸

We break into cases. Case a . If { i } ⊕ { i 1 } ⊕ · · · ⊕ { i r } = S i , then S i \ { i } ⊂ { i 1 , . . . , i r } , so 1 + ∑ r l =1 o i l ≥ o i , and so no new terms of degree less than o i are added. Moreover, the only way the third term has degree o i in ζ is when i = i 1 = . . . = i r while still having { i, i 1 , . . . , i r } = S i ; otherwise the power of ζ is 1 + ∑ r l =1 o i l > o i . In that case, letting r ′ = | S i | -1 , the coefficient of ζ o i in the third term is

̸

̸

̸

Case b . If { i } ⊕ { i 1 } ⊕ · · · ⊕ { i r } = S i ′ for some i ′ = i , then either i ∈ { i 1 , . . . , i r } , in which case 1 + ∑ r l =1 o i l > o i . Otherwise, we must have i ′ > i . But in this case o i ′ > o i since i ∈ S i ′ , so we also have ∑ r l =1 o i l > o i and again no new terms of degree less than o i are added. In fact, only terms of degree strictly more than o i are added. Thus, in either case the coefficient of the term with degree o i in ζ is simply 0.

Finally, we showed that each of the three terms have at least the degree o i in ζ . Moreover, each term can be decomposed as g i ( γ )ˆ q ( ρ , γ ) for a certain ˆ q . Thus, overall we can decompose

˜ q k,i ( ϕ , ρ , γ ) = g i ( γ ) · ˆ q k +1 ,i ( ϕ , ρ ) , concluding the proof of the lemma.

Claim 4. Let

Then, for each distinct pair z , z ′ ∈ { +1 , -1 } P , we have r z ( ζ, ϕ , m ) -r z ′ ( ζ, ϕ , m ) ̸≡ 0 as a polynomial in ζ and γ .

̸

Proof. Recall the definition of o i from Lemma E.14. Fix i ∈ [ P ] be such that z i = z ′ i and o i is minimized. It is always possible to choose such an i since z = z ′ . Again ˜ r z ( ϕ , ρ , γ ) is the term with degree o i in ζ . Using Lemma E.14 then

̸

̸

but ˜ q k 1 ,i ′ are non-zero polynomials in γ with different dependence that is captured by g i ′ ( γ ) according to Lemma E.14. Thus, r z ( ζ, ϕ , m ) -r z ′ ( ζ, ϕ , m ) ̸≡ 0 .

E.5.3 Linear Independence of Powers of Polynomials

Similar to Abbe et al. [2022], we finish our proof using the classical result of Newman and Slater [1979] about the linear independence of large powers of polynomials.

̸

Proposition E.15 (Remark 5.2 in Newman and Slater [1979]) . Let R 1 , . . . , R m ∈ C [ ζ ] be nonconstant polynomials such that for all i = i ′ ∈ [ m ] we have R i ( ζ ) is not a constant multiple of R i ′ ( ζ ) . Then for L ≥ 8 m 2 we have that ( R 1 ) L , . . . , ( R m ) L ∈ C [ ζ ] are C -linearly independent.

We will finally show that det( M ( ζ , α , m , γ )) ̸≡ 0 .

̸

Proof of Lemma E.13. Let us fix κ ∈ R P such that for all z = z ′ , we have r z ( ζ, α , m , κ ) -r z ′ ( ζ, α , m , κ ) ̸≡ 0 as polynomials in ζ . This can be ensured by drawing κ ∼ Unif([0 . 5 , 1 . 5] P ) , since for all z = z ′ we have r z ( ζ, α , m , γ ) -r z ′ ( ζ, α , m , γ ) ̸≡ 0 as polynomials in ζ, γ by Claim 4. We now rewrite ˜ r z ( ζ ) = r z ( ζ, α , m , κ ) to further indicate that the variables ϕ = α , ρ = m and γ = κ are instantiated, and that we are looking at a polynomial over ζ . We have also chosen m i = i ! ( L i ) for all i ∈ { 0 , . . . , L } . We then have

̸

̸

Also, observe from the recurrence relations ζ divides q k 1 ,i ( ζ, α , m , κ ) for each i ∈ [ P ] , so ζ j divides ˜ r z ( ζ j ) = ∑ P i =1 z i q k 1 ,i ( ζ j , α , m , κ ) . Therefore, polynomials (1 + ˜ r z ( ζ j )) and , (1 + ˜ r z ′ ( ζ j )) are not constant multiples of each other for any z = z ′ .

Now see ζ as variables, and construct the Wronskian matrix over the L -th power polynomials { (1 + ˜ r z ( ζ )) L } z ∈{ +1 , -1 } P . This is a 2 P × 2 P matrix H ( ζ ) whose entries are indexed by z and l ∈ [2 P ] and defined by:

Applying Proposition E.15 implies the polynomials { (1 + ˜ r z ( ζ )) L } z ∈{ +1 , -1 } P are linearlyindependent, and so the Wronskian determinant is nonzero as a polynomial in ζ :

Also, observe that we can write det( H ( ζ )) = ∂ ∂ζ 2 ∂ 2 ∂ζ 2 3 . . . ∂ 2 P -1 ∂ζ 2 P -1 2 P det( M ( ζ , α , m , κ )) | ζ = ζ 1 = ··· = ζ 2 P . This finally gives us det( M ( ζ , α , m , κ )) ̸≡ 0 as a polynomial in ζ , and thus, det( M ( ζ , α , m , γ )) ̸≡ 0 as a polynomial in ζ and γ .

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, it does.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We do specify the limitations of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, we do provide all the assumptions and correct proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We do provide the code and the experiments are fully reproducible.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We attach the code in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify our training algorithm and related details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We do report the error bars whenever necessary.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments are relatively small scale and can be reproduced under 1 hr of compute on a standard computer with 16 GB RAM.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: It does.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work with no societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not have models or data with safety concerns.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use any existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new asset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This does not require crowd sourcing nor research with human subject.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our study does not require IRB approval.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning juntas from the perspective of restricted statistical query algorithms. Concretely, for some d-dimensional product distribution \\mu_x^d and some conditional distribution \\mu_{y|z}, the learner is given access (via samples or, in this paper, certain queries) to the distribution of (y,x) where x ~ \\mu_x^d and y|x has distribution \\mu_{y|z} where z is some tuple of P unknown coordinates of x. That set of coordinates that influence y is called the support of the distribution, and the goal is to learn the support.\n\nA Q-restricted statistical query algorithm is one that only makes queries to E[phi(y,x)] for functions phi in Q. Examples include SQ and CSQ; this paper introduces, for any fixed loss function \\ell, another class DLQ_\\ell, where the allowable queries are gradients of \\ell(f(x,w),y) (w.r.t. w) for arbitrary functions f.\n\nThe main result of this paper is a characterization for any {SQ, CSQ, DLQ} of the query complexity of the support learning problem, when P, \\mu_x, \\mu_{y|z} are fixed but d is growing: up to lower-order terms it has the form d^k for a \"leap exponent\" k that can be determined from \\mu_x,\\mu_{y|z}, and the query class Q. As a consequence, the paper gives separations and equivalences among these classes: notably, CSQ is equivalent to DLQ for \\ell_2 loss (Proposition 6.2a), but SQ is equivalent to DLQ for \\ell_1 loss (Proposition 6.1a). \n\nAdditionally, the paper has some results showing that if one performs SGD on a two-layer neural network in the mean-field regime and with loss function \\ell, then DLQ_\\ell (rather than SQ or CSQ) is what generically characterizes success of the dynamics (more precisely, Leap(DLQ_\\ell) = 1 is what characterizes success with O(d) samples).\n\nSTRENGTHS:\n- This paper extends prior work by Abbe, Boix-Adsera, and Misiakiewicz to a more general setting (specifically, the prior work considered the special case of CSQ and the uniform distribution on the hypercube). While CSQ has long been motivated by the fact that it captures gradient descent, a key takeaway of this paper is that this is only for \\ell_2 loss, and for other losses a different characterization is needed. This seems like a useful observation for the community.\n- The paper also contrasts the power of adaptive queries versus non-adaptive queries, and gives a characterization of query complexity for both.\n- The paper is written carefully and rigorously.\n\nWEAKNESSES:\n- I spent a while trying to parse the notation for the proof of Theorem 5.1(a). Ultimately I think it is not super complicated, but some intuitive, high-level explanation of the construction would have been very helpful in addition to the formal definitions. Also, the proof outline explains the boilerplate elements that appear in any SQ lower bound argument but gives no intuition/proof sketch for the crucial and presumably novel inequality (25).\n\nQUESTIONS:\n- Does the characterization extend beyond just {SQ, CSQ, DLQ} to any query class Q, or if not what breaks down?\n- The equivalence between SQ and DLQ for ell_1 loss rests on a universal approximation result. I am curious if there is intuition for why the analogous approximation result for ell_2 does not hold? Is it just that there is no non-linearity in the gradient? If this is the intuition, it seems to me that in some sense \"most\" loss functions besides ell_2 ought to induce an equivalence between DLQ_ell and SQ? Proposition 6.2(c) is interesting because it gives a loss function that does not degenerate to SQ nor CSQ: is there no universal approximation here because it's a polynomial?\n- I assume that CSQ and SQ provably *do not* capture SGD on neural networks, but I didn't see a formal statement to this effect; does this follow from the separations in Section 6?\n- This is a minor technical confusion: in lines 505-506, I see why the null responses are compatible with nu^\\sigma, but since nu_{S^\\star} is not in the hypothesis class (since it's a partially \"decoupled\" distribution), I don't see why this immediately gives failure of the learning algorithm? I would have thought one needs two hypothesis in the hypothesis class to be indistinguishable. Of course I think this should also follow from (26) but I just wanted to check if I am misunderstanding.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies the computational complexity of learning low-dimensional functions (juntas) and examines a new type of statistical query termed DLQ, which captures the difficulty of learning using gradient-based queries. The authors identify quantities that dictate the computational lower- and upper-bounds based on the properties of the link function and show that the DLQ complexity may match either SQ or CSQ depending on the loss function. This generalizes the earlier leap exponent in (Abbe et al. 2023), which is specific to CSQ models. Finally, in a simplified hypercube setting, the authors show that a mean-field neural network can learn functions with DLQ-leap exponent 1 using $n=O(d)$ samples, which matches the DLQ complexity.\n\nSTRENGTHS:\nThis submission makes a number of contributions to the learning of low-dimensional functions on product distributions:\n\n* The proposed DLQ \"interpolates\" between SQ and CSQ depending on the structure of the loss function. This provides insights into the class of low-dimensional functions that can be learned by gradient descent algorithms.\n\n* The authors also distinguish between adaptive and non-adaptive queries, providing a separation between algorithms with one gradient step versus consecutive steps.\n\n* The mean-field analysis extends (Abbe et al. 2022) to different loss functions and confirms that gradient descent training of a two-layer neural network learns juntas with DLQ leap exponent 1.\n\nWEAKNESSES:\nI have the following concerns/questions:\n\n1. The class of functions with the prescribed DLQ leap exponent is rather opaque, even in the simple hypercube setting. For Gaussian single-index models, the SQ-leap exponent is completely characterized in (Damian et al. 2024), and it is known that all polynomial links have SQ exponent 1 or 2. But in the current submission, it is not clear what functions have DLQ leap exponent 1. The authors provided one example in Proposition 6.2(c) for one specific loss and link function, but there is no general principle that determines the DLQ hardness.  \nIntuitively, if the output is discrete, then the detectable set for SQ versus CSQ may not differ too much, so it is not easy to see that the SQ/DLQ-learnable function class is significantly larger on hypercube input. Can the authors clarify?\n\n2. The current submission does not take the rotational symmetry into account, and the query dependence in the lower bound is polynomial as opposed to logarithmic in the Gaussian case (Damian et al. 2022). For algorithms that are not axis-aware, such as SGD from rotationally invariant initialization, do we expect a lower bound that is stronger in terms of query dependence?\n\n3. Assumption 2.1 needs more explanation beyond \"*it always holds when $\\mathcal{X}$ is discrete*\". Can the authors comment on how to translate this assumption to a condition on the link function when the input is continuous, such as Gaussian? \n\n4. How do the authors relate the number of queries in the SQ lower bound to the sample size in the mean-field analysis?\n\n---\nThe authors’ response addressed most of my concerns. I have updated my evaluation accordingly.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the complexity of learning sparse functions (i.e., juntas) using statistical and gradient queries. For that, it introduces Differentiable Learning Queries (DLQ) that can be seen as a generalization of correlation statistical queries beyond square loss. \n\nWith the notion, the paper gives a characterization of the query complexity for learning the support of a sparse function over generic product distributions. The characterization is in terms of leap exponent introduced from Abbe et al earlier. \n\nFinally, the paper also introduces cover exponent that characterizes non-adaptive query complexity.\n\nSTRENGTHS:\nThe paper studies a well-motivated question of query complexity of learning juntas, a long established question in ML theory. The paper builds upon the prior work of Abbe et al and provides a generalization beyond CSQ and square loss function. I find the results strong at a technical level. To my knowledge, the main theorems are novel. \n\nThe work proves general relations between SQ, CSQ and DLQ.\n\nThe paper also provides experimental validation of the theoretical results, though it's somewhat limited.\n\nThe paper is well written.\n\nWEAKNESSES:\nFigure 1 is an experimental study. I believe it'd be nice to expand this into a small section that extends beyond learning y_2 (2) and using the 2 loss functions here. I also can't find more details: what is the architecture of this two-layer net? \n\nAbbe et al [2023] has some nicer numerical results. I'd recommend check it out.\n\nCompared with Abbe et al 2023, the paper only tackles the case when the target function is junta. It doesn't extend much beyond sparse functions. \n\nMinor comments follow:\n* Line 18: remove “that is”\n* Line 24: it should be clarified that Abbe et al. [2022, 2023] studies learning juntas using neural networks via SGD\n\nQUESTIONS:\nAny upper bound results or conjectures on how to match the lower bound in section 5?\n\nLIMITATIONS:\nThe paper lists several limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}