Title: Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputs
Venue: NeurIPS 2023
Accepted: yes
Keywords: curriculum learning

parities

time complexity

sample complexity

neural networks

SGD
Abstract: Experimental results have shown that curriculum learning, i.e., presenting simpler examples before more complex ones, can improve the efficiency of learning. Some recent theoretical results also showed that changing the sampling distribution can help neural networks learn parities, with formal results only for large learning rates and one-step arguments. Here we show a separation result in the number of training steps with standard (bounded) learning rates on a common sample distribution: if the data distribution is a mixture of sparse and dense inputs, there exists a regime in which a 2-layer ReLU neural network trained by a  curriculum noisy-GD (or SGD) algorithm that uses sparse examples first, can learn parities of sufficiently large degree, while any fully connected neural network of possibly larger width or depth trained by noisy-GD on the unordered samples cannot learn without additional steps. We also provide experimental results supporting the qualitative separation beyond the specific regime of the theoretical results.

Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputs

Abstract

Experimental results have shown that curriculum learning, i.e., presenting simpler examples before more complex ones, can improve the efficiency of learning. Some recent theoretical results also showed that changing the sampling distribution can help neural networks learn parities, with formal results only for large learning rates and one-step arguments. Here we show a separation result in the number of training steps with standard (bounded) learning rates on a common sample distribution: if the data distribution is a mixture of sparse and dense inputs, there exists a regime in which a 2-layer ReLU neural network trained by a curriculum noisy-GD (or SGD) algorithm that uses sparse examples first, can learn parities of sufficiently large degree, while any fully connected neural network of possibly larger width or depth trained by noisy-GD on the unordered samples cannot learn without additional steps. We also provide experimental results supporting the qualitative separation beyond the specific regime of the theoretical results.

1 Introduction

Starting with easy concepts is an effective way to facilitate learning for humans. When presented with simpler ideas or tasks, individuals can develop a foundation of knowledge and skills upon which more complex concepts can be built. This approach allows learners to gradually gain competence and accelerate learning [EA84, RK90, AKB + 97, SGG14].

For machine learning algorithms, this is typically referred to as Curriculum Learning (CL) [BLCW09]. Several empirical studies have shown that presenting samples in a meaningful order can improve both the speed of training and the performance achieved at convergence, compared to the standard approach of presenting samples in random order [WCZ21, SIRS22, GBM + 17, HW19, PSL15].

The theoretical study of CL is mostly in its infancy. While some works have analytically shown benefits of CL for certain targets [SMS22, WCA18, MKAS21, CM23] (see Section 1.1 for a discussion on these), it remained open to provide a separation result between curriculum training and standard training on a common sampling distribution. Specifically, one seeks to demonstrate how a standard neural network, trained using a gradient descent algorithm on a given dataset and horizon, can successfully learn the target when certain samples are presented first during training, whereas learning does not occur if samples are presented in a random order.

This paper provides such an instance. We focus on the case of parities, as a classical case of challenging target for differentiable learning. In the experimental part of the paper, we report results suggesting that a similar picture can hold for other large leap functions as defined in [ABAM23], although more investigations are required to understand when and how curriculum learning should be developed for more general functions.

We assume that the network is presented with samples ( x, χ S ( x )) , with x ∈ {± 1 } d and χ S ( x ) := ∏ i ∈ S x i , for some set S ⊆ { 1 , . . . , d } of fixed size. We consider datasets where a small fraction ρ of

Aryo Lotfi EPFL

the inputs are sparse , meaning they have, on average, few negative coordinates and mostly positive coordinates, and the remaining inputs are dense , with an average of half negative and half positive coordinates. We refer to Section 2 for the definition of the input distribution. Parities seem to be promising candidates for theoretical investigations into CL, as they are hard to learn in the statistical query learning model [Kea98], or on dense data by any neural network trained with gradient-based methods having limited gradient precision [AS20], but they are efficiently learnable on sparse data by regular architectures [MKAS21] (see Section 1.1 for further discussion).

We say that a neural network is trained with curriculum if it goes through an initial phase where it is exposed to batches consisting only of sparse samples (see discussion below on the interpretation of these as 'simpler' examples). Following this phase, the network is trained using batches sampled from the entire dataset. We compare this curriculum training approach with standard training, where the network receives batches of samples selected randomly from the entire dataset at each step. Our intuition is as follows. If the fraction of sparse samples is small, without curriculum, the network is exposed to batches of mostly uniform inputs, and the contribution of sparse inputs in estimating the gradients will be limited. As a result, learning parities on the mixed distribution without curriculum is roughly as hard as learning parities on dense inputs (Theorem 2). On the contrary, if the network is initially exposed to batches consisting of sparse inputs, it can efficiently identify the support of the parity, and subsequent fitting using batches sampled from the entire dataset enables learning the target with fewer training steps compared to standard training (Theorem 1).

In particular, we show that if the fraction of sparse samples ρ is small enough, (e.g., ρ < d -4 . 5 , where d is the input dimension), a 2 -layer fully connected network of size θ ( d ) trained by layerwise SGD (or noisy-SGD, see Def. 1) with curriculum can learn any target k -parity, with k bounded, in T = θ ( d ) steps, while the same network (or any network of similar size) trained by noisy-SGD without curriculum cannot learn the target in less than Ω( d 2 ) steps. We refer to Section 2 for a more detailed explanation of our results. While our theoretical results use simplifying assumptions (e.g., layerwise training), our experiments show separations between curriculum and standard training in a broader range of settings.

1.1 Related Literature

Curriculum Learning (CL) was first introduced in the machine learning context by the seminal work of [BLCW09]. This paper along with many other subsequent works presented empirical evidence that training neural networks with a curriculum can be beneficial in both the training speed and the performance achieved at convergence in certain instances in different domains such as computer vision [JMMH14, GHZ + 18, CG15, STD19], natural language processing (NLP) [KB17, GBM + 17, ZKK + 18, PSN + 19], and reinforcement learning [FHW + 17, NPL + 20]. We refer to relevant surveys [WCZ21, SIRS22] for a more comprehensive list of applications of CL in different domains. In general, defining a complexity measure on samples and consequently an ordering of training inputs is not trivial. Some works determine the easiness of samples based on predefined rules (e.g., shape as used in [BLCW09]), while others also use the model's state and performance to evaluate the samples dynamically as in self-paced learning (SPL) [KPK10]. In this work, we follow the former group as we define the hardness of samples based on the number of -1 bits, i.e., the Hamming weight. Note that in parity targets, +1 is the identity element. Thus, one can view the number of -1 bits as the length of the sample. In this sense, our work resembles [SAJ10, ZS14, KB17, PSN + 19] which use the length of the text inputs as a part of their curriculum.

Despite the considerable empirical work on CL, there appears to be a scarcity of theoretical analysis. Few theoretical works focused on parity targets, as in this paper. While parities are efficiently learnable by specialized algorithms that have access to at least d samples (e.g. Gaussian elimination over the field of two elements), they are hard to learn under the uniform distribution (i.e. on dense data) by any neural network trained with gradient-based methods with limited gradient precision [AS20, AKM + 21]. Parities on sparse data can be learned more efficiently on an active query model, which can be emulated by a neural network [AS20]. However, such emulation networks are far from the practical and homogeneous networks, and distribution class-dependent. A natural question that has been considered is whether more standard architectures can also benefit from a simpler data distribution. In [MKAS21], the authors make a first step by showing that parities on sparse inputs are efficiently learnable on a 1-layer network with a parity module appended to it, with a one-step argument. In [DM20], it is shown that for a two-layer fully connected network, sparse inputs help but

with a variant of the distribution that allows for leaky labels. [ABLR23] obtains experimental results for a curriculum algorithm presenting sparse samples first with heuristics for regular networks and hyperparameters, but without proofs. In [CM23], the authors prove that a two-layer fully connected net can benefit from sparse inputs, with a one-step gradient argument requiring a large learning rate. This paper improves on [CM23] not only in terms of having more natural training settings and hyperparameters (i.e., we use SGD with bounded batch size and bounded learning rate), it is also establishing a separation between learning with and without curriculum on a common sampling distribution, while [CM23] does not (in fact in their work curriculum involves both sparse and dense inputs while standard training involves dense inputs only). To the best of our knowledge, our paper is the first work that establishes a rigorous separation between training on adequately ordered samples and randomly ordered samples drawn from the same distribution, and our theorem gives a condition on the mixture under which this separation holds (interestingly the separation does not take place for all mixtures parameters). Beyond parities, [SMS22] studies a teacher-student model, where the target depends on a sparse set of features and where the variance on the irrelevant features. Furthermore, [WCA18, WA20] show that on some convex models, CL provides an improvement in the speed of convergence of SGD. In contrast, our work covers an intrinsically non-convex problem.

2 Setting and Informal Contributions

We consider the problem of learning a target function f : {± 1 } d →{± 1 } that belongs to the class of k -parities on d bits, i.e.,

where we denoted by [ d ] := { 1 , . . . , d } . We assume that the network has access to a dataset ( X,Y ) = { ( x s , y s ) } s ∈ [ m ] where the inputs x s are sampled i.i.d. from a mixed distribution defined as follows:

where D µ = Rad( µ +1 2 ) ⊗ d 1 , for some µ ∈ [ -1 , 1] , D u = Unif {± 1 } d 2 and ρ ∈ [0 , 1] is some fixed parameter, and y s = f ( x s ) . We will assume µ ∈ (0 , 1) . In other words, we assume that the dataset contains approximately ρm samples with in average 1 -µ 2 d negative bits (sparse samples), and (1 -ρ ) m samples with in average half negative bits (dense samples). We consider training the neural network with stochastic gradient descent, with or without gradient noise (SGD or noisy-SGD, see Def. 1).

Informal Description of the Curriculum and Standard Training Algorithms. We compare two training strategies: standard training and curriculum training. In the standard training, at each step, mini-batches are drawn from the whole dataset. For curriculum training, we first need to isolate the set of sparse inputs in our dataset. Let us denote this set by X 1 := { x s ∈ X : H ( x s ) < ( 1 2 -µ 4 ) d } , where H ( x ) := ∑ j ∈ [ d ] 1 ( x j = -1) denotes the Hamming weight of x , which is the count of negative bits in x , and by Y 1 = { y s ∈ Y : x s ∈ X 1 } the corresponding set of labels. Our curriculum strategy consists of training on only sparse samples for the first T 1 steps, where T 1 ≤ T is some well-chosen time horizon and T is the total number of training steps. After this initial phase, the network is trained on samples belonging to the whole dataset. To sum up, we compare the following two training strategies:

(standard training): At all t ∈ [ T ] , mini-batches are sampled from ( X,Y ) ;

(curriculum training): At t ≤ T 1 , mini-batches are sampled from ( X 1 , Y 1 ) and at T 1 < t ≤ T mini-batches are sampled from ( X,Y ) .

Informal Description of the Positive and Negative Results. In this paper, we prove that if the fraction of sparse samples ρ is small enough, there is a separation in the number of training steps needed to learn the target parity up to a given accuracy. We say that an algorithm on a neural network

learns a target function f : {± 1 } d →{± 1 } up to accuracy 1 -ϵ , for ϵ ≥ 0 , in T steps, if it outputs a network NN( x ; θ T ) such that:

where sgn( . ) denotes the sign function. In our main positive result, we consider training with the covariance loss (Def. 2) and we refer to Remark 2 for considerations on other losses. Let us state our main positive result informally.

Theorem 1 (Theorem 3, Informal) . Consider the input distribution D mix with mixture parameters ρ = ˜ O ( d -1 / 2 ) 3 , µ = θ (1) . Let ϵ > 0 . Then, a 2 -layer ReLU neural network of size O ( d ) initialized with an isotropic distribution, trained with the covariance loss, standard (i.e., non-diverging) learning rate schedule and either (i) a layer-wise curriculum-noisy-SGD algorithm with gradient range A = ˜ O ( √ d ) , noise-level τ = ˜ O ( d -1 ) and sparse samples first, or (ii) a layer-wise curriculum-SGD algorithm with sparse samples first, can learn any k -parities, k = θ (1) , up to accuracy at least 1 -ϵ in T = ˜ O ( d ) /ϵ 2 steps.

We further show an additional positive result valid for a layer-wise curriculum SGD with the hinge loss, which holds for large batch size and diverging learning rate (Theorem 4). On the flip side, we have the following.

Theorem 2 (Theorem 5, Informal) . Consider the input distribution D mix with mixture parameters ρ = ˜ O ( d -3 . 5 -δ ) , δ > 0 , and µ = θ (1) . Then, any neural network of size ˜ O ( d ) (any depth), with any initialization and activation, trained by a noisy-SGD algorithm (layer-wise or joint) with gradient range A = ˜ O ( √ d ) , noise-level τ = ˜ Ω( d -1 ) , batch size B = ˜ Ω( ρ -2 ) , any almost surely differentiable loss function and any learning rate schedule, will need at least T = ˜ Ω( ϵd 1+ δ ) steps to achieve accuracy 1 2 + ϵ for any k -parities ( k ≥ 6 ).

Conclusion of Results. When using noisy-SGD, the curriculum learning algorithm that takes sparse samples first, for the first layer training, and then full samples, can learn in a regime of hyperparameters and training steps (Theorem 1) that is contained in the hyperparameter regime of the negative result (Theorem 2), for which it is shown that the absence of curriculum cannot learn without additional training steps. Further, it is shown in part (ii) of the first theorem, that using SGD rather than noisy-SGD still allows to learn with the curriculum training; this result does however not come with a counter-part for the lower-bound, i.e., we cannot show that SGD without curriculum could not learn such parities, as there is currently no proof technique to obtain rigorous lower-bounds for SGD algorithms (as for the case of noiseless honest-SQ algorithms); see for instance discussions about this in [AKM + 21, ABA22]. However, it is not expected that SGD can do better than noisy-SGD on 'regular' neural networks [ABA22, ABAM22, ABAM23].

3 Positive Results for Curriculum Training

The Algorithm. Let us define the noisy-SGD algorithm, that is used in both our positive and negative results.

Definition 1 (Noisy-SGD) . Consider a neural network NN( . ; θ ) , with initialization of the weights θ 0 , and a dataset ( X,Y ) = { ( x s , y s ) } s ∈ [ m ] . Given an almost surely differentiable loss function L , the updates of the noisy-SGD algorithm with learning rate γ t and gradient range A are defined by

where for all t ∈ { 0 , . . . , T -1 } , Z t are i.i.d. Unif[ -τ, τ ] , for some noise level τ , and they are independent from other variables, B is the batch size, and [ z ] A := argmin | y |≤ A | y -z | , i.e., whenever the gradient exceeds A (resp. -A ) it is rounded to A (resp. -A ).

For brevity, we will write L ( θ t , y, x ) := L (NN( x ; θ t ) , y, x ) . We assume that for all s ∈ [ B ] , ( x s,t , y s,t ) are chosen from ( X t , Y t ) , where ( X t , Y t ) can be either the whole dataset ( X,Y ) , or a subset of it. We include the noise Z t and the gradient range A to cover the noisy-SGD algorithms,

used in SQ lower bounds e.g. in [AS20, AKM + 21, MKAS21, ABA22, ACHM22]. Note that if we set τ = 0 and A large enough, we recover the standard SGD algorithm without noise.

We assume that our dataset is drawn from the mixed distribution defined in (2), with parameters ρ, µ . For the purposes of this section, we assume ρ = ˜ O ( d -1 / 2 ) and µ ∈ (0 , 1) . As a preliminary step, we isolate the sparse inputs from our dataset, by mean of Hamming weight. In particular, we define X 1 := { x s ∈ X : H ( x s ) < ( 1 2 -µ 4 ) d } , where H ( x ) := ∑ j ∈ [ d ] 1 ( x j = -1) denotes the Hamming weight of x , and Y 1 = { y s ∈ Y : x s ∈ X 1 } . Note that, due to the concentration of Hamming weight, for d large enough, there will be approximately mρ samples in X 1 (see Lemma 1). We assume that the training set is large enough, such that each sample is observed at most once during training.

In this section, we consider a 2 -layer neural network NN( x ; θ ) , with θ = ( a, w, b ) , defined by: NN( x ; θ ) := ∑ N i =1 a i σ ( w i x + b i ) , where σ ( x ) = ReLU( x ) := max { x, 0 } . We assume that the number of hidden units is N ≥ k + 1 , where k is the degree of the target parity. We initialize w 0 ij = 0 , a 0 i = κ and b 0 i = ∆ d + 1 , for some κ, ∆ > 0 , for all i, j . We refer to κ and ∆ as the second layer initialization scale and the bias initialization scale, respectively. We adopt a layer-wise curriculum training approach. During the initial phase of training, we exclusively train the first layer of the network using samples taken from ( X 1 , Y 1 ) . Furthermore, we project the weights of the first layer, to guarantee that they stay bounded. In the second phase, we train only the second layer using the entire merged dataset. We refer to Algorithm 1 in Appendix A for the pseudo-code of the algorithm used in our proof. In the context of standard learning without curriculum, similar layer-wise training strategies are considered in e.g., [MSS20, BEG + 22, ABAM23]. [CM23] uses a layer-wise curriculum training, but with a one-step argument that requires a diverging learning rate. In our analysis, we keep the bias weights fixed to specific values during the whole training. We believe that one could extend the result to include scenarios with more general initializations and where the bias weights are trained, but this would require further technical work.

The Covariance Loss. We train NN( x ; θ ) using noisy-SGD with a specific loss function, namely the covariance loss , that we define here. The covariance loss appears in [CM23] with a slightly different definition: in particular, the following definition does not depend on the average estimator output, as opposed to their definition.

Definition 2 (Covariance Loss) . Let ( X,Y ) = { ( x s , y s ) } s ∈ [ m ] be a dataset, where for all s , x s ∈ X and y s ∈ {± 1 } , and let ¯ y = 1 m ∑ s ∈ [ m ] y s . Assume | ¯ y | < 1 . Let ˆ f : X → {± 1 } be an estimator. We define the covariance loss as

where ( a ) + := max { a, 0 } .

The covariance loss can be applied to classification datasets with non-negligible fractions of +1 and -1 labels. In Proposition 1, we show that a small covariance loss implies a small classification error.

̸

Proposition 1. Let ( X,Y ) = { ( x s , y s ) } s ∈ [ m ] be a dataset, where for all s , x s ∈ X and y s ∈ {± 1 } . Let ¯ y = 1 m ∑ s ∈ [ m ] y s and assume | ¯ y | < 1 -δ , for some δ > 0 . Let ˆ f := X → {± 1 } be an estimator. If 1 m ∑ s ∈ [ m ] L cov ( ˆ f, x s , y s ) < ϵ, then 1 m ∑ s ∈ [ m ] 1 (sgn( ˆ f ( x s )) = y s ) < ϵ δ .

The proof of Proposition 1 can be found in Appendix D.

Remark 1 . We remark that for balanced datasets ( ¯ y = 0 ), the covariance loss coincides with the hinge loss. On the other hand, note that the right hand side of (5) can be rewritten as ( (1 -y s ¯ y ) · (1 -y s ˆ f ( x s )) ) + . Thus, in some sense, the samples in the under-represented class (where y s ¯ y ≤ 0 ) are assigned a higher loss compared to those in the over-represented class (where y s ¯ y ≥ 0 ).

Main Theorem. We are now ready to state our main theorem.

Theorem 3 (Main Positive Result) . Let NN( x ; θ ) be a 2 -layer ReLU neural network with N ≥ k +1 hidden units, 2 nd layer init. scale κ > 0 and bias init. scale ∆ > 0 . Let D mix be a mixed distribution, as defined in (2) , with parameters 0 < ρ ≤ ∆ 4 N √ log( d ) √ d , and µ ∈ (0 , 1) . Assume we have access to

a dataset ( X,Y ) = { x s , y s } s ∈ [ m ] , where x s iid ∼ D mix , y s = χ S ( x s ) , with | S | = k . Assume NN is trained according to the layer-wise curriculum-SGD algorithm (Algorithm 1), with the covariance loss, batch size B , noise level τ and gradient range A .

√

For all ϵ > 0 , there exist C, C 1 , C ∗ > 0 such that if κ ≤ 1 /N (2∆ d + 2) , τ ≤ κ , A = ˜ Ω( d ) , m ≥ 4 C ∗ log(3 d ) dρ -1 B , and if

where L µ = µ k -1 -µ k +1 , then, with probability at least 1 -6 Nd -C ∗ , the algorithm outputs a network such that 1 m ∑ s ∈ [ m ] L cov ( θ T 1 + T 2 , x s , y s ) ≤ ϵ.

The following corollary is a direct consequence of Theorem 3 and Proposition 1.

Corollary 1. Let the assumptions of Theorem 3 be satisfied and assume that τ = O ( κ ) and ∆ , k constants. There exists a 2 -layer ReLU network of size at most ( d +2)( k +1) and initialization invariant to permutations of the input neurons, such that for all ϵ > 0 the layer-wise curriculum noisy-SGD algorithm with gradient range A = ˜ O ( √ d ) , after at most T = ˜ O ( d ) steps of training with bounded learning rates, outputs a network such that P x ∼D mix (sgn(NN( x ; θ T )) = f ( x )) ≥ 1 -ϵ, with high probability.

Remark 2 . The choice of the covariance loss is motivated by a simplification of the proof of Theorem 3. Indeed, with this loss, we can show that in the first part of training the weights incident to the relevant coordinates move non-negligibly, while the other weights stay close to zero (see proof outline). If we used another loss, say the hinge loss, all weights would move non-negligibly. However, we believe that one could still adapt the argument to cover the hinge loss case, by adding another projection to the iterates. In the following, we also give a positive result for the hinge loss, with a different proof technique, that holds for large batch size and large learning rate (Theorem 4).

Remark 3 . While we show that with appropriate hyperparameters, the layer-wise curriculum-SGD can learn k -parities in ˜ O ( d ) steps, we need to notice that to implement the curriculum, one has to first retrieve ( X 1 , Y 1 ) . This involves computing the Hamming weight of d/ρ samples, which adds computational cost if the curriculum learning includes this part. Nevertheless, in the experiments that we performed, the recovery of ( X 1 , Y 1 ) was consistently efficient, with an average of less than 1 second running time for a dataset of 10 million samples of dimension 100 . On the other hand, the reduction of training steps due to the curriculum yields significant time savings.

Remark 4 . Theorem 3 provides an upper bound on the sample complexity required to learn using a curriculum, of ˜ O ( dB/ρ ) for SGD with batch size B . Consequently, while the number of iterations needed for learning remains unaffected by ρ , the sample complexity increases as ρ decreases. Additionally, we do not currently have a lower bound on the sample complexity for learning parities with standard training with offline SGD. However, our experiments in Section 5 demonstrate a noticeable difference in the sample complexity required for learning with and without a curriculum. These findings may encourage future work in establishing a theoretical advantage in the number of samples needed to learn with curriculum.

Remark 5 . We further remark that the proof method of Theorem 3 yields a tighter upper bound on curriculum learning's sample complexity than the 1-step argument used in [CM23][Thm 4]. Specifically, using the technique of [CM23][Thm 4], we achieve an upper bound of ˜ O ( d 2 /ρ ) , as opposed to the ˜ O ( d/ρ ) given by Theorem 3 for SGD with bounded batch size.

Remark 6 . T 1 depends on the degree of the parity k through the parameter L µ . We believe that such dependence cannot be removed, however, we remark that choosing µ sufficiently close to 1 (e.g. µ = 1 -1 k ) allows bounding L µ independently from k . Indeed, our experiments (Figure 4) show mild dependence on the parity degree for curriculum learning with large µ .

Proof Outline. The proof of Theorem 3 follows a similar strategy as in [AGJ21, ABAM23, TV19]. We decompose the dynamics into a drift and a martingale contribution, and we show that the drift allows us to recover the support of the parity in ˜ O ( d ) steps, while in the same time horizon, the martingale contribution remains small, namely of order O (1 / √ d log( d )) . In particular, we show that

after the first phase of training, with high probability, all weights corresponding to coordinates in the parity's support are close to ∆ and the others are close to zero. The fit of the second layer on the whole dataset corresponds to the analysis of a linear model and follows from standard results on the convergence of SGD on convex losses (e.g., in [SSBD14]). The formal proof can be found in Appendix A.

Hinge Loss. While the use of the covariance loss simplifies considerably the proof of Theorem 3, we believe that the advantages of curriculum extend beyond this specific loss. Here, we present a positive result that applies to a more commonly used loss function, namely the hinge loss.

Theorem 4 (Hinge Loss) . Let NN( x ; θ ) = ∑ N i =1 a i σ ( w i x + b i ) be a 2-layer fully connected network with activation σ ( y ) := Ramp( y ) and N = ˜ θ ( d 2 log(1 /δ )) . Assume we have access to a dataset ( X,Y ) = { ( x s , y s ) } s ∈ [ m ] , with x s iid ∼ D mix , with parameters ρ > 0 , µ ∈ (0 , 1) , and y s = χ S ( x ) , for some target parity χ S , with | S | = k . Then, there exists an initialization and a learning rate schedule, such that for any target parity χ S and for any ϵ > 0 , the layer-wise curriculum SGD on the hinge loss with batch size B = ˜ θ ( d 10 /ϵ 2 log(1 /δ )) , noise level τ = ˜ θ ( ϵµ k log 1 /δ d 6 ) , T 1 = 1 and T 2 = ˜ θ ( d 6 /ϵ 2 ) with probability 1 -3 δ , outputs a network such that P x ∼D mix (sgn(NN( x ; θ T )) = f ( x )) ≥ 1 -ϵ.

As opposed to Theorem 3, Theorem 4 only holds for SGD with large batch size and unbounded learning rate schedules. The proof of Theorem 4 follows a 1-step argument, similarly to [CM23], and it is deferred to Appendix B.

4 Negative Result for Standard Training

In this section, we present a lower bound for learning k -parities on the mixed distribution of eq. (2), without curriculum.

Theorem 5 (Main Negative Result) . Assume we have access to a dataset ( X,Y ) = { ( x , y ) } s ∈ [ m ] , with x s iid ∼ D mix , with parameters ρ = o d (1) , µ ∈ (0 , 1) , and y s = χ S ( x ) , with | S | = k . Let NN( x ; θ ) be a fully-connected neural network of size P , with initialization that is invariant to permutations of the input neurons. Then, the noisy-SGD algorithm with noise level τ , gradient range A , batch size B , any learning rate schedule, and any loss function, after T steps of training without curriculum, outputs a network such that

where the probability is over x ∼ D mix and any randomness in the algorithm, and where C k is a constant that depends on k .

Remark 7 . For the purposes of Theorem 5, the network can have any fully connected architecture and any activation such that the gradients are well-defined almost everywhere. Furthermore, the loss can be any function that is differentiable almost everywhere.

Theorem 5 follows from an SQ-like lower bound argument and builds on previous results in [AS20][Theorem 3]. We defer the formal proof to Appendix C. Theorem 5 implies the following corollary.

Corollary 2. Under the assumptions of Theorem 5, if ρ = ˜ O ( d -3 . 5 -δ ) and k ≥ 6 , any fully connected network of size at most ˜ O ( d ) with any permutation-invariant initialization and activation, trained by the noisy-SGD algorithm with batch size B = ˜ Ω( ρ -2 ) , noise-level τ = ˜ O (1 /d ) and gradient range A = ˜ O ( √ d ) , after T = ˜ O ( d ) steps of training, will output a network such that P x ∼D mix (sgn(NN( x ; θ T )) = f ( x )) ≤ 1 2 + Cd -δ log( d ) c , for some c, C > 0 .

In particular, in Corollary 2, the network can have any width and depth (even beyond 2 layers), as long as the total number of parameters in the network scales as ˜ O ( d ) in the input dimension. Combining Corollary 1 and Corollary 2, we get that if ρ = ˜ O ( d -3 . 5 -δ ) , for some δ > 0 , and k ≥ 6 , and for ϵ = 1 / 10 , say, a 2 -layer fully connected network can learn any k parities up to accuracy 1 -ϵ in T = θ ( d ) steps with curriculum, while the same network in the same number of steps without curriculum, can achieve accuracy at most 1 -2 ϵ , for d large enough.

10 4

10 5

10 6

Number of samples

0.6

0.8

1.0

Gen. accuracy

=0.0001

Curr.

10 4

10 5

10 6

Number of samples

0.6

0.8

1.0

Gen. accuracy

=0.001

Curr.

10 4

10 5

10 6

Number of samples

0.6

0.8

1.0

Gen. accuracy

=0.01

Curr.

10 4

10 5

10 6

Number of samples

0.6

0.8

1.0

Gen. accuracy

=0.1

Curr.

10 4

10 5

10 6

Number of samples

0.8

0.9

1.0

Gen. accuracy

=0.5

Curr.

10

4

10

3

10

2

10

1

10 5

10 6

No. of samples

Curr.

Remark 8 . While our results show a first separation in the number of training steps between curriculum and standard training, we do not claim that the given range of ρ is maximal. For instance, one could get a tighter negative bound by considering Gaussian, instead of uniform, noise in the noisy-SGD iterates. However, getting a positive result with the curriculum for Gaussian noise would require a more involved argument. Our experiments show separation in the number of training steps and in the sample complexity for different values of ρ .

5 Experiments

In this section, we present empirical results demonstrating the benefits of the proposed curriculum method. 4 We use a multi-layer perceptron (MLP) with 4 hidden layers of sizes 512 , 1024 , 512 , and 64 as our model. We optimize the model under the ℓ 2 loss using mini-batch SGD with batch size 64. We also train all layers jointly. Each of our experiments is repeated with 10 different random seeds, and the results are reported with 95% confidence intervals. Note that generalization error is always computed based on D mix . Here, we mostly focus on the common settings not covered by our theoretical results. We present additional results on other architectures, namely, two-layer mean-field networks [MMN18] and Transformers [VSP + 17] and covariance and hinge loss functions in Appendix E.2.

Number of Samples. First, we focus on the difference in sample complexity, between curriculum and standard training. We consider learning f ( x 1 , . . . , x 100 ) = x 1 x 2 x 3 x 4 x 5 , with inputs sampled from a mixed distribution with µ = 0 . 98 and different values of ρ = 10 -4 , 10 -3 , 10 -2 , 0 . 1 , 0 . 5 . For each ρ , we train the network on training sets of different sizes (between 10 4 and 10 6 ) with and without curriculum. When using the curriculum, we first select the sparse samples as those with Hamming weight smaller than ( 1 2 -µ 4 ) d , and we train on those until convergence (specifically, training loss smaller than 10 -2 ). We then train the model on the entire training set until convergence. Similarly, when the curriculum is not used, the model is trained using all the available samples until convergence.

Figure 1 shows the validation accuracy achieved for different values of ρ, m . It can be seen that for some values of ρ , there is a reduction in the number of samples needed to achieve validation accuracy close to 1 when the curriculum is used. For the purpose of visualization, in Figure 1 (bottom-right), we sketch the number of samples needed to achieve accuracy close to 1 for different values of ρ , based on the previous plots. We notice that the sample complexity decreases as ρ is increased, for both curriculum and standard training. Among the ρ values that we tried, the gap between the two training strategies is maximal for ρ = 0 . 01 , and it decreases as ρ gets larger or smaller. Also, note

10

4

10

3

10

2

10

1

10 6

Steps to learn

Standard

Curriculum

10

3

10

2

10

1

10 6

Steps to learn

.98

.9

.5

.02

Curr.

10 4

10 5

10 6

Number of training samples

0.5

0.6

0.7

0.8

0.9

1.0

Gen. accuracy

k

=5

k

=7

k

=10

Curr.

10 4

10 5

10 6

Number of training samples

0.5

0.6

0.7

0.8

0.9

1.0

Gen. accuracy

.98

.9

.5

.02

Curr.

that for small values of ρ the sample complexity for standard training is constant and equals 10 6 . We note that with 10 6 samples, we can learn f ( x ) even with uniform inputs ( ρ = 0 ).

Number of Training Steps. Now we focus on the number of iterations needed for learning the same f as above. In order to study the number of iterations independently of the number of samples, we draw fresh mini-batches at each step. When using the curriculum, we sample from D µ in the initial phase and from D mix in the second phase, while when using standard training we sample from the mixed distribution at each step. In all cases, we train until convergence. The number of iterations needed for learning f for different values of ρ is depicted in Figure 2 (top-left). It can be seen that the number of training steps needed for learning with the curriculum is constant w.r.t. ρ . In contrast, for standard training, the number of iterations increases with the decrease of ρ . Both these observations are consistent with our theoretical results. In particular, for small values of ρ , the curriculum significantly reduces the number of training steps, while for large ρ it provides little or no benefit, and for ρ = 0 . 5 , it is harmful.

Dependency on µ and k . Next, we investigate the dependency of our results on µ and on the degree of parity k . For evaluating dependency on µ , we consider the function f as above and different values of µ . First, we consider the number of training steps needed for learning f based on ρ . The results are shown in Figure 2 (top-right). It can be seen that while the curriculum is not beneficial for µ = 0 . 02 (where D µ is close to D u and all samples are dense), it can significantly reduce the number of training steps for moderate and large values of µ . Further, we fix ρ = 0 . 01 and compare the performance of the curriculum and standard training for different training set sizes. The results are demonstrated in Figure 2 (bottom-right). It is shown that for large values of µ such as 0 . 98 and 0 . 90 , there is a gap in the number of samples needed for achieving generalization accuracy close to 1 between curriculum and standard training. On the other hand, for smaller values of µ , this gain is diminished. Nonetheless, note that the relationship between the performance of the curriculum method and µ is not monotonous (e.g., see the number of steps plot), and this is indeed reflected in our analysis. More precisely, in Theorem 1, T 1 scales with 1 /L 2 µ where L µ = µ k -1 -µ k +1 .

10 4

10 5

10 6

Number of training samples

0

1

2

Gen. loss

Curr.

10 4

10 5

Number of training samples

0.5

1.0

1.5

2.0

Gen. loss

Curr.

10 4

10 5

10 6

Number of training samples

0.0

0.2

0.4

Gen. loss

Standard

Curriculum

10

3

10

2

10

1

10 6

3×10 5

4×10 5

6×10 5

Steps to learn

Standard

Curriculum

10

3

10

2

10

1

10 5

10 6

Steps to weakly learn

Curr.

10

3

10

2

10

1

4×10 5

6×10 5

Steps to learn

Curr.

To analyze the effect of the degree of parity, k , we fix µ = 0 . 98 , ρ = 0 . 01 and the input dimension d = 100 , and we compare the performance of the model trained with different amounts of training data. The results are portrayed in Figure 2 (bottom-left). One can see that, while without curriculum the parity problem becomes harder to learn as k increases, the complexity with curriculum has mild dependence on k and it allows one to learn all the parities considered with 10 5 samples. The plot exhibiting the dependence of the number of optimization steps on k is reported in Appendix E.2.

Beyond Parities. Finally, we study target functions composed of multiple monomials. We consider three examples in 100-dimensional space: (i) f left ( x ) = x 1 x 2 x 3 x 4 x 5 + 1 2 x 1 x 2 x 3 x 4 x 5 x 6 , (ii) f middle ( x ) = x 1 x 2 x 3 x 4 x 5 + 1 2 x 6 · · · x 11 , and (iii) f right ( x ) = 1 2 ( x 1 x 2 + x 1 · · · x 6 ) . We also fix µ = 0 . 98 . In Figure 3, we investigate the gain of curriculum method in sample complexity for ρ = 0 . 01 (top row) and the number of training steps (bottom row) needed to learn. The plots in the bottom row are obtained by training the network with fresh batches. The left, middle, and right column correspond to f left , f middle , and f right , respectively. Note that curriculum is beneficial for f left , while it hardly helps with f right . Interestingly, for f middle , curriculum is advantageous for weak learning of the function. More specifically, it helps learning x 1 x 2 x 3 x 4 x 5 monomial of f middle . Consequently, for the corresponding iterations plot (Figure 3 bottom-middle), we plot the number of iterations needed for getting a loss below 0 . 26 . This suggests that the curriculum method put forward in this paper mostly helps with learning of the monomial with the lowest degree. We leave improvements of our curriculum method to future work.

6 Conclusion

In this paper, we introduced a curriculum strategy for learning parities on mixed distributions. Our approach involves training on sparse samples during the initial phase, which leads to a reduction in the number of required training steps and sample complexity compared to standard training. In the experiments section, we presented some results for certain functions with multiple monomials. Although a complete picture requires further investigation, these results suggest that starting from sparse samples provides advantages in the early stages of training. It would be interesting to investigate what variation of our curriculum strategy fits best the case of multiple monomials, beyond the initial phase. For example, one could explore multiple curriculum phases where training progresses from samples with increasing Hamming weight, or employ a self-paced curriculum that dynamically selects samples during training. Such analyses could offer valuable insights into the broader applicability of our approach. Here we focused mainly on establishing a formal separation in the number of training steps for the canonical case of parities.

References

[ABA22] Emmanuel Abbe and Enric Boix-Adsera. On the non-universality of deep learning: quantifying the cost of symmetry. arXiv preprint arXiv:2208.03113 , 2022.

[ABAM22] Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory , pages 4782-4887. PMLR, 2022.

[ABAM23] Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. arXiv preprint arXiv:2302.11055 , 2023.

[ABLR23] Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization on the unseen, logic reasoning and degree curriculum. arXiv preprint arXiv:2301.13105 , 2023.

[ACHM22] Emmanuel Abbe, Elisabetta Cornacchia, Jan Hazla, and Christopher Marquis. An initial alignment between neural network and target is needed for gradient descent to learn. In International Conference on Machine Learning , pages 33-52. PMLR, 2022.

[AGJ21] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. The Journal of Machine Learning Research , 22(1):4788-4838, 2021.

[AKB + 97] Judith Avrahami, Yaakov Kareev, Yonatan Bogot, Ruth Caspi, Salomka Dunaevsky, and Sharon Lerner. Teaching by examples: Implications for the process of category acquisition. The Quarterly Journal of Experimental Psychology Section A , 50(3):586606, 1997.

[AKM + 21] Emmanuel Abbe, Pritish Kamath, Eran Malach, Colin Sandon, and Nathan Srebro. On the power of differentiable learning versus PAC and SQ learning. In Advances in Neural Information Processing Systems , volume 34, 2021.

[AS20] Emmanuel Abbe and Colin Sandon. On the universality of deep learning. In Advances in Neural Information Processing Systems , volume 33, pages 20061-20072, 2020.

[AS23] Emmanuel Abbe and Colin Sandon. Poly-time universality and limitations of deep learning. to appear in CPAM , 2023.

[BEG + 22] Boaz Barak, Benjamin L Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. arXiv preprint arXiv:2207.08799 , 2022.

[BLCW09] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48, 2009.

[CG15] Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In Proceedings of the IEEE international conference on computer vision , pages 14311439, 2015.

[CM23] Elisabetta Cornacchia and Elchanan Mossel. A mathematical model for curriculum learning. arXiv preprint arXiv:2301.13833 , 2023.

[DBK + 20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

[DM20] Amit Daniely and Eran Malach. Learning parities with neural networks. Advances in Neural Information Processing Systems , 33:20356-20365, 2020.

[EA84] Renee Elio and John R Anderson. The effects of information order and learning mode on schema abstraction. Memory & cognition , 12(1):20-30, 1984.

[FHW + 17] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning , pages 482-495. PMLR, 2017.

[GBM + 17] Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning , pages 1311-1320. PMLR, 2017.

[GHZ + 18] Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from largescale web images. In Proceedings of the European conference on computer vision (ECCV) , pages 135-150, 2018.

[HW19] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In International Conference on Machine Learning , pages 2535-2544. PMLR, 2019.

[JMMH14] Lu Jiang, Deyu Meng, Teruko Mitamura, and Alexander G Hauptmann. Easy samples first: Self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on Multimedia , pages 547-556, 2014.

[KB14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

[KB17] Tom Kocmi and Ondrej Bojar. Curriculum learning and minibatch bucketing in neural machine translation. arXiv preprint arXiv:1707.09533 , 2017.

[Kea98] Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45(6):983-1006, 1998.

[KPK10] M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances in neural information processing systems , 23, 2010.

[MKAS21] Eran Malach, Pritish Kamath, Emmanuel Abbe, and Nathan Srebro. Quantifying the benefit of using differentiable learning over tangent kernels. In International Conference on Machine Learning , pages 7379-7389. PMLR, 2021.

[MMN18] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences , 115(33):E7665-E7671, 2018.

[MSS20] Eran Malach and Shai Shalev-Shwartz. Computational separation between convolutional and fully-connected networks. arXiv preprint arXiv:2010.01369 , 2020.

[NPL + 20] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. The Journal of Machine Learning Research , 21(1):7382-7431, 2020.

[PGM + 19] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

[PSL15] Anastasia Pentina, Viktoriia Sharmanska, and Christoph H Lampert. Curriculum learning of multiple tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5492-5500, 2015.

[PSN + 19] Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell. Competence-based curriculum learning for neural machine translation. arXiv preprint arXiv:1903.09848 , 2019.

[RK90] Brian H Ross and Patrick T Kennedy. Generalizing from the use of earlier examples in problem solving. Journal of Experimental Psychology: Learning, Memory, and Cognition , 16(1):42, 1990.

[RSR + 19] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 , 2019.

[SAJ10] Valentin I Spitkovsky, Hiyan Alshawi, and Dan Jurafsky. From baby steps to leapfrog: How 'less is more' in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pages 751-759, 2010.

[SGG14] Patrick Shafto, Noah D Goodman, and Thomas L Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive psychology , 71:55-89, 2014.

[SIRS22] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision , pages 1-40, 2022.

[SMS22] Luca Saglietti, Stefano Sarao Mannelli, and Andrew Saxe. An analytical theory of curriculum learning in teacher-student networks. Journal of Statistical Mechanics: Theory and Experiment , 2022(11):114014, 2022.

[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

[STD19] Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning a differentiable curriculum. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

[TV19] Yan Shuo Tan and Roman Vershynin. Online stochastic gradient descent with arbitrary initialization solves non-smooth, non-convex phase retrieval. arXiv preprint arXiv:1910.12837 , 2019.

[VSP + 17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

[WA20] Daphna Weinshall and Dan Amir. Theory of curriculum learning, with convex loss functions. Journal of Machine Learning Research , 21(222):1-19, 2020.

[WCA18] Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In International Conference on Machine Learning , pages 5238-5246. PMLR, 2018.

[WCZ21] Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

[ZKK + 18] Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation. arXiv preprint arXiv:1811.00739 , 2018.

[ZS14] Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4615 , 2014.

A Proof of Theorem 3

Algorithm 1 Layer-wise curriculum-SGD. Init. scales κ, ∆ > 0 , learning rates γ 1 , γ 2 > 0 , step counts T , T , input param. ρ, µ , batch size B , noise level τ and gradient range A .

In this section, we report the proof of Theorem 3. The proof follows a similar argument used in [TV19, AGJ21, ABAM23]: we decompose the dynamics into a drift and a martingale contribution. Our work differs from the ones above in considering a mixed input distribution and the covariance loss. For the purposes of this section, we consider a 2-layer neural network: NN( x ; θ ) = ∑ N i =1 a i σ ( w i x + b i ) , with N = k + 1 hidden neurons and activation σ := ReLU . We consider the following initialization:

We train the neural network using the layer-wise curriculum-SGD algorithm (see Algorithm 1 for pseudo-code). Without loss of generality, we assume that the target parity is χ [ k ] := ∏ j ∈ [ k ] x j . The result for other k -parities follows directly since the initialization and the curriculum-SGD algorithm are invariant to permutations of the input neurons. We show the following:

With a large enough training set, there are enough sparse inputs, which can be identified by computation of the Hamming weight.

Wetrain the first layer's weights w ij on the sparse data, keeping the a i , b i fixed. We first show that with enough sparse samples, the labels' average will concentrate around E x ∼D µ [ f ( x )] , with high probability. This term is needed in the computation of the covariance loss. Then, we show that after T 1 = ˜ O ( d ) steps, with an appropriate learning rate, with high probability, the target parity will belong to the linear span of the hidden units. This is obtained by showing that the weights corresponding to the first k input coordinates move noticeably, while the ones corresponding to the other d -k coordinates remain small.

We train the second layer's weights a i on the full data, keeping the first layer's weights and biases fixed. Established results on the convergence of SGD on convex losses allow us to conclude.

Bounding the Number of Samples. In the following lemma, we show that if the training set is large enough, the fraction of sparse inputs is large, and their Hamming weights will concentrate around expectation with high probability as d increases.

Lemma 1. Let { x s } s ∈ [ m ] be m i.i.d. inputs sampled from D , with parameters ρ, µ . Let X 1 := { x s : H ( x s ) < d ( 1 2 -µ 4 ) } and let S 1 := { x s : x s ∼ D µ } . If d ≤ m ≪ exp( d ) and m ≥ 2 m 1 ρ , for C ∗ > 0 ,

With probability at least 1 -2 d -C ∗ , | S 1 | ≥ m 1 .

ii) With probability at least 1 -d -C ∗ , S 1 = X 1 .

Proof. i) For all s ∈ [ m ] , P ( x s ∈ S 1 ) = ρ , thus,

for some constant C ∗ , that depends only on ρ , and in ( b ) we used Hoeffding's inequality.

ii) It is enough to show that for all x s ∈ S 1 , H ( x s ) < ¯ h , for ¯ h = d ( 1 2 -µ 4 ) and for all x s ̸∈ S 1 , H ( x s ) ≥ ¯ h . Thus, for x s ∈ S 1 ,

Similarly, for x s ̸∈ S 1 , P ( H ( x s ) < ¯ h ) < exp( -C ′ d ) , for some C ′ > 0 . The result follows from union bound.

In the rest of the proof, we will assume that the event in Lemma 1 holds.

A.1 First Layer Training

For the first training phase, we only use the sparse samples.

Computing ¯ y 1 . As a first step, we need to estimate the average of y over the sparse set. To that purpose, we sample one batch of size B 1 from X 1 . We denote by ¯ y 1 such estimate

Proof. By construction, and by Lemma 1, ¯ y 1 is the mean of B 1 i.i.d random variables sampled from D µ . Thus, by Hoeffding's inequality,

where in ( a ) we used that

We assume that each sample is presented to the network at most once. For instance, this can be done by discarding each sample after it is used, assuming that the number of samples is large enough for covering all training steps. Denoting by G w t ij the gradient computed at step t < T 1 for weight w t ij , the updates of the first layer weights are given by:

where by [ . ] ∆ we denote the projection of all coordinates that exceed ∆ (resp. -∆ ) to ∆ (resp. -∆ ). Note that with our assumption on the initialization, | NN( x ; θ t ) | < 1 for all t < T 1 . Thus, the gradients at each step are given by:

We decompose the dynamic into a drift and a martingale contribution:

where we denoted by

the (deterministic) population gradients.

Drift Contribution. In the following, we denote by w t j = w t ij for an arbitrary neuron i ∈ [ N ] . We first bound the drift term.

Lemma 3. Let L µ := µ k -1 -µ k +1 . Let ζ > 0 . If B 1 ≥ 2 C ∗ log( d ) /ζ 2 , with prob. 1 -d -C ∗ for all t ≤ T 1 ,

Proof. With our initialization, for all t < T 1 ,

Applying Lemma 2, with probability 1 -ϵ ′ ,

thus, applying one more time Lemma 2, we get the result.

Martingale Contribution. We consider w t j = w t ij for an arbitrary neuron i ∈ [ N ] . For t ∈ N and for all j , we define the martingale contribution to the dynamics up to time t by

We show that for appropriate time horizon and learning rate, the martingale contribution stays small.

Thus, for all j ∈ [ k ] ,

Similarly, for all j ̸∈ [ k ] ,

Lemma 4 (Lemma 4 in [ABAM23]) . Fix T ≤ C 0 d log( d ) C 0 , for some C 0 > 0 . For all C ∗ , there exists C that depends on C 0 , C ∗ such that if:

with probability at least 1 -d -C ∗ :

Proof. Note that in our setting (Boolean inputs, ReLU activation) we have that for all t < T 1

Thus, by Azuma-Hoeffding inequality:

The result follows by choosing ϵ = 1 / √ d log( d ) and by union bound on all t ≤ C 0 d log( d ) C 0 and j ∈ [ d ] .

Bounding the Contributions to the Dynamics. We define the following stopping times:

We show that for t ≥ τ ∆ j , w t j stays close to ∆ .

Lemma 5. Fix T ≤ C 0 d log( d ) C 0 , for some C 0 > 0 . Let ζ > 0 and let B 1 ≥ 2 C ∗ log( d ) /ζ 2 . For all C ∗ , there exists C such that if γ 1 satisfies (34) and if ζ ≤ 1 √ d log( d ) C 0 , with probability 1 -3 d -C ∗ for all j ∈ [ d ] :

Proof. For all j , let s = sup { t ′ ≤ t : w t ′ j ≥ ∆ -γ 1 (4 κ + τ ) } , then with probability 1 -3 d -C ∗ for all j ∈ [ d ] :

where in ( a ) we used that by Lemma 3 with prob. 1 -2 d -C ∗ , ¯ G w l j < -κL µ + κµζ < κµζ for all l ∈ [ t ] and Lemma 4, in ( b ) we used that by (34), γ 1 (4 κ + τ ) ≤ γ 1 (4 κ + τ ) √ t ≤ 1 / √ d log( d ) , and in ( c ) we used the assumptions on T and ζ . Thus, the result holds.

We will now establish a high-probability bound on the time that it takes for all weights associated with coordinates in the support of the target parity to approach proximity with ∆ .

Lemma 6. Let ζ > 0 and let B 1 ≥ 2 C ∗ log( d ) /ζ 2 . For all C ∗ , there exists C such that if γ 1 satisfies (34) , ζ ≤ 1 √ d log( d ) C 0 and d is large enough, such that d log( d ) ≥ ( √ C 0 µ +1) 2 ∆ 2 , with probability 1 -3 d -C ∗ :

Proof. Let T ≤ C 0 d log( d ) C 0 , for some C 0 > 0 . Let α t = min { w t j : j ∈ [ k ] , t ≤ τ ∆ j , t ≤ T } . Then, with probability 1 -3 d -C ∗ :

where in ( a ) we used that by (34), √ tγ 1 κ ≤ 1 / √ d log( d ) , and in ( b ) we used the assumptions on T and ζ . Thus if t ≥ 2∆ γ 1 κL µ and d is large enough, such that √ C 0 µ +1 √ d log( d ) < ∆ , we have

Thus, the result holds.

As our last lemma, we show that during the same time horizon, all coordinates that are not in the support of the parity will stay close to zero.

Lemma 7. Fix T ≤ C 0 d log( d ) C 0 , for some C 0 > 0 . Let ζ > 0 and let B 1 ≥ 2 C ∗ log( d ) /ζ 2 . For all C ∗ , there exists C such that if γ 1 satisfies (34) , ζ ≤ 1 √ d log( d ) C 0 and d is large enough, such that √

d log( d ) ≥ ( C 0 µ +1) 2 ∆ 2 , with probability 1 -3 d -C ∗ :

Proof. Let | β t | = max {| w t j | : j ∈ [ k +1 , ..., d ] , t ≤ T } . Thus, with probability 1 -3 d -C ∗ :

Thus, combining Lemmas 5, 6 and 7 and by union bound on i ∈ [ N ] , we get that for all C ∗ , there exist C 1 , C 2 such that if T 1 and γ 1 are such that

for some C 0 , C , with prob. 1 -3 Nd -C ∗ , for all i ∈ [ N ] ,

For τ ≤ κ , we can choose

where C ′ = C/ 50 , and the result follows.

A.2 Second Layer Training

As mentioned, the second phase of training consists of training the second layer's weights a i , keeping w ij and b i fixed. We do not train the biases, and directly assign b T 1 i to be well spread in the interval [∆(2 -k ) , ∆(2 + k )] , so that the target parity belongs to the linear span of the hidden units at time T 1 . We believe that one could extend the argument to cover scenarios where biases are trained, but this would require further technical work. To train the second layer, we use samples drawn from the entire dataset. Again, we take a large enough dataset and we assume that each sample is used at most once. In the following, we assume that the events in (58)-(59) hold. Below, we show that there exist a ∗ i such that ∑ N i =1 a ∗ i σ ( w T 1 i x + b i ) achieves small expected loss. For simplicity, we restrict our attention to k even, but an analogous Lemma to the below can be written for k odd. For a function f , an estimator ˆ f and input distribution D , let us define

Lemma 8. Assume k is an even integer. Let N = k +1 and for i ∈ { 0 , ..., k } let b i = -∆ k +2( i + 1)∆ . For all ϵ > 0 , there exist ∆ > 0 , D > max { C 1 , C 2 } such that if ρ < ( ∆ 2 N -D ) √ log( d ) √ dµ , and if d is large enough:

Proof. We follow a similar proof scheme as in [BEG + 22], and choose a ∗ i such that

We denote f ∗ ( x ) = ∑ k i =0 a ∗ i σ ( w T 1 i x + b i ) . One can verify that 2 χ [ k ] ( x ) = ∑ k i =0 a ∗ i σ (∆ ∑ k j =1 x j + b i ) . For all i ∈ [ N ] , let us denote ˆ w ij = w T 1 ij -∆ for all j ∈ [ k ] and ˆ w ij = w T 1 ij for all j ̸∈ [ k ] . Thus, E x ∼D [ ∑ d j =1 ˆ w ij x j ] = ρµ ∑ d j =1 ˆ w ij conditioned on the event in (58)-(59), ∥ ˆ w i ∥ 1 ≤ C 3 √ d √ log( d ) for all

i ∈ [ N ] , where C 3 = max { C 1 , C 2 } . Let δ = ∆ 8 N . Then,

where in ( a ) we used the assumption on ρ and in ( b ) we used Hoeffding's inequality and the fact that ∑ d j =1 (2 ˆ w ij ) 2 ≤ 4 C 2 3 log( d ) . Thus, with probability 1 -2 Nd -D 2 2 C 2 3 over x for all i ∈ { 0 , ..., k } :

Consequently,

Thus, for d large enough, with probability 1 -2 Nd -D 2 2 C 2 3 we have f ∗ ( x ) χ [ k ] ( x ) ≥ 1 . Moreover, for all x ,

Then,

for d large enough.

To conclude, we apply an established result from [SSBD14] on the convergence of SGD on convex losses.

Theorem 6 ([SSBD14]) . Let L be a convex function and let a ∗ ∈ argmin ∥ a ∥ 2 ≤B L ( a ) , for some B > 0 . For all t , let α t be such that E [ α t | a t ] = -∇ a t L ( a t ) and assume ∥ α t ∥ 2 ≤ ξ for some ξ > 0 . If a (0) = 0 and for all t ∈ [ T ] a t +1 = a t + γα t , with γ = B ξ √ T , then

We refer to [SSBD14] for a proof. Let us take L ( a ) := L cov (( a, w T 1 , b T 1 ) , χ [ k ] , D ) . Then, L is convex in a and for all t ∈ [ T 2 ] ,

and, recalling σ := ReLU , we have

and ∥ α t ∥ 2 ≤ √ N ∥ α t ∥ ∞ . Thus, if we choose

we obtain

The result follows under the assumption that the training process is terminated once the training loss falls below ϵ , thus guaranteeing that L cov ( θ T 1 + T 2 , χ [ k ] , x ) ≤ ϵ.

B Proof of Theorem 4

Theorem 7 (Theorem 4, restatement) . Let d, N be two positive integers. Let NN( x ; θ ) = ∑ N i =1 a i σ ( w i x + b i ) be a 2-layer fully connected network with activation σ ( y ) := Ramp( y ) (as defined in (84) ) and N ≥ ( d +1)( d -k +1)log(( d +1)( d -k +1) /δ ) . Consider training NN( x ; θ ) with layerwise curriculum-SGD on the hinge loss with batch size B ≥ (4 ζ 2 N 2 ) -1 log( Nd + N δ ) , noise level τ ≤ ζ log ( Nd + N δ ) -1 / 2 , with ζ ≤ ϵµ k 24( d +1) 2 ( d -k +1) 2 N and µ = √ 1 -1 2( d -k ) . Then, there exists an initialization and a learning rate schedule such that after T 1 = 1 step on the sparse data and T 2 = 64 ϵ 2 ( d -k +1) 3 ( d +1) N (1 + τ ) 2 steps on the full data, with probability 1 -3 δ , the trained network has generalization error at most ϵ .

This proof is similar to the proof of Theorem 3 in [CM23], with the following two modifications: 1) the use of a mixed input distribution in the second part of training, and, 2) the addition of uniform gradient noise in the iterates.

B.1 Proof Setup

We consider a 2-layer neural network, defined as NN( x ; θ ) = ∑ N i =1 a i σ ( w i x + b i ) , where N ≥ ( d + 1)( d -k + 1) log(( d + 1)( d -k + 1) /δ ) is the number of hidden units, θ = ( a, b, w ) and σ := Ramp denotes the activation defined as:

Similarly as before, without loss of generality, we assume that the labels are generated by χ [ k ] ( x ) := ∏ k i =1 x i . Our proof scheme is the following:

We train only the first layer of the network for one step on data ( x, χ [ k ] ( x )) with x ∼ D µ , with µ := 2 p -1 = √ 1 -1 2( d -k ) ;

We show that after one step of training on such biased distribution, the target parity belongs to the linear span of the hidden units of the network;

We train only the second layer of the network on ( x, χ [ k ] ( x )) with x ∼ D , until convergence;

We use established results on the convergence of SGD on convex losses to conclude.

We train our network with noisy-GD on the hinge loss. Specifically, we apply the following updates, for all t ∈ { 0 , 1 , . . . , T -1 } :

where L ( θ t , χ [ k ] , x ) = max { 0 , 1 -χ [ k ] ( x ) NN( x ; θ t ) } and Z w t i,j , Z a t i and Z b t i are i.i.d. Unif[ -τ, τ ] for some τ > 0 . Following the proof strategy introduced above, we set D 0 = D µ , where µ = √ 1 -1 2( d -k ) , and D t = D , for all t ≥ 1 . We set the parameters of (85) to:

and we consider the following initialization scheme:

where we define

Note that such initialization is invariant to permutations of the input neurons.

B.2 First Step: Recovering the Support

As mentioned above, we train our network for one step on ( x, χ [ k ] ( x )) with x ∼ D µ .

Population Gradient at Initialization. Let us compute the population gradient at initialization. Since we set ξ 0 = 0 , we do not need to compute the initial gradient for a . Note that at initialization

| NN( x ; θ 0 ) | < 1 . Thus, the initial population gradients are given by

Lemma 9. Initialize a, b, w according to (92) . Then,

Proof. If we initialize according to (92), we have ⟨ w i , x ⟩ + b i ∈ [0 , 1] for all i . The results holds since E x ∼D µ [ χ S ( x )] = µ | S | .

Effective Gradient at Initialization.

Lemma 10 (Noisy-GD) . Let

be the effective gradients at initialization, where G w i,j , G b i are the population gradients at initialization and Z w i,j , Z b i are i.i.d. Unif[ -τ, τ ] . If B ≥ (4 ζ 2 N 2 ) -1 log( Nd + N δ ) and τ ≤ ζ log ( Nd + N δ ) -1 / 2 , with probability 1 -2 δ :

Proof. By Hoeffding's inequality

The result follows by union bound.

Lemma 11. Let

(108)

If B ≥ (4 ζ 2 N 2 ) -1 log( Nd + N δ ) and τ ≤ ζ log ( Nd + N δ ) -1 / 2 , with probability 1 -2 δ ,

For all j ∈ [ k ] , i ∈ [ N ] , | w 1 i,j -1 | ≤ 2 Nζ µ k -1 ;

ii) For all j ̸∈ [ k ] , | w 1 i,j -(1 -1 2( d -k ) ) | ≤ 2 Nζ µ k -1 ;

For all j ∈ [ k ] , i ∈ [ N ] , | ˆ w 1 i,j -1 | = γ 0 | ˆ G w i,j -G w i,j | ≤ 2 Nζ µ k -1 ; ii) For all j ̸∈ [ k ] , i ∈ [ N ] , | ˆ w 1 i,j -(1 -1 2( d -k ) ) | = γ 0 | ˆ G w i,j -G w i,j | ≤ 2 Nζ µ k -1 ; iii) For all i ∈ [ N ] , | ˆ b 1 i -( d +1)( b 0 i -1 2 ) | = | λ 0 ( b 0 i + ψ 0 ˆ G b i ) -λ 0 ( b 0 i + ψ 0 G b i ) | (109) ≤ | λ 0 | · | ψ 0 | · | ˆ G b i -G b i | (110) ≤ N ( d +1) ζ µ k . (111)

Lemma 12. If N ≥ ( d +1)( d -k +1)log(( d +1)( d -k +1) /δ ) , then with probability 1 -δ , for all l ∈ { 0 , ..., d } , and for all m ∈ {-1 , ..., d -k } there exists i such that b 0 i = b lm d +1 + 1 2 .

Proof. The probability that there exist l, m such that the above does not hold is

The result follows by union bound.

Lemma 13. Let σ lm ( x ) = Ramp ( ∑ d j =1 x j -1 2( d -k ) ∑ j>k x j + b lm ) , with b lm given in (93) . If B ≥ (4 ζ 2 N 2 ) -1 log( Nd + N δ ) , τ ≤ ζ log ( Nd + N δ ) -1 / 2 and N ≥ ( d +1)( d -k +1)log(( d +1)( d -k +1) /δ ) , with probability 1 -3 δ , for all l, m there exists i such that

Proof. By Lemma 12, with probability 1 -δ , for all l, m there exists i such that b (0) i = b lm d +1 + 1 2 . For ease of notation, we replace indices i ↦→ ( lm ) , and denote ˆ σ lm ( x ) = Ramp ( ∑ d j =1 w 1 lm,j x j + b 1 lm ) . Then, by Lemma 11 with probability 1 -2 δ ,

We make use of Lemma 6 from [CM23], for which we rewrite the statement here for completeness.

Lemma 14 (Lemma 6 in [CM23]) . There exists a ∗ with ∥ a ∗ ∥ ∞ ≤ 4( d -k ) such that

Lemma 15. Let f ∗ ( x ) = ∑ l,m a ∗ lm σ lm ( x ) and let ˆ f ( x ) = ∑ l,m a ∗ lm ˆ σ lm ( x ) , with σ lm , ˆ σ lm defined in Lemma 13 and a ∗ defined in Lemma 14. If B ≥ (4 ζ 2 N 2 ) -1 log( Nd + N δ ) , τ ≤ ζ log ( Nd + N δ ) -1 / 2 and N ≥ ( d +1)( d -k +1)log(( d +1)( d -k +1) /δ ) , with probability 1 -3 δ for all x ,

Proof.

Thus,

which implies the result.

B.3 Second Step: Convergence

To conclude, we use one more time Theorem 6. Let L ( a ) := E x ∼D [ L (( a, b 1 , w 1 ) , χ [ k ] , x ) ] . Then, L is convex in a and for all t ∈ [ T ] ,

Thus, recalling σ = Ramp , we have ∥ α t ∥ ∞ ≤ 1 + τ and ∥ α t ∥ 2 ≤ √ N (1 + τ ) . Let a ∗ be as in Lemma 14. Clearly, ∥ a ∗ ∥ 2 ≤ 4( d -k +1) 3 / 2 ( d +1) 1 / 2 . Moreover, a 1 = 0 . Thus, we can apply Theorem 6 with B = 4( d -k +1) 3 / 2 ( d +1) 1 / 2 and ξ = √ N (1 + τ ) , and obtain that if:

then, with probability 1 -3 δ over the initialization

C Proof of Theorem 5

Let us recall the definition of Cross-Predictability (CP) from [AS20].

Definition 3 (Cross-Predictability (CP), [AS20]) . Let P F be a distribution over functions from {± 1 } d to {± 1 } and P X a distribution over {± 1 } d . Then,

We invoke Theorem 3 from [AS20], which we restate here.

Theorem 8 ([AS20]) . Let P F be a distribution on the class of parities on d bits, and let P X be a distribution over {± 1 } d . Assume that P F ∼P F ,x ∼P X ( F ( x ) = 1) = 1 2 + o d (1) . Consider any neural network NN( x ; θ ) with P edges and any initialization. Assume that a function f is chosen from P F and then T steps of noisy-SGD (Def. 1) with learning rate γ , gradient range A , batch size B and noise level τ are run on the network using inputs sampled from P X and labels given by f . Then, in expectation over the initial choice of f , the training noise, and a fresh sample x ∼ P X , the trained network satisfies:

In Theorem 8 we imported Theorem 3 from [AS20] with 1) the tighter bound for the specific case of parities, which gives a term (CP+1 /B ) 1 / 2 instead of (CP+1 /B ) 1 / 4 (see Theorem 4 and remarks in [AS20]); 2) the bound to the junk flow term JF T for noisy-SGD with uniform noise, instead of Gaussian noise, which gives a term P instead of √ P (see comments in [AS23]).

We thus compute the cross-predictability between P X = D and P F being the uniform distribution on the set of parities of degree k .

where C ′ k is a constant that depends only on k .

Proof.

̸

̸

where in ( a ) we used that for S and S ′ two randomly sampled sets of k coordinates, we have:

where C k is such that C k ≥ k ( k l ) Γ( k ) Γ(1+ k -l ) for all l ∈ [ k -1] .

If µ ≫ d -1 / 4 , then

where C ′ k depends only on k .

Finally, we need to discuss that Theorem 5 applies to any fully connected network with weights initialized from a distribution that is invariant to permutation of the input neurons. Let χ S ( x ) = ∏ j ∈ S x j , with | S | = k , and let π be a random permutation of the input neurons. Theorem 8 and Lemma 16 imply that after T steps of training with noisy-SGD with the parameters A,B and τ , on data generated by D and χ S , the trained network is such that

Recall that the network is fully connected and the weights outgoing from the input neurons are initialized from a distribution that is invariant to permutation of the input neurons. Consider the action induced by π on the weights outgoing from the input neurons. By properties of SGD, it follows that each conditional probability on π contributes equally to the left hand side of (137), thus the same bound holds also for the single function:

D Proof of Proposition 1

Note that

Since | ¯ y | < 1 , sgn(1 -y s ¯ y ) = 1 , and sgn((1 -y s ˆ f ( x s ))(1 -y s ¯ y )) = sgn(1 -y s ˆ f ( x s )) . Thus,

E Experiment Details and Additional Experiments

In this section, we provide more details on the implementation of the experiments. Moreover, we present further experiments on the comparison of the curriculum strategy and standard training.

E.1 Experiment Details

E.1.1 Architectures and Loss Functions

Architectures. We used an MLP model trained by SGD under the ℓ 2 loss for the results presented in the main part. In this appendix, we further present experimental results using other models, particularly, mean-field [MMN18] and Transformer [VSP + 17]. Below, we describe each in detail.

MLP. The MLP model is a fully-connected architecture consisting of 4 hidden layers of sizes 512, 1024, 512, and 64. The ReLU activation function has been used for each of the layers (except the last one). Moreover, we have used PyTorch's default initialization which initializes weights of each layer with U ( -1 √ dim in , 1 √ dim in ) where dim in is the input dimension of the corresponding layer.

Mean-field. We also use a two-layer neural network with mean-field [MMN18] parametrization. Particularly, we define f MF ( x ) = 1 N ∑ N i =1 a i σ ( ⟨ w i , x ⟩ + b i ) where N = 2 16 and σ = ReLU are the number of neurons and the activation function respectively. We also initialize the weights following a i ∼ U ( -1 , 1) , w i ∼ U ( -1 √ d , -1 √ d ) ⊗ d , and b i ∼ U ( -1 √ d , -1 √ d ) , where d is the dimension of the input. Note that with this parametrization, one should employ large learning rates (e.g., 1000) assuming SGD is used as the optimizer.

Transformer. We use the standard encoder part of Transformer architecture [VSP + 17] which is commonly used in language modeling ([RSR + 19]) and vision applications (particularly, Vision Transformers [DBK + 20]). More specifically, we first embed ± 1 into a 256-dimensional vector using a shared embedding layer. Then, the embedded input is processed through 6 transformer layers. In each transformer layer, the size of the MLP hidden layer is also 256, and 6 attention heads are used. Finally, a linear layer is utilized for computing the output. We also use learnable positional embedding.

Loss Functions. We have mainly used the ℓ 2 loss for our experiments. Nonetheless, we additionally present results on the hinge and covariance loss which are used in our theoretical results. We briefly review each of these losses below.

ℓ 2 loss. We use the squared loss (ˆ y -y ) 2 for each sample where ˆ y and y are the predicted and the true values respectively.

Hinge loss. We use the hinge loss defined as max(0 , 1 -ˆ yy ) for each sample where the true label y ∈ {± 1 } and the predicted value ˆ y ∈ R .

Covariance loss. Wealso use the covariance loss as defined in Definition 2, i.e., max(0 , ( y -¯ y )( y -ˆ y )) = max(0 , 1 -y ¯ y -ˆ y ( y -¯ y )) , where y ∈ {± 1 } , ¯ y ∈ ( -1 , 1) and ˆ y ∈ R present the true label, the average of the true label (on the training distribution), and the predicted value.

E.1.2 Procedure

We have implemented the experiments using the PyTorch framework [PGM + 19]. The experiments were executed (in parallel) on NVIDIA A100, RTX3090, and RTX4090 GPUs and consumed approximately 200 GPU hours (excluding the selection of hyperparameters). Note that each experiment has been repeated using 10 different random seeds and the results are reported with 95% confidence intervals.

Curriculum Strategy / Standard Training. Here we explain the general setting of training. When using the curriculum strategy, the model is first trained on the sparse samples until the training loss reaches below 10 -2 . Then, in the second phase of the curriculum, the training is continued on the mixed distribution until the training loss drops below 10 -3 . On the other hand, during the standard training, the training is done on the mixed distribution, again, until the training loss becomes less than 10 -3 .

Number of Samples/Number of Iterations. We generally focused on two components to compare the curriculum and standard training: sample complexity and the number of iterations needed for convergence. To compare the sample complexity, for a given number of training samples, we first generate a training set according to ρ and µ . In the case of using the curriculum, the sparse samples are then selected by checking if their Hamming weight satisfies H ( x ) < d ( 1 2 -µ 4 ) . Afterward, training is done as described earlier with or without the curriculum. Finally, the test accuracy (or the test loss) is compared. We are also interested in the number of steps needed for the convergence of models whether the curriculum is used or not. To study the number of optimization steps independently of the number of samples, we consider an online setting in which each mini-batch is freshly sampled and there is no fixed training set. Note that in this case, when in the curriculum's first phase, we directly sample the sparse distributions from D µ . Similarly, the model is trained until the training loss drops below 10 -3 , and then the number of iterations is compared. The only exception is f middle in Figure 3, where we were mainly interested in the weak learning of the target function. In that case, the training is stopped when the training loss reaches below 0 . 26 .

Hyperparameter Tuning. Note that the experiments are aimed at comparing the standard and curriculum-based training in a fair and similar context. As a result, we did not perform extensive hyperparameter tuning for our experiments. (Also note that the task and/or the training distribution varies from one experiment to another.) Nonetheless, we tried different batch sizes and learning rates to ensure the robustness of our findings. Generally, we did not observe any significant change in the patterns and potential gains of the curriculum method (of course changes like using a smaller learning rate would make both curriculum and standard training slower). Consequently, we used a moderate batch size of 64 for all of our experiments (other than Transformers) and we selected the learning rate based on the speed and the stability of the convergence. Particularly, we used a learning rate of 0 . 003 when optimizing the MLP model with SGD under the ℓ 2 loss (which covers all experiments presented in the main part). For the mean-field model, we use a learning rate of 1000. Also for the experiments of Transformers, we used the Adam [KB14] optimizer with batch size 512 and learning rate 0 . 0001 .

E.2 Additional Experiments

First, we complete Figure 2 by presenting the plot that shows the gain of the curriculum method in the number of iterations needed for convergence for different parity degrees in Figure 4. Similar to the previous experiments, we consider the parity embedded in dimension d = 100 . As expected, it is shown that parities of higher degree require more iterations to be learned in general. However, this increase in iterations is mild when the curriculum is employed, which leads to an increasing gap between the curriculum and standard training as the degree of parity is increased.

10

3

10

2

10

1

10 6

Steps to learn

Parity degree

k

=5

k

=7

k

=10

Standard

Curriculum

Next, we move to the mean-field model. Similar to the other experiments, we evaluate the benefit of the curriculum strategy for both the sample complexity and the number of steps needed for convergence. We empirically observed that the mean-field model is able to learn parity functions more easily (with fewer samples and iterations) than the MLP model used in the main text. We thus use parity of degree k = 7 in dimension 100 and µ = 0 . 98 to demonstrate the gains of the curriculum strategy more clearly. Particularly, we report the accuracy of the mean-field model trained under the ℓ 2 loss for ρ = 0 . 01 and different training set sizes in Figure 5 (left). Likewise, the number of iterations needed for convergence for different values of ρ is presented in 5 (right). As expected, the gap between curriculum strategy and standard training in both the sample complexity and number of optimization steps are similarly exhibited if a mean-field model is used.

10 4

10 5

10 6

Number of training samples

0.5

0.6

0.7

0.8

0.9

1.0

Gen. acc

Curr.

10

3

10

2

10

1

10 5

Steps to learn

Curr.

Moreover, we study other loss functions studied in this paper. To this end, we keep the same setting as above, considering the mean-field model, parity of 7 bits, and d = 100 , µ = 0 . 98 , and ρ = 0 . 01 (unless varying). We report the results of the hinge and covariance loss in Figure 6 and Figure 7 respectively. It can be seen that the patterns are generally similar to the case that ℓ 2 loss was being used (Figure 5). In sum, the curriculum method makes the learning of the parity function possible with fewer samples and iterations for small enough values of ρ .

10 4

10 5

10 6

Number of training samples

0.5

0.6

0.7

0.8

0.9

1.0

Gen. acc

Curr.

10

3

10

2

10

1

10 5

10 6

Steps to learn

Curr.

10 4

10 5

10 6

Number of training samples

0.5

0.6

0.7

0.8

0.9

1.0

Gen. acc

Curr.

10

3

10

2

10

1

10 5

10 6

Steps to learn

Curr.

Finally, we evaluate the curriculum strategy for the Transformer model trained under the ℓ 2 loss. To make the optimization of the model faster, we use Adam [KB14] optimizer with batch size 512 for this experiment. We also stop the training when the training loss reaches below 10 -2 instead of 10 -3 . Similar to the above-mentioned setting, we set µ = 0 . 98 , ρ = 0 . 01 (unless varying), d = 100 , and we consider learning the parity of degree k = 7 . The validation accuracy of the Transformer for different training set sizes is presented in Figure 8 (left). It can be seen that there is a significant, albeit weak, advantage in terms of sample complexity when the curriculum is used. Similarly, the number of iterations with fresh samples needed for convergence of the model is shown in Figure 8 (right). The gain in the number of steps is also weaker than the other models, nonetheless, it is significant (especially for small values of ρ such as 0 . 001 ).

10 4

10 5

Number of training samples

0.5

0.6

0.7

0.8

0.9

1.0

Gen. acc

Curr.

10

3

10

2

10

1

10 4

Steps to learn

Curr.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical analysis of curriculum learning, in which a neural network first presented with “easier” examples is able to more efficiently learn a target over more “complex” examples. The authors specifically consider the problem of learning $k$-sparse parities over a distribution which is a mixture of the uniform distribution over the hypercube and a biased distribution over the hypercube (the easy examples). The main positive results study a two-layer neural network trained via a layer-wise curriculum version of SGD (Algorithm 1), where first the first layer weights are trained on samples from the biased distribution, and next the second layer weights are trained on samples from the mixture distribution. Theorems 3 shows that this algorithm learns $k$-sparse parities with respect to the mixture distribution in $\\tilde O(d)$ steps of SGD. In contrast, the paper presents a lower bound showing that noisy SGD on the mixture distribution alone requires $\\Omega(d^{1+\\delta})$ timesteps to learn a $k$-sparse parity up to nontrivial error. The proof for the upper bound relies on the martingale plus drift argument to show that the $k$ relevant coordinates have been learned after the first stage, while the lower bound uses similar techniques to proving SQ lower bounds.\n\nSTRENGTHS:\n- The paper is well written.\n- I skimmed the proofs in the appendix and they appear to be sound. The experiments section demonstrates that the proposed curriculum strategy is effective in settings beyond those considered in the theory.\n- Curriculum learning is a common procedure in practice, and yet there is limited theoretical understanding of its behavior. The sparse parity problem presented here is an interesting setting to study curriculum learning, with a clear measure of “easier” samples. The results here adequately show that a neural network can obtain an efficiency improvement from first training on these easier samples, which I find to be a very nice result.\n\nWEAKNESSES:\n- The novelty of the paper, in comparison to the prior work [CM23], is a bit difficult to discern. It appears that the main difference is that the current paper uses SGD for multiple steps in the first stage and uses the “drift plus martingale” technique for the proof, while [CM23] considers a single large step of GD with large batch size in the first stage and shows that is sufficient for learning the support. However, Theorem 3 here seems very similar to Theorem 3.4 in [CM23]. Also, Theorem 4 still uses the one-step algorithm. Could you please comment further on the novelty of the current paper in comparison to the prior work?\n- A minor weakness is that the algorithm is a bit restrictive and has some nonstandard modifications. In particular, during the first stage the biases are chosen to be very large ($\\Theta(d)$), and then resampled to different deterministic values before the second stage. Next, there is an $\\ell_\\infty$ projection on the weights during each step of stage 1. Could you please comment on the necessity of these modifications? I think the paper could benefit from additional discussion of these modifications.\n\nQUESTIONS:\n- It appears that in Theorem 3, the algorithm receives far more samples from the biased distribution than the uniform distribution. In particular, during the first stage the algorithm receives $\\tilde O(d)$ biased samples, while in the second stage receives $\\tilde O(d)$ uniform samples. In contrast, when running SGD on the entire mixture distribution (like for the lower bounds in Theorem 5), the algorithm receives only $\\rho$-times as many biased samples as uniform samples. In the regimes where there is a separation, $\\rho$ is very small, in which case curriculum SGD is receiving far more samples from $D_\\mu$. It thus seems like the success of curriculum SGD is not necessarily due to viewing the “easy” examples first, but rather from having access to far more “easy” examples than in regular SGD. Is my understanding here correct, and can you please comment on this further?\n\nI am open to increasing my score after further clarifications on the above point and my concerns in the weaknesses section.\n\n- Minor typos: In Algorithm 1, should line 3 be $\\frac12 - \\frac{\\mu}{4}$ instead?\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a separation result between curriculum learning and standard learning in the number of noisy-(S)GD training steps for a 2-layer ReLU network for case of learning noiseless k-parity. While the precise training algorithm deviates from the common deep learning setup, experiments verify that learning k-parity can benefit from learning sparse inputs before harder dense inputs - mostly in terms of an improved sample complexity.\n\nSTRENGTHS:\n- Theoretical evidence for benefits of curriculum learning.\n- Additional positive results for the hinge loss.\n- Bound on sample complexity for the positive case.\n- The negative result allows for a comparison with the positive one in some settings.\n- The experimental results suggest a strong difference between curriculum learning and standard learning (yet, not exactly in the theoretical setup so far) in the context of learning a parity function.\n\nWEAKNESSES:\n- The analysis is limited to a single and not very general learning problem (learning k-parity).\n\n- The analysis makes assumptions that are quite different from practical deep learning setups:\n1) Non-standard neural network initialization. \n2) Non-standard loss (covariance loss) that penalizes errors on samples with true negative labels higher than errors on truly positive samples. The hinge loss results rely on an uncommon activation function.\n3) The hinge loss results are relatively weak as they just state the existence of an initialization etc. that leads to successful training. Essentially, this is not stronger than an expressiveness results, because the initialization could just correspond to the optimal (potentially learned) parameters.\n4) No noise in the data.\n5) Curriculum learning and dense learning use learning algorithms that are different also in other aspects than just the order of samples. For instance, the batch size is required to be quite large in case of dense learning. Most importantly, the theorem for curriculum learning assumes a much larger proportion of easy examples $\\rho$ than the standard learning. This is important because $\\rho$ essentially controls how easy the learning problem is. In the experiments, for large enough $\\rho$ dense learning seems to perform better than curriculum learning. (Btw, this observation is not explained by the theory.)\n6) The learning algorithm is not standard SGD or GD but learns the layers separately.\n7) The bias parameters are not trained (and it is left unclear in the main paper how they are initialized).\n\n- Mismatch between experimental validation and theory:\n1) different loss (l2 in experiments, mostly covariance loss in theory)\n2) k = 5 in most experiments, while theory assumes k >=6 for separation result.\n3) MLP with four layers instead of 2.\nExperiments should also validate the theory and not only study fairly unrelated cases where we can make similar observations. The direct validation would be important to understand how relevant the constants in the theoretical results are, for instance.\n\n- The theorem statements are hard to understand because they rely on notation that has not been introduced in the main text. (Examples: $\\mu^{k-1}$ or $\\Delta$ or P.)\n\n- The proof techniques do not appear to be very novel but similar to previous work.\n\n- Missing information: How is the generalisation accuracy computed? I assume samples were drawn from the mixed data distribution, which depends on $k$ and $\\rho$? How many samples were used in the experiments?\nThis is important because it also influences how hard the learning problem is. \n\n- The theoretical and experimental results study the performance with respect to a test distribution that depends on the amount of easy versus hard examples.\nThis is suboptimal because it can therefore make no claims about the question what kind of data would benefit learning the actual parity function (on all possible inputs).\nIn contrast, the central question underlying curriculum learning is what kind of data (and which order) is best for learning a task.\n\n\n**Points of minor critique:**\n\n- Literature discussion:\nGiven the literature discussion, the study of the specific problem of learning parities in the context of curriculum learning does not seem to be in its infancy, as claimed.  \nFurthermore, [SMS22] does not really make the difficulty level criterion label dependent but dependent on the (irrelevant features). In some sense, the same holds also true for the submitted work, as the definition depends on the negative coordinates that also inform the label.\n(Yet, this does not change the fact that [SMS22] and the presented work are sufficiently different.)\n\n- More elements of the notation could be explained. For instance, Rad could be introduced as Rademacher distribution; Unit as uniform distribution; neural networks could be defined formally earlier, []_A could be defined mathematically, etc. The bias scale was not defined. \nHow are the biases initialized? \nThe size P of a neural network has not been defined precisely. Does it refer to the number of hidden neurons or number of parameters?\nWeak learning has not been defined/explained.\n\n- The set of sparse inputs does not have size $\\rho$ on average according to the definition of $X_1$ on page 3 (different from how it is frequently discussed). Also $D_u$ contributes to $X_1$, while $D_{\\mu}$ does not always. Later, $d$ is assumed to be large enough that the statement is approximately correct, but that comes a bit late (and is not very precise).\n\n- Only even values of $\\mu$ have been studied in the experiments. Uneven averages should also be considered.\n\n- Not all curves in Figure 2 (middle and right) are visible.\n\nQUESTIONS:\n- To which degree does the performance actually depend on being exposed to the hard samples?\nCurriculum learning is based on the hypothesis that it is not only beneficial to learn from easy examples early but that learning from hard samples later also boosts the performance. \nI am aware the the negative result needs $\\rho$ to be small. Yet, I suspect that learning only on \"easy\" examples would already be sufficient to learn a parity function because the easy examples carry all relevant information about the parity function.\nTo analyze this question in more detail, it would be good to 1) make the generalisation accuracy independent of the mixed distribution and test whether the correct target function has been learned; 2) report how the performance changes during training.\n\n- Figure 2 (middle) implies a non-monotonous dependence of the accuracy in $\\mu$. It would be good to study this in more detail and show a figure with $\\mu$ on the x-axis. What could be an explanation? Does it matter whether $d\\mu$ is even or uneven?\n\n- To which degree are the experimental results dependent on hyperparameter tuning? Where the standard learning and curriculum learning be tuned independently?\n\nLIMITATIONS:\nThe limitations have been pointed out under weaknesses and were mostly discussed by the authors. I do not foresee a major or immediate societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper presents provable results showing the efficiency of curriculum learning for a specific problem setting with training data $(x,y)$, where $x\in(\pm1)^d$ is mixed distributed and $y=\Pi_{j\in\mathcal{S}} x_j\in(\pm1)$. What's more, the study utilizes a 2-layer fully connected network and the noisy-SGD training method

STRENGTHS:
1. This paper clearly proves the advantages of curriculum learning over standard learning by analyzing a well-defined problem.
2. Numerous results are also provided and highlight the reduced sample and iteration requirements of curriculum learning.

WEAKNESSES:
Please see Questions.

QUESTIONS:
1. This paper introduces a highly specific data setting. How can this setting be applied to real-world applications or more general settings?
2. Is there a particular significance to the notion of "sparsity"? I am curious about the evidence if the model was trained on dense data instead of sparse data initially.
3. In Theorem 3, why is $T_1$ independent to the first layer model size $Nd$, while $T_2$ is much larger than the second layer size $N$? Additionally, is the result evaluated using training accuracy?  
4. In my opinion, a more appropriate point of comparison for standard training would be: **randomly** sample an equivalent dataset size (as in curriculum learning) to train the first layer, followed by training the second layer using the entire dataset. Since the first layer contains significantly more parameters, standard training might introduce overfitting due to the layer-wise training method rather than curriculum learning.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the impact of curriculum learning for the family of parity functions. Compared to previous work, in which their settings have been distanced from a realistic setting, e.g., considering a non-common activation function with a particular initialization, or a non bounded learning rate, this paper solves all of those issues, although still in special case of covariance loss. They have been successful to show how the order of samples given to the model will effect the sample complexity, both in theory and experiments. Besides, to the downsides of the standard SGD algorithm that samples are completely shuffled, they have provided quite tight upper bounds for the final accuracy of a general fully connected neural network with limited total size, when assuming the procedure is noisy to make it resemble SQ-learning setting and use similar proofs.\n\nSTRENGTHS:\n1- Authors' suggested setting has been closest to reality so far. Unlike previous work they use a bounded learning rate throughout the training process, whereas the former algorithm had proceeded by using only one step of sgd with a learning rate that can get arbitrarily large. \n\n2- The combination of a lower bound for test accuracy when using curriculum learning (giving correlated samples at first so that it approximately learns the support) and an upper bound in case of standard random batch sgd, illustrates advantages of the proposed algorithm. \n\n3. Experiments are in fully agreement with results of theories, and they have implemented various settings for this: Comparison on number of samples, dependence on training steps, and different parameters.\n\nWEAKNESSES:\n1- Improvements over previous work is not very conspicuous. Essentially there is no new point in the paper but having former arguments in a more rigorous manner. \n\n2- Additionally, they don't give any assuring bounds besides two layer neural network, which is the exact model their reference work has considered. \n\n3- There are many non-free hyper parameters in their training process. Having assumed that the number of training steps for the first part of algorithm (T_1) depends on parity size is questionable. This can be the reason why previous work assumed only one step of SGD on sparse data. \n\n4- When attempting to come up with similar bounds besides the covariance loss, which is not common between experimentalists, they only consider hinge loss. And this causes their advantage over previous work to disappear because learning rates gets unbounded.\n\nQUESTIONS:\n1 - Would you please make a clarification on new points of your paper? It seems like your reference(CM23) has covered most of the claims as in your paper. \n\n2- I suggest to include other tasks that curriculum would come to be handful in your work.\n\nLIMITATIONS:\nThe main limit of this paper is not to go anywhere else other than the parity task. Many papers have investigated the same object before.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}