Title: Matching the Statistical Query Lower Bound for $k$-Sparse Parity Problems with Sign Stochastic Gradient Descent
Venue: NeurIPS 2024
Accepted: yes
Keywords: k-Parity

Optimization

Generalization
Abstract: The $k$-sparse parity problem is a classical problem in computational complexity and algorithmic theory, serving as a key benchmark for understanding computational classes. In this paper, we solve the $k$-sparse parity problem with sign stochastic gradient descent, a variant of stochastic gradient descent (SGD) on two-layer fully-connected neural networks. We demonstrate that this approach can efficiently solve the $k$-sparse parity problem on a $d$-dimensional hypercube ($k\le O(\sqrt{d})$) with a sample complexity of $\tilde{O}(d^{k-1})$ using $2^{\Theta(k)}$ neurons, matching the established $\Omega(d^{k})$ lower bounds of Statistical Query (SQ) models. Our theoretical analysis begins by constructing a good neural network capable of correctly solving the $k$-parity problem. We then demonstrate how a trained neural network with sign SGD can effectively approximate this good network, solving the $k$-parity problem with small statistical errors. To the best of our knowledge, this is the first result that matches the SQ lower bound for solving $k$-sparse parity problem using gradient-based methods.

Matching the Statistical Query Lower Bound for k -Sparse Parity Problems with Sign Stochastic Gradient Descent

Abstract

The k -sparse parity problem is a classical problem in computational complexity and algorithmic theory, serving as a key benchmark for understanding computational classes. In this paper, we solve the k -sparse parity problem with sign stochastic gradient descent, a variant of stochastic gradient descent (SGD) on two-layer fullyconnected neural networks. We demonstrate that this approach can efficiently solve the k -sparse parity problem on a d -dimensional hypercube ( k ≤ O ( √ d ) ) with a sample complexity of ˜ O ( d k -1 ) using 2 Θ( k ) neurons, matching the established Ω( d k ) lower bounds of Statistical Query (SQ) models 2 . Our theoretical analysis begins by constructing a good neural network capable of correctly solving the k -parity problem. We then demonstrate how a trained neural network with sign SGD can effectively approximate this good network, solving the k -parity problem with small statistical errors. To the best of our knowledge, this is the first result that matches the SQ lower bound for solving k -sparse parity problem using gradientbased methods.

1 Introduction

The k -parity problem, defined on a binary sequence of length d , is a fundamental problem in the field of computational complexity and algorithmic theory. This problem involves finding a subset of cardinality k by assessing if the occurrence of 1 's in this subset is even or odd. The complexity of the problem escalates as the parameter k increases. Its significance, while evidently practical, is rooted in its theoretical implications; it serves as a vital benchmark in the study of computational complexity classes and has profound implications for our understanding of P versus NP (Vardy, 1997; Downey et al., 1999; Dumer et al., 2003) and other cornerstone questions in computational theory (Blum, 2005; Klivans et al., 2006). Furthermore, the k -parity problem's complexity underpins many theoretical models in error detection and information theory (Dutta et al., 2008), and is instrumental

in delineating the limitations and power of algorithmic efficiency (Farhi et al., 1998). This paper tackles the k -sparse parity problem (Daniely and Malach, 2020), where the focus is on the parity of a subset with cardinality k ≪ d .

Recent progress in computational learning theory has focused on improving the sample complexity guarantees for learning k -sparse parity functions using stochastic gradient descent (SGD). Under the framework of the Statistical Query (SQ) model (Kearns, 1998), it has been established that learning the k -sparse parity function requires a minimum of Ω( d k ) queries (Barak et al., 2022), highlighting the challenge in efficiently learning these functions. On the other hand, considerable effort has been devoted to establishing sample complexity upper bounds for the special XOR case ( k = 2 ), with notable successes including O ( d ) sample complexity using infinite-width or exponential-width (i.e., O (2 d ) ) two-layer neural networks trained via gradient flow (Wei et al., 2019; Chizat and Bach, 2020; Telgarsky, 2022), and O ( d 2 ) sample complexity with polynomial-width networks via SGD (Ji and Telgarsky, 2020; Telgarsky, 2022). A significant advancement in solving the 2 -parity problem was recently introduced by Glasgow (2023). They proved a sample complexity bound of ˜ O ( d ) using a two-layer ReLU network with logarithmic width trained by SGD, thus matching the SQ lower bound when k = 2 .

In the general case of k ≥ 2 , Barak et al. (2022) has made significant progress, achieving a sample complexity of ˜ O ( d k +1 ) with a network width requirement of 2 Θ( k ) , which is independent of the input dimension d . Additionally, Barak et al. (2022) demonstrated that the neural tangent kernel (NTK)-based method (Jacot et al., 2018) requires a network of polynomial width d Ω( k ) to solve the k -parity problem. Recently, Suzuki et al. (2023) achieved a sample complexity of O ( d ) by the mean-field Langevin dynamics (MFLD) (Mei et al., 2018; Hu et al., 2019), which requires neural networks with an exponential width in d , i.e., O ( e d ) , and an exponential number of iterations (i.e., O ( e d ) ) to converge. Thus, their method is not computationally efficient and does not match the SQ lower bound. Notably, Abbe et al. (2023a) introduced the leapk function for binary and Gaussian sequences, which extends the scope of the k -parity problem. They also proved Correlational Statistical Query (CSQ) (Kearns, 1998; Bshouty and Feldman, 2002) lower bounds for learning leapk function for both Gaussian and Boolean inputs. In detail, they proved CSQ lower bounds of Ω( d k -1 ) for Boolean input and Ω( d k/ 2 ) for Gaussian input, which suggests that learning from Boolean input can be substantially harder than learning from Gaussian input. They also proved that SGD can learn low dimensional target functions with Gaussian isotropic data and 2-layer neural networks using n ≳ d Leap -1 examples. However, their upper bound analysis is based on the assumption that the input data x follows a Gaussian distribution and relies on Hermite polynomials, making it unclear how to extend it to analyze Boolean input. Based on the above review of existing literature, it raises a natural but unresolved question:

Is it possible to match the statistical query lower bound for k -sparse parity problems with stochastic gradient descent?

In this paper, we give an affirmative answer to the above question. In particular, we consider the standard k -sparse parity problem, where the input x is drawn from a uniform distribution over d -dimensional hypercube Unif( {-1 , 1 } d ) . Our approach involves training two-layer fully-connected neural networks with m = 2 Θ( k ) width using sign SGD (Bernstein et al., 2018) with batch size B = O ( d k -1 polylog( d )) . We prove that the neural network trained by SGD can achieve a constantorder positive margin with high probability after T = O (log d ) iterations. Therefore, the total number of examples required in our approach is n = BT = ˜ O ( d k -1 ) . Thus, the total number of scalar-valued queries required in our paper is m · d · n = 2 Θ( k ) · d · ( d k -1 · polylog d ) = 2 Θ( k ) d k · polylog d , where m is the number of neurons, d is the input dimension, and n is the total number of fresh examples seen by the algorithm 3 . Abbe et al. (2023a) also proved a CSQ lower bound Ω( d k ) for learning d -dimensional k -parity problems, which implies the sample complexity lower bound n ≳ d k -1 . Thus, our sample complexity result also matches the CSQ lower bound in Abbe et al. (2023a).

1.1 Our Contributions

The Statistical Query (SQ) lower bound indicates that, regardless of architecture, SGD requires a query complexity of Ω( d k ) for learning k -sparse d -dimensional parities under a constant noise level.

We push the sample complexity frontier of k -sparse parity problem to ˜ O ( d k -1 ) via SGD, specifically with online stochastic sign gradient descent. Our main result is stated in the following informal theorem:

Theorem 1.1 (Informal) . For a two-layer fully-connected neural networks of width 2 Θ( k ) , online sign SGD with batch size ˜ O ( d k -1 ) can find a solution to the k -parity problem with a small test error within O ( k log d ) iterations.

The above theorem improves the sample complexity in Barak et al. (2022) from ˜ O ( d k +1 ) to ˜ O ( d k -1 ) . Moreover, the total number of queries required is ˜ O ( d k ) , which matches the SQ/CSQ lower bound up to logarithmic factors. Additionally, under the standard basis setting, our result matches the sample complexity in Glasgow (2023) for solving the XOR (i.e., 2 -parity) problem with sign SGD. It is worth noting that our result only requires two-layer fully connected neural networks with 2 Θ( k ) width and sign SGD training with O ( k log d ) iterations, which gives a computationally efficient algorithm. Finally, we empirically verify our theory in Appendix A, showcasing the efficiency and efficacy of our approach.

Notation. We use [ N ] to denote the index set { 1 , . . . , N } . We use lowercase letters, lowercase boldface letters, and uppercase boldface letters to denote scalars, vectors, and matrices, respectively. For a vector v = ( v 1 , · · · , v d ) ⊤ , we denote by ∥ v ∥ 2 := ( ∑ d j =1 v 2 j ) 1 / 2 its L 2 norm. For a vector v = ( v 1 , · · · , v d ) ⊤ , we denote by v [ i 1 : i 2 ] := ( v i 1 , · · · , v i 2 ) ⊤ its truncated vector ranging from the i 1 -th coordinate to the i 2 -th coordinate. We denote by 0 a vector of all zeros. For two sequence { a k } and { b k } , we denote a k = O ( b k ) if | a k | ≤ C | b k | for some absolute constant C , denote a k = Ω( b k ) if b k = O ( a k ) , and denote a k = Θ( b k ) if a k = O ( b k ) and a k = Ω( b k ) . We also denote a k = o ( b k ) if lim | a k /b k | = 0 . We use ˜ O ( · ) and ˜ Ω( · ) to omit logarithmic terms in the notation. Finally, we denote x n = poly( y n ) if x n = O ( y D n ) for some positive constant D , and x n = polylog( y n ) if x n = poly(log( y n )) .

2 Related Work

XOR Problem. The performance of two-layer neural networks in the task of learning 2 -parity has been the subject of extensive research in recent years. Wei et al. (2019); Chizat and Bach (2020); Telgarsky (2022) employed margin techniques to establish the convergence toward a global margin maximization solution, utilizing gradient flow and sample complexity of O ( d ) . Notably, Wei et al. (2019) and Chizat and Bach (2020) employed infinite-width neural networks, while Telgarsky (2022) employed a more relaxed width condition of O ( d d ) . A significant breakthrough in this domain was achieved by Glasgow (2023), who demonstrated a sample complexity of ˜ O ( d ) by employing SGD in conjunction with a ReLU network of width polylog( d ) . Furthermore, several other studies have shown that when input distribution follows Gaussian distribution, neural networks can be effectively trained to learn the XOR cluster distribution (Frei et al., 2022; Meng et al., 2023; Xu et al., 2023). A comparison between the results in this paper and those of related work solving the XOR problem can be found in Table 1.

k -parity Problem. The challenge of training neural networks to learn parities has been explored in previous research from diverse angles. Several papers studied learning the k -parity function by using two-layer neural networks. Daniely and Malach (2020) studied learning k -parity function by applying gradient descent on the population risk (infinite sample size). Notably, Barak et al. (2022) presented both empirical and theoretical evidence that the k -parity function can be effectively learned using SGD and a neural network of constant width, demonstrating a sample complexity of O ( d k +1 ) and query complexity of O ( d k +2 ) . Edelman et al. (2024) demonstrated that sparse initialization and increased network width lead to improvements in sample efficiency. Specifically, they showed that the sample complexity can be reduced at the cost of increasing the width. However, the best statistical query complexity they can achieve is O ( d k +2 ) , which is the same as that in Barak et al. (2022). Suzuki et al. (2023) reported achieving a sample complexity of O ( d ) by employing mean-field Langevin dynamics (MFLD). Furthermore, Abbe et al. (2022) and Abbe et al. (2023a) introduced a novel complexity measure termed 'leap' and established that leapk (with k -parity as a special case) functions can be learned through SGD with a sample complexity of ˜ O ( d max( k -1 , 1) ) . Additionally, Abbe et al. (2023b) demonstrated that a curriculum-based noisy-GD (or SGD) approach

.

could attain a sample complexity of O ( d ) , provided the data distribution comprises a mix of sparse and dense inputs. The conditions outlined in this paper are compared to those from related work involving uniform Boolean data distribution, as presented in Table 2.

3 Problem Setup

In this section, we introduce the k -sparse parity problem and the neural network we consider in this paper.

Definition 3.1 ( k -parity ) . Let each data point ( x , y ) with x ∈ R d and y ∈ {-1 , 1 } be generated from the following distribution D A , where A is a non-empty set satisfying A ⊆ [ n ] :

x j ∼ {-1 , 1 } as a uniform random bit for j ∈ [ d ] .

The label y is generated as Π j ∈ A x j .

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

x

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

y

Modified vs. Standard Sign Function

sign(

x

)

sign(x)

The k -parity problem with dimension d is defined as the task of recovering A , where | A | = k , using samples from D A .

Without loss of generality, we assume that A = { 1 , . . . , k } if | A | = k . Under this assumption, we denote D A by D for simplicity. This k -parity problem in Definition 3.1 is a classical one, which has been studied by Daniely and Malach (2020); Barak et al. (2022) using neural network learning. When restricted to the 2 -parity function, the problem is reduced to the XOR problem (Wei et al., 2019).

Two-layer Neural Networks. We consider a two-layer fully-connected neural network, which is defined as follows:

where m is the number of neurons. Here, we employ a polynomial activation function defined by σ ( z ) = z k . The term w r ∈ R d represents the weight vector for the r -th neuron, and W denotes the aggregate of all first-layer model weights. The second-layer weights a r 's are sampled uniformly from the set {-1 , 1 } and fixed during training.

Algorithm. We train the above neural network model by minimizing the correlation loss function:

where ℓ ( z ) = 1 -z . We consider binary initialization with w (0) r ∼ Unif( {± 1 } d ) , which is widely used for neural networks solving parity problem (Barak et al., 2022; Abbe and Boix-Adsera, 2022). We then employ the stochastic sign gradient descent with constant step size and weight decay (i.e., ℓ 2 norm regularization on the first-layer weights) to minimize the correlation loss as follows:

where λ > 0 is the weight decay parameter, η > 0 is the step size, and ˜ sign( x ) is the modified sign function defined as:

Here, ρ > 0 is a threshold parameter. In this context, L ( t ) is computed using a randomly sampled online batch S t with batch size | S t | = B :

Consequently, the update rule for each w r is given by:

where ˜ sign is applied on an element-wise basis.

Remark 3.2. Sign SGD has been previously studied in Riedmiller and Braun (1993); Bernstein et al. (2018). Recently, it has become increasingly popular and has been utilized in adaptive optimizers for training large models (Chen et al., 2024; Liu et al., 2023). Previous studies (Balles and Hennig, 2018; Bernstein et al., 2018; Zou et al., 2021) have demonstrated that Sign SGD behaves similarly to Adam when using sufficiently small step sizes or small moving average parameters, β 1 and β 2 . In our work, the choice of sign SGD over standard SGD stems primarily from our adoption of the polynomial activation function σ ( z ) = z k . As later explained in Section 4, this specific activation function is pivotal in constructing a neural network that accurately tackles the k -parity problem. However, it introduces a trade-off: the gradient's dependency on the weights becomes polynomial rather than linear. Sign SGD addresses this issue by normalizing the gradient, ensuring that all neurons progress uniformly towards identifying the parity. Moreover, incorporating a threshold within the sign function plays a crucial role as it effectively nullifies the gradient of noisy coordinates. This, together with weight decay, aids in reducing noise, thereby enhancing the overall performance of the network.

4 Main Results

In this section, we begin by demonstrating the capability of the two-layer fully connected neural network (3.1) to classify all examples correctly. Specifically, we construct the following good network:

where { w ∗ r, [1: k ] ∣ ∣ r ∈ [2 k ] } = {± 1 } k , a ∗ r = ∏ k j =1 sign( w ∗ r,j ) and w ∗ r, [ k +1: d ] = 0 d -k for any r ∈ [2 k ] . Notably, leveraging the inherent symmetry within our neural network model, we can formally assert the following proposition: yf ( W ∗ , x ) = y ′ f ( W ∗ , x ′ ) for any ( y, x ) and ( y ′ , x ′ ) generated from D A . The subsequent proposition demonstrates the precise value of the margin.

Proposition 4.1. For any data point ( x , y ) generated from the distribution D A , it holds that

Proof. Given a ( y, x ) ∈ D A , we have that y = Π k i =1 x i . We divide the neurons into ( k +1) groups Ω i , i ∈ { 0 , . . . , k } . A neuron r ∈ Ω i if and only if ∑ k s = j 1 ( w ∗ r = x j ) = i . Then we have that

where the third equality is due to the fact that y = ∏ k j =1 sign( x j ) and a ∗ r = ∏ k j =1 sign( w ∗ r,j ) , the fourth equality is due to the definition of Ω i , the last equality holds because σ is k -th order polynomial activation function and Lemma F.2.

Therefore, we can conclude that for any ( x , y ) , we have yf ( W ∗ , x ) = k ! · 2 k > 0 . We will demonstrate in the next section that training using large batch size online SGD, as long as Condition 4.2 is met, will lead to the trained neural network f ( W ( T ) , x ) approximating ( m/ 2 k +1 ) · f ( W ∗ , x ) effectively after T = O ( k log( d )) iterations. Our main theorem is based on the following conditions on the training strategy.

Condition 4.2. Suppose there exists a sufficiently large constant C , such that the following conditions hold:

Neural network width m satisfies m ≥ C · 5 k log(1 /δ ) .

Dimension is sufficiently large: d ≥ C log 2 (2 m/ϵ ) .

Online SGD batch size B ≥ C 2 k (( k -1)!) -2 d k -1 log k -1 (16 mdBT/δ ) log 2 (8 mdT/δ ) .

Learning rate η satisfies η ≤ C -1 .

Regularization parameter λ is taken as λ = 1 .

The threshold ρ for the modified sign function satisfies ρ = 0 . 1 k ! .

In the k -parity problem, the label y is determined by a set of k bits. Consequently, the total count of distinct features is 2 k , reflecting all possible combinations of these bits. The condition of m is established to guarantee a roughly equal number of neurons within the good neuron class, each correctly aligned with distinct features. The condition of d ensures that the problem is in a sufficiently high-dimensional setting. The condition of m,d implies that d ≥ Ω(log 2 m ) ≥ Ω( k 2 ) , which is a mild requirement for the sparsity k . In comparison, Barak et al. (2022) requires d ≥ Ω( k 4 ) for neural networks solving k -parity problem. By Stirling's approximation, the condition of B can be simplified to B ≥ ˜ Ω ( k (2 e log(16 mdBT/δ ) d/k 2 ) k -1 ) . Therefore, the conditional batch size B will exponentially increase as parity k ≤ O ( √ d ) goes up, which ensures that the stochastic gradient can sufficiently approximate the population gradient. Finally, the conditions of η, λ ensure that gradient descent with weight decay can effectively learn the relevant features while simultaneously denoising the data. Finally, the threshold condition ρ increases as sparsity k increases to accommodate the increase of the population gradient. Based on these conditions, we give our main result on solving the parity problem in the following theorem.

Theorem 4.3. Under Condition 4.2, we run online SGD for iteration T = Θ ( kη -1 λ -1 log d ) iterations. Then with probability at least 1 -δ we can find W ( T ) such that

where γ = 0 . 25 k ! is a constant.

Theorem 4.3 establishes that, under certain conditions, a neural network is capable of learning to solve the k -parity problem within Θ( kη -1 λ -1 log d ) iterations, achieving a population error of at most ϵ . According to Condition 4.2, the total number of examples utilized amounts to BT = ˜ O ( d k -1 ) given a polynomial logarithmic width requirement of m = O (1) with respect to d .

Remark 4.4. While Theorem 4.3 works for fixed second-layer training, we demonstrate that comparable results can be obtained when the second layer of the network is simultaneously trained with a lower learning rate. Detailed results and further elaboration of this aspect are provided in Appendix E. Our findings present a sample complexity of ˜ O ( d k -1 ) , aligning with Conjecture 2 posited by Abbe et al. (2023a), which suggests a sample complexity lower bound of ˜ Ω( d k -1 ) . Besides, our results for the uniform Boolean distribution match the complexity achieved by Abbe et al. (2023a) under the isotropic Gaussian scenario. Despite these similarities in outcomes, our methodology diverges significantly: we employ online sign SGD utilizing a large batch size of ˜ O ( d k -1 ) and conduct training over merely ˜ O (1) iterations. In contrast, Abbe et al. (2023a) implement projected online SGD with a minimal batch size of 1 , extending training over ˜ O ( d k -1 ) iterations. Abbe et al. (2023a) also requires a two-phase training process for the first and second layer weights, requiring them to be trained separately.

5 Overview of Proof Technique

In this section, we discuss the main ideas used in the proof. Based on these main ideas, the proof of our main Theorem 4.3 will follow naturally. The complete proofs of all the results are given in the appendix. Section 5.1 serves as a warmup by examining population sign gradient descent. Here, three pivotal ideas crucial to the proof of stochastic sign gradient descent are introduced:

The impact of the initialization's positivity or negativity on the trajectory of neuron weights.

The divergence between feature coordinates and noise coordinates of different neurons.

How a trained neural network can effectively approximate the good neural network (4.1).

Moving on to Section 5.2, we delve into the analysis of sign SGD. Contrasting with population GD, the addition in SGD analysis involves accounting for the approximation error between the population gradient and the stochastic gradient. This consideration leads to the stipulation of the batch size B outlined in Condition 4.2.

5.1 Warmup: Population Gradient Descent

For population gradient, we perform the following updates:

where L D ( W ) = E ( x ,y ) ∼D [ ℓ ( y, f ( W , x ))] = 1 -E ( x ,y ) ∼D [ yf ( W , x )] . Then, the following coordinate-wise population gradient update rules hold:

In the preceding discussion of the update rule, we have identified that the noise coordinates ( j / ∈ [ k ] ) exhibit exponential decay, characterized by a decay constant of 1 -ηλ . To further dissect the dynamics of this system, we turn our attention to the behavior of feature coordinates. We categorize neurons into two distinct types based on their initial alignment: a neuron is classified as a good neuron if a r = ∏ k j =1 sign( w (0) r,j ) , and conversely, as a bad neuron if a r = -∏ k j =1 sign( w (0) r,j ) . This distinction is pivotal, as it divides the neuron population into two distinct classes: good and bad . Neurons in the good class are integral to the functionality of the final trained neural network, playing a significant role in its test accuracy. Conversely, neurons classified as bad tend to diminish in influence over the course of training, ultimately contributing minimally to the network's overall performance. For good neurons, the update rules for feature coordinates can be reformulated to

For neurons classified as bad , the update rules for feature coordinates can be rewritten as:

Comparing equations (5.3) and (5.4), it becomes apparent that the feature coordinates of good and bad neurons exhibit divergent behaviors. Consequently, the feature coordinates of good neurons will significantly outweigh those of bad neurons in the long term. With the regularization parameter λ set as 1 in Condition 4.2, we derive the following lemma illustrating the divergent trajectories of population gradient descent for both neuron types:

Lemma 5.1. Under Condition 4.2, for good neurons r ∈ Ω g := { r ∈ [ m ] : a r = ∏ k j =1 sign( w (0) r,j ) } , the feature coordinates will remain the same as initialization throughout the training:

For bad neurons r ∈ Ω b := { r ∈ [ m ] : a r = -∏ k j =1 sign( w (0) r,j ) } , the feature coordinates will decay faster than noise coordiantes:

According to (5.2) and Lemma 5.1, after training T = Θ( kη -1 λ -1 log( d )) iterations, bad neurons and noise coordinates in good neurons diminish to a magnitude of Θ(1 / poly( d )) , as shown in Lemma 5.2. In contrast, the feature coordinates of good neurons remain unchanged.

Lemma 5.2. Under Condition 4.2, for T ≥ ( k +1) η -1 λ -1 log( d ) , it holds that

This leads to the following approximation for the trained neural network:

Under Condition 4.2, the condition on m ensures a balanced distribution of neurons across different initializations, approximately m/ 2 k +1 , given 2 k +1 kinds of possible initializations. This results in the trained neural network f ( W ( T ) , x ) closely approximating ( m/ 2 k +1 ) · f ( W ∗ , x ) .

5.2 Stochastic Sign Gradient Descent

Transitioning from the trajectory trained by population gradient descent, this section delves into the dynamics under sign stochastic gradient descent (Sign SGD). We commence by presenting a lemma that estimates the approximation error between the population gradient and the stochastic gradient.

Lemma 5.3. Under Condition 4.2, with probability at least 1 -δ with respect to the online data generation, the stochastic gradient approximates the population gradient well:

where L ( t ) is the loss of randomly sampled online batch S t and

The choice of batch size B = O ( d k -1 ) in our algorithm is crucial for gradient concentration. According to Lemma 5.3, the gap between stochastic gradient and population gradient is bounded by ϵ 1 · ∥ w r ∥ k -1 2 . At initialization, the absolute value of the population gradient on the signal coordinate is approximately ˜ O ( d -( k -1) / 2 ) · ∥ w r ∥ k -1 2 . To ensure the stochastic sign gradient matches the population sign gradient, the approximation error ϵ 1 must be smaller than this value, which requires ϵ 1 = ˜ O ( d -( k -1) / 2 ) . Solving B -1 / 2 + d ( k -3) / 2 B -1 = d -( k -1) / 2 yields our sufficient batch size.

Building upon this approximation guarantee from Lemma 5.3, and considering the established order of ρ, ϵ 1 and ∥ w ( t ) r ∥ 2 , we arrive at an important corollary.

Corollary 5.4. Under Condition 4.2, given the same initialization, with probability at least 1 -δ , the stochastic sign gradient is the same as the population sign gradient:

This corollary suggests that, under identical initialization, the trajectory of a model trained using population gradient descent will, with high probability, align with the trajectory of a model trained using stochastic gradient descent.

6 Conclusion and Future Work

In our study, we have conducted a detailed analysis of the k -parity problem, investigating how sign Stochastic Gradient Descent (sign SGD) can effectively learn intricate features from binary datasets. Our findings reveal that sign SGD, when employed in two-layer fully-connected neural networks solving k -sparse parity problem, is capable of achieving a sample complexity ˜ O ( d k -1 ) . Remarkably, this result matches the theoretical expectations set by the Statistical Query (SQ) model, underscoring the efficiency and adaptability of sign SGD.

Looking ahead, an intriguing direction for future research is to explore the possibility of learning k -parity using SGD with even smaller queries that surpass the SQ lower bond, and understand whether more standard neural network architectures allow such improvement. This potential advancement could pave the way for developing more efficient algorithms capable of tackling complex problems with weaker data requirements. Another promising direction is to extend our results to non-isotropic data settings (Nitanda et al., 2024), where sign gradient descent with momentum could be effective in handling the transformed feature space.

References

ABBE, E., ADSERA, E. B. and MISIAKIEWICZ, T. (2022). The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory . PMLR.

ABBE, E., ADSERA, E. B. and MISIAKIEWICZ, T. (2023a). Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory . PMLR.

ABBE, E. and BOIX-ADSERA, E. (2022). On the non-universality of deep learning: quantifying the cost of symmetry. Advances in Neural Information Processing Systems 35 17188-17201.

ABBE, E., CORNACCHIA, E. and LOTFI, A. (2023b). Provable advantage of curriculum learning on parity targets with mixed inputs. In Thirty-seventh Conference on Neural Information Processing Systems .

BALLES, L. and HENNIG, P. (2018). Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning . PMLR.

BARAK, B., EDELMAN, B. L., GOEL, S., KAKADE, S., MALACH, E. and ZHANG, C. (2022). Hidden progress in deep learning: Sgd learns parities near the computational limit. arXiv preprint arXiv:2207.08799 .

BERNSTEIN, J., WANG, Y.-X., AZIZZADENESHELI, K. and ANANDKUMAR, A. (2018). signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning . PMLR.

BLUM, A. (2005). On-line algorithms in machine learning. Online algorithms: the state of the art 306-325.

BSHOUTY, N. H. and FELDMAN, V. (2002). On using extended statistical queries to avoid membership queries. Journal of Machine Learning Research 2 359-395.

CHEN, X., LIANG, C., HUANG, D., REAL, E., WANG, K., PHAM, H., DONG, X., LUONG, T., HSIEH, C.-J., LU, Y. ET AL. (2024). Symbolic discovery of optimization algorithms. Advances in Neural Information Processing Systems 36 .

CHIZAT, L. and BACH, F. (2020). Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory . PMLR.

DANIELY, A. and MALACH, E. (2020). Learning parities with neural networks. Advances in Neural Information Processing Systems 33 20356-20365.

DOWNEY, R. G., FELLOWS, M. R., VARDY, A. and WHITTLE, G. (1999). The parametrized complexity of some fundamental problems in coding theory. SIAM Journal on Computing 29 545-570.

DUMER, I., MICCIANCIO, D. and SUDAN, M. (2003). Hardness of approximating the minimum distance of a linear code. IEEE Transactions on Information Theory 49 22-37.

DUTTA, C., KANORIA, Y., MANJUNATH, D. and RADHAKRISHNAN, J. (2008). A tight lower bound for parity in noisy communication networks. In Proceedings of the nineteenth annual ACM-SIAM symposium on Discrete Algorithms .

EDELMAN, B., GOEL, S., KAKADE, S., MALACH, E. and ZHANG, C. (2024). Pareto frontiers in deep feature learning: Data, compute, width, and luck. Advances in Neural Information Processing Systems 36 .

FARHI, E., GOLDSTONE, J., GUTMANN, S. and SIPSER, M. (1998). Limit on the speed of quantum computation in determining parity. Physical Review Letters 81 5442.

FREI, S., CHATTERJI, N. S. and BARTLETT, P. L. (2022). Random feature amplification: Feature learning and generalization in neural networks. arXiv preprint arXiv:2202.07626 .

GLASGOW, M. (2023). Sgd finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the xor problem. arXiv preprint arXiv:2309.15111 .

HU, K., REN, Z., SISKA, D. and SZPRUCH, L. (2019). Mean-field langevin dynamics and energy landscape of neural networks. arXiv preprint arXiv:1905.07769 .

JACOT, A., GABRIEL, F. and HONGLER, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems .

JI, Z. and TELGARSKY, M. (2020). Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks. In International Conference on Learning Representations .

KEARNS, M. (1998). Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) 45 983-1006.

KLIVANS, A. R., SERVEDIO, R. A. and RON, D. (2006). Toward attribute efficient learning of decision lists and parities. Journal of Machine Learning Research 7 .

LIU, H., LI, Z., HALL, D., LIANG, P. and MA, T. (2023). Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342 .

MEI, S., MONTANARI, A. and NGUYEN, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences 115 E7665-E7671.

MENG, X., ZOU, D. and CAO, Y. (2023). Benign overfitting in two-layer relu convolutional neural networks for xor data.

MILNE-THOMSON, L. M. (2000). The calculus of finite differences . American Mathematical Soc.

NITANDA, A., OKO, K., SUZUKI, T. and WU, D. (2024). Improved statistical and computational complexity of the mean-field langevin dynamics under structured data. In The Twelfth International Conference on Learning Representations .

RIEDMILLER, M. and BRAUN, H. (1993). A direct adaptive method for faster backpropagation learning: The rprop algorithm. In IEEE international conference on neural networks . IEEE.

SUZUKI, T., WU, D., OKO, K. and NITANDA, A. (2023). Feature learning via mean-field langevin dynamics: classifying sparse parities and beyond. In Thirty-seventh Conference on Neural Information Processing Systems .

TELGARSKY, M. (2022). Feature selection with gradient descent on two-layer networks in lowrotation regimes. arXiv preprint arXiv:2208.02789 .

VARDY, A. (1997). The intractability of computing the minimum distance of a code. IEEE Transactions on Information Theory 43 1757-1766.

WEI, C., LEE, J. D., LIU, Q. and MA, T. (2019). Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel. Advances in Neural Information Processing Systems .

XU, Z., WANG, Y., FREI, S., VARDI, G. and HU, W. (2023). Benign overfitting and grokking in relu networks for xor cluster data.

ZOU, D., CAO, Y., LI, Y. and GU, Q. (2021). Understanding the generalization of adam in learning neural networks with proper regularization. arXiv preprint arXiv:2108.11371 .

Limitations

While our study provides valuable insights into the effectiveness of SGD in learning the k -parity problem, there are some limitations:

Our study focuses on sign gradient descent (Sign SGD). This approach normalizes the gradient, ensures uniform neuron updates toward identifying parity, and effectively nullifies noisy coordinate gradients. However, it's worth noting that data may be presented in non-standard or unknown coordinate systems, which could limit Sign SGD's effectiveness. To address this limitation, future work could explore alternatives such as normalized gradient descent with an adaptive learning rate or incorporating momentum into Sign SGD.

Our analysis is primarily based on polynomial activation functions. While effective for the standard k-parity problem, extending our approach to other activation functions like sigmoid or ReLU presents challenges. This extension could potentially be achieved through polynomial function approximation. However, the main challenge lies in identifying an appropriate functional decomposition and accurately characterizing the approximation error during training.

By acknowledging these limitations, we aim to provide a transparent assessment of our work's scope and potential areas for future exploration.

A Experiments

In this section, we present a series of experiments designed to empirically validate the theoretical results established in our main theorem. The primary objectives of these experiments are to (1) assess the test accuracy of the trained neural network, thereby corroborating the theorem's results, and (2) verify the key lemmas concerning the behavior of good and bad neurons. Specifically, we aim to demonstrate that for good neurons, the feature coordinates remain largely unchanged from initialization while the noise coordinates decay exponentially. Conversely, for bad neurons, we expect both feature and noise coordinates to exhibit exponential decay.

Model. We generated synthetic k -parity data based on Definition 3.1. Each data point ( x , y ) with x ∈ R d and y ∈ {± 1 } is produced from distribution D A , where A specifically is taken as [ k ] . We utilized two-layer fully-connected neural networks with m number of neurons. The network employs a polynomial activation function σ ( z ) = z k . The first layer weights for the r -th neuron, w (0) r ∈ R d , were initialized following a binary scheme, where w (0) r ∼ Unif ( {± 1 } d ) . The second layer weights for the r -th neuron, a r , is randomly initialized as 1 or -1 with equal probability.

Computation Resources The experiments are conducted on an A6000 server. As these are synthetic experiments, the requirement for computational resources is minimal.

Training. Our model was trained to minimize the empirical correlation loss function, incorporating L 2 regularization with a regularization parameter set at λ = 1 . The training process utilized online stochastic sign gradient descent (Sign SGD) with a fixed step size η and a predetermined batch size B . The principal metric for assessment was test accuracy. Our experiments were conducted considering parity k ∈ { 2 , 3 , 4 } .

For k = 2 , the model configuration included a data dimension of d = 8 , a hidden layer width of m = 12 , a total of T = 25 epochs, a learning rate η = 0 . 1 , an online batch size of B = 64 , and a threshold for ˜ sign set at ρ = 0 . 3 . In the case of k = 3 , we employed a data dimension of d = 16 , increased the hidden layer width to m = 48 , extended the training to T = 50 epochs, adjusted the learning rate to η = 0 . 05 , used an online batch size of B = 256 , and set the threshold for ˜ sign at ρ = 1 . For k = 4 , the model was further scaled up with a data dimension of d = 20 , a hidden layer width of m = 128 , a training epoch of T = 100 , a smaller learning rate of η = 0 . 02 , an online batch size of B = 2048 , and a threshold for ˜ sign established at ρ = 3 .

Experimental Results. The evaluation of test accuracy across various configurations is presented in Table 3. Using neural networks with merely 2 Θ( k ) neurons, we observed high test accuracy for k -parity problem with k ∈ { 2 , 3 , 4 } , confirming the results of our main theorem (Theorem 4.3).

These empirical results validate the efficacy of our studied model architecture (3.1) and training methodology in tackling the k -sparse parity problem.

To further validate our theoretical findings, we examined the change of feature and noise coordinates for the first neuron w ( t ) 1 over multiple iterations, focusing specifically on the setting k ∈ { 2 , 3 , 4 } . Figures 2, 3, 4, 5, 6, and 7 visually represent these trajectories. Our empirical findings reveal a consistent pattern: the feature coordinates ( w ( t ) 1 , 1 , . . . , w ( t ) 1 ,k ) of the neurons identified as good (with initialization satisfying a r = ∏ k j =1 w (0) r,j ) exhibit relative stability, while their noise coordinates ( w ( t ) 1 ,k +1 , . . . ) show a decreasing trend over time. In contrast, the trajectories for neurons classified as bad (with initialization satisfying a r = -∏ k j =1 w (0) r,j ) indicate a general reduction in all coordinate values.

These empirical observations support our theoretical analyses, as outlined in Lemma 5.1 and Lemma 5.2, showcasing the consistency between the theoretical foundations of our model and its practical performance. The disparities between good and bad neurons, as evidenced by their feature and noise coordinate behaviors, underscore the nuanced dynamics inherent in the learning process of k -parity problems.

0

5

10

15

20

25

Epochs

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Change of the First Neuron in the Hidden Layer

Feature 1:

w

(

t

)

1, 1

Feature 2:

w

(

t

)

1, 2

Noise 1:

w

(

t

)

1, 3

Noise 2:

w

(

t

)

1, 4

0

10

20

30

40

50

Epochs

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Change of the First Neuron in the Hidden Layer

Feature 1:

w

(

t

)

1, 1

Feature 2:

w

(

t

)

1, 2

Feature 3:

w

(

t

)

1, 3

Noise 1:

w

(

t

)

1, 4

Noise 2:

w

(

t

)

1, 5

Noise 3:

w

(

t

)

1, 6

0

5

10

15

20

25

Epochs

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Change of the First Neuron in the Hidden Layer

Feature 1:

w

(

t

)

1, 1

Feature 2:

w

(

t

)

1, 2

Noise 1:

w

(

t

)

1, 3

Noise 2:

w

(

t

)

1, 4

0

10

20

30

40

50

Epochs

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Change of the First Neuron in the Hidden Layer

Feature 1:

w

(

t

)

1, 1

Feature 2:

w

(

t

)

1, 2

Feature 3:

w

(

t

)

1, 3

Noise 1:

w

(

t

)

1, 4

Noise 2:

w

(

t

)

1, 5

Noise 3:

w

(

t

)

1, 6

0

20

40

60

80

100

Epochs

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Change of the First Neuron in the Hidden Layer

Feature 1:

w

(

t

)

1, 1

Feature 2:

w

(

t

)

1, 2

Feature 3:

w

(

t

)

1, 3

Feature 4:

w

(

t

)

1, 4

Noise 1:

w

(

t

)

1, 5

Noise 2:

w

(

t

)

1, 6

Noise 3:

w

(

t

)

1, 7

Noise 4:

w

(

t

)

1, 8

B Preliminary Lemmas

During the initialization phase of a neural network, neurons can be categorized into 2 k distinct groups. This classification is based on whether each feature coordinate is positive or negative. We define these groups as follows:

where b 1 , b 2 , · · · , b k ∈ {± 1 } . To illustrate with specific examples, consider the following special cases:

In these cases, Ω 11 ··· 1 represents the group of neurons where all initial weights are positive across the k features, while Ω -1 -1 ···-1 consists of neurons with all initial weights being negative. Within each group of neurons, we can further subdivide them into two subgroups based on the value of a r . Let's denote

where Ω g denotes the good neuron set and Ω b denotes the bad neuron set. We will later demonstrate in the proof and experiments that neurons in Ω g and neurons in Ω b exhibit distinct behaviors during the training process. Specifically, for neurons in Ω g , the feature coordinates will remain largely unchanged from their initial values throughout training, while the noise coordinates will decrease to a lower order compared to the feature coordinates. On the other hand, for neurons in Ω b , both feature coordinates and noise coordinates will decrease to a lower order compared to their initial values.

In order to establish the test error result, it is essential to impose a condition on the initialization. Specifically, the number of neurons for each type of initialization should be approximately equal.

Lemma B.1. With probability at least 1 -δ for the randomness in the neural network's initialization, the sizes of the sets Ω g and Ω b are bounded as follows:

Besides, the intersections of Ω b 1 b 2 ··· b k with both Ω g and Ω b are also bounded within a specified range:

where

0

20

40

60

80

100

Epochs

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Change of the First Neuron in the Hidden Layer

Feature 1:

w

(

t

)

1, 1

Feature 2:

w

(

t

)

1, 2

Feature 3:

w

(

t

)

1, 3

Feature 4:

w

(

t

)

1, 4

Noise 1:

w

(

t

)

1, 5

Noise 2:

w

(

t

)

1, 6

Noise 3:

w

(

t

)

1, 7

Noise 4:

w

(

t

)

1, 8

Proof. Let X r = 1 [ w (0) r, 1 > 0 , · · · , w (0) r,k > 0 , a r = 1 ] . Then, by Chernoff bound, we have

Then, with probability at least 1 -δ , we have

By applying union bound to all 2 k +1 kinds of initialization, with probability at least 1 -δ it holds that for any Ω b 1 b 2 ··· b k ∩ Ω g and Ω b 1 b 2 ··· b k ∩ Ω b ( ( b 1 , · · · , b k ) ∈ {± 1 } k )

where

C Warmup: Population Sign GD

In this section, we train a neural network with gradient descent on the distribution D . Here we use correlation loss function ℓ ( y, ̂ y ) = 1 -y ̂ y . Then, the loss on this attribution is L D ( W ) = E ( x ,y ) ∼D [ ℓ ( y, f ( W , x ))] , and we perform the following updates:

We assume the network is initialized with a symmetric initialization: for every r ∈ [ m ] , initialize w (0) r ∼ Unif ( {-1 , 1 } d ) and initialize a r ∼ Unif ( {-1 , 1 } ) .

Lemma C.1. The following coordinate-wise population sign gradient update rules hold:

Proof. For population gradient descent, we have

Notice that for j / ∈ [ k ] , we have

where

where the last equality is because { j 1 , · · · , j k -1 } ⊊ { 1 , · · · , k, j } . This implies that

For j ∈ [ k ] , we have

̸

where the last inequality is because E ( x ,y ) ∼D [ x 1 x 2 · · · x k x j x j 1 · · · x j k -1 ] = 0 if and only if { j, j 1 , · · · , j k -1 } = { 1 , 2 , · · · , k } . It follows that

Given the update rules in Lemma C.1, we observe distinct behaviors for good neurons ( Ω g ) and bad neurons ( Ω b ). The following corollary illustrates these differences:

Corollary C.2. For any neuron r ∈ Ω b 1 ··· b k ∩ Ω g , the update rule for feature coordinates ( j ∈ [ k ]) is given by:

For any neuron r ∈ Ω b 1 ··· b k ∩ Ω b , the update rule for feature coordinates ( j ∈ [ k ]) is given by:

By setting the regularization parameter λ to 1 , we observe a noteworthy property of the weight of feature coordinates in good neurons. This is formalized in the following lemma:

Lemma C.3. Assume λ = 1 and ρ < k ! . For a neuron r ∈ Ω b 1 ··· b k ∩ Ω g , the weight associated with any feature coordinate remains constant across all time steps t ≥ 0 . Specifically, it holds that:

Proof. We prove this by using induction. The result is obvious at t = 0 . Suppose the result holds when t = ˜ t . Then, according to Corollary C.2, we have

The following lemma demonstrates that for bad neurons, the weights of feature coordinates tend to shrink over time. The dynamics of this shrinking are characterized as follows:

Lemma C.4. Assume λ = 1 and η/ (1 -ηλ ) < ( ρ/k !) 1 k -1 . For a neuron r ∈ Ω b 1 b 2 ··· b k ∩ Ω b , the weights of any two feature coordinates j and j ′ (where j, j ′ ∈ [ k ] ) are equal at any time step, that is, b j w ( t ) r,j = b j ′ w ( t ) r,j ′ . Furthermore, the weight of any feature coordinate j ∈ [ k ] evolves according to the following inequality for any t ≥ 0 :

Proof. We prove this by using induction. We prove the following three hypotheses:

We will show that H 1 (0) and H 2 (0) are true and that for any t ≥ 0 we have

H 2 ( t ) = ⇒ H 3 ( t ) ,

H 1 ( t ) , H 2 ( t ) = ⇒ H 1 ( t +1) ,

H 1 ( t ) , H 2 ( t ) = ⇒ H 2 ( t +1) .

H 1 (0) and H 2 (0) are obviously true since b j w (0) r,j = 1 for any j ∈ [ k ] . Next, we prove that H 2 ( t ) = ⇒ H 3 ( t ) and H 1 ( t ) , H 2 ( t ) = ⇒ H 1 ( t +1) . According to Corollary C.2, we have

where the second equality is by H 2 ( t ) . This verifies H 3 ( t ) . Besides, given H 1 ( t ) , we have

Plugging this into (C.1), we can get

which verifies H 1 ( t +1) . Finally, we prove that H 1 ( t ) , H 2 ( t ) = ⇒ H 2 ( t +1) . By (C.1) and H 1 ( t ) , we have

If | w ( t ) r,j | < ( ρ/k !) 1 k -1 , we can get

If | w ( t ) r,j | ≥ ( ρ/k !) 1 k -1 , given that

we can get

Given Lemma C.3 and Lemma C.4, we can directly get the change of neurons of all kinds of initialization.

Corollary C.5. Assume λ = 1 , ρ < k ! and η/ (1 -ηλ ) < ( ρ/k !) 1 k -1 . For any fixed ( b 1 , b 2 , · · · , b k ) ∈ {± 1 } k , considering r ∈ Ω b 1 b 2 ··· b k , we have the following statements hold.

For any neuron r ∈ Ω b 1 b 2 ··· b k ∩ Ω g , the weights of feature coordinates remain the same as initialization: w ( t ) r,j = b j for any t ≥ 0 and j ∈ [ k ] .

For any neuron r ∈ Ω b 1 b 2 ··· b k ∩ Ω b , the weights of feature coordinates will shrink simultaneously over time: b j w ( t ) r,j = b j ′ w ( t ) r,j ′ for any t ≥ 0 and j, j ′ ∈ [ k ] and

Building on Corollary C.5, we can now characterize the trajectory of all neurons over time. Specifically, after a time period T = Θ( kη -1 λ -1 log( d )) , the following observations about neuron weights hold:

Lemma C.6. Assume λ = 1 , ρ < k ! and η/ (1 -ηλ ) < ( ρ/k !) 1 k -1 . For T ≥ ( k +1) η -1 λ -1 log d , it holds that

Proof. The first equality is obvious according to Corollary C.5. We only need to prove the inequalities. According to Lemma C.1, we have for any r ∈ [ m ] and j ∈ [ d ] \ [ k ] that

where the last inequality is by T ≥ kη -1 λ -1 log( d ) . According to Corollary C.5, for any r ∈ Ω b and j ∈ [ k ] , we have that

Lemma C.7. Under Condition 4.2, with a probability of at least 1 -δ with respect to the randomness in the neural network's initialization, trained neural network f ( W ( T ) , x ) approximates accurate classifier ( m/ 2 k +1 ) · f ( W ∗ , x ) well:

Proof. First, we can rewrite (4.1) as follows:

where w ∗ b 1 ··· b k = [ b 1 , b 2 , · · · , b k , 0 , · · · , 0] ⊤ . To prove this lemma, we need to estimate the noise part of the inner product ⟨ w ( T ) r , x ⟩ . By Hoeffding's inequality, we have the following upper bound for the noise part ∑ d j = k +1 w ( T ) r,j x j :

Then with probability at least 1 -ϵ/m we have that for fixed r ∈ [ m ]

where the last inequality is by Condition 4.2. By applying union bound to all m neurons, with probability at least 1 -ϵ we have that for any r ∈ [ m ]

where the first three inequalities are by triangle inequality and Lemma B.1; the fourth inequality is due to

by (C.2), mean value theorem and Lemma C.6 and for r ∈ Ω b

Since ( m/ 2 k +1 ) · | f ( W ∗ , x ) | = 0 . 5 k ! · m , then as long as

we have

where the second inequality is by Stirling's approximation. Then it follows that

D Stochastic Sign GD

In this section, we consider stochastic sign gradient descent for learning k -parity function. The primary aim of this section is to demonstrate that the trajectory produced by SGD closely resembles that of population GD. To begin, let's recall the update rule of SGD:

where | S t | = B . Our initial step involves estimating the approximation error between the stochastic gradient and the population gradient, detailed within the lemma that follows.

Lemma D.1. With probability at least 1 -δ with respect to the randomness of online data selection, for all t ≤ T , the following bound holds true for each neuron r ∈ [ m ] and for each coordinate j ∈ [ d ] :

where ϵ 1 is defined as

Proof. To prove (D.1), let us introduce the following notations:

where g r,j ( x , y ) represents the gradient at the point ( x , y ) , and h r,j ( x , y ) denotes the truncated version of g r,j ( x , y ) , which is employed for the convenience of applying the Hoeffding's inequality. Firstly, utilizing Hoeffding's inequality, we can assert the following:

Furthermore, we can establish an upper bound for the difference between the expectations of h r,j ( x , y ) and g r,j ( x , y ) :

where the first inequality is by Cauchy inequality and the second inequality is by Hoeffding's inequality. Additionally, with high probability, the gradient and the truncated gradient are identical:

where the second inequality applies Hoeffding's inequality. Combining inequalities (D.2), (D.3), and (D.4), we can assert with probability at least

that the following inequality holds:

By setting γ = √ 2 log(16 B/δ ) ∥ w ( t ) r ∥ 2 and x = √ 2 kγ k -1 log(8 /δ ) / √ B , we establish that with probability at least 1 -δ , the following bound is true:

Applying a union bound over all indices r ∈ [ m ] , j ∈ [ d ] , and iterations t ∈ [0 , T -1] , we conclude with probability at least 1 -δ that

Based on Lemma D.1, we can get the following lemma showing that with high probability, the stochastic sign gradient follows the same update rule as the population sign gradient.

Lemma D.2. Under Condition 4.2, with probability at least 1 -δ with respect to the randomness of online data selection, the following sign SGD update rule holds:

Proof. We prove this by using induction. We prove the following hypotheses:

We will show that H 2 (0) , H 3 (0) , H 4 (0) and H 5 (0) are true and for any t ≥ 0 we have

H 2 ( t ) , H 4 ( t ) = ⇒ H 4 ( t +1) . (This can be established by adapting the proof of Lemma C.3; hence, we omit the proof details here.)

H 2 ( t ) , H 5 ( t ) = ⇒ H 5 ( t +1) , H 6 ( t ) . (This can be shown by following the proof of Lemma C.4, so the proof details are omitted here.)

H 3 ( t ) , H 4 ( t ) , H 4 ( t +1) , H 6 ( t ) = ⇒ H 1 ( t ) .

{ H 1 ( s ) } t s =0 = ⇒ H 2 ( t +1) , H 3 ( t +1) .

H 4 (0) and H 5 (0) are obviously true since w (0) r,j = b j for any r ∈ Ω b 1 b 2 ··· b k and j ∈ [ k ] . To prove that H 2 (0) and H 3 (0) are true, we only need to verify that

By Lemma D.1, we have for j / ∈ [ k ]

leading to (D.8). For j ∈ [ k ] , we have

Since k ! -ϵ 1 · d k -1 2 ≥ ρ , we can get

which verifies (D.7). Next, we verify that H 3 ( t ) , H 4 ( t ) , H 4 ( t +1) , H 6 ( t ) = ⇒ H 1 ( t ) . For r ∈ Ω g , given H 3 ( t ) , H 4 ( t ) and H 4 ( t +1) , we can get

For r ∈ Ω b , given H 3 ( t ) and H 6 ( t ) , we can get

Finally, we verify that { H 1 ( s ) } t s =0 = ⇒ H 2 ( t +1) , H 3 ( t +1) . Notice that ∥ w ( t +1) r ∥ 2 ≤ ∥ w (0) r ∥ 2 given { H 1 ( s ) } t s =0 , we can prove H 3 ( t +1) and H 2 ( t +1) by following the prove of (D.7) and (D.8) given Lemma D.1.

Based on Lemma D.2 and the proof of Lemma C.6, Lemma C.7, we can get the following lemmas and theorems aligning with the result of population sign GD.

Lemma D.3. Under Condition 4.2, for T = Θ( kη -1 λ -1 log( d )) ,with a probability of at least 1 -δ with respect to the randomness of the online data selection, it holds that

LemmaD.4. Under Condition 4.2, with a probability of at least 1 -2 δ with respect to the randomness in the neural network's initialization and the online data selection, trained neural network f ( W ( T ) , x ) approximates accurate classifier ( m/ 2 k +1 ) · f ( W ∗ , x ) well:

Based on Lemma D.4, we are now ready to prove our main theorem.

Theorem D.5. Under Condition 4.2, we run mini-batch SGD for T = Θ( kη -1 λ -1 log( d )) iterations. Then with probability at least 1 -2 δ with respect to the randomness of neural network initialization and the online data selection, it holds that

where γ = 0 . 25 k ! is a constant.

Proof. Given Lemma D.4, we have

According to Proposition 4.1, we can get

which completes the proof.

E Trainable Second Layer

In this section, we consider sign SGD for training the first and second layers together. In this scenario, we have the following sign SGD update rule:

where | S t | = B . For training the neural network over T = Θ( k ( ηλ ) -1 log( d )) iterations, we adopt a small learning rate for the second layer, adhering to the condition:

The network is initialized symmetrically: for every r ∈ [ m ] , initialize w (0) r ∼ Unif ( {-1 , 1 } d ) and initialize a (0) r ∼ Unif ( {-1 , 1 } ) . Under this setting, denote

Similar to the fix-second-layer case, our initial step involves estimating the approximation error between the SGD gradient and the population gradient, detailed within the lemma that follows.

Lemma E.1. With probability at least 1 -δ with respect to the randomness of online data selection, for all t ≤ T , we have for any r ∈ [ m ] and j ∈ [ d ] that

where

Proof. To prove (E.3), we denote

Initially, by invoking Hoeffding's inequality, we have the following probability bound:

Next, we establish an upper bound for the difference between the expected values of h r,j ( x , y ) and g r,j ( x , y ) :

where the first inequality follows from the Cauchy-Schwarz inequality and the second from Hoeffding's inequality. With high probability, the gradient and the truncated gradient coincide:

Combing (E.4), (E.5) and (E.6), it holds with probability at least

that

By taking γ = √ 2 log(16 B/δ ) ∥ w ( t ) r ∥ 2 and x = √ 2 kγ k -1 | a ( t ) r | log(8 /δ ) / √ B , then with probability at least 1 -δ it holds that

Then, by applying a union bound to all r ∈ [ m ] , j ∈ [ d ] and iterations t ∈ [0 , T -1] , it holds with probability at least 1 -δ that

Lemma E.2 (Stability of Second Layer Weights) . For t ≤ T = Θ( k ( ηλ ) -1 log( d )) , the magnitude of change in the second layer weights from their initial values is bounded as follows:

Consequently, the sign of each weight remains consistent over time:

Proof. Notice that by (E.2) and ˜ sign( · ) ∈ {-1 , 0 , 1 } , we can get for any t ≥ 0 that

where the second inequality is by t ≤ T , and the last inequality is by the condition η 2 ≤ 1 4 T √ πk 8 ( e + e -1 2 ) -k .

Lemma E.3. With probability at least 1 -δ with respect to the randomness of online data selection, the following sign SGD update rule holds:

Proof. We prove this by using induction. We prove the following hypotheses:

We will show that H 2 (0) , H 3 (0) , H 4 (0) and H 5 (0) are true and for any t ≥ 0 we have

H 2 ( t ) , H 4 ( t ) = ⇒ H 4 ( t +1) .

H 2 ( t ) , H 5 ( t ) = ⇒ H 5 ( t +1) , H 6 ( t ) .

H 3 ( t ) , H 4 ( t ) , H 4 ( t +1) , H 6 ( t ) = ⇒ H 1 ( t ) .

{ H 1 ( s ) } t s =0 = ⇒ H 2 ( t +1) , H 3 ( t +1) .

where which implies that

H 4 (0) and H 5 (0) are obviously true since w (0) r,j = b j for any r ∈ Ω b 1 b 2 ··· b k and j ∈ [ k ] . To prove that H 2 (0) and H 3 (0) are true, we can follow the proof of Lemma D.2 by noticing that | a (0) r | = 1 . Now, we verify that H 2 ( t ) , H 4 ( t ) = ⇒ H 4 ( t +1) . By H 2 ( t ) and sign( a ( t ) r ) = sign( a (0) r ) according to Lemma E.2, we have for any neuron r ∈ Ω b 1 b 2 ··· b k ∩ Ω g that

where the last equality is by ρ ≤ k !(1 -c ) ≤ k ! · | a ( t ) r | . H 2 ( t ) , H 5 ( t ) = ⇒ H 5 ( t +1) , H 6 ( t ) can be verified in the same way as Lemma C.4 by noticing that ρ ≤ k !(1 -c ) ≤ k ! · | a ( t ) r | . H 3 ( t ) , H 4 ( t ) , H 4 ( t +1) , H 6 ( t ) = ⇒ H 1 ( t ) can be proved by following exactly the same proof as Lemma C.4. { H 1 ( s ) } t s =0 = ⇒ H 2 ( t +1) , H 3 ( t +1) be verified in the same way as Lemma C.4 by noticing that ϵ 1 · (1 + c ) · d k -1 2 < ρ and k ! -ϵ 1 · (1 + c ) · d k -1 2 > ρ .

Based on Lemma E.2 and Lemma E.3, we can get the following lemmas and theorems aligning with the result of the fixed second-layer case.

Lemma E.4. For T = Θ( kη -1 λ -1 log( d )) ,with a probability of at least 1 -δ with respect to the randomness of the online data selection, it holds that

Lemma E.5. With a probability of at least 1 -2 δ with respect to the randomness in the neural network's initialization and the online data selection, trained neural network f ( W ( T ) , x ) approximates accurate classifier ( m/ 2 k +1 ) · f ( W ∗ , x ) well:

Proof. Let

By the proof of Lemma C.7, we can get

To prove the result, we need to estimate the difference between ˜ f ( W ( T ) , x ) and f ( W ( T ) , x ) :

where the first inequality is by triangle inequality; the second inequality is by Lemma E.2. Then, we provide upper bounds for terms I 1 and I 2 respectively. For I 1 , we have the following upper bound:

where the first inequality is by mean value theorem; the second inequality is by (C.2) and Lemma E.4; the third inequality is by Lemma B.1; the last inequality is by Lemma F.3. For I 2 , we have the following upper bound

By plugging (E.8) and (E.9) into (E.7), we can get:

Therefore, we have

where the second inequality is by Stirling's approximation, and the last equality is due to c = 1 4 √ πk 8 ( e + e -1 2 ) -k .

F Auxiliary Lemmas

We first introduce a finite difference operator with step h and order n as follows,

The following Lemma calculates the value finite difference operating on the polynomial function.

Lemma F.1. (Milne-Thomson, 2000) For f ( x ) = x n , we have that ∆ n h [ f ]( x ) = h n n ! .

Based on Lemma F.1, we have the following Lemma, which calculates the margin of good NNs defined in (3.1).

Lemma F.2. For any integer k , we have

Proof. Applying Lemma F.1 with f ( x ) = x k , n = k in Lemma F.1 gives,

where the first equality is due to the definition of finite difference operator and the last equality is due to Lemma F.1.

Lemma F.3. For any positive integer k , it holds that

Proof. We can establish the following inequality:

where the first inequality is by ( k i ) = ( k k -i ) , the second inequality is by 1 -t ≤ exp( -t ) , ∀ t ∈ R , the last inequality is by ( k i ) exp( -2 i ) > 0 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in abstract and Section 1 are supported by results in Section 4 and also the additional expeiremnts in the Appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation section is presented in page 12.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, the full set of assumptions are provided in Section 3 and the complete (and correct) proof are provided in the Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper provides detailed experiment settings in Section A, which should be sufficient for reproducing the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The experiments in this paper are based on synthetic data generated to support the theoretical findings. As such, there is no real-world dataset or code to be made openly accessible. The question of open access to data and code is not applicable in this case, as the main contributions are purely theoretical and do not rely on empirical results from specific datasets.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This paper provides detailed experiment settings in Section A, which should be sufficient for reproducing the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: : This paper provides detailed experiment settings in Section A, which include important details such as the error bars reported in Table 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The requirement of Compute Resources are stated in Section A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: As a theoretical paper, the research presented in this paper has been conducted with the highest ethical standards and is fully compliant with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work provides the theoretical understanding of generic optimization algorithm (SGD). Although there might be some potential social impacts on applications, according to the guidelines, we believe our result does not have a direct connection with these issues.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: As a theoretical paper, this work does not involve the development or release of any models or datasets, particularly those that might be considered high-risk for misuse, such as pretrained language models, image generators, or scraped datasets. Therefore, the question of safeguards for responsible release does not apply.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper is a theoretical work that does not utilize or rely on any existing assets such as code, data, or models from other sources. Therefore, the question of crediting creators or detailing licenses and terms of use for such assets does not apply.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: As this is a theoretical paper, it does not introduce or release any new assets such as datasets, code, or models. Thus, there is no need for documentation related to new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This theoretical paper does not conduct any experiments involving crowdsourcing or research with human subjects, thus there are no participants, instructions, or compensation details to report.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: As this paper is purely theoretical and does not involve any research with human subjects, there are no study participants, thus no associated risks or requirements for Institutional Review Board (IRB) approvals.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning the $k$-sparse parity problem over the $d$-dimensional boolean hypercube, using a two-layer neural network. The main result is that a specific modification of online SGD, called \"sign SGD,\" can learn the k-parity problem $n = \\tilde O(d^{k-1})$ samples and a network width of $2^{\\Theta(k)}$; the number of scalar queries to the model is $m\\cdot n \\cdot d = \\tilde O(d^k)$, and thus this algorithm matches the SQ lower bound of $d^k$.\n\nSTRENGTHS:\n- The k-sparse parity problem has attracted much recent interest in the deep learning theory community, with many prior works focusing on both lower bounds (via the SQ/CSQ framework) and learning guarantees for neural networks. This paper makes good progress by showing that the SQ lower bound of $d^k$ queries can indeed be achieved by a gradient-based algorithm.\n- The paper is well written and easy to understand, and I was able to follow the proof sketch.\n- I have skimmed the appendices, and the proofs appear to all be correct.\n\nWEAKNESSES:\n- My main issue with this paper is the choice of the Sign SGD algorithm. Unlike vanilla SGD, Sign SGD is not invariant to the choice of the basis, and is implicitly taking advantage of the structure of the problem being k-parity (where in the ground truth solution each neuron weight is either -1, 0, or 1). For example, the Sign SGD algorithm here would fail if the task was instead k-parity in an unknown basis. Many of the prior works discussed in Tables 1 and 2, however, are rotationally invariant. \n- In particular, the paper (Glasgow, 2023) had the more ambitious goal to show that vanilla SGD, with minimal modifications, can learn the 2-parity problem. The algorithm in that paper is rotationally invariant and will succeed if the target was XOR in an unknown basis. I thus find the comparisons to (Glasgow, 2023) in the paper, such as \"our result matches the best sample complexity for solving the XOR (i.e., 2-parity) problem (Glasgow, 2023)\" (line 84) to be a bit misleading.\n- On a more minor note, I also find the choice of activation $\\sigma(z) = z^k$ to be unrealistic, as this again is essentially hardcoding the fact that the target problem is $k$-parity. As seen in Tables 1 and 2, most of the prior work uses the ReLU activation.\n\nQUESTIONS:\n- Can you include more intuition in the main text for Lemma 5.3, specifically why a batch size of $\\tilde O(d^{k-1})$ is sufficient for gradient concentration?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of learning a $k$-parity function on the $d$-dimensional hypercube using SGD on a two-layer neural network. They consider a specific choice of activation ($\\sigma (x) = x^k$) and training dynamics (correlation loss, online sign SGD, fixed second layer weights) and show the following: for $m = 2^{\\Theta (k)}$ neurons, batch size $B = \\Theta (d^{k-1} \\text{polylog} (d))$, then $T = \\Theta (log (d))$ steps of online sign SGD achieves small $0/1$-test error. In particular, the total computation time scales as $\\Theta (mTB) = \\Theta (d^{k} \\text{polylog} (d))$. This is noteworthy as it matches the lower bound for learning parities from a large class of learning algorithms, namely statistical queries.\n\nSTRENGTHS:\n- The paper is well written, clear, and a very pleasant read (thank you!). Remarks and discussions provide background and comparison with other works, and help outline the contributions.\n\n- The theoretical analysis and statements are clean, and the proof quite compact and elegant, thanks to some simplifying assumptions in the model..\n\n- The fact that SGD on a (quite regular) neural network can match the runtime complexity of the SQ class of algorithms is a nice message. There is a growing literature trying to understand the power of learning with SGD on neural networks, compared to other classes of algorithms, and I believe this paper is a useful addition to the literature.\n\nWEAKNESSES:\nI am not sure the paper offers striking novel insights compared to existing literature, or that the analysis is particularly interesting in terms of proof techniques. This is the reason for my rating, but I remain open to changing it during the discussion period.\n\n- I think it was already quite widely believed that SGD on NNs can match the SQ lower bound (this is indicated in many papers on learning sparse functions on the hypercube/multi-index functions on Gaussian data). The case of $k$-parities was not written/proven before, and I think this paper makes the necessary effort to settle it. However, I believe some existing analysis can be extended to this case (e.g., [Abbe2023a], [Margalit et al., 2023]), even though they will have drawbacks specific to their choice of simplifications.\n\n- The analysis and proofs are quite nice, but it feels like a game of “there exists a specific choice of hyperparameters so that an analysis goes through”. Given that the paper doesn’t put forth particularly novel insights/phenomena, the analysis of a more vanilla training setting (non-monomial activation, more general loss, and standard GD) would have felt more substantial.\n\n- There are several questions about this setting that remain completely not understood (e.g., what happens when reusing batches) which further contributes to the impression that this work is incremental.\n\nQUESTIONS:\nAbout the conclusion on going beyond SQ: the paper “On the power of differentiable learning versus PAC and SQ learning” shows that it is indeed possible with enough machine precision: we can emulate Gaussian elimination. Of course, this is not a very interesting result, as it uses a highly non-standard network that encodes an algorithm in its architecture. There are other noise-tolerant algorithms that do slightly better than SQ, it would be interesting to show that regular architectures prevent their emulation (besides low machine precision).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper addresses the k-sparse parity problem, a fundamental one in computational complexity and algorithmic theory, by using stochastic gradient descent (SGD) on two-layer fully-connected neural networks. The authors demonstrate that SGD can efficiently solve the problem on a d-dimensional hypercube with a sample complexity that matches the established  lower bounds of Statistical Query (SQ) models.

STRENGTHS:
The paper provides a solid theoretical foundation by matching the SQ lower bound for learning k-sparse parity problems, which is a significant achievement in this area. Given the recent interest in SQ and CSA bounds and the amount of works on single and multi-index models folllowing the work of Abbe et al, I find these results pertinent.

WEAKNESSES:
The results, both in their scope and sytle, are more aligned with theoretical computer science (CS) than machine learning (ML). While the mathematical contributions are significant, the immediate applicability to practical ML problems is very limited.

QUESTIONS:
A variant of the parity problem that is not on the hypercube would be if one uses y = sign(z1 z2 z3) with z_mu = x*w_mu, with w_mu a hidden direction on the hypercube. I was wondering how different would the SG bounds and the results of the algorithm be in this case.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers learning k-sparse parities with two layer neural networks trained with stochastic signed gradient descent, where the second layer is fixed at initialization and the first layer is trained. It is shown that with the activation $\\sigma(z) = z^k$, a network with width $2^{\\Theta(k)}$ can solve the k-sparse parity problem with $\\tilde{O}(d^{k-1})$ samples in $O(\\log d)$ iterations, almost matching the Statistical Query lower bound for this problem in terms of the number of queries.\n\nSTRENGTHS:\nThe theoretical analysis of the paper is conceptually simple and clean, and there is sufficient discussion on the intuition behind the proof techniques in the main text, easily conveying the ideas to the reader.\n\nWEAKNESSES:\nWhile the goal of this work is to understand the training dynamics of modern neural networks, given the use of a polynomial activation and signed gradients, it is not clear to what extent the intuitions from this work extend to more mainstream algorithms, such as ReLU networks trained with SGD. \n\nFurther, it is possible to more explicitly discuss some limitations of the work. For example, it appears that the algorithm is not rotationally invariant, therefore is using the knowledge of the coordinate system, in comparison with the work of Glasgow (2023) which does not require this knowledge (for the simpler XOR problem). It might also give readers a more accurate picture if “signed SGD” is used in the title and abstract instead of “SGD”.\n\nQUESTIONS:\n1. Is there a way to trade off compute (iterations/neurons) against samples in upper bounds while maintaining the optimal number of queries?\n\n2. The authors mention non-i.i.d. data as room for future research. There are perhaps even more accessible settings to be investigated. For example, can these results be extended to non-isotropic data, similar to the setting of [1]?\n\n3. The summation notation in Line 183 might need clarification. I might have missed this, but I didn't see the definition of $s$ used here.\n\n\n[1] Nitanda et al. \"Improved statistical and computational complexity of the mean-field Langevin dynamics under structured data.\" ICLR 2024.\n\nLIMITATIONS:\nSome limitations could be more explicitly discussed as described in the “Weaknesses” section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}