Title: Pareto Frontiers in Deep Feature Learning: Data, Compute, Width, and Luck
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep learning

feature learning

parity

grokking

lottery tickets

scaling
Abstract: In modern deep learning, algorithmic choices (such as width, depth, and learning rate) are known to modulate nuanced resource tradeoffs. This work investigates how these complexities necessarily arise for feature learning in the presence of computational-statistical gaps. We begin by considering offline sparse parity learning, a supervised classification problem which admits a statistical query lower bound for gradient-based training of a multilayer perceptron. This lower bound can be interpreted as a *multi-resource tradeoff frontier*: 
successful learning can only occur if one is sufficiently rich (large model), knowledgeable (large dataset), patient (many training iterations), or lucky (many random guesses). We show, theoretically and experimentally, that sparse initialization and increasing network width yield significant improvements in sample efficiency in this setting. Here, width plays the role of parallel search: it amplifies the probability of finding "lottery ticket" neurons, which learn sparse features more sample-efficiently. Finally, we show that the synthetic sparse parity task can be useful as a proxy for real problems requiring axis-aligned feature learning. We demonstrate improved sample efficiency on tabular classification benchmarks by using wide, sparsely-initialized MLP models; these networks sometimes outperform tuned random forests.

Pareto Frontiers in Deep Feature Learning: Data, Compute, Width, and Luck

Abstract

In modern deep learning, algorithmic choices (such as width, depth, and learning rate) are known to modulate nuanced resource tradeoffs. This work investigates how these complexities necessarily arise for feature learning in the presence of computational-statistical gaps. We begin by considering offline sparse parity learning, a supervised classification problem which admits a statistical query lower bound for gradient-based training of a multilayer perceptron. This lower bound can be interpreted as a multi-resource tradeoff frontier : successful learning can only occur if one is sufficiently rich (large model), knowledgeable (large dataset), patient (many training iterations), or lucky (many random guesses). We show, theoretically and experimentally, that sparse initialization and increasing network width yield significant improvements in sample efficiency in this setting. Here, width plays the role of parallel search: it amplifies the probability of finding 'lottery ticket' neurons, which learn sparse features more sample-efficiently. Finally, we show that the synthetic sparse parity task can be useful as a proxy for real problems requiring axis-aligned feature learning. We demonstrate improved sample efficiency on tabular classification benchmarks by using wide, sparsely-initialized MLP models; these networks sometimes outperform tuned random forests.

1 Introduction

Algorithm design in deep learning can appear to be more like 'hacking' than an engineering practice. Numerous architectural choices and training heuristics can affect various performance criteria and resource costs in unpredictable ways. Moreover, it is understood that these multifarious hyperparameters all interact with each other; as a result, the task of finding the 'best' deep learning algorithm for a particular scenario is foremost empirically-driven. When this delicate balance of considerations is achieved (i.e. when deep learning works well), learning is enabled by phenomena which cannot be explained by statistics or optimization in isolation. It is natural to ask: is this heterogeneity of methods and mechanisms necessary?

This work studies a single synthetic binary classification task in which the above complications are recognizable, and, in fact, provable . This is the problem of offline (i.e. small-sample) sparse parity learning: identify a k -way multiplicative interaction between n Boolean variables, given m random examples. We begin by interpreting the standard statistical query lower bound for this problem as a multi-resource tradeoff frontier for deep learning, balancing between the heterogeneous resources of dataset size, network size, number of iterations, and success probability. We show that in different regimes of simultaneous resource constraints (data, parallel computation, sequential computation,

10 3

10 4

10 5

10 6

dataset size

m

10 1

10 2

10 3

10 4

10 5

MLP width

r

76

78

96

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

2

4

6

10

18

12

26

50

56

82

96

28

98

40

56

98

16

40

66

88

32

50

60

82

92

38

52

70

84

40

56

86

46

58

92

98

50

64

94

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

% of converged runs (n=100, k=3, ReLU MLP, unif. init)

0.0

0.5

r

=3e4,

m

=1e3

r

=3e4,

m

=3e3

r

=3e4,

m

=1e4

r

=3e4,

m

=3e4

r

=3e4,

m

=1e5

r

=3e4,

m

=3e5

0.0

0.5

r

=3e3,

m

=1e3

r

=3e3,

m

=3e3

r

=3e3,

m

=1e4

r

=3e3,

m

=3e4

r

=3e3,

m

=1e5

r

=3e3,

m

=3e5

1

10 5

0.0

0.5

r

=300,

m

=1e3

1

10 5

r

=300,

m

=3e3

1

10 5

r

=300,

m

=1e4

1

10 5

r

=300,

m

=3e4

1

10 5

r

=300,

m

=1e5

1

10 5

r

=300,

m

=3e5

Example train & test error curves

training iterations

and random trials), the standard algorithmic choices in deep learning can succeed by diverse and entangled mechanisms. Specifically, our contributions are as follows:

Multi-resource lower and upper bounds. We formulate a 'data × width × time × luck' lower bound for offline sparse parity learning with feedforward neural nets (Theorem 3). This barrier arises from the classic statistical query (SQ) lower bound for this problem. We show that under different resource constraints, the tractability of learning can be 'bought' with varied mixtures of these resources. In particular, in Theorems 4 and 5 we prove that by tuning the width and initialization scheme, we can populate this frontier with a spectrum of successful models ranging from narrow networks requiring many training samples to sample-efficient networks requiring many neurons, as summarized by the following informal theorem statement:

Informal Theorem 1. Consider the problem of learning ( n, k ) -parities from m i.i.d. samples with a 2 -layer widthr ReLU MLP, whose first-layer neurons are initialized with sparsity s . After O n (1) steps of gradient descent, the k relevant coordinates are identified with probability 0.99 when (1) s > Ω( k ) , r = Θ(( n/s ) k ) and m = Θ( n 2 ( s/k ) k -1 ) , and when (2) s < k , r = Θ(( n/k ) s ) and m = Θ(( n/k ) k -s -1 ) .

Intuitively, this analysis reveals a feature learning mechanism by which overparameterization (i.e. large network width) plays a role of parallel search over randomized subnetworks. Each individual hidden-layer neuron has its own sample complexity for identifying the relevant coordinates, based on its Fourier gap (Barak et al., 2022) at initialization. Trained with parallel gradient updates, the full network implicitly acts as an ensemble model over these neurons, whose overall sample complexity is determined by the 'winning lottery tickets' (Frankle and Carbin, 2018) (i.e. the lucky neurons initialized to have the lowest sample complexity). This departs significantly from the neural tangent kernel (Jacot et al., 2018) regime of function approximation with wide networks, in which overparameterization removes data-dependent feature selection (rather than parallelizing it across subnetworks).

Empirical study of neural nets' statistical thresholds for sparse parity learning. We corroborate the theoretical analysis with a systematic empirical study of offline sparse parity learning using SGD on MLPs, demonstrating some of the (perhaps) counterintuitive effects of width, data, and initialization. For example, Figure 1 highlights our empirical investigation into the interactions between data, width, and success probability. The left figure shows the fractions of successful training runs as a function of dataset size (x-axis) and width (y-axis). Roughly, we see a 'success frontier', where having a larger width can be traded off with smaller sample sizes. The right figure depicts some training curves (for various widths and sample sizes). Grokking (Power et al., 2022; Liu et al., 2022) (discontinuous and delayed generalization behavior induced by optimization dynamics) is evident in some of these figures.

'Parity2real' transfer of algorithmic improvements. It is often observed that deep learning methods underperform tree-based methods (e.g. random forests) on tabular datasets, particularly those where the target function depends on a few of the input features in a potentially non-smooth manner; see Grinsztajn et al. (2022) for a recent discussion. Motivated by our findings in the synthetic parity setting (and the observation that the sparse parity task possesses a number of the same properties of these problematic real-world datasets), we then turn to experimentally determine the extent to which our findings also hold for real tabular data. We evaluate MLPs of various depths and initialization sparsities on 16 tabular classification tasks, which were standardized by Grinsztajn et al. (2022) to compare neural vs. tree-based methods. Figure 3 shows that wider networks and sparse initialization yield improved performance, as in the parity setting. In some cases, our MLPs outperform tuned random forests.

1.1 Related work

In the nascent empirical science of large-scale deep learning, scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) have been shown to extrapolate model performance with remarkable consistency, revealing flexible tradeoffs and Pareto frontiers between the heterogeneous resources of data and computation. The present work reveals that in the simple synthetic setting of parity learning, the same intricacies can be studied theoretically and experimentally. In particular, viewing model size × training iterations × random restarts as a single 'total FLOPs' resource, our study explains why data × compute can be a necessary and sufficient resource, through the lens of SQ complexity.

Analyses of deep feature learning. Formally characterizing the representation learning mechanisms of neural networks is a core research program of deep learning theory. Many recent works have analyzed gradient-based feature learning (Wei et al., 2019; Barak et al., 2022; Zhenmei et al., 2022; Abbe et al., 2022; Damian et al., 2022; Telgarsky, 2022), escaping the 'lazy' neural tangent kernel (NTK) regime (Jacot et al., 2018; Chizat et al., 2019), in which features are fixed at initialization.

Learning parities with neural networks. The XOR function has been studied as an elementary challenging example since the dawn of artificial neural networks (Minsky and Papert, 1969), and has been revisited at various times: e.g. neural cryptography (Rosen-Zvi et al., 2002); learning interactions via hints in the input distribution (Daniely and Malach, 2020); a hard case for selfattention architectures (Hahn, 2020). Closest to this work, Barak et al. (2022) find that in the case of online (infinite-data) parity learning, SGD provides a feature learning signal for a single neuron , and can thus converge at a near-optimal computational rate for non-overparameterized networks. They note computational-statistical tradeoffs and grokking in the offline setting, which we address systematically. Merrill et al. (2023) examine the same problem setting empirically, investigating a mechanism of competing sparse and dense sub-networks. Abbe et al. (2023) provide evidence that the time complexity required by an MLP to learn an arbitrary sparse Boolean function is governed by the largest 'leap' in the staircase of its monomials, each of which is a sparse parity. Telgarsky (2022) gives a margin-based analysis of gradient flow on a two-layer neural network that achieves improved sample complexity ( ˜ O ( n/ϵ ) ) for the 2-sparse parity problem, at the cost of exponential width.

Neural nets and axis-aligned tabular data. Decision tree ensembles such as random forests (Breiman, 2001) and XGBoost (Chen and Guestrin, 2016) remain more popular among practitioners than neural networks on tabular data (Kaggle, 2021), despite many recent attempts to design specialized deep learning methods (Borisov et al., 2022). Some of these employ sparse networks (Yang et al., 2022b; Lutz et al., 2022) similar to those considered in our theory and experiments.

We refer the reader to Appendix A for additional related work.

2 Background

2.1 Parities, and parity learning algorithms

A light bulb is controlled by k out of n binary switches; each of the k influential switches can toggle the light bulb's state from any configuration. The task in parity learning is to identify the subset of k important switches, given access to m i.i.d. uniform samples of the state of all n switches. Formally, for any 1 ≤ k ≤ n and S ⊆ [ n ] such that | S | = k , the parity function χ S : {± 1 } n → ± 1 is

defined as χ S ( x 1: n ) := ∏ i ∈ S x i . 1 The ( n, k ) -parity learning problem is to identify S from samples ( x ∼ Unif( {± 1 } n , y = χ S ( x ))) ; i.e. output a classifier with 100 % accuracy on this distribution, without prior knowledge of S .

This problem has a rich history in theoretical computer science, information theory, and cryptography. There are several pertinent ways to think about the fundamental role of parity learning:

(i) Monomial basis elements: A k -sparse parity is a degreek monomial, and thus the analogue of a Fourier coefficient with 'frequency' k in the harmonic analysis of Boolean functions (O'Donnell, 2014). Parities form an important basis for polynomial learning algorithms (e.g. Andoni et al. (2014)).

(ii) Computational hardness: There is a widely conjectured computational-statistical gap for this problem (Applebaum et al., 2009, 2010), which has been proven in restricted models such as SQ (Kearns, 1998) and streaming (Kol et al., 2017) algorithms. The statistical limit is Θ(log( number of possibilities for S )) = Θ( k log n ) samples, but the amount of computation needed (for an algorithm that can tolerate an O (1) fraction of noise) is believed to scale as Ω( n ck ) for some constant c > 0 - i.e., there is no significant improvement over trying all subsets.

(iii) Feature learning: This setting captures the learning of a concept which depends jointly on multiple attributes of the inputs, where the lower-order interactions (i.e. correlations with degreek ′ < k parities) give no information about S . Intuitively, samples from the sparse parity distribution look like random noise until the learner has identified the sparse interaction. This notion of feature learning complexity is also captured by the more general information exponent (Ben Arous et al., 2021).

2.2 Notation for neural networks

For single-output regression, a 2-layer multi-layer perceptron (MLP) is a function class, parameterized by a matrix W ∈ R r × n , vectors v, b ∈ R r and scalar β ∈ R , defined by

where σ is an activation function, usually a scalar function applied identically to each input. For rows w i of W , the intermediate variables σ ( w ⊤ i x + b i ) are thought of as hidden layer activations or neurons . The number of parallel neurons is often called the width .

3 Theory

In this section we theoretically study the interactions between data, width, time, and luck on the sparse parity problem. We begin by rehashing Statistical Query (SQ) lower bounds for parity learning in the context of gradient-based optimization, showing that without sufficient resources sparse parities cannot be learned. Then, we prove upper bounds showing that parity learning is possible with correctly scaling either width (keeping sample size small), sample size (keeping width small), or a mixture of the two.

Statistical query algorithms. A seminal work by Kearns (1998) introduced the statistical query (SQ) algorithm framework, which provides a means of analyzing noise-tolerant algorithms. Unlike traditional learning algorithms, SQ algorithms lack access to individual examples but can instead make queries to an SQ oracle , which responds with noisy estimates of the queries over the population. Notably, many common learning algorithms, including noisy variants of gradient descent, can be implemented within this framework. While SQ learning offers robust guarantees for learning in the presence of noise, there exist certain problems that are learnable from examples but not efficiently learnable from statistical queries (Blum et al., 1994, 2003). One notable example of such a problem is the parity learning problem, which possesses an SQ lower bound. This lower bound can be leveraged to demonstrate the computational hardness of learning parities with gradient-based algorithms (e.g., Shalev-Shwartz et al. (2017)).

3.1 Lower bound: a multi-resource hardness frontier

We show a version of the SQ lower bound in this section. Assume we optimize some model h θ , where θ ∈ R r (i.e., r parameters). Let ℓ be some loss function satisfying: ℓ ′ (ˆ y, y ) = -y + ℓ 0 (ˆ y ) ; for example, one can choose ℓ 2 (ˆ y, y ) = 1 2 ( y -ˆ y ) 2 . Fix some hypothesis h , target f , a sample S ⊆ {± 1 } n and distribution D over {± 1 } n . We denote the empirical loss by L S ( h, f ) = 1 |S| ∑ x ∈S ℓ ( h ( x ) , f ( x )) and the population loss by L D ( h, f ) := E x ∼D [ ℓ ( h ( x ) , f ( x ))] . We consider SGD updates of the form:

for some sample S t , step size η t , regularizer R ( · ) , and adversarial noise ξ t ∈ [ -τ, τ ] r . For normalization, we assume that ∥∇ h θ ( x ) ∥ ∞ ≤ 1 for all θ and x ∈ X .

Denote by 0 the constant function, mapping all inputs to 0 . Let θ ⋆ t be the following trajectory of SGD that is independent of the target. Define θ ⋆ t recursively s.t. θ ⋆ 0 = θ 0 and

Assumption 2 (Bounded error for gradient estimator) . For all t , suppose that

Remark: If m = ˜ O (1 /τ 2 ) , 2 Assumption 2 is satisfied w.h.p. for: 1) S t ∼ D m ('online' SGD), 2) S t = S for some S ∼ D m ('offline' GD) and 3) S t is a batch of size m sampled uniformly at random from S , where S ∼ D M and M ≥ m ('offline' SGD). Indeed, in all these cases we have E S t [ ∇ L S t ( h θ ⋆ t , 0 ) ] = ∇ L D ( h θ ⋆ t , 0 ) , and the above follows from standard concentration bounds.

The lower bound in this section uses standard statistical query arguments to show that gradient-based algorithms, without sufficient resources , will fail to learn ( n, k ) -parities. We therefore start by stating the four types of resources that impact learnability:

The number of parameters r - equivalently, the number of parallel 'queries'.

The number of gradient updates T - i.e. the serial running time of the training algorithm.

The gradient precision τ - i.e. how close the empirical gradient is to the population gradient. As discussed above, ˜ O (1 /τ 2 ) samples suffice to obtain such an estimator.

The probability of success δ .

The following theorem ties these four resources together, showing that without a sufficient allocation of these resources, gradient descent will fail to learn:

Proposition 3. Assume that θ 0 is randomly drawn from some distribution. For every r, T, δ, τ > 0 , if rT τ 2 δ ≤ 1 2 ( n k ) , then there exists some ( n, k ) -parity s.t. with probability at least 1 -δ over the choice of θ 0 , the first T iterates of SGD are statistically independent of the target function.

The proof follows standard SQ lower bound arguments (Kearns, 1998; Feldman, 2008), and for completeness is given in Appendix B. The core idea of the proof is the observation that any gradient step has very little correlation (roughly n -k ) with most sparse-parity functions, and so there exists a parity function that has small correlation with all steps. In this case, the noise can force the gradient iterates to follow the trajectory θ ⋆ 1 , . . . , θ ⋆ T , which is independent of the true target. We note that, while the focus of this paper is on sparse parities, similar analysis applies for a broader class of functions that are characterized by large SQ dimension (see Blum et al. (1994)).

Observe that there are various ways for an algorithm to escape the lower bound of Theorem 3. We can scale a single resource with ( n k ) , keeping the rest small, e.g. by training a network of size n k , or using a sample size of size n k . Crucially, we note the possibility of interpolating between these extremes: one can spread this homogenized 'cost' across multiple resources, by (e.g.) training a network of size n k/ 2 using n k/ 2 samples. In the next section, we show how neural networks can be tailored to solve the task in these interpolated resource scaling regimes.

3.2 Upper bounds: many ways to trade off the terms

As a warmup, we discuss some simple SQ algorithms that succeed in learning parities by properly scaling the different resources. First, consider deterministic exhaustive search, which computes the error of all possible ( n, k ) -parities, choosing the one with smallest error. This can be done with constant τ (thus, sample complexity logarithmic in n ), but takes T = Θ( n k ) time. Since querying different parities can be done in parallel, we can reduce the number of steps T by increasing the number of parallel queries r . Alternatively, it is possible to query only a randomly selected subset of parities, which reduces the overall number of queries but increases the probability of failure.

The above algorithms give a rough understanding of the frontier of algorithms that succeed at learning parities. However, at first glance, they do not seem to reflect algorithms used in deep learning, and are specialized to the parity problem. In this section, we will explore the ability of neural networks to achieve similar tradeoffs between the different resources. In particular, we focus on the interaction between sample complexity and network size, establishing learning guarantees with interpolatable mixtures of these resources.

Before introducing our main positive theoretical results, we discuss some prior theoretical results on learning with neural networks, and their limitations in the context of learning parities. Positive results on learning with neural networks can generally be classified into two categories: those that reduce the problem to convex learning of linear predictors over a predefined set of features (e.g. the NTK), and those that involve neural networks departing from the kernel regime by modifying the fixed features of the initialization, known as the feature learning regime.

Kernel regime. When neural networks trained with gradient descent stay close to their initial weights, optimization behaves like kernel regression on the neural tangent kernel (Jacot et al., 2018; Du et al., 2018): the resulting function is approximately of the form x ↦→ ⟨ ψ ( x ) , w ⟩ , where ψ is a data-independent infinite-dimensional embedding of the input, and w is some weighting of the features. However, it has been shown that the NTK (more generally, any fixed kernel) cannot achieve low ℓ 2 error on the ( n, k ) -parity problem, unless the sample complexity grows as Ω( n k ) (see Kamath et al. (2020)). Thus, no matter how we scale the network size or training time, neural networks trained in the NTK regime cannot learn parities with low sample complexity, and thus do not enjoy the flexibility of resource allocation discussed above.

Feature learning regime. Due to the limitation of neural networks trained in the kernel regime, some works study learning in the 'rich' regime, quantifying how hidden-layer features adapt to the data. Among these, Barak et al. (2022) analyze a feature learning mechanism requiring exceptionally small network width: SGD on 2-layer MLPs can solve the online sparse parity learning problem with network width independent of n (dependent only on k ), at the expense of requiring a suboptimal ( ≈ n k ) number of examples. This mechanism is Fourier gap amplification , by which SGD through a single neuron σ ( w ⊤ x ) can perform feature selection in this setting, via exploiting a small gap between the relevant and irrelevant coordinates in the population gradient. The proof of Theorem 4 below relies on a similar analysis, extended to the offline regime (i.e., multiple passes over a dataset of limited size).

3.2.1 'Data × model size' success frontier for sparsely-initialized MLPs

In this section, we analyze a 2-layer MLP with ReLU ( σ ( x ) = max(0 , x ) ) activation, trained with batch ('offline') gradient-descent over a sample S with ℓ 2 -regularized updates 3 :

We allow learning rates η , and weight decay coefficients λ to differ between layers and iterations. For simplicity, we analyze the case where no additional noise is added to each update; however, we believe that similar results can be obtained in the noisy case (e.g., using the techniques in Feldman et al. (2017)). Finally, we focus on ReLU networks with s -sparse initialization of the first layer: every weight w i has s randomly chosen coordinates set to 1 , and the rest set to 0 . Note that after initialization, all of the network's weights are allowed to move, so sparsity is not necessarily preserved during training.

Over-sparse initialization ( s > Ω( k ) ). The following theorem demonstrates a 'data × width' success frontier when learning ( n, k ) -parities with sparsely initialized ReLU MLPs at sparsity levels s > Ω( k ) .

Theorem 4. Let k be an even integer, and ϵ ∈ (0 , 1 / 2) . Assume that n ≥ Ω(1 /ϵ 2 ) . For constants c 1 , c 2 , c 3 , c 4 depending only on k , choose the following: (1) sparsity level: s ≥ c 1 /ϵ 2 , for some odd s , (2) width of the network: r = c 2 ( n/s ) k , (3) sample size: m ≥ c 3 ( s/k ) k -1 n 2 log n , and (4) number of iterations: T ≥ c 4 /ϵ 2 . Then, for every ( n, k ) -parity distribution D , with probability at least 0 . 99 over the random samples and initialization, gradient descent with these parameter settings returns a function h T s.t. L D ( h T ) ≤ ϵ .

Intuitively, by varying the sparsity parameter in Theorem 4, we obtain a family of algorithms which smoothly interpolate between the small-data/large-width and large-data/small-width regimes of tractability. First, consider a sparsity level linear in n (i.e. s = α · n ). In this case, a small network (with width r independent of the input dimension) is sufficient for solving the problem, but the sample size must be large ( Ω( n k +1 ) ) for successful learning; this recovers the result of Barak et al. (2022). At the other extreme, if the sparsity is independent of n , the sample complexity grows only as O ( n 2 log n ) 4 , but the requisite width becomes Ω( n k ) .

Proof sketch. The proof of Theorem 4 relies on establishing a Fourier anti-concentration condition, separating the relevant (i.e. indices in S ) and irrelevant weights in the initial population gradient, similarly as the main result in Barak et al. (2022). When we initialize an s -sparse neuron, there is a probability of ≳ ( s/n ) k that the subset of activated weights contains the 'correct' subset S . In this case, to detect the subset S via the Fourier gap, it is sufficient to observe s k -1 examples instead of n k -1 . Initializing the neurons more sparsely makes it less probable to draw a lucky neuron, but once we draw a lucky neuron, it requires fewer samples to find the right features. Thus, increasing the width reduces overall sample complexity, by sampling a large number of 'lottery tickets'.

Under-sparse initialization ( s < k ). The sparsity parameter can modulate similar data vs. width tradeoffs for feature learning in the 'under-sparse' regime. We provide a partial analysis for this more challenging case, showing that one step of gradient descent can recover a correct subnetwork. Appendix B.3 discusses the mathematical obstructions towards obtaining an end-to-end guarantee of global convergence.

Theorem 5. For even k, s , sparsity level s < k , network width r = O (( n/k ) s ) , and ε -perturbed 5 s -sparse random initialization scheme s.t. for every ( n, k ) -parity distribution D , with probability at least 0 . 99 over the choice of sample and initialization after one step of batch gradient descent (with gradient clipping) with sample size m = O (( n/k ) k -s -1 ) and appropriate learning rate, there is a subnetwork in the ReLU MLP that approximately computes the parity function.

Here, the sample complexity can be improved by a factor of n s , at the cost of requiring the width to be n s times larger. The proof of Theorem 5 relies on a novel analysis of improved Fourier gaps with 'partial progress': intuitively, if a neuron is randomly initialized with a subset of the relevant indices S , it only needs to identify k -s more coordinates, inheriting the improved sample complexity for the ( n -s, k -s ) -parity problem. Note that the probability of finding such a lucky neuron scales as ( n/k ) -s , which governs how wide (number of lottery tickets) the network needs to be.

Remarks on the exact-sparsity regime. Our theoretical analyses do not extend straightforwardly to the case of s = Θ( k ) . Observe that if the value of k is known, initializing a network of size r = ˜ O ( n k ) with sparsity s = k gives w.h.p. a subnetwork with good first-layer features at initialization. We believe that with a proper choice of regularization and training scheme, it is possible to show that such a network learns to select this subnetwork with low sample complexity. We leave the exact details of this construction and end-to-end proofs for future work.

Analogous results for dense initialization schemes? We believe that the principle of 'parallel search with randomized per-subnetwork sample complexities' extends to other initialization schemes

10 3

10 4

10 5

10 6

dataset size

m

10 1

10 2

10 3

10 4

10 5

MLP width

r

1000

2000

3000

4000

5000

6000

0.0

0.2

0.4

0.6

0.8

1.0

n=100 k=3, dense init

1000

2000

3000

4000

5000

6000

0.0

0.2

0.4

0.6

0.8

1.0

n=100 k=4, dense init

1000

2000

3000

4000

5000

6000

0.0

0.2

0.4

0.6

0.8

1.0

n=100 k=3, sparse init

1000

2000

3000

4000

5000

6000

0.0

0.2

0.4

0.6

0.8

1.0

n=100 k=4, sparse init

r

=100000

r

=30000

r

=10000

r

=3000

r

=1000

r

=300

Empirical statistical threshold vs. width+init

fraction of converged runs

dataset size

m

(linear scale)

10 3

10 4

10 5

10 6

10 3

10 4

10 5

n=100 k=3, dense init

10 3

10 4

10 5

10 6

10 3

10 4

10 5

n=100 k=3, sparse init

r

=100000

r

=300

Computation time vs. width+init

median # iterations to convergence

dataset size

m

(log scale)

(including those more commonly used in practice), and leads to analogous success frontiers. To support this, our experiments investigate both sparse and uniform initialization, with qualitatively similar findings. For dense initializations, the mathematical challenge lies in analyzing the Fourier anti-concentration of general halfspaces (see the discussion and experiments in Appendix C.1 of (Barak et al., 2022)). The axis-aligned inductive biases imparted by sparse initialization may also be of independent practical interest.

4 Experiments

A high-level takeaway from Section 3 is that when a learning problem is computationally difficult but statistically easy, a complex frontier of resource tradeoffs can emerge; moreover, it is possible to interpolate between extremes along this frontier using ubiquitous algorithmic choices in deep learning, such as overparameterization, random initialization, and weight decay. In this section, we explore the nature of the frontier with an empirical lens-first with end-to-end sparse parity learning, then with natural tabular datasets.

4.1 Empirical Pareto frontiers for offline sparse parity learning

We launch a large-scale ( ∼ 200K GPU training runs) exploration of resource tradeoffs when training neural networks to solve the offline sparse parity problem. While Section 3 analyzes idealized variants of SGD on MLPs which interpolate along the problem's resource tradeoff frontier, in this section we ask whether the same can be observed end-to-end with standard training and regularization.

On various instances of the ( n, k ) -sparse parity learning problem, we train a 2-layer MLP with identical hyperparameters, varying the network width r ∈ { 10 , 30 , 100 , . . . , 10 5 } and the dataset size m ∈ { 100 , 200 , 300 , 600 , 1000 , . . . , 10 6 } . Alongside standard algorithmic choices, we consider one non-standard augmentation of SGD: the under-sparse initialization scheme from Section 3.2.1; we have proven that these give rise to 'lottery ticket' neurons which learn the influential coordinates more sample-efficiently. Figure 1 (in the introduction) and Figure 2 illustrate our findings at a high level; details and additional discussion are in Appendix C.1). We list our key findings below:

A 'success frontier': large width can compensate for small datasets. We observe convergence and perfect generalization when m ≪ n k . In such regimes, which are far outside the

online setting considered by Barak et al. (2022), high-probability sample-efficient learning is enabled by large width. This can be seen in Figure 1 (left), and analogous plots in Appendix C.1.

Width is monotonically beneficial, and buys data, time, and luck. In this setting, increasing the model size yields exclusively positive effects on success probability, sample efficiency, and the number of serial steps to convergence (see Figure 2). This is a striking example where end-to-end generalization behavior runs opposite to uniform convergence-based upper bounds, which predict that enlarging the model's capacity worsens generalization.

Sparse axis-aligned initialization buys data, time, and luck. Used in conjunction with a wide network, we observe that a sparse, axis-aligned initialization scheme yields strong improvements on all of these axes; see Figure 2 (bottom row). In smaller hyperparameter sweeps, we find that s = 2 (i.e. initialize every hidden-layer neuron with a random 2 -hot weight vector) works best.

Intriguing effects of dataset size. As we vary the sample size m , we note two interesting phenomena; see Figure 2 (right). The first is grokking (Power et al., 2022), which has been previously documented in this setting (Barak et al., 2022; Merrill et al., 2023). This entails a data vs. time tradeoff: for small m where learning is marginally feasible, optimization requires significantly more training iterations T . Our second observation is a 'sample-wise double descent' (Nakkiran et al., 2021): success probability and convergence times can worsen with increasing data. Both of these effects are also evident in Figure 1).

Lottery ticket neurons. The above findings are consistent with the viewpoint taken by the theoretical analysis, where randomly-initialized SGD plays the role of parallel search , and a large width increases the number of random subnetworks available for this process-in particular, the 'winning lottery ticket' neurons, for which feature learning occurs more sample-efficiently. To provide further evidence that sparse subnetworks are responsible for learning the parities, we perform a smaller-scale study of network prunability in Appendix C.2.

4.2 Sample-efficient deep learning on natural tabular datasets

Sparse parity learning is a toy problem, in that it is defined by an idealized distribution, averting the ambiguities inherent in reasoning about real-life datasets. However, due to its provable hardness (Theorem 3, as well as the discussion in Section 2.1), it is a maximally hard toy problem in a rigorous sense 6 . In this section, we perform a preliminary investigation of how the empirical and algorithmic insights gleaned from Section 4.1 can be transferred to more realistic learning scenarios.

To this end, we use the benchmark assembled by Grinsztajn et al. (2022), a work which specifically investigates the performance gap between neural networks and tree-based classifiers (e.g. random forests, gradient-boosted trees), and includes a standardized suite of 16 classification benchmarks with numerical input features. The authors identify three common aspects of tabular 7 tasks which present difficulties for neural networks, especially vanilla MLPs:

(i) The feature spaces are not rotationally invariant. In state-of-the-art deep learning, MLPs are often tasked with function representation in rotation-invariant domains (token embedding spaces, convolutional channels, etc.).

(ii) Many of the features are uninformative. In order to generalize effectively, especially from a limited amount of data, it is essential to avoid overfitting to these features.

(iii) There are meaningful high-frequency/non-smooth patterns in the target function. Combined with property (i), decision tree-based methods (which typically split on axis-aligned features) can appear to have the ideal inductive bias for tabular modalities of data.

Noting that the sparse parity task possesses all three of the above qualities, we conduct a preliminary investigation on whether our empirical findings in the synthetic case carry over to natural tabular data. In order to study the impact of algorithmic choices (mainly width and sparse initialization) on sample efficiency, we create low-data problem instances by subsampling varying fractions of each dataset for training. Figure 3 provides a selection of our results. We note the following empirical findings, which are the tabular data counterparts of results (2) and (3) in Section 4.1:

10 0

10

1

10

2

10

3

10

4

0.3

0.4

0.5

dense init

Task 1

10 0

10

1

10

2

10

3

10

4

0.35

0.40

0.45

0.50

Task 10

10 0

10

1

10

2

10

3

10

4

0.3

0.4

0.5

Task 13

10 0

10

1

10

2

10

3

10

4

0.2

0.3

0.4

0.5

Task 14

10 0

10

1

10

2

10

3

10

4

0.3

0.4

0.5

sparse init

Task 1

10 0

10

1

10

2

10

3

10

4

0.35

0.40

0.45

0.50

Task 10

10 0

10

1

10

2

10

3

10

4

0.3

0.4

0.5

Task 13

10 0

10

1

10

2

10

3

10

4

0.2

0.3

0.4

0.5

Task 14

MLPs (varying width & init sparsity) on tabular data

fraction of full training data

test error (median of 100 runs)

dense init, r=100

dense init, r=300

dense init, r=1000

dense init, r=3000

dense init, r=10000

dense init, r=30000

sparse init, r=100

sparse init, r=300

sparse init, r=1000

sparse init, r=3000

sparse init, r=10000

sparse init, r=30000

(

see above

/

below

)

best MLP

best RF

[GOV22] best MLP

[GOV22] best RF

(2T) Wide networks generalize on small tabular datasets. Like in the synthetic experiments, width yields nearly monotonic end-to-end benefits for learning. This suggests that the 'parallel search + pruning' mechanisms analyzed in our paper are also at play in these settings. In some (but not all) cases, these MLPs perform competitively with tuned tree-based classifiers.

(3T) Sparse axis-aligned initialization sometimes improves end-to-end performance. This effect is especially pronounced on datasets which are downsampled to be orders of magnitude smaller. We believe that this class of drop-in replacements for standard initialization merits further investigation, and may contribute to closing the remaining performance gap between deep learning and tree ensembles on small tabular datasets.

5 Conclusion

We have presented a theoretical and empirical study of offline sparse parity learning with neural networks; this is a provably hard problem which admits a multi-resource lower bound in the SQ model. We have shown that the lower bound can be surmounted using varied mixtures of these resources, which correspond to natural algorithmic choices and scaling axes in deep learning. By investigating how these choices influence the empirical 'success frontier' for this hard synthetic problem, we have arrived at some promising improvements for MLP models of tabular data (namely, large width and sparse initialization). These preliminary experiments suggest that a more intensive, exhaustive study of algorithmic improvements for MLPs on tabular data has a chance of reaping significant rewards, perhaps even surpassing the performance of decision tree ensembles.

Broader impacts and limitations. The nature of this work is foundational; the aim of our theoretical and empirical investigations is to contribute to the fundamental understanding of neural feature learning, and the influences of scaling relevant resources. A key limitation is that our benchmarks on tabular data are only preliminary; it is a significant and perennial methodological challenge to devise fair and comprehensive comparisons between neural networks and tree-based learning paradigms.

References

Abbe, E., Adsera, E. B., and Misiakiewicz, T. (2022). The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory , pages 4782-4887. PMLR.

Abbe, E., Boix-Adsera, E., and Misiakiewicz, T. (2023). Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. arXiv preprint arXiv:2302.11055 .

Andoni, A., Panigrahy, R., Valiant, G., and Zhang, L. (2014). Learning polynomials with neural networks. In International Conference on Machine Learning , pages 1908-1916. PMLR.

Applebaum, B., Barak, B., and Wigderson, A. (2010). Public-key cryptography from different assumptions. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 171-180.

Applebaum, B., Cash, D., Peikert, C., and Sahai, A. (2009). Fast cryptographic primitives and circular-secure encryption based on hard learning problems. In Annual International Cryptology Conference , pages 595-618. Springer.

Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, G. (2022). High-dimensional asymptotics of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems , 35:37932-37946.

Bachmann, G., Anagnostidis, S., and Hofmann, T. (2023). Scaling mlps: A tale of inductive bias. arXiv preprint arXiv:2306.13575 .

Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. (2021). Explaining neural scaling laws. arXiv preprint arXiv:2102.06701 .

Barak, B., Edelman, B., Goel, S., Kakade, S., Malach, E., and Zhang, C. (2022). Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems , 35:21750-21764.

Ben Arous, G., Gheissari, R., and Jagannath, A. (2021). Online stochastic gradient descent on non-convex losses from high-dimensional inference. The Journal of Machine Learning Research , 22(1):4788-4838.

Bietti, A., Bruna, J., Sanford, C., and Song, M. J. (2022). Learning single-index models with shallow neural networks. Advances in Neural Information Processing Systems , 35:9768-9783.

Blum, A., Furst, M., Jackson, J., Kearns, M., Mansour, Y., and Rudich, S. (1994). Weakly learning dnf and characterizing statistical query learning using fourier analysis. In Proceedings of the twenty-sixth annual ACM symposium on Theory of computing , pages 253-262.

Blum, A., Kalai, A., and Wasserman, H. (2003). Noise-tolerant learning, the parity problem, and the statistical query model. Journal of the ACM (JACM) , 50(4):506-519.

Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., and Kasneci, G. (2022). Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems .

Breiman, L. (2001). Random forests. Machine learning , 45:5-32.

Chen, S. and Meka, R. (2020). Learning polynomials in few relevant dimensions. In Conference on Learning Theory , pages 1161-1227. PMLR.

Chen, T. and Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , pages 785-794.

Chizat, L., Oyallon, E., and Bach, F. (2019). On lazy training in differentiable programming. Advances in neural information processing systems , 32.

Damian, A., Lee, J., and Soltanolkotabi, M. (2022). Neural networks can learn representations with gradient descent. In Conference on Learning Theory , pages 5413-5452. PMLR.

Damian, A., Nichani, E., Ge, R., and Lee, J. D. (2023). Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models. arXiv preprint arXiv:2305.10633 .

Daniely, A. and Malach, E. (2020). Learning parities with neural networks. Advances in Neural Information Processing Systems , 33:20356-20365.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 .

Du, S. S., Zhai, X., Poczos, B., and Singh, A. (2018). Gradient descent provably optimizes overparameterized neural networks. arXiv preprint arXiv:1810.02054 .

Feldman, V. (2008). Evolvability from learning algorithms. In Proceedings of the fortieth annual ACM symposium on Theory of computing , pages 619-628.

Feldman, V., Guzman, C., and Vempala, S. (2017). Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1265-1277. SIAM.

Frankle, J. and Carbin, M. (2018). The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 .

Frei, S., Chatterji, N. S., and Bartlett, P. L. (2022). Random feature amplification: Feature learning and generalization in neural networks. arXiv preprint arXiv:2202.07626 .

Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. (2021). Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems , 34:18932-18943.

Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems , volume 35, pages 507-520. Curran Associates, Inc.

Hahn, M. (2020). Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171.

Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020). Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701 .

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 .

Hutter, M. (2021). Learning curve theory. arXiv preprint arXiv:2102.04074 .

Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31.

Kaggle (2021). State of data science and machine learning 2021.

Kamath, P., Montasser, O., and Srebro, N. (2020). Approximate is good enough: Probabilistic variants of dimensional and margin complexity. In Conference on Learning Theory , pages 2236-2262. PMLR.

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 .

Kearns, M. (1998). Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45(6):983-1006.

Kol, G., Raz, R., and Tal, A. (2017). Time-space hardness of learning sparse parities. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing , pages 1067-1080.

Liu, Z., Michaud, E. J., and Tegmark, M. (2022). Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117 .

Lutz, P., Arnould, L., Boyer, C., and Scornet, E. (2022). Sparse tree-based initialization for neural networks. arXiv preprint arXiv:2209.15283 .

Malach, E., Kamath, P., Abbe, E., and Srebro, N. (2021). Quantifying the benefit of using differentiable learning over tangent kernels. In International Conference on Machine Learning , pages 7379-7389. PMLR.

Merrill, W., Tsilivis, N., and Shukla, A. (2023). A tale of two circuits: Grokking as competition of sparse and dense subnetworks. arXiv preprint arXiv:2303.11873 .

Michaud, E. J., Liu, Z., Girit, U., and Tegmark, M. (2023). The quantization model of neural scaling. arXiv preprint arXiv:2303.13506 .

Minsky, M. and Papert, S. (1969). Perceptrons: an introduction to computational geometry . MIT Press.

Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003.

O'Donnell, R. (2014). Analysis of Boolean functions . Cambridge University Press.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). PyTorch: An imperative style, highperformance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830.

Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 .

Rosen-Zvi, M., Klein, E., Kanter, I., and Kinzel, W. (2002). Mutual learning in a tree parity machine and its application to cryptography. Physical Review E , 66(6):066135.

Shalev-Shwartz, S. and Ben-David, S. (2014). Understanding machine learning: From theory to algorithms . Cambridge university press.

Shalev-Shwartz, S., Shamir, O., and Shammah, S. (2017). Failures of gradient-based deep learning. In International Conference on Machine Learning , pages 3067-3075. PMLR.

Shamir, O. and Zhang, T. (2013). Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In International conference on machine learning , pages 71-79. PMLR.

Shi, Z., Wei, J., and Liang, Y. (2021). A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features. In International Conference on Learning Representations .

Telgarsky, M. (2022). Feature selection with gradient descent on two-layer networks in low-rotation regimes. arXiv preprint arXiv:2208.02789 .

Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. (2021). MLP-Mixer: An all-MLP architecture for vision. Advances in neural information processing systems , 34:24261-24272.

Wei, C., Lee, J. D., Liu, Q., and Ma, T. (2019). Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Systems , 32.

Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. (2022a). Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466 .

Yang, J., Lindenbaum, O., and Kluger, Y. (2022b). Locally sparse neural networks for tabular biomedical data. In International Conference on Machine Learning , pages 25123-25153. PMLR.

Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. (2022). Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12104-12113.

Zhenmei, S., Wei, J., and Liang, Y. (2022). A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features. In International Conference on Learning Representations .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers learning parity functions with neural networks and particularly studies the tradeoff between size of the network and the sample size. The paper also explore a connection between the network size the lottery ticket hypothesis. The neural network studied in this paper has a sparse, specific, and symmetric structure. One step gradient arguments under the hinge loss have been used for the training. Finally, the paper proposes a promising direction for processing tabular datasets using neural networks.\n\nSTRENGTHS:\n- The paper shows that with sparse initialization, one can decide between wide network and smaller number of samples, or narrower network and higher number of samples and basically move between the two regime. \n- The connection of width to the lottery ticket hypothesis in the sparse initialization setting. \n- The experiments are quite extensive for the 2-layer MLP model with sparse initialization. \n- Observation of grokking and sample-wise double descent in the experiments is also interesting. This shows a potential avenue for future theoretical research on grokking.\n\nWEAKNESSES:\n- In the abstract, the lower-bound is stated for the number of training samples. However, the lower bound with SQ only contains the gradient precision. (Large number of samples gives is sufficient for the condition on gradient precision but not necessary). So to be exact, there is no bound on the number of samples. Moreover, the SQ bound does not give information about the SGD algorithm. \n- It would have been great if there were experiments checking the results of this paper beyond that particular setting of theoretical results. For example, it would have been nice to have the same experiments on Transformers and mean-field networks. Also the batch size is always 32 in the experiments; it would be nice to also try larger and smaller (maybe one sample at time) batch-sizes as well. Further, what happens if we do not have the initialization sparsity?\n\nQUESTIONS:\nThe setting considered in the paper is quite specific (e.g., the initialization and the training of the network). So is it possible to theoretically/empirically discuss the limitations of this study. For example if we keep the parity function as the target, do we expect these result to generalize to other architecture with more standard training methods?\n\nSuggestion:\\\nThe appendix part for the proof of the positive part is a bit hard to read. Particularly, if would be great if the exact training algorithm is once explained and also there is an overview over the proof.\n\nLIMITATIONS:\nThis is a theoretical study and there is no negative societal impact. However, the potential limitations of the findings could be discussed more extensively.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study the tradeoff between various resources in feature learning---data, compute, width. The authors focus on the fundamental problem of learning parity functions using a two-layer MLP. The degree $k$ of a parity function of $n$ variables controls the hardness of the problem: generally a bigger $k$ means more difficult. Learning parity function is a fundamental problem which has been used to study deep learning in the literature. The authors consider learning $(n,k)$-parities using gradient descent (GD) sparsely initialized at sparsity level $s$.\n\nThe first main result shows that if $s > k$ (over-sparse initialization), then there is a tradeoff between network width $r$ and sample size $m$ required to learn a $(n,k)$-parity with high probability. This result interpolates existing feature-learning results as far as I know. The second main result shows that if $s<k$ (under-sparse initialization), there is a similar width vs. sample size tradeoff for one-step GD.\n\nExtensive simulations and experiments on natural tabular datasets are presented to complement the theory.\n\nSTRENGTHS:\nI find this paper impressive and enjoyable to read. Understanding the tradeoff of different resources (time, memory, data) is important for large-scale machine learning. Focusing on the fundamental problem of learning parity functions, the authors present solid theoretical results. I think both theoretical and empirical contributions are significant.\n\nThe theory also provides perspectives for the lottery ticket hypothesis, by showing that a large width is beneficial for finding the right \"lottery ticket\".\n\nThe writing is very clear. High-level ideas are well explained and figures are easy to understand.\n\nWEAKNESSES:\nI cannot evaluate the technical novelty of this paper as I do not claim expertise in learning parity functions, while some of the analytical strategies seem to be based on existing papers.\n\nOne minor comment is that theory is established for shallow networks, and it cannot reflect \"Deep Feature Learning\" in the title.\n\nQUESTIONS:\nI am curious (1) if the authors could comment on the role of depth in neural networks, and (2) how do the results connect to the staircase phenomenon?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDisclaimer: My expertise in this domain is limited and understanding is highly superficial.\n\nThe manuscript presents a detailed take on addressing the tradeoffs on 4 axes, namely model size, dataset size, training epochs, and stochasticity. It offers a well-rounded analysis of the area and provides empirical and theoretical evidence for intuitive ideas, increasing network width leading to better sample efficiency. They use sparse parity learning as a proxy for real world feature learning problems which might be aligned on the above mentioned 4-axes. They also provide curious insights like the network width improves the possibility of “winning lottery ticket” neurons. The wide, sparsely initialized MLPs also outperform the very strong RFs.\n\nSTRENGTHS:\n- Section 4.1 and C.1: The statements on interplay between width and dataset size for example are a good starting point for further focused research.\n- Section B: Detailed theoretical contributions in the supplementary section verify the experiments and intuition.\n- Highly exhaustive experiments, analyzing and investigating important questions.\n\nWEAKNESSES:\n- The paper, while having a good base, introduction and motivation, becomes hard to follow and the organization seems a bit unnatural. For example, in my opinion, moving the insight from Lines 823 to 853 to the main paper at the expense of the rather verbose Section 3.2, before the “actual” analysis starts in Line 223 might help pitch the paper better. \n- It’s not clear to me how this work is different from the rich vein of work in learning parities and providing a SQ learning bound. It might be worthwhile adding a section comparing against previous attempts and drawing focus to distinct contributions, or the work might seem incremental in several directions without making a significant difference in one direction.\n\nQUESTIONS:\n- See Weaknesses.\n- How is the work distinct from the previous works like [1], especially as theorem 4 seems like a trivial extension of informal theorems and analysis from [1]? A bit more formalization of the distinction would be helpful.\n- More comparisons on the nC2 combinations of data-width-time-luck would provide one-on-one insights on the interplay and could be a good addition to the work. Almost a matrix of sorts, highlighting the relationship between two axes and their effect overall on the network could summarize the findings well.\n\n[1] Barak, Boaz, et al. \"Hidden progress in deep learning: Sgd learns parities near the computational limit.\" Advances in Neural Information Processing Systems 35 (2022): 21750-21764.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn an attempt to explore the mechanisms behind generalization and training of neural networks and different elements that play a role in it,  this paper investigates the impact of four resources of training MLPs on the famous and well-studied (n, k)-parity learning problem. The authors conduct a massive grid of experiments to analyze the trade-offs between data, compute, width (model size) and luck (sparsity at initialization) when training MLPs on the mentioned problem. The experimental evaluation suggests existence of \"frontiers\" of resource trade-offs such that decrease any resource at the frontier would likely result in incomplete or non-existent generalization. They further theoretically analyze this frontier and prove that under the necessary assumptions, this frontier can be recovered (at least in the over-sparse initialization case) theoretically. Based on the observed patterns, the authors show that under correct combination of resources, neural networks are capable of achieving similar or in some cases superior performance than tree-based methods on small tabular tasks.\n\nSTRENGTHS:\n* The diverse and large-scale experiments show clear patterns of existence of the trade-offs proposed by the authors, and provide thorough evaluation of different combinations of the four mentioned resources.\n* The theoretical results are supportive of the patterns present in the experiments and agree with the results from recent work.\n* Although the presented results may not seem novel, and arguably have been speculated before, a study that extensively studies the concept of \"success frontier\" was lacking.  \n* This work addresses an important aspect of using neural networks in practice: tabular data and suggests that further improvements can be made in applying neural networks on these kind of problems. This is in contrary to the popular belief that neural networks are inferior than tree-based models on these datasets.\n* The manuscript is well-presented, and to my knowledge, the authors have covered most related work and included necessary discussion and comparison with related work, which helps in conveying the importance of this setup and the findings.\n\nWEAKNESSES:\nI believe that this work has the potential to be very influential, but there are still some aspects that should be improved and some points that could be better justified.\n\n### Major concerns:\n* It's not clear if the study would apply to other problems or not. Hence, claiming that the results apply to \"deep feature learning in general\" sounds very exaggerated. It is definitely intuitive that the observed phenomena should extend to other problems to some extent, but we can't claim anything just based on intuition. For instance, it has been observed that width is not necessarily monotonically beneficial in all problems, and this could apply to other resources discussed in the paper as well. As a suggestion, a few ablation studies of smaller scale on other problems could be a better support for hypothesizing that the observations convey to other settings as well. \n* Section 3.1: Correct me if I'm getting this wrong, but based on Proposition 3 it seems like there \"exists\" settings in which SGD can't find the correct solution. If I'm right, this doesn't mean \"GD will fail\" at all, and it just means that GD is not guaranteed to converge. \n* In Theorem 4, setting error close to zero would result in $s \\to \\infty$, and $r \\to 0$. How is this possible? Isn't $r$ required to be at least as large as $s$? Could the authors clarify the setting of this theorem?\n* I'm not convinced about some references made to the lottery ticket hypothesis and \"good neurons\", and that width buys \"Luck\" in general. Although LTH could be a plausible explanation for the observations in the experiments, totally attributing the affect of width to LTH and \"good neurons\" would require thorough evidence, may it be theoretical or empirical.\n\n### Minor issues:\n* The authors have mentioned that they use bootsrapping for neural nets to solve the low-data problem. Have the authors used bootstrapping for other models that don't inherently perform it to have a fair comparison?\n* In theorem 5, and theorem 21 (the formal version), it is mentioned that the problem will be solved \"approximately\". Can the authors please clarify this approximation?\n\nQUESTIONS:\nI have mentioned my questions in the \"Weaknesses\" section.\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}