SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
T = \Omega \left(\frac {1}{\epsilon} \left(\frac {\sqrt {r} B _ {a 2} B _ {b} B _ {x 1}}{(m p) ^ {\frac {1}{4}}} + m \tilde {b}\right) \left(\frac {\sqrt {\log m}}{\sqrt {B _ {b} B _ {G}}} + \frac {1}{B _ {x 1} (m p) ^ {\frac {1}{4}}}\right)\right),
$$
$$
\frac {n}{\log n} = \tilde {\Omega} \left(\frac {m ^ {3} p B _ {x} ^ {2} B _ {a 2} ^ {4} B _ {b}}{\epsilon^ {2} r ^ {2} B _ {a 1} ^ {2} B _ {G}} + \frac {(m p) ^ {\frac {1}{2}} B _ {x 2}}{B _ {b} B _ {G}} + \frac {B _ {x} ^ {2}}{B _ {x 2}} + \frac {1}{p} + \left(\frac {1}{B _ {G} ^ {2}} + \frac {1}{B _ {x 1} ^ {2}}\right) \frac {B _ {x 2}}{| \ell^ {\prime} (0) | ^ {2}} + \frac {T m}{\delta}\right),
$$
then with initialization (8) and proper hyper-parameter values, we have with probability  $\geq 1 - \delta$  over the initialization and training samples, there exists  $t\in [T]$  in Algorithm 1 with:
$$
\begin{array}{l} \Pr [ \operatorname {s i g n} (f _ {\Xi^ {(t)}} (\mathbf {x})) \neq y ] \leq \mathcal {L} _ {\mathcal {D}} (f _ {\Xi^ {(t)}}) \\ \leq \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + r B _ {a 1} B _ {x 1} \sqrt {2 \gamma + O \left(\frac {\sqrt {B _ {x 2} \log n}}{B _ {G} | \ell^ {\prime} (0) | n ^ {\frac {1}{2}}}\right)} + \epsilon . \\ \end{array}
$$
Intuitively, the theorem shows when a data distribution admits a small approximation error by some "ground-truth" network with  $r$  neurons using gradient features from  $S_{p,\gamma ,B_G}$  (i.e., a small optimal approximate loss  $\mathrm{OPT}_{d,r,B_F,S_p,\gamma ,B_G}$ ), the gradient descent training can successfully learn good neural networks with sufficiently many  $m$  neurons.
Now we discuss the requirements and the error guarantee. Viewing boundedness parameters  $B_{a1}, B_{x1}$  etc. as constants, then the number  $m$  of neurons learned is roughly  $\tilde{\Theta}\left(\frac{r^4}{pe^4}\right)$ , a polynomial overparameterization compared to the "ground-truth" network. The proof shows that such an overparameterization is needed such that some neurons can capture the gradient features given by gradient descent. This is consistent with existing analysis about overparameterization network learning, and also consistent with existing empirical observations.
The error bound consists of three terms. The last term  $\epsilon$  can be made arbitrarily small, while the other two depend on the concrete data distribution. Specifically, with larger  $r$  and  $\gamma$ , the second term increases. While the first term (the optimal approximation loss) decreases, since a larger  $r$  means a larger "ground-truth" network family, and a larger  $\gamma$  means a larger Gradient Feature set  $S_{p,\gamma,B_G}$ . So, there is a trade-off between these two terms. When we later apply the framework to concrete problems (e.g., mixtures of Gaussians, parity functions), we will show that depending on the specific data distribution, we can choose the proper values for  $r,\gamma$  to make the error small. This then leads to error guarantees for the concrete problems and demonstrates the unifying power of the framework. Please refer to Appendix D.3 for more discussion about our problem setup and our core concept, e.g., parameter choice, early stopping, the role of  $s$ , activation functions, and so on.
Proof Sketch. The intuition in the proof of Theorem 3.12 is closely related to the notion of Gradient Features. First, the gradient descent will produce gradients that approximate the features in  $S_{p,\gamma ,B_G}$ . Then, the gradient descent update gives a good set of neurons, such that there exists an accurate classifier using these neurons with loss comparable to the optimal approximation loss. Finally, the training will learn to approximate the accurate classifier, resulting in the desired error guarantee. The complete proof is in Appendix D (the population version in Appendix D.2 and the empirical version in Appendix D.4), including the proper values for hyper-parameters such as  $\eta^{(t)}$  in Theorem D.17. Below, we briefly sketch the key ideas and omit the technical details.
We first show that a large subset of neurons has gradients at the first step as good features. (The claim can be extended to multiple steps; for simplicity, we follow existing work (e.g., [33, 105]) and present only the first step.) Let  $\nabla_{i}$  denote the gradient of the  $i$ -th neuron  $\nabla_{\mathbf{w}_i} \mathcal{L}_{\mathcal{D}}(f_{\Xi^{(0)}})$ . Denote the subset of neurons with nice gradients approximating feature  $(D, s)$  as:
$$
G _ {(D, s), N i c e} := \left\{i \in [ 2 m ]: s = \mathbf {b} _ {i} / | \mathbf {b} _ {i} |, \langle \nabla_ {i}, D \rangle > (1 - \gamma) \| \nabla_ {i} \| _ {2}, \| \nabla_ {i} \| _ {2} \geq \left| \mathbf {a} _ {i} ^ {(0)} \right| B _ {G} \right\}. \tag {9}
$$
Lemma 3.13 (Feature Emergence). For any  $r$  size subset  $\{(D_1, s_1), \ldots, (D_r, s_r)\} \subseteq S_{p, \gamma, B_G}$ , with probability at least  $1 - re^{-\Theta(mp)}$ , for all  $j \in [r]$ , we have  $|G_{(D_j, s_j), Nice}| \geq \frac{mp}{4}$ .
This is because  $\nabla_{i} = \ell^{\prime}(0)\mathbf{a}_{i}^{(0)}\mathbb{E}_{(\mathbf{x},y)}\left[y\sigma^{\prime}\left[\left\langle \mathbf{w}_{i}^{(0)},\mathbf{x}\right\rangle -\mathbf{b}_{i}\right]\mathbf{x}\right] = \ell^{\prime}(0)\mathbf{a}_{i}^{(0)}G(\mathbf{w}_{i}^{(0)},\mathbf{b}_{i})$  Now consider  $s_j = +1$  (the case  $-1$  is similar). Since  $\mathbf{w}_i$  is initialized by Gaussians, by  $\nabla_{i}$  's connection to Gradient Features, we can see that for all  $i\in [m]$ $\operatorname *{Pr}\left[i\in G_{(D_j, + 1),Nice}\right]\geq \frac{p}{2}$  . The lemma follows from concentration via a large enough  $m$  , i.e., sufficient overparameterization. The gradients allow obtaining a set of neurons approximating the "ground-truth" network with comparable loss:
Lemma 3.14 (Existence of Good Networks). For any  $\delta \in (0,1)$ , with proper hyper-parameter values, with probability at least  $1 - \delta$ , there is  $\tilde{\mathbf{a}}$  such that  $\| \tilde{\mathbf{a}} \|_0 = O(r\sqrt{mp})$  and  $f_{(\tilde{\mathbf{a}},\mathbf{W}^{(1)},\mathbf{b}})(\mathbf{x}) = \sum_{i=1}^{4m} \tilde{\mathbf{a}}_i \sigma\left(\left\langle \mathbf{w}_i^{(1)}, \mathbf{x} \right\rangle - \mathbf{b}_i\right)$  satisfies
$$
\mathcal {L} _ {\mathcal {D}} \left(f _ {(\bar {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \leq \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + \sqrt {2} r B _ {a 1} B _ {x 1} \left(\sqrt {\gamma} + \sqrt {\frac {2 B _ {b}}{\sqrt {m p} B _ {G}}}\right).
$$
Given the good set of neurons, we finally show that the remaining gradient steps can learn an accurate classifier. Intuitively, with small step sizes  $\eta^{(t)}$ , the weights of the first layer  $\mathbf{w}_i$  do not change too much (stay in a neighborhood) while the second layer weights grow, and thus the learning is similar to convex learning using the good set of neurons. Technically, we adopt the online convex optimization analysis (Theorem D.5) in [33] to get the final loss guarantee in Theorem 3.12.
4 Applications in Special Cases
In this section we will apply the gradient feature learning framework to some specific problems, corresponding to concrete data distributions  $\mathcal{D}$ . We primarily focus on prototypical problems for analyzing feature learning in networks. We will present here the results for mixtures of Gaussians and parity functions, and include the complete proofs and some other results in Appendix E.
4.1 Mixtures of Gaussians
Mixtures of Gaussians are among the most fundamental and widely used statistical models. Recently, it has been used to study neural network learning, in particular, the effect of gradient descent for feature learning of two-layer neural networks and the advantage over fixed feature methods [46, 99].
Data Distributions. We follow notations from [99]. The data are from a mixture of  $r$  high-dimensional Gaussians, and each Gaussian is assigned to one of two possible labels in  $\mathcal{Y} = \{\pm 1\}$ . Let  $S(y) \subseteq [r]$  denote the set of indices of Gaussians associated with the label  $y$ . The data distribution is then:  $q(\mathbf{x},y) = q(y)q(\mathbf{x}|y), q(\mathbf{x}|y) = \sum_{j \in S(y)} p_j \mathcal{N}_j(\mathbf{x})$ , where  $\mathcal{N}_j(\mathbf{x})$  is a multivariate normal distribution with mean  $\mu_j$ , covariance  $\Sigma_j$ , and  $p_j$  are chosen such that  $q(\mathbf{x},y)$  is correctly normalized. We will make some assumptions about the Gaussians, for which we first introduce some notations.
$$
D _ {j} := \frac {\mu_ {j}}{\| \mu_ {j} \| _ {2}}, \quad \tilde {\mu} _ {j} := \mu_ {j} / \sqrt {d}, \quad B _ {\mu 1} := \min _ {j \in [ r ]} \| \tilde {\mu} _ {j} \| _ {2}, \quad B _ {\mu 2} := \max _ {j \in [ r ]} \| \tilde {\mu} _ {j} \| _ {2}, \quad p _ {B} := \min _ {j \in [ r ]} p _ {j}.
$$
Assumption 4.1. Let  $8 \leq \tau \leq d$  be a parameter that will control our final error guarantee. Assume
- Equiprobable labels:  $q(-1) = q(+1) = 1/2$ .
- For all  $j \in [r]$ ,  $\Sigma_{j} = \sigma_{j} I_{d \times d}$ . Let  $\sigma_{B} := \max_{j \in [r]} \sigma_{j}$  and  $\sigma_{B+} := \max \{ \sigma_{B}, B_{\mu 2} \}$ .
-  $r \leq 2d$ ,  $p_B \geq \frac{1}{2d}$ ,  $\Omega\left(1 / d + \sqrt{\tau \sigma_{B+}^2 \log d / d}\right) \leq B_{\mu 1} \leq B_{\mu 2} \leq d$ .
- The Gaussians are well-separated: for all  $i \neq j \in [r]$ , we have  $-1 \leq \langle D_i, D_j \rangle \leq \theta$ , where
$$
0 \leq \theta \leq \min  \left\{\frac {1}{2 r}, \frac {\sigma_ {B +}}{B _ {\mu 2}} \sqrt {\frac {\tau \log d}{d}} \right\}.
$$
Remark 4.2. The first two assumptions are for simplicity; they can be relaxed. We can generalize our analysis to the mixture of Gaussians with unbalanced label probabilities and general covariances. The third assumption is to make sure that each Gaussian has a good amount of probability mass to be learned. The remaining assumptions are to make sure that the Gaussians are well-separated and can be distinguished by the learning algorithm.
We are now ready to apply the framework to these data distributions, for which we only need to compute the Gradient Feature set and the corresponding optimal approximation loss.
Lemma 4.3 (Mixtures of Gaussians: Gradient Features).  $(D_{j}, + 1)\in S_{p,\gamma ,B_{G}}$  for all  $j\in [r]$ , where
$$
p = \frac {B _ {\mu 1}}{\sqrt {\tau \log d} \sigma_ {B +} \cdot d ^ {\Theta (\tau \sigma_ {B +} {} ^ {2} / B _ {\mu 1} ^ {2})}}, \quad \gamma = \frac {1}{d ^ {0 . 9 \tau - 1 . 5}}, \quad B _ {G} = p _ {B} B _ {\mu 1} \sqrt {d} - O \left(\frac {\sigma_ {B +}}{d ^ {0 . 9 \tau}}\right).
$$
Let  $f^{*}(\mathbf{x}) = \sum_{j=1}^{r} \frac{y_{(j)}}{\sqrt{\tau \log d}\sigma_{B+}} \left[ \sigma \left( \langle D_{j}, \mathbf{x} \rangle - 2\sqrt{\tau \log d}\sigma_{B+} \right) \right]$  whose hinge loss is at most  $\frac{3}{d^{\tau}} + \frac{4}{d^{0.9\tau-1}\sqrt{\tau \log d}}$ .
Given the values on gradient feature parameters  $p, \gamma, B_G$  and the optimal approximation loss  $\mathrm{OPT}_{d,r,B_F,S_p,\gamma,B_G}$ , the framework immediately leads to the following guarantee:
Theorem 4.4 (Mixtures of Gaussians: Main Result). Assume Assumption 4.1. For any  $\epsilon, \delta \in (0,1)$ , when Algorithm 1 uses hinge loss with
$$
m = \operatorname {p o l y} \left(\frac {1}{\bar {\delta}}, \frac {1}{\epsilon}, d ^ {\Theta \left(\tau \sigma_ {B +} {} ^ {2} / B _ {\mu 1} ^ {2}\right)}, r, \frac {1}{p _ {B}}\right) \leq e ^ {d}, \quad T = \operatorname {p o l y} (m), \quad n = \operatorname {p o l y} (m)
$$
and proper hyper-parameters, then with probability at least  $1 - \delta$ , there exists  $t \in [T]$  such that
$$
\operatorname * {P r} [ \mathrm {s i g n} (f _ {\Xi^ {(t)}} (\mathbf {x})) \neq y ] \leq \frac {\sqrt {2} r}{d ^ {0 . 4 \tau - 0 . 8}} + \epsilon .
$$
The theorem shows that gradient descent can learn to a small error via learning the gradient features, given proper hyper-parameters. In particular, we need sufficient overparameterization (a sufficiently large number  $m$  of neurons). When  $\sigma_{B^{+}}^{2} / B_{\mu 1}^{2}$  is a constant which is the prototypical interesting case, and we choose a constant  $\tau$ , then  $m$  is polynomial in the key parameters  $\frac{1}{\delta}, \frac{1}{\epsilon}, d, r, \frac{1}{p_{B}}$ , and the error bound is inverse polynomial in  $d$ . The complete proof is given in Appendix E.2.
[46] studies (almost) linear separable cases while our setting includes non-linear separable cases, e.g., XOR. [99] mainly studies neural network classification on 4 Gaussian clusters with XOR structured labels, while our setting is much more general, e.g., our cluster number can extend up to  $2d$ .
4.1.1 Mixtures of Gaussians: Beyond the Kernel Regime
As discussed in the introduction, it is important for the analysis to go beyond fixed feature methods such as NTK (i.e., the kernel regime), so as to capture the feature learning ability which is believed to be the key factor for the empirical success. We first review the fixed feature methods. Following [33], suppose  $\Psi$  is a data-independent feature mapping of dimension  $N$  with bounded features, i.e.,  $\Psi : \mathcal{X} \to [-1,1]^N$ . For  $B > 0$ , the family of linear models on  $\Psi$  with bounded norm  $B$  is  $\mathcal{H}_B = \{h(\tilde{\mathbf{x}}): h(\tilde{\mathbf{x}}) = \langle \Psi(\tilde{\mathbf{x}}, w\rangle, \|w\|_2 \leq B\}$ . This can capture linear models on fixed finite-dimensional feature maps, e.g., NTK, and also infinite dimensional feature maps, e.g., kernels like RBF, that can be approximated by feature maps of polynomial dimensions [64, 98, 105].
Our framework indeed goes beyond fixed features and shows features from gradients are more powerful than features from random initialization, e.g., NTK. Our framework can show the advantage of network learning over kernel methods under the setting of [99] (4 Gaussian clusters with XOR structured labels). For large enough  $d$ , our framework only needs roughly  $\Omega$  ( $\log d$ ) neurons and  $\Omega((\log d)^2)$  samples to achieve arbitrary small constant error (see Theorem E.18 when  $\sigma_B = 1$ ), while fixed feature methods need  $\Omega(d^2)$  features and  $\Omega(d^2)$  samples to achieve nontrivial errors (as proved in [99]). Moreover, [99] uses ODE to simulate the optimization process for the 2-layer networks learning XOR-shaped Gaussian mixture with  $\Omega(1)$  neurons and gives convincing evidence that  $\Omega(d)$  samples is enough to learn it, yet they do not give a rigorous convergence guarantee for this problem. We successfully derive a convergence guarantee and we require a much smaller sample size  $\Omega((\log d)^2)$ . For the proof (detailed in Appendix E.3), we only need to calculate the  $p, \gamma, B_G$  of the data distribution carefully and then inject these numbers into Theorem 3.12.
4.2 Parity Functions
Parity functions are a canonical family of learning problems in computational learning theory, usually for showing theoretical computational barriers [103]. The typical sparse parties over  $d$ -dim binary inputs  $\phi \in \{\pm 1\}^d$  are  $\prod_{i \in A} \phi_i$  where  $A \subseteq [d]$  is a subset of dimensions. Recent studies have shown that when the distribution of inputs  $\phi$  has structures rather than uniform, neural networks can perform feature learning and finally learn parity functions with a small error, while methods without feature learning, e.g. NTK, cannot achieve good results [33, 76, 105]. Thus, this has been a prototypical setting for studying feature learning phenomena in networks. Here we consider a generalization of this problem and show that our framework can show successful learning via gradient descent.
Data Distributions. Suppose  $\mathbf{M} \in \mathbb{R}^{d \times D}$  is an unknown dictionary with  $D$  columns that can be regarded as patterns. For simplicity, assume  $d = D$  and  $\mathbf{M}$  is orthonormal. Let  $\phi \in \mathbb{R}^d$  be a hidden representation vector. Let  $A \subseteq [D]$  be a subset of size  $rk$  corresponding to the class relevant patterns and  $r$  is an odd number. Then the input is generated by  $\mathbf{M}\phi$ , and some function on  $\phi_A$  generates the label. WLOG, let  $A = \{1, \dots, rk\}$ ,  $A^\perp = \{rk + 1, \dots, d\}$ . Also, we split  $A$  such that for all  $j \in [r]$ ,  $A_j = \{(j - 1)k + 1, \dots, jk\}$ . Then the input  $\mathbf{x}$  and the class label  $y$  are given by:
$$
\mathbf {x} = \mathbf {M} \phi , y = g ^ {*} \left(\phi_ {A}\right) = \operatorname {s i g n} \left(\sum_ {j \in [ r ]} \operatorname {X O R} \left(\phi_ {A _ {j}}\right)\right), \tag {10}
$$
where  $g^{*}$  is the ground-truth labeling function mapping from  $\mathbb{R}^{rk}$  to  $\mathcal{V} = \{\pm 1\}$ ,  $\phi_{A}$  is the sub-vector of  $\phi$  with indices in  $A$ , and  $\mathrm{XOR}(\phi_{A_j}) = \prod_{l \in A_j} \phi_l$  is the parity function. We still need to specify the distribution  $\mathcal{X}$  of  $\phi$ , which determines the structure of the input distribution:
$$
\mathcal {X} := \left(1 - 2 r p _ {A}\right) \mathcal {X} _ {U} + \sum_ {j \in [ r ]} p _ {A} \left(\mathcal {X} _ {j, +} + \mathcal {X} _ {j, -}\right). \tag {11}
$$
For all corresponding  $\phi_{A^{\perp}}$  in  $\mathcal{X}$ , we have  $\forall l\in A^{\perp}$ , independently:  $\phi_l = \left\{ \begin{array}{ll} + 1, & \mathrm{w.p.} p_o\\ -1, & \mathrm{w.p.} p_o\\ 0, & \mathrm{w.p.} 1 - 2p_o \end{array} \right.$
where  $p_{o}$  controls the signal noise ratio: if  $p_{o}$  is large, then there are many nonzero entries in  $A^{\perp}$  which are noise interfering with the learning of the ground-truth labeling function on  $A$ . For corresponding  $\phi_{A}$ , any  $j \in [r]$ , we have
- In  $\mathcal{X}_{j, + },\phi_{A_j} = [+1, + 1,\dots , + 1]^\top$  and  $\phi_{A\setminus A_j}$  only have zero elements.
- In  $\mathcal{X}_{j, - }$ ,  $\phi_{A_j} = [-1, - 1,\dots , - 1]^\top$  and  $\phi_{A\setminus A_j}$  only have zero elements.
- In  $\mathcal{X}_U$ , we have  $\phi_A$  draw from  $\{+1, -1\}^{rk}$  uniformly.
In short, we have  $r$  parity functions each corresponding to a block of  $k$  dimensions;  $\mathcal{X}_{j, + }$  and  $\mathcal{X}_{j, - }$  stands for the component providing a strong signal for the  $j$ -th parity;  $\mathcal{X}_U$  corresponds to uniform distribution unrelated to any parity and providing weak learning signal;  $A^{\perp}$  is the noise part. The label depends on the sum of the  $r$  parity functions.
Assumption 4.5. Let  $8 \leq \tau \leq d$  be a parameter that will control our final error guarantee. Assume  $k$  is an odd number and:  $k \geq \Omega(\tau \log d)$ ,  $d \geq rk + \Omega(\tau r \log d)$ ,  $p_{o} = O\left(\frac{rk}{d - rk}\right)$ ,  $p_{A} \geq \frac{1}{d}$ .
Remark 4.6. We set up the problem to be more general than the parity function learning in existing work. If  $r = 1$ , the labeling function reduces to the traditional  $k$ -sparse parties of  $d$  bits. The assumptions require  $k, d,$  and  $p_A$  to be sufficiently large so as to provide enough large signals for learning. Note that when  $k = \frac{d}{16}$ ,  $r = 1$ ,  $p_o = \frac{1}{2}$ , our analysis also holds, which shows our framework is beyond the kernel regime (discuss in detail in Section 4.2.1).
To apply our framework, again we only need to compute the Gradient Feature set and the corresponding optimal loss. We first define the Gradient Features: For all  $j \in [r]$ , let  $D_{j} = \frac{\sum_{l \in A_{j}} \mathbf{M}_{l}}{\|\sum_{l \in A_{j}} \mathbf{M}_{l}\|_{2}}$ .
Lemma 4.7 (Parity Functions: Gradient Features). We have  $(D_{j}, + 1), (D_{j}, - 1) \in S_{p,\gamma ,B_{G}}$  for all  $j \in [r]$ , where
$$
p = \Theta \left(\frac {1}{\sqrt {\tau r \log d} \cdot d ^ {\Theta (\tau r)}}\right), \quad \gamma = \frac {1}{d ^ {\tau - 2}}, \quad B _ {G} = \sqrt {k} p _ {A} - O \left(\frac {\sqrt {k}}{d ^ {\tau}}\right). \tag {12}
$$
With gradient features from  $S_{p,\gamma ,B_G}$ , let  $f^{*}(\mathbf{x}) = \sum_{j = 1}^{r}\sum_{i = 0}^{k}(-1)^{i + 1}\sqrt{k}\Big[\sigma \left(\langle D_j,\mathbf{x}\rangle -\frac{2i - k - 1}{\sqrt{k}}\right) - 2\sigma \left(\langle D_j,\mathbf{x}\rangle -\frac{2i - k}{\sqrt{k}}\right) + \sigma \left(\langle D_j,\mathbf{x}\rangle -\frac{2i - k + 1}{\sqrt{k}}\right)\Big]$  whose hinge loss is 0.
Above, we show that  $D_{j}$  is the "indicator function" for the subset  $A_{j}$  so that we can build the optimal neural network based on such directions. Given the values on gradient feature parameters and the optimal approximation loss, the framework immediately leads to the following guarantee:
Theorem 4.8 (Parity Functions: Main Result). Assume Assumption 4.5. For any  $\epsilon, \delta \in (0,1)$  when Algorithm 1 uses hinge loss with
$$
m = \operatorname {p o l y} \left(\frac {1}{\delta}, \frac {1}{\epsilon}, d ^ {\Theta (\tau r)}, k, \frac {1}{p _ {A}}\right) \leq e ^ {d}, \quad T = \operatorname {p o l y} (m), \quad n = \operatorname {p o l y} (m)
$$
$$
\Pr [ \operatorname {s i g n} (f _ {\Xi^ {(t)}} (\mathbf {x})) \neq y ] \leq \frac {3 r \sqrt {k}}{d ^ {(\tau - 3) / 2}} + \epsilon .
$$
The theorem shows that gradient descent can learn to a small error in this problem. We also need sufficient overparameterization: When  $r$  is a constant (e.g.,  $r = 1$  in existing work), and we choose a constant  $\tau$ ,  $m$  is polynomial in  $\frac{1}{\delta}$ ,  $\frac{1}{\epsilon}$ ,  $d$ ,  $k$ ,  $\frac{1}{p_A}$ , and the error bound is inverse polynomial in  $d$ . The proof is in Appendix E.4. Our setting is more general than that in [33, 76] which corresponds to  $\mathbf{M} = I$ ,  $r = 1$ ,  $p_A = \frac{1}{4}$ ,  $p_o = \frac{1}{2}$ . [105] study single index learning, where one feature direction is enough for a two-layer network to recover the label, while our setting considers  $r$  directions  $D_1, \ldots, D_r$ , so the network needs to learn multiple directions to get a small error.
4.2.1 Parity Functions: Beyond the Kernel Regime
Again, we show that our framework indeed goes beyond fixed features under parity functions. Our problem setting in Section 4.2 is general enough to include the problem setting in [33]. Their lower bound for fixed feature methods directly applies to our case and leads to the following:
Proposition 4.9. There exists a data distribution in the parity learning setting in Section 4.2 with  $\mathbf{M} = I, r = 1, p_A = \frac{1}{4}, k = \frac{d}{16}, p_o = \frac{1}{2}$ , such that all  $h \in \mathcal{H}_B$  have hinge-loss at least  $\frac{1}{2} - \frac{\sqrt{N}B}{2^k\sqrt{2}}$ .
This means to get an inverse-polynomially small loss, fixed feature models need to have an exponentially large size, i.e., either the number of features  $N$  or the norm  $B$  needs to be exponential in  $k$ . In contrast, Theorem 4.8 shows our framework guarantees a small loss with a polynomially large model, runtime, and sample complexity. Clearly, our framework is beyond the fixed feature methods.
Parities on Uniform Inputs. When  $r = 1, p_A = 0$ , our problem setting will degenerate to the classic sparse parity function on a uniform input distribution. This has also been used for analyzing network learning [16]. For this case, our framework can get a  $k2^{O(k)}\log (k)$  network width bound and a  $O(d^{k})$  sample complexity bound, matching those in [16]. This then again confirms the advantage of network learning over kernel methods that requires  $d^{\Omega (k)}$  dimensions as shown in [16]. See the full statement in Theorem E.31, details in Appendix E.5, and alternative analysis in Appendix E.6.
5 Further Implications and Conclusion
Our general framework sheds light on several interesting phenomena in NN learning observed in practice. Feature learning beyond the kernel regime has been discussed in Section 4.1.1 and Section 4.2.1. Here we discuss the LTH and defer more implications such as simplicity bias, learning over different data distributions, and new perspectives about roadmaps forward in Appendix C.