SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and  $\| \alpha^t\| _2\leq \sqrt{N}\| \alpha^t\|_{\infty}$  . Thus, if we choose
$\mathcal{B} = \frac{4\sqrt{N}}{\Delta}$
-  $\xi = 2(C_3 + 2\Delta k + \tau)\sqrt{\frac{dN}{\log(d)}}$
$T_{2} = \frac{256(C_{3} + 2\Delta k + \tau)^{2}N^{2}d}{\epsilon^{2}\Delta^{2}\log(d)}$
$\cdot \gamma_{2} = \frac{\epsilon\log(d)}{8(C_{3} + 2\Delta k + \tau)^{2}dN}$
$$
\min  _ {t \in \left[ T _ {1} + T _ {2} \right]} L _ {\text {c o v}} \left(\theta^ {t}, \chi_ {[ k ]}, x\right) \leq \frac {\epsilon}{2} + \frac {\epsilon}{2} = \epsilon . \tag {83}
$$
The result follows under the assumption that the training process is terminated once the training loss falls below  $\epsilon$ , thus guaranteeing that  $L_{\mathrm{cov}}\left(\theta^{T_1 + T_2}, \chi_{[k]}, x\right) \leq \epsilon$ .
B Proof of Theorem 4
Theorem 7 (Theorem 4, restatement). Let  $d, N$  be two positive integers. Let  $\mathrm{NN}(x; \theta) = \sum_{i=1}^{N} a_i \sigma(w_i x + b_i)$  be a 2-layer fully connected network with activation  $\sigma(y) := \mathrm{Ramp}(y)$  (as defined in (84)) and  $N \geq (d+1)(d-k+1) \log((d+1)(d-k+1)/\delta)$ . Consider training  $\mathrm{NN}(x; \theta)$  with layerwise curriculum-SGD on the hinge loss with batch size  $B \geq (4\zeta^2 N^2)^{-1} \log\left(\frac{Nd+N}{\delta}\right)$ , noise level  $\tau \leq \zeta \log\left(\frac{Nd+N}{\delta}\right)^{-1/2}$ , with  $\zeta \leq \frac{\epsilon \mu^k}{24(d+1)^2(d-k+1)^2N}$  and  $\mu = \sqrt{1 - \frac{1}{2(d-k)}}$ . Then, there exists an initialization and a learning rate schedule such that after  $T_1 = 1$  step on the sparse data and  $T_2 = \frac{64}{\epsilon^2} (d-k+1)^3 (d+1)N(1+\tau)^2$  steps on the full data, with probability  $1 - 3\delta$ , the trained network has generalization error at most  $\epsilon$ .
This proof is similar to the proof of Theorem 3 in [CM23], with the following two modifications: 1) the use of a mixed input distribution in the second part of training, and, 2) the addition of uniform gradient noise in the iterates.
B.1 Proof Setup
We consider a 2-layer neural network, defined as  $\mathrm{NN}(x;\theta) = \sum_{i=1}^{N}a_i\sigma(w_i x + b_i)$ , where  $N \geq (d+1)(d-k+1)\log((d+1)(d-k+1)/\delta)$  is the number of hidden units,  $\theta = (a,b,w)$  and  $\sigma := \mathrm{Ramp}$  denotes the activation defined as:
$$
\operatorname {R a m p} (x) = \left\{ \begin{array}{l l} 0 & x \leq 0, \\ x & 0 <   x \leq 1, \\ 1 & x > 1 \end{array} \right.. \tag {84}
$$
Similarly as before, without loss of generality, we assume that the labels are generated by  $\chi_{[k]}(x) \coloneqq \prod_{i=1}^{k} x_i$ . Our proof scheme is the following:
1. We train only the first layer of the network for one step on data  $(x, \chi_{[k]}(x))$  with  $x \sim \mathcal{D}_{\mu}$ , with  $\mu := 2p - 1 = \sqrt{1 - \frac{1}{2(d - k)}}$ ;
2. We show that after one step of training on such biased distribution, the target parity belongs to the linear span of the hidden units of the network;
3. We train only the second layer of the network on  $(x,\chi_{[k]}(x))$  with  $x\sim \mathcal{D}$ , until convergence;
4. We use established results on the convergence of SGD on convex losses to conclude.
We train our network with noisy-GD on the hinge loss. Specifically, we apply the following updates, for all  $t \in \{0, 1, \dots, T - 1\}$ :
$$
w _ {i, j} ^ {t + 1} = w _ {i, j} ^ {t} - \gamma_ {t} \left(\frac {1}{B} \sum_ {s = 1} ^ {B} \nabla_ {w _ {i, j} ^ {t}} L \left(\theta^ {t}, \chi_ {[ k ]}, x _ {s} ^ {t}\right) + Z _ {w _ {i, j} ^ {t}}\right),
$$
$$
a _ {i} ^ {t + 1} = a _ {i} ^ {t} - \eta_ {t} \left(\frac {1}{B} \sum_ {s = 1} ^ {B} \nabla_ {a _ {i} ^ {t}} L \left(\theta^ {t}, \chi_ {[ k ]}, x _ {s} ^ {t}\right) + Z _ {a _ {i} ^ {t}}\right) + c _ {t}, \tag {85}
$$
$$
b _ {i} ^ {t + 1} = \lambda_ {t} \left(b _ {i} ^ {t} + \psi_ {t} \frac {1}{B} \sum_ {s = 1} ^ {B} \nabla_ {b _ {i} ^ {t}} L (\theta^ {t}, \chi_ {[ k ]}, x _ {s} ^ {t}) + Z _ {b _ {i} ^ {t}}\right) + d _ {t},
$$
where  $L(\theta^t, \chi_{[k]}, x) = \max \{0, 1 - \chi_{[k]}(x) \mathrm{NN}(x; \theta^t)\}$  and  $Z_{w_{i,j}^t}, Z_{a_i^t}$  and  $Z_{b_i^t}$  are i.i.d. Unif  $[- \tau, \tau]$  for some  $\tau > 0$ . Following the proof strategy introduced above, we set  $\mathcal{D}^0 = \mathcal{D}_\mu$ , where  $\mu = \sqrt{1 - \frac{1}{2(d - k)}}$ , and  $\mathcal{D}^t = \mathcal{D}$ , for all  $t \geq 1$ . We set the parameters of (85) to:
$$
\gamma_ {0} = \mu^ {- (k - 1)} 2 N, \quad \gamma_ {t} = 0 \quad \forall t \geq 1, \tag {86}
$$
$$
\eta_ {0} = 0, \quad \eta_ {t} = \frac {\epsilon}{2 N (1 + \tau) ^ {2}} \quad \forall t \geq 1, \tag {87}
$$
$$
\psi_ {0} = \frac {N}{\mu^ {k}}, \quad \psi_ {t} = 0 \quad \forall t \geq 1, \tag {88}
$$
$$
c _ {0} = - \frac {1}{2 N}, \quad c _ {t} = 0 \quad \forall t \geq 1, \tag {89}
$$
$$
\lambda_ {0} = (d + 1), \quad \lambda_ {t} = 1 \quad \forall t \geq 1, \tag {90}
$$
$$
d _ {0} = 0, \quad d _ {t} = 0 \quad \forall t \geq 1, \tag {91}
$$
and we consider the following initialization scheme:
$$
w _ {i, j} ^ {0} = 0 \quad \forall i \in [ N ], j \in [ d ];
$$
$$
a _ {i} ^ {0} = \frac {1}{2 N} \quad \forall i \in [ N ]; \tag {92}
$$
$$
b _ {i} ^ {0} \sim \operatorname {U n i f} \left\{\frac {b _ {l m}}{d + 1} + \frac {1}{2}: l \in \{0, \dots , d \}, m \in \{- 1, \dots , d - k \} \right\},
$$
$$
b _ {l m} := - d + 2 l - \frac {1}{2} + \frac {m + 1}{d - k}. \tag {93}
$$
Note that such initialization is invariant to permutations of the input neurons.
B.2 First Step: Recovering the Support
As mentioned above, we train our network for one step on  $(x,\chi_{[k]}(x))$  with  $x\sim \mathcal{D}_{\mu}$
Population Gradient at Initialization. Let us compute the population gradient at initialization. Since we set  $\xi_0 = 0$ , we do not need to compute the initial gradient for  $a$ . Note that at initialization
$|\mathrm{NN}(x;\theta^0)| < 1$ . Thus, the initial population gradients are given by
$$
\forall j \in [ k ], i \in [ N ] \quad \bar {G} _ {w _ {i, j}} = - a _ {i} \mathbb {E} _ {x \sim \mathcal {D} _ {\mu}} \left[ \prod_ {l \in [ k ] \backslash j} x _ {l} \cdot \mathbf {1} \left(\langle w _ {i}, x \rangle + b _ {i} \in [ 0, 1 ]\right) \right] \tag {94}
$$
$$
\forall j \notin [ k ], i \in [ N ] \quad \bar {G} _ {w _ {i, j}} = - a _ {i} \mathbb {E} _ {x \sim \mathcal {D} _ {\mu}} \left[ \prod_ {l \in [ k ] \cup j} x _ {l} \cdot \mathbb {1} \left(\left\langle w _ {i}, x \right\rangle + b _ {i} \in [ 0, 1 ]\right) \right] \tag {95}
$$
$$
\forall i \in [ N ] \quad \bar {G} _ {b _ {i}} = - a _ {i} \mathbb {E} _ {x \sim \mathcal {D} _ {\mu}} \left[ \prod_ {l \in [ k ]} x _ {l} \cdot \mathbb {1} \left(\langle w _ {i}, x \rangle + b _ {i} \in [ 0, 1 ]\right) \right] \tag {96}
$$
Lemma 9. Initialize  $a, b, w$  according to (92). Then,
$$
\forall j \in [ k ], \quad \bar {G} _ {w _ {i, j}} = - \frac {\mu^ {k - 1}}{2 N}; \tag {97}
$$
$$
\forall j \notin [ k ], \quad \bar {G} _ {w _ {i, j}} = - \frac {\mu^ {k + 1}}{2 N}; \tag {98}
$$
$$
\bar {G} _ {b _ {i}} = - \frac {\mu^ {k}}{2 N}. \tag {99}
$$
Proof. If we initialize according to (92), we have  $\langle w_i, x \rangle + b_i \in [0,1]$  for all  $i$ . The results hold since  $\mathbb{E}_{x \sim \mathcal{D}_{\mu}}[\chi_S(x)] = \mu^{|S|}$ .
Effective Gradient at Initialization.
Lemma 10 (Noisy-GD). Let
$$
\hat {G} _ {w _ {i, j}} := \frac {1}{B} \sum_ {s = 1} ^ {B} \nabla_ {w _ {i, j} ^ {0}} L \left(\theta^ {0}, \chi_ {[ k ]}, x ^ {s, 0}\right) + Z _ {w _ {i, j}}, \tag {100}
$$
$$
\hat {G} _ {b _ {i}} := \frac {1}{B} \sum_ {s = 1} ^ {B} \nabla_ {b _ {i} ^ {0}} L \left(\theta^ {0}, \chi_ {[ k ]}, x ^ {s, 0}\right) + Z _ {b _ {i}}, \tag {101}
$$
be the effective gradients at initialization, where  $G_{w_{i,j}}, G_{b_i}$  are the population gradients at initialization and  $Z_{w_{i,j}}, Z_{b_i}$  are i.i.d. Unif[-τ,τ]. If  $B \geq (4\zeta^2 N^2)^{-1}\log \left(\frac{Nd + N}{\delta}\right)$  and  $\tau \leq \zeta \log \left(\frac{Nd + N}{\delta}\right)^{-1/2}$ , with probability  $1 - 2\delta$ :
$$
\left\| \hat {G} _ {w _ {i, j}} - \bar {G} _ {w _ {i, j}} \right\| _ {\infty} \leq \zeta , \tag {102}
$$
$$
\left\| \hat {G} _ {b _ {i}} - \bar {G} _ {b _ {i}} \right\| _ {\infty} \leq \zeta . \tag {103}
$$
Proof. By Hoeffding's inequality
$$
\mathbb {P} \left(\left| \hat {G} _ {w _ {i, j}} - \bar {G} _ {w _ {i, j}} \right| \geq \zeta\right) \leq 2 \exp \left(- \frac {2 \zeta^ {2}}{\left(4 N ^ {2} B\right) ^ {- 1} + \tau^ {2}}\right) \leq \frac {2 \delta}{N d + N}, \tag {104}
$$
$$
\mathbb {P} \left(\left| \hat {G} _ {b _ {i}} - \bar {G} _ {b _ {i}} \right| \geq \zeta\right) \leq 2 \exp \left(- \frac {2 \zeta^ {2}}{\left(4 N ^ {2} B\right) ^ {- 1} + \tau^ {2}}\right) \leq \frac {2 \delta}{N d + N}. \tag {105}
$$
The result follows by union bound.
Lemma 11. Let
$$
w _ {i, j} ^ {1} = w _ {i, j} ^ {0} - \gamma_ {0} \hat {G} _ {w _ {i, j}} \tag {106}
$$
$$
b _ {i} ^ {1} = \lambda_ {0} \left(b _ {i} ^ {0} - \psi_ {0} \hat {G} _ {b _ {i}}\right) \tag {107}
$$
If  $B \geq (4\zeta^2 N^2)^{-1} \log \left(\frac{Nd + N}{\delta}\right)$  and  $\tau \leq \zeta \log \left(\frac{Nd + N}{\delta}\right)^{-1/2}$ , with probability  $1 - 2\delta$ ,
i) For all  $j\in [k],i\in [N],|w_{i,j}^{1} - 1|\leq \frac{2N\zeta}{\mu^{k - 1}};$
ii) For all  $j \notin [k]$ ,  $|w_{i,j}^{1} - (1 - \frac{1}{2(d - k)})| \leq \frac{2N\zeta}{\mu^{k - 1}}$ ;
iii) For all  $i\in [N]$ ,  $|b_i^1 -(d + 1)(b_i^0 -\frac{1}{2})|\leq \frac{N(d + 1)\zeta}{\mu^k}$ .
Proof. We apply Lemma 11:
i) For all  $j \in [k]$ ,  $i \in [N]$ ,  $|\hat{w}_{i,j}^1 - 1| = \gamma_0 |\hat{G}_{w_{i,j}} - G_{w_{i,j}}| \leq \frac{2N\zeta}{\mu^{k-1}}$ ;
ii) For all  $j \notin [k]$ ,  $i \in [N]$ ,  $|\hat{w}_{i,j}^1 - (1 - \frac{1}{2(d - k)})| = \gamma_0 |\hat{G}_{w_{i,j}} - G_{w_{i,j}}| \leq \frac{2N\zeta}{\mu^{k-1}}$ ;
iii) For all  $i \in [N]$ ,
$$
\begin{array}{l} \left| \hat {b} _ {i} ^ {1} - (d + 1) \left(b _ {i} ^ {0} - \frac {1}{2}\right) \right| = \left| \lambda_ {0} \left(b _ {i} ^ {0} + \psi_ {0} \hat {G} _ {b _ {i}}\right) - \lambda_ {0} \left(b _ {i} ^ {0} + \psi_ {0} G _ {b _ {i}}\right) \right| (109) \\ \leq | \lambda_ {0} | \cdot | \psi_ {0} | \cdot | \hat {G} _ {b _ {i}} - G _ {b _ {i}} | (110) \\ \leq \frac {N (d + 1) \zeta}{\mu^ {k}}. (111) \\ \end{array}
$$
Lemma 12. If  $N \geq (d + 1)(d - k + 1) \log ((d + 1)(d - k + 1) / \delta)$ , then with probability  $1 - \delta$ , for all  $l \in \{0, \dots, d\}$ , and for all  $m \in \{-1, \dots, d - k\}$  there exists  $i$  such that  $b_i^0 = \frac{b_{lm}}{d + 1} + \frac{1}{2}$ .
Proof. The probability that there exist  $l, m$  such that the above does not hold is
$$
\left(1 - \frac {1}{(d + 1) (d - k + 1)}\right) ^ {N} \leq \exp \left(- \frac {N}{(d + 1) (d - k + 1)}\right) \leq \frac {\delta}{(d + 1) (d - k + 1)}. \tag {112}
$$
Lemma 13. Let  $\sigma_{lm}(x) = \mathrm{Ramp}\left(\sum_{j=1}^{d} x_j - \frac{1}{2(d-k)} \sum_{j>k} x_j + b_{lm}\right)$ , with  $b_{lm}$  given in (93). If  $B \geq (4\zeta^2 N^2)^{-1} \log \left(\frac{Nd+N}{\delta}\right)$ ,  $\tau \leq \zeta \log \left(\frac{Nd+N}{\delta}\right)^{-1/2}$  and  $N \geq (d+1)(d-k+1) \log ((d+1)(d-k+1)/\delta)$ , with probability  $1-3\delta$ , for all  $l, m$  there exists  $i$  such that
$$
\left| \sigma_ {l m} (x) - \operatorname {R a m p} \left(\sum_ {j = 1} ^ {d} \hat {w} _ {i, j} ^ {1} x _ {j} + \hat {b} _ {i} ^ {1}\right) \right| \leq 3 N (d + 1) \zeta \mu^ {- k}. \tag {113}
$$
Proof. By Lemma 12, with probability  $1 - \delta$ , for all  $l, m$  there exists  $i$  such that  $b_i^{(0)} = \frac{b_{lm}}{d + 1} + \frac{1}{2}$ . For ease of notation, we replace indices  $i \mapsto (lm)$ , and denote  $\hat{\sigma}_{lm}(x) = \mathrm{Ramp}\left(\sum_{j=1}^{d} w_{lm,j}^1 x_j + b_{lm}^1\right)$ . Then, by Lemma 11 with probability  $1 - 2\delta$ ,
$$
\begin{array}{l} \left| \sigma_ {l m} (x) - \hat {\sigma} _ {l m} (x) \right| \leq \left| \sum_ {j = 1} ^ {k} \left(w _ {l m, j} ^ {1} - 1\right) x _ {j} + \sum_ {j = k + 1} ^ {d} \left(w _ {l m, j} ^ {1} - \left(1 - \frac {1}{2 (d - k)}\right)\right) x _ {j} + b _ {l m} ^ {1} - b _ {l m} \right| \\ \leq k 2 N \zeta \mu^ {- (k - 1)} + (d - k) 2 N \zeta \mu^ {- (k - 1)} + N (d + 1) \zeta \mu^ {- k} \\ \leq 3 N (d + 1) \zeta \mu^ {- k}. \\ \end{array}
$$
We make use of Lemma 6 from [CM23], for which we rewrite the statement here for completeness.
Lemma 14 (Lemma 6 in [CM23]). There exists  $a^*$  with  $\| a^* \|_{\infty} \leq 4(d - k)$  such that
$$
\sum_ {l = 0} ^ {d} \sum_ {m = - 1} ^ {d - k} a _ {l m} ^ {*} \sigma_ {l m} (x) = \chi_ {[ k ]} (x). \tag {114}
$$
Lemma 15. Let  $f^{*}(x) = \sum_{l,m}a_{lm}^{*}\sigma_{lm}(x)$  and let  $\hat{f} (x) = \sum_{l,m}a_{lm}^{*}\hat{\sigma}_{lm}(x)$ , with  $\sigma_{lm},\hat{\sigma}_{lm}$  defined in Lemma 13 and  $a^*$  defined in Lemma 14. If  $B\geq (4\zeta^{2}N^{2})^{-1}\log (\frac{Nd + N}{\delta})$ ,  $\tau \leq \zeta \log \left(\frac{Nd + N}{\delta}\right)^{-1 / 2}$  and  $N\geq (d + 1)(d - k + 1)\log ((d + 1)(d - k + 1) / \delta)$ , with probability  $1 - 3\delta$  for all  $x$
$$
L \left(\hat {f}, f ^ {*}, x\right) \leq (d + 1) ^ {2} (d - k + 1) ^ {2} 1 2 N \zeta \mu^ {- k}. \tag {115}
$$
$$
\begin{array}{l} \left| f ^ {*} (x) - \hat {f} (x) \right| = \left| \sum_ {l, m} a _ {l m} ^ {*} \left(\sigma_ {l m} (x) - \hat {\sigma} _ {l m} (x) \right. \right| (116) \\ \leq d (d - k + 1) \| a ^ {*} \| _ {\infty} \sup  _ {l m} | \sigma_ {l m} (x) - \hat {\sigma} _ {l m} (x) | (117) \\ \leq (d + 1) ^ {2} (d - k + 1) ^ {2} 1 2 N \zeta \mu^ {- k}. (118) \\ \end{array}
$$
$$
\begin{array}{l} (1 - \hat {f} (x) f ^ {*} (x)) _ {+} \leq | 1 - \hat {f} (x) f ^ {*} (x) | (119) \\ = \left| f ^ {*} ^ {2} (x) - \hat {f} (x) f ^ {*} (x) \right| (120) \\ = | f ^ {*} (x) | \cdot | f ^ {*} (x) - \hat {f} (x) | \leq (d + 1) ^ {2} (d - k + 1) ^ {2} 1 2 N \zeta \mu^ {- k}, (121) \\ \end{array}
$$
B.3 Second Step: Convergence
To conclude, we use one more time Theorem 6. Let  $\mathcal{L}(a) \coloneqq \mathbb{E}_{x \sim \mathcal{D}}\left[L((a,b^1,w^1),\chi_{[k]},x)\right]$ . Then,  $\mathcal{L}$  is convex in  $a$  and for all  $t \in [T]$ ,
$$
\alpha^ {t} = - \frac {1}{B} \sum_ {s = 1} ^ {B} \nabla_ {a ^ {t}} L \left(\left(a ^ {t}, b ^ {1}, w ^ {1}\right), \chi_ {[ k ]}, x\right) + Z _ {a ^ {t}}. \tag {122}
$$
Thus, recalling  $\sigma = \mathrm{Ramp}$ , we have  $\| \alpha^t \|_{\infty} \leq 1 + \tau$  and  $\| \alpha^t \|_2 \leq \sqrt{N} (1 + \tau)$ . Let  $a^*$  be as in Lemma 14. Clearly,  $\| a^* \|_2 \leq 4(d - k + 1)^{3/2}(d + 1)^{1/2}$ . Moreover,  $a^1 = 0$ . Thus, we can apply Theorem 6 with  $\mathcal{B} = 4(d - k + 1)^{3/2}(d + 1)^{1/2}$  and  $\xi = \sqrt{N} (1 + \tau)$ , and obtain that if:
-  $T = \frac{64}{\epsilon^2} (d - k + 1)^3 (d + 1)N(1 + \tau)^2$ ;
-  $\eta_{t} = \frac{\epsilon}{2N(1 + \tau)^{2}}$  for  $t\in \{1,\dots ,T\}$
then, with probability  $1 - 3\delta$  over the initialization
$$
\mathbb {E} _ {x \sim \mathcal {D}} \left[ \min  _ {t \in [ T ]} L \left(\theta^ {t}, \chi_ {[ k ]}, x\right) \right] \leq \frac {\epsilon}{2} + \frac {\epsilon}{2} = \epsilon . \tag {123}
$$
Let us recall the definition of Cross-Predictability (CP) from [AS20].
Definition 3 (Cross-Predictability (CP), [AS20]). Let  $P_{\mathcal{F}}$  be a distribution over functions from  $\{\pm 1\}^d$  to  $\{\pm 1\}$  and  $P_{\mathcal{X}}$  a distribution over  $\{\pm 1\}^d$ . Then,
$$
\mathrm {C P} \left(P _ {\mathcal {F}}, P _ {\mathcal {X}}\right) = \mathbb {E} _ {F, F ^ {\prime} \sim P _ {\mathcal {F}}} \left[ \mathbb {E} _ {x \sim P _ {\mathcal {X}}} [ F (x) F ^ {\prime} (x) ] ^ {2} \right]. \tag {124}
$$
We invoke Theorem 3 from [AS20], which we restate here.
Theorem 8 ([AS20]). Let  $\mathcal{P}_{\mathcal{F}}$  be a distribution on the class of parities on  $d$  bits, and let  $\mathcal{P}_{\mathcal{X}}$  be a distribution over  $\{\pm 1\}^d$ . Assume that  $\mathbb{P}_{F \sim \mathcal{P}_{\mathcal{F}}, x \sim \mathcal{P}_{\mathcal{X}}} (F(x) = 1) = \frac{1}{2} + o_d(1)$ . Consider any neural network  $\mathrm{NN}(x; \theta)$  with  $P$  edges and any initialization. Assume that a function  $f$  is chosen from  $\mathcal{P}_{\mathcal{F}}$  and then  $T$  steps of noisy-SGD (Def. 1) with learning rate  $\gamma$ , gradient range  $A$ , batch size  $B$  and noise level  $\tau$  are run on the network using inputs sampled from  $\mathcal{P}_{\mathcal{X}}$  and labels given by  $f$ . Then, in expectation over the initial choice of  $f$ , the training noise, and a fresh sample  $x \sim \mathcal{P}_{\mathcal{X}}$ , the trained network satisfies:
$$
\mathbb {P} (\operatorname {s g n} (\mathrm {N N} (x; \theta^ {T})) = f (x)) \leq \frac {1}{2} + \frac {T P A}{\tau} \cdot \left(\mathrm {C P} \left(\mathcal {P} _ {\mathcal {F}}, \mathcal {P} _ {\mathcal {X}}\right) + \frac {1}{B}\right) ^ {1 / 2} \tag {125}
$$
In Theorem 8 we imported Theorem 3 from [AS20] with 1) the tighter bound for the specific case of parities, which gives a term  $(\mathrm{CP} + 1 / B)^{1 / 2}$  instead of  $(\mathrm{CP} + 1 / B)^{1 / 4}$  (see Theorem 4 and remarks in [AS20]); 2) the bound to the junk flow term  $\mathrm{JF_T}$  for noisy-SGD with uniform noise, instead of Gaussian noise, which gives a term  $P$  instead of  $\sqrt{P}$  (see comments in [AS23]).
We thus compute the cross-predictability between  $\mathcal{P}_{\mathcal{X}} = \mathcal{D}$  and  $\mathcal{P}_{\mathcal{F}}$  being the uniform distribution on the set of parities of degree  $k$ .
Lemma 16. If  $\mu = \theta (1)$
$$
\operatorname {C P} \left(\text {U n i f} \{k - p a r i t i e s \}, \mathcal {D}\right) \leq \binom {d} {k} ^ {- 1} + C _ {k} ^ {\prime} \cdot \rho^ {2} \cdot \mu^ {4 k}, \tag {126}
$$
where  $C_k^\prime$  is a constant that depends only on  $k$ .
$$
\begin{array}{l} \operatorname {C P} \left(\text {U n i f} \{\mathrm {k} - \text {p a r i t i e s} \}, \mathcal {D}\right) = \mathbb {E} _ {S, S ^ {\prime}} \mathbb {E} _ {x \sim \mathcal {D}} \left[ \chi_ {S} (x) \chi_ {S ^ {\prime}} (x) \right] ^ {2} (127) \\ = \binom {d} {k} ^ {- 1} + \mathbb {E} _ {S, S ^ {\prime}: S \neq S ^ {\prime}} \mathbb {E} _ {x \sim \mathcal {D}} \left[ \chi_ {S} (x) \chi_ {S ^ {\prime}} (x) \right] ^ {2} (128) \\ = \binom {d} {k} ^ {- 1} + \mathbb {E} _ {S, S ^ {\prime}: S \neq S ^ {\prime}} \rho^ {2} \mathbb {E} _ {x \sim \mathcal {D} _ {\mu}} [ \chi_ {S} (x) \chi_ {S ^ {\prime}} (x) ] ^ {2} (129) \\ \stackrel {(a)} {\leq} \binom {d} {k} ^ {- 1} + \rho^ {2} \cdot \sum_ {l = 0} ^ {k - 1} d ^ {- l} \mu^ {4 (k - l)} \left(C _ {k} + O \left(d ^ {- 1}\right)\right) (130) \\ = \binom {d} {k} ^ {- 1} + \rho^ {2} \cdot \mu^ {4 k} \cdot \sum_ {l = 0} ^ {k - 1} \left(\frac {1}{d \mu^ {4}}\right) ^ {l} \left(C _ {k} + O \left(d ^ {- 1}\right)\right) (131) \\ \sim \binom {d} {k} ^ {- 1} + C _ {k} \cdot \rho^ {2} \cdot \frac {d ^ {- k} - \mu^ {4 k}}{\left(d \mu^ {4}\right) ^ {- 1} - 1}, (132) \\ \end{array}
$$
where in  $(a)$  we used that for  $S$  and  $S^{\prime}$  two randomly sampled sets of  $k$  coordinates, we have:
$$
\begin{array}{l} \mathbb {P} \left(\left| S \cap S ^ {\prime} \right| = l\right) = \frac {\binom {k} {l} \binom {d - k} {k - l}}{\binom {d} {k}} (133) \\ = d ^ {- l} \left(\frac {k \binom {k} {l} \Gamma (k)}{\Gamma (1 + k - l)} + O (1 / d)\right) (134) \\ \leq d ^ {- l} \left(C _ {k} + O (1 / d)\right), (135) \\ \end{array}
$$
where  $C_k$  is such that  $C_k \geq \frac{k\binom{k}{l}\Gamma(k)}{\Gamma(1+k-l)}$  for all  $l \in [k-1]$ .
If  $\mu \gg d^{-1 / 4}$ , then
$$
\operatorname {C P} \left(\text {U n i f} \{\mathrm {k} \text {- p a r i t i e s} \}, \mathcal {D}\right) \leq \binom {d} {k} ^ {- 1} + C _ {k} ^ {\prime} \cdot \rho^ {2} \cdot \mu^ {4 k}, \tag {136}
$$
where  $C_k^\prime$  depends only on  $k$
Finally, we need to discuss that Theorem 5 applies to any fully connected network with weights initialized from a distribution that is invariant to permutation of the input neurons. Let  $\chi_S(x) = \prod_{j\in S}x_j$ , with  $|S| = k$ , and let  $\pi$  be a random permutation of the input neurons. Theorem 8 and Lemma 16 imply that after  $T$  steps of training with noisy-SGD with the parameters  $A,B$  and  $\tau$ , on data generated by  $\mathcal{D}$  and  $\chi_S$ , the trained network is such that
$$
\mathbb {E} _ {\pi} \mathbb {P} (\mathrm {N N} (x; \theta^ {T}) = (\chi_ {S} \circ \pi) (x)) \leq \frac {1}{2} + \frac {T P A}{\tau} \left(\binom {d} {k} ^ {- 1} + C _ {k} ^ {\prime} \rho^ {2} \mu^ {4 k} + \frac {1}{B}\right) ^ {1 / 2}. \tag {137}
$$
Recall that the network is fully connected and the weights outgoing from the input neurons are initialized from a distribution that is invariant to permutation of the input neurons. Consider the action induced by  $\pi$  on the weights outgoing from the input neurons. By properties of SGD, it follows that each conditional probability on  $\pi$  contributes equally to the left hand side of (137), thus the same bound holds also for the single function:
$$
\mathbb {P} (\mathrm {N N} (x; \theta^ {T}) = \chi_ {S} (x)) \leq \frac {1}{2} + \frac {T P A}{\tau} \left(\binom {d} {k} ^ {- 1} + C _ {k} ^ {\prime} \rho^ {2} \mu^ {4 k} + \frac {1}{B}\right) ^ {1 / 2}. \tag {138}
$$
D Proof of Proposition 1
$$
\begin{array}{l} 1 - y ^ {s} \bar {y} - \hat {f} \left(x ^ {s}\right) \left(y ^ {s} - \bar {y}\right) = \left(y ^ {s}\right) ^ {2} - y ^ {s} \bar {y} - \hat {f} \left(x ^ {s}\right) \left(y ^ {s} - \bar {y}\right) (139) \\ = \left(y ^ {s} - \hat {f} \left(x ^ {s}\right)\right) \left(y ^ {s} - \bar {y}\right) (140) \\ = \left(1 - y ^ {s} \hat {f} \left(x ^ {s}\right)\right) \left(1 - y ^ {s} \bar {y}\right) (141) \\ \end{array}
$$
Since  $|\bar{y}| < 1$ ,  $\operatorname{sgn}(1 - y^s\bar{y}) = 1$ , and  $\operatorname{sgn}((1 - y^s\hat{f}(x^s))(1 - y^s\bar{y})) = \operatorname{sgn}(1 - y^s\hat{f}(x^s))$ . Thus,
$$
\begin{array}{l} \epsilon > \frac {1}{m} \sum_ {s \in [ m ]} \left((1 - y ^ {s} \hat {f} \left(x ^ {s}\right)) (1 - y ^ {s} \bar {y})\right) _ {+} (142) \\ = \frac {1}{m} \sum_ {s \in [ m ]} \left((1 - y ^ {s} \hat {f} (x ^ {s})) (1 - y ^ {s} \bar {y})\right) _ {+} \mathbb {1} \left(y ^ {s} \hat {f} \left(x ^ {s}\right) <   1\right) (143) \\ \geq \frac {1}{m} \sum_ {s \in [ m ]} \left((1 - y ^ {s} \hat {f} (x ^ {s})) (1 - y ^ {s} \bar {y})\right) _ {+} \mathbb {1} \left(y ^ {s} \hat {f} (x ^ {s}) <   0\right) (144) \\ \geq \delta \frac {1}{m} \sum_ {s \in [ m ]} \mathbb {1} \left(y ^ {s} \hat {f} \left(x ^ {s}\right) <   0\right). (145) \\ \end{array}
$$
In this section, we provide more details on the implementation of the experiments. Moreover, we present further experiments on the comparison of the curriculum strategy and standard training.
E.1 Experiment Details
E.1.1 Architectures and Loss Functions
Architectures. We used an MLP model trained by SGD under the  $\ell_2$  loss for the results presented in the main part. In this appendix, we further present experimental results using other models, particularly, mean-field [MMN18] and Transformer  $\mathrm{VSP}^{+}17$ . Below, we describe each in detail.
- MLP. The MLP model is a fully-connected architecture consisting of 4 hidden layers of sizes 512, 1024, 512, and 64. The ReLU activation function has been used for each of the layers (except the last one). Moreover, we have used PyTorch's default initialization which initializes weights of each layer with  $U\left(\frac{-1}{\sqrt{\dim_{\mathrm{in}}}}, \frac{1}{\sqrt{\dim_{\mathrm{in}}}}\right)$  where  $\dim_{\mathrm{in}}$  is the input dimension of the corresponding layer.