SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $L_{\mu} = \mu^{k - 1} - \mu^{k + 1}$ , then, with probability at least  $1 - 6Nd^{-C^*}$ , the algorithm outputs a network such that  $\frac{1}{m}\sum_{s\in [m]}L_{\mathrm{cov}}(\theta^{T_1 + T_2},x^s,y^s)\leq \epsilon$ .
The following corollary is a direct consequence of Theorem 3 and Proposition 1.
Corollary 1. Let the assumptions of Theorem 3 be satisfied and assume that  $\tau = O(\kappa)$  and  $\Delta, k$  constants. There exists a 2-layer ReLU network of size at most  $(d + 2)(k + 1)$  and initialization invariant to permutations of the input neurons, such that for all  $\epsilon > 0$  the layer-wise curriculum noisy-SGD algorithm with gradient range  $A = \tilde{O}(\sqrt{d})$ , after at most  $T = \tilde{O}(d)$  steps of training with bounded learning rates, outputs a network such that  $\mathbb{P}_{x \sim \mathcal{D}_{\mathrm{mix}}}\left(\operatorname{sgn}(\mathrm{NN}(x; \theta^T)) = f(x)\right) \geq 1 - \epsilon$ , with high probability.
Remark 2. The choice of the covariance loss is motivated by a simplification of the proof of Theorem 3. Indeed, with this loss, we can show that in the first part of training the weights incident to the relevant coordinates move non-negligibly, while the other weights stay close to zero (see proof outline). If we used another loss, say the hinge loss, all weights would move non-negligibly. However, we believe that one could still adapt the argument to cover the hinge loss case, by adding another projection to the iterates. In the following, we also give a positive result for the hinge loss, with a different proof technique, that holds for large batch size and large learning rate (Theorem 4).
Remark 3. While we show that with appropriate hyperparameters, the layer-wise curriculum-SGD can learn  $k$ -parities in  $\tilde{O}(d)$  steps, we need to notice that to implement the curriculum, one has to first retrieve  $(X_1, Y_1)$ . This involves computing the Hamming weight of  $d / \rho$  samples, which adds computational cost if the curriculum learning includes this part. Nevertheless, in the experiments that we performed, the recovery of  $(X_1, Y_1)$  was consistently efficient, with an average of less than 1 second running time for a dataset of 10 million samples of dimension 100. On the other hand, the reduction of training steps due to the curriculum yields significant time savings.
Remark 4. Theorem 3 provides an upper bound on the sample complexity required to learn using a curriculum, of  $\tilde{O}(dB / \rho)$  for SGD with batch size  $B$ . Consequently, while the number of iterations needed for learning remains unaffected by  $\rho$ , the sample complexity increases as  $\rho$  decreases. Additionally, we do not currently have a lower bound on the sample complexity for learning parities with standard training with offline SGD. However, our experiments in Section 5 demonstrate a noticeable difference in the sample complexity required for learning with and without a curriculum. These findings may encourage future work in establishing a theoretical advantage in the number of samples needed to learn with curriculum.
Remark 5. We further remark that the proof method of Theorem 3 yields a tighter upper bound on curriculum learning's sample complexity than the 1-step argument used in [CM23][Thm 4]. Specifically, using the technique of [CM23][Thm 4], we achieve an upper bound of  $\tilde{O}(d^2/\rho)$ , as opposed to the  $\tilde{O}(d/\rho)$  given by Theorem 3 for SGD with bounded batch size.
Remark 6.  $T_{1}$  depends on the degree of the parity  $k$  through the parameter  $L_{\mu}$ . We believe that such dependence cannot be removed, however, we remark that choosing  $\mu$  sufficiently close to 1 (e.g.  $\mu = 1 - \frac{1}{k}$ ) allows bounding  $L_{\mu}$  independently from  $k$ . Indeed, our experiments (Figure 4) show mild dependence on the parity degree for curriculum learning with large  $\mu$ .
Proof Outline. The proof of Theorem 3 follows a similar strategy as in [AGJ21, ABAM23, TV19]. We decompose the dynamics into a drift and a martingale contribution, and we show that the drift allows us to recover the support of the parity in  $\tilde{O}(d)$  steps, while in the same time horizon, the martingale contribution remains small, namely of order  $O(1 / \sqrt{d\log(d)})$ . In particular, we show that
after the first phase of training, with high probability, all weights corresponding to coordinates in the parity's support are close to  $\Delta$  and the others are close to zero. The fit of the second layer on the whole dataset corresponds to the analysis of a linear model and follows from standard results on the convergence of SGD on convex losses (e.g., in [SSBD14]). The formal proof can be found in Appendix A.
Hinge Loss. While the use of the covariance loss simplifies considerably the proof of Theorem 3, we believe that the advantages of curriculum extend beyond this specific loss. Here, we present a positive result that applies to a more commonly used loss function, namely the hinge loss.
Theorem 4 (Hinge Loss). Let  $\mathrm{NN}(x;\theta) = \sum_{i=1}^{N} a_i \sigma(w_i x + b_i)$  be a 2-layer fully connected network with activation  $\sigma(y) \coloneqq \operatorname{Ramp}(y)$  and  $N = \tilde{\theta}(d^2 \log(1/\delta))$ . Assume we have access to a dataset  $(X,Y) = \{(x^s, y^s)\}_{s \in [m]}$ , with  $x^s \stackrel{iid}{\sim} \mathcal{D}_{\mathrm{mix}}$ , with parameters  $\rho > 0$ ,  $\mu \in (0,1)$ , and  $y^s = \chi_S(x)$ , for some target parity  $\chi_S$ , with  $|S| = k$ . Then, there exists an initialization and a learning rate schedule, such that for any target parity  $\chi_S$  and for any  $\epsilon > 0$ , the layer-wise curriculum SGD on the hinge loss with batch size  $B = \tilde{\theta}(d^{10}/\epsilon^2 \log(1/\delta))$ , noise level  $\tau = \tilde{\theta}(\frac{\epsilon \mu^k \log 1/\delta}{d^6})$ ,  $T_1 = 1$  and  $T_2 = \tilde{\theta}(d^6/\epsilon^2)$  with probability  $1 - 3\delta$ , outputs a network such that  $\mathbb{P}_{x \sim \mathcal{D}_{\mathrm{mix}}}(\operatorname{sgn}(\mathrm{NN}(x; \theta^T)) = f(x)) \geq 1 - \epsilon$ .
As opposed to Theorem 3, Theorem 4 only holds for SGD with large batch size and unbounded learning rate schedules. The proof of Theorem 4 follows a 1-step argument, similarly to [CM23], and it is deferred to Appendix B.
4 Negative Result for Standard Training
In this section, we present a lower bound for learning  $k$ -parities on the mixed distribution of eq. (2), without curriculum.
Theorem 5 (Main Negative Result). Assume we have access to a dataset  $(X,Y) = \{(x^{s},y^{s})\}_{s\in [m]}$ , with  $x^{s} \stackrel{iid}{\sim} \mathcal{D}_{\mathrm{mix}}$ , with parameters  $\rho = o_{d}(1), \mu \in (0,1)$ , and  $y^{s} = \chi_{S}(x)$ , with  $|S| = k$ . Let  $\mathrm{NN}(x;\theta)$  be a fully-connected neural network of size  $P$ , with initialization that is invariant to permutations of the input neurons. Then, the noisy-SGD algorithm with noise level  $\tau$ , gradient range  $A$ , batch size  $B$ , any learning rate schedule, and any loss function, after  $T$  steps of training without curriculum, outputs a network such that
$$
\mathbb {P} _ {x \sim \mathcal {D} _ {\mathrm {m i x}}} (\operatorname {s g n} (\mathrm {N N} (x; \theta^ {T})) = f (x)) \leq \frac {1}{2} + \frac {T P A}{\tau} \cdot \left(\binom {d} {k} ^ {- 1} + C _ {k} \rho^ {2} \mu^ {4 k} + \frac {1}{B}\right) ^ {1 / 2}, \tag {8}
$$
where the probability is over  $x \sim \mathcal{D}_{\mathrm{mix}}$  and any randomness in the algorithm, and where  $C_k$  is a constant that depends on  $k$ .
Remark 7. For the purposes of Theorem 5, the network can have any fully connected architecture and any activation such that the gradients are well-defined almost everywhere. Furthermore, the loss can be any function that is differentiable almost everywhere.
Theorem 5 follows from an SQ-like lower bound argument and builds on previous results in [AS20][Theorem 3]. We defer the formal proof to Appendix C. Theorem 5 implies the following corollary.
Corollary 2. Under the assumptions of Theorem 5, if  $\rho = \tilde{O}(d^{-3.5 - \delta})$  and  $k \geq 6$ , any fully connected network of size at most  $\tilde{O}(d)$  with any permutation-invariant initialization and activation, trained by the noisy-SGD algorithm with batch size  $B = \tilde{\Omega}(\rho^{-2})$ , noise-level  $\tau = \tilde{O}(1 / d)$  and gradient range  $A = \tilde{O}(\sqrt{d})$ , after  $T = \tilde{O}(d)$  steps of training, will output a network such that  $\mathbb{P}_{x \sim \mathcal{D}_{\mathrm{mix}}}(\mathrm{sgn}(\mathrm{NN}(x; \theta^T)) = f(x)) \leq \frac{1}{2} + C d^{-\delta} \log(d)^c$ , for some  $c, C > 0$ .
In particular, in Corollary 2, the network can have any width and depth (even beyond 2 layers), as long as the total number of parameters in the network scales as  $\tilde{O}(d)$  in the input dimension. Combining Corollary 1 and Corollary 2, we get that if  $\rho = \tilde{O}(d^{-3.5 - \delta})$ , for some  $\delta > 0$ , and  $k \geq 6$  and for  $\epsilon = 1/10$ , say, a 2-layer fully connected network can learn any  $k$  parities up to accuracy  $1 - \epsilon$  in  $T = \theta(d)$  steps with curriculum, while the same network in the same number of steps without curriculum, can achieve accuracy at most  $1 - 2\epsilon$ , for  $d$  large enough.
Figure 1: Comparison of the performance of the curriculum strategy and standard training for different training set sizes and values of  $\rho$ . In the bottom-left plot, we report the number of samples needed to achieve accuracy close to 1 for different values of  $\rho$  based on the other plots.
Remark 8. While our results show a first separation in the number of training steps between curriculum and standard training, we do not claim that the given range of  $\rho$  is maximal. For instance, one could get a tighter negative bound by considering Gaussian, instead of uniform, noise in the noisy-SGD iterates. However, getting a positive result with the curriculum for Gaussian noise would require a more involved argument. Our experiments show separation in the number of training steps and in the sample complexity for different values of  $\rho$ .
In this section, we present empirical results demonstrating the benefits of the proposed curriculum method. We use a multi-layer perceptron (MLP) with 4 hidden layers of sizes 512, 1024, 512, and 64 as our model. We optimize the model under the  $\ell_2$  loss using mini-batch SGD with batch size 64. We also train all layers jointly. Each of our experiments is repeated with 10 different random seeds, and the results are reported with  $95\%$  confidence intervals. Note that generalization error is always computed based on  $\mathcal{D}_{\mathrm{mix}}$ . Here, we mostly focus on the common settings not covered by our theoretical results. We present additional results on other architectures, namely, two-layer mean-field networks [MMN18] and Transformers  $[\mathrm{VSP}^{+}17]$  and covariance and hinge loss functions in Appendix E.2.
Number of Samples. First, we focus on the difference in sample complexity, between curriculum and standard training. We consider learning  $f(x_{1},\ldots ,x_{100}) = x_{1}x_{2}x_{3}x_{4}x_{5}$ , with inputs sampled from a mixed distribution with  $\mu = 0.98$  and different values of  $\rho = 10^{-4}$ ,  $10^{-3}$ ,  $10^{-2}$ , 0.1, 0.5. For each  $\rho$ , we train the network on training sets of different sizes (between  $10^{4}$  and  $10^{6}$ ) with and without curriculum. When using the curriculum, we first select the sparse samples as those with Hamming weight smaller than  $\left(\frac{1}{2} -\frac{\mu}{4}\right)d$ , and we train on those until convergence (specifically, training loss smaller than  $10^{-2}$ ). We then train the model on the entire training set until convergence. Similarly, when the curriculum is not used, the model is trained using all the available samples until convergence.
Figure 1 shows the validation accuracy achieved for different values of  $\rho, m$ . It can be seen that for some values of  $\rho$ , there is a reduction in the number of samples needed to achieve validation accuracy close to 1 when the curriculum is used. For the purpose of visualization, in Figure 1 (bottom-right), we sketch the number of samples needed to achieve accuracy close to 1 for different values of  $\rho$ , based on the previous plots. We notice that the sample complexity decreases as  $\rho$  is increased, for both curriculum and standard training. Among the  $\rho$  values that we tried, the gap between the two training strategies is maximal for  $\rho = 0.01$ , and it decreases as  $\rho$  gets larger or smaller. Also, note
$^{4}$ Code: https://github.com/aryol/parity-curriculum
Figure 2: (Top-left) Number of training steps needed for learning 5-parity for different values of  $\rho$ . The gap between curriculum and standard training decreases as  $\rho$  increases. (Top-right) The number of training steps for different values of  $\mu$  based on  $\rho$ . (Bottom-right) Performance for different values of  $\mu$  and different sample sizes. Curriculum boosts the learning for large values of  $\mu$ . (Bottom-left) Performance for different parity functions. (The curves of standard training for  $k = 7$  and  $k = 10$  overlap.) The benefit of curriculum potentially increases with the degree of the parity.
that for small values of  $\rho$  the sample complexity for standard training is constant and equals  $10^{6}$ . We note that with  $10^{6}$  samples, we can learn  $f(x)$  even with uniform inputs  $(\rho = 0)$ .
Number of Training Steps. Now we focus on the number of iterations needed for learning the same  $f$  as above. In order to study the number of iterations independently of the number of samples, we draw fresh mini-batches at each step. When using the curriculum, we sample from  $\mathcal{D}_{\mu}$  in the initial phase and from  $\mathcal{D}_{\mathrm{mix}}$  in the second phase, while when using standard training we sample from the mixed distribution at each step. In all cases, we train until convergence. The number of iterations needed for learning  $f$  for different values of  $\rho$  is depicted in Figure 2 (top-left). It can be seen that the number of training steps needed for learning with the curriculum is constant w.r.t.  $\rho$ . In contrast, for standard training, the number of iterations increases with the decrease of  $\rho$ . Both these observations are consistent with our theoretical results. In particular, for small values of  $\rho$ , the curriculum significantly reduces the number of training steps, while for large  $\rho$  it provides little or no benefit, and for  $\rho = 0.5$ , it is harmful.
Dependency on  $\mu$  and  $k$ . Next, we investigate the dependency of our results on  $\mu$  and on the degree of parity  $k$ . For evaluating dependency on  $\mu$ , we consider the function  $f$  as above and different values of  $\mu$ . First, we consider the number of training steps needed for learning  $f$  based on  $\rho$ . The results are shown in Figure 2 (top-right). It can be seen that while the curriculum is not beneficial for  $\mu = 0.02$  (where  $\mathcal{D}_{\mu}$  is close to  $\mathcal{D}_u$  and all samples are dense), it can significantly reduce the number of training steps for moderate and large values of  $\mu$ . Further, we fix  $\rho = 0.01$  and compare the performance of the curriculum and standard training for different training set sizes. The results are demonstrated in Figure 2 (bottom-right). It is shown that for large values of  $\mu$  such as 0.98 and 0.90, there is a gap in the number of samples needed for achieving generalization accuracy close to 1 between curriculum and standard training. On the other hand, for smaller values of  $\mu$ , this gain is diminished. Nonetheless, note that the relationship between the performance of the curriculum method and  $\mu$  is not monotonous (e.g., see the number of steps plot), and this is indeed reflected in our analysis. More precisely, in Theorem 1,  $T_1$  scales with  $1 / L_{\mu}^2$  where  $L_{\mu} = \mu^{k - 1} - \mu^{k + 1}$ .
Figure 3: Considering gains in sample complexity (top row) and number of steps (bottom row) beyond parity targets. Plots in the left (corresponding to  $f_{\mathrm{left}}(x) = x_1x_2x_3x_4x_5 + \frac{1}{2} x_1x_2x_3x_4x_5x_6$ ), middle ( $f_{\mathrm{middle}}(x) = x_1x_2x_3x_4x_5 + \frac{1}{2} x_6 \cdots x_{11}$ ), and right ( $f_{\mathrm{right}}(x) = \frac{1}{2} x_1x_2 + \frac{1}{2} x_1 \cdots x_6$ ) columns present cases in which curriculum is beneficial, is only beneficial for weak learning, and is not beneficial, respectively.
To analyze the effect of the degree of parity,  $k$ , we fix  $\mu = 0.98$ ,  $\rho = 0.01$  and the input dimension  $d = 100$ , and we compare the performance of the model trained with different amounts of training data. The results are portrayed in Figure 2 (bottom-left). One can see that, while without curriculum the parity problem becomes harder to learn as  $k$  increases, the complexity with curriculum has mild dependence on  $k$  and it allows one to learn all the parities considered with  $10^{5}$  samples. The plot exhibiting the dependence of the number of optimization steps on  $k$  is reported in Appendix E.2.
Beyond Parities. Finally, we study target functions composed of multiple monomials. We consider three examples in 100-dimensional space: (i)  $f_{\mathrm{left}}(x) = x_1x_2x_3x_4x_5 + \frac{1}{2} x_1x_2x_3x_4x_5x_6$ , (ii)  $f_{\mathrm{middle}}(x) = x_1x_2x_3x_4x_5 + \frac{1}{2} x_6 \cdots x_{11}$ , and (iii)  $f_{\mathrm{right}}(x) = \frac{1}{2}(x_1x_2 + x_1 \cdots x_6)$ . We also fix  $\mu = 0.98$ . In Figure 3, we investigate the gain of curriculum method in sample complexity for  $\rho = 0.01$  (top row) and the number of training steps (bottom row) needed to learn. The plots in the bottom row are obtained by training the network with fresh batches. The left, middle, and right column correspond to  $f_{\mathrm{left}}$ ,  $f_{\mathrm{middle}}$ , and  $f_{\mathrm{right}}$ , respectively. Note that curriculum is beneficial for  $f_{\mathrm{left}}$ , while it hardly helps with  $f_{\mathrm{right}}$ . Interestingly, for  $f_{\mathrm{middle}}$ , curriculum is advantageous for weak learning of the function. More specifically, it helps learning  $x_1x_2x_3x_4x_5$  monomial of  $f_{\mathrm{middle}}$ . Consequently, for the corresponding iterations plot (Figure 3 bottom-middle), we plot the number of iterations needed for getting a loss below 0.26. This suggests that the curriculum method put forward in this paper mostly helps with learning of the monomial with the lowest degree. We leave improvements of our curriculum method to future work.
In this paper, we introduced a curriculum strategy for learning parities on mixed distributions. Our approach involves training on sparse samples during the initial phase, which leads to a reduction in the number of required training steps and sample complexity compared to standard training. In the experiments section, we presented some results for certain functions with multiple monomials. Although a complete picture requires further investigation, these results suggest that starting from sparse samples provides advantages in the early stages of training. It would be interesting to investigate what variation of our curriculum strategy fits best the case of multiple monomials, beyond the initial phase. For example, one could explore multiple curriculum phases where training progresses from samples with increasing Hamming weight, or employ a self-paced curriculum that dynamically selects samples during training. Such analyses could offer valuable insights into the broader applicability of our approach. Here we focused mainly on establishing a formal separation in the number of training steps for the canonical case of parities.
$\left[\mathrm{RSR}^{+}19\right]$  Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
[SAJ10] Valentin I Spitkovsky, Hiyan Alshawi, and Dan Jurafsky. From baby steps to leapfrog: How "less is more" in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 751-759, 2010.
[SGG14] Patrick Shafto, Noah D Goodman, and Thomas L Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive psychology, 71:55-89, 2014.
[SIRS22] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, pages 1-40, 2022.
[SMS22] Luca Saglietti, Stefano Sarao Mannelli, and Andrew Saxe. An analytical theory of curriculum learning in teacher-student networks. Journal of Statistical Mechanics: Theory and Experiment, 2022(11):114014, 2022.
[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[STD19] Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning a differentiable curriculum. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[TV19] Yan Shuo Tan and Roman Vershynin. Online stochastic gradient descent with arbitrary initialization solves non-smooth, non-convex phase retrieval. arXiv preprint arXiv:1910.12837, 2019.
$\left[\mathrm{VSP}^{+}17\right]$  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[WA20] Daphna Weinshall and Dan Amir. Theory of curriculum learning, with convex loss functions. Journal of Machine Learning Research, 21(222):1-19, 2020.
[WCA18] Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In International Conference on Machine Learning, pages 5238-5246. PMLR, 2018.
[WCZ21] Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
$\left[\mathrm{ZKK}^{+}18\right]$  Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy G Winnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation. arXiv preprint arXiv:1811.00739, 2018.
[ZS14] Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4615, 2014.
A Proof of Theorem 3
Algorithm 1 Layer-wise curriculum-SGD. Init. scales  $\kappa, \Delta > 0$ , learning rates  $\gamma_1, \gamma_2 > 0$ , step counts  $T_1, T_2$ , input param.  $\rho, \mu$ , batch size  $B$ , noise level  $\tau$  and gradient range  $A$ .