SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Mean-field. We also use a two-layer neural network with mean-field [MMN18] parametrization. Particularly, we define  $f_{\mathrm{MF}}(x) = \frac{1}{N}\sum_{i=1}^{N}a_i\sigma(\langle w_i,x\rangle + b_i)$  where  $N = 2^{16}$  and  $\sigma = \mathrm{ReLU}$  are the number of neurons and the activation function respectively. We also initialize the weights following  $a_i \sim U(-1,1)$ ,  $w_i \sim U(\frac{-1}{\sqrt{d}},\frac{-1}{\sqrt{d}})^{\otimes d}$ , and  $b_i \sim U(\frac{-1}{\sqrt{d}},\frac{-1}{\sqrt{d}})$ , where  $d$  is the dimension of the input. Note that with this parametrization, one should employ large learning rates (e.g., 1000) assuming SGD is used as the optimizer.
- Transformer. We use the standard encoder part of Transformer architecture  $\left[\mathrm{VSP}^{+}17\right]$  which is commonly used in language modeling  $\left[\left[\mathrm{RSR}^{+}19\right]\right]$  and vision applications (particularly, Vision Transformers  $\left[\mathrm{DBK}^{+}20\right]$ ). More specifically, we first embed  $\pm 1$  into a 256-dimensional vector using a shared embedding layer. Then, the embedded input is processed through 6 transformer layers. In each transformer layer, the size of the MLP hidden layer is also 256, and 6 attention heads are used. Finally, a linear layer is utilized for computing the output. We also use learnable positional embedding.
Loss Functions. We have mainly used the  $\ell_2$  loss for our experiments. Nonetheless, we additionally present results on the hinge and covariance loss which are used in our theoretical results. We briefly review each of these losses below.
-  $\ell_2$  loss. We use the squared loss  $(\hat{y} - y)^2$  for each sample where  $\hat{y}$  and  $y$  are the predicted and the true values respectively.
- Hinge loss. We use the hinge loss defined as  $\max (0,1 - \hat{y} y)$  for each sample where the true label  $y\in \{\pm 1\}$  and the predicted value  $\hat{y}\in \mathbb{R}$ .
- Covariance loss. We also use the covariance loss as defined in Definition 2, i.e.,  $\max(0, (y - \bar{y})(y - \hat{y})) = \max(0, 1 - y\bar{y} - \hat{y}(y - \bar{y}))$ , where  $y \in \{\pm 1\}$ ,  $\bar{y} \in (-1, 1)$  and  $\hat{y} \in \mathbb{R}$  present the true label, the average of the true label (on the training distribution), and the predicted value.
E.1.2 Procedure
We have implemented the experiments using the PyTorch framework  $\left[\mathrm{PGM}^{+}19\right]$ . The experiments were executed (in parallel) on NVIDIA A100, RTX3090, and RTX4090 GPUs and consumed approximately 200 GPU hours (excluding the selection of hyperparameters). Note that each experiment has been repeated using 10 different random seeds and the results are reported with  $95\%$  confidence intervals.
Curriculum Strategy / Standard Training. Here we explain the general setting of training. When using the curriculum strategy, the model is first trained on the sparse samples until the training loss reaches below  $10^{-2}$ . Then, in the second phase of the curriculum, the training is continued on the mixed distribution until the training loss drops below  $10^{-3}$ . On the other hand, during the standard training, the training is done on the mixed distribution, again, until the training loss becomes less than  $10^{-3}$ .
Number of Samples/Number of Iterations. We generally focused on two components to compare the curriculum and standard training: sample complexity and the number of iterations needed for convergence. To compare the sample complexity, for a given number of training samples, we first generate a training set according to  $\rho$  and  $\mu$ . In the case of using the curriculum, the sparse samples are then selected by checking if their Hamming weight satisfies  $H(x) < d\left(\frac{1}{2} - \frac{\mu}{4}\right)$ . Afterward, training is done as described earlier with or without the curriculum. Finally, the test accuracy (or the test loss) is compared. We are also interested in the number of steps needed for the convergence of models whether the curriculum is used or not. To study the number of optimization steps independently of the number of samples, we consider an online setting in which each mini-batch is freshly sampled and there is no fixed training set. Note that in this case, when in the curriculum's first phase, we directly sample the sparse distributions from  $\mathcal{D}_{\mu}$ . Similarly, the model is trained until the training loss drops below  $10^{-3}$ , and then the number of iterations is compared. The only exception is  $f_{\mathrm{middle}}$  in Figure 3, where we were mainly interested in the weak learning of the target function. In that case, the training is stopped when the training loss reaches below 0.26.
Hyperparameter Tuning. Note that the experiments are aimed at comparing the standard and curriculum-based training in a fair and similar context. As a result, we did not perform extensive hyperparameter tuning for our experiments. (Also note that the task and/or the training distribution varies from one experiment to another.) Nonetheless, we tried different batch sizes and learning rates to ensure the robustness of our findings. Generally, we did not observe any significant change in the patterns and potential gains of the curriculum method (of course changes like using a smaller learning rate would make both curriculum and standard training slower). Consequently, we used a moderate batch size of 64 for all of our experiments (other than Transformers) and we selected the learning rate based on the speed and the stability of the convergence. Particularly, we used a learning rate of 0.003 when optimizing the MLP model with SGD under the  $\ell_2$  loss (which covers all experiments presented in the main part). For the mean-field model, we use a learning rate of 1000. Also for the experiments of Transformers, we used the Adam [KB14] optimizer with batch size 512 and learning rate 0.0001.
E.2 Additional Experiments
First, we complete Figure 2 by presenting the plot that shows the gain of the curriculum method in the number of iterations needed for convergence for different parity degrees in Figure 4. Similar to the previous experiments, we consider the parity embedded in dimension  $d = 100$ . As expected, it is shown that parities of higher degree require more iterations to be learned in general. However, this increase in iterations is mild when the curriculum is employed, which leads to an increasing gap between the curriculum and standard training as the degree of parity is increased.
Figure 4: Dependency of the gains of the curriculum strategy in the number of training steps for different degrees of the parity function. The gain of the curriculum method is potentially increased if parities of higher degrees are used.
Next, we move to the mean-field model. Similar to the other experiments, we evaluate the benefit of the curriculum strategy for both the sample complexity and the number of steps needed for convergence. We empirically observed that the mean-field model is able to learn parity functions more easily (with fewer samples and iterations) than the MLP model used in the main text. We thus use parity of degree  $k = 7$  in dimension 100 and  $\mu = 0.98$  to demonstrate the gains of the curriculum strategy more clearly. Particularly, we report the accuracy of the mean-field model trained under the  $\ell_2$  loss for  $\rho = 0.01$  and different training set sizes in Figure 5 (left). Likewise, the number of iterations needed for convergence for different values of  $\rho$  is presented in 5 (right). As expected, the gap between curriculum strategy and standard training in both the sample complexity and number of optimization steps are similarly exhibited if a mean-field model is used.
Figure 5: Comparison of curriculum and standard training for the mean-field model trained under the  $\ell_2$  loss; (left) accuracy of the model trained on training sets with varying sizes, (right) the number of iterations needed for convergence of the model. The general picture of the potential benefits of the curriculum method remains unchanged.
Moreover, we study other loss functions studied in this paper. To this end, we keep the same setting as above, considering the mean-field model, parity of 7 bits, and  $d = 100$ ,  $\mu = 0.98$ , and  $\rho = 0.01$  (unless varying). We report the results of the hinge and covariance loss in Figure 6 and Figure 7 respectively. It can be seen that the patterns are generally similar to the case that  $\ell_2$  loss was being used (Figure 5). In sum, the curriculum method makes the learning of the parity function possible with fewer samples and iterations for small enough values of  $\rho$ .
Figure 6: Potential benefits of the curriculum strategy for the mean-field model trained with the hinge loss; (left) accuracy for different training set sizes, (right) the number of iterations (with fresh samples) needed for the convergence of the model. Similar to the case of the  $\ell_2$  loss, the curriculum is beneficial for small enough values of  $\rho$ .
Figure 7: Comparing curriculum strategy and standard training for the mean-field model trained with the covariance loss; (left) accuracy for different training set sizes, (right) the number of optimization steps needed for learning the parity function. Similar to experiments where  $\ell_2$  or hinge loss are used, curriculum potentially reduces the number of iterations and also the number of samples needed for learning.
Finally, we evaluate the curriculum strategy for the Transformer model trained under the  $\ell_2$  loss. To make the optimization of the model faster, we use Adam [KB14] optimizer with batch size 512 for this experiment. We also stop the training when the training loss reaches below  $10^{-2}$  instead of  $10^{-3}$ . Similar to the above-mentioned setting, we set  $\mu = 0.98$ ,  $\rho = 0.01$  (unless varying),  $d = 100$ , and we consider learning the parity of degree  $k = 7$ . The validation accuracy of the Transformer for different training set sizes is presented in Figure 8 (left). It can be seen that there is a significant, albeit weak, advantage in terms of sample complexity when the curriculum is used. Similarly, the number of iterations with fresh samples needed for convergence of the model is shown in Figure 8 (right). The gain in the number of steps is also weaker than the other models, nonetheless, it is significant (especially for small values of  $\rho$  such as 0.001).
Figure 8: Evaluating the curriculum for the Transformer model in terms of sample complexity (left) and the number of iterations (right). It can be seen that the curriculum can still be significantly beneficial, although the difference between the curriculum and standard training is weaker in comparison to the MLP and mean-field model.