An Analytical Theory of Curriculum Learning in Teacher-Student Networks
Authors: Luca Saglietti
Stefano Sarao Mannelli
Andrew M Saxe
Venue: NeurIPS 2022 Accept (2022)
Keywords: learning
curriculum learning
theory
statistical mechanics
generalization model
fading
structured data

Abstract:     In animals and humans, curriculum learning---presenting data in a curated order---is critical to rapid learning and effective pedagogy. 
    A long history of experiments has demonstrated the impact of curricula in a variety of animals but, despite its ubiquitous presence, a theoretical understanding of the phenomenon is still lacking. 
    Surprisingly, in contrast to animal learning, curricula strategies are not widely used in machine learning and recent simulation studies reach the conclusion that curricula are moderately effective or ineffective in most cases. 
    This stark difference in the importance of curriculum raises a fundamental theoretical question: when and why does curriculum learning help? 
    In this work, we analyse a prototypical neural network model of curriculum learning in the high-dimensional limit, employing statistical physics methods. 
    We study a task in which a sparse set of informative features are embedded amidst a large set of noisy features. We analytically derive average learning trajectories for simple neural networks on this task, which establish a clear speed benefit for curriculum learning in the online setting. However, when training experiences can be stored and replayed (for instance, during sleep), the advantage of curriculum in standard neural networks disappears, in line with observations from the deep learning literature. 
    Inspired by synaptic consolidation techniques developed to combat catastrophic forgetting, we investigate whether consolidating synapses at curriculum change points can boost the benefits of curricula. We derive generalisation performance as a function of consolidation strength (implemented as a Gaussian prior connecting learning phases), and show that this consolidation mechanism can yield a large improvement in test performance.
    Our reduced analytical descriptions help reconcile apparently conflicting empirical results, trace regimes where curriculum learning yields the largest gains, and provide experimentally-accessible predictions for the impact of task parameters on curriculum benefits. More broadly, our results suggest that fully exploiting a curriculum may require explicit consolidation at curriculum boundaries.

An Analytical Theory of Curriculum Learning in Teacher-Student Networks
Luca Sagliettil Stefano Sarao Mannellil:", and Andrew Saxe
Abstract
In animals and humans, curriculum preneentnngg data in a curated order- is critical to rapid learning and effective pedagogy. A long history of experiments has demonstrated the impact of curricula in a variety of animals but, despite its ubiquitous presence, theoretical understanding of the phenomenon is still lacking. Surprisingly, in contrast to animal learning, curricula strategies are not widely used in machine learning and recent simulation studies reach the conclusion that curric- ula are moderately effective or even ineffective in most cases. This stark difference in the importance of curriculum raises fundamental theoretical question: when and why does curriculum learning help? In this work, we analyse a prototypical neural network model of curriculum learning in the high-dimensional limit, employ- ing statistical physics methods. We study a task in which a sparse set of informative features are embedded amidst large of noisy features. We analytically derive average learning trajectories for simple neural networks on this task, which estab- lish clear speed benefit for curriculum learning in the online setting. However, when training experiences can be stored and replayed the advantage of curricu- lum in standard neural networks disappears, in line with observations from the deep learning literature. Inspired by synaptic consolidation techniques developed to combat catastrophic forgetting, we propose curriculum-awar algorithms that consolidate synapses at curriculum change points and investigate whether this can boost the benefits of curricula. We derive generalisation performance as a function of consolidation strength (implemented as an L2 regularisation/elasiic coupling algorithms can yield
connecting learning phases), and show that curriculum-awas large improvement in test performance. Our reduced analytical descriptions help reconcile apparently confliciing empirical results, trace regimes where curriculum learning yields the largest gains, and provide pperimeettally-accssiii predictions for the impact of task parameters on curriculum benefits. More broadly, our results suggest that fully exploiting curriculum may require explicit adjustments in the loss.
1 Introduction
Presenting learning materials in meaningful order according curriculum greatly helps learning in animals and humans [T 2 and is considered an essential aspect of good pedagogy [5]. For example, humans have been shown to learn visual discriminations faster when presented with examples that exaggerate the relevant difference between classes, a phenomenon known as "fading" Beyond humans, curricula in the form of "shaping" "staircase" procedures are near- universal feature task designs in animal studies, without which training often fails entirely. For
6 3
Department of Computing Sciences, Bacconi University.
$ Gatsby Computational Neuroscience Unit & Sainsbury Wellcome Centre, University College London.
3 FAIR, Meta Al Equal contributions.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

instance, the International Brain Laboratory task, standardised perceptual Decision-making training paradigm in mice, involves six stages of increasing difficulty before reaching final performance (
Building from this intuition, a seminal series of papers proposed a similar curriculum learning approach for machine learning (ML) TO 12]. In striking contrast to the clear benefits of curriculum in biological systems, however, curriculum learning has generally yielded equivocal benefits in artificial systems. Experiments in a variety of domains [13 [4] have found usually modest speed and generalisation improvements from curricula. Recent extensive empirical analyses have found minimal benefits on standard datasets [T5]. Indeed, a common intuition in deep learning practice holds that training distributions should ideally be as close as possible to testing distributions, a notion which runs counter curriculum. Perhaps the only areas where curricula are actively used are in large language models [[6] and certain reinforcement learning settings [17].
This gap between the effect of curriculum in biological and artificial learning systems poses puzzle for theory. When and why is curriculum learning useful? What properties of a task determine the extent of possible benefits? What ordering of learning material is most beneficial? And can new learning algorithms better exploit curricula? Compared to the empirical investigations of curriculum learning, theoretical results on curriculum learning remain sparse. Most notably, [8] [9] show that curriculum can lead to faster learning in a simple setting, but the effects of curriculum on asymptotic generalisation and the dependence on task structure remain unclear. A hint that indeed curriculum learning might lead to statistically different minima comes from a connection between constraint- satisfaction problems and physics results on flow networks (20] but to our knowledge no direct result has been reported in the modern theoretical MI. literature.
In this work we study the impact of curriculum using the analytically tractable teacher-student
framework and the tools of statistical physics 21 22 23 24]. High-dimensional teacher-student
models are popular approach for systematically studying learning behaviour in neural networks
125 26 22], and have recently been leveraged to analyse a variety of phenomena [27 [8 29 30 31
32.
Using simple model to build structured data [12. we examine the impact of ordering examples by increasing difficulty (curriculum), decreasing difficulty (anti-curris cunticcuri, or standard shuffled training. We derive exact expressions for the online learning dynamics and the performance of batch learning. However, in the latter, curriculum confers no benefit under standard training in our model setting. Motivated by theories of synaptic consolidation and elastic weight consolidation 33 we introduce elastic penalties (Gaussian priors) that regularise training toward solutions obtained in prior curriculum phases, instantiating a long-term memory effect. With these priors, curriculum yields benefits both in the online 3 and in the batch 4 settings. Settings
Further related work. The first empirical investigation of curriculum learning appeared in 1927 35). consisting in a visual discrimination task for dogs under curriculum and no-curriculum paradigms. Later behavioural studies proved curricula to be beneficial independent of the ani- mal (dogs, mice, rats, pigcons, humans) and the data modality (visual, auditory, or tactile stimuli) 36 2 37 38 6). However, these experimental observations were not observed in standard artificial neural networks (ANNs). Several ideas in the connectionist community were proposed in order to show curriculum effects in the learning dynamics of ANNs (39 40 10 IT]. While these studies were able to match previous experimental data, they also required substantial changes in the architecture of the ANN and/or in the learning rule.
Except for very few instances [6 17), standard ML practice tends to avoid taking curricula into account. An obvious obstacle is the fact that most datasets do not provide meta-data about sample difficulties. An interesting line of research pointed out the possible relevance of implicit curricula, based on the observation that neural networks tend to consistently Icarn the samples in a certain order 41.. Thus, a possible way of addressing the lack of difficulty labels would be to use the natural learning order indicative of the various difficulties of the training samples. However, recent work [5], which compared several heuristics for curriculum learning including implicit curricula- in variety of settings, showed limited benefits with this strategy.
The picture that emerges from the literature seems contradictory: on the one hand, curricula appear fundamental to biological learning; on the other hand, curricula appear largely irrelevant in many machine learning settings. The core motivation behind our work is to reconcile these views and contribute theoretical understanding of curriculum learning.

---

(B)
Teacher
(b)
(c)
Curiculum
Input
W
y
Easy
Hard
70
Reieoun
Anii-curiculum
83
Student
Hand
Easy
Imokcont
No carriculum
W
Randomll shutlled
arti-cumtuloo
curricue
Example #
Figure 1: Teacher-student setting for curriculum learning. (a) Illustration of teacher-student
setting in which a "student" network is trained from i.id. inputs with labels from a "teacher" network. Since the teacher network is sparse, its output depends only subset of relevant input features. (b) We consider curricula which order examples by difficulty, here taken to be the variance in the irrelevant feature dimensions. We refer to increasing, decreasing, and random difficulty order as curriculum, anti-curriculum, and no curriculum, respectively. (c) Example test error on hard examples for the student over training. The switch- point between easy and hard samples lies at a = 1/2. Solid lines show numerical simulations, while dashed lines show theoretical predictions derived in Section 3 For this particular parameter setting, curriculum speeds learning but only modestly improves final performance at a = 1. Parameters: an = 1. 02 = 1. A = 0, D2 1, 7 = 10-5, 77 = 3.
Model definition and overview of approach
In the following, we revisit prototypical model of curriculum learning from [2] that finds corre- spondence to the fading literature [6] as highlighted in Sec. [ Our setting is summarised in Fig. [ The model entails simple teacher-student setup, where teacher and student are each shallow 1-layer neural networks of size N (also known perceptrons). The learning task for the student is a binary classification problem, with dataset D = {vV (v*) } where the ground-truth labels are produced by the teacher network y" sign W T x"" The student learns via empirical risk minimisation of an L2 regularised convex loss.
A key feature of this model is that the teacher network is sparse, with only fraction P of ~ (O0,, 1) non-zero components. Therefore, in order to achieve a good test accuracy, the student has to guess which components should be set to zero and align the relevant weights in the correct direction. A large range of 0 < < could give rise to the phenomenology we seek to analyse. In the remainder of the paper we will focus on the case p 0.5.
We model the variable degree of difficulty in the samples by decomposing each input vector as x" [xf" x R.N. where x e RON denotes the relevant components of the input, and x ((l-))N the irrelevant ones. Note that, erucially, the sparse teacher network is completely blind to the irrelevant part of the input: y sign Σø Wr.jz" While a4, i.i.d. W/ 1). Vu we consider the variance for the irrelevant components to be sample-dependent wft ~ (V(0,, A A A smaller variance in the irrelevant part induces higher SNR in the student learning problem.
The dataset is partitioned according to difficulty levels given by the variances of the irrelevant inputs. For simplicity we consider only two partitions in most of our analysis, but generalisations to multiple difficulty levels follow straightforwardly. We thus have a dataset with M = (an + o2) = aN samples in total. In the first 0r N samples the irrelevant inputs have variance A, while for the remaining a2 samples the variance 2₂ > A₁- In the curriculum learning condition we present the easy examples first, while in the anti-curriculum condition we present the hard examples first. Standard learning presents examples shuffled in random order.
In 12 the input distribution is uniform between O and but this does qualitatively change the results.
3

---

3 Online dynamical solution in the large input limit
We start by focusing on the same online learning setting explored in [12.. We consider a 1-layer student network with sigmoidal activation function, (.) = erf(./v2) that learns to minimise a mean square emor loss with L2 regularisation of intensity y, using gradient descent. This yields the updates
WF-x" (a W zz"
W WH+LWH
y*
x WW.
The dynamics of the model can be analysed in the high-dimensional limit N, M + oo with a M/N (1). Generalising the results of [26 42 on the online stochastic gradient descent dynamics in single-layer regression problems, we obtain precise description of the performance at all times, as function of several order parameters: the squared norm of the relevant and irrelevant part of the student weights Q, = Ñ W W" and Q; = N W W', respectively; the overlap of the relevant weights of the student and teacher R W WT; and the squared norm of the teacher vector T = W Wr. In particular, given Qr, Qi R and T, the test loss (i.e. average loss new example) on a dataset with variance A in the irrelevant inputs is given by
1 sin LSSE
Q AQ:
2 π
RIVT
sin
1 1+Q, AQi
VQ, + AQ.+1
the accuracy by
1 2 + sin-1
R VTQQ, + +Q,)
A= = E 2 y signý + +1)
(2)
If the dataset contains a random mixture of different difficulty levels A 2 the loss and accuracy can be obtained by taking weighted average over the partitions.
To understand how test performance changes through learning, we study the evolution of the order parameters. Combining their definition with the definition of the dynamics [I] and the fact that the random variables concentrate in the high-dimension as N - CO, we obtain an analytic form for the updates: Q, f fa. (Qr. Q, R, T). Qi fq. (Qr. Qs R) R + fR (Qr, R, T);; where fq.. fq, and In are long but explicit expressions that are reported in the supplementary material (SM).
Dynamical advantages of curriculum. With these theoretical results in hand, we can now char- acterise the performance of curricula in the online setting. We obtain a description of the learning trajectories for each learning protocol, yielding the evolution of training and test accuracies, and of other observables such as the norm of the student and its overlap with the teacher.
Solving the dynamical equations gives two key advantages relative to simulating models in this setting. First, they are free of finite size effects and stochastic fluctuations. And second, their evaluation is very fast (up to 6 orders of magnitude in simulation time reduction see SM ??), enabling systematic exploration of the parameter space of the problem, along with fine- aneeggrinee optimisation over paper-pprameters such as learning rate, weight decay and scaling in the initialisation.
Optimising final test accuracy separately for each curriculum strategy, we find that curriculum learning is the optimal strategy, followed by baseline (no-curriculum) and lastly anti-curriculum. In Fig. I we show typical learning trajectories for a dataset with equal numbers of easy and hard samples. The results of the simulations (solid lines) are well-described by our theoretical equations (dashed lines), and show that the curriculum strategy leads to better performance throughout training. Fig. I shows the evolution during training of the test accuracy computed on the whole dataset.
Next, we systematically trace the effect of curriculum for a range of total dataset sizes (a) + az2) and number of easy examples aj in the phase diagram in Fig. This diagram shows in panels (a) and (b) the accuracies on hard instances reached the end of training, by curriculum learning and anti-curriculum learning respectively, normalised by the accuracy reached by the standard strategy. The two heatmaps show that curriculum learning always outperforms standard learning and that, on the other hand, anti-curriculum learning outperforms standard learning only in part of the diagram. Comparing the two strategies, in Fig. (c), we can observe that there a region for small a and a where anti-curricllum learning is the best strategy, while in the majority of the situations curriculum

---

2,00
2.00
2.00
1.010
3.75
175
175
1.50
1.50
130
1.25
1.25
1.25
1.00
1.00
1:40
1,500
0.75
0.75
0.50
0.0
0,090
0.5
1.0
2.5
0.5
1.0
1.5
0.5
1.0
3.5
01.1%
01:110
01
(a) Curriculum learning.
(b) Anti-curriculum learning. (c) Curriculum vs anti-curriculum.
Figure 2: Phase diagram of online learning performance gap with optimal parameters. The colour scale shows the ratio of the accuracy on hard instances reached by curriculum over no- curriculum (a), anti-curriculum function of the total dataset size (a1 + a2) and easy dataset size (a1). Curriculum broadly benefits performance and anti-curriculum effective in certain regions, but the size of the improvement modest. Parameters: 0.50, A = 0, A2
no-curriculum (b), and currieulum
anti-curriculum (c),
learning is best, Interestingly, there a sizeable region of the diagram in which both curriculum and anti-curriclum help. possibly explaining why both have been recommended in prior work (55. A possible intuition behind this counter-intuitive phenomenon highlighted by our analysis that, in some settings, the large amount of noise contained in the hard data will always be too disruptive for effective learning. Thus, leaving the easy (cleaner) data for last could allow the model to better exploit it.
12 43 44
Further, we find that our setting, which small task-relevant signal is embedded in large task- irrelevan variation, is critical to the benefit of curriculum. Fig. showws performance as a function of sparsity P. additional details are deferred the SM ??. Non-sparse tasks do benefit. Hence curriculum aids tasks with many irrelevant factors of variation. Interestingly, the literature from human psychology shows precisely this: no curriculum benefits for low-dimensional tasks or tasks with variation in irrelevant dimensions [6]
Our results also highlight the intricate dependence of curriculum on parameters the learning setup. If not all parameters are correctly optimised, we can observe more complex scenarios. For instance, the initialisation condition for the norm of the weights of the student plays an important role. We explore this dependence by changing the variance of the normal distribution from which the initial weights are sampled from. We observe that anti-curriculum learning becomes the best strategy when the variance is large, as shown in Fig. 3 for weights of order 1. In this case, curriculum learning shows an advantage only in the first phase when easy examples are shown, which consistent with the results of [19]. However, in the next phase when hard examples are shown, the curriculum strategy does not extract enough information and it outperformed by the other two strategies. The fact that curriculum anti-curriculum can look better depending
the parameter setting might help explain
the confusion the literature over the best protocol [12 14 43 44 45 At least in this model, better performance from unti-curricllum is signature of sub sub-optimal choice of the parameters.
To summarise our findings in this online learning setting, curriculum mainly offers a dynamical advantage: it speeds up learning but has minimal impact on asymptotic performance.
4 Batch learning solution
The previous section discussed the online case where each example is used once and then discarded. However, in common machine learning practice, neural networks typically revisit each sample repeat- edly until convergence. Therefore an important question is: can curricula lead to a generalisation improvement when trained on the same dalase! until convergence?
investigate this question by considering student that learns from slices of a dataset in distinct optimisation phases, where in each phase the student optimises -errulllrrseed logistic loss. Without
5

---

2.00
2.00
1.75
175
70
1.5
1.50
1.25
100
3.00
0.79
55
0.50
curicuuum anti-curtcauum curricuds
1.0
1.5
1.5
2.0
(a) Curriculum learning.
(b) Anti-curriculum learning.
(c) Accuracy trajectories.
Figure 3: Performance gap starting from high initialisation norm. The first two figures show the accuracy-gap on hard instances between curriculum learning and the baseline (a) and cuti-curriuulum learning and the baseline (b). Contrary to the phase diagram Fig. 2 curriculum learning is not always the optimal and anti-curriculum is not always the worst strategy. The right panel shows the accuracy evaluated on the hard samples for a a2 = 0.5.
further modification, curriculum can have no effect in this setting: due to the convex nature of the teacher-student setup [22]] the network is bound converge minimum uniquely determined by the final slice of data, with no memory of the progress made at intermediate steps. This simple observation may help explain empirical observations on real data, such [15], which find no benefit of curriculum in standard settings. In fact, in principle curriculum could still influence non-convex problems 12 but empirical results the MI. field are not showing clear signals of memory retention. dynamical memory effects requires careful tuning
A possible explanation of this is that relying of the learning rate and of the number of training epochs, while typical choices for these hyper- parameters could lead to memory loss and performance inconsistencies. These observations raise the theoretical question how to better implement curriculum learning to induce a non-vanishing effect also in batch learning settings.
To instantiate long-term memory effect our model, we propose biasing the optimisation landscape via Gaussian prior, centred around the optimiser of the previous learning phase. The additional term the loss acts as an elastic coupling between the successive phases, and the associated intensity 712 is then an additional hyper-paree parameter of the model. This scheme is similar to regularisation methods proposed against catastrophic interference in continual learning, such as Synaptic Intelligence [46]. Changing the loss according to the curriculum prescription effectively makes the learning algorithm aware of the different levels of difficulty the dataset.
Tools from statistical physics can be used to analytically compute test performance under this scheme. In order simplify the presentation, we first consider just two learning phases. is natural to frame this setting as 2-level problem, involving two systems with independent copies of the network weights Wi and W2. In a typical statistical physics approach, we associate Boltzmann- Gibbss measure to the systems, with an energy function determined by the regularised logistic loss C.. While the statistical properties of the first system can be determined caff-cnnsitennly, the added elastic interaction creates dependence of the second measure on the configurations of the first system. In mathematical terms, the coupled system is represented by the following partition function:
(Z(W2, W Wi; of dd,,
/WI e -B,,, (W,,
log / (WW2e 92(( (Wa.Da)+ [W2-Wil33
(W,
(3)
where D1, D2 denote the dataset slices. This object represents the normalisation of the Boltzmann- Gibbs measure, and allows one to extract relevant information on the asymptotic behaviour of our model. The optimisations entailed in each learning phase be described in the "low noise" limit of B1, 32 - oc, where the measures focus on the minimisers of the respective losses. In order to study aver-iver quantity that does not depend on specific realisation of the dataset, we aim to compute the associated average free-energy:
= lim lim
(log (Z 2, W Di, (2,,)))). w)))),)),,..)
(4)
N 91,97 +0x N,N

---

1.05
0.84
accurscy fand samnels Sandeeell
0.62
1.02
0.00
0.78
Curficalam
0.95
0.75
onti curriculum
random spliting
0.2
culum
0.2
2.4
0.6
0.8
0gF2
(a) Learning with curricula
(b) Curriculum vs no-curriculum
Figure Effect of elastic coupling (Gaussian prior) between curriculum phases. (a) comparison
between asymptotic performance of curricula (full lines) and single batch learning, at Q1 = 1 0r 1, with regularisation ?/ that yields the best generalisation when learning the entire dataset (in principle not optimal for the other strategies). The points represent the results from 10 numerical simulations at size N 2000. Parameters:
0.50, A and A₂ 1. (b) ratio between the accuracy reached by curriculum learning over anti-curriculum of dimension a₁ + Q2 = and of the sparsity level of the teacher p. Note that p can also be seen as
function of the number of easy samples in a dataset
the fraction of relevant components in the inputs. A 0 and A 1. 71 722 and 7/1 where set values that optimise test performance.
This quantity can be seen as a special case of the so-called Franz-Parisi potential computation 447 48 and the entailed double average can be evaluated through the replica method. Refer to SM for details.
Similar the online case, in high-dimensions the free-entropy concentrates function that depends on several order parameters that capture the geometrical distribution of teacher and student configurations. In addition those already introduced in Sec. we also have 5Q. which is linked to the variance of the student norm. Moreover, for each order parameter we also need to introduce a conjugate parameter, denoted in the following with the hat symbol. The final expression for the free-energy reads:
deterministic
Φ = extr -AARR [AR+ + (1950- (080-500) ) + gs(11 (2,11,2,100 72, 71) + an gE (A1) + a2 g8 (A2)
(5)
where gs and Jø are two scalar functions, often called entropic and energetic channels, that encode the dependence of the optimisation problem the Gaussian prior and the logistic loss respectively. The extremum condition for the free-energy yields a system fixed-point equations that converge asymptotic prediction for the order parameters, comparable with the results of numerical simulations on large instances, Fig. 4 At convergence, the order parameters can be inserted again in Eq. to obtain estimate of the accuracy. Note that this formalism is not limited to phases, but can be extended to the case of discrete number of sequential stages.
The importance sparsity. Sparsity key ingredient in determining the impact of curriculum strategies. It naturally introduces notion of relevant and irrelevant inputs, and defines a secondary learning goal: identifying what part of the presented data should be disregarded by the model. Curriculum learning can aid this identification process, since the easy samples are more transparent to this structure. This is also observed in human experiments [6]. However, the relative difficulty of the problem of inferring the support of the teacher and the problem of aligning with its non-zero components depends on the degree of sparsity p, so effectiveness of curriculum can vary with
In the right panel of Fig. 4 we explore the interplay between the sparsity of the teacher and the fraction of easy samples in the dataset a1, comparing curriculum with the no-curriculum baseline. The phase diagram highlights the variability in the impact of the curriculum ordering:
Curriculum most effective at low values of p and close to the diagonal, where the fraction of easy examples in the dataset is comparable to the fraction of relevant dimensions.

---

When p 0.5, the possible gain from ordering the samples according to difficulty is counterbalanced by the instrinsic cost of splitting the information content into two blocks, thus curriculum can become detrimental.
When a1 too small compared to (above diagonal), the first stage in the curriculum strategy only help in the support identification problem, but will not allow a good estimation of the direction the teacher. Because of the elastic prior, the second stage cannot improve too much
it and the effect of curriculum small.
When is larger than the sparsity (below diagonal), the easy examples contain sufficient information for solving both the support and the teacher estimation problems, and this information is also exploited by the baseline. Thus the improvement of curriculum becomes negligible.
We refer to the SM for an in-depth comparison with aniiccurriullm.
2.00
1.03
1.50
1.02
1.2
1.0
8
5
1.03
0.75
0.50
1.5
0.5
0.96
(a) Curriculum learning.
(b) Anti-curriculum learning.
(c) Curriculum anti-curriculum.
Figure 5: Phase diagram for the performance gap in the batch setting. The colour scale shows no-curriculum (a), anti-curriculum
the ratio of the accuracy hard instances for curriculum over no-curriculum (b), and curriculum over anti-curric aati-curriculum (c), as function of the total dataset size (01 + a2)) and easy dataset size (a1). In contrast the online case, performance benefits are greater and curriculum is strictly better than anti- cutiicurriulumm. Both 71 = and 7/2 are optimised point-wise, in order to yield the best test accuracy. Parameters: p 0.50, A 0, Ag =
Asymptotic advantages of curriculum. Contrary to the case of online SGD, if the fraction of relevant directions small, batch learning with clastic coupling notably improves test accuracy of both curriculum and anti-curricllum above the baseline. This confirms the utility of curriculum strategies when the signal partially "hidden in clutter" [49
Fig. 5 shows similar phase diagrams to Fig. but for the batch setting. At each point in the phase diagram the regularisation level ?1 "2 and the coupling 7/22 are optimised yield the best accuracy. We find that the performance order is nearly always preserved: curriculum followed by anti-curriculum followed by baseline. In the SM we see similar improvements by applying the elastic coupling strategy both in the online setting and on real data.
In summary, in the batch setting, splitting the learning process in stages might not be advantageous per se. However, our observations show that if the loss is modified to reduce memory loss between the learning stages, curriculum learning strategies can offer measurable asymptotic advantage.
5 Connection with experimental literature
Recent work has suggested that curriculum learning could provide an important window into the learning algorithms at work biology [5T]. Our analysis makes several predictions for curriculum effects. In this section assess these predictions based on connections to extant experiments and propose future experimental tests.
First, we find that curriculum strategy yields speed up in learning in all the tested settings (see Fig. [p). This acceleration broadly consistent with the findings from cognitive science 2 6. By contrast, results show that the speed improvement does not necessarily translate into sizcable

---

New
Old
1.03
0.03
0.10
1.02
0.30
0.70
Easy
1.03
0.2
0.4
A
Hard
1.5
12-
1.0
10
11.0
11.4
Nöden contusion
dificcutty
(a) Generalisation gap per difficulty.
(b) Rolc of sparsity.
(c) Fading experiment [6]
Figure 6: Connection with psychology experiments. (a) Top: Accuracy ratio of different strategies in the model, with ourriulun/nn-ccrriclbee in green and aurriculum/ani-curiialll
in orange. The
ratio shows non-monotonic behaviour. Bottom: The accuracy ratio obtained by [50]. Parameters d2 1 and optimal learning rate, norm at initialisation and weight decay intensity. (b) Top: Dependence on the sparsity of the generalisation gain of curriculum over no-curriculum, measured ratio between final accuracy, for fixed total dataset size (o) + 02 1). Bottom: The ratio obtained from experiments 3 and 4 of [6] (c) Example cartoon stimuli from the "fading" paradigm used in [6]. where participants distinguish daemons of
0.5, A1 0.0, A2 1.0, an
the old world from daemons of the new world. The distinguishing feature (horn length) is diluted
among many irrelevant features (colour, eye size, mouth size). Highlighting the relevant feature to participants leads to better and faster learning.
generalisation eтror improvement, and the performance achieved at the end of training can even deteriorate when learning hyperparameters are not fully optimised (c.f. Fig. 3 Deterioration due to curricula has generally not been reported
psychology literature, though it has been observed in ML [15). This fact may suggest that animals naturally learn with near-optimal hyperparameters such that curricula generally confer benefits.
A more specific observation concerns the performance on different difficulties after learning. As reported [50], human and rodent subjects trained an auditory task using curricula showed the greatest improvement for intermediate levels of difficulty as depicted in Fig. (o bottom panel. The same conclusion can be drawn from the experiment of [7С. where, surprisingly, subjects trained with curricula to classify medical images showed poor performance in hard tasks compared to the control group. To address this phenomenon, we calculate accuracy as function of difficulty in the model in Fig- [6 top panel. Consistent with these experiments, we find regimes where the gap between curriculum learning and the baseline non-monotonic, with the largest performance gain for intermediate difficulties.. Contrary to 778, however, do not observe negative effects of curriculum for high difficultics. Further experiments that more systematically manipulate training and transfer difficulties could provide stronger test of these predictions.
key ingredient in our model the role of sparsity, such that a small signal is embedded amidst many irrelevant features. Experimentally, the importance of having many factors of variation to obtaining curriculum effect has been documented in the "fading" experiments of [6]. Human subjects were trained on classification tasks involving stimuli with one task-relevant feature dimension and a variable number of task-irrelevant feature dimensions. Example cartoon "daemon" stimuli are depicted in Fig. GC. where for instance horn height might be the distinguishing feature while colour, eye size, and mouth size might constitute fask-irreeevant features. Without any irrelevant factors of variation (p 1), they report curriculum benefit. By contrast when 75% of features irrelevant (p = 255, they record a strong curriculum effect, as shown in Fig. 6b bottom. This qualitative trend also observed in our model (Fig. 6 top). While these experiments tested only two sparsity levels, further experiments could sample this dimension more extensively and test for interactions with the fraction of easy and hard examples. We note that while the connectionist literature has addressed the effect of curriculum in several settings [39 40 10 [T].
we found that easy-to-han tord effects appear even in simple setup without need for complex networks and/or dynamics.
Finally, our results may shed light on self- generaeded curricula during human development Children undergo vocabulary spurt that coincides with their ability to grasp and centre objects in the

---

50000
1.06
Easy dataset size
40000
1.04
1.02
30000
1.00
20000
0.98
10000
0.96
0
20000
40000
Total dataset size
(al
(6)
[c]
[d]
Figure 7: Experimental setting CIFAR1O-derived data. (a) Input samples combine task- relevant image with a distractor image, and become progressively harder from left to right. (b) Ratio between final accuracy hard instances for curriculum learning versus curriculum. 7/ Y 712 init, and stopping time are optimised.
visual field 5 Quantitative estimates of the amount of clutter (irrelevant objects) in self-generated views decrease due to this grasping ability, yielding self-generated curriculum [49 [4] Our model similarly predicts that reducing clutter should improve learning speed and performance.
Real-World Demonstration. To verify this prediction simple cluttered object classification task from the CIFAR1O 10 dataset [55] by patching two images together into a 32 x input image (Fig. D.. The task to produce the class label of the image on the left. The right image is distractor that is irrelevant to the classification. To vary difficulty, we scale the contrast of the irrelevant image (Fig. 77) We train a single-layer network with the cross-entropy loss and the curriculum protocol with Gaussian prior between two curriculum stages, implemented Pytorch Lightning to ensure that training parameters accord with standard practice. We optimised hyperparameters in each curriculum phase separately. We trained all combinations of five clastic penalties log spaced between 1e and 1e2, and weight decay parameters {0, .2 .5). We then compute the best performing model for five random seeds and take the mean over seeds. Further dataset, model and experimental details are given in Appendix ??. As shown in Fig. 7p. curriculum improves performance, particularly when easy examples make up large proportion of the dataset, confirming that curricula that reduce clutter can benefit learning.
richer visual setting, we construct
Conclusions
We analysed a model curriculum learning introduced by 12 and amenable of analytical treatment. This simple setting sheds light on results observed in the cognitive science and machine learning literature, and the theoretical tractability allows for exploration of a wide range of parameters that would be costly to obtain through experiments. Future work will need to move beyond models with simple loss landscapes to address the impact of curricula complex tasks like reinforcement learning. Nevertheless, the model recapitulates a variety of observations in the literature (50) 56, (57), revealing that casy-to-hard effects can appear when a sparse signal is embedded in many irrelevant dimensions of variation. We find that making the algorithm curriculum-awace by modifying the loss can better exploit curricula, offering a potential route for improved practical algorithms. Other curriculum-awan approaches are possible such adapting the learning algorithm (58) the architecture [10]. On the psychology side, our predictions can help in designing new experiments, for instance testing the counter-intuitiv benefit of curticcuriuulum learning for intermediate sparsity.
Acknowledgments and Disclosure of Funding
We thank Miguel Quiz-Garcin and Ronald Dekker for important discussions. L.S. acknowledges funding from the ERC European Union Horizon 2020 Research and Innovation Program Grant Agreement 714608 Dale Fellowship (21 (21338/7//1//2) and Sainsbury Wellcome Centre Core Grant (219627/7/1977. GAT3755). A.S. is CIFAR Azricli Global Scholar in the Learning in Machines & Brains programme.
S.S.M. & A.S. were supported by Wellcome and Royal Society Henry
10

---

References
[1] Douglas H Lawrence. The transfer of a discrimination along continuum. Journal of Compara- live and Physiological Psychology, 45(6):511, 1952.
[2] Robert A Baker and Stanley W Osgood. Discrimination transfer along pitch continuum. Journal of Experimental Psychology, 48(4):241, 1954.
[3] Renee Elio and John Anderson. The effects of information order and learning mode on schema abstraction. Memory & Cognition, 12:20-30, January 1984.
[4] Robert C. Wilson, Amitai Shenhav, Mark Straccia, and Jonathan D. Cohen. The Eighty Five Percent Rule for optimal learning. Nature Communications, 10(1):6646, November 2019.
[5] Judith Avrahami, Yaakov Kareev, Yonatan Bogot, Ruth Caspi, Salomka Dunaevsky, and Sharon Lerner. Teaching by Examples: Implications for the Process of Category Acquisition. The Quar- terly Journal of Experimental Psychology Section A, 5063):586-606, August 1997. Publisher: SAGE Publications.
[6] Harold Pashler and Michael C. Mozer. When does fading enhance perceptual category learning? Journal of Experimental Psychology: Learning, Memory, and Cognition, 162-:1162-1173, 2013.
[7] Adam N. Hornsby and Bradley Love. Improved classification of mammograms following idealized training. Journal of Applied Research in Memory and Cognition, 3(2):72-76, June 2014.
[8] Brett D. Roads, Buyun Xu, June K. Robinson, and James W. Tanaka. The easy-to-hard training advantage with real-world medical images. Cognitive Research: Principles and Implications, October 2018.
[9] The International Brain Laboratory, Valeria Aguillon-Rodriguee Dora Angelaki, Hannah Bayer, Niccolo Bonacchi, Matteo Carandini, Fanny Cazettes, Gaelle Chapuis, Anne K Churchland, Yang Dan, Eric Dewitt, Mayo Faulkner, Hamish Forrest, Laura Haetzel, Michael Häusser, Sonja Hofer, Fei Hu, Anup Khanal, Christopher Krasniak, Ines Laranjeira, Zachary Mainen, Guido Meijer, Nathaniel J Miska. Thomas D Mrsic-Flogel, Masayoshi Murakami, Jean-Paul Noel, Alejandro Pan- Van-Vazueez Cyrille Rossant, Joshua Sanders, Karolina Socha, Rebecca Terry, Anne E Urai, Hernando Vergara, Miles Wells, Christian J Wilson, Ilana B Witten, Lauren E Wool, and Anthony Zador. Standardized and reproducible measurement of decision-making in mice. eLife, 10:e63711, 1, May 2021. Publisher: eLife Sciences Publications, Ltd.
[10] Jeffrey L. Elman. Learning and development in neural networks: the importance of starting small. Cognition, 48(1):71-99) July 1993.
[11] Kai A. Krueger and Peter Dayan. Flexible shaping: how learning in small steps helps. Cognition, 11003):380-394, March 2009.
[12] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
[13] Anastasia Pentina, Viktoriia Sharmanska, and Christoph H. Lampert. Curriculum learning of multiple tasks. pages 5492-5500. IEEE Computer Society, June 2015, ISSN: 1063-6919.
[14] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In ICML, volume 97, pages 2535 2544. PMLR, 2019.
[15] Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? ICLR, 2020.
[16] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Berbert-Vosss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 18:7--19 2020.
[17] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. 20X1:20010003934 [cs], January 2021. arXiv: 2010.03934.
Prioritized Level Replay.

---

[18] Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In International Conference on Machine Learning, pages 1588-5246. PMLR, 2018.
[19] Daphna Weinshall and Dan Amir. Theory of curriculum learning, with convex loss functions. Journal of Machine Learning Research, 21(22):11-19 2020.
[20] Miguel García, Andrea Liu, and Eleni Katifori. Tuning and jamming reduced to their minima. Physical Review E, 10065)0052088 2019.
[21] Mare Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications, volume 9. World Scientific Publishing Company, 1987.
[22] Andreas Engel and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001.
[23] Lenka Zdeborová and Florent Krzakala. Statistical physics of inference: Thresholds and algorithms. Advances in Physics, 65(5):453-552, 2016.
[24] Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl- Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 2020.
[25] Leticia F Cugliandolo and Jorge Kurchan. Analytical solution of the off-equilibrium dynamics of a long-range spin-glass model. Physical Review Letters, 71(1):173, 1993.
[26] Michael Biehl and Holm Schwarze, Learning by on-line gradient descent. Journal of Physics A: Mathematical and general, 28(3):643, 1995.
[27] M.S. Advani, A.M. Saxe, and H. Sompolinsky, High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428- 446, 2020.
[28] Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[29] Stefano Sarao Mannelli, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborova. Passed & spurious: Descent algorithms and local minima in spiked matrix-tensor models. 97:4333- 4342, 09-15 Jun 2019.
[30] Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, and Lenka Zde- borová. Who is afraid of big bad minima? analysis of gradient-flow in spiked matrix-tensor models. In Advances in Neural Information Processing Systems, pages
2019.
[31] Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Complex dynamics in simple neural networks: Understanding gradient flow phase retrieval. 33:3265-3274, 2020.
[32] Hugo Cui, Luca Saglictti, and Lenka Zdeborová. Large deviations for the perceptron model and consequences for active learning. In Mathematical and Scientific Machine Learning, pages 390-430. PMLR, 2020.
[33] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987-3995. PMLR, 2017.
[34] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Gtrabska-Barwinskee et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (4(13):3521-3526 2017.
[35] P Ivan Paviov. Conditioned reflexes: an investigation of the physiological activity of the cerebral cortex. Annals of neurosciences, 17(3):136, 2010.
[36] Burrhus Frederic Skinner. The behavior of organisms: An experimental analysis. BF Skinner Foundation, 2019.
[37] Merav Ahissar and Shaul Hochstein. Task difficulty and the specificity of perceptual learning. Nature, (87 1997.
12

---

[38] C Donald Morris, John D Bransford, and Jeffery J Franks. Levels of processing versus transfer appropriate processing. Journal of verbal learning and verbal behavior, 16(5):519-55) -333,, 1977.
[39] Kim Plunkett, Virginia Marchman, and Steen Ladegaard Knudsen. From rote learning system building: acquiring verb morphology in children and connectionist nets. In Connectionist Models, pages 201-219. Elsevier, 1991.
[40] Kim Plunkett and Virginia Marchman. U-shaped learning and frequeney effects in multi- layered perception: Implications for child language acquisition. Cognition, 38(1 1:331223 1991.
[41] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In 7th International Conference on Learning Representations, ICLR 20/9, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[42] David Saad and Sara A Solla. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21))4337, 1995.
[43] Tom Koemi and Ondfej Bojar. Curriculum Learning and Minibatch Bucketing in Neural Machine Translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 379-386, Varna, Bulgaria, September 2017. INCOMA Ltd.
[44] Nathan Schneider, Dirk Hovy, Anders Johannsen, and Marine Carpuat. Semeval-2016 task 10: Detecting minimal semantic units and their meanings (dimsum). In Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch, editors, Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval @ NAACL-HLT 2016, Sun Diego, CA, USA, June 16-17, 2016, pages 546-559. The Association for Computer Linguisties, 2016.
[45] Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. Curriculum Learning for Domain Adaptation in Neural Machine Translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1903- 1915, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[46] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3987-3995, International Convention Centre, Sydney, Australia, 06 11 Aug 2017. PMLR.
[47] Silvio Franz and Giorgio Parisi. Phase diagram of coupled glassy systems: A mean-field study. Physical review letters, 79(13)2448, 1997.
[48] Luca Saglietti and Lenka Zdeborová. Solvable model for inheriting the regularization through knowledge distillation. CoRR, abs/2012.00199 2020.
[49] Elizabeth M. Clerkin, Elizabeth Hart, James M. Rehg, Chen Yu, and Linda B. Smith. Real- world visual statistics and infants' first-learned object names. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 372(1711), January 2017.
[50] Estella H Liu, Eduardo Mercado III, Barbara A Church, and Itzel Orduña. The casy-to-hard effect in human (homo sapiens) and rat (rattus norvegicus) auditory identification. Journal of Comparative Psychology, 122(2):132, 2008
[51] Daniel R. Kepple, Rainer Engelken, and Rajan Kanaka. Curriculum learning a tool to uncover learning principles the brain. ICIR, 2022.
[52] Hadar Karmazyn Raz, Drew H. Abney, David Crandall, Chen Yu, and Linda B. Smith. How do infants start learning object names in a sea of clutter? Annual Conference of the Cognitive Science Society, 2019:521-526. July 2019.
[53] Linda B. Smith and Lauren K. Slone. A Developmental Approach to Machine Learnin?? Frontiers in Psychology, 8, 2017.
[54] Chen Yu and Linda B. Smith. Embodied attention and word learning by toddlers. Cognition, 125(2):244 262, November 2012.
[55] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009.
13

---

[56] Itzel Orduña, Estella H Liu, Barbara A Church, Ann C Eddins, and Eduardo Mercado III. Evoked-potential changes following discrimination learning involving complex sounds. Clinical neurophysiology, 123(4):711-719 2012.
[57] Barbara Church, Eduardo Mercado III, Matthew G Wisniewski, and Estella H Liu. Temporal dynamics in auditory perceptual learning: impact of sequencing and incidental learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39(1):270, 2013.
[58] Miguel Ruiz- Marzi,, Ge Zhang, Samuel S Schoenholz, and Andrea J Liu. Tilting the playing field: Dynamical loss functions for machine learning. In International Conference Machine Learning, pages 9157-9167. PMLR, 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts your work? [No] The model
is theoretical in nature and we do not foresee any negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Refer to sections "model definition".
(b) Did you include complete proofs of all theoretical results? [N/A] In the SM we provide details about the derivation.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] The architecture used is very simple to simulate and do not provide code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All the figures resulting from the experiments contain details to produce them. Full parameters for the experiments on real data are given in the SM.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We report estimated total amount of compute and type of compute in the SM for the experiment real data (s 10000 GPU hours, ss 1110 kg CO2 eq). The data for each theory-based figure can be obtained in few hours of laptop simulation.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use and cite the CIFAR1O dataset.
(b) Did you mention the license of the assets? [No] is widely known that CIFARIO an MIT License dataset.
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "px9QJD64VZL", "reviewer": "Reviewer_Cgtb", "summary": "The paper is motivated by the gap in our understanding of the importance of curriculum learning in biological models vs machine learning models. The main question the paper tries to address is, when and why curriculum learning helps. \nThey consider a teacher, student setup where both teacher and student are shallow 1-layer neural networks of size N and the teacher network is sparse, with only a fraction ρ < 1 of ∼ N (0, 1) non-zero components. The student model has to guess which components should be set to zero and align the relevant weights in the correct direction.\nThe findings of the paper are as follows:\n* Curriculum learning is most effective when a small signal is embedded amidst many irrelevant feature.\n* The effect of curriculum or anti-curriculum depends on other learning parameters such as initialisation (anti-curriculum can lead to better results of hyper-parameters are not optimal).\n* Curriculum mainly offers a dynamical advantage: it speeds learning, with only minimal impact on asymptotic performance.\n* Curriculum-aware algorithms can better exploit curricula.", "questions": "- One way to explain why it seems that explicit curriculum learning doesn't help in many cases in ML, is that the models could potentially be impacted by some imiplict curriculum, e.g., curriculum  imposed by the architecture of the model or the optimization algorithm. What do you think about this, and how does it fit to the dicussions in this paper?\n\n- What does \"curriculum-aware algorithms\" mean?", "limitations": "I don't have any comments on this.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The topic of the paper is very interesting and inspiring: linking psycological experiments and theories of how curriculum learning works/helps humans/animals to our understanding of effects and benefits of curriculum learning in machine learning. \n\nThe main issue I think is that the emprical experiments are too simple and far away from the state of the art ML techniques. I wonder how, conceptually, the findings could be replicated in scaled up settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kPrFAs98poA", "reviewer": "Reviewer_MUSa", "summary": "This work introduces an analytical framework that eases the study of the effect of data curricula in teach-student networks. The authors present a set of formulas that can produce the final accuracy of a model based on the initial parameters of the student and teacher networks without the need to perform the simulation (training). The analytical solution is an iterative method that is applied to the online learning case (optimize on one sample at a time, and each sample is used only once) and batch learning case (train until convergence). They evaluate their methods and study the benefits of curricula using a synthetic dataset of Gaussian samples whose label is determined by a sparse teacher network and difficulty is determined by the variance in irrelevant samples. Additionally, they evaluate on a custom-designed task using CIFAR-10 data, where an irrelevant image is concatenated to the target image, and difficulty is based on the opacity of the irrelevant image. The study the effect of curricula based on model initialization scale, teacher network sparsity level, and variance amount in irrelevant features (SNR).", "questions": "- *\"due to the convex nature of the teacher-student setup [22], the network is bound to converge to a minimum uniquely determined by the final slice of data, with no memory of the progress made at intermediate steps\"* ... Is this a property of all student-teacher networks or just the the setup you are using? Because if it isn't, then curriculum learning can have a significant effect to the learning trajectory in the parameter spaces, and the optimum points able to be reached. \n\n- *\"the first stage in the curriculum strategy can only help in the support identification problem\"* ... Shouldn't the identification of the support lead to easy, successful learning of the simple remaining task? \n\n- What do you mean by training trajectories? Is it the accuracy curve over the course of training?", "limitations": "- I am not sure how generalization the teacher-student network analysis is, as compared to standard learning approaches. In particular, this setup with a convex network, simple optimizer, and simple synthetic dataset. I think the learning dynamics can be significantly different in a deep, non-convex network and using a modern momentum-based optimizer. ", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- An analytical theorem, with a rigorous proof, which can be of great use to the study of curricula and learning theory.\n- The connection of machine learning results with experiments of psychology is interesting, and convincing, and attests to the soundness of the author's methods. \n- Well-written introduction about curriculum learning with references to biological, psychological and artificial learning backgrounds.\n- Investigates an important problem and presents a solid problem definition, particularly in the batch learning section.\n\n**Weaknesses**\n\n- The benefits of the analytical solution are states as a) \"free of finite size effects and stochastic fluctuation\", and b) \"their evaluation is very fast\". Both of them require further elaboration. I would like to see a comparison of the speed of the analytical solution and full training, and a break-down of how the analytical solution is faster. In the supplementary material I see that the analytical forms of the updates are computed using matrix multiplications and activation functions on a scale comparable to the student network's computations.\n- Section two is title \"Online dynamical solution in the large input limit\", but then the relevance of the large input limit (or even its definition) is not explained. Furthermore, the relevance and effect of high-dimensional input is not clearly shown. \n- Between the analytical theory, experiments on the benefit of CL, introduction of elastic coupling method, and connections between machine learning and psychology, the contributions of the paper seem scattered and unfocused.\n- The organization and writing throughout the paper is quite confusing.\n  - Figure 1 is based on notations and definitions introduced in section 3 and is not easily understood at the point it is presented.\n  - Line 137: \"Fig. 5 shows performance as a function of sparsity ρ\". Figure 5 does not show that. I think you referenced the wrong figure.\n  - Line 140: \"low-D tasks .. [6]\". The notation low-D is not introduced, not even in [6]. I assumed it means low-dimensional.\n  - Line 143: \"starting from a large initialisation\", I assume it means the random initialization scale of the student or teacher network. It is not entirely clear. Please include more details about this initialization. Does it use a normal random distribution?\n  - Figure 1: \"The curriculum boundary lies at α = 1/2\". What is the curriculum boundary? It is never defined. The abstract talks about \"curriculum boundary consolidation\", but it is not further elaborated. \n  - What is α in Fig 1c x-axis and other figures? $\\alpha_1$ and $\\alpha_2$ are defined, but $\\alpha$ never is, and I am not able to understand what it means. \n  - Half-way through the paper section numbers stop being used. What I assume are sections 5 and 6 do not have numbers. And I am not sure if the subsections of Section 4 belong there. \n  - Figure 3c \"Accuracy hard samples.\" is not an accurate title. It is accuracy of all samples, easy and hard. It's just that anti-curriculum performs best, which is influenced by hard samples.\n  - The figure captions are inconsistent in the uses of (a), (b), (left, center, right), and (top, bottom). For example Figure 3 says (a), (b) and then \"the right panel\".\n\n  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2Qlm_J20ChP", "reviewer": "Reviewer_S1ip", "summary": "The paper explores the mechanism of curriculum learning, specifically when will, or will not, the curriculum setting work. The paper investigates two kinds of demos, online learning dynamics, and batch learning, as well as proposes an elastic penalty regularization, which demonstrates that the curriculum setting could only be beneficial when the task is sparse. The paper also validates the observation on CIFAR10, which further illustrates the investigation over real datasets.", "questions": "1. In Figure 2, the paper presents that under some circumstances, anti-curriculum learning could perform better than curriculum or non-curriculum learning (b,c bottom left part), which is anti-intuitive, can the paper give some explanations of such phenomenon?\n2. The paper reveals that with too less easy data ($\\alpha_1$), the performance of curriculum learning can be degraded, can it be related to overfitting?\n\n", "limitations": "See Sec 'Strengths And Weaknesses'\n\nIn addition, the Appendix is not well related to the main text, please use markers in the main text to direct the appendix list.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenght:\n1. The paper is well organized, and clearly presented, in which it goes from easy to deep to introduce what curriculum learning is, its limitations, and investigate when curriculum settings will benefit or not.\n2. The paper investigates the relations between curriculum learning and sparsity, and demonstrates that the effectiveness of curriculum learning heavily depends on the sparsity.\n\nWeaknesses:\n1. Though the paper introduces the experiments over CIFAR10, the evaluation over the real dataset is still somehow limited, it could be better to have more evaluations of curriculum learning on other real data with non-convex settings, or at least make the explanation of the evaluation on CIFAR10 more detailed.\n2. It can be better (though not necessarily to have) if some methodologies could be proposed to improve the performance of curriculum learning under non-sparse cases, which could make it more practical.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept