When Do Curricula Work?
Authors: Xiaoxia Wu
Ethan Dyer
Behnam Neyshabur
Venue: ICLR 2021 Oral (2021)
Keywords: Curriculum Learning
Understanding Deep Learning
Empirical Investigation

Abstract: Inspired by human learning, researchers have proposed ordering examples during training based on their difficulty. Both curriculum learning, exposing a network to easier examples early in training, and anti-curriculum learning, showing the most difficult examples first, have been suggested as improvements to the standard i.i.d. training. In this work, we set out to investigate the relative benefits of ordered learning. We first investigate the implicit curricula resulting from architectural and optimization bias and find that samples are learned in a highly consistent order. Next, to quantify the benefit of explicit curricula, we conduct extensive experiments over thousands of orderings spanning three kinds of learning: curriculum, anti-curriculum, and random-curriculum -- in which the size of the training dataset is dynamically increased over time, but the examples are randomly ordered. We find that for standard benchmark datasets, curricula have only marginal benefits, and that randomly ordered samples perform as well or better than curricula and anti-curricula, suggesting that any benefit is entirely due to the dynamic training set size. Inspired by common use cases of curriculum learning in practice, we investigate the role of limited training time budget and noisy data in the success of curriculum learning. Our experiments demonstrate that curriculum, but not anti-curriculum or random ordering can indeed improve the performance either with limited training time budget or in the existence of noisy data.

Published as a conference paper at ICLR 2021
WHEN Do CURRICULA WORK?
Xiaoxia Wu" UChicago and TTIC cru@ttic.edu
Ethan Dyer Blueshift, Alphabet edyer@google.com com
Behnam Neyshabur Blueshift, Alphabet
neyshabur@googgee.com
ABSTRACT
Inspired by human learning, researchers have proposed ordering examples during training based on their difficulty. Both curriculum learning, exposing a network easier examples early in training, and anti-curriculum learning, showing the most difficult examples first, have been suggested as improvements to the stan- dard i.i.d. training. In this work, we set out to investigate the relative benefits of ordered learning. We first investigate the implicit curricula resulting from ar- chitectural and optimization bias and find that samples are learned in highly consistent order. Next, to quantify the benefit of explicit curricula, we conduct extensive experiments over thousands of orderings spanning three kinds of learn- ing: curriculum, anti-curriculum, and random-curriculun
in which the size of the training dataset is dynamically increased over time, but the examples are ran- domly ordered. We find that for standard benchmark datasets, curricula have only marginal benefits, and that randomly ordered samples perform as well or better than curricula and anti-curricula, suggesting that any benefit entirely due the dynamic training set size, Inspired by common use cases of curriculum learning in practice, we investigate the role of limited training time budget and noisy data the success of curriculum learning. Our experiments demonstrate that curriculum, but not anticcurriculum can indeed improve the performance either with limited training time budget or in existence of noisy data.'
INTRODUCTION
Inspired by the importance properly ordering information when teaching humans (Avrahami al., 1997), curriculum learning (CL) proposes training models by presenting easier examples carlier during training (Elman, 1993; Sanger, 1994; Bengio al., 2009). Previous empirical studies have shown instances where curriculum learning can improve convergence speed and/or generalization in domains such as natural language processing (Cirik et al., 2016; Platanios al., 2019), computer vision (Pentina et al., 2015; Sarafianos et al., 2017; Guo et al., 2018; Wang et al., 2019), and neural evolutionary computing (Zaremba & Sutskever, 2014). In contrast to curriculum learning, anti- curriculum learning selects the most difficult examples first and gradually exposes the model to casier ones. Though counter intuiivve, empirical experiments have suggested that anti-curriculum learning can be good 2017; Zhang et al., 2018; 2019b). This is in tension with experiments other contexts, however, which demonstrate that anti-curricula under perform standard or curriculum training (Bengio al., 2009; Hacohen & Weinshall, 2019).
better than curriculum learning in certain scenarios (Koemi & Bojar,
As explained above, empirical observations on curricula appear to be in conflict. Moreover, de- spite a rich literature (see Section A), no ordered learning method is known to improve consistently across contexts, and curricula have been widely adopted in machine learning. This suggest rul- ing out curricula as beneficial practice for learning. In certain contexts, however, for large-scale text models such as GPT-3 (Brown al., 2020) and T5 (Raffel al., 2019), non-uniform mixing strategies are standard practice. These contradicting observations contribute to confusing picture on the usefulness of curricula.
This work is attempt to improve our understanding of curricula systematically. We start by asking a very fundamental question about a phenomenon that we call implicil curricula. Are examples
Work performed while Xiaoxia Wu was student at UT Austin and interning at Blueshift. Code https : htt.bttb...t.////// www./oo//ooo/ooooooo research/unde http:///wddnr.aaaainoooooooooooo

---

Published as a conference paper at ICLR 2021
Standard Training Time
Limited Training Time
Data with Noisy Labels
41nd
24.0 74.5
75.0 75.5 Test Accuracy
78.0 76.5 71.0
77.5
12.5 15.0 17.5 20.0 22.5 26.0
27.5 33.0
56
50
50
PJ7 6.1 155
02
70
Test Accuracy
Test Accur acy
Figure I: Curricula help for time limited or noisy training, but not standard training, Each point repre- sents an independent learning ordering CIFAR 100 and is mean over three independent runs with the same hyper-parameters. Color represents the type of learning, from bottom to top, standard i.i.d. training (grey), curriculum (blue), anti-curriculum (purple), and random curriculum (green). The solid orange line the best test accuracy for standard i..dd training. The left, middle and right plots represent standard time, short time, and noisy training. We find that for the original dataset and learning constraints there are no statistically sig- nificant benefits from anti, random, curriculum learning (left). We find that for training with limited time budget (center) or with noisy data (right) curriculum learning can be heneficial.
learned in a consistent order across different runs, architectures, and tasks? If such a robust notion exists, is it possible to change the order in which the examples are learned by presenting them in different order? The answer to this question determines there exists a robust notion of example difficulty that could be used to influence training.
We then look into different ways of associating difficulty to examples using scoring functions and a variety of schedules known as pacing functions for introducing examples to the training procedure. We investigate if any of these choices can improve over the standard full-data i.i.d. training pro- cedure commonly used in machine learning. Inspired by the success of CL in large scale training scenarios, we train in settings intended to emulate these large scale settings. In particular, we study the effect of curricula when training with training time budget and training in the presence of noise.
Contributions. In this paper, we systematically design and run extensive experiments to gain better understanding of curricula. We train over 25,000 models over four datasets, CIFAR 10/100, FOOD101, and FOOD101N covering wide range of choices in designing curricula and arrive at the following conclusions:
Implicit Curricula: Examples are learned in a consistent order (Section 2). We show that the order in which examples learned is consistent across runs, similar training methods, and similar architectures, Furthermore, we show that it is possible to change this order by changing the order in which examples are presented during training. Finally, we establish that well-known notions of sample difficulty are highly correlated with each other.
Curricula achieve (almost) no improvement in the standard setting (Section 4 and 6). We show curriculum learning, random, and anti-curriculum learning perform almost equally well in the standard setting."
Curriculum learning improves over standard training when training time is limited (Section 5 and 6). Imitating the large data regime, where training for multiple epochs is not feasible, we limit the number of iterations in the training algorithm and compare cur- riculum, random and anti-curriculum ordering against standard training. Our experiments reveal a clear advantage of curriculum learning over other methods.
Curriculum learning improves over standard training in noisy regime (Section and 6). Finally, we mimic noisy data by adding label noise to CIFARI00 and also use a natural noisy dataset - FrOODOII. Similar to Jiang et al. (2018); Saxena et al. (2019); Guo et al. (2018), our experiments indicate that curriculum learning has a clear advantage over other curricula and standard training.
Related Work. Bengio et al. (2009) is perhaps the most prominent work curriculum learning where the "difficulty" of examples is determined by the loss value of a pre-trained model. Toneva
"ee the first paragraph of Section B for details of the standard-time experimental setup.
2

---

Published as a conference paper at ICLR 2021
-1.0
Image orderss (carned neration
VEG PC
3.9
8000
2.8
3.7
-6000
3.6
4400
Nett Batch
3.5
0.4
0.3
3.
FC
VGG
Nets Eatch-Norm
(optimizaiions
Figure 2: Implicit Curricula: Images are learned in a similar order for similar architectures and train- ing methods. (Left) Epoch/Iteration at which each image is learned across 142 different architectures and optimization procedures. Each row is CIFAR 10 image ordered by its average learned epoch. The columns from left right, are fully-connected (FC) nets, VGG nets (VGGI1 and VGG19), and nets with Batch-Norm (loffe & Szegedy, 2015) including ResNet18, ResNet50, WideResNet288- 10, WideResNer48- 10 DenseNet121 EfficientNet BO, VGG11-BN and VGG19-BN, (Right) The Spearman correlation matrix shows high correlation between orderings within architecture families.
et al. (2019) instead suggested using the first iteration in which an example is learned and remains learned after that. Finally, Jiang et (2020b) has proposed using a consistency score (c-score) calculated based the consistency of a model in correctly predicting particular example's label trained on i.i.d. draws of the training set. When studying curriculum learning, we look into all of the suggessd measures of sample difficulty. We further follow Hacohen & Weinshall (2019) in defining the notion pacing function and use it to schedule how examples are introduced to the training procedure. However, we look into much more comprehensive set of pacing functions and different tasks in this work. Please see Section A for a comprehensive review of the literature on curricula.
IMPLICIT CURRICULA
Curriculum learning is predicated on the expectation that we can adjust the course of learning by controlling the order of training examples. Despite the intuitive appeal, the connection between the order which examples are shown to a network during training and the order which a network learns classify these examples correctly is not apriori obvious. To better understand this con- nection, we first study the order in which a network learns examples under traditional stochastic gradient descent with i..d. data sampling. We refer to this ordering which results from the choice of architecture and optimization procedure - as an implicit curriculum.
To quantify this ordering define the learned iteration of sample for given model as the epoch for which the model correctly predicts the sample for that and all subsequent epochs. Explicitly, ming- {t" ((): (1); yi, v'* ( < T} where y; and (1); are the correct label and the predictions of the model for i-th data point (see the detailed mathematical description in Section 3.1).
We study wide range of model families including fully-connected networks, VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016), Wide-ResNet (Zagoruyko & Komodakis, 2016), DenseNet (Huang et al., 2017) and EfficientNet (Tan Le. 2019) models with different optimiza- tion algorithms such as Adam (Kingma & Ba, 2014) and SGD with momentum (see Section B for details). The results Figure 2 for CIFARIO (Krizhevsky & Hinton, 2009) show that the implicit curricula are broadly consistent within model families. In particular, the ordering in which images are learned within convolutional networks much more consistent than between convolutional net- works and fully connected networks,? and the learned ordering within each sub-type of CNN is even more uniform. The robustness of this ordering, at least within model types, allows us to talk with less ambiguity about the difficulty of given image without worrying that the notion of difficulty is highly model-dependent.
Thhe difference between shallow fully connected nets and deep convolutional nets, to some extent, matches what Mangalam & Prabhu (2019) has found when comparing shallow and deep networks.
3

---

Published as a conference paper at ICLR 2021
143
10
1:0
0:03
0:00
0:22
6.30
0.73
0.03
0:02
0.69
0.66
2:23
0:25
0.34
0.3
333
0.56
1.00
0.60
6:65
4.00
0.22
0.09
0.79
1
6.72
0
0.70
6:72
9.00
635
342
0.00
PECOF
345
0:53
0.39
9:00
SU72
0.70
0.72
1.00
5:72
0.5
0.33
0.00
0.24
000000 240
Iem
D4
0.50
4.33
0.51
0.05
0:00
0.55
2:5
0:56
0.6
9.55
S:7
6.76
9.77
0:83
6:72
0.09
1,00
242
022
0:01
karn
0.14
0.00
0.24
0.22
0.04
0.25
6.2
3:51
TearTT
hann OUCOE ORCOM
Ed
IMrT
DAE
Oepore
WHIT bi WAYT bu BeoT
Figure 3: Scoring functions show a high correlation the standard training, but perceived difficulty depends on the training order. (Left) Six scoring functions computed on CIFARIO using the standard i.i.d. training algorithms. Columns from left to right show VGG-11 loss, ResNet-18 loss. VGG-11 11 iteration learned, ResNet-18 iteration learned, c-score, and loss based c-score. Here the order given by VGG-11 loss ResNet- 18 loss uses the recorded loss at epoch 10. (Right) Loss-based difficulty and learned iteration difficulty when performing non-standard training from left to right columns are: c-score baseline, curriculum- basnd training, random-training, and anti-curriclum training. The last three used the same pacing function step with a = 0.8 and b = 0.2. (Center) The Spearman's rank correlation is high between all scoring functions computed from ordinary curriculum training, but lower for random training and significantly lower for anti-curriculum, indicating that the three orderings lead to networks learning samples different orders.
We will see in the next section (and Figure 3) that, as expected, the choice of explicit curriculum can alter the order in which a network learns examples. The most dramatic manifestation of this is unti-curriculu learning where showing the network images in the reverse order indeed causes the network to learn more difficult images first. Next, we introduce the class of curricula will consider for the remainder of the paper.
PUTTING CURRICULA THROUGH THEIR PACES
Many different approaches have been taken implement curricula in machine learning. Here we focus on a particular widely used paradigm introduced in Bengio et al. (2009) and used in Hacohen & Weinshall (2019). In this setup. a curriculum is defined by specifying three ingredients.
The scoring function: The scoring function is map from an input example, æ. numerical score, s(x) E R.. This score is typically intended to correspond to a notion of difficulty, where higher score corresponds to a more difficult example.
The pacing function:
The pacing function (t) specifies the size of the training data-set used at each step, t. The training at step t consists of the g(t) lowest scored examples. Training batches are then sampled uniformly from this set.
The order: from lowest score to highest score, anti-curriculum- ordering examples from highest score to lowest, or random. Though technically redundant with redefining the scoring function, we maintain the convention that the score is ordered from casiest to hardest.
Additionally we specify an order of either curriculum ordering examples
This procedure is summarized in Algorithm 1. It is worth emphasizing that due to the pacing func- tion, using random ordering in Algorithm 1 is not the same as traditional i.i.d. training the full training dataset, but rather corresponds to i.i.d. training on a training dataset with dynamic size. We
Algorithm (Random-/Anti-) Curriculum learning with pacing and scoring functions
1: Input: Initial weights w training set (xI....,Xx) pacing function g : [T] + [N], scoring function $ : [N] + R, order o E ("ascending", "descending", "random")
2: (X1...., XN) < (ort(xx.....XN/ s,,0,
3: for = 1,...,T' T do
4: ((!) train-one-epoch(( (t-1), fxi,..., (x1,),
xx(1))
5: end for
4

---

Published as a conference paper at ICLR 2021
stress that the scoring and pacing function paradigm for curriculum learning is inherently limited. In this setup, the scoring function is computed before training over all of the data and thus the algorithm cannot implement self- and training-dependers curriculum has been considered in Kumar et al. (2010); Jiang et al. (2015); Platanios et al. (2019). Additionally, the dynamic training dataset is built by including all examples within fixed score window (from lowest score up in curricula and highest score down in anti-curricula) and does not accommodate more flexible subsets. Further- more, the form of curriculum discussed here only involves ordering examples from fixed training dataset, rather than more drastic modifications of the training procedure, such as gradually increas- ing image resolution (Vogelsang et al., 2018) 8) the classes (Weinshall et al., 2018). Nonetheless, is commonly studied and serves as a useful framework and control study to empirically investigate the relative benefits of training orderings.
Next, we describe scoring and pacing functions that will be used in our empirical investigation.
3.1 SCORING FUNCTIONS: DIFFICULTY SCORES ARE BROADLY CONSISTENT
In this section, we investigate three families of scoring functions. As discussed above, we define the scoring function s(x, y) E R to return a measure of an example's 's difficulty. We say that an example 2y is more difficult than an example 2; if s(xy yj) > s(z;,, ys). In this work, we consider three scoring functions:
Loss function. In this case samples are scored using the real-valued loss of reference model that is trained on the same training data,e.g. given a trained model fw X + y we set s(,, Vi) = {(fw (lff(())) (.))
Learned epoch/iteration. This metric has been introduced in Section 2.
We let s(x;, yi) =
min,-(1"" (1 (4); yı, At" S t S T) (see Algorithm 3 and Figure 11 for example).
Estimated c-score. c-score (Jiang et al., 2020b) designed to capture the consistency of a reference model correctly predicting particular example's s label when trained on indepen- dent i.i.d. draws of a fixed sized dataset not containing that example. Formally, s(x,, 34) = y: D)] where D, with |D|
is training data set sampled from the data pool without the instance (x,, y4) and @w, is the reference model prediction. [6( DD), where / is We also consider loss based c-score, s(x,, 34) LE" DD ((=).:))) the loss function. The pseudo-code is described in Algorithm 4.
D2 ((m..:)) [P(0w,
To better understand these three scoring functions, we evaluate their consistency in CIFARIO im- ages. In particular, for VGG-11 and ResNet-18, we averaged the learned epoch over five random seeds and recorded the loss epochs 2, 10, 30, 60, 90, 120, 150, 180, 200. We also compute the c-score using multiple independently trained ResNet-18 models and compare it to the reference c- scores reported in (Jiang et al., 2020b). The main results of these evaluations are reported in the left and middle panels of Figure 3. We see that all six scores presented have high Spearman's 's rank correlation suggesting a consistent notion of difficulty across these three scores and two models. For this reason, we use only the c-score scoring function in the remainder of this paper.
Additional model seeds and training times are presented Figure 12 and 13 in the appendix. Among all cases, we found one notable outlier. For ResNet-1 88 after 180 epochs, the training loss no longer correlates with the other scoring functions. We speculate that this is perhaps due to the model memorizing the training data and achieving near-zero loss all training images.
3.2 PACING FUNCTIONS: FORCING EXPLICIT CURRICULA
The pacing function determines the size of training data to used at iteration t (see Algorithm 1). We consider six function families: logarithmic, exponential, step, linear, quadratic, and root. Table 2 illustrates the pacing functions used for our experiments which is parameterized by (a, b). Here a is the fraction of training needed for the pacing function to reach the size of the full training set, and b is the fraction of the training set used at the start training, thus any pacing function with a = or b = 1 is equivalent to standard training. We denote the full training set size by N and the total number of training steps by T. Explicit expressions for the pacing functions we use, (), and examples shown in Figure 4.
5

---

Published as a conference paper at ICLR 2021
In order to many possible choices of pacing function, in all remaining experiments, select b E {0.0025, 0.1, 0.2, 0.4, 0.8) and a E {0.01, 0.1, 0.2, 0.4, 0.8, 1.6) for our empirical study. Pacing functions with these parameters are plotted in Figure 14.
Given these pacing functions, we now ask the explicit curricula enforced by them can change the order in which examples are learned. The right panel in Figure 3 indicates what is the impact of using curriculum, anti-curriculum or random ordering on the order in which examples are learned. All curricula use the step pacing functions with curriculum and random ordering do not change the order compared to c-score but anti-curriculum could indeed force the model learn more difficult examples sooner. This is also demonstrated clearly in the Spearman's 's correlation of matiicurricllum with other methods shown in the middle panel of Figure 3.
0.8 and b 0.2. The result indicates that
Name
Expression ((a,6)(t)
exp
size of data
linear
log
Nb + N(1 - b) (1 + .1 log a7 + e 10)
log
exp
Nb NO b) (exp
10A WT
1))
6N
quad
step
Nb + N
root
step
polynomial
Nb N
(uT 1
- p- 1121/2,1,2
0
step
Figure 4: Pacing functions (Left) pacing function definitions for the six families of pacing functions used throughout. (Right) example, pacing function curves from each family. The parameter determines the fraction of training time until all data used. The parameter b sets the initial fraction of the data used.
4 PACING FUNCTIONS GIVE MARGINAL BENEFIT, CURRICULA GIVE NONE
currkuuum unti-cunicuuum
random curricutum standaret
standard2 standards
exp
inear
log
quad
root
step
rand
72.
TE.0
155
CUT
Aandd
C 15,00
(a)
CIPARLDO 15200
CIAARED lest Ace
59.0
(b)
LIARR10 lest Acc
Figure 5: Curricula provide little benefit for standard learning. (a) Bar plots showing the best curacy, for curriculum (blue), anti-curriculum (purple), random-curriculum (green), and standard i.idd training (grey) with three ways of calculating the standard training bascline for CIFARIO (left) and CIFAR100 (right). (b) Means over three seeds for all 540 strategies, and 180 means over three standard training runs (grey) for CIFARIO (left) and CIFARI00 (right). The x-axis is the test accuracy. Marker shape signifies the pacing func- tion family. Black solid and dashed lines give the mean and standard deviation over standard training runs. The solid orange line is the standard2 baseline test accuracy. We observe no statistically significant improvement from curriculum, anti-curricllum, random training.
Equipped with the ingredients described in the previous sections, we investigated the relative benefits of (random-/anti-) curriculum learning. As discussed above, we used the c-score as our scoring function and performed hyper-parameter sweep over the 180 pacing functions and three orderings described in Section 3. We replicated this procedure over three random seeds and the standard benchmark datasets, CIFAR10 and CIFAR100. For each experiment in this sweep, we trained a ResNet-50 model for 35200 total training steps*. We selected the best configuration and stopping time using the validation accuracy and recorded the corresponding test accuracy. The results of these runs for CIFAR1O/100 are shown in Figure 5.
To understand whether (anti-) curriculum, or random learning provides any benefit over ordinary training, we ran 540 standard training runs. From these 540 runs, we created three baselines. The standard/ baseline is the mean performance over all 540 runs. The standard2 baseline splits the 540 *This step number corresponds 100 epochs of standard training (with batch size 128).

---

Published as a conference paper at ICLR 2021
standard:
acrderco
CloA302 Ind
CTAR300 57000 degn
CPMR000 1700
37002
Figure 6: Curriculum-learning helps when training with limited time budget. CIFAR100 performance when training with 17600, 1760 and 352 total steps (see Figure 17 for CIFARIO). Curriculum learning provides a robust benefit when training for 1760 and 352 steps. See Figure 5 for additional plotting details.
runs into 180 groups of 3; the mean is taken over each group of 3 and the maximum is taken over the 180 means. This setup is intended pacing parameters (a, b) each with three seeds. Lastly, standard3 is the mean value of the top three values over all 540 runs.
stand in for the hyperparameter search of the 30 choices of
Marginal value of ordered learning. When comparing our full search of pacing functions to the naive mean, standard1, all three sample orderings have many pacing functions that outperform the baseline. We find, however, that this is consistent with being an artifact of the large search space. When comparing the performance of all three methods to less crippled baseline, which considers the massive hyperparameter sweep, we find that none of the pacing functions or orderings statis- tically significantly outperforms additional sampling of ordinary SGD training runs, Furthermore, establish that using similar techniques remove examples from the training set (as opposed to introducing them) also does not help (see Figure 16 in the appendix).
Perhaps most striking is that performance shows dependence on the three different orderings (and thus scoring function). For example, in the CIFAR 10 runs, the best mean accuracy is achieved via random ordering, while in CIFAR100, the best single run has random ordering. This suggests that for these benchmark datasets, with standard training limitations, any marginal benefit of ordered learning is due entirely to the dynamic dataset size (pacing function).
CURRICULA FOR SHORT-TIME TRAINING AND NOISY DATA
In the previous section, we found little evidence for statistically significant benefits from curricula or pacing. This observation is consistent with the fact that curricula have not become a standard part of supervised image classification. In other contexts, however, curricula are standard. Notably, in practice, many large scale text models are trained using curricula (Brown et al., 2020; Raffel et al., 2019). These models are typically trained in a data-rich setting where multiple epochs of training is not feasible. Furthermore, the data for training is far less clean than standard image benchmarks.
To emulate these characteristics and investigate whether curricula are beneficial in such contexts, applied the same empirical analysis used in Section 4 to training with a limited computational budget and training on noisy data, It should be noted that the benefits of curricula for noisy data have been studied previously, for example, in (Jiang al., 2018; Saxena et al., 2019; Guo et al., 2018).
Limited training time budget. For shorter time training, we follow the same experiment setup described in Section 4 but modify the total number of steps, T. Instead of using 35200 steps, we consider training with T = 352, 1760 or 1 17600 steps. We use cosine learning rate decay, decreasing monotonically
over the T steps. The results are given in Figure 6, and in supplementary Figures 20 and 21 for CIFARIO/100.
We see that curricula are still no better than standard training for 17600 iterations. However, when drastically limiting the training time to 1760 or 352 steps, curiculun--earring but not anti- curriculum can indeed improve performance, with increasing performance gains as the the training time budget is decreased. Surprisingly. at the smallest time budget, 352 steps, we also see random- learning consistently outperforms the baseline, suggesting that the dynamic pacing function on its own helps improve performance. We depict the best pacing functions for 1760 and 352 steps in Figure& 8. In the 2nd plot of Fig (left, limit training), many pacing functions start with a relatively small dataset size and maintain a small dataset for large fraction of the total training time.
Data with noisy labels. To study curricula in the context of noisy data, we adopted a common procedure of generating artificial label noise by randomly permuting the labels (Zhang al., 2017;
7

---

Published as a conference paper at ICLR 2021
and canooin
35% hole
home
Figure 7: Curriculum-eearning helps when training with noisy labels. Performance on CIFAR 100 with the addition of 20%, 40%. 60% and 80% label noise shows robust benefits when using curricula.
2223022
eng CAS 42.0. lig. 8823
40.560 upo
23 3233
MMc 8.5033
64-4044
46 42-.422.
190 $48585
2000
20000
1428
122
12:43
17.07
20103
10:3
100R3
20093
Figure 8: Top performing pacing functions for limited time and noisy training. Top performing pacing functions from the six families considered for CIFARI00 (from left right) finite time training with 1760 and 352 steps, 20% label noise and 60% label noise.
CITARIO with 20 mp.
CPAROO with S.e..
CIASSD with ep
CITARICO with 32 sp
CITARIOO with S.p.
CIAR1DC with eb.
305
339
Storebnn
C.score
CIAR1100 noha
CITAP10O (2% nEss
CIARLDO 50%
CIAA1100 nobe
2 150
3 120
50
02
C.score
0.Scomt tnn
C. bin
Figure 9: Label noise and limited training time shift the e-score distributions towards more difficult examples. We compute loss-based c-score limited time training on CIFARIO (top left three). CIFAR100 (top right three) and for CIFAR100 models trained with label noise (bottom row).
Saxena et al., 2019; Jiang et al., 2020b). We considered CIFARI00 datasets with 20%, 40%, 60%, and 80% label noise and otherwise use the same experimental setup described in Section As the training data has been modified, we must recompute the c-score. The results are shown in Figure 9.
Equipped with the new ordering, we repeat the same set of experiments and obtain the results shown in Figure 7 and supplementary Figure 22. Figure shows that curriculum learning outperforms other methods by a large margin across all noise levels considered. The best pacing function in each family is shown in Figure 8. We see that the best overall pacing functions for both 20% and 60% noise are the step and exponential pacing functions corresponding to simply ignoring all noisy data during training. For 40% noisy labels, this strategy was not contained in sweep of and b values.
To understand why the reduced training time and label noise benefit from curricula, we plot in Figure the loss-based c-score distributions for CIFAR10/100 trained with 30, 5 and 1 epochs and CIFAR100 100 trained with noise 20%, 40% 60%, and 80%. For the distribution of clean CIFAR100/ 10 with 30 epochs (see title). significant number of images concentrate around zero. However, the concentration slowly shifts to larger values as the the total training epoch decreases or the label noise increases. This suggests that both label noise and reduced time training induce more difficult c-score distribution and curricula can help by focusing first on the easier examples.
CURRICULA IN THE LARGE DATA REGIME
In this section, we are interested in investigating whether the previous conclusions we made for CIFAR10/ generalize to larger scale datasets. We conduct experiments FOOD101 (Bossard
8

---

Published as a conference paper at ICLR 2021
Kuluum arti-curncuuum
random- kuulum standarct
standard2 standard3
exp
lineer
log
qued
root
step
$0:00
AMM
47U
C347
70.5
1000101 29600 steps
FOOD101 1480 1teps
FOOD 0IN 321300 steps
PC00100 24e60 Stons
Val ACC
FOCD:01 54B0 Stare FOOCCLDIN 221300 stor
Vol AcC
V2I ACC
(a)
(b)
Figure 10: Large data regime- FOOD101 and FOODI0IN, (a) Bar plots showing the best mean accu- racy, for curriculum (blue), anti-curricllum (purple), candom-curriculm (green), and standard i..d. training (grey) with three ways calculating the standard training baseline for FOOD101 standard -imee training (left), FOOD101 short-time training (middle), and FOOD101N training (right)". (b) For FOOD101 standard-time training (left), we plot the means over three seeds training runs (grey). For FOOD101 short-time training (middle), we plot the means over three seeds for all 216 strategies, and 72 means over three standard training runs For FOODI0IN (right), we reported the values for all 216 strategies and 72 standard training runs. See Figure 5 for additional details. In FOOD101 standard- time training. we observe no statistically significant improvement from curriculum, anti-curriculum, or random training. However, for FOOD101 short-time training and FOODIOIN training, CL provides robust benefit. Since only nandom seed for FOODIOIN, we plot the avenge of all standand runs (standand) and the best run 72 strilgiis (standan(2).
540 strategies, and 180 means over three standard
et al., 2014) which consists of 75, 000 training and 25, 000 validation images and FOOD10IN (Lee et al., 2018) which consists 310, 000 training images and the same validation images as FOOD101. The maximum length of the image dimensions are 512 pixels. We train the ResNet-50 model by resizing these images to be 250 and applying random horizontal flips and random crops with size 224 as in the PyTorch example for ImageNet. See Section B for more experiment details.
Standard Training Time. For handard-time training. the total number of training steps is 296 100, which corresponds to 100 epochs of standard i.i.d. training. Similar to Section 4, we train 180 x 3 models respectively for standard i.i.d., curriculum, anti-curricllum and random-curricllum learning over three random seeds. The result is presented in Figure 10. Again, we observe none of the orderings or pacing functions significantly outperforms standard i.i.d. SGD learning.
Limited time Training and Noisy Labels Training, For FOODI01 short-time training, models are trained for 296 x 5 steps. We do not search over 180 pacing functions. Instead, we consider 72 pacing functions with a E {0.1, 0.4, 0.8, 1.6} and b E {0.1. 0.4, 0.8} over the six pacing function families. For FOOD10IN training, models are trained for 121, 100 steps with single fixed random seed 111 and the same reduced pacing function search space as FOOD101. Although the hyper-parameter search space for both FOOD101 and FOOD101N training smaller than the one used Section 5, the conclusion is similar- Figure 10 shows curriculum still perform better than anti-curriclumm, random curriculum and standard SGD training.
In summary, the observations CIFARIO and CIFARI00 in three training regime standard-time, short-time and noisy training- generalize to larger-scale data like FOOD101 and FOODIOIN.
DISCUSSION
In this work, we established the phenomena of implicit curricula, which suggests that examples are learned in a consistent order. We further ran extensive experiments over four datasets, CIFARIO I0, CIFAR100, FOOD101 and FOOD101N and showed that while curricula are not helpful in standard training settings, masy-to-difficul ordering can be beneficial when training with limited time budget or noisy labels.
We acknowledge that despite training more than 25,000 models on CIFAR-like and FOOD101(N) datasets, our empirical investigation is still limited by the computing budget. That limitation forced us to choose between diversity in the choice of orderings and pacing functions as opposed to diversity in the tasks type, task scale, and model scale. Our strategic choice of limiting the datasets to CIFAR- 10/100 and FOOD-101(N) allowed us to focus on a wide range of curricula and pacing functions, which was very informative. As result, our general statements choice of curricula - which is the main object of study in this paper - but one should be careful about generalizing them tasks/training regimes that are not similar to those included in this work.
more robust with respect to the

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
We would like to thank for the discussions with the teammates at Google. We owe particular grati- tude to Mike Mozer, Vaishnavh Nagarajan, Preetum Nakkiran, Hanie Sedghi, and Chiyuan Zhang. We appreciate the help with the experiments from Zachary Cain, Sam Ritchie, Ambrose Slone, and Vaibhav Singh. Lastly, XW would like to thank the team for all of the hospitality.
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks. In International Conference on Machine Learning, pp- 233-242, 2017.
Judith Avrahami, Yaakov Kareev, Yonatan Bogot, Ruth Caspi, Salomka Dunaevsky, and Sharon Lerner Teaching by examples: Implications for the process of category acquisition. The Quarterly Journal of Experimental Psychology Section A, 50(3):586-600 1997.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of International Conference Machine Learning, 2009.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101 mol-ming discriminative compo- nents with random forests. In European conference computer vision, pp. 446-461. Springer, 2014.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2))223-311 --12,, 2018.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? volume 97 of Proceedings of Machine Learning Research, pp. 872-881, Long Beach, California, USA, 09-15 Jun 2019, PMLR.
Satrajit Chatterjee. Learning and memorization. International Conference on Machine Learning, pp. 755-763, 2018.
Volkan Cirik, Eduard Hovy, and Louis- Couis-Phllippe Morency. Visualizing and understanding curriculum learning for long short-tern memory networks. arXiv preprint arXiv:1611. 1.02200 2016.
Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99, 1993.
Yanbo Fan, Siwei Lyu, Yiming Ying, and Baogang Hu. Learning with average top-k loss. In
Advances in neural information processing systems, pp. 497-505, 2017.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 128):128-135, 1999.
Rong Ge, Runzhe Wang, and Haoyu Zhao. Mildly overparametrized neural nets can memorize training data efficiently. arXiv preprint arXiv: /1877 1837, 2019.
Ian Goodfellow, Mehdi Mirza, Aaron Courville Da Xiao, and Yoshua Bengio. An empirical investigation of catastrophic forgeting in gradientbased neural networks. In In Proceedings of International Conference on Learning Representations (ICLR. Citeseer, 2014.
Alex Graves, Mare G Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In International Conference on Machine Learning, pp- 1311-1320, 2017.
10

---

Published as a conference paper at ICLR 2021
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4)5,, 2009.
Jindong Gu and Volker Tresp. arXiv:1911.09530 2019.
Neural network memorization dissection. arXiv preprint
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R. Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep net- works. ICML, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
5901-5910, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.
Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary Lipton, al. Accelerating deep learning by focusing the biggest losers. arXiv preprint urXiv: 1910.0072.. 2019.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-paced cur- riculum learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2694 2700. AAAI Press, 2015.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pp. 2304-2313, 2018.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In ICML, 2020a. URL ht wawiiiiiiiivv/iii//eaaaaaaaaavva http::/s///xxxv.ooo////ss..//// / 9111 09781.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Exploring the memorization- generalization continuum in deep learning. arXiv preprint arXiv:2002.03206, 2020b.
Tyler B Johnson and Carlos Guestrin. Training deep models faster with robust, approximate impor- tance sampling. In Advances in Neural Information Processing Systems, pp. 7265-72755 2018.
Kenji Kawaguchi and Haihao Lu. Ordered sgd: A new stochastic optimization framework for em- pirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pp. 669-679, 2020.
Faisal Khan, Bilge Mutlu, and Jerry Zhu. How do humans teach: On curriculum learning and teaching dimension. In Advances in neural information processing systems, pp. 1449-1457, 2011.
Justin Khim, Liu Leqi, Adarsh Prasad, and Pradeep Ravikumar. Uniform convergence of rank- weighted learning. In International Conference on Machine Learning, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6 442266900, 2014.
11

---

Published as a conference paper at ICLR 2021
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3522 2017.
Tom Koemi and Ondřej Bojar. Curriculum learning and minibatch bucketing in neural machine translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pp. 379-386, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in neural information processing systems, pp. 1189-1197, 2010.
Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: 02862, 2017.
Yueming Lyu and Ivor W. Tsang. Curriculum loss: Robust learning and generalization against label corruption. In International Conference on Learning Representations, 2020.
Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? In ICML workshop, 2019.
Tambet Matiisen. Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learn- ing. IEEE transactions on neural networks and learning systems, 2019.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204, 2013.
Deanna Needell, Rachel Ward, and Nati Srebro. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. In Advances in neural information processing systems, pp. 1017-1025, 2014.
Cuong Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Ste- fano Soatto. Toward understanding catastrophic forgetting in continual learning. arXiv preprint arXiv: /0ti::900.000 00...9988999999 0109, 2019.
Diego Ortego, Eric Arazo, Paul Albert, Noel E O'Conoor, and Kevin McGuinness. Towards robust learning with different label noise distributions. arXiv preprint 11X1v:1912.08741, 2019.
Anastasia Pentina, Viktoriaa Sharmanska, and Christoph H Lampert. Curriculum learning of multi- ple tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5492-5500, 2015.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. Competence-based curriculum learning for neural machine translation. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1162-1172, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. 10683, 2019.
Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu, Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint arXiv:2007.07400, 2020.
Mengye Ren. Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference Machine Learning, pp. 4334 4343, 2018.
12

---

Published as a conference paper at ICLR 2021
Sam Ritchie, Ambrose Slone, and Vinay Ramasesh. Caliban: Docker-based job manager for producible workflows. Journal of Open Source Software, 5(53):2403, 2020. doi: 10.21 105/joss. 02403. URL /ttpp:://oi.... wttp:/..rg................ joss 02 403.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, pp- 3738-3748, 2018.
Terence Sanger. Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE transactions on Robotics and Automation, 10(3):323-333, 1994.
Nikolaos Sarafianos, Theodore Giannakopoulos, Christophoros Nikou, and Ioannis A Kakadiaris. Curriculum learning for multi-task classification of visual attributes. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 2608-2615, 2017.
Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning differentiable curriculum. In Advances in Neural Information Processing Systems, pp. 11095-11105, 2019.
Vatsal Shah, Xiaoxia Wu, and Sujay Sanghavi. Choosing the sample with lowest loss makes sgd robust. volume 108 of Proceedings of Machine Learning Research, pp. 2130-2130, Online, 26-28 Aug 2020. PMLR.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss mini- mization. In International Conference on Machine Learning, pp. $739-5744 PMLR, 2019.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1e::::409955555 2014.
Sainbayar Sukhbaatar and Rob Fergus. Learning from noisy labels with deep neural networks. arXiv preprint arXiv: 2080, 2(3):4, 2014.
Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference Machine Learning, pp. 6105-6114, 2019.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. ICLR, 2019.
Lukas Vogelsang, Sharon Gilad-Gutnick, Evan Ehrenberg, Albert Yonas, Sidney Diamond, Richard Held, and Pawan Sinha. Potential downside of high initial visual acuity. Proceedings of the National Academy of Sciences, 115(44): 11333-11333 2018
Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In Proceedingy the IEEF/CVF International Conference on Computer Vision (ICCV), October 2019.
Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In International Conference on Machine Learning, pp- 2018.
Lilian Weng. Curriculum for reinforcement learning. lilianweng, lilianwengggtthubioo/// githubbio//illllog 2020.
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small relu networks are powerful memorizers: tight analysis of memorization capacity. In Advances in Neural Information Processing Systems, pp. 15558 15569, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.12. BMVA Press, September 2016.
Wojeiech Zaremba and Ilya Sutskever. Learning execute. arXiv preprint arXiv: 1410.4410, 2014.
13

---

Published as a conference paper at ICLR 2021
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameterization. In International Confer- ence on Learning Representations, 2020.
Jiong Zhang, Hsiang-Fu Yu, and Inderjit S Dhillon. Autoassist: A framework accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2019a.
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianaa J Mar- tindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation. arXiv preprint ar 1011vv111100011 2018.
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. Curriculum learning for domain adaptation in neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguis- tics: Human Language Technologies, Volume (Long Papers), June 2019b.
A LITERATURE ON CURRICULUM LEARNING
The related work can be divided into the following four main subparts. With the best of our efforts, we mainly cover the recent ones among the numerous papers.
Understanding the learning dynamics of individual samples. is critical to understand the learn- ing dynamics of model and so bridge the gap between human learning and curriculum learning (Khan et al., 2011). I). An interesting phenomenon for the standard i.i.d. training algorithm is observed and termed "catastrophic forgetting" " neural networks cannot learn in sequence manner and suffer significant drop in generalization
earlier tasks (French, 1999; Goodfellow et al., 2014; Kirk- patrick et al., 2017; Ritter et al., 2018; Nguyen et al., 2019; Ramasesh et al., 2020). Even for a single task, forgetting events occur with high frequency some examples, which could lie in the subset of "hard" samples (Toneva et al., 2019). To further understand how the forgettable examples are related to "hard" examples and unforgettable examples "easy" ones, we select commonly used metric, loss functions, to define the difficulty (Bengio et al.. 2009; Hacohen & Weinshall, 2019). One criticism of using loss functions as metric that wver-aaameeterized neural networks can memorize and completely overfit the training data reaching the interpretation region if trained long enough (Arpit al., 2017; Zhang et al., 2017), which has triggered plenty of theoretical and empir- ical analysis (Chatterjee, 2018; Yun et al., 2019; Ge et al., 2019; Gu & Tresp, 2019; Zhang et al., 2020). Thus, in addition to loss functions, we measure the difficulty of examples with a consistency score (c-score) metric proposed in Jiang et al. (2020b), in which the model should never see the estimated instances, and the estimation should be accurate, close to full expectation.
Improved methods of curriculum learning. Fruitful methods have been proposed to improve scor- ing functions and pacing functions in various applications. One significant improvement about dy- namically updating the ordering and pacing based on the current models or learned teacher model. This idea can be positively related to importance sampling methods for empirical risk minimization (Gretton al., 2009; Shimodaira, 2000; Needell al., 2014; Johnson & Guestrin, 2018; Byrd & Lipton, 2019: 9; Khim et al., 2020). CL in the earlier or short-time training assigns the accessible instances with higher weight than difficult ones. Self celf-paced curriculum learning (Jiang et al., 2015; 2018), based on self- celf-paced learning (Kumar et al.. 2010), formulate a concise optimization problem that considers knowledge before and during training. Huang et al. (2020) adapts CL to define new loss function such that the relative importance of easy and hard samples are adjusted during different training stages, and demonstrated its effectiveness on popular facial benchmarks. The curriculum of teacher's guidance (Graves et al., 2017) teacher-student collaboration (Matüsenn et al., 2019) have been empirically proven useful in reinforcement learning for addressing sequential decision tasks. A comprehensive survey on curriculum reinforcement learning can be found in Weng (2020) and Platanios et al. (2019). Our work is not about proposing a method to improve CL training but empirically exploring CL's performance to clear the pictures when curricula help and when not.
14

---

Published as a conference paper at ICLR 2021
Algorithm 2 Loss function
Input: Initial weights w training (..,..., xN)
2: for t = 1,.... I"' do
3: w(t) train-one-epoch((ee (w(t-1) fxI...., xN
4: end for
5: for = =I....,N do
6: s(i) < (x;
7: end for
Algorithm Learned Epoch
1: Input: Initial weights w training set (xI...., XNN}
2: for I.....N do
3: s(i) - 0
4: c(i) (T 1)t0/1 (x))))))
5: end for
I,...,, do 6: for (()) train--ne--pocc ,
7:
8: f = I.....N do
9: if (fwie) (x,), yk) then
10:
c(i) < min(c(i),
11:
else
12:
c(i) + T +
13:
end
14: s(i) < sii) E(fw(n) U:)TT
15: end for
16: end for
17: for i 1-..,.., N do
s(i) s(i) ci)
19: end for
Curriculum Learning with label noise. CL has been widely adopted with new methods to attain positive results in the learning region with artificial or natural label noises (Natarajan et al., 2013; Sukhbaatar & Fergus, 2014; Zhang et al., 2017; Jiang et al., 2020a). Guo al. (2018) splits the data into three subsets according to noisy level and applies different weights for each subset during different training phases, which achieved the state-of-the-art result the Web Vision data set (Li al., 2017). Ren al. (2018) uses meta-learning learn the weights for training examples based on their gradient directions. Saxena et al. (2019) obtains significant improvement by equipping each instance with learnalle parameter governing their importance in the learning process. Shen & Sanghavi (2019) and Shah et al. (2020) suggest iteratively discards those high current losses to make the optimization robust to noisy labels. Lyu & Tsang (2020) presents (noise pruned) curriculum loss function with theoretical guarantees and empirical validation for robust learning. Given these success cases, empirical investigation with label noise for CL is within our scope.
Anti-curricllum v.s. large current loss minimization. Although both anti-curriculum learning and large current loss minimization focus on "difficult" examples, the two ideas could be orthogonal, particularly in longer training time. The former relies on static pre-defined ordering while the latter requires the order iteratively updated. For a short time learning, the two could be similar. Among much work on minimizing large current loss (i.e., Fan et al. (2017); Zhang et al. (2019a); Jiang et al. (2019); Ortego et al. (2019) and reference therein), Kawaguchi stochastic gradient descent (SGD), which purposely biased toward those instances with higher cur- rent losses. They empirically show that the ordered SGD outperforms the standard SGD (Bottou et al., 2018) when decreasing the learning rate, though it is not better initially. Supported with big loss minimization (short-time training). hard tata-mining methods, and some empirical success of anti-curriculum (mentioned in the introduction), investigate anti-curriculum.
Lu (2020) introduced ordered
15

---

Published as a conference paper at ICLR 2021
Algorithm 4 Estimated C-Score
1: Input: training set S {x1.... xN) ratio a, number of repeated experiments
2: for
3: for 1,2,... 7 do
[1/a] do 4: Sies = (X((-1)aa))
(3aN))
5:
w f- train(S/SS
6:
for x; Swa do
7: s(i) + s(t) + (x,)
8:
end for
9: end for
end for
Test 0-1 predictions during training process
800
600
ordering by
400
200
200
400
6d0
800
1000
1200
(x30)
Test loss during training process
1000
The ardering by Ieam'
600
0
200
600
1000
1200
Iterations (x30)
Figure 11: The top figure plots the 0 (dark blue) and (yellow) prediction for the images (y-axis) during the iterations (x-axis). The bottom figure plots the corresponding loss values. The images belong to class in the testing of CIFARIO. We the standard time training with batch-size 256 (Section B) and record the loss 0-1 predication every 30 iterations.
B EXPERIMENT DETAILS
We summarize the common experimental setup first and give details for each figure. We first clarify standard training that appears in most the figures (such as Figure 1, and 7), where we are based on the code in Py Torch and use ResNet50 model with default initialization." The data augmentation includes random horizontal flip and normalization, and the random training seeds are fixed to be (111, 222, 333). We choose batch size to be 128 and use one NVIDIA Tesla V100 GPU for each experiment. We use Caliban (Ritchie et al., 2020) and Google cloud AI platform to submit the jobs. The optimizer is SGD with 0.9 momentum, weight decay 5 x 10-5 and learning rate scheduler
'https: : /
thub. wtt.ttot...ooooooootootooeoco httpbccc//ggrrrrrr/////a///////a
htw.:/asttterreeeeeeeeeeee iwmammagggemmnmmeee in.
16

---

Published as a conference paper at ICLR 2021
EXI
CM
S-
SA4
SA-1
4EA
CXT
SC2
S/4
CN
3
SKC
IVD
CM
27
CM
PN
6:8
871
S49
140
Figure 12: The left plot is the visualization of the ten orderings of CIFAR10 10 using the standard i.i.d. training algorithms I.O.V, L60.V V and 1.000 V: the loss function of VGG11 epoch 10, 60 and 200. (b) LJO.R: the loss function by ResNet18 at epoch 10, 60 and 200. (c) T.Vgg: the learned iteration by VGG-11 (d) ResNe:: learned iteration by ResNet18 (f) C.ResNer: the c-score by ResNet18 with much less com- putation than (e). The Spearman correlation matrix for the orderings is plotted on the right. Note that we select LIO.V, RIO.R and the last four ordering to present in Figure 3.
50000
1.0
The ordering by column
40000
0.8
12
30000
18
0.6
21
27
20000
30
36
10000
39
24
30
33
36
Scoring Funcions
Figure 13: The left plot is the visualization of the ten orderings of CIFAR1O using the standard i.i.d. training al- gorithms, curriculum, citriiurrculu and random- curriculum learning. All columns but the last four are using the loss function as scoring metric (a) 1-9 columns are VGG11 at epoch 2, 10, 30, 60, 90, 120, 150, 180, 200. (b) 10-18 columns are ResNet18 at epoch 2, 10, 30, 60, 90, 120, 150, 180, 200. (c) 19-27, 28-36, 37-45 columns are anti-curiclum, curriculum and random curriculum learning with step-pacing function parame- ter a = 1.0 = 0.2 step 2291, 7031, 11771, 16511, 21251, 25991, 28361, 30731 and 39100 (d) 46-49 are Vega...VVgg.... ResNNe, c-score.loss and c-score as described in Figure 3 and Figure 12. The right plot the corresponding Spearman correlation matrix.
random-curriculum
cosine decay - with an initial value of 0.1. We use half- nref-pre for training. In most of our cases, when using "standard training setup", we will implicitly mean the total training epoch is 100. For the figures in Section and 3. we use the full training samples 50000. For figures Section 4 and 5, we use training samples 45000 and validation samples 5000. We look for the best test emror of these 5000 validation samples and plot the corresponding test error/prediction.
For other figures such as Figure 3, 12 and 13. We vary the optimizers by either completely switching to Adam, or using different learning rate schedulers such as step-size decay every 30 epochs. The batch size are chosen to be 128 and 256, and weight decay 5 x 10 1 x 10 1 10-6 or 0.
40000
20000
dep
4000
Figure 14: Visualization of the six types of pacing functions for T 39100 and N 50000 with a E
(0.01. 0.11,. 0.4, 1.0, 1.6) and (0.0025, 0.1, 0.4, 1.0).
17

---

Published as a conference paper at ICLR 2021
When using fully- fulle-co nets, VGG, and other networks, we pick the learning rates 0.01 among grid search in {0.5, 0.1, 0.001).
For (anti-/random-) curriculum learning, move the torch. utils. data. Dato in the training loop so that to update the size of training data dynamically. We also refer to the code in Hacohen & Weinshall (2019).9 In particular, each time load data, we make class balance a priority, which means we do not completely follow the exact ordered examples. However, our ordering does imply the ordering within class.
In the following subparts, we give details for each figure.
Figure 2 in Section 3. Algorithm 3 illustrate the main steps we take to measure the order. We now present the configuration for each experiment that translates to the ordering. We design three layers for the architecture in fully connected networks (with hidden unites 5120 for the first layer and 500 or 1000 for the second one). ReLU activation function is used between the layers. For VGG networks, we use VGG11 and VGG19. For both networks, we use a learning rate of 0.01 and weight-decay 0.0005. We vary the random seeds and the optimizers, as mentioned above. For the Adam opti- mizer, we set epoch to be 100 and initialize the learning rate 0.001 decaying by a factor of 10 after 50 epochs. For SGD with 0.9 momentum (100 epochs), we use cosine learning rate schedules or decaying schedule with a factor of 10 after 100//3 /3 epochs. Moreover, we widen the choice of con- volutional networks including VGGI (GG)1(19)-BN) denoted by VGG11(19) with batch normalization, ResNet18, ResNet50, WideResNNe28- 10, WideResNNet4--Ue 10 DenseNet121, EfficientNet BO. We set the learning rate for standard SGD training- 0.1. Finally, we vary the training epochs and choose a longer one, 200 epochs, to check the consistency. In summary, we present all the experiments in Figure 15 where we have seed_# arc_# scheduler # with # indicating the configuration for SGD with 0.9 momentum training, and seed_# arc_# adam for Adam. We add 11 to indicate those longer 200-epoch training.
Figure 3, 12 and 13. We mainly focus on filling the details on how to train the models since Algo- rithm 2 and are straightforward to implement. For VGGI I and ResNet18, we use the "standard training setup" (the random seed is 111) but with epoch to be 200 to record more orderings. Here, we use two data loaders to the same training data one is augmented for training. while the other is clean for ordering. In this way, we minimize the effect caused by other factors. For c-score.loss, we separate the training data into four parts and use 3/4 of the data to train new random initialized model - ResNet18. 8, and record the value for the remainder 1/4 data. A shorter epoch of 30 is used for each training with cosine decaying learning rate (reaching zero after 30 epoch).
the order c-score.loss right above. We follow the standard setup with step-type pacing. The hyper- parametesss are = 0.2 and a = 0.8. Algorithm and 3 are used to measure the ordering during training.
For titrrandom--curriculle
learning,
Figures in Section 4 and 5. The experiments are explained partially in the main text. Due to the nature of the (anti-/random-) curriculum learning that has different iterations in epoch (as update the size of training data dynamically). we use the notation "iteration/step" instead of "epoch". We want to emphasize that when we decrease the number of training steps from 35200 in Figure to 17600, 1760 and 352 in Figure 6, we implicitly accelerate the decrease in the learning rate as it eventually drops to zero within the steps we set. For fair comparison, we set the configuration for each run the same as "standard training setup" (mentioned at the beginning of this section), including the data- aatta-augmentttion, batch-size, size, optimizer (weight-decay and learning rate), and random seeds.
F Figure 10. For FOOD101 and FOODIOIN, we choose a batch size to be 256 and use NVIDIA Tesla 8x V 100 GPU for each experiment. The data augmentation is the same as the ImageNet training in PyTorch example (see Footnote 5). The optimizer is SGD with 0..9 momentum, weight decay 10-5, and a learning rate scheduler cosine decay- with an initial value of 0.1. We use half-precision for training.
C FIGURES
hnttp:://gi thub. gei fwany/ciaar-aeaeea
18

---

Published as a conference paper at ICLR 2021
10000_ 111-rrt__hhh J01003--5on
223 Inwooreddder.oo
2 201.222---e331333333-3
04 sead 222-arch 1o0 11,3
103 seer 222-cc
205erct 301303--30n
06 seed 11-4rc 110100000 by
5seed 365-arh 5::003-a5am
027 edue oodkn
7esss 111-arth 1=:003--cted zer codra
77 seed arch vep1R Invecred stop
seed 333-ach
seed vigto
305-rch 2:1003-ddealun aur.aasr.aaae
80
pesd, 115 Lnw.oaa000
aae. 222 100000000000000000000000000
--000-atmm
sead vpp Inveradderr
ocheddar u0R0
12 2000 -11- -000-
72 aead 305-rrch Simmare
14 00000 222 cmo 1.00 osion
74 sccdl Donsone
Lookin
16 /000 300 em 30000 s5om
75 seed anth Densoner edh scuie coomne
17 s.0: 772-anth 1-003-oched far zodin
77 sead 222-877 deroerne
reed 313-wchh
sonoretto thadeen
28s0 19
1000 0000-cceed...
TO see0 11 11-rrch Simmonet
222 www.ncabt eoddam
20 _..: 305 1-00--uche
527 edkle o
373-acch wwwreeffo
22 2... [11-art >pp =s011--ccam 1--eaa
121 seed Tanch ww0mmetBbs1
24 af1- -co1f-ecam.1
denuanet-
122 seed 111-dh ochedas ww0.W/G5110
25 0081
>55 loom
02 seed onc Dencono dit scuee clap
27 1004 222 erce .5211 1-lto-mmmm
D4 seed 203 arch Sensonat echaccler stop
0888 28 [11
see0 erce Domsorot
30 sssd -5811 eevesseer coehne
I7 seed 633- modil edww
seed 100000220110
32 2est 222 >50 1i-e-eeeeerr 2
(80 seed a Stnwweb
120 seed beech www. 1D-
2... -oo11 oereooke 2 34
vokde 10
111-ach 10
55 0000 am -0815 coom
G27 educ oodin
Schodan g0pp1
37 0000 aco-rce (il---eeesber coemn
R27 arde osdin
scheddee dep-1
30000 ame_ 1211 wew.essooooooeooo 30
seed
slap2
43 nead Witeemeet
232-athh
sosorettt
42 20st 222 ar +2818-000 deleoetteer slep2
34 seed 222-rrch aft5en
132 seed 003-ach wewseet0x1
2880 111 .0811 Im-dda
sokle costmo1 05
s4ed
<odee
45 s00d erc 10811 odan
G07 eolcdaa .top
6eed a22 wwoooooo00001100
47 seed 272-arth v0011 tevecheckde oodes
527 rree dee..
135 seed (1hrch Wo
2880 a0111 49
poo8ns
seed (1
50 seed w0th wf% 10
schedder
conocurr .ocino1
09 seed 220 arce www.eenn00ii1
51 [_aes__11-mm 111-4mh_-02111_10 1_0 ooetn
sesd_ 31 Schedder
333 arch bf
02 3000 308-4r0 >g811 couur step
GCT cold depp1
133 6000 111
www.:11110oooooooooo111111 w00000000000
54 298t 111 arc- +5811 snccheeu deep
102 reed 11 arch prewenette adom
140 reed 111-ach weo0et00:1D-
19 3888
*9811 couoy sepp1
seed arch pww.occttttt. cdam
sotcolas 222-ech tomereeettt 1D=
Inechactde
104 222
07 sse mr =9811 socou step
607 eddedda oodin
6chedaer 3000
09 sest 111 amn 12815 odam
Sclodday _np.1
60 senz 222-mm 10810 Invedem owdin
62 00s0_111-4000 -2 111111111000_11..____1 Jn--cohooooo__oodrr
107 seed 303 anch pww.erewettttteooo adom
This figure describes the key hyper-parameters used for each of the columns Figure 3. For example, the first top left: lseed. 111 arch. fc1000 - -0110000-ammmmm corresponds to the result of 1st column in Figure 3 where we use random seed 111 and fully-connected layers with Adam as an optimizer. The ambiguous one where there is adam.11 means that we use twice a longer time than those with configuration ends with adam.
Figure 15:
365
3600
2004 20p4
Figure 16: Standard time training for CIFARIO0 with total 39100 steps. The training setup different from the figures in the main sections, where trained 45000 images and then 5000 validation images to pick the best test accuracy. Here, we use the entire training samples 50000 and directly pick the best test accuracy from these 5000 test samples. Thus, the accuracy is higher in general. The left bar (dot) plot increasing pacing functions. The right bar (dot) plot uses the reverse pacing functions (see the 5th plot for an example). That is, start with full data set and then slowly discard the images in [9()), .V] as g(t) is now decreasing, See Figure 5 for detailed description for each plot.
the normal
19

---

Published as a conference paper at ICLR 2021
comkubm unticcumiuuum
Mandardt
csp
1g
quad
stop
CiAnio 1999
<1 17660 ShN0
chvc0 152 Stepe.
10022
25d store
Figure 17: Short time training for CIFARIO with total steps equal to 19550, 3910 and 391 (see x-label). See Figure for detailed description for each plot.
Balky Taaning 4tt% Randeon Labels
hidgy Training 89t boncommLatet
anding haiilllayyyy
paiing
sog:: 03 120.5
coed 63 $20.2
ang: 27.3203
quad 25.18100
lnner 02 320.
rost: 38.020.3
root: 25120.
log: 33.13/9
step 63. 3e0
log 25:080
stes: 27.4
00000
20000
10000
258
7040
14100
20109
7043
1-:80
28.60
Step
saag
Figure 18: Top performing pacing functions for noisy training. Top performing pacing functions from the six families considered for CIFAR100 with 40% label noise (left) and 80% label noise (right).
CPARIO 35.2 stops
CIARSO 35,24 steps
TAB
WE ETTLOUT appertt Grandor
pecing
Ensar paotig
pacing
quid pacing
rocl puct
JAB pectigg
CHATEOO 23OD
CHARISE
25 Jloey
12 200
Standard time training with steps 35200. The top panel is for CIFARIO and the bottom for CIFARI00. In each panel, we have 3 x plots where each row from 1st to 3nd are curriculum, and curriculum learning (see the y-label). Each column means the type of pacing functions: from Ist to the 6th are exponential, linear, log quadratic, root, and step basic functions (see x label). Each plot inside a panel is heat-map for the parameter b E {0.0025, 0. 0.2, 0.4, 1.0} at y-axis describing the best accuracy for the corresponding (x, y) = (a, b). The dark blue light blue means the lowest value the median one, while the light red dark red means the median to the highest one. So the darkest blue is the lowest, and the darkest red is the highest.
Figure 19:
anti-
(0.01, 0. 0.2, 0.4, 1.0, 1.6) x-axis and the parameter
20

---

Published as a conference paper at ICLR 2021
CENRIO 17
OITRIO
GG4
BooM pedng
lgg pactig >
reel pet Fi
SEEP peoing
F2
Enear pactig
pacing
03440 pacing
FDCT Ei
1180 puciii
CRATID JNG ALTO
CIATTEO
AC
144
A+
peciigia
Figure 20: CIFARIO short time training. Top, middle and bottom panels are total steps 17600, 1760 and 352.
21

---

Published as a conference paper at ICLR 2021
Bnoo peding
sociing
reel
peoing
CHARGES Stors
ALR
LCC
SUP
deaas
22
11
11
2+
Figure 21: CIFAR100 short-time training. Top, middle and bottom panels are total steps 17600, 1760 and 352. See Figure 19 for detailed description.
22

---

Published as a conference paper at ICLR 2021
CAMESTSMI
SI0e
CRAFIECHIZ
2B stosss
14
CIARSSOMM 35 shae
CENEICONDE 25 steat
pecingie
CEARSECMIN 23 IhAps
CENRICONOS 24 steas
packng
paong
paorga
COMISSEMM
CRARSEONOS 23 stop
214
Enear pedtine
12.60 pacingg
step pactip a
Figure 22: Noisy training for CIFAR100 with 35200 steps. From top to bottom plots are for 20%, 40%, 60% and 80% label noise; See Figure 19 for detailed description.
23


Reviews:
[{"review_id": "thGXJ_h8wSr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n \nThe paper provides a comprehensive analysis of the benefits of curriculum learning in different application scenarios. This includes investigating the phenomenon of implicit curricula, showing if the examples are learned in a consistent order across different architectures, and exploring the influences of explicit curricula in the standard and emulation settings. The paper empirically shows that curriculum learning has marginal benefits for standard training, but is helpful when the training time is limited or the training data is noisy.\n\n##########################################################################\n\nReasons for score: \n\nI vote for accepting the paper. I believe the analyses presented in the paper can be valuable for the community. I like the implicit curricula experiments, showing that the difficulty of an example is somewhat independent of the training method. Other empirical observations are also interesting. In general, I think the paper provides a satisfactory answer to the question raised in the paper title (when do curricula work?) \n\n##########################################################################\n\nPros: \n\nThis paper has extremely comprehensive evaluations, examining the influence of curriculum learning (curriculum/anti-curriculum/random-curriculum) in diverse settings (standard/limited training time budget/noisy data). The methodology for the evaluations is technically sound;\n\nThe findings presented in the paper can be valuable for the community: (1) the difficulty of an example is somewhat independent of the training method; (2) curriculum learning provides little benefit for standard training, but help for limited time and noisy training;\n\nThe paper is well written. It is a thoroughly enjoyable experience to read the paper.\n\n##########################################################################\n\nCons:\n\nI found few weaknesses in the paper. I include a question below which I hope could be clarified:\n\nThe learned iteration of a sample is defined by the first epoch from which the prediction remains correct for all subsequent epochs. I wonder if there is any sample that is predicted correctly in earlier steps but incorrectly in later steps (e.g.  the forgettable examples). How to handle them in the implicit curricula experiment?\n\n\n\n##########################################################################\n\nMinor comments: \n\nPage 16: two data loader → two data loaders\n\n#########################################################################\n\nFinal recommendation:\n\nI have read the authors' responses as well as the comments from my fellow reviewers. I would like to keep my rating of the paper (8).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "tTVUDbmJwtR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper conducts a large-scale evaluation of the impact of curriculum learning (CL) in image classification. The paper progresses nicely through a sequence of well-thought research questions and experiments, with the key findings stated up front. In particular, the notion of \"implicit curriculum\" is shown to exist. Prior findings around when CL is helpful (limited training, label noise) are confirmed, which is nice. Overall, this methodical empirical evaluation comes away with a clear set of takeaways, empirically \"summarizing\" a lot of prior work on CL and the training of deep models. Some discussion about why CL helps when training is limited or data has label noise (or next steps) would strengthen the paper a bit more.\n\nStrengths:\n  + Extremely well-written and easy to read. Key findings are summarized and visualized up front.\n  + Well-designed large-scale empirical investigation into important open questions for training image classifiers.\n\nAreas for improvement\n  - Can't think of too many. I suppose the paper could have included a bit more discussion into why the reduced training or label noise benefits from curriculum learning. I'm also curious to see how these findings compare with a similar study on sequence (especially text) data but as the paper mentions, it's outside the scope of this paper.\n  - A pointer up front directing the reader to the appendix where \"standard training\" is defined would have been nice to have.\n\nQuestions:\n  * On Page 5, I didn't follow the sentence \"Given **these pacing functions**, we can now ask if the explicit curricula enforced by them can change the order in which examples are learned.\". I agree that Fig 3-right shows that the difficulty ordering (e2d, rand, d2e) can change the order in which examples are learned when using the step pacing function. What other pacing functions are used in Fig 3-right? Is there a different interpretation of the sentence involving the pacing functions?\n\nMinor comments\n  - A few typos\n    - Fig 2's caption (\"ReseNet50\", \"EfficeientNet\")\n    - Page 6 (\"CIAFR10\")\n\n\nUPDATE: I thank the authors for their detailed response and updated paper. I'm now more inclined to accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "223Yf19Th9K", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a comprehensive empirical study on the effects of the ordering of training samples in curriculum learning. The authors designed extensive experiments and obtained some interesting results: 1. the models with a similar architecture learn samples in a consistent order; 2. with enough iterations curriculum learning has no gain on performance comparing with random ordering; 3. curriculum learning outperforms others when the training time is limited; 4. curriculum learning is more robust with noisy samples. \nIn general, I think this is a quite practical work that could be beneficial to the community. The experiments are carefully designed and the results are sound, and the paper is well structured. \n\nThere are just some minor issues that may need some clarification:\n1. As the authors mentioned in Sec.3, the random ordering is not as the same as i.i.d. training because it corresponds to dynamic training size. Isn't it more similar with bootstrapping? It would be interesting if the authors can have some discussion in this point of view. \n2. The parameters of pacing function a, b should be introduced with the pacing function in the beginning of Sec.3 , as they were mentioned before Sec. 3.2 without explanation. \n3. Subfigures in Fig.5 should have subcaptions.\n4. In the experiments with noisy labels,  the best pacing functions ignore all noisy data, does it mean their gain of performance is simply from filtering out the noisy samples? Is there any other influence from the formulation of the pacing function itself? \n5. In the paragraph under Fig.8, the last sentence 'a trend to start and maintain ...' is a bit confusing to understand, could authors clarify it a bit?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This nice paper gives a better understanding of how Curriculum Learning (CL) affects image classification. In particular, it gives insight into cases such as noisy training data and limited training time. It shows that examples can be rated by difficulty to some extent, in that the order in which examples are learned seems to be consistent across runs. The paper is thorough and well-written.
Decision: Accept (Oral)