When Do Curricula Work?

1 INTRODUCTION . Inspired by the importance of properly ordering information when teaching humans ( Avrahami et al. , 1997 ) , curriculum learning ( CL ) proposes training models by presenting easier examples earlier during training ( Elman , 1993 ; Sanger , 1994 ; Bengio et al. , 2009 ) . Previous empirical studies have shown instances where curriculum learning can improve convergence speed and/or generalization in domains such as natural language processing ( Cirik et al. , 2016 ; Platanios et al. , 2019 ) , computer vision ( Pentina et al. , 2015 ; Sarafianos et al. , 2017 ; Guo et al. , 2018 ; Wang et al. , 2019 ) , and neural evolutionary computing ( Zaremba & Sutskever , 2014 ) . In contrast to curriculum learning , anticurriculum learning selects the most difficult examples first and gradually exposes the model to easier ones . Though counter-intuitive , empirical experiments have suggested that anti-curriculum learning can be as good as or better than curriculum learning in certain scenarios ( Kocmi & Bojar , 2017 ; Zhang et al. , 2018 ; 2019b ) . This is in tension with experiments in other contexts , however , which demonstrate that anti-curricula under perform standard or curriculum training ( Bengio et al. , 2009 ; Hacohen & Weinshall , 2019 ) . As explained above , empirical observations on curricula appear to be in conflict . Moreover , despite a rich literature ( see Section A ) , no ordered learning method is known to improve consistently across contexts , and curricula have not been widely adopted in machine learning . This suggest ruling out curricula as a beneficial practice for learning . In certain contexts , however , for large-scale text models such as GPT-3 ( Brown et al. , 2020 ) and T5 ( Raffel et al. , 2019 ) , non-uniform mixing strategies are standard practice . These contradicting observations contribute to a confusing picture on the usefulness of curricula . This work is an attempt to improve our understanding of curricula systematically . We start by asking a very fundamental question about a phenomenon that we call implicit curricula . Are examples ∗Work performed while Xiaoxia Wu was student at UT Austin and interning at Blueshift . 1Code at https : //github.com/google-research/understanding-curricula learned in a consistent order across different runs , architectures , and tasks ? If such a robust notion exists , is it possible to change the order in which the examples are learned by presenting them in a different order ? The answer to this question determines if there exists a robust notion of example difficulty that could be used to influence training . We then look into different ways of associating difficulty to examples using scoring functions and a variety of schedules known as pacing functions for introducing examples to the training procedure . We investigate if any of these choices can improve over the standard full-data i.i.d . training procedure commonly used in machine learning . Inspired by the success of CL in large scale training scenarios , we train in settings intended to emulate these large scale settings . In particular , we study the effect of curricula when training with a training time budget and training in the presence of noise . Contributions . In this paper , we systematically design and run extensive experiments to gain a better understanding of curricula . We train over 25,000 models over four datasets , CIFAR10/100 , FOOD101 , and FOOD101N covering a wide range of choices in designing curricula and arrive at the following conclusions : • Implicit Curricula : Examples are learned in a consistent order ( Section 2 ) . We show that the order in which examples are learned is consistent across runs , similar training methods , and similar architectures . Furthermore , we show that it is possible to change this order by changing the order in which examples are presented during training . Finally , we establish that well-known notions of sample difficulty are highly correlated with each other . • Curricula achieve ( almost ) no improvement in the standard setting ( Section 4 and 6 ) . We show curriculum learning , random , and anti-curriculum learning perform almost equally well in the standard setting.2 • Curriculum learning improves over standard training when training time is limited ( Section 5 and 6 ) . Imitating the large data regime , where training for multiple epochs is not feasible , we limit the number of iterations in the training algorithm and compare curriculum , random and anti-curriculum ordering against standard training . Our experiments reveal a clear advantage of curriculum learning over other methods . • Curriculum learning improves over standard training in noisy regime ( Section 5 and 6 ) . Finally , we mimic noisy data by adding label noise to CIFAR100 and also use a natural noisy dataset – FOOD101N . Similar to Jiang et al . ( 2018 ) ; Saxena et al . ( 2019 ) ; Guo et al . ( 2018 ) , our experiments indicate that curriculum learning has a clear advantage over other curricula and standard training . Related Work . Bengio et al . ( 2009 ) is perhaps the most prominent work on curriculum learning where the “ difficulty ” of examples is determined by the loss value of a pre-trained model . Toneva 2See the first paragraph of Section B for details of the standard-time experimental setup . et al . ( 2019 ) instead suggested using the first iteration in which an example is learned and remains learned after that . Finally , Jiang et al . ( 2020b ) has proposed using a consistency score ( c-score ) calculated based on the consistency of a model in correctly predicting a particular example ’ s label trained on i.i.d . draws of the training set . When studying curriculum learning , we look into all of the above-suggested measures of sample difficulty . We further follow Hacohen & Weinshall ( 2019 ) in defining the notion of pacing function and use it to schedule how examples are introduced to the training procedure . However , we look into a much more comprehensive set of pacing functions and different tasks in this work . Please see Section A for a comprehensive review of the literature on curricula . 2 IMPLICIT CURRICULA . Curriculum learning is predicated on the expectation that we can adjust the course of learning by controlling the order of training examples . Despite the intuitive appeal , the connection between the order in which examples are shown to a network during training and the order in which a network learns to classify these examples correctly is not apriori obvious . To better understand this connection , we first study the order in which a network learns examples under traditional stochastic gradient descent with i.i.d . data sampling . We refer to this ordering – which results from the choice of architecture and optimization procedure – as an implicit curriculum . To quantify this ordering we define the learned iteration of a sample for a given model as the epoch for which the model correctly predicts the sample for that and all subsequent epochs . Explicitly , mint∗ { t∗|ŷw ( t ) i = yi , ∀t∗ ≤ t ≤ T } where yi and ŷw ( t ) i are the correct label and the predictions of the model for i-th data point ( see the detailed mathematical description in Section 3.1 ) . We study a wide range of model families including fully-connected networks , VGG ( Simonyan & Zisserman , 2014 ) , ResNet ( He et al. , 2016 ) , Wide-ResNet ( Zagoruyko & Komodakis , 2016 ) , DenseNet ( Huang et al. , 2017 ) and EfficientNet ( Tan & Le , 2019 ) models with different optimization algorithms such as Adam ( Kingma & Ba , 2014 ) and SGD with momentum ( see Section B for details ) . The results in Figure 2 for CIFAR10 ( Krizhevsky & Hinton , 2009 ) show that the implicit curricula are broadly consistent within model families . In particular , the ordering in which images are learned within convolutional networks is much more consistent than between convolutional networks and fully connected networks,3 and the learned ordering within each sub-type of CNN is even more uniform . The robustness of this ordering , at least within model types , allows us to talk with less ambiguity about the difficulty of a given image without worrying that the notion of difficulty is highly model-dependent . 3The difference between shallow fully connected nets and deep convolutional nets , to some extent , matches what Mangalam & Prabhu ( 2019 ) has found when comparing shallow and deep networks . We will see in the next section ( and Figure 3 ) that , as expected , the choice of explicit curriculum can alter the order in which a network learns examples . The most dramatic manifestation of this is anti-curriculum learning where showing the network images in the reverse order indeed causes the network to learn more difficult images first . Next , we introduce the class of curricula we will consider for the remainder of the paper . 3 PUTTING CURRICULA THROUGH THEIR PACES . Many different approaches have been taken to implement curricula in machine learning . Here we focus on a particular widely used paradigm introduced in Bengio et al . ( 2009 ) and used in Hacohen & Weinshall ( 2019 ) . In this setup , a curriculum is defined by specifying three ingredients , • The scoring function : The scoring function is a map from an input example , x , to a numerical score , s ( x ) ∈ R. This score is typically intended to correspond to a notion of difficulty , where a higher score corresponds to a more difficult example . • The pacing function : The pacing function g ( t ) specifies the size of the training data-set used at each step , t. The training set at step t consists of the g ( t ) lowest scored examples . Training batches are then sampled uniformly from this set . • The order : Additionally we specify an order of either curriculum – ordering examples from lowest score to highest score , anti-curriculum – ordering examples from highest score to lowest , or random . Though technically redundant with redefining the scoring function , we maintain the convention that the score is ordered from easiest to hardest . This procedure is summarized in Algorithm 1 . It is worth emphasizing that due to the pacing function , using a random ordering in Algorithm 1 is not the same as traditional i.i.d . training on the full training dataset , but rather corresponds to i.i.d . training on a training dataset with dynamic size . We Algorithm 1 ( Random-/Anti- ) Curriculum learning with pacing and scoring functions 1 : Input : Initial weights w0 , training set { x1 , . . . , xN } , pacing function g : [ T ] → [ N ] , scoring function s : [ N ] → R , order o ∈ { “ ascending ” , “ descending ” , “ random ” } . 2 : ( x1 , . . . , xN ) ← sort ( { x1 , . . . , xN } , s , o ) 3 : for t = 1 , . . . , T do 4 : w ( t ) ← train-one-epoch ( w ( t−1 ) , { x1 , . . . , xg ( t ) } ) 5 : end for stress that the scoring and pacing function paradigm for curriculum learning is inherently limited . In this setup , the scoring function is computed before training over all of the data and thus the algorithm can not implement a self-paced and training-dependent curriculum as has been considered in Kumar et al . ( 2010 ) ; Jiang et al . ( 2015 ) ; Platanios et al . ( 2019 ) . Additionally , the dynamic training dataset is built by including all examples within a fixed score window ( from lowest score up in curricula and highest score down in anti-curricula ) and does not accommodate more flexible subsets . Furthermore , the form of curriculum discussed here only involves ordering examples from a fixed training dataset , rather than more drastic modifications of the training procedure , such as gradually increasing image resolution ( Vogelsang et al. , 2018 ) or the classes ( Weinshall et al. , 2018 ) . Nonetheless , it is commonly studied and serves as a useful framework and control study to empirically investigate the relative benefits of training orderings . Next , we describe scoring and pacing functions that will be used in our empirical investigation .

Summary: The paper provides a comprehensive analysis of the benefits of curriculum learning in different application scenarios. This includes investigating the phenomenon of implicit curricula, showing if the examples are learned in a consistent order across different architectures, and exploring the influences of explicit curricula in the standard and emulation settings. The paper empirically shows that curriculum learning has marginal benefits for standard training, but is helpful when the training time is limited or the training data is noisy.