Curriculum Discovery through an Encompassing Curriculum Learning Framework

We describe a curriculum learning framework capable of discovering optimal curricula in addition to performing standard curriculum learning . We show that this framework encompasses existing curriculum learning approaches such as difficulty-based data sub-sampling , data pruning , and loss re-weighting . We employ the proposed framework to address the following key questions in curriculum learning : ( a ) what is the best curriculum to train a given model on a given dataset ? and ( b ) what are the characteristics of optimal curricula for different datasets and difficulty metrics ? We show that our framework outperforms competing stateof-the-art curriculum learning approaches in natural language inference and two other text classification tasks . Exhaustive experiments illustrate the generalizability of the discovered curricula across the datasets and difficulty metrics . 1 INTRODUCTION . Curriculum Learning ( CL ) is a technique in Machine Learning that mimics human education systems . To learn a complex subject , students must first learn the foundational and basic materials before learning more complex ones . Without a curriculum , learning may be intractable , inefficient and learners may never reach a full understanding of a topic due to lack of the required background knowledge . Machine learning optimization through stochastic gradient descent trains models by observing example data instances . Some data instances are harder than others and require background knowledge , which could be acquired by observing and being adept at easier examples before harder ones . CL techniques seek to order examples according to their difficulty for training to generate better models . It has been shown that CL improves performance in solving harder tasks , or in cases of limited or noisy data ( Wu et al. , 2021 ) . CL research has made significant progress in the last decade through the work of Bengio et al . ( 2009 ) , although the principle of ordering training samples from easier to harder was introduced in the 1990s ( Elman , 1993 ; Sanger , 1994 ; Rohde & Plaut , 1999 ) . A curriculum can be defined by ordering training samples based on their difficulty to learn . Given a measure of difficulty , there are different types of CL approaches for ordering the data : sub-sampling techniques , which sample the easiest or hardest data points at every iteration for training ( Zhou et al. , 2020 ; Bengio et al. , 2009 ; Xu et al. , 2020 ; Guo et al. , 2018 ; Platanios et al. , 2019 ) , sample weighting techniques , which use the complete data at every iteration but weight data points differently according to their difficulty ( Castells et al. , 2020 ; Kumar et al. , 2010 ; Jiang et al. , 2015 ; 2018 ; Zhou et al. , 2020 ; Yang et al. , 2019 ) , and pruning techniques , which prune the hard or noisy samples from the dataset prior to training ( Northcutt et al. , 2021 ; Guo et al. , 2018 ) . Sub-sampling methods can be cumulative , exclusive or a combination of both . Cumulative approaches add new data points to the ones that have previously been used for training . On the other hand , exclusive approaches create a new subset of data at every training stage . Such methods can introduce samples from easy to hard ( Bengio et al. , 2009 ; Kumar et al. , 2010 ) or hard to easy , which can be an effective learning strategy in some specific tasks ( Kocmi & Bojar , 2017 ; Zhang et al. , 2018 ; 2019 ) . In addition , CL methods may impose a curriculum by adjusting model ’ s capacity according to the difficulty of their inputs ( Karras et al. , 2018 ; Sinha et al. , 2020 ; Morerio et al. , 2017 ) or schedule the order of tasks in the context of multi-task learning ( Caubrière et al. , 2019 ; Sarafianos et al. , 2017 ; Florensa et al. , 2017 ) . Other CL approaches such as ( Zhou et al. , 2020 ; Saxena et al. , 2019 ) use O ( n ) , where n is the number of training samples , extra parameters for learning curricula . Current CL approaches calculate difficulty scores for training samples based on the loss of a trained model ( Xu et al. , 2020 ; Wu et al. , 2021 ) , trainable parameters that weight samples ( Kumar et al. , 2010 ; Jiang et al. , 2015 ; Castells et al. , 2020 ) , loss value during training ( Wu et al. , 2021 ) , moving average of loss during training ( Zhou et al. , 2020 ) , transformations of the loss during training ( Jiang et al. , 2018 ; Castells et al. , 2020 ) , and consistency in the correct classification of samples ( Amiri et al. , 2017 ; Xu et al. , 2020 ) . The difficulty scores will then be used to order samples for training . In this work , we focus on an alternative approach , where examples can be ordered based on prior knowledge about their difficulty , e.g. , object shape or orientation in image classification ( Bengio et al. , 2009 ) . We will demonstrate a CL framework that encompasses existing CL approaches through an effective and flexible data partitioning and weighting scheme . Our framework provides a new paradigm for selecting an ordering strategy . Instead of a pre-determined strategy , the framework allows searching over the curriculum space to identify the best curriculum for a particular dataset and model . It partitions training data into several groups of training samples , e.g . { easy , medium , hard } samples , according to a difficulty scoring function . Parameterized weighting functions will then be defined for each data group to specify the weight of its samples during training . Each weight function is controlled by two parameters , which can be set empirically or adjusted using Bayesian optimization . In addition , the framework discovers optimal curricula by optimizing the parameters of the weight functions ( only 2 parameters per function ) using a hyper-parameter optimization algorithm . Furthermore , the curricula identified through this search provide useful insight about the dataset , such as the relative importance of different samples or knowledge dependency between samples , e.g . which samples should be learned first . We begin by explaining our framework and showing how it is capable of approximating existing CL approaches . Then , in the context of the proposed framework , we investigate curriculum discovery , characteristics of discovered curricula , and generalizability of curricula with respect to their datasets . 2 CURRICULUM LEARNING FRAMEWORK . 2.1 WEIGHTING FUNCTIONS . We define the curriculum using generalized logistic functions ( Richards , 1959 ) of the form : w ( t ; r , s ) = 1 1 + exp ( −r ∗ ( t− s ) ) , ( 1 ) where r ∈ R is the rate-of-change parameter , which specifies how fast the weight can increase ( r > 0 ) or decrease ( r < 0 ) ; t ∈ [ 0 , 1 ] is the training progress ( iteration number divided by max iterations ) ; and s ∈ R shifts the pivot weight of the logistics function ( w ( . ) = .5 ) to the left or right such that at t = s the weight is 0.5 . Figure 1a illustrates the effect of varying these parameters . Greater absolute values for the rate parameter enforce greater slope and faster rate of changes in weights , while greater values of the shift parameter enforce longer delays in reaching the pivot weight of 0.5 . These parameters provide flexibility in controlling sample weights during training , which is critical for deriving effective curricula . Given the weighting function , we encode prior knowledge about sample difficulty into the training paradigm of our CL framework by splitting the data into several partitions of increasing difficulty according to the prior knowledge . Then , we define a weight function by a pair of parameters ( r , s ) for each partition , The proposed sample weighting framework provides flexibility in sample-ordering according to difficulty . It can be learned to approximate existing predetermind curricula . For example , it is possible to begin with only easy instances or only difficult instances ( as an anti-curriculum ) or a combination of both . Figure 1b shows a specific configuration for the logistic functions based on standard CL ( Bengio et al. , 2009 ; Kumar et al. , 2010 ) , where training starts with easier samples and gradually proceeds with harder ones . In addition , our approach enables discovering new data-driven curricula from data . Prior to training , the difficulty scores of samples are computed and each sample is assigned to a difficulty class c ∈ { easy , medium , hard } , see §2.2 . In addition , the hyper-parameters of our three weight functions are optimized prior to training and kept fixed throughout training { ( re , se ) , ( rm , sm ) , ( rh , sh ) } . During training , the weighted loss is computed as follows . l̂i = w ( t ; rc , sc ) ∗ li ( 2 ) Where li is the unweighted and instantaneous loss of instance i , l̂i is the weighted loss , t is the current training iteration divided by the maximum number of iterations , c is the difficulty class of instance i , and ( rc , sc ) are the corresponding rate and shift parameters for the difficulty class c . 2.2 SCORING FUNCTIONS . Ground-truth labels for many datasets are often obtained through human annotation and crowdsourcing . This is achieved by collecting multiple annotations per data sample and aggregating the results , typically by majority voting . We use sample-level annotator disagreement to define a difficulty score for each sample using Shannon entropy ( Shannon , 2001 ) , where higher disagreement among annotators corresponds to higher sample difficulty . Entropy is a natural measure of difficulty ( for human population ) and may serve as a reliable prior knowledge for partitioning data by difficulty . Entropy of each sample xi is calculated as H ( xi ) = − ∑ l pc log pc where c is a class category and pc is the fraction of annotators who chose label c for the sample . The use of entropy is supported by Nie et al . ( 2020 ) , who studied the correlation between human agreement and model performance and reported a consistent positive correlation between model accuracy and level of human agreement , showing that model performs better on samples with a high level of agreement . Furthermore , training loss contains valuable information about difficulty with respect to the model ( learner ) , which may be different among architectures and tasks , and indicative of the model ’ s specific needs . However , loss at a particular step ( e.g. , final loss ) is dictated by the stochastic gradient descent and mini-batching dynamics and therefore is not a good indicator of difficulty ( Zhou et al. , 2020 ; Wu et al. , 2021 ) . Using a baseline model , trained with no curriculum and with default hyperparameters , we collect the loss values of all training instances at intervals of 0.5 epochs and use the average loss to estimate sample difficulty . In our experiments , we obtain twenty observations of the loss and compute the average for each instance . Such an estimation is supported by Zhou et al . ( 2020 ) who showed that the moving average of a sample ’ s instantaneous loss is a good metric for difficulty . Partitioning the data into three groups of increasing difficulty can be done using difficulty score percentiles , or 1-dimensional k-means clustering of the scores . Examples of data partitions using entropy and loss are shown in Figure 2 .

Summary: The paper proposes to learn training curricula by considering 3 sigmoids (representing "easy", "mid" and "hard" difficulty levels), that would define instances' weight as a function of time. The sigmoids' parameters are fitted to 3-bucket values of entropy of multi-annotator labels (that need to pre-exist), or of the instances' loss value dynamics (as provided by a baseline, no-curriculum, model that also needs to be trained). The paper emphasizes that the approach can replicate some of existing curriculum heuristics (e.g. easy-to-hard, hard-to-easy etc.), which however so far failed to produce a general recipe of training deep models, -- this diminishes the value of this flexibility. Finally, the approach is illustrated on 3 small to mid-size datasets with largest gaps to no-curriculum achieved on reduced datasets to re-balance classes.