A Simple Approach To Define Curricula For Training Neural Networks

1 INTRODUCTION . Stochastic Gradient Descent ( SGD ) ( Robbins & Monro , 1951 ) is a simple yet widely used algorithm for machine learning optimization . There have been many efforts to improve its performance . A number of such directions , such as AdaGrad ( Duchi et al. , 2011 ) , RMSProp ( Tieleman & Hinton , 2012 ) , and Adam ( Kingma & Ba , 2014 ) , improve upon SGD by fine-tuning its learning rate , often adaptively . However , Wilson et al . ( 2017 ) has shown that the solutions found by adaptive methods generalize worse even for simple overparameterized problems . Reddi et al . ( 2019 ) introduced AMSGrad hoping to solve this issue . Yet there is performance gap between AMSGrad and SGD in terms of the ability to generalize ( Keskar & Socher , 2017 ) . Further , Choi et al . ( 2019 ) shows that more general optimizers such as Adam and RMSProp can never underperform SGD when all their hyperparameters are carefully tuned . Hence , SGD still remains one of the main workhorses of the ML optimization toolkit . SGD proceeds by stochastically making unbiased estimates of the gradient on the full data ( Zhao & Zhang , 2015 ) . However , this approach does not match the way humans typically learn various tasks . We learn a concept faster if we are presented the easy examples first and then gradually exposed to examples with more complexity , based on a curriculum . An orthogonal extension to SGD ( Weinshall & Cohen , 2018 ) , that has some promise in improving its performance is to choose examples according to a specific strategy , driven by cognitive science – this is curriculum learning ( CL ) ( Bengio et al. , 2009 ) , wherein the examples are shown to the learner based on a curriculum . 1.1 RELATED WORKS . Bengio et al . ( 2009 ) formalizes the idea of CL in machine learning framework where the examples are fed to the learner in an order based on its difficulty . The notation of difficulty of examples has not really been formalized and various heuristics have been tried out : Bengio et al . ( 2009 ) uses manually crafted scores , self-paced learning ( SPL ) ( Kumar et al. , 2010 ) uses the loss values with respect to the learner ’ s current parameters , and CL by transfer learning uses the loss values with respect to a pre-trained learner to rate the difficulty of examples in data . Among these works , what makes SPL particular is that they use a dynamic CL strategy , i.e . the preferred ordering is determined dynamically over the course of the optimization . However , SPL does not really improve the performance of deep learning models , as noted in ( Fan et al. , 2018 ) . Similarly , Loshchilov & Hutter ( 2015 ) uses a function of rank based on latest loss values for online batch selection for faster training of neural networks . Katharopoulos & Fleuret ( 2018 ) and Chang et al . ( 2017 ) perform importance sampling to reduce the variance of stochastic gradients during training . Graves et al . ( 2017 ) and Matiisen et al . ( 2020 ) propose teacher-guided automatic CL algorithms that employ various supervised measures to define dynamic curricula . The most recent works in CL show its advantages in reinforcement learning ( Portelas et al. , 2020 ; Zhang et al. , 2020 ) . The recent work by Weinshall & Cohen ( 2018 ) introduces the notion of ideal difficult score to rate the difficulty of examples based on the loss values with respect to the set of optimal hypotheses . They theoretically show that for linear regression , the expected rate of convergence at a training step t for an example monotonically decreases with its ideal difficulty score . This is practically validated by Hacohen & Weinshall ( 2019 ) by sorting the training examples based on the performance of a network trained through transfer learning . However , there is a lack of theory to show that CL improves the performance of a completely trained network . Thus , while CL indicates that it is possible to improve the performance of SGD by a judicious ordering , both the theoretical insights as well as concrete empirical guidelines to create this ordering remain unclear . While the previous CL works employ tedious methods to score the difficulty level of the examples , Hu et al . ( 2020 ) uses the number of audio sources to determine the difficulty for audiovisual learning . Liu et al . ( 2020 ) uses the norm of word embeddings as a difficulty measure for CL for neural machine translation . In light of these recent works , we discuss the idea of using task-specific statistical ( unsupervised ) measures to score examples making it easy to perform CL on real image datasets without the aid of any pre-trained network . 1.2 OUR CONTRIBUTIONS . Our work proposes two novel algorithms for CL . We do a thorough empirical study of our algorithms and provide some more insights into why CL works . Our contributions are as follows : • We propose a novel dynamic curriculum learning ( DCL ) algorithm to study the behaviour of CL . DCL is not a practical CL algorithm since it requires the knowledge of a reasonable local optima as needs to compute the gradients of full data after ever training epoch . DCL uses the gradient information to define a curriculum that minimizes the distance between the current weight and a desired local minima . However , this simplicity in the definition of DCL makes it easier to analyze its performance formally . • Our DCL algorithm generates a natural ordering for training the examples . Previous CL works have demonstrated that exposing a part of the data initially and then gradually exposing the rest is a standard way to setup a curriculum . We use two variants of our DCL framework to show that it is not just the subset of data which is exposed to the model that matters , but also the ordering within the data partition that is exposed . We also analyze how DCL is able to serve as a regularizer and improve the generalization of networks . • We contribute a simple , novel and practical CL approach for image classification tasks that does the ordering of examples in a completely unsupervised manner using statistical measures . Our insight is that statistical measures could have an association with the difficulty of examples in real data . We empirically analyze our argument of using statistical scoring measures ( especially standard deviation ) over permutations of multiple datasets and networks . Additionally , we study why CL based on standard deviation scoring works using our DCL framework . Algorithm 1 Approximate greedy dynamic curriculum learning ( DCL+ ) . Input : Data X , local minima w̃ , weight wt , batch size b , and pacing function pace . Output : Sequence of mini-batches Bt for the next training epoch . 1 : ãt ← w̃ −wt 2 : ρt ← [ ] 3 : Bt ← [ ] 4 : for ( i = 0 ; N ; 1 ) do 5 : append − ã T t · ∇fi ( wt ) ‖ãt‖2 to ρt 6 : end for 7 : X̃ ← X sorted according to ρt , in ascending order 8 : size← pace ( t ) 9 : for ( i = 0 ; size ; b ) do 10 : append X̃ [ i , ... , i+ b− 1 ] to Bt 11 : end for 12 : return Bt 2 PRELIMINARIES . At any training step t , SGD updates the weight wt using ∇fi ( wt ) which is the gradient of loss of example xi with respect to the current weight . The learning rate and the data are denoted by η and X = { ( xi , yi ) } N−1i=0 respectively , where xi ∈ R d denotes a data point and yi ∈ [ K ] its corresponding label for a dataset with K classes . We denote the learner as hϑ : Rd → [ K ] . Generally , SGD is used to train hϑ by giving the model a sequence of mini-batches { B0 , B1 , ... , BT−1 } , where Bi ⊆ X ∀i ∈ [ T ] . Each Bi is generated by uniformly sampling examples from the data . We denote this approach as vanilla . In CL , the curriculum is defined by two functions , namely the scoring function and the pacing function . The scoring function , scoreϑ ( xi , yi ) : Rd× [ K ] → R , scores each example in the dataset . Scoring function is used to sort X in an ascending order of difficulty . A data point ( xi , yi ) is said to be easier than ( xj , yj ) if scoreϑ ( xi , yi ) < scoreϑ ( xj , yj ) , where both the examples belong to X . Unsupervised scoring measures do not use the data labels to determine the difficulty of data points . The pacing function , paceϑ ( t ) : [ T ] → [ N ] , determines how much of the data is to be exposed at a training step t ∈ [ T ] . We define speedup for CL model as its improvement over vanilla model ( in terms of the number of training steps ) to achieve a given test accuracy . For example , CL has 2× speedup if vanilla model achieves 90 % test accuracy in 100 training steps while CL achieves the same 90 % test accuracy in 50 training steps . 3 DYNAMIC CURRICULUM LEARNING . For DCL algorithms ( Kumar et al. , 2010 ; Graves et al. , 2017 ; Matiisen et al. , 2020 ) , examples are scored and sorted after every few training steps since the parameters of the scoring function change dynamically with the learner as training proceeds . Hacohen & Weinshall ( 2019 ) and Bengio et al . ( 2009 ) use a fixed scoring function and pace function for the entire training process . They empirically show that a curriculum helps to learn fast in the initial phase of the training process . In this section , we propose and analyze our novel DCL algorithm that updates the difficulty scores of all the examples in the training data at every epoch using their gradient information . We hypothesize the following : Given a weight initialization and a local minima obtained by full training of vanilla SGD , the curriculum ordering determined by our DCL variant leads to speedup in training . We first describe the algorithm , then the underlying intuition , and finally validate the hypothesis using experiments . Our DCL algorithm iteratively works on reducing the L2 distance , Rt , between the weight parameter wt and a given optimal weight w̄ at any training step t. Suppose , for any t̃ < t , St̃ , t is the ordered set containing the ( t− t̃+1 ) indices of training examples that are to be shown to the learner from the training steps t̃ through t. Let us define at = ( w̄ −wt ) , Rt = ‖at‖2 , and θt̃i as the angle between ∇fi ( wt ) and at̃ . Then , using a geometrical argument , ( see Figure 1 ) , R2t = ( Rt̃ − η j=t−1∑ j=t̃ , i∈St̃ , t−1 ( ‖∇fi ( wj ) ‖2 cos θt̃i ) ) 2 + η2 ( j=t−1∑ j=t̃ , i∈St̃ , t−1 ( ‖∇fi ( wj ) ‖2 sin θt̃i ) ) 2 = R2t̃ − 2ηRt̃ j=t−1∑ j=t̃ , i∈St̃ , t−1 ( ‖∇fi ( wj ) ‖2 cos θt̃i ) + η2 ( j=t−1∑ j=t̃ , i∈St̃ , t−1 ( ‖∇fi ( wj ) ‖2 cos θt̃i ) ) 2 + η2 ( j=t−1∑ j=t̃ , i∈St̃ , t−1 ( ‖∇fi ( wj ) ‖2 sin θt̃i ) ) 2 ( 1 ) For a vanilla model , S0 , T is generated by uniformly sampling indices from [ N ] with replacement . Since , finding a set S0 , T to minimize R2T and an optimal w̄ are intractable for nonconvex optimization problems , we approximate the DCL algorithm ( DCL+ , see Algorithm 1 ) . We approximate w̄ with w̃ , which is a local minima obtained from training the vanilla SGD model . Also , to reduce computational expense while sampling examples , we neglect the terms with coefficient η2 in equation 1 while designing our algorithm . Algorithm 1 uses a greedy approach to minimize R2t by sampling examples at every epoch using the scoring function scoret ( xi ) = −‖∇fi ( wt ) ‖2 cos θti = − aTt · ∇fi ( wt ) ‖at‖2 = ρt , i . ( 2 ) Let us denote the models that use the natural ordering of mini-batches greedily generated by Algorithm 1 for training networks as DCL+ . DCL- uses the same sequence of mini-batches that DCL+ exposes to the network at any given epoch , but the order is reversed . We empirically show that DCL+ achieves a faster and better convergence with various initializations of w0 . We use learning rates with an exponential step-decay rate for the optimizers in all our experiments as traditionally 0 1000 2000 3000 4000 5000 Training step 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 To p- 1 te st a cc ur ac y k = 0.2 k = 0.4 k = 0.6 k = 0.8 k = 1.0 Figure 3 : Learning curves for experiment 2 with varying pace ( t ) = bkNc for DCL+ . The parameter k needs to be finely tuned for improving the generalization of the network . A low k value exposes only examples with less noise to the network at every epoch whereas a high k value exposes most of the dataset including highly noisy examples to the network . A moderate k value shows less noisy examples along with some examples with moderate level of noise to the learner . Here , a moderate k = 0.6 generalizes the best . done ( Simonyan & Zisserman , 2014 ; Szegedy et al. , 2016 ) . For a fair comparison , we tune the learning rates and decay rates of the models . Experimental setup : In our experiments , we set pace ( t ) = bkNc ∀t , where k ∈ [ b/N , 1 ] is a tunable hyper-parameter . We use a 2-layer fully-connected network ( FCN ) with 10 hidden neurons and Exponential Linear Unit ( ELU ) nonlinearities to empirically validate our algorithms ( k = 0.9 ) on a subset of the MNIST dataset with class labels 0 and 1 ( Experiment 1 ) . Since , this is a very easy task ( as the vanilla model accuracy is as high as ∼ 99.9 % ) , we compare the test loss values across training steps in Figure 2a to see the behaviour of DCL on an easy task . DCL+ shows the fastest convergence , although all the networks achieve the same test accuracy . DCL+ achieves vanilla ’ s final test loss score at training step 682 ( ∼ 30 % speedup ) . In Experiment 2 , we use a 2-layered FCN with 128 hidden neurons and ELU nonlinearities to evaluate our DCL algorithms ( k = 0.6 ) on a relatively difficult small mammals dataset ( Krizhevsky et al. , 2009 ) , a super-class of CIFAR-100 . Figure 2b shows that DCL+ achieves a faster and better convergence than vanilla with respect to the test set accuracy in experiment 2 . DCL+ achieves vanilla ’ s convergence test accuracy score at training step 1896 ( ∼ 60 % speedup ) . Further experimental details are deferred to Appendix B.1 . Since , DCL is computationally expensive , we perform DCL experiments only on small datasets . Fine-tuning of k is crucial for improving the performance of DCL+ on the test set ( see Figure 3 ) . We fine-tune k by trial-and-error over the test accuracy score .

Summary: This work studies a number of curriculums for faster training of neural networks. They first propose a curriculum named DCL+ that is designed to order data points based on their alignment of gradient with the direction of optimization. This curriculum depends on the evaluation of individual gradients of datapoints as well as an approximation to a local optima. Next, they study a number of easy-to-compute statistical measures for ordering data points.