Rethinking Curriculum Learning With Incremental Labels And Adaptive Compensation

1 INTRODUCTION . Deep networks have seen rich applications in high-dimensional problems characterized by a large number of labels and a high volume of samples . However , successfully training deep networks to solve problems under such conditions is mystifyingly hard ( Erhan et al . ( 2009 ) ; Larochelle et al . ( 2007 ) ) . The go-to solution in most cases is Stochastic Gradient Descent with mini-batches ( simple batch learning ) and its derivatives . While offering a standardized solution , simple batch learning often fails to find solutions that are simultaneously stable , highly generalizable and scalable to large systems ( Das et al . ( 2016 ) ; Keskar et al . ( 2016 ) ; Goyal et al . ( 2017 ) ; You et al . ( 2017 ) ) . This is a by-product of how mini-batches are constructed . For example , the uniform prior assumption over datasets emphasizes equal contributions from each data point regardless of the underlying distribution ; small batch sizes help achieve more generalizable solutions , but do not scale as well to vast computational resources as large mini-batches . It is hard to construct a solution that is a perfect compromise between all cases . Two lines of work , curriculum learning and label smoothing , offer alternative strategies to improve learning in deep networks . Curriculum learning , inspired by strategies used for humans ( Skinner ( 1958 ) ; Avrahami et al . ( 1997 ) ) , works by gradually increasing the conceptual difficulty of samples used to train deep networks ( Bengio et al . ( 2009 ) ; Florensa et al . ( 2017 ) ; Graves et al . ( 2017 ) ) . This has been shown to improve performance on corrupted ( Jiang et al . ( 2017 ) ) and small datasets ( Fan et al . ( 2018 ) ) . More recently , deep networks have been used to categorize samples ( Weinshall et al . ( 2018 ) ) and variations on the pace with which these samples were shown to deep networks were analyzed in-depth ( Hacohen & Weinshall ( 2019 ) ) . To the best of our knowledge , previous works assumed that samples cover a broad spectrum of difficulty and hence need to be categorized and presented in a specific order . This introduces computational overheads e.g . pre-computing the relative difficulty of samples , and also reduces the effective amount of data from which a model can learn in early epochs . Further , curriculum learning approaches have not been shown to compete with simple training strategies at the top end of performance in image benchmarks . A complementary approach to obtaining generalizable solutions is to avoid over-fitting or getting stuck in local minima . In this regard , label smoothing offers an important solution that is invariant to the underlying architecture . Early works like Xie et al . ( 2016 ) replace ground-truth labels with noise while Reed et al . ( 2014 ) uses other models ’ outputs to prevent over-fitting . This idea was extended in Bagherinezhad et al . ( 2018 ) to an iterative method which uses logits obtained from previously trained versions of the same deep network . While Miyato et al . ( 2015 ) use local distributional smoothness , based on the robustness of a model ’ s distribution around a data point , to regularize outcomes , Pereyra et al . ( 2017 ) penalized highly confident outputs directly . Closest in spirit to our work is the label smoothing method defined in Szegedy et al . ( 2016 ) , which offers an alternative target distribution for all training samples with no extra data augmentation . In general , label smoothing is applied to all examples regardless of how it affects the network ’ s understanding of them . Further , in methods which use other models to provide logits/labels , often the parent network used to provide those labels is trained using an alternate objective function or needs to be fully re-trained on the current dataset , both of which introduce additional computation . In this work , we propose LILAC , Learning with Incremental Labels and Adaptive Compensation , which emphasizes a label-based curriculum and adaptive compensation , to improve upon previous methods and obtain highly accurate and stable solutions . LILAC is conceived as a method to learn strong embeddings by using the recursive training strategy of incremental learning alongside the use of unlabelled/wrongly-labelled data as hard negative examples . It works in two key phases , 1 ) incremental label introduction and 2 ) adaptive compensation . In the first phase , we incrementally introduce groups of labels in the training process . Data , corresponding to labels not yet introduced to the model , use a single fake label selected from within the dataset . Once a network has been trained for a fixed number of epochs with this setup , an additional set of ground-truth labels is introduced to the network and the training process continues . In recursively revealing labels , LILAC allows the model sufficient time to develop a strong understanding of each class by contrasting against a large and diverse set of negative examples . Once all ground-truth labels are revealed the adaptive compensation phase of training is initiated . This phase mirrors conventional batch learning , except we adaptively replace the target one-hot vector of incorrectly classified samples with a softer distribution . Thus , we avoid adjusting labels across the entire dataset , like previous methods , while elevating the stability and average performance of the model . Further , instead of being pre-computed by an alternative model , these softer distributions are generated on-the-fly from the outputs of the model being trained . We apply LILAC to three standard image benchmarks and compare its performance to the strongest known baselines . While incremental and continual learning work on evolving data distributions with the addition of memory constraints ( ( Rebuffi et al. , 2017 ; Castro et al. , 2018 ) and derivative works ) , knowledge distillation ( ( Schwarz et al. , 2018 ; Rolnick et al. , 2018 ) and similar works ) or other requirements , this work is a departure into using negative mining and focused training to improve learning on a fully available dataset . In incremental/continual learning works , often the amount of data used to retrain the network is small compared to the original dataset while in LILAC we fully use the entire dataset , distinguished by Seen and Unseen labels . Thus , it avoids data deficient learning . Further , works like Bucher et al . ( 2016 ) ; Li et al . ( 2013 ) ; Wang & Gupta ( 2015 ) emphasize the importance of hard negative mining , both in size and diversity , in improving learning . Although the original formulation of negative mining was based on imbalanced data , recent object detection works have highlighted its importance in contrasting and improving learning in neural networks . To summarize , our main contributions in LILAC are as follows , • we introduce a new take on curriculum learning by incrementally learning labels as opposed to samples , • our method adaptively compensates incorrectly labelled samples by softening their target distribution which improves performance and removes external computational overheads , • we improve average recognition accuracy and decrease the standard deviation of perfor- mance across several image classification benchmarks compared to batch learning , a property not shared by other curriculum learning and label smoothing methods . 2 LILAC . In LILAC , our main focus is to induce better learning in deep networks . Instead of the conventional curriculum learning approach of ranking samples , we consider all samples equally beneficial . Early on , we focus on learning labels in fixed increments ( Section 2.1 ) . Once the network has had a chance to learn all the labels , we shift to regularizing the network to prevent over-fitting by providing a softer distribution as the target vector for previously misclassified samples ( Section 2.2 ) . An overview of the entire algorithm discussed is available in the appendix as Algorithm 1 . 2.1 INCREMENTAL LABEL INTRODUCTION PHASE . In the incremental phase , we initially replace the ground-truth labels of several class using a constant held-out label . Gradually , over the course of several fixed intervals of training we reveal the true label . Within a fixed interval of training , we keep constant two sets of data , ” Seen ” , whose groundtruth labels are known and ” Unseen ” , whose labels are replaced by a fake value . When training , Full Dataset ( Virtual Data Partition in LILAC ( : Seen : Unseen Incremental Step 1 Incremental Step 2 ( b ) Evolution of data partition over each incremental step , ( a ) Data setup in LILAC Uses Ground-truth labels Uses a fake label Incremental Step 3 ( Final Incremental Step ) = 4 ) = 1 ) ( ) ( ) − = 1 mini-batches are uniformly sampled from the entire training set , but the instances from ” Unseen ” classes use the held-out label . By the end of the final interval , we reveal all ground-truth labels . We now describe the incremental phase in more detail . At the beginning of the incremental label introduction phase , we virtually partition data into two mutually exclusive sets , S : Seen and U : Unseen , as shown in Fig . 1 . Data samples in S use their ground-truth labels as target values while those in U use a designated unseen label , which is held constant throughout the entire training process . LILAC assumes a random ordering of labels , Or ( M ) , where M denotes the total number of labels in the dataset . Within this ordering , the number of labels and corresponding data initially placed in S is defined by the variable b . The remaining labels , M − b , are initially placed in U and incrementally revealed in intervals of m labels , a hyper-parameter defined by the user . Training in the incremental phase happens at fixed intervals of E epochs each . Within a fixed interval , the virtual data partition is held constant . Every mini-batch of data is sampled uniformly from the entire original dataset and within each mini-batch , labels are obtained based on their placement in S or U . Then the number of samples from U is reduced or augmented , using a uniform prior , to match the number of samples from S. This is done to ensure no unfair skew in predictions towards U since all data points use the same designated label . Finally , the curated mini-batches of data are used to train the neural network . At the end of each fixed interval , we reveal another set of m groundtruth labels and move samples of those classes from U to S after which the entire data curation and training process is repeated for the next interval . 2.2 ADAPTIVE COMPENSATION . Once all the ground-truth labels are available to the deep network , we begin the adaptive compensation phase of training . The main idea behind adaptive compensation is , if the network is unable to correctly predict a sample ’ s label even after allowing sufficient training time , then we alter the target vector to a less peaked distribution . Compared to learning one-hot vectors , this softer distribution can more easily be learned by the network . Unlike prior methods we adaptively modify the target vector only for incorrectly classified samples on-the-fly . In this phase , the network is trained for a small number of epochs using standard batch learning . Let T be the total number of training epochs in the incremental phase and batch learning . During the adaptive compensation phase , we start at epoch e , where e > T . For a mini-batch of samples in epoch e , predictions from the model at e− 1 are used to determine the final target vector used in the objective function ; specifically , we soften the target vector for an instance iff it was misclassified by the model at the end of epoch e − 1 . The final target vector for the ith instance at epoch e , te , i , is computed based on the model φe−1 using Equation 1. te , i = { ( M−1M−1 ) δyi + ( 1− M−1 ) 1 , argmax ( φe−1 ( xi ) ) 6= y i δyi , otherwise . ( 1 ) Here , ( xi , yi ) denote a training sample and its corresponding ground-truth label for sample index i while δyi represents the corresponding one-hot vector . 1 is a vector ofM dimensions with all entries as 1 and is a scaling hyper-parameter .

Summary: This paper makes the observation that a curriculum need not depend on the difficulty of examples, as most (maybe all) prior works do. They suggest instead a curriculum based on learning one class at a time, starting with one and masking the label of all others as 'unknown' (i.e. treating them as negative examples), and unmasking classes as learning progresses. This is the "incremental labels" part. They make another observation, that label smoothing is applied to all examples regardless of difficulty, and propose an alternative "adaptive" version where labels are smoothed only for difficult examples. This is the "adaptive compensation" part.