The Negative Pretraining Effect in Sequential Deep Learning and Three Ways to Fix It

1 INTRODUCTION . Lifelong learning ( Thrun & Mitchell , 1995 ; Silver et al. , 2013 ) holds both the promise of benefitting from past experiences and the challenge of having to continually adapt to new problem settings . The characteristics of this intriguing sequential learning problem make it a balancing act between retaining knowledge and adapting to new experiences referred to as the stability-plasticity dilemma ( Mermillod et al. , 2013 ) . Similar to a coach or teacher , we want to understand when a sequence of tasks helps or hinders further learning and what the underlying mechanism are that influence this process given that the cost of training very large models from scratch such as GPT-3 ( Brown et al. , 2020 ) has increased considerably ( Sharir et al. , 2020 ) . This setting of training a model on a sequence of tasks to maximize its performance on a target task we coin the sequential learning problem . The existing literature on sequential learning sheds some light on the characteristics of this problem setting . In some cases , following a sequence of learning tasks can lead to better results than simply training on the target task from scratch ( Bengio et al. , 2009 ) . Achille and Soatto ( Achille et al. , 2017 ) and Wang et al . ( 2019 ) contrast this picture by highlighting that neural networks can suffer from pretraining on some tasks . When learning data is corrupted for a sufficiently long period , even switching back to clean data does not allow the model to recover its original performance ( Maennel et al. , 2020 ) . Until compared , it may unforuntely be unclear that a model is corrupted . Contribution : We formalize the basic ingredients of sequential learning as following a path on a learning manifold as depicted in Fig . 1 and provide three ways to remove the negative pretraining effect that can occur for certain task changes in neural network training as depicted in Fig . 2 . We establish experimentally for a variety of task changes and datasets that this final performance can be highly dependent on the way the chosen model is trained , the sequence of training tasks , and how the model is perturbed during the training process . First , we demonstrate that increasing learning rates after pretraining can remove negative pretraining effects . Secondly , we show that continuously changing from source to target task instead of “ jumping ” can reap significant benefits in performance and does not incur a deficit penalty . Thirdly , we display that resetting only the biases for each task in the training process appears to decrease negative pretraining effects . Finally we speculate on plausible reasons such as qualitatively different training behavior during the “ high learning rate phase ” ( Lewkowycz et al. , 2020 ) that may explain the observed effects . 2 RELATED WORK . Sequential learning involves choosing a sequence of tasks and learning processes and following this learning path to improve performance on a single or set of target tasks . This strongly resembles the objectives of lifelong learning ( Thrun & Mitchell , 1995 ) , continual learning ( Lesort et al. , 2019 ) and meta-learning ( Thrun & Pratt , 2012 ; Schaul & Schmidhuber , 2010 ; Hochreiter et al. , 2001 ) and connects to many time-dependent learning aspects in relation to neural networks . Time-dependent learning : We are aware of several time-dependent phenomena in training neural networks such as catastrophic forgetting ( French , 1999 ) , critical learning periods ( Achille et al. , 2017 ) , and time sensitivity of regularization and data augmentation ( Golatkar et al. , 2019 ) . Achille et al . ( 2017 ) show that by dropping out certain frequencies during the beginning of training , networks are unable to recover to full performance , even when those frequencies are reintroduced . Similarly , Liu et al . ( 2019 ) show that first optimizing on random labels ruins performance on clean data shown indefinitely to the same network afterwards . Recent information-theoretic analysis ( Shwartz-Ziv & Tishby , 2017 ) suggests that neural network have distinct learning phases : First , a phase where network parameters grow in information content , before , in a second phase , self-regularizing and pruning away irrelevant information . More commonly known time-dependent or sequential heuristics are learning rate schedules ( Darken & Moody , 1991 ) , cyclical learning rates ( Smith , 2017 ) , and regularization annealing such as dropout annealing ( Rennie et al. , 2014 ) , student-teacher model transfer ( Vicente & Caticha , 1997 ) , and pretraining ( Erhan et al. , 2009 ) . Curriculum learning and generation : Curriculum learning ( Elman , 1993 ; Bengio et al. , 2009 ) is the technique of training agents by showing them a progression of tasks from easier variants to harder examples to attempt to solve the final target task . Both the utility of transferring between similar or dissimilar tasks is debated , with some results pointing towards similarity ( Achille et al. , 2019 ; Ammar et al. , 2014 ; Ritter et al. , 2018 ) and others towards dissimilarity ( Farquhar & Gal , 2018 ; Nguyen et al. , 2019 ) . Self-paced learning ( Jiang et al. , 2015 ) extends curriculum learning by letting the model itself influence the sequence of tasks ( Graves et al. , 2017 ) . The order of tasks , the curriculum , is generally an input that is hand-designed by human experimenters and often based on heuristics to guide the scheduling ( i.e . it may be easier to make an agent jump over small gaps before moving to larger ones ( Heess et al. , 2017 ) ) . Meta-learning ( Schaul & Schmidhuber , 2010 ; Vilalta & Drissi , 2002 ) is similarly related by focusing on speeding up or enhancing performance on a target task by pretraining on other tasks to first learn to learn . Biological analogies : It is through neuroplasticity ( Dayan & Cohen , 2011 ) that animals and humans learn new skills and recover from changes or damage to their sensory organs or brain . Famously , Erismann and Kohler ( Kohler & Pissarek , 1960 ) asked study participants to wear special mirror goggles which flipped the participants ’ visual field upside down as summarized in ( Sachse et al. , 2017 ) . Remarkably , the participants visual perception adjusted after a few days by returning the visual field back to their “ upright ” standard perception . In contrast to this , there are other stimuli people learn to discard in childhood if they are not needed . Young children can differentiate phonemes of both their own and foreign languages ; however , older children are better at distinguishing native phonemes and have more difficulty distinguishing foreign phonemes ( Cheour et al. , 1998 ; Kuhl et al. , 1992 ) . A similar effect of “ perceptual narrowing ” is found for 6-month-old and 9-month-old children when comparing the ability to discriminate between human faces to the ability to discriminate between primate faces ( Pascalis et al. , 2002 ) . Younger children are again found to be less specialized to human faces than slightly older ones . Summarizing many results ( Costandi , 2016 ) , we currently know that neuroplasticity is “ largest ” at an early age yet generally does not end in adulthood . It is instead always present in reorganization processes involved in new experiences and when adjusting to injury . 3 SEQUENTIAL LEARNING SETTING . We are interested in better understanding what influences the outcome of a sequential learning problem . In the following , we say that a pretrained model has plasticity if it can generalize similarly well when trained on a target task as an uncorrupted model trained directly on the target task . If it can not generlize similarly well , we refer to this as a negative pretraining effect . To facilitate further reading , we summarize essential elements such as the definition of a learning task ⌧ , constrained to supervised tasks in this work , the learning mapping ! , the learning path as an ordered sequence of task-learning process pairs T , and the setup of its repeated application in a sequential learning process ⌦ in Tab . 1 and visualize the overall setting in Fig . 1 . A task ⌧ consists of a probability distribution p ( x , y ) of data points and desired outputs as well as a loss function L that maps an input-output behavior of a model to a cost score . The learning mapping ! uses the information in task ⌧ and initial model finit to train a model optimized for the task flearn until convergence . Applying the learning mappings and tasks ( ⌧i , ! i ) 2 T sequentially starting from an initial model finit is summarized via the symbol ⌦ . Based on these elements , we ask what influences the final generalization outcome on a target task when following a learning path of task-learn process pairs T . Specifically , we examine these question via three interventions aimed at distinct ingredients of the learning process as visualized in Fig . 2 . 3.1 GENERAL EXPERIMENTAL BENCHMARK SETUP . We will focus on stochastic gradient descent ( SGD ) learning applied to convolutional neural networks . As a stopping criterion , we consider the scenario where training has already converged , where the validation loss can not be decreased further by changing the model f via SGD . Empirical task transfer often deals with tasks that are separate ( Zamir et al. , 2018 ) , and while fruitful empirically , it becomes difficult to understand conceptually . Instead we employ controlled customized task changes , allowing us to better gauge the effect of task changes . With the basic building blocks in place , we ask in which cases , even with an infinite number of data points , previous tasks can hinder learning and generalization on new tasks . We examine three intervention scenarios acting on various levels of abstraction as visualized in Fig . 2 : A ) Learning rate changes : Applying changes to the learning process , we observe how employing different learning rates both on the first and second task can influence pretraining effects . B ) Task discretization : Intervening on the task-level , we consider the influence of inserting more tasks between first and last task by discretizing the changes from first to last task . C ) Model biases resetting : Finally , we act on the model and examine how final performance is affected when resetting biases before tasks . We deal with four visual supervised classification datasets of which we use three in every experimental section below thereby significantly extending smaller scale experiments by Achille et al . ( 2017 ) : • CIFAR-10 ( Krizhevsky , 2009 ) : A 32x32 full-color dataset with 10 classes . • FashionMNIST ( Xiao et al. , 2017 ) : A 28x28 , black-and-white dataset with 10 classes . • MNIST ( LeCun et al. , 1998 ) : A 28x28 , black-and-white dataset with 10 classes . • SVHN ( Sermanet et al. , 2012 ) : A 32x32 full-color dataset with 10 classes . For all experiments if not mentioned otherwise , we use Pytorch ( Paszke et al. , 2019 ) and run a ResNet-18 ( He et al. , 2016 ) with a fixed initial learning rate of 10e-4 with momentum 0.9 and weight decay 1e-3 . The networks are initialized using the standard Pytorch layer intialization discussed in Appendix 8.2 . Each task is trained until convergence , for a maximum of 200 epochs per task , using early stopping on a validation set with patience 50 . As baseline , the model was trained only on the target task - the standard uncorrupted dataset .

Summary: The authors study what they call a negative pretraining effect = models pretrained on task 1 and tuned on task 2 sometimes underperform compared to just training on task 2 from scratch. This is an important factor in many forms of life long learning, multi-task learning and curriculum learning. They investigate 3 potential remedies / setups: a) using different learning rates for task 1 and task 2, b) changing from task 1 to task 2 more smoothly, and c) resetting network biases at different stages of the process. The perform with a single ResNet18 architecture on MNIST, Fashion MNIST, SVHN and CIFAR-10.