Learn the Time to Learn: Replay Scheduling for Continual Learning

1 INTRODUCTION . Many organizations deploying machine learning systems receive large volumes of data daily where these new data are often associated with new tasks . Although all historical data are stored in the cloud in practice , retraining machine learning systems on a daily basis is prohibitive both in time and cost . In this setting , the systems must continuously adapt to new tasks without forgetting the previously learned abilities . Continual learning methods ( De Lange et al. , 2019 ; McCloskey & Cohen , 1989 ; Parisi et al. , 2019 ) address this challenge where , in particular , replay-based methods ( Chaudhry et al. , 2019 ; Hayes et al. , 2020 ) have shown to be very effective in achieving great prediction performance and retaining knowledge of old tasks . Replay-based methods mitigate catastrophic forgetting by revisiting a small set of samples , which is feasible to process compared to the size of the historical data . In traditional continual learning literature , the replay memory is limited due to the assumption that historical data are not available . In the real-world setting where historical data are in fact always available , the requirement of small memory remains due to processing time and cost issues . Most research on replay-based continual learning has been focused on the sample quality in the memory ( Aljundi et al. , 2019 ; Borsos et al. , 2020 ; Chaudhry et al. , 2019 ; Chrysakis & Moens , 2020 ; Nguyen et al. , 2017 ; Rebuffi et al. , 2017 ; Yoon et al. , 2021 ) or data compression to increase the memory capacity ( Hayes et al. , 2020 ; Iscen et al. , 2020 ; Pellegrini et al. , 2019 ) . Common for these methods is that the memory allocates an equal amount of space for storing samples from old tasks . When learning new tasks , the whole memory is replayed to mitigate catastrophic forgetting . However , in life-long learning settings , this simple strategy would be inefficient as the memory must store a large number of tasks . Furthermore , these methods ignore the time to learn old tasks again which is important in human learning . Humans are continual learning systems , and different methods have been developed to enhance memory retention , such as spaced repetition ( Dempster , 1989 ; Ebbinghaus , 2013 ; Landauer & Bjork , 1977 ) which is used often in education . These education methods focus on the scheduling of learning and rehearsal of previous learned knowledge . In this work , we argue that finding the proper schedule of what tasks to replay in the fixed memory setting is critical for continual learning . To demonstrate our claim , we perform a simple experiment on the Split MNIST ( Zenke et al. , 2017 ) dataset where each task consists of learning the digits 0/1 , 2/3 , etc . arriving in sequence . The replay memory contains data from task 1 and can only be replayed at one point in time . Figure 1 shows how the task performances progress over time when the memory is replayed at different time steps . In this example , the best final performance is achieved when the memory is used when learning task 5 . Note that choosing different time points to replay the same memory leads to noticeably different results in the final performance . These results indicate that scheduling the time when to apply replay can influence the final performance significantly of a continual learning system . To this end , we propose learning the time to learn , in which we learn replay schedules of which tasks to replay at different times inspired from human learning ( Dempster , 1989 ) . We illustrate the advantages with replay scheduling by using Monte Carlo tree search ( MCTS ) ( Coulom , 2006 ) to learn policies for replay . More specifically , we train a neural network on the current task dataset mixed with the scheduled replay samples and measure the final performance of the network to evaluate the replay schedules selected by MCTS . In summary , our contributions are : • We demonstrate the importance of replay scheduling in continual learning and propose to learn the time to learn which tasks to replay ( Section 3.1 and 3.2 ) . • We use MCTS as an example method to illustrate how replay schedules for continual learning can be learned by establishing a finite set of memory compositions that can be replayed at every task ( Section 3.3 ) . • We demonstrate with several benchmark datasets that learned scheduling can improve the continual learning performance significantly in the fixed size memory setting ( Section 4.1 and 4.4 ) . Furthermore , we show that our method can be combined with any other memory selection methods ( Section 4.3 ) , as well as being efficient in situations where the memory size is even smaller than the number of classes ( Section 4.5 ) . 2 RELATED WORK . In this section , we give a brief overview of continual learning methods , essentially replay-based methods , as well as spaced repetition techniques for human continual learning . Continual Learning . Traditional continual learning can be divided into three main areas , namely regularization-based , architecture-based , and replay-based approaches . Regularization-based methods aim to mitigate catastrophic forgetting by protecting parameters influencing the predictive performance from wide changes and use the rest of the parameters for learning the new tasks ( Adel et al. , 2019 ; Chaudhry et al. , 2018a ; Kirkpatrick et al. , 2017 ; Li & Hoiem , 2017 ; Nguyen et al. , 2017 ; Rannen et al. , 2017 ; Schwarz et al. , 2018 ; Zenke et al. , 2017 ) . Architecture-based methods isolate task-specific parameters by either increasing network capacity ( Rusu et al. , 2016 ; Yoon et al. , 2019 ; 2017 ) or freezing parts of the network ( Mallya & Lazebnik , 2018 ; Serra et al. , 2018 ) to maintain good performance on previous tasks . Replay-based methods mix samples from old tasks with the current dataset to mitigate catastrophic forgetting , where the replay samples are either stored in an external memory ( Chaudhry et al. , 2019 ; Hayes et al. , 2020 ; Isele & Cosgun , 2018 ; Lopez-Paz & Ranzato , 2017 ) or generated using a generative model ( Shin et al. , 2017 ; van de Ven & Tolias , 2018 ) . Regularization-based approaches and dynamic architectures have been combined with replay-based approaches to methods to overcome their limitations ( Chaudhry et al. , 2018a ; b ; Douillard et al. , 2020 ; Ebrahimi et al. , 2020 ; Joseph & Balasubramanian , 2020 ; Mirzadeh et al. , 2020 ; Nguyen et al. , 2017 ; Pan et al. , 2020 ; Pellegrini et al. , 2019 ; Rolnick et al. , 2018 ; von Oswald et al. , 2019 ) . Our work relates most to replay-based methods with external memory which we spend more time on describing in the next paragraph . Replay-based Continual Learning . A commonly used memory selection strategy of replay samples is random selection . Much research effort has focused on selecting higher quality samples to store in memory ( Aljundi et al. , 2019 ; Borsos et al. , 2020 ; Chaudhry et al. , 2019 ; Chrysakis & Moens , 2020 ; Hayes et al. , 2019 ; Isele & Cosgun , 2018 ; Lopez-Paz & Ranzato , 2017 ; Nguyen et al. , 2017 ; Rebuffi et al. , 2017 ; Yoon et al. , 2021 ) . Chaudhry et al . ( 2019 ) reviews several selection strategies in scenarios with tiny memory capacity , e.g. , reservoir sampling ( Vitter , 1985 ) , first-in first-out buffer ( Lopez-Paz & Ranzato , 2017 ) , k-Means , and Mean-of-Features ( Rebuffi et al. , 2017 ) . However , more elaborate selection strategies have been shown to give little benefit over random selection for image classification problems ( Chaudhry et al. , 2018a ; Hayes et al. , 2020 ) . More recently , there has been work on compressing raw images to feature representations to increase the number of memory examples for replay ( Hayes et al. , 2020 ; Iscen et al. , 2020 ; Pellegrini et al. , 2019 ) . Our approach differs from the above mentioned works since we focus on learning to select which tasks to replay at the current task rather than improving memory selection or compression quality of the samples in the memory . Replay scheduling can however be combined with any selection strategy as well as storing feature representations . Human Continual Learning . Humans are continual learning systems in the sense of learning tasks and concepts sequentially . Furthermore , humans have an impressive ability to memorize experiences but can forget learned knowledge gradually rather than catastrophically ( French , 1999 ) . Different learning techniques have been suggested for humans to memorize better ( Dunlosky et al. , 2013 ; Willis , 2007 ) . An example is spaced repetition which gradually increases time-intervals between rehearsals for retaining long-term memory ( Dempster , 1989 ) . This technique has been studied frequently and was inspired from the works of Ebbinghaus ( 2013 ) on memory retention . For example , Landauer & Bjork ( 1977 ) demonstrated that memory training schedules using adjusted spaced repetition were better at preserving memory than uniformly spaced training . Hawley et al . ( 2008 ) studies the efficacy of spaced repetition on adults with probable Alzheimer ’ s disease for learning face-name association . Several works in continual learning with neural networks are inspired by or have a connection to human learning techniques , including spaced repetition ( Amiri , 2019 ; Amiri et al. , 2017 ; Feng et al. , 2019 ; Smolen et al. , 2016 ) , mechanisms of sleep ( Ball et al. , 2020 ; Mallya & Lazebnik , 2018 ; Schwarz et al. , 2018 ) , and reactivation of memories ( Hayes et al. , 2020 ; van de Ven et al. , 2020 ) . Our replay scheduling method is inspired by spaced repetition ; we learn schedules of which memory samples to use for replay at different time steps . 3 METHOD . In this section , we describe our method for learning replay schedules for continual learning . The idea is to learn schedules of which memory examples the network should rehearse at different times . We use Monte Carlo tree search ( MCTS ) ( Browne et al. , 2012 ; Coulom , 2006 ) to learn a scheduling policy by encouraging searches for promising replay schedules based on the classification accuracy . 3.1 PROBLEM SETTING . We focus on the setting considering the real-world continual learning needs where all historical data are available but are prohibitively large . Therefore , only a small amount of historical data can be used when adapting the model to new data due to processing capability consideration . Thus , the goal is to learn how to select subsets of historical data to efficiently mitigate catastrophic forgetting when learning new tasks . We refer to these subsets of historical data as the replay memory throughout the paper , where the size of the replay memory affects the processing time when learning a new task t. Moreover , we focus on composing the replay memory based on the seen tasks in the historical data rather than single stored instances . Next , we introduce the notation of our problem setting which resembles the traditional continual learning setting for image classification . We let a neural network fθ , parameterized by θ , learn T tasks sequentially given their corresponding task datasets D1 , . . . , DT arriving in order . The t-th datasetDt = { ( x ( i ) t , y ( i ) t ) } Nt i=1 consists of Nt samples where x ( i ) t and y ( i ) t are the i-th data point and class label respectively . The training objective at task t is given by min θ Nt∑ i=1 ` ( fθ ( x ( i ) t ) , y ( i ) t ) , ( 1 ) where ` ( · ) is the loss function , e.g. , cross-entropy loss in our case . Since Dt is only accessible at time step t , the network fθ is at risk of catastrophically forgetting the previous t − 1 tasks when learning the current task . Replay-based continual learning methods mitigate the forgetting of old tasks by storing old examples in an external replay memory , that is mixed with the current task dataset during training . Next , we describe our method for constructing this replay memory . We assume that historical data from old tasks are accessible at any time step t. However , since the historical data is prohibitively large , we can only fill a small replay memoryM with M historical samples for replay due to processing time constraints . The challenge is how to fill the replay memory withM samples that efficiently retain the knowledge of old tasks when learning new tasks . We focus on selecting the samples on task-level by deciding on the task proportion ( a1 , . . . , at−1 ) of samples to fetch from each task , where ai ≥ 0 is the proportion ofM examples from task i and ∑t−1 i=1 ai = 1 . Consequently , we need a method for choosing these task proportions of which old tasks to replay . To simplify this selection , we construct a discrete set of choices for possible task proportions telling how many samples from each task to use when constructing the replay memoryM .

Summary: This paper proposes a new continual learning method that learns to select samples for the replaying process. The replay memory is filled with samples from previous tasks according to a specific proportion corresponding to the action performed at the current task. The paper proposes to find the optimal sequence of actions by using Monte-Carlo Tree Search (MCTS) with the reward as the average accuracy over all tasks after seeing the final task. The experiments demonstrate that MCTS improves the continual learning performance especially when the memory size is small and is compatible with some sample-selection strategies.