Three continual learning scenarios and a case for generative replay

1 INTRODUCTION . Current state-of-the-art deep neural networks can be trained to impressive performance on a wide variety of individual tasks . Learning multiple tasks in sequence , however , remains a substantial challenge for deep learning . When trained on a new task , standard neural networks forget most information related to previously learned tasks , a phenomenon referred to as “ catastrophic forgetting ” . In recent years , numerous methods for alleviating catastrophic forgetting have been proposed . However , due to the wide variety of experimental protocols used to evaluate them , many of these methods claim “ state-of-the-art ” performance ( e.g. , Kirkpatrick et al. , 2017 ; Rebuffi et al. , 2017 ; Nguyen et al. , 2017 ; Masse et al. , 2018 ; Kemker & Kanan , 2018 ; Wu et al. , 2018 ) . To obscure things further , some methods shown to perform well in some experimental settings are reported to dramatically fail in others : compare the performance of elastic weight consolidation in Kirkpatrick et al . ( 2017 ) and Zenke et al . ( 2017 ) with that in Kemker et al . ( 2017 ) and Kamra et al . ( 2017 ) . To enable a fairer and more structured comparison of methods for reducing catastrophic forgetting , as a first contribution this paper identifies three distinct continual learning scenarios of increasing difficulty . These scenarios are distinguished by whether at test time task identity is provided and , if it is not , whether task identity needs to be inferred . We show that such differences in experimental design can explain seemingly contradictory results reported in the recent literature : even for experimental protocols involving the relatively simple classification of MNIST-digits , methods that perform well in one continual learning scenario can completely fail in another . Using these three scenarios , a second contribution of this paper is to provide an extensive comparison of recently proposed methods . These experiments reveal that generative replay , especially when combined with distillation techniques , has the capability to perform well on all three scenarios . An important disadvantage of this approach , however , is that it can be computationally very costly . As a third contribution , this paper proposes a way to reduce these computational costs . Current approaches using generative replay train two separate models : a main model for solving the tasks and a generative model for sampling examples representative of previous tasks . We merge the generative model into the main model by equipping it with backward connections that are trained to have generative capability ( e.g. , a variational autoencoder with added softmax layer ) . We demonstrate that this can substantially reduce training time , with no or negligible loss in performance . 2 CONTINUAL LEARNING SCENARIOS . We consider the continual learning problem in which a single model needs to sequentially learn a series of tasks , whereby it is not allowed to store raw data . This continual learning framework has been actively studied in recent years : many methods for alleviating catastrophic forgetting are being proposed , with almost as many different experimental protocols being used for their evaluation . We found that an important difference between these experimental protocols is whether at test time information about the task identity is available and—if it is not—whether the model is required to identify the identity of the task it has to solve . Yet , this crucial experimental design consideration is not always clearly stated and differences in this regard are sometimes not appreciated . For example , in Masse et al . ( 2018 ) a substantial improvement over state-of-the-art is reported , while their method assumes task identity is always available and the compared methods operate without this assumption . To enable more meaningful comparisons , we identify three distinct scenarios for continual learning . In the first scenario , models are always informed about which task needs to be performed . This is the easiest continual learning scenario , and we refer to it as incremental task learning . Since task identity is always provided , it is possible to train models with task-specific components . A typical neural network architecture used in this scenario has a “ multihead ” output layer , meaning that each task has its own output units but the rest of the network is ( potentially ) shared between tasks . In the second scenario , which we refer to as incremental domain learning , task identity is not available at test time . Models however only need to solve the task at hand ; they are not required to infer which task it is . Typical examples of this scenario are protocols whereby the structure of the tasks is always the same , but the input-distribution is changing . A classical example of such a task protocol is ‘ permuted MNIST ’ ( Goodfellow et al. , 2013 ) , in which all tasks involve classifying MNIST-digits but with a different permutation applied to the pixels for each new task ( Figure 2 ) . Although permuted MNIST is most naturally performed according to the incremental domain learning scenario , it can be performed according to the other scenarios too ( Table 2 ) . Finally , in the third scenario , models need to be able both to solve each task seen so far and to infer which task they are presented with . We refer to this scenario as incremental class learning , as it includes protocols in which new classes need to be learned incrementally . An example task protocol most naturally performed under this scenario is sequentially learning MNIST-digits ( ‘ split MNIST ’ ; Figure 1 ) , although this protocol has also been performed under the other two scenarios ( Table 1 ) . 3 CONTINUAL LEARNING STRATEGIES . A simple and intuitive explanation for catastrophic forgetting is that after a neural network is trained on a new task , its parameters are now optimized for the new task and no longer for the previous one ( s ) . This formulation highlights two strategies for alleviating catastrophic forgetting : ( 1 ) not freely optimizing the entire network on each task , and ( 2 ) modifying the training data to make it more representative for previous tasks . 3.1 NOT OPTIMIZING ENTIRE NETWORK / REGULARIZED OPTIMIZATION . A straightforward way of not optimizing the full network on every task is to explicitly define a different subnetwork per task . Several recent papers use this strategy , with different approaches for selecting the parts of the network for each task . A simple approach is to randomly assign which nodes participate in each task ( Context-dependent Gating [ XdG ] ; Masse et al. , 2018 ) . Other approaches use evolutionary algorithms ( Fernando et al. , 2017 ) or gradient descent ( Serrà et al. , 2018 ) to learn which units to employ for each task . By design , these approaches are limited to the incremental task learning scenario , as task identity is required to select the correct task-specific components . A modification to make this strategy applicable in the other scenarios is to preferentially train a different part of the network for each task , but to always use the entire network for execution . One way to do this is by differently regularizing the network ’ s parameters during training on each new task , which is the approach of Elastic Weight Consolidation ( EWC ; Kirkpatrick et al. , 2017 ) and Synaptic Intelligence ( SI ; Zenke et al. , 2017 ) . Both methods estimate for all parameters of the network how important they are for the previously learned tasks and penalize future changes to them accordingly ( i.e. , learning is slowed down for parts of the network important for previous tasks ) . 3.2 MODIFYING TRAINING DATA . A second strategy is to complement the training data for each new task to be learned with “ pseudodata ” representative of the previous tasks . We refer to this strategy as replay . An early implementation of this strategy , called pseudo-rehearsal , generated completely random inputs as pseudo-data and labeled them based on the predictions of a copy of the model stored after finishing training on the previous task ( Robins , 1995 ) . This approach had some success with very simple , artificial inputs , but does not work with more complicated inputs ( Atkinson et al. , 2018 ) . An alternative is to take the input data of the current task , label them using the model trained on the previous tasks , and use the resulting input-target pairs as pseudo-data . This is the approach of Learning without Forgetting ( LwF ; Li & Hoiem , 2017 ) . Another important aspect of this method is that instead of labeling the inputs to be replayed as the most likely category according to the previous tasks ’ model ( i.e. , “ hard targets ” ) , it pairs them with the by that model predicted probabilities for all target classes ( i.e. , “ soft targets ” ) . The objective for the replayed data is then to match the probabilities predicted by the model being trained to these target probabilities . The approach of matching predicted ( and typically temperature-raised ) probabilities of one network to those of another network had previously been used to compress ( or “ distill ” ) information from one ( large ) network to another ( smaller ) network ( Hinton et al. , 2015 ) . Another option is to generate the input data to be replayed . For this , besides the main model for task performance ( e.g. , classification ) , a separate generative model is sequentially trained on all tasks to generate samples from their input data distributions . For the first application of this approach , which was called Deep Generative Replay ( DGR ) , the generated input samples were paired with “ hard targets ” provided by the main model ( Shin et al. , 2017 ) . We note that it is possible to combine LwF and DGR by replaying input samples from a generative model and pairing them with soft targets ( see also Wu et al. , 2018 ; Venkatesan et al. , 2017 ) . We include this hybrid method in our comparison under the name DGR+distill . A final option is to store examples from previous tasks and replay those . Such “ exact replay ” can substantially boost performance ( Rebuffi et al. , 2017 ; Nguyen et al. , 2017 ; Kemker & Kanan , 2018 ; see Appendix C ) , but due to privacy concerns or memory constraints , it is not always possible to do so . In this paper we restrict ourselves to the case where storing raw data is not allowed . 4 EXPERIMENTAL DETAILS . To compare the performance of the above discussed approaches , we used two different task protocols that were both performed according to all three continual learning scenarios defined in section 2 . 4.1 TASK PROTOCOLS . The first task protocol was split MNIST ( Zenke et al. , 2017 ; Figure 1 ) . For this , the original MNISTdataset was split into five tasks , where each task was a two digit classification . The original 28x28 pixel grey-scale images were used without pre-processing . The standard training/test-split was used resulting in 60,000 training images ( ∼6000 per digit ) and 10,000 test images ( ∼1000 per digit ) . The second task protocol was permuted MNIST ( Goodfellow et al. , 2013 ; Figure 2 ) . The tasks of this protocol were classifying MNIST-digits ( every task now had all ten digits ) , whereby in each task the pixels of the MNIST-images were permutated in a different way . We used a sequence of ten such tasks . To generate the permutated images , the original images were first zero-padded to 32x32 pixels . For each task , a random permutation was then generated and applied to these 1024 pixels . No other pre-processing was performed . Again the standard training/test-split was used .

Summary: This paper summarizes previous lifelong learning methods and identifies three different continual learning scenarios. Based on that, it draws a conclusion that DGR+distill outperforms other methods on all these scenarios. Further, the paper proposes unified model that combines a replay generator and a classification model. The proposed RTF model achieves comparable performance with DGR+distill and is approximately two times faster than DGR+distill.