Generative Negative Replay for Continual Learning

1 INTRODUCTION . The majority of neural networks training approaches assume that is feasible to build a set of independent and identically distributed ( i.i.d . ) samples to train the model . This assumption is in contrast with biological learning since intelligent beings observe the world as an ordered sequence of highly correlated data . When state-of-the-art deep neural networks are trained continually , and the whole data can not be accessed at once , the model suffers from the catastrophic forgetting problem ( McCloskey & Cohen , 1989 ) , and the knowledge about old data ( old experiences ) tend to be overwritten by new examples . Storing part of past data and replaying them interleaved with new data proved to be an effective approach to mitigate forgetting ( see Hayes et al . ( 2021 ) for a comprehensive survey ) . However , in some applications , the storage overhead together with privacy issues make replay techniques unfeasible . Therefore , generative replay has been recently explored , where a generative model is trained to produce data from past experiences ( see Lesort et al . ( 2018 ) and Shin et al . ( 2017 ) ) . Besides solving the replay memory issue , generative replay can theoretically be capable of generating more general and novel examples not included in past experiences , thus potentially overcoming replay methods . Unfortunately , generative replay introduces much complexity due to the need for an interleaved incremental training of both a classifier and a generator . Moreover , generative models are usually complex and unstable to train , especially in incremental scenarios . Several researchers have shown that generative replay fails in complex CL scenarios with high-dimensional data ( see Aljundi et al . ( 2019 ) ; Lesort et al . ( 2018 ) and van de Ven et al . ( 2020 ) ) mainly due to the inaccuracies in the data generation that progressively grows across the experiences if a single generator is incrementally updated ( see related works in section 5 for more details ) . The photocopy example helps to understand why . Let us consider a high-quality photocopy machine : when a picture is initially copied the output looks very similar to the original , but if the process is repeated several times by using as input the output of the previous step , some artifacts will soon appear and , after many iterations , the result will be highly compromised . Hence , even if some state-of-the-art models have been proved to be effective in generating also high dimensional data ( Huang et al . ( 2018 ) Karras et al . ( 2019 ) ) , the continual training of such generators remains a challenging problem . Although generative models are a hot research topics and we can expect improved methods in the future , as of today we must deal with imperfect generated data and try to exploit them at best when a classifier is incrementally trained . The proposed approach , denoted as Generative Negative Replay , does not attempt to improve the knowledge of old classes using the generated data because it assumes that the data quality is not enough for this purpose . Nevertheless , it makes use of generated ( latent ) data as negative examples to better learn the classes of current experience , especially when the number of classes per experience is small and we incur in the “ learning in isolation ” problem . We experimentally demonstrate , on complex benchmarks such as CORe50 and ImageNet-1000 , where ( positive ) generative replay fails , that negative replay is effective to contrast the learning in isolation problem , allowing to train a classifier incrementally across a high number of experiences ( see Figure 1 ) . We also investigate the impact of data quality on negative replay with an ablation study ( section 4.4 ) where negative examples are sampled from original past patterns ( upper bound ) and randomly generated . 2 PROBLEM FORMULATION . A continual learning ( CL ) problem consists of a number NE of experiences , each containing a subset of data that is only accessible during the corresponding experience : CL = { e1 , e2 , ... , eNE } , ( 1 ) each of them is composed of several data points and the corresponding labels : ek = ( Xk , Yk ) , Xk = { xk1 , xk2 , ... , xkNk } , Yk = { y k 1 , y k 2 , ... , y k Nk } ( 2 ) where xki and y k i are the data points and the associated labels contained in the k-th experience and Nk is the number of samples in the k-th experience . Let D = ( X , Y ) be the entire dataset , then X = ⋃NE i=1 Xi and Y = ⋃NE i=1 Yi . We can define three different scenarios for supervised continual learning ( Maltoni & Lomonaco , 2019 ; van de Ven & Tolias , 2018 ) : New Instances ( NI ) , New Classes ( NC ) , and New Instances and Classes ( NIC ) . In NI ( also known as domain incremental ) all the classes are introduced in the first experience , and only new examples of the same classes are included in the following experiences . In NC ( also known as class incremental ) each experience contains only examples from classes never seen before . NIC is the combination of NI and NC , so each experience can be composed of new examples of already seen classes and/or examples from new classes . A formal definition of the above scenarios can be found in Appendix A . Given the above definitions , our goal is to fit a function f , parametrized by Θ , to the sequence of experiences . A naive approach is finding the best parameters Θ∗ that minimizes : Θ∗ = arg min Θ L ( fΘ ( Xi ) , Yi ) for i = { 1 , ... , NE } , ( 3 ) where L ( · ) is a loss function ( e.g . cross entropy loss ) . As first pointed out by McCloskey & Cohen ( 1989 ) , this simple approach is prone to catastrophic forgetting , thus the model fΘ is not able to learn the experiences { e1 , e2 , ... , eNE } sequentially . 2.1 CONTINUAL LEARNING WITH GENERATIVE REPLAY . Generative replay requires to train simultaneously and incrementally a classifier and a generative model ( Shin et al. , 2017 ; Wu et al. , 2018 ; Thandiackal et al. , 2021 ) The generative model g , parametrized by Ω provides surrogate data similar to the past experiences ’ data . In the case of a conditional generative model ( in which we can control the class of the generated data ) , the optimal parameters of the classifier can be derived using a replay memory as follows : Θ∗ = arg min Θ L ( fΘ ( Xi ∪Mxi , Yi ∪M y i ) for i = { 1 , ... , NE } , ( 4 ) whereMxi andM y i are the datapoints and labels contained in the replay memory during the training on experience i when the replay memory is populated as : Mxi ← gΩ ( zj |cj ) ; M y i ← cj ; cj ∈ i−1⋃ k=1 Yk , j = { 1 , ... , R } , ( 5 ) where R is the number of generated replay patterns ( size of memory ) , z is a latent random input vector to the generative model , c is a label sampled from the set of labels encountered in the past experiences , and “ ← ” indicates the insertion of an element in the memory . The same generated data fed to the classifier can be used to control forgetting in the generative model as well . Instead of a generic generative model , suppose we have a conditional generative model composed of an encoder qγ parametrized by γ and a decoder pξ parametrized by ξ , such that gΩ = pξ ◦ qγ , Ω = ( γ , ξ ) . The optimal parameters of the generative model can be obtained requiring that the generated data are similar ( L2 loss ) to the original ones : γ∗ , ξ∗ = arg min γ , ξ ‖pξ ( qγ ( Xi ∪Mxi ) |Yi ∪M y i ) −Xi ∪M x i ‖22 for i = { 1 , ... , NE } , ( 6 ) where qγ ( Xi ) is forced to follow a target distribution , typically N ( 0 , 1 ) . 3 GENERATIVE NEGATIVE REPLAY . As discussed before , generative replay is an appealing strategy for continual learning , but , to exploit it in complex scenarios with many experiences , we need to overcome the data degradation issue . Since this problem is not easily addressable on the generator side , we propose to circumvent it by changing the way the classifier makes use of generated data . Let us suppose the classifier fΘ can be divided into a feature extractor fφ , parametrized by φ and a classification head cψ parametrized by ψ , so that fΘ = cψ ◦ fφ , Θ = ( φ , ψ ) . The parameters ψ of the classification head can be divided into C groups , where C is the number of classes . The groups , denoted as ( ψ1 , ψ2 , ... , ψC ) represents the parameters associated to the connections between the features extracted by fφ and the output neuron of the corresponding class . For simplicity , let us assume that the feature extraction weights φ are frozen ( after an initial pretraining ) and , across the experiences , we only learn the classification head weights ψ . As explained in Section section 3.3 , this assumption is not necessary and our experiments were carried out by learning both φ and ψ . 3.1 LEARNING CLASSES IN ISOLATION . Learning in isolation is one of the main causes of catastrophic forgetting , especially in the NC or NIC scenarios where only a limited number of classes are present in a single experience , and the parameters of the classification head are learned without negative examples that counteract the “ greediness ” of the optimization . As an example , let us consider an NC scenario where only one class is present in each experience . Suppose that c is the only class in the experience k , then the best way to optimize the model is to change the parameters ψc to maximize the output of the output neuron c for every input and change the rest of ψj , j 6= c to minimize the output for remaining classes . This still holds if in the experience are present only a few classes , since the model is only optimized to discriminate between the present classes and has no interest in maintaining the past acquired knowledge . 3.2 POSITIVE AND NEGATIVE REPLAY . Replay can be used to counteract the learning in isolation problem , however , when the replay data comes from a generative model , the data quality degradation has a negative impact on the classifier training . The aforementioned problem is typical of the standard generative replay approach ( hereafter denoted as generative positive replay ) , where replay data is used by the classifier in the same manner of the current experience ’ s data , and therefore the classification head ’ s weights associated to the replay classes are optimized based on the replay data . On the contrary , in the proposed generative negative replay approach the replay patterns are used to counteract the detrimental effects of the training in isolation but they are not used to modify the parameters ψ associated with the replay classes . The key idea ( validated experimentally ) is that the generated patterns are valid antagonists to mitigate the learning in isolation problem , but their quality is not enough to improve the knowledge of classes originally learned on real data . It is well known that one class learning approaches are in general less effective than discriminative learning because the presence of both positive and negative examples allows to better characterize the classification boundaries ( Hempstalk & Frank , 2008 ) . Therefore , the proposed approach exploits generated data to constrain the classification boundary and to avoid that the real data in the current experience pull it too much in their direction .

Summary: In Continual Learning (CL) scenarios, storing data from previous experience is helpful to mitigate catastrophic forgetting. However, privacy issues or storage overhead makes replay methods impractical. Aware of this problem, generative models have been proposed in previous work to generate data that represent previous experiences. Still, these methods suffer from low performance because of the continual training of the generating model and/or the generation of high dimensionality data. In this paper, the authors propose Generative Negative Replay. Instead of using the generated data as a positive example of previous classes, they used it as a negative example for the classes present in the current experience. While training with the negative examples, the authors propose to freeze the weights of the classification head corresponding to the previous classes, similar to what happens in CWR. This solution is particularly relevant in situations where there are not as many classes from experience.