RMM: Reinforced Memory Management for Class-Incremental Learning

1 Introduction . Ideally , AI systems should be adaptive to ever-changing environments—where the data are continuously observed by sensors . Their models should be capable of learning new concepts from data while maintaining the ability to recognize previous ones . In practice , the systems often have constrained memory budgets because of which most of the historical data have to be abandoned [ 20 ] . However , deep-learning-based AI systems , when continuously updated using new data and limited historical data , often suffer from catastrophic forgetting , as the updates can override knowledge acquired from previous data [ 33 , 34 , 39 ] . To encourage research on the forgetting problem , Rebuffi et al . [ 40 ] defined a standard protocol of class-incremental learning ( CIL ) for image classification , where the training data of different object classes come in phases . In each phase , the classifier is evaluated on all classes observed so far . As the total memory size is limited [ 40 ] , CIL systems abandon the majority of the data and only preserve a small number of exemplars , e.g. , 20 exemplars per class , which will be used for replaying in subsequent phases . Replaying usually happens for multiple epochs [ 13 , 18 , 30 , 40 ] , so both the old 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . class exemplars and new class data need to be stored in the limited memory . Existing CIL methods allocate memory between the old and new classes in an arbitrary and static fashion , e.g. , 20 per old class vs. 1 , 300 per new class for the ImageNet-Full dataset . This causes a serious imbalance between the old and new classes and can exacerbate the problem of catastrophic forgetting . To address this , we propose to learn an optimal memory management policy for each incremental phase with continuously reinforced model performance and call our method reinforced memory management ( RMM ) . Detailed actions include 1 ) allocating the memory between the existing ( old ) and the coming ( new ) data for each phase , and 2 ) specifying the memory for each old class according to its recognition difficulty before abandoning any of its data . To this end , we leverage reinforcement learning [ 26–28 , 51 , 59 ] and design a new policy function to contain two sub-functions that propagate two levels of actions in a hierarchical way . Level-1 function determines how to split memory between the old and new data . Its output action is then inputted into the Level-2 function to determine how to allocate memory for each old class . The overall objective of the function is to maximize the cumulative evaluation accuracy across all incremental phases . However , this is not naturally compatible with the standard protocol of CIL [ 40 ] where neither past nor future data are accessible for evaluation . To tackle this issue , we propose to pre-train the function on pseudo CIL tasks and then adopt it in the learning process of our target task . In principle , we can build such pseudo tasks using any available categorical data , e.g. , the data in the 0-th phase of the target CIL task or the data from another dataset . Even though this is a non-stationary reinforcement learning problem , we can regard the pseudo and target CIL tasks as a sequence of stationary tasks and train the policy function to exploit the dependencies between these consecutive tasks . Such continuous adaptation in non-stationary environments is feasible based on the empirical analysis given in [ 2 ] . Technically , we propose the following method to guarantee the transferability of policy functions between pseudo and target CIL tasks . We take a Level-1 action based on the ratio of the number of new classes to the total number of classes observed so far . A lower ( higher ) ratio will result in weakening the stability ( plasticity ) of the classification model . Then , we take a Level-2 action for each individual class conditioned on both the Level-1 action and the training entropy of that class . A higher entropy denotes a more difficult class , leading to more memory allocated to the class . For evaluation , we conduct extensive CIL experiments by plugging RMM into two top-performing methods ( LUCIR+AANets , POD+AANets ) and testing them on three benchmarks ( CIFAR-100 , ImageNet-Subset , and ImageNet-Full ) . Our results show the clear and consistent superiority of RMM , e.g. , it boosts the state-of-the-art POD+AANets by 3.6 % , 4.4 % , and 1.9 % in the 25-Phase settings of the above benchmarks , respectively . Our technical contribution is three-fold . 1 ) A hierarchical reinforcement learning algorithm called RMM to manage the memory in a way that can be conveniently modified through incremental phases and for different classes . 2 ) A pseudo task generation strategy that requires only in-domain available data ( small-scale ) or cross-domain datasets ( large-scale ) , relieving the data incompatibility between reinforcement learning and class-incremental learning . 3 ) Extensive experiments , visualization , and interpretation for RMM in three CIL benchmarks and using two top models as baselines . 2 Related Work . Incremental Learning [ 22 , 36 , 48 , 53 , 58 ] continuously updates the model using data coming in a sequence of phases . Similar tasks are also referred to as continual learning [ 12 , 32 ] and lifelong learning [ 3 , 9 ] . Recent papers are either task-incremental learning—each phase corresponds to a task ( dataset ) that contains new data of all seen classes [ 7 , 10 , 19 , 29 , 43 , 47 , 57 ] , or classincremental learning ( CIL ) —each phase contains data of a new set of classes , i.e. , classes are unseen [ 4 , 6 , 18 , 25 , 31 , 37 , 40 , 41 , 49 , 52 , 55–57 ] . This paper is concerned with CIL . The key challenge of CIL is the forgetting problem—older classes are forgotten in later phases . Existing methods tackling this can be divided into three categories : memory-based , regularization-based , and network-architecture-based [ 11 , 35 ] . Memory-based methods preserved a small subset of the old class data ( exemplars ) to replay the model on them ( together with the new class data ) , in order to relieve the forgetting of the old classes . Some work [ 18 , 40 ] proposed heuristic strategies to select more representative exemplars from the old class data , and others [ 31 , 47 ] tried to generate exemplars in optimizable frameworks . None of them changed the allocation of memory for different classes , i.e. , all used an arbitrary and static scheme for memory allocation . Regularization-based methods introduce regularization terms in the loss function to consolidate previous knowledge when training the model on new data . The key idea is to enforce predicted label logits [ 29 , 40 ] , features maps [ 13 , 18 ] , or the topology in the feature space [ 49 ] of the new model to be close to that of the previous model . Network-architecture-based methods aim to design “ incremental network architectures ” . Some work [ 45 , 54 ] gradually extended the network capacity for new data , while others proposed to freeze partial network parameters [ 1 , 30 ] to preserve the knowledge of the old classes . Reinforcement Learning defines an agent that needs to decide its actions in an unknown environment by maximizing the expected cumulative reward . It has been widely applied to many optimization problems , e.g. , neural architecture search [ 54 , 59 ] and neural machine translation [ 38 , 46 ] . Reinforcement learning has also been introduced to solve incremental learning problems . Xu et al . [ 54 ] proposed to increase convolution filters once a new task arrives and optimize the increased number by reinforcement learning . Gao et al . [ 14 ] proposed an improved version that makes the minimal expansion of the network , reducing memory and computing overheads . Veniat et al . [ 50 ] introduced a modular architecture , where each module represents a different atomic skill , and used the REINFORCE algorithm [ 51 ] to optimize it . Huang et al . [ 21 ] combined reinforcement learning with Net2Net [ 8 ] and designed a NAS-based CIL method . In our work , we also use the REINFORCE algorithm [ 51 ] , but differ in three aspects . First , we are the first to optimize memory allocation for CIL in a reinforced way . Second , we learn the policy functions on generated pseudo CIL tasks , where we can access both past , and future data ( for each incremental phase ) and thus are able to compute the cross-phase ( long-term ) rewards . In contrast , the related work [ 14 , 54 ] could use only current-phase data to estimate a short-term reward . Third , our reinforcement learning has a hierarchical structure that specially fits the nature of the data stream in the CIL settings . 3 Preliminaries . Class-Incremental Learning ( CIL ) usually assumes ( N+1 ) learning phases : an initial phase and N incremental phases during which the number of classes gradually increases till the maximum [ 13 , 18 , 20 , 31 ] . We assume that total memoryM is bounded and fixed for all incremental phases [ 40 ] .M is used to store the exemplars and new coming data as both kinds of data need to be loaded repeatedly during training epochs . In the initial ( 0-th ) phase , data D0 , containing the training samples of C0 classes , are used to learn the initial classification model Θ0 . In the i-th incremental phase , we splitM into two dynamic partitions : the exemplar memoryMold and new data memoryMnew . We select Et as representative samples of the data seen in the t-th phase , and denote total exemplars E0 ∼ Ei−1 shortly as E0 : i−1 . We save E0 : i−1 intoMold and freeMnew . Then , we observe new data that contain Ci new classes . We randomly load new data intoMnew untilMnew is full , and all the other new data are discarded . We denote the loaded new data as Di . Then , we initialize Θi with Θi−1 , and train it using E0 : i−1 ∪ Di . The resulting model Θi will be evaluated with a test set containing all classes observed so far . We repeat this training and testing , and report the average accuracy across all phases . Reinforcement Learning ( RL ) aims to learn an optimal policy function π for an agent interacting in an unknown environment [ 51 , 54 , 59 ] . In the CIL scenario , in each incremental phase , the agent observes the current state si from the environment , and then takes an action ai ( how to allocate memory ) according to the policy function π ( ai|si ) . Subsequently , the environment is updated to a new state si+1 and the reward ri is calculated to optimize the parameters of π ( ai|si ) through back-propagation . Specifically , the learning objective of π ( ai|si ) is to maximize the expected cumulative reward Ri = ∑∞ t=i γ t−irt , where γ ∈ [ 0 , 1 ) is a discounting factor that determines the weights of future rewards . Please note that in our case , the ( N+1 ) -phase CIL task is a finite horizon problem [ 15 , 59 ] , so we remove the discounting factor and use R = ∑N t=0 rt , which is actually the cumulative validation accuracy of all training CIL tasks . In Section 4 , we discuss the proposed RL algorithm for memory allocation and how to generate pseudo tasks for training its policy function .

Summary: This work addresses the problem of memory management in Class Incremental Learning (CLI). In this setup, models store exemplars from old classes and must decide what exemplars to drop to make space to new memories. The authors propose a hierarchical reinforcement learning approach where an agent first chooses how much of the old memory to release to make space for new samples and then it chooses how much of the new space allocate to high-entropy samples and low-entropy samples (they divide new classes in two groups). Experiments show that the the proposed method attains better performance than POD-AANets and LUCIR on CIFAR and ImageNet. In the supplementary material they show that the proposed memory management system also improves the performance of already-existing systems. Additional studies show how policies transfer between datasets, the impact of hierarchical RL, and how old and new memories balance over time.