RMM: Reinforced Memory Management for Class-Incremental Learning

1 Introduction . Ideally , AI systems should be adaptive to ever-changing environments—where the data are continuously observed by sensors . Their models should be capable of learning new concepts from data while maintaining the ability to recognize previous ones . In practice , the systems often have constrained memory budgets because of which most of the historical data have to be abandoned [ 20 ] . However , deep-learning-based AI systems , when continuously updated using new data and limited historical data , often suffer from catastrophic forgetting , as the updates can override knowledge acquired from previous data [ 33 , 34 , 39 ] . To encourage research on the forgetting problem , Rebuffi et al . [ 40 ] defined a standard protocol of class-incremental learning ( CIL ) for image classification , where the training data of different object classes come in phases . In each phase , the classifier is evaluated on all classes observed so far . As the total memory size is limited [ 40 ] , CIL systems abandon the majority of the data and only preserve a small number of exemplars , e.g. , 20 exemplars per class , which will be used for replaying in subsequent phases . Replaying usually happens for multiple epochs [ 13 , 18 , 30 , 40 ] , so both the old 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . class exemplars and new class data need to be stored in the limited memory . Existing CIL methods allocate memory between the old and new classes in an arbitrary and static fashion , e.g. , 20 per old class vs. 1 , 300 per new class for the ImageNet-Full dataset . This causes a serious imbalance between the old and new classes and can exacerbate the problem of catastrophic forgetting . To address this , we propose to learn an optimal memory management policy for each incremental phase with continuously reinforced model performance and call our method reinforced memory management ( RMM ) . Detailed actions include 1 ) allocating the memory between the existing ( old ) and the coming ( new ) data for each phase , and 2 ) specifying the memory for each old class according to its recognition difficulty before abandoning any of its data . To this end , we leverage reinforcement learning [ 26–28 , 51 , 59 ] and design a new policy function to contain two sub-functions that propagate two levels of actions in a hierarchical way . Level-1 function determines how to split memory between the old and new data . Its output action is then inputted into the Level-2 function to determine how to allocate memory for each old class . The overall objective of the function is to maximize the cumulative evaluation accuracy across all incremental phases . However , this is not naturally compatible with the standard protocol of CIL [ 40 ] where neither past nor future data are accessible for evaluation . To tackle this issue , we propose to pre-train the function on pseudo CIL tasks and then adopt it in the learning process of our target task . In principle , we can build such pseudo tasks using any available categorical data , e.g. , the data in the 0-th phase of the target CIL task or the data from another dataset . Even though this is a non-stationary reinforcement learning problem , we can regard the pseudo and target CIL tasks as a sequence of stationary tasks and train the policy function to exploit the dependencies between these consecutive tasks . Such continuous adaptation in non-stationary environments is feasible based on the empirical analysis given in [ 2 ] . Technically , we propose the following method to guarantee the transferability of policy functions between pseudo and target CIL tasks . We take a Level-1 action based on the ratio of the number of new classes to the total number of classes observed so far . A lower ( higher ) ratio will result in weakening the stability ( plasticity ) of the classification model . Then , we take a Level-2 action for each individual class conditioned on both the Level-1 action and the training entropy of that class . A higher entropy denotes a more difficult class , leading to more memory allocated to the class . For evaluation , we conduct extensive CIL experiments by plugging RMM into two top-performing methods ( LUCIR+AANets , POD+AANets ) and testing them on three benchmarks ( CIFAR-100 , ImageNet-Subset , and ImageNet-Full ) . Our results show the clear and consistent superiority of RMM , e.g. , it boosts the state-of-the-art POD+AANets by 3.6 % , 4.4 % , and 1.9 % in the 25-Phase settings of the above benchmarks , respectively . Our technical contribution is three-fold . 1 ) A hierarchical reinforcement learning algorithm called RMM to manage the memory in a way that can be conveniently modified through incremental phases and for different classes . 2 ) A pseudo task generation strategy that requires only in-domain available data ( small-scale ) or cross-domain datasets ( large-scale ) , relieving the data incompatibility between reinforcement learning and class-incremental learning . 3 ) Extensive experiments , visualization , and interpretation for RMM in three CIL benchmarks and using two top models as baselines . 2 Related Work . Incremental Learning [ 22 , 36 , 48 , 53 , 58 ] continuously updates the model using data coming in a sequence of phases . Similar tasks are also referred to as continual learning [ 12 , 32 ] and lifelong learning [ 3 , 9 ] . Recent papers are either task-incremental learning—each phase corresponds to a task ( dataset ) that contains new data of all seen classes [ 7 , 10 , 19 , 29 , 43 , 47 , 57 ] , or classincremental learning ( CIL ) —each phase contains data of a new set of classes , i.e. , classes are unseen [ 4 , 6 , 18 , 25 , 31 , 37 , 40 , 41 , 49 , 52 , 55–57 ] . This paper is concerned with CIL . The key challenge of CIL is the forgetting problem—older classes are forgotten in later phases . Existing methods tackling this can be divided into three categories : memory-based , regularization-based , and network-architecture-based [ 11 , 35 ] . Memory-based methods preserved a small subset of the old class data ( exemplars ) to replay the model on them ( together with the new class data ) , in order to relieve the forgetting of the old classes . Some work [ 18 , 40 ] proposed heuristic strategies to select more representative exemplars from the old class data , and others [ 31 , 47 ] tried to generate exemplars in optimizable frameworks . None of them changed the allocation of memory for different classes , i.e. , all used an arbitrary and static scheme for memory allocation . Regularization-based methods introduce regularization terms in the loss function to consolidate previous knowledge when training the model on new data . The key idea is to enforce predicted label logits [ 29 , 40 ] , features maps [ 13 , 18 ] , or the topology in the feature space [ 49 ] of the new model to be close to that of the previous model . Network-architecture-based methods aim to design “ incremental network architectures ” . Some work [ 45 , 54 ] gradually extended the network capacity for new data , while others proposed to freeze partial network parameters [ 1 , 30 ] to preserve the knowledge of the old classes . Reinforcement Learning defines an agent that needs to decide its actions in an unknown environment by maximizing the expected cumulative reward . It has been widely applied to many optimization problems , e.g. , neural architecture search [ 54 , 59 ] and neural machine translation [ 38 , 46 ] . Reinforcement learning has also been introduced to solve incremental learning problems . Xu et al . [ 54 ] proposed to increase convolution filters once a new task arrives and optimize the increased number by reinforcement learning . Gao et al . [ 14 ] proposed an improved version that makes the minimal expansion of the network , reducing memory and computing overheads . Veniat et al . [ 50 ] introduced a modular architecture , where each module represents a different atomic skill , and used the REINFORCE algorithm [ 51 ] to optimize it . Huang et al . [ 21 ] combined reinforcement learning with Net2Net [ 8 ] and designed a NAS-based CIL method . In our work , we also use the REINFORCE algorithm [ 51 ] , but differ in three aspects . First , we are the first to optimize memory allocation for CIL in a reinforced way . Second , we learn the policy functions on generated pseudo CIL tasks , where we can access both past , and future data ( for each incremental phase ) and thus are able to compute the cross-phase ( long-term ) rewards . In contrast , the related work [ 14 , 54 ] could use only current-phase data to estimate a short-term reward . Third , our reinforcement learning has a hierarchical structure that specially fits the nature of the data stream in the CIL settings . 3 Preliminaries . Class-Incremental Learning ( CIL ) usually assumes ( N+1 ) learning phases : an initial phase and N incremental phases during which the number of classes gradually increases till the maximum [ 13 , 18 , 20 , 31 ] . We assume that total memoryM is bounded and fixed for all incremental phases [ 40 ] .M is used to store the exemplars and new coming data as both kinds of data need to be loaded repeatedly during training epochs . In the initial ( 0-th ) phase , data D0 , containing the training samples of C0 classes , are used to learn the initial classification model Θ0 . In the i-th incremental phase , we splitM into two dynamic partitions : the exemplar memoryMold and new data memoryMnew . We select Et as representative samples of the data seen in the t-th phase , and denote total exemplars E0 ∼ Ei−1 shortly as E0 : i−1 . We save E0 : i−1 intoMold and freeMnew . Then , we observe new data that contain Ci new classes . We randomly load new data intoMnew untilMnew is full , and all the other new data are discarded . We denote the loaded new data as Di . Then , we initialize Θi with Θi−1 , and train it using E0 : i−1 ∪ Di . The resulting model Θi will be evaluated with a test set containing all classes observed so far . We repeat this training and testing , and report the average accuracy across all phases . Reinforcement Learning ( RL ) aims to learn an optimal policy function π for an agent interacting in an unknown environment [ 51 , 54 , 59 ] . In the CIL scenario , in each incremental phase , the agent observes the current state si from the environment , and then takes an action ai ( how to allocate memory ) according to the policy function π ( ai|si ) . Subsequently , the environment is updated to a new state si+1 and the reward ri is calculated to optimize the parameters of π ( ai|si ) through back-propagation . Specifically , the learning objective of π ( ai|si ) is to maximize the expected cumulative reward Ri = ∑∞ t=i γ t−irt , where γ ∈ [ 0 , 1 ) is a discounting factor that determines the weights of future rewards . Please note that in our case , the ( N+1 ) -phase CIL task is a finite horizon problem [ 15 , 59 ] , so we remove the discounting factor and use R = ∑N t=0 rt , which is actually the cumulative validation accuracy of all training CIL tasks . In Section 4 , we discuss the proposed RL algorithm for memory allocation and how to generate pseudo tasks for training its policy function .

Summary: The paper proposes a RL-based memory management policy for CIL problem. They have two-step formulation of policy, which first determines how to allocate the memory to old and new tasks, then determines how many samples to store per each class within the task. A standard RL framework of policy gradient method is used to optimize their policy. Experimental results how that their RMM can further improve the recent state-of-the-art methods, such as LUCIR or PODNet.