RMM: Reinforced Memory Management for Class-Incremental Learning
Year: 2022

Abstract:
Class-Incremental Learning (CIL) [40] trains classifiers under a strict memory budget: in each incremental phase, learning is done for new data, most of which is abandoned to free space for the next phase. The preserved data are exemplars used for replaying. However, existing methods use a static and ad hoc strategy for memory allocation, which is often sub-optimal. In this work, we propose a dynamic memory management strategy that is optimized for the incremental phases and different object classes. We call our method reinforced memory management (RMM), leveraging reinforcement learning. RMM training is not naturally compatible with CIL as the past, and future data are strictly non-accessible during the incremental phases. We solve this by training the policy function of RMM on pseudo CIL tasks, e.g., the tasks built on the data of the 0-th phase, and then applying it to target tasks. RMM propagates two levels of actions: Level-1 determines how to split the memory between old and new classes, and Level-2 allocates memory for each specific class. In essence, it is an optimizable and general method for memory management that can be used in any replaying-based CIL method. For evaluation, we plug RMM into two top-performing baselines (LUCIR+AANets and POD+AANets [30]) and conduct experiments on three benchmarks (CIFAR-100, ImageNet-Subset, and ImageNet-Full). Our results show clear improvements, e.g., boosting POD+AANets by 3.6%, 4.4%, and 1.9% in the 25-Phase settings of the above benchmarks, respectively. The code is available at https://class-il.mpi-inf.mpg.de/rmm/.

heading: 1 Introduction
text: Ideally, AI systems should be adaptive to ever-changing environments—where the data are continuously observed by sensors. Their models should be capable of learning new concepts from data while maintaining the ability to recognize previous ones. In practice, the systems often have constrained memory budgets because of which most of the historical data have to be abandoned [20]. However, deep-learning-based AI systems, when continuously updated using new data and limited historical data, often suffer from catastrophic forgetting, as the updates can override knowledge acquired from previous data [33, 34, 39].
To encourage research on the forgetting problem, Rebuffi et al. [40] defined a standard protocol of class-incremental learning (CIL) for image classification, where the training data of different object classes come in phases. In each phase, the classifier is evaluated on all classes observed so far. As the total memory size is limited [40], CIL systems abandon the majority of the data and only preserve a small number of exemplars, e.g., 20 exemplars per class, which will be used for replaying in subsequent phases. Replaying usually happens for multiple epochs [13, 18, 30, 40], so both the old
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
class exemplars and new class data need to be stored in the limited memory. Existing CIL methods allocate memory between the old and new classes in an arbitrary and static fashion, e.g., 20 per old class vs. 1, 300 per new class for the ImageNet-Full dataset. This causes a serious imbalance between the old and new classes and can exacerbate the problem of catastrophic forgetting.
To address this, we propose to learn an optimal memory management policy for each incremental phase with continuously reinforced model performance and call our method reinforced memory management (RMM). Detailed actions include 1) allocating the memory between the existing (old) and the coming (new) data for each phase, and 2) specifying the memory for each old class according to its recognition difficulty before abandoning any of its data. To this end, we leverage reinforcement learning [26–28, 51, 59] and design a new policy function to contain two sub-functions that propagate two levels of actions in a hierarchical way. Level-1 function determines how to split memory between the old and new data. Its output action is then inputted into the Level-2 function to determine how to allocate memory for each old class. The overall objective of the function is to maximize the cumulative evaluation accuracy across all incremental phases. However, this is not naturally compatible with the standard protocol of CIL [40] where neither past nor future data are accessible for evaluation. To tackle this issue, we propose to pre-train the function on pseudo CIL tasks and then adopt it in the learning process of our target task. In principle, we can build such pseudo tasks using any available categorical data, e.g., the data in the 0-th phase of the target CIL task or the data from another dataset. Even though this is a non-stationary reinforcement learning problem, we can regard the pseudo and target CIL tasks as a sequence of stationary tasks and train the policy function to exploit the dependencies between these consecutive tasks. Such continuous adaptation in non-stationary environments is feasible based on the empirical analysis given in [2].
Technically, we propose the following method to guarantee the transferability of policy functions between pseudo and target CIL tasks. We take a Level-1 action based on the ratio of the number of new classes to the total number of classes observed so far. A lower (higher) ratio will result in weakening the stability (plasticity) of the classification model. Then, we take a Level-2 action for each individual class conditioned on both the Level-1 action and the training entropy of that class. A higher entropy denotes a more difficult class, leading to more memory allocated to the class. For evaluation, we conduct extensive CIL experiments by plugging RMM into two top-performing methods (LUCIR+AANets, POD+AANets) and testing them on three benchmarks (CIFAR-100, ImageNet-Subset, and ImageNet-Full). Our results show the clear and consistent superiority of RMM, e.g., it boosts the state-of-the-art POD+AANets by 3.6%, 4.4%, and 1.9% in the 25-Phase settings of the above benchmarks, respectively.
Our technical contribution is three-fold. 1) A hierarchical reinforcement learning algorithm called RMM to manage the memory in a way that can be conveniently modified through incremental phases
and for different classes. 2) A pseudo task generation strategy that requires only in-domain available data (small-scale) or cross-domain datasets (large-scale), relieving the data incompatibility between reinforcement learning and class-incremental learning. 3) Extensive experiments, visualization, and interpretation for RMM in three CIL benchmarks and using two top models as baselines.

heading: 2 Related Work
text: Incremental Learning [22, 36, 48, 53, 58] continuously updates the model using data coming in a sequence of phases. Similar tasks are also referred to as continual learning [12, 32] and lifelong learning [3, 9]. Recent papers are either task-incremental learning—each phase corresponds to a task (dataset) that contains new data of all seen classes [7, 10, 19, 29, 43, 47, 57], or classincremental learning (CIL)—each phase contains data of a new set of classes, i.e., classes are unseen [4, 6, 18, 25, 31, 37, 40, 41, 49, 52, 55–57]. This paper is concerned with CIL. The key challenge of CIL is the forgetting problem—older classes are forgotten in later phases. Existing methods tackling this can be divided into three categories: memory-based, regularization-based, and network-architecture-based [11, 35]. Memory-based methods preserved a small subset of the old class data (exemplars) to replay the model on them (together with the new class data), in order to relieve the forgetting of the old classes. Some work [18, 40] proposed heuristic strategies to select more representative exemplars from the old class data, and others [31, 47] tried to generate exemplars in optimizable frameworks. None of them changed the allocation of memory for different classes, i.e., all used an arbitrary and static scheme for memory allocation. Regularization-based methods introduce regularization terms in the loss function to consolidate previous knowledge when training the model on new data. The key idea is to enforce predicted label logits [29, 40], features maps [13, 18], or the topology in the feature space [49] of the new model to be close to that of the previous model. Network-architecture-based methods aim to design “incremental network architectures”. Some work [45, 54] gradually extended the network capacity for new data, while others proposed to freeze partial network parameters [1, 30] to preserve the knowledge of the old classes.
Reinforcement Learning defines an agent that needs to decide its actions in an unknown environment by maximizing the expected cumulative reward. It has been widely applied to many optimization problems, e.g., neural architecture search [54, 59] and neural machine translation [38, 46]. Reinforcement learning has also been introduced to solve incremental learning problems. Xu et al. [54] proposed to increase convolution filters once a new task arrives and optimize the increased number by reinforcement learning. Gao et al. [14] proposed an improved version that makes the minimal expansion of the network, reducing memory and computing overheads. Veniat et al. [50] introduced a modular architecture, where each module represents a different atomic skill, and used the REINFORCE algorithm [51] to optimize it. Huang et al. [21] combined reinforcement learning with Net2Net [8] and designed a NAS-based CIL method. In our work, we also use the REINFORCE algorithm [51], but differ in three aspects. First, we are the first to optimize memory allocation for CIL in a reinforced way. Second, we learn the policy functions on generated pseudo CIL tasks, where we can access both past, and future data (for each incremental phase) and thus are able to compute the cross-phase (long-term) rewards. In contrast, the related work [14, 54] could use only current-phase data to estimate a short-term reward. Third, our reinforcement learning has a hierarchical structure that specially fits the nature of the data stream in the CIL settings.

heading: 3 Preliminaries
text: Class-Incremental Learning (CIL) usually assumes (N+1) learning phases: an initial phase and N incremental phases during which the number of classes gradually increases till the maximum [13, 18, 20, 31]. We assume that total memoryM is bounded and fixed for all incremental phases [40].M is used to store the exemplars and new coming data as both kinds of data need to be loaded repeatedly during training epochs. In the initial (0-th) phase, data D0, containing the training samples of C0 classes, are used to learn the initial classification model Θ0. In the i-th incremental phase, we splitM into two dynamic partitions: the exemplar memoryMold and new data memoryMnew. We select Et as representative samples of the data seen in the t-th phase, and denote total exemplars E0 ∼ Ei−1 shortly as E0:i−1. We save E0:i−1 intoMold and freeMnew. Then, we observe new data that contain Ci new classes. We randomly load new data intoMnew untilMnew is full, and all the other new data
are discarded. We denote the loaded new data as Di. Then, we initialize Θi with Θi−1, and train it using E0:i−1 ∪ Di. The resulting model Θi will be evaluated with a test set containing all classes observed so far. We repeat this training and testing, and report the average accuracy across all phases.
Reinforcement Learning (RL) aims to learn an optimal policy function π for an agent interacting in an unknown environment [51, 54, 59]. In the CIL scenario, in each incremental phase, the agent observes the current state si from the environment, and then takes an action ai (how to allocate memory) according to the policy function π(ai|si). Subsequently, the environment is updated to a new state si+1 and the reward ri is calculated to optimize the parameters of π(ai|si) through back-propagation. Specifically, the learning objective of π(ai|si) is to maximize the expected cumulative reward Ri = ∑∞ t=i γ
t−irt, where γ ∈ [0, 1) is a discounting factor that determines the weights of future rewards. Please note that in our case, the (N+1)-phase CIL task is a finite horizon problem [15, 59], so we remove the discounting factor and use R = ∑N t=0 rt, which is actually the cumulative validation accuracy of all training CIL tasks. In Section 4, we discuss the proposed RL algorithm for memory allocation and how to generate pseudo tasks for training its policy function.

heading: 4 Reinforced Memory Management (RMM)
text: Our RMM approach learns policy functions that propagate two levels of actions in a hierarchical way, specially designed for CIL. As illustrated in Figure 1 (b), Level-1 determines the memory split between exemplars and new data, and Level-2 allocates the memory for each individual class. We motivate and introduce the formulation of RMM, including the definitions of states, actions, rewards, and hierarchical policy functions in Section 4.1. In Section 4.2, we detail the steps of creating pseudo CIL tasks on which we learn the policy functions. In Section 4.3, we summarize the algorithm.

heading: 4.1 Formulation
text: In the i-th incremental phase CIL, we manage the memory for two kinds of data: exemplars E0:i−1 and new data Di. For the former, we have access to their images and labels so we can allocate a different memory size to a different class, e.g., based on its recognition difficulty. For the latter, we do not have such access before loading the data (otherwise, causing a violation to the CIL protocol), so we are only able to learn a total memory size, i.e., the memory size for all new classes (and then split it evenly for each individual class). Therefore, the memory management in CIL settings is inherently hierarchical: 1) coarse memory allocation between exemplars and new data; and then 2) fine-grained memory allocation among specific classes. To this end, we modify the standard reinforcement learning into a hierarchical structure.
As illustrated in Figure 2 (a), in the i-th incremental phase of CIL (i.e., the environment), the argent receives a state value si. Level-1 policy πη takes si as the input to produce an action a [1] i ∼ πη(si). a [1] i determines how to split memory between the exemplars and new data. After that, Level-2 policy πφ takes si and a [1] i as inputs to produce the second action a [2] i ∼ πφ(si, a [1] i ) that distributes the exemplar memory for each individual class.
States, defined for our CIL settings, should have two properties. 1) Being transferable between CIL tasks, e.g., from a small-scale CIL task including 50 classes (in total) to a large one including 100 classes. The reason is that we need to transfer the policy functions learned from pseudo CIL tasks (defined in Section 4.2) to the target task. The states, the inputs of policy functions, should also be transferable. 2) Being distinct in each incremental phase. This is to enable the state variable to represent a specific forgetting or data imbalance degree at each different learning phase of the CIL model. To fulfill these properties, we formulate the state in the i-th phase as si = ( Ci∑i−1 t=0 Ct , |Mold||M| ) , where Ci denotes the number of classes in Di,Mold denotes the memory allocated to exemplars E0:i−1, andM is the total memory.
Level-1 Actions. In the 1-st incremental phase, our Level-1 policy function produces an action to allocate the memory for exemplars E0 and new data D1. We denote this action as a[1]1 and assign its value with the ratio of the number of the exemplars |E0| to the memory size |M|, so we have a [1] 1 ∈ (0, 1). In the i-th phase (i ≥ 2), the definition of a [1] i is different to a [1] 1 as it is a relative change over a[1]i−1. Specifically, a [1] i is the ratio of increased (if its value is positive) or decreased (if
negative) memory size ofMold compared to the (i-1)-th phase. Using this definition aims for smooth and continuous memory management. In the formulation, the memory sizes of exemplars E0:i−1 and new data Di are, respectively,
|Mold| = |E0:i−1| = i∑ t=1 a [1] t |M|, |Mnew| = |Di| =
( 1−
i∑ t=1 a [1] t
) |M|. (1)
We set a constrain a[1]i ∈ [−0.1, 0.1] for i ≥ 2. Otherwise, if a [1] i is too big, there are not enough exemplars to fill the memory, as most old-class data has been abandoned. If a[1]i is too small, many exemplars will be permanently deleted in this phase, making it hard or even impossible to adjust Mold back to a high value in the future phases. If ∑i t=1 a [1] t > 1,Mnew will be negative. So, we
force ∑i t=1 a [1] t ≤ 1 by rejection sampling [5], i.e., using πη to output another action until it is feasible to execute. Note that this situation rarely happens in real training, because whenMnew becomes very low, πη tends to produce an action to increase it.
Level-2 Actions. Here, we elaborate on how to get class-specific memory allocation. In the (i−1)-th phase, we split the classes for Di−1 into two groups evenly according to training entropy values: classes with higher values (difficult classes) are in one group and the rest in the other group. Therefore, Level-2 action a[2]i ∈ (0, 1) determines how to split memories between harder and easier classes. During initial experiments, we observed that using two groups already yields improved results and using more groups causes a decrease.
Let MAj and MBj denote the memory allocated for the high-entropy and low-entropy groups, respectively, in the j-th phase (j ≤ i):
|MAj | = a [2] j+1|Ej | = a [2] j+1Cj∑i t=1 Ct |Mold|, |MBj | = (1− a [2] j+1)|Ej | = (1− a[2]j+1)Cj∑i t=1 Ct |Mold|. (2)
Then, we allocate memory evenly to the classes within the group, e.g., if the high-entropy group has 10 classes, each class will have a memory size of 110 |M A j |.
Rewards. The objective of CIL is that the trained model (in any phase) should be efficient to recognize all classes seen so far. It is intuitive and convenient to use the validation accuracy as the reward in each phase. In the i-th phase, the objective of RMM is to maximize the expected cumulative reward, i.e., R = ∑N i=0 ri, where ri denotes the validation accuracy in the i-th phase.
4.2 Optimization
In the CIL protocol, it is impossible to see past or future data in any incremental phase. It is thus not intuitive how to compute cumulative rewards till the last phase. We propose to solve the issue by generating pseudo CIL tasks (where all data are accessible).
Pseudo CIL Tasks should meet two requirements: 1) their training and validation data are fully accessible for computing cumulative rewards, and 2) they have the same format (e.g., the same number of phases) of the target CIL task. Data Sources: For requirement 1, an intuitive solution is to use D0 (available in the 0-th phase). Based on the CIL protocol [13, 18, 20, 30], D0 contains half of the classes of the whole dataset, e.g., 50 classes on CIFAR-100, which supplies enough data to build downsized CIL tasks. When building the tasks, we randomly choose 10% training samples of each
class (from D0) to compose a pseudo validation set (note that we are not allowed to use the original validation set in training). When aiming for larger-scale data in CIL, we can leverage smaller datasets. For example, the pseudo tasks for ImageNet-Subset can be built on the data of CIFAR-100. This is also meaningful to evaluate the transferability of RMM policy functions (discussed in the Ablation Study). Task Generation Protocol is based on requirement 2. If using another dataset, we simply follow its original CIL protocol. If using the data accessed in the 0-th phase (i.e., D0), we can reduce the number of classes (in each phase) by half. For example, for CIFAR-100, we use 50-class D0 to generate a 5-phase pseudo CIL task as follows: loading 25 classes in the 0-th phase, and after that, five classes per phase. To generate another pseudo task, we simply change the order of classes.
Training. We elaborate the steps of learning Level-1 policy πη and Level-2 policy πφ in the following. The goal is to optimize the parameters η and φ by maximizing the expected cumulative reward J(η, φ). We denote any pseudo CIL task and its cumulative reward as T and R, respectively, and have,
J(η, φ) = ET Eπη,πφ [R]. (3)
Policy Gradient Estimation. According to the policy gradient theorem [51], we can compute the gradients for J(η, φ) as follows,
∇η,φJ(η, φ) = ET [ N∑ i=1 Eπη,πφ [∇η,φ log(πη(a [1] i |si)πφ(a [2] i |si, a [1] i ))R] ] . (4)
Following the REINFORCE algorithm [51], we replace the expectations ET [·] and Eπη,πφ [·] with sample averages using the Monte Carlo method [16]. Specifically, in each epoch, we create K pseudo tasks and run each task for Z times, as shown in Figure 3. Thus we can derive the empirical approximation of ∇η,φJ(η, φ) as,
∇η,φJ(η, φ) = 1
ZK K∑ k=1 Z∑ z=1 N∑ i=1 ∇η,φ log(πη(a[1]i |si)πφ(a [2] i |si, a [1] i ))(R k z − b), (5)
where Rkz denotes the z-th reward for the k-th pseudo task Tk, and b denotes the baseline function— the moving average of previous rewards. Using this baseline function is a common trick in RL to reduce the variance of estimated policy gradients [23, 42, 59].
Updating Parameters. We update η and φ in each epoch according to the gradient ascent rule [54, 59]:
η := η + β1∇ηJ(η, φ), φ := φ+ β2∇φJ(η, φ), (6) where β1 and β2 are the learning rates. We iterate this update for m epochs in total.
4.3 Algorithm
Algorithm 1 Learning policy functions in RMM 1: Input: Data D for generating pseudo CIL tasks. 2: Output: Policy functions πη , πφ. 3: Initialize η and φ; 4: for m epochs do 5: for k in 1, ...,K do 6: Create a new pseudo task Tk using D; 7: for z in 1, ..., Z do 8: Initialize classification model Θ0; 9: for i in 0, ..., N do
10: if i ≥ 1 do 11: Observe si and produce a [1] i ∼ πη(si); 12: AllocateMold andMnew using Eq. 1; 13: Produce a[2]i ∼ πφ(a [1] i , si); 14: Allocate {MAj }ij=0 and {MBj }ij=0 using Eq. 2; 15: Update E0:i−1 using herding [40]; 16: Save E0:i−1 inMold and freeMnew; 17: Observe new data and load Di intoMnew randomly; 18: Initialize Θi with Θi−1 and train it using E0:i−1 ∪Di; 19: Compute validation accuracy ri; 20: Compute Rkz = ∑N i=0 ri and update b; 21: Compute∇η,φJ(η, φ) using Eq. 5; 22: Update η and φ using Eq. 6.
Algorithm 1 summarizes the overall training steps of the proposed RMM. There are four loops in the algorithm: 1) we train the RMM agent for m epochs; 2) we create K pseudo CIL tasks in each epoch; 3) we run each pseudo CIL task for Z times; and 4) there are N+1 learning phases each time. Specifically, Line 3 initializes the parameters of policy functions. Line 6 creates the k-th pseudo CIL task. Line 8 initializes the classification model. Lines 10-16 allocate the memory according to the actions produced by RMM policy. Line 17 loads new data. Lines 18-19 train the classification model and compute the accuracy. Line 20 estimates the z-th cumulative reward. Lines 21-22 compute the gradients and update policy functions.

heading: 5 Experiments
text: We evaluate the proposed RMM method on three CIL benchmarks: CIFAR-100 [24], ImageNet-Subset [40], and ImageNet-Full [44], and use two top performing methods LUCIR+AANets and POD+AANets [30] as baselines. Below we introduce the datasets and implementation details (Section 5.1), followed by the experimental results and analyses (Section 5.2).

heading: 5.1 Datasets and Implementation Details
text: Datasets. We use three benchmarks based on two datasets, CIFAR-100 [24] and ImageNet [44], following common settings [13, 18, 40, 30]. CIFAR-100 [24] contains 60, 000 samples of 32× 32 color images from 100 classes. There are 500 training and 100 test samples for each class. ImageNet (ILSVRC 2012) [44] contains around 1.3 million samples of 224 × 224 color images from 1, 000 classes. There are about 1, 300 training and 50 test samples for each class. ImageNet has two CIL settings: ImageNet-Subset is based on a subset of 100 classes; and ImageNet-Full uses the full set of 1, 000 classes. The 100-class data for the ImageNet-Subset are sampled from ImageNet. For the experiments on PODNet [13] and POD-AANets [30], we use the same class orders and hyperparameters as [13]. For the experiments on LUCIR [18] and LUCIR-AANets [30], we use the same class orders and hyperparameters as [18].
Benchmarks. We follow the benchmark protocol used in [13, 18, 30, 31]. Given a dataset, the initial (the 0-th phase) model is trained on the data of half of the classes. Then, it learns the remaining classes evenly in the subsequentN phases. Assume there is an initial phase andN incremental phases in the CIL system. The total number of incremental phases N is set to be 5, 10 or 25 (for each the setting is called “N -phase” setting). At the end of each individual phase, the learned model in each phase is evaluated on the test set containing all seen classes. In the tables, we report average accuracy over all phases and the last-phase accuracy, where the latter indicates the degree of forgetting.
Network Architectures. Following [18, 30, 40, 52], we use a 32-layer ResNet [40] for CIFAR-100 and an 18-layer ResNet [17] for ImageNet. Please note that it is standard to use a shallower ResNet for ImageNet. The 32-layer ResNet consists of an initial convolution layer and three residual blocks (in a single branch). Each block has ten convolution layers with 3× 3 kernels. The number of filters starts from 16 and is doubled every next block. After these three blocks, there is an average-pooling layer to compress the output feature maps to a feature embedding. The 18-layer ResNet follows the standard settings in [17]. We deploy AANets using the same parameters as its original paper [30].
Method CIFAR-100 ImageNet-Subset ImageNet-Full
N=5 10 25 5 10 25 5 10 25
LwF [29] 56.79 53.05 50.44 58.83 53.60 50.16 52.00 47.87 47.49 iCaRL [40] 60.48 56.04 52.07 67.33 62.42 57.04 50.57 48.27 49.44 LUCIR [18] 63.34 62.47 59.69 71.21 68.21 64.15 65.16 62.34 57.37 Mnemonics [31] 64.59 62.59 61.02 72.60 71.66 70.52 65.40 64.02 62.05 PODNet [13] 64.60 63.13 61.96 76.45 74.66 70.15 66.80 64.89 60.28
For policy functions πη and πφ, we use two-layer FC networks. All actions are discretized at 0.1 intervals to reduce the search space and get a tolerable training overhead.
Hyperparameters and Configuration. The training of the classification model Θ exactly follows the uniform setting in [13, 18, 30, 31]. On CIFAR-100 (ImageNet-Subset/Full), we train it for 160 (90) epochs in each phase, and divide the learning rate by 10 after 80 (30) and then after 120 (60) epochs. Then, we fine-tune the model for 20 epochs using only exemplars (including the preserved exemplars of the new data to be used in future phases). We use an SGD optimizer and an ADAM optimizer for the classification model and policy functions, respectively. More details are given in the supplementary.
Memory Budget. There are two popular settings about memory budget in related work. One uses a bounded memory budget with a fixed capacity for all phases [18, 31, 40]. Another one allows the memory budget to grow along with phases [18, 20, 49]. The first one is more strict and thus used as the major setting in our paper (note that the results and analyses using the second setting are given in the supplementary materials). In every benchmark, the total budget of memory depends on the phase number N . For example, on CIFAR-100, the total memory budget is set as 7, 000 samples when N=5 (7, 000 samples = 10 classes/phase × 500 samples/class + 2, 000 samples). Please note that 2, 000 is a bounded memory budget allocated since the 0-th phase for saving exemplars. More clarifications about memory budget are given in the supplementary. For fair comparison, we re-implement related methods and report the results in Table 1 if their original results (in the respective papers) were obtained in a different setting of memory budget.

heading: 5.2 Results and Analyses
text: Table 1 presents the results of two state-of-the-art methods (LUCIR+AANets and POD+AANets [30]) w/ and w/o our RMM plugged in, and some recent CIL work [13, 18, 29, 31, 40]. Table 2 shows the ablation study in 6 settings. Figure 4 plots the changes of the average number of exemplars per old/new class for the incremental phases.
Comparing to the State-of-the-Art. From Table 1, we make the following observations. 1) Our RMM consistently improves the two top baselines LUCIR+AANets and POD+AANets [30] in all settings. E.g., LUCIR-AANets w/ RMM and POD-AANets w/ RMM respectively get 2.7% and 3.1% average improvements on the ImageNet-Subset. 2) Our POD-AANets w/ RMM achieves the best performances. Interestingly, we find that our RMM can boost performance more when the number of phases is larger. For example, when N=25, RMM improves POD-AANets by 3.6% and 4.4% on CIFAR-100 and ImageNet-Subset, respectively. These two numbers are 2.3% and 2.1% when N=5.
This indicates that the superiority of our RMM is more obvious in challenging settings (where the forgetting problem is more serious due to the more frequent model re-training through phases).
Ablation Settings. Table 2 shows the results of our ablation study. Row 1 is for the baseline method POD-AANets [30]. Row 2 is for learning only Level-1 policy πη (where each class gets an even split of the memory). Row 3 is for learning both Level-1 policy πη and Level-2 policy πφ in our proposed hierarchical method, and its results are used in Table 1 as “ours”. Row 4 is for using Policy functions Transferred from another dataset (T.P.), which means on the target CIL dataset there is no training of RMM. Here, for CIFAR-100, we use the policy functions learned on ImageNet-Subset, and vice versa. On the last two rows, we show two oracle settings. Row 5 is the upper bound that assumes all past and future data are accessible during training RMM on the target CIL dataset. Row 6 is for using cross-validation (i.e., all past, future, and validation data are accessible) to find the best fixed memory split between old and new class data, e.g., oldnew = 0.7 is chosen and then used in all phases. The details of chosen split rates are given in the supplementary materials.
Ablation Results. Hierarchical: In Table 2, when comparing Row 2 to Row 1, it is clear that leveraging reinforcement learning yields better results as it can derive adaptive memory allocation between old and new data. Using class-specific memory management further increases the model performance (i.e., comparing Row 3 to Row 2), even though we divide the classes into only two groups. T.P. (Transferred Policy functions): Comparing Row 4 to Row 3, we can see that using transferred policy functions (trained on another dataset) achieves comparable performance, and Row 4 does not require any reinforcement learning on the target CIL dataset. Oracle: Comparing Row 3 to Row 5, we see that learning RMM on pseudo CIL tasks is comparable to the upper bound case where all training and validate data are accessible, given the fact that the latter needs higher computational overhead and violates the standard CIL protocol. Row 6 results are consistently lower than ours in Row 3, although cross-validation has access to all past, future, and validation data.
Allocated Memory. Figure 4 shows the change of the average number of samples per class in three ablative settings. Solid and dashed lines represent old and new classes, respectively. From the plots, we have two observations. 1) Learning RMM on the pseudo or target CIL tasks (green and orange lines), we can obtain similar memory management results (i.e., actions). This means the learned policy is transferrable in non-stationary continuous environments. This matches the conclusion of continuous adaptation in [2]. 2) Using our RMM method achieved more balanced memory sizes between exemplars and new data. For example, in the 1-st phase of the 5-phase setting, “UpperBound
1 2 3 4 5 phase (N=5)
RL” and “Two-level RL” allocate around 100 samples for both exemplars and new data. While the baseline setting has 40 and 500 samples for them, respectively. It thus addresses the data imbalance problem for CIL in a learnable way.

heading: 6 Conclusions
text: We propose the reinforced memory management (RMM) method specially for tackling CIL tasks. The hierarchical reinforcement learning (RL) framework (two levels) in RMM is capable of making more adaptive memory allocation actions than using standard RL (one level). Using the generated pseudo tasks in RMM solves the issue of data incompatibility between CIL and RL. Corresponding experimental results show that the policy trained on these pseudo tasks can be directly applied to target tasks without any computational overhead. Our overall method of RMM is generic, and its trained policy (with or without using an in-domain dataset) can be easily incorporated into exemplar replaying-based CIL methods to boost performance.
Limitations and Societal Impact
We analyse the limitations and potential negative societal impact in the following three aspects.
• Complexity. Training RMM takes an additional time cost. According to Algorithm 1, the cost is O(mKZ) times higher than the time used for the target CIL task. However, the training of RMM policy is offline and can use a different dataset (see Table 2) — RMM pre-learns a robust policy from synthesized pseudo tasks and can be directly applied for memory management in real CIL tasks. The overhead of applying this policy is very little, e.g., 0.63% and 1.12% of the total training time respectively on CIFAR-100 and ImageNet (Subset and Full), taking POD+AANets as the baseline.
• Technical assumptions. We build the framework of RMM based on a series of technical assumptions, which might not directly hold for all real-world continual-learning applications. When applying our method to mission-critical problems, particular care is required when modeling the system.
• Privacy issues. Keeping the old class exemplars has the issue of data privacy. This calls for future research that explicitly forgets or mitigates the identifiable feature of the data.
Acknowledgments and Disclosure of Funding
This research was supported by A*STAR under its AME YIRG Grant (Project No. A20E6c0101), Alibaba Innovative Research (AIR) programme, and Max Planck Institute for Informatics.

Review summaries:
The paper proposes a RL-based memory management policy for CIL problem. They have two-step formulation of policy, which first determines how to allocate the memory to old and new tasks, then determines how many samples to store per each class within the task. A standard RL framework of policy gradient method is used to optimize their policy. Experimental results how that their RMM can further improve the recent state-of-the-art methods, such as LUCIR or PODNet. 

The paper proposes a new memory sampling strategy which replaces random/herding. The strategy consists of two parts: first level determining the distribution of #samples of each class in memory and second level determining samples to be selected for every class. This is learnt via RL policy. The sampler is trained by emulating a pseudo-CL problem with the data available in the pretraining phase-- where there are an equal number of tasks made as expected in the CL setup to enable transfer. Given a sampling configuration, a CL method trains the model and the performance on the pseudo-CL test set is given as reward to the RL algorithm. This trains the policy components, which optimize sample size and samples per class and train CL. Then, the best policy is used to create the memory which is used to train the actual CL problem with state-of-the-art continual learning approaches. This memory selection achieves better performance than previously used memories across CIFAR100, Imagenet100 and Imagenet1000 datasets.

This work addresses the problem of memory management in Class Incremental Learning (CLI). In this setup, models store exemplars from old classes and must decide what exemplars to drop to make space to new memories. The authors propose a hierarchical reinforcement learning approach where an agent first chooses how much of the old memory to release to make space for new samples and then it chooses how much of the new space allocate to high-entropy samples and low-entropy samples (they divide new classes in two groups). Experiments show that the the proposed method attains better performance than POD-AANets and LUCIR on CIFAR and ImageNet. In the supplementary material they show that the proposed memory management system also improves the performance of already-existing systems. Additional studies show how policies transfer between datasets, the impact of hierarchical RL, and how old and new memories balance over time. 

A common approach in class-incremental learning is to retain a fixed set of examples of the old classes to help avoid forgetting. The paper presents an approach to dynamically retain exemplars for the class-incremental learning problem. Reinforcement learning is used for dynamic memory management.