RMM: Reinforced Memory Management for Class-Incremental Learning
Authors: Yaoyao Liu
Bernt Schiele
Qianru Sun
Venue: NeurIPS 2021 Poster (2021)
Keywords: incremental learning
class-incremental learning
reinforcement learning

Abstract: Class-Incremental Learning (CIL) [38] trains classifiers under a strict memory budget: in each incremental phase, learning is done for new data, most of which is abandoned to free space for the next phase. The preserved data are exemplars used for replaying. However, existing methods use a static and ad hoc strategy for memory allocation, which is often sub-optimal. In this work, we propose a dynamic memory management strategy that is optimized for the incremental phases and different object classes. We call our method reinforced memory management (RMM), leveraging reinforcement learning. RMM training is not naturally compatible with CIL as the past, and future data are strictly non-accessible during the incremental phases. We solve this by training the policy function of RMM on pseudo CIL tasks, e.g., the tasks built on the data of the zeroth phase, and then applying it to target tasks. RMM propagates two levels of actions: Level-1 determines how to split the memory between old and new classes, and Level-2 allocates memory for each specific class. In essence, it is an optimizable and general method for memory management that can be used in any replaying-based CIL method. For evaluation, we plug RMM into two top-performing baselines (LUCIR+AANets and POD+AANets [28]) and conduct experiments on three benchmarks (CIFAR-100, ImageNet-Subset, and ImageNet-Full). Our results show clear improvements, e.g., boosting POD+AANets by 3.6%, 4.4%, and 1.9% in the 25-Phase settings of the above benchmarks, respectively.  The code is available at https://class-il.mpi-inf.mpg.de/rmm/.

RMM: Reinforced Memory Management for Class-Incremental Learning
Yaoyao Liu
Bernt Schiele'
Qianru Sun?
Max Planck Institute Informatics, Saarland Informatics Campus
School of Computing and Information Systems, Singapore Management University
(yaoyao. liu, schiele)@mpi @mpi-inf mpg.de qianrusun@emu. edu. ag
Abstract
Inssemnncremttll Learning (CIL) 40 trains classifiers under a strict memory budget: in each incremental phase, learning is done for new data, most of which is abandoned to free space for the next phase. The preserved data are exemplars used for replaying. However, existing methods for memory allocation, which often sub-optimal. In this work, we propose a dynamic memory management strategg that is optimized for the incremental phases and different object classes. We call our method reinforced memory management (RMM), leveraging reinforcement learning. RMM training is not naturally com- patible with CIL as the past, and future data are strictly non-accessible during the incremental phases. We solve this by training the policy function of RMM pseudo CIL tasks, e.g., the tasks built on the data of the 0-th phase, and then applying target tasks. RMM propagates two levels of actions: Level-1 de- termines how to split the memory between old and new classes, and Level-2 allocates memory for each specific class. essence, it is an optimizable and general method for memory management that can be used in any replaying replaying-based CII. method. For evaluation, we plug RMM into two top-performing baselines (LUCIR+AANets and POD+AANets (30) and conduct experiments benchmarks (CIFAR- 100, ImageNet- Subset, and ImageNet-Full), Our results show clear improvements, e.g., boosting POD+AANets by 3.6%, 4.4%, and 1.9% in the 25-Phase settings of the above benchmarks, respectively. The code is available at https://class- wttsss.1..ppppii mpi-inf mpg. de/ /rermm/
static and ad hoc strategy
three
Introduction
Ideally, Al systems should be adaptive to ever-changing environments- the data are continu- ously observed by sensors. Their models should be capable learning new concepts from data while maintaining the ability to recognize previous ones. In practice, the systems often have constrained memory budgets because of which most the historical data have be abandoned [20]. However, deep-learning- bbased AI systems, when continuously updated using new data and limited historical data, often suffer from catastrophic forgetting, as the updates can override knowledge acquired from
previous data 33 34 39].
To encourage research on the forgetting problem, Rebuffi et al. 40 defined standard protocol of class-incremental learning (CIL.) for image classification, where the training data of different object classes come phases. In each phase, the classifier evaluated on all classes observed so far. As the total memory size is limited [40]. CIL systems abandon the majority of the data and only preserve a small number of exemplars, e.g., 20 exemplars per class, which will be used for replaying in subsequent phases. Replaying usually happens for multiple epochs 13
40 so both the old
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Phose
loud
000.0 00000
train
model
Phose
load
10000
000
train
model
states
randomly
Level-1 policy
Level-1 action
exemplar selection
old/new allocation
0000 2000
Level-2 policy
Level-2 action
class-specific allocation & exemplar selection
Phase i+1
load
train
Phose i+1
load
train
model
model
randomly
(a) Bascline
(b) Reinforeed Memory Management (ours)
Figure I: (a) Existing CIL methods [8
allocate memory between old and new classes in an arbitrary and frozen way, causing the data imbalance between old and new classes and exacerbating the catastrophic forgetting of old knowledge in the learned model. (b) Our proposed method- Reinforced Memory Management (RMM)- able to learn the optimal and class-specific memory sizes in different incremental phases. Please note we use orange, blue, and green dots to denote the samples observed in the (i-1)-th i-th, and (i+1)-th phases, respectively.
30
40
class exemplars and new class data need to be stored in the limited memory. Existing CIL methods allocate memory between the old and new classes in an arbitrary and static fashion, e.g., 20 per old class vs. 1, 300 per new class for the ImageNet-Füu Full dataset. This causes a serious imbalance between the old and new classes and can exacerbate the problem of catastrophic forgetting.
To address this, we propose to learn an optimal memory management policy for each incremental phase with continuously reinforced model performance and call our method reinforced memory management (RMM). Detailed actions include 1) allocating the memory between the existing (old) and the coming (new) data for each phase, and 2) specifying the memory for each old class according to its recognition difficulty before abandoning any of its data. To this end, we leverage reinforcement learning (2-228) 51 59] and design a new policy function to contain two levels of actions in a hierarchical way. Level-1 function determines how to split memory between the old and new data. Its output action is then inputted into the Level-2 function to determine how to allocate memory for each old class. The overall objective of the function is to maximize the cumulative evaluation accuracy across all incremental phases. However, this not naturally compatible with the standard protocol of CIL [40] where neither past nor future data are accessible for evaluation. To tackle this issue, we propose to pre-train the function on pseudo CIL tasks and then adopt it in the learning process of our target task. In principle, we can build such pseudo tasks using any available categorical data, c.g., the data in the 0-th phase of the target CIL task or the data from another dataset. Even though this is non-stationary reinforcement learning problem, can regard the pseudo and target CIL tasks as a sequence of stationary tasks and train the policy function to exploit the dependencies between these consecutive tasks. Such continuous adaptation in non-stationary environments is feasible based on the empirical analysis given in (2].
sub-functions that propagate
Technically, we propose the following method to guarantee the transferability of policy functions between pseudo and target CIL tasks. We take a Level-1 action based on the ratio of the number of new classes to the total number of classes observed so far. A lower (higher) ratio will result in weakening the stability (plasticity) of the classification model. Then, we take a Level-2 action for each individual class conditioned on both the Level-1 action and the training entropy of that class. A higher entropy denotes a more difficult class, leading to more memory allocated to the class. For evaluation, we conduct extensive CIL experiments by plugging RMM into two top-performing methods (LUCIR+AANet, POD+AANets) and testing them on three benchmarks (CIFAR-100) ImageNet-Subsee and ImageNet-Fulld Full). Our results show the clear and consistent superiority of RMM, e.g., boosts the state-of-the POD+AANets by 3.6%, 4.4%, and 1.9% in the 25-Phase settings of the above benchmarks, respectively.
Our technical contribution is three-fold. 1) A hierarchical reinforcement learning algorithm called RMM to manage the memory in way that can be conveniently modified through incremental phases

---

and for different classes. 2) A pseudo task generation strategy that requires only in in-domain available data (small- -cale) or cross-domain datasets (large-scale), relieving the data incompatibllity between reinforcement learning and class-incremental learning. 3) Extensive experiments, visualization, and interpretation for RMM in three CIL benchmarks and using two top models as baselines.
Related Work
Incremental Learning 22 53 58 continuously updates the model using data coming in
sequence of phases. Similar tasks are also referred to as continual learning [12
32 and lifelong
learning [ 9 Recent papers are either task-incremental learning each phase corresponds to
57 or class- incremental learning (CIL) each phase contains data of a new set of classes, i.e., classes are 52 (7) This paper is concerned with CIL. The key challenge of CIL is the forgetting problem older classes are forgotten in later phases. Existing methods tackling this can be divided into three categories: memory- -aandd, regularization-batt basedd, and network-architer architccture--baseeee
task (dataset) that contains new data of all seen classes 77 10 19 29
43
47
unseen 4 6 18, 25 31 37
40 41
49
[TT 33]. Memory-based methods preserved a small subset of the old class data (exemplars) to replay the model on them (together with the new class data), in order to relieve the forgetting of the old classes. Some work [18 40) proposed heuristic strategies to select more representative exemplars from the old class data, and others [31 47 tried to generate exemplars in optimizable frameworks. None of them changed the allocation of memory for different classes, i.c., all used an arbitrary and static scheme for memory allocation. Regularization- methods introduce regularization terms in the loss function to consolidate previous knowledge when training the model on new data. The key idea is to enforce predicted label logits [29 40).
features maps [3] 18), or the topology in the feature space [49] of the new model to be close that of the previous model. architectures". Some work 45 54] gradually extended the network capacity for new data, while others proposed freeze partial network parameters classes.
methods aim to design "incremental network
30 preserve the knowledge of the old
Reinforcement Learning defines an agent that needs to decide its actions in an unknown environ- ment by maximizing the expected cumulative reward. It has been widely applied to many optimization problems, e.g., neural architecture search [54] [5] and neural machine translation [38 46). Rein- forcement learning has also been introduced to solve incremental learning problems. Xu et al. [54]
proposed to increase convolution filters once a new task arrives and optimize the increased number by reinforcement learning. Gao et al. [14] proposed an improved version that makes the minimal expansion of the network, reducing memory and computing overheads. Veniat et al. [50] introduced modular architecture, where each module represents different atomie skill, and used the RE- INFORCE algorithm [51] to optimize it. Huang et al. (1 combined reinforcement learning with Net2Net [ and designed a NASS CIL method. In work, we also the REINFORCE algorithm [5I]. but differ in three aspects. First, we are the first to optimize memory allocation for CIL in reinforced way. Second, we learn the policy functions on generated pseudo CIL tasks, where we can access both past, and future data (for each incremental phase) and thus are able to compute the cross-phase (long-term) rewards. In contrast, the related work [4 [44 could use only current-phase data to estimate short-term reward. Third, our reinforcement learning has a hierarchical structure that specially fits the nature of the data stream in the CIL settings.
3 Preliminaries
Class-nncremental Learning (CIL) usually assumes (N+1) learning phases: an initial phase and N
incremental phases during which the number of classes gradually increases till the maximum [13] 18
(I]. We assume that total memory M is bounded and fixed for all incremental phases 40]. MM is
used to store the exemplars and new coming data as both kinds of data need to be loaded repeatedly
during training epochs. In the initial (0-th) phase, data Do. containing the training samples of Co classes, are used to learn the initial classification model Oy. In the i-th incremental phase, we split M into two dynamic partitions: the exemplar memory Mald and new data memory Mnew- We select E, as representative samples of the data seen in the 1-th phase, and denote total exemplars E E shortly as En--1- We save E into Moid and free Maew- Then, we observe new data that contain C, new classes. We randomly load new data into Mnew until Mnew is full, and all the other new data
3

---

are discarded. We denote the loaded new data as D. Then, we initialize θ with O;- and train it using E D. The resulting model O will be evaluated with a test set containing all classes observed so far. We repeat this training and testing, and report the average accuracy across all phases.
Reinforcement Learning (RL) aims to learn an optimal policy function x for an agent interacting in an unknown environment [ST 54 59]. In the CIL scenario, in each incremental phase, the agent observes the current state 8; from the environment, and then takes an action 0g (how to allocate memory) according to the policy function [s), Subsequently, the environment is updated to a new state 8;+1 and the reward r; is calculated to optimize the parameters of π T(a (sy) through back-propane propppoopagttion Specifically, the learning objective of T(a;(s;) is to maximize the expected cumulative reward R. = >i= "r,"r; where 7 E [0, 1) is a discounting factor that determines the weights of future rewards. Please note that in case, the (N+1 )-phase CII. task is finite horizon problem 59), so we remove the discounting factor and use R Σ cumulative validation accuracy of all training CIL tasks. In Section 4 we discuss the proposed RL algorithm for memory allocation and how to generate pseudo tasks for training its policy function.
t== Pt, which actually the
4 Reinforced Memory Management (RMM)
Our RMM approach learns policy functions that propagate two levels of actions in a hierarchical way. specially designed for CIL. As illustrated in Figure (b). Level-1 determines the memory split between exemplars and new data, and Level-2 allocates the memory for each individual class. We motivate and introduce the formulation of RMM, including the definitions of states, actions, rewards,
and hierarchical policy functions in Section 4.1 In Section 4.2 we detail the steps of creating pseudo we summarize the algorithm.
CIL tasks on which we learn the policy functions. In Section
4.1 Formulation
In the i-th incremental phase CIL. we manage the memory for two kinds of data: exemplars E and new data D,. For the former, have access to their images and labels so can allocate different memory size to a different class, e.g., based on its recognition difficulty. For the latter, we do not have such access before loading the data (otherwise, causing violation the CII, protocol), so we are only able to learn a total memory size, i.e., the memory size for all new classes (and then split it evenly for each individual class). Therefore, the memory management in CIL settings is inherently hierarchical: 1) coarse memory allocation between exemplars and new data; and then 2) fine-grained memory allocation among specific classes. To this end, we modify the standard reinforcement learning into a hierarchical structure.
As illustrated in Figure (a), in the i-th incremental phase of CIL (i.e., the environment), the argent receives a state value 8, Level-1 policy Ty takes 8; as the input to produce an action a!" ~ Ty (s:). a: !I determines how to split memory between the exemplars and new data. After that, Level-2 policy
(!l) To takes 8; and !!! as inputs to produce the second action a: Ta(si, a that distributes the exemplar memory for each individual class.
States, defined for our CIL settings, should have two properties. 1) Being transferable between CIL tasks, e.g., from a small-scale CIL task including 50 classes (in total) to a large one including 100 classes. The reason is that we need to transfer the policy functions learned from pseudo CIL tasks (defined in Section ( to the target task. The states, the inputs of policy functions, should also be transferable. 2) Being distinct in each incremental phase. This is to enable the state variable to represent a specific forgetting or data imbalance degree each different learning phase of the CIL
model. To fulfill these properties, formulate the state in the i-th phase as s; = (5 where C2 denotes the number of classes in D;, Mold denotes the memory allocated to exemplars Eu--1, and M is the total memory.
[Maia]] ) C. [M]
Level-1 Actions. In the 1-st incremental phase, our Level-1 1 policy function produces action to allocate the memory for exemplars E, and new data Dj. We denote this action as a!") and assign its value with the ratio of the number of the exemplars |Eo] to the memory size [M]. so we have aj
[1] (0, 1). the i-th phase (i 2 2), the definition of at is different to att as it is a relative change over [1] Specifically, a; is the ratio of increased (if its value is positive) decreased (if

---

Agent
Observe $1
State 4i
Observe
Take Actions "4
Pseudo CIL Task T
Pseudo CIL Task Ta
Level-1 Palicy Ky
Level-1 Action df
Environment:
Environment
Cumulative Reward R
Take
Agent
Action
Observe
Take Actions d/
Level-2 Policy Ripi
Level-2 Action of"
Take
Action
(a) The i-th incremental phase the k-th pseudo CIL task
(b) The k-th pseudo CIL task
Figure 2: (a) In the i-th phase of the k-th pseudo CIL task, Level-1 policy Th takes 8; as the input, and produces action a!" Level-2 policy T takes and a;" the inputs, then produces action a; (b) For the k-th pseudo CIL task, we allocate memory for N times (i.c., in N phases) using the policies ny and TT. and compute the cumulative reward R.
negative) memory size of Mood compared to the (i-1)-th phase. Using this definition aims for smooth and continuous memory management. In the formulation, the memory sizes of exemplars Eq.i-1 i and new data D, are, respectively,
|Moaal = 1Ep.
[1 a
MM
[Mnew| = |D,| 1
1
Σ
[1] 0
IM
(1)
t=1
We set constrain a E (-1. for 2 2. Otherwise, if af" is too big, there are not enough exemplars to fill the memory, as most old-class data has been abandoned. a!! is too small, many exemplars will be permanently deleted in this phase, making it hard even impossible adjust Mola back to a high value in the future phases. If Ei= a!" 1, Mnew will be negative. So, we force E af < 1 by rejection sampling [5] i.e., using n, output another action until it is feasible to execute. Note that this situation rarely happens in real training, because when Maew becomes very low, T. tends to produce an action to increase it.
Level-2 Actions. Here, we elaborate on how to get class-specific memory allocation. In the (i (i-1) 1)-th phase, we split the classes for D.-1 into two groups evenly according to training entropy values: classes with higher values (difficult classes) are in one group and the rest in the other group. Therefore, Level-2 action a;") (0, 1) determines how split memories between harder and easier classes. During initial experiments, we observed that using two groups already yields improved results and using more groups causes decrease.
Let MA and MB denote the memory allocated for the high-entropy and low-entropy groups, respectively, in the j-th phase (j < i):
I = a (1 a 1 Motal-
2 |M,1 - 11
C Maial, C. C.
Ei C.
Then, we allocate memory evenly to the classes within the group, e.g., if the high-entropy group has 10 classes, each class will have memory size of 3 |M
Rewards. The objective of CIL is that the trained model (in any phase) should be efficient to recognize all classes seen so far. is intuitive and convenient use the validation accuracy as the reward in each phase. In the i-th phase, the objective of RMM is to maximize the expected cumulative reward, i.c., R Eizo Ti, where denotes the validation accuracy in the i-th phase.
5

---

4.2 Optimization
In the CIL protocol, impossible see past or future data in any incremental phase. It is thus not intuitive how to compute cumulative rewards till the last phase. We propose to solve the issue by generating pseudo CII. tasks (where all data are accessible).
Pseudo CIL Task Ti
Cumulative Reward R
Agent
Pseudo CIL Task Tk
Cumulative Reward R
Pseudo CIL Tasks should meet two requirements: 1) their training and validation data are fully cessible for computing cunulative rewards, and 2) they have the same format (e.g., the same number of phases) the target CII. task. Data Sources: For requirement 1, an intuitive solution is use Do (available in the 0-th phase). Based on the CIL protocol [13 18 20 30), Do contains half of the classes of the whole dataset, c.g., 50 classes on CIFAR-100, which supplies enough data build downsized CIL tasks. When building the tasks, we randomly choose 10% training samples of each class (from Do) to compose pseudo validation set (note that we are not allowed to use the original validation in training). When aiming for larger-scale data CIL, we can leverage smaller datasets. For example, the pseudo tasks for ImageNet-Subset can be built on the data of CIFAR- 100. This is also meaningful to evaluate the transferability of RMM policy functions (discussed in the Ablation Study). Task Generation Protocol is based on requirement 2. If using another dataset, we simply follow its original CIL protocol. using the data accessed the O-th phase (i.e., Do), we can reduce the number of classes (in each phase) by half. For example, for CIFAR-100, we use 50-class Do to generate a 5-phase pseudo CIL task as follows: loading 25 classes the 0-th phase, and after that, five classes per phase. To generate another pseudo task, simply change the order of classes.
xZ
Update
Vg. J(7 ②)
Figure 3: Updating and Φi in one epoch. To get stable gradients for J [[n, Φ), we create K different pseudo CII. tasks, and run each task for Z times.
Training. We elaborate the steps of learning Level-1 policy F and Level-2 policy T& the following. The goal is to optimize the parameters and é by maximizing the expected cumulative reward [(,, Φ). We denote any pseudo CII. task and its cumulative reward as and R, respectively, and have,
J(n の) Ey E,, (R).
Policy Gradient Estimation. According to the policy gradient theorem gradients for [[n, () as follows,
51 we can compute the
N
V no Tn, = ET
Ban V Nn log(n,(( (1 (8)) ( a. ))]
(4)
Following the REINFORCE algorithm [5I], we replace the expectations Ey [+] and Ex,, s, with sample averages using the Monte Carlo method (16). Specifically, in each epoch, we create K pseudo tasks and run each task for Z times, shown in Figure Thus we can derive the empirical approximation of V Vn (n,. ②) as,
K
Z
N
V (n.
1
V n. log(TT, |s)TT( (2) 1l1 at ( b),
(5)
ZK
where Rht denotes the 2-th reward for the k-th pseudo task Tk, and b denotes the baseline function the moving average of previous rewards. Using this baseline function is common trick in RL to reduce the variance of estimated policy gradients (23 42 559].
Updating Parameters. We update y) and ő in each epoch according to the gradient ascent rule 54
59
7 := + V, Φ). o = Φ
Φ),
(6)
where 31 and 3g are the learning rates. We iterate this update for m epochs in total.

---

4.3 Algorithm
Algorithm| D summmarizes the overall training steps of the proposed RMM. There are four loops in the algorithm: 1) we train the RMM agent for epochs; 2) we create K pseudo CIL tasks in each epoch; 3) we run each pseudo CIL task for Z times; and 4) there are N+1 learning phases each time. Specifi- cally, Line initializes the parameters of pol- icy functions. Line 6 creates the k-th pseudo CIL task. Line initializes the classifica- tion model. Lines 10-16 allocate the memory according the actions produced by RMM policy. Line 17 loads new data. Lines 18-19 train the classification model and compute the accuracy. Line 20 estimates the z-th cu- mulative reward. Lines 21-22 compute the gradients and update policy functions.
Algorithm Learning policy functions in RMM
1: Input: Data D for generating pscudo CIL tasks.
Output: Policy functions Typ Tg.
Initialize 7 and db:
for m epochs do
for in 1,.., do Create a pseudo task Tx using D; for in 1, ,..,, Z do Initialize classification model Og:
8:
9:
for i in O,..,, .N do
10:
i 2 1 do
11:
Observe 8; and produce a; (,
12:
Allocate Mold and Muew using Eq.
Produce afth xx(aa s;):
13:
14:
Allocate (M and {M 1j-0 using Eq.
15:
Update Eoi-1 using herding :0:
16:
Save Eou-1 in Mold and free MMew;
Experiments
17:
Observe new data and load D, into M randomly;
18:
Initialize O, with 0;-1 and train it using E U U,;
We evaluate the proposed RMM method on three CIL benchmarks: CIFAR-100 (44).
19:
Compute validation accuracy ry;
20:
Compute R Ee F2 and update b;
ImageNet-Subset [40]. and ImageNet-Full
44 and use two top performing methods
21:
Compute V, Vn, () using Eq.
LUCIR+AANets and POD+AANets [30 as
22: Update v and & using Eq.
baselines. Below we introduce the datasets
and implementation details (Section 5.1) followed by the experimental results and analyses (Sec-
tion 5.2
5.1 Datasets and Implementation Details
Datasets. We use three benchmarks based on two datasets, 100 [24] and ImageNet [44]. following common settings [13 18 40 30J. CIFAR-100 [24] contains 60, 000 samples of 32 32 color images from 100 classes. There are 500 training and 100 test samples for each class. ImageNet
44 contains around 1.3 million samples of 224 x 224 color images from 000 classes. There are about 1, 300 training and 50 test samples for each class. ImageNet has two CIL settings: ImageNet-Subbet is based on a subset of 100 classes; and ImageNet- Full uses the full set of 1, 000 classes. The 100-class data for the ImageNet-Subset are sampled from ImageNet. For the experiments on PODNet 133 and POD-AAI AANets (3) we use the same class orders and hyperparameters as 13. For the experiments on LUCIR [18 and LUCIR-AANets 30), we use the same class orders and hyperparameters as [8]
(ILSVRC
31.. Given dataset, the initial (the 0-th phase) model is trained on the data of half of the classes. Then, learns the remaining classes evenly in the subsequent N phases. Assume there is initial phase and N incremental phases in the CIL system. The total number of incremental phases N is set to be 5, 10 or 25 (for each the setting is called "N-phase" setting). At the end of each individual phase, the learned model in each phase is evaluated on the test set containing all seen classes In the tables, we report average accuracy over all phases and the last-phase accuracy, where the latter indicates the degree of forgetting.
Benchmarks. We follow the benchmark protocol used in 13 18 30
we use 32-layer ResNet 40 for CIFAR- 100 and an 18-layer ResNet [17] for IetgeNet. Please note that it standard to use a shallower ResNet for ImageNet. The 32-layer ResNet consists of an initial convolution layer and three residual blocks (in a single branch). Each block has ten convolution layers with 3 x 3 kernels. The number of filters starts from 16 and is doubled every next block. After these three blocks, there is an average-pooling layer to compress the output feature maps to a feature embedding. The 18-layer ResNet follows the standard settings in [17]. We deploy AANets using the same parameters its original paper [30].
Network Architectures. Following 18
30 40
7

---

CIFAR-100
ImageNet- Subsrr
ImageNet Full
Method
N=
10
25
5
10
25
5
10
25
LwF 29
56.79
53.05
50.44
58.83
53.60
50.16
52.00
47.87
47.49
iCaRI. 40
60.48
56.04
52.07
67.33
62.42
57.04
50.57
48.27
49.44
LUCIR 18
63.34
62.47
59.69
71.21
68.21
64.15
65.16
62.34
57.37
Mnemonics 31
64.59
62.59
61.02
72.60
71.66
70.52
65.40
64.02
62.05
PODNet 13
64.60
63.13
61.96
76.45
74.66
70.15
66.80
64.89
60.28
LUCIR -ANett 30]
66.88
65.53
63,92
72.80
69.71
68.07
65.31
62,99
61.21
w/ RMM (ours)
68.42
67.17
64,56
73.58
72.83
72.30
65.81
64.10
62.23
POD-AANeus
30
66.61
64.61
62.63
77.36
75.83
72.18
67.97
65.03
62.03
w/ RMM (ours)
68.86
67.61
66.21
79.52
78.47
76.54
69.21
67.45
63.93
Table I: Average accuracies (%) across all phases using two state-of-the-ar methods (LU- CIR+AANets and POD+AANets 30) w/ and w/o our RMM plugged in. The upper block is for recent CIL methods. For fair comparison, budget (see "Memory Budget" in Section 5.T) based on the public code. The results of using another common budget setting and the detailed numbers (confidence intervals and last- last-phase accuracies) provided in the supplementary materials.
re-implement these methods using our strict memory
For policy functions Th and Ta, intervals to reduce the search space and get a tolerable training overhead.
two-layer FC networks. All actions are discretized at 0.1
Hyperparameters and Configuration. The training of the classification model O exactly follows On CIFAR-100 MamageNer-Subse//uul (90) epochs in each phase, and divide the learning rate by 10 after 80 (30) and then after 120 (60) epochs. Then, we fine-tune the model for 20 epochs using only exemplars (including the preserved exemplars of the new data to be used in future phases). We use an SGD optimizer and an ADAM optimizer for the classification model and policy functions, respectively. More details are given in the supplementary.
the uniform setting in 13 30 31).
we train it for 160
Memory Budget. There are two popular settings about memory budget in related work. One uses bounded memory budget with fixed capacity for phases [8 31 40). Another one allows the memory budget to grow along with phases [8 20 49] The first one is more strict and thus used as the major setting in our paper (note that the results and analyses using the second setting are given in the supplementary materials). In every benchmark, the total budget of memory depends on the phase number N. For example, on CIFAR-100 100, the total memory budget is set as 7, 000 samples when N= 500 samples/class 2, 000 samples). Please note that 2, 000 is a bounded memory budget allocated since the (-th phase for saving exemplars. More clarifications about memory budget are given in the supplementary. For fair comparison, we re-implement related methods and report the results in Table if their original results (in the respective papers) were obtained in different setting of memory budget.
(7, 000 samples 10 classes/phase
5.2 Results and Analyses
Table presents the results of two -fate-of-the- art methods (LUCIR+AANets and POD+AANets (0 w/ and w/o our RMM plugged in, and some recent CIL work [ 18 29
40) Table shows
the ablation study in settings. Figure[pplots the changes of the average number of exemplars per old/new class for the incremental phases.
Comparing to the State-of-the-Art. From Table I we make the following observations. 1) Our RMM consistently improves the two top baselines LUCIR+AANets and POD+AANets [30] in all settings. E.g., LUCIR-UANNANN w/ RMM and POD-AANets w/ RMM respectively get 2.7% and 3.1% average improvements on the ImageNet-Subset. 2) Our POD-AANets w/ RMM achieves the best performances. Interestingly, we find that our RMM can boost performance more when the number of phases larger. For example, when N=25, RMM improves POD-AANets by 3.6% and 4.4% on CIFAR- 100 and ImageNet-Subset, respectively. These two numbers are 2.3% and 2.1% when =5
8

---

CIFAR-100
Ablation Setting
N=
10
25
10
25
Avg
Last
Avg
Last
Avg
Last
Avg
Last
Avg
Last
Avg
Last
1 BaseRow
66.61 57.81
64.61 55.70
62.63
52.53
77.36 70.02
75.83
68.97
72.18 63.89
2 One-level RL
67.92
58.61
66.94
58.31
65,95
56.44
78.50
72.00
78.15
71.00
75.47
67.47
3 Two-level RL (Used)
68.86
59.00
67.61
59.03
66.21
56.50
79.52
73.80
78.47
71.40
76.54
68.84
margin
+2.3
+1.2
+3.3
+3.6
+4
+2.1
+3.8
+2.6
+2.4
+4.4
+5
4 Two-level RL (T.P.)
68.62
59.40
67.22
58.20
65,82
56.20
78.81
72.42
77.68
70.77
75.29
68.81
margin
+1.6 +2
+2.6
+2.5
+3,2
+3.7
+1.5
+2.4
+1.9
+1.8
+3.1
+4.9
5 UpperBound RL
70.00
61.12
68.36
60.00
66,56
56.74
80.01
74.31
78.95
71.97
76.99
69.14
6 CrossVal Val Fixed
67.50 58.48
66.69
57.19
65.73
55.51
77.96
70.31
76.70
69.08
74.18
66.10
Table 2:
The evaluation results in the ablation study (%). "T.P." denotes our results using the Policy functions Transferred from another dataset. "Avg", "Last", and "Used" denote the average accuracy over all phases, the last-phase accuracy, and the results used as ours in Table respectively. BaseRow is from the sota method POD-AANets B0). Row 2 is for learning Level-1 policy. Row 3 is for learning Level-1 and Level-2 policies in hierarchical way. Row 4 is for using Transferred Policies (from the other dataset in the table), when RL is costly or impossible on target CIL tasks. The bottom lines are two oracles: training the RL model on the target CIL task (Row 5) and using cross- validatio to find the best fixed memory allocation between old and new classes (Row 6).
This indicates that the superiority of our RMM is more obvious in challenging settings (where the forgetting problem is more serious due to the more frequent model re-training through phases).
Ablation Settings. Table 2 shows the results of our ablation study. Row 1 is for the baseline method POD-AANets [30]. Row 2 is for learning only Level-1 policy T, (where each class gets an even split of the memory). Row is for learning both Level-1 policy T, and Level-2 policy T in our proposed hierarchical method, and its results are used in Table as "ours". Row 4 is for using Policy functions Transferred from another dataset (T.P.), which means on the target CIL dataset there is no training and vice
of RMM. Here, for CIFAR-100, we use the policy functions learned on ImageNet-Subseee versa. On the last two rows, we show two oracle settings. Row 5 is the upper bound that assumes all past and future data are accessible during training RMM on the target CIL dataset. Row 6 is for using cross-validation (i.e., all past, future, and validation data are accessible) to find the best fixed memory split between old and new class data, e.g., new of chosen split rates are given in the supplementary materials.
old
= 0.7 is chosen and then used in all phases. The details
Ablation Results. Hierarchical: In Table 2 when comparing Row 2 to Row 1, it is clear that leveraging reinforcement learning yields better results as it can derive adaptive memory allocation between old and new data. Using class- flas-sppccific memory management further increases the model performance (i.c., comparing Row 3 to Row 2), even though we divide the classes into only two groups. T.P. (Transferred Policy functions): Comparing Row 4 to Row 3, we can that using transferred policy functions (trained on another dataset) achieves comparable performance, and Row 4 does not require any reinforcement learning on the target CIL dataset. Oracle: Comparing Row 3 to Row 5, we see that learning RMM on pseudo CIL tasks is comparable to the upper bound case where all training and validate data are accessible, given the fact that the latter needs higher computational overhead and violates the standard CII, protocol. Row 6 results are consistently lower than ours in Row 3, although cross-validation has access to all past, future, and validation data.
Allocated Memory. Figure 4 shows the change of the average number of samples per class in three ablative settings. Solid and dashed lines represent old and new classes, respectively. From the plots, we have two observations. 1) Learning RMM on the pseudo or target CIL tasks (green and orange lines), we can obtain similar memory management results (i.e., actions). This means the learned policy is transferrable in non-stationary continuous environments. This matches the conclusion of continuous adaptation in 2 2) Using our RMM method achieved more balanced memory sizes between exemplars and new data. For example, in the 1-st phase of the 5-phase setting, "UpperBound

---

Upperllound RI., Old UpperrBound RL, New
Two-level RL, Old
-Two-level RL, New
Baseline, Baseline, New
Samples per class
500
500
400
300
300
200
200
100
100
4
3 4
5
10
15
20
25
5
10
15 20
25
phase (N=
phase (N=25)
Figure 4: The memory allocated for "Old" and "New" across different phases CIFAR-100. The second and fourth plots are enlarged versions of the first and third plots, respectively. Solid and dashed lines denote old and new classes, respectively. The baseline is POD-AANets [30]. "Two-level RL" and "UpperBound RL" correspond to Row 3 and Row Table respectively.
RL" and "Two-level RL" allocate around 100 samples for both exemplars and data. While the baseline setting has 40 and 500 samples for them, respectively. It thus addresses the data imbalance problem for CIL in learnable way.
Conclusions
We propose the reinforced memory management (RMM) method specially for tackling CIL tasks. The hierarchical reinforcement learning (RL) framework (two levels) in RMM is capable of making more adaptive memory allocation actions than using standard RL (one level). Using the generated pseudo tasks in RMM solves the issue of data incompatibility between CIL and RL. Corresponding experimental results show that the policy trained on these pscudo tasks can be directly applied to target tasks without any computational overhead. Our overall method of RMM is generic, and its trained policy (with without using in-domain dataset) can be easily incorporated into exemplar replaying-
aapse based CIL methods to boost performance.
Limitations and Societal Impact
We analyse the limitations and potential negative societal impact the following three aspects.
Complexity. Training RMM takes an additional time cost. According to Algorithm T the cost is Z) times higher than the time used for the target CIL task. However, the training RMM policy is offline and can use different dataset (see Table 2 RMM pre-learns robust policy from synthesized pseudo tasks and can directly applied for memory management real CIL tasks. The overhead of applying this policy is very little, e.g., 0.63% and 1. 1.12% of the total training time respectively on CIFAR-100 and ImageNet (Subset and Full), taking POD+AANets as baseline.
Technical assumptions. We build the framework of RMM based on series of technical assump- tions, which might directly hold for all real-world continual-leariing applications. When applying our method to mission-critical problems, particular care is required when modeling the system.
Privacy issues. Keeping the old class exemplars has the issue of data privacy. This calls for future research that explicitly forgets or mitigates the identifiable feature of the data.
Acknowledgments and Disclosure of Funding
This research was supported by *STAR under its AME YIRG Grant (Project No. A20E6c0101), A Aiiaaa Innovative Research (AIR) programme, and Max Planck Institute for Informatics.
10

---

References
[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3931-3940, 2020.
[2] Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In ICLR, 2018.
[3] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. CVPR, pages 3366-3375, 2017.
[4] Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In CVPR, pages 583-592, 2019.
[5] Christopher M Bishop. Pattern recognition and machine learning- springer, 2006.
[6] Francisco M. Castro, Manuel J. Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV, pages 241-257, 2018.
[7] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. In ICLR, 2019.
[8] Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. In Yoshua Bengio and Yann LeCun, editors, ICLR, 2016.
[9] Zhiyuan Chen and Bing Liu. Lifelong machine learning, Synthesis Lectures Artificial Intelligence and Machine Learning, 12(3):1-207, 2018.
[10] Guy Davidson and Michael C Mozer. Sequential mastery of multiple visual tasks: Networks naturally learn to learn and forget forget. In CVPR, pages 9282-9293, 2020.
[11] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars, Continual learning: A comparative study on how to defy forgetting classification tasks. arXiv, 9909.08383, 2019.
[12] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. arXiv, 1909.08355 2019.
[13] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-task. incremental learning. In ECCV, 2020.
[14] Qiang Gao, Zhipeng Luo, and Diego Klabjan. Efficient architecture search for continual learning. arXiv preprint 2XXx:2006.04027. 2020.
[15] Pierre Yves Glorennec. Reinforcement learning: An overview. In Proceedings European Symposium on Intelligent Techniques (ESIT-00), Aachen, Germany, pages 14-15. Citeseer, 2000.
[16] John Hammersley. Monte carlo methods. Springer Science & Business Media, 2013.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[18] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In CVPR, pages 831-839, 2019.
[19] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma. Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In ICLR, 2019.
[20] Xinting Hu, Kaihua Tang, Chunyan Miao, Xian- Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in class-incremental learning. In CVPR, 2021.
11

---

[21] Shenyang Huang, Vincent Frangois-Lavet, and Guillaume Rabusseau. Neural architecture search for class-incremental learning. arXiv preprint arXiv:1909.00 06686, 2019.
[22] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through feature adaptation. In ECCV, pages 699-715, 2020.
[23] Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! In ICLR Workshops, 2019.
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. Technical report, Citeseer, 2009.
[25] Anna Kukleva, Hilde Kuehne, and Bernt Schiele. Generalized and incremental few-shot learning by explicit learning and calibration without forgetting. In ICCV, pages 9020-9029, 2021.
[26] Yingying Li, Xin Chen, and Na Li. Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. In NeurIPS, pages 14858-14870, 2019.
[27] Yingying Li and Na Li. Online learning for markov decision processes in nonstationary environments: A dynamic regret analysis. In ACC, pages 1232-1237. IEEE, 2019.
[28] Yingying Li, Aoxiao Zhong, Guannan Qu, and Na Li. Online markov decision processes with time-varying transition probabilities and rewards. In ICML workshop on Real-world Sequential Decision Making, 2019.
[29] Zhizhong Li and Derek Hoiem. Learning without forgetting, IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2947, 2018.
[30] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class- incremental learning. In CVPR, 2021.
[31] Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. In CVPR, pages 12245-12254, 2020.
[32] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In NIPS, pages 6467-6476, 2017.
[33] Michael McCloskey and Neal J Cohen. Catastrophic interference connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109-165. Elsevier, 1989.
[34] K. McRae and P. Hetherington. Catastrophic interference is eliminated in pre-trained networks. In CogSei, 1993.
[35] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress continual learning. In ECCV, 2020.
[36] Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Random path selection for incremental learning. NeurIPS, 2019.
[37] Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. In CVPR, pages 13588- 13597, 2020.
[38] Mare Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. In Yoshua Bengio and Yann LeCun, editors, JCLR, 2016.
[39] R. Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97:285-308, 1990.
[40] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. iCaRL: Incremental classifier and representation learning. In CVPR, pages 5533-5542, 2017.
12

---

[41] Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard S Zemel. Incremental few-shot learning with attention attractor networks. NeurlPS, pages 2276-5286, 2019.
[42] Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In CVPR, pages 7008-7024, 2017.
[43] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2019.
[44] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):111-25,, 2015.
[45] Andrei Rusu, Neil Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv, 1606.04671, 2016.
[46] Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In ACL, 2016.
[47] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NIPS, pages 2990-2999, 2017.
[48] Christian Simon, Piotr Koniusz, and Mehrtash Harandi. On learning the geodesic path for incremental learning. In CVPR, pages 1591-1600, 2021.
[49] Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. Topology- preserving class-incremental learning. In ECCV, 2020.
[50] Tom Veniat, Ludovic Denoyer, and Marc' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. In ICLR, 2021.
[51] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):22--256, 1992.
[52] Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, pages 374-382, 2019.
[53] Ziyang Wu, Christina Back, Chong You, and Yi Ma. Incremental learning via rate reduction. In CVPR, pages 1125-1133, 2021.
[54] Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurlPS, pages 899-908, 2018.
[55] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremennal learning. In CVPR, pages 6982-6991, 2020.
[56] Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In CVPR, pages 12455-12464, 2021.
[57] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and faimess in class incremental learning. In CVPR, pages
2020.
[58] Fei Zhu, Xu- Yao Zhang, Chuang Wang, Fei Yin, and Cheng Lin Liu. Prototype augmentation and self-supervision for incremental learning. In CVPR, pages 5871-5880, 2021.
[59] Barret Zoph and Quoc V, Le. Neural architecture search with reinforcement leaminng In ICLR, 2017.
13


Reviews:
[{"review_id": "xoqrr0cZbSM", "reviewer": "Reviewer_j8oJ", "summary": "The paper proposes a RL-based memory management policy for CIL problem. They have two-step formulation of policy, which first determines how to allocate the memory to old and new tasks, then determines how many samples to store per each class within the task. A standard RL framework of policy gradient method is used to optimize their policy. Experimental results how that their RMM can further improve the recent state-of-the-art methods, such as LUCIR or PODNet. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Applying RL for the memory management is novel, and it does seem that the CIL performance improves with the proposed method. However, my major concern is that the policy should be learned with a pseudo-CIL tasks, utilizing the separate or base classes. Namely, one needs to build exact same setting (i.e., with the same number of incremental learning phases, N) as the target CIL setting, and it does not seem to be practical. At least, it would have been nice if the experiments have the results for the mismatched scenarios -- namely, for the case in which the setting of target and pseudo-CIL is different. \n\nFollowings are some more comments:\n\nPro:\n   - the problem of memory management for CIL and applying RL is new. \n   - Hierarchical formulation of RL is also new. \n   - Positive experimental results are shown. (Improvements over the state-of-the-art)\n\nCon & Questions:\n   - Major concern is mentioned above. CIL capability should not be confined to the benchmark data setting (Requirement 2) in line 206). Namely, what if the number of phases N becomes 13 or 17, for example? It seems like the whole RL policy should be re-trained, and this is a critical limitation. \n   - How does the performance of RMM change with the total memory budget? Maybe, if the total memory budget increases, the effect of RMM would decrease? \n\nOverall, I think the paper is on the borderline. While the setting and methodology is novel, the practicality of the method seems to be limited. I would also see what other reviewers think and finalize my rating. N/A\n\n----\nI have read the author's rebuttal and finalized my score to 6. Thanks. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N.A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hour"}, "extra_text": {}}, {"review_id": "9HnlOFMlsgc", "reviewer": "Reviewer_pi5D", "summary": "A common approach in class-incremental learning is to retain a fixed set of examples of the old classes to help avoid forgetting. The paper presents an approach to dynamically retain exemplars for the class-incremental learning problem. Reinforcement learning is used for dynamic memory management.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ The idea of using reinforcement learning for memory allocation is interesting. The way in which policy function is learned on pseudo-CIL tasks is novel to my knowledge. \n+ A bi-level action space is defined i.e., first deciding the split between old and new classes and then the memory allocated to each specific class. \n+ Extensive experiments are provided on three splits and two datasets. \n+ The generalizability of the proposed approach is showcased by plugging it in two recent baseline CIL approaches [25+18, 25+13]. \n+ Experiments with fixed and growing memory budgets are also performed, the proposed approach appears to help in both cases. \n\nCons/Questions:\n- One main issue I see with the current approach is that the policy is learned on pseudo tasks which may have totally different task distributions, intrinsic properties as compared to the future tasks that we are going to encounter. How much the approach is sensitive to such distribution shifts, e.g., can we learn a robust policy on an altogether different distribution or a set of classes that are much different from the ones going to be encountered in future tasks (as it is quite likely in a realistic setting)?\n- RL component will result in additional computational costs. I could not find any comparisons on that. It will be useful to compare the training times with recent approaches and also report the overhead caused by policy learning.\n- The paper mentions that the policy can be learned with either an in-domain or out-of-domain dataset, there is a single ablation experiment shown to support this. Can authors provide additional experimental results to support this claim, e.g., leveraging a medical-scan dataset, stylized ImageNet or iNaturalist?\n- Since Table 1 reports average accuracies, can the authors report confidence intervals or standard deviations over all tasks (for all/available methods)?\n- The memory budget in Table S4 seems to have used high storage. Can the authors confirm its in line with the literature? My understanding is that it's the total memory and not necessarily the memory used for storing the exemplars only. If this is the case, it will be clear to provide exemplars only memory statistics as well. \n- In figure 1, it will be helpful to annotate the meaning of different colors (i.e., old exemplars, new exemplars etc.). \n- The related works section can be improved with related papers:\n-- Zhu et al. Prototype Augmentation and Self-Supervision for Incremental Learning. CVPR'21.\n-- Wu et al. Incremental Learning via Rate Reduction. CVPR'21.\n-- Simon et al. On Learning the Geodesic Path for Incremental Learning. CVPR'21.\n-- Iscen et al. Memory-Efficient Incremental Learning Through Feature Adaptation. ECCV'20.\n-- Rajasegaran et al. Random path selection for incremental learning. NeurIPS'19.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are not listed. Societal impact questionairre is referred to as supplementary material, it would have been nice to answer the questions in the main paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "8jGFQ2GYD1", "reviewer": "Reviewer_Y6tA", "summary": "The paper proposes a new memory sampling strategy which replaces random/herding. The strategy consists of two parts: first level determining the distribution of #samples of each class in memory and second level determining samples to be selected for every class. This is learnt via RL policy. The sampler is trained by emulating a pseudo-CL problem with the data available in the pretraining phase-- where there are an equal number of tasks made as expected in the CL setup to enable transfer. Given a sampling configuration, a CL method trains the model and the performance on the pseudo-CL test set is given as reward to the RL algorithm. This trains the policy components, which optimize sample size and samples per class and train CL. Then, the best policy is used to create the memory which is used to train the actual CL problem with state-of-the-art continual learning approaches. This memory selection achieves better performance than previously used memories across CIFAR100, Imagenet100 and Imagenet1000 datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "2.1) Well written: The paper is very well written and was an enjoyable read. The contributions are clearly stated, acknowledgement is given to the existing body of work to the best of my knowledge. Experiments back motivation the problem as well, demonstrating the need. The paper then proceeds to describe a sufficient solution, results are clear and well-presented.\n\n2.2) Novel idea: The paper introduced a an interesting idea which might seem currently inefficient but can potentially be the first step which can result in nice & efficient future works about memory selection aspects of CL (similar to NASNet {1} for the NAS domain)\n\n2.3) Thorough ablations and supporting experiments: The paper analyzed the proposed approach well ablating each of the components and comparing with a maximum limit (Table 2) and exploring the memory size (Figure 4)/growing memory (supplementary) pre-emptively answering many of my questions.\n\nWeaknesses:\n\n3.1) Potentially violating the CL protocol [Critical]: Since the future is unknown-- continual learning formulation and in real world, formulations don’t have access to how many tasks will arrive in the future. Isn't using this information violating the CL protocol? (and this seems to be critical to the proposed method)\n\n3.2) Complexity and expensiveness don’t justify performance improvement [Major]: The introduced method requires extensive retraining of models to learn the optimal policy function. Furthermore, the memory selection process seems very complex in design for a memory sampler. The accuracy increase is relatively minor, which makes me far less excited about the proposed approach. Another major shortcoming is this requires datasets which have good transferability properties (NAS literature demonstrates that CIFAR and Imagenet happen to have similar properties so transfer works between them, but it's often unlikely)\n\n3.2.1) Why not only use the first level and not second-- it would massively simplify the learning complexity and time while still preserving most performance increases (acc. to Table 2).\n\nOverall:\n\nThis seems to be a well-written and explained paper with a novel idea which bridges two domains together and could potentially allow interesting directions to explore. However it seems to me that (i) it violates the CL protocol in a way which is critical to the method’s performance (ii) performance improvements does not justify the complexity introduced: it's very expensive to train and complex in design, the improvement in accuracy is not as great which makes me less excited (if more parameters & information are introduced, improvements in performance are expected). However, I might have misunderstood some of these claims. I request the authors to clarify the points presented in the weaknesses, so that I can consider it in my final rating. I am not familiar with RL literature and hence my feedback is primarily about the CL aspects of the work.\n\n{1} Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le, Learning Transferable Architectures for Scalable Image Recognition, CVPR18\n\nPost rebuttal:\n\nI am somewhat convinced by the author's rebuttal. I presumed assuming the number of tasks was critical, but it was not. I am quite surprised by the generalizability of the proposed approach. \n\nI suspect that there might be a simpler phenomena happening here aiding the generalizability-- eg: balancing to keep extra samples of older classes in a certain manner or picking certain kind of samples. Discovering what it is might vastly simplify the approach and lead to better understanding of the underlying mechanism. This is potentially nice future direction to build upon, and I encourage the authors to pursue it if possible.\n\nOverall, the method seems too complicated for its performance gains currently. However, I think this provides a method for exploring this space and uncovering better sampling strategies. It is generalizable enough to be valuable tool for the community without violating any critical assumptions. I would upgrade my rating to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No substantial negative societal impact that must've been discussed in my view but the section was missing from the draft.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "3xSR5_qf4F", "reviewer": "Reviewer_79eo", "summary": "This work addresses the problem of memory management in Class Incremental Learning (CLI). In this setup, models store exemplars from old classes and must decide what exemplars to drop to make space to new memories. The authors propose a hierarchical reinforcement learning approach where an agent first chooses how much of the old memory to release to make space for new samples and then it chooses how much of the new space allocate to high-entropy samples and low-entropy samples (they divide new classes in two groups). Experiments show that the the proposed method attains better performance than POD-AANets and LUCIR on CIFAR and ImageNet. In the supplementary material they show that the proposed memory management system also improves the performance of already-existing systems. Additional studies show how policies transfer between datasets, the impact of hierarchical RL, and how old and new memories balance over time.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overall Review\nThe proposed method is sound, interesting, and it improves the performance of class-incremental learners on multiple datasets. The authors provide ablations, a detailed description of their algorithm, and code. I have some questions about the level-2 actions and the chosen baselines but for the moment I recommend to accept this work. \n\n## Strengths\n* The proposed method is technically sound and improves the performance of CI learners.\n* The text is clear and easy to understand, the authors did a good job explaining their method.\n* The authors provide an ablation study\n\n## Weaknesses\n* In the text you cite other CIL works such as DER [45] that you do not include in your tables. Could you explain why? I suspect it is due to a difference in setups but some explanation would be welcome by me and those who are not experts in the particular field of CIL.\n* The use of the entropy for splitting the data in groups is not well-motivated. Could you reference previous work or include some experiment showing that using entropy is better than directly learning to predict a score?\n\n# Detailed comments\n## Originality\n* The authors apply existing techniques in RL to the memory management problem in CIL but to the best of my knowledge they are the first to do such thing.\n\n## Quality\n* The overall technical quality is good. My only concern is that the group separation by entropy is not explained.\n\n## Clarity\n* In general, the text is well-written and easy to follow. \n\n## Significance\n* The proposed idea is sound and interesting for the research community. The experiments show significant improvement. \n\n## Reproducibility\n* The authors did a good job towards reproducibility. They include an Algorithm, the code, the implementation details, and how to execute the code in the supplementary material.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors vaguely discuss limitations and potential negative societal impact in the supplementary material. They could broaden the discussion by including some discussion about the additional computational cost of training a model with their method or the privacy issues when keeping exemplar data.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a reinforcement learning-based for memory management in Class-Incremental Learning (CIL). The reviewers find the paper well written and the idea novel. They also think that the experiments are thorough and positive. There were initially some concerns on the generalizability of the proposed method. The reviewers are generally satisfied by the author responses on the issue. Despite remaining reservations regarding the complexity of the method relative to the performance gains, there is a clear consensus among the reviewers that the paper should be accepted.


Decision: Accept (Poster)