Closed-loop Control for Online Continual Learning
Authors: Yaqian Zhang
Eibe Frank
Bernhard Pfahringer
Albert Bifet
Nick Jin Sean Lim
Alvin Jia
Venue: ICLR 2022 Submitted (2022)
Keywords: Continual Learnig
Reinforcement learning
Class-incremental Continual Learning
Online Learning

Abstract: Online class-incremental continual learning (CL) deals with the sequential task learning problem in a realistic non-stationary setting with a single-pass through of data. Replay-based CL methods have shown promising results in several online class-incremental continual learning benchmarks. However, these replay methods typically assume pre-defined and fixed replay dynamics, which is suboptimal. This paper introduces a closed-loop continual learning framework, which obtains a real-time feedback learning signal via an additional test memory and then adapts the replay dynamics accordingly. More specifically, we propose a reinforcement learning-based method to dynamically adjust replay hyperparameters online to balance the stability and plasticity trade-off in continual learning. To address the non-stationarity in the continual learning environment, we employ a Q function with task-specific and task-shared components to support fast adaptation. The proposed method is applied to improve state-of-the-art replay-based methods and achieves superior performance on popular benchmarks.

Under review as a conference paper at ICLR 2022
CLOSED-LOOP CONTROL FOR ONLINE CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Online insseinnremettll continual learning (CL) deals with the sequential task learning problem in realistic non-stationary setting with a single-pass through of data. Replay-basee CL methods have shown promising results several online class-incremental continual learning benchmarks. However, these replay meth- ods typically assume pre-defined and fixed replay dynamics, which is suboptimal. This paper introduces closed-loop continual learning framework, which obtains a real-time feedback learning signal via an additional test memory and then adapts the replay dynamics accordingly. More specifically, propose reinforcement learning- based method to dynamically adjust replay hyperparameters online to balance the stability and plasticity trade-off in continual learning. To address the non-stationarity in the continual learning environment, we employ a Q function with task-specific and task-shared components to support fast adaptation. The proposed method is applied to improve sfte-of-the-aa replay-based methods and achieves superior performance on popular benchmarks.
INTRODUCTION
A major challenge in research artificial neural networks is to develop incremental learning abil- ity to accumulate knowledge over time from a non-stationary stream of data. Most deep learning techniques are designed with the assumption that the training samples are drawn independently and identically from fixed data distribution, but this assumption is often violated in ever-changing real- world environments. Most successful deep learning results are achieved by training on pre-collected datasets. The resulting model is static and incapable of adapting its behavior to the non-stationary environment over time (Delange et al.) 2021). When new data becomes available, the training process needs to restart again. In non- con-stationary environment, training on the new data can seri- ously undermine the model's previously acquired knowledge of past data, which may cause what is known "catastrophic forgetting" (Parisi et al. 2019). In many real-world applications, large amount new data are generated continually, and it has become increasingly important to ensure that the learning agent has lifelong learning capabilities learn new knowledge and maintain the old knowledge.
Our work focuses online class incremental (OCI) continual learning problems to learn sequence of classification tasks with a single pass over the data. To address the problem catastrophic forget- ting in this setting, continual learning algorithms need to balance between stability and plasticity: the ability to robustly maintain old knowledge while being able to quickly incorporate new knowledge. Previous work continual learning deals with this problem by modifying the standard training process using memory replay-based, rggularization-based or parameter isolation- The work presented in this paper rests on the observation that, besides requiring changes to the train- ing algorithm, the sequential learning process also brings special challenges to the hyperparameter tuning process, which has not been extensively studied far. In fact, a large number of CL methods have to rely on well-selected hyperparameters to effectively balance the trade-off between stability and plasticity; to search for the best hyperparameters, most research in this area follows an offline approach to go through the whole data sequence several times measuring performance against rep- resentative, held-out validation set. This violates the fundamental assumption of continual learning, namely that there is access to previous task data. There is some work on heuristic hyperparameter tuning specifically designed for online CL (Aljundi et al. 2019 Mai et al. 2021a), but it requires extra held-out task validation data, which may not be available.
techniques.

---

Under review as a conference paper at ICLR 2022
To address the hyperparameter tuning problem, we propose closed-loop continual learning frame- work that is able to balance stability and plasticity without access to an external held-out validation set. Specifically,, we build real-time feedback signal that measures the performance of the CL agent and enables online hyperparameter tuning using reinforcement learning (RL). The proposed closed-loop CL. framework is combined with replay-based continual learning achieve adaptive memory replay for online class-incremental continual learning.
Replay-based CL methods prevent forgetting of previously learned knowledge by maintaining small memory of past data and performing joint training on incoming and memorized data during the optimization process. Despite their simplicity, these methods have significantly outperformed CL approaches without episodic memory in the online clus-incrrmmenal setting (Chaudhry et al.) 2019). Compared to standard supervised learning, two main challenges exist in memory replay methods. One is the risk of overfitting the memory, as a relatively small amount of memorized samples are repeatedly rehearsed in the training process. The other challenge the class imbalance problem in the joint training process: generally, only a relatively small amount of old-class data is available in the memory. Because of these challenges, it is important to be able to modulate the replay dynamics of these methods by hyperparameterss such as the replay step size and the number of replay iterations. However, the values assigned to these hyperparameters are often hand-crafted and kept fixed during the whole learning process in previous research memory replay methods. In this work, we address this problem by proposing a method that is able to dynamically adapt the hyperparameters of memory replay methods in an online manner. Specifically, based on closed- loop CL framework, we apply reinforcement learning to adjust replay hyperparameters to achieve a dynamic adjustment between stability and plasticity. We formulate the problem of adjusting replay hyperparameters as Markov decision process and use purpose-buil state space to capture the cur- rent performance of the CL model and the relevant input data information for replay hyperparameter decision-makin making.. To address the non-stationarity of the environment, we propose Q function with task-shared and task-specific components. The proposed RL-based methods are used to improve the two replay- -ased methods ER (Chaudhry et al. 2019) and SCR (Mai al., 2021b). and achieve adaptive memory replay.
2
RELATED WORK
2.1 CONTINUAL LEARNING
Continual learning is concerned with learning sequence of tasks. In each task t, data (+) is sampled from distribution [tt); the corresponding class labels are y)t) The goal is to minimize the loss on all seen tasks given limited or no access to data (1)), ()0)) ) from previous tasks & < T: EL W. (x00,)0) (x00))00 [e (f (x(0;0) ,))))) ])(] with loss function CL parameters 0, T the number of tasks seen so far, and ff representing the CL network function. In recent years, continual learning methods have been proposed to deal with different problem settings and evaluation protocols. Van de Ven & Tolias (2019) summarize three continual learning scenarios based whether task identity is provided at test time. Notably, there are substantial differences between the difficulties these three scenarios. The first scenario is task-incremental, which assumes the task identity is provided at test time. Methods proposed for this scenario usually train task-specific components to prevent the interference between new tasks and old task (Yoon al., 2020 2018 Mallya & Lazebnik 2018). A common practice in this setting is to use meat evaluation with separate output layer (head) assigned to each task. The task ID is used determine the head use for inference. This scenario is the easiest since the model just needs to distinguish classes within task. The second scenario is domain incremental, where the structure of the task is always the same and the input distribution changing. The last scenario is class-incremental, with new tasks containing new classes. This setting employs single-head evaluation, meaning the CL agent needs to classify all the classes seen so far, without known task information. This setting can be used as a realistic framework of the common real-world problem of incrementally learning new classes of objects.
Another aspect of CL is whether offline online learning is considered. The offline CL setting assumes full access to the whole data for a task at once. Therefore, training on each single task can be performed over multiple epochs, before moving on to the next task (Rebuffi et al. 2017).
2

---

Under review as a conference paper at ICLR 2022
The online CL setting is more challenging: stream of samples is seen only once and the sampling distribution is non-IID. In this paper, we focus on this more challenging and realistic setting.
To combat catastrophic forgetting, CL methods can be categorized into three families: replay meth-
ods, regularization-basee methods, and parameter isolation methods (Delange et al. [02T]] Parisi
et al.) 2019). The regularization methods, such EWC++ (Chaudhry et al.) 2018a) and LWF (Li
& Hoiem 2017) introduce a regularization term in the loss function to prevent dramatic changes
to important parameters and consolidate previous knowledge. Parameter isolation-based methods assign different subsets of model parameters to different tasks to prevent the interference between tasks. Some examples include methods that dynamically expand the network, such as RCL (Xu & Zhu 2018)) APD ( (Yoon et al., (020). and DEN (Yoon et 2018;; alternatively, one can use fixed network (Mallya Lazebnik 2018). This line of work generally requires knowledge of the task ID at test time and thus is applied in the task-incremental scenario. Replay-based methods retain a memory to store past data for joint training(Chaudrry et al., 2019 [Prabhu et al.] 2020). Some methods in this area focus on how to store and sample informative memory samples, such as MIR (Aljundi et al., 2019). Recent work uses supervised contrastive loss in memory-replay to help learn a high-quality feature representation (Mai et al. (021b). Some work also combines the regularization and replay approaches, such as MC-SGD (Mizaddeh et al. (020). ICARL (Rebuffi et al., 2017).
In terms of hyperparameter tuning for continual learning, most work in CL simply employs offline hyperparameter optimization using a held-out validation set. However, there is some recent work that discusses hyperparameter tuning frameworks specifically designed for continual learning set- tings without representative validation set. For offline continual learning, Delange et al. (202I) proposes method to dynamically balance the stability-plasticity trade-off via a maximal plasticity search followed by stability decay. Instead of requiring a validation set covering all the tasks, this method makes use of a held-out validation set for a new task; however, some parameters, like the tolerance threshold and decaying speed. need to be manually set in advance. Considering the online continual learning setting, Chaudhry et al. (20Tbb) and Mai et al. (2021a) employ a hyperparameter tuning protocol targeting this setting. These approaches use external cross-validation data streaming with a small number of tasks. Offline hyperparameter tuning is applied on this external validation data with multiple passes to identify optimal values, which are then used for the actual online contin- ual learning tasks. A limitation of this work is that it relies on external validation data. Depending on the similarity between the validation tasks and the actual continual learning tasks, the chosen hyperparameters may not be appropriate.
2.2 REINFORCEMENT LEARNING
Reinforcement learning (RL) concerns the problem of sequential decision-making in a stochastic environment (Sutton & Barto, 2018). In standard reinforcement learning settings, an RL agent interacts with the environment to maximize the long-term rewards. Typically, this interactive process is formulated as a Markov decision process (MDP) described by a tuple < S, A, R, P, 7 >, where S is the state space, A is the action space, R: SxA- + R is the reward function, P S - [0, 1] is the state transition probability, and 7 is the discount factor. The agent behavior is defined as the policy л 5 A that determines how and action is selected during the interaction. The goal of the RL agent is to learn a policy that maximizes the cumulative rewards in an episode. The action-value function @, (s, a) used represent the expected long-term reward of executing action a in state s, where so is the initial state distribution.
Although reinforcement learning has achieved superior performance in many challenging applica- tions such as board games, continual locomotion control, and complex computer games (Schrit- twieser et al. 2020; Mnih et al., 2013) Vinyals et al., 2019), its application adapting the learn- ing process of other machine learning algorithms is relatively limited. Reinforcement learning and bandits algorithms have been used in the area of neural network architecture search and offline hy- perparameter optimization in IID environments where a held-out validation set is available and the reward distribution is static (Li al. 2017 Parker-Holder et al., 2020 Hoffman et al. 2014). To our knowledge, the only work applying reinforcement learning in continual learning is in the parameter isolation- based method, which applies it to dynamically expand a network (Xu & Zhu 2018). This
approach is applied in the offline incrementtt continual learning setting and assumes access to a held-out validation set. There does not appear to be any work that applies reinforcement learning to replay-based continual learning methods or online continual learning settings.
3

---

Under review as a conference paper at ICLR 2022
mocming (ton RD)
Incaaning (rao)
Merrory
Continaal Leamiin Agent U
Teat menorr
Continual Learring Aqunt B0DXBu 8.55
Meroiy
Closed Momrary Rep
Figure 1: Closed-loop continual learning framework: an additional test memory is constructed to approximate validation data and used as feedback signal guide online hyperparameter adaptation.
CLOSED-LOOP CONTINUAL LEARNING
To tune the hyperparameters of continual learning for never-ending data stream, we propose closed-loop continual learning framework, as shown in Figure [ The main idea is to design quantitative feedback signal measure the performance of CL agent in an online manner and then use this feedback signal to guide the hyperparameter selection. To this end, we propose to build a test memory online that includes a subset of samples of all the classes seen so far and use the performance of the model this test memory to approximate its performance Evaluating the CL model using the test memory leads to an online feedback signal, which can be used adjust the hyperparameters of the CL model and construct closed-loop control system. To construct such test memory. we follow a procedure similar to the one employed in the management of replay memory: Given incoming batch of data, non-overlapping subsets of samples are stored in the replay memory and test memory respectively. To make the test memory representative, we employ class-balanced greedy sampling method (Prabhu al., 2020) greedily store samples from the data stream with the constraint to asymptotically balance the class distribution. Specifically, when data from a new class arrives, the sampler simply creates new bucket for that class and starts deleting samples from the class that currently contains the maximum number of samples. Every sample in this class is assumed be equally important and is randomly chosen for removal.
the test data.
Notably, a significant difference between the replay and test memory is that the samples in the replay memory will be repeatedly drawn to update model parameters via gradient descent, while the test memory will not take part in the direct update of model parameters, but only be used to evaluate the model's s performance and indirectly affect the hyperparameters in the model update.
To investigate the effectiveness of test memory, we examine the correlation between the model's accuracy on the test data and the model's accuracy on the test memory. Our goal is to design a test memory that is representative and correlates well with test data. Meanwhile, since the use of extra test memory leads to a space overhead, we are particularly concerned with whether the test memory can perform well with tiny test memory size.
REINFORCEMENT LEARNING-BAR BASED ADAPTIVE REPLAY
We follow the aforementioned closed- loop continual learning framework to treat the test memory performance as a feedback signal and use reinforcement learning as the controller to adjust replay hyperparameters accordingly, shown in Figure 2 We apply the closed-loop CL. framework to balance the stability and plasticity trade-off by dynamically adapting the replay dynamics in terms of the replay ratio and the number of replay iterations. based adaptive memory replay framework in Section 4.1 and then discuss the design of the Markov decision process to dynamically adjust the replay ratio and the replay iterations in Section 4.2 and Section 4.3 respectively. Lastly, Section 4.4 presents the reinforcement learning algorithm to solve the sequential decision-making problem in the non-stationary continual learning environment.
this section, we firstly describe the overall RL-
4.1 RL-BASED ADAPTIVE MEMORY REPLAY FRAMEWORK
The key element in replay-basel continual learning algorithms is the joint training on memory data of old classes and incoming data of new classes to mitigate catastrophic forgetting. The main chal- lenge is data imbalance between the new classes and old classes in the joint training, with only
4

---

Under review as a conference paper at ICLR 2022
Reinforcement Learning Environment
Incoming Data (non id)
Memory
Test
Continual Learning
Memory
Agent
State
Reward
Action
Reiffrrement Learning Agent
Task
Task specific
head
State
Task lask-shared
retwork
Ourput
head
Figure 2: Reinforcement learning-based adaptive replay framework: the RL agent selects replay hy- perparameters and the CL agent executes the replay update and evaluates the model on test memory to output reward and state for the RL agent.
small number of old classes data stored in the memory. This class imbalance leads to bias towards
the new classes in the final output layers (Mai et al., 202 Wu et al.) 2019). Another challenge
is the possible overfitting of memory data as a small amount of memory data needs to be replayed
multiple times (Verwimp et al., (02T). Ad-hoe replay dynamics have been designed to improve the performance of the replay method. For example, Review Trick (Mai al. 2020) is applied in the winning solution of the continual learning challenge at CVPR2020, which employs a small learning rate on the memory at the end of learning each task. And Verwimp et al. (2021) find that isolated training on the memory may be used as a way to move away from the high-loss ridge and reduce the overfitting of the replay memory. Yet other research (Aljundi al. 2019; Bang et al. (02T) consid- ers how to effectively store or select representative subset of memory samples for joint training. In contrast, our work focuses on the step size and the number of replay iterations.
of internal replay dynamics, including replay
Table presents the process of dynamically adjusting a replay hyperparameter with reinforcement learning, based treating the replay ratio as the hyperparameter of concern. At each joint training step, the RL agent analyzes the current state the environment and selects a hyperparameter value based on the current state. Then, the CL agent executes the joint training step following the specified hyperparameter value. Lastly, the CL model is evaluated on the test memory to generate reward feedback representing the goodness of the hyperparameter choice for the RL agent. To instantiate this algorithm for adaptive memory replay, we need to design the state, reward, and action space. We treat the adaptation of replay ratio and the number of replay iterations as two separate problems as the information used to tune the two hyperparameters is different. The MDP design details are discussed in the following two sections.
4.2 ADAPTIVE MEMORY REPLAY RATIO
To formulate the problem of adjusting the replay ratio as a sequential decision-making process, define the Action space as the set of possible values of the replay ratio-the the ratio between the stepsize used for training on the incoming batch and the stepsize used for training on the memory batch. As shown in Equation | the step size on the memory batch is one, and we use a to denote the step size on the incoming batch, which is equal to the replay ratio.
Σ (f((;0)))
Σ ((f(r;)))))) rEB.
Jreplay.ratio
ZEBM
With higher replay ratio, the learning process is more focused on the incoming batch and the plasticity of the model is enhanced. Conversely, with a lower replay ratio, the learning process is more focused on the memory samples and the stability of the CL agent is increased. The replay
5

---

Under review as a conference paper at ICLR 2022
Table 1: RL-based memory replay hyperparameter adaptation
Algorithm: Memory Replay with adaptive replay hyperparameter
Input:
M train memory, Miest test memory,
B, incoming batch data,
0 parameter of CL network, & parameter of RL agent,
a replay ratio
function RL_ ReplRayy,,MMMMee B;
BM ~ M >> Sample memory batch from train memory
8 4 RT. state(0, Mtest: B2, BM) >> Compute state
0. d(als) > Sample RI. action (i.e. replay hyperparameter)
0 + trinnt,rrainin(( (BM U U. a) D- Update CL model
r 4 ReL_Reearr(O,MMa
D> Compute Reward
è 4 UpdateRLie r, 8) D Update RL agent
M B, Update train memory
Miest + B, D Update test memory
ratio is typically set to in previous CL methods. The goal of reinforcement learning is to adapt the replay ratio for each incoming batch to maximize the performance on the test memory.
Our goal is to learn to make a deaendeenn replay ratio decision. The intuition is that if the current incoming batch is challenging and important for the CL agent, a high ratio should be employed. An incoming batch is considered being informative and important for the training of the CL agent if it contains some new classes or classes that the CL agent has not learned to recognize well. Following this idea, we design the State space to capture the relevant information to measure the importance of the replay batch in order to support the selection of replay ratio. To measure the importance of a replay batch, compute weighted model accuracy based the class distribution of the replay batch and the current class-specific accuracy on the test memory. Specifically, we first compute the current model's accuracy on the test memory for each class. We use C denote the classes that have been seen by test memory and A(c), c E C is the current prediction accuracy for each class measured using test memory. We use B [xr y Bily E CJ denote the subset of incoming batch excluding the samples from the classes that have not been seen by the test memory before. The weighted accuracy of incoming batch defined as AB. 1/ Σx.VEBE Aly). If batch of data contains lot of classes for which the CL model currently performs poorly (as indicated by the test memory), the weighted accuracy is expected be low. The weighted accuracy of memory batch AB,, is defined a similar way. This information is included in the state space to help adjust the step size for this batch. The overall state space includes the current performance of the CL agent measured by the average loss and accuracy using the test memory, and the weighted accuracy/loss and the number of new classes for the incoming batch and memory batch. A detailed description is shown in the appendix (Table 3)
Following the closed-loop CL framework, use the test memory to construct a reward signal. Reward is defined as the decrease of the loss on the test memory after one gradient update step with memory replay:
R =
Σ ((((;;0),))) - Σ
e(f(I; ((f(z;0'+1),
(2)
ZEMi
zEMion
where 01+1 --- 1 VolErEDY (f(x;)'),)) + ExeBy ((((;;
The reward is affected by several factors, including the CL network networks parameters 8, the sampled memory batch BM, the incoming batch B, and the test memory Miest- In Section 4.4 we discuss how to address the non-stationarity of the reward function. To simplify the problem, we employ one-step MDP to maximize the reward for each memory replay step.
4.3 ADAPTIVE MEMORY REPLAY ITERATION
We formulate online tuning of the number of replay iterations as a reinforcement learning problem by defining the Action space the number of additional memory iteration steps for each incoming

---

Under review as a conference paper at ICLR 2022
batch. For each incoming batch, the CL agent first performs vanilla memory replay step. After that, the RL agent decides how many more memory replay iterations will be conducted for this incoming batch. During each memory iteration step, the CL agent repeats the process of replaying memory, i.e., sampling a memory batch and concatenating with the incoming hatch to perform gradient updates.
Previous work has been largely restricted to performing only a single memory iteration on incoming samples. For online continual learning, multiple memory iterations can be particularly important, as this can help maximally extract information from a single pass through the incoming data samples. Nevertheless, with more iterations, the risk of overfitting the memory data increases and can hurt the performance in some cases. To support the selection of the number of memory iterations, we again use state space design that captures relevant information, including the loss and accuracy on the test memory batch, and the training loss and accuracy on the incoming batch. Similar to the replay ratio adaptation, Reward is defined as the decrease of CL. loss measured on the test memory (see Equation? 2) and we employ one-step MDP to maximize the reward for each memory replay step.
4.4 REINFORCEMENT LEARNING ALGORITHM
To solve the two sequential decision- making problems designed above, DQN (Van Hasselt et al., 2016) is used to update the RL agent, and the Q function ((s, q) learned with experience replay techniques. We use Q function to model the relationship between the choice of replay hyperpa- rameters and the resulting model performance improvement measured on the test memory.
It is worth mentioning that the reinforcement learning environment in the closed-loop continual learning framework contains several important components: incoming data, memory, test memory, and CL agent (see Figure 2) The reward function is influenced by these factors as shown in Equa- tion 2 In the continual learning environment, the distribution of incoming data is changing from task to task. As the training continues, the state of the CL agent is also changing as it moves to the local minimum. Due to these factors, the influence of the replay hyperparameters on the model performance may change dynamically during different continual learning stages. Therefore, it is reasonable to assume the Q function may not be totally the same for different tasks, although it does share some common properties. To address this non-stationarity of the environment, we propose task-dependent Q learning structure.
As shown in the RL agent part of Figure 2 when a new CL task arrives, instead of training a com- pletely new Q function or totally copying the old Q function from previous tasks, we propose Q network with a task-shared component and task-specific component. The task-shared component is used to capture the shared closed-loop control information across tasks and the task-specific com- ponent is randomly initialized for each new task. It should be noted that although the goal of CL is to perform well on all the tasks seen before, the goal of RL is to estimate an accurate Q function for the current task, regardless of previous tasks. This is because the Q function is used to adapt and improve the training process of the current CL task. The process of learning the Q function is similar to transfer learning. There is no need to store the task-specific head for the previous tasks. Also, although there is task-specific component in the RL network, there is task-specific component in the CL. network. Thus, unlike in the expansion-basee CL methods, no task information is needed at inference time.
5 EXPERIMENTS
In our evaluations, we focus on the comparison of the RL-based method to random sampling in the experience replay (ER) (Chaudhry et al., 2019) and SCR (Mai et al., (02Ibb approaches, which our method directly modifies. ER employs vanilla memory replay with a replay ratio of 1 and memory iteration. Supervised Contrastive Replay (SCR) applies supervised contrastive loss, instead of cross-entropy loss, for the joint training and improves the performance of vanilla replay by a large margin [1 We also consider the following reference baselines:
Note that SCR employs Searest-Class Mean classifier which is expensive for online learning and testing since the feature space changing during CL. training and the class centers need to be recomputed repeatedly. To employ SCR in closed-loop CL. setting, we add softmax head for SCR.
7

---

Under review as a conference paper at ICLR 2022
finetune: Trains continuously new tasks without any forgetting avoidance strategy.
MIR (Aljundi et al. 2019) Maximally Interfered Retrieval, another replay-based method
that improves ER by using a controlled sampling of memories for replay.
AGEM (Chaudhry et al. 2018b): Averaged Gradient Episodic Memory, which utilizes the samples in the memory to constrain the CL parameter updates instead of using direct replay.
LWF (Li & Hoiem 2017 2 Learning Without Forgetting, a regularization method that utilizes knowledge distillation to penalize feature drift.
The experiments are conducted using three continual learning benchmarks. Implementation details be found in the appendix.
CIFAR-100- divides CIFAR-100 100 dataset into 20 disjoint tasks. Each task contains 5 classes. There are 500 images for training and 100 images for testing in each class.
CIFAR-1O-split splits CIFAR-10 dataset into 5 disjoint tasks. Each task contains 2 classes. There are 5000 images for training and 1000 images for testing in each class.
CLRS (Li et al. 2020) a continual learning benchmark for remote sensing image scene classification. It contains 25 scene classes, split into 5 disjoint tasks. Each class has 450 training data and 150 test data.
5.1 CLOSED-LOOP CONTINUAL LEARNING
We first examine the effectiveness of the test memory. We compare the loss the memory with the loss on the test data. We use training memory loss as a baseline for approximating the test loss. As shown in the Figure the loss computed on test memory is closely related to loss test data. Notably, the loss computed on training memory is quite different from the loss on test data, especially when more continual learning tasks are encountered. This result shows the introduce of additional test memory is necessary.
nameyy
dra
CIFAR dataset
(b) CLRS dataset
(c) CIFARIO dataset
Figure 3: Approximation of test loss using test memory and training memory.
To study whether small test memory is effective, we further investigate the influence of test mem- ory size on the approximation quality of test loss. Specifically, we analyze the correlation between the model's performance lation between the test data and test memory is above 0.9 when the test memory size is above 100, as shown in Figure 5 This result suggests that small test memory is enough to present reliable feedback on the model's performance.
the test memory and the test data. In all three datasets, the corre-
5.2 ADAPTIVE REPLAY RATIO
We now consider the performance of our RL-based methods relative the baselines. The results in Table showss that our approach can improve accuracy for both ER and SCR. The replay ratio chosen by the ER-RL agent as the greedy action in each task shown in Figure In the first few tasks, the RI. agent explores all the choices randomly and then slowly focuses on more desirable ratio values. In this case, we can see that RL favors smaller step size on the incoming batch. As mentioned earlier, one challenge in the replay-based method is the bias toward new classes due to class imbalance in the joint training. With small replay ratio, the joint training process gives a higher weight on the old classes and can help to reduce the bias towards catastrophic forgetting. However, should be noted that although the agent is mostly focused on smaller values of incoming ratio, large values still considered be the greedy action once in
classes and mitigate
8

---

Under review as a conference paper at ICLR 2022
200
20
20
(a) Ea
(b) ER-RL.iter
Figure 4: RL-based replay hyperparameter adaptation (CIFARI00); a) replay ratio selection per- centage in each task; b) replay iteration selection percentage in each task.
while by the RL agent. The selection of large incoming ratio once in while is likely beneficial to combat outfitting on replay memory.
Table 2: Continual learning performance in three datasets across 3 runs.
Methods
CIFAR100
CLRS
CIFAR10
finetune
2.5 (#00.4)
(+0.3)
13.4 (+1.5)
LWF
8,8 (#0.3)
9.2
18.5 (#1.3)
AGEM
3.2 (20.3)
(.4.6)))
17.2 (#1.2)
MIR
21.5 (+1.5)
37.2 12..3)
ER
20.8 (#1.5)
(+1.1)
54.1 (#3.3)
ER-RL (ours)
22.9 (#1.8
16.8 (#0.8)
59.2 (#3.3)
SCR
38.4 (-L.8)
24.1 (EE.5)
70.2 (+3.0)
(Cr-RL_ratio (ours)
39.3 (+5.))
26.0 (#2.1)
73.3 (#1.1)
Multiple Iterations
ER-3iter
27.9 (+0.7)
17.8 (+1.8)
66.7 (#1.8)
EL_ERiiter (ours)
28.1 (+0.7)
17.0 (#0.7)
68.0 (#2.2)
SCR-3iter
46.6 (+.55)
30.7 (=600)
78.2 (+5))
RLLSCRiie iter (ours)
44.2 (+0.1)
32.0 (#2.1)
77.0 (+122)
5.3 ADAPTIVE NUMBER OF REPLAY ITERATIONS
Previous replay methods usually employ memory replay with only one iteration for online continual learning. Our results in Table 2 show that multiple iterations can improve results by large margin for both ER and SCR. Thus, the number of memory iterations should be regarded as an important hyperparameter in online continual learning. Table also shows that the reinforcement learning- based method can identify a suitable number of memory iterations in an online manner and achieve performance close the best possible performance (which is attained by using fixed number of iterations throughout). Figure 4bb) shows the number of memory iterations selected by the ER-RL agent the greedy action each task. We can see that for the first few tasks, the RL agent explores the environment and selects all the choices with nearly equal probability. As the learning process continues, the RL agent gradually focuses on the more promising number of memory iterations (note that the figure shows the number of additional iterations selected).
CONCLUSION
This paper proposes a closed-loop continual learning framework to adjust replay dynamics in an online manner. We provide empirical evidence to show that tiny test memory can provide reliable feedback signal with a strong correlation to ground truth and can be used to guide the learning process of CL agents. We present concrete example of employing closed-loop continual learning to achieve adaptive memory replay for online maas-inncemmntal continual learning. Reinforcement learning methods applied autonomously adjust the replay ratio and the number of replay

---

Under review as a conference paper at ICLR 2022
iterations and dynamically balance the plasticity and stability trade-off. Future work will investigate the optimization of other hyperparameters such as the learning rate and regularization parameters.
REFERENCES
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in Neural Information Processing Systems, 32:1 1849-144111111 11800, 2019.
Jihwan Bang. Heesu Kim, YoungJoon Yoo, Jung-War Woo Ha, and Jonghyun Choi. Rainbow mem- ory: Continual learning with memory of diverse samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8218-8227, 2021.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018a
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018h bb...
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprini arXiv: 10486, 2019.
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Matthew Hoffman, Bobak Shahriari, and Nando Freitas. On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning. Artificial Intelligence and Statistics, pp. 365-374. PMLR, 2014.
Haifeng Li, Hao Jiang, Xin Gu, Jian Peng, Wenbo Li, Liang Hong, and Chao Tao. Clrs: Continual learning benchmark for remote sensing image scene classification, Sensors, 20(4): 2020.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit- based approach to hyperparameter optimization. The Journal of Machine Learning Research. (8(1):6765-6811 2017.
Z Li and D Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, (1112)))2935-247,,211 2017.
Zheda Mai, Hyunwoo Kim, Jihwan Jeong, and Scott Sanner. Batch-level experience replay with review for continual learning. arXiv e-prints. pp. arXiv-2007, 2020.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey, arXiv preprint arXiv:2101.1042 2021a.
Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3589-3599, 2021b.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Scyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In International Conference on Learning Representations, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(754))559-533, 2015.
10

---

The MDP design for adaptation of Under review as a conference paper at ICLR 2022
Genman Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Jack Parker-Holder, Vu Nguyen, and Stephen J Roberts. Provably efficient online hyperparameter optimization with population based bandits. Advances in Neural Information Processing Systems, 33, 2020.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference on computer vision, pp. 524-540. Springer, 2020.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Julian Schrittiesser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):004-60) 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. NeurIPS Continual Learning workshop, 2019.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-
learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Eli Verwimp, Matthias De Lange, and Tinne Tuytelaars. Rehearsal revealed: The limits and merits of revisiting samples continual learning. arXiv preprini arXiv:2104.074400 2021.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, Nature, 575(7782)350-354, 2019.
Jeffrey S Vitter. Random sampling with reservoir. ACM Transactions Mathematical Software (TOMS), 11(1):37-57, 1985.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374 382, 2019.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurlPS, 2018
JacHong Yoon, Jeongtae Lee, Eunho Yang, and Sung Ju Hwang. Lifelong learning with dynamically expandable network. In International Conference on Learning Representations. International Conference on Learning Representations, 2018.
Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In Eighth International Conference Learning Representations, ICLR 2020. ICLR, 2020.
A APPENDIX
A. MDP DESIGN
The MDP design for replay ratio adaptation shown in Table the number of replay iterations is shown in Table

---

Under review as a conference paper at ICLR 2022
Table 3: MDP design for RI.-based replay ratio adaptation
State:
average loss and accuracy of test memory [RMegee
Ene ([f(2;),,1)
weighted loss and accuracy of incoming batch AB,
weighted loss and accuracy of memory batch ABM number of unseen classes in the incoming batch |B, B number of unseen classes in the memory batch MBM B Action: replay ratio [0.1, 0.2, 0.3, 0.5, 0.75, 1.0, 1.2, 1.5) decrease in the test memory loss
Reward:
Table 4: MDP design for RL-based replay iteration adaptation
State:
average loss and accuracy of test memory train loss and accuracy of incoming batch [B]]] ZzEB, ((f(x;0),))
[RMend EPEBmun {(f(z;0),))
Action: additional replay iteration [0, 1, 2]
Reward: decrease in the test memory loss
A.2 IMPLEMENTATION DETAILS
Following (Chaudhry al. 2019 Mai 2021b), reduced ResNet18 is used as the backbone model for all datasets. We use stochastic gradient descent with learning rate of 0.1, and the model receives batch with size 10 a time from the data stream. All the methods are trained with cross- entropy loss except that SCR is trained with supervised contrastive loss. The softmax classifier is used for classification. We use softmax head for SCR which includes hidden layer of 1024. We use reservoir sampling (Vitter 1985) for memory management and greedy sampling (Prabhu et al.) 2020). We use a memory batch size 10, except that SCR uses a memory batch size TO0. All the experiments are across three seeds.
In the reinforcement learning, the shared component of the Q network is a 3-layer MLP with size 32. The task-specific component and the output head is linear layer with size 10. Replay buffer size is 100. Q learning batch size is 50. The learning rate for updating the RL agent 0.001. The weight decay factor is 0.0001. e-greddy is used for exploration. The exploration rate is reduced from 1 to 0.2 at 20% time steps, then reduced from 0.2 to 0.05 at 50% time steps, and fixed at 0.02 after that.
A.3 ALGORITIM
A.4 SUPPLEMENTARY RESULTS
(a) CIFARI00 dataset
(b) CLRS dataset
(c) CIFARIO dataset
Figure 5: Quality of feedback signal with respect to test buffer size terms of correlation with real test loss.
12

---

Under review as a conference paper at ICLR 2022
Table RL-based memory replay with adaptive replay ratio
Algorithm: Joint training with adaptive replay ratio
Input: M train memory, Meest test memory, B2 incoming batch data,
0 parameter of CL network, ó parameter RL agent, replay ratio
function RL. Replay SoppaayrratooMMMM. ratio(,M,,Mest B; )
BM MM i Sample memory batch from train memory Bi, BM) >> Compute state
8 - FRL.stale(O,Marn
<(als) Sample RL action
- Set. Replay (heplay_Para(a) >> Set replay memory iterations
0 SGD(BM U L,, 0, a)) >> Update CL model
r f RL Reward(0, Mtest) >> Compute Reward
+- UpdateRL
s) >> Update RL agent
M + B; >> Update train memory
Meest + B, Update test memory
Table 6: Memory replay with adaptive replay iteration
Algorithm: Joint training with adaptive replay iteration
Input: M train memory, Meest test memory, B; incoming batch data,
i parameter of CL network, & parameter of RL agent,
k additional replay iteration
function RL Replay iter(MM.Messt. B;
Bm .M b> Sample memory batch from train memory 0 SGD(Bm, U B,, 0) t- Update CL model
S - RL-state (0,.stateDD,Meett B,, BM) D Compute state
d(als) D- Sample RI. action
+ Set- Sitt-ReplyyPaar for in range(k):
D Set replay memory iterations
Bn ~ M b- Sample memory batch from train memory
0 + SGD(Bm U B,, 0) Update CL model
+ Reward(d Mrest) D> Compute Reward
t UpdateRLaa s) Update RL agent
M < B, D Update train memory
Meast B, Update test memory
IR-
BV
0.6
IR-1ber
18-41 her
fvernge socuree
6.5
0.00
0.
6.00
02-
0:5
last
Task X2
Task
(a) 10FAR100 dataset
(b) CLRS dataset
(c) CIFARIO dataset
Figure 6: Adaptive memory iteration.
13


Reviews:
[{"review_id": "exaMaH_-BYA", "reviewer": "Reviewer_vSnQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Choosing appropriate hyper-parameters for replay differently for every task, and doing this using closed-loop control is an interesting idea.\n\nBut it's not clear to me if the improvement in performance is worth all the extra machinery required to train a DQN to choose the hyper-parameters. Moreover, from the results of Fig. 4, it seems like the DQN network seems to go towards a fixed action rather than adaptively change the hyper-parameters in a complex way. This indicates that there might be a way to choose these hyper-parameters using a much simpler setup.\n\nThe authors do not discuss the computational cost-performance tradeoff of using a DQN to choose the hyper-parameters, vs using a simpler open-loop scheme to choose them. This is the key issue that requires addressing.\n\nThe paper is generally clearly written, and their algorithm itself is relatively clearly explained. \nBut the authors go through a big chunk of related work, including some which are not of immediate relevance to the algorithm in the paper. This section can easily be shortened.\n\nThe empirical evaluation of their method covers quite a lot of ground, and makes a convincing case that their method does have some advantage in terms of performance. \n\nA couple of minor comments: \n- Fig 3 is tiny and the plot is really hard to read without zooming in a lot.\n- Fig 4: x-axis is not labelled, and not clear what it is.", "summary_of_the_paper": "In this paper, the authors use reinforcement learning (specifically Q-learning) to choose certain hyper-parameters of replay for online continual learning. These hyper-parameters are chosen online using a DQN network as it sees new data, continuously adapting the choice of hyper-parameters based on a separate test memory.", "summary_of_the_review": "Overall, the significance and novelty of the work is not so clear to me, since it seems like it's a straightforward application of an off-the-shelf RL algorithm for choosing parameters of online continual learning. On the other hand, it is interesting that this setup does give performance improvements. But the biggest reason for my score is that the tradeoff between adding the complexity of DQN for choosing hyper-parameters vs performance gains is not discussed or quantified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "df8Hx3uk-O", "reviewer": "Reviewer_GhFg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "############## Strengths ##############\n\n1. The motivation for the problem of hyper-parameter tuning in a continual learning (CL) setting is strong and appealing, and has been understudied in the literature\n2. The proposed approach of RL-based search seems promising and is (for the most part) described in a good level of detail\n3. The empirical evaluation goes beyond looking at performance and presents a deeper study of why and how certain aspects of the approach are working\n\n############## Weaknesses ##############\n\n1. The description of the approach as class-incremental appears to be incorrect\n2. Some crucial details of the experimental setting are unclear in terms of the fairness of comparison against baselines\n3. Some of the primary empirical results are inconsistent with the motivation throughout the paper of using RL for hyper-parameter tuning\n\n############## Arguments ##############\n\nThe problem setting itself studied in the submission is fairly novel: how to tune the hyper-parameters in a CL setting without \"cheating\" and looking at future tasks while tuning the hyper-parameters. Even though this sounds trivial in hindsight, most current CL papers unfortunately tune hyper-parameters in the entire set of tasks, which is akin to training on the test set in standard single-task supervised learning. The authors then adequately motivate the use of RL-based search as a tool for hyper-parameter tuning, explaining that a replay \"test\" memory can be used as a proxy for CL performance. The approach itself is then clearly explained and motivated: there are pieces of information like performance on the current task and performance on previous tasks that should intuitively impact the best choice for CL hyper-parameters at the current training step. At a high level, this description is clear, insightful, and easy to follow. As a side note, I quite like the idea of using test memory as a proxy for test data. It's interesting because this actually comes from training data: the model had initially used these data points for training. So the only reason this is not training accuracy is that the model eventually would forget these samples if not for the replay strategy. Basically it's a \"test-for-forgetting\" memory. I wonder if models typically forget more easily test samples vs training samples (e.g., if training accuracy usually decays differently than test accuracy). I would encourage the authors to include a discussion about this in a later revision of their work, though I did not think the lack of it was a major shortcoming of the submission. \n\nFor all these positive factors, unfortunately the paper has some fundamental flaws that would need to be addressed prior to being ready for publication. \n\nThe first one is the incorrect use of the term class-incremental learning as a descriptor for their method. While the authors discuss the definition of class-incremental learning correctly, their defense for why their method using task-specific Q-functions does not break the basic assumptions of class-incremental learning is incomplete. It boils down to the difference discussed by Van de Ven et al. between two categorization schemes: single-head vs multi-head [1,2] and task- vs domain- vs class-incremental. The latter categorization, which the authors assume in this work, is _not_ about the model used for learning (e.g., whether task-specific output heads are used or not), but rather about the problem setting itself. In particular, the class-incremental setting demands that approaches are not given access to task descriptors _even during training_. And this goes against the proposed method for using task-specific Q-functions. While this may seem like a simple notation disagreement, empirically all class-incremental baselines are trained without access to task indicators, which is unfair. \n\nMoreover, on the point of task-specific Q-functions, it remains unclear exactly _why_ the authors chose to train task-specific Q-networks. It is also unclear exactly _how_ this is done: is the replay buffer for Q learning cleared upon the start of each new task? If so, why? One would think intuitively that, if the task-agnostic state space defined in Sec 4.2 and 4.3 indeed captures a relation between test-memory performance and hyper-parameters, then this should be roughly agnostic to the task. It is unclear in the paper why the authors chose not to. Back to the previous point, if this choice is indeed ideal, then other methods should also have access to task indicators.\n\nAnother crucial detail that is somewhat unclear in the paper is the relation between replay memory size and test memory size. In particular, the combined sizes of these two memories should be equal to other methods' total replay memory. However, some of the wording in Sec 3 seems to suggest that this is not the case, and the test memory used for RL training uses _additional_ memory that the base methods are not allowed to use. In order to properly assess the benefits of the proposed approach, it should be evaluated under the same conditions as other approaches, concretely in terms of storing the exact same number of samples in memory. The evaluation should gauge whether increasing the replay memory size is better than using the additional memory as a test memory to follow the RL approach. \n\nOne more choice remains unclear: why only test the benefits of RL-based search for individual hyper-parameters? Would the method not be applicable to searching over multiple hyper-parameters simultaneously (e.g., by training multiple RL agents in parallel)? And somewhat related: why the choice of using discrete actions? While it seems valid, since it is akin to doing a grid-search over a choice of possible hyper-parameters, it is unclear why not just use a continuous action space which RL supports. \n\nIt is also never stated what the exploration strategy is for DQN. I assume epsilon-greedy? This choice seems to be quite relevant for the problem at hand, because the RL agent is being evaluated from the start (i.e., if it is bad initially, that will degrade the overall performance of the CL agent, as it will likely never recover performance on the earlier tasks). It seems like the algorithm may only work because all the choices of hyper-parameters are reasonably good. What if we don't know a priori a good way to restrict the space of possible hyper-parameters? This is for example very hard if we apply the proposed ideas to something like EWC, where the possible range of \\lambda is huge. Even in the case of replay methods, it's unclear what strategy the authors used to restrict the action space to the selected values. \n\nIn terms of results, unfortunately the findings do not seem to entirely support the claims made throughout the paper, and in particular the motivation itself for using RL. Concretely, the trend of hyper-parameter adaptation in Figure 4 suggests that the optimal hyper-parameters are _fixed_ and not state-based. Therefore, there doesn't seem to be a point at all in using RL, since the actions should not be state-dependent. This conclusion is further validated by the fact that ER and SR with a fixed value of 3 replay iterations outperform the RL-based method. This, along with the fact that the authors in the end use a single-step MDP, suggests that perhaps a multi-armed bandit formulation would have been more appropriate. \n\n############## Additional feedback ##############\n\nThe following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.\n\nIntro\n- The first paragraph is a bit disorganized\n- Connection to other works for online hyper-parameter tuning seems weak: only downside pointed is that there is no held-out validation data. It becomes clear only later that this validation data comes from _other_ tasks beyond those in the CL problem, which is not always available.\n- I wonder how specific the approach is to replay. Could it be used directly for other techniques, e.g. regularization-based?\n- The intro overall is quite complete, albeit a bit disorganized. The examples in the first paragraph seem to come out of nowhere, the method is introduced twice seemingly independently in the penultimate and last paragraph.\n\nSec 2\n- Overall the connections in the CL section are drawn nicely\n- The explanation here of using \"external\" validation data explains why this is stated as a weakness in the Intro: external means from tasks _not_ in the sequence, which may or may not require different hyper-parameters (and there may not be access to such external tasks)\n- The connection to RL-based hyper-parameter tuning strategies is also nice\n\nSec 4.1\n- The definition of replay ratio is quite confusing. It's not really a step size, and a ratio suggests a ratio of \"the amount of replay vs new data\". From Eq. 1, it seems that it's instead a weighting of new vs replayed data.\n\nSec 4.2\n- The state space is carefully designed and the intuition behind it is clearly explained.\n- The choice of doing one-step MDP is buried at the end of the section. I think this choice is valid, because it might not be necessary to tune hyper-parameters for future learning. However, the use of an RL formulation throughout the paper suggests that this is exactly what this work is doing. I suggest placing this caveat much earlier in the paper and discuss the implications of this choice.\n\nSec 4.3\n- Overall 4.2 and 4.3 are well written and it's nice that the \"default\" action is explained. One comment is that the state space for 4.3 is not described in nearly as much detail as in 4.2. In particular, it's not explained why the state spaces are different.\n\n\nSec 5\n- Task sequences are fairly short, but I don't think that matters much since replay has been shown to work well on long sequences and I see no technical reason why this approach would be more susceptible to the sequence length\n- Figure 4 needs x-axis label (I assume # task?)\n\nAppendices\n- Many appendices are never mentioned in the main paper, including the forgetting curves of Fig 5\n\n[1] Farquhar and Gal. Towards robust evaluations of continual learning. arXiv:1805.09733, 2018.\n\n[2] Chaudhry et al. Riemannian walk for incremental learning: Understanding forgetting and intransigence. arXiv:1801.10112, 2018.", "summary_of_the_paper": "This submission proposes an RL-based method for tuning hyper-parameters of continual learning (CL) methods. The rationale is that tuning hyper-parameters on the entire set of tasks off-line invalidates the CL assumption of online learning, and therefore online methods are needed. The authors discuss the elements that make up the state space for making hyper-parameter choices, such as various performance metrics, and use use (deep) Q-learning to train an RL agent to make decisions about the hyper-parameter to use at each training step. The approach is applied on top of two existing replay-based methods, varying two hyper-parameters (one each in separate evaluations), and demonstrated to achieve improved performance w.r.t. the _un-tuned_ variants.\n", "summary_of_the_review": "Unfortunately, I recommend the rejection of this paper. I believe that the problem that the authors are proposing is insightful in itself, and certainly captures one current issue with most CL evaluations today: that most approaches tune hyper-parameters in an off-line setting, which is inconsistent with the CL problem formulation. However, in my opinion there are a number of shortcomings (primarily empirical) that should be addressed for turning this into a complete conference publication. Most critically, there are a number of issues in the evaluation setting and in the results themselves that should be addressed prior to publishing the work. I do strongly encourage the authors to continue improving upon their work, as I believe that it could be highly impactful.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PmyHf-guIYY", "reviewer": "Reviewer_ng9G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the positive side, the approach presented for the control of the parameters is well described.  The datasets used and the selection of baselines is in line with the state of the art in this problem area.  And the quality of presentation is quite good with few noticeable errors or confusing passages.  \n\nOn the other hand, the presented approach is built off of ER or SCR as the backbone and using DQN to perform the required reinforcement learning.  So the theoretical contribution is mostly centered in the formulation of the reward signal which is then solved in a known way.  The reward function under test, in this case, is also not particularly sophisticated either, boiling down to measuring the difference in total risk from one time step to another on a random holdout set.  Alternatively, other additional ways to encapsulate the risk might have been good to test to characterize how this RL technique is affected by these choices or bringing in a more elaborate method of choosing the replay set would have been possible.  Figure 4 is somewhat confusing matters also in that it appears to almost contradict the non-stationary nature that we are trying to capture, as it appears that at least for this example that over time the parameters mostly converge towards a small region.  Finally, unless I am mistaken in my reading of Table 2, but it appears that on cifar when multiple iterations are used this method does no better than plain SCR and shows almost no difference with the ER method if multiple iterations are allowed. \n", "summary_of_the_paper": "The authors present an approach for continual learning based on replay.  By finding a way to represent hyperparameters of the replay learning process, specifically the replay ratio and the amount of replay iterations, the authors use a reinforcement learning approach to attempt to find the optimal parameters for each epoch.  The impetus for this design is ascribed to the fact that in the continual learning regime it is expected that the input distribution is not stationary and so we would also like to be able to modify the parameters automatically in response to the change in present data.  The authors provide experimental results for this method on cifar10, cifar100, and CLRS and by using their approach in complement to Experience Replay (ER) and Supervised Contrastive Replay (SCR). ", "summary_of_the_review": "While the paper was well written and the ideas made sense as presented, however, my concerns with the amount of theoretical novelty combined with an empirical situation in which I have some questions is going to make my recommendation at 5, marginally below threshold.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GU3lu7sFHg", "reviewer": "Reviewer_hbHU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe main strength of the paper is addressing an important problem in continual learning: how to balance between online and memory-based updates, and how many times to iterate when a mini-batch is drawn from the memory.\n\nThe paper is well written and mostly understandable. There are many variants of continual learning tasks, among which this paper focuses on an important kind, where a stream of samples is seen only once, and new tasks may contain new classes but no known task information.\n\nAnother strength of the work is the simplicity of the idea, which is as simple as using reinforcement learning to learn the hyperparameters. The ingenuity is the employment of a test memory to structure this learning.\n\nWeaknesses:\n\nThe main weakness of the work is the lack of clarity of the MDP formulation. In order to formulate a problem with an MDP, there need to be meaningful state-to-state transitions, and the optimal policy needs to be a function of the state. To understand whether those are present in the adopted continual learning setup, one could take a simplified scenario where the overall process of the continual learning setup is stationary. In that scenario, could you give a sensible description of the state-to-state transition dynamics that is contingent on the choice of actions? \n\nWhat would an optimal policy look like that would choose different actions under different states? If the optimal policy isn’t a function of states, how would you motivate the MDP formulation? Even if the optimal choices of the hyper-parameters aren’t state-dependent, do they have fixed points under some simplified setup? For example, can you describe a simple setup where an optimal number of replay iterations exists and explain what it means for a particular number of replay iterations to be optimal?\n\nA second weakness is the lack of some crucial details. For example, as a DQN-style RL is used, it involved a replay buffer of its own. Details of the size of that replay aren’t given.\n\nMoreover, using RL on top of CL increases the computation compared to the base CL methods with fixed hyper-parameters. Depending on how much extra computation RL is using, it can have implications on the fairness of the experimental setup. Would it be fair to compare methods with vastly different amounts of computation, especially, if the competing methods are using substantially less computation?\n\nIt will be much appreciated if you could provide some details on the experimental setup. When a task is presented, how many samples of that task are presented to the learner at a time? Are the samples presented online to the learner iid or sequential? The paper indicates that they are sequential: “The online CL setting is more challenging: … the sampling distribution is non-IID”. There could be pseudocode describing this process of interaction, not just the algorithm.\n\nThere are other works that use a validation set for continual learning such as “Large Scale Incremental Learning” by Wu et al. These should be discussed and compared with the proposed approach.\n\nThe symbols of Equation (1) aren’t described.\n\nThe x-axis labels of plots in Figure 4 aren’t given. Variable names are used for plot titles instead.\n", "summary_of_the_paper": "This paper takes a particular continual learning setup and addresses the challenge of choosing two important hyper-parameters of standard continual learning approaches based on experience replay: the online-to-memory loss ratio, and the number of replay iterations. A DQN-style reinforcement learning approach is proposed to adapt these two parameters online. An additional small test memory is introduced, which is used to construct the reward and states. The efficacy of the approach is demonstrated using CIFAR datasets.\n", "summary_of_the_review": "The paper addresses an important variant of continual learning, and the proposed method is simple and applicable on top of many existing approaches. On the other hand, the application of RL isn’t well-motivated, and some crucial details are missing that may warrant reevaluation of the proposed methods, such as by achieving a computational parity with competing methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript studies the problem of continual learning and introduces a reinforcement learning agent to select hyperparameters for replay/training. Ordinarily, replay based mechanisms for continual learning use settings and hyperparameters that are chosen and fixed through training. If it was possible to adjust replay dynamics online (in this case by looking at performance on a held-aside test set), performance might be improved. This is the approach taken by this manuscript. 
Reviewers were generally happy with the writing of the paper and presentation of the material. At the same time, more than one reviewer worried about the novelty of the approach. In essence, the proposal amounts to using a black-box optimizer (in this case RL) to adjust online the hyperparameters (e.g. the replay ratio) for continual learning (of-the-shelf ER and SCR). Viewed through this lens, and given that the optimizer in this case was a straightforward application of DQN, this concern is potentially well founded. The primary novelty then is the construction of the reward function to be optimized: in this case defined as the decrease of the CL loss measured on a held aside test set that is constructed online. Nevertheless, novelty is only part of the equation and strong empirical results can easily be a deciding factor in readiness for publication. On this front, reviewer GhFg points out that the empirical results and comparisons with baseline methods are not as clear as they need to be. Several issues are raise in discussion: the primary one is around the question of how the authors have allowed task-specific information for the Q functions used by RL, and what the implications of this might be. The baselines compared against do not use any task-specific information, which muddies the waters when trying to understand the comparisons. I agree with the reviewer that the manuscript needs to do a better job of making the empirical setting and comparisons as transparent and fair as possible. Given this, and the fact (raised by several reviewers) that some empirical evidence presented in the manuscript actually points to RL selecting near-static parameters over time, I recommend that the manuscript be rejected. At the same time, I want to encourage the authors to focus on a streamlined version of the manuscript that addresses the issues raised by GhFg, as I believe that if the concerns can be addressed the work is close to making a compelling contribution for the field.
Decision: Reject