Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present an approach for continual learning based on replay.  By finding a way to represent hyperparameters of the replay learning process, specifically the replay ratio and the amount of replay iterations, the authors use a reinforcement learning approach to attempt to find the optimal parameters for each epoch.  The impetus for this design is ascribed to the fact that in the continual learning regime it is expected that the input distribution is not stationary and so we would also like to be able to modify the parameters automatically in response to the change in present data.  The authors provide experimental results for this method on cifar10, cifar100, and CLRS and by using their approach in complement to Experience Replay (ER) and Supervised Contrastive Replay (SCR). 
main_review: On the positive side, the approach presented for the control of the parameters is well described.  The datasets used and the selection of baselines is in line with the state of the art in this problem area.  And the quality of presentation is quite good with few noticeable errors or confusing passages.  

On the other hand, the presented approach is built off of ER or SCR as the backbone and using DQN to perform the required reinforcement learning.  So the theoretical contribution is mostly centered in the formulation of the reward signal which is then solved in a known way.  The reward function under test, in this case, is also not particularly sophisticated either, boiling down to measuring the difference in total risk from one time step to another on a random holdout set.  Alternatively, other additional ways to encapsulate the risk might have been good to test to characterize how this RL technique is affected by these choices or bringing in a more elaborate method of choosing the replay set would have been possible.  Figure 4 is somewhat confusing matters also in that it appears to almost contradict the non-stationary nature that we are trying to capture, as it appears that at least for this example that over time the parameters mostly converge towards a small region.  Finally, unless I am mistaken in my reading of Table 2, but it appears that on cifar when multiple iterations are used this method does no better than plain SCR and shows almost no difference with the ER method if multiple iterations are allowed. 

summary_of_the_review: While the paper was well written and the ideas made sense as presented, however, my concerns with the amount of theoretical novelty combined with an empirical situation in which I have some questions is going to make my recommendation at 5, marginally below threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors use reinforcement learning (specifically Q-learning) to choose certain hyper-parameters of replay for online continual learning. These hyper-parameters are chosen online using a DQN network as it sees new data, continuously adapting the choice of hyper-parameters based on a separate test memory.
main_review: Choosing appropriate hyper-parameters for replay differently for every task, and doing this using closed-loop control is an interesting idea.

But it's not clear to me if the improvement in performance is worth all the extra machinery required to train a DQN to choose the hyper-parameters. Moreover, from the results of Fig. 4, it seems like the DQN network seems to go towards a fixed action rather than adaptively change the hyper-parameters in a complex way. This indicates that there might be a way to choose these hyper-parameters using a much simpler setup.

The authors do not discuss the computational cost-performance tradeoff of using a DQN to choose the hyper-parameters, vs using a simpler open-loop scheme to choose them. This is the key issue that requires addressing.

The paper is generally clearly written, and their algorithm itself is relatively clearly explained. 
But the authors go through a big chunk of related work, including some which are not of immediate relevance to the algorithm in the paper. This section can easily be shortened.

The empirical evaluation of their method covers quite a lot of ground, and makes a convincing case that their method does have some advantage in terms of performance. 

A couple of minor comments: 
- Fig 3 is tiny and the plot is really hard to read without zooming in a lot.
- Fig 4: x-axis is not labelled, and not clear what it is.
summary_of_the_review: Overall, the significance and novelty of the work is not so clear to me, since it seems like it's a straightforward application of an off-the-shelf RL algorithm for choosing parameters of online continual learning. On the other hand, it is interesting that this setup does give performance improvements. But the biggest reason for my score is that the tradeoff between adding the complexity of DQN for choosing hyper-parameters vs performance gains is not discussed or quantified.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This submission proposes an RL-based method for tuning hyper-parameters of continual learning (CL) methods. The rationale is that tuning hyper-parameters on the entire set of tasks off-line invalidates the CL assumption of online learning, and therefore online methods are needed. The authors discuss the elements that make up the state space for making hyper-parameter choices, such as various performance metrics, and use use (deep) Q-learning to train an RL agent to make decisions about the hyper-parameter to use at each training step. The approach is applied on top of two existing replay-based methods, varying two hyper-parameters (one each in separate evaluations), and demonstrated to achieve improved performance w.r.t. the _un-tuned_ variants.

main_review: ############## Strengths ##############

1. The motivation for the problem of hyper-parameter tuning in a continual learning (CL) setting is strong and appealing, and has been understudied in the literature
2. The proposed approach of RL-based search seems promising and is (for the most part) described in a good level of detail
3. The empirical evaluation goes beyond looking at performance and presents a deeper study of why and how certain aspects of the approach are working

############## Weaknesses ##############

1. The description of the approach as class-incremental appears to be incorrect
2. Some crucial details of the experimental setting are unclear in terms of the fairness of comparison against baselines
3. Some of the primary empirical results are inconsistent with the motivation throughout the paper of using RL for hyper-parameter tuning

############## Arguments ##############

The problem setting itself studied in the submission is fairly novel: how to tune the hyper-parameters in a CL setting without "cheating" and looking at future tasks while tuning the hyper-parameters. Even though this sounds trivial in hindsight, most current CL papers unfortunately tune hyper-parameters in the entire set of tasks, which is akin to training on the test set in standard single-task supervised learning. The authors then adequately motivate the use of RL-based search as a tool for hyper-parameter tuning, explaining that a replay "test" memory can be used as a proxy for CL performance. The approach itself is then clearly explained and motivated: there are pieces of information like performance on the current task and performance on previous tasks that should intuitively impact the best choice for CL hyper-parameters at the current training step. At a high level, this description is clear, insightful, and easy to follow. As a side note, I quite like the idea of using test memory as a proxy for test data. It's interesting because this actually comes from training data: the model had initially used these data points for training. So the only reason this is not training accuracy is that the model eventually would forget these samples if not for the replay strategy. Basically it's a "test-for-forgetting" memory. I wonder if models typically forget more easily test samples vs training samples (e.g., if training accuracy usually decays differently than test accuracy). I would encourage the authors to include a discussion about this in a later revision of their work, though I did not think the lack of it was a major shortcoming of the submission. 

For all these positive factors, unfortunately the paper has some fundamental flaws that would need to be addressed prior to being ready for publication. 

The first one is the incorrect use of the term class-incremental learning as a descriptor for their method. While the authors discuss the definition of class-incremental learning correctly, their defense for why their method using task-specific Q-functions does not break the basic assumptions of class-incremental learning is incomplete. It boils down to the difference discussed by Van de Ven et al. between two categorization schemes: single-head vs multi-head [1,2] and task- vs domain- vs class-incremental. The latter categorization, which the authors assume in this work, is _not_ about the model used for learning (e.g., whether task-specific output heads are used or not), but rather about the problem setting itself. In particular, the class-incremental setting demands that approaches are not given access to task descriptors _even during training_. And this goes against the proposed method for using task-specific Q-functions. While this may seem like a simple notation disagreement, empirically all class-incremental baselines are trained without access to task indicators, which is unfair. 

Moreover, on the point of task-specific Q-functions, it remains unclear exactly _why_ the authors chose to train task-specific Q-networks. It is also unclear exactly _how_ this is done: is the replay buffer for Q learning cleared upon the start of each new task? If so, why? One would think intuitively that, if the task-agnostic state space defined in Sec 4.2 and 4.3 indeed captures a relation between test-memory performance and hyper-parameters, then this should be roughly agnostic to the task. It is unclear in the paper why the authors chose not to. Back to the previous point, if this choice is indeed ideal, then other methods should also have access to task indicators.

Another crucial detail that is somewhat unclear in the paper is the relation between replay memory size and test memory size. In particular, the combined sizes of these two memories should be equal to other methods' total replay memory. However, some of the wording in Sec 3 seems to suggest that this is not the case, and the test memory used for RL training uses _additional_ memory that the base methods are not allowed to use. In order to properly assess the benefits of the proposed approach, it should be evaluated under the same conditions as other approaches, concretely in terms of storing the exact same number of samples in memory. The evaluation should gauge whether increasing the replay memory size is better than using the additional memory as a test memory to follow the RL approach. 

One more choice remains unclear: why only test the benefits of RL-based search for individual hyper-parameters? Would the method not be applicable to searching over multiple hyper-parameters simultaneously (e.g., by training multiple RL agents in parallel)? And somewhat related: why the choice of using discrete actions? While it seems valid, since it is akin to doing a grid-search over a choice of possible hyper-parameters, it is unclear why not just use a continuous action space which RL supports. 

It is also never stated what the exploration strategy is for DQN. I assume epsilon-greedy? This choice seems to be quite relevant for the problem at hand, because the RL agent is being evaluated from the start (i.e., if it is bad initially, that will degrade the overall performance of the CL agent, as it will likely never recover performance on the earlier tasks). It seems like the algorithm may only work because all the choices of hyper-parameters are reasonably good. What if we don't know a priori a good way to restrict the space of possible hyper-parameters? This is for example very hard if we apply the proposed ideas to something like EWC, where the possible range of \lambda is huge. Even in the case of replay methods, it's unclear what strategy the authors used to restrict the action space to the selected values. 

In terms of results, unfortunately the findings do not seem to entirely support the claims made throughout the paper, and in particular the motivation itself for using RL. Concretely, the trend of hyper-parameter adaptation in Figure 4 suggests that the optimal hyper-parameters are _fixed_ and not state-based. Therefore, there doesn't seem to be a point at all in using RL, since the actions should not be state-dependent. This conclusion is further validated by the fact that ER and SR with a fixed value of 3 replay iterations outperform the RL-based method. This, along with the fact that the authors in the end use a single-step MDP, suggests that perhaps a multi-armed bandit formulation would have been more appropriate. 

############## Additional feedback ##############

The following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.

Intro
- The first paragraph is a bit disorganized
- Connection to other works for online hyper-parameter tuning seems weak: only downside pointed is that there is no held-out validation data. It becomes clear only later that this validation data comes from _other_ tasks beyond those in the CL problem, which is not always available.
- I wonder how specific the approach is to replay. Could it be used directly for other techniques, e.g. regularization-based?
- The intro overall is quite complete, albeit a bit disorganized. The examples in the first paragraph seem to come out of nowhere, the method is introduced twice seemingly independently in the penultimate and last paragraph.

Sec 2
- Overall the connections in the CL section are drawn nicely
- The explanation here of using "external" validation data explains why this is stated as a weakness in the Intro: external means from tasks _not_ in the sequence, which may or may not require different hyper-parameters (and there may not be access to such external tasks)
- The connection to RL-based hyper-parameter tuning strategies is also nice

Sec 4.1
- The definition of replay ratio is quite confusing. It's not really a step size, and a ratio suggests a ratio of "the amount of replay vs new data". From Eq. 1, it seems that it's instead a weighting of new vs replayed data.

Sec 4.2
- The state space is carefully designed and the intuition behind it is clearly explained.
- The choice of doing one-step MDP is buried at the end of the section. I think this choice is valid, because it might not be necessary to tune hyper-parameters for future learning. However, the use of an RL formulation throughout the paper suggests that this is exactly what this work is doing. I suggest placing this caveat much earlier in the paper and discuss the implications of this choice.

Sec 4.3
- Overall 4.2 and 4.3 are well written and it's nice that the "default" action is explained. One comment is that the state space for 4.3 is not described in nearly as much detail as in 4.2. In particular, it's not explained why the state spaces are different.


Sec 5
- Task sequences are fairly short, but I don't think that matters much since replay has been shown to work well on long sequences and I see no technical reason why this approach would be more susceptible to the sequence length
- Figure 4 needs x-axis label (I assume # task?)

Appendices
- Many appendices are never mentioned in the main paper, including the forgetting curves of Fig 5

[1] Farquhar and Gal. Towards robust evaluations of continual learning. arXiv:1805.09733, 2018.

[2] Chaudhry et al. Riemannian walk for incremental learning: Understanding forgetting and intransigence. arXiv:1801.10112, 2018.
summary_of_the_review: Unfortunately, I recommend the rejection of this paper. I believe that the problem that the authors are proposing is insightful in itself, and certainly captures one current issue with most CL evaluations today: that most approaches tune hyper-parameters in an off-line setting, which is inconsistent with the CL problem formulation. However, in my opinion there are a number of shortcomings (primarily empirical) that should be addressed for turning this into a complete conference publication. Most critically, there are a number of issues in the evaluation setting and in the results themselves that should be addressed prior to publishing the work. I do strongly encourage the authors to continue improving upon their work, as I believe that it could be highly impactful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper takes a particular continual learning setup and addresses the challenge of choosing two important hyper-parameters of standard continual learning approaches based on experience replay: the online-to-memory loss ratio, and the number of replay iterations. A DQN-style reinforcement learning approach is proposed to adapt these two parameters online. An additional small test memory is introduced, which is used to construct the reward and states. The efficacy of the approach is demonstrated using CIFAR datasets.

main_review: Strengths:
The main strength of the paper is addressing an important problem in continual learning: how to balance between online and memory-based updates, and how many times to iterate when a mini-batch is drawn from the memory.

The paper is well written and mostly understandable. There are many variants of continual learning tasks, among which this paper focuses on an important kind, where a stream of samples is seen only once, and new tasks may contain new classes but no known task information.

Another strength of the work is the simplicity of the idea, which is as simple as using reinforcement learning to learn the hyperparameters. The ingenuity is the employment of a test memory to structure this learning.

Weaknesses:

The main weakness of the work is the lack of clarity of the MDP formulation. In order to formulate a problem with an MDP, there need to be meaningful state-to-state transitions, and the optimal policy needs to be a function of the state. To understand whether those are present in the adopted continual learning setup, one could take a simplified scenario where the overall process of the continual learning setup is stationary. In that scenario, could you give a sensible description of the state-to-state transition dynamics that is contingent on the choice of actions? 

What would an optimal policy look like that would choose different actions under different states? If the optimal policy isn’t a function of states, how would you motivate the MDP formulation? Even if the optimal choices of the hyper-parameters aren’t state-dependent, do they have fixed points under some simplified setup? For example, can you describe a simple setup where an optimal number of replay iterations exists and explain what it means for a particular number of replay iterations to be optimal?

A second weakness is the lack of some crucial details. For example, as a DQN-style RL is used, it involved a replay buffer of its own. Details of the size of that replay aren’t given.

Moreover, using RL on top of CL increases the computation compared to the base CL methods with fixed hyper-parameters. Depending on how much extra computation RL is using, it can have implications on the fairness of the experimental setup. Would it be fair to compare methods with vastly different amounts of computation, especially, if the competing methods are using substantially less computation?

It will be much appreciated if you could provide some details on the experimental setup. When a task is presented, how many samples of that task are presented to the learner at a time? Are the samples presented online to the learner iid or sequential? The paper indicates that they are sequential: “The online CL setting is more challenging: … the sampling distribution is non-IID”. There could be pseudocode describing this process of interaction, not just the algorithm.

There are other works that use a validation set for continual learning such as “Large Scale Incremental Learning” by Wu et al. These should be discussed and compared with the proposed approach.

The symbols of Equation (1) aren’t described.

The x-axis labels of plots in Figure 4 aren’t given. Variable names are used for plot titles instead.

summary_of_the_review: The paper addresses an important variant of continual learning, and the proposed method is simple and applicable on top of many existing approaches. On the other hand, the application of RL isn’t well-motivated, and some crucial details are missing that may warrant reevaluation of the proposed methods, such as by achieving a computational parity with competing methods.
review_title: null

Meta-review: This manuscript studies the problem of continual learning and introduces a reinforcement learning agent to select hyperparameters for replay/training. Ordinarily, replay based mechanisms for continual learning use settings and hyperparameters that are chosen and fixed through training. If it was possible to adjust replay dynamics online (in this case by looking at performance on a held-aside test set), performance might be improved. This is the approach taken by this manuscript. 
Reviewers were generally happy with the writing of the paper and presentation of the material. At the same time, more than one reviewer worried about the novelty of the approach. In essence, the proposal amounts to using a black-box optimizer (in this case RL) to adjust online the hyperparameters (e.g. the replay ratio) for continual learning (of-the-shelf ER and SCR). Viewed through this lens, and given that the optimizer in this case was a straightforward application of DQN, this concern is potentially well founded. The primary novelty then is the construction of the reward function to be optimized: in this case defined as the decrease of the CL loss measured on a held aside test set that is constructed online. Nevertheless, novelty is only part of the equation and strong empirical results can easily be a deciding factor in readiness for publication. On this front, reviewer GhFg points out that the empirical results and comparisons with baseline methods are not as clear as they need to be. Several issues are raise in discussion: the primary one is around the question of how the authors have allowed task-specific information for the Q functions used by RL, and what the implications of this might be. The baselines compared against do not use any task-specific information, which muddies the waters when trying to understand the comparisons. I agree with the reviewer that the manuscript needs to do a better job of making the empirical setting and comparisons as transparent and fair as possible. Given this, and the fact (raised by several reviewers) that some empirical evidence presented in the manuscript actually points to RL selecting near-static parameters over time, I recommend that the manuscript be rejected. At the same time, I want to encourage the authors to focus on a streamlined version of the manuscript that addresses the issues raised by GhFg, as I believe that if the concerns can be addressed the work is close to making a compelling contribution for the field.
Decision: Reject