Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes 6 rules for continual learning experiments.
Specifically, they propose to 
- 1) test for extreme continual learning settings, 
- 2) report random guess performance 
- 3) evaluate naive/simple methods 
- 4) revisit old data 
- 5) test for different task orders and 
- 6) define performance that derive from the data.

Then,  pan empirical study on different scenarios that emerge from the combinations of those rules is provided.
main_review: **Strengths**
- All rules are sound
- enough baselines are provided

**Weaknesses**
I have some problems with the rules
- Rule 1
    - I think this rule as merit, but the authors should provide some motivation for it
- Rule 2
    - although a good rule, this is already reported in most high-quality research papers, or it is trivial to compute.
- Rule 3
    - also reported in most high-quality research papers
- Rule 4
    - good rule to bring up, although, as the authors pointed out, it has been studied in OSAKA.
- Rule 5 
    - same as above
- Rule 6
    - not sure I understood fully this rule. I think the text could be improved

also, related to the experiments and discussion:
- "The availability of task labels for testing simplifies the learning problem significantly in task oracle base methods." 
    - this is not a new finding, it's well reported in the literature. Actually, lots of papers report both performance as class-incremental vs task-incremental learning.
- "in OSAKA (Caccia et al., 2020). The work considers task transition to be model by a Markov chain and uses $\alpha$ parameter to decide how likely the next data will be drawn from the current task. Hence, recapitulation occurs when it transitions back to previously visited task. However, OSAKA did not consider an extreme scenario as suggested by rule 1 by considering very low $\alpha$ values where transitions between tasks are prevalent."
    - IIUC, $\alpha$ controls the non-stationarity of the distribution and lowering $\alpha$ brings us closer to a stationary data distribution, which is by definition outside the scope of Continual learning.
summary_of_the_review: I think this work as merit and could potentially make a good contribution to the field  if improved.
I don't think it's top-conference material in its current form, however.
I am sorry to the authors, but I do not have more constructive criticism to provide at the moment.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposed 6 rules for robust continual learning evaluation and conducted some experiments. 
main_review: Personally, I think the paper makes limited contributions to the evaluation of continual learning. I am very surprised that the paper didn’t mention different settings of continual learning and seems to mix up the different settings. There are three main continual learning setups, class incremental learning, task incremental learning, and domain incremental learning. Different settings require different evaluation methods. Without discussing the settings, the value of the paper drops dramatically. 
 
I think the proposed rules have some merits but quite limited. Most rules are related to the sequence of tasks, e.g., rule 1 and rule 5. But different sequences of tasks have been considered in testing by several existing papers, e.g., Ke et al. Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks. NeurIPS-2020. 

I also have issues with some aspects of some rules. For example, rule 2 says that providing a random guess classifier conforming to the experimental protocol is important to demonstrate the performance improvement attributable to the method. Personally, I do not see the value of this with the current state of the art. If a system does not do better than random guessing, it cannot beat any baselines. I also do not see that separating a problem into different chunks or tasks has anything wrong because many practical problems are like that. I agree that some mixing of tasks will be useful in evaluation, but that is similar to a replay-based method. 

Rule 3 says testing simple methods, but again several papers have already done so, e.g., multitask learning, building a separate model for each task, and training all tasks incrementally without any forgetting prevention mechanism. 

Regarding task identifiers, I do not see any issue of providing them in training. But for testing, it depends on what setting of continual learning is used. For task incremental learning, task identifiers should be provided because they represent completely independent tasks, e.g., given the same image of a dog, one task may want to classify whether it is an animal or not, but another task may want to classify it as a specific breed of dog. Without the task identifier provided, how to classify it? For class and domain incremental learning, task identifiers should not be provided in testing. 


summary_of_the_review: I think the paper made limited contributions to the evaluation of robust continual learning. It doesn't contain much more than what are already known to the research community. This reviewer also doesn't agree with some points.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a experimental design unification scheme for continual learning methods by introducing six rules and highlight the implications of the experimental settings on the accuracy of different methods. The effect of four types of sequences of incoming data and three types of task identifier is empirically investigated on the accuracy of different continual learning methods.
main_review: The paper addresses an important problem in continual learning and the introduced rules are clear and intuitive. Authors also have done a great job capturing various types of data sequences and task identifiers. The following are the ares of improvement:
- Discussions in the paper are mostly dry and are raised in the vacuum. Providing examples would have been helpful for the following sample cases:
-- P4: "However, in many applications, tasks are either not known". what kind of applications?
-- P6: What are some examples of tasks when "unrestricted scenario" becomes important?
- Naive exploration of the hyperspace: The paper identifies a few experimental design dimensions and multiple variations for each, however, it does not provide much suggestion about how these dimensions and their different variations should explored or which combination should be used to evaluate different continual learning methods. Without shedding light into this matter, the paper is regarded more as a study providing evidence for the role of experimental design on continual learning accuracy which is not a surprise.
- Limited hindsight analysis of the existing work: Section 4 provides experimental results for a few scenarios including unrestricted and reappearing classes. However, the flow of the experiments is not well justified and only limited hindsight analysis is provided (e.g. P7: In the unrestricted setting for splitMNIST, most regularisation results are around 20% which is a similar founding as Farqhuar Farquhar & Gal (2018) as their single head setting) where the previous findings seem to be aligned with the experimental results reported in the paper. It would have been more useful to report cases of misalignment between the experimental results reported in the paper and the existing literature to highlight the importance of the experimental settings on the  previous bodies of works' conclusions.
- Minor comments
-- P2: ensure that the method => ensures that the method
-- Tables in section 4 are not sell explanatory. It's not clear what are the numbers reported in the table and one need to read through the text to figure what they actually represent.
summary_of_the_review: Overall my recommendation is to reject the submission in it's current form, however, if the authors can address the comments I am fine with accepting it as a marginally above the threshold submission. The rationale is that:
1- Robustness of experimental settings for continual learning is important and the paper takes steps towards formalizing and unifying it.
2- Reported experimental results and the discussions (although limited) are still useful for the community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper a) attempts to clarify the methodological assumptions that underlie the experimental results of many papers in the continual learning field, b) proposes a number of rules for benchmark settings and evaluation protocols in order to show the true capabilities of continual learning methods under challenging conditions, and c) benchmarks a number of existing methods following the guidelines in (a) and (b).
main_review: The continual learning literature uses a proliferation of different benchmark settings and evaluation protocols, and it’s often difficult to determine which of these a paper is using without delving into code or reading appendices or even having to email the authors. As the authors of this paper state, these choices can have a dramatic impact on results. So efforts to tidy up this methodolical confusion and to set out best practice, such as the present paper, are very much needed. However, papers of this sort are, in my view, not appropriate for a research-focused conference such as ICLR, as they are purely methodological and don’t offer new techniques or results or analysis. The paper reads more like a thesis chapter than a research paper. It might be better suited as part of a longer journal paper whose main contribution is a new continual learning method. Or perhaps an arXiv paper. The area chair may disagree, of course, as (I’m sure) will the authors. Notwithstanding its suitability for an ML conference, I also feel the paper covers ground that has already been covered by Hsu et al (2018) and van de Ven & Tolias (2019). (Both are cited in this paper. Notably, these are both arXiv-only papers, as far as I know.) Although the present paper offers some further thoughts and some useful new benchmark figures, I don’t feel this is enough novelty to warrant an ICLR paper.

The standard of English is poor. Mostly this isn’t a barrier to understanding the paper (and it doesn’t impact my score). However, there are a few sentences where I cannot even guess the intended meaning. Here are some examples:

p.5, fig.1, caption: “Contrasting, the task identifier have task 1 being larger than other tasks”

p.5, last para.: “We also include entropy for model selection for multi-model and does not have any restriction on prediction”

p.6: “In Section 4, the second case is used and labelled iCaRL using exe.”

p.9: “Reducing assumptions improves the flexibility of the implementations as it remove optimization that rarely holds.”
summary_of_the_review: The paper is a laudable attempt to improve the methodology of continual learning research. But a lot of the ground it covers has been covered in other papers. And it’s more like a thesis chapter / journal paper section than a standalone contribution to the field.
review_title: null

Meta-review: This paper is a scholarly examination for how to conduct continual learning evaluations, proposing six rules that in large part synthesize work from other papers. While there is certainly scholarly benefit to such an exploration, all reviewers believe that the contribution is not substantial enough in its current form to warrant acceptance. It is certainly true that not all continual learning papers follow all of the guidelines/rules for evaluation, and consequently, papers such as this are useful to improve the scientific process.  However, the contribution needs to be substantially deepened, including more extensive and in-depth experiments with novel insights as described in the reviews, before the paper is ready for publication.
Decision: Reject