Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper argues that the goal of precisely formulating abstract desiderata for continual learning (CL) is ill-posed because different applications may always favour distinct scenarios. Following this logic, the paper proposes a categorisation system which aims to make CL research comparisons more structured. The paper goes on to review the challenges of meaningful evaluation of CL algorithms, particularly in terms of systematic comparisons of different works with somewhat different sets of assumptions, using the notion of “catastrophic forgetting” as a working example. Aspects which sometimes are overlooked in other machine learning research, e.g. dataset preprocessing and balancing, become important aspects in CL research and cannot be ignored. Furthermore, many machine learning paradigms may have natural extensions to non-i.i.d. learning scenarios, or at least various multi-task formulations of interest. Since many of these satellite formulations inherit metrics and assumptions of their respective paradigms, it naturally creates a problem formulation challenge for CL research.
The paper proposes a multi-objective radar/spider web chart (CLEVA-Compass) as a classification heuristic for CL research evaluation setups. The diagram is based on a well conceived collection of conceptual and practical desiderata for CL algorithms. The paper proposes how such a diagram could be used by future CL research and anticipates unintended uses. Limitations of the proposed classification strategy, as well as complementary challenges of dataset design, are discussed towards the end of the paper.

main_review: Pros:

The paper correctly identifies current challenges of defining CL research evaluations. The hypothesis that the abstract CL setting is a non-trivial multi-objective optimization problem has been well articulated in a review paper in Trends in Cognitive Sciences by Hadsell et al. 2020, but probably it has been given in several forms much earlier. The non-trivial term here has the technical meaning that no one solution exists which dominates all other solutions in terms of all objectives, hence trade-offs are needed. The only way to set trade-offs is to go to specific application domains and linearise the multiple objectives according to the needs of each domain. In that sense, the paper is correct that several rational definitions of CL are needed, assuming that the hypothesis outlined by Hadsell et al. is correct, as mounting evidence seems to suggest.

The paper coherently, albeit briefly, reviews the literature, ultimately arriving at a classification strategy. This type of review works help illuminate other sciences, e.g. neuroscience, and usually have dedicated publishing tracks in journals. Indeed, most practitioners in those disciplines relish review papers. Perhaps it is time for such tracks in machine learning main conferences, especially for subfields which deal with fundamental issues such as agreeing on definitions and evaluations.

Cons:

No formal system is given to precisely define terms, or at least outline how one should be developed. While this may be out of scope for one conference paper, the lack of such a system is an important source of confusion in CL research. At the end of the day any learning system should either optimise a precise criterion, at least approximately, or no criterion at all. This should be made clear upfront, with a discussion of all the assumptions such learning settings make.



Note to authors:

While the authors wish that the proposed categorization will not lead to a flurry of SOTA claims, it is safe to say that SOTA claims have already been made for most ways to cut the CL “cake”; even worse, SOTA claims are anecdotally thought to be a requirement for CL works to be published in main ML conferences. Hence, while the authors’ intentions are noble, the incentive structure may already be established against such hopes. At the end of the day, if quasi vacuous SOTA claims are what it takes for progress to be made in CL research, it is hard to argue against it, noble principles be damned. One may also take the view that SOTA claims made based on apple-oranges comparisons are not productive and may even stifle progress. However, the authors of this paper find a third option, proposing that such comparisons are productive exactly because they are correct if the “cake” is cut finely enough. While this entire paragraph of the review is somewhat in gest, it is included in order to serve as one of many possible subjective evaluations of the state of affairs in CL research, a path which was paved with good intentions.

Perhaps a fourth possible solution to this conundrum would be to acknowledge limitations of current algorithms and evaluation metrics, and naturally replace grand claims with precisely defined setups which aim to solve CL problems which other ML researchers encounter today:
For example:
Supervised or semi-supervised learning of very large-scale models on very large-scale (mostly unbalanced) and ad-hoc created datasets requires many passes through these datasets. Very little progress can be made in one pass, and SOTA continual learning algorithms do not seem to help.
Standard deep reinforcement learning *is* a continual learning problem with its own set of trade-offs due to agents being required to sample their own training data. Yet most current CL research does not seem to apply or help in practice.
Multi-agent RL is a highly complex data distribution shift problem. Training GANs or adversarially robust models against many types of attacks are natural dataset shift training scenarios, yet little help seems to be available from CL research.

The question arises: is a more structured classification of ongoing CL research into more fine-grained and fragmented  tracks going to improve the current state of affairs, making CL research more useful for ML in general? I personally find this question difficult to answer ahead of time, and will leave it for the research community.


Misc:
The A-GEM reference on page 8 (Chaudhry et al., 2018) may not be entirely accurate. I believe A-GEM was introduced in (Chaudhry et al., 2019), perhaps a small confusion could have occurred there.

summary_of_the_review: I believe it is timely to allow review papers into main ML conferences, so on balance I vote for acceptance. It is worth mentioning that several position papers cited in the text, e.g. Pineau et al. 2021, Mitches et al. 2019, Bender & Friedman, 2018 have all been published in journals and/or conference proceedings.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach for a more nuanced assessment of continual learning which provides a visual representation that enables the identification of a given method's context with regards to the broader literature, and enables the comparison of two methods in terms of reported metrics.
main_review: The problem of relevant evaluation of continual learning approaches is a major one, and worth addressing. The authors do a great job presenting the different aspects of the evaluation (Fig 2) and explaining why they are meaningful. The distinction between static and continual evaluation is important, and definitely impacts the evaluation methods chosen and the metrics that can be used for comparing across them, as the authors rightly note.
I particularly liked this statement: "it is typical for approaches to be validated within train-val-test splits, where either a model-centric approach investigates optimizer variants and new model architectures, or alternatively, a data-centric approach analyzes how algorithms for data curation or selection can be improved for given models" -- it is a great way of comparing and contrasting different approaches. (I would argue that many of the arguments that the authors raise for continual learning apply for ML in general to different extents, actually, since in practice ML approaches are deployed in a way that is similar to continual learning -- i.e. production-ready ML systems are a often tacitly a form of continual learning without explicitly stating so.
I appreciated the open and transparent approach proposed for CLEVA. One thing I'm not 100% sure about is how practically it can be used, i.e. operationalized. This will undoubtedly become clearer when the code is released, but will it be via a UI or something similar? I feel like just showing the compass itself would not be enough for a full understanding of its scope, and how it can be applied. 
I do find that there is a gap between theory and practice in the current proposal (which is highlighted quite well by the authors in section 4), and additional work is needed in order to better plan out its application in practice. i.e. "The CLEVA-Compass should thus not be used to conclude a method’s superiority and its utility will depend on faithful use in the research community." -- this is an important aspect of model evaluation that researchers will be looking for, and so if CLEVA doesn't allow it, this can get in the way of its widespread application.
This statement is not clear to me: "We believe it is best to avoid attempts at combining the prior works of the above paragraph with
the CLEVA-Compass." -- why is this the case?

summary_of_the_review: This is an interesting approach to evaluating continual learning. The transparent and multi-faceted approach is interesting and unique, and can be truly useful for those working in the community.
I feel that there is, however, a gap to be addressed between the more theoretical (and informative) usage of CLEVA and the more applied usage of it as a tool in guiding ML research. While section 4 of the paper does address this to some extent, the fact that the authors do not recommend using CLEVA to compare models, or alongside other works cited in that paragraph -- this would merit some additional thought and development.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes CLEVA-compass which provides visual means to easily compare different continual learning works and provides a checklist to promote results reproducibility. Indeed, existing variations in the problem setup and evaluation of continual learning make the direct comparison of works in this field challenging. CLEVA tries to summarize each work in two different level: 1) each work's influential paradigms and items and 2) the measures reported in the work that can be used for reproducibly.
main_review: The work has nicely depicted different machine learning paradigms and their relationships with continual learning. Also it represents the difficulties and challenges arises from the continuous nature of this paradigm which makes the comparison of existing works challenging. The figures have efficiently summarize the main components in the paper. However, some main points of the research still remains unclear after reading the paper. The unclear parts are mentioned here:

1) In CLEVA there are two inner and outer circles which show whether a component is achieved in a supervised way or without supervision. Even-though an example is provided further explanation of this two layers can be beneficial to get the importance and difference of this two levels.

2) Highlighting influential paradigms and items that their extraction from text is challenging and sometimes misinterpreted is the only benefit of using CLEVA? The main positive effects and gains that CLEVA has still is not clear to me. 

3) Authors stated that CLEVA can not be used for indicating the context that the proposed continual learning works can be applied, or showing superiority of one work versus other than, therefore how can it be used for assessment? In other word, what are the main advantages of CLEVA which can encourage researchers to use that for presenting their works.


  
summary_of_the_review: The main motivation and the benefits of the proposed compass is not clear. As an assessment tool the main advantages that this work can provide to the community is not clear to me. The better comparison of four works in terms of this compass in the text could be one way to show the superiority of the world of using CLEVA versus not using it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work, the question arises of a better way to evaluate different methods in Continual Learning. The authors explain the problems of having a correct evaluation in Continual Learning, describing the complex scenario that Continual Learning confronts by being the intersection of multiple related areas. With this motivation, the authors present CLEVA-Compass or Continual Learning EValuation Assessment Compass, a visual representation that improves the comparison and transparency of different methods in Continual Learning. CLEVA-Compass includes two levels. The inner level shows which paradigms influence the method, and the outer level shows the setup and evaluation metrics presented in the method.
main_review: Strength:
- The paper is very well structured, clearly explaining the authors' concerns about the evaluation in Continual Learning. This structure helps to motivate the problem and present the solution adequately.
- I agree with the authors that comparing methods is not trivial. Different methods take different assumptions on the same problem, focusing on different metrics, which cause confusion and make the comparison tricky. CLEVA-Compass is a clever way to mitigate this problem by clearly identifying which metrics are reported and the contexts of the experiments.  
- A significant problem raised by the authors is the transparency of the evaluation. For the most part, current incentives are to improve accuracy and decrease forgetfulness, often making unfair comparisons between methods or creating new metrics to "surpass" previous methods. Using CLEVA-Compass makes it more explicit about the context that each method is experimenting with, mitigating unfair comparisons.
- The CLEVA-Compass visualization is an excellent way to summarize information in a single visualization. Comparing more than two methods on the same inner level can be confusing, but multiple visualizations can solve this problem.

Concerns:
- My main concern with CLEVA-Compass is the selection of the different paradigms and metrics of the inner and outer levels. I understand that this is a vision of the authors, showing the areas and metrics relevant to Continual Learning. However, Continual Learning is inevitably evolving and merging with new scenarios, for example, self-supervision. As mentioned in the paper, CLEVA-Compass can evolve to adapt, but this can encourage people to create new factors (or metrics) that benefit their proposed method. Understanding that the latter is part of what you want to avoid with CLEVA-Compass: How are you planning to adapt your compass continuously? Or do you hope that the community will manage these modifications?
- A minor concern I have with CLEVA-Compass is the definition of the inner level points. I may not have understood it correctly, but I have two doubts:
    1. The definitions of each paradigm depend only on the definition given in Figure 3? These points may have ambiguous areas where concepts overlap and thus generate confusion.
    2. Regarding the 3 phases in the inner level, I think it can generate confusion when an unsupervised method is used in one area and not in another (as shown in the example of page 7, paragraph 1). If the area is ambiguous, in which part of the visualization the point is.
I am not an expert in all paradigms mentioned, so this ambiguity may not exist. Still, it is a concern that I have.

summary_of_the_review: The paper addresses a crucial topic in Continual Learning, and I think it may interest many people. Evaluating and comparing different methods transparently is relevant not only for Continual Learning but for all areas of Machine Learning, as mentioned by the authors. My only concern is the questions I leave to the authors, which I hope they can answer.
review_title: null

Meta-review: This review paper presents a way of comparative assessment of continual learning. Reviewers all agreed that this work is interesting, unique with comprehensive coverage of the CL space. The proposed categorization, CLEVA-Compass, and its GUI have great potential to facilitate future CL work.
Decision: Accept (Poster)