Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper argues and demonstrates that in a task-incremental continual learning setting, in many cases it is beneficial to split up the capacity of a learning algorithm and to learn separate models for different tasks. Building on this insight, and taking inspiration from the boosting literature, the paper then proposes the method Model Zoo, in which a new model is learned for each new task and whereby each new model is trained on multiple tasks. During evaluation, the predictions of all models trained on the task under consideration are averaged. The paper reports surprisingly large improvements over existing continual learning methods on several different task-incremental learning methods, both with and without utilizing replay.
main_review: Strengths:
-	The proposed boosting-style algorithm for task-incremental learning is an original and important contribution to the continual learning literature.
-	The surprisingly large gap in performance between the Isolated learner and the compared existing methods is intriguing.
-	The theoretical analyses and considerations in Section 2 are interesting, and I think they provide important insights and food-for-thought for many continual learning researchers.

As stated above, I believe this paper makes important and insightful contributions, but there are several issues that should be addressed before this paper could be published at this conference:
-	The paper suggests that it covers virtually all continual learning settings out there, but the paper ignores several important settings. Specifically, a critical assumption that this paper makes for all its experiments is that task identity is always provided to the learner (i.e. task-incremental learning). The paper does not consider settings where task identity is not provided (i.e. domain- and class-incremental learning; https://arxiv.org/abs/1904.07734). In a footnote it is argued that without the assumption of task identity being provided continual learning is impossible in the case of permutation of classes; but it is unclear why this argument is relevant as none of the experiments considered involve such a permutation of classes. It is important that it is made clear that the insights and observations of this paper are specific to the task-incremental learning setting.
-	At several places, the paper makes claims about performing better than all existing methods, on all datasets and on all benchmarks. Based on the reported experiments, these claims are too strong (e.g. it is clear not all existing methods are compared against), and I also think that these claims are unlikely to be true (e.g. see below). These claims should therefore be moderated.
-	Perhaps most importantly, I have serious concerns about whether the comparisons against existing continual learning methods (e.g. in Fig 1A, Table 1) are done fairly. Firstly, for some results it is stated that they are taken from other publications. This seems problematic, as the mentioned publications use different architectures and optimization methods. It is therefore unclear whether the reported difference is due to these differences (which are arguable unrelated to continual learning), or due to the continual learning strategy. Secondly, I wonder how the other results (i.e. those for which it is not stated that they are taken from other publications) were obtained. I could not find any description of this in the paper, and also in the submitted code there does not seem to be any code for these other methods.
I am particularly worried about this because from Fig 1A, it seems to be the case that the main part of the reported improvement over existing methods is due to an improved ability to learn individual task (e.g. compare all accuracies after just the first), rather than due to an improvement in forward/backward transfer. That is, I am afraid that the main improvements reported in this paper might be due to network architecture/optimization choices that are not interesting from a continual learning perspective.
-	An important class of methods that is not considered in this paper is generative replay (Mocanu et al., 2016 https://arxiv.org/abs/1610.05555; Shin et al., 2017 https://arxiv.org/abs/1705.08690), which does not store data and could be interpreted as adhering to the “strict formulation” of continual learning. I think it is important to at least discuss these methods as I am positive that some more recent generative replay variants (e.g. van de Ven et al., 2020 https://www.nature.com/articles/s41467-020-17866-2; Liu et al., 2020 https://arxiv.org/abs/2004.09199) could outperform the Isolated baseline.
-	It is claimed that the “Multi-Head (multi-task)” baseline is not a continual learner because it is jointly trained on all tasks. However, it seems straight-forward to adapt this baseline in such a way that it is, for the purposes of this paper, a “legitimate continual learner”: the tasks are visited sequentially as in the other baselines, and on each new task, the model is trained using the data of all tasks seen so far. (Or in other words, always all previous tasks are replayed fully.)
-	It is unclear to me how Model Zoo with limited replay is performed, and it would be good to provide some more details. In the text it is stated that only a subset of the data is used in equation (7), but I think the authors probably meant to refer to another equation?
-	The claim in the title that the proposed Model Zoo is a “Growing Brain” does not appear to be supported or justified anywhere in the text. I think it would be good to either modify the title, or to justify and discuss the reasons for this comparison in the text.

summary_of_the_review: Although I believe this paper provides several important insights, there are several critical issues that prevent me from supporting acceptance of this paper. I am happy to engage in a discussion with the authors, to see whether my concerns can be taken away or whether the paper could be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper begins with a clear exposition of continual learning and an analysis from a statistical learning theory perspective.

It then introduces the Model Zoo: A sequence of small models that are each trained on a subset of the available tasks at that point (with a separate linear classifier for each task). The tasks selected for training are those that have high loss under the models trained so far (similar to boosting).

The following section contains an exhaustive set of experiments that compares the Model Zoo with a variety of baselines (with and without replay, with and without limited training, a single model with multiple heads, etc.). Surprisingly, the "Isolated" model (i.e., a separate model is trained on each task) does better than most baselines that train on multiple tasks. The Model Zoo outperforms all other baselines and even outperforms the multi-head baseline on ImageNet.

The paper ends with a critical discussion on how the problem of continual learning should be approached.
main_review: I am not an expert in continual learning, so there are details that I might have overlooked. Overall I think this is a strong paper.

One of the confusions I had as a reader: The authors talk a lot about task synergy and competition (e.g., sections 2.2 and 2.3, theorem 2, figure 3, etc.) However, in the end it seems that the method does not use task synergies/competitions at all to decide which tasks to train on. Instead, tasks are more likely to be chosen if they have high loss under the existing model (equation 8). I would appreciate a bit more discussion about this. Is the current method a proxy for selecting synergistic tasks? If not, how is the preceding discussion relevant?
summary_of_the_review: I think this is an excellent paper. It is well-written, the proposed algorithm has strong results, it asks important and critical questions about continual learning research, it validates the proposed model with a large amount of experiments, is well-referenced, and provides source code and 14 pages of appendices to ensure reproducibility of the results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes yet another definition of continual learning (CL) and uses it to motivate a divide-and-rule approach. The algorithm exploits all assumptions of the adopted CL setting, including the relaxation of data access rules, in order to perform better compared to algorithms designed for more restrictive settings. The paper concludes that previous CL settings were holding back progress. Empirical evaluations and ablations are given using the new CL setting proposed in this paper, with the help of datasets favorable to this approach, namely splits of existing i.i.d. datasets.
main_review: Pros:

The paper makes a very good case for the difficulties of sharing parameters between different tasks, which are amplified by sequential data access restrictions, especially without task labels. However, the paper doesn’t address many such issues, and simply goes on to assume many of these limitations don’t exist. That is fine, as long as the setup is clear from the start, and the tone of the paper reflects the reality of results, i.e. a simpler problem can be solved better than much harder problems.

The results provided show that even in favorable settings, e.g. splits of i.i.d. datasets, parameter sharing may not be optimal, even if data access restrictions are confined only to future data in sequential learning. This is an important contribution, but may not support all the claims.


Cons:

According to this paper, a continual learner’s only limitation is not knowing all data from all the tasks not yet seen; essentially, according to this definition, a continual learner’s only limitation is not knowing the future. Standard sequential learning settings used in the literature are borrowed and simplified by removing data access restrictions and giving full information about past tasks. Other implicit assumptions are made but not all are properly discussed: 
- Data imbalance issues are not mentioned explicitly. What is the meaning of learning both MNIST and ImageNet?
- Task sequences considered are favorable to transfer and no attempt to study adversarial tasks is made;
- It is assumed past data can be neatly sorted into i.i.d. buckets (separate tasks) with known labels;
- Task losses aren’t always comparable in general, e.g. in terms of scale or learning dynamics; MNIST and CIFAR-10 have quite different error curves during learning; what is the meaning of the average joint objective?
In my humble opinion the paper should give a thorough discussion of limitations and appropriate qualifications for all claims.

Writing and framing issues:

The paper simply redefines continual learning in contradiction to a majority of recent literature. For example, the first paragraph of the paper gives a non-standard definition of continual learning: a continual learner isn’t widely defined as an algorithm which “leverages past data to learn new tasks”. While proposing a new definition is very interesting and indeed a potentially important contribution, the presentation of this new definition is at best misleading, significantly reducing the accuracy of the writing.

Many terms, e.g. “task relatedness”, task “competition for capacity”, “synergistic data” are offered without precise definitions or associated references. While difficulties in producing such precise definitions may be well understood, they don’t seem to temper the tone of the paper, nor do they seem to inspire the authors towards ideally a more humble, or at least balanced view of existing literature and the plurality of opinions within. The authors describe different viewpoints as an “intellectual gap” in research, which is not very helpful.

The authors appear to misattribute properties of the learning setting as qualities of the proposed algorithm, which I believe is done in error. The adopted CL setting offers data neatly split into labelled i.i.d. buckets. This does not mean that the data selection heuristic will always be effective, especially when these implicit assumptions are violated. However, the paper doesn't discuss such issues at length.

While the comments about focusing on backward and forward transfer are well taken, it is important to note that the proposed solution heavily relies on the definition of multi-task learning given originally by Caruana (1994), i.e. learning from multiple tasks independently as best suits each individual task. There is no guarantee that the proposed algorithm achieves this, however. Empirical results provided do not give good enough reasons to believe so because all experiments rely on splits of single datasets into well balanced bins with similar data, e.g. preprocessed in the same way, similarly centered and scaled, etc. Because the dataset shift is quite low between splits of X, it is really hard to tell how effective the proposed method is in the general case under its own assumed (much more permissive) CL setting. Indeed, at the beginning of section 3 the authors express hope that interference doesn’t happen, but offer no formal guarantee and do not take on an empirical challenge where the algorithm could demonstrate performance under less than ideal conditions.

It seems that the choice of how much data to use from tasks with higher loss is left as a hyper-parameter, and hence depends strongly on the data stream. If one can expect little interference, then many sources can be combined without much hindrance, as the paper observes; but this is not necessarily true in general. In fact, such hyper-parameters express the worst case expectations, which in general can be very negative. The paper does not discuss this, or report such results.

summary_of_the_review: This could be a great paper. It is important to understand what works in practice with current algorithms and architectures. Analysis of worst case performance is not the only important case in many practical settings. Hence, I lean towards acceptance despite the gross lack of nuance and generally imprecise writing. As I said, the authors could turn this paper into a great contribution by providing nuance instead of hyperbole, deference to the audacity of other researchers to face challenges which the authors avoid axiomatically, instead of off hand comments. I will (reluctantly) not oppose publication close to current form, but I cannot confidently recommend acceptance before the writing is improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides two main contributions on the topic of continual learning: (i) the first is to propose a definition of relatedness between tasks in the framework of statistical learning theory, and use it to perform a theoretical analysis of when it can be fruitful to train multiple tasks with the same model and when it can be detrimental to do so; (ii) the second is to propose an architecture for continual learning called ModelZoo, which maintains an ensemble of models that grows as each new task is introduced. Each time a new task arrives, a feature generator h and k task-specific classifiers g_k are initialised, and training proceeds on the combination of h and each g_k on the current task and data from k-1 previously seen tasks. At any point, the performance on a previously seen task can be evaluated by averaging over all small models previously trained on that task. ModelZoo takes inspiration from AdaBoost in how it selecting which previous tasks to train concurrently with the current one, which is by preferentially sampling tasks that have a bad performance under the current ensemble. An empirical analysis is performed by training on various standard continual learning image classification tasks, which demonstrate that Model Zoo is on par with and sometimes outperforms the single model multi-task approach (both , which is often used as an “upper bound” on continual learning methods. The results also demonstrate that a simple baseline of an ensemble of small isolated learners outperforms a selection of existing continual learning methods (with results reported reported from other papers).

main_review: This paper provides both theoretical and empirical insights to support the claim that splitting the capacity of a model into several small learners can benefit continual learning, which have the potential to guide future continual learning research. While the empirical results to some extent corroborate the theory, the link between the outperformance of Model Zoo and the reasons predicted by the theory were not entirely obvious and could potentially be clarified with some additional experiments. Additionally, the experiments were independently lacking in some ways detailed below. I would consider increasing my score if these issues are largely addressed with revisions. 

Comments:

- The theoretical analysis appears accurate to me and it demonstrates that if tasks are sufficiently related, then training them together with a shared feature extractor can benefit improve generalisation, but that if they are not, it can harm generalisation. The benefits of a shared feature extractor among some tasks is reflected in the design of ModelZoo, which trains a shared feature extractor along with task-specific classifiers for a subset of tasks at each round. Furthermore, the empirical results that the ensemble of isolated learners outperforms many existing methods and that ModelZoo sometimes outperforms the multi-task model provide evidence to the theoretical result that claims that it is not always beneficial to train all tasks in a single model. What was not entirely clear was why selecting tasks with high empirical risk under the ensemble is a good heuristic for selecting a subset that are synergistic in the way described in the theoretical analysis. Could it not be that, to paraphrase Tolstoy, each unhappy task is unhappy in its own way? The intuition here is was not clear to me. Two possible ways to empirically demonstrate that this method selects tasks that are more likely to be synergistic are:
    - Perform an analysis to show that groups of tasks selected by ModelZoo exhibit more transfer between each other than randomly selected groups of tasks.
    - Train an ablation to ModelZoo where the tasks are uniformly selected for training, rather than biased by empirical risk, and compare performance to model zoo.
- The experiments were lacking in a number of ways:
    - It is claimed that this paper is the first to perform continual learning on coarse cifar-100 (according to super-classes) but this does not appear to be true (see [1,2,3]).
    - In Table 1, no accuracy numbers are given for any of the competing methods on Split-CIFAR 10 nor for Coarse CIFAR-100, and only two methods are reported for Split MNIST, which is a very common benchmark in the literature. Split CIFAR-10, for example, was trained on in [7,8,9].
    - In Table 2, for the multi-epoch runs, no accuracy or forward/backward transfer metrics are reported for the competing methods, so it is difficult to compare them in terms of computation and storage to the methods proposed in the paper.
    - The definition of forward transfer is unconventional in that is simply the “accuracy on a new task when it is first seen”. Typically, as in [7], you would compare the accuracy on a task after having trained on each of the previous tasks compared to a baseline performance on a randomly initialised network.
    - None of the competing methods are reimplemented - all results are taken from the original papers or other ones that have reimplemented them. This is okay but it makes it difficult to compare them directly to the methods in the paper, e.g. to compare inference times, the models should ideally be trained on the same hardware or if not the differences should be clarified. Also, it is not clear if the training is equal across approaches in terms of how much replay data is afforded to each of them.
    - It would have been useful to empirically compare to at least one or two other continual learning methods that use ensembles (e.g. [4,5,6]) or at least to mention that other ensemble-based approaches to continual learning exist in the related work.

[1] Yoon, Jaehong, et al. "Scalable and Order-robust Continual Learning with Additive Parameter Decomposition." International Conference on Learning Representations. 2020.

[2] Yoon, Jaehong, et al. "Federated continual learning with weighted inter-client transfer." International Conference on Machine Learning. PMLR, 2021.

[3] Shanahan, Murray, Christos Kaplanis, and Jovana Mitrović. "Encoders and Ensembles for Task-Free Continual Learning." arXiv preprint arXiv:2105.13327 (2021).

[4] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3366–3375, 2017.

[5] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations (ICLR), 2020.

[6] Germán Kruszewski, Ionut-Teodor Sorodoc, and Tomas Mikolov. Evaluating online continual learning with calm. ArXiv preprint arXiv:2004.03340, 2021.

[7] Lopez-Paz David and Marc'Aurelio Ranzato. "Gradient Episodic Memory for Continual Learning". European Conference on Computer Vision. Advances in Neural Information Processing Systems. 2017.

[8] Rebuffi Sylvestre-Alvise, Alexander Kolesnikov and Christoph H. Lampert. "iCaRL: Incremental classifier and representation learning." arXiv preprint arXiv:1611.07725, 2016.

[9] Zenke, Friedemann, Ben Poole, and Surya Ganguli. "Continual learning through synaptic intelligence". International Conference on Machine Learning. 2017.

summary_of_the_review: This paper provides both theoretical and empirical insights to support the claim that splitting the capacity of a model into several small learners can benefit continual learning, which have the potential to guide future continual learning research. While the empirical results to some extent corroborate the theory, the link between the outperformance of Model Zoo and the reasons predicted by the theory were not entirely obvious and could potentially be clarified with some additional experiments. Additionally, the experiments were independently lacking in some ways detailed below. I would consider increasing my score if these issues are largely addressed with revisions.
review_title: null

Meta-review: The paper sparked a very substantial discussion, not just about its scientific content, but also, as the authors will have seen, from the narrative standpoint. I would like to thank the reviewers who devoted time and efforts to discuss the paper’s content. I encourage the authors to polish further their paper prior to camera ready, using the numerous scientific comments made (e.g. R6VS, KWuM, the refs of Cn5V). 

AC.
Decision: Accept (Poster)