Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper provides metrics, benchmarks, and algorithms that can evaluate transferability, not just catastrophic forgetting in continual learning. These methods are reasonable, but there seems to be a need for more justification for the five stream cases used in the benchmark for transfer evaluation. This is the main reason for not giving this paper a higher score.

This paper provides a set of benchmarks for evaluating continual learning algorithms in terms of transfer and scalability, in addition to catastrophic forgetting, which has been mainly used in the past. In particular, five specific task streams were defined, and it is argued that these can be used to assess the transferability, which was not distinguished in the existing benchmarks. Additionally, the authors propose MNTDP, analogous to the modular network, as a new continual learning algorithm to better meet the metrics assessed in the new benchmark.

This paper is well-organized and well-written. Defining metrics for continual learning in Section 3 is not very new, but I think it is necessary. I personally like the authors' claim that transferability should also be an important measure in continual learning. However, there seems to be something that needs to be clearer in terms of motivation in the part that proposes five task streams for benchmarking. It would have been nice if there was a discussion on whether the 5 cases provided as a measure for transfer overlap each other or whether there is an area that cannot be covered. For example, in the case of (MNIST -> CIFAR-10 -> ... -> F-MNIST), it is unclear where it corresponds to the five stream cases. It is also unclear how much the performance improvement of the T(S^{}) values in Table 1 actually represents. I think that referencing the transfer learning benchmarks can help both justify stream case classification and compare transfer performance results.

Another question is, in the case of scalability, what does it mean for regularization-based or replay buffer -based rather than modular network-based continual learning?
review_title: This paper provides metrics, benchmarks, and algorithms that can evaluate transferability, not just catastrophic forgetting in continual learning. These methods are reasonable, but there seems to be a need for more justification for the five stream cases used in the benchmark for transfer evaluation. This is the main reason for not giving this paper a higher score.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes modular networks with task driven prior (MNTNP) for continual learning. For each new task, the model will try to add new modules to each layer, and then search for a best path that leads to the best performance for the current task. During this process only the newly added modules are tuned, therefore, the method does not have catastrophic forgetting by construction. The model size is likely to grow through this process. However, due to similarity among tasks, it is likely that some of the modules will be shared between different tasks. Hence, the growth of the size of the model is likely to be sub-linear with respect to the number of tasks.

+ Pros
+ The idea of exploring different modules and structures for each task is interesting

+ The paper is in general well written and easy to follow

- Cons
- The novelty of the proposed method is limited. The only difference it has as compared to [1] seems to be the data prior that it used to get the previous path to start the search.  There is no comparison with [1] or the other similar methods such as [2], and [3].  It is hard to see if the proposed change resulted in a significant benefit for the learning. In addition, the comparison with other methods are not complete. Because some of the method does not add additional parameter for each of the task, it is hard to see if the gain from the proposed method or is more of a matter of adding additional parameters to the model.

- The setting to freeze old path does prevent forgetting by construction, however, it also prevents positive backward transfer as a whole. This takes away half of the benefit from continual learning, and the only thing left is forward transfer. This scheme also limits forward transfer. For example, in the $S^+$ setting, in an ideal case the model can only tune the final classification layer according to this method and lose the chance to improve the overall model. The authors argue that the optimal model for permuted MNIST and split CIFAR are independent models because each task is distinct. I would argue that, for permuted MNIST the semantic information is consistent across tasks, so ideally a continual learning model would only need to re-arrange the initial layers to accommodate the change in surface form and keep the rest of the model intact. For CIFAR the sharing would be more significant, because the input were mostly drawn from the same distribution and it is likely to have sharable information across tasks. In fact, I would argue the split to independent model is no longer continual learning, as the models are already independent, we completely lost the continual part.

- The proposed approach also requires a task identifier to work with. I think in classification setting, which is the experiments focused in the paper, providing this identifier alone already solves a large part of the problem. The setting is more realistic in reinforcement learning setting, however, no experiments were carried out under that setting. 

- It is not clear what algorithm is used in the experiments for the proposed method. Is the data driven prior always applied?


References:
[1] Li, X., Zhou, Y., Wu, T., Socher, R., & Xiong, C. (2019, May). Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting. In International Conference on Machine Learning
[2] Serra, J., Suris, D., Miron, M., & Karatzoglou, A. (2018, July). Overcoming Catastrophic Forgetting with Hard Attention to the Task. In International Conference on Machine Learning
[3] Yoon, J., Yang, E., Lee, J., & Hwang, S. J. (2018, February). Lifelong Learning with Dynamically Expandable Networks. In International Conference on Learning Representations.
review_title: Modularized network for continual learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary of paper**

This paper has two main contributions. Firstly, it introduces a new set of benchmarks designed to proble different qualities that may be important for continual learning methods. Secondly, it introduces a new modular method for continual learning, which learns to add modules when learning on a new task. The method reduces the exponential search-space by only considering a few options for how to add the module, leveraging which task the new task is most similar to. Experimental results are provided on the new benchmarks as well as Permuted MNIST and Split CIFAR100.

**Review summary**

I really like the idea behind the benchmark suite for continual learning. Although the algorithm CTrL is neat, I think the experiments/explanation could do with more work (see later in the review). Therefore I am currently leaning towards a weak reject, pending a discussion with the authors.

**Pros of paper**

1. Testing different properties of a continual learning algorithm, one by one, with different benchmarks, is a very nice idea. In fact, it might be nice for the authors to consider expanding the suite of benchmarks to test even more desirable aspects of continual learning / properties of continual learning methods. Some ideas:
(i) Multiple tasks which are from the same dataset, to test how continual learning systems react. Some will deal very well with this, others will not.
2. It's great to see memory and compute reported as benchmarks for continual learning.
3. The proposed modular method is very nicely presented, and is a neat, relatively simple, method. It has nice ideas to it, such as starting with the modular architecture of the most related task only. 

**Cons of paper**

4. The paper defines a transfer metric, using only the final task. This works well with the specific benchmarks they propose. This is very similar to a "Forward Transfer Metric" from Pan et al., 2020 ("Deep Continual Learning by Functional Regularisation of Memorable Past"), which averages the metric over all tasks (except the first task). I was wondering why the authors restricted the transfer metric to be just for the last task.
5. Although the related work section is long with some nice detail in places, there could be more comparisons in the paper to model-based continual learning methods. The paper could also mention methods based on IBP priors.
6. I would like more of a discussion relating to Li et al., 2019, which the authors say is the most related work. It would have been nice to use that as a baseline in the experiments. (Although I understand it is difficult if they do not have code available online.) In general, I would have liked to see some stronger / more recent baselines in the experiment section: Independent and PNNs are known to have bad memory growth; ER is the simplest idea in the replay-based methods; Online EWC is also very simple (and old).
7. I did not understand why the "Independent" baseline in Table 1 has non-zero transfer metrics? (I thought that, because it is independent models, the metric should be 0?)

**Update to review**

I have increased the score from 5 to 6. The authors have made the paper stronger with the inclusion of stronger baselines, cleaned-up presentation, and backward transfer. I am particularly excited by the new suite of benchmarks.
review_title: AnonReviewer2 Review

final_score: rating: 7: Good paper, accept
review_content: The authors investigate modular neural networks in a continual learning (CL) setting, where multiple classification tasks have to be learned in sequence. To avoid the well-known problem of catastrophic forgetting, parameters used by previous tasks are simply frozen, and new tasks are learned by expanding the architecture (as in e.g. PNN, Rusu et al., 2016). The authors focus on efficient forward transfer to new tasks. A many-tasks benchmark is evaluated on the authors' method as well as a number of baselines.

The paper is very well written and the explanations are clear. The experiments are fair and well-executed. The proposed long-sequence streams provide useful baselines that can be used in future papers, as the authors will share their code.

The main weaknesses of the paper lie in its originality and applicability. Dynamic architecture expansion for CL is a well-studied approach (as acknowledged by the authors), and while picking an appropriate initial path with a cheap classifier is a good idea, determining how to expand from there remains difficult and largely expensive in the more effective MNTDP-D version.

Here are some suggested improvements for the authors:
- Baselines: since PNN, improved methods for architecture expansion have been developed. While the provided baselines are interesting and evaluated in a fair manner, I think that the paper would be much stronger if the authors considered as well stronger, more comparable baselines (for example, DEN, Yoon et al., 2018, or CLAW, Adel et al., 2020). 

- Other methods have been developed which leverage reinforcement learning to help determine how to expand a network in a CL setting. These should be at least discussed in the light of MNTDP-S (e.g., Xu and Zhu, NeurIPS 2018 and NHuang, Lavet, Rabusseau, arXiv 2019)

- The proposed method cannot achieve backward transfer. This should be noted.

- It should be discussed that the presented method requires knowledge of task identity at test time. This is not always possible.

- "Finally, the CL algorithm has to yield predictors that scale sublinearly with the number of tasks both in terms of memory and compute."
To make the statement general, I would note that sublinear memory and time complexity scaling with the number of tasks should be the aim whenever the continual learner is exposed to tasks that are not independent.

Edit: I have raised my score from 6 to 7 after the rebuttal.

review_title: Continual learning by expanding modular networks

Meta-review: This paper presents a new benchmark for evaluating continual learning(CL) algorithms on transferability and scalability. It also introduces a data-driven prior to reduce the architecture searching space. Experiments show the new benchmark helps to analyze the properties of CL algorithms and the proposed algorithm performs better than baselines.

The reviewers raised concerns about evaluation metric, weak baselines, and limited experimental cases for evaluating transferability. The authors added more experiments with stronger baselines and revised the paper based on the reviewers' suggestions. However, the authors also admit that how to evaluate transferability is still an open question. 

Despite the concerns, the reviewers generally agreed that the paper is well written,
and the new benchmark is an important contribution for evaluating continual learning algorithms on transferability and scalability.  Hence it makes a worthwhile contribution to ICLR and I'm recommending acceptance of the paper.
Decision: Accept (Poster)