Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new algorithm for addressing continual learning (CL) by leveraging modular architectures. In contrast to (the few) existing modular CL works, this one focuses on the task-agnostic setting, where the agent is not informed of the task ID. The proposed architecture assigns two components to each module: a functional component and a structural component. The latter is in charge of locally determining whether each data point (actually each mini-batch, in practice) is in-distribution for the given module. The training proceeds in stages, adding new modules whenever the new data point is out-of-distribution for all modules, then training the module such that its output is in-distribution for modules at later depths, and finally training the entire model together. Extensive experiments are carried out to demonstrate the strengths of the proposed approach.
main_review: ############## Strengths ##############

1. The problem setting is highly relevant and vastly understudied
2. The proposed approach addresses one particular shortcoming of existing modular CL methods: the ability to work without task IDs
3. The experiments are extensive, comprehensive, and well described and analyzed

############## Weaknesses ##############

1. The description of the technical approach is too short and somewhat hard to follow, making several choices unclear and unjustified
2. The continual meta-learning experiments seem disconnected from the rest of the paper to me
3. The method has no mechanism for limiting the amount of experts that are added, which is empirically demonstrated to lead to a large number of modules in the only experiment where this is evaluated

############## Recommendation ##############

This is a solid submission and I recommend accepting it. While technically the submission doesn't introduce a fancy new method with many technical contributions, it does demonstrate a clear understanding of the challenges of modular CL, and addresses one of the shortcomings of existing modular CL approaches. To me the biggest plus of the manuscript is the quality of the empirical evaluation, with well-chosen baselines evaluated on a variety of settings. 

############## Arguments ##############

On the technical side, the key contribution is proposing for each module to individually select whether each data point should be routed to it. This is a relatively minor technical contribution, but I do not think that is necessarily problematic given that this is a fairly new problem setting. What does concern me is that this section is very short and I don't believe that it is described in enough detail. In particular, no intuition behind the structural loss is given. I make out between the main text and Appendix A.3 that the idea is for this module to serve as a density estimator that detects out-of-distribution samples. However, it is unclear how the losses described in Sec 3 achieve this, or why different losses must be used for the feature extractor and the outputs. 

One interesting technical insight is that training the structural component locally and with a separate loss than the prediction loss enables fully-differentiable training even if using hard modular selections. The authors also clearly explain that the key challenge for learning a task-agnostic modular component is that this component itself can suffer from forgetting if using standard gating functions, which is not the case with the local approach proposed here. The projection phase is an interesting idea to ensure that the new module is compatible with other modules, and this is possible also due to the choice of local module selection.

The empirical evaluation is very strong. The first experiment on CTrL (a benchmark specifically designed to test properties that modular CL methods should exhibit) compares against a wide array of well-chosen baselines, including the two primary related works (Mendez & Eaton and Veniat et al.). Results in Table 1 are strong and convincing. Then the second experiment proposes a benchmark specifically for out-of-distribution generalization, which was not studied in either of the preceding modular CL works, and the authors again demonstrate a strong ability to generalize OOD. The third experiment demonstrates that independently trained models can later be combined to achieve equal-or-better performance than the original separate models. 

That being said, I recommend removing the final experiment on continual meta-learning. The problem setting for this experiment is unclear (even though it's somewhat described in Appendix E). But the main problem is that this is a completely different problem setting than studied in the rest of the paper, plus even the algorithm is modified to deal with the modified problem setting. It seems disconnected and contributes to the extreme length of the paper (26 pages including appendices).

One concern which the authors only mention in the conclusion is that the model grows quite rapidly. This is demonstrated in the results of Table 1, where the proposed method uses nearly one new module per task, just like the MoE. While in principle using modular architectures enable sub-linear growth (as the authors mention) it isn't clear whether their proposed method is actually sub-linear in practice. Note that having fewer than one new module per task is _not_ equivalent to being sub-linear (e.g., maybe the model learns 0.5 new modules per task, which is still linear). Part of what makes this such a major issue is that the model is allowed to grow at every sample (or mini-batch), which is quite frequent.

############## Additional feedback ##############

The following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.


Intro
- "Modules self-determine their relevance given an input" -- sounds very similar to the competitive mechanism of RIMs [1]. Perhaps a comparison is warranted.

Sec 3
- Equation 7 doesn't make it clear at all that the structural components are only trained via the structural loss. With soft modularity (as used in most experiments), the first term (the overall loss) is also differentiable w.r.t. the structural parameters, but later it is made clear that this is not used. It might be best to present this as two separate objectives used to train disjoint sets of parameters.

Conclusion
- I like how the conclusion points to actually relevant directions for future work.

Typos
- CtRL is used multiple times, but it's CTrL

Appendices
- The entire meta-CL addition is very long, and I would frankly recommend removing it from the paper. It's a whole new problem setting (which I don't think is described in full detail) with some changes in the algorithm to address this particular problem setting.

[1] Goyal et al. "Recurrent Independent Mechanisms," (2020).
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a continual learning method by learning module networks and a way to compose them. One notable difference is that the proposed approach does not reply on an external task id as the input. Instead, it learns to predict whether the input batch can be composed with the current modules or need to add another module. The authors show several experiments to validate the performance.
main_review: This proposed method serves as an important step toward a more general continual learning setting where task identity is unknown. And the idea is pretty simple and reasonable: by tracking the mean and variance of the structural score output by each module, the model itself could predict whether new module is needed.

However, I had a hard time understanding the key part of the proposed method, which makes me less confident about such an assessment. Specifically, I’m confused about section 3.2 and appendix A.2. Could the authors clarify how the projection and accumulation work? When does the projection and accumulation happen? 
- Does that happen at every iteration?
- If not, does that only happen during a new task is added (or during an outlier z score detected)? If so, in figure 2, why the projection only happen in the last layer for t=1 but happen two times for t=3 (I think it’s typo and should be t=2)?
- And what is the output space of the whole network? For example, in the CTrL benchmark, is the final layer of your network a 5-way classifier? If so, does that mean the expansion of final layer is not according to the outlier detection but the task identity?

Another weakness is that the method only shows the continual transfer learning in MNIST. In contrast, the recent method (such as [63]) do the same task on CIFAR and miniImageNet. I would highly encourage the author do more experiments on larger scale and more complex datasets.

Overall, I think this is an important methodology improvement. However, I don’t think I fully understand the details so I will keep my score as borderline for now.
limitations_and_societal_impact: I didn't find any potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tries to tackle continual learning with a modular approach and proposes the local module composition (LMC) approach. LMC composes modules dynamically by computing local relevance scores for each layer, computed with the input. This architecture makes LMC perform well for OOD tasks as well as outperforming existing approaches on a continual learning benchmark.
main_review: Originality & Significance
- Although modular approach is not a new concept in continual learning, LMC allows a soft-gating algorithm that is input-dependent rather than task-dependent. Combining modular expansion with the gating algorithm is the primary contribution of the work in my opinion. The gating algorithm is primarily based on the work of Hocquet et al. [30] and Aljundi et al. [2]. The results demonstrate how LMC can be useful for OOD tasks, but in the usual CL case where task identities are known, it does not show significantly superior performance over other methods.  Overall, this combination of modular architecture with the gating algorithm leads to some interesting results.

Quality
- This paper proposes a new modular approach for CL and shows outstanding performance especially for OOD tasks which I find interesting.
- One of the main benefits of modular approaches is how certain modules can be reused for quite different tasks enabling better parameter sharing and generalization. By looking at Appendix B.5, it seems modules typically specialize into different tasks without much reuse except for very related tasks. An analysis on this would be beneficial.
- This leads to a question, can the superior performance of LMC versus other baseline models be attributed to more trainable parameters? - it is hard to judge without knowing what kind of architectures/parameter counts were used for baseline models. 
-  The l2-norm structural objective function could cause dead modules in principle - certain modules may always output 0 and not used, for example at some layer, out of N available modules, only one might be always chosen if N-1 modules are optimized to always output 0s. This could limit the capacity of the network even though the whole network has a large amount of trainable parameters. Was this observed in experiments or how was this avoided?
- In my opinion, the major benefit of modular approach in continual learning is how it does not forget when given a lot of different tasks. However, only five tasks are considered in this work. It would be great to consider the S^{long} setting as in MNTDP paper which this work compares with the most.
- LMC has the additional step of adapting to a new dataset through the projection phase - which I like. To demonstrate the usefulness of the whole architecture, it would be good to see if it can significantly outperform MNTDP when MNTDP is given a chance of adaptation too - for example by having a normalization layer for the new dataset to match the statistics of the previously seen datasets. 
- Figure 3 has the same tables for (a) and (d) and this should not be the case.

Clarity
- The paper is well-organized and written. It was easy to read
limitations_and_societal_impact: Yes, they are included.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors propose an algorithm to perform continual learning without task-id labels at inference. The method involves local module-level OOD estimators which then output linear coefficients that informs the model how to combine outputs. The model also has the option of creating new modules which are first trained in a projection phase from downstream layer signals before starting training on their own. The proposed method is tested in standard continual learning benchmarks as well as OOD detection and meta-learning settings.

main_review: Overall: The proposed method is simple and combines technologies in a reasonable way to achieve good results on a number of settings. I appreciated the breadth of experiments and various supplementary experiments (OOD detection, module coefficient visualizations in the appendix, etc.). I think that the formulation with the two stages of training and the projection phase is fairly elegant. I do have questions about the method and think a few details are inconsistent, but overall enjoyed the paper and feel it’s of sufficient quality to publish at NeurIPS.

Here are a few comments that I would appreciate some feedback on in the author response:

(1) One of the main contributions the paper claims to have is to not require the task ID at testing time. But task-agnostic continual learning is not a new topic, and the authors themselves cited works in this area (e.g. [28] and [30]). So is the main contribution of the paper just the combination of the task-agnosticity with the particular formulation of the module-level training? I’d like to understand in a more concise manner the core impact that the work has from a fundamental science perspective.

(2) Why do the authors not compare to other established task-agnostic baselines? The comparisons are mostly to heavily modified versions of task-id-required methods, and these modifications are not too convincing (for example, using MNTDP with entropy is a bit dicey as entropy is a fairly unrobust statistic). 

(3) To what extent is task-agnosticity a common scenario? Usually the model user is pretty aware of which task they would like to perform, especially if the tasks are different (for example, if you need to do segmentation and detection within a single model). One scenario that might make it difficult to have task ID at inference is if you have a distribution of input data to perform the same task (e.g. digit classification with MNIST + SVHN), but this feels more like a domain adaptation setting to me. Can the authors expand on why task agnosticity is widely practical, and have the authors considered comparing to some domain adaptation baselines?

(4) In the section on OOD detection with colored MNIST, why was the task-agnostic version of LMC not tested? It seems to detract from one of the central points of the paper to only analyze this setting with task-id-required LMC.

(5) Some discussion on how the z’ threshold is set and how robust results are to this hyperparameter would be welcome.

**Post Rebuttal**: Thank you to the authors for the detailed response. My questions/concerns were largely addressed. I think it is an interesting paper with good evidence backing up claims. I will keep my rating a WA (6). 

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper originally received three borderline positive reviews but without high confidence, and so a fourth review was solicited after the regular review period from a reviewer who is extremely knowledgeable on the topic.  From this fourth review, it is clear that this paper should be accepted.

This paper focuses on continual learning of modular representations in task-agnostic settings, develops an approach that addresses this challenging, important, and understudied problem, and performs an extensive empirical analysis on a variety of well-chosen baselines. 

The authors should be aware that there are some unclear parts, identified in the reviews, that even the most expert of the reviewers found difficult to follow at times.  These need to be clarified for the final version. 
Decision: Accept (Poster)