Continual Learning via Local Module Composition
Authors: Oleksiy Ostapenko
Pau Rodriguez
Massimo Caccia
Laurent Charlin
Venue: NeurIPS 2021 Poster (2021)
Keywords: continual learning
modularity
compositionality
neverending learning
lifelong learning
plasticity
plasticity-stability dilemma
task incremental learning
multitask learning
routing
dynamic architectures
model growing
out-of-distribution generalization
OOD
parameter sharing

Abstract: Modularity is a compelling solution to continual learning (CL), the problem of modeling sequences of related tasks. Learning and then composing modules to solve different tasks provides an abstraction to address the principal challenges of CL including catastrophic forgetting, backward and forward transfer across tasks, and sub-linear model growth. We introduce local module composition (LMC), an approach to modular CL where each module is provided a local structural component that estimates a module’s relevance to the input. Dynamic module composition is performed layer-wise based on local relevance scores. We demonstrate that agnosticity to task identities (IDs) arises from (local) structural learning that is module-specific as opposed to the task- and/or model-specific as in previous works, making LMC applicable to more CL settings compared to previous works. In addition, LMC also tracks statistics about the input distribution and adds new modules when outlier samples are detected. In the first set of experiments, LMC performs favorably compared to existing methods on the recent Continual Transfer-learning Benchmark without requiring task identities. In another study, we show that the locality of structural learning allows LMC to interpolate to related but unseen tasks (OOD), as well as to compose modular networks trained independently on different task sequences into a third modular network without any fine-tuning. Finally, in search for limitations of LMC we study it on more challenging sequences of 30 and 100 tasks, demonstrating that local module selection becomes much more challenging in presence of a large number of candidate modules. In this setting best performing LMC spawns much fewer modules compared to an oracle based baseline, however, it reaches a lower overall accuracy. The codebase is available under https://github.com/oleksost/LMC. 

Continual Learning via Local Module Composition
Oleksiy Ostapenko 12
Massimo Caccia 123 Laurent Charlin 145 Mila Quebec Al Institute, Uniierrsité de Montréal, Serviiceow, *HEC Montréal,
Pau Rodríguez'
$Canada CIFAR AI Chair
Abstract
Modularity is compelling solution to continual learning (CL), the problem of modeling sequences of related tasks. Learning and then composing modules solve different tasks provides an abstraction to address the principal challenges of CL including catastrophic forgetting, backward and forward transfer across tasks, and sub-linear model growth. We introduce local module composition (LMC), an approach to modular CL where each module is provided a local structural component that estimates module's 's relevance to the input. Dynamic module com- position is performed layer-wise based on local relevance scores. We demonstrate that agnosticity to task identities (IDs) arises from (local) structural learning that is module-specific as opposed to the task- and/or model-specific as in previous works, making LMC applicable to more CL settings compared to previous works. In addi- tion, LMC also tracks statistics about the input distribution and adds new modules when outlier samples are detected. In the first set of experiments, LMC performs favorably compared to existing methods on the recent Continual Transfer-leamning Benchmark without requiring task identities. In another study, show that the locality of structural learning allows LMC to interpolate to related but unseen tasks different
(OOD), well compose modular networks trained independently task sequences into a third modular network without any fine-tuning. Finally, search for limitations of LMC study it more challenging sequences of 30 and 100 tasks, demonstrating that local module selection becomes much more challenging presence of large number of candidate modules. In this setting best performing LMC spawns much fewer modules compared to an oracle based baseline, however reaches lower overall accuracy. The codebase is available under ht https: / com/ www.oooeessssseeeeeeeeeoooooosse
Introduction
The goal of continual learning (CL) to learn efficiently from non-stationary stream of tasks without (catastrophically) forgetting previous tasks [62]. CL is often modeled as tradc-off between knowledge retention (stability) and knowledge expansion (plasticity) [26, 64]. Parameter sharing can provide control over this trade-off. For example, learning a single model shared across tasks results in better knowledge transfer and faster learning at the expense of forgetting [46, 57]. Conversely, learning separate model per task eliminates forgetting but minimizes transfer and data efficiency [2, 41).
Modular learning aims at balaneing transfer and forgetting by learning a of specialized modules that can be recomposed to solve (new) tasks while only updating a subset of relevant modules or adding new modules [6, 47, 27]. In principle, a modular learner capable of composing modules in meaningful structures can provide additional benefits including (i) computational gains due to only executing modules that are relevant a task [47, 4]: (ii) memory gains due instantiating sub-linear number of modules w.r.t. the number of tasks; (iii) systematic [8] and out-of distribbutt (OOD) generalization [18] through knowledge recombination; and (iv) biological plausibility [91, 90, 96].
corresponding author: tocksiy.ostapenkooeeeo do-ooiiicc
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Designing modular methods for CL comes with two main challenges. The first is how and when to add new modules to ensure sufficient plasticity to learn new tasks. Existing modular methods use greedy search variants, expanding the model when it improves validation performance [92, 63]. The second challenge how to compose that is, retrieve task-specific structural knowledge given a new task (previously seen or not).
Existing methods rely on a task's identifier (ID) to retrieve task-specific structural knowledge, which comes either in the form of an optimal module layout [92] or as a model- and task-specific controller network that generates modular layouts [63]. Unfortunately, in many realistic CL scenarios task identities are unavailable at test time [23, 35, 14]. Lifting this limitation is challenging since standard mechanisms for task inference, for example, leveraging a fask-inference model, could be subject to forgetting themselves.
To address both challenges, equip each module with local structural component that predicts score indicating how relevant the module is for a given input. In-distribution inputs result in high scores, while out-o--distribution inputs result in low scores. In other words, modules self-determine their relevance given an input.
This local component is used for composing modules: for each datum, modules are combined at each layer according to their normalized scores without requiring a task's ' ID ($3). The local component is module is instantiated if all the current modules flag their input
also used for module expansion: as being locally out-of-distribution ($3.1). 1). Further, new shallow modules (i.e. closer to the input) are first trained in a projection phase maximize the relatedness scores of subsequent, deeper, modules ($3.2). This process projects the output of new modules into the representation space expected by the subsequent modules and ensures the compatibility between low- and high-level modules.
In set of studies, we explore the performance and versatility of our local structural approach, which we call Local Module Composer (LMC). First, we show that LMC reaches superior or comparable performance to existing modular and non-modula, methods without requiring task IDs at test time using the Continual Transfer Learning (CTrL) benchmark, designed to evaluate transfer and forgetting in CL [92] ($4.1). Then, we demonstrate how LMC, relying on its projection phase, can solve out-of- distribution (OOD) tasks not seen during the continual training ($4.2). We also show it is possible to combine modules from independently trained models into a new model to solve tasks seen by each of the independent models without any finetuning ($4.3). Finally, analysis of longer task sequences (30 and 100 tasks) reveals that LMC tends to spawn much fewer modules to reach good performance than the fully task-aware MNTDP [92] counterpart. However, LMC reaches slightly lower accuracy on longer sequences than MNTDP, which highlights the difficulty of automatic task-ID agnostic module selection in the presence of large number of candidate modules. In Appendix F we demonstrate the applicability of LMC in the meta-continual learning (meta-CL) setting, a task-agnostic setting by nature.
We highlight that by relying on a local (per-module) structural component, LMC offers a modular CL approach that i) does not require task IDs during test in the standard task incremental settings; ii) balances parameter sharing to yield strong CL performances compared to baselines that require access the task ID; iii) in our experiments instantiates sub-linear number of modules; iv) permits recombination of modules at test time enabling OOD generalization as well as (v) the ability to combine independently trained models in third model without fine-tuning. Notably, the OOD generalization is only possible if the agent is task-agnostic in the module selection process, since OOD tasks were not observed at training, the learner has to interpolate between the learned tasks, and a (categorial) task ID is of no use.
2 Background: Modular Continual Learning
Let F(r;; 0): X y be a learner parametrized with a set of parameters 0. In task-incremental CL, the learner is exposed to a sequence of tasks. Each task is composed of a training set D, of (4, v) pairs and task identifier (ID) [92, 46). The goal is to learn an optimal 0* that minimizes the loss L for all observed tasks:
8* = arg min
E ((F(xx))))))))))))))))))))))))))
(1)

---

z0())
Weighted Sum
Em
wm
sm
(m((xll11)
w(()
softmar(>
2()
!))
fo
f1
81
fm
8m
m
m
x:(1 (1-11)
Figure 1: Modular Layer Scheme. Each black rectangle a module. Inside each module, the functional component fm receives the input (!(1)) and feeds its output to the structural component s. The output of the structural component is used to calculate the importance score Ymu using Eq. 5,
(t)
which are normalized to obtain the attention vector w(!) The layer output is the weighted sum of the functional outputs of each module. l and are the running mean and variance of the scores 7m used to detect outlier inputs and to trigger module addition.
(l)
The parameter sharing trade-off between tasks can be addressed through different architectural design choices for J. For example, can be monolithic network that shares parameters across all tasks. Most existing task incremental CL methods use task-speciff output head, requiring the task ID to select the output head corresponding the task hand [46, 87, 1].
At the other end of the spectrum are the expert based solutions that learn an independent model, a.k.a. expert, for each task [2. 83]. In this case, each expert trains task specifcii parameters = [T(
To balance parameter sharing and transfer, modular methods organize their parameters in series of modules M {mm]) with parameters 0 {o()), ), where d denotes the parameters of module k at layer in F. In general, a module can be any parametric function. In our experiments, unless otherwise stated, a module consists of single convolutional layer followed by batch-norm, ReLU activation, and a max- mnax-pooling operation.
sample, batch of samples, task. Let denote a specific composition of modules that gives rise to a distinct prediction function; we make this dependence explicit: F(x; 0, vb). Importantly, sharing modules across tasks should lead to desirable transfer properties.
Modules be composed conditioned
Veniat et [92] frames modular CL as finding an optimal layout v:(() for each task, where each layout selects single module per layer per task (hard selection):
0-, V arg min 8
Σ Be..))-, (()),
In this case the set of layouts V = (1 grows with the number of tasks, while modules can be reused across different task-specific layouts resulting in sub-linear growth pattern. They design method called MNTDP search the exponentially large space of modular layouts by only considering layouts resulting from adding a new module per layer to the best prior path (past task's 's path with the highest nearest neighbor accuracy IDs to retrieve (1))) at test time.
new task) starting at the top layer. This solution relies task
Another way of composing modules uses dynamic routing [63, 82, 47, 65]. The module layout is generated by a structural function & = s(x), hence different inputs take different routes through F. It standard to approximate the structural function using a neural network = s(x; Φ) with structural parameters Φb This framework has been applied to CL in [63] by learning separate structural function per task (5()) s(2; 619). The task IDs are used to retrieve the correct structural function:
0", -.
arg min
Σ (e ((;; )())), v)),
6
where & (o(1))
3 is the set of structural parameters for all tasks.

---

[=0
[=3
(A) Accumulation
(B) Projection
(C) Accumulation
(D) Projection
(E) Projection
(F) Accumulation
Figure 2: Two-phase training. Each module contains a functional (rectangle) and structural (trape- zoid) component. Their color intensity denotes the strength of their activation. Components with dashed contours are trained, solid contours represent fixed components, arrows show the gradient flow: black arrow finctional signal, pink structural signal. (A) All modules are trained on task (B) Task arrives, a new module is added at layer 1, which is first trained to project its output into the representation of the modules above via the structural signal (the functional signal is optional). No module addition allowed during the projection phase. (C) Module addition is allowed again, both signals used for training.
(D) As task arrives, new module is added at layer 1 again, projection phase is triggered. (E) A new module is added at the layer 2, both new modules are now trained in the second projection phase. (F) Both modules are trained using both signals.
The above methods require task IDs at both training and testing time. Next we introduce our modular CL approach that only relies task IDs during training.
3 Local Module Composer (LMC)
We propose LMC, a modular approach where each module consists of functional component (a; e and a structural component s(x; ( see Figure 1. The functional components are responsible for learning to solve the prediction task and are trained via the usual task loss C (c.g. cross-entropy loss for classification). The structural components receive the corresponding functional output as their input (see Figure 1) and are responsible for dynamic routing through F. Structural parameters & are trained using structural loss (stt.) computed locally at each module.
Intuitively, the structural component of module should serve as a density estimator of the outputs of the functional component. The module's contribution to the layer's output is proportional to the likelihood of the input sample under the estimated density. our instantiation, the structural component produces relatedness score: a lower score for inputs that are more likely to belong to the distribution on which a given module was trained, and a higher score for inputs that are out-of-distributio for the given module. Hence, the likelihood of the input sample is approximated by the negative relatedness score.
Given input data sample 2²()) xr, the output (1) of a layer is defined the weighted sum of the functional outputs of all [M local modules and used as input to the subsequent layer 7 + 1:
(1
Σ m==1
(.) (1x(1-1) (00)
(4)
The functional output of the network is equal to the output of the final layer: F(x; =) (L), In the last layer F implements a single output-head per task. At training time, the task ID is available and we update only the output-head corresponding the currently leamned task. At time, the task ID not available and we select the output head with the highest activation weight w((L) i.e., the last layer performs hard module selection.
The module activation weights wi( are computed by normalizing the vector of local relatedness scores 7()) E R 1 Each element of (()) is obtained from the negative structural loss which approximates the likelihood of each module:
(!)
L(st.) (s|f(x(1-1;; 1);000)0)0)))))))00)))))) d!)
(5)
w m
(()
softmax(7
(6)

---

Modules with lower structural loss get higher activation weights. Note that in practice, it can be useful bias the module selection towards the expected module selection batch. assuming that samples within batch are likely to belong to the same task. We discuss this point further in A.1.
Instead of using the softmax function, it is possible perform hard selection taking the module with the highest score [82], or alternatively selecting top-k modules [88]. In both cases, LMC's structural parameters stay differentiable due the local nature of structural learning. Note that in the case of global structural objective, hard module selection would require applying tools for non-differentiable learning such as Expectation Maximization [47] or canforcement-learniin based methods [82, 5].
The overall LMC objective consists of optimizing both functional and structural losses:
M
0',0" arg min
Bla.yDD. (F(F(x;0,)) +
cla (s[((x (; 1MD
0.0
As in [92], learning is performed w.r.t. only newly introduced modules to prevent forgetting.
Structural component. We test two instantiations of the structural component and loss Clate) In the first one, 8 is an invertible neural network [80]. Here we use the invertible architecture proposed by Dinh et al. [21]. As shown by Hocquet et al. [37], for this invertible architecture the structural objective can be defined as Clat)) (x) = ||x||2. Intuitively, an invertible architecture prevents C(st.)
from collapsing to all-zeros solution.
In the second instantiation, and f form an autoencoder and clat)) (xx ||x((111) x:| is the reconstruction error with respect to the module's input 2((-1), Aljundi al. [2] used similar idea was for selecting the most relevant expert network conditioned on a task. Unless stated otherwise, modules in the feature extractor use the autoencoder their structural component, while output heads use invertible these combinations worked well in practice.
3.1 Expansion strategy
It is necessary to expand F as new tasks arrive to acquire new knowledge. A new module is added to a layer when all modules in this layer detect an outlier input. To this end, we track the running statisties of the relatedness score ry for each module mean ju and variance a (see Figure 1), and calculate z-score for each sample in the batch and each module at layer:
W
Pm
Zym
(8)
m
An input considered an outlier if its z-score is larger than predefined threshold e' (see Ap- pendix B.6 for an ablation study of s' values). The expansion decision can be made on the per-sample (i.e., if an outlier sample is detected) or a per-batch basis (i.e., 2 is averaged over the mini-batch). Unless stated otherwise, in our experiments, the decision was made on per- basis. Additionally, at training the parameters of existing modules are fixed once the task changes. during forward pass through F module addition is triggered at multiple layers, we start adding modules at the layer closest to the input.
3.2 Training
Each module in LMC receives two types of learning signal: a structural signal resulting from minimizing glat.), and functional signal resulting from minimizing the global functional loss L. All structural components & are trained only with the structural signal that is calculated locally to each module.
The training of functional components proceeds in two phases: projection and accumulation. Whenever the expansion strategy triggers the addition of new module (i.e., 2m > z' Vm (0. ... MM()))), starting with layers closest to the input, LMC initiates the projection phase. During this phase, the new module is trained to minimize the structural loss from all the layers above and new-module addition is allowed. This procedure makes the representation of new modules
5

---

5
S+
S
SPI
MODEL
A
5
M
A
M
J
F
HAT[S7]
63.7.0.7
-1.3
24
61.
41.5
-0.220.2
24"
30.1
0.0g0.1
24"
61.9
-3.2a1.3
24
61. 210.7
-0.1
20"
BWC]46]
62.7.0.3
-3 -3.620.9
24
53.4
-2.320.4
24"
56.3
-9.133.3
24"
62.3
-3.620.9
24"
54.243.1
-4.42..
20"
O-EWC[855
62.0+0.7
-3.2 3.2
24"
54.6.0.7
-1.3+1.0
24
34.2+1.1
-10.8-3.1
24"
62.4+0.6
-3.0+0.9
24"
4
-5.7
20"
ER[81. 16]
60.6.0.7
-2. 110.9
4"
63.0e0.6
3.8
4"
63.8
-1.9
4"
60.7a1.0
-1.5
4"
60.5a1.0
0.5.0.9
4"
EXPERTS
62.7
0.0
63,2
0.0
24
63.1
24
13.1
0.0
24
63,9920.5
0.0
MNTDP
66.3
0.0
13.7
62.6
0.0
21.0
67.920.9
16.0
0.0
64.064.,a00
0.0
17.2
SG-F[63]
63.6&1.5
0.0
14.7
61.520.6
0.0
20.8
65.5
0.0
17.5
14.1
0.0
16.2
02.0
0.0
16.0
LMC
66.6&1.5
-0.0
15.3
60.1
-1,4
21.3
69.5+1.0
0.0.0.1
20.0
66.7a2.2
10.1
15.5
-3.523.1
18.2
MNTDP(A)
41.9+2.5
-2.8+0.6
14.8 43.2+1.3
10.80....00
20.7
322.7+1
6
17.2
37. 27.
-5.++3.5
13.3
35. 1+3.6
-66.444.6
15.8
LMC(AI
67.2
-0.5
15.7
62.2s4.5
2.3
22.3
68.5
-0.120.1
19.7
55.1a3.4
-7.184.0
15.5
63.5a1.9
-1.0a1.5
19.0
LMC(A.H)
64.921.5
-0.240.2
16,2
55.8
-0.3e1.2
15.3
67.6+2.7
-0.8s1.0
21.5
54.2a3.6
-2.922.0
15.9
53.8
3.125.5
10.8
SG-F(A)
29.5
-35.3
14.3
20.424.4
-39.346.7
16.0
24.415.6
-38.7.4.0
18.7
30.524.5
-33.0...-34.0,5.5
12.2
19.421.0
-41.811.6
15.5
ERIA.SHBL. 16]
60.4
-0.510.7
65.320.9 4"
6.0s1.0
4"
58.8.3.2
-4.2a3.7
47.621.5 4"
-7.621.6
4"
$8.641.3
-1.241.5
4"
FINETUNE
47.5=1.5 14.9-1.4 4"
31..
29..3+3.8
4"
39.7+5.0
-.3.9-5.
4"
45.4e4.0
15.5+3..7 4'
1
29.2e3.2
4"
P.NETUNE
52. -15.7-1.1
24"
38.2+3.2
25..843.
24"
49.3+2.0
-18.4=2.0
24"
49.3+2.1
18.442.0
24"
37.
1+2.1 -26.0+2.2 20
Table 1: CTrL results: report accuracy (f A), forgetting (1 JF) with standard deviations calculated over 6 different runs. We report the mean number of modules (M) over these runs, where marks methods with fixed capacity. The first block comprises set of standard CL baselines including regularization and replay based methods. The second block are the modular methods, third modular and replay based methods that task ID agnostic (A), and the last block the two finetuning baselines. (H) indicates hard module selection. (S) indicates single-head as detailed in the main text.
compatible with subsequent modules and enables their composition. This procedure "encourages" already-learned modules to be reused, preventing over-spawning new modules. The functional signal is optional during projection (we kept it in all experiments unless otherwise stated).
In the accumulation phase. new module addition is allowed again and all non-frozen modules are trained with both signals. The functional components of new modules still receive signal from the structural components of modules above. The two-phase training is explained schematically in Figure and implementation details are provided in Appendix A.
Experiments
We now evaluate the performance, empirical capabilities, and properties of LMC in four different CL settings. First, in $ 4.1 we study a standard lask-incremental CL setting (task-ID agnostic and aware) using the Continual Transfer Learning Benchmark (CTrL) [92]. Next, we explore the properties of LMC through other CL settings. In Š 4.2 we evaluate the continual OOD generalization ability of the proposed LMC. In $ 4.3 we show the ability of LMC combine modules form independently trained models. In Appendix F, we evaluate LMC in the Continual Meta- Meta-Learning setting.
4.1 Continual transfer learning using the CTrL benchmark
The CTrL benchmark was proposed to systematically evaluate properties of CL methods with a focus on modular architectures [92]. It consists of 5 streams of visual image classification tasks. The first stream S (ti t2, t3, ta, ts. f1) consists of sequence of 6 tasks, where the first and last task are the same except the first has an order of magnitude more training samples ("+") than other tasks. This stream is designed to evaluate the direct transfer ability of models, i.e. modular learner should be able to reuse the first task's 's modules for the last task. The S+ (11, f2, la. l4, Is. ft) stream is similar to S but now the last task comes with more data than the other tasks (including the first one). Here, the modular learner should be able to update its knowledge, i.c. performance on the first task should improve after learning the last task. In the Siin = (f1, 12, 13, 14, ts (t) stream the first i and the last & tasks are similar, with slight input distribution change (e.g. different background color). In the Sout (L+, /2, la, la, Is, (") stream the first task 4 and the last task '" differ in the amount of training data and the output distribution, i.e. the labels of the last task are randomly permuted. The plasticity stream Sol (t1, t2, t3, t4, ts) evaluates the ability to leam stream of unrelated and

---

potentially interfering tasks, i.e., transfer from unrelated tasks can harm performance. Descriptive statistics for all datasets are in Appendix B.I.
We compare to several baselines. Finetune: trains a single model (wider model marked with L) for all tasks. Experts: trains model per task. We also compare with the several recently proposed modular CL baselines, which achieve competitive results in CTrL and require task IDs at test time. MNTDP [92]: a recent search-based module selection approach described in more detail in $ 2. MNTDP requires the task ID to retrieve the previously found best structure for each test task. MN a task ID agnostic version of MNTDP we created, which selects the path with the lowest entropy in the output distribution. SG-F [63]] Soft-gating with fixed modules, a modular method mentioned in 2. It relies on a task-specific structural network that generates soft-gating vectors for each layer of the modular learner and fixes learned modules when new tasks arrive. We slightly adapted the original expansion strategy of SG-F order to conform to our experimental setup; details are in Appendix B.2. SG-F(ay version of SG-F with single structural network shared across tasks. HAT[87]: learns attention masks for activations that gate the gradients to prevent forgetting. The task ID is used to select task-specific attention mask for inference.
We also compare to several standard CL methods. EWC [46]: trains a single model for all tasks while applying parameter-regularizate
to minimize forgetting. O-EWC:(85] online version of EWC that does not require storing a separate approximation of the Fisher information matrix per task. ER [16]: trains a single model while replaying samples from previously seen tasks. The size of the replay buffer corresponds to the memory size of the LMC assuming the worse case linear growth pattern (i.e., LMC with 24 modules on a 6-task sequence). ERAs: task II) agnostic version of ER that uses a single output head to classify all classes from all tasks: i.e. after learning stream S the output head has 50 output neurons and the output classes of the last task are considered the same as the ones of the first task.
We use several versions of LMC. LMC a version of LMC that uses the task ID for output head selection (not module selection as MNTDP). LMCoN: the default version of LMC. It equips output heads with structural components and is therefore task ID agnostic at test time. LMCAA. version of task ID agnostic LMC that performs hard module selection, i.c., taking the module with the highest relevance score per layer. All methods use the same architecture (described in Appendix A.4) together with the Adam [44] optimizer. HAT the only method that uses SGD.
Similar to Veniat et al. [92], we use the following evaluation metrics: (AA) average accuracy on all seen tasks at the end of CL training, Forgetting (F) difference between accuracy the end of the training and accuracy after learning the task averaged across tasks [60]; Number of modules (M) at the end of the continual training procedure. Formal definitions of all metrics are in Appendix B.3.
Table reports performance using the CTrL benchmark. Overall, modular methods tend to outperform ER and the regularization based methods (HAT and EWC). Among the modular methods, soft-gating SG-FAu, with a single controller shared among all the tasks performed the worst. This baseline showcases the problem of forgetting in the global structural component ka.k.a. controller) of dynamic routing methods such as the one proposed by Mendez and Eaton [63]. A version of LMC performs the best on the S 5 and Sout streams.
Notably, LMCA,, which does not rely on task IDs at test time, outperformed all other task ID agnostic methods such as MNTDP and ERAs.) on all streams but S+, and always performed on par with task ID aware methods. On the Sout stream low performance is expected for task-ID agnostic methods due to output distribution shift: i..., at test time we notice that LMC correctly assigns samples from the last task t" to the first task's s tf output head. However, the resulting classification accuracy is low because the labels of the last task are randomly permuted in this stream.
The task ID agnostic LMCas) outperforms task ID aware LMC on the S and 5+ streams. Here, LMCAs selects modules (and the output head) which were predominantly trained on the task that provided more training data (e.g. tj in S stream), hence transferring knowledge between the first and the last tasks. In contrast, LMC Ai when tested on the last task Ł is forced to select the output head belonging to this task, which was trained on less data than the output head of Lf task, leading to lower accuracy. In addition, we observed that versions of LMC often exhibit high variance (e.g. see S+, Sout and Sol streams). This may be caused by the larger amount of trainable parameters compared to other models and relatively small amount of training data. Finally, low performance of
7

---

103
70.0
94.1
93.7
91.3
02.9
07.5
96.3
07-3
100.0
91.7
99.1
82.9
62.6
65.0
76
-28.4
+6.0
27.1
-16.5
#5.6
80.1
$8.0
223.3
35.4
=12.2
80.0
#0.5
50.
80.1
80.5
20.
$16.0
18.3
121
$11.8
01i0o Maxx
75.7
95.5
94.7
:20,5
10.1
500,5
=13.3
69.3
65.3
80.5
+20.4
23.0
115.0
60.2
80.3
19.1
$0.0
60.4
#14.1
=21
112
=21 1
11.44
70.4
97.7
63.1
+30.0
88.1
110.0
128.8
10.3
$3.0
115.9
10.3
10.1
+00.0
+15
+15
+'2.7
Modd-Ret
65.0
06.1
86.8
89.8
92.6
75.0
75.0
64.3
86
77.8
89.5
97.8
87.3
89.3
87.6
00.1
65.2
61
986
76.6
63.B1
65.1
50.0
89.5
9P.
89.4
78.9
52.5
56.8
10.3
98-4
98.0
58.
59.1
10.3
81.8
70-1
85.3
50.9
59.2
=14.0
=15.7
=22.8
#22
+9.9
=0.4
60.6
=17.0
+17.6
#20.0
(0.1)
(2.3)
(4.5)
(6.7)
(8.8)
(9.1)
12.30
(4,5)
(6.7)
(8.9)
(0.1)
(2.3)
(4,5) (6.7) (8.8)
(0.1)
12.30
(4,5]
(6.7]
(0,8)
(a) EWC
(b) MNTDP
(c) LMC -A)
(d) LMC -Ai (no proj.)
Figure 3: Continual DOD--enerallizttion matrices show test accuracy on seen and unseen tasks (on- and off-diagonal tasks respectively). In each sub-figure x-axis shows the MNIST class-combination used to build the task, y-axis gives the foregrounddbcckgruundd colors. Only diagonal tasks are learned continually. While non-modular EWC (a) and modular MNTDP (b) can prevent forgetting, LMC, A (c) is able to generalize to OOD tasks as well. LMC(-N) without projection phase performs poorly as shown in (d).
LMCaaa emphasizes the importance of soft modular attention for LMC. Additional results, including a transfer metric [92], are in Appendix B.4.
4.2 Compositional OOD generalization
This second study tests the ability of LMC to recombine modules for OOD generalization. We use colored-MNIST dataset- - a variation the standard MNIST dataset of hand- digits from to 9 [43] in which digits are colorized. We design a simple sequence of tasks as follows. First, we define two high-level features: the breground-backgruon
color combination (using the colors red, black, green, blue) and the class (0-9). Then, we create five con-overlapping tasks of two (digit) classes each: {0-1,
8 8- At training time the model is continually trained using a sequence of these tasks, however, each task is only seen in one of five different foreground backoruud-b combina- tions tred-black, green-blace test time we measure the generalization ability to seen and unseen combinations of classes and colors.
bluebblcc, b Black-red, bl black-gee black-green) At
In Figure we present the accuracy matrices for different learners when tested on all 25 combinations of colors and classes after it has been trained only on the 5 tasks on the diagonal.
We compare the performance of LMC(-A) with EWC [46]. MNTDP [92], and an ablated version of LMC without the projection phase. We observe that the OOD accuracy attained by LMC is significantly higher than EWC and MNTDP. Since the model trained with EWC is monolithic, the digit-background color combinations are entangled with the digits' shape for each task, hindering OOD generalization. In contrast, modular approaches such as MNTDP and LMC learn a different module combination for each task. In contrast to MNTDP, LMC's s module selection does not rely on task identifiers and each module is selected
local manner based on its compatibility with the current input. This allows LMC to interpolate between previously seen tasks being able to dynamically compose existing modules to adapt to tasks that have not been seen at training. Because MNTDP's module selection relies on a database of task-specific structures found to be optimal for the corresponding task at training, this method must reuse the predefined module compositions based on task IDs. This forces MNTDP to use modules that were trained using different color combination, and results in e.g. a 24% accuracy drop with respect to LMC on [0, 1] when the foreground and background colors are inverted w.r.t. the seen combination.
In Figure 3d we report results for LMC without applying the projection phase. The projection phase adapts the representation of newly introduced modules to match the distribution expected by the subsequent modules. As expected, we found that skipping severely degrades performance. This result validates the usefulness of the projection phase to achieve an efficient local module selection. In Appendix we plot the average module selection for all 25 test tasks, showing how modules are reused for the OOD tasks.
8

---

AvV. test acc. (4)%
68
Avv. test aoc. (A)%
70
Method LMCI A)
Method
66-
LMC(-N)
MNTDPP
00
62
0
20
30
50
60
70
20
40
60
80
100
120
140
160
180
200
Number of modules (M)
Number of modules (M)
(a) tasks sequence glongā0
(b) 100 tasks sequence glong
Figure 5: Results on glong and glong30 sequences for different hyperparameter values (we select only runs with reasonably good performance, i.e. A>60%) same plots plots for all conducted runs can be found in Appendix C).
4.3 Combining modular learners
In earlier sections we show cross-task reusabil- ity of modules, here we test the model reusability. We motivate the practical impor- tance of this kind of reusability with federated learning example: a privacy preserving training might be required for LMCI and LMC2, trained on the premises of customers 1 and 2, after which their modules can be combined in single central entity LMC3, located on premises of the cloud service provider. LMC3 required to perform tasks seen by both independent LMCs but can not be finetuned as it has no access to the original training data distributions.
100
LMC1
LMC3
Test Acouracy
80
MNST
IMNIT+
dMNIST-r
MNNST-
<MNIST-g
SVHN
Figure 4: Performance of combining indepen-
dently trained LMC1 and LMC2 into LMC3.
the ability of LMC to pre- serve and transfer knowledge in such setting. To this end, we design the following tasks: 1 and IMNIST- Both are sampled from the fashion-MNIST dataset [98] but the latter comes with an order of magnitude less training data. cMNIST-r is variant of the colored-MNIST dataset where the background of 95% of the training samples is colored in red and 5% in green. For the cMNIST-g dataset these proportions are inverted and 95% of the training samples is colored in green. The test set contains 50% of samples with green and 50% with red background. We trained LMC1 continually on MNIST, fMNIST+, and cMNIST-r tasks. We trained LMC2 on IMNIST- cMNIST-g, and SVHN. We then combined the modules of both LMCs layer-wise to obtain LMC3.
In Figure
We observed positive transfer for both cMNIST and IMNIST- tasks with LMC3, We found that LMC3 selects different modules originating from different LMCs conditioned test samples with different background colors - LMCI's 's modules were specialized on red background while LMC2's i on green (selected paths presented in Appendix D). Notably, cross-model reusability without fine-tuning is novel to LMC and can be attributed to the local nature of the structural component. Using global structural component
in [63] would require tuning separate structural component specifically for LMC3. In case of task-specific routing of proposed for MNTDP [92]. additional search would be needed discover task-specific paths through the consolidated LMC3 model. In both cases the access to the orinal training data distributions would be required and privacy would not be preserved.
4.4 Longer task sequences
Here we study the performance LMC on longer task sequences consisting of 30 -loog30 and 100- glong tasks. The Slong sequence corresponds to the one proposed by Veniat al. [92] as part of the CTrL benchmark. gtong30 is a 30-tasks subset of glong (see Appendix B. for details).
We first report the average test accuracy (AA) and the total number of modules (M) of the models selected through vooss-validttion. glong30, MNTDP A=64.58, M=64; LMC-AA A=62.44, M=50. glong; 6-68994 M=142; LMC A=63.88, M=32. While the gap between the accuracy

---

of LMC and MNTDP on Stong30 is only 1.86% in the case of Slong this gap grows to 6.58%- points. is important to highlight that in contrast to LMC, MNTDP's module selection is performed by task ID aware oracle. We further analyze the trade-off between the number of modules and accuracy in Figure 5, where we plot the number of modules (M) against average test accuracy (AA) for models that resulted from training with different hyperparameters. For both streams, we observe that LMC tends to spawn much fewer modules than MNTDP. However, MNTDP shines in the presence of large number of modules and achieves higher overall test accuracy on these streams. Interestingly, as can be clearly observed on the Stong stream, LMC reaches higher accuracy with smaller number of modules: e.g. ~64% with 32 modules, while adding modules leads to lower accuracy: e.g. ~61 % with 98 modules. This result suggests that local task ID agnostic module selection becomes more challenging for LMC in presence of a large number of modules.
5 Related work
Modularity in neural networks is studied in the context of scalability [9], and more recently as way to achieve compositionality and systematic generalization [6, 47, 15, 8, 27, 19] well as for multi-task learning [65, 82]. From the causal point of view, a data generation process could be thought composition of independent causal modules [75]. Researchers model these kinds of systems using a set of independent modules, where each module is invariant to changes in the other modules induced by e.g. distribution shifts [84, 76). This idea is crystallized by Parascandolo et al. [74], who propose a way to learn a set of causal independent mechanisms as mixture-of-experts Building up on this ideas, others show evidence of compositional OOD generalization [61]. Recently, [66] argue for more wholistic view on CL including OOD generalization as an important desiderata.
Continual learning methods typically address the problem of forgetting through parameter regu- larization [46, 70, 99], replay [89, 78, 3. 73, 54, 12, 97, 39, 81, 13] or dynamic architectures (and MoEs) [83, 87, 52, 85, 53, 41]. Our work falls under the umbrella of the latter and shares its advan- tage of having the capacity to adapt to a large number of related tasks. Our focus is on improving modular CL approaches, which despite their advantages, have only recently been studied in the CL literature [63, 92]. The main difference with our work is that we use a local composition mechanism instead of global one. We detail this difference in $2 and also compare these methods in $4.1.
Continual learning focuses on fast learning and remembering [25, 35, 33, 41], often empha- sising the online performance OOD tasks [14]. As argued by Jerfel et al. [41] modularity can be useful in this setting to minimize interference between tasks. They proposed a way to train a MoE model, with each expert focusing LMC aims at decomposing knowledge into layer-wise composable modules further reducing modular granularity. Continual-man learning is often confused with its counterpart meta-continual learning [40, 11, 93], in which algorithm are learning to continually learn.
cluster of tasks leveraging Bayesian nonparametrics.
The rapid growth of continual learning has lead researchers to work on empirical studies [20, 58, 56), surveys [32, 42, 55, 66, 67] as well as CL-specific software [72, 22, 59].
6 Conclusion
We develop LMC, method to learn and compose a series of modules on a continual stream of tasks fulfilling some of the basic desiderata of modular CL such as module specialization, avoidance of collapse, and sublinear growth. In LMC, structural information is learned and stored locally for each module. It is the locality of the structural component that enables generalization to related but unseen tasks, and that permits combining different LMCs without fine-tuning.
Future work could focus on achieving more efficient sub-linear model growth through OOD general- ization and reusability of modules. Additionally, while the benefits of modularity for CL are well understood, the implications of the CL regime on modularity and compositionality have not been studied extensively. It is possible that providing knowledge to the learner in incremental chunks results in the implicit supervision needed to better disentangle it into specialized and composable modules. Another promising direction is removing the need for task boundaries during training and developing more robust architectures for the local structural component (related discussions are in Appendix A.3, and limitations in Appendix G).
10

---

References
[1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) forget. CoRR, abs/171 1.09601, 2017.
[2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 3366-3377 3375, 2017.
[3] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximal interfered retrieval. In Advances Neural Information Processing Systems 32, pages 11849 11860. Curran Associates, Inc., 2019.
[4] Mohammed Amer and Tomás Maul. review of modularization techniques in artificial neural networks. Artificial Intelligence Review, 52(1):527-555 2019.
[5] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. arXiv preprint 101.006060000000. 2016.
[6] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39 48, 2016.
[7] Antreas Antoniou, Harrison Edwards, and Amos Storkey, How to train your maml. arXiv preprint vrXiv:1810.09502. 2018
[8] Demitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations, 2018.
[9] Dana H Ballard. Modular learning in neural networks. In AAAL, pages 279-284, 1987.
[10] Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and residual networks. In International Conference Machine Learning, pages 573-582, PMLR, 2019.
Jacobsen. Invertible
[11] Lucas Caccia and Joelle Pineau. Special: Self-supervised pretraining for continual learning. arXiv preprint arXiv:2 090600 2021.
[12] Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. Online learned continual compres- sion with stacked quantization module. ArXiv, abs/191 1.00019. 2019.
[13] Lucas Caccia, Rahaf Aljundi, Tinne Tuytelaars, Joelle Pincau, and Eugene Belilovsky, Reducing represen- tation drift in online continual lcarning, arXiv preprint arXiv:2/04.05023 2021.
[14] Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fahrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Trina Rish, Alexande Lacoste, David Vazquez, et al. Online fast adaptation and knowledge accumulation: a new approach to continual learning. arXiv preprint arXiv:2003. 05856, 2020.
[15] Michael Chang. Abhishek Gupta, Sergey Levine, and Thomas L Griffiths. Automatically composing representation transformations as means for generalization. In International Conference on Learning Representations. 2018
[16] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Punect K. Dokania, Philip HS Torr, and M Ranzato. Continual Icarning with tiny episodic memories. 2019.
[17] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In P Proceedings of the IEEE Conf. Computer Vision and Puttern Recognition (CVPR). 2014.
[18] Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. Modularity improves out out-of-domain instruction following. arXiv preprint arXiv:2010.12766 2020.
[19] Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference on Learning Representations, 2021. URL ht http///peeeeeeieeeeeee/ee net wtt.oooummmmduuueeoaa
[20] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia. Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Comtinual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint 1rXiv:1909.08338 2019.
[21] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non- lonelr independent components estimation. arXiv preprint 111...4400.8510, 2014.
11

---

[22] Arthur Douillard and Timothée Lesort. Continuum: Simple management of complex continual learning scenarios. ArXiv, abs/2102.06253, 2021.
[23] Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint 18XX::1805009733, 2018.
[24] Chelsea Finn, Picter Abbeel, and Sergey Levine. Modcl-agnostic meta-lcarning for fast adaptation of deep networks, International Conference on Machine Learning (ICML), 2017.
[25] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. Interna- tional Conference on Machine Learning, pages 1920-1930. PMLR, 2019.
[26] Robert M. French. Pseudo-recurrent connectionist networks: An approach to the sensitivity- stabilit'dilemmma. dilemma. Connection Science, 9(4):333380, 1997.
[27] Anirudh Goyal, Alex Lamb, Jordan Hoffmaann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint (9KIv:990..10800 2019.
[28] Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Kc, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordination among neural modules through a shared global workspace. arXiv preprint arXiv:2103.011 2021.
[29] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- -asse meta-learning as hierarchical bayes. arXiv preprint arXiv:1801.0892 2018.
[30] Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint mrXiv:2007004434 2020.
[31] Sharut Gupta, Praveer Singh, Ken Chang, Mehak Aggarwal, Nishanth Arun, Liangqiong Qu, Katharina Hoebel, Jay Patel, Mishka Gidwani, Ashwin Vaswani, et al. The unreasonable effectiveness of batch-norm statistics in addressing catastrophic forgetting across medical institutions. arXiv preprint arXiv:201 100896,, 2020.
[32] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 2020.
[33] James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone, Continuous meta-lcarning without tasks, arXiv preprint arXiv: 191.00866, 2019.
[34] Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A. Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. ArXiv, abs/1906.05201, 2019.
[35] Xu He, Jakub Sygnowski, Alexandre Galashov. Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv:1906.05220 2019.
[36] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 1rXiv:I812.00mm 2.06006, 2018
[37] Guillaume Hocquet, Olivier Bichler, and Damien Querlioz. Ova-inn: Continual learning with invertible neural networks. arXiv preprint arXiv:2006. 13772, 2020.
[38] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[39] David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In AAAI conference on artificial intelligence, 2018
[40] Khurram Javed and Martha White. Meta-lcarning representations for continual learning. In Advances in Neural Information Processing Systems 32, pages 1820- 1830. Curran Associates, Inc., 2019. URL http:/ http::://ppppss.eeee niw.....pps...mm wtt.eerssiiss.ccppppppeeeee cccpppper/ 8458- 1e-8-meta--am aeeetallaaae
for conttnnaall.... -laannnneerrnnnne... .df..
[41] Ghassen Jerfel, Erin Grant, Tom Grifliths, and Katherine A Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. In Advances in Neural Information Processing Systems, pages 9122-9133, 2019.
[42] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives, 2020.
12

---

[43] Byungju Kim. Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9012-9020, 2019.
[44] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL htt.ttttp////xii.orrr abss / 1412. 6980.
[45] Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 2013.
[46] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):352) -3526, 2017.
[47] Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neural computation. Advances in Neural Information Processing Systems, 31 31:2408-2411 2018.
[48] Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[49] Alex Krizhovsky, Geoffrey Hinton, ct al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[50] Brenden Lake, Ruslan Salakhutdinov. and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 3cience,35066666)113333 35066266) 2015.
[51] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann lecun. cwn. httpb/xdb///tte /miit/..
[52] Jeongtae Lee, Jachong Yoon, Eunho Yang, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. CoRR, abs/1708.0154l 2017.
[53] Snochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. arXiv preprint arXiv:200J.0068 2020.
[54] Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Jean-François Goudou, and David Filliat. Generative Models from the perspective of Continual Learning. In International Joint Conference on Neural Networks (IJCN),, 2019.
[55] Timothée Lesort, Massino Caccia, and Irina Rish. Understanding continual learning settings with data distribution drift analysis, 2021.
[56] Timothée Lesort, Thomas George, and Irina Rish. Continual learning in deep networks: an analysis of the last layer, 2021.
[57] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40( 12:22935522947 2017.
[58] Vincenzo Lomonaco, Lorenzo Pellegrini. Pau Rodríguez, Massimo Caccia, Qi She, Yu Chen, Quentin Jodelet, Ruiping Wang, Zheda Mai, David Vázquez, German Ignacio Parisi, Nikhil Churamani, Mare Pick- ett, Issam H. Laradji, and Davide Maltoni. Cvpr 2020 continual learning in computer vision competition: Approaches, results, current challenges and future directions. ArXiv, abs/2009.09929, 2020.
[59] Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Mare Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andrcas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: an end-to-end library for continual learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2nd Continual Learning in Computer Vision Workshop, 2021.
[60] David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pages 6467-6476, 2017.
[61] Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Fast and slow learning of recurrent independent mechanisms. In International Conference on Learning Representations. 2021. URL ht tps:: http://ppneeeee..ecceee net wtteee/oooooonoooneeeeeee
13

---

[62] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[63] Jorge A Mendez and Eric Eaton. Lifelong learning of compositional structures. arXiv preprint arXiv:2007.077 077322 2020.
[64] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
[65] Elliot Meyerson and Risto Miikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. arXiv preprint arXiv:1711.00108, 2017.
[66] Martin Mundt, Yong Won Hong, Iuliia Pliushch, and Visvanathan Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning, 2020.
[67] Martin Mundt, Steven Lang, Quentin Delfosse, and Kristian Kersting. Cleva-compass: continual learning evaluation assessment compass to promote research transparency and comparability, 2021.
[68] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't I know? arXiv preprint 1810000 2018.
[69] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[70] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations (ICLR), 2018.
[71] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint arXiv: arXiv:1803.00 1803.0299, 2018
[72] Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko, Pau Rodriguez, Matthew D Riemer, Julio Hurtado, Khimya Khetarpal, Dominic Zhao, Ryan Lindeborg. Timothée Lesort, Laurent Charlin, Irina Rish, and Massimo Caccia. Sequoia: A software framework to unify continual learning research, 2021.
[73] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11321-11329, 2019.
[74] Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, pages 4036-4044. PMLR, 2018
[75] Judea Pearl. Causality. Cambridge university press, 2009.
[76] Jonas Peters, Dominik Janzing, and Berhhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[77] Mohammad Pezeshki, Sékou- Aurou Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoic. Gradient starvation: A lcarning proclivity in neural networks. arXiv preprint arXiv:201 1.09468, 2020.
[78] Sylvestre-Alvis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Computer Vision and Pattern Recognition (CVPR), 2017.
[79] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv: 00676, 2018
[80] Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMIR, 2015.
[81] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne, Experience replay for continual learning. In Advances in Neural Information Processing Systems, 2019.
[82] Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint IrXiv:1904.12774, 2019.
14

---

[83] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukeuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 04671, 2016.
[84] B Schölkopf. D Janzing, J Peters, Sgouritsa, K Zhang, and J Mooij. On causal and anticausal learning. In 29th International Conference on Machine Learning (ICML 2012), pages 1255-1262. International Machine Learning Society, 2012.
[85] Jonathan Schwarz, Jelena Luketina, Wojeiech M Czamecki, Agnieszka Grabska- Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. arXiv preprint arXiv: 06370, 2018.
[86] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. International Conference on Machine Learning (ICML), 2018.
[87] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pages 4548 4557. PMLR, 2018.
[88] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff' Dean. Outrageously large neural networks: The sparsely- gated -eetret- layer, 2017.
[89] Andrea Soltoggio. Short-term plasticity as cause-effect hypothesis testing in distal reward learning. Biological cybernetics, 109(1):75-99) 94, 2015.
[90] Olaf Spors and Richard F Betzel. Modular brain networks. Annual review of psychology, 67:613-640 2016.
[91] Saul Sternberg. Modular processes in mind and brain. Cognitive newropssychlloy. 2563-4) 2011.
[92] Tom Veniat, Ludovic Denayer, and Mare' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. arXiv preprint arXiv:2012.12631, 2020.
[93] Johannes von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where to learn: Gradient sparsity in meta and continual learning. NewrlPS 2021, 2021.
[94] Yaqing Wang, Quamming Yao. James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey few-shot learning. ACM Computing Surveys (CSUR). 53(3)1-34, 2020.
[95] Ziyu Wang, Bin Dai, David Wipf, and Jun Zhu. Further analysis of outlier detection with deep generative models. arXiv preprint urXiv:2010.13066 2020.
[96] James CR Whittington and Rafal Bogacz. An approximation of the error backpropagation algorithm in predictive coding network with local hebbian synaptic plasticity. Neural computation, 1229)1222- 2017.
[97] Chenshen Wu, Luis Herranz, Xialei Liu, Joost de Weijer. Bogdan Raducanu, al. Memory replay gans: Learning to generate new categories without forgetting. Advances in Neural Information Processing Systems, 31:5962-5972. 2018
[98] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 17XVv:1708.07747,222
[99] Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes, 2018
Acknowledgments and Disclosure of Funding
Laurent Charlin holds CIFAR AI Chair Program and acknowledges support from Samsung Electron- ies Co., Ldt., Google, and NSERC. Massimo Caccia was supported through MITACS during his part time employment with Element Al the ServiceNow company. Massimo Caccia was also supported by Amazon, during his part time employment there. We would like to thank Mila and Compute Canada for providing computational resources. We also would like to thank Irina Rish for useful discussions.
15


Reviews:
[{"review_id": "vJFnnWTgGuA", "reviewer": "Reviewer_eYQc", "summary": "Authors propose an algorithm to perform continual learning without task-id labels at inference. The method involves local module-level OOD estimators which then output linear coefficients that informs the model how to combine outputs. The model also has the option of creating new modules which are first trained in a projection phase from downstream layer signals before starting training on their own. The proposed method is tested in standard continual learning benchmarks as well as OOD detection and meta-learning settings.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall: The proposed method is simple and combines technologies in a reasonable way to achieve good results on a number of settings. I appreciated the breadth of experiments and various supplementary experiments (OOD detection, module coefficient visualizations in the appendix, etc.). I think that the formulation with the two stages of training and the projection phase is fairly elegant. I do have questions about the method and think a few details are inconsistent, but overall enjoyed the paper and feel it’s of sufficient quality to publish at NeurIPS.\n\nHere are a few comments that I would appreciate some feedback on in the author response:\n\n(1) One of the main contributions the paper claims to have is to not require the task ID at testing time. But task-agnostic continual learning is not a new topic, and the authors themselves cited works in this area (e.g. [28] and [30]). So is the main contribution of the paper just the combination of the task-agnosticity with the particular formulation of the module-level training? I’d like to understand in a more concise manner the core impact that the work has from a fundamental science perspective.\n\n(2) Why do the authors not compare to other established task-agnostic baselines? The comparisons are mostly to heavily modified versions of task-id-required methods, and these modifications are not too convincing (for example, using MNTDP with entropy is a bit dicey as entropy is a fairly unrobust statistic). \n\n(3) To what extent is task-agnosticity a common scenario? Usually the model user is pretty aware of which task they would like to perform, especially if the tasks are different (for example, if you need to do segmentation and detection within a single model). One scenario that might make it difficult to have task ID at inference is if you have a distribution of input data to perform the same task (e.g. digit classification with MNIST + SVHN), but this feels more like a domain adaptation setting to me. Can the authors expand on why task agnosticity is widely practical, and have the authors considered comparing to some domain adaptation baselines?\n\n(4) In the section on OOD detection with colored MNIST, why was the task-agnostic version of LMC not tested? It seems to detract from one of the central points of the paper to only analyze this setting with task-id-required LMC.\n\n(5) Some discussion on how the z’ threshold is set and how robust results are to this hyperparameter would be welcome.\n\n**Post Rebuttal**: Thank you to the authors for the detailed response. My questions/concerns were largely addressed. I think it is an interesting paper with good evidence backing up claims. I will keep my rating a WA (6). \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ZRc3RBYUt3Z", "reviewer": "Reviewer_Vtim", "summary": "This paper tries to tackle continual learning with a modular approach and proposes the local module composition (LMC) approach. LMC composes modules dynamically by computing local relevance scores for each layer, computed with the input. This architecture makes LMC perform well for OOD tasks as well as outperforming existing approaches on a continual learning benchmark.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality & Significance\n- Although modular approach is not a new concept in continual learning, LMC allows a soft-gating algorithm that is input-dependent rather than task-dependent. Combining modular expansion with the gating algorithm is the primary contribution of the work in my opinion. The gating algorithm is primarily based on the work of Hocquet et al. [30] and Aljundi et al. [2]. The results demonstrate how LMC can be useful for OOD tasks, but in the usual CL case where task identities are known, it does not show significantly superior performance over other methods.  Overall, this combination of modular architecture with the gating algorithm leads to some interesting results.\n\nQuality\n- This paper proposes a new modular approach for CL and shows outstanding performance especially for OOD tasks which I find interesting.\n- One of the main benefits of modular approaches is how certain modules can be reused for quite different tasks enabling better parameter sharing and generalization. By looking at Appendix B.5, it seems modules typically specialize into different tasks without much reuse except for very related tasks. An analysis on this would be beneficial.\n- This leads to a question, can the superior performance of LMC versus other baseline models be attributed to more trainable parameters? - it is hard to judge without knowing what kind of architectures/parameter counts were used for baseline models. \n-  The l2-norm structural objective function could cause dead modules in principle - certain modules may always output 0 and not used, for example at some layer, out of N available modules, only one might be always chosen if N-1 modules are optimized to always output 0s. This could limit the capacity of the network even though the whole network has a large amount of trainable parameters. Was this observed in experiments or how was this avoided?\n- In my opinion, the major benefit of modular approach in continual learning is how it does not forget when given a lot of different tasks. However, only five tasks are considered in this work. It would be great to consider the S^{long} setting as in MNTDP paper which this work compares with the most.\n- LMC has the additional step of adapting to a new dataset through the projection phase - which I like. To demonstrate the usefulness of the whole architecture, it would be good to see if it can significantly outperform MNTDP when MNTDP is given a chance of adaptation too - for example by having a normalization layer for the new dataset to match the statistics of the previously seen datasets. \n- Figure 3 has the same tables for (a) and (d) and this should not be the case.\n\nClarity\n- The paper is well-organized and written. It was easy to read", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they are included.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "M7HkYZwJRSb", "reviewer": "Reviewer_KtPP", "summary": "This paper proposes a continual learning method by learning module networks and a way to compose them. One notable difference is that the proposed approach does not reply on an external task id as the input. Instead, it learns to predict whether the input batch can be composed with the current modules or need to add another module. The authors show several experiments to validate the performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This proposed method serves as an important step toward a more general continual learning setting where task identity is unknown. And the idea is pretty simple and reasonable: by tracking the mean and variance of the structural score output by each module, the model itself could predict whether new module is needed.\n\nHowever, I had a hard time understanding the key part of the proposed method, which makes me less confident about such an assessment. Specifically, I’m confused about section 3.2 and appendix A.2. Could the authors clarify how the projection and accumulation work? When does the projection and accumulation happen? \n- Does that happen at every iteration?\n- If not, does that only happen during a new task is added (or during an outlier z score detected)? If so, in figure 2, why the projection only happen in the last layer for t=1 but happen two times for t=3 (I think it’s typo and should be t=2)?\n- And what is the output space of the whole network? For example, in the CTrL benchmark, is the final layer of your network a 5-way classifier? If so, does that mean the expansion of final layer is not according to the outlier detection but the task identity?\n\nAnother weakness is that the method only shows the continual transfer learning in MNIST. In contrast, the recent method (such as [63]) do the same task on CIFAR and miniImageNet. I would highly encourage the author do more experiments on larger scale and more complex datasets.\n\nOverall, I think this is an important methodology improvement. However, I don’t think I fully understand the details so I will keep my score as borderline for now.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I didn't find any potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.0 hours"}, "extra_text": {}}, {"review_id": "ABU_B8rCqha", "reviewer": "Reviewer_Hxbt", "summary": "The paper proposes a new algorithm for addressing continual learning (CL) by leveraging modular architectures. In contrast to (the few) existing modular CL works, this one focuses on the task-agnostic setting, where the agent is not informed of the task ID. The proposed architecture assigns two components to each module: a functional component and a structural component. The latter is in charge of locally determining whether each data point (actually each mini-batch, in practice) is in-distribution for the given module. The training proceeds in stages, adding new modules whenever the new data point is out-of-distribution for all modules, then training the module such that its output is in-distribution for modules at later depths, and finally training the entire model together. Extensive experiments are carried out to demonstrate the strengths of the proposed approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "############## Strengths ##############\n\n1. The problem setting is highly relevant and vastly understudied\n2. The proposed approach addresses one particular shortcoming of existing modular CL methods: the ability to work without task IDs\n3. The experiments are extensive, comprehensive, and well described and analyzed\n\n############## Weaknesses ##############\n\n1. The description of the technical approach is too short and somewhat hard to follow, making several choices unclear and unjustified\n2. The continual meta-learning experiments seem disconnected from the rest of the paper to me\n3. The method has no mechanism for limiting the amount of experts that are added, which is empirically demonstrated to lead to a large number of modules in the only experiment where this is evaluated\n\n############## Recommendation ##############\n\nThis is a solid submission and I recommend accepting it. While technically the submission doesn't introduce a fancy new method with many technical contributions, it does demonstrate a clear understanding of the challenges of modular CL, and addresses one of the shortcomings of existing modular CL approaches. To me the biggest plus of the manuscript is the quality of the empirical evaluation, with well-chosen baselines evaluated on a variety of settings. \n\n############## Arguments ##############\n\nOn the technical side, the key contribution is proposing for each module to individually select whether each data point should be routed to it. This is a relatively minor technical contribution, but I do not think that is necessarily problematic given that this is a fairly new problem setting. What does concern me is that this section is very short and I don't believe that it is described in enough detail. In particular, no intuition behind the structural loss is given. I make out between the main text and Appendix A.3 that the idea is for this module to serve as a density estimator that detects out-of-distribution samples. However, it is unclear how the losses described in Sec 3 achieve this, or why different losses must be used for the feature extractor and the outputs. \n\nOne interesting technical insight is that training the structural component locally and with a separate loss than the prediction loss enables fully-differentiable training even if using hard modular selections. The authors also clearly explain that the key challenge for learning a task-agnostic modular component is that this component itself can suffer from forgetting if using standard gating functions, which is not the case with the local approach proposed here. The projection phase is an interesting idea to ensure that the new module is compatible with other modules, and this is possible also due to the choice of local module selection.\n\nThe empirical evaluation is very strong. The first experiment on CTrL (a benchmark specifically designed to test properties that modular CL methods should exhibit) compares against a wide array of well-chosen baselines, including the two primary related works (Mendez & Eaton and Veniat et al.). Results in Table 1 are strong and convincing. Then the second experiment proposes a benchmark specifically for out-of-distribution generalization, which was not studied in either of the preceding modular CL works, and the authors again demonstrate a strong ability to generalize OOD. The third experiment demonstrates that independently trained models can later be combined to achieve equal-or-better performance than the original separate models. \n\nThat being said, I recommend removing the final experiment on continual meta-learning. The problem setting for this experiment is unclear (even though it's somewhat described in Appendix E). But the main problem is that this is a completely different problem setting than studied in the rest of the paper, plus even the algorithm is modified to deal with the modified problem setting. It seems disconnected and contributes to the extreme length of the paper (26 pages including appendices).\n\nOne concern which the authors only mention in the conclusion is that the model grows quite rapidly. This is demonstrated in the results of Table 1, where the proposed method uses nearly one new module per task, just like the MoE. While in principle using modular architectures enable sub-linear growth (as the authors mention) it isn't clear whether their proposed method is actually sub-linear in practice. Note that having fewer than one new module per task is _not_ equivalent to being sub-linear (e.g., maybe the model learns 0.5 new modules per task, which is still linear). Part of what makes this such a major issue is that the model is allowed to grow at every sample (or mini-batch), which is quite frequent.\n\n############## Additional feedback ##############\n\nThe following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.\n\n\nIntro\n- \"Modules self-determine their relevance given an input\" -- sounds very similar to the competitive mechanism of RIMs [1]. Perhaps a comparison is warranted.\n\nSec 3\n- Equation 7 doesn't make it clear at all that the structural components are only trained via the structural loss. With soft modularity (as used in most experiments), the first term (the overall loss) is also differentiable w.r.t. the structural parameters, but later it is made clear that this is not used. It might be best to present this as two separate objectives used to train disjoint sets of parameters.\n\nConclusion\n- I like how the conclusion points to actually relevant directions for future work.\n\nTypos\n- CtRL is used multiple times, but it's CTrL\n\nAppendices\n- The entire meta-CL addition is very long, and I would frankly recommend removing it from the paper. It's a whole new problem setting (which I don't think is described in full detail) with some changes in the algorithm to address this particular problem setting.\n\n[1] Goyal et al. \"Recurrent Independent Mechanisms,\" (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper originally received three borderline positive reviews but without high confidence, and so a fourth review was solicited after the regular review period from a reviewer who is extremely knowledgeable on the topic.  From this fourth review, it is clear that this paper should be accepted.

This paper focuses on continual learning of modular representations in task-agnostic settings, develops an approach that addresses this challenging, important, and understudied problem, and performs an extensive empirical analysis on a variety of well-chosen baselines. 

The authors should be aware that there are some unclear parts, identified in the reviews, that even the most expert of the reviewers found difficult to follow at times.  These need to be clarified for the final version. 
Decision: Accept (Poster)