Continual Learning via Local Module Composition
Year: 2022

Abstract:
Modularity is a compelling solution to continual learning (CL), the problem of modeling sequences of related tasks. Learning and then composing modules to solve different tasks provides an abstraction to address the principal challenges of CL including catastrophic forgetting, backward and forward transfer across tasks, and sub-linear model growth. We introduce local module composition (LMC), an approach to modular CL where each module is provided a local structural component that estimates a module’s relevance to the input. Dynamic module composition is performed layer-wise based on local relevance scores. We demonstrate that agnosticity to task identities (IDs) arises from (local) structural learning that is module-specific as opposed to the taskand/or model-specific as in previous works, making LMC applicable to more CL settings compared to previous works. In addition, LMC also tracks statistics about the input distribution and adds new modules when outlier samples are detected. In the first set of experiments, LMC performs favorably compared to existing methods on the recent Continual Transfer-learning Benchmark without requiring task identities. In another study, we show that the locality of structural learning allows LMC to interpolate to related but unseen tasks (OOD), as well as to compose modular networks trained independently on different task sequences into a third modular network without any fine-tuning. Finally, in search for limitations of LMC we study it on more challenging sequences of 30 and 100 tasks, demonstrating that local module selection becomes much more challenging in presence of a large number of candidate modules. In this setting best performing LMC spawns much fewer modules compared to an oracle based baseline, however it reaches a lower overall accuracy. The codebase is available under https://github.com/oleksost/LMC.

heading: 1 Introduction
text: The goal of continual learning (CL) is to learn efficiently from a non-stationary stream of tasks without (catastrophically) forgetting previous tasks [62]. CL is often modeled as a trade-off between knowledge retention (stability) and knowledge expansion (plasticity) [26, 64]. Parameter sharing can provide control over this trade-off. For example, learning a single model shared across tasks results in better knowledge transfer and faster learning at the expense of forgetting [46, 57]. Conversely, learning a separate model per task eliminates forgetting but minimizes transfer and data efficiency [2, 41].
Modular learning aims at balancing transfer and forgetting by learning a set of specialized modules that can be recomposed to solve (new) tasks while only updating a subset of relevant modules or adding new modules [6, 47, 27]. In principle, a modular learner capable of composing modules in meaningful structures can provide additional benefits including (i) computational gains due to only executing modules that are relevant to a task [47, 4]; (ii) memory gains due to instantiating a sub-linear number of modules w.r.t. the number of tasks; (iii) systematic [8] and out-of-distribution (OOD) generalization [18] through knowledge recombination; and (iv) biological plausibility [91, 90, 96].
corresponding author: oleksiy.ostapenko@t-online.de
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Designing modular methods for CL comes with two main challenges. The first is how and when to add new modules to ensure sufficient plasticity to learn new tasks. Existing modular methods use greedy search variants, expanding the model when it improves validation performance [92, 63]. The second challenge is how to compose that is, retrieve task-specific structural knowledge given a new task (previously seen or not).
Existing methods rely on a task’s identifier (ID) to retrieve task-specific structural knowledge, which comes either in the form of an optimal module layout [92] or as a model- and task-specific controller network that generates modular layouts [63]. Unfortunately, in many realistic CL scenarios task identities are unavailable at test time [23, 35, 14]. Lifting this limitation is challenging since standard mechanisms for task inference, for example, leveraging a task-inference model, could be subject to forgetting themselves.
To address both challenges, we equip each module with a local structural component that predicts a score indicating how relevant the module is for a given input. In-distribution inputs result in high scores, while out-of-distribution inputs result in low scores. In other words, modules self-determine their relevance given an input.
This local component is used for composing modules: for each datum, modules are combined at each layer according to their normalized scores without requiring a task’s ID (§3). The local component is also used for module expansion: a new module is instantiated if all the current modules flag their input as being locally out-of-distribution (§3.1). Further, new shallow modules (i.e. closer to the input) are first trained in a projection phase to maximize the relatedness scores of subsequent, deeper, modules (§3.2). This process projects the output of new modules into the representation space expected by the subsequent modules and ensures the compatibility between low- and high-level modules.
In a set of studies, we explore the performance and versatility of our local structural approach, which we call Local Module Composer (LMC). First, we show that LMC reaches superior or comparable performance to existing modular and non-modular methods without requiring task IDs at test time using the Continual Transfer Learning (CTrL) benchmark, designed to evaluate transfer and forgetting in CL [92] (§4.1). Then, we demonstrate how LMC, relying on its projection phase, can solve out-ofdistribution (OOD) tasks not seen during the continual training (§4.2). We also show it is possible to combine modules from independently trained models into a new model to solve tasks seen by each of the independent models without any finetuning (§4.3). Finally, an analysis of longer task sequences (30 and 100 tasks) reveals that LMC tends to spawn much fewer modules to reach good performance than the fully task-aware MNTDP [92] counterpart. However, LMC reaches slightly lower accuracy on longer sequences than MNTDP, which highlights the difficulty of automatic task-ID agnostic module selection in the presence of a large number of candidate modules. In Appendix F we demonstrate the applicability of LMC in the meta-continual learning (meta-CL) setting, a task-agnostic setting by nature.
We highlight that by relying on a local (per-module) structural component, LMC offers a modular CL approach that i) does not require task IDs during test in the standard task incremental settings; ii) balances parameter sharing to yield strong CL performances compared to baselines that require access to the task ID; iii) in our experiments instantiates a sub-linear number of modules; iv) permits recombination of modules at test time enabling OOD generalization as well as (v) the ability to combine independently trained models in a third model without fine-tuning. Notably, the OOD generalization is only possible if the agent is task-agnostic in the module selection process, since OOD tasks were not observed at training, the learner has to interpolate between the learned tasks, and a (categorial) task ID is of no use.

heading: 2 Background: Modular Continual Learning
text: Let F(x; θ) : X → Y be a learner parametrized with a set of parameters θ. In task-incremental CL, the learner is exposed to a sequence of tasks. Each task is composed of a training set Dt of (x, y) pairs and a task identifier (ID) t [92, 46]. The goal is to learn an optimal θ∗ that minimizes the loss L for all observed tasks:
θ∗ = arg min θ T∑ t=1 E(x,y)∼Dt [L(F(x; θ), y)]. (1)
The parameter sharing trade-off between tasks can be addressed through different architectural design choices for F . For example, F can be a monolithic network that shares parameters θ across all tasks. Most existing task incremental CL methods use a task-specific output head, requiring the task ID to select the output head corresponding to the task at hand [46, 87, 1].
At the other end of the spectrum are the expert based solutions that learn an independent model, a.k.a. expert, for each task [2, 83]. In this case, each expert trains task-specific parameters θ = {θ(t)}Tt=1. To balance parameter sharing and transfer, modular methods organize their parameters in a series of modules M = {m(l)k } with parameters θ = {θ (l) k }, where θ (l) k denotes the parameters of module k at layer l in F . In general, a module can be any parametric function. In our experiments, unless otherwise stated, a module consists of a single convolutional layer followed by batch-norm, ReLU activation, and a max-pooling operation.
Modules can be composed conditioned on a sample, a batch of samples, or a task. Let ψ denote a specific composition of modules that gives rise to a distinct prediction function; we make this dependence explicit: F(x; θ, ψ). Importantly, sharing modules across tasks should lead to desirable transfer properties.
Veniat et al. [92] frames modular CL as finding an optimal layout ψ(t) for each task, where each layout selects a single module per layer per task (hard selection):
θ∗,Ψ∗ = arg min θ,Ψ T∑ t=1 E(x,y)∼Dt [L(F(x; θ, ψ (t)), y)]. (2)
In this case the set of layouts Ψ = {ψ(t)} grows with the number of tasks, while modules can be reused across different task-specific layouts resulting in sub-linear growth pattern. They design a method called MNTDP to search the exponentially large space of modular layouts by only considering layouts resulting from adding a new module per layer to the best prior path (past task’s path with the highest nearest neighbor accuracy on a new task) starting at the top layer. This solution relies on task IDs to retrieve ψ(t) at test time.
Another way of composing modules uses dynamic routing [63, 82, 47, 65]. The module layout is generated by a structural function ψ = s(x), hence different inputs take different routes through F . It is standard to approximate the structural function using a neural network ψ = s(x;φ) with structural parameters φ. This framework has been applied to CL in [63] by learning a separate structural function per task ψ(t) = s(x;φ(t)). The task IDs are used to retrieve the correct structural function:
θ∗,Φ∗ = arg min θ,Φ T∑ t=1 E(x,y)∼Dt [L(F(x; θ, s(x;φ (t))), y)], (3)
where Φ = {φ(t)} is the set of structural parameters for all tasks.
The above methods require task IDs at both training and testing time. Next we introduce our modular CL approach that only relies on task IDs during training.

heading: 3 Local Module Composer (LMC)
text: We propose LMC, a modular approach where each module consists of a functional component f(x; θ (l) m ) and a structural component s(x;φ (l) m ), see Figure 1. The functional components are responsible for learning to solve the prediction task and are trained via the usual task loss L (e.g. cross-entropy loss for classification). The structural components receive the corresponding functional output as their input (see Figure 1) and are responsible for dynamic routing through F . Structural parameters φ are trained using a structural loss L(st.) computed locally at each module. Intuitively, the structural component of a module should serve as a density estimator of the outputs of the functional component. The module’s contribution to the layer’s output is proportional to the likelihood of the input sample under the estimated density. In our instantiation, the structural component produces a relatedness score: a lower score for inputs that are more likely to belong to the distribution on which a given module was trained, and a higher score for inputs that are out-of-distribution for the given module. Hence, the likelihood of the input sample is approximated by the negative relatedness score.
Given an input data sample x(0) = x, the output x(l) of a layer l is defined as the weighted sum of the functional outputs of all |M (l)| local modules and used as input to the subsequent layer l + 1:
x(l) = |M(l)|∑ m=1 w(l)m · f(x(l−1); θ(l)m ). (4)
The functional output of the network is equal to the output of the final layer: F(x; θ, φ) = x(L). In the last layer F implements a single output-head per task. At training time, the task ID is available and we update only the output-head corresponding to the currently learned task. At test time, the task ID is not available and we select the output head with the highest activation weight w(L)m , i.e., the last layer performs hard module selection.
The module activation weights w(l)m are computed by normalizing the vector of local relatedness scores γ(l) ∈ R|M(l)|. Each element of γ(l) is obtained from the negative structural loss which approximates the likelihood of each module:
γ(l)m = −L(st.) ( s [ f(x(l−1); θ(l)m );φ (l) m ]) , (5)
w(l)m = softmax(γ (l))m. (6)
Modules with lower structural loss get higher activation weights. Note that in practice, it can be useful to bias the module selection towards the expected module selection in a batch, assuming that samples within a batch are likely to belong to the same task. We discuss this point further in § A.1.
Instead of using the softmax function, it is possible perform hard selection taking the module with the highest score [82], or alternatively selecting top-k modules [88]. In both cases, LMC’s structural parameters stay differentiable due to the local nature of structural learning. Note that in the case of global structural objective, hard module selection would require applying tools for non-differentiable learning such as Expectation Maximization [47] or reinforcement-learning based methods [82, 5].
The overall LMC objective consists of optimizing both functional and structural losses:
θ∗, φ∗ = arg min θ,φ T∑ t=1 E(x,y)∼Dt [ L ( F(x; θ, φ), y ) + L∑ l=0 |M(l)|∑ m=0 L(st.)m ( s[f(x(l−1); θ(l)m );φ (l) m ] )] .
(7)
As in [92], learning is performed w.r.t. only newly introduced modules to prevent forgetting.
Structural component. We test two instantiations of the structural component s and loss L(st.)m . In the first one, s is an invertible neural network [80]. Here we use the invertible architecture proposed by Dinh et al. [21]. As shown by Hocquet et al. [37], for this invertible architecture the structural objective can be defined as L(st.)m (x) = ||x||2. Intuitively, an invertible architecture prevents L(st.)m from collapsing to an all-zeros solution.
In the second instantiation, s and f form an autoencoder and L(st.)m (x) = ||x(l−1) − x||2 is the reconstruction error with respect to the module’s input x(l−1). Aljundi et al. [2] used a similar idea was for selecting the most relevant expert network conditioned on a task. Unless stated otherwise, modules in the feature extractor use the autoencoder as their structural component, while output heads use invertible s — these combinations worked well in practice.

heading: 3.1 Expansion strategy
text: It is necessary to expand F as new tasks arrive to acquire new knowledge. A new module is added to a layer when all modules in this layer detect an outlier input. To this end, we track the running statistics of the relatedness score γ for each module — mean µ and variance σ (see Figure 1), and calculate a z-score for each sample in the batch and each module at a layer:
zm = wm − µm
σm . (8)
An input is considered an outlier if its z-score is larger than a predefined threshold z′ (see Appendix B.6 for an ablation study of z′ values). The expansion decision can be made on the per-sample (i.e., if an outlier sample is detected) or a per-batch basis (i.e., z is averaged over the mini-batch). Unless stated otherwise, in our experiments, the decision was made on a per-batch basis. Additionally, at training the parameters of existing modules are fixed once the task changes. If during a forward pass through F module addition is triggered at multiple layers, we start adding modules at the layer closest to the input.

heading: 3.2 Training
text: Each module in LMC receives two types of learning signal: a structural signal resulting from minimizing L(st.)m , and a functional signal resulting from minimizing the global functional loss L. All structural components s are trained only with the structural signal that is calculated locally to each module.
The training of functional components proceeds in two phases: projection and accumulation. Whenever the expansion strategy triggers the addition of a new module (i.e., zm > z′ ∀m ∈ {0, . . . , |M (l)|}), starting with layers closest to the input, LMC initiates the projection phase. During this phase, the new module is trained to minimize the structural loss from all the layers above and no new-module addition is allowed. This procedure makes the representation of new modules
compatible with subsequent modules and enables their composition. This procedure “encourages” already-learned modules to be reused, preventing over-spawning new modules. The functional signal is optional during projection (we kept it in all experiments unless otherwise stated).
In the accumulation phase, new module addition is allowed again and all non-frozen modules are trained with both signals. The functional components of new modules still receive a signal from the structural components of modules above. The two-phase training is explained schematically in Figure 2 and implementation details are provided in Appendix A.

heading: 4 Experiments
text: We now evaluate the performance, empirical capabilities, and properties of LMC in four different CL settings. First, in § 4.1 we study a standard task-incremental CL setting (task-ID agnostic and aware) using the Continual Transfer Learning Benchmark (CTrL) [92]. Next, we explore the properties of LMC through other CL settings. In § 4.2 we evaluate the continual OOD generalization ability of the proposed LMC. In § 4.3 we show the ability of LMC to combine modules form independently trained models. In Appendix F, we evaluate LMC in the Continual Meta-Learning setting.

heading: 4.1 Continual transfer learning using the CTrL benchmark
text: The CTrL benchmark was proposed to systematically evaluate properties of CL methods with a focus on modular architectures [92]. It consists of 5 streams of visual image classification tasks. The first stream S− = (t+1 , t2, t3, t4, t5, t1) consists of a sequence of 6 tasks, where the first and last task are the same except the first has an order of magnitude more training samples (“+”) than other tasks. This stream is designed to evaluate the direct transfer ability of models, i.e. a modular learner should be able to reuse the first task’s modules for the last task. The S+ = (t1, t2, t3, t4, t5, t+1 ) stream is similar to S−, but now the last task comes with more data than the other tasks (including the first one). Here, the modular learner should be able to update its knowledge, i.e. performance on the first task should improve after learning the last task. In the Sin = (t1, t2, t3, t4, t5, t′1) stream the first t1 and the last t′1 tasks are similar, with a slight input distribution change (e.g. different background color). In the Sout = (t+1 , t2, t3, t4, t5, t ′′ 1) stream the first task t1 and the last task t ′′ 1 differ in the amount of training data and the output distribution, i.e. the labels of the last task are randomly permuted. The plasticity stream Spl = (t1, t2, t3, t4, t5) evaluates the ability to learn a stream of unrelated and
potentially interfering tasks, i.e., transfer from unrelated tasks can harm performance. Descriptive statistics for all datasets are in Appendix B.1.
We compare to several baselines. Finetune: trains a single model (wider model marked with L) for all tasks. Experts: trains a model per task. We also compare with the several recently proposed modular CL baselines, which achieve competitive results in CTrL and require task IDs at test time. MNTDP [92]: a recent search-based module selection approach described in more detail in § 2. MNTDP requires the task ID to retrieve the previously found best structure for each test task. MNTDP(A): a task ID agnostic version of MNTDP we created, which selects the path with the lowest entropy in the output distribution. SG-F [63]: Soft-gating with fixed modules, a modular method mentioned in § 2. It relies on a task-specific structural network that generates soft-gating vectors for each layer of the modular learner and fixes learned modules when new tasks arrive. We slightly adapted the original expansion strategy of SG-F in order to conform to our experimental setup; details are in Appendix B.2. SG-F(A): a version of SG-F with a single structural network shared across tasks. HAT[87]: learns attention masks for activations that gate the gradients to prevent forgetting. The task ID is used to select a task-specific attention mask for inference.
We also compare to several standard CL methods. EWC [46]: trains a single model for all tasks while applying parameter-regularization to minimize forgetting. O-EWC:[85] online version of EWC that does not require storing a separate approximation of the Fisher information matrix per task. ER [16]: trains a single model while replaying samples from previously seen tasks. The size of the replay buffer corresponds to the memory size of the LMC assuming the worse case linear growth pattern (i.e., LMC with 24 modules on a 6-task sequence). ER(A,S): task ID agnostic version of ER that uses a single output head to classify all classes from all tasks: i.e. after learning stream S− the output head has 50 output neurons and the output classes of the last task are considered the same as the ones of the first task.
We use several versions of LMC. LMC(¬A) a version of LMC that uses the task ID for output head selection (not module selection as MNTDP). LMC(A): the default version of LMC. It equips output heads with structural components and is therefore task ID agnostic at test time. LMC(A,H): a version of task ID agnostic LMC that performs hard module selection, i.e., taking the module with the highest relevance score per layer. All methods use the same architecture (described in Appendix A.4) together with the Adam [44] optimizer. HAT is the only method that uses SGD.
Similar to Veniat et al. [92], we use the following evaluation metrics: (A) average accuracy on all seen tasks at the end of CL training; Forgetting (F ) — difference between accuracy at the end of the training and accuracy after learning the task averaged across tasks [60]; Number of modules (M) at the end of the continual training procedure. Formal definitions of all metrics are in Appendix B.3.
Table 1 reports performance using the CTrL benchmark. Overall, modular methods tend to outperform ER and the regularization based methods (HAT and EWC). Among the modular methods, soft-gating SG-F(A,F) with a single controller shared among all the tasks performed the worst. This baseline showcases the problem of forgetting in the global structural component (a.k.a. controller) of dynamic routing methods such as the one proposed by Mendez and Eaton [63]. A version of LMC performs the best on the S−, Sin and Sout streams.
Notably, LMC(A), which does not rely on task IDs at test time, outperformed all other task ID agnostic methods such as MNTDP(A) and ER(A,S) on all streams but S+, and always performed on par with task ID aware methods. On the Sout stream low performance is expected for task-ID agnostic methods due to output distribution shift: i.e., at test time we notice that LMC correctly assigns samples from the last task t′′1 to the first task’s t + 1 output head. However, the resulting classification accuracy is low because the labels of the last task are randomly permuted in this stream.
The task ID agnostic LMC(A) outperforms task ID aware LMC on the S− and S+ streams. Here, LMC(A) selects modules (and the output head) which were predominantly trained on the task that provided more training data (e.g. t+1 in S
− stream), hence transferring knowledge between the first and the last tasks. In contrast, LMC(¬A) when tested on the last task t1 is forced to select the output head belonging to this task, which was trained on less data than the output head of t+1 task, leading to lower accuracy. In addition, we observed that versions of LMC often exhibit high variance (e.g. see S+, Sout and Spl streams). This may be caused by the larger amount of trainable parameters compared to other models and relatively small amount of training data. Finally, low performance of
LMC(A,H) emphasizes the importance of soft modular attention for LMC. Additional results, including a transfer metric [92], are in Appendix B.4.

heading: 4.2 Compositional OOD generalization
text: This second study tests the ability of LMC to recombine modules for OOD generalization. We use a colored-MNIST dataset — a variation of the standard MNIST dataset of hand-written digits from 0 to 9 [43] in which digits are colorized. We design a simple sequence of tasks as follows. First, we define two high-level features: the foreground-background color combination (using the colors red, black, green, blue) and the class (0–9). Then, we create five non-overlapping tasks of two (digit) classes each: {0-1, ..., 8-9}. At training time the model is continually trained using a sequence of these tasks, however, each task is only seen in one of five different foreground-background combinations {red-black, green-black, blue-black, black-red, black-green}. At test time we measure the generalization ability to seen and unseen combinations of classes and colors.
In Figure 3 we present the accuracy matrices for different learners when tested on all 25 combinations of colors and classes after it has been trained only on the 5 tasks on the diagonal.
We compare the performance of LMC(¬A) with EWC [46], MNTDP [92], and an ablated version of LMC without the projection phase. We observe that the OOD accuracy attained by LMC is significantly higher than EWC and MNTDP. Since the model trained with EWC is monolithic, the digit-background color combinations are entangled with the digits’ shape for each task, hindering OOD generalization. In contrast, modular approaches such as MNTDP and LMC learn a different module combination for each task. In contrast to MNTDP, LMC’s module selection does not rely on task identifiers and each module is selected in a local manner based on its compatibility with the current input. This allows LMC to interpolate between previously seen tasks being able to dynamically compose existing modules to adapt to tasks that have not been seen at training. Because MNTDP’s module selection relies on a database of task-specific structures found to be optimal for the corresponding task at training, this method must reuse the predefined module compositions based on task IDs. This forces MNTDP to use modules that were trained using a different color combination, and results in e.g. a 24% accuracy drop with respect to LMC on [0,1] when the foreground and background colors are inverted w.r.t. the seen combination.
In Figure 3d we report results for LMC without applying the projection phase. The projection phase adapts the representation of newly introduced modules to match the distribution expected by the subsequent modules. As expected, we found that skipping it severely degrades performance. This result validates the usefulness of the projection phase to achieve an efficient local module selection. In Appendix E we plot the average module selection for all 25 test tasks, showing how modules are reused for the OOD tasks.
4.3 Combining modular learners
In earlier sections we show cross-task reusability of modules, here we test the cross-model reusability. We motivate the practical importance of this kind of reusability with a federated learning example: a privacy preserving training might be required for LMC1 and LMC2, trained on the premises of customers 1 and 2, after which their modules can be combined in a single central entity — LMC3, located on premises of the cloud service provider. LMC3 is required to perform tasks seen by both independent LMCs but can not be finetuned as it has no access to the original training data distributions.
In Figure 4 we test the ability of LMC to preserve and transfer knowledge in such setting. To this end, we design the following tasks: fMNIST+ and fMNIST-. Both are sampled from the fashion-MNIST dataset [98] but the latter comes with an order of magnitude less training data. cMNIST-r is a variant of the colored-MNIST dataset where the background of 95% of the training samples is colored in red and 5% in green. For the cMNIST-g dataset these proportions are inverted and 95% of the training samples is colored in green. The test set contains 50% of samples with green and 50% with red background. We trained LMC1 continually on MNIST, fMNIST+, and cMNIST-r tasks. We trained LMC2 on fMNIST-, cMNIST-g, and SVHN. We then combined the modules of both LMCs layer-wise to obtain LMC3.
We observed positive transfer for both cMNIST and fMNIST- tasks with LMC3. We found that LMC3 selects different modules originating from different LMCs conditioned on test samples with different background colors — LMC1’s modules were specialized on red background while LMC2’s on green (selected paths presented in Appendix D). Notably, cross-model reusability without fine-tuning is novel to LMC and can be attributed to the local nature of the structural component. Using a global structural component as in [63] would require tuning a separate structural component specifically for LMC3. In case of task-specific routing of proposed for MNTDP [92], additional search would be needed to discover task-specific paths through the consolidated LMC3 model. In both cases the access to the orinal training data distributions would be required and privacy would not be preserved.

heading: 4.4 Longer task sequences
text: Here we study the performance LMC on longer task sequences consisting of 30 – Slong30, and 100 – Slong tasks. The Slong sequence corresponds to the one proposed by Veniat et al. [92] as part of the CTrL benchmark. Slong30 is a 30-tasks subset of Slong (see Appendix B.1 for details).
We first report the average test accuracy (A) and the total number of modules (M) of the models selected through cross-validation. Slong30: MNTDP A=64.58, M=64; LMC(¬A): A=62.44, M=50. Slong: MNTDP A=68.92, M=142; LMC(¬A): A=63.88, M=32. While the gap between the accuracy
of LMC and MNTDP on Slong30 is only 1.86%-points, in the case of Slong this gap grows to 6.58%- points. It is important to highlight that in contrast to LMC, MNTDP’s module selection is performed by a task ID aware oracle. We further analyze the trade-off between the number of modules and accuracy in Figure 5, where we plot the number of modules (M) against average test accuracy (A) for models that resulted from training with different hyperparameters. For both streams, we observe that LMC tends to spawn much fewer modules than MNTDP. However, MNTDP shines in the presence of large number of modules and achieves higher overall test accuracy on these streams. Interestingly, as can be clearly observed on the Slong stream, LMC reaches higher accuracy with smaller number of modules: e.g. ∼64% with 32 modules, while adding modules leads to lower accuracy: e.g. ∼61% with 98 modules. This result suggests that local task ID agnostic module selection becomes more challenging for LMC in presence of a large number of modules.

heading: 5 Related work
text: Modularity in neural networks is studied in the context of scalability [9], and more recently as a way to achieve compositionality and systematic generalization [6, 47, 15, 8, 27, 19] as well as for multi-task learning [65, 82]. From the causal point of view, a data generation process could be thought as a composition of independent causal modules [75]. Researchers model these kinds of systems using a set of independent modules, where each module is invariant to changes in the other modules induced by e.g. distribution shifts [84, 76]. This idea is crystallized by Parascandolo et al. [74], who propose a way to learn a set of causal independent mechanisms as mixture-of-experts. Building up on this ideas, others show evidence of compositional OOD generalization [61]. Recently, [66] argue for a more wholistic view on CL including OOD generalization as an important desiderata.
Continual learning methods typically address the problem of forgetting through parameter regularization [46, 70, 99], replay [89, 78, 3, 73, 54, 12, 97, 39, 81, 13] or dynamic architectures (and MoEs) [83, 87, 52, 85, 53, 41]. Our work falls under the umbrella of the latter and shares its advantage of having the capacity to adapt to a large number of related tasks. Our focus is on improving modular CL approaches, which despite their advantages, have only recently been studied in the CL literature [63, 92]. The main difference with our work is that we use a local composition mechanism instead of a global one. We detail this difference in §2 and also compare to these methods in §4.1.
Continual-meta learning focuses on fast learning and remembering [25, 35, 33, 41], often emphasising the online performance on OOD tasks [14]. As argued by Jerfel et al. [41] modularity can be useful in this setting to minimize interference between tasks. They proposed a way to train a MoE model, with each expert focusing on a cluster of tasks leveraging Bayesian nonparametrics. LMC aims at decomposing knowledge into layer-wise composable modules further reducing modular granularity. Continual-meta learning is often confused with its counterpart meta-continual learning [40, 11, 93], in which algorithm are learning to continually learn.
The rapid growth of continual learning has lead researchers to work on empirical studies [20, 58, 56], surveys [32, 42, 55, 66, 67] as well as CL-specific software [72, 22, 59].

heading: 6 Conclusion
text: We develop LMC, a method to learn and compose a series of modules on a continual stream of tasks fulfilling some of the basic desiderata of modular CL such as module specialization, avoidance of collapse, and sublinear growth. In LMC, structural information is learned and stored locally for each module. It is the locality of the structural component that enables generalization to related but unseen tasks, and that permits combining different LMCs without fine-tuning.
Future work could focus on achieving more efficient sub-linear model growth through OOD generalization and reusability of modules. Additionally, while the benefits of modularity for CL are well understood, the implications of the CL regime on modularity and compositionality have not been studied extensively. It is possible that providing knowledge to the learner in incremental chunks results in the implicit supervision needed to better disentangle it into specialized and composable modules. Another promising direction is removing the need for task boundaries during training and developing more robust architectures for the local structural component (related discussions are in Appendix A.3, and limitations in Appendix G).

Review summaries:
The paper proposes a new algorithm for addressing continual learning (CL) by leveraging modular architectures. In contrast to (the few) existing modular CL works, this one focuses on the task-agnostic setting, where the agent is not informed of the task ID. The proposed architecture assigns two components to each module: a functional component and a structural component. The latter is in charge of locally determining whether each data point (actually each mini-batch, in practice) is in-distribution for the given module. The training proceeds in stages, adding new modules whenever the new data point is out-of-distribution for all modules, then training the module such that its output is in-distribution for modules at later depths, and finally training the entire model together. Extensive experiments are carried out to demonstrate the strengths of the proposed approach.

Authors propose an algorithm to perform continual learning without task-id labels at inference. The method involves local module-level OOD estimators which then output linear coefficients that informs the model how to combine outputs. The model also has the option of creating new modules which are first trained in a projection phase from downstream layer signals before starting training on their own. The proposed method is tested in standard continual learning benchmarks as well as OOD detection and meta-learning settings. 

This paper tries to tackle continual learning with a modular approach and proposes the local module composition (LMC) approach. LMC composes modules dynamically by computing local relevance scores for each layer, computed with the input. This architecture makes LMC perform well for OOD tasks as well as outperforming existing approaches on a continual learning benchmark.

This paper proposes a continual learning method by learning module networks and a way to compose them. One notable difference is that the proposed approach does not reply on an external task id as the input. Instead, it learns to predict whether the input batch can be composed with the current modules or need to add another module. The authors show several experiments to validate the performance.