Lifelong Learning of Compositional Structures
Authors: Jorge A Mendez
ERIC EATON
Venue: ICLR 2021 Poster (2021)
Keywords: lifelong learning
continual learning
compositional learning
modular networks

Abstract: A hallmark of human intelligence is the ability to construct self-contained chunks of knowledge and adequately reuse them in novel combinations for solving different yet structurally related problems. Learning such compositional structures has been a significant challenge for artificial systems, due to the combinatorial nature of the underlying search problem. To date, research into compositional learning has largely proceeded separately from work on lifelong or continual learning. We integrate these two lines of work to present a general-purpose framework for lifelong learning of compositional structures that can be used for solving a stream of related tasks. Our framework separates the learning process into two broad stages: learning how to best combine existing components in order to assimilate a novel problem, and learning how to adapt the set of existing components to accommodate the new problem. This separation explicitly handles the trade-off between the stability required to remember how to solve earlier tasks and the flexibility required to solve new tasks, as we show empirically in an extensive evaluation.

Published as a conference paper at ICLR 2021
LIFELONG LEARNING OF COMPOSITIONAL STRUCTURES
Jorge A. Mendez and Eric Faton Department of Computer and Information Science University of Pennsylvania {mendezme, (eatonn@seeae upenn edu
ABSTRACT
A hallmark of human intelligence is the ability to construct self-contained chunks of knowledge and adequately reuse them in novel combinations for solving differ- ent yet structurally related problems. Learning such compositional structures has been significant challenge for artificial systems, due to the combinatorial nature the underlying search problem. To date, research into compositional learning has largely proceeded separately from work on lifelong or continual learning. We integrate these two lines of work to present a general-purposs. framework for life- long learning of compositional structures that can be used for solving stream of related tasks. Our framework separates the learning process into two broad stages: learning how to best combine existing components in order assimilate novel problem, and learning how to adapt the set of existing components to accommo- date the new problem. This separation explicitly handles the trade-off between the stability required to remember how to solve earlier tasks and the flexibility required to solve new tasks, as we show empirically in an extensive evaluation.
INTRODUCTION
A major goal of artificial intelligence is to create an agent capable of acquiring a general understand- ing of the world. Such an agent would require the ability to continually accumulate and build upon its knowledge as it encounters new experiences. Lifelong machine learning addresses this setting, whereby an agent faces a continual stream of diverse problems and must strive capture the knowl- edge necessary for solving each new task it encounters. If the agent is capable of accumulating knowledge in some form of compositional representation (e.g., neural net modules), it could then selectively reuse and combine relevant pieces of knowledge to construct novel solutions.
Various compositional representations for multiple tasks have been proposed recently (Zarembal al. 2016; (uu et al., 20177] Kirsch et al., 2018 Meyerson & Miikkulannen 2018). We address the novel question of how to learn these compositional structures in a lifelong learning setting. We design general-purpose framework that is agnostic to the specific algorithms used for learning and the form of the structures being learned. Evoking Piaget's (1976) assimilation and accommodation stages of intellectual development, this framework embodies the benefits of dividing the lifelong learning process into two distinct stages. In the first stage, the learner strives to solve a new task by combining existing components it has already acquired. The second stage uses discoveries from the new task to improve existing components and to construct fresh components if necessary.
Our proposed framework, which we depict visually in Appendix A is capable of incorporating various forms of compositional structures, as well as different mechanisms for avoiding catastrophic forgetting (McCloskey Cohen 1989). As examples of the flexibility of our framework. it can incorporate naive fine-tuning, experience replay, and clastic weight consolidation (Kirkpatrick et al., 2017 as knowledge retention mechanisms, and linear combinations of linear models (Kumar & Daumé III, 2012) Ruvolo & Eaton 2013), soft layer ordering (Meyerson & Miikkulainen, 2018), and a soft version of gating networks (Kirsch et al. 20188) Rosenbaum et al. 2018)) as the compositional structures. We instantiate our framework with the nine combinations of these examples, and evaluate it on eight different data sets, consistently showing that separating the lifelong learning process into two stages increases the capabilities of the learning system, reducing catastrophic forgetting and achieving higher overall performance. Qualitatively, we show that the components learned by an algorithm that adheres to framework correspond self-contained, reusable functions.

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
Lifelong learning In continual or lifelong learning, agents must handle a variety of tasks over their lifetimes, and should accumulate knowledge in a way that enables them to more efficiently learn to solve new problems. Recent efforts have mainly focused avoiding catastrophic forget- ting. At a high level, algorithms define parts of parametric models (e.g., deep neural networks) to be shared across tasks. As the agent encounters tasks sequentially, it strives retain the knowledge that enabled it to solve earlier tasks. One common approach is to impose regularization to prevent parameters from deviating in directions that are harmful for performance on the early tasks (Kirk- patrick et al. 2017)] Zenke et al., [017:] Li & Hoiem 2017, Ritter et al.) 2018). Another approach retains a small buffer of data from all tasks, and continually updates the model parameters utilizing data from all tasks, thereby maintaining the knowledge required to solve them (Lopez-Paz & Ran- zato 2017: Nguyen et al. 2018: Isele Cosgun| 2018). related technique to learn generative model to "hallucinate" data, reducing the memory footprint at the cost of using lower-quality data and increasing the cost of accessing data (Shin et al. 2017: Achille et al. 2018 Rao et al. 2019)).
These approaches, although effective in avoiding the problem of catastrophic forgetting, make no substantial effort toward the discovery of reusable knowledge. One could argue that the model pa- rameters are learned in such way that they are reusable across all tasks. However, it is unclear what the reusability of these parameters means, and moreover the way in which parameters are reused is hard-coded into the architecture design. This latter issue is major drawback when attempting to learn tasks with a high degree of variability, as the exact form in which tasks are related is often unknown. Ideally, the algorithm would be able to determine these relations autonomously.
Other methods learn a set of models that are reusable across many tasks and automatically select how reuse them (Ruvolo & Eaton| 2013 Nagabandi et al. 2019). However, such methods selectively reuse entire models, enabling knowledge reuse, but not explicitly in a compositional manner.
mostly distinct line of parallel work has explored the learning of compositional knowledge. The majority of such methods either learn the structure for piecing to- gether a given set of components (Cai et al.) 2017: Xu et al. 2018 Bunel et al.] 2018) or learn the set of components given a known structure for how to compose them (Bošnjak et al.. 2017).
Compositional knowledge
more interesting case when neither the structure nor the set of components are given, and the agent must autonomously discover the compositional structure underlying a set of tasks. Some ap- proaches for solving this problem assume access to a solution descriptor (e.g., in natural language), which can be mapped by the agent to a solution structure (Hu et al., 2017 Johnson et al. 2017 Pahuja et al., 2019). However, many agents (e.g., service robots) are expected to learn more au- tonomous settings, where this kind of supervision is not available. Other approaches instead learn the structure directly from optimization of a cost function (Rosenbaum et al. 2018) Kirsch et al.) 2018 Meyerson & Miikkulainer can be viewed as instances of neural architecture search, a closely related area (Elsken et al.) 2019).
2018 Alet et al. 2018 Chang al. 2019). Many of these works
However, note that the approaches above assume that the agent will have access to large batch of tasks, enabling it to evaluate numerous combinations of components and structures on all tasks simultaneously. More realistically, the agent faces a sequence of tasks in lifelong learning fashion. Most work in this line assumes that each component can be fully learned by training on single task, and then can be reused for other tasks (Reed & de Freitas 2016 Fernando et al. 2017 Valkov al.) 2018). Unfortunatcly, this is infeasible in many real-works scenarios in which the agent has access to little data for each of the tasks. One notable exception was proposed by Dayaaunt et al. (0117). which improves early components with experience in new tasks, but is limited to very simplistic settings.
Unlike prior work, our approach explicitly learns compositional structures a lifelong learning setting. We do not assume access to a large batch of tasks or the ability to learn definitive components after training on a single task. Instead, we train on a small initial batch of tasks (four tasks, in our experiments), and then autonomously update the existing components to accommodate new tasks.
Our framework also permits incorporating new components over time. Related work has increased network capacity in the non-compositional setting (Yoon et al. 2018) or in a compositional setting where previously learned parameters are kept fixed (Li et al. 2019). Another method enables adap- tation of existing parameters (Rajasegaran et al. 2019), but requires expensively storing and training multiple models for each task to select the best one before adapting the existing parameters, and is designed for a specific choice of architecture, unlike our general framework.
2

---

Published as a conference paper at ICLR 2021
3 THE LIFELONG LEARNING PROBLEM
We frame lifelong learning as online multi-task learning. The agent will face sequence of tasks T()) T(2) over its lifetime. Each task will be learning problem defined by cost function ( (f(1), where the agent must learn prediction function f(()) E F:; x²() + y)( to minimize the cost, where a function class, and (²(*) and y)(+) are the instance and label spaces, respectively. Each task's solution is parameterized by ((), such that f()) leco- The goal of the lifelong learner to find parameters 0(1), ..... O that minimize the cost across all tasks: Σ'= ( (f()). The
number of tasks, the order in which tasks will arrive, and the task relationships are all unknown.
Given limited data for each new task, the agent will strive to discover any relevant information to 1) relate it to previously stored knowledge in order to permit transfer and 2) store any new knowledge for future reuse. The agent may be evaluated on any previous task, requiring it to perform well on all tasks. In consequence, it must strive to retain knowledge from even the earliest tasks.
4 THE LIFELONG COMPOSITIONAL LEARNING FRAMEWORK
Our framework for lifelong learning of compositional structures (illustrated in Appendix A stores knowledge in set of k shared components M = {m1.... ....,........................... ) that acquired and refined over the agent's lifetime. Each component m, = ma E .M is a self-contained, reusable function parameterized by Φ; that can be combined with other components. The agent reconstructs each task's predictive function (() via task-specific structure g((²) x((() M F, with M being the set of possible sequences of k components, such that (x) g! (zz (z)),, where g( is parameterized by vector vb((), Note that s(1) yields a function from F. The structure functions select the components from M and the order in which to compose them to construct the model for each task (the (())s)) Specific examples of components and structures are described in Section 4.1
The intuition behind our framework is that, at any point in time f, the agent will have acquired set of components suitable for solving tasks it encountered previously (T(2), (((-1)). If these components, with minor adaptations, can be combined to solve the current task T(t), then the agent should first learn how reuse these components before making any modifications to them. The rationale for this idea of keeping components fixed during the early stages of training on the current task (+), before the agent has acquired sufficient knowledge to perform well on (+),, is that pre- mature modification could be catastrophically damaging to the set of existing components. Once the structure g(t) has been learned, we consider that the agent has captured sufficient knowledge about the current task, and it would be sensible to update the components to better accommodate that knowledge. If, instead, it is not possible to capture the current task with the existing components, then new components should be added. These notions loosely mirror the stages of assimilation and accommodation in Piaget's (1976) theories intellectual development, and so we adopt those terms. Algorithms under our framework take the form of Algorithm | split into the following steps.
Initialization The components M should be ini- tialized encouraging reusability, both across tasks and within diffferent structural configurations of task models. The former signifies that the components should solve particular sub-problem regardless of the objective of the task. The latter means that com- ponents may be reused multiple times within the structure for a single task's model, or at different structural orders across different tasks. For example, in deep nets, this means that the components could be used at different depths. We achieve this by train- ing the first few tasks the agent encounters jointly initialize M, keeping a fixed, but random, structure that reuses components encourage reusability.
Algorithm Lifelong Comp. Learning
Initialize components M while T()) +- getTask()
Freeze M for = I,..., structUpdates
Assimilation step on structure s(t)
if mod adaptFreq
Freeze s((), unfreeze M
for j ll compUpdates Adaptation step on M Freeze M. unfreeze s!!) Add components via expansion Store info. for future adaptation
Assimilation Algorithms for finding compositional knowledge vary in how they optimize each task's structure. In modular nets, component selection can be learned via reinforcement learning
3

---

Published as a conference paper at ICLR 2021
(Johnson al.) 2017: Rosenbaum et al., 2018 Chang et al. 2019 Pahuja al.) 2019), stochastic search (Femando et al. [017 Alet et al. 2018). or backpropagation (Shazeer et al. 2017 Kirsch
al. Meyerssn & Mikkullinen 2018). Our framework will use any of these approaches to assimilate the current task by keeping the components M fixed and learning only the structure s(t) Approaches supported by our framework must accept decoupling the learning of the structure from the learning of the components themselves; this requirement holds for all the above examples.
An effective approach should maintain performance on earlier tasks, while be- ing flexible enough incorporate new knowledge. To accommodate new knowledge from the cur- rent task, the learner may adapt existing components or expand to include new components:
Accommodation
Adaptation step Approaches for non-compositional structures have been to naively fine- tune models with data from the current task, to impose regularization to selectively freeze weights (Kirkpatrick et al.] 2017] Ritter et al., 2018), or to store a portion of data from previ- ous tasks and use experience replay (Lopez-Paz & Ranzato 2017] Iscle & Cosgun 2018). We will instantiate our framework by using any of these methods to accommodate new knowledge into existing components once the current task has been assimilated. For this to be possible, require that the method can be selectively applied to only the component parameters Φb
Expansion step Often, existing components, even with some adaptation, are insufficient to solve the current task. In this case, the learner would incorporate novel components, which should encode knowledge distinct from existing components and combine with those components to solve the new task. The ability to discover new components endows the learner with the flexibility required to learn over a lifetime. For this, we create component dropout, described in Section 4.2
Concrete instantiations of Algorithm are described in Section 5.1 with pseudocode in Appendix B
4.1 COMPOSITIONAL STRUCTURES
We now present three compositional structures that can be learned within our framework.
Linear combinations of models In the simplest setting, each component is a linear model, and they are composed via linear combinations. Specifically, we assume that x²( C Rd, and each task- specific function is given by faco (a:) (()) æ, with ( E Rdd The predictive functions are con- structed from a set of linear component functions ma. (x) = Φ; æ, with Φ, E Rd, by linearly com- bining them via a task-specific weight vector 2b(() E R yielding: (()( = Saptes (x, Mx) = (t) x)), where we have constructed the matrix + [eI...., Φu to collect all components.
Soft layer ordering In order to handle more complex models, we construct compositional deep nets that compute each layer's output as a linear combination the outputs of multiple modules. As proposed by Meyerson & Miikkulainen (2018), number of components matches the network's 's depth, and all components share the imput and output dimensions. Concretely, each component is a deep net layer mæ (x) = a(p x)) where a is any nonlinear activation and Φi E R A set of parameters wh(t) E k weights the output of the components at each depth: ((() = D(1) Σk= V o.... Ei= V/, g(o, where
assume that each module is one layer, the
(()) and D(t) are task-specifie input and output transformations such that E(1) x00 + Rd and D(t) 2 + y(t), and the weights are restricted to sum to one at each depth j: Σ Vi)
(
1
Soft gating In the presence of large data, it is often beneficial to modify the network architecture for each input æ: (Rosenbaum al. 2018 Kirsch al. 2018). unlike both approaches above which use constant structure for each task. We modify the soft layer ordering architecture by weighting each component's 's output at depth by deput-depnndent soft gating net s/0 :x(t)) + R giving predictive function s(9) =D() Ein [s] (#))im, Σ Ex-ls (æ)))m, g(, As above, we (
restrict the weights to sum to one at each depth: E [s) (x));=1.
4.2 EXPANSION OF THE SET OF COMPONENTS M VIA COMPONENT DROPOUT
To enable our deep learners to discover new components, we created an expansion step where the agent considers adding a single new component per task. In order to assess the benefit of the new
4

---

Published as a conference paper at ICLR 2021
component, the agent learns two different networks: with and without the novel component. Dropout enables training multiple neural networks without additional storage (Hinton et al. 2012 and has been used to prune neural net nodes in non-compositional settings (Gomez et al. 2019). Our pro- posed dropout strategy deterninistically alternates backpropagation steps with and without the new component, which we call component dropout. Intermittently bypassing the new component ensures that existing components can compensate for it if it is discarded. After training, we apply a post hoc criterion (in our experiments, validation error check) potentially prune the new component.
4.3 COMPUTATIONAL COMPLEXITY
Approaches to lifelong learning tend to be computationally intensive, revisiting data or parameters from previous tasks at each training step. Our framework only carries out these expensive operations during (infrequent) adaptation steps. Table T summarizes the computational complexity per epoch of the algorithms described in Section 5.1 The assimilation step of our method with expansion (dynamic + compositional) is comparable to compositional baselines in the worst case (one new component per task), and our method without expansion (compositional) is always at least as fast.
Table 1: Time complexity per epoch (of assimilation, where applicable) for n samples of d features, k components of d nodes, T tasks, and 7.m replay samples per task. Derivations in Appendix
Dyn. + Comp.
Compositional
Joint
No Comp.
ER
o ((Tnm + nnd dk2 - o((T + m)d + d) O(nd + dk + d)) (nd (dk: + d))
EWCL
O (nd (ikT + d)) (nd(átk + d))
O (nd((TeRk + dh2² + d))
NFT
Ond + d))
5
EXPERIMENTAL EVALUATION
5.1 FRAMEWORK INSTANTIATIONS AND BASELINES
We evaluated our framework with the three compositional structures of Sec- tion 4.1 All methods assimilate task (() via backpropagation on the structure's parameters (t()), For each, we trained three instantiations of Algorithm I, varying the method used for adaptation:
Instantiations
Naïve fine-tuning (NFT) updates components via standard backpropagation, ignoring past tasks. Elastic weight consolidation (EWC, Kirkpatrick et al. 2017) penalizes modifying model param-
eters via > |0 |2 1 where F(!) the Fisher information around (()) Backpropa-
F(!) with Kronecker factors. Experience replay (ER) stores n'm samples per task in a replay buffer, and during adaptation takes backpropagation steps with data from the replay buffer along with the current task's 's data. We explored variations with and without the expansion step: dynamic- + compositional methods use component dropout add new components, while compositional methods keep fixed-size set.
gation is carried out on the regularized loss, and we approximated
For every adaptation method listed above, we constructed two baselines. Joint base- lines use compositional structures, but do not separate assimilation and accommodation, and instead update components and structures jointly. In contrast, no-components baselines optimize a single architecture to be used for all tasks, with additional task-specific input and output mappings, g(t) and ( The latter baselines correspond to the most common lifelong learning approach, which learns monolithic structure shared across tasks, while the former the naïve extensions of those methods to a compositional setting. We also trained an ablated version of our framework that keeps all components fixed after initialization (FM), only taking assimilation steps for each task.
Baselines
5.2 RESULTS
We evaluated these methods on tasks with no evident compositional structure demonstrate that there is no strict requirement for a certain type of compositionality. compositional data set, and shows that our results naturally extend to that setting. We repeated
simple
While it is theoretically possible for EWC to operate in constant time w.r.t. T, practical implementations use per-task Kronecker factors due the enormous computational requirements of the constant-time solution.
5

---

Published as a conference paper at ICLR 2021
Table Average final performance across tasks using factored linear and Landmine (higher is better) and RMSE for Schools (lower is better). Standard errors after +.
for FERA
Base
Algorithm
FERA
Landmine
Schools
Compositional
79.0 0.4%
93.6 + 0.1%
10.65 + 0.04
ER
Joint
78.2 + 0.4%
90.5 + 0.3%
11.55 = 0.09
No Comp.
66.4 + 0.3%
93.5 + 0.1%
10.34 + 0.02
Compositional
79.0 0.4%
93.7 0.1%%
10.55 1 0.03
EWC
Joint
72.1 + 0.7%
10.73 - 0.17
No Comp.
60.1 0.5%
03.5+0.111
10.35 0.02
Compositional
79.0 0.4%
93.7 + 0.1%
10.87 + 0.07
NFT
Joint
67.9 + 0.6%
72.8 + 2.5%
25.80 + 2.35
No Comp. experiments ten times with varying random seeds. For details on data sets and hyper-parameters Code and data sets are available at htttps:// www.tthbb.ccmm cwm/t. L. fe Lon--ML/ git. Additional results, beyond those presented in this section,
57.0 + 0.9%
2.47%
18.01 = 1.04
see Appendix [
given in Appendix
5.2. 1 LINEAR COMBINATIONS OF MODELS
We first evaluated linear combinations of models on three data sets used previously for evaluating linear lifelong learning (Ruvolo & Eaton 2013). The Facial Recognition (FERA) data set tasks involve recognizing one of three facial expression action units for one of seven people, for total of T = 21 tasks. The Landmine data set consists of T = 29 tasks, which require detecting land mines in radar images from different regions. Finally, the London Schools (Schools) data set contains T = 139 regression tasks, each corresponding to exam score prediction in a different school.
Table summarizes the results obtained with linear models. The compositional versions of ER, EWC, and NFT clearly outperformed all the joint versions, which learn the same form of models but by jointly optimizing structures and components. This suggests that the separation of the learning process into assimilation and accommodation stages enables the agent to better capture the structure of the problem. Interestingly, the no-components variants, which learn a single linear model for all tasks, performed better than the jointly trained versions in out of the three data sets, and even outperformed our compositional algorithms in one. This indicates that the tasks in those two data sets (Landmine and Schools) are so closely related that single model can capture them.
5.2.2 DEEP COMPOSITIONAL LEARNING WITH SOFT LAYER ORDERING
We then evaluated how the different algorithms performed when learning deep nets with soft layer ordering, using five data sets. Binary MNIST (MNIST) is a common lifelong learning bench- mark, where each task is a binary classification problem between a pair of digits. We constructed T' = 10 tasks by randomly sampling the digits with replacement across tasks. The Binary Fash- ion MNIST (Fashion) data set is similar to MNIST, but images correspond to items of clothing. For these two data sets, all models used a task-specific input transformation layer g() initialized at random and kept fixed throughout training, to ensure that the input spaces were sufficiently differ- ent (Meyerson & Mikkuulainn 2018). J. A more complex lifelong learning problem commonly used in the literature is Split CUB-200 (CUB), where the agent must classify bird species. We created = 20 tasks by randomly sampling ten species for each, without replacement across tasks. All agents used a frozen ResNet-1s 18 pre-trained on ImageNet as a feature extractor {() shared across all tasks. For these first three data sets, all architectures were fully connected networks. To show that our framework supports more complex convolutional architectures, we used two additional data sets. We constructed lifelong learning version of CIFAR-100 (CIFAR) with T 20 tasks by randomly sampling five classes per task, without replacement across tasks. Finally, we used the Omniglot data set, which consists of T = 50 multi-class classification problems, each corresponding detecting handwritten symbols in a given alphabet. The inputs to all architectures for these two data sets were the images directly, without any transformation (t))
Results in Table g show that all the algorithms conforming to our framework outperformed the joint and no-components learners. In four out of the five data sets, the dynamic addition of new components yielded either or marginal improvements. However, CIFAR it was crucial for

---

Published as a conference paper at ICLR 2021
Table 3: Average final accuracy across tasks using soft layer ordering. Standard errors after +.
Base
Algorithm
MNIST
Fashion
CUB
CIFAR
Omniglot
Dyn. + Comp.
97.6 0.2%
96.6 + 0.4%
79.0 + 0.5%
77.6 + 0.3%
71.7 + 0.5%
Compositional
96.5 + 0.2%
95.9 + 0.6%
80.6 + 0.3%
58.7 + 0.5%
71.2 + 1.0%
ER
Joint
94.2 0.3%
95.1 0.7%
77.7 0.5%
70.7 65.8 0.4%
No Comp.
91.2 0.3%
93.6 0.6%
44.0 0.9%
51.6 0.6%
43.2 4.2%
Dyn. + Comp.
97.2 = 0.2%
73.9 9..5
0..6...3%
71.5 0.5%
Compositional
96.7 0.2%
95.9 0.6%
73.6 0.9%
48.0
53.4 + 5.2%
EWC
Joint
66.4 1.4%
69.6 1 1.6%
55.4
42.9 + 0.4%
58.6 + 1.1%
No Comp.
66.0 1.1%
1.26 68.8 1.1%
0.7%
68.8 = 0.4%
Dyn. + Comp.
97.3 = 0.2%
9.44
+.70%
73.0 0.4%
69.4 0.4%
Compositional
96.5 0.2%
95.9
74.5 +
54.8 1.2%
68.9 + 0.9%
NFT
Joint
67.4 1.4%
1.9%
65.1
43.9 : 0.6%
63.1 = 0.9%
No Comp.
64.4 1.1%
67.0 = 1.3%
49.1
36.6 1 0.6%
68.9 + 1.0%
Dyn. + Comp.
99.1 0.0%
97.3 + 0.3%
78.3 0.4%
78.4 + 0.3%
71.0 + 0.4%
FM
Compositional
84.1 + 0.8%
86.3 = 1.3%
80.1 + 0.3%
48.8 = 1.6%
63.0 = 3.3%
Soft ordering
1.5
1.5
1.5
Forward
D+C
NC
D+C
C
1
NC
D+C
J
NC
Final
Soft gating
1.
1.5
1.5
D+C
NC
D+C
NC
D+C
NC
(a) ER
(b) EWC
(c) NFT
Figure 1: Average gain w.r.t no-components NFT across tasks and data sets immediately after training on each task (forward) and after all tasks had been trained (final), using soft ordering (top) and soft gating (bottom). Algorithms within our framework (C and D+C) outperformed baselines. Gaps between forward and final performance indicate that our framework exhibits less forgetting.
1.0
8.8
3.8
54
0.4
04
200
450
$50
420
602
200
450
(d) ER No Components Figure 2: Learning curves averaged across MNIST and Fashion using ER and soft ordering. Each curve shows single task trained for 100 epochs and continually evaluated during and after training. Algorithms under our framework displayed no forgetting. For ER dynamic compositional, as more tasks were seen and accommodated, assimilation performance of later tasks improved. Joint and no-components versions dropped performance of early tasks during the learning of later tasks.
(a) ER Dyn. Comp.
(b) FR Compositional
(c) FR Joint
the agent to be capable of detecting when new components were needed. This added flexibility enables our learners to handle more varied tasks, where new problems may not be solved without substantially new knowledge. Algorithms with adaptation outperformed the ablated compositional FM agent, showing that it is necessary accommodate new knowledge into the set of components in order to handle a diversity of tasks. When FM was allowed to dynamically add new components (keeping old ones fixed), it yielded the best performance on MNIST and Fashion by adding far more components than methods with adaptation, as we show in Appendix F as well as on CIFAR.
To study how flexibly our agents learn new tasks and how stably they retain knowledge about earlier tasks, Figure shows accuracy gains immediately after each task was learned (forward) and af- ter all tasks had been learned (final), w.r.t. no-components NFT (final). Compositional learners with no dynamically added components struggled to match the forward performance of joint baselines, indicating that learning the ordering over existing layers during much of training is less flexible than modifying the layers themselves, as expected. However, the added stability dramatically decreased forgetting w.n.t joint methods. The dynamic addition of new layers yielded substantial improve- ments in the forward stage, while still reducing catastrophic forgetting w.r.t. baselines. Figure 2 shows the learning curves of MNIST and Fashion tasks using ER, the best adaptation method. Per- formance jumps in 100-epoch intervals show adaptation steps incorporating knowledge about the current task into the existing components without noticeably impacting earlier tasks' performance. Compositional and dynamic compositional ER exhibit almost performance drop after training
7

---

Published as a conference paper at ICLR 2021
Table Base
Average final accuracy across all tasks using soft gating. Standard errors after the +. Algorithm
MNIST
Fashion
CIFAR
Omniglot
Dyn. Comp.
98.2 + 0.1%
97.1 + 0.4%
74.9 4 0.3%
73.7 + 0.3%
Compositional
98.0 + 0.2%
97.0 + 0.4%
75.9 + 0.4%
73.9 + 0.3%
ER
Joint
93.8 + 0.3%
94.6 0.7%
72.0 1.000.40 0.4%
72.6 0.2%
No Comp.
01.200.0 0.3%
-..6%
51.6 0.6%
43.2 4.2%
Dyn. + Comp.
98.2 0.1%
0.4%%
76.6 0.5%
0.44%
Compositional
98.0 + 0.2%
97.0 0.4%
76.9 : 0.3%
EWC
Joint
68.6 + 0.9%
49.9 1.1%
No Comp.
66.0 1.1%
68.8 1.11
36.0
0.4%
Dyn. + Comp.
98.2 -: 0.1%
97.1 + 0.4%
0.7%
69.1 0.9%
Compositional
0.2%
96.9 0.5%
68.2 0.5%
72.1 = 0.3%
NFT
Joint
67.3 1.7%
66.4 1.9%
$1.0 0.8%
65.8 + 1.3%
No Comp. Dyn. Comp.
64.4 1.1%
+ 1.3%
36.6 + 0.6%
68.9 + 1.0%
98.4 0.1%
97.0 : 0.4%
77.2 = 0.3%
74.0 + 0.4%
FM
Compositional
94.8 + 0.4%
96.3 - 0.4%
77.2 = 0.3%
74.1 = 0.3%
Table 5: Average final accuracy across tasks on the Combined data set. Each column shows accuracy on the subset of tasks from each given data set, as labeled. Standard errors after +.
Base
Algorithm
All data sets
MNIST
Fashion
CUB
Dyn. + Comp.
86.5 + 1.8%
99.5 + 0.0%
98.0 + 0.3%
74.2 + 2.0%
Compositional
82.1 2.5%
99.5 0.0%
97.8 + 0.3%
65.5 + 2.4%
ER
Joint No Comp.
72.8 + 4.1%
98.9 + 0.3%
97.0 0.7%
47.6 + 6.2%
47.4 4.5%
91.8 + 1.3%
2.5%
7.1 0.4%
on a task, whereas accuracy for the joint and no-components versions diminishes as the agent learns seen by dynamic ER, the existing components
subsequent tasks. Most notably, more tasks became better able to assimilate new tasks, shown by the trend of increasing performance as the number of tasks increases. This suggests that the later tasks' accommodation stage can successfully determine which new knowledge should be incorporated into existing components (enabling them to better generalize across tasks), and which must be incorporated into new component.
In Appendix F we show that our methods do not forget early tasks, and outperform baselines even in small data settings. We also found that most components learned by our methods are reused by multiple tasks, as desired. Morcover, analysis of various accommodation schedules revealed that infrequent adaptation leads best results, informing future algorithm design choices.
5.2.3 DEEP COMPOSITIONAL LEARNING WITH SOFT GATING
Finally, we tested our algorithms when the compositional structures were given by soft gating net. Table 4 shows substantial improvement of compositional algorithms w.r.t. baselines. We hypothesized that the gating net granted our assimilation step more flexibility, which is confirmed in Figure Fioure[[totttom): (bottom): the forward accuracy of compositional methods was nearly identical to that of jointly trained and no-components versions. This added flexibility enabled our simplest version of a compositional aaaaalaaggooo algorithm, FM, to perform better than the full versions of our algorithm the CIFAR data set with convolutional gating nets, showing that even the components initialized with only a few tasks are sufficient for top lifelong learning performance. We attempted to run experiments with this method on the CUB data set, but found that all algorithms were incapable of generalizing to test data. This is consistent with findings in prior work, which showed that gating nets require vast amounts of data, unavailable in CUB (Rosenbaum et al. 2018 Kirsch et al. 2018).
5.2.4 DEEP COMPOSITIONAL LEARNING OF SEQUENCES OF DIVERSE TASKS
One of the key advantages of learning compositional structures is that they enable learning a more diverse set of tasks, by recombining components in novel ways to solve each problem. In this setting, non-compositional structures struggle to capture the diversity of the tasks in single monolithic architecture. To verify that this is indeed the case, we created a novel data set that combines the MNIST, Fashion, and CUB data sets into single Combined data set of T 40 tasks. We trained all our instantiations and baselines with soft layer ordering, using the same architecture as used for CUB in Section 5.2.2 Agents were given indication that each task came from different data
8

---

Published as a conference paper at ICLR 2021
intensity
intensity
4
4
4
w
4
4
4
4
4
0f
44
4 depth
4
4
4
4
4
4f
4
4
4
4
4
4
4
4
4
4
4
4
4
4
w
Task
Task 10
Figure 3: Generated MNIST "4" digits on the last two tasks seen by compositional ER with soft ordering, varying the intensity with which single specific component is selected. The learned component performs a functional primitive: the more the component is used (left to right on each row), the thinner the lines of the digit become. The magnitude of this effect decreases with depth (moving from top to bottom), with the digit completely disappearing at the earliest layers, but only becoming slightly sharper at the deepest layers. This effect is consistent across both tasks.
set, and they were all trained following the exact same setup of Section 5.2.2 Table 5 summarizes the results for ER-based algorithms, showing that our method greatly outperforms the baselines. In particular, ER with no components was completely incapable of learning the CUB tasks, showing that compositional architectures are required to handle this more complex setting. Even jointly training the components and structures performed poorly. Algorithms under our framework instead performed remarkably well, especially the complete version with dynamically added components. The remaining instantiations and baselines exhibited a similar behavior (see Appendix G).
5.3 VISUALIZATION OF THE LEARNED COMPONENTS
We now visually inspect the components learned by our framework to verify that they are indeed self-contained and reusable. Similar Meyerson Miikkulainen (2018), we created T 10 generative tasks, where each pixel in an image of the digit "4" constitutes one data point, using the coordinates as features and the pixel intensity as the label. We trained a soft ordering net with = components via compositional ER, and shared the input and output transformations across tasks to ensure the only differences across task models are due to the structure s(t) of each task. Varying the intensity ( with which component is selected at various depths gives information about the effect of the component in different contexts. Figure 3 shows generated digits as the intensity of component i 0 varies different depths, revealing that the discovered component learned to vary the thickness of the digit regardless of the task at hand, with a more extreme effect at the initial layers. Additional details and more comprehensive visualizations are included in Appendix
CONCLUSION
We presented general framework for learning compositional structures in lifelong learning set- ting. The key piece of our framework is the separation of the learning process into two stages: assimilation of new problems with existing components, and accommodation of newly discovered knowledge into the set of components. These stages have connections to Piagetian theory of devel- opment, opening the door for future investigations that bridge between lifelong machine learning and developmental psychology. We showed the flexibility of our framework by capturing nine different concrete algorithms within our framework, and demonstrated empirically in an extensive evaluation that these algorithms are stronger lifelong learners than existing approaches. More concretely, we demonstrated that both learning traditional monolithic architectures and naively training composi- tional structures via existing methods lead to substantially degraded performance. Our framework is simple conceptually, easy to combine with existing continual or compositional learning techniques, and effecctive in trading off the ficxibility and stability required for lifelong learning.
In this work, we showed the potential of compositional structures to enable strong lifelong learning. One major line of open work remains properly understanding how to measure the quality of the ob- tained compositional solutions, especially in settings without obvious decompositions, like those we consider in Section 5.2 While visualizations in Section 5.3 and results in Table F.4 suggest that our method obtains reusable components, we currently lack proper metric to assess the degree to which the learned structures are compositional. We leave this question open for future investigation.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like to thank Seungwon Lee, Boyu Wang. and Oleh Rybkin for their feedback on various versions of this draft. We would also like to thank the anonymous reviewers for their valuable feed- back and suggestions. The research presented in this paper was partially supported by the DARPA Lifelong Learning Machines program under grant FA8750-18-2-0111 the DARPA SAIL-ONN pro- gram under contract HR001 120000000 and the Army Research Office under MURI grant W911NF 20-1-0080. The views and conclusions in this paper are those of the authors and should not be in- terpreted as representing the official policies, either expressed or implied, of the Defense Advanced Research Projects Agency, the Army Research Office, or the U.S. Government. The U.S. Govern- ment is authorized to reproduce and distribute reprints for Govermment purposes notwithstanding any copyright notation herein.
REFERENCES
Alessandro Achille, Tom Eccles, Loic Matthey, Chris Burgess, Nicholas Watters, Alexander Lerch- ner, and Irina Higgins. Life-long disentangled representation learning with cross-domain latent homologies. In Advances in Neural Information Processing Systems 31 (NewrIPS-18), pp. 9873- 9883, 2018.
Ferran Alet, Tomas Lozano-Perr Perezz and Leslie P Kaelbling. Modular meta-learning, In Proceedings of the 2nd Conference on Robot Learning (CoRL-19), pp.
2018.
Matko Bošnjak, Tim Rocktäschel, Jason Naradowsky, and Sebastian Riedel. Programming with a differentiable forth interpreter. In Proceedings of the 34th International Conference on Machine Learning (ICML-17), pp. 547-556, 2017.
Rudy Bunel, Matthew Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Leveraging grammar and reinforcement learning for neural program synthesis. In 6th International Confer- ence on Learning Representations (ICLR-18). 18), 2018.
Jonathon Cai, Richard Shin, and Dawn Song. Making neural programming architectures generalize via recursion. In 5th International Conference Learning Representations (ICLR- 17), 2017.
Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas I.. Griffiths. Automatically compos- ing representation maanformations as a means for generalization. In 7th International Conference on Learning Representations (ICLR-19) 2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey.
Journal of Machine Learning Research, 20(55):1-21 2019.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. PathNet: Evolution channels gradient descent in super neural networks. arXiv preprint 1:Xiv:1701.08755 2017.
Alexander L Gaunt, Mare Brockschmidt, Nate Kushman, and Daniel Tarlow. Differentiable pro- grams with neural libraries. In Proceedings of the 34th International Conference on Machine Learning (ICML-17), pp. 1213- 2017.
Aidan N Gomez, Ivan Zhang, Siddhartha Rao Kamalakara, Divyam Madaan, Kevin Swersky, Yarin Gal, and Geoffrey Hinton. Learning sparse networks using targeted dropout. arXiv preprint 1905 2019.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:/207.055: 2012.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV-17), pp. 804-813, 2017.
David lsele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI-18), pp. 3302-3309, 2018.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reason- ing. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV-17),
10

---

Published as a conference paper at ICLR 2021
pp. 29889-998,, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinee et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 11<(13):521-3322, 2017.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neural computation. In Advances in Neural Information Processing Systems 31 (NeurIPS-18) pp. 2408- 2418, 2018.
Abhishek Kumar and Hal Daumé III. Learning task grouping and overlap in multi-task learning. In Proceedings of the 29th International Coference on International Conference on Machine Learn- ing (ICML-12), pp. 1723-1730, 2012.
Seungwon Lee, James Stokes, and Eric Faton. Learning shared knowledge for deep lifelong learning using deconvolutional networks. In Proceedings of the 28th International Joins Conference on Artificial Intelligence (ICCAI-19, pp. 2837-2844, 2019.
Xilai Li. Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the 36th International Conference Machine Learning (ICML-19), pp- 3925-3934, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligncce (TPAMI), 40(12):2935-29947 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30 (NeurIPS-17), pp. 6467-6476, 2017.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp. 109-165. Elsevier, 1989.
Elliot Meyerson and Risto Mikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. In 6th International Conference on Learning Representations (ICLR- 18), 2018.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based RL. In 7th International Conference Learning Repre- sentations (ICLR-19), 2019.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learn- ing. In 6th International Conference on Learning Representations (ICLR-18) 2018.
Vardaan Pahuja, Jie Fu, Sarath Chandar, and Christopher Pal. Structure learning for neural module networks. In Proceedings of the Beyond Vision and LANguage: inTEgrating Real-world kNowl- edge (LANTERN), pp. 1-10. Association for Computational Linguistics. 2019.
Jean Piaget. Piaget's theory. In Bärbel Inhelder, Harold H. Chipman, and Charles Zwingmann (eds.), Piaget and His School: A Reader in Developmental Psychology, pp. 11-23. Springer, Berlin, Heidelberg, 1976.
Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Ran- dom path selection for incremental learning. In Advances in Neural Information Processing Sys- tems 32 (NewrIPS-19), pp. 12669-12679, 2019.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee whye Tch, and Raia Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems 32 (Newr/PS-19), pp. 7647-7657, 2019.
Scott Reed and Nando de Freitas. Neural gegraammer--nntepreeten In 4th International Conference on Learning Representations (ICLR-16), 2016.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured Laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems 31 (NeurIPS-18) pp. 3738-3748, 2018.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In 6th International Conference on Learning Representations (1CLR-18), 2018.
11

---

Published as a conference paper at ICLR 2021
Paul Ruvolo and Eric Eaton. ELLA:
An efficient lifelong learning algorithm. In Proceedings the 30th International Conference on Machine Learning (ICML-13), pp. 507-515, 2013.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expert layer. In 5th International Conference on Learning Representations (CC.R-I7). 2017.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems 30 (NeurIPS-17), pp. 2990-2999, 2017.
Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning as program synthesis. In Advances in Neural Information Processing Systems 31 (NeurIPS-I8), pp. 8687-8698, 2018.
Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural
task programming: Learning to generalize across hierarchical tasks. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA-18), pp. 3795-3802, 2018.
JacHong Yoon, Jeongtae Lee, Eunho Yang, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations (ICLR-18), 2018.
Wojeiech Zaremba, Tomas Mikolov, Armand Joulin, and Rob Fergus. Learning simple algorithms from examples. Proceedings of the 33rd International Conference on Machine Learning (ICML-16), pp. 421-429, 2016.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning (ICML-17), pp. 3987-
3995, 2017.
12

---

Published as a conference paper at ICLR 2021
APPENDICES TO
" "LIFELONG LEARNING OF COMPOSITIONAL STRUCTURES"
by Jorge A. Mendez and Eric Eaton
A DEPICTION OF OUR COMPOSITIONAL LIFELONG LEARNING FRAMEWORK
ceneral-puppose puppsee framework for lifelong learning of com- positional structures. Figure A. illustrates our proposed framework, split into four learning stages: 1) initialization of components, 2) assimilation of new tasks with existing components, 3) adaptation of existing components with new knowledge, and 4) expansion of the set of components.
Section the main paper presented our general
previous tasks
current task
future tasks
tasks
2 t-1
2
training data
1. Initialize components
learned task model (()
3. Improve old components
existing components to solve new tasks
4. Add new components
previously learned components M
Lifelong Learning System
Figure A.1: 1: Our framework for lifelong compositional learning initializes of components by training on small set tasks (1). Each new task is learned by composing the relevant compo- nents (2). Subsequently, the agent improves imperfect components with newly discovered knowl- edge (3), and adds any new components that were discovered during training on the current task (4).
B FRAMEWORK INSTANTIATIONS
In this section, we describe in more detail the specific implementations used in the experiments of Section 5 in the main paper. To simplify evaluation, we fixed structUpdates, adaptFreq. and compUpdates such that the lcarning process would be split into multiple epochs of assimilation followed by single final epoch adaptation. This is the most extreme separation of the learning into assimilation and accommodation:
knowledge is accommodated into existing components until after assimilation has finished. We study the effects this choice Appendix
Algorithms B.5 1.100.ssmmmi summarize the implementations of all instantiations of our framework used in and we leave blank lines
our experiments. Shared subroutines are included as Algorithms/B.. in compositional methods to highlight missing steps from their dynamic compositional counter-
13

---

Published as a conference paper at ICLR 2021
parts. The learner first initializes the components by jointly training the first Timit tasks it en- counters. At every subsequent time t, during the first (numEpochs - epochs, the agent assimilates the new task by training the task-specific structure parameters vp(+) via backpropagation, learning how to combine existing components for task T(t), For dynamic + compositional methods, this as- similation step is done with component dropout, and simultaneously optimizes the parameters of the newly added component Φ The adaptation step varies according to the base lifelong learning method, applying techniques for avoiding forgetting to the whole set of component parameters & for one epoch. This step is done via component dropout for dynamic- + compositional methods. Finally, dynamic compositional methods discard the fresh component if it does not improve performance by more than threshold 7 the current task ((1), and otherwise keep it for future training.
Algorithm B.1 Initialization
A Algorithm B.2 Expansion
1: (1)) + randommnittializationn
1: di f accuracy wwwp:/ee..wlaailiaiaaaa .validation)
2: init_buft +- init.buff u 7
hideComponent (A:
3: t = Timit -1
d2 + accuracy(T(t) validattoo) )
4:
for i 1,..., nunEpochs
4: recoverComponent
5: for i, t init_buff
5: if #L-02 <
validation error check
6:
Φ - V
ag discardComponent((
7: end for
7: k + k ++
8: end for
backprop on components
end if
9:
end if
Algorithm B.3 Assimilation (Comp.)
Algorithm B.4 Assimilation (Dyn. + Comp.)
randomVVecoor) 1: 4 ---
D new comp.
(b())
+k+lk f-1
1: for -,..., numEpochs - 2: for 4- (+), train -iraiin
3: for i == 1...., numEpochs -
4: for +- -()
train
3: 26(1) t- ab(0) - n/ si C( (x()
5:
21(t) + --blt) n win C (F((())
6:
Φ + Φx+1 Von
C ((())()))
hideComponent(A-
(((1) < wb(t) v paD (((()(
recoverComponent +1) 10: end for
4: end for
5: end for
b backprop on structure
11: end for
component dropout
Algorithm B.5 Compositional ER
Algorithm B.6 Dynamic + Compositional ER
while ( < getTask()
while T()) <- getTask()
2:
if't < Tinit
2:
if f < Tinit
3:
Call Algorithm B.I
initialization
3: Call Algorithm
B.I
> initialization
4:
else
4:
else
5:
Call Algorithm B.3
assimilation
5:
Call Algorithm
B.4
assimilation
6:
for ī, + (1, T 1 train) U buffer
for x <- (1, T ( train) buffer
7:
Φ -- 4 - - c10
7:
< + v c()) (f (z))
hideComponent (k
$ V Ct) (z))))
10:
recoverComponent(/ end for
8: end for
b adaptation
11:
adaptation
12: Call Algorithm B.2
expansion
9: end if
13:
end
10: buffer(th it] sample(T()) train, nm)
14:
buffer(t] +- sample(T()) wet.ttrain........ n.,)
11: end while
15:
end while
14

---

Published as a conference paper at ICLR 2021
Algorithm B.7 Compositional EWC
Algorithm B.8 Dynamic + Compositional EWC
1:
while (()) + getTask()
while T < getTask()
2:
if < Tinit
if f < Tinit
3:
Call Algorithm B.1
> initialization
3: Call Algorithm B.1
b initialization
4:
else
else
5:
Call Algorithm
B.3
D assimilation
5:
Call Algorithm B.4
assimilation
6:
for x f- T train
for x T trainn
7:
A + Σ (((()))))
A < Σi s--) a (0066
8:
g k (f + B)
g ( (x))))) X(A B)
9:
Φ < + 119
9:
+ t $ 119
10:
hideComponent +1)
+ Σ (()) (b(
11:
12:
g + V a C (f (x)) + A(A B)
13:
+ 71
14:
recoverComponent// (++1)
end for
b adaptation
15:
end for
adaptation
16:
Call Algorithm
B.2
expansion
11: end if
17:
end
12: ( (t)) + KFaaCTW..raiinn .rrinn, ))
18:
( b(t) KFAC(T (4) www.ii, +)
13: B B -
19:
B f- B (b(())
14: end while
20:
end while
Algorithm B.9 Compositional NFT
Algorithm B.10 Dynamic + Compositional NFT
1:
while ( getTask()
1:
while T -getTask()
2:
if t't < Timit
Tinit
3:
Call Algorithm
B.I
initialization
3:
Call Algorithm B.1
initialization
4:
else
4:
else
5:
Call Algorithm B.3
assimilation
5:
Call Algorithm B.4
assimilation
6:
for T train
6:
for <- ( train
7:
+ - - v (100
7:
1 - 1 v, ()
8:
hideComponent
9:
$ - + C (1 (x)
10:
recoverComponent (k 11: end for
8: end for
> adaptation
adaptation
12: Call Algorithm B.2
b expansion
9: end if
13: end if
10: end while
14: end while
C COMPUTATIONAL COMPLEXITY DERIVATION
We now derive asymptotic bounds for the computational complexity of all baselines and instantia- tions of our framework presented in Section 4.3 in the main paper. We assume the network archi- tecture uses fully connected layers, and soft layer ordering for compositional structures. Extending these results to convolutional layers and soft gating is straightforward.
single forward and backward pass through standard fully connected layer of inputs and outputs requires Oiio) computations, and is additive across layers. Assuming binary classification net, the no-components architecture contains one input layer g(t) with d inputs and d outputs, k layers with d inputs and d outputs, and one output layer ((t) with d inputs and one output. Training such net in the standard single-task setting then requires O (dd Zh + d) computations per input point. For a full epoch of training on a data set with n data points, the training cost would then be O Ond dk + d)). This is exactly the computational cost no-components NFT, since it ignores any information from past tasks during training, and leverages only the initialization of parameters.
15

---

Published as a conference paper at ICLR 2021
On the other hand, soft layer ordering net evaluates all k layers of size d x d every one of the k depths in the network, resulting in cost of (2) for those layers. This results in an overall cost per epoch of (ndd + d)) for single-task training, and therefore also for joint NET training. Since compositional methods do not use information from earlier tasks during assimilation, because they only train the task task-specific structure g(t) during this stage, then the cost per epoch of assimilation is also (nd((k² + d)). Dynamic + compositional methods can at most contain T components if they add one new component for every seen task. This leads to a cost of ( for the shared layers, and an overall cost per epoch of assimilation of O (nd dkTT + d)).
Kroneceer-factored EWC requires computing two O d) matrices, a and b(9, for every ob- served task. At each training iteration, EWC modifies the gradient of component i by adding E a (1(() a Φb; ())(() where Φ; (t) are the parameters of component i obtained after training on task (+)) While the second term of this
can be pre-computed and stored in mem- ory, it is not possible to pre-compute the first term. Theoretically, one can apply Kronecker product properties to store a (prohibitively large) O ( x d²) matrix and avoid computing the per-task sum, but practical implementations avoid this and instead compute the sum for every task, at cost of (TT)k) per mini-batch. With (n) mini-batches per epoch, we obtain an additional cost with respect to joint and no-components NFT of ((TT Note that this step is carried out after obtaining the gradients for each layer, and thus there is no additional k22 term for joint EWC.
Deriving the complexity bound of ER simply requires extending the size of the batch of data from n to (Tnm + 12) for replay buffer size of T'm per task.
To put the computational complexity of dynamic + compositional methods into perspective, we compute the number of components required to solve T tasks. We consider networks with hard layer ordering, and assume that all T tasks can be represented by different orders over the same set of components. Given network with depths and & components, it is possible to create k* different layer orderings. If all T tasks require different orderings, then we require at least k = VT components. Designing a lifelong learning algorithm that can provably attain this bound in the number of components, or any sublinear growth in T, remains an open problem.
For completeness, we note that the (very infrequent) adaptation steps for compositional methods incur the same computational cost as any epoch of joint methods. On the other hand, to obtain the cost of adaptation steps for dynamic + compositional methods, we need to replace k² terms in the expressions for joint methods by kT, again noting that this corresponds to the worst case, where the agent adds new component for every single task it encounters.
D EVALUATION ON A TOY COMPOSITIONAL DATA SET
The results of Section 5.2 in the main paper were obtained on a suite of data sets that does not explicitly require any compositional structure. This deliberate choice enabled us study the gen- crality of our framework, and we found that algorithms that instantiate it work well across data sets with a range of feature representations, relations across tasks, number of tasks, and sample sizes. In this section, we introduce a data set that explicitly assumes a compositional structure that intu- itively matches the assumptions of our soft layer ordering architecture, and we show that the results obtained for non-compositional data sets still hold for this class of problems.
We created the Objects data set with 48 classes, each corresponding to an object composed of a shape (circle, triangle, or square), color (orange, blue, pink, or green), and location (each of the for quadrants in the image). We generated n = 100 images of size 28 x 28 per class. We uniformly sampled the center of the object from [c, 3, ca + 3], [c, 3, Cy + 3). where ex and cy the centers of the quadrant for each class, respectively. The RGB values were uniformly sampled from [r 16, r 16), [g 16, g + 16]. [b 16, g 16), where r, 9, and b are the nominal RGB values for the color of the class. Finally, we uniformly sampled the size of the objects from [3. 7] pixels.
To test our approach in this setting, created a lifelong version of the Objects data set by randomly splitting the data into 16 three-way classification tasks. 50% of the instances for each class were used as training data, 20% validation data, and 30% as test data. We used soft ordering nets with A
16

---

Published as a conference paper at ICLR 2021
Table D.I: Average final accuracy across tasks on the compositional Objects data using soft layer ordering. Column labels indicate which component was held out for final tasks. Std. errors after +.
Base
Algorithm
Circle
Top-left
Orange
Random
Dyn. Comp.
93.4
85.9 + 1.0%
89.4 1.0%
91.8 + 0.6%
Compositional
92.2 - 0.9%
84.9 = 1.2%
88.7 + 1.2%
90.9 + 0.9%
ER
Joint
92.0 0.8%
83.5 1.0%
8
89.1 0.110.69 0.6%
No Comp.
91.2
88.4 0.8%
89.8 1.0%
Dyn. + Comp.
98.4
85.9 1.0%
=
91.6 0.7%
Compositional
92.0 1.1%
85.3 1.3:1.29 1.2%
1.2%
90.9
EWC
Joint
91.1 + 0.8%
82.4 1.3%
90.1
No Comp.
88.1 + 1.8%
81.0 + 1.5%
83.3 2.5%
86.3 + 2.1%
Dyn. Comp.
9.73+
86.3 + 1.0%
89.6 1.1%%
91.5 + 0.6%
Compositional
92.3 0.8%
85.7 + 1.1%
88.7
90.6 + 0.9%
NFT
Joint
90.6 -
81.8 + 1.2%
86.6 1 1.3%
88.4 1.2%
No Comp.
89.2 + 2.0%
77.5 + 1.9%
86.8 + 1.2%
85.7 + 1.5%
Dyn. Comp.
93.0 + 0.7%
86.0 + 1.0%
89.5 + 1.1%
91.4 + 0.5%
FM
Compositional
90.8 + 1.4%
83.8 +: 1.5%
88.1 + 1.2%
89.4 + 0.9%
components of 64 fully connected hidden units shared across tasks, and a linear input transformation g() trained for each task. All agents trained for 100 epochs per task using mini-batch of size 32, with compositional agents using 99 epochs for assimilation and single epoch for adaptation. The regularization hyper-parameter for EWC was set to À = IE - 3, and ER was given a a replay buffer of size 7mm. 5. We ran 50 trials of each experiment with different random seeds controlling class splits for each task, maining/validationn/ee splits for each class, and the ordering of tasks.
We evaluated all methods in four different settings. In the Random setting, classes were randomly split and ordered, matching the experimental setting of Section 5.2 The other, more challenging settings created by holding and location) or five (for shape) tasks, requiring the agents to adapt to never-seen components dy- namically. Results Table [aIle show each of our methods outperforms all baselines in all settings, showcasing the ability of our framework to discover the underlying compositional structures.
one shape, location, color only for the final four (for color
E EXPERIMENTAL SETTING
Below, we give additional details describing the experimental setting used in the main paper.
E. DATA SETS
The data used for linear experiments underwent the same processing and train/test split of Ruvolo & Eaton (0133). For MNIST and Fashion, we randomly sampled pairs of digits to act as the positive and negative classes in each task, and allowed digits be reused across tasks. For CUB and CIFAR, ten and five classes were used per task, respectively. without reusing of classes across different tasks. CUB images were cropped by the provided bounding boxes and resized to 224 224. For these four data sets, we used the standard train/test split, and further divided the training set into 80% for training and 20% for validation. Finally, for Omniglot, we used each alphabet as one task, and split the data into 80% for training, 10% for validation, and 10% for test, for each task. For each of the ten trials, we varied the random seed which controlled the tasks (whenever the tasks were not fixed by definition). the random splits for seining/validatiionness. and the order in which the tasks were presented to the agent. Validation sets were only used by dynamic + compositional learners for selecting whether keep component. Details are summarized in Table R.2
E.2 NETWORK ARCHITECTURES
We used k= 4 components for all compositional algorithms with fixed k. This is the only architec- tural choice for linear models. Below, describe the architectures used for other experiments.
17

---

Published as a conference paper at ICLR 2021
Table E.2: Data set details summary,
FERA
Landmine
Schools
MNIST
Fashion
CUB
CIFAR
Omniglot
tasks
21
29
139
10
10
20
20
50
classes
2
2
2
10
5
14-55
features
100
9
27
784
784
512
32x32x3
105x 105
feat. extract.
PCA
ResNet-18
train
225-499
222-345
11-125
~9500
9500
~20
~200
224-880
val
2500
2500
~30
500
28-110
test
225-500
223 345
11-126
2000
2000
~150
500
28-110
Soft layer ordering We based our soft layer ordering architectures on those used by Meyerson & Miikkulainen (2018), whenever possible. For MNIST and Fashion, we used a random and fixed linear input transformation g()) for each task, and each component was a fully connected layer of 64 units. For CUB, all tasks shared fixed ResNet-18 pre-trained on ImageNe as an input transformation, followed by a task-specific input transformation g(4) given by a linear trained layer, and each component was a fully connected layer of 256 units. For CIFAR, there was no input transformation, and each component was a convolutional layer of 50 channels with and padding of pixel, followed by a max max-pooling layer of size 2 x 2. Finally, for Omniglot, there was also no input transformation, and each component was a convolutional layer of 53 channels with 3 x 3 kernels and no padding, followed by max- max-pooling of 2 x 2 patches. The input images to the convolutional nets CIFAR and Omniglot were padded with all-zero channels in order to match the number of channels required by all component layers (50 and 53, respectively). All component layers were followed by ReLU activation and a dropout layer with dropout probability = 0.5. The output of each network was a linear task-specific output transformation D(t) trained individually on each task. The architectures for jointly trained baselines were identical to these, and those for no-components basclines had the same layers but no mechanism to select the order of the layers.
3 kernels
Soft gating The soft gating architectures mimicked those of the soft layer ordering architectures closely, all having the same input and output transformations, as well as the same components. The only difference
in the structure architectures. For fully connected nets, each depth, the structure function s( was linear layer that took as input the previous depth's output and whose output was a soft selection over the component layers for the current depth. For convolutional nets, there was one gating net per task with the same architecture as the main network. The structure s(t) was computed by passing the previous depth's 's output in the main network through the remaining depths in the gating network (e.g., the output depth in the original network was passed through depths 3 and 4 in the gating network to compute the structure over modules at depth 3).
E.3 ALGORITHM DETAILS
All agents trained for 100 epochs on each task, with mini-batch of 32 samples. Compositional agents used the first 99 epochs solely for assimilation and the last epoch for adaptation. Dynamic + compositional agents followed this same process, but every assimilation step was done via com- ponent dropout; after the adaptation step, the agent kept the new component if its validation per- formance with the added component represented at least a 5% relative improvement over the per- formance without the additional component. Joint agents trained all components and the structure for the current task jointly during all 100 epochs, keeping the structure for the previous tasks fixed, while no-components agents trained the whole model every epoch.
ER-bd algorithms used a replay buffer of a single mini-batch per task. Similarly, EWC-based algorithms used single mini-batch to compute the approximate Fisher information matrix required for regularization, and used a fixed regularization parameter À = 10-3.
To ensure a fair comparison, all algorithms, including our baselines, used the same initialization procedure by training the first Timie =4 4 tasks jointly, in order to encourage the network to generalize across tasks. For soft ordering nets, the order of modules for the initial tasks was initialized a random one-hot vector for each task at each depth, ensuring that each component was selected at
The pre-trained Res is provided by PyTorch, and we followed the pre-processing recommended org wtt.ee/s//aaaaatbbbbeebeee wtw.//iiiiioooo/eeeeeeo/////e html
https::/ppytecch.orgg//
18

---

Published as a conference paper at ICLR 2021
1.00
0.30
3,0
Dyn Comt
Compostona
0.00
pest
No Choyp
Comp peoerraraa
Dyn. Compp
Comprsttinn
Compostion
0.00
cenp
puut Compostinn
Dys Camp
Goop Cani
Gengostonn
Dyn Comtr
Cenpootion
Dyn Cempusttmm
chip
pove
Compl
Cany poor
No Cenve
povt
Comprotinn comprrttinn
Dat
Dyn.
Dyn
MNIST
(b) Fashion
(c) CUB
0.8
Accrog
Forward Final
Byn conpp
0.0
Nr. Aive
We comp Compostinn
Dyn Componstion
Ne comp Compottoon
Just
Dye. Comp
boet Comp
Camprotton
Compa
(d)
CIFAR
Omniglot
Figure F.2: Soft layer ordering accuracy. Compositional agents outperformed baselines in most data
sets for every adaptation method. Dyn. comp. agents further improved performance, leading to
our methods being strongest. Error bars denote standard errors.
FR Dyn. + Comp.
Compositional
ER Joint
ER No Components
0.5
0.5
MNIST
200
400
200
030
220
810
202
002
14
1.0
0.8
0.5
0.3
0.0
Fashion
0.5
CUB
0.4
0.4
1.00
1300
1000
2000
0.0
CIFAR
0.2
503
1936
1500
503
1350
1503
5000
1000
1500
0.0
303
1030
1536
0.0
0.5
0.6
Omniglot
0.4
0.0
1000
2003
1003
4002
0.0
1000
2003 2010
1020
0.7
1030
2000
7000
7.0
7002
2000
3200
4200
spact
eacchu
Figure F.3: Smoothed learning curves with soft layer ordering using ER. Compositional methods did exhibit decaying performance of early tasks, while joint and no-components baselines did.
least once, and for soft gating nets, the gating nets were randomly initialized. The structures over initial tasks were kept fixed during training, modifying only the weights of the components.
F ADDITIONAL RESULTS FOR QUANTITATIVE EXPERIMENTS
We now present detailed results that expand upon those presented in Section 5.2 in the main paper.
For completeness, we include expanded results from Figures D and in the main paper, correspond- ing to soft layer ordering. Figure F2 is more detailed version of Figure I and shows the test accuracy immediately after each task was trained and after all tasks had been trained, separately for each data set. Compositional algorithms conforming to proposed framework achieve better
19

---

Published as a conference paper at ICLR 2021
1.0
1.0
Rouurcy Netartion 1ar2
1.00
0.9
0.95
0.90
0.85
0.80
0.5
Task
Task
Task
(a) ER Soff Ordering
(b) BWC Soft Ordering
(c) NFT-Soof Soft Ordering
1.00
1.0
accurok Mitentton re50
0.95
0.9
0.8
0.90
0.7
0.85
0.6
0.80
0.5
0.75
0
4
4
Task
Task
Task
(d) ER
Soft Gating
(e) EWC Soft Gating
(1) NFT
Soft Gating
Dyn. + Comp.
Compositional
Joint
No Comp.
Figure F4:: Catastrophic forgetting across data sets. Ratio of accuracy when a task was first trained to
when all tasks had been trained. For data sets with more than tasks, sampled ten interleaved tasks to match all the x-axes. Compositional algorithms had practically no forgetting, whereas jointly trained and no-components baselines forgot knowledge required solve earlier tasks.
Table F.3: Number of learned components. Standard errors reported after the +.
Structure
Base
MNIST
Fashion
CUB
CIFAR
Omniglot
ER
5.2 0.3
4.9 0.3
5.9 E.9.E 0.3
19.1 + 0.3
9.3 = 0.3
Soft ordering
EWC
0.3
4.7
5.8 0.2
19.6 1.6 0.2
00.10.33
NFT
0.2
4.8
6.1 0.3
17.7 0.3
1.70
FM
1.00
8.8
6.5
19.1 + 0.4
1.6
ER
4.0
0.1
0.1
0.1
Soft gating
EWC
4.1
4.0
4.8
7.4
NFT
4.1
4.1
4.1
7.2
FM
0.4
4.2
4.4
7.3
trade-off than others in flexibility and stability, leading to good adaptability to each task with little forgetting of previous tasks. Similarly, Figure F.3 learning curves similar to those in Figure [ in the main paper, for each data set. Baselines that train components and structures jointly all exhibit a decay in the performance of earlier tasks as learning of future tasks progresses, whereas methods conforming
framework do not. Results for soft gating nets display similar behavior.
The gap between the first and second bar for each algorithm Figure F22 is an indicator of the amount of catastrophic forgetting. However, it hides details of how forgetting affects each indi- vidual task. On the other hand, the decay rate of each task in Figure F.3 shows how each task is forgotten over time, but does measure quantitatively how much forgetting occurred. Based on prior work (Lee et al. 2019), we evaluated the ratio of performance after each task was trained to after all tasks had been trained metric knowledge retention. Results Figure [F.4] show that compositional methods exhibit substantially less catastrophic forgetting, particularly for the earlier tasks seen during training.
In our experiments, it was in many cases necessary to incorporate an expansion step in order for our algorithm to be sufficiently flexible to handle the stream incoming tasks. This expansion step enables our methods to dynamically add new components if the existing ones are insufficient to achieve good performance the new task. Table F3 shows the number of components learned by each dynamic algorithm using both soft ordering and soft gating, averaged across all ten trials. Notably, in the soft ordering case, in order for our methods to work on the CIFAR data set, they required learning almost component per task. This explains why compositional algorithms
20

---

Published as a conference paper at ICLR 2021
0.8
0.955
1925
Acturaay
1990
0.05
1875
0.80
1.50
0.2
0.75 16
data points
data points
ddata
(a) MNIST
(b) Fashion
(c) CUB
Dyn. + Comp.
Compositional
+
Joint
No Comp.
Figure F.5: Accuracy of ER-based methods with varying data sizes. Compositional methods per-
formed better even with extremely little data per task. Shaded area represents standard errors.
Table F.4: Number of tasks that reuse a component. task reuses a component its accuracy drops by more than 5% relative when the component is dropped. Standard errors reported after the +. Algorithm
Comp.
MNIST
Fashion
CUB
CIFAR
Omniglot
6.40 + 0.43
6.00 0.24
13.40 + 0.78
18.90 + 0.30
46.40 + 0.98
1
4.90 + 0.26
4.70 + 0.28
7.90 = 0.41
16.20 + 0.80
30.60 = 2.86
Compositional
4.10 + 0.26
4.10
5.90 0.57
11.90 + 1.14
18.80 3.76
3
3.00 + 0.32
2.60 +
3.20 + 0.49
5.70 + 0.97
10.90 + 2.17
4.70 0.388
5.10
9.80 0.98
13.30 1.27
21.90 + 1.82
1
3.60 0.25
3.90
6.20 0.56
6.20 + 0.61
12.30 + 0.68
2.80 + 0.28
3.30 0.28
4.40 = 0.62
4.00 0.35
9.20 = 0.61
1.90 + 0.26
2.10 + 0.22
2.70 0.20
3.10
7.40 + 0.47
4
3:00 0.32
6.50 + 0.49
1.80
5.00 0.51
6
1.50 0.16
4.20 = 0.46
Dyn. + Comp.
1.10 0.09
3.40 0.47
8
1.00 + 0.00
1.90 + 0.30
1.00 0.00
10
1.00 + 0.00
1.00 + 0.00
12
1.00 + 0.00
13
0.90 0.09
14
0.90 + 0.09
without dynamic component additions were incapable of performing well CIFAR. It is also worth noting that soft gating nets typically required adding fewer new components, which is to be expected, since the gating structure gives the learner substantially more flexibilityy Recall that, mentioned in Section 5.2.3) soft gating networks were unable to perform well on the CUB data set because of the small sample size, so the corresponding column omitted from the table.
One of the key aspects of lifelong learning is the ability to learn in the presence of little data for each task, using knowledge acquired from previous tasks to acquire better generalization for new tasks. To evaluate the sample efficiency of our algorithm, we varied the number of data points used for training for MNIST, Fashion, and CUB using the soft ordering structure and ER. We repeated the evaluation for 50 trials, each with different random seed controlling the selection of classes and samples for each task, and the order over tasks. Learners were trained for 1,000 epochs, with our compositional methods alternating nine epochs of assimilation and one epoch of adaptation. We for each function of the
used a batch size of b = 32, and limited the replay buffer size min (max (0oxx[(0.1 In..1n], 1),
data size Figure F.5 shows the learning accuracy for ER- ER-based algorithms number training points, revealing that compositional algorithms work better than baselines even in the presence of very little data.
Our approach was designed to discover a set of components that are reusable across multiple tasks. To verify that this effectively occurs, we evaluated how many tasks reuse each component. Taking the models pre-trained via compositional and dynamic + compositional ER with soft layer ordering, evaluated the accuracy of the model each task if any individual component removed
21

---

Published as a conference paper at ICLR 2021
0.98
0.9
0.9
Accurscy
0.94
000
0.92
0.7
0.7
50
100
adaptFieg
adaptfreg
adaptireg
(a) ER on MNIST
(b) EWC on MNIST
(c) NFT on MNIST
0.97
0.96
0.80
0.85
0.94
0.75
0.75
0.70
0.70
10
20
50
100
10
20
50
100
10
20
50
100
dddppffee reg
adaptiree Feg
adapt req
(d) ER on Fashion
(e) EWC on Fashion
(1) NFT on Fashion
4.75
0.60
0.55
0.5
0.45
2
50
100
20
50
100
0.4
20
50
100
adaptfrec
acaptfreg
adaptfreq
(g) ER on CUB
(h) EWC on CUB
(i) NET on CUB
Dyn. + Comp.
Compositional + Joint
- No Comp.
Figure F.6: Effect of the assimilation and accommodation schedule. Average accuracy across tasks w.r.t. the number of assimilation epochs between accommodation epochs. Broadly, methods under our framework performed better with a scheduled that favored stability, taking more assimilation steps before accommodating any new knowledge into the set of existing components.
from the model. We then considered task to reuse a given component if removing it caused a relative drop in accuracy of more than 5%. Table [F4 shows the number of tasks that reuse each component. Since there is no fixed ordering over components across trials, we sorted each trial's components in descending order of the number of tasks that reused each component. Moreover, for dynamic compositional ER, we only consider components that are created across all trials for given data set ensure all averages and algorithms, all & = 4 components available from initialization were used by multiple tasks. For the Omniglot data set, find that this behavior persists even for components that are dynamically added in the expansion step. However, this is not the case for the CIFAR data set, for which the first few dynamically added components are indeed reused by multiple tasks, but subsequent ones are used by a single task. This indicates that those components were added merely for increasing performance of that individual task, but found no reusable knowledge useful for future tasks.
statistically significant. We found that across all data sets
When designing algorithms under our framework, one needs to choose how to alternate the pro- cesses of assimilation and accommodation. In most experiments so far, we considered the simplest case, where adaptation is entirely carried out after assimilation is finished. However, it is possi- ble that other choices yield better results, enabling the learner incorporate knowledge about the current task that further enables it to assimilate it better. To study this question, we carried out ad- ditional experiments using ER variants on the MNIST, Fashion, and CUB data sets with soft layer ordering. Instead of executing the adaptation step only after completing assimilation, we alternated epochs of assimilation with epochs of adaptation with various frequencies. Results are displayed found that it was beneficial to carry adaptation steps infrequently,
in Figure [.6] Generally, with a clear increasing trend in performance as the learner took more assimilation steps before each adaptation step. For MNIST and Fashion, found that all choices of schedule led to improved per-
22

---

Published as a conference paper at ICLR 2021
Table G.5: Average final accuracy across tasks on the Combined data set. Each column shows accuracy on the subset of tasks from each given data set, as labeled. Standard errors after +.
Base
Algorithm
All data sets
MNIST
Fashion
CUB
Dyn. Comp.
1.8%
98.0 0.3%
74.2 + 2.0%
Compositional
82.1 - 2.5%
0.0%
97.8 - 0.3%
ER
Joint
72.8 4.1%
98.9
97.0 0.7%
6.2%
No Comp. Dyn. + Comp.
+..4%%
1.3%
83.5
7.1%
7..1
08.7
9..1
52.4 2.9%
Compositional
4.00%
9.0% 5.00%
96.1
EWC
Joint
85.1 +: 5.5%
+.8%
47.51..55
No Comp.
+.89%
52.9 + 1.6%
0.4%
Dyn. Comp.
7.2%%
99.1 + 0.3%
9.92+
53.3 2.8%
Compositional
7
98.5 + 0.5%
+
3.5%
NFT
Joint
52.7 4.9%%
85.5 + 4.9%
8.75
1.7%
No Comp.
3.7%%
61.3 3.8%
3.6%
8.7 0.5%
Dyn. Comp.
83.8 2.0%
99.6 6 + 0.0%
98.3 0.3%
68.7 :: 1.5%
FM
Compositional
74.6 + 3.1%
99.5 + 0.0%
98.1 + 0.3%
50.3 + 2.0%
formance over baselines, highlighting the benefits of splitting the learning process into assimilation and accommodation. For CUB, the results were more nuanced, with very fast accommodation rates achieving lower accuracy than the baselines. This is consistent with our results in Table 3 where compositional FM, equivalent to compositional ER with a schedule of infinite assimilation steps per accommodation step. performed nearly as well as compositional ER with a single adaptation epoch.
G COMPLETE RESULTS ON SEQUENCES OF DIVERSE TASKS
We now describe in more detail the experimental setting used obtain the results in Section 5.2.4 in the main paper, and provide the complete table of results using all our instantiations and basclines on the Combined data set.
We combined the 10 MNIST tasks, 10 Fashion tasks, and 20 CUB tasks into a single lifelong learn- ing data set with T 40 tasks, and trained all our methods on this new Combined data set. None of the methods were informed in any way that the tasks came from different data sets, and each learner was simply required to learn all tasks consecutively exactly as the remaining experiments. We used the soft layer ordering structure, with the architecture used for the CUB tasks described in Appendix B Only CUB images were processed with the pre-trained ResNet-18, whereas MNIST and Fashion images where fed directly to the task-specific input transformation {())
The results are summarized in Table G.5 As expected, our methods clearly outperformed all base- lines, by much wider margin than in the single-data-set settings of Section 5.2.2 in the main pa- per. In particular, no-components baselines (those with monolithic architectures) were completely incapable of learning to solve the CUB tasks. Even the jointly trained variants, which have com- positional structures but learn them naively with existing lifelong methods, failed drastically. Our methods were far better, especially when using ER as the base adaptation method.
Note that, in order to match the requirements of the CUB data set, the architecture we used gave MNIST and Fashion a higher capacity (layers of size 256 vs 64) and the ability to train the input transformation for each task individually (instead of keeping it fixed) compared to the architecture described in Appendix H This explains the higher performance of most methods those two data sets compared to the results in Table 3 the main paper.
H VISUALIZATION OF THE LEARNED COMPONENTS
The primary motivation for our framework was the creation of lifelong learning algorithms capable of discovering self-contained, reusable components, useful for solving variety of tasks. In this section, provide additional details about the visualization experiment of Section [33] as well as a more comprehensive study of various components and comparison the joint ER baseline.
23

---

Published as a conference paper at ICLR 2021
intensity
intensity
4
4 d depth
4
4
#
4
.
1
1
4
4
4
4
4
4
4
4
4
4
4
41
44
7
2
4
4
4
4
4
4
4
4
44
44
A
A4
A4
Task
9
Task
9
4
1
4
4
4
2
7
1
4 + depth
44
4
4
4
24
2
2
4
4
4
4
4
4
4
4
4
4
4
4
1
4
4
4
4
4
4
4
Task
10
Task 10
(a) ER Compositional
(b) ER Joint
Figure H.7: Visualization of reconstructed MNIST "4" digits on the last two tasks seen by the
compositional and joint variants of ER with soft layer ordering, varying the intensity with which component 0 is selected. Compositional ER learned a component that performs a functional primitive: the more intensely the component is selected (moving from left to right on each row), the thinner the lines of the digit become. The magnitude of this effect decreases with depth (moving from top to bottom), with the digit completely disappearing selected at the earliest layers, but only becoming slightly sharper with intensity at the deepest layers. This effect is consistent across both tasks. Joint ER did not exhibit this consistent behavior, with different effects observed at different depths and for the different tasks.
the component is more intensely
intensity
0
intensity
4
4
+
+
4
4
4
4
4
4
4
4
4
4
4
4
+
#
#
#
#
/
4 depth
4
4
4
4
4
4
4
4
4
4
14
4
4
4
4
K4
4
4
4
4
4
4
4
4
4
#4
A
A/f
4#
Task
9
Task
9
4
4
4
+
4
#
-f
-f
1
1
5/
54
24
24
c
J
J
4
4
4
4
4
4
/
4
depth
44
44
44
4
4
4
4
4
4
4
4
4
4
4
4
6
E
E
E
4
4
4
4
#
4
4
Task 10
Task 10
(a) ER Compositional
(b) ER Joint
Figure H.8: Visualization of reconstructed MNIST "4" digits, varying the intensity of component
i = 1. The component learned via compositional ER consistently decreases the length of the left side of the digit and increases that of the right side. Again, we were unable to detect any consistency in the effect of the component learned via joint ER.
We followed the visualization experiment of Meyerson & Miikkulainen (008)). where each task corresponded to a single image of the digit "4", and each pixel in the image constituted one data- point. The x, y coordinates of the pixel were used as features, and the pixel's intensity was the associated label. Pixel coordinates and intensities were normalized [o, 1]. All pixels in the image were treated as training data, since we were interested in understanding the learned representations, as opposed to generalizing
unseen data. Our network had = components shared across all tasks, and used soft layer ordering learn the structure (!) for each task. We used a linear input transformation layer g()) shared across all tasks, and shared sigmoid output transformation layer
24

---

Published as a conference paper at ICLR 2021
intensity
0
intensity
4
H
#
F
F
4
4
4
4
4
4
4
4
#
#
4 t depep
4
4
4
4
4
4
4
4
4
4
4
4
A4
L
7
4
4
4
4
4
4
4
4
4
4
44
#4
A
14
Task
9
Task
9
dedephh 1
4
5/4
Y
Y
Y
4
4
4
V
6
6
54
44
4
4
4
4
4
4
4
A
A
P
F
F
4
4
4
44
E
E
E
E/
E
E
4/4
4
4
4
4
4
4f
4
4 +
4
4
E
E
4
4
44
4
4
4
4
4
4
4
4
4
4
Task
10
Task 10
(a) ER Compositional
(b) ER Joint
Figure H.9: Visualization of reconstructed MNIST "4" digits, varying the intensity of component
i= 2. As the intensity of the component leamned via compositional ER increased, the digit changed
from very sharp to very smooth. Joint ER again did not exhibit any consistent behavior.
intensity
intensity
4 f depth +1
4
4
4
#
#
*
+
4
4
4
4
1
4
4
4
4
4
4
4
4
44
14
#4
24
V.4
4
4
4
4
4
4
#4
#4
Task
9
Task
9
5
44
4/4
L
4
*
,
4 depth
4
44
5/
5
5/
#
4
44
E4
54
E4
U
4
4
4
4
4
4/4
7/4
4
4
E
E
4
4
4
4
4
4
#
4
4
4
Task 10
Task 10
(a) ER Compositional
(b) ER Joint
Figure 1010: Visualization of reconstructed MNIST "4" digits, varying the intensity of component
i = 3. This component also interpolates between sharper and smoother digits, while also rotating
the digit. There was consistency in the behavior of the component learned by Joint ER.
D(O), Sharing the input and output transformations across tasks ensures that the only differences across the models of the different tasks are due the structure of each task over the components. We trained the network to minimize the binary cross-entropy loss on T = 10 tasks for 1 1,000 epochs via the compositional and jointly trained versions of ER with replay buffer and batch size of 32 pixel instances, updating the components of the compositional version every 100 epochs.
To evaluate the ability of compositional ER to capture reusable functional primitives, we observed the reconstructed images output by our network as we varied the intensity v with which one specific component is chosen at different depths j in the network. We focused our evaluation on the last two tasks seen by the learner, in order to disregard the effects of catastrophic forgetting, which rendered the visualizations of the outputs of the joint ER baseline incomprehensible for earlier tasks. Figures HL. these reconstructions as the intensity of each component individually varies at different depths. The components learned with compositional ER learned to produce effects on the digits consistent across tasks, with more extreme effects at the initial layers. In contrast, joint ER learned components whose effects are different for different tasks and at different depths.
25


Reviews:
[{"review_id": "yOmre4iEObR", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe paper attempts to solve lifelong/continual learning (CL) by building reusable components and learning both the way of combining them and the components themselves. To do this, the authors present a framework of algorithms which is based on three abstract elements which are iterated:\na. Updating the components itself\nb. Updating the way the components are combined (structure) for a given task\nc. Adding new components\n\nThe framework is instantiated as a concrete algorithm in several different ways whose performance is evaluated through extensive experiments.\n\n### Concern\nMy biggest concern regarding the work is lack of comparison with previous CL literature. Authors do a good work listing previous contributions in CL (in Sec. 2), and provide two claims, which, as implied, would make the previous models significantly different from their work:\nA: \"it is unclear what the reusability of these parameters [of previous works] mean\"\nB: \"the way in which parameters are reused is hard-coded into the architecture design\"\n\nA. Many of the previous works (in particular Achille et al. 2018 and Rao et al. 2019) contain a shared sub-model which maps data into a representation which is further processed to predict task-specific labels. The ability to extract structure from data coming from various tasks is, on the level of the objective, the same \"reusable knowledge\" that the authors' work attempts to gather. One could argue that the way previous work learn this knowledge could be improved: to do this, one should compare the new model with the previous ones using the existing objectives which try to assess the level of forward/backward transfer (which, arguably, is an established way of measuring \"reusability of the knowledge\") or by proposing a different metric for quantifying the \"discovery of reusable knowledge\". The work under review does neither.\nB. Authors claim that hard-coded network architecture is an unsatisfactory element of previous work, in particular because it makes it harder \"to learn tasks with high degree of variability\". Deciding which part of the NN-based algorithm is \"architecture\" is fuzzy at best. I believe that the authors' work (in particular \"soft layer ordering\" compositional structure) can also be interpreted as learning a neural network with a fixed architecture, sharing many layers, and placing non-linearities every second layer (see * below). I would like to ask the authors to clarify what formally they mean by \"hard-coding architecture\". Furthermore, the hypothesis of \"methods with hard-coded architecture have a disadvantage at 'hard' tasks\" needs experimental verification, which I found lacking in the paper.\n\n### Smaller statements\n1. The paper is well-written and relatively easy to understand.\n2. The work seems very related to the domain of Neural Architecture Search: I believe a reference to this line of work is warranted.\n3. If I understood correctly, the method is trained on batches of 4 tasks (or are 4 tasks present only in the first batch?) at a time. This is a significant simplification of the classical CL setting where only one task can be observed at a time. I believe a big part of the CL problem is that one never has an access to the level of \"task variability\" and thus needs to trade-off overfitting and underfitting without having access to the meta-task.\n4. The work is missing a clear message: the presented framework is a generalization of some previous methods, but it's far from clear why the algorithms fitting to the framework would be beneficial over the others solving the same problem of CL nor what other benefit does thinking in terms of the proposed framework gives.\n\n### A question\nwhy $\\psi$ and $s_j$ sum to 1 in Sec. 4.1?\n\n### Review summary\nI find the overall method interesting and a potentially advantageous approach to CL. At the same time, despite extensive experiments and a detailed writeup (spilling, a bit unfairly, crucial parts of the work (like an algorithm diagram) into a 10+ page-long appendix), I feel that placing the work in the contemporary CL+NAS literature is inadequate, making it impossible to appreciate the benefits of the work.\n\n*\nLet $x \\in R^d$: a data point, $\\sigma$: a non-linear function, $A_{1, 2}$: activation matrix $\\in R^{d \\times d}$.\n\nA typical, 2-layer network could be described as:\n$$\nf(x) = \\sigma(A_2 \\sigma (A_2 x))\n$$\n\nYour soft layer ordering model is (dropping some indices):\n$$\nf(x) = \\sum_i \\psi_{i,1} m_i \\circ \\sum_j \\psi_{j, 2} m_j(x) =\n$$\n$$\n= \\sum_i \\psi_{1, i} \\sigma \\Big( \\phi_i^T \\times \\big(\\sum_j \\psi_{2, j} \\sigma(\\phi_j^T x)\\big)\\Big) =\n$$$$\n= \\psi_1 \\sigma(\\phi \\psi_2 \\sigma(\\phi x))))\n$$\nWhere $\\phi$ is of shape $k \\times d \\times d$ and $\\psi_1, \\psi_2 \\in R^k$\nIn other words, the proposed model behaves like a neural network with interleaving $d$- and $k$-units-wide hidden layers, with and without non-linearities and some extra weight sharing and broadcasting.", "extra_scores": {}, "extra_text": {}}, {"review_id": "p2aV2nYkdAM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses lifelong learning of compositional structures by proposing a general-purpose framework, which separates the learning process to two stages: combine existing components for assimilation; adapt existing components for accommodation and optionally add new components.\n\nThe strong point is that the approach is general for model architectures and can be combined with different catastrophic forgetting mechanisms. There are many evaluation datasets and settings. Also, it has visualization to analyze the learnt models. The weak point is that the arguments and evaluations do not support the claims very well.\n\nMy recommendation is that this paper is below the acceptance bar.\n\nThe reasons are mainly from concerns for compositional structure learning and catastrophic forgetting evaluation.\n1. The paper does not clearly explain why separating assimilation and accommodation leads to learning compositional structure and components from a machine learning perspective. Even an intuitive explanation or a concrete example may be more helpful.\n2. The evaluation does not directly support that compositional the structure is learnt. The main evaluation metric is the final accuracy, which may tell that the proposed approach works as a good pre-training algorithm for reusable modules, but this does not mean the compositional structure and components are learnt. Though there are visualizations, it still lacks numerical evaluation for this purpose.\n3. If this paper emphasizes ability as a pre-training algorithm, it should compare with other pre-training algorithms.\n4. This paper does not explain why the proposed approach is helpful for catastrophic forgetting.\nThe experiment setting has the first 99 epochs solely for assimilation and the last epoch for adaptation. So a possible explanation is that the proposed approach has 99% less iteration to update the parameters then baselines, hence it better avoids catastrophic forgetting, and just stores task specific parameters. This seems trivial and is not a stand-alone significant contribution.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "j6eaxhAwM1L", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Summary **\n\nThe paper proposes a novel approach to lifelong learning, which builds upon the idea of gradually construcing a set of reusable components.\nThrough extensive experiments on different underlying architectures, the authors demonstrate the promise of their approach.\n\n** Strengths **\n\nThe paper addresses a highly relevant problem.\nThe paper provides a very thorough evaluation of the proposed approach in a range of conditions.\nThe paper is very well written and is a pleasure to read.\nIn general, the contribution approaches the ideal when it comes to result reporting, ensuring reproducibility, and the thoroughness of experimental evaluation.\n\n** Weaknesses **\n\nWhile extremely thorough, the experiments are somewhat repetitive. I.e. overall we can confidently say that the method performs well on the selected tasks, but it seems that these experiments do not fully investigate the potential of the proposed architecture in diverse enough conditions.\n\nThe main selling point of the contribution is the ability to acquire compositional problem-solving components, but the qualitative analysis into whether the problem-solving components are actually compositional is very limited. The results given in appendix G and on the figure 3 are not very convincing, in my opinion. Similarly, in appendix D, when the architecture is applied in an intentionally compositional setting, the authors only report performance metrics.\n\nOverall, it remains largely unclear whether the proposed method is truly leveraging compositionality, or is more of an efficient way of constructing an ensemble of classifiers.\n\nNevertheless, I believe that this paper remains above the acceptance threshold, despite these limitations.\n\n** Typos **\n\n\"Qualitatively, we show that the components learned by an algorithm that subscribes our framework correspond to self-contained, reusable functions.\" - \"subscribes\" seems like a typo in this context.\n\n** Questions **\n\n- Intuitively, it seems that using component dropout introduces a conflict with the goal of obtaining disentangled/compositional components.\nI.e. the authors say \"Intermittently bypassing the new component ensures that existing components can compensate for it if it is discarded\", \nbut at the same time it seems that for compositionality, we want different components to be unique and independent of each other. I would like the authors to clarify their reasoning behind using component droupout.\n\n** Conclusion **\n\nThe paper is an extremely thorough and a very polished piece of work, which compensates what it somewhat lacks in \"wow-effect\" with general thoroughness and high quality of experiments, writing, and reporting.\nAfter reading the paper I felt genuinely grateful to authors for putting this work together in such a thorough and thoughtful manner.\n\nOverall, in my opinion, the contribution is above the acceptance threshold.\n\n** Suggestions **\n\nThe paper abruptly ends with no summary or conclusion, which, in my opinion, severely affects its perception. While I understand that\nthis is due to the strict page limit, in my opinion, sacrificing conclusion is not reasonable. It is possible to move of the tables 3 or 4 to the appendix,\nsince they convey the same general message (the message being that the model performs well).\n\nIf the paper is not accepted to publication now, I would suggest moving some of the experimental evaluations tables into appendix and putting more effort into testing the model on qualitatively different tasks. It would be very interesting to see how the model performs on nontrivial problems requiring compositional reasoning,\nand maybe less directly rooted in the visual domain. Even if staying in the visual domain is desirable, there is still room for improvement. For example, maybe would be possible to adapt the model to the RAVEN (https://arxiv.org/abs/1903.02741) dataset (after adapting it to the lifelong-learning setting) or similar tasks.\n\n** Update after rebuttal **\n\nI appreciate that the authors addressed my comments. After reading the authors response and other reviews, I still believe that it is a good paper that should be accepted to the conference.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ajiJlmznRMm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThe paper introduces a framework for lifelong learning of compositional structures. The algorithm is loosely inspired by biological learning and consists of two main steps. The step of component selection relies on existing methods that can learn task-specific structure. In the next step (adaptation), the algorithm adapts the knowledge from the previous tasks to the current task and if that is insufficient to solve the task, new components are added. Adaptation step relies on existing methods for adapting the knowledge state given a new task in continual learning (component parameters are updated). Knowledge expansion (adding new components) uses component dropout, a method proposed by the authors which combines pruning and alternating backpropagation steps with and without the potential new component. The proposed method is beneficial in terms of computational complexity in comparison with the standard lifelong learning methods. The authors evaluate the method on three compositional structures and show that it outperforms the baselines. The paper includes visualisation of the learned components, extensive appendix with additional experiments and ablation studies, and a systematic overview of the prior work in learning compositional structures and lifelong learning.\n\nScore justification:\n\nThe paper is exceptionally well-written and rich in terms of experimental results and ablation studies. The proposed algorithm combines existing methods in a novel way and extends them with component dropout. The topics of learning compositional structures and lifelong/continual learning are of high interest to the community. The new procedure (Algorithm 1) balances out the key problems of retaining existing knowledge and expanding it without frequent expensive data revisiting steps. The only concern would be the lack of conclusion and sporadic vague phrasing. Overall the paper is very interesting to read and it should have a strong impact on the research in lifelong learning. \n\nPros: \n\nThe justified parallel between the algorithm and children development research\n\nThe contributions are well-positioned in comparison to the existing work (in the Related work section and throughout the paper)\n\nComputational complexity is discussed\n\nExtensive experiments which show strong results in learning components of an increasing complexity \n\nCode is released for reproducibility\n\nCons:\n\nThe main paper combined with the appendix is quite long for a conference paper\n\nConclusion is missing\n\nFrom the sentence “Our framework separates the learning process into two broad stages:learning how to best combine existing components in order to assimilate a novel problem, and learning how to adapt the set of existing components to accommodate the new problem.” it is unclear how the two stages differ\n\nIn Table 1, you could mention the meaning of the symbols which parametrize the complexities in the caption so that it is possible to better understand the computational complexity analysis without the appendix\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above  \n\nTypos: None were found\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6jj1Zp4Fktk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new framework for compositional lifelong learning. In the proposed approach, the composition and adaptation parts are separated when a lifelong learner faces a new task: first, learn the best way to compose all existing components for the new task (and train an optional new component if exiting components aren't sufficient to reach a good performance), and only then adapt the components parameters to better fit the new problem. This new framework is validated on extensive experiments, using three composition and 3 adaptation strategies from the literature on 9 datasets. The paper is pleasing to read, each choice is discussed and justified\n\nMy main concern is about the scalability and the resilience to harder streams of tasks. Compositional and dynamic approaches are necessary when tackling complex streams of tasks, for example when having to deal with different modality or a large number of tasks.\nAll experiments in the main paper use relatively short sequences of tasks in which it makes sense to reuse all components on the new tasks since all tasks are strongly related (all coming from the same data set). I would be interested to see the performance of standard continual learning techniques from the literature on the same streams of tasks to demonstrate that there is indeed a gain in using this approach even on standard streams.\nI would also be interested in seeing how sharp the component selection is on each task, is the assimilation step selecting only one or two components per layer or is it blending them all? We could expect a sharp component selection on the compositional dataset (appendix D), which would confirm that the different components are indeed specialized to handle specific properties of the tasks. \n\nMinor questions and remarks:\n- Appendix E.1 mentions that the validation set is only used for selecting whether to keep the new component. How are the hyper-parameters ( $\\tau$, learning rates, number of samples to keep around for ER, $\\lambda$, $T_{init}$, number of assimilation/adaptation epochs, ...) selected?\n- Some papers on dynamic/compositional methods for continual learning should be discussed in the related work [1, 2, 3]\n\n\n[1] Lifelong Learning with Dynamically Expandable Networks. Yoon et. al. [2017]\n\n[2] Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting. Li et. al. [2019]\n\n[3] An Adaptive Random Path Selection Approach for Incremental Learning. Rajasegaran et. al. [2019]\n\nEdit: Upgrade from 5 to 6 after the author's response.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses lifelong/continual learning (CL) by combining reusable components. The algorithm is based on, updating components, updating how they are combined for a given task and adding new components. 

Reviewers had concerns about the learning workflow, how it could scale to harder CL streams and how it differs from existing LL/CL work.  They also asked for clarifications about compositionality. They highlighted the experiments as a point of strength.  After the rebuttal, all reviewers found the paper to be above the acceptance bar. 
Decision: Accept (Poster)