Lifelong Learning of Compositional Structures

1 INTRODUCTION . A major goal of artificial intelligence is to create an agent capable of acquiring a general understanding of the world . Such an agent would require the ability to continually accumulate and build upon its knowledge as it encounters new experiences . Lifelong machine learning addresses this setting , whereby an agent faces a continual stream of diverse problems and must strive to capture the knowledge necessary for solving each new task it encounters . If the agent is capable of accumulating knowledge in some form of compositional representation ( e.g. , neural net modules ) , it could then selectively reuse and combine relevant pieces of knowledge to construct novel solutions . Various compositional representations for multiple tasks have been proposed recently ( Zaremba et al. , 2016 ; Hu et al. , 2017 ; Kirsch et al. , 2018 ; Meyerson & Miikkulainen , 2018 ) . We address the novel question of how to learn these compositional structures in a lifelong learning setting . We design a general-purpose framework that is agnostic to the specific algorithms used for learning and the form of the structures being learned . Evoking Piaget ’ s ( 1976 ) assimilation and accommodation stages of intellectual development , this framework embodies the benefits of dividing the lifelong learning process into two distinct stages . In the first stage , the learner strives to solve a new task by combining existing components it has already acquired . The second stage uses discoveries from the new task to improve existing components and to construct fresh components if necessary . Our proposed framework , which we depict visually in Appendix A , is capable of incorporating various forms of compositional structures , as well as different mechanisms for avoiding catastrophic forgetting ( McCloskey & Cohen , 1989 ) . As examples of the flexibility of our framework , it can incorporate naı̈ve fine-tuning , experience replay , and elastic weight consolidation ( Kirkpatrick et al. , 2017 ) as knowledge retention mechanisms , and linear combinations of linear models ( Kumar & Daumé III , 2012 ; Ruvolo & Eaton , 2013 ) , soft layer ordering ( Meyerson & Miikkulainen , 2018 ) , and a soft version of gating networks ( Kirsch et al. , 2018 ; Rosenbaum et al. , 2018 ) as the compositional structures . We instantiate our framework with the nine combinations of these examples , and evaluate it on eight different data sets , consistently showing that separating the lifelong learning process into two stages increases the capabilities of the learning system , reducing catastrophic forgetting and achieving higher overall performance . Qualitatively , we show that the components learned by an algorithm that adheres to our framework correspond to self-contained , reusable functions . 2 RELATED WORK . Lifelong learning In continual or lifelong learning , agents must handle a variety of tasks over their lifetimes , and should accumulate knowledge in a way that enables them to more efficiently learn to solve new problems . Recent efforts have mainly focused on avoiding catastrophic forgetting . At a high level , algorithms define parts of parametric models ( e.g. , deep neural networks ) to be shared across tasks . As the agent encounters tasks sequentially , it strives to retain the knowledge that enabled it to solve earlier tasks . One common approach is to impose regularization to prevent parameters from deviating in directions that are harmful for performance on the early tasks ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Li & Hoiem , 2017 ; Ritter et al. , 2018 ) . Another approach retains a small buffer of data from all tasks , and continually updates the model parameters utilizing data from all tasks , thereby maintaining the knowledge required to solve them ( Lopez-Paz & Ranzato , 2017 ; Nguyen et al. , 2018 ; Isele & Cosgun , 2018 ) . A related technique is to learn a generative model to “ hallucinate ” data , reducing the memory footprint at the cost of using lower-quality data and increasing the cost of accessing data ( Shin et al. , 2017 ; Achille et al. , 2018 ; Rao et al. , 2019 ) . These approaches , although effective in avoiding the problem of catastrophic forgetting , make no substantial effort toward the discovery of reusable knowledge . One could argue that the model parameters are learned in such a way that they are reusable across all tasks . However , it is unclear what the reusability of these parameters means , and moreover the way in which parameters are reused is hard-coded into the architecture design . This latter issue is a major drawback when attempting to learn tasks with a high degree of variability , as the exact form in which tasks are related is often unknown . Ideally , the algorithm would be able to determine these relations autonomously . Other methods learn a set of models that are reusable across many tasks and automatically select how to reuse them ( Ruvolo & Eaton , 2013 ; Nagabandi et al. , 2019 ) . However , such methods selectively reuse entire models , enabling knowledge reuse , but not explicitly in a compositional manner . Compositional knowledge A mostly distinct line of parallel work has explored the learning of compositional knowledge . The majority of such methods either learn the structure for piecing together a given set of components ( Cai et al. , 2017 ; Xu et al. , 2018 ; Bunel et al. , 2018 ) or learn the set of components given a known structure for how to compose them ( Bošnjak et al. , 2017 ) . A more interesting case is when neither the structure nor the set of components are given , and the agent must autonomously discover the compositional structure underlying a set of tasks . Some approaches for solving this problem assume access to a solution descriptor ( e.g. , in natural language ) , which can be mapped by the agent to a solution structure ( Hu et al. , 2017 ; Johnson et al. , 2017 ; Pahuja et al. , 2019 ) . However , many agents ( e.g. , service robots ) are expected to learn in more autonomous settings , where this kind of supervision is not available . Other approaches instead learn the structure directly from optimization of a cost function ( Rosenbaum et al. , 2018 ; Kirsch et al. , 2018 ; Meyerson & Miikkulainen , 2018 ; Alet et al. , 2018 ; Chang et al. , 2019 ) . Many of these works can be viewed as instances of neural architecture search , a closely related area ( Elsken et al. , 2019 ) . However , note that the approaches above assume that the agent will have access to a large batch of tasks , enabling it to evaluate numerous combinations of components and structures on all tasks simultaneously . More realistically , the agent faces a sequence of tasks in a lifelong learning fashion . Most work in this line assumes that each component can be fully learned by training on a single task , and then can be reused for other tasks ( Reed & de Freitas , 2016 ; Fernando et al. , 2017 ; Valkov et al. , 2018 ) . Unfortunately , this is infeasible in many real-world scenarios in which the agent has access to little data for each of the tasks . One notable exception was proposed by Gaunt et al . ( 2017 ) , which improves early components with experience in new tasks , but is limited to very simplistic settings . Unlike prior work , our approach explicitly learns compositional structures in a lifelong learning setting . We do not assume access to a large batch of tasks or the ability to learn definitive components after training on a single task . Instead , we train on a small initial batch of tasks ( four tasks , in our experiments ) , and then autonomously update the existing components to accommodate new tasks . Our framework also permits incorporating new components over time . Related work has increased network capacity in the non-compositional setting ( Yoon et al. , 2018 ) or in a compositional setting where previously learned parameters are kept fixed ( Li et al. , 2019 ) . Another method enables adaptation of existing parameters ( Rajasegaran et al. , 2019 ) , but requires expensively storing and training multiple models for each task to select the best one before adapting the existing parameters , and is designed for a specific choice of architecture , unlike our general framework . 3 THE LIFELONG LEARNING PROBLEM . We frame lifelong learning as online multi-task learning . The agent will face a sequence of tasks T ( 1 ) , . . . , T ( T ) over its lifetime . Each task will be a learning problem defined by a cost function L ( t ) ( f ( t ) ) , where the agent must learn a prediction function f ( t ) ∈ F : X ( t ) 7→ Y ( t ) to minimize the cost , where F is a function class , and X ( t ) and Y ( t ) are the instance and label spaces , respectively . Each task ’ s solution is parameterized by θ ( t ) , such that f ( t ) = fθ ( t ) . The goal of the lifelong learner is to find parameters θ ( 1 ) , . . . , θ ( T ) that minimize the cost across all tasks : ∑T t=1 L ( t ) ( f ( t ) ) . The number of tasks , the order in which tasks will arrive , and the task relationships are all unknown . Given limited data for each new task , the agent will strive to discover any relevant information to 1 ) relate it to previously stored knowledge in order to permit transfer and 2 ) store any new knowledge for future reuse . The agent may be evaluated on any previous task , requiring it to perform well on all tasks . In consequence , it must strive to retain knowledge from even the earliest tasks . 4 THE LIFELONG COMPOSITIONAL LEARNING FRAMEWORK . Our framework for lifelong learning of compositional structures ( illustrated in Appendix A ) stores knowledge in a set of k shared components M = { m1 , . . . , mk } that are acquired and refined over the agent ’ s lifetime . Each component mi = mφi ∈ M is a self-contained , reusable function parameterized by φi that can be combined with other components . The agent reconstructs each task ’ s predictive function f ( t ) via a task-specific structure s ( t ) : X ( t ) ×Mk 7→ F , withMk being the set of possible sequences of k components , such that f ( t ) ( x ) = s ( t ) ( x , M ) ( x ) , where s ( t ) is parameterized by a vector ψ ( t ) . Note that s ( t ) yields a function from F . The structure functions select the components from M and the order in which to compose them to construct the model for each task ( the f ( t ) ’ s ) . Specific examples of components and structures are described in Section 4.1 . The intuition behind our framework is that , at any point in time t , the agent will have acquired a set of components suitable for solving tasks it encountered previously ( T ( 1 ) , . . . , T ( t−1 ) ) . If these components , with minor adaptations , can be combined to solve the current task T ( t ) , then the agent should first learn how to reuse these components before making any modifications to them . The rationale for this idea of keeping components fixed during the early stages of training on the current task T ( t ) , before the agent has acquired sufficient knowledge to perform well on T ( t ) , is that premature modification could be catastrophically damaging to the set of existing components . Once the structure s ( t ) has been learned , we consider that the agent has captured sufficient knowledge about the current task , and it would be sensible to update the components to better accommodate that knowledge . If , instead , it is not possible to capture the current task with the existing components , then new components should be added . These notions loosely mirror the stages of assimilation and accommodation in Piaget ’ s ( 1976 ) theories on intellectual development , and so we adopt those terms . Algorithms under our framework take the form of Algorithm 1 , split into the following steps . Algorithm 1 Lifelong Comp . Learning Initialize components M while T ( t ) ← getTask ( ) Freeze M for i = 1 , . . . , structUpdates Assimilation step on structure s ( t ) if i mod adaptFreq = 0 Freeze s ( t ) , unfreeze M for j = 1 , . . . , compUpdates Adaptation step on M Freeze M , unfreeze s ( t ) Add components via expansion Store info . for future adaptation Initialization The components M should be initialized encouraging reusability , both across tasks and within different structural configurations of task models . The former signifies that the components should solve a particular sub-problem regardless of the objective of the task . The latter means that components may be reused multiple times within the structure for a single task ’ s model , or at different structural orders across different tasks . For example , in deep nets , this means that the components could be used at different depths . We achieve this by training the first few tasks the agent encounters jointly to initialize M , keeping a fixed , but random , structure that reuses components to encourage reusability . Assimilation Algorithms for finding compositional knowledge vary in how they optimize each task ’ s structure . In modular nets , component selection can be learned via reinforcement learning ( Johnson et al. , 2017 ; Rosenbaum et al. , 2018 ; Chang et al. , 2019 ; Pahuja et al. , 2019 ) , stochastic search ( Fernando et al. , 2017 ; Alet et al. , 2018 ) , or backpropagation ( Shazeer et al. , 2017 ; Kirsch et al. , 2018 ; Meyerson & Miikkulainen , 2018 ) . Our framework will use any of these approaches to assimilate the current task by keeping the components M fixed and learning only the structure s ( t ) . Approaches supported by our framework must accept decoupling the learning of the structure from the learning of the components themselves ; this requirement holds for all the above examples . Accommodation An effective approach should maintain performance on earlier tasks , while being flexible enough to incorporate new knowledge . To accommodate new knowledge from the current task , the learner may adapt existing components or expand to include new components : • Adaptation step Approaches for non-compositional structures have been to naı̈vely fine- tune models with data from the current task , to impose regularization to selectively freeze weights ( Kirkpatrick et al. , 2017 ; Ritter et al. , 2018 ) , or to store a portion of data from previous tasks and use experience replay ( Lopez-Paz & Ranzato , 2017 ; Isele & Cosgun , 2018 ) . We will instantiate our framework by using any of these methods to accommodate new knowledge into existing components once the current task has been assimilated . For this to be possible , we require that the method can be selectively applied to only the component parameters φ . • Expansion step Often , existing components , even with some adaptation , are insufficient to solve the current task . In this case , the learner would incorporate novel components , which should encode knowledge distinct from existing components and combine with those components to solve the new task . The ability to discover new components endows the learner with the flexibility required to learn over a lifetime . For this , we create component dropout , described in Section 4.2 . Concrete instantiations of Algorithm 1 are described in Section 5.1 , with pseudocode in Appendix B .

Summary: The paper introduces a framework for lifelong learning of compositional structures. The algorithm is loosely inspired by biological learning and consists of two main steps. The step of component selection relies on existing methods that can learn task-specific structure. In the next step (adaptation), the algorithm adapts the knowledge from the previous tasks to the current task and if that is insufficient to solve the task, new components are added. Adaptation step relies on existing methods for adapting the knowledge state given a new task in continual learning (component parameters are updated). Knowledge expansion (adding new components) uses component dropout, a method proposed by the authors which combines pruning and alternating backpropagation steps with and without the potential new component. The proposed method is beneficial in terms of computational complexity in comparison with the standard lifelong learning methods. The authors evaluate the method on three compositional structures and show that it outperforms the baselines. The paper includes visualisation of the learned components, extensive appendix with additional experiments and ablation studies, and a systematic overview of the prior work in learning compositional structures and lifelong learning.