Lifelong Learning of Compositional Structures

1 INTRODUCTION . A major goal of artificial intelligence is to create an agent capable of acquiring a general understanding of the world . Such an agent would require the ability to continually accumulate and build upon its knowledge as it encounters new experiences . Lifelong machine learning addresses this setting , whereby an agent faces a continual stream of diverse problems and must strive to capture the knowledge necessary for solving each new task it encounters . If the agent is capable of accumulating knowledge in some form of compositional representation ( e.g. , neural net modules ) , it could then selectively reuse and combine relevant pieces of knowledge to construct novel solutions . Various compositional representations for multiple tasks have been proposed recently ( Zaremba et al. , 2016 ; Hu et al. , 2017 ; Kirsch et al. , 2018 ; Meyerson & Miikkulainen , 2018 ) . We address the novel question of how to learn these compositional structures in a lifelong learning setting . We design a general-purpose framework that is agnostic to the specific algorithms used for learning and the form of the structures being learned . Evoking Piaget ’ s ( 1976 ) assimilation and accommodation stages of intellectual development , this framework embodies the benefits of dividing the lifelong learning process into two distinct stages . In the first stage , the learner strives to solve a new task by combining existing components it has already acquired . The second stage uses discoveries from the new task to improve existing components and to construct fresh components if necessary . Our proposed framework , which we depict visually in Appendix A , is capable of incorporating various forms of compositional structures , as well as different mechanisms for avoiding catastrophic forgetting ( McCloskey & Cohen , 1989 ) . As examples of the flexibility of our framework , it can incorporate naı̈ve fine-tuning , experience replay , and elastic weight consolidation ( Kirkpatrick et al. , 2017 ) as knowledge retention mechanisms , and linear combinations of linear models ( Kumar & Daumé III , 2012 ; Ruvolo & Eaton , 2013 ) , soft layer ordering ( Meyerson & Miikkulainen , 2018 ) , and a soft version of gating networks ( Kirsch et al. , 2018 ; Rosenbaum et al. , 2018 ) as the compositional structures . We instantiate our framework with the nine combinations of these examples , and evaluate it on eight different data sets , consistently showing that separating the lifelong learning process into two stages increases the capabilities of the learning system , reducing catastrophic forgetting and achieving higher overall performance . Qualitatively , we show that the components learned by an algorithm that adheres to our framework correspond to self-contained , reusable functions . 2 RELATED WORK . Lifelong learning In continual or lifelong learning , agents must handle a variety of tasks over their lifetimes , and should accumulate knowledge in a way that enables them to more efficiently learn to solve new problems . Recent efforts have mainly focused on avoiding catastrophic forgetting . At a high level , algorithms define parts of parametric models ( e.g. , deep neural networks ) to be shared across tasks . As the agent encounters tasks sequentially , it strives to retain the knowledge that enabled it to solve earlier tasks . One common approach is to impose regularization to prevent parameters from deviating in directions that are harmful for performance on the early tasks ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Li & Hoiem , 2017 ; Ritter et al. , 2018 ) . Another approach retains a small buffer of data from all tasks , and continually updates the model parameters utilizing data from all tasks , thereby maintaining the knowledge required to solve them ( Lopez-Paz & Ranzato , 2017 ; Nguyen et al. , 2018 ; Isele & Cosgun , 2018 ) . A related technique is to learn a generative model to “ hallucinate ” data , reducing the memory footprint at the cost of using lower-quality data and increasing the cost of accessing data ( Shin et al. , 2017 ; Achille et al. , 2018 ; Rao et al. , 2019 ) . These approaches , although effective in avoiding the problem of catastrophic forgetting , make no substantial effort toward the discovery of reusable knowledge . One could argue that the model parameters are learned in such a way that they are reusable across all tasks . However , it is unclear what the reusability of these parameters means , and moreover the way in which parameters are reused is hard-coded into the architecture design . This latter issue is a major drawback when attempting to learn tasks with a high degree of variability , as the exact form in which tasks are related is often unknown . Ideally , the algorithm would be able to determine these relations autonomously . Other methods learn a set of models that are reusable across many tasks and automatically select how to reuse them ( Ruvolo & Eaton , 2013 ; Nagabandi et al. , 2019 ) . However , such methods selectively reuse entire models , enabling knowledge reuse , but not explicitly in a compositional manner . Compositional knowledge A mostly distinct line of parallel work has explored the learning of compositional knowledge . The majority of such methods either learn the structure for piecing together a given set of components ( Cai et al. , 2017 ; Xu et al. , 2018 ; Bunel et al. , 2018 ) or learn the set of components given a known structure for how to compose them ( Bošnjak et al. , 2017 ) . A more interesting case is when neither the structure nor the set of components are given , and the agent must autonomously discover the compositional structure underlying a set of tasks . Some approaches for solving this problem assume access to a solution descriptor ( e.g. , in natural language ) , which can be mapped by the agent to a solution structure ( Hu et al. , 2017 ; Johnson et al. , 2017 ; Pahuja et al. , 2019 ) . However , many agents ( e.g. , service robots ) are expected to learn in more autonomous settings , where this kind of supervision is not available . Other approaches instead learn the structure directly from optimization of a cost function ( Rosenbaum et al. , 2018 ; Kirsch et al. , 2018 ; Meyerson & Miikkulainen , 2018 ; Alet et al. , 2018 ; Chang et al. , 2019 ) . Many of these works can be viewed as instances of neural architecture search , a closely related area ( Elsken et al. , 2019 ) . However , note that the approaches above assume that the agent will have access to a large batch of tasks , enabling it to evaluate numerous combinations of components and structures on all tasks simultaneously . More realistically , the agent faces a sequence of tasks in a lifelong learning fashion . Most work in this line assumes that each component can be fully learned by training on a single task , and then can be reused for other tasks ( Reed & de Freitas , 2016 ; Fernando et al. , 2017 ; Valkov et al. , 2018 ) . Unfortunately , this is infeasible in many real-world scenarios in which the agent has access to little data for each of the tasks . One notable exception was proposed by Gaunt et al . ( 2017 ) , which improves early components with experience in new tasks , but is limited to very simplistic settings . Unlike prior work , our approach explicitly learns compositional structures in a lifelong learning setting . We do not assume access to a large batch of tasks or the ability to learn definitive components after training on a single task . Instead , we train on a small initial batch of tasks ( four tasks , in our experiments ) , and then autonomously update the existing components to accommodate new tasks . Our framework also permits incorporating new components over time . Related work has increased network capacity in the non-compositional setting ( Yoon et al. , 2018 ) or in a compositional setting where previously learned parameters are kept fixed ( Li et al. , 2019 ) . Another method enables adaptation of existing parameters ( Rajasegaran et al. , 2019 ) , but requires expensively storing and training multiple models for each task to select the best one before adapting the existing parameters , and is designed for a specific choice of architecture , unlike our general framework . 3 THE LIFELONG LEARNING PROBLEM . We frame lifelong learning as online multi-task learning . The agent will face a sequence of tasks T ( 1 ) , . . . , T ( T ) over its lifetime . Each task will be a learning problem defined by a cost function L ( t ) ( f ( t ) ) , where the agent must learn a prediction function f ( t ) ∈ F : X ( t ) 7→ Y ( t ) to minimize the cost , where F is a function class , and X ( t ) and Y ( t ) are the instance and label spaces , respectively . Each task ’ s solution is parameterized by θ ( t ) , such that f ( t ) = fθ ( t ) . The goal of the lifelong learner is to find parameters θ ( 1 ) , . . . , θ ( T ) that minimize the cost across all tasks : ∑T t=1 L ( t ) ( f ( t ) ) . The number of tasks , the order in which tasks will arrive , and the task relationships are all unknown . Given limited data for each new task , the agent will strive to discover any relevant information to 1 ) relate it to previously stored knowledge in order to permit transfer and 2 ) store any new knowledge for future reuse . The agent may be evaluated on any previous task , requiring it to perform well on all tasks . In consequence , it must strive to retain knowledge from even the earliest tasks . 4 THE LIFELONG COMPOSITIONAL LEARNING FRAMEWORK . Our framework for lifelong learning of compositional structures ( illustrated in Appendix A ) stores knowledge in a set of k shared components M = { m1 , . . . , mk } that are acquired and refined over the agent ’ s lifetime . Each component mi = mφi ∈ M is a self-contained , reusable function parameterized by φi that can be combined with other components . The agent reconstructs each task ’ s predictive function f ( t ) via a task-specific structure s ( t ) : X ( t ) ×Mk 7→ F , withMk being the set of possible sequences of k components , such that f ( t ) ( x ) = s ( t ) ( x , M ) ( x ) , where s ( t ) is parameterized by a vector ψ ( t ) . Note that s ( t ) yields a function from F . The structure functions select the components from M and the order in which to compose them to construct the model for each task ( the f ( t ) ’ s ) . Specific examples of components and structures are described in Section 4.1 . The intuition behind our framework is that , at any point in time t , the agent will have acquired a set of components suitable for solving tasks it encountered previously ( T ( 1 ) , . . . , T ( t−1 ) ) . If these components , with minor adaptations , can be combined to solve the current task T ( t ) , then the agent should first learn how to reuse these components before making any modifications to them . The rationale for this idea of keeping components fixed during the early stages of training on the current task T ( t ) , before the agent has acquired sufficient knowledge to perform well on T ( t ) , is that premature modification could be catastrophically damaging to the set of existing components . Once the structure s ( t ) has been learned , we consider that the agent has captured sufficient knowledge about the current task , and it would be sensible to update the components to better accommodate that knowledge . If , instead , it is not possible to capture the current task with the existing components , then new components should be added . These notions loosely mirror the stages of assimilation and accommodation in Piaget ’ s ( 1976 ) theories on intellectual development , and so we adopt those terms . Algorithms under our framework take the form of Algorithm 1 , split into the following steps . Algorithm 1 Lifelong Comp . Learning Initialize components M while T ( t ) ← getTask ( ) Freeze M for i = 1 , . . . , structUpdates Assimilation step on structure s ( t ) if i mod adaptFreq = 0 Freeze s ( t ) , unfreeze M for j = 1 , . . . , compUpdates Adaptation step on M Freeze M , unfreeze s ( t ) Add components via expansion Store info . for future adaptation Initialization The components M should be initialized encouraging reusability , both across tasks and within different structural configurations of task models . The former signifies that the components should solve a particular sub-problem regardless of the objective of the task . The latter means that components may be reused multiple times within the structure for a single task ’ s model , or at different structural orders across different tasks . For example , in deep nets , this means that the components could be used at different depths . We achieve this by training the first few tasks the agent encounters jointly to initialize M , keeping a fixed , but random , structure that reuses components to encourage reusability . Assimilation Algorithms for finding compositional knowledge vary in how they optimize each task ’ s structure . In modular nets , component selection can be learned via reinforcement learning ( Johnson et al. , 2017 ; Rosenbaum et al. , 2018 ; Chang et al. , 2019 ; Pahuja et al. , 2019 ) , stochastic search ( Fernando et al. , 2017 ; Alet et al. , 2018 ) , or backpropagation ( Shazeer et al. , 2017 ; Kirsch et al. , 2018 ; Meyerson & Miikkulainen , 2018 ) . Our framework will use any of these approaches to assimilate the current task by keeping the components M fixed and learning only the structure s ( t ) . Approaches supported by our framework must accept decoupling the learning of the structure from the learning of the components themselves ; this requirement holds for all the above examples . Accommodation An effective approach should maintain performance on earlier tasks , while being flexible enough to incorporate new knowledge . To accommodate new knowledge from the current task , the learner may adapt existing components or expand to include new components : • Adaptation step Approaches for non-compositional structures have been to naı̈vely fine- tune models with data from the current task , to impose regularization to selectively freeze weights ( Kirkpatrick et al. , 2017 ; Ritter et al. , 2018 ) , or to store a portion of data from previous tasks and use experience replay ( Lopez-Paz & Ranzato , 2017 ; Isele & Cosgun , 2018 ) . We will instantiate our framework by using any of these methods to accommodate new knowledge into existing components once the current task has been assimilated . For this to be possible , we require that the method can be selectively applied to only the component parameters φ . • Expansion step Often , existing components , even with some adaptation , are insufficient to solve the current task . In this case , the learner would incorporate novel components , which should encode knowledge distinct from existing components and combine with those components to solve the new task . The ability to discover new components endows the learner with the flexibility required to learn over a lifetime . For this , we create component dropout , described in Section 4.2 . Concrete instantiations of Algorithm 1 are described in Section 5.1 , with pseudocode in Appendix B .

Summary: The authors propose a new framework for compositional lifelong learning. In the proposed approach, the composition and adaptation parts are separated when a lifelong learner faces a new task: first, learn the best way to compose all existing components for the new task (and train an optional new component if exiting components aren't sufficient to reach a good performance), and only then adapt the components parameters to better fit the new problem. This new framework is validated on extensive experiments, using three composition and 3 adaptation strategies from the literature on 9 datasets. The paper is pleasing to read, each choice is discussed and justified