Provable hierarchical lifelong learning with a sketch-based modular architecture

1 INTRODUCTION . How can complex concepts be learned ? Human experience suggests that hierarchical structure is key : the complex concepts we use are no more than simple combinations of slightly less complex concepts that we have already learned , and so on . This intuition suggests that the learning of complex concepts is most tractably approached in a setting where multiple tasks are present , where it is possible to leverage what was learned from one task in another . Lifelong learning ( Silver et al. , 2013 ; Chen & Liu , 2018 ) captures such a setting : we are presented with a sequence of learning tasks and wish to understand how to ( selectively ) transfer what was learned on previous tasks to novel tasks . We seek a method that we can analyze and prove leverages what it learns on simple tasks to efficiently learn complex tasks ; in particular , tasks that could not be learned without the help provided by learning the simple tasks first . In this work , we propose an architecture for addressing such problems based on creating new modules to represent the various tasks . Indeed , other modular approaches to lifelong learning ( Yoon et al. , 2018 ; Rusu et al. , 2016 ) have been proposed previously . But , these works did not consider what we view as the main advantage of such architectures : their suitability for theoretical analysis . We prove that our architecture is capable of efficiently learning complex tasks by utilizing the functions learned to solve previous tasks as components in an algorithm for the more complex task . In addition to our analysis proving that the complex tasks may be learned , we also demonstrate that such an approach can learn functions that standard training methods fail to learn in practice , including some that are believed not to be learnable , even in principle ( Klivans & Sherstov , 2009 ) . We also consider methods for automatically identifying whether a learning task posed to the agent matches a previously learned task or is a novel task . We note briefly that a few other works considered lifelong learning from a theoretical perspective . An early approach by Solomonoff ( 1989 ) did not seriously consider computational complexity aspects . Ruvolo & Eaton ( 2013 ) gave the first provable lifelong learning algorithm with such an analysis . But , the transfer of knowledge across tasks in their framework was limited to feature learning . In particular , they did not consider the kind of deep hierarchies of tasks that we seek to learn . 1.1 OVERVIEW OF THE ARCHITECTURE . The main technical novelty in our architecture over previous modular lifelong learners is that ours uses a particular type of internal data structure called sketches ( Ghazi et al. , 2019 ; Panigrahy , 2019 ) . All such data , including inputs from the environment , outputs from a module for another task , decisions such as choosing an action to take , or even descriptions of the modules themselves , are encoded as such sketches . Although sketches have a dense ( vector ) representation , they can also be interpreted as a kind of structured representation ( Ghazi et al. , 2019 , Theorem 9 ) and are recursive ; that is , they point to the previous modules/events that they arose from ( Figure 1 , right ) . However , in order to construct these sketches in Ghazi et al . ( 2019 ) , the structure of the network is assumed to be given . No algorithms for constructing such a hierarchical network of modules from training data were known . In this work we show a method to construct such a hierarchical network from training data . We provide an architecture and algorithms for learning from a stream of training inputs that produces such a network of modules over time . This includes challenges of identifying each module , and discovering which other modules it depends on . Our architecture can be viewed as a variant of the Transformer architecture ( Radford et al. , 2021 ; Shazeer et al. , 2017 ) , particularly the Switch Transformer ( Fedus et al. , 2021 ) in conjunction with the idea of Neural Memory ( Wang et al. , 2021 ) . Instead of having a single feedforward layer , the Switch Transformer has an array of feedforward layers that an input can be routed to at each layer . Neural Memory on the other hand is a large table of values , and one or a few locations of the memory can be accessed at each layer of a deep network . In a sense the Switch Transfomer can be viewed as having a memory of possible feedforward layers ( although they use very few ) to read from . It is viewing the memory as holding “ parts of a deep network ” as opposed to data , although this difference between program and data is artificial : for example , embedding table entries can be viewed as “ data ” but are also used to alter the computation of the rest of the network , and in this sense act as a “ program modifier ” . The key component of our architecture is a locality sensitive hash ( LSH ) table based memory ( see Wang et al . ( 2021 ) ) that holds sketches of data ( such as inputs ) and modules or programs ( think of an encoding of a small deep network ) that handles such sketches ( Figure 1 , left ) . The “ operating system ” of our architecture executes the basic loop of taking sketches ( either from the environment or from internal modules ) and routing/hashing them to the LSH table to execute the next module that processes these sketches . These modules produce new sketches that are fed back into the loop . New modules ( or concepts ) are formed simply by instantiating a new hash bucket whenever a new frequently-occurring context arises , i.e . whenever several sketches hash to the same place ; the context can be viewed as a function-pointer and the sketch can be viewed as arguments for a call to that function . Frequent subsets of sketches may be combined to produce compound sketches . Finally we include pointers among sketches based on co-occurrence and co-reference in the sketches themselves . These pointers form a knowledge graph : for example if the inputs are images of pairs of people where the pairs are drawn from some latent social network , then assuming sufficient sampling of the network , this network will arise as a subgraph of the graph given by these pointers . The main loop allows these pointers to be dereferenced by passing them through the memory table , so they indeed serve the intended purpose . The main idea of the architecture is that all events produce sketches , which can intuitively be thought of as the “ mind-state ” of the system when that event occurs . The sketch-to-sketch similarity property ( see below ) combined with a similarity-preserving hash function ensures that similar sketches go to the same hash bucket ( Appendix B ) ; thus the hash table can be viewed as a content addressed memory . See Figure 1 for an illustration of this . We remark that the distances between embeddings of scene representations were used to automatically segment video into discrete events by Franklin et al . ( 2020 ) , and obtained strong agreement with human annotators . The thresholded distance used to obtain the segmentation is analogous to our locality-sensitive hashes , which we use as context sketches . A sketch can be viewed at different levels of granularity before using it to access the hash table ; this becomes the context of the sketch . Each bucket contains a program that is executed when a sketch arises that indexes into that bucket . The program in turn produces outputs and new sketches that are routed back to the hash table . The system works in a continuous loop where sketches are coming in from the environment and also from previous iterations ; the main structure of the loop is : Phenomena sketch context bucket program Phenomena input f h produces output Thus external and internal inputs arrive as sketches that are converted into a coarser representation using a function f ( see Section 2.1 below ) and then hashed to a bucket using a locality-sensitive hash function h. The program at that bucket is executed to produce an output-sketch that is fed back into the system and may also produce external outputs . This basic loop ( described in Algorithm 1 ) is executed by the routing module , which can be thought of as the operating system of the architecture . 2 SKETCHES REVIEW . Our architecture relies on the properties of the sketches introduced in Ghazi et al . ( 2019 ) . In this section we briefly describe the key properties of these sketches ; the interested reader is referred to Ghazi et al . ( 2019 ) ; Wang et al . ( 2021 ) for details . A sketch is a compact representation of a possibly exponentially-wide ( d×N ) matrix in which only a small number k of the columns are nonzero , that supports efficient computation of inner products , and for which encoding and decoding are performed by linear transformations . For concreteness , we note that sketches may be computed by random projections to Rd′ for d′ ≥ kd logN ; the JohnsonLindenstrauss Lemma then guarantees that inner-products are preserved . For our purposes , we suppose modules M1 , . . . , MN produce vectors x1 , . . . , xN ∈ Rd as output , where only k of the modules produce ( nonzero ) outputs . We view the sparse collection of module outputs as a small set of pairs of the form { [ Mi1 , xi1 ] , . . . , [ Mik , xik ] } : For example an input image has a sketch that can be thought of as a tuple [ IMAGE , 〈bit-map-sketch〉 ] . An output by an image recognition module that finds a person in the image can be represented as [ PERSON , [ 〈personsketch〉 , 〈position-in-image-sketch〉 ] ) ; here IMAGE and PERSON can be thought of as “ labels ” . If the outputs of these modules are vector embeddings in the usual sense , then indeed the inner products measure the similarity of the objects represented by the output embeddings . Observe that the constituent individual vectors xj in a sketch may themselves be sketches . For example , 〈person-sketch〉 could in turn be set of such pairs { [ NAME , 〈name-sketch〉 ] , [ FACIALFEATURES , 〈facial-sketch〉 ] , [ POSTURE , 〈posture-sketch〉 ] } , and an image consisting of multiple people could be mapped by our recognition module to a set { 〈person-1-sketch〉 , 〈person-2sketch〉 , . . . , 〈person-k-sketch〉 } . Note if the if the tuple is very large , we will not be able to recover the sketch of each of its members but only get a “ average ” or summary of all the sketches – however if a member has high enough relative weight ( see ( Ghazi et al. , 2019 , Section 3.3 ) ) it can be recovered . Appendix C.1 discusses how large objects can be stored as sketches hierarchically . Indeed , following Ghazi et al . ( 2019 ) , the outputs of modules in our architecture will be tuples that , in addition to an “ output ” component , represent the input sketches which , in turn , represent the modules that produced those inputs , e.g. , { [ MODULE-ID , 〈module-id〉 ] , [ OUTPUT-SKETCH , 〈outputsketch〉 ] , [ RECURSIVE-INPUT-SKETCH , 〈recursive-input-sketch〉 ] } . By recursively unpacking the input sketch , it is possible to reconstruct the history of computation that produced the sketch . Algorithm 1 Informal presentation of the main execution loop Input : input sketch T ( this sketch may contain a desired output for training ) 1 current-sketches← { T } 2 while current-sketches is not empty : 3 current-programs← ∅ 4 foreach sketch S in current-sketches do 5 extract context C = f ( S ) 6 update access-frequency-count of bucket h ( C ) 7 if bucket h ( C ) has a program P : 8 append ( S , P ) to current-programs 9 else : 10 if bucket h ( C ) is frequently accessed : 11 initialize program at h ( C ) with some random program and mark it for training . 12 Fetch programs Pi ( possibly by some similarity criterion ) , append those ( S , Pi ) to current-programs 13 Routing module chooses some subset of current-programs , runs each program on its associated sketch , appends outputs to current-sketches 14 Append sketches on outgoing edges of accessed buckets to current-sketches 15 if any of the programs are marked for training : 16 routing module picks one or some of them and trains them , and may choose to stop execution loop 17 if any of the sketches is of ( a special ) type OUTPUT sketch : 18 routing module picks one such , outputs that sketch or performs that action , and may choose to stop execution loop 19 if any of the sketches is of type REWARD sketch ( say for correct prediction or action ) : 20 updates the reward for this bucket and propagates those rewards to prior buckets 21 Routing module picks k combinations of sketches in current-sketches , and combine them into compound sketches : S1 , . . . , Sk ( may produce 0 sketches ) 22 current-sketches← { S1 , . . . , Sk }

Summary: The authors propose a modular architecture for multitask learning in the presence of hierarchically structured tasks leveraging sketches and locally sensitive hashing. They further provide proofs and error bounds about how the proposed architecture can learn tasks while leveraging access to previously learned subroutines. The authors additionally provide extensions to the proposed architecture in the case of tasks depending on other tasks and lack of clear task boundaries. Empirical studies are conducted for this modular approach in comparison to end-to-end models on synthetic domains related to learning the intersection of half-spaces and MNIST digits.