Continual learning with hypernetworks

1 INTRODUCTION . We assume that a neural network f ( x , Θ ) with trainable weights Θ is given data from a set of tasks { ( X ( 1 ) , Y ( 1 ) ) , . . . , ( X ( T ) , Y ( T ) ) } , with input samples X ( t ) = { x ( t , i ) } nti=1 and output samples Y ( t ) = { y ( t , i ) } nti=1 , where nt ≡ |X ( t ) | . A standard training approach learns the model using data from all tasks at once . However , this is not always possible in real-world problems , nor desirable in an online learning setting . Continual learning ( CL ) refers to an online learning setup in which tasks are presented sequentially ( see van de Ven & Tolias , 2019 , for a recent review on CL ) . In CL , when learning a new task t , starting with weights Θ ( t−1 ) and observing only ( X ( t ) , Y ( t ) ) , the goal is to find a new set of parameters Θ ( t ) that ( 1 ) retains ( no catastrophic forgetting ) or ( 2 ) improves ( positive backward transfer ) performance on previous tasks compared to Θ ( t−1 ) and ( 3 ) solves the new task t potentially utilizing previously acquired knowledge ( positive forward transfer ) . Achieving these goals is non-trivial , and a longstanding issue in neural networks research . Here , we propose addressing catastrophic forgetting at the meta level : instead of directly attempting to retain f ( x , Θ ) for previous tasks , we fix the outputs of a metamodel fh ( e , Θh ) termed task-conditioned hypernetwork which maps a task embedding e to weights Θ . Now , a single point has to be memorized per task . To motivate such approach , we perform a thought experiment : we assume that we are allowed to store all inputs { X ( 1 ) , . . . , X ( T ) } seen so far , and to use these data to compute model outputs corresponding to Θ ( T−1 ) . In this idealized setting , one can avoid forgetting by simply mixing data from the current task with data from the past , { ( X ( 1 ) , Ŷ ( 1 ) ) , . . . , ( X ( T−1 ) , Ŷ ( T−1 ) ) , ( X ( T ) , Y ( T ) ) } , where Ŷ ( t ) refers to a set of synthetic targets generated using the model itself f ( · , Θ ( t−1 ) ) . Hence , by training to retain previously acquired input-output mappings , one can obtain a sequential algorithm in principle as powerful as multi-task learning . Multi-task learning , where all tasks are learned simultaneously , can be seen as a CL upper-bound . The strategy described above has been termed rehearsal ( Robins , 1995 ) . However , storing previous task data violates our CL desiderata . Therefore , we introduce a change in perspective and move from the challenge of maintaining individual input-output data points to the problem of maintaining sets of parameters { Θ ( t ) } , without explicitly storing them . To achieve this , we train the metamodel parameters Θh analogous to the above outlined learning scheme , where synthetic targets now correspond to weight configurations that are suitable for previous tasks . This exchanges the storage of an entire dataset by a single low-dimensional task descriptor , yielding a massive memory saving in all but the simplest of tasks . Despite relying on regularization , our approach is a conceptual departure from previous algorithms based on regularization in weight ( e.g. , Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ) or activation space ( e.g. , He & Jaeger , 2018 ) . Our experimental results show that task-conditioned hypernetworks do not suffer from catastrophic forgetting on a set of standard CL benchmarks . Remarkably , they are capable of retaining memories with practically no decrease in performance , when presented with very long sequences of tasks . Thanks to the expressive power of neural networks , task-conditioned hypernetworks exploit task-totask similarities and transfer information forward in time to future tasks . Finally , the task-conditional metamodelling perspective that we put forth is generic , as it does not depend on the specifics of the target network architecture . We exploit this key principle and show that the very same metamodelling framework extends to , and can improve , an important class of CL methods known as generative replay methods , which are current state-of-the-art performers in many practical problems ( Shin et al. , 2017 ; Wu et al. , 2018 ; van de Ven & Tolias , 2018 ) . 2 MODEL . 2.1 TASK-CONDITIONED HYPERNETWORKS . Hypernetworks parameterize target models . The centerpiece of our approach to continual learning is the hypernetwork , Fig . 1a . Instead of learning the parameters Θtrgt of a particular function ftrgt directly ( the target model ) , we learn the parameters Θh of a metamodel . The output of such metamodel , the hypernetwork , is Θtrgt . Hypernetworks can therefore be thought of as weight generators , which were originally introduced to dynamically parameterize models in a compressed form ( Ha et al. , 2017 ; Schmidhuber , 1992 ; Bertinetto et al. , 2016 ; Jia et al. , 2016 ) . Continual learning with hypernetwork output regularization . One approach to avoid catastrophic forgetting is to store data from previous tasks and corresponding model outputs , and then fix such outputs . This can be achieved using an output regularizer of the following form , where past outputs play the role of pseudo-targets ( Robins , 1995 ; Li & Hoiem , 2018 ; Benjamin et al. , 2018 ) : Loutput = T−1∑ t=1 |X ( t ) |∑ i=1 ‖f ( x ( t , i ) , Θ∗ ) − f ( x ( t , i ) , Θ ) ‖2 , ( 1 ) In the equation above , Θ∗ is the set of parameters before attempting to learn task T , and f is the learner . This approach , however , requires storing and iterating over previous data , a process that is known as rehearsing . This is potentially expensive memory-wise and not strictly online learning . A possible workaround is to generate the pseudo-targets by evaluating f on random patterns ( Robins , 1995 ) or on the current task dataset ( Li & Hoiem , 2018 ) . However , this does not necessarily fix the behavior of the function f in the regions of interest . Hypernetworks sidestep this problem naturally . In target network weight space , a single point ( i.e. , one set of weights ) has to be fixed per task . This can be efficiently achieved with task-conditioned hypernetworks , by fixing the hypernetwork output on the appropriate task embedding . Similar to Benjamin et al . ( 2018 ) , we use a two-step optimization procedure to introduce memorypreserving hypernetwork output constraints . First , we compute a candidate change ∆Θh which minimizes the current task loss L ( T ) task = Ltask ( Θh , e ( T ) , X ( T ) , Y ( T ) ) with respect to Θ . The candidate ∆Θh is obtained with an optimizer of choice ( we use Adam throughout ; Kingma & Ba , 2015 ) . The actual parameter change is then computed by minimizing the following total loss : Ltotal = Ltask ( Θh , e ( T ) , X ( T ) , Y ( T ) ) + Loutput ( Θ∗h , Θh , ∆Θh , { e ( t ) } ) = Ltask ( Θh , e ( T ) , X ( T ) , Y ( T ) ) + βoutput T − 1 T−1∑ t=1 ‖fh ( e ( t ) , Θ∗h ) − fh ( e ( t ) , Θh + ∆Θh ) ) ‖2 , ( 2 ) where Θ∗h is the set of hypernetwork parameters before attempting to learn task T , ∆Θh is considered fixed and βoutput is a hyperparameter that controls the strength of the regularizer . On Appendix D , we run a sensitivity analysis on βoutput and experiment with a more efficient stochastic regularizer where the averaging is performed over a random subset of past tasks . More computationally-intensive algorithms that involve a full inner-loop refinement , or use secondorder gradient information by backpropagating through ∆Θh could be applied . However , we found empirically that our one-step correction worked well . Exploratory hyperparameter scans revealed that the inclusion of the lookahead ∆Θh in ( 2 ) brought a minor increase in performance , even when computed with a cheap one-step procedure . Note that unlike in Eq . 1 , the memory-preserving term Loutput does not depend on past data . Memory of previous tasks enters only through the collection of task embeddings { e ( t ) } T−1t=1 . Learned task embeddings . Task embeddings are differentiable deterministic parameters that can be learned , just like Θh . At every learning step of our algorithm , we also update the current task embedding e ( T ) to minimize the task loss L ( T ) task . After learning the task , the final embedding is saved and added to the collection { e ( t ) } . 2.2 MODEL COMPRESSION WITH CHUNKED HYPERNETWORKS . Chunking . In a straightforward implementation , a hypernetwork produces the entire set of weights of a target neural network . For modern deep neural networks , this is a very high-dimensional output . However , hypernetworks can be invoked iteratively , filling in only part of the target model at each step , in chunks ( Ha et al. , 2017 ; Pawlowski et al. , 2017 ) . This strategy allows applying smaller hypernetworks that are reusable . Interestingly , with chunked hypernetworks it is possible to solve tasks in a compressive regime , where the number of learned parameters ( those of the hypernetwork ) is effectively smaller than the number of target network parameters . Chunk embeddings and network partitioning . Reapplying the same hypernetwork multiple times introduces weight sharing across partitions of the target network , which is usually not desirable . To allow for a flexible parameterization of the target network , we introduce a set C = { ci } NCi=1 of chunk embeddings , which are used as an additional input to the hypernetwork , Fig . 1b . Thus , the full set of target network parameters Θtrgt = [ fh ( e , c1 ) , . . . , fh ( e , cNC ) ] is produced by iteration over C , keeping the task embedding e fixed . This way , the hypernetwork can produce distinct weights for each chunk . Furthermore , chunk embeddings , just like task embeddings , are ordinary deterministic parameters that we learn via backpropagation . For simplicity , we use a shared set of chunk embeddings for all tasks and we do not explore special target network partitioning strategies . How flexible is our approach ? Chunked neural networks can in principle approximate any target weight configuration arbitrarily well . For completeness , we state this formally in Appendix E .

Summary: Paper proposes a method for CL. The method is based on hypernetworks. These networks are a metamodel, which produce the parameters (from a task-conditioned embedding) which will be used in the main network. Preventing forgetting in the main network is now, replaced by preventing forgetting in the hypernetwork. This is done by imposing a regularization on the hypernetwork outcome, imposing that the generated weights should be similar for previous tasks (similar to Li & Hoiem who impose this on the network outputs). In addition, the paper proposes chunking, which refers to using an additional set of chunk, embeddings which are shared for all tasks, which allow compressing the hypernetwork. Furthermore, they propose an extension that allows for image replay (this is not an easy extension and an impressive contribution on itself, but maybe confusing for the current paper).