Continual learning in recurrent neural networks

1 INTRODUCTION . The ability to continually learn from a non-stationary data distribution while transferring and protecting past knowledge is known as continual learning ( CL ) . This ability requires neural networks to be stable to prevent forgetting , but also plastic to learn novel information , which is referred to as the stability-plasticity dilemma ( Grossberg , 2007 ; Mermillod et al. , 2013 ) . To address this dilemma , a variety of methods which tackle CL for static data with feedforward networks have been proposed ( for reviews refer to Parisi et al . ( 2019 ) and van de Ven and Tolias ( 2019 ) ) . However , CL for sequential data has only received little attention , despite recent work confirming that recurrent neural networks ( RNNs ) also suffer from catastrophic forgetting ( Schak and Gepperth , 2019 ) . A set of methods that holds great promise to address this problem are regularization methods , which work by constraining the update of certain parameters . These methods can be considered more versatile than competing approaches , since they do not require rehearsal of past data , nor an increase in model capacity , but can benefit from either of the two ( e.g. , Nguyen et al. , 2018 ; Yoon et al. , 2018 ) . This makes regularization methods applicable to a broader variety of situations , e.g . when issues related to data privacy , storage , or limited computational resources during inference might arise . The most well-known regularization methods are weight-importance methods , such as elastic weight consolidation ( EWC , Kirkpatrick et al . ( 2017a ) ) and synaptic intelligence ( SI , Zenke et al . ( 2017 ) ) , which are based on assigning importance values to weights . Some of these have a direct probabilistic interpretation as prior-focused CL methods ( Farquhar and Gal , 2018 ) , for which solutions of upcoming tasks must lie in the posterior parameter distribution of the current task ( cf . Fig . 1b ) , highlighting the stability-plasticity dilemma . Whether this dilemma differently affects feedforward networks and RNNs , and whether weight-importance based methods can be used off the shelf for sequential data has remained unclear . Here , we contribute to the development of CL approaches for sequential data in several ways . • We provide a first comprehensive comparison of CL methods applied to sequential data . For this , we port a set of established CL methods for feedforward networks to RNNs and assess their performance thoroughly and fairly in a variety of settings . • We identify elements that critically affect the stability-plasticity dilemma of weightimportance methods in RNNs . We empirically show that high requirements for working memory , i.e . the need to store and manipulate information when processing individual samples , lead to a saturation of weight importance values , making the RNN rigid and hindering its potential to learn new tasks . In contrast , this trade-off is not directly affected by the sheer recurrent reuse of the weights , related to the length of processed sequences . We complement these observations with a theoretical analysis of linear RNNs . • We show that existing CL approaches can constitute strong baselines when compared in a standardized setting and if equivalent hyperparameter-optimization resources are granted . Moreover , we show that a CL regularization approach based on hypernetworks ( von Oswald et al. , 2020 ) mitigates the limitations of weight-importance methods in RNNs . • We provide a code base1 comprising all assessed methods as well as variants of four well known sequential datasets adapted to CL : the Copy Task ( Graves et al. , 2014 ) , Sequential Stroke MNIST ( Gulcehre et al. , 2017 ) , AudioSet ( Gemmeke et al. , 2017 ) and multilingual Part-of-Speech tagging ( Nivre et al. , 2016 ) . Taken together , our experimental and theoretical results facilitate the development of CL methods that are suited for sequential data . 2 RELATED WORK . Continual learning with sequential data . As in Parisi et al . ( 2019 ) , we categorize CL methods for RNNs into regularization approaches , dynamic architectures and complementary memory systems . Regularization approaches set optimization constraints on the update of certain network parameters without requiring a model of past input data . EWC , for example , uses weight importance values to limit further updates of weights that are considered essential for solving previous tasks ( Kirkpatrick et al. , 2017b ) . Throughout this work , we utilize a more mathematically sound and less memoryintensive version of this algorithm , called Online EWC ( Huszár , 2018 ; Schwarz et al. , 2018 ) . Although a highly popular approach in feedforward networks , it has remained unclear how suitable EWC is in the context of sequential processing . Indeed , some studies report promising results in the context of natural language processing ( NLP ) ( Madasu and Rao , 2020 ; Thompson et al. , 2019 ) , while others find that it performs poorly ( Asghar et al. , 2020 ; Cossu et al. , 2020a ; Li et al. , 2020 ) . Here , we conduct the first thorough investigation of EWC ’ s performance on RNNs , and find that it can often be a suitable choice . A related CL approach that also relies on weight importance values is SI ( Zenke et al. , 2017 ) . Variants of SI have been used for different sequential datasets , but have not been systematically compared against other established methods ( Yang et al. , 2019 ; Masse et al. , 2018 ; Lee , 2017 ) . Fixed expansion layers ( Coop and Arel , 2012 ) are another method to limit the plasticity of weights and prevent forgetting , and in RNNs take the form of a sparsely activated layer between consecutive hidden states ( Coop and Arel , 2013 ) . Lastly , some regularization approaches rely on the use of non-overlapping and orthogonal representations to overcome catastrophic forgetting ( French , 1992 ; 1994 ; 1970 ) . Masse et al . ( 2018 ) , for example , proposed the use of context-dependent random subnetworks , where weight changes are regularized by limiting plasticity to task-specific subnetworks . This eliminates forgetting for disjoint networks but leads to a reduction of available capacity per task . In concurrent work , Duncker et al . ( 2020 ) introduced a learning rule which aims to optimize the use of the activity-defined subspace in RNNs learning multiple tasks . When tasks are different , 1Source code for all experiments ( including all baselines ) is available at https : //github.com/ mariacer/cl_in_rnns . catastrophic interference is avoided by forcing the use of task-specific orthogonal subspaces , whereas the reuse of dynamics is encouraged across tasks that are similar . Dynamic architecture approaches , which rely on the addition of neural resources to mitigate catastrophic forgetting , have also been applied to RNNs . Cossu et al . ( 2020a ) presented a combination of progressive networks ( Rusu et al. , 2016 ) and gating autoencoders ( Aljundi et al. , 2017 ) , where an RNN module is added for each new task and the reconstruction error of task-specific autoencoders is used to infer the RNN module to be used . Arguably , the main limitation of this type of approach is the increase in the number of parameters with the number of tasks , although methods have been presented that add resources for each new task only if needed ( Tsuda et al. , 2020 ) . Finally , complementary memory systems have also been applied to the retention of sequential information . In an early work , Ans et al . ( 2004 ) proposed a secondary network that generates patterns for rehearsing previously learned information . Asghar et al . ( 2020 ) suggested using an external memory that is progressively increased when new information is encountered . Sodhani et al . ( 2020 ) combined an external memory with Net2Net ( Chen et al. , 2016 ) , such that the network capacity can be extended while maintaining memories . The major drawback of complementary memory systems is that they either violate CL desiderata by storing past data , or rely on the ability to learn a generative model , a task that arguably scales poorly to complex data . We discuss related work in a broader context in supplementary materials ( SM D ) . Hypernetworks . Introduced by Ha et al . ( 2017 ) , the term hypernetwork refers to a neural network that generates the weights of another network . The idea can be traced back to Schmidhuber ( 1992 ) , who already suggested that a recurrent hypernetwork could be used for learning to learn ( Schmidhuber , 1993 ) . Importantly , hypernetworks can make use of the fact that parameters in a neural network possess compressible structure ( Denil et al. , 2013 ; Han et al. , 2015 ) . Indeed , Ha et al . ( 2017 ) showed that the number of trainable weights of feed-forward architectures can be reduced via hypernetworks . More recently , hypernetworks have been adapted for CL ( He et al. , 2019 ; von Oswald et al. , 2020 ) , but not for learning with sequential data . 3 METHODS . Recurrent Neural Networks . We consider discrete-time RNNs . At timestep t , the network ’ s output ŷt and hidden state ht are given by ( ŷt , ht ) = fstep ( xt , ht−1 , ψ ) , where xt denotes the input at time t and ψ the parameters of the network ( Cho et al. , 2014 ; Elman , 1990 ; Hochreiter and Schmidhuber , 1997a ) . In this work , we consider either vanilla RNNs ( based on Elman ( 1990 ) ) , LSTMs ( Hochreiter and Schmidhuber , 1997a ) or BiLSTMs ( Schuster and Paliwal , 1997 ) . Naive baselines . We consider the following naive baselines . Fine-tuning refers to training an RNN sequentially on all tasks without any CL protection . Each task has a different output head ( multi-head ) , and the heads of previously learned tasks are kept fixed . Multitask describes the parallel training on all tasks ( no CL ) . To keep approaches comparable , the multitask baseline uses a multi-head output . Because we focus on methods with a comparable number of parameters , we summarize approaches that allocate a different model per task in the From-scratch baseline , where a different model is trained separately for each task , noting that performance improvements are likely to arise in related methods ( such as Cossu et al . ( 2020a ) ) whenever knowledge transfer is possible . Continual learning baselines . We consider a diverse set of established CL methods and investigate their performance in RNNs . Online EWC ( Huszár , 2018 ; Kirkpatrick et al. , 2017a ; Schwarz et al. , 2018 ) and SI ( Zenke et al. , 2017 ) are different weight-importance CL methods . A simple weighted L2 regularization ensures that the neural network is more rigid in weight directions that are considered important for previous tasks , i.e. , the loss for the K-th task is given by L ( ψ , DK ) = Ltask ( ψ , DK ) + λ |ψ|∑ i=1 ωi ( ψi − ψ̃ ( K−1 ) i ) 2 ( 1 ) where λ is the regularization strength , ωi is the importance associated with ψi ( cf . SM B.5 and B.6 ) and ψ̃ ( K−1 ) denotes the main network weights ψ that were checkpointed after learning task K − 1 . We denote by HNET a different regularization approach based on hypernetworks that was recently proposed by von Oswald et al . ( 2020 ) . A hypernetwork ( Ha et al. , 2017 ) is a neural network ψ = h ( e , θ ) with parameters θ and input embeddings e that generates the weights of a main network . This method sidesteps the problem of finding a compromise between tasks with a shared model ψ , by generating a task-specific model ψ ( k ) from a low-dimensional embedding space via a shared hypernetwork in which the weights θ and embeddings e are continually learned . In contrast to von Oswald et al . ( 2020 ) , we focus here on RNNs as main networks : fstep ( xt , ht−1 , ψ ) = fstep ( xt , ht−1 , h ( e , θ ) ) ( Fig . 1a ) . Crucially , this method has the advantage of not being noticeably affected by the recurrent nature of the main network , since CL is delegated to a feedforward metamodel , where forgetting is avoided based on a simple L2-regularization of its output . For a fair comparison , we ensure that the number of trainable parameters is comparable to other baselines by focusing on chunked hypernetworks ( von Oswald et al. , 2020 ) , and enforcing : ∣∣θ ∪ { ek } Kk=1∣∣ ≤ |ψ| . Further details can be found in SM B.4 . Masking ( or context-dependent gating , Masse et al . ( 2018 ) ) applies a binary random mask per task for all hidden units of a multi-head network , and can be seen as a simple method for selecting a different subnetwork per task . Since catastrophic interference can occur because of the overlap between subnetworks , this method can be combined with other CL methods such as SI ( Masking+SI ) . We also consider methods based on replaying input data from previous tasks , either via a sequentially trained generative model ( Shin et al. , 2017 ; van de Ven and Tolias , 2018 ) , denoted Generative Replay , or by maintaining a small subset of previous training data ( Rebuffi et al. , 2017 ; Nguyen et al. , 2018 ) , denoted Coresets-N , where N refers to the number of samples stored for each task . Target outputs for replayed data are obtained via a copy of the main network , stored before training on the current task ( detailed baseline descriptions in SM B ) . Task Identity . We assume that task identity is provided to the system during training and inference , either by selecting the correct output head or by feeding the correct task embedding ek into the hypernetwork , and elaborate in SM G.12 on how to overcome this limitation .

Summary: This paper provides a systematic evaluation of the performance of different CL methods on RNN. The study suggests that high working memory requirements increase difficulty of learning new tasks, while the average length of input sequence is not strictly related to the difficulty of learning new tasks. The author proposes to overcome this problem by using a hypernetwork-based CL approach, which shows promising results in the experiments.