Continual learning in recurrent neural networks
Authors: Benjamin Ehret
Christian Henning
Maria Cervera
Alexander Meulemans
Johannes von Oswald
Benjamin F Grewe
Venue: ICLR 2021 Poster (2021)
Keywords: Recurrent Neural Networks
Continual Learning

Abstract: While a diverse collection of continual learning (CL) methods has been proposed to prevent catastrophic forgetting, a thorough investigation of their effectiveness for processing sequential data with recurrent neural networks (RNNs) is lacking. Here, we provide the first comprehensive evaluation of established CL methods on a variety of sequential data benchmarks. Specifically, we shed light on the particularities that arise when applying weight-importance methods, such as elastic weight consolidation, to RNNs. In contrast to feedforward networks, RNNs iteratively reuse a shared set of weights and require working memory to process input samples. We show that the performance of weight-importance methods is not directly affected by the length of the processed sequences, but rather by high working memory requirements, which lead to an increased need for stability at the cost of decreased plasticity for learning subsequent tasks. We additionally provide theoretical arguments supporting this interpretation by studying linear RNNs. Our study shows that established CL methods can be successfully ported to the recurrent case, and that a recent regularization approach based on hypernetworks outperforms weight-importance methods, thus emerging as a promising candidate for CL in RNNs. Overall, we provide insights on the differences between CL in feedforward networks and RNNs, while guiding towards effective solutions to tackle CL on sequential data.

Published as a conference paper at ICLR 2021
CONTINUAL LEARNING IN RECURRENT NEURAL NETWORKS
Benjamin Ehret*, Christian Henning*, Maria R. C Alexander Meulemans, Johannes von Oswald, Benjamin E. Grewe *Equal contribution
Institute of Neuroinformatics University of Zürich and ETH Zürich Zürich, Switzerland (behret, henninge, amaulee,,,mm voswald), borewe @ethz. ch
ABSTRACT
While diverse collection of continual learning (CL) methods has been proposed to prevent catastrophic forgetting, a thorough investigation of their effectiveness for processing sequential data with recurrent neural networks (RNNs) is lacking. Here, we provide the first comprehensive evaluation of established CL methods variety of sequential data benchmarks. Specifically, shed light the particularities that arise when applying weight-importance methods, such as clas- tic weight consolidation, to RNNs. In contrast feedforward networks, RNNs iteratively reuse shared set of weights and require working memory to process input samples. We show that the performance of not directly affected by the length of the processed sequences, but rather by high working memory requirements, which lead to an increased need for stability at the cost of decreased plasticity for learning subsequent tasks. We additionally pro- vide theoretical arguments supporting this interpretation by studying linear RNNs. Our study shows that established CL methods can be successfully ported the recurrent case, and that a recent regularization approach based on hyperneworks outperforms weight-importance methods, thus emerging for CL in RNNs. Overall, we provide insights on the differences between CL in feedforward networks and RNNs, while guiding towards effective solutions tackle CL on sequential data.
methods is
promising candidate
INTRODUCTION
The ability to continually learn from non-stationary data distribution while transferring and protect- ing past knowledge is known as continual learning (CL). This ability requires neural networks to be stable prevent forgetting, but also plastic learn novel information, which is referred as the stability-plaaticin dilemma (Grossberg.) 2007 Mermillod et af.) 2013). To address this dilemma, a variety of methods which tackle CL for static data with feedforward networks have been proposed (for reviews refer to Parisi et al. al.] (2019) and van de Ven and Tolias (2019)). However, CL for sequential data has only received little attention, despite recent work contirming that recurrent neural networks (RNNs) also suffer from catastrophic forgetting (Schak and Gepperth 2019).
set of methods that holds great promise address this problem are regularization methods, which work by constraining the update of certain parameters. These methods can be considered more versatile than competing approaches, since they do not require rehearsal of past data, nor an increase in model capacity, but can benefit from either of the two (e.g., [guyen et al.) 2018 Yoon ct al. 2018). This makes regularization methods applicable to a broader variety of situations, e.g. when issues related to data privacy, storage, or limited computational resources during inference might arise. The most well-known regularization methods are tright-importance methods, such as elastic weight consolidation (EWC, KKrkpatrick et al. (2017a)) and synaptic intelligence (SI, Zenke jet al. (2017)), which are based on assigning importance values to weights. Some of these have a direct probabilistic interpretation as prior-focused CL. methods (Farquhar and Gall 2018), for which

---

Published as a conference paper at ICLR 2021
solutions of upcoming tasks must lie in the posterior parameter distribution of the current task (cf. Fig. [bb. highlighting the plaattittt dilemma. Whether this dilemma differently affects feedforward networks and RNNs, and whether weight-importance based methods can be used off the shelf for sequential data has remained unclear.
Here, contribute to the development of CL. approaches for sequential data in several ways.
We provide a first comprehensive comparison of CL methods applied to sequential data. For this, we port a set established CL methods for feedforward networks to RNNs and assess their performance thoroughly and fairly in a variety of settings.
We identify elements that critically affect the kability-plasticii dilemma of weight- importance methods in RNNs. We empirically show that high requirements for working memory, i... the need to store and manipulate information when processing individual sam- ples, lead to a saturation of weight importance values, making the RNN rigid and hindering its potential to learn new tasks. In contrast, this trade-off is not directly affected by the sheer recurrent reuse of the weights, related to the length of processed sequences. We complement these observations with a theoretical analysis of linear RNNs.
We show that existing CL approaches can constitute strong baselines when compared in standardized setting and if equivalent sypperarraeeeeroopimiizae Moreover, we show that a CL regularization approach based hypemetworks (von Oswald et al. 2020p mitigates the limitations of
resources are granted.
methods in RNNs.
We provide code base comprisnng all assessed methods as well as variants of four well known sequential datasets adapted to CL: the Copy Task (Graves al.) (0144). Sequential Stroke MNIST (Gulcehre et al., 2017). AudioSet (Gemmeke et al. 2017] and multilingual
Part-of-Speech tagging (Nivre et al. 2016).
Taken together, our experimental and theoretical results facilitate the development of CL methods that are suited for sequential data.
2 RELATED WORK
Continual learning with sequential data. As in Parisi et al. (0199), categorize CL methods for RNNs into regularization approaches, dynamic architectures and complementary memory systems.
Regularization approaches optimization constraints on the update of certain network parameters without requiring a model of past input data. EWC, for example, uses weight importance values to limit further updates of weights that are considered essential for solving previous tasks (Kirkpatrick) et al. 20176). Throughout this work, we utilize a more mathematically sound and less memory- intensive version of this algorithm, called Online EWC (Huszár 2018 (Shwwrz et al.. 2018). Although highly popular approach in feedforward networks, it has remained unclear how suitable EWC is in the context of sequential processing. Indeed, some studies report promising results in the context of natural language processing (NLP) (Madasu and Rao 2020; Thompson et al. 2019). while others find that performs poorly (Asghar et al.) 2020 Cossu et al. 2020a; Li et al. 2020). Here, we conduct the first thorough investigation of EWC's performance on RNNs, and find that it can often be a suitable choice. A related CL approach that also relies on weight importance values SI Zenke et al., 2017). Variants of SI have been used for different sequential datasets, but have not been systematically compared against other established methods (Yang et al.) 2019 Masse et al.) 2018 Lee| 2017). Fixed expansion layers (Coop and Arel 2012) are another method to limit the plasticity of weights and prevent forgetting, and in RNNs take the form of a sparsely activated layer between consecutive hidden states (Coop and Arel 2013). Lastly, some regularization approaches rely on the use of overllallg and orthogonal representations to overcome catastrophic forgetting (French) 1992 1994, 1970). Masse et al. (2018), for example, proposed the use of context-dependen. random subnetworks, where weight changes are regularized by limiting plasticity to task-specific subnetworks. This eliminates forgetting for disjoint networks but leads to a reduction of available capacity per task. In concurrent work, Duncker al. (2020) introduced learning rule which aims to optimize the use of the activity-define subspace in RNNs learning multiple tasks. When tasks are different,
Source code for all experiments (including all baselines) is available https: :/gi thub. com/
haris mariacer/cl_in__
2

---

Published as a conference paper at ICLR 2021
catastrophic interference is avoided by forcing the use of task-specific orthogonal subspaces, whereas the reuse of dynamics is encouraged across tasks that are similar.
Dynamic architecture approaches, which rely on the addition of neural resources to mitigate catas- trophic forgetting, have also been applicd to RNNs. [Cossu et al. (2020)) presented combination of progressive networks (Rusu et al. 2 and gating autoencoders (Aljundi et al. (017), where
RNN module is added for each new task and the reconstruction error of task-specific autoencoders used infer the RNN module to be used. Arguably, the main limitation of this type of approach is the increase in the number of parameters with the number of tasks, although methods have been presented that add resources for each new task only needed (Tsuda et al. 2020).
Finally, complementary memory systems have also been applied to the retention of sequential information. In an early work,[Am al. (2004) proposed a secondary network that generates patterns for rehearsing previously learned information. AAsghar et al. (2020) suggested using an external memory that is progressively increased when new information encountered. [Sodhani et al. (2020) combined an external memory with Net2Net (Chen et al. 2016), such that the network capacity can be extended while maintaining memories. The major drawback of complementary memory systems is that they either violate CL desiderata by storing past data, or rely on the ability to learn a generative model, a task that arguably scales poorly to complex data. We discuss related work in a broader context in supplementary materials (SM D).
Hypernetworks. Introduced by Ha et al. (2017), the term hypernetwork refers to neural network that generates the weights of another network. The idea can be traced back to SSchmidhuberr (9922),
who already suggested that recurrent hypernetwork could be used for learning to learn (Schmidhuber 1993). Importantly. hypernetworks can make use of the fact that parameters in a neural network possess compressible structure (Denil et al. 2013 Han et al. 2015). Indeed, Ha et al. (2017) showed that the number of trainable weights of feed-forwat architectures can be reduced via hypernetworks. More recently, hypernetworks have been adapted for CL (He et al. 2019 von Oswald et al. 2020).
but not for learning with sequential data.
3 METHODS
Recurrent Neural Networks. We consider discrete-time RNNs. At timestep 6, the network's
output Yd and hidden state h, are given by (fe, h2) = Seep (x, h 1, 1.), where x, denotes the input time t and & the parameters of the network (Cho et al., 2014 Elman 1990 Hochreiter and Schmidhuber 1997a). In this work, we consider either vanilla RNNs (based on Elman (1900),
LSTMs (Hochreiter and Schmidhuber 1997a) or BiLSTMs (Schuster and Paliwal 1997).
Naive baselines. We consider the following naive baselines. Fine-tuning refers to training an RNN sequentially on all tasks without any CL protection. Each task has a different output head (multi-head), and the heads of previously learned tasks kept fixed. Multitask describes the parallel training on all tasks (no CL). To keep approaches comparable, the multitask baseline uses multi-head output. Because we focus methods with comparable number of parameters, summarize approaches that allocate a different model per task in the From-scratch baseline, where a different model is trained separately for each task, noting that performance improvements are likely to arise in related methods (such as Cossu et al. (020a)) whenever knowledge transfer is possible.
Continual learning baselines. We consider a diverse set of established CL methods and investigate their performance in RNNs. Online EWC (Huszár 2018 Kirkpatrick al. 2017a Schwarz et al. 2018) and SI (Zenke et al. (017)) are different weight- importancotteeee CL methods. A simple weighted L2 regularization ensures that the neural network is more rigid in weight directions that are considered important for previous tasks, i.e., the loss for the K-th task is given by
C(v, Dk) Cask (v., DK) + + with - J -11-
where is the regularization strength, wi is the importance associated with dy (cf. SM SM5 and B.6) and [(K-)) denotes the main network weights v that were checkpointed after learning task K --1 1. We denote by HNET different regularization approach based on hypernetworks that was recently proposed by yon Oswald et al. (2020). A hypernetwork (Ha et al. 2017) is a neural network / h(e, 0) with parameters 0 and input embeddings that generates the weights of
3

---

Published as a conference paper at ICLR 2021
ye
Low error for task 1
EWC Hypernetwork
Low enror for task 2
EWC solution space
h(e, 0)
fstep
1/1
2/2
Xt
(a)
(b)
Figure 1: (a): A hypemnetorr h(e, 8) produces the weights v of recurrent main network conditioned on e. (b): Here, we illustrate a hypemnetoork based CL approach versus a weight- importance method (such EWC). Both methods start learning second task from common solution 1/1 EWC is rigid along certain directions in weight space, which leads to a trade-off solution w.w when seeking good optima for the upcoming task (cf. Farquhar and Gall (2018)). approach (Eq, 2 has no such trade-off built into its objective; it is only limited by the optimization algorithm and network capacity, and capable to output the task-specific solutions 1 and HHETT (figure adapted from Kirkpatrick et al. (2017a))
The hypernetwork-bore
main network. This method sidesteps the problem of finding a compromise between tasks with shared model ds, by generating a task-specific model wilk) from low-dimensional embedding space via a shared hypernetwork in which the weights 0 and embeddings e are continually learned. In contrast to von Oswald et al. (2020), we focus here on RNNs as main networks: Ssep (xe: h v/)
faop (x,, h 0)) (Fig. lak. Crucially, this method has the advantage of not being noticeably affected by the recurrent nature of the main network, since CI. is delegated to a feedforward meta- model, where forgetting is avoided based on simple 1.2rreguarizztioon of its output. For a fair comparison, we ensure that the number of trainable parameters is comparable to other baselines by focusing chunked hypernetworks (von Oswald et al. 2020), and enforcing: |0 fex | < [bb Further details can found in SM B.4 Masking (or context-dependen gating, Masse et al. (0018) applies a binary random mask per task for all hidden units of multi-head network, and be as a simple method for selecting a different subnetwork per task. Since catastrophic interference occur because of the overlap between subnetworks, this method can be combined with other CL. methods such as SI (Masking+SI), We also consider methods based on replaying input data from previous tasks, either via sequentially trained generative model (Shin et al. 2017 van de Ven and Tolias 2018), denoted Generative Replay, or by maintaining small subset of previous training data (Rebulfi et al. 2017 Nguyen et al. 22018), denoted Coresets-N where N refers to the number of samples stored for each task. Target outputs for replayed data are obtained via copy of the main network, stored before training on the current task (detailed baseline descriptions in SM B
Task Identity. We assume that task identity is provided to the system during training and inference, either by selecting the correct output head or by feeding the correct task embedding e₁ into the hypernetwork, and elaborate in G.12 how overcome this limitation.
4 ANALYSIS OF WEIGHT-IIPORTANCE METHODS
cright-importance methods have widely been used in feedforward networks, but whether or not they are suited for tackling CL in RNNs has remained unclear. Here, we investigate the particularities that weight-importance methods face when applied to RNNs. As opposed to feedforward networks, RNNs provide a natural way to process temporal sequences. Because their hidden states are function of newly incoming inputs as well as their own activity in the previous timestep, RNNs are able to store and manipulate sample-specific information within their hidden activity, thus providing a form of working memory. Importantly, processing input sequences
sequence element at time results in the reuse of recurrent weights for number of times that is equal to the length of the input sequence. In this section, investigate whether weight importance values directly affected by working
4

---

Published as a conference paper at ICLR 2021
(a) Copy Task
(b) Padded
(c) Permuted (d) Patter Manipulation
4 14446
f 4 F1 12
Figure 2: Variants of the Copy Task. (a): Example pattern of the basic Copy Task where input and output patterns are identical. (b): Padded Copy Task, the input padded with zeros, but the output consists only of the pattern itself, i.e. to (a). (c): In Permuted Copy, the output is time- permuted version of the presented input in (a). (d): In the Pattern Manipulation Task (r = 1), the output
corresponds to XOR operation between the original input (a) and a permuted version of the input (c). The blue rectangle highlights this operation for two specific bits.
memory requirements or by the length of processed sequences. For this, we develop some intuitions by studying linear RNNs, which we then test in non-linear RNNs using synthetic dataset.
First, we obtain some theoretical insights by analysing how linear RNNs can learn to solve a set of tasks (for details refer to SMC). Whenever task-specific output heads are not rich enough model task variabilities, RNNs trained with methods whose recurrent computation is not task-conditioned (e.g- weight-importance methods) must solve all tasks simultaneously within their hidden space. In an extreme scenario where tasks are so different that they cannot share any useful computation, becomes clear that task interferences can be prevented if the information relevant to each task resides in task-specific orthogonal subspaces (refer to SM SM G. for a discussion of scenarios with task similarity). Maintaining this structure within the hidden space imposes certain constraints on the recurrent weights. Our theory shows these constraints increase with the number of tasks and with the dimensionality of the task-specific subspaces. Based on these theoretical insights, we hypothesize that also in nonlinear RNNs increasing working memory requirements cause high weight rigidity (as illustrated by high importance values), whereas the recurrent reuse of weights is not a driving factor.
To test this hypothesis, Task (Graves et al.) 2014), in which a random binary input pattern has to be recalled by the network after stop bit is observed (see Fig. 2 and SM E. for details). For all Copy Task experiments, vanilla RNNs combined with orthogonal regularization (see SM G.2.. We denote the length (number of timesteps) of the binary input pattern to be copied by p. and the actual number of timesteps until the stop bit by (examples can be found in SM Fig. (ST). This distinction allows us to consider two variants, the basic Copy Task where p i, and the Padded Copy Task where i p (Fig. 2 a and b). In this variant, we zero-pad a binary input pattern of length p for i i p timesteps until the occurrence of the stop bit, resulting an input sequence with timesteps. Specifically, we consider set of Copy Task?]] with varying input lengths i and, either fixed pattern length p = 5, or pattern length tied to the input length (p = i). This allows us to disentangle how sequence length and memory load affect weight importance.
explore synthetic dataset consisting of several variations of the Copy
As in Online ine EWC, we calculate weight importance as the diagonal elements the empirical Fisher information matrix (see SM B.5). To quantify memory load, we study the intrinsic dimensionality of the hidden state of the RNN using principal component analysis (PCA), once networks have been trained to achieve near optimal performance (above 99%). We define the intrinsic dimensionality as the number of principal components that are needed to explain 75% of the variance.
As expected, the intrinsic dimensionality of the hidden space increases during input pattern presenta- tion and peaks after p timesteps, i.e. the stop bit for tasks with p i (Fig. [aa and i p timesteps before the stop bit if p = 5 remains fixed (Fig. 3bb Weight importance values rapidly increase with memory requirement (p), but not sequence length (inereasing i, fixed p) (Fig. 3c). The same trend is
Note, these tasks are learned independently isolate the effects of and p on weight importance values. NNote, Fig. 3a shows a decreased dimensionality at t = 0 for = 35 or i = 40 compared to i = 30. We hypothesize that this is due to need to non-linearly encode information into the hidden state for large and verified that the dimensionality increases with when using Kernel PCA (Schölkopf et al. 1997), data not shown.
5

---

Published as a conference paper at ICLR 2021
100
100
number of did.
number of d'dmm.
mcan Fisher
varying
mmni length
15
100
fixed p
50
50-
20
25
50-
0
40
0
-20
20
40
-20
10
20
30
40
t
input length
(a)
(b)
(c)
Figure 3: (a) Intrinsic dimensionality per timestep of the 356-dimensional RNN hidden space h, for the basic Copy Task, where input and pattern lengths are tied (i = p). The stop bit (dotted black line) shown time where the pattern length is fixed (p 5) but input length i varies. In (a) and (b), dimensionality of the hidden state space increases only during input pattern presentation. (c) Mean Fisher values (weight-impootance values in Cnline EWC) of recurrent weights after learning the Copy Task (solid line, p = 5, 40) or the Padded Copy Task (dotted line, p = 5) independently for an increasing set of sequence lengths (Mean + SD, n 5).
(Mean SD, n 5). (b) Same as (a) for the Padded Copy Task,
observed when computing weight-importance values according (cf. SMG.33, and when using LSTMs instead of vanilla RNNs (cf. SM We extend this analysis to CL. setting in SM G.5
Overall, this analysis reveals that impooraane methods are affected by the processing and storage required by the task, but not by the sequential nature of the data, even if the same set of weights is reused over many timesteps. This could cause weight-importance methods to suffer from a saturation of importance values when working memory requirements are high, which could in turn reduce their plasticity for learning new tasks. We explore this in the next section.
CONTINUAL LEARNING EXPERIMENTS
To highlight strengths and weaknesses of different CL methods in various settings, we performed experiments one synthetic and two real-world sequential datasets, using different types of RNNs. We distinguish between du Buring and Final accuracies. The during accuracy a CL experiment is obtained by taking the mean over the test accuracy from each task right after it has been trained on, ie., when tasks have not yet been subject to forgetting. The final accuracy describes the mean test accuracy over all tasks obtained after the last task has been learned.
For all reported methods, results were obtained via an extensive hyperparameter search, where the hy- perparameter configuration of the with best final accuracy selected and subsequently tested on multiple random seeds (experimental details in FM. We provide additional CL experiments on multilingual NLP data in SM G.9
5.1
VARIATIONS OF THE COPY TASK
After exposing the challenges that reight-importance methods face when dynamically processing data, we explore how these manifest in a CL scenario. We compare weight- weight-imports importance methods against other CL approaches, with partieular focus HNET, which in principle bypass those challenges. We transform the Copy Task into a set of CL tasks by applying for each task a different random time-permutation N (A)) In this setting, which we denote Permuted Copy Task, each timestep t, E (1,.....) from the input pattern has to be recalled at output timestep f, x (1.) (Fig. 2b). We perform these experiments on vanilla RNNs First, we evaluate all methods in relatively simple scenario with five tasks using p = i 5 (Table D}. Online EWC achieves very high performance, and HNET reaches close to 100% accuracy. The random subnetworks in Masking can learn individual tasks to perfection. However, weight changes within subnetworks, which result from random overlaps, cause severe performance drops and show the need to add stabilization mechanisms, e.g., Mask ikkng+Si. Since the input data distribution is relatively simple and identical across tasks, learning generative model is feasible, which is illustrated by the performance of Gen. Replay.

---

Published as a conference paper at ICLR 2021
Table 1: Mean during and final accuracies for the Permuted Copy Task with i = 5 using 5 tasks each (Mean SEM in %, n 10).
Table 2: Detailed Online EWC/HNET compar- isons (Mean + SEM in %, n 5). is the number of task-specific random permutations.
during
final
during
final
Multitask
N/A
99.87 + 0.05
Padded Copy Task
From scratch
N/A
100.00 0.00
HNET
100.00 = 0.00
100.00 + 0.00 97.89 + 0.10
Fine-tuning
99.99 0.00
71.05 0.13
Online ENC
97.94 = 0.09
HNRT
99.98 0.00
99.66 0.01
Pattern Manipulation Task r
Online EWC
99.93 = 0.01
98.66 0.14
HNET
100.00 = 0.00
99.84 + 0.15
SI
98.41 = 0.06
94.03 + 0.24
Online EWC
98.52 0.27
95.45 + 0.17
Masking
99.53 = 0.26
72.31 0.82
Masking+SI
99,40 =: 0.25
99.40
Pattern Manipulation Task r 5
Gen. Replay
100.00 - 0.00
100.00 0.00
HNET
95.73 + 1.44
93.87 + 1.24
Coresets-100
100.00 1 0.00
99.94 1 0.00
Online EWC
87.40 + 4.53
81.80 + 3.25
In the following, we focus on a comparison between Online EWC and HNET to further investigate how these methods affected by sequence length and working memory requirements. We first test whether On nn EWC is affected by sequence length by investigating the Permuted Copy Task at the performance of both methods is not markedly
p 5, = 25 using 5 tasks. As T affected by sequence length. Interestingly, the results are slightly better for longer sequences with both methods, which can be due an increased processing time between input presentation and recall. Next, we compare the performance of Online EWC and HNET in a set of tasks for which working memory requirements can be easily controlled. In this setting, referred to as Pattern Manipulation Task, difficulty is controlled by set of task-specific random permutations along the time axis (Fig. 21.. The output computed from the input pattern by applying a binary XOR operation iteratively with all of its permutations (i.e. the result of the XOR between the input and its first permutation will then undergo a second XOR operation with the second permutation of the input, and so on). Note that this variant substantially differs from previous Copy Task variations, since the processing of input patterns is now both input- and task-dependent. As shown in Table 2 Onl
EWC experiences larger drop with increased task difficulty than HNET, confirming that it is more severely affected by working memory requirements. Finally, we investigate the difference between single-head and multi-head settings, as well as task conditional processing in G.13
5.2 SEQUENTIAL STROKE MNIST
To test whether the results from the synthetic Copy Task hold true for real world data we turned to sequential digit recognition task where task difficulty can be directly controlled. In the Stroke MNIST (SMNIST) dataset (de Jong 2016), MNIST images (LeCun et al.] 19988) are represented as sequences of pen displacements, that result in the original digits when drawn in order. We adapt this dataset to CL scenario by splitting it into five binary classification problems (digits 0 vs 1, vs 3, etc.), reminiscent of the popular Split-MNIST experiment commonly used to benchmark CL methods on static data ZZnke et al. 2017). Interestingly, this dataset allows exploring how the performance of different CL methods depends on the difficulty of individual tasks by generalizing the notion of Split-SMNIST sequences of m SMNIST samples (ef. Gulcehre al. 2017), where each sequence contains only two types of digits (e.g. 2332 or 7767 for m = 4). To obtain a binary decision problem, we randomly group all 2d possible sequences within task into two classes. This ensures that despite increasing levels of task difficulty, as determined by m. chance level is not affected. Crucially, an increase in m leads an increase in the amount of information that needs be stored and manipulated per input sequence, and therefore allows exploring the effect that increasing working memory requirements have on different CL methods in a real-world dataset.
We train LSTMs on five tasks for an increasing number of digits per sequence (m = 2, 3, 4) and observe that methods are differently affected by the task difficulty level (see Fig. 4 For m = 1 Online inn EWC, SI and HNET all achieve above 97% performance. However for m 4 the performance of Online EWC and SI drops to 73.16 + 0.98% and 69.58 + 0.55% respectively, while the hypernetwork approach successfully classifies 94.42 + 1. 1.85% of all inputs. Thus weight- importance methods seem more strongly affected than ENET by an increase in task complexity and working memory requirements. Interestingly, the performance gap depends setup as outlined in S.G.. Coresets perform slightly worse, especially when comp-ccopyy increases. Mask i ST, which trades-off network capacity for the ability of finding solutions in
the experimental
7

---

Published as a conference paper at ICLR 2021
100
HNET
Online EWC
HNET
accuracy (%)
Online EWC
10
accuracy (%)
-100
SI
80
Masking SI
classes
-80
Coresets
Fine-tuning
3
From scratch
2 2
60
digits per sequence (m)
Multitask
tasks
10
tasks
10
Figure 4: Mean final accuracies for four Split Sequential-SMNIST experiments, each compris- ing five tasks (Mean + SEM, n = 10).
Figure 5: Mean final accuracies for Split- AudioSet experiments with varying number of classes and tasks.
less rigid subnetwork, emerges as the preferable method for this experiment. We additionally list during accuracies for all methods in SM Table[Jaan discuss the use of replay for Split-MMNIST in Finally we show that, consistent with our Copy Task results, the performance of weight- importance methods not significantly affected when sequence lengths are increased without concomitant increase in working memory (cf. SM. G.8).
SM SM
5.3 AUDIOSET
AudioSet (Gemmeke al., 2017) dataset of manually annotated audio events. It consists of 10-second audio snippets which have been preprocessed by VGG network extract 128- dimensional feature vectors at Hz. This dataset has been previously adapted for CL by Kemker and Kanan (2018) and Kemker et al. (2018). whose particular split has been made pub- lic, and by Cossu et al. (2020b), for which the test set size largely differed across classes. We therefore created new variant, which we call Split-AudioSet-I0 containing tasks with 10 classes each (see SM
Table 3: Mean during and final accuracies for the Split-AudioSe-100 experiments (Mean + SEM in %, 10).
during
final
Mu1 i task From-scratch Fine-tuning HNET Online ENC
N/A
77.31 + 0.10
N/A
79.06 + 0.11
71.95 0.24
49.02 1.00
73.05 0.45
71.76 1 0.62
68.82 = 0.20
65.56 + 0,35
SI
67.66 0.10
66.92 0.04
Masking Masking+SI
75.81 0.15
50.87 = 1.09
64.88 = 0.19
64.86 0.20
F33 for details).
Coresets-100
74.25 - 0.11
72.30 + 0.11
The results obtained this dataset using LSTMs
Coreseta-500
77.03 0.08
73.90 + 0.07
are listed in Table 3 HNET is the strongest among regularization based methods, and is only outperformed by Coresets, which rely on storing past data. Masking during accuracies indicate that random subnetworks have enough ca- pacity to learn individual tasks, but low final accuracies suggest that catastrophic forgetting occurs, presumably because of the overlap between subnetworks. This is partly solved in Mask maakiing Masking+SI by introducing stabilization which, however, reduces plasticity for learning new tasks. In contrast to the findings of Sec. [.2 and SM Masking+SI performs worst among regularization approaches, indicating that trading-off capacity for complex datasets can be harmful. The F From scrat ch baseline outperforms other methods, which is explained by the fact that it trains a separate model per task, leading to 10 times more network capacity. Notably, we were not able to successfully train Generative Replay ay model on this dataset despite extensive hyperparameter search. Together with the results in Sec. 5.1 this highlights that the performance of Genera i ve Repl ay depends on the complexity of the input data distribution, and not necessarily on the CL nature of the problem. To further investigate the ptaaiiity trade-off, we tested HNET and Online EWC across a range of difficulty levels in individual tasks. This can be controlled by the number of classes to be learned within each task, which varied from two to ten. For both methods hyperparameters found for Split-AudioSer 10. Since the performance of On line EWC strongly depends the regularization parameter X, we tuned this value achieve optimal results in each setting (cf. Fig. S2) Fig. showw the task-averaged f final 1 accuracies for the different task-difficulty settings. While HNET performance is primarily affected by task difficulty but not by the number of tasks, results for Online EWC show an interplay between task difficulty and the ability to retain good performance on many tasks. These results provide further evidence that the hypernetwork- approach can resolve the limitations of ingootiimpottacce CL methods for sequential data.
and
used the best
8

---

Published as a conference paper at ICLR 2021
6 DISCUSSION
The stability dilemma with sequential data. Weight-importance methods address CL by progressively constraining network's weights, directly trading plasticity for stability. In the case of RNNs, weights are subject to additional constraints, since the same set of weights is reused across time to dynamically process an input stream of data. We show that increased working memory requirements, resulting from more complex processing within individual tasks, lead to high weight- importance values and can hinder the ability learn future tasks (cf. Sec. 44. On the contrary, find that longer sequence lengths do not impact performance for a fixed level of task complexity (cf. Fig. 3b Table 2). suggesting that weight reuse doesn't ''t interfere with the RNN's ability retain previous knowledge. These observations are consistent with our theoretical analysis of linear RNNs (SMC) which predicts that more challenging processing within individual tasks leads to increased interference between tasks. This aggravates the staiility-plasticit dilemma in iniigtttmportancee based methods, which we confirm in a range of experiments.
Benefits of a hypernetwwok-bbssed CL approach for sequential data. We propose that dilemma
hypernetwork-baacee approach (von Oswald et al.) 2020) can alleviate the when continually learning with RNNs. Since stability is outsourced to a regularizer that does not directly limit the plasticity of main network weights (cff. SM Eq. 2). this approach has more flexibility than reight-importance methods for finding new solutions, as shown by our experiments. Although Coresets and Generative Replay perform better, these approaches might not always be applicable: Coresets rely on the storage of past data (which might not always be feasible for privacy storage reasons), and Generative ve Replay scales poorly to complex data. On the contrary, an approach based on hypernetworks has the versatility of regularization approaches and can be used in variety of situations.
Future avenues for CL with RNNs. As discussed in SMG.11. an interesting avenue for weight- importance methods is the use of task-conditional processing in order to overcome the need of solving all learned tasks in parallel. Although hypernetworks overcome this limitation by design, they introduce additional optimization challenges, especially in conjunction with vanilla RNNs (cf. SM G.2 which leaves room for future improvements. An interesting direction is the use of recurrent hypernetwork to generate timestep-specific weights in the main RNN (Ha et al. 2017: Suarez 2017). Although naive application of this combination for CL using SM Eq. 2 would come at the cost of a linear increase in computation with the number of timesteps, this problem can be elegantly sidestepped by the use of a feed-forward hypernetwork that generates the weights of the recurrent hypernetwork. SM Eq.2can then simply be applied to the static output of this Shpeer-oyprrerwork, protecting a set of timestep-specific weights per task without the need increase the regularization budget.
CONCLUSION
Our work advances the CL community in three ways. First, by systematically evaluating the performance of established CL methods when applied to RNNs, we provide extensive baselines that can serve as reference for future studies on CL with sequential data. Second, we use theoretical arguments derived from linear RNNs to hypothesize limitations of weight-importance based CL in the context of recurrent computation, and provide empirical evidence to support these statements. Third, derived from these insights, we suggest that an approach based on hypernetworks mitigates the stability- plalstityy dilemma, and show that it outperforms well as real- world data. Finally, our work discusses several future improvements and directions of CL approaches for sequential data.
methods on synthetic as
ACKNOWLEDGEMENTS
This work was supported by the Swiss National Science Foundation (B.F.. CRSII5-173721 and 189251). 3, ETH project funding (B.F.G. ETH-20 19-01) and funding from the Swiss Data Science Center (B.FG, C17-18, J. O. P18-03). We especially thank João Sacramento for insightful discussions, ideas and feedback. We would also like to thank Nikola Nikolov and Seijin Kobayashi for helpful advice and James Runnalls for proofreading our manuscript.

---

Published as a conference paper at ICLR 2021
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3563-3593, 2014.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3366-3375, 2017.
Bernard Ans, Stéphane Rousset, Robert M. French, and Serban Musca. Self-refreshing memory in artificial neural networks: learning temporal sequences without catastrophic forgetting. Connection Science, 16(2):71-99) 99, Jun 2004.
Nabiha Asghar, Lili Mou, Kira A. Selby, Kevin D. Pantasdo, Pascal Poupart, and Xin Jiang. Progres- sive memory banks for incremental domain adaptation. In International Conference on Learning Representations, 2020.
Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv preprint arXiv: 1:Xiv:4411.76111 2014.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi. Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):224018, 2019.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2Net: Accelerating Learning via Knowledge Transfer. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Kyunghyun Cho, B van Merrienboer, Caglar Gulcehre, F Bougares, H Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Conference Empirical Methods in Natural Language Processing (EMNLP 2014), 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances Neural Information Processing Systems 28, pages 2980-2988. Curran Associates, Inc., 2015.
Robert Coop and Itamar Arel. Mitigation of catastrophic interference in neural networks using fixed expansion layer. 2012 IEEE 55th International Midwest Symposium on Circuits and Systems (MWSCAS), pages 726-729. IEEE, 2012.
Robert Coop and Itamar Arel. Mitigation of catastrophic forgetting in recurrent neural networks using a fixed expansion layer. In The 2013 International Joint Conference on Neural Networks (IJCN), pages 1-7. IEEE, 2013.
Andrea Cossu, Antonio Carta, and Davide Bacciu. Continual Learning with Gated Incremental Memories for sequential data processing. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN 2020), April 2020a.
Andrea Cossu, Antonio Carta, and Davide Bacciu. Continual learning with gated incremental memories for sequential data processing, 2020b.
Edwin D. de Jong. Incremental Sequence Learning. arXiv, Nov 2016.
Misha Denil, Babak Shakibi, Laurent Dinh, Mare' Aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning. In Advances in neural information prucessing systems, pages 2148- 2156, 2013.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties.
10

---

Published as a conference paper at ICLR 2021
Lea Duncker, Laura Driscoll, Krishna V Shenoy, Maneesh Sahani, and David Sussillo. Organizing recurrent network dynamics by tosk-compttation to enable continual learning. Advances in Neural Information Processing Systems, 33, 2020.
Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179 211, 1990.
Sebastian Farquhar and Yarin Gal. A unifying bayesian view continual learning. Bayesian Deep Learning Workshop at NeurIPS, 2018.
Robert M. French. Using Pseudo-Recurrent Connectionist Networks to Solve the Problem of Sequential Learning. ResearchGate, Feb 1970.
Robert M. French. Semi-distributed Representations and Catastrophic Forgetting in Connectionist
Networks. Connection Science, 4(3-4):365-37) Jan 1992.
Robert M. French. Dynamically Constraining Connectionist Networks to Produce Distributed, Orthogonal Representations to Reduce Catastrophic Interference. ResearchGate, Aug 1994.
Jort F. Gemmeke, Daniel P. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. IEEE ICASSP 20/7, New Orleans, LA, 2017.
Alex Graves, Greg Wayne, and Ivo Danihelka. 1xXiv:410..444 2014.
Neural turing machines. arXiv preprint
Stephen Grossberg. Conseiousness CLEARS the mind. Neural Networks, 2009):0400-1553, Nov 2007.
Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio. Memory augmented neural networks with wormhole connections. arXiv preprint 170100770.. 2017.
David Ha, Andrew Dai, and Quoe Le. Hypernetworks. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pages 1135-1143, 2015.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Tch, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv: 1906609200000 2019.
Benjamin Heinzerling and Michael Strube. Sequence tagging with contextual and non-contextual subword representations: A multilingual evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 273-291, Florence, Italy, July 2019. Association for Computational Linguistics.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253: 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997a.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1-42, January 1997b.
Ferenc Huszár. Note on the quadratic penalties elastic weight consolidation. Proceedings of the National Academy of Sciences, (15(11):22466EE3) March 2018.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In International Conference Learning Representations, 2018.
Ronald Kemker, Mare McClure, Angelina Abitino. Tyler L Hayes, and Christopher Kanan. Mea- suring catastrophic forgetting in neural networks. Thirty-second AAAI conference on artificial intelligence, 2018.
11

---

Published as a conference paper at ICLR 2021
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings. 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan. Tiago Ramalho, Agnieszka Barnskkaaaa Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetring in neural networks. Proceedings of the National Academy of Sciences, 114(13)33521-33 March 2017a.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. U.S.A., 114(13)3521-3356 Mar 2017b.
Germán Kruszewski, lonut-Teodor Sorodoc, and Tomas Mikolov. Class-agnostic continual learning of alternating languages and domains. arXiv preprint arXiv:2004.033444 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(1) 96(11):2278-2324 1998.
Sungjin Lee. Toward Continual Learning for Conversational Agents. arXiv, Dec 2017.
Yuanpeng Li, Liang Zhao, Kenneth Church, and Mohamed Elhoseiny. Compositional language continual learning. In International Conference on Learning Representations, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40 40(12):2935-2247 2017.
Guangyi Lv, Shuai Wang, Bing Liu, Enhong Chen, and Kun Zhang. Sentiment classification by leveraging the shared knowledge from a sequence of domains. In International Conference on Database Systems for Advanced Applications, pages 795-811. Springer, 2019.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computa- tion, 4(3):488 472, 1992.
Avinash Madasu and Vijjii Anvesh Rao. Sequential domain adaptation through elastic weight consolidation for sentiment analysis. arXiv preprint vrXiv:2007.01189 2020.
Nicolas Y Masse, Gregory D Grant, and David Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, (11444): 2018.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: investigat- ing the continuum from catastrophic forgetting to age-limited learning effects. Front. Psychol., 4, Aug 2013.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard F. Turner. Variational continual learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Track Proceedings, 2018.
Joakim Nivre, Carie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. Universal dependencies vl: A multilingual treebank collection. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi. Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may 2016. European Language Resources Association (ELRA).
12

---

Published as a conference paper at ICLR 2021
Aaron Van Oord, Nal Kalchbrenner. and Koray Kavukcuoglu. Pixel recurrent neural networks. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1747-1756, New York, New York, USA, 20-22 Jun 20.6a. PMLR.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03490 2016b.
Alexander Ororbia, Ankur Mali, C Lee Giles, and Daniel Kifer. Continual learning of recurrent neural networks by locally aligning distributed representations. IEEE Transactions on Neural Networks and Learning Systems, 2020.
German I. Parisi, Ronald Kemker, Jose L.. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, May 2019.
Daniel Philps, Artur d' Avila Garcez, and Tillman Weyde. Making good on Istms unfulfilled promise. arXiv preprint arXiv: 191.:499 2019.
Barbara Plank, Anders Søgaard, and Yoav Goldberg. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412-418, Berlin, Germany, August 2016. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31 International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1278-1286, Bejing, China, 22-24 Jun 2014. PMLR.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, ttss/600.04677. 2016.
Monika Schak and Alexander Gepperth. A Study on Catastrophic Forgetting in Deep LSTM Networks. SpringerLink,, pages 714-728, Sep 2019.
Jürgen Schmidhuber. 'self-referential" weight matrix. In International Conference on Artificial Neural Networks, pages 446-450. Springer, 1993.
Jürgen Schmidhuber. Learning to Control Fast- Weight Memories: An Alternative to Dynamic Recurrent Networks. Neural Comput., 4(1):131-139, Jan 1992.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernel principal component analysis. In International conference on artificial neural networks, pages 583-588. Springer, 1997.
M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673-268, 1997.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska- Barrkksska Yee Whye Teh, Razvan Pascanu. and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4528 4537, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual Learning with Deep Generative Replay. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 2990-2999. Curran Associates, Inc., 2017.
13

---

Published as a conference paper at ICLR 2021
Jasper Snock, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D Sculley, Joshua Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pages 13969-13339 13990, 2019.
Shagun Sodhani, Sarath Chandar, and Yoshua Bengio. Toward training recurrent neural networks for lifelong learning. Neural computation, 32(1):1-35, 2020.
Joseph Suarez. Language modeling with recurrent highway hypernetworks. I. Guyon, Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in
Neural Information Processing Systems 30, pages 3267-3276. Curran Associates, Inc., 2017.
Brian Thompson, Jeremy Gwinnup. Huda Khayrallah, Kevin Duh, and Philipp Koehn. Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation. ACthoohology pages 2062-2068, Jun 2019.
Ben Tsuda, Kay M. Tye, Hava T. Siegelmann, and Terrence J. Sejnowski. A modeling framework for adaptive lifelong learning with transfer and savings through gating in the prefrontal cortex. bioRxiv, page 2020.03.1 11 119987577 Mar 2020.
Gido M. van de Ven and Andreas S. Tolias. Generative replay with feedback connections general strategy for continual learning. arXiv preprint arXiv:1809, 1063,, 2018
Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. arXiv, Apr 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learn- ing recurrent networks with long term dependencies. In Proceedings of the 34th International Conference on Machine Searning-Volume 70, pages 3570-3578, JMLR. org, 2017.
Thomas Wolf, Julien Chaumond, and Clement Delangue. Continuous learning in a hierarchical multiscale neural network. arXiv preprint /805. 057558 2018.
Guangyu Robert Yang, Madhura R Joglekar, Francis Song, William Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297-306, 2019.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference Learning Representations, 2018.
Wojeiech Zaremba and Ilya Sutskever. Learning to execute, 2014.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual Learning Through Synaptic Intelligence. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17 17, pages 3987-3995, JMLR.org, 2017.
14


Reviews:
[{"review_id": "yvG2fdeCsSO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros: Most work on continual learning addresses only feedforward networks. This paper\nprovides apparently the first systematic discussion and comparison of CL methods for RNNs\nThereby it provides an important service to the community. The material is presented thoroughly in the Suppl. \n\nCons:\nProvides on the other hand less algorithmic innovation. In particular, it focuses on methods related to (Oswald et al., 2020), a paper that was accidentally omitted from the reference list, but apparently is this ICLR 2020 paper that contains related material:\nvon Oswald, J., Henning, C., Sacramento, J., & Grewe, B. F. (2019). Continual learning with hypernetworks. arXiv preprint arXiv:1906.00695. ICLR 2020.\n\nI am uncertain about the generalizability of results that were demonstrated for the chosen benchmark tasks. In particular, the conceptually important distinction between challenges arising from working memory load and sequence length is tested by variations of the copy task with padded inputs, where relevant and irrelevant input bits are distinguished in a very simple way that is hardly met by real-world scenarios. \n\nThere may also be differences arising from different types of RNNs, and it is not clear to me to what extent one can make conclusions about all of them by testing on just one type.\n\nI tend to vote for accept.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RPpv2sMbfTx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a systematic evaluation of the performance of different CL methods on RNN. The study suggests that high working memory requirements increase difficulty of learning new tasks, while the average length of input sequence is not strictly related to the difficulty of learning new tasks. The author proposes to overcome this problem by using a hypernetwork-based CL approach, which shows promising results in the experiments.\n\nStrength:\n* The paper provides extensive study to compare different continual learning methods.\n* The conclusion is well supported by analysis of intrinsic dimension and performance on different tasks.\n* The paper is well written and easy to follow\n\nWeaknesses\n* It would be interesting to see results on more realistic tasks, like sentence classification.\n* The conclusion on working memory requirement didn’t consider the possibility of knowledge sharing between tasks. For example, two complicated tasks may share a common sub-network that is essential for solving both tasks. Such that, in an ideal situation, the model doesn't need to allocate large amounts of extra resources to learn the second task. It would be interesting to see how different CL methods can reuse knowledge learnt from previous tasks.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2x05K3R-8TX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors do an evaluation of the application of weight-importance continual learning methods to recurrent neural networks (RNNs). They draw out the tradeoff between complexity of precessing and just remembering (working memory) in terms of the applicability of these weight importance methods. They also provide some theoretical interpretation based on stying linear RNNs.\n\nOverall, I vote for accepting this paper because the work is well-motivated, thorough, and provides useful insights. My major concerns are listed below. \n\nStrengths:\n\n+ The paper is very well written, and the motivation, methods and inferences are quite clearly described. The main question the authors are considering is very clear.\n+ The results around use of existing continual learning methods to RNNs is timely and relevant.\n+ The insight into the tradeoff between complexity of processing and working memory requirements and its effect on the ability of the network to continually learn is very interesting. Similarly the fact that hypernetwork based approaches work better than other approaches most of the time is useful.\n+ The analysis of the above tradeoff using a linear RNN is also interesting since it provides a nice intuition for why the tradeoff exists.\n\nWeaknesses:\n\n- The motivation and conclusions from the ssMNIST task is not very evident and the tasks doesn't seem to make a clear point. \n- Readability of some parts of the paper depend very heavily on the supplement, and the paper itself doesn't stand by itself. For example, the linear RNN analysis, description of some tasks (I had to read the supplement to actually understand the permuted copy and the pattern manipulation tasks). As an aside, I hope the authors submit an extended version to an appropriate venue, because I think many of the results and discussions relegated to the supplement seem quite interesting and relevant for the community (e.g. the task-conditional processing).\n- An analysis of why hypernetworks perform better would be interesting. So would have been some proposals for methods designed specifically for CL with RNNs.\n\nMinor:\n\nThe main text doesn't clearly mention that vanilla RNNs are used in section 5.1", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I agree with the reviewers, and I find the careful analysis of CL approaches relying on regularization for RNN useful and insightful. I do feel that a lot of the interesting content is still in the appendix (from a quick skim and looking at the plots in the appendix) but I think something like this can potentially be unavoidable. 

I do like the separation between sequence length and memory requirements. I think making observations about different types of recurrent architectures is hard, but I think the paper does a good job to raise some interesting questions. 

A note that I would make (that I haven't seen raised through a quick look in the paper) is that is not clear how the Fisher Information Matrix should be computed in case of a recurrent model (which is a problem in general). E.g. a typical thing is to compute it as for a feed-forward model (using the gradients coming from BPTT) which is feasible computationally, but actually that is problematic as you first sum gradients before taking their outer-product rather than summing the outer-products corresponding of the different terms in the gradient. I'm wondering if that plays a role here as well.

Overall I think the paper does careful analysis and ablation studies and raises some interesting observation of how one should approach CL algorithms for RNN models.


Decision: Accept (Poster)