Continual Learning with Gated Incremental Memories for Sequential Data Processing
Authors: Andrea Cossu
Antonio Carta
Davide Bacciu
Venue:  (2020)
Keywords: continual learning
recurrent neural networks
progressive networks
gating autoencoders
sequential data processing

Abstract: The ability to learn over changing task distributions without forgetting previous knowledge, also known as continual learning, is a key enabler for scalable and trustworthy deployments of adaptive solutions. While the importance of continual learning is largely acknowledged in machine vision and reinforcement learning problems, this is mostly under-documented for sequence processing tasks. This work focuses on characterizing and quantitatively assessing the impact of catastrophic forgetting and task interference when dealing with sequential data in recurrent neural networks. We also introduce a general architecture, named Gated Incremental Memory, for augmenting recurrent models with continual learning skills, whose effectiveness is demonstrated through the benchmarks introduced in this paper.

Under review as a conference paper at ICLR 2020
CONTINUAL LEARNING WITH GATED INCREMENTAL MEMORIES FOR SEQUENTIAL DATA PROCESSING
Anonymous authors Paper under double-blind review
ABSTRACT
The ability to learn over changing task distributions without forgetting previous knowledge, also known as continual learning, is key enabler for scalable and trustworthy deployments of adaptive solutions. While the importance of continual learning is largely acknowledged in machine vision and reinforcement learning problems, this is mostly under-documented for sequence processing tasks. This work focuses on characterizing and quantitatively assessing the impact of catas- trophic forgetting and task interference when dealing with sequential data in re- current neural networks. We also introduce a general architecture, named Gated Incremental Memory, for augmenting recurrent models with continual learning skills, whose effectiveness is demonstrated through the benchmarks introduced in this paper.
INTRODUCTION
Continual Learning (CL) can be defined "the unending process of learning new things top of what has already been learned" (Ringl 2011).
A more formal definition centered around the computational aspects of CL algorithm is presented
in Lesort et al. (2019)
Definition 1 Given potentially infinite sequence of unknown distributions D = [Di, D2, Da....
where each D, = {X,, Yd) includes the input data X; and the target labels Y,. a Continual Learning algorithu AºT. is defined by the following signature:
YD, ED D,
AÇ" h4-1, TR4, M, 14-1,44-44 h4, M, >>
where TR, is the training set drawn from the corresponding data distribution Di, hi is the function that the model has learned after having seen all the training sets up to the i-th, M, is an external memory in which store patterns from TR; and t, is the label identifying the task.
In this paper, the notion of task is associated to a specific objective (e.g. learn to classify MNIST digits). An inpul distribution, instead, generates data related to a particular task (e.g. subsets of MNIST digits). Hence, each task is associated to multiple input distributions (also called subtasks). In the CL scenario. learning model is required to incrementally build and dynamically update internal representations as the distribution tasks varies across its lifetime. Ideally, part of such in- ternal representations will be general and invariant enough be reusable across similar tasks, while another part should preserve and encode task specific representations. Unfortunately, continuous plasticity of internal representations under drifting task distributions is widely known to suffer from negative interference between the tasks that are incrementally presented to the model, yielding to the well known stability-plastiiity dilemma (Parisi et al. 2019; Zhou al. 2012) of connectionist models. The result of this being models which catastrophicall forget previously acquired knowl- edge (Robins 1995; French 1999) as new tasks become available, that one of the main challenges faced by CL. algorithms.
While current trends CL put particular emphasis on computer vision applications or reinforce- ment learning scenarios, sequential data processing is rarely taken into consideration (see Sec- tion Sequential data processing in CL involves a stream D of distributions
sequences D3 ( 1, 3........ where X, sequences of vectors and Y, are the corresponding targets. Machine Learning (ML), sequential data processing plays a fundamental role impor- tant fields like Natural Language Processing (NIP),, signal processing, bioinformatics and many

---

Under review as a conference paper at ICLR 2020
others. In this context, Recurrent Neural Networks (RNNs) provide an effective means to deal with sequential data, thanks to their adaptive memory. that is the ability of developing neural represen- tations that capture the history of their imputs. Learning proper memory representations is major challenge for RNNs. Especially in a CL setting, where RNNs have also to deal with drifts in task distributions which can greatly affect their capability of developing robust and effective memory representations.
In this work, we provide threefold contribution to the discussion concerning CL in sequential data processing. First, we introduce benchmarks for the evaluation of CL skills by adapting three sequen- tial datasets to CL scenarios. These benchmarks can be easily tuned to varying degrees of complexity and hardness, and they have been devised to provide an assessment that is not tailored to a specific application domain (e.g. NLP, RL or computer vision). Rather, they aim at evaluating the intrin- sic effectiveness of recurrent models in CL. setting. Second, we define new dynamic approach that imbues RNN architectures with CL skills by incrementally adding new modules to capture the shift in task distribution while avoiding catastrophic forgetting. This architecture, named Gated In- cremental Memory, leverages gating autoencoders to automatically recognize input distributions in order avoid any explicit supervision concerning their identity at inference time. Third, eval- uate baseline RNN models on the proposed benchmarks, comparing their performance against our enhanced architecture. Source code and full details regarding the experimental setup are provided as supplemental material, to ease replicability of the results and allow other researchers a direct and fair comparison of their own CL models on our benchmarks.
The results of our empirical analysis confirm the ad advantages of using our enhanced architecture over standard recurrent models. Such advantages are particularly clear when testing enhanced architec- ture on old distributions, since it successfully prevents forgetting, when learning sequences with increasing length (in the Copy Task) and when the input sequences end with a common suffix (in the Sequence Classification SSMNIST Task). These results highlight some of the key differences between feedforward and recurrent CL techniques, and pinpoint the need for solutions specifically designed for recurrent architectures.
2 RELATED WORK
CL literature mostly focuses on computer vision and reinforcement learning applications. with ap- proaches ranging from regularization methods (Kirkpatrick et al., 2017 Zenke al. 2017), to dual
models (Hinton & Plaut 1987 French 1997). to dynamic architectures (Rusu et al. 2016 Yoon al. 2018). The work of Coop Arel (2013) is one of the first attempts to apply RNNs in CL scenarios. In their paper. they employed a RNN with Fixed Expansion Layer (FEL) (Coop & Arel 2013) and trained the resulting model to reconstruct auto-associative binary sequences. The popular Copy task (Graves et al., 2014) and Sequential Stroke MNIST dataset (de Jong. 2016) have been used by (Sodhani et al., 2018) to test the capabilities of dynamic recurrent models in presence of increasingly longer sequences. By using dynamic external memories, Asghar et al. (2018) success- fully applied RNNs to CL on NLP tasks with MNLI dataset. Finally, Kemker et al. (011)) applied feedforward networks to the Audioset dataset (Gemmeke et al. [017]] in CL setting. Our dynamic RNN architecture is inspired by Progressive networks (Rusu et al. (0166), a CL approach popular in the feedforward domain that deals with new distributions by dynamically adding new hidden units to feedforward layer and connecting them to the existing hidden units through lateral connections. The gating autoencoders are adapted to sequential scenarios from Aljundi et al. (2017). where they are first introduced for feedforward models.
GATED INCREMENTAL MEMORIES FOR CONTINUAL LEARNING WITH RECURRENT NEURAL NETWORKS
In this section, we introduce Gated Incremental Memory (GIM), a novel CL architecture designed for recurrent neural models and sequential data. In particular, we show how GIM can be obtained by combining together recurrent version of the Progressive network (Rusu al. 2016) and a set of gating autoencoders (Aljundi et al. 2017) in order to avoid, at test time, any explicit supervision about subtask label. We denote the recurrent version of the Progressive network as Augmented models (A-LMN and A-LSTM), while the entire architecture, including both augmented archi- tectures and gating autoencoders, called GIM. To highlight the generality of our approach,
2

---

Under review as a conference paper at ICLR 2020
detail the application of GIM to two substantially different recurrent neural models from literature (GIM-LMN, GIM-LSTM). In the following, we indicate an entire sequence with bold notation (e.g. x). while single vector with plain formatting (e.g. xz).
3.1 RECURRENT NEURAL NETWORKS
The proposed approach is independent on the underlying recurrent architecture. In this work, we focus our study two different classes of RNNs, using either gated and non-gated approaches. Gated models, like LSTM (Hochreiter Schmidhuber 1997) and GRU (Cho et al. (014), leverage adaptive gates enable selective memory cell updates. In our analysis, representative of gated architectures, given its popularity in literature and its state-of-the-art perfor- mances in several sequential data processing benchmarks. Non-gat Mon-gated approaches rely on different mechanisms to solve the vanishing gradient problem, like parameterizing recurrent connections with an orthogonal matrix (Mhammedi al. 2017). 7). our analysis, we consider the Linear Memory Net- work (LMN) (Bacciu et al. (2199)
consider LSTM
representative of non-gated approaches. LMNs leverage conceptual separation between nonlinear feedforward mapping computing the hidden state hy and linear dynamic memory computing the history dependent state h." Briefly, in formulas:
h = o (Wah, x4 + With (G-1)
1 = Whm h4 + Wm 1
where It, he and ht" are the input vector, the functional component activation and memory state at time L. The memory state hT" is the final output of the layer, which is given input to subsequent layers. It is then useful to study how both models behave in CL environments and how their different memory representations respond to phenomena like drifting tasks distribution, eventually resulting in catastrophic forgetting.
3.2 GATED INCREMENTAL. MEMORY (GIM)
GIM represents general class of dynamic, recurrent architectures that be built top of any recurrent model. GIM relies on progressive memory (Rusu et al. (016) extension of the underlying RNN model that, for the sake of this work, are LSTMs or LMNs. We also leverage set of gating autoencoders, one for each distribution, to automatically select the module which has the best match the current input distribution. Figures] and provide an overview of the entire GIM architecture during training and test. The main component of GIM is the RNN module. As soon as a new distribution arrives, a new RNN module added on top of the existing architecture and connected to the previous one. The exact inter-modules connections are slightly different depending on the underlying recurrent model. When using the A-I.STM, at each timestep. the new module takes as additional input the current hidden state of the previous module. In the A-LMN, the hidden state h' of the new module computed by taking as additional input the concatenation of both previous module's memory hi" and functional activation ht- In order to prevent forgetting, when new module is attached to the existing architecture, the previous module's parameters are frozen and no longer updated. Therefore, each module becomes an expert of its own domain. The input vector & at each time step / is forwarded to all modules. Each module has its own output layer and, during training, the last module added the network is used to generate the final output y. The forward pass for an A-LMN with N modules can then be formalized as follows:
1T loT.1 = LMN, (x, AT.1) WT hT LMN, ([x;
2,..., N
where hef; and lr,; are the memory states and functional activations of module i after having seen the entire input sequence x of length 7 and [ is the concatenation operator between vectors. The aggregated output y is computed by passing the final memory state h7, through linear layer. The A-LSTM forward pass follows the same process, substituting the final hidden state by to the
3

---

Under review as a conference paper at ICLR 2020
SubTask
SunTask
SubTesk
AALSTM
LSTTA1
A-LSTM2
ALSTI
ALLST14
A-LSTA
AE1
AEZ
Figure Training of GIM-LSTM on subtasks.
memory state and functional activations of A-LMN.
Each module of the augmented architecture is associated single LSTM autoencoder (Srivastaval iet al. 2015), which is a to reconstruct the input The training process involves only the autoencoder associated to the last added module. The loss function used to train the autoencoders is the reconstruction error, that is the mean squared error (MSE) applied element-wise to the input and reconstructed sequence. The Augmented models are trained by using standard Cross Entropy loss for classification tasks. The following equations provide formal description of the GIM-LMN training when tackling the i-th distribution (the GIM-LSTM behaves in the same way):
model trained jointly to the corresponding module
y A-LMN(x)
X (x)
where AE; identifies the autoencoder associated to module i. The input is sampled from the i-th distribution. The output and the reconstruction X are then used in the corresponding loss functions (Cross Entropy and MSE, respectively) train the A-LMN and the autoencoder by gradient descent. At inference time, the input x is forwarded to all autoencoders and the module associated to the one producing the minimum reconstruction error is selected to produce the final output, as described by the following equations:
x; (x), NV
k = arg min MSE(X,,
y = LMN (x).
The algorithms describing the forward pass for both A-LMN and A-LSTM and the training and test procedure GIM are provided in Appendix(A.1.
GIM, like Progressive networks, is capable of learning multiple distributions without being affected by forgetting. addition, GIM overcomes one of the major drawbacks of Progressive networks (Rusu al., 2016): it does not require explicit knowledge about input distributions at test time, since gating autoencoders are able to autonomously recognize current input and use the appropri- ate module for output. GIM also simplifies the inter-modules connections: while in Progressive networks they are feedforward networks, created between a module and all the next ones, GIM employs a simpler, non adaptive, version and connects only adjacent modules. Therefore, the con- nections do not contribute to the total number of adaptive parameters, which scales linearly with the number of modules. Finally, as described in Aljjundii et al. (2017). is possible to compute a related- ness measure between subtasks using the autoencoders reconstruction errors. Given a new subtask Tk. an old subtask T. and their associated autoencoders, the relatedness computed by using the reconstruction errors Er, and E,, of the autoencoders measured validation data for subtask Tę. Formally:
Rel Ta) = (
Er. -Er.-Er
Erz
Er.
Notice that this definition is not entirely equivalent to the one reported in Aljundi et al. (2017). Since we assume that each autoencoder achieves the lowest error on its own training data, we have Er. > Erg. Aljundi et al. (2017) normalize relatedness by having Er, in the denominator of (T). in place of E,,- In our experiments, we noticed that this led to large negative values for the relatedness, which not particularly insightful. Therefore, we propose different normalization coefficient, using the largest reconstruction error E,, in place of Er,-
4

---

orthogonal Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
One of the main contributions of this pa- per is the definition of set of learn- ing tasks that can be used as standard benchmarks for CL algorithms on se- quence processing problems. We be- lieve that the current literature is lack- ing a set of simple, application-agnossee benchmarks that can provide a fair com- parison and evaluation of different ap- proaches with a minimal experimental setup. The final appendices provide all the details regarding the experimental setup needed reproduce the results. The experiments rely on three datasets: the Copy dataset (Graves et al., 22144). the Sequential Stroke MNIST' dataset (de Jong 2016) and the Audioset dataset (Gemmeke al., 2017). We adapted each dataset to classification tasks in CL environments. We split each task in multiple subtasks, associating to each one a subset the possible classes. The complexity of each task can be adjusted and controlled by the experimenter. The tasks are not designed for a specific application domain, but instead, they general enough to stress the main difficulties of RNN training and CL scenarios. This is particularly useful to test the limitation of each model and allows great degree of flexibility. The source code is available online and we plan to release more datasets in the future
Test
MLSTM1 A-L5MM2 A-LTM--3
AE1
AE2
Figure 2: Test of GIM-LSTM. Input x is presented to all autoencoders. In this case, AE2 obtains the minimum re- constructione error. Hence, the second module (dashed line) is chosen to produce the output. The input is propa- gated up until the second module.
4.1 COPY TASK
The Copy task (Graves et al. 2014) con- sists in the reconstruction of sequences of random binary vectors. This task could be considered a simple CL ex- ample, since the data distribution for each subtask only difffers for the se- quences' length. However, generaliz- ing to long sequence lengths is a dif- ficult problem for traditional recurrent architectures (Graves et al.] 2014) and therefore the same techniques used in more standard CL settings can be used
Table I: Accuracy (in percentage) on Copy task, mea- sured at the end of each subtask training. Column labels indicate sequence length for each subtask.
%
5
12
19
26
33
40
A-LMN
100
100
100
100
95
95
A-LSTM
100
100
100
95
90
85
LMN
100
100
100
78
LSTM
100
97
85
78
to solve this problem by incrementally increasing the memory capacity of the final model (Sodhani et al. 2018). Furthermore, it is useful to evaluate the memory capacity of the Augmented archi- tecture compared against the standard RNNs. In the Copy Task, LMN and A-LMN initialization of the memory component (see Appendix A.2 for more details). Gating autoencoders have not been employed in this task, since its objective is exactly the reconstruction of the input se- quence. The subtasks are organized in curriculum (Bengio et al. (209) following sequence length, from sequences with 5 + 2 vectors to sequences with 26 + 2, steps of 7 vectors subtasks total). Results, summarized in Table [ show that LMN performs slightly better than L.TTM (using more hidden units can bridge this gap). This is expected since orthogonal models typically outperform LSTMs pure memorization problems (Vorontsov et al. 2017). Augmented models outperformed both them, with A-LMN converging faster than A-L.TM. Augmented models have been trained also with longer sequences (up to length 40). without sensible drops in accuracy. All models ob- tained an accuracy on par with a random classifier when tested on previous subtasks (a well- well-known problem for RNNs). The Copy Task highlights the behavior of recurrent CL models trained grow- ing sequence lengths and proves their effectiveness when compared to standard (non CL oriented) recurrent models. The use of dynamic modules in A-LMN and A-LSTM positively contributes to the memory capacity, justifying the additional cost required by the Augmented architectures.
Public link redacted to preserve anonymity, reviewers find source code in the supplementary material
5

---

Under review as a conference paper at ICLR 2020
Tokens
1
1
2
ECO
E0D
E00
EDD
EDD
E0D
SubTask 1
Class 44
EOD
Sequence classification with 5 digit classes Additional
Digit Segment
Digits
2
Single digit sequence
EDD
EGD
EGD
EDD
EDD
EOD
(a) single digit sequence SubTask 2 is classified into its cor-
Class 44
responding digit class.
The vertical segments
Sequence classificattin digit classes
Additional Digtt Segment
correspond to single input vector. The one eod.
Digits
(b) sequence of digits (including an additional segment), is classified into its corresponding class. The tokens are uniquely associated the class index 44. The digits corresponding to the tokens are explicitly showed for each subtask. The last token common all the sequenees belonging to the subtask and its length affects the learning behavior, showed in Table
Figure 4: SSMNIST Tasks description
4.2 SSMNIST TASK
Sequential Stroke MNIST (SSMNIST) (de Jong 2016 is variation of the popular MNIST dataset of handwritten digits. In this version, each digit is composed by sequence of pen strokes, where each pen stroke represented by two coordi- nates (I: and y, scalars), digit bit eod. We devised two different kind of experiments, aimed both evaluating gating autoencoders capabilities and GIMs classification performances.
A-LSTM accuracy
A-LSTM-1
A-L5TM-2
end of stroke bit eos and end of
0.6
A-STTM-3
0.4
In Sodhani et al. (2018), the SSMNIST dataset has already been employed in CL environments. However, the authors build the CL strategy by progressively increasing the se- quence length (e.g. by concatenating together multiple digits). Here, we chose a different approach by limiting each subtask to fixed set of digit classes. Our approach is consistent with the literature about CL in computer vision, where the different subtasks designed over different object classes within the images (Maltoni & Lomonaco 2019: Lopez-Paz & Ranzato 2017)
0.2
0.0
0
10
15
Figure 3: A-LSTM learning curve
on Sequence Classification. Final concatenated length of 12.
4.2.1 SINGLE DIGIT SSMNIST
The Single Digit task (Fig. 4a requires the models to predict the digit class associated to the input sequence. The task has been partitioned into subtasks, each of which selects input patterns belonging to 2 digit classes. There is no intersection in the digit classes between different subtasks. Classification performance easily reached top (100%) accuracy for all models. The only exception is the LMN, which obtained accuracy par with random classifier subtask 3. LMN and LSTM experienced complete forgetting of old knowledge and were not able to address any the previous subtasks. GIMs. instead, completely prevent forgetting through the use of gating autoencoders, that were always able to distinguish between the subtasks at test time. There was no clear difference between GIM-L and GIM-LMN performances in this task. GIMs useful in this context

---

Under review as a conference paper at ICLR 2020
Table 3: The table reports the values of the final digit segment length that produce, for each model, the behavior specified by the top row. L (Learning) means that the model learns all the subtasks, NL (No Learning) means that the model does not learn any subtask and PL (Partial Learning) means that the model learns only one subtask.
L
NL
PL
GIM-LMN
S 22
35
GIM-LSTM
$ 11
> 14
12 - 13
LMN
16
24
22
LSTM
1
12
only achieve top accuracy on all subtasks (e.g. GIM-LMN against LMN subtask 3), but also to prevent forgetting, Final test performances are summarized in T
4.2.2 SEQUENCE CLASSIFICATION SSMNIST
The Sequence Classification task (Fig.
4b partitions the 10 digit classes in 3 subtasks of 3 digit classes each, without intersections (see 4).
Table 2: Accuracies (in percentage) on test sets after
training on all subtasks on Single Digit SSMNIST.
Sequences are composed by tokens (either O, 1 or 2), each one associated to specific digit, depending on the sub- task. The association between tokens and digits for each subtask described by Table 4 Sequences are grouped into 100 different classes, according to the sequence of tokens they represent. As an example, class 44 consists of tokens (1,0, 2,0,), The complete list of classes is presented in Appendix(A.4) The models are trained to classify each sequence into its corresponding class. This task is more dif- ficult than the Single Digit, since the models have first to decode the digits sequence from the input sequence (e.g. (4, 6, 4, 8, 6)), then associate such sequence the appropriate class (44) by inferring the correct tokens, without any intermediate help.
%
T1
T2
T3
T4
T5
GIM-LMN GIM-LSTM LMN
100
100
100
100
100
100
100
100
100
100
0
0
100
LSTM
0
0
100
Gating autoencoders were still able recognize the subtask, even if, with respect the Single Digit, this task increases the number of digit classes per subtask (from 2 to 3) and it is based on longer sequences (made by multiple digit sequences). As in the Single Digit, GIMs successfully overcame catastrophic forgetting, while LMN and LSTM were not able to address previous subtasks.
By adding a common digit segment at the end of each sequence, the classification problem becomes much more difficult. The common suffix between all sequences requires larger memory capacity to distinguish the inputs. Depending on the length of the final segment, different behaviors emerged, as showed in Table Some lengths completely prevented learning (No Learning column), while some others did not affect the performances (Learning column), still obtaining an accuracy above 95%. The most interesting behavior (Partial Learning column) has been observed for narrow range of lengths. In this case, learning is prevented on the first subtasks. On the third subtask, the models suddenly achieved the top accuracy, showed in Fig- The Partial I Learning behavior highlights how the use of GIMs enables learning of more articulated input distributions, compared baseline RNNs.
4.3 AUDIOSET TASK
Audioset (Gemmeke et al. 2017) is collection of annotated audio events, extracted from 10 seconds audio clips and organized hierarchically in classes. The objective is the classification of sound from its audio clip source, embedded through a VGG-acoustic model into 10 vectors, one per second. In order to implement CL scenario, we selected 40 audio classes and split them among 4 subtasks (10 classes per subtask). We selected the 40 classes according to the procedure outlined by Kemker et al. (2018). Since their classes have not been published. randomly selected them from the superset resulting from their preprocessing pipeline. See Appendix [A.5] for a complete description. Results showed that gating autoencoders were perfectly capable of detecting the
7

---

Under review as a conference paper at ICLR 2020
correct data distributions at inference time. Therefore, GIMs successfully prevented forgetting: they obtained a final test performance comparable to the validation accuracy measured at the end of each subtask training. Standard RNN architectures, on the contrary. were prone to catastrophic forgetting, achieving 0% accuracy on previous subtasks. 3 summariizes these results.
Audioset data has already been used in literature assess CL. Kemker et al. (2018). However, the work by Kemker et al. (2018) focused on the task from a static perspective, relying on the use of feedforward models only. Since the preprocessing step provides, for each audio clip, a sequence of fixed-size embeddings, it is possible to concatenate the vectors into a single large vector and feed it to the network. The sequential aspect of the task, however, completely lost. At the best of our knowledge, we are the first to tackle Audioset in CL scenarios with recurrent models. It is also important to notice that the task difficulty is increased when using recurrent networks, since the model is not able to see the input in its entirety (like in feedforward networks), but it has to scan it one timestep at a time.
4.4 RELATEDNESS ANALYSIS
In our experiments, large relatedness values are associated "heteroge- neous" distributions. We call heteroge- neous distributions the ones that gener- ate patterns from many different classes (as in Audioset) or from a complex mix- ture of few classes (as in Sequence Clas- sification SSMNIST). We observed that, in correspondence with heterogeneous distributions, autoencoders training con- verges larger values of reconstruc- tion error than the ones obtained with homogeneous distributions (as Single Digit SSMNIST). However, reconstruc- tion error, hence relatedness, does not tell anything about gating capabilities (i.e. the ability to select the appropriate module at test time). Autoencoders, in fact, always performed correctly in all exper- iments, preventing forgetting. A complete account of relatedness measure is provided Appendix
Tokens-to-Digits
1
T1
2
0
T2
4
T3
3 1
Table 4: This table associates to each subtask of Se- quence Classification SSMNIST (TI, T2, T3) the val- ues taken by the tokens (0, 1, 2) in the sequence classes. As an example, sequence (1, 0, 1, 2,0) (class 44) corre- sponds different multi digit sequences: (0, 20 0, 7, for subtask and (4, 6, 4, 8, 6) for subtask 2.
CONCLUSIONS
The main objective of this work is to draw attention on the problem of CL. for sequential data processing. We pro- posed set of benchmarks against which future models can assess their perfor- mances. These benchmarks test the main difficulties of the sequential and CL domains and their difficulty be carefully tuned. We introduced GIM, a recurrent CL architecture for sequential data processing, and showed that it con- sistently outperforms baseline RNNs on all the proposed benchmarks. The results support the claim that recurrent architectures need to be adapted to manage CL. scenarios and encourage future works towards an in-depth study of their behaviors. In the future, GIM could be applied to Reinforcement Learning tasks, where there more opportu- nities to compare performances with existing works. In order to provide a formal characterization of the learning process, it would be useful to apply existing metrics to better describe the type of tasks addressed by the model, in terms both of transfer learning opportunities and subtasks interference. New metrics can also be developed, specifically tailored to sequential learning problems.
Table 5: Accuracies (in percentage) on test sets after training all subtasks on Audioset.
% GIM-LMNN
T1
T2
T3
T4
61
65
53
50
GIM-LSTM LMN LSTM
69
60
47
0
0
0
43
0
42
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
R. Aljundi, P. Chakravarty, and T. Tuytelaars. Expert Gate: Experts. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 7120-7129, July 2017.
Lifelong Learning with a Network of
Nabiha Asghar, Lili Mou, Kira A. Selby, Kevin D. Pantasdo, Pascal Poupart, and Xin Jiang. Pro- gressive Memory Banks for Incremental Domain Adaptation. arXiv: 181 1.00239 [es], November 2018. arXiv: 1811 1811.00239.
Davide Bacciu, Antonio Carta, and Alessandro Sperduti. Linear Memory Networks. In Proceedings of the 28th International Conference Artificial Neural Networks (ICANN 2019), Lecture Notes in Computer Science. Springer-Verlat Verlag, September 2019.
Yoshua Bengio, Jrme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning ICML '09, pp- 1-8, Montreal, Quebec, Canada, 2009, ACM Press.
Kyunghyun Cho, Bart van Merrinboer, Dzmitry Bahdanau, and Yoshua Bengio. On the Properties of Neural Machine Translation: EncoderDecoder Approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/ URL httph:tp//// www aclweb. wwtp/anthooogy//.e--444
Robert Coop and Itamar Arel. Mitigation of catastrophic forgetting in recurrent neural networks using a Fixed Expansion Layer. In The 2013 International Joint Conference on Neural Networks (IJNN), pp. 1-7, Dallas, TX, USA, August 2013. IEEE.
Edwin D. de Jong. Incremental Sequence Learning. arXiv: 1611.03068 [cs], November 2016.
arXiv: 1611 1.03668.
Robert M. French. Pseudo-recurrent Connectionist Networks: An Approach to the 'Sensitivity- Stability' Dilemma. Connection Science, 94):3533-33 December 1997. ISSN 0954-0091.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128-135, April 1999. ISSN 1364-6613, 1879-307X.
Gemmeke, D. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio Set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, March 2017.
Alex Graves, Greg Wayne, and Danihelka. Neural Turing Machines. arXiv: 1410.5401 [cs], October 2014. arXiv: 1410.5401.
Geoffrey E Hinton and David C Plaut. Using Fast Weights Deblur Old Memories. Proceedings of the ninth annual conference of the Cognitive Science Society, pp. 177-186, 1987.
Sepp Hochreiter and Jurgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9: 1735-1780, 1997.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan. Measur- ing Catastrophic Forgetting in Neural Networks. Thirty-Second AAAI Conference on Artificial Intelligence, April 2018.
James Kirkpatrick. Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinnkk. Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521, March 2017.
Timothe Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Daz-Roo RazzRudr Continual Learning for Robotics. arXiv: 1907.00182 [cs], June 2019. 1907-001 00182.
arXiv:

---

Under review as a conference paper at ICLR 2020
David Lopez-Paz and Mare'Aurelic Aurelio Ranzato. Gradient Episodic Memory for Continual Learning. NIPS, 2017. arXiv: 1706.08840.
Davide Maltoni and Vincenzo Lomonaco. Continuous learning in single- ingele-incremenal-tteeee scenarios.
Neural Networks, 1 116:56-73, August 2019. ISSN 0893-6080.
Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections. In International Conference Machine Learning, pp. 2401-24499 July 2017. httphhh:://roceeeeeee
mlr. press/veevv 70 /hmammediiiii 1 7a html
German I. Parisi, Ronald Kemker, Jose L.. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 11::54-71, May 2019. ISSN
0893-6080.
Mark Ring. Recurrent Transition Hierarchies for Continual Learning: A General Overview. Work- shops al the Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
Anthony Robins. Catastrophic Forgetting; Catastrophic Interference; Stability; Plasticity; Rehearsal. Connection Science, 7(2):123-146, June 1995. ISSN 0954-0091, 1360-0494.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. arXiv: 1606.04671 [cs], June 2016. arXiv: 1606.04671.
Shagun Sodhani. Sarath Chandar, and Yoshua Bengio. On Training Recurrent Neural Networks for Lifelong Learning. arXiv: 1811.07017 [cs, stat], November 2018. arXiv: 1811.07017.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised Learning of Video Representations using LSTMs. ICML, 2015. arXiv: 1502.04681.
Eugene Vorontsov. Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and leam- ing recurrent networks with long term dependencies. In International Conference on Machine Learning, pp. 3570-3578, July 2017.
Jaehong Yoon, Eunho Yang. Jeongtae Lee, and Sung Ju Hwang. Lifelong Learning With Dynami- cally Expandable Networks. ICLR, pp. 11, 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual Learning Through Synaptic Intelli- gence. International Conference on Machine Learning, pp. 3987-3995, July 2017.
Guanyu Zhou, Kihyuk Sohn, and Honglak Lee. Online Incremental Feature Learning with Denois- ing Autoencoders. Proceedings of Machine Learning Research, 22:9, 2012.
A APPENDIX
A.1 GATED INCREMENTAL MEMORIES
Algorithms and outline the forward pass for A-LSTM and A-LMN architectures. Algorithm describes the training and test process for the GIM architecture.
Algorithm 1 A-LSTM Forward Pass
Require: A-LSTM with N modules, input x Ensure: N >
1: Initialize hidden state of module 1: h1,
2: hi, = A-LSTM, (x,
> Compute final state of module
3: for d 4 2, N do
4: Initialize hidden state of module has
5: X = [x; ha-1,fl
Concatenate hidden state with input D Obtain final state of module d
6: hd.f A-LSTM,(X, ha,i)
7: end for
8: Output final hidden state of last module: hN.f
10

---

Under review as a conference paper at ICLR 2020
Algorithm 2 A-LMN Forward Pass
Require: A-LMN with N modules, input
Ensure: >
1: Initialize memory state of module 1: MY.
2: hr h1, A-LMN, (T.)))
Compute final memory and functional state of module
3: for d + 2, . do
4: Initialize memory state of module ha,
5: x= x=xx h ha-1.f)
> Concatenate functional and memory state with input
6:
h ha, = A-LMN1 (x, h2,))
Compute final memory and functional state of module
7: end for
8: yx oWWN
Output of the final module
Algorithm Training and testing of LSTM Autoencoders, including output selection from Aug- mented architecture
Require: A sequence of distributions D = (D1, D2, D
Ensure:
| |>
1: lae
Create empty list
2: while new distribution Dj: is available do
3: LSTM LSTMaee init-autoencoder((
4: heeaa (...ppend (LSTMene, LSTM)ee))
B Append new autoencoder
5:
Initialize the hidden state of the encoder henc,
6:
for training batch x E D2 do
7:
h Me,, henc,)) hdee. LSTM (0, hene.f)
Compute encoder state
8:
b. Compute decoder state
9:
Wyhdec.f
Compute output
10:
J = MSE(x,X) x)
>> Compute reconstruction loss
11:
Take an optimization step using Tw
0J computed with backpropagation
end for
13:
end while
14:
for test batch x E D do
Trec {-
Create empty list
16: for LSTMencLS LSTMadc lae do
17:
+ output of current autoencoder on x
18:
(re-aappen(MME(( X))
Append the reconstruction loss
19:
end for
20:
Retrieve minimum loss autoencoder's index Select the output produced by the m-th module of the Augmented architecture
arg min free
21:
22:
end for
A.2 EXPERIMENT SETTINGS
A.2.1 COPY
The experiment settings used for the Copy task are listed below:
1. Each vector in the sequence has 8 binary values. The last bit is always 0, except for the delimiter vector which is vector of zeros with the last bit set to 1.
2. The subtasks draw sequences with lengths from 5 + 2 to 26 + 2 with steps of 2.
3. The sequence length for each training step is drawn at random according to the lengths corresponding the current subtask.
4. Binary Cross Entropy (BCE) loss, RMSProp optimizer with learning rate of 3e 5, mo- mentum of 0.9 and no weight decay. The gradient clipping is set to 5.0. The mini batch size is 3.
11

---

Under review as a conference paper at ICLR 2020
11
12
T3
14
T5
C1
2
0
4
6
8
C2
3
7
9
Table 6: This table associates to the five subtasks (T) of Single Digit SSMNIST, the corresponding digit classes (C) (2 classes per subtask).
5. The LSTM and GIM-LSTM use 256 hidden units, while the LMN and the GIM-LMN use 126 memory units and 126 feedforward units. With this setting the models are better comparable since they use the same number of units in total.
6. For what concerns the LMN and the GIM-LMN, the memory matrices have been initialized as orthogonal, due the benefits the norms of the matrices explained in Vorontsov al. (2017). Moreover, in order to constrain the memory matrices to remain approximately orthogonal during training, the BCE loss has been augmented with an additional penalty term in order to penalize non-orthogonal memory matrices (A || WWT - |]). The hyperparameter À associated with this penalty term has been set 0.1.
We tried use different number hidden units all models. Such changes affect the final result only for LSTM and LMN with 512 hidden units, leading to better performance (90% accuracy) on sequences of length 19 and 26. With this configuration, LMN and LSTM obtain comparable performances. We tried configuration with 256 and 512 hidden units for both functional and memory component for LMN and A-LMN, and with 128 and 512 for LSTM and A-LSTM. We varied the hyperparameter À in the range [0.01, 0.1, 0.2, 1].
A.2.2 SSMNIST
In the Single Digit task, subtasks are numbered from (following the order of the training phase) and are associated to the digit classes according to Table
The experiment settings for all SSMNIST tasks are listed below:
LSTM and GIM-LSTM with 64 hidden units and 1 hidden layer
LMN and GIM-LMN with 64 hidden units both for the memory component and the func- tional component
RMSProp optimizer with learning rate of 3e -,-5, L2 regularizer of le e--3 3, gradient clipping with norm 5.0 and momentum 0.9
Mini-batch size of sequences
The LSTM autoencoders 40 hidden units for both the encoder and the decoder in the Single Digit SSMNIST, and 200 hidden units for the Sequence Classification SSMNIST. They are optimized with Adam optimizer, learning rate of le and L2 decay of le
We use 60% of dataset for training. 20% for both validation and test.
For what concerns Sequence Classification SSMNIST Task during training, sequences are dynam- ically generated each step by randomly selecting a batch of class labels and then by randomly generating sequences belonging to those classes. Each digit in the sequence is randomly sampled from the training set. During validation and final test, we generate 5000 sequences with the same procedure. The classes equally balanced in the validation and test sets (50 patterns per class). The average sequence length in SSMNIST is 40 vectors per digit, with 4 scalars in each vector. We compact the representation in the LSTM autoencoders by a factor of 4, by using 40 hidden units per digit. Hence, in Sequence Classification SSMNIST Task we use 40 5 = 200 hidden units for autoencoders.
A.2.3 AUDIOSET
The experiment settings are listed below:
12

---

Under review as a conference paper at ICLR 2020
Table 7: Relatedness values between subtasks on Single Digit SSMNIST.
T1
T2
T3
T4
TS
T1
1.0000
T2
0.1703
1.0000
T3
0.5407
0.4762
1.0000
T4
0.3642
0.2876
0.4403
1.0000
T5
0.0102
0.4432
0.7379
0.5164
1.0000
Table 8: Relatedness values between subtasks on Sequence Classification SSMNIST.
T1
T2
T3
T1
1.0000
T2
0.9899
1.0000
T3
0.9867
0.9668
1.0000
Standard LSTM and GIM-LSTM with 32 hidden units and hidden layer
LMN and GIM-LMN with 32 hidden units both for the memory component and the func- tional component
RMSProp optimizer with learning rate of -, L2 regularizer of 1, gradient clipping with norm 5.0 and momentum of 0.9
Mini-batch size of 4 audio clips
The LSTM autoencoders use 200 hidden units for both the encoder and the decoder and they are optimized with Adam optimizer, learning rate of 3e and no decay.
We tested configurations with different number of hidden units, in the range [8, 16, 32, 64] for all models. In the LMN and GIM-LMN the same number of hidden units has been used in both functional and memory component. Results showed that even models with 8 hidden units achieve accuracies that are only slightly below the ones achieved by the models with 32 units. By increasing the units to 16, we obtained the same results reported in the paper.
Each Audioset pattern is composed by 10 vectors with 128 elements, for total number of elements equal to 1280. We use 200 hidden units in the LSTM autoencoders, thus compressing the representation by factor of 6.4.
A.3 RELATEDNESS BETWEEN SUBTASKS
The relatedness values between subtask are showed by Table 7
A.4 SSMNIST CLASSES
Table 10 reports the sequence classes used in the SSMNIST Sequential classification task.
A.5
AUDIOSET CLASSES
The input patterns used in the experiments have been restricted to the ones labeled with only class. Each class has restrictions based on the associated ontology, is marked with reliability > 70%
Table 9: Relatedness values between subtasks on Audioset.
TI
T2
T3
T4
T1
1.0000
T2
0.8415
1.0000
T3
0.8713
0.9673
1.0000
T4
0.9126
0.9444
0.9541
1.0000
13

---

Under review as a conference paper at ICLR 2020
Table 10: SSMNIST classes
ID
2
4
5
ID
2
4
5
ID
2
3
4
5
67
2
0
0
2
2
2
0
68
2
1
2
2
0
0
2
0
0
35
2
2
69
2
2
0
3
1
2
70
0
2
4
0
0
0
0
2
37
0
2
1
0
71
5
38
1
72
2
6
0
0
0
39
0
0
2
0
73
7
2
0
40
2
2
8
2
0
2
2
41
2
2
75
2
0
2
0
9
2
2
2
42
0
0
2
76
0
2
10
2
0
43
1
77
0
0
2
0
11
0
2
2
44
1
0
78
2
0
12
0
45
2
2
79
2
0
2
13
0
46
0
0
80
14
0
47
0
81
0
15
2
2
0
2
48
0
2
2
2
82
2
16
49
2
0
83
0
0
17
2
0
0
50
0
2
84
0
2
51
0
85
0
19
0
0
52
2
0
86
2
0
2
20
0
53
0
2
2
21
0
54
0
2
88
0
0
22
0
0
55
0
2
0
23
56
0
1
90
2
0
24
0
2
2
57
2
2
2
91
2
25
2
58
0
92
26
0
1
2
2
0
59
2
2
93
0
27
1
0
2
94
0
28
2
1
2
61
2
2
1
0
2
95
0
0
29
0
96
2
30
0
0
0
63
2
2
2
97
2
31
0
98
0
32
0
2
65
2
2
99
2
0
0
33
2
0
66
0
100
0
and is not sub super class of any other, following the procedure adopted by Kemker et al. 2018)) Table reports the 40 classes (N column) selected from Audioset dataset.
The ID is the corresponding class code and Category is its description, in the official Audioset documentation. Train and Test list the number of patterns of the corresponding class in the entire dataset.
The first 10 (N from 1 to 10) classes are used the first subtask, the classes from 20, from 21 to 30 and from 31 to 40 are used in the second, third and fourth subtasks.
14

---

Under review as a conference paper at ICLR 2020
Table 11: Audioset classes
N
ID
Train
Test
Category
4
154
6
Conversation
2
6
175
12
Babbling
10
75
3
Whoop
N
ID
Train
Test
Category
525
476
10
Radio
22
407
337
15
Tick
14
296
16
Screaming
23
411
732
16
Sewing machine
15
436
8
Whispering
24
414
64
14
Cash register
17
4
Baby laughter
25
415
2005
41
Printer
18
124
7
Giggle
26
419
382
Hammer
21
95
3
Chuckle chortle
27
421
85
Sawing
Baby cry
10
23
483
14
28
442
146
12
Chink clink
infant cry
29
450
382
12
Trickle dribble
Microwave
368
393
11
30
452
94
Fill (with liquid)
oven
31
453
1056
18
Spray
369
836
58
Blender
32
454
163
Pump (liquid)
377
1396
59
Vacuum cleaner
33
467
283
52
Slap smack
Electric shaver
14
382
201
11
34
468
1294
37
Whack thwack
electric razor
35
469
339
Smash crash
Computer
15
386
318
4
36
470
99
10
Breaking
keyboard
37
471
37
20
Bouncing
387
271
18
Writing
38
481
652
13
Beep bleep
Telephone bell
17
390
80
14
39
483
384
17
Ding
ringing
40
493
103
18
Rumble
18
391
243
9
Ringtone
19
398
332
21
Buzzer
20
400
284
13
Fire alarm
21
403
175
18
Steam whistle
15


Reviews:
[{"review_id": "r1xv8YTatr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed an interesting continual learning approach for sequential data processing with recurrent neural network architecture. \nThe authors provide a general application on sequential data for continual learning, and show their proposed model outperforms baseline.\n\nIt is natural that their naive baseline shows poor performance since they do not consider any continual learning issues like the catastrophic forgetting problem. Then, I hesitate to evaluate the model in terms of performance. In that sense, it would be much crucial to show more meaningful ablation studies and analysis for proposed model. However, there is a few of thing about them. \n\nThen, I decide to give a lower score that even the authors suggest that the main contribution is a definition of problem setting. It requires more detailed and sophisticated analysis.\n          \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1lsUfWpKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nIn this paper, the authors propose a new method to apply continual learning on sequential data. The model is constructed by combining an Autoencoder and LSTM/LMN for each task. The experiments on several datasets show the proposed model outperforms basic LSTM/LMN.\n\n\nStrength:\n\n+ Sequential data widely exist in the real world, e.g., text, health records. Thus, It is interesting to see that continual learning is used in sequential data. \n\n+ The motivation of the proposed model is clear. The authors save the learned knowledge in the hidden representation of LSTM/LMN.\n\nWeakness:\n- In this paper, the model size linearly increases since the number of LSTM/LMN and AE increases when a new task comes in. Thus, if the number of tasks is too large, the model size is quite big. In traditional continual learning settings, researchers may not always increase the model size for overcoming catastrophic forgetting. For example, if task 1 and task 2 sample from the same distribution, they can share the same LSTM/LMN and AE. Thus, it would be better if the authors can consider how to reduce the model size in the future version.\n\n- In the experiments, the authors only compare the proposed model with simple LSTM or LMN. However, most continual learning methods can still be applied in this scenario, at least regularization based methods [1,2] can be simply applied in this scenario. The authors may need to compare the proposed method with them in the future version.\n\n- It is better to compare it with a larger dataset. For example, in the natural language processing field, we can regard sentiment analysis on one language as one task. Then, we can construct the continual learning dataset for sentiment analysis.\n\nMinor Comments:\nIt is better to improve Figure 3 by adding the x-axis label and y-axis label.\n\n\n[1] Kirkpatrick, James, et al. \"Overcoming catastrophic forgetting in neural networks.\" Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.\n[2] Zenke, Friedemann, Ben Poole, and Surya Ganguli. \"Continual learning through synaptic intelligence.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1esT_GTKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of this work is to best understand the performance and benchmarking of continual learning algorithms when applied to sequential data processing problems like language or sequence data sets. The contributions of the paper are 3 fold - new benchmarks for CL with sequential data for RNN processing, new architecture introduced for more effective processing and a thorough empirical evaluation. \n\nIntroduction: \nI think a little more insight into why the sequential data processing CL scenario is any different than the vision scenario would be quite helpful. Specifically, it would be quite impactful to tell us more about what the additional challenges with RNNs for CL vs feedforward for CL are in the intro. \n\nThe paper is written as if the benchmark is the main contribution and the architecture improvement is just a delta on top of this, but it gets confusing when the methods section starts off with just directly stating the new architecture. \n\nThe algorithm seems like a straightforward combination of recurrent progressive nets and gated autoencoders for CL. Can the authors provide more justification if that is the contribution or there is more to the insight than has been previously suggested in prior work?\n\nFigure 1 has a very uninformative caption. It also doesn’t show how modules feed into one another properly. \n\nThe motivation for why one needs GIM after one already has A-LSTM or A-LMN is not very clear?\n\nOverall the contribution does seem a bit incremental based on prior work and the description lacks enough detail to properly indicate why this is a very important contribution?\n\nExperiments:\nWhat does it mean to be application agnostic but restricted to particular datasets and losses? This doesn’t quite parse to me. \n\nThe description of the tasks is very informal and hard to follow. It’s not clear what exactly the tasks and datasets look like \n\n“using morehidden units can bridge this gap” -> why not just do it? Its a benchmark after all. \n\nOverall the task descriptions should be in a separate section where the setup is described in a lot of detail and motivated properly. \n\nThe results in the experiments section are very hard to parse. The captions need much more detail for eg Table 2. \n\nCould we also possibly have more baselines from continual learning? For instance EWC (Kirkpatrick) or generative replay might be competitive baselines. \n\nOverall I think that the GIM and A-LMN and A-LSTM methods are reasonable although somewhat incremental. But the proposed benchmarks are pretty unclear and the results are a bit hard to really interpret well. It would also be important to run comparisons with more baselines and to provide more ablation/analysis experiments to really see the benefit of GIM/A-LMN or A-LSTM. I also think that the task descriptions should be much earlier in the paper and desribed in much more rigorous detail. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript describes a continual learning approach where individual instances consist of sequences, such as language modeling. The paper consists of a definition of a problem setting, tasks in that problem setting, baselines (not based on existing continual learning approaches, which the authors argue is to highlight the need for such techniques, but with which the reviewers took issue), and a novel architecture.

Reviews focused on the gravity of the contribution. R1 and R2, in particular, argued that the paper is written as though the problem/benchmark definition is the main contribution. R2 mentions that in spite of this, the methods section jumps directly into the candidate architecture. As mentioned above, several reviewers also took issue with the fact that existing CL techniques are not employed as baselines. The authors engaged with reviewers and promised updates, but did not take the opportunity to update their paper.

As many of the reviewers' comments remain unaddressed and the authors' updates did not materialize, I recommend rejection, and encourage the authors to incorporate the feedback they have received in a future submission.
Decision: Reject