Title: CONTINUAL LEARNING WITH GATED INCREMENTAL MEMORIES FOR SEQUENTIAL DATA PROCESSING

Abstract: The ability to learn over changing task distributions without forgetting previous knowledge, also known as continual learning, is a key enabler for scalable and trustworthy deployments of adaptive solutions. While the importance of continual learning is largely acknowledged in machine vision and reinforcement learning problems, this is mostly under-documented for sequence processing tasks. This work focuses on characterizing and quantitatively assessing the impact of catastrophic forgetting and task interference when dealing with sequential data in recurrent neural networks. We also introduce a general architecture, named Gated Incremental Memory, for augmenting recurrent models with continual learning skills, whose effectiveness is demonstrated through the benchmarks introduced in this paper.

1. INTRODUCTION:

Continual Learning (CL) can be defined as "the unending process of learning new things on top of what has already been learned" (Ring, 2011) . A more formal definition centered around the computational aspects of a CL algorithm is presented in Lesort et al. (2019) :

Definition 1 Given a potentially infinite sequence of unknown distributions D = {D 1 , D 2 , D 3 , ...}, where each D i = {X i , Y i } includes the input data X i and the target labels Y i , a Continual Learning algorithm A CL i is defined by the following signature:

∀D i ∈ D, A CL i ≡ < h i-1 , T R i , M i-1 , t i > → < h i , M i >

where T R i is the training set drawn from the corresponding data distribution D i , h i is the function that the model has learned after having seen all the training sets up to the i-th, M i is an external memory in which to store patterns from T R i and t i is the label identifying the task.

In this paper, the notion of task is associated to a specific objective (e.g. learn to classify MNIST digits). An input distribution, instead, generates data related to a particular task (e.g. subsets of MNIST digits). Hence, each task is associated to multiple input distributions (also called subtasks). In the CL scenario, a learning model is required to incrementally build and dynamically update internal representations as the distribution of tasks varies across its lifetime. Ideally, part of such internal representations will be general and invariant enough to be reusable across similar tasks, while another part should preserve and encode task specific representations. Unfortunately, continuous plasticity of internal representations under drifting task distributions is widely known to suffer from negative interference between the tasks that are incrementally presented to the model, yielding to the well known stability-plasticity dilemma (Parisi et al., 2019; Zhou et al., 2012) of connectionist models. The result of this being models which catastrophically forget previously acquired knowledge (Robins, 1995; French, 1999) as new tasks become available, that is one of the main challenges faced by CL algorithms. While current trends in CL put particular emphasis on computer vision applications or reinforcement learning scenarios, sequential data processing is rarely taken into consideration (see Section 2). Sequential data processing in CL involves a stream D of distributions of sequences D i = (X i , Y i ), i = 1, 2, 3, ..., where X i are sequences of vectors and Y i are the corresponding targets. In Machine Learning (ML), sequential data processing plays a fundamental role in important fields like Natural Language Processing (NLP), signal processing, bioinformatics and many others. In this context, Recurrent Neural Networks (RNNs) provide an effective means to deal with sequential data, thanks to their adaptive memory, that is the ability of developing neural representations that capture the history of their inputs. Learning proper memory representations is a major challenge for RNNs. Especially in a CL setting, where RNNs have also to deal with drifts in task distributions which can greatly affect their capability of developing robust and effective memory representations. In this work, we provide a threefold contribution to the discussion concerning CL in sequential data processing. First, we introduce benchmarks for the evaluation of CL skills by adapting three sequential datasets to CL scenarios. These benchmarks can be easily tuned to varying degrees of complexity and hardness, and they have been devised to provide an assessment that is not tailored to a specific application domain (e.g. NLP, RL or computer vision). Rather, they aim at evaluating the intrinsic effectiveness of recurrent models in a CL setting. Second, we define a new dynamic approach that imbues RNN architectures with CL skills by incrementally adding new modules to capture the shift in task distribution while avoiding catastrophic forgetting. This architecture, named Gated Incremental Memory, leverages gating autoencoders to automatically recognize input distributions in order to avoid any explicit supervision concerning their identity at inference time. Third, we evaluate baseline RNN models on the proposed benchmarks, comparing their performance against our enhanced architecture. Source code and full details regarding the experimental setup are provided as supplemental material, to ease replicability of the results and allow other researchers a direct and fair comparison of their own CL models on our benchmarks. The results of our empirical analysis confirm the advantages of using our enhanced architecture over standard recurrent models. Such advantages are particularly clear when testing enhanced architecture on old distributions, since it successfully prevents forgetting, when learning sequences with increasing length (in the Copy Task) and when the input sequences end with a common suffix (in the Sequence Classification SSMNIST Task). These results highlight some of the key differences between feedforward and recurrent CL techniques, and pinpoint the need for solutions specifically designed for recurrent architectures.

2. RELATED WORK:

CL literature mostly focuses on computer vision and reinforcement learning applications, with approaches ranging from regularization methods (Kirkpatrick et al., 2017; Zenke et al., 2017) , to dual models (Hinton & Plaut, 1987; French, 1997) , to dynamic architectures (Rusu et al., 2016; Yoon et al., 2018) . The work of Coop & Arel (2013) is one of the first attempts to apply RNNs in CL scenarios. In their paper, they employed a RNN with Fixed Expansion Layer (FEL) (Coop & Arel, 2013) and trained the resulting model to reconstruct auto-associative binary sequences. The popular Copy task (Graves et al., 2014) and Sequential Stroke MNIST dataset (de Jong, 2016) have been used by (Sodhani et al., 2018) to test the capabilities of dynamic recurrent models in presence of increasingly longer sequences. By using dynamic external memories, Asghar et al. (2018) successfully applied RNNs to CL on NLP tasks with MNLI dataset. Finally, Kemker et al. (2018) applied feedforward networks to the Audioset dataset (Gemmeke et al., 2017) in a CL setting. Our dynamic RNN architecture is inspired by Progressive networks (Rusu et al., 2016) , a CL approach popular in the feedforward domain that deals with new distributions by dynamically adding new hidden units to a feedforward layer and connecting them to the existing hidden units through lateral connections. The gating autoencoders are adapted to sequential scenarios from Aljundi et al. (2017) , where they are first introduced for feedforward models.

3. GATED INCREMENTAL MEMORIES FOR CONTINUAL LEARNING WITH RECURRENT NEURAL NETWORKS:

In this section, we introduce Gated Incremental Memory (GIM), a novel CL architecture designed for recurrent neural models and sequential data. In particular, we show how GIM can be obtained by combining together a recurrent version of the Progressive network (Rusu et al., 2016) and a set of gating autoencoders (Aljundi et al., 2017) in order to avoid, at test time, any explicit supervision about subtask label. We denote the recurrent version of the Progressive network as Augmented models (A-LMN and A-LSTM), while the entire architecture, including both augmented architectures and gating autoencoders, is called GIM. To highlight the generality of our approach, we detail the application of GIM to two substantially different recurrent neural models from literature (GIM-LMN, GIM-LSTM). In the following, we indicate an entire sequence with bold notation (e.g. x), while a single vector with plain formatting (e.g. x i ).

3.1. RECURRENT NEURAL NETWORKS:

The proposed approach is independent on the underlying recurrent architecture. In this work, we focus our study on two different classes of RNNs, using either gated and non-gated approaches. Gated models, like LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014) , leverage adaptive gates to enable selective memory cell updates. In our analysis, we consider LSTM as a representative of gated architectures, given its popularity in literature and its state-of-the-art performances in several sequential data processing benchmarks. Non-gated approaches rely on different mechanisms to solve the vanishing gradient problem, like parameterizing recurrent connections with an orthogonal matrix (Mhammedi et al., 2017) . In our analysis, we consider the Linear Memory Network (LMN) (Bacciu et al., 2019) as a representative of non-gated approaches. LMNs leverage a conceptual separation between a nonlinear feedforward mapping computing the hidden state h t and a linear dynamic memory computing the history dependent state h m t . Briefly, in formulas:

h t = σ(W xh x t + W mh h m t-1 ) h m t = W hm h t + W mm h m t-1

where x t , h t and h m t are the input vector, the functional component activation and the memory state at time t. The memory state h m t is the final output of the layer, which is given as input to subsequent layers. It is then useful to study how both models behave in CL environments and how their different memory representations respond to phenomena like drifting tasks distribution, eventually resulting in catastrophic forgetting.

3.2. GATED INCREMENTAL MEMORY (GIM):

GIM represents a general class of dynamic, recurrent architectures that can be built on top of any recurrent model. GIM relies on a progressive memory (Rusu et al., 2016) extension of the underlying RNN model that, for the sake of this work, are LSTMs or LMNs. We also leverage a set of gating autoencoders, one for each distribution, to automatically select the module which has the best match to the current input distribution. Figures 1 and 2 provide an overview of the entire GIM architecture during training and test. The main component of GIM is the RNN module. As soon as a new distribution arrives, a new RNN module is added on top of the existing architecture and connected to the previous one. The exact inter-modules connections are slightly different depending on the underlying recurrent model. When using the A-LSTM, at each timestep, the new module takes as additional input the current hidden state of the previous module. In the A-LMN, the hidden state h t of the new module is computed by taking as additional input the concatenation of both previous module's memory h m t and functional activation h t . In order to prevent forgetting, when a new module is attached to the existing architecture, the previous module's parameters are frozen and no longer updated. Therefore, each module becomes an expert of its own domain. The input vector x t at each time step t is forwarded to all modules. Each module has its own output layer and, during training, the last module added to the network is used to generate the final output y. The forward pass for an A-LMN with N modules can then be formalized as follows:

h m T,1 , h T,1 = LMN 1 (x, h m 0,1 ) h m T,j , h T,j = LMN j ([x; h m T,j-1 ; h T,j-1 ], h m 0,j ), j = 2, ..., N

where h m T,i and h T,i are the memory states and functional activations of module i after having seen the entire input sequence x of length T and [•; •] is the concatenation operator between vectors. The aggregated output y is computed by passing the final memory state h m T,N through a linear layer. The A-LSTM forward pass follows the same process, substituting the final hidden state h T to the Each module of the augmented architecture is associated to a single LSTM autoencoder (Srivastava et al., 2015) , which is a sequence-to-sequence model trained jointly to the corresponding module to reconstruct the input x. The training process involves only the autoencoder associated to the last added module. The loss function used to train the autoencoders is the reconstruction error, that is the mean squared error (MSE) applied element-wise to the input and reconstructed sequence. The Augmented models are trained by using standard Cross Entropy loss for classification tasks. The following equations provide a formal description of the GIM-LMN training when tackling the i-th distribution (the GIM-LSTM behaves in the same way):

y = A-LMN(x) x = AE i (x)

where AE i identifies the autoencoder associated to module i. The input x is sampled from the i-th distribution. The output y and the reconstruction x are then used in the corresponding loss functions (Cross Entropy and MSE, respectively) to train the A-LMN and the autoencoder by gradient descent. At inference time, the input x is forwarded to all autoencoders and the module associated to the one producing the minimum reconstruction error is selected to produce the final output, as described by the following equations:

xi = AE i (x), i = 1, ..., N k = arg min i MSE(x i , x) y = LMN k (x).

The algorithms describing the forward pass for both A-LMN and A-LSTM and the training and test procedure of GIM are provided in Appendix A.1. GIM, like Progressive networks, is capable of learning multiple distributions without being affected by forgetting. In addition, GIM overcomes one of the major drawbacks of Progressive networks (Rusu et al., 2016) : it does not require explicit knowledge about input distributions at test time, since gating autoencoders are able to autonomously recognize current input and use the appropriate module for output. GIM also simplifies the inter-modules connections: while in Progressive networks they are feedforward networks, created between a module and all the next ones, GIM employs a simpler, non adaptive, version and connects only adjacent modules. Therefore, the connections do not contribute to the total number of adaptive parameters, which scales linearly with the number of modules. Finally, as described in Aljundi et al. (2017) , it is possible to compute a relatedness measure between subtasks using the autoencoders reconstruction errors. Given a new subtask T k , an old subtask T a and their associated autoencoders, the relatedness is computed by using the reconstruction errors E r k and E ra of the autoencoders measured on validation data for subtask T k . Formally:

EQUATION

Notice that this definition is not entirely equivalent to the one reported in Aljundi et al. (2017) .

Since we assume that each autoencoder achieves the lowest error on its own training data, we have E ra > E r k . Aljundi et al. (2017) normalize relatedness by having E r k in the denominator of (1), in place of E ra . In our experiments, we noticed that this led to large negative values for the relatedness, which are not particularly insightful. Therefore, we propose a different normalization coefficient, using the largest reconstruction error E ra in place of E r k .

4. EXPERIMENTS:

Figure 2 : Test of GIM-LSTM. Input x is presented to all autoencoders. In this case, AE2 obtains the minimum reconstructione error. Hence, the second module (dashed line) is chosen to produce the output. The input is propagated up until the second module.

One of the main contributions of this paper is the definition of a set of learning tasks that can be used as standard benchmarks for CL algorithms on sequence processing problems. We believe that the current literature is lacking a set of simple, application-agnostic, benchmarks that can provide a fair comparison and evaluation of different approaches with a minimal experimental setup. The final appendices provide all the details regarding the experimental setup needed to reproduce the results.

The experiments rely on three datasets: the Copy dataset (Graves et al., 2014) , the Sequential Stroke MNIST dataset (de Jong, 2016) and the Audioset dataset (Gemmeke et al., 2017) . We adapted each dataset to classification tasks in CL environments. We split each task in multiple subtasks, associating to each one a subset of the possible classes. The complexity of each task can be adjusted and controlled by the experimenter. The tasks are not designed for a specific application domain, but instead, they are general enough to stress the main difficulties of RNN training and CL scenarios. This is particularly useful to test the limitation of each model and allows a great degree of flexibility. The source code is available online and we plan to release more datasets in the future1 . The Copy task (Graves et al., 2014) consists in the reconstruction of sequences of random binary vectors. This task could be considered a simple CL example, since the data distribution for each subtask only differs for the sequences' length. However, generalizing to long sequence lengths is a difficult problem for traditional recurrent architectures (Graves et al., 2014) and therefore the same techniques used in more standard CL settings can be used to solve this problem by incrementally increasing the memory capacity of the final model (Sodhani et al., 2018) . Furthermore, it is useful to evaluate the memory capacity of the Augmented architecture compared against the standard RNNs. In the Copy Task, LMN and A-LMN use orthogonal initialization of the memory component (see Appendix A.2 for more details). Gating autoencoders have not been employed in this task, since its objective is exactly the reconstruction of the input sequence. The subtasks are organized in a curriculum (Bengio et al., 2009) following sequence length, from sequences with 5 ± 2 vectors to sequences with 26 ± 2, steps of 7 vectors (4 subtasks total). Results, summarized in Table 1 , show that LMN performs slightly better than LSTM (using more hidden units can bridge this gap). This is expected since orthogonal models typically outperform LSTMs on pure memorization problems (Vorontsov et al., 2017) . Augmented models outperformed both of them, with A-LMN converging faster than A-LSTM. Augmented models have been trained also with longer sequences (up to length 40), without sensible drops in accuracy. All models obtained an accuracy on par with a random classifier when tested on previous subtasks (a well-known problem for RNNs). The Copy Task highlights the behavior of recurrent CL models trained on growing sequence lengths and proves their effectiveness when compared to standard (non CL oriented) recurrent models. The use of dynamic modules in A-LMN and A-LSTM positively contributes to the memory capacity, justifying the additional cost required by the Augmented architectures. The digits corresponding to the tokens are explicitly showed for each subtask. The last token is common to all the sequences belonging to the same subtask and its length affects the learning behavior, as showed in Table 3 . In this version, each digit is composed by a sequence of pen strokes, where each pen stroke is represented by two coordinates (x and y, scalars), an end of stroke bit eos and an end of digit bit eod. We devised two different kind of experiments, aimed at both evaluating gating autoencoders capabilities and GIMs classification performances.

4.1. COPY TASK:

In Sodhani et al. (2018) , the SSMNIST dataset has already been employed in CL environments. However, the authors build the CL strategy by progressively increasing the sequence length (e.g. by concatenating together multiple digits).

Here, we chose a different approach by limiting each subtask to a fixed set of digit classes. Our approach is consistent with the literature about CL in computer vision, where the different subtasks are designed over different object classes within the images (Maltoni & Lomonaco, 2019; Lopez-Paz & Ranzato, 2017) .

4.2.1. SINGLE DIGIT SSMNIST:

The Single Digit task (Fig. 4a ), requires the models to predict the digit class associated to the input sequence. The task has been partitioned into 5 subtasks, each of which selects input patterns belonging to 2 digit classes. There is no intersection in the digit classes between different subtasks. Classification performance easily reached top (100%) accuracy for all models. The only exception is the LMN, which obtained an accuracy on par with a random classifier on subtask 3. LMN and LSTM experienced complete forgetting of old knowledge and were not able to address any of the previous subtasks. GIMs, instead, completely prevent forgetting through the use of gating autoencoders, that were always able to distinguish between the 5 subtasks at test time. There was no clear difference between GIM-LSTM and GIM-LMN performances in this task. GIMs are useful in this context not The Sequence Classification task (Fig. 4b ) partitions the 10 digit classes in 3 subtasks of 3 digit classes each, without intersections (see Table 4 ). Sequences are composed by 5 tokens (either 0, 1 or 2), each one associated to a specific digit, depending on the subtask. The association between tokens and digits for each subtask is described by Table 4 . Sequences are grouped into 100 different classes, according to the sequence of tokens they represent. As an example, class 44 consists of tokens (1, 0, 1, 2, 0). The complete list of classes is presented in Appendix A.4. The models are trained to classify each sequence into its corresponding class. This task is more difficult than the Single Digit, since the models have first to decode the digits sequence from the input sequence (e.g. (4, 6, 4, 8, 6) ), then associate such sequence to the appropriate class (44) by inferring the correct tokens, without any intermediate help.

Gating autoencoders were still able to recognize the subtask, even if, with respect to the Single Digit, this task increases the number of digit classes per subtask (from 2 to 3) and it is based on longer sequences (made by multiple digit sequences). As in the Single Digit, GIMs successfully overcame catastrophic forgetting, while LMN and LSTM were not able to address previous subtasks. By adding a common digit segment at the end of each sequence, the classification problem becomes much more difficult. The common suffix between all sequences requires a larger memory capacity to distinguish the inputs. Depending on the length of the final segment, different behaviors emerged, as showed in Table 3 . Some lengths completely prevented learning (No Learning column), while some others did not affect the performances (Learning column), still obtaining an accuracy above 95%. The most interesting behavior (Partial Learning column) has been observed for a narrow range of lengths. In this case, learning is prevented on the first 2 subtasks. On the third subtask, the models suddenly achieved the top accuracy, as showed in Fig. 3 . The Partial Learning behavior highlights how the use of GIMs enables learning of more articulated input distributions, compared to baseline RNNs.

4.3. AUDIOSET TASK:

Audioset (Gemmeke et al., 2017) is a collection of annotated audio events, extracted from 10 seconds audio clips and organized hierarchically in classes. The objective is the classification of a sound from its audio clip source, embedded through a VGG-acoustic model into 10 vectors, one per second. In order to implement a CL scenario, we selected 40 audio classes and split them among 4 subtasks (10 classes per subtask). We selected the 40 classes according to the procedure outlined by Kemker et al. (2018) . Since their classes have not been published, we randomly selected them from the superset resulting from their preprocessing pipeline. See Appendix A.5 for a complete description. Results showed that gating autoencoders were perfectly capable of detecting the correct data distributions at inference time. Therefore, GIMs successfully prevented forgetting: they obtained a final test performance comparable to the validation accuracy measured at the end of each subtask training. Standard RNN architectures, on the contrary, were prone to catastrophic forgetting, achieving 0% accuracy on previous subtasks. Table 5 summarizes these results. Audioset data has already been used in literature to assess CL skills Kemker et al. (2018) . However, the work by Kemker et al. (2018) focused on the task from a static perspective, relying on the use of feedforward models only. Since the preprocessing step provides, for each audio clip, a sequence of fixed-size embeddings, it is possible to concatenate the vectors into a single large vector and feed it to the network. The sequential aspect of the task, however, is completely lost. At the best of our knowledge, we are the first to tackle Audioset in CL scenarios with recurrent models. It is also important to notice that the task difficulty is increased when using recurrent since the model is not able to see the input in its entirety (like in feedforward networks), but it has to scan it one timestep at a time.

4.4. RELATEDNESS ANALYSIS:

Tokens-to-Digits 0 1 2 T1 2 0 7 T2 6 4 8 T3 3 1 5

Table 4 : This table associates to each subtask of Sequence Classification SSMNIST (T1, T2, T3) the values taken by the tokens (0, 1, 2) in the sequence classes.

As an example, sequence (1, 0, 1, 2, 0) (class 44) corresponds to different multi digit sequences: (0, 2, 0, 7, 2) for subtask 1 and (4, 6, 4, 8, 6) for subtask 2.

In our experiments, large relatedness values are associated to "heterogeneous" distributions. We call heterogeneous distributions the ones that generate patterns from many different classes (as in Audioset) or from a complex mixture of few classes (as in Sequence Classification SSMNIST). We observed that, in correspondence with heterogeneous distributions, autoencoders training converges to larger values of reconstruction error than the ones obtained with homogeneous distributions (as in Single Digit SSMNIST). However, reconstruction error, hence relatedness, does not tell anything about gating capabilities (i.e. the ability to select the appropriate module at test time). Autoencoders, in fact, always performed correctly in all experiments, preventing forgetting. A complete account of relatedness measure is provided in Appendix A.3. The main objective of this work is to draw attention on the problem of CL for sequential data processing. We proposed a set of benchmarks against which future models can assess their performances. These benchmarks test the main difficulties of the sequential and CL domains and their difficulty can be carefully tuned. We introduced GIM, a recurrent CL architecture for sequential data processing, and showed that it consistently outperforms baseline RNNs on all the proposed benchmarks. The results support the claim that recurrent architectures need to be adapted to manage CL scenarios and encourage future works towards an in-depth study of their behaviors.

5. CONCLUSIONS:

In the future, GIM could be applied to Reinforcement Learning tasks, where there are more opportunities to compare performances with existing works. In order to provide a formal characterization of the learning process, it would be useful to apply existing metrics to better describe the type of tasks addressed by the model, in terms both of transfer learning opportunities and subtasks interference. New metrics can also be developed, specifically tailored to sequential learning problems.

A APPENDIX:

A.1 GATED INCREMENTAL MEMORIES Algorithms 1 and 2 outline the forward pass for A-LSTM and A-LMN architectures. Algorithm 3 describes the training and test process for the GIM architecture.

Algorithm 1 A-LSTM Forward Pass:

Require: A-LSTM with N modules, input x Ensure: N > 1 1: Initialize hidden state of module 1: Initialize the hidden state of the encoder h enc,i 6:

h 1,i 2: h 1,f = A-LSTM 1 (x,

for training batch x ∈ D k do Take an optimization step using ∂J ∂w computed with backpropagation 12:

end for 13: end while for LSTM enc LSTM dec ∈ l ae do 17:

x ← output of current autoencoder on x 1. Each vector in the sequence has 8 binary values. The last bit is always 0, except for the delimiter vector which is a vector of zeros with the last bit set to 1. 

:

Public link redacted to preserve anonymity, reviewers can find source code in the supplementary material

The subtasks draw sequences with lengths from 5 ± 2 to 26 ± 2 with steps of 2.

The sequence length for each training step is drawn at random according to the lengths corresponding to the current subtask.

Binary Cross Entropy (BCE) loss, RMSProp optimizer with learning rate of 3e -5, momentum of 0.9 and no weight decay. The gradient clipping is set

to 5.0. The mini batch size is 3.

Figures and Tables:

Figure 1: Training of GIM-LSTM on 3 subtasks.

(a) A single digit sequence is classified into its corresponding digit class. The vertical segments correspond to a single input vector. The last one is the eod.(b) A sequence of digits (including an additional final segment), is classified into its corresponding class. The tokens are uniquely associated to the class index 44.

Figure 4: SSMNIST Tasks description

enc,f = LST M enc (x, h enc,i ) Compute encoder state 8: h dec,f = LST M dec (0, h enc,f )

min l rec Retrieve minimum loss autoencoder's index 21: Select the output produced by the m-th module of the Augmented architecture 22: end for A.2 EXPERIMENT SETTINGS A.2.1 COPY The experiment settings used for the Copy task are listed below:

Accuracy (in percentage) on Copy task, measured at the end of each subtask training. Column labels indicate sequence length for each subtask.

Table content: <table><tr><td>%</td><td>5</td><td>12</td><td>19</td><td colspan="2">26 33 40</td></tr><tr><td colspan="6">A-LMN 100 100 100 100 95 95</td></tr><tr><td colspan="6">A-LSTM 100 100 100 95 90 85</td></tr><tr><td>LMN</td><td colspan="4">100 100 100 78</td><td>--</td></tr><tr><td>LSTM</td><td colspan="2">100 97</td><td>85</td><td>78</td><td>--</td></tr></table>

The table reports the values of the final digit segment length that produce, for each model, the behavior specified by the top row. L (Learning) means that the model learns all the subtasks, NL (No Learning) means that the model does not learn any subtask and PL (Partial Learning) means that the model learns only one subtask.

Table content: <table><tr><td/><td>L</td><td>NL</td><td>PL</td></tr><tr><td colspan="2">GIM-LMN ≤ 22</td><td>-</td><td>35</td></tr><tr><td colspan="4">GIM-LSTM ≤ 11 ≥ 14 12 -13</td></tr><tr><td>LMN</td><td>16</td><td>≥ 24</td><td>22</td></tr><tr><td>LSTM</td><td colspan="2">≤ 11 ≥ 12</td><td>-</td></tr><tr><td colspan="4">only to achieve top accuracy on all subtasks (e.g. GIM-LMN against LMN on subtask 3), but also</td></tr><tr><td colspan="4">to prevent forgetting. Final test performances are summarized in Table 2.</td></tr><tr><td colspan="2">4.2.2 SEQUENCE CLASSIFICATION SSMNIST</td><td/><td/></tr></table>

Accuracies (in percentage) on test sets after training on all subtasks on Single Digit SSMNIST.

Table content: <table><tr><td>%</td><td>T1</td><td>T2</td><td>T3</td><td>T4</td><td>T5</td></tr><tr><td colspan="6">GIM-LMN 100 100 100 100 100</td></tr><tr><td colspan="6">GIM-LSTM 100 100 100 100 100</td></tr><tr><td>LMN</td><td>0</td><td>0</td><td>0</td><td>0</td><td>100</td></tr><tr><td>LSTM</td><td>0</td><td>0</td><td>0</td><td>0</td><td>100</td></tr></table>

Accuracies (in percentage) on test sets after training on all subtasks on Audioset.

Table content: <table><tr><td>%</td><td colspan="4">T1 T2 T3 T4</td></tr><tr><td>GIM-LMN</td><td colspan="4">61 65 53 50</td></tr><tr><td colspan="5">GIM-LSTM 60 69 60 47</td></tr><tr><td>LMN</td><td>0</td><td>0</td><td>0</td><td>43</td></tr><tr><td>LSTM</td><td>0</td><td>0</td><td>0</td><td>42</td></tr></table>

Obtain final state of module d 7: end for 8: Output final hidden state of last module: h N,f Training and testing of LSTM Autoencoders, including output selection from Augmented architecture Require: A sequence of distributions D = {D 1 , D 2 , D 3 , ...} Ensure: | D |> 1 1: l ae ← [] Create empty list 2: while a new distribution D k is available do 3: LSTM enc , LSTM dec = init-autoencoder() 4: l ae .append((LSTM enc , LSTM dec ))

Table content: <table><tr><td colspan="2">Algorithm 2 A-LMN Forward Pass</td><td/></tr><tr><td colspan="3">Require: A-LMN with N modules, input x</td></tr><tr><td colspan="2">Ensure: N &gt; 1</td><td/></tr><tr><td colspan="3">1: Initialize memory state of module 1: h m 1,i</td></tr><tr><td colspan="2">2: h m 1,f , h 1,f = A-LMN 1 (x, h m 1,i )</td><td>Compute final memory and functional state of module 1</td></tr><tr><td colspan="2">3: for d ← 2, N do</td><td/></tr><tr><td>4:</td><td colspan="2">Initialize memory state of module d: h m d,i</td></tr><tr><td>5:</td><td>x = [x; h m d-1,f ; h d-1,f ]</td><td>Concatenate functional and memory state with input</td></tr><tr><td>6:</td><td>h m d,f , h d,f = A-LMN 1 (x, h m d,i )</td><td>Compute final memory and functional state of module d</td></tr><tr><td colspan="2">7: end for</td><td/></tr><tr><td colspan="2">8: y m N = σ(W mo N h m N,f )</td><td>Output of the final module</td></tr><tr><td colspan="3">Algorithm 3 Append new autoencoder</td></tr><tr><td>5:</td><td/><td/></tr><tr><td/><td>h 1,i )</td><td>Compute final state of module 1</td></tr><tr><td colspan="2">3: for d ← 2, N do</td><td/></tr><tr><td>4:</td><td colspan="2">Initialize hidden state of module d: h d,i</td></tr><tr><td>5:</td><td>x = [x; h d-1,f ]</td><td>Concatenate hidden state with input</td></tr><tr><td>6:</td><td/><td/></tr></table>

Audioset classes

Table content: <table><tr><td>N</td><td colspan="3">ID Train Test</td><td>Category</td><td/><td/><td/></tr><tr><td>1</td><td>4</td><td>154</td><td>6</td><td>Conversation</td><td/><td/><td/></tr><tr><td>2</td><td>6</td><td>175</td><td>12</td><td>Babbling</td><td/><td/><td/></tr><tr><td>3</td><td>10</td><td>75</td><td>3</td><td>Whoop</td><td>N</td><td colspan="2">ID Train Test</td><td>Category</td></tr><tr><td colspan="3">4 5 6 7 8 9 10 11 368 525 14 15 17 18 21 23 12 369 13 377 1396 476 296 436 72 124 95 483 393 836 14 382 201 15 386 318 16 387 271 17 390 80 18 391 243</td><td>10 16 8 4 7 3 14 11 58 59 11 4 18 14 9</td><td>Radio Screaming Whispering Baby laughter Giggle Chuckle chortle Baby cry infant cry Microwave oven Blender Vacuum cleaner Electric shaver electric razor Computer keyboard Writing Telephone bell ringing Ringtone</td><td colspan="2">22 407 23 411 24 414 25 415 2005 337 732 64 26 419 382 27 421 85 28 442 146 29 450 382 30 452 94 31 453 1056 32 454 163 33 467 283 34 468 1294 35 469 339 36 470 99 37 471 37 38 481 652 39 483 384 40 493 103</td><td>15 16 14 41 48 3 12 12 5 18 6 52 37 42 10 20 13 17 18</td><td>Tick Sewing machine Cash register Printer Hammer Sawing Chink clink Trickle dribble Fill (with liquid) Spray Pump (liquid) Slap smack Whack thwack Smash crash Breaking Bouncing Beep bleep Ding Rumble</td></tr><tr><td colspan="2">19 398</td><td>332</td><td>21</td><td>Buzzer</td><td/><td/><td/></tr><tr><td colspan="2">20 400</td><td>284</td><td>13</td><td>Fire alarm</td><td/><td/><td/></tr><tr><td colspan="2">21 403</td><td>175</td><td>18</td><td>Steam whistle</td><td/><td/><td/></tr></table>

Peer review (iclr2020):
Summary: In this paper, the authors propose a new method to apply continual learning on sequential data. The model is constructed by combining an Autoencoder and LSTM/LMN for each task. The experiments on several datasets show the proposed model outperforms basic LSTM/LMN. Strength: + Sequential data widely exist in the real world, e.g., text, health records. Thus, It is interesting to see that continual learning is used in sequential data. + The motivation of the proposed model is clear. The authors save the learned knowledge in the hidden representation of LSTM/LMN. Weakness: - In this paper, the model size linearly increases since the number of LSTM/LMN and AE increases when a new task comes in. Thus, if the number of tasks is too large, the model size is quite big. In traditional continual learning settings, researchers may not always increase the model size for overcoming catastrophic forgetting. For example, if task 1 and task 2 sample from the same distribution, they can share the same LSTM/LMN and AE. Thus, it would be better if the authors can consider how to reduce the model size in the future version. - In the experiments, the authors only compare the proposed model with simple LSTM or LMN. However, most continual learning methods can still be applied in this scenario, at least regularization based methods [1,2] can be simply applied in this scenario. The authors may need to compare the proposed method with them in the future version. - It is better to compare it with a larger dataset. For example, in the natural language processing field, we can regard sentiment analysis on one language as one task. Then, we can construct the continual learning dataset for sentiment analysis. Minor Comments: It is better to improve Figure 3 by adding the x-axis label and y-axis label. [1] Kirkpatrick, James, et al. *Overcoming catastrophic forgetting in neural networks.* Proceedings of the national academy of sciences 114.13 (2017): 3521-3526. [2] Zenke, Friedemann, Ben Poole, and Surya Ganguli. *Continual learning through synaptic intelligence.* Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.