Title: OVERCOMING CATASTROPHIC FORGETTING FOR CONTINUAL LEARNING VIA MODEL ADAPTATION

Abstract: Learning multiple tasks sequentially is important for the development of AI and lifelong learning systems. However, standard neural network architectures suffer from catastrophic forgetting which makes it difficult for them to learn a sequence of tasks. Several continual learning methods have been proposed to address the problem. In this paper, we propose a very different approach, called Parameter Generation and Model Adaptation (PGMA), to dealing with the problem. The proposed approach learns to build a model, called the solver, with two sets of parameters. The first set is shared by all tasks learned so far and the second set is dynamically generated to adapt the solver to suit each test example in order to classify it. Extensive experiments have been carried out to demonstrate the effectiveness of the proposed approach.

1. INTRODUCTION:

It is well-known that neural networks (NNs) suffer from catastrophic forgetting (CF) (McCloskey & Cohen, 1989) , which refers to the phenomenon that when learning a sequence of tasks, the learning of each new task may cause the NN to forget the models learned for the previous tasks. Without solving this problem, an NN is hard to adapt to lifelong or continual learning, which is important for AI.

Problem Statement: Given a sequence of supervised learning tasks T = (T 1 , T 2 , . . . , T N ), we want to learn them one by one in the given sequence such that the learning of each new task will not forget the models learned for the previous tasks.

In recent years, many approaches (often called continual learning) have been proposed to lessen the effect of CF (Chen & Liu, 2018; Parisi et al., 2018) , e.g., dynamically expandable network (DEN) (Yoon et al., 2018) , learning without forgetting (LWF) (Li & Hoiem, 2016) ), elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) , incremental moment matching (IMM) (Lee et al., 2017) , gradient episodic memory (GEM) (Lopez-Paz et al., 2017) , generative replay (GR) (Shin et al., 2017) , etc. These existing studies except DEN and LWF focused on learning a model parameterized by a single joint set of parameters θ * which is assumed to work well for all tasks. We call them joint parameterization (JP) methods. DEN and LWF require constantly increasing the number of parameters and thus can result in a huge and complex model. JP methods, however, suffer from accuracy deterioration. Assume we have two tasks A and B that need to be learned sequentially. Let θ * A and θ * B be the optimal parameters for Task A and Task B respectively when each of them is learned individually. Let θ * be the joint parameters learned by the existing approaches to perform Tasks A and B in sequence. Inevitably, θ * is different from θ * A and/or θ * B and is highly likely to result in more errors for the two tasks than θ * A and θ * B individually.

In this paper, we propose a different approach, called Parameter Generation and Model Adaptation (PGMA), to dealing with CF and to significantly reducing accuracy deterioration. PGMA does not learn a joint parameter set θ * . Instead, it learns a parameter generator f (•) and a shared parameter set θ 0 . The key idea of PGMA is as follows: The overall classification network, called the solver S, has two disjoint subsets/parts of parameters. The first subset is θ 0 , which is shared by all tasks learned so far. The second subset is just a place holder H that will be filled by parameters generated by f (•) for each test instance. That is, given a test instance, a set of parameters will be generated by the learned parameter generator f (•) to replace H. Solver S then combines θ 0 and the generated parameters for the test instance to classify it. Clearly, unlike existing JP approaches, we do not have a network with a set of fixed parameters for S. The idea is that θ 0 contains the common features of all tasks, and the generated parameters for each test instance adapt S for each test instance in order to classify it.

Since f (•) and the shared parameters θ 0 will change during training for each new task, forgetting can occur for previous tasks. To deal with it, in training f (•) and S for each task T i , in addition to the training data of T i , a small number of replayed samples will be generated by a data generation network for the previous tasks to ensure that the knowledge learned for previous tasks remain stable/unforgotten. Compared with the existing generative replay methods, our method does not need labels for the replayed data because we use the replayed data only to constrain the training of S and f (•) rather than to treat them as surrogates of labeled training data from previous tasks. Further, since in the existing replay methods, the labels are produced by the current learned network itself, the labels can be noisy and biased, which may result in errors being accumulated and propagated to subsequent tasks. Our method does not have this problem.

Apart from reducing the effect of accuracy deterioration of the existing approaches, the proposed approach also has some other advantages. First, no parameter increase or network expansion is needed to learn new tasks. Second, no previous data needs to be stored to enable the system to remember the previously learned models or knowledge.

Experiments conducted using two image datasets (MNIST and CIFAR-10) and two text datasets (DBPedia ontology (Lehmann et al., 2015) and THUCNews (Li et al., 2006) ) show that the proposed approach PGMA works well for different scenarios and different types of datasets, and outperforms the existing strong baselines markedly.

2. PROPOSED PGMA FRAMEWORK:

Let the sequence of supervised learning tasks be T = (T 1 , T 2 , . . . , T N ). Each task T i is represented by

T i = {x ij , y ij |j ∈ (1, . . . , N i )}

, where x ij is the j-th example/sample of T i , and y ij is its label. We use (x t , y t ) to denote a test instance/example. To simplify the notation, we will just use (x i , y i ) to denote a training example from task T i , omitting the second subscript j. Note that in the paper, we use the terms example, sample, and instance interchangeably.

The proposed parameter generation and model adaptation (PGMA) architecture has three main components:

• Solver S: It is the main classification model. As mentioned in Section 1, the parameter set of S consists of two subsets, θ 0 that is shared by all tasks (and instances) and H, a parameter place holder, which will be replaced by the generated parameters set p i (or p t ) for each training (or testing) example x i (or x t ). p i (or p t ) basically serves to adapt the solver S to classify the example in training or testing. This is the key idea of our approach. We adopt this parameter split as several studies (Yoon et al., 2018; Li & Hoiem, 2016; Kirkpatrick et al., 2017) have shown that only part of parameters of a neural network needs to be adjusted when learning a new task. See Section 2.2

• Dynamic Parameter Generator (DPG) f (•): It takes the embedding z i (or z t ) of each input training (or testing) example x i (or x t ) to generate the parameters p i (or p t ) for solver S. Note that we use z i (or z t ) rather than the raw data x i (or x t ) because the raw data's dimension can be very high. The embedding z i (or z t ) as a low dimensional dense representation reduces the mapping space for DPG and thus reduces the difficulty in its parameter generation. See Section 2.2.

• Data Generator (DG): It has two functions. The main function is to generate a set of replayed data or samples {x m } M m=1 using its decoder DG D for previous tasks to deal with catastrophic forgetting. The other function is to generate the embedding z i (or z t ) of each input training (or testing) example x i (or x t ) using its encoder DG E . See Section 2.3.

2.1. OVERALL APPROACH OF PGMA:

We work backward by describing testing first before training. Given a test instance x t , DG E first generates its embedding z t , which is fed to f (•) to generate a set of parameters p t . Solver S then takes x t as input and uses the trained/learned shared parameters θ 0 and p t to classify x t . θ 0 contains the common features of all tasks learned so far. p t simply adapts S for x t in order to classify x t .

For training, the pipeline of the proposed PGMA framework is shown in Figure 1 . Given a new task T i with its data (x i , y i ), solver S and DPG f (•) are jointly trained to learn T i and also not to forget the previously learned tasks. In each iteration, a set of parameters p i is generated by the current DPG f (z i , µ) for each training instance x i (z i being its embedding), where µ is the set of parameters of the DPG network, which is trained. In training the solver S and DPG for the new task T i , both f (•) and the shared parameters θ 0 will change, which can cause forgetting in DPG for previous tasks. To keep DPG remembering the acquired knowledge for previous tasks, we minimize the variation of certain layers' output caused by the changes of θ 0 and f (•) using the set of replayed samples {x m } M m=1 generated by DG.

PGMA has two training components: (1) DPG and solver S (DPG&S for short), i.e., DPG is optimized together with solver S. (2) DG. The two components have their respective objective functions (see Sections 2.2 and 2.3), and are trained alternately. This is because DPG takes input z i (which is produced by DG) to generate parameters, and alternating training ensures consistent convergence rates of DG and DPG.

2.2. DYNAMIC PARAMETER GENERATOR (DPG) AND SOLVER S IMPLEMENTATION:

Several neural networks can be used to implement DPG for parameter generation, e.g., convolutional neural network (CNN), recurrent neural network (RNN) and multilayer perceptron (MLP). The objective of this paper is not to explore all possible implementations. We found that a straightforward implementation using MLP can already achieve good results. Formally, DPG can be written as:

p i = f (z i , µ) = σ(w D z i + b) (1)

where σ is the activation function, and w D and b are the parameters of DPG and denoted by µ.

For the implementation of solver S, several deep learning networks can be used as well. Again, a MLP is employed in this work. Each layer of the MLP is a perceptron and can be formalized by output = σ(wx input ), where x input denotes the input of the particular perceptron. We also call a perceptron a basic unit1 . In general, for each basic unit k of the solver S, we can have a shared portion of the parameters θ 0,k and a generated portion of the parameters p i,k , i.e.,

θ * i = combine(θ 0 , p i ) = {[w * i,k ]} K k=1 = {[θ 0,k ; p i,k ]} K k=1 (2)

where combine(.) is concatenation and K is the number of basic units that the solver has. In our case, K is the number of hidden layers (basic units) of the solver MLP.

It is important to note that the above is the most general case. In practice, there is no need to adapt all the basic units in the solver, but only a subset of them. In our experiments (Section 3), we will see that adapting only the final layer of the solver MLP can already achieve good results.

However, one may still ask whether the proposed combination method has sufficient capacity to adjust the solver S in general because only part of the parameters are generated. Actually, p ik can affect all dimensions of the output of its corresponding basic unit k:

w * i,k x input = [θ 0,k ; p i,k ]x input = θ 0,k x 1 input + p i,k x 2 input (3)

where x 1 input and x2 input are the block vectors of x input . p i,k x 2 input can be regarded as the bias and can adapt the output vector to any point in the vector space.

To train DPG&S, we use cross entropy loss (L ce ). The objective function of the solver S including DPG f (z i , µ) and θ 0 for learning each new classification task T i is defined as:

minimize µ,θ0 L ce (S(x i , θ * i ), y i ) s.t. M m=1 ||R(x m , θ * i ) -R(x m , θ * i-1 )|| < r (4)

where L ce is the cross entropy loss, R(•) denotes the output of all basic units (which can be any layer or a set of layers 2 ) in solver S, which we will introduce later, and θ * i is the whole set of parameters of S (θ * i = combine(p i , θ 0 )) representing the adaptation of S. We can see that the generated replayed samples x m are used as constraints to alleviate DPG&S's forgetting. Specially, we extend the knowledge distillation loss (Hinton et al., 2015) to the general situation. That is, to keep the past learned knowledge, the output of the basic units in the solver should not change much when learning a new task with the help of the generated data. If we do not consider the activation function, the constraints in Eq. 4 can also be written as:

EQUATION

where K again denotes the number of basic units and M denotes the number of replayed samples.

The basic unit with a smaller k is at the relative lower layer of the solver network.3 x m,k is the input of the k th basic unit and is calculated through forward propagation, except x m,1 = DG D (z sample m , θ d ) which is the initial replayed sample x m generated by DG (before optimizing the current task T i ). w * i,k is the combined parameter introduced in Section 2.2. Two questions may be asked about Eqs. 4 and 5. (1) Since the replayed data x (denoting all x m ) is used in Eqs. 4 and 5, does it play the same role as the original data x from previous tasks? (2) What is the impact of the constraints on learning the new task? We answer the two questions now.

Question 1: The answer is positive. Since the purpose of the constraints is to maintain the learned effect of the old data, clearly, using the original old data x in the constraints is better. However, we proved the positive answer to question 1, which is given in Appendix.

Question 2: In calculating Eq. 5, if we stack all {x m,k } M m=1 (column vectors) into a matrix X k , Eq. 5 can be regarded as constraining w * i,k X k to remain unchanged. If X k is a row low rank matrix, w * i,k can be trained to fit the new tasks. Otherwise, especially if X k is a row full rank matrix, w * i,k cannot be trained and therefore cannot learn new tasks. However, benefiting from dynamic parameter generation, our approach will not suffer from this problem. That is because w * i,k is specially generated through DPG to perform well with input X k . And the new parameters will be generated to fit new tasks while w * i,k is able to remain unchanged.

2.3. DATA GENERATOR (DG):

As indicated earlier, DG has two functions. First, it compresses the original input data x i to z i using its encoder to reduce the number of dimensions of x i and consequently reduces the mapping space of DPG to make the generation of the parameters p i easier. The compression is formulated by:

EQUATION

where DG E is the encoder of DG with parameters θ e .

Second, it generates the replayed data for previous tasks to deal with forgetting in solver S and DPG.

Note that DG differs from data generators in the existing generative replay (GR) methods (Shin et al., 2017) as GR needs to generate both the replay data x m and their labels y m (using the solver learned so far), which can be noisy. DG only generates the data x m but not the labels (which are not needed by our approach), and then the labeling errors won't affect our model PGMA, but will hurt GR.

Each replayed sample x m is generated by the decoder of DG, called DG D :

EQUATION

where z sample m is the m th sample sampled from the multivariate normal distribution.4 θ d is the set of parameters of DG D before optimizing the current task T i .

DG can be implemented with an auto-encoder, e.g., VAE-like (Variational Auto-Encoder (Kingma & Welling, 2013)) and WAE-like (Wasserstein Auto-Encoder) auto-encoders. We use WAE in DG since it can let different examples get a chance to stay far away from each other, which promotes better reconstruction (Tolstikhin et al., 2017) .5 Note that DG also suffers from forgetting, which is dealt with like DPG and also using specially designed losses (see below).

To train DG, we use mean square error as the reconstruction loss to enable its replay ability of the past data, and add to it the penalized form of the Wasserstein distance between the distribution of z i and multivariate normal distribution to help generate data (Tolstikhin et al., 2017 ) (together denoted as L wae ). where x m is the replayed data (see Eq. 7), θ e and θ d are the encoder's and decoder's parameters of DG in the process of learning the new task T i , respectively, and z sample m and M have the same meanings as they are in DPG&S. Eq. 8 constrains the consistency of DG's decoder and encoder over the randomly sampled z sample m . Eq. 9 ensures that DG's decoder can still remember the old data. Using Eqs. 8 and 9, we can maintain DG's ability to reflect the old data. Overall, the final objective function for DG is composed by L wae , Eq. 8 and 9.

2.4. THE COMPLETE TRAINING PROCEDURE:

Finally, we summarize the whole training procedure of PGMA in Algorithm 1, which is selfexplanatory. "//" is followed by comments. The test procedure is straightforward, see Section 2.1. 

3. EXPERIMENTS:

We now evaluate the proposed approach PGMA6 and compare it with state-of-the-art baselines using two image datasets and two text datasets.

Datasets:

• Two image datasets: (1) MNIST: this dataset consists of 70,000 images of handwritten digits from 0 to 9. We use 60,000/3000/7000 images for training/validation/testing respectively. (2) CIFAR-10: this dataset consists of 60,000 32x32 color images of 10 classes, with 6000 images per class. There are 50,000/3000/7000 images for training/validation/testing respectively.

• Two text datasets: (1) DBPedia ontology: this is a crowd-sourced dataset (Lehmann et al., 2015) with 560,000 training samples and 70,000 test samples. Out of the 70,000 test samples, we use 10,000 for validation and 60,000 for test. (2) THUCNews: this dataset consists of 65,000 sentences of 10 classes (Li et al., 2006) . We randomly select 50,000/5000/10,000 sentences for training/validation/testing respectively.

Experiment Settings:

Data Preparation: To simulate sequential learning, we adopt the same two data processing methods as in (Lee et al., 2017) , named disjoint and shuffled.

• Disjoint: This method divides each dataset into several subsets of classes. Each subset is a task. For example, we divide the MNIST dataset into two tasks (or subsets of classes). The first task consists of digits (classes) {0, 1, 2, 3, 4} and the second task consists of the remaining digits (classes) {5, 6, 7, 8, 9}. The systems learn the two subsets as two tasks in a sequential fashion and regard them together as 10-class classification. In order to consider more tasks in testing, for MNIST, CIFAR-10, and THUCNews, which all have 10 classes, we created two experiment settings of 2 tasks (5 classes per task) and 5 tasks (2 classes per task). For DBPedia, which has 14 classes, we created three experiment settings, 2 tasks (7 classes per task), 3 tasks (5, 5, and 4 classes for the three tasks respectively), and 5 tasks (3, 3, 3, 3, and 2 classes for the 5 tasks respectively).

• Shuffled: This method shuffles the input pixels of an image with a fixed random permutation. Two experiment settings were created: 3 tasks and 5 tasks. In both cases, the dataset for the first task is the original dataset. The datasets for the rest of the tasks are constructed through shuffling. Since shuffling of words in a sentence will change the sentence meaning and results in confusion, thus this experiment is not done on text datasets.

Baselines: We use three state-of-the-art baselines that are representative of the current approaches: 1) EWC (elastic weight consolidation) (Kirkpatrick et al., 2017) ; 2) IMM (incremental moment matching) (Lee et al., 2017) ; 3) GR (generative replay) (Shin et al., 2017) . We use the open source code released by the authors or the third party for comparison. We use Adam algorithm to update the parameters and also use Adam as a baseline to show how serious the forgetting problem is.

Training Details: For fair comparison, our proposed approach uses the same solver (or classifier) as the baselines. That is, a multilayer perceptron is adopted as the solver/classifier (as the baselines all use this method), which is a 3-layer network (i.e., two basic units with each hidden layer as a unit) followed by a softmax layer. For our approach, the total number of parameters in the solver includes both the generated parameters p and the shared parameters θ 0 . Due to the differences among different datasets, we adopt different settings for them, see Table 5 in Appendix for details. All baselines and our approach use the same setting for the same dataset. We use a 3-layer perceptron (with 2 hidden layers) network (we also call it T-net) for DPG and set the size of each hidden layer to 1000. Each T-net can generate 100 parameters at a time. We can parallel several T-nets in the DPG to generate more parameters when needed. The network parameters are updated using the Adam algorithm with a learning rate of 0.001.

Results and Analysis:

Figure 3 shows the test accuracy plots per 40 training steps of each method for each task as the tasks are sequentially learned. IMM is not shown here because IMM works by combining well trained models of individual tasks to form one joint model for all tasks. Thus, we cannot draw an accuracy curve with increased training steps like others. However, its final results and those of the other systems are given in Tables 1 and 2 . Shuffled CIFAR-10 is also not included in the figure because our experiments show that it cannot be learned by CNN. It is also not used in any existing paper. Note that due to space limitations, Figure 3 only plots the results of those settings with only 2 and 3 tasks.

From Figure 3 , we can make the following observations:

(1). The proposed PGMA method consistently outperforms the baselines in overcoming forgetting, by a big margin in most cases.

(2). EWC does not perform well for the disjoint setting. GR is better but still poorer than our method. Adam's results show that forgetting is very serious for all datasets and settings.

Time efficiency: Due to DPG, our method uses slightly more time than baselines, under 25% for training and under 28% for testing compared to GR, which is efficient. This is a small price to pay for the major gain in accuracy in dealing with catastrophic forgetting.

Memory usage: There is no need for our method to save data, which results in a large memory saving. For more details about time efficiency and memory usage, please see Appendix (Section C). From Table 1 , we can see that our PGMA method is consistently superior to EWC, GR, and IMM on different datasets with 2 or 3 tasks (5 tasks results are given in Table 2 ). We believe that is because our PGMA model can reduce the effect of the accuracy deterioration problem discussed in Section 1. Adam's results show that forgetting is very serious as it does not deal with the forgetting problem. EWC's performance is poor for the disjoint case (a more realistic setting in practice), which is also reported by other researchers (Lee et al., 2017) . Appendix has the results for only the last task. Table 2 shows the results for 5 tasks. EWC and Adam are not included as they performed poorly. Again, we observe that our PGMA method is markedly better than the baselines.

Published as a conference paper at ICLR 2019 

Ablation Study:

Here we study how the system behaves with less and less parameters in θ 0 or more and more parameters replaced by the parameters generated by DPG. We selected the disjoint MNIST 2 tasks setting to conduct the experiment as it is more useful than the shuffled setting and also more difficult. Table 3 shows the accuracy results (averaged in the same way as in Table 1 ) when a portion of the parameters in only the last layer of the solver is replaced by the parameters generated by DPG. We observe that the accuracy improves with increased percentages of parameters being replaced.

The best accuracy is obtained when 80% of the parameters in the last layer are replaced through DPG, and the accuracy won't further improve with more replaced parameters. This observation indicates that replacing a part of parameters in solver to adapt new input tasks is sufficient. The same conclusion can also be made by replacing the parameters in the first hidden layer of the solver (which has 2 hidden layers). We fix the replacing percentage of the last layer to 20%, and then increase the replacing percentages of the first layer. The best accuracy reaches 94.31%(±0.84%) when replacing 40% parameters of the first layer, which gains only 1.87% in accuracy compared with no replacement. This result indicates that it suffices to replace the parameters in the last layer. Table 4 shows the contribution of different components. We can see a significant drop if DPG is removed. The second row gives the performance of our model when only DG and constraints are used. The third row shows the result without using constraints (DG+label(like GR)) but replacing them with the replay method. We can see that constraints work better than predicting labels.

4. RELATED WORK:

Many approaches have been proposed to deal with catastrophic forgetting (CF), which is one of the challenging problems of neural networks for lifelong learning (Chen & Liu, 2018) . EWC (Kirkpatrick et al., 2017) quantifies the importance of weights to previous tasks, and selectively alters the learning rates of weights. Following EWC, Zenke et al. (2017) measured the synapse consolidation strength in an online fashion and used it as regularization. Learning without forgetting (LWF) (Li & Hoiem, 2016) feeds the old network with new training data in new tasks and regards the output as "pseudolabels". In Incremental Moment Matching (IMM) (Lee et al., 2017) , each trained network on one task is preserved and all networks are merged into one at the end of the sequence of tasks.

Above approaches focus on adjusting the network weights. Another main approach is to add some data of past tasks to the new task training to prevent forgetting the past. Gradient Episodic Memory (GEM) (Lopez-Paz et al., 2017) stores a subset of training data for every finished task, and limits the loss function on these so-called "memories". Instead of keeping some real data from previous tasks, Generative Replay (GR) (Shin et al., 2017) keeps data generators for previous tasks and learns using a mix of real data of the new tasks and replayed data of previous tasks. Seff et al. (2017) proposed to solve continual generative modeling by combining the ideas of data generation and EWC.

Other existing approaches include iCaRL (Rebuffi et al., 2017) , Pathnet (Fernando et al., 2017) , memory aware synapses (Aljundi et al., 2017) , phantom sampling (Venkatesan et al., 2017) , active long term memory networks (Furlanello et al., 2016) , conceptor-aided backprop (He & Jaeger, 2018) , gating networks (Masse et al., 2018; Serrà et al., 2018) , dynamically expandable networks (DEN) (Yoon et al., 2018 ), progress & compress (Schwarz et al., 2018) (active column is distilled into the knowledge base, taking care to protect any previously acquired skills), and incremental regularized least squares (Camoriano et al., 2017) . Most of those works suffer from accuracy deterioration as discussed in Section 1 while some dynamically expanding the network size (e.g., DEN and LWF), which results in a huge and complex model. Our method is different from these existing approaches.

Several existing works generate parameters, but they are for different purposes. Denil et al. (2013) trained several different architectures by learning only a small number of weights and predicting the rest. Ha et al. (2016) used a small network to generate the weights for a larger network. Brock et al. (2017) learned an auxiliary HyperNet to generate weights for the main model. Our method uses the generated parameters for model adaptation for continual learning.

5. CONCLUSION:

This paper proposed a novel approach PGMA to dealing with catastrophic forgetting. The approach learns to build a model with two sets of parameters. The first set is shared by all tasks learned so far and the second set is dynamically generated to adapt the model (solver) to suit each individual test example. As we discussed in related work, this is different from all existing methods. Experimental results showed that the proposed approach outperformed the existing baseline methods markedly. ADAM gets the best accuracy on the last task as Adam optimizer learns the new task only and does not need to worry about the forgetting problem on the old tasks. For the last task, our PGMA system obtained comparable accuracy on average with the baselines designed for overcoming forgetting. However, we are much better at preventing forgetting of old tasks. The results for the last task are given in Table 7 . Note that although EWC is quite good at the last task but it suffers seriously from forgetting as shown in Table 1 or Figure 3 , which is consistent with results reported in other papers. 

:

This is because most of the existing networks can be regarded as the composition of multiple basic units, even for those with complex structures, e.g., seq2seq(Sutskever et al.,

2014), R-Net(Wang et al., 2017), Resnet(He et al., 2016) and Densenet(Huang et al., 2017). Further, complex structures give us more freedom to decide which part of the parameters should be generated.

Note that in this case, the parameters in R is a subset of parameters (θ * i ) for solver S. To simplify the expression and without causing confusion, we don't introduce new symbols and let the symbols unchanged.

Note that constraining only the units in the last layer can already achieve good results.

In training, we force zi to satisfy the multivariate normal distribution. We can then sample many z sample m from the multivariate normal distribution.

Our approach is not limited to using WAE. Other auto-encoders may also be applied.

https://github.com/morning-dews/PGMA tensorflow

Figures and Tables:

Figure 1: Sequential training in the PGMA framework.

Figure 2: Data Generator training pipeline demonstration.

Figure 3: Accuracy curves -we test the system's accuracy per 40 training steps on the test set of each method. Y-axis shows the accuracy of different tasks as the training time (steps) increases. Note that Shuffled CIFAR-10 is not included as CNN cannot learn it directly.

Algorithm 1 PGMA (Parameter Generation and Model Adaptation) training

Table content: <table><tr><td/><td>Input: T = {Ti} N i=1 , where Ti = {xij, yij}</td><td colspan="2">15: // Learning the subsequent tasks</td></tr><tr><td/><td>Initial: Randomly initialize f (•), DG, S;</td><td/><td>for all i = 2; i ≤ N ; i + + do</td></tr><tr><td/><td>// Learning the first task</td><td/><td>Generate replayed samples x m by DG;</td></tr><tr><td/><td>for all n = 0, . . . , until convergence do</td><td/><td>for all n = 0, . . . , until convergence do</td></tr><tr><td>5:</td><td>Sample a mini-batch from T1;</td><td/><td>Sample a mini-batch from Ti;</td></tr><tr><td/><td>// Training DG</td><td>20:</td><td>Minimize Lwae, Eqs. 8 &amp;9 and then update</td></tr><tr><td/><td>Minimize Lwae and then update DG;</td><td/><td>DG;</td><td>// Section 2.3</td></tr><tr><td/><td>// Training DPG and S. pn and Sn below</td><td/><td>Compute pn using Eqs. 1 &amp; 6;</td></tr><tr><td/><td>// denote a batch of generated parameters</td><td/><td>Construct Sn using pn and θ0;</td></tr><tr><td>10:</td><td>// and adapted solvers, respectively</td><td/><td>Minimize Lce, Eq. 5, and then update f (•)</td></tr><tr><td/><td>Compute pn using Eqs. 1 &amp; 6; // Section 2.2</td><td/><td>and S; // Section 2.2</td></tr><tr><td/><td>Construct Sn using pn and θ0; // Section 2.2</td><td/><td>end for</td></tr><tr><td/><td>Minimize Lce then update f (•) and S;</td><td colspan="2">25: end for</td></tr><tr><td/><td>end for</td><td/></tr></table>

Average accuracy over all tasks in a sequence after the tasks have all been learned.

Table content: <table><tr><td>Model</td><td>shuffled MNIST (3 tasks)</td><td>disjoint MNIST (2 tasks)</td><td>disjoint CIFAR-10 (2 tasks)</td><td>THUCNews (2 tasks)</td><td>DBPedia (2 tasks)</td><td>DBPedia (3 tasks)</td></tr><tr><td>Adam</td><td>91.46</td><td>48.64</td><td>41.99</td><td>46.78</td><td>47.66</td><td>41.10</td></tr><tr><td>EWC</td><td>96.70</td><td>48.96</td><td>37.75</td><td>45.02</td><td>53.95</td><td>35.89</td></tr><tr><td>GR</td><td>97.57</td><td>89.96</td><td>65.11</td><td>81.55</td><td>88.41</td><td>82.14</td></tr><tr><td>IMM</td><td>97.92</td><td>94.12</td><td>62.98</td><td>80.32</td><td>92.65</td><td>85.31</td></tr><tr><td>Our PGMA</td><td>98.14</td><td>96.53</td><td>69.51</td><td>85.12</td><td>94.70</td><td>88.06</td></tr></table>

Average accuracy over 5 tasks in a sequence after the tasks have all been learned.

Table content: <table><tr><td>Model</td><td>shuffled MNIST</td><td>disjoint MNIST</td><td>DBPedia</td><td>CIFAR-10</td><td>THUCNews</td></tr><tr><td>GR</td><td>94.54</td><td>75.47</td><td>63.71</td><td>31.09</td><td>47.35</td></tr><tr><td>IMM</td><td>96.09</td><td>67.25</td><td>64.04</td><td>32.36</td><td>46.61</td></tr><tr><td>Our PGMA</td><td>96.77</td><td>81.70</td><td>69.68</td><td>40.47</td><td>52.93</td></tr></table>

Average accuracy with different % of parameters in solver's last layer (one basic unit) being replaced by the parameters generated by DPG. Each score followed by a 95% confidence interval.

Table content: <table><tr><td>Model</td><td>GR</td><td>0%</td><td>20%</td><td>40%</td><td colspan="2">Our PGMA</td><td>60%</td><td>80%</td><td>100%</td></tr><tr><td>Accuracy</td><td>88.71 (±2.64)</td><td>90.96 (±0.69)</td><td>92.44(±1.31)</td><td colspan="2">93.96(±0.63)</td><td colspan="2">94.51(±0.87)</td><td>96.07(±0.62)</td><td>96.05(±0.64)</td></tr></table>

Empirical evaluation of different components

Table content: <table><tr><td>Components</td><td>Acc (%)</td></tr><tr><td colspan="2">DPG + DG + Constraints 96.07 ± 0.62</td></tr><tr><td colspan="2">DG + Constraints 90.96 ± 0.69</td></tr><tr><td colspan="2">DG + label (like GR) 88.21 ± 2.81</td></tr></table>

Empirical evaluation of different components

Table content: <table><tr><td>Time EWC</td><td>IMM</td><td>GR</td><td>Our PGMA</td></tr><tr><td colspan="3">Training Time/per epoch (s) 2.460 4.930 8.673</td><td>10.836</td></tr><tr><td colspan="3">Test time(s) 0.703 0.707 0.728</td><td>0.930</td></tr></table>

Accuracy of the last task after all tasks have been learned.

Table content: <table><tr><td>Model</td><td>shuffled MNIST</td><td>shuffled MNIST</td><td>disjoint MNIST</td><td>disjoint MNIST</td><td>disjoint CIFAR-10</td><td>THUCNews</td><td>DBPedia</td><td>DBPedia</td><td>DBPedia</td></tr><tr><td/><td>3 tasks</td><td>5 tasks</td><td>2 tasks</td><td>5 tasks</td><td>2 tasks</td><td>2 tasks</td><td>2 tasks</td><td>3 tasks</td><td>5 tasks</td></tr><tr><td>Adam</td><td>98.30</td><td>98.27</td><td>97.28</td><td>95.78</td><td>83.98</td><td>93.56</td><td>95.30</td><td>97.52</td><td>98.45</td></tr><tr><td>EWC</td><td>98.12</td><td>97.99</td><td>96.97</td><td>95.61</td><td>75.38</td><td>86.72</td><td>96.11</td><td>96.55</td><td>93.32</td></tr><tr><td>GR</td><td>98.10</td><td>97.18</td><td>96.34</td><td>92.31</td><td>77.82</td><td>87.20</td><td>93.18</td><td>89.40</td><td>98.76</td></tr><tr><td>IMM</td><td>97.06</td><td>95.36</td><td>95.07</td><td>86.94</td><td>48.74</td><td>65.08</td><td>93.84</td><td>85.72</td><td>70.19</td></tr><tr><td>Our PGMA</td><td>98.15</td><td>97.26</td><td>96.22</td><td>93.47</td><td>73.75</td><td>86.36</td><td>96.01</td><td>91.14</td><td>82.99</td></tr></table>

Peer review (iclr2019):
 This paper proposes a method for continual learning. The model has three components: a) a data generator to be used at training time to replay past examples, b) a parameter generator that takes the input observation to produce parameters for c) the actual classifier. The authors demonstrate the method on simple datasets with a stream of 2 or 3 tasks. Strenghts: - the combination of components is novel - the method does not rely on task descriptors neither at training nor test time Weaknesses: - the paper needs a major rewrite to improve fluency and to better organize and describe the proposed approach - the empirical validation is weak. Relevance Learning in a continual setting is certainly very relevant for this venue. Novelty While each component is by itself not very novel (replay methods for continual learning have already been used, networks predicting parameters have also become a fairly common approach in meta-learning literature), the proposed combination is novel in this sub-field. Clarity Clarity is very poor and definitely does not meet the acceptance bar for this conference. I believe that the authors would need to make a major revision to address this issue. While ICLR allows authors to revise papers, I think the revision needed to fix this draft goes beyond the acceptable limit, as reviewers would then need to make a whole new revision. First, fluency is very poor. There are lots of grammatical errors (see first sentence of introduction "neural networks suffers.."), a plethora of un-necessary acronyms which force the reader to go back and forth to figure out what they refer to (MA, DG, DPG, DPG&S, ...), and several sentences are not well formed (e.g., read first sentence of introduction). Second, some statements are contradictory; e.g., the authors define "basic unit" as "simple MLP with one hidden layer", but then say it "is an activation function plus a matrix transformation".. Third, graphics and formulas are too small and not legible. Fourth, the organization of the paper is poor, it is very wordy yet vague. For instance, the authors should precisely describe how the data generator is trained in sec. 2.3. The authors should provide an algorithm summarizing how the different components interplay both at training and test time. At present, I am making educated guesses about how this system works. For instance, how are real and generated examples interleaved? how is forgetting prevented in the data generator? References to prior work While there are lots of references in sec. 4, they are not sufficiently well described - see third paragraph of sec. 4 where the authors cite almost 20 papers by simply saying they are "some other approaches". Also, I did not find mention to methods predicting parameters in the meta-learning community but also others like: Denil et al. "Predicting network parameters in deep learning" NIPS 2013 Empirical validation The empirical evaluation does show an advantage of the proposed approach on some simple streams composed by up to three tasks. However, a) the tasks are really simple because of the small number of tasks considered and b) the baselines are weak. For instance, EWC is now a bit out-dated as there are variants that work a bit better, like: Chaundry et al. "Riemannian Walk ..." ECCV 2018 and there are other methods like "Progress and Compress" Schwartz ICML 2018 the authors could have compared against. Besides, the authors do not mention anything about memory and time cost both at training and test time, possibly including the time to cross validate all the hyper-parameters of this method. Overall, I am left with the sense that the proposed approach will be hard to scale to many more tasks and more realistic images (for which we do not quite know how to train efficiently and well generators). Other comments I did not find the formalization in eq. 9 very useful. The first and last term in that equation can be very big and there is no sense of how lose this bound is. Also, it is not clear whether there is a general principle to partition the set of parameters (to determine which ones should be shared).