Continual Learning via Principal Components Projection
Authors: Gyuhak Kim
Bing Liu
Venue:  (2020)
Keywords: Neural network
continual learning
catastrophic forgetting
lifelong learning

Abstract: Continual learning in neural networks (NN) often suffers from catastrophic forgetting. That is, when learning a sequence of tasks on an NN, the learning of a new task will cause weight changes that may destroy the learned knowledge embedded in the weights for previous tasks. Without solving this problem, it is difficult to use an NN to perform continual or lifelong learning. Although researchers have attempted to solve the problem in many ways, it remains to be challenging. In this paper, we propose a new approach, called principal components projection (PCP). The idea is that in learning a new task, if we can ensure that the gradient updates will only occur in the orthogonal directions to the input vectors of the previous tasks, then the weight updates for learning the new task will not affect the previous tasks. We propose to compute the principal components of the input vectors and use them to transform the input and to project the gradient updates for learning each new task. PCP does not need to store any sampled data from previous tasks or to generate pseudo data of previous tasks and use them to help learn a new task. Empirical evaluation shows that the proposed method PCP markedly outperforms the state-of-the-art baseline methods.

Under review as a conference paper at ICLR 2020
CONTINUAL LEARNING VIA PRINCIPAL COMPONENTS PROJECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Continual learning in neural networks (NN) often suffers from catastrophic for- getting. That is, when learning sequence of tasks an NN, the learning of a new task will cause weight changes that may destroy the learned knowledge embedded in the weights for previous tasks. Without solving this problem, it is difficult use an NN to perform continual or lifelong learning. Although researchers have attempted to solve the problem in many ways, remains to be challenging. In this paper, we propose a new approach, called principal components projection (PCP). The idea is that in learning new task, if we can ensure that the gradient updates will only occur in the orthogonal directions to the input vectors of the previous tasks, then the weight updates for learning the new task will not affect the previous tasks. We propose to compute the principal components of the input vectors and use them to transform the input and to project the gradient updates for learning each new task. PCP does not need to store any sampled data from previ- ous tasks or to generate pseudo data of previous tasks and use them to help learn new task. Empirical evaluation shows that the proposed method PCP markedly outperforms the state-of-the-ar baseline methods.
INTRODUCTION
Human brains have this extraordinary ability to learn large number of tasks incrementally with high accuracy. Both the learning process of and the learned knowledge for the tasks have little negative interference of each other. In fact, the learned knowledge earlier can even help the learning of new tasks later. Continual learning or lifelong learning attempts to make the computer to do the same using neural networks (NNs). However, it has been proven challenging. One of the key problems or obstacles is the issue of catastrophic forgetting (CF). This problem was first reported by McCloskey & Cohen 1989). In their work NN, they found that when learning a sequence of tasks incrementally in an NN, the training of each new task can cause the network to forget the information or knowledge learned from previous tasks. This usually means that the new task training may significantly modify the weights that have been learned for previous tasks, and thus degrade the model performance for the previous tasks. Without solving this problem, an NN is unsuitable for continual learning as it forgets the existing knowledge when it learns new things.
In the past few years, CF has attracted a great deal of research attention (Parisi et al. 2018a). A literature review will be given in Section 2. There are two main setups in continual learning: class continual learning (CCL.) and task continual learning (TCL) (Dhar et al. 2019). The general term continual learning has been used for both, which has caused some confusion when comparing results in some cases. In the CCL setup, each task consists of one or more classes to be learned together and the learning process builds single model for all tasks for classifying all the classes of the tasks learned so far. In testing, a test instance from any class may be presented to the model for it to classify. There is no prior information about which task the test instance comes from. Formally, CCL can be defined as follows.
Class continual learning (CCL). Given sequence of datasets D (D,, D2, Dr) of T tasks. The dataset of task & is Dx { y ). where each æ/. E X is a feature vector from task and y E Y& is the corresponding target class label of æi.. All Y 's are disjoint and Uh_ Y. Y. The goal of CCL is to construct a single prediction function or model f X + Y that can correctly identify the target class y for a given test instance xc.

---

Under review as a conference paper at ICLR 2020
In the TCL setup, each task is separate classification problem (e.g., one task could be classify different breeds of dogs and another task could be to classify different types of birds). Here, one model is built for each task, although all the models reside in the same neural network. In testing, the system knows which task each test instance belongs to and uses only the specific model for the task (dog bird classification) classify the test instance. Formally, TCL. is defined as follows.
Task continual learning (TCL). Given a sequence of datasets D = { D...... Dr} of T tasks. The dataset of task is Dx = {((art (*)
(-))im) where each æ E X is feature vector from task k, e E T is the indicator of task k; and y/ Y2 Y is the corresponding target class label. The goal of TCL is to construct a predictor f X x T + Y to identify the correct class label E Y& for (xx, th) (the given test instance x from task k).
This paper works in the TCL setting and proposes a more effective technique to deal with CF. The proposed method is called principal components projection (PCP), which is inspired by the OWM technique in (Zeng et al.) 2019). The idea is that in learning a new task, if we can ensure that gradient updates will occur only in the orthogonal directions to the input vectors of the previous tasks, then the weight updates for the new task will have little effect on the weights for the previous tasks.
However, OWM (Zeng et al. 2019 has a key weakness. Its weight update in learning the new task occurs only in the solution space of the previous tasks, which restricts the solution space for the new task. When the new task is quite different from previous tasks, the learning accuracy will suffer (see Section 3.1). The proposed PCP method deals with the problem. The key idea of PCP is that computes the principal components of the input vectors (as in PCA) for each task at each level of the network and uses them to transform the input and then to project the gradient updates to the orthogonal directions in learning the new task so that the updates will not affect the previous tasks and can also go beyond the solution space of previous tasks (see the details in Section 3.2). This results in significantly better results.
Moreover, unlike many existing TCL methods (see Section 2), PCP does not need to store any sampled data from previous tasks or generate pseudo past data and use them to help learn a new task. Empirical evaluation shows that PCP outperforms existing stat-of-the-art baselines considerably.
2 RELATED WORK
Catastrophic forgetting (CF) has been dealt with since early 1990s. Li & Hoiem 2016) gave a good summary of the early work. The paper also proposed new technique called LwF to deal with forgetting. It learns the parameters for the new task On with the help of shared parameters for all tasks 0, and parameters for old tasks 0, without degrading much of the performance of the old tasks. The idea is to optimize 6g and 6n on the new task with the constraint that the predictions on the new task's examples using 0₂ and 6, do not shift much. Rusu et al. (2016) proposed a progressive neural network that retains pool of pre-trained models and learns lateral connections among them. Kirkpatrick et al. (2017) proposed model called EWC that quantifies the importance of weights to previous tasks, and selectively adjusts the plasticity of weights. This method is also used in (Schwarz et al., 2018). Motivated by EWC (Kirkpatrick et al., 2017). Zenke et al. (2017) measured the synapse consolidation strength in an online fashion and used it regularization to deal with CF. Rebufh et al. (2017) retained an exemplar set that best approximates the previous tasks to help train the new task. Similar ideas were also used in (Lopez-Paz & Ranzato 2017
Chaudhry et al., 2019 Ans et al. (2044). (Shin et al. 2017 Kamra et 20.|[017]|Rstamm Rostami et al. 2019)
followed the Generative Adversarial Networks (GANs) framework (Goodfellow) (016) to produce generators for previous tasks, and learn parameters that fit a mixed set of real data of the new task and replayed/generated data of previous tasks. A network of experts is proposed by (Aljundi al. 2016) to measure task relatedness to deal with catastrophic forgetting. Rannen Ep Triki et al. (2017) used an autoencoder to extend the LwF method (Li & Hoiem| 2016). Jung et al. (2016) proposed a less-forgetful learning that regularizes the final hidden activations. [Roseneeld & Tootsos (2017) proposed controller modules that optimize loss on the new task with representations learned from previous tasks. They found that they could achieve satisfactory performance while only requiring about 22% of parameters of a fine-tuning method. Jin & Sendhoff (2006) modeled the catastrophic forgetting problem rehearsal framework to interleave base patterns with new patterns during optimization. Nguuyn et al. (217)) proposed the variational continual learning by combining online variational inference (VI)
multi-objective learning problem and proposed multi-objectivv pseudo-
2

---

Under review as a conference paper at ICLR 2020
and Monte Carlo VI for neural networks. Seff et al. (2017) proposed to solve continual generative modeling by combining the ideas of GANs (Goodfellow 2016) and EWC (Kirkpatrick et al. 2017).
Apart from regularization based approaches mentioned above (e.g., LwF (Li & Hoiem 2016), EWC (Kirkpatrick et al. 2017) and their variants), dual-memory based learning systems have also been proposed. They are inspired by the complementary learning systems (CLS) theory (McClel- land et al. 1995] Kumaran et al.) (016) in which memory consolidation and retrieval are related to the interplay of the mammalian hippocampus (short-term memory) and neocortex (long-term mem- ory). Gepperth & Karaoguz (20T6) proposed to use a modified self-organizing map (SOM) as the long-term memory. To complement it, short-term memory (STM) is added to store novel exam- ples. During the sleep phase, the whole content of STM is replayed to the system. This process is known as intrinsic replay or pseudo-rehearsal (Robins, 1995). trains all the nodes in the network with new data (e.g.. from STM) and the replayed examples/samples from previously seen classes or distributions on which the network has been trained. The replayed samples prevents the network from forgetting. A related work is that in Wu et al. 2018)) Kemker & Kanan (2018) proposed similar dual-memory system called FearNet. It uses a hippocampal network for short-term memory, medial prefrontal cortex network for long-term memory, and a third neural network to determine which memory to use for prediction.
More recently, Hu et al. (2019) proposed technique that learns to generate a part of the parameters while sharing the other part. Forgetting is dealt with by generating some previous parameters and regularization. ZZng et al. (2019) proposed method that tries to learn the new task by revising the weights in the directions that are orthogonal to the data of the previous tasks so that the new task will not interfere with the old tasks. The work in Xu & Zhu (2018) searches for the best neural architec- ture for each coming task via reinforcement learning. |Ritter et al. (2018) proposed an online Laplace approximation method. Dhar et al. (2019) proposed method that combines three loss functions to encourage the model resulted from the new task (class) to be similar to the previous model. Some other related works include Learn+++ (Polikar et al. 200T), Pathnet (Fernando et al., (017). Memory Aware Synapses (Aljundi et al., al.]|201 2017). One Big Net for Everything (Schmidhuber 2018). Phantom Sampling (Venkatesan et al.J (017), Active Long Term Memory Networks (Furlancllo et al. 2016).
Conceptor-Aided Backprop (He & Jaeger 2018), Gating Networks (Masse et al. 2018 Serrà al.
2018 PackNet (Mallya & Lazebnik] 2017). Diffusion-based Neuromodulation (Velez & Clune,
2017
Incremental Moment Matching (Lee al. 2017b). Dynamically Expandable Networks (Lee
et al.
2017a Yoon et al. 2018) ei al. 2019). Incremental Regularized Least Squares (Camoriano
al. 2017), and Dual- Recurrent Seif-Organizationn (Parisi al. 2018b). An excellent
survey of continual learning can be found in (Parisi et al. 2018a).
Most continual learning works deal with CCL (class continual learning) where each task consists of one or more classes that are incrementally learned. This paper focuses on task continual learning (TCL), where each task is a separate classification problem. GEM (Gradient Episodic Memory) (Lopez-Paz & Ranzato 2017) is such method, which takes task id in addition to the training data of the specific task as input. GEM deals with forgetting when learning a new task by jointly training the current task using its training data and also sampled data from the previous tasks to ensure the past knowledge is maintained. It also has a cross-validation step to tune the network using the first few tasks, which most other continual learning methods do not have. A-GEM (Chaudhry cet al. (2119) significantly improved GEM's efficiency while achieving comparable or better results. Another recent work in this direction is 'learn to grow" (Li et al. 2019). which grows the network with each new task while encouraging sharing of network parameters using a new penalized loss function.
3 MODEL DEVELOPMENT
Traditionally, a neural network is trained to perform a single task. Training the network to perform an additional task causes catastrophic forgetting (CF) where the network is no longer able to perform the previous task well. CF is caused by the standard gradient descent method. We use learning of two tasks k- 1 and & to illustrate. Learning task & in task k I's network can be formulated as
(1) where À is the learning rate and W k,* and W - are the optimal parameters for task k and task 1-1 of layer = 0,..., respectively. From here, the subscript / will be omitted since the analysis
Whi W W A
3

---

Under review as a conference paper at ICLR 2020
is invariant with layers. When the model predicts for the previous task i 1, may not produce the optimal result for task because
W 1h-1 o W Øn-1
(2)
where #:-1 is sample from task k - and Wk-1, is the optimal parameter for task k 1.
3.1 PROJECTED GRADIENT UPDATE
Following the example above, if the parameters for learning task & are updated in the directions perpendicular to the column space of data samples from the previous task, then forgetting can be avoided. This is formulated as
W w XV
where pk-1 is projection matrix perpendicular to the column space of samples --- With this update, the new parameters for task lo also work for task k - 1 because
W zk-1 ==-===1 Wk xx-1 -V p ppppppl--- xx-1 W
Σt-1
(4)
Zeng et al. 2219 proposes the orthogonal weight modification (OWM) method which constructs P by using the recursive least square algorithm. The recursive algorithm updates a linear model which gives the least square error of sequentially arriving data. Instead of discounting the previous errors, OWM discounts the current error and solves
min Σ |e(0))2- al]e())]
(5)
where e is the error and < a < is a parameter. By the matrix inversion lemma, it updates the projection for sample F& of task i (superscript" means transpose)
ph = pk-1 (a + x. ph ph-ppp))-- pk
xxx)
pk-1-pa Ph-1
(6)
and updates the weight as W = W - V Since P projects V onto the input space, the weight is modified within the subspace where a good performance was attained. By construction, if the optimal weight space for task 1 is span span(W")). the updated 2 for task 2 lies in span(W") O span (W2) * U. This makes sure that the model finds the solution without affecting the previous task performance, but this restricts the solution space for new task. Thus, OWM is effective tasks are similar. If the tasks are dissimilar, it will not learn well. For instance, in the extreme case, if span 3 O (span(W' (W') n span (
0, the obtained 3 will perform for task 1 and task 2. but will not perform for task 3. Even if there is intersection but the intersection is small, it will also be hard to find good solution.
Therefore, an ideal projection should be such that 1) it is resistant to forgetting and 2) does not restrict the solution space for the new task. Below, propose projection method achieve both objectives based on changing the data representation.
3.2 LEARNING WITH ORTHOGONAL REPRESENTATION
Let the data from task & be (Xk, t*t where X₃ is the training data and t is the indicator of task k. We use task number k as the indicator, which is also used in our system. The proposed learning framework PCP is defined as follows.
For the data of task k. we change its representation with an orthogonal basis (c(9)... denote this basis matrix for task k as C* and train the model using the transformed data C X The projection matrix pk is constructed such that it maps to the space perpendicular to the eurrent and all the previous orthogonal basis matrices C'l C'..... O.k. Let C' [C' .... C the projection can be computed in a standard way as follows (again, superscript means transpose):
(()) }. We
p = I -(C
where I is the identity matrix.
4

---

Under review as a conference paper at ICLR 2020
Suppose that the model is trained with the aforementioned framework from task to task k. The parameter W for task lo + 1 obtained by
W = W AVW* pk
(8)
where V is the backpropagation with respect X we have
Then for sample æ, from task k
-'''''
(9)
-W'C'a, - VW'p'C'..
(10)
W'C'a, V
(11)
(12)
where Eq. 10 is because W = W AVWk-1pk-1 = Wk-2 - V
VP
W - A VW p and Eq. is because pl is a projection orthogonal to the basis matrices
including C
In general, the data matrix X E Rexd spans the column space. This restricts the choice of C such that (1) it is an orthogonal matrix and (2) it sufficiently represents X with d vectors, where d the imput vectors. We choose C to be m principal component vectors of X in principal component analysis (PCA). The new representation Ca captures the covariance, so reduces the loss of information when the data reduced to m dimensions.
the dimensionality
The projection matrix orthogonal C' ensures that the forward- pass returns optimal y² if the input is represented in column space of C. This is shown from Eq. 9 to Eq. [2] However, it does not restrict solution space because, by construction, the later projection made of previous and new principal vectors so span( C span(P)) for < j. In addition, during forward pass, W++1x W'x VW' P'zz, the projection maps input to the space orthogonal columns of C' for i [1, t] and V W also changes P'æ x to a different space. To put in another way, let the system learn new task 1, which follows a different distribution from task k and cannot be expressed by the previous change of basis matrices C C'..... C Then, there exists basis B whose elements are linearly independent of C²,, C'...., Ct Denote x&+1 by a linear combination vectors in B. That is
Tk++1 Ba
(13)
for some coefficient vector a. Since p is not orthogonal to B, the forward for sample # becomes
Wk+ x++1
w PW*p*1441
(14)
w Ba VW*
(15)
(16)
for any j E [1, k). Thus, the parameter will be updated during back-propagation to find Di+t * vī- That is, we are learning in a space beyond the existing solution space. We also note that Øx+1 is partially explained by previous principal vectors, i.c., 2b+1 C² + ...... ......a..... + Bax+1 for some unknown change of basis B, then Pt P*æ++1 = Ba * 0. Thus, the model learns the new part Bax+1- These cases mean that the new task doesn't 't have to be learned in the existing solution space like OWM (Zeng et al. (019),, but separately. Thus the proposed method is able to learn better we will see in the experiment section.
The detailed learning algorithm given in Algorithm and the testing process is given in Algorithm 2 In the algorithms, comments started with "//". With the comments, both algorithms self- explanatory
EXPERIMENTS
In this section, we empirically evaluate the proposed method PCP and compare it with the latest state-of-the-arr baselines.
'The principal component vectors saved for each task during learning they also used in testing to transform the test data (see Algorithm 2)
5

---

Under review as a conference paper at ICLR 2020
Algorithm Principal Component Projection (PCP) training algorithm
Require: P/' for layers 1 = 0..... L of task
1: // The number of TI tasks is unknown
2: for all k 1, ... do
3: // Train the model to correctly identify {( k), Uk
4: Compute loss with the standard forward pass
5: In backward pass, update parameters W W XVW WWWFPP P
6: !! After forward pass, obtain the principal component vectors neurons including input and hidden layers
7:
The number of principal vectors may vary for different layers (in our experiments, they are automatically selected based the validate data)
8:
For example, choose my principal vectors for the n samples of activated neurons A, E Faxd /th layer (in our experiments, the number of principal vectors my is also automatically selected based on the validation data) for all 0..... do,
9:
Obtain principal component vectors C = (c(1)... c(mi..)))k
10:
11:
// Denote the principal vectors from task 1 to k at layer by C₁ = [C],... C
12:
Construct P/ I Ci(C C)) C', where is transpose
Change of basis C [c(1).k clmL.)) o. ,o Raxdd where is the
vector
end for
15:
// Until convergence, train neural net f to identify {(C* x , k), y In forward pass, apply C to imput neurons of each layer.
In backward pass, update parameters as W W AVWA P.
18: end for
return Co, C₁.
Algorithm PCP test algorithm
Require: The matrices, denoted by Co...., Cr. of principal component vectors each layer to
T,
// Retrieve corresponding principal vectors of task & from C for each layer
2: // Make prediction by changing the representations of input and activated neurons during forward pass
3: In forward pass, apply C of C to input neurons of each layers.
4: if ff (C) == Y/ then
5: Correct prediction
6: else
7:
Incorrect prediction
8: end if'
4.1
EXPERIMENT DATASETS
Four image classification benchmark datasets are used in our experiments.
(1). MNIST (LeCun et al. 1998 F This is handwritten digit classification dataset of 10 digits. The dataset has 70,000 examples/instances,
use 60,000 for training and 10,000 for testing. 20% of the training data is also used as the validation set. Each class has the same number of examples training, validation or testing as the dataset is balanced. This is also the case for the rest of the datasets.
(2). CIFAR-10 (Krizhevsky & Hinton 2009 This is an image classification dataset consisting of 60,000 32x32 color images of 10 classes, with 6,000 images per class. We use 50,000 for training and 10,000 for testing, Again, 20% of the training data is used as the validation set.
'http://yann.leco http///www..ccc/ccccscctcte https://www.c.. www.otoooooooooooooooooooo edu/ krizleifar.hml

---

Under review as a conference paper at ICLR 2020
(3). EMNIST-26 (Cohen et al. 2017 EMNIST-26 consists of 26 letters, i.., 26 balanced classes. has 145,600 examples. We used 124,800 for training and the rest for testing. Again, 20% of the training data is also used the validation set.
(3). EMNIST--7 (Cohen et al., 2017 E EMNIST-47 consists of 47 characters, i.c., 47 balanced classes It has 131,600 examples. We use 1 112,000 examples for training and the rest for testing. Again, 20% of the training data is used as the validation set.
4.2 BASELINE SYSTEMS
Our main baseline the recent algorithm OWM (Zeng et al. 2019) published in Nature Machine Intelligence. We mainly compare with it due to two reasons. The first reason is that this is strong baseline. Most of existing techniques for dealing with catastrophic forgetting (CF) tries keep the weights learned for previous tasks unchanged or minimally changed so that the previously learned knowledge is not forgotten (see Section 2). This paper gives a novel and principled idea on how the previous weights can be kept while learning new task. It also proposed novel technique to implement the idea to deal with CF. The system OWM outperforms some recent techniques dra- matically. For example, it outperforms the PGMA method in (Hu et al. 2019) by more than 10% (from 42.47% to (2.3%) ) for tasks of CIFAR-10. 10. PGMA already outperforms IMM (incremental moment matching) (Lee al., 2017b) by 8%. The second reason that our technique is inspired by their idea and it tries to address its shortcoming as discussed in Section 3.2. As we will see below, the idea in the paper a good one. If the weakness is addressed, the results can be quite strong.
We also include the EWC method (Kirkpatrick et al. 2017 6 as baseline in comparison as EWC commonly used in about all papers dealing with CF. We will not directly compare the proposed system PCP with PGMA and IMM as they are significantly inferior to OWM. All our results from OWM are obtained by running the code preeaaeee by the authors based on their tuned parameters.
Although OWM works in the CCL (class continual learning) setting (not task continual learning (TCL)), it is easy to use in the TCL setting by restricting testing of each test instance to the classes of the specific task. That is, if test instance is from task 1, we will only use the final classification nodes corresponding to the classes of task to predict the class of the test instance. Classifications of other nodes are ignored regardless what their values may be for the test instance. Thus, in the TCL setting the classification results are usually better than those of the CCL setting because the task id is known for each test instance in the TCL setting. For fair comparison, the proposed PCP method uses exactly the same network and the same network size to learn and to test. As mentioned above, also their released code produce the results reported below.
4.3 TRAINING DETAILS
Since we mainly compare with OWM, we use the same architecture as OWM. For MNIST, we use 3 fully connected layers with size 784 800 800 10. For CIFAR-IO, we used Alexnet pre-trained feature extractor, which is fixed through out the training. Fully connected layers of sizes 9216 4096- 1000- 1000 10 are used for classification. 9216 the number of output neurons of the feature extractor. OWM uses the same pre-trained feature extractor. For EMNIST-26 and EMNIST-47, we used fully connected layers of sizes 784-8811 800 - 26 and 784-800 - 8000 - 47, respectively. We did not use convolution layers for MNIST and EMNIST because the two datasets consist of single channel images and the model could attain high accuracy without using it. We used stochastic gradient descent with minibatch size of 40, 64, and 40 for MNIST, CIFAR-10, and EMNIST, respectively. A momentum value of 0.9 was chosen for better convergence. Regularization methods such as dropout were not used in any model.
http:://wwww.niss.owwwdeeeeeee/ wttpt//wwwde//9////0/ -daansttd
"https://github.c phttps//gitubb.om//ooskommmm wwwp///wwwwoooo/////.e/.oooaooni
$ 'We are unable to compare with the recent method in (Li et al. 2019) as its code was not released and we also did not manage to get the code from the authors via email. Systems that store sampled original data of previous tasks (Chaudhry al. 2019 Rebuffi et al. (017) are not comparable with These systems also require a pre-cross- -alisatttoooooi phase using the first few tasks to pre-set the network. We don't need either.
7

---

Under review as a conference paper at ICLR 2020
Table 1: Average accuracy over all tasks after the last task is learned
MNIST (5 tasks)
CIFARTO
EMNIST-26 tasks)
EMNIST-26 (10 tasks)
EMNIST-47
EMNIST-47
Model
(5 tasks)
(5 tasks)
(10 tasks)
EWC
96.14
67.33
51.16
42.83
48,98
52.11
OWM
98.89
78,29
42.04
50,00
24.61
38.39
PCP
98.54
83,43
71.88
74.25
65.61
74.60
For the experiments with PCP, we used 20% of training data as the validation set to automatically select good set of parameters, which includes the number of epochs, learning rates and the number of principal vectors at each level.
4.4 TEST RESULTS AND ANALYSIS
The experimental results in accuracy are reported in Table [ which are obtained as follows. We first incremenaally learn each task one by one. After all tasks are learned, we test all tasks using their respective test data. As noted above, the parameters and training epochs are selected using the validation set of each task. The test results are given in Table 1.
In Table D the first row gives the dataset names and number of tasks for cach dataset. For MNIST and CIFAR-IO of 5 tasks, each task consists of 2 classes. For example, for MNIST, the tasks are sets of two digits/classes, i.e., {0. 1). (2, 3), {4, 5), {6, 7), and {8, 9}. Since EMNIST-26 has 26 classes, the tasks consist of 5, 5, 5, 5, and 6 classes. For 10 tasks, they have 2, 2, 2, 2, 3, 3, 3, 3, 3, and 3 classes. For EMNIST-47, the 47 classes are split into 10, 10, 10, 9, and classes for 5 tasks, and 3, 4, 5, 5, 5, 5, 5, and 5 classes for 10 tasks.
From Table T can observe that PCP consistently outperforms baselines on all datasets and tasks except MNIST, for which OWN and PCP are about the same. All three systems perform well on MNIST it relatively easy dataset. Although OWM performs well for CIFAR-I0. it does poorly for EMNIST-26 and EMNIST-47. We tuned OWM extensively, but were not able to obtain better results. We believe that it is due to the restricted solution space weakness of the system identified in this paper (Section 3.2). In the case of CIFAR- 10, the solution spaces of the tasks are similar for OWM as ImageNet contains a large number of similar images as those CIFAR-10. 10. However, ImageNet feature extractor is not useful for EMNIST as ImageNet does not have similar images to handwritten characters from EMNIST (thus not used), which causes OWM to fail due to the weakness identified. Note that CIFAR-10'S result reported here for 5 tasks is better than that in the OWM paper because the TCL setting is easier than the CCL setting, as at the test time, the system knows the task id of the test instance, but in the CCL setting, that knowledge is not available.
Note that for both EMNIST-26 and EMNIST-47, the results for tasks are poorer than for 10 tasks. This is because for the 10 tasks case, each task has fewer classes and it is easy get higher accuracy with fewer classes.
The proposed system PCP is about 1.5 times slower than OWM. The main reason is that PCP needs to train the mapping from æ to and from C. to y. The projection matrix requires matrix inversion. This needs to be computed once after a task is learned. Obtaining the principal vectors requires computing eigenvectors. It is also computed only once after training. EWC is about 5 times slower than OWM.
Execution time:
5 CONCLUSION
To make a deep neural network model learn new tasks continually or incrementally, we have to deal with catastrophic forgetting. This paper proposed novel method, called principal components projection (PCP), which is based on the idea that if network weights are updated only for the part of training data that the model has not seen in the orthogonal directions to those of previous tasks, interfering with the previously leamned knowledge can be largely avoided. The proposed method works by transforming the input vectors and activated neurons for each task using orthogonal basis matrices, which lists of principal components of the corresponding neurons. We modify the gradient if the sample does not lie on the column space spanned by all the previous task data. Experimental results showed superior performance of the proposed method PCP compared to the
8

---

Under review as a conference paper at ICLR 2020
existing state-of- the-arts. Our future work will focus further improving the aceuracy and studying how to enable and exploit knowledge sharing across tasks to improve the performance all tasks.
REFERENCES
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2, 2016.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory Aware Synapses: Learning what (not) to forget. arXiv preprint arXiv: 2017.
Bernard Ans, Stéphane Rousset, Robert M French, and Serhan Musca. Self-refreshing memory in artificial neural networks: Learning temporal sequences without catastrophic forgetting. Connec- tion Science, 16(2):71-99, 2004.
Raffaello Camoriano, Giulia Pasquale, Carlo Ciliberto, Lorenzo Natale, Lorenzo Rosasco, and Gior- gio Metta. Incremental robot learning of new objects with fixed update time. IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 3207-32012 IEEE, 2017.
Arslan Chaudhry, MarcAurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. In ICLR, 2019.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre van Schaik. Emnist: an extension of mnist to handwritten letters. http://axxi..orrraa///7000022... 2017.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In CVPR, 2019.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super 2017.
neural networks. arXiv preprint arXiv:
Tommaso Furlanello, Jiaping Zhao, Andrew M Saxe, Laurent Itti, and Bosco S Tjan. Active long term memory networks. arXiv preprint arXiv:1606.0255 02355, 2016.
Alexander Gepperth and Cem Karaoguz. A bio-inspired incremental learning architecture for ap- plied perceptual problems. Cognitive Compulation, 8(5):924-934, 2016.
Ian Goodfellow. NIPS 2016 tutorial: Generative adversarial networks.
arXiv preprint
arXiv:1701.00166) 00100000006, 2016
Xu He and Herbert Jaeger. Overcoming Catastrophic Interference using Conceptor-Aided Back- propagation. In International Conference Learning Representations, 2018.
Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In ICLR, 2019.
Yaochu Jin and Bernhard Sendhoff. Alleviating catastrophic forgetting via multi-objective learning. In IJCNN, pp. 2335-3342. 2006.
Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forge learning in deep neural networks. arXiv preprint arXiv: 1000.000700222 2016.
Nitin Kamra, Umang Gupta, and Yan Liu. Deep Generative Dual Memory Network for Continual Learning. arXiv preprint arXiv: 2017.
Ronald Kemker and Christopher Kanan. FearNet: Brain-Inspired Model for Incremental Learning. In JCLR, 2018
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka -aaoskaa-BBwiska,, and Others. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 14((1):3521-3526, 2017.

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report TR- TR-2009, University of Toronto, Toronto., 2009.
Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends in cognitive sciences, 20 (7):512-5344 2016.
Yann LeCun, Corinna Cortes, and Christopher JC Burges. The mnist database of handwritten digits. http://yyan.lecum wttpw./ww/wedb///iis...eee 1998.
Jeongtae Lee, Jaehong Yun, Sungju Hwang, and Eunho Yang. Lifelong Learning with Dynamically Expandable Networks. arXiv preprint arXiv: 1eXiiv:708.001 708.01147 2017a.
Sang-Woo Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances Neural Information Processing Systems, pp. 4655-4665, 2017b.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In ICML, 2019.
Zhizhong Li and Derek Hoiem. Learning Without Forgetting. In European Conference on Computer Springer, 2016.
Vision, pp.
David Lapez-Paz and MarcAurelio Ranzato. Gradient Episodic Memory for Continual Learning. In Advances in Neural Information Processing Systems, pp. 6470-6479, 2017.
Arun Mallya and Svetlana Lazehnik. PackNet: Adding Multiple Tasks Single Network by Iterative Pruning. arXiv preprint
2017.
Nicolas Y Masse, Gregory D Grant, and David J Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. arXiv preprint arXiv:1802.0156)) 2018.
James I. McClelland, Bruce I. McNaughton, and Randall C O'reeilly Why there are complementary learning systems the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen, Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational Continual Learn- ing. arXiv preprint arXiv:1710.106:: 2017.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual
Lifelong Learning with Neural Networks: A Review. arXiv preprint 1802.0800.07555 2018a.
German Parisi, Jun Tani, Comellus Weber, and Stefan Wermter. Lifelong Learning of Spa- tiotemporal Representations with Dual-Memory Recurrent Self-Organization. arXiv preprint 10X.v:1805.10966 2018b
Robi Polikar, Lalita Upda, Satish S Upda, and Vasant Honavar. Learn++: An incremental learning algorithm for supervised neural networks. IEEE transactions on systems, man, and cybernetics, part C (applications and reviews), 31(4):497-508 2001.
Amal Rannen Ep Triki, Rahaf Aljundi, Matthew Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In ICCV 20/7, pp. 1320-1328, 2017.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In CVPR, pp. 5533-5542, 2017.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, pp- 3738-3745 2018.
10

---

Under review as a conference paper at ICLR 2020
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorchearsal. Connection Science, 7(2): 123-146, 1995.
Amir Rosenfeld and John K Tsotsos. Incremental Learning Through Deep Adaptation. arXiv preprint arXiv:1705.044: 2017.
Mohammad Rostami, Soheil Kolouri, and Praveen K. Pilly. Complementary learning for overcom- ing catastrophic forgetting using experience replay. In IJCAI, 2019.
Andrei A Rusu, Neil C Rabinowitz. Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.0000 04671, 2016.
Juergen Schmidhuber. One Big Net For Everything. (1991):1-117 2018.
Jonathan Schwarz, Jelena Luketina, Wojeiech M Czarnecki, Agnieszka Trabska-Barriinka, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning. arXiv preprint arXiv: /805, 503.06370, 2018.
Ari Seff, Alex Beatson, Daniel Suo, and Han Liu. Continual learning generative adversarial nets. arXiv preprint arXiv:1705.08.0 2017.
Joan Serrà, Dídac Surís, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML, 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NIPS, pp. 2994-3003, 2017.
Roby Velez and Jeff Clune. Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networks. PloS one. 12(1 1):00187736, 2017.
Ragav Venkatesan, Hemanth Venkateswara, Sethuraman Panchanathan, and Baoxin Li. A Strategy for an Uncompromising Incremental Learner arXiv preprint arXiv:1705.00744, 2017.
Chenshen Wu, Luis Herranz, Xialei Liu, Joost van de Weijer, Bogdan Raducanu, et al. Memory replay gans: Learning to generate new categories without forgetting. In Advances In Neural Information Processing Systems, pp. 5962-5972, 2018.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Pro- cessing Systems, pp. 899-908, 2018
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. 2018.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continuous learning of context- processing in neural networks. Nature Machine Intelligence, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995, 2017.


Reviews:
[{"review_id": "r1lB7pU2FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The method proposes a method for continual learning. The method is an extension of recent work, called orthogonal weights modification (OWM) [Zheng,2019]. This method aims to find gradient updates which are perpendicular to the input vectors of previous tasks (resulting in less forgetting). However, the authors argue, that the learning of new tasks is happening in the solution space of the previous tasks, which might severely limit the ability to adapt to new tasks. The authors propose a ‘principal component’-based solution to this problem. The method is considering the ‘task continual learning’ scenario (also known as task-aware) which means that the task label is given at inference time.\n\nConclusion:\n\n1. The paper is not well-positioned in related works. I think the work is more related to works with ‘parameter isolation methods’ such as Piggyback, Packnet, HAT. These methods reserve part of the capacity of the network for tasks. I think the authors should relate their work with these methods, and provide an argument of the problem with these previous methods, which is addressed by their approach. I can see that rather than freezing weights (PackNet) or features (HAT) , the method freezes linear combinations of features. But it is for me not directly clear that that is desirable. In HAT the backpropagated vector is projected on the mask vector which coincides with the neurons (activations). \n\n2. The experimental verification of the paper is too weak, and only comparison to EWC and OWM (not well known) are provided. At least a comparison with the more related works PackNet and HAT should be included. For more recent method for task-aware CL see also ‘Continual learning: A comparative study on how to defy forgetting in classification tasks’. Also results seem bad. For example on CIFAR10, 5 tasks in TCL setting is two-class problem per task; I would expect better results. \n\n3. The authors claim that OWM is effective if tasks are similar, but not when dissimilar. And the proposed PCP solves this problem. However, all experiments are on similar tasks, and no cross domain tasks are considered, e.g. going from MNIST (task1) to EMNIST-26 (task2) etc. This would empirically support the claim. Also, the authors expect the difference between PCP and OWM to be even larger then. \n\n4. Some more analysis of the success of PCA in representing the distribution would be appreciated, e.g. the percentage of total energy which is captured (sum of selected eigenvalues divided by sum of all eigenvalues). Such an analysis of P_l^k as a function of the tasks (and for several layers) would be interesting to see, for example for EMNIST-47(10 tasks). \n\n5. Novelty with respect to OWM is rather small.\n\n6. The authors should mention that the method is pretrained on ImageNet in section 4.3. Given these datasets, I think it makes more sense to train from scratch and I would like to see those results. \n\nMinor remarks:\n- I wonder if you use OWM or PCP you discard the possibility of positive backward transfer. Maybe the authors could comment on that. \n\n- The authors write that ‘TCL setting the classification results are usually better than those of the CCL’ is that not per definition true ? Anything correctly classified under CCL is correctly classified under TCL but not the other way around. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SyxXQhYAYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to solve the catastrophic forgetting issue in the continual learning problem. The authors propose a method based on principal components projection to tackle this issue.  The authors conduct experiments on image classification tasks to show the performance of the proposed method and compare it with two other baselines EWC and OWM.\n\nStrong points:\n1. This paper tries to solve an important problem.\n2. The intuition of applying principal components projection is straightforward.\n\nWeak points:\n1. The most concerned point about this paper is the experiment. It is not convincing. The authors claim that OWM is one of the strongest baselines, but actually it perform really badly on EMNIST-26 (5 tasks),  EMNIST-47 (5 tasks) and EMNIST-47 (10 tasks). What is the reason? Is it because of insufficient parameter tuning? If different methods perform differently on various datasets, it is really necessary to show more baseline methods to illustrate that the proposed method has universally good performance on different datasets.\n2. It might strengthen the paper if the authors can show the comparison results on more other datasets, e.g., other image classification tasks. It would be better if the authors can show the proposed method can generalize to other tasks.\n3. The authors point out that one key drawback of OWM is that, if the tasks are not quite related, OWM may perform very badly. Is this the case in the experiment?\n4. It is not clear why the proposed method can solve the issue that OWM faces with (bad accuracy when tasks are not quite related).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gezXh3tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces Principal Components Projection, a method that computes the principal components of input vectors, using them to train on a transformed input space and to project gradient updates. Experiments show improved results over OWM (the method that this paper builds on) and EWC.\n\nIf I understand correctly (which I think may not be the case), the principal component vectors are computed after the first forward/backward pass of each task, for the inputs to each layer (C_l^k). These principal components are then fixed, the orthogonal projection matrix P_l^k is then found, and then normal training is iterated until convergence using this C_l^k and P_l^k.\n\nQuestions:\n- Seeing as (especially for the first task), weights are initialised randomly, why does this method provide reasonable principal components for layers after the first layer? \n- I also do not understand why the dxd projection matrix P, which is orthogonal to all previous basis matrices C, has the property span(P^i) \\subset span(P^j) for i < j. Surely as more basis matrices are found, then the orthogonal space restricts in size.\n- I also do not understand Equation 1. What is \\grad{W}? If it is, as defined 2 pages later, 'the backpropagation with respect to X_{k+1}' [or X_k here], then is Equation 1 saying that only one gradient step is used per task?\n\nThe experiments seem reasonable, except that there are no standard deviations on the results. However, as far as I'm aware, these experimental protocols (dataset and model size) are not used in other papers: it would be nice to see experiments which match previous papers' protocols, for example with MNIST and CIFAR-10 at least (other papers use smaller model sizes).\n\nAs it is currently, I am unable to understand the paper despite spending some time trying to understand it. I am therefore giving the paper a weak reject. Hopefully the authors can answer my questions.\n\nFinally, some minor specific suggestions for improving the writing:\n- Immediately after Equation 12, there is \\grad{P^j} instead of \\grad{W^j}{P^{k-2}}\n- The paragraph before Equation 13 uses 't' instead of 'k' sometimes for task index\n- Use `   not ' for open quotation marks", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is no author response for this paper. The paper addresses the issue of catastrophic forgetting in continual learning. The authors build upon the idea from [Zheng,2019], namely finding gradient updates in the space perpendicular to the input vectors of the previous tasks resulting in less forgetting, and propose an improvement, namely to use principal component analysis to enable learning new tasks without restricting their solution space as in [Zheng,2019]. 
While the reviewers acknowledge the importance to study continual learning, they raised several concerns that were viewed by the AC as critical issues: (1) convincing experimental evaluation -- an analysis that clearly shows how and when the proposed method can solve the issue that [Zheng,2019] faces with (task similarity/dissimilarity scenario) would substantially strengthen the evaluation and would allow to assess the scope and contributions of this work; also see R3’s detailed concerns and questions on empirical evaluation, R2’s suggestion to follow the standard protocols, and R1’s suggestion to use PackNet and HAT as baselines for comparison;  (2) lack of presentation clarity -- see R2’s concerns how to improve, and R1’s suggestions on how to better position the paper. 
A general consensus among reviewers and AC suggests, in its current state the manuscript is not ready for a publication. It needs clarifications, more empirical studies and polish to achieve the desired goal.

Decision: Reject