Learning to Remember from a Multi-Task Teacher
Authors: Yuwen Xiong
Mengye Ren
Raquel Urtasun
Venue:  (2020)
Keywords: Meta-learning
sequential learning
catastrophic forgetting

Abstract: Recent studies on catastrophic forgetting during sequential learning typically focus on fixing the accuracy of the predictions for a previously learned task. In this paper we argue that the outputs of neural networks are subject to rapid changes when learning a new data distribution, and networks that appear to "forget" everything still contain useful representation towards previous tasks. We thus propose to enforce the output accuracy to stay the same, we should aim to reduce the effect of catastrophic forgetting on the representation level, as the output layer can be quickly recovered later with a small number of examples. Towards this goal, we propose an experimental setup that measures the amount of representational forgetting, and develop a novel meta-learning algorithm to overcome this issue. The proposed meta-learner produces weight updates of a sequential learning network,  mimicking a multi-task teacher network's representation. We show that our meta-learner can improve its learned representations on new tasks, while maintaining a good representation for old tasks.

Under review as a conference paper at ICLR 2020
LEARNING TO REMEMBER FROM A MULTI-TASK TEACHER
Anonymous authors Paper under double-blind review
ABSTRACT
Recent studies catastrophic forgetting during sequential learning typically focus on fixing the accuracy of the predictions for a previously learned task. this paper we argue that the outputs of neural networks are subject to rapid changes when learning a new data distribution, and networks that appear to "forget" everything still contain useful representation towards previous tasks. Instead of enforcing the output accuracy to stay the same, we propose to reduce the effect of catastrophic forgetting on the representation level, as the output layer can be quickly recovered later with a small number of examples. Towards this goal, we propose an experi- mental setup that measures the amount of representational forgetting, and develop a novel meta-learning algorithm to overcome this issue. The proposed meta-learner produces weight updates of sequential learning network, mimicking multi-task teacher network's 's representation. We show that our meta-learner can improve its learned representations on new tasks, while maintaining good representation for old tasks.
INTRODUCTION
An intelligent agent needs to deal with a dynamic world and is typically presented with sequential tasks that are highly correlated in time yet constantly changing. Newborns learn to build generic representations from video and audio streaming input. Kids can learn highly skilled tasks such as skiing and swimming sequentially without worrying about forgetting one another. Humans seem to have a robust way of learning representations from sequential inputs (and tasks), yet state-of-the- art machine learning algorithms rely heavily on uniformly sampled training examples from the same distribution.
One of the major challenges in sequential learning of neural networks the issue of catastrophic forgetting (McCloskey & Cohen 1989)-afte after neural network is trained on new task, its perfor- mance on old tasks drops significantly. Despite several attempts, this problem remains unsolved. Explicit weight regularization methods (Evgeniou & Pontil 2004 Kirkpatrick et al. 2016 Lee et al., 2017) often rely on simplistic assumptions on the shape of the weight posterior distribution. Model compression methods (Serrà et al. [0188 Fernando et al.) 2017: Mallya & Lazebnik (018)) seem promising on existing benchmarks, however the underlying mechanism is to train small individual networks that may lack global cooperation, limiting factor when learning a large number of classes towards generic representation. Generative models (Shin et al. [0177] Kemker & Kanan 2018 Venkatesan al. 2017) seem to be natural choice; however, training high quality generative model is far from trivial, oftentimes more complex than training the original network itself.
Despite the variety of models that have been proposed, there seems be lack of general under- standing what kind of knowledge is being forgotten and to what extent it can be recovered. Recent research places much of its focus on maintaining the output performance of previous tasks. In this paper we argue that this can be misleading since the output layer of network is very sensitive to changes the output distribution. Instead, here we would like to understand how much of the performance drop is related to the lack of training on previous output layers versus the loss of information in the newly learned representation. Towards this goal, we exploit linear decoding layer to measure the amount of catastrophic forgetting on the representation level. This gives insights on whether the drop in performance is likely to be recovered by re-leaming the output layer from very few examples.
Motivated by recent progress on meta-learning (Jaderberg et al. 2017 Andrychowicz et al, 2016 Metz et al. 2018), in this paper we propose to learn weight update rule overcome representational

---

Under review as a conference paper at ICLR 2020
forgetting. meta-learning algorithm learn such rule by rolling out many sequential learning experiences during training. In human language acquisition, it is found that children who lost their first language maintain similar brain activation to bilingual speakers (Pierce et al., (0144) Inspired by this fact, we propose a novel meta-learning algorithm that tries to mimic a multi-task teacher network's 's representation, an offline oracle in sequential learning setup, since multi-task learning has simultaneous access to all tasks whereas our sequential learning algorithm only has access to one
task time.
In summary, the contributions of our paper are two-fold. First, we propose a measure of catas- trophic forgetting at the representation level, which provides more insight on the amount previous knowledge forgotten in the new task. Second, we develop a new meta-learning algorithm that can predict weight updates that are less prone to catastrophic forgetting than standard backpropagation. We demonstrate the effectiveness of our approach on the MNIST (Lecun et al. (998), FashionM- NIST (Xiao al. 2017). CIFAR-10, and CIFAR-101 (Krizhevsky 2009) datasets, and find that our meta-learner able to generalize to unseen object classes from meta-training.
The rest of the paper is organized as follows: we first survey existing literature in Section 2 Section describes representational forgetting in deep neural networks, and our experimental setup to measure details our proposed meta-learning algorithm, followed by experimental results in Section
it; Section 4
2 RELATED WORK
Catastrophic forgetting in sequential learning has been studied in the early literature of neural networks (McCloskey & Cohen 1989 Ratcliff 1990, French 1991] Mcrae & Hetherington 1993 French 1999). The degree of forgetting usually measured in terms of the amount of time, typically the number of iterations, saved to relearn the old task. This metric has several drawbacks as the number of iterations can be fairly sensitive to the choice of optimization hyper-parameters and network architecture. Furthermore, the network may never recover fully the original performance.
With the recent success of deep learning, the issue of catastrophic forgetting has regained attention in the research community. Unlike classical methods, recent papers measure the old task performance immediately after training on the new task (Kirkpatrick et al., 2016 Goodfellow et al. 2013 Li
Hoiem 2018). In this setting, networks that are trained in a sequential manner over tasks suffer dramatically, due to the fact that the previous output classifier branches are no longer tuned to the newly learned representations.
In this paper we follow the early literature, and allow the network to relearn on old tasks, as we believe it is natural to review the old task before testing on it again. But instead of counting the number of iterations to recover the old task performance, we propose to train on top the newly learned representation using small decoding model to perform the old task. We believe this measure well captures the amount of representational forgetting. To this end, we exploit a simple linear readout layer, it is relatively fast to train and is more robust than measuring the number of recovering steps.
One way to address catastrophic forgetting is through explicit regularization. Evgeniou & Pontil (2004) add 1.2 regularizer to ensure that the new weights do not "drift" away from the old weights. Elastic weight consolidation (Kirkpatrick et al. (016) computes the strength of the regularizer on each weight dimension using a diagonal approximation of the Fisher information matrix. Zenke iet all]|(220) (2017) propose to directly approximate the regularization strength online. Lee et al. (2017) incrementally match the moments of posterior Gaussian distributions of new tasks. These methods are often motivated from a simple quadratic loss surface, and can be potentially limiting their ability to deal with more complex learning dynamics. Regularization can also be imposed on the activation level: I Learning without Forgetting (Li & Hoiem] (018)) regularizes the network such that the logits of the new examples on the old classifier remain similar. While this framework is more flexible, using the old activations to distill new tasks can be less informative if the network has not seen enough classes.
In constrast to continuous regularization, model compression based approaches (Mallya & Lazebnik 2018 Fernando et al. 2017 Rusu et al. 2016:) Yoon et al.] 2018) discretely allocate certain capacity of network towards learning tasks. PackNet (Mallya & Lazebnik] 2018) applies network pruning in between sequential tasks, so that the pruned neurons can be re-allocated. PathNet (Fernando et al.
2017 uses genetic algorithms select pathways of the network for reuse. HAT (Serrà et al. 2018)
2

---

Under review as a conference paper at ICLR 2020
A
CIFAR-SA
B
CIFAR-BB
oo O
los Imagees
Readout
Acc
Readout
-24
Original
Task A
Tesk
Figure I: Using a readout layer to measure catastrophic forgetting on representations. A) A network first pre-trained on Task A. B) Then is finetuned on Task B. C) We then feed in Task A training data to the network, and record the representation at the last layer prior to the classification layer. We re-train& readout layer to recover Task A output. Test accuracy on Task A is recorded as "Readout", and original classification layer accuracy is recorded as "Original". Chance is 20%.
learns hard binary mask for each weight connection. |usu et al. (016) add connections from old frozen modules towards newly allocated modules, at the price of learning more intermediate layers, thus scaling quadratically with the number of tasks. Yoon al. (2018) propose dynamically prune and allocate neurons at the same time. In comparison, our meta-learner can also be interpreted as implicitly learning to perform dynamic capacity re-allocation without increasing the network size; however, our approach does not choose a discrete set of neurons synapses to update, but try to mimic the activation responses from multi-task network. This allows the algorithm to learn to implicitly allocate weight subspaces for learning the new tasks.
Another class of methods store a subset of the old data, so that the old task can be jointly trained. ieeuf. et (2017) propose to choose representative exemplars of old tasks. Gradient episodic memory (Lopez-Paz & Ranzato 2017) stores old examples and makes sure that the new example only updates in the direction that agrees with the gradient directions of old examples, [Sprechmann et al. (2018) explore a learnable memory architecture to dynamically store and retrieve examples facilitating sequential learning. Though effective, storing data points costs additional storage and it may also lack biological plausibility. To address the issue of data storage, generative models have also been used avoid storing old raw data (Shin al. 2017: Venkatesan al.) 2017 Kemker & Kanan, 2018 van der Ven & Tolias 2018). Although generative models enjoy the benefits of data storage based models, the final performance heavily depends on the quality of the generated data, because training a competitive generative model itself may be more complex and take more capacity
than the original network.
Our proposed model is inspired from prior work in meta- meta-learming using a learned parameterized weight update rule (Bengio et al.] 1990) instead of standard optimization methods. Synthetic gradient (Jaderberg et al.) (017)) uses an MLP to predict the gradient direction when performing forward passes, allowing asynchronous weight updates across layers. (Adrychhwiczz et al. (2016);
Ravi & Larochelle (2017) use recurrent network to predict updates. Metz et al. (2018) propose to learn an unsupervised learning rule based solcly on activations. [Mconi et al. 2018 2019) combine a learned Hebbian plasticity rule with learned weights. The largest difference between our proposed model and prior work is the fact that instead of predicting the gradients or improve the task performance at the end of the training episode, our meta-leamnr is trained with a teacher network's activation as supervision.
3 REPRESENTATIONAL FORGETTING IN SEQUENTIAL LEARNING
Sequential learning is the process of learning tasks sequentially without revisiting previous tasks. For simplicity, in this paper we study the setting of two tasks, where network is first presented with Task A and then Task The exposure to Task A can bring positive benefits towards learning Task B if the two tasks are similar. This is a property often studied in the transfer learning literature. Unfortunately, it is challenging to maintain the initial performance of the model learned on Task A (when updating it to perform well task B), especially when the old task environment is not available to the agent. For example, a commercial robot needs to adapt and learn in new environments while the original training data cannot be shipped together with the learning algorithm.
3

---

Under review as a conference paper at ICLR 2020
[000000
Legend
Teacher (mudi-ttok))
300
RF
Shudent (eeeg aentiss)
Mess-lear
N/ 300
9/4
R]
T
3, U.x
Figure 2: Overview of our purposed method during one training step: 1) the teacher network is updated using SGD on multi-task data; 2) the student network is updated using the meta-learner on Task B data only; 3) the multi-task data are fed into both networks and we record the representations as h and h; 4) the meta module then updated to minimize the difference between and h. At meta-test time, the teacher network is no longer present, and we update the student network solely with the trained meta-learner for the entire training sequence.
LSTM
LSTM
W
Weightt
Wy
Gradlentt
8
Weegh Liddat
W TrhhAn Aw
Freeatts
Post act
g
Before update
Meta-learner
Alter update
Figure 3: Our meta-learne module is a stacked LSTM network. For a given output neuron hi, the meta-network takes its activation, its pre-activations, current weights and gradients, and output the weight updates assiciated with h². Weights in the same layer will be shared single meta-network, while different layers will have different nmeta- nmeta-networks.
Catastrophic forgetting (McCloskey & Cohen 1989) occurs when learning agent forgets the old task after adapting to the new task. While recent literature solely focuses on the output performance of agent the previous tasks, it is unclear whether the agent "truly forgets" the prior experiences, or only the output layers are miscalibrated due to learning the new task. The latter issue has an easy solution as simply training the output classifier on the old task for few iterations can recover most of the performance loss. This is similar to human revisiting previously learned skills.
In this paper we study how much forgetting occurs the representation level. Towards this goal, propose the experiment setup illustrated in Figure [ where a network first trained on Task A, and learns the representation of the inputs, referred to the final layer prior to the classification head. Next, a different task, Task B is introduced. In the second stage of training, the model no longer has access Task A. It then fine-tunes all its layers on Task B, since learning better representation will be useful for the new task. To test the amount of forgetting on Task A, we train a linear readout layer on the newly learned representations, using training examples from Task A, and evaluate the test performance.
We repeatedly do this readout training throughout the learning of task B. the beginning, the readout performance is close to the original Task A performance, and later, the features become less selective towards Task A. As shown in Figure [ we indeed observe that the readout accuracy of Task A is constantly decreasing the learning process goes In contrast measuring the output performance directly ("Readout" vs. "Original" in Figure as was done prior catastrophic forgetting literature, the level of forgetting here is not dramatic. suggesting portion of forgetting happens due to the mis-calibration of the output layer. Representational forgetting is still significant-for the binary classification problem illustrated in Figure readout accuracy Task A drops 15% after 300 iterations of training on Task B.
Meta-leaming is general tool for to learn a new learning algorithm with desired properties. In the next section, we propose a novel meta-learning algorithm that directly addresses the issue of representational forgetting.
4

---

Under review as a conference paper at ICLR 2020
4 LEARNING FROM A MULTI- TASK TEACHER
As the learning process goes on in Task B, each gradient descent step can potentially erase useful features for the old task. Continuous regularization (Evgeniou & Pontil 2004 Kirkpatrick et al. 20166) Zenke al. 2017 7) or discrete model pruning based methods (Mallya & Lazebnik] 2018) Fernando et al. 2017 Rusu et al., [20T6] Yoon et al., 2018), as discussed in Section 2 can hurt the capacity of learning new tasks and limit the sharing of distributed representation. Manually designing a sophisticated learning objective is not obvious, thus we are interested in learning learning rule that can predict dynamic weight updates alongside the training of Task B.
For simplicity, describe the setting of fully connected layer, shown in Figure B where indexes pre-activations and post-activations. A weight synapse Wa, connects to its pre-activation h, and post- activation hz. Let It.j be the gradient of the connection obtained through regular backpropagation. Our meta-learner f is implemented as long short-term memory (LSTM) network that takes as inputs the weight connection, gradients, inputs and outputs, similar to what has been done by Metz et al.) (2018). The network intuitively predicts a dynamic gating that modulates the plasticity of synapses: o., = f(W.jj J. h., hy;s 0), which is then multiplied with the original gradients form the updates of the synapse: Aw., 5.jj B.j- Finally, the weights are updated in a similar way to SGD, W.j= = .. -&A., .,j where layers as well by considering all filter locations together, with more details in Section 5.1
the learning rate. We can generalize this setup to convolutional
Learning this meta-learner, is however a non-trivial task. In (Metz et al., 2018), the learning process is unrolled just like a re- current neural network for a large number of steps, and meta- meta-learning is done using backpropagation through time, which is in- efficient since each meta-update can only be done in the outer loop.
Algorithm Learning to Remember from a Multi-Tann Task Teacher
Require: wo, DA, DB Ensure: 0 (Meta-parameterss
1: for i = 1... N do
2:
// Reinitialize teacher and student networks
3:
f- W0o
4: i < w0o
key insight of our paper is that, we can actually learn from multi-task teacher network as if it is an oracle to our se- quential learner. Similar to knowledge distillation (Hinton et al., 2015), where a student network learns activation patterns from a teacher network, here, we would like to train the meta-learner such that the student network has similar activation pattern compared to the teacher network.
5: Mcta-lcarner resets hidden state;
6:
// Reinitialize T-BPTT step to zero
7:
4- 0
for t T(i)
9:
Ta yu + GetMiniBBtthh (Da):
10:
yb + (PD)))
11:
TeachenNetBackwarrd/22 2a U 2b, ya U yo
12:
I StudentNerRackkwrr((rr ye; w);
13:
Teacher update with SGD
14:
w + og:
15:
# I takes weights, gradients and activations
This avoids using the final accuracy of an episode as the supervision signal, which can be very inefficient.
16:
Aú + 9 f g. d);
17:
// Student update with meta-learner f
18:
10 +- n0 oAw;
19:
TeacherNet(ar. U 36: w);
The idea of using a multi-task network as a teacher is inspired by the human language acquisition literature. It was found that adopted children who were separated from
20:
à StudentNet(z,, U zu; i)):
21:
L <- HuberLoss(h,, A);
22:
+ 1;
23:
Meta-learner update, backprop thru time steps + - n/T-BPTT(L, 8, s):
the Chinese language (their birth language)
24:
around one year old on average, still 25:
if L > Loss Tosshhreshold()) then
maintain activations in their left superior 26:
break; // Restart learning end if
temporal gyrus similar to French/Chinese 27:
bilingual speakers at the age of 9 to 17, even though they have no exposure Chi-
28:
if 2 T-BPTTStepS(i) then + 0; Reset T-BPTT steps
29:
30:
end if
nese for 13 years average (Pierce et al. 2014). We draw a parallel in our sequential learning framework here: whereas the net- work is allowed to forget the old task on the output level, the learned representation should resemble the one learned by multi-task network.
31:
end for
32:
end for
The overall algorithm has two nested loops, like many other meta-learning algorithms. The inner loop simulates the experience of doing a single learning process, and the outer loop rolls the network
5

---

Under review as a conference paper at ICLR 2020
back to its initial point and re-starts the learning again. Different from standard syper-paramette optimization, in our proposed algorithm, the meta-learner updated every step in the inner loop, which makes the training more efficient. The meta- meta-training procedure is detailed in Algorithm
In the beginning each inner loop, the teacher network and the student network share the same initialization that is pre-trained Task A. For every step in the inner loop, illustrated in Figure 2 the teacher network performs a regular SGD update using data from both the old and new tasks. The student network, who does have access the old task, computes the gradients on the new task and sends them to the meta-learner network, which then predicts a multiplicative gating. Now using the newly updated weights. we compare the representation difference between the teacher and student networks, and update the meta-learner to minimize this difference.
We use Huber loss multiplied with a scalar hyperparameter the objective for minimizing the representational differences. To help the meta-learner gradually make progress, we set up curriculum such that whenever the loss is greater than certain threshold, we will reinitialize the learning process to prevent the meta-learner deviating too far (see Line 23 Algorithm I. To speed up learning, we perform truncated backpropagation through time on the meta-learner LSTM. Gradient accumulation is reset whenever the number of unrolled steps is longer than the truncation steps (see Line 28 of Algorithm
5 EXPERIMENTS
In this section, we first give implementation details of our algorithm, and then report experimental results on three sets of experiments. the first set, we test whether our meta- veeta-leanning algorithm can learn useful learning rules through many sequential training episodes. In the second and third sets of experiments, we verify the generalization ability of our meta-learner, by using unseen classes and unseen initialization checkpoints for evaluation.
5.1 IMPLEMENTATION DETAILS
A separate three-layer stacked LSTM meta-leamer is learned for each layer of the student network (except the classification layer). The meta-learner uses ReLU activation functions in the hidden layers and tanh in the output layer. The weights and biases of the output layer are initialized to 0 and respectively, to produce a reasonable value the starting time. A learnable scaling coefficient is then applied to the output to adjust the range. For the classification layer, apply standard SGD without a meta-learner.
Meta-learner specification: For convolutional layers, we take the average over the spatial window of the pre- and post- activations. The convolutional kernel of size kH kw x Can flattened to vector as the input to the meta-learner which outputs the weight updates of size kn x kw Oin- For example, for a 3 x
10 convolutional layer, the input dimension of the meta-learner 3 x 10 (weights) +3 3 x 10 (gradients) +10 (pre-act)
(post-act) = 191, and the output dimension is 3 3 x = 90. For other layers, kH X kw is not applicable, and the input is Cin x 3 (weight, gradient, pre-act) +1 (post-act) and the output is C.n
Baselines: We compare our proposed method to several baselines:
SGD performs standard SGD on new tasks with the same learning rate as Task A.
SGD 0.1 is standard SGD with 0.1 learning rate. This see whether forgetting on Task A can be traded off with learning progress on Task B.
Learning without Forgetting (LwF) (Li & Hoiem 2018) distills new data the old classification branch as additional regularization. We validate the regularization coefficient and set to 1.0.
Elastic Weight Consolidation (EWC) (Kirkpatrick et al. (016)) adds quadratic regularizer on the weights, where the regularization strength computed as diagonal approximation of the Fisher information matrix. We validate the regularization coefficient and set it to 1.0.
5.2 EXPERIMENT 1: SEQUENTIAL LEARNING ON TWO TASKS
We first conduct experiments on MNIST and CIFAR- 10 to verify the effectiveness our meta- neaaneearner. To ensure that the meta-learner does not overfit to the training examples, split the data of Task B into two parts evenly, denoted as DB, and D D,, respectively. At meta-training time, only DB, will be used; at meta-test time, only DP, will be used.

---

Under review as a conference paper at ICLR 2020
Task
Task B
Task
Task B
0.5
0.0
12.5
6.-
-1.0-
10.0
8Acc
4Acc (SS)
2.--
4.0
Treeoe Teacher
adcc
-.0
4.0
3.0
4Acc (95)
3.0
7.5-
Frieze
0.2
SCD
5.0
23
50D
Teacher
-2.-
4.0
7.0-
2.5
-6.0
100
520
530
520
500
Update Staps
Update Steps
Update
Figure 4: Left: Exp 1: MNIST FNshionMNIST, "Freeze" has 98.37% on Task A and 75.64% on
Task B; Right: Exp I: CIFAR 5A 5B, "Freeze" has 92.31% % on Task A and 74.81 % on Task B. Error bar denotes standard error of the mean of runs.
Task
0.0
0.0
15.0
2.0
15.0
10.0
Freeor Texchar 10.0
40cc
3.0
Teachar
Freep
(Acc
.3.
10.0
5CD
6.1
10.0
SCD
5CD L.
Lif
LieF
IWC
EWC
44.
Dus
34.0
Oun
150
200
300 400
530
150
200
300
4io
530
100
203
300
450
500
10c 203
300
45o
500
Update Steps
Update Steps
Update Steps
Update Steps
Figure 5: Left: Exp 2: CIFAR 5A + 100 with unseen classes; Right: Exp 3: CIFAR 5A -> 100
with unseen initialization checkpoints. "Freeze" has 92.25% on Task and 71.68% on Task Error bar denotes standard error of the mean of 5 runs.
The network first pre-trained on MNIST (Lecun et al.) 1998 (i.e.. Task A) and then trained on FashionMNIST (Xiao et al., (017) (i.e.. Task B). The main network contains two 5x5 conv layers and two fully-connected layers with dimensions [20, 50, 500, 10]. respectively, and the meta-learner is a 3-layer LSTM with 32 hidden unit. SGD with learning rate 1e-2 and momentum of 0.5 is used throughout pre-training. During meta-learning. we use learning rate 5e-2 without momentum. We set the curriculum threshold to 20 and T-BPTT step to 5 initially and increase them by 5 and 2 every 1000 episodes until 35 and 11, respectively. The loss coefficient is set to 300. We train the meta-learner using the Adam optimizer with learning rate 1e-3 for a total of 4000 episodes. At meta-test time, we take the activations before the last layer and train a linear readout layer with 100 Adam optimization steps using learning rate 1e-1.
MNIST FashionMNIST::
We split the CIFAR-10 dataset into two subsets, the first subset (CIFAR-5A) consists of the first classes ("airplane", "automobile", "bird", "'att", "deer") and the second subset (CIFAR-5B consists of the remaining 5 classes (" ("dog". "frog", "horse", "ship", "truck"). ")uck"). A ResNet-32 He et al., 2016) network is first pre-trained on CIFAR-5A, with all BatchNorm (Toffe & Szegedy 2015) layers replaced by GroupNorm (Wu & He| 2018), using a learning rate of 0.1 and momentum of 0.9. During meta-lcarning, we use learning rate 0.1 without momentum. For pre-traiin pre-traniig we use 128 examples as mini-batch and for meta-learning The meta-learner is 3-layer LSTM with 64 hidden units for the convolutional kernels and 32 hidden units for y or /3 of the GroupNorm layers. We set the curriculum threshold to 20 and T-BPTT step to 5 initially and increase them by 5 and 2 every 300 episodes until 30 and 9, respectively. The loss scaling coefficient is set 300. We train the meta-learner using the Adam optimizer with learning rate 1e-3 for a total of 900 episodes. At meta-test time, we take the activations before the last layer and train linear readout network using 500 Adam optimizer steps with learning rate of le-1.
CIFAR-10:
use 128 for the teacher and 64 for the student (Task B only).
Results: Figure shows results on MNIST + FashionMNIST and CIFAR-10. The error bar is the standard error of the mean of 5 runs. All entries in the table use the proposed readout measure. We found that reducing learning rate cannot help prevent from catastrophic forgetting. Our meta-learner outperforms other methods by large margin on Task A. On Task B, our leta-learner has very similar performance to the teacher network, which matches our expectation.
5.3 EXPERIMENT 2: GENERALIZING TO UNSEEN CLASSES
In Experiment 1, Task is the same for both training and testing. To verify the generalization ability of our model to unseen classes, we utilize the CIFAR-100 dataset, to construct two different tasks, B, and B2 from disjoint subset of data. We start from an initial model trained on CIFAR-5A. at meta-training time we train meta-learner on Task B,, while meta-test time we evaluate meta-
7

---

Under review as a conference paper at ICLR 2020
Exp MNIST/Fushion
Exp CIFAR-SA/SB
Exp CIFAR-SA/100
Exp
CIEAR-5-50
Freeze
84.8
65.1
68.6 + 2.9
68.6 + 2.9
SGD
86.8 +: 1.0
68.7 1.5
71.0 +: 2.1
71.0 4: 2.1
SGD x 0.1
85.0 + 0.7
65.4 +: 2.1
69.7 + 2.3
69.7 + 2.3
LwF
87.2 +: 1.1
69.3 = 0.7
73.1 + 1.7
73.1 4 1.7
BWC
86.9 1.7
67.7 1.0
72.9
2.1
Ours
87.3
0.7
70.5 1.1
75.0 1.3
73.6 + 2.6
Teacher
88.0 + 1.1
76.7 0.3
81.9 + 1.2
81.9 + 1.2
Table 1: Joint task performance Experiment 1, 2 & 3, after 500 update steps. + denotes standard error of the mean of 5 runs.
55
1200
50
Comtuuu bhizhole
10-
urrell stop
1000
-0.5 0.5 convi
0.5
800
res2a_1
res2a_2
600
400
10
10
10
500
1000
1500
2000
training episode
rws2b_
Figure Training curve on Experiment 2
Figure Visualization of the meta-lcamer outputs
learner on Task B2. Unlike the previous experiment, the class definition changes from meta-trainin to meta-testing. This is a more practical setting since in reality we do not know priori which new task the model needs to adapt to.
We split CIFAR-100 dataset into two subsets. The first subset consists of the first 50 classes, the second subset consists of the remaining 50 classes. At vetta-trainnn time we randomly sample 5 classes from the first subset to constitute Task B at the beginning of every episode. And at meta-test time, we randomly sample classes from the second subset to constitute Task B2, and repeat it for 5 times to take the average performance. Task B1 and Task B2 have overlap for both images and classes. We set the curriculum threshold to 20 and the T-BPTT step to initially and increase them by 5 and 2 every 300 episodes until 50 and 17 and train the meta-learner for total of 2k episodes. Figure illustraaess the curriculum threshold schedule and number unrolled steps indicate the training progress. Other paprr-parrmeters are keep the same CIFAR-10 in this experiment.
CIFAR-100:
Results: As shown in Figure 5 our meta-learner generalizes well unseen classes and clearly outperforms other baselines. By the end of 500 update steps, the representation forgetting on Task A is 4% for model, compared over 10% for SGD, EWC, and LwF; meanwhile Task B our model performs much than SGD x0.1, only =2% behind other baselines.
5.4 EXPERIMENT 3: GENERALIZING TO UNSEEN INITIALIZATION
In Experiment remove the assumption on fixed initialization checkpoint. In order for the meta- learner to generalize to different initialization state, during training time we provide 100 different pretrained checkpoints and each training episode uses different checkpoint. At time, an unseen checkpoint is provided.
Results:
As shown in Figure 5 meta-learner generalizes well unseen initialization (as well as unseen classes) and still outperforms other baselines. The performance drops a little as expected, since the meta-leamer model has learn adapt different initializations during meta training/test.
Joint task performance: We include the joint task performance Table I for all Experiments 1-3, where train joint classifier on the representations learned after 500 update steps Task B. It can be confirmed that the sequentially learned representations produced by the meta-learner are useful to both tasks, outperforming other sequential learning baselines.
5.5 VISUALIZATION OF META-LEARNER OUTPUTS
To further understanding the behavior of our meta-learner, we visualize the distribution the output of the meta-learner (i.e. the gradient multiplier d) in Figure Our meta-learner produces non-trivial outputs that are not simply a global scaling of the learning rate. Sometimes the multiplier can be negative, which means the final update direction is opposite to the gradient descent direction. It shows that the meta-learner can learn to dynamically modify the gradient direction to prevent catastrophic forgetting.
8

---

Under review as a conference paper at ICLR 2020
6 CONCLUSION AND FUTURE WORK
Catastrophic forgetting handicaps state-of-the- art deep neural networks from learning online tasks in the wild. This paper studies the effect of representational forgetting in a sequential learning framework. In particular, we propose to add a linear readout layer to test the amount of forgetting at the representation level, where a significant drop in performance on old tasks is still observed, consistent with prior literature. We then propose to train a meta-learner to predict the weight updates, with supervision from multi-task teacher network. Our meta-learner is able to overcome catastrophic forgetting while improving its performance on new tasks. We further verify that our meta- meta-learner has the ability generalize to unseen classes and checkpoint initializations. Currently made the meta-learner successful at predicting weight updates for up to 500 steps, but we still find challenging to let it generalize even longer sequences. In the future, expect these issues can be addressed by potentially training the meta-learner with longer sequences and more sequential tasks with more computational time, and combining meta-learning techniques for dealing with longer horizon (c.g. Liao et al. (2018); Metz et al. (2019)).
have
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems 29, NIPS, 2016.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Université de Montréal, Département d' d'informatique de recherche .... 1990.
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In Proceedings of the tenth ACM SIGKDD international conference Knowledge discovery and data mining. ACM, 2004.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, 0tbs/1701.087344 2017.
Robert M French. Using semi-distributed representations to overcome catastrophic forgetting in connectionist networks. 1991.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-135, 115,, 1999.
Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv:1312.62111 2013.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, CVPR, 2016.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, Abs//5033.0251,, 2015.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML, 2015.
Max Jaderberg. Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In 6th International Conference on Learning Representations, ICLR, 2018.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barbbkkk,,rriska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. CoRR, abs/161 2.00796, 2016.

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-232, Nov 1998. ISSN 0018-9219. doi: 10.1109/5.726791.
Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Wer Moo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching, In Advances in Neural Information Processing Systems 30, NIPS, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2935-2947, 2018.
Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urtasun, and Richard S. Zemel. Reviving and improving recurrent back-propagation. In Proceedings of the 35th International Conference on Machine Learning, JCML, 2018.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning, In Advances in Neural Information Processing Systems 30, NIPS, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Ken Merae and Phil A. Hetherington. Catastrophic interference is eliminated in pretrained networks. In Proceedings of the 15h Annual Conference of the Cognitive Science Society. Erlbaum, 1993.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Leanstin unsupervised learning rules. CoRR,
2018.
Luke Metz, Niru Maheswaranathan. Jeremy Nixon, C. Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In Proceedings of the 36th International Conference on Machine Learning, JCML,, 2019.
Thomas Miconi, Kenneth O. Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. In Proceedings of the 35th International Conference on Machine Learning, ICML, 2018.
Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth O. Stanley. Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity. In 7th International Conference on Learning Representations, ICLR, 2019.
Lara J. Pierce, Denise Klein, Jen-Kai Chen, Audrey Delcenserie, and Fred Genesee. Mapping the unconscious maintenance of lost first language. Proceedings of the National Academy of Sciences, 111(4):1731441731 2014. ISSN 0027-8424.
doi: 10. 10.1073/pnas. 14094114011111
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2)2285, 1990.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In 5th International Conference on Learning Representations, ICIR, 2017.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR, 2017.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, Abs/1606.04671 2016.
10

---

Under review as a conference paper at ICLR 2020
Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of the 35th International Conference on Machine Learning, ICML, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems 30, NIPS, 2017.
Pablo Sprechmann, Siddhant M. Jayakumar, Jack W. Rae, Alexander Pritzel, Adrià Puigdomênech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell. Memory-based parameter adaptation. In 6th International Conference Learning Representations, ICLR, 2018.
Gido M. van der Ven and Andreas S. Tolias. Generative replay with feedback connections as a general strategy for continual learning. CoRR, abx/1809.10635, 2018.
Ragav Venkatesan, Hemanth Venkateswara, Sethuraman Panchanathan, and Baoxin Li. A strategy for an uncompromising incremental learner. CoRR, abs/1705.00744, 2017.
Yuxin Wu and Kaiming He. Group normalization. In 15th European Conference Computer Vision, ECCV, 2018
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, ICLR, 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.


Reviews:
[{"review_id": "rJgWaM12KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###  Summary\n\n- The paper demonstrates that neural networks that appear to have forgotten an old task still contain useful information of that task in their representation layers. \n- The paper proposes to meta-learn an update rule (parameterized by an LSTM) that acts as a gating mechanism (or plasticity) for each learnable parameter at meta-test time. \n- To meta-learn the update rule, the paper proposes minimizing the difference between representations of a teacher and student neural network. The teacher neural network learns from a batch of data sampled IID from the distribution of the complete dataset whereas the student neural network samples a batch only from the current task.\n\n### Decision with reasons \n\nI vote for rejecting the paper.\n\n1- The claim that neural networks forget mostly due to a miscalibration of the output layer is not well supported empirically (The drop in readout accuracy in Figure 1 is still significant). If the claim is only to the extent that the drop in readout accuracy is slower than original accuracy, then it's not interesting or new. (This is what I believed in before reading the paper as well). \n\n2- While the underlying idea in the paper for learning an update rule is promising and sound, the paper is missing baselines that also use the meta-training dataset in some way. Moreover, a meta-learned update rule is only useful if it can discover some general underlying learning principles. In this paper, the meta-train and meta-test settings are too similar to see if that is the case. \n\n\n### Supporting arguments for the reasons for the decision.\n\n1- The paper claims that catastrophic forgetting in a neural network is partly due to miscalibration of the last layer, and the representation layer of the neural network still contain useful information. However, the only supporting evidence for this claim is that readout accuracy does not drop as quickly as the original accuracy (Figure 1). \n\nFirst, the drop in readout accuracy is still significant to term forgetting 'catastrophic.' Secondly, figure 1 only report results after 300 steps. A more interesting question is the difference between the accuracies when the network has been trained on Task B till convergence. Secondly, it is important to report the read-out accuracy for task A on a random Neural Network of the same architecture to see if the Neural network is maintaining information in the representation layer (as the authors claim), or if a linear classifier on a random CNN is just a strong baseline (Shown to be a strong baseline in many recent papers. One example is Anand et.al 2019 [1])\n\n2- The motivation behind meta-learning an update rule is to discover underlying learning principles that generalize to new settings. Metz et. al. 2019, for example, showed that their learned update rule could be applied to networks with different architecture, non-linearities, and datasets (They went as far as showing it worked on different data modalities.)\n\nAll the results in this paper, however, are for a fixed architecture (The authors do look at generalization to unseen classes, but we care about generalization to arbitrary architectures/problems when meta-learning an update rule). The data at meta-train and meta-test time are also very similar (Different parts of the same dataset). The empirical results, consequently, are not very convincing. Moreover, by reading between the lines, it can be inferred that the learned update rule is very finicky. For instance, to generalize just to unseen initializations, the authors had to use 100 different initializations at meta-training time. That does not instill a lot of confidence in me about the stability of the learned update rule. \n\nFinally, the paper proposes the complex student-teacher learning paradigm while skipping a simple baseline: training on the student model by using data from Task B in the support set and using data from A and B in the query set during meta-training. A similar procedure was proposed by Javed and White 2019 [3]. (Note that the current baselines in the paper do not use the meta-training data at all which makes the comparison extremely unfair. Moreover, even a simple baseline such as LwF that does not use meta-training performs almost as well (See Table 1).) \n\n### Additional evidence that can change my evaluation\n \n1- Showing that the meta-learned update rule can be applied to different architectures/non-linearities/datasets (Train on one dataset, test on another). \n\n### Minor comments that did not play a part in my decision, but should be addressed nonetheless. \n\nThe paper should cite the classic paper by Yoshua. et.al (1991) which proposed the idea of meta-learning an update rule [2]. \n\nThe paper, in its current form, needs to be proofread and reorganized. There are many errors in the grammar (For example just in the first paragraph, New borns -> Newborns, a same -> the same (or 'a distribution')). I find passing my writing through the free version of Grammarly very helpful in getting rid of most such errors. \n\nThe organization of the paper is also not very clear. For example, the third paragraph in \"Related Work\" is about the method proposed in the paper whereas the second and fourth are about related work. \n\nThe writing is also occasionally ambigious. For instance: \n\n\"In human language acquisition, it is found that children who lost their first language maintain similar brain activation to bilingual speakers (Pierce et al., 2014).\n\nInspired by this fact, we propose a novel meta-learning algorithm that tries to mimic a multi-task teacher network’s representation, an offline oracle in our sequential learning setup, since multi-task learning has simultaneous access to all tasks whereas our sequential learning algorithm only has access to one task at a time.\"\n\nIt is not clear how the method in the second paragraph is inspired from the statement in the first paragraph. \n\nI did not take writing quality in account when giving my score because openreview allows updating the paper during the review process. I hope that authors would fix these issues during the writing process. \n\nOn an unrelated note, the figures in the paper are well made and clear.  It is possible to understand the proposed methodology just from the figures. \n\n[1] Unsupervised State Representation Learning in Atari https://arxiv.org/abs/1906.08226\n\n[2] Learning a Synaptic Learning Rule https://mila.quebec/wp-content/uploads/2019/08/bengio_1991_ijcnn.pdf\n\n[3] Meta-Learning Representations for Continual Learning https://arxiv.org/abs/1905.12588\n\n\n#### UPDATE\nI gave the paper a 3 for the following reasons: \n\n1. The baselines do not use the meta-training dataset at all. This makes the comparisons unfair (Is the update rule learning some general learning principles or learning or induce good representations for the meta-training dataset?) \n\n2. The meta-train and meta-test settings are too similar. Learning an update rule only makes sense if we can discover some underlying learning principles. If the update rule is tied to a data-distribution, it is not extremely useful.\n\nIn the public discussion phase, the authors addressed both of my concerns. They added baselines which uses the meta-training dataset for learning a representation, and they added an experiment in which meta-testing is done on a different dataset. However: \n\n1. The baselines perform very close to the proposed method at a fraction of meta-training cost.  Rep*, even before doing any steps on Task B, results in better average performance than the method proposed in the paper. Moreover, the authors do not combine these baselines with existing methods to mitigate interference (Such as LwF) which can easily be done (and would probably increase the performance of the baseline noticeably)\n\n2. The meta-training and meta-testing datasets are still too similar in the new experiment (A downscaled TinyImagenet is very similar to CIFAR). Even though the results are more promising given the added experiment, I don't think they answer if the LSTM is learning some general learning principle or some task-specific heuristic for performing slightly better. \n\nR2's review (and the response to the review) also highlights an important problem -- the authors didn't tune the optimizers used by the baselines on the meta-training dataset. I suspect that a well-tuned adaptative optimizer (Like Adam) would reduce the gap between baselines and the author's method significantly. \n\nIs hundreds-of-hours of GPU compute worth negligible (or no) performance improvement in a very restricted setting (Since the learning rule doesn't seem to generalize based on existing results)? I'm inclined to say that it is not. As a result, I'm keeping my initial score. \n\nI do encourage the authors to investigate the proposed method more (It is a reasonable method) and try to empirically demonstrate that the LSTM is discovering some general learning principle.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkxKwvdptB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper explores learning without forgetting / the online learning setting. They employ a novel meta-learned learning algorithm to this end.\n\nWriting:\nFor the most part the writing was clear and easy to follow. There where a couple typos on the top of page 2 that should be fixed.\n\nMotivation:\nThe motivation for wanting meta-learning as well as various algorithmic choices are clear. \nThe one piece of motivation I did not fully understand is why not forgetting on the feature space is so important. My understanding of the method is that it should be applicable in both settings (with and without relearning the last layer). Infact, I would expect the difference between the meta-learned method and the baselines to only increase in this setting.\n\nI find the distillation based learning to be a clever alternative to the computationally heavy optimizing over past performance.\n\nExperiments:\nThis work provides a nice build up of experiments.\nExperiment 1 demonstrates the principles. In my opinion you should caution the reader given the meta-train, meta-test split. D_{B_1} and D_{B_2} are the same distribution and thus it will be easy for the learned update rule to memorize features. Given your learned update rule \narchitecture I doubt this will be the case though. I believe the authors are aware of this though as this issue is addressed in experiments 2 and 3.\n\nPlease include what the error bars are over in the captions.\n\nExperiments 2 and 3 are interesting and demonstrate the method on a more realistic setting. From the details it seems like this was difficult to get to work -- needing a complex schedule for example. Further elaboration or study of these details (e.g. ablations) would help the field. Also please include what the +- is for the experiments in table 1.\n\nFigure 6 is not referenced in the text. It was also difficult for me to understand though I finally got it.\n\nOverall, I believe the baselines could be made considerably stronger. Meta-learning expends considerable compute to find a good learned update rule. Spending similar amounts of compute tuning the baselines would be appreciated. Second, the meta-learned update rule presented here is essentially a learned optimizer and thus considerably more powerful than SGD. What optimizers did you use for LwF and EWC? Where the hyper parameters tuned here in an attempt to use similar compute? Where there learning rate schedules also tuned? \n\nQuestions / concerns:\n \nCost of running this not discussed. I would expect that both meta-training, and training are considerably more expensive. I am curious in particular \n\nOne motivation for meta-learning update rules in this way is that this cost can be amortized ahead of time and the learned update rule can transfer to new very different tasks. Without transfer like this, however, it's unclear if a method such as this is useful in general. Some discussion to this end I think would be helpful. I am not docking this work for not doing this type of generalization work though as we must start someplace and meta-training on similar data distributions is a logical place to do so.\n\nI am unclear as to your exact meta-training setup from algorithm 1. Does your meta-gradient (DL/dtheta) get computed every inner iteration (iteration of t)? If so how many steps do you back prop through? As of now it looks like your only backpropping a single iteration / application of f. Second, when computing this meta-gradient do you compute the true derivative or a first order approximation common in other work?\n \nOverall:\nI would recommend this paper for acceptance as it presents an interesting approach to solving the catastrophic forgetting issue with a compelling set of diverse experiments.  \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HklvmbYRYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper introduces a variation on measuring catastrophic forgetting in sequential learning at the representation level and attempts to resolve forgetting issue with the help of a meta-learner that predicts weight updates for previous tasks while it receives supervision from a multi-task learner teacher. The new method is evaluated on sequences of two tasks while task 1 data remains available at all times to the teacher.\n\nPros:\n(+): This paper is very well-written and very well-motivated. \n(+): Tackling continual learning from a meta-learning approach is novel and not yet well-explored. \n(+): Literature review is done precisely well.\n\nCons that significantly affected my score and resulted in rejecting the paper are two-fold. \n\nFirst, based on my understanding from the paper, it appears that this work has a significant contradictory assumption with a regular continual learning setup and that is to provide access to the entire dataset from an old task while we learn a new task. This changes the problem from continual/sequential/lifelong learning to multi-task learning. All the prior work that were beautifully reviewed in section 1 and 2 obey this assumption where access to previous tasks’ data is either impossible (ex. [1,3,4,5,6,7,8] in the below list ) or is very limited (ex. [2]). \n\nSecond, is the experimental setting. The experiments are accurately described and performed but authors have only considered sequence of 2 tasks which is far from being considered as a continual learning setting. I would like to ask the authors to explain how this method can be extended to multiple tasks and how much of the past data they should provide while training? Another drawback in the experiments is about the baselines. Despite addressing the most recent papers in section 2, authors have only made comparison against two relatively old approaches (EWC by Kirkpatrickthat et al from 2016 as well as LwF by Li & Hoiem presented at ECCV 2016, I believe the authors have cited the journal version of the work published in 2018 but the work is actually from ECCV 2016). Although these methods are still included as baselines in the literature, more recent approaches which have outperformed these need to be provided as well. I have provided a list of papers which achieved superior performance to the current baselines below which is arranged chronologically and is indeed not limited to this list as it is not realistic to list all prior work since 2016 in here. \n\nI would be happy to change my score if authors can address the above concerns about considering distinguishing multi-task learning from continual learning and providing a realistic evaluation setup with more than 2 tasks and comparison with current state of the art methods.\n\n[1] Zenke, Friedemann, Ben Poole, and Surya Ganguli. \"Continual learning through synaptic intelligence.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.\n[2] Lopez-Paz, David, and Marc'Aurelio Ranzato. \"Gradient episodic memory for continual learning.\" Advances in Neural Information Processing Systems. 2017.\n[3] Shin, Hanul, et al. \"Continual learning with deep generative replay.\" Advances in Neural Information Processing Systems. 2017.\n[4] Nguyen, Cuong V., et al. \"Variational continual learning.\" arXiv preprint arXiv:1710.10628 (2017).\n[5] Serrà, J., Surís, D., Miron, M. & Karatzoglou, A.. (2018). Overcoming Catastrophic Forgetting with Hard Attention to the Task. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:4548-4557\n[6] Schwarz, Jonathan, et al. \"Progress & compress: A scalable framework for continual learning.\" arXiv preprint arXiv:1805.06370 (2018).  \n[7] Mallya, Arun, and Svetlana Lazebnik. \"Packnet: Adding multiple tasks to a single network by iterative pruning.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n[8] Ebrahimi, Sayna, et al. \"Uncertainty-guided Continual Learning with Bayesian Neural Networks.\" arXiv preprint arXiv:1906.02425 (2019).\n[9] Aljundi, Rahaf, et al. \"Online continual learning with no task boundaries.\" arXiv preprint arXiv:1903.08671 (2019).\n\n------------------------------------------------------------------------------------------------------------------------------------------------------\n------------------------------------------------------------------------------------------------------------------------------------------------------\n------------------------------------------------------------------------------------------------------------------------------------------------------\nPOST-REBUTTAL review:\nI disagree with the authors claiming that this work is continual learning (sequential learning + avoiding forgetting).\nDespite introducing 9 recent continual learning work to authors in my initial review, they added 2 meta-learning baselines (MAML,REP), keeping 2 naive and old CL baselines is not acceptable. I reply to authors comment below regarding the baselines:\n\n[Authors' reply:] Lopez-Paz, David, and Marc'Aurelio Ranzato. \"Gradient episodic memory for continual learning.\" and Shin, Hanul, et al. \"Continual learning with deep generative replay.\" and Nguyen, Cuong V., et al. \"Variational continual learning” and Aljundi, Rahaf, et al. \"Online continual learning with no task boundaries.\" We argue that these paper have a different setting compared to ours since they require a buffer whereas our method has no storage of past data/gradient. Having past data storage can usually improve the performance and our method can potentially also get a boost. Having a data buffer can also cost a lot of memory storage depending on input/weight dimension. Therefore, we argue it won’t be a fair setting to compare with methods with data buffers.\n\n[Reviewer's reply:] GEM (Lopez-Paz et al., 2017) and its faster version (A-GEM) (Chaudhry, et al. 2018) and other memory based methods  such as MER (Riemer et al. 2018), ER-RES (Chaudhry et al. 2019), they use memory sizes of at most 6MB to store samples but they only do a **single epoch** through the data. So if it is not fair, it would be for those methods given the computational expenses of this paper. VCL (Nguyen et al. 2018) in its vanilla version does not use coreset memory if that is still your concern.\n\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n[Authors' reply:] Zenke, Friedemann, Ben Poole, and Surya Ganguli. \"Continual learning through synaptic intelligence.\" has very similar performance to EWC in their paper. We have cited this work already.\n\n[Reviewer's reply:] This method is an online version of EWC which is faster despite the on-par performance. So it has its own advantage.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n[Authors' reply:] Serrà, J., Surís, D., Miron, M. & Karatzoglou, A.. (2018). Overcoming Catastrophic Forgetting with Hard Attention to the Task and other pruning based papers. Thanks for pointing out. We have cited and will compare to them in the future. One thing to note is that, in these works, the model needs to know which task ID it is currently dealing with, and thus can turn on the pruning procedure for the next session. This can potentially be a limitation for dynamic incoming tasks.\n\n[Reviewer's reply:] Comparing with HAT paper (Serrà et al. 2018) is really easy using their provided code and is one of the strongest baselines in continual leaning literature. They do NOT do any pruning. Their approach simply learns an attention mask which regularizes weights and prevent changes on them without using any memory. Regarding the task number, this is indeed not an issue for your approach with 2 tasks.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\n[Authors' reply:] Ebrahimi, Sayna, et al. \"Uncertainty-guided Continual Learning with Bayesian Neural Networks.\" has not been published in a conference venue, and it may be too early to compare with it. \n\n[Reviewer's reply:] I agree that this work is not published and hence can't be asked for comparison but I encourage authors to read it.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nAuthors are neglecting one important difference between meta-learning and continual learning: in MAML and REP, it is assumed that we have access to ALL tasks distributions from which we sample from in the beginning (look at page 3, algorithm 1, line 3). This is in contrast to continual learning where one cannot even assume how many tasks will be given. Moreover, the computational expense of this work which causes performing more than 2 tasks to be a future work is also not acceptable when there are significantly cheaper and are able to do a lot more than 2. (In all the references I mentioned, the length of the sequence in experiments is at least 5.)\n\nWhile this work might be interesting to meta-learning community, I think it is far from being introduced as a method that prevents catastrophic forgetting and hence be included in the CL literature. Therefore, I intend to keep my score as reject. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the setting of continual learning. Instead of focusing on catastrophic forgetting measured in terms of the output performance of the previous tasks, the authors tackle forgetting that happens at the level of the feature representation via a meta-learning approach. As rightly acknowledged by R2, from a meta-learning perspective the work is quite interesting and demonstrates a number of promising results. 
However the reviewers have raised several important concerns that placed this work below the acceptance bar:
 (1) the current manuscript lacks convincing empirical evaluations that clearly show the benefits of the proposed approach over SOTA continual learning methods; specifically the generalization of the proposed strategy to more than two sequential tasks is essential; also see R1’s detailed suggestions that would strengthen the contributions of this approach in light of continual learning;
(2) training a meta-learner to predict the weight updates with supervision from a multi-task teacher network as an oracle, albeit nicely motivated, is unrealistic in the continual learning setting -- see R1’s detailed comments on this issue. 
(3) R2 and R3 expressed concerns regarding i) stronger baselines that are tuned to take advantage of the meta-learning data and ii) transferability to the different new tasks, i.e. dissimilarity of the meta-train and meta-test settings. Pleased to report that the authors showed and discussed in their response some initial qualitative results regarding these issues. An analysis on the performance of the proposed method when the meta-training and testing datasets are made progressively dissimilar would strengthen the evaluation the proposed meta-learning approach. 
There is a reviewer disagreement on this paper. AC can confirm that all three reviewers have read the rebuttal and have contributed to a long discussion. Among the aforementioned concerns, (3) did not have a decisive impact on the decision, but would be helpful to address in a subsequent revision. However, (1) and (2) make it very difficult to assess the benefits of the proposed approach, and were viewed by AC as critical issues. AC suggests, that in its current state the manuscript is not ready for a publication and needs a major revision before submitting for another round of reviews. We hope the reviews are useful for improving and revising the paper.

Decision: Reject