Continual Learning with Adaptive Weights (CLAW)
Authors: Tameem Adel
Han Zhao
Richard E. Turner
Venue:  (2020)
Keywords: Continual learning

Abstract: Approaches to continual learning aim to successfully learn a set of related tasks that arrive in an online manner. Recently, several frameworks have been developed which enable deep learning to be deployed in this learning scenario. A key modelling decision is to what extent the architecture should be shared across tasks. On the one hand, separately modelling each task avoids catastrophic forgetting but it does not support transfer learning and leads to large models. On the other hand, rigidly specifying a shared component and a task-specific part enables task transfer and limits the model size, but it is vulnerable to catastrophic forgetting and restricts the form of task-transfer that can occur. Ideally, the network should adaptively identify which parts of the network to share in a data driven way. Here we introduce such an approach called Continual Learning with Adaptive Weights (CLAW), which is based on probabilistic modelling and variational inference. Experiments show that CLAW achieves state-of-the-art performance on six benchmarks in terms of overall continual learning performance, as measured by classification accuracy, and in terms of addressing catastrophic forgetting. 

Published as a conference paper at ICLR 2020
CONTINUAL LEARNING WITH ADAPTIVE WEIGHTS (CLAW)
Tameem Adel
Han Zhao
Department of Engineering, University of Cambridge tah4 70 wec.m.ac..k
Carnegie Mellon University han. zhao@cs... cmu edu
Richard E. Turner
Department of Engineering, University of Cambridge Microsoft Research ret 2 60 cam. ac.uk
ABSTRACT
Approaches to continual learning aim to successfully learn a set of related tasks that arrive in online manner. Recently, several frameworks have been developed which enable deep learning to be deployed in this learning scenario. A key mod- elling decision
what extent the architecture should be shared across tasks. On the one hand, separately modelling each task avoids catastrophic forgetting but it does not support transfer learning and leads large models. On the other hand, rigidly specifying a shared component and a task-specific part enables task transfer and limits the model size, but is vulnerable to catastrophic forgetting and restricts the form of task-transfer that can occur. Ideally, the network should adaptively identify which parts of the network to share in a data driven way. Here we intro- duce such an approach called Continual Learning with Adaptive Weights (CLAW), which is based on probabilistic modelling and variational inference. Experiments show that CLAW achieves state-of-the-arr performance on six benchmarks in terms of overall continual learning performance, as measured by classification accuracy, and in terms of addressing catastrophic forgetting.
INTRODUCTION
Continual learning (CL), sometimes called lifelong or incremental learning, refers an online framework where the knowledge acquired from learning tasks in the past is kept and accumulated so that can be reused in the present and future. Data belonging to different tasks could potentially
be non i..dd (Schlimmer & Fisher 1986 Sutton & Whitehead 1993 Ring 1997, Schmidhuber
2013; Nguyen et al. 2018; Schmidhhberr 2018). A continual learner must be able to learn new task, crucially, without forgetting previous tasks (Ring) 1995 Srivastava et al., 2013 Schwarz et al. 2018 Serra et al. 2018 Hu et al., 2019). In addition, CL. frameworks should continually adapt to any domain shift occurring across tasks. The learning updates must be incremental i.e, the model is updated at each task only using the new data and the old model, without access to all previous data (from earlier tasks) - due speed, security and privacy constraints. A compromise must be found between adapting to new tasks and enforcing stability to preserve knowledge from previous tasks. Excessive adaptation could lead inadvertent forgetting of how perform earlier tasks. Indeed, catastrophic forgetting is one of the main pathologies in continual learning (McCloskey & Cohen
1989 |atellif]] 1990, Robins 1993. 1995| French] 1999| Pape et al. 201T Goodfellow et 2014 4a
Achille et a 2018 Kemker al. 2018 Kemker & Kanan 2018 Diaz- Rodriguez et al. 2018 Zeno
ct al. 2018 Ahn et 2019 Parisi et al. 2019: Pfulb & Gepperth 2019 Rajasegaran et al. 2019).
Many approaches to continual learning employ architecture which is divided priori into (i) slowly evolving, global part; and (ii) a quickly evolving, task-specific, local part. This is one way to enable multi-task transfer whilst mitigating catastrophic forgetting, which has proven to be effective (Rusu et al., 2016b; Fernando et al. 2017]] Yoon et al. 2018). albeit with limitations. Specifying priori the shared global, and task-specific local parts in the architecture restricts fiexibility. As

---

Published as a conference paper at ICLR 2020
more complex and heterogeneous tasks are considered, one would like more flexible, data-driven approach to determine the appropriate amount of sharing across tasks. Here, we aim at automating the architecture adaptation process so that each neuron of the network either be kept intact, i.. acting as global, or adapted to the new task locally. Our proposed variational inference framework is flexible enough to learn the range within which the adaptation parameters can vary. We introduce for each neuron one binary parameter controlling whether or not to adapt, and two parameters to control the magnitude of adaptation. All parameters are learnt via variational inference. We introduce our framework as an expansion of the variational continual learning algorithm (Nguyen al., 2018), whose variational and sequential Bayesian nature makes it convenient for our modelling and architecture adaptation procedure. Our modelling ideas can also be applied to other continual learning frameworks, see the Appendix for a brief discussion.
We highlight the following contributions: (1) A modelling framework which flexibly automates the adaptation of local and global parts of the (multi-task) continual architecture. This optimizes the tradeoff' between mitigating catastrophic forgetting and improving task transfer. (2) A probabilistic variational inference algorithm which supports incremental updates with adaptively learned param- eters. (3) The ability to combine modelling and inference approaches without any significant augmentation of the architecture (no new neurons are needed). (4) Sfate-of-the- art results in six experiments on five datasets, which demonstrate the effectiveness of our framework in terms of overall accuracy and reducing catastrophic forgetting.
2 BACKGROUND ON VARIATIONAL CONTINUAL LEARNING (VCL)
In this paper, we use Variational Continual Learning (VCL., Nguyen et al. 2018) as the underlying continual learning framework. However, our methods apply to other frameworks, (Section A.I... VCL is a variational Bayesian framework where the posterior of the model parameters
Appendix
0 is learnt and updated continually from sequence of T datasets, (x!"),
(n) yt ( ) where t
1.......7 . and Nr is the size of the dataset associated with the 1-th task. More specifically, denote by ply(0 (0,0 .)) the probability distribution returned by discriminative classifier with input æ; output
and parameters 0. For Dr = {v: (m) N J we approximate the intractable posterior pl p(0)Dr:t) after
observing the first t datasets via a tractable variational distribution g: as?
(i(
Z, 91 (0) (()p))))))
(1)
where qu is the prior p. p() = II
N: Ih n= p(y
p(y/ 0, अ. ()) and Z₁ the normalizing constant which
does not depend on O but only on the data D. This framework allows the approximate posterior qı (0) to be updated incrementally from the previous approximate posterior (0) in an online fashion.
In VCL., the approximation in (T) is performed by minimizing the following KI--divergence over family Q of tractable distributions:
q: (0) argmin KI. KI. (9(0)
||
1 Ze (0)
Q
This framework can be enhanced to further mitigate catastrophic forgetting by using coreset (Nguyen al., 2018)), i.e. representative set of data from previously observed tasks that can serve as memory and can be revisited before making decision. As discussed in the Related Work, this leads to overhead costs of memory and optimisation (selecting most representative data points). Previous work VCL considered simple models without automatic architecture building or adaptation.
OUR CLAW APPROACH
In earlier CL. approaches, the parts of the network architecture that are shared among the learnt tasks are designated a priori. To alleviate this rigidity and effectively balance adaptation and stability, propose multi-task, continual model in which the adaptation of the architecture is data-driven by learning which neurons need to be adapted as well as the maximum adaptation capacity for each. All the model parameters (including those used for adaptation) are estimated via an efficient variational
Here we suppress the dependence on the inputs in (1:) and p(D.(0) to lighten notation.
2

---

Published as a conference paper at ICLR 2020
inference algorithm which incrementally learns from data of the successive tasks, without need to store (nor generate) data from previous tasks and with no expansion in the network size.
3.1 MODELLING
With model parameters 0, the overall variational objective we aim at maximising at task with index t is equivalent to the following online marginal likelihood:
( -KL
-KL((q(((0 (q:(0)) qc-1(0))
Σ (q) logp(y (m(") 0)
(3)
We propose a framework where the architecture, whose parameters are 0, is flexibly adapted based on the available tasks, via learning procedure that will be described below. With each task, automate the adaptation of the neuron contributions. Both the adaptation decisions (i.c. whether or adapt) and the maximum allowed degree of adaptation for every neuron learnt. We refer to the binary adaptation variable as a. There another variable s that is learnt in a multi-task fashion to control the maximum degree of adaptation, such that the expression the task task-specific weights can differ from the global weights, in case the respective neuron is to be adapted. The parameter a depicts unconstrained adaptation, as described later
Ite-a limits how far
We illustrate the proposed model to perform this adaptation by learning the probabilistic contributions of the different neurons within the network architecture on a task-by-task basis. We follow this with the inference details. Steps of the proposed modeling are listed follows:
For task the classifier that we are modeling outputs: ΣNrr [log (1) w w")]
The task-specifet weights w can be expressed in terms of their global counterparts follows:
w" = -1 =11+b'aa)) o w.
(4)
The symbol denotes an element-wise (Hadamard) multiplication.
For each task T and each neuron j at layer i, ef is binary variable which indicates whether the corresponding weight adapted (a 1) unadapted (a7' = 0). Initially assume that the adaptation probability of follows a Bernoulli distribution with probability pi of's ~ Bernoulli(P:) (P..)) Since this Bernoulli is not straightforward to optimise, and to adopt a scalable inference procedure based on continuous latent variables, we replace this Bernoulli with Gaussian that has equivalent mean and variance from which we draw af For the sake of attaining higher fidelity than what is granted by standard Gaussian, we base our inference on variational Gaussian estimation. Though in a context different from continual learning and with different estimators, idea of replacing Bernoulli with an equivalent Gaussian has proven be effective with dropout (Srivastava et al. 2014 Kingma et al. 2015).
The approximation of the Bernoulli distribution by the corresponding Gaussian distribution is achieved by matching the mean and variance. The mean and variance of the Bernoulli distribution are Pi.i Pi.j (1 pij), respectively. A Gaussian distribution with the same mean and variance is used fit o:
atis NP PLJ(1 (Pj)))
(5)
The variable bT controls the strength of the adaptation and it limits the range of adaptation via:
1 b
Ite T
So that the maximum adaptation is s. The variable a an unconstrained adaptation value, similar Renals|2014p. The addition of 1 is to facilitate the usage of probability
to that in (Swietojanski & Renals [2014). distribution while still keeping an adaptation range allowing for the attenuation or amplification of each neuron's 's contribution.
We learn parameters a, and a, with corresponding for each neuron but avoid using the neuron indices here, as well as in other locations like 4 (6) and 77. for better readability.
"There can be parameter pi per layer instead of one Pi.j for each neuron j each layer but opt for
the latter for the sake of gaining further adaptation flcxibility.
3

---

Published as a conference paper at ICLR 2020
Before facing the first dataset and learning task t 1, the prior the weights (w(w) p(w) is chosen to be log-scale prior, which can be expressed as: plog w|) x c, where is a constant. The tog-scale prior can alternatively be described as:
p((wl) &
W
At a high level, adapting neuron contributions can be seen as a generalisation of attention mechanisms in the context of continual learning. Applying this adaptation procedure to the input leads to an attention mechanism. However, our approach is more general since we do not apply only to very bottom (i.e. input) layer, but throughout the whole network. We next show how our variational inference mechanism enables us to learn the adaptation parameters.
3.2 INFERENCE
We describe details related to the proposed variational inference mechanism. The adaptation parameters are included within the variational parameters.
The (unadapted version of the) model parameters 0 consist of the weight vectors w. To automate adaptation, perform inference Pi.. which would have otherwise been hyperparameter of the prior (Louizos et al. 2017 Molchanov et al. 2017 Ghosh et al. 2018). Multiplying w by (1 + bo) where is distributed according (5), then from with random noise variable N 1):
wh, ni (I [+ bijPis bus Pis(1 pijJe)
q(wiJ ni) ~ ~N ((null bLsPL)), bij rispis( -Pi))
(8)
From 7 and 8 the corresponding KL-divergence between the variational posterior of w, q(wl (w)r)
and the prior p( (w) is as follows. The subscripts are removed when q in turn is used as a subscript for improved readability. The variational parameters are Yi and Pi.j-
KL (q(Wis))))))) || pPwLs)) = Ba(w)>) (ogl((wi)/ns))/p(m....
Ba(win) (ogg(wi.lhgg) Eq(win) log p(WIJ) H(a(wi)lma))) Ba(why log p(W1.)
(9)
- (1 11+ + Pog -pii)) Eqinns log |e]
(10)
=o bij - 0.5 log Pij 0.5 0.5 - Pi.) +c+ Eqgtt)) log |e],
(11)
where the switch from to (10) is due to the entropy computation (Bernardo & Smith 2000) of
the Gaussian qi defined (8). The switch from (10) to (11 is due to using a log-scale
prior, similar to Appendix C in (Kingma et al.) 2013) and to Section 4.2 in (Molchanov et al. 2017).
Eq/iwly) log [c] computed via an accurate approximation similar equation (14) in (Molchanoy
et al. 2017). ,, with slightly different values of K1, k2 and lys, This is a very close approximation via
numerically pre-computing (q(wln) log |e| using a third degree polynomial (Kingma et al. 2015
Molchanov et a 2017).
This is the form of the KL-divergence between the approximate posterior after the first task and the prior. Afterwards, straightforward to see how this KL-divergence applies for the subsequent tasks in manner similar to 2 but while taking into account the posterior form and original prior.
The KL-divergence expression derived to Pi.j and then using samples from the respective distributions to assign values to diij adapted contributions of each neuron at each layer of the network are learnt per task. Values of pij are constrained between and during training via projected gradient descent.
is to be minimised. By minimising (I]) with respect
4

---

Published as a conference paper at ICLR 2020
Algorithm 1 Continual Learning with Adaptive Weights (CLAR)
Input: A sequence of T' datasets, {xx(", (m)
J Jn=i where = 1,,,....................... T and N₂ is the size of the
dataset associated with the t-th task.
Output: qe (0)),, where 0 are the model parameters. Initialise all p( Wi.jl) with log-scale prior, as in for I. 1...7 do Disclose the dataset {a;
(m) Vi Jn= for the current task t.
for i 1 ... # layers do for = 1 1... # neurons at layer i do
Compute Pı.j using stochastic gradient descent on
Compute Si,jt using (13).
Update the corresponding general value Si using 14 end for end for end for
3.2.1 LEARNING THE MAXIMUM ADAPTATION VALUES
Using (6) express the value of bI and neglecting the constant term therein since it does not affect the optimisation, the Kl.-divergence in (T)) is equivalent to:
KL (L(((wIJlmn)) || p(W1J))
slogs1j + log(1 c 0.5 log Pıj 00.5 1og Pı.) + Eq(wy) log [c].
(12)
Values of ji,j are learnt by minimising 12) with respect to ai.j- This subsection explains how to learn the maximum adaptation variable Sij- Values of the maximum Sij of the logistic function defined in (6) are learnt from multiple tasks. For each neuron j at layer i, there is general value Sjj and another value that is specific for each task t, referred to as Sij.t- This is similar to the meta-learning procedure proposed in (Finn et al. (0077). The following procedure to learn s is performed for each task such that: (i) the optimisation performed learn task-speeific value Si,,t benefits from the warm initialisation with the general value Si,j rather than a random initial condition; and then (ii) the new information obtained from the current task t reflected back to update the general value Si.j-
First divide the sample N₂ into two halves. For the first half, depart from the general value of Sijj as an initial condition, and use the assigned data examples from task t to learn the task- specific values Sj for the current task t.. For neuron at layer refer to the second term in (3). En Eq)(0) (0) [og p(y (m)) 0)] as (x, s1j). The set of parameters @ contains as well as other parameters, but we focus here on s the f notation since the following procedure is developed
to optimise s. Also, refer to the loss of the (classification) function f as Err(f) CE(f(2, 0) y))
where CE stands for the cross-entropy:
N₂/2
201 N₂
Sijjt Sijj
V
Σ
Err(f.(aa ya, S1j)).
(13)
d=1
Now use the second half of the data from task t. to update the general learnt value Sijj
2w02
Sij
Sij
V
Err(f((aa, Da, Sija)))
(14)
N
d=
12/21/2
Where w1 and We step-size parameters.
When testing samples from task t after having faced future tasks + +,, 1 2, the value of si.j used is the learnt s.j.t. There is only one value per neuron, so the overhead resulting from storing such values is negligible. The key steps of the algorithm are listed in Algorithm
At task t, the algorithmic complexity of a single joint update of the parameters based on the additive terms in (2) is OMF..))) where L is the number of layers in the network, D is the
5

---

Published as a conference paper at ICLR 2020
(largest) number of neurons within a single layer, E is the number of samples taken from the random noise variable e, and M is the minibatch size. Each a is obtained by taking one sample from the corresponding p. so that does not result in an overhead in terms of the complexity.
EXPERIMENTS
Our experiments mainly aim evaluating the following: (i) the overall performance of the introduced CLAW, depicted by the average classification accuracy over all the tasks; (ii) the extent to which catastrophic forgetting can be mitigated when deploying CLAW; and (iii) the achieved degree of positive forward transfer. The experiments demonstrate the effectiveness of CLAW in achieving state-of-the-an continual learning results measured by classification accuracy and by the achieved reduction in catastrophic forgetting. We also perform ablations in Section D in the Appendix which exhibit the relevance of each of the proposed adaptation parameters.
We perform six experiments on five datasets. The datasets in use are: MNIST (LeCun et al. 1998). notMNIST (Butalov (001T), Fashion- Reshion-MNIST (Xiao et al.) (017). Omniglot (I Laake al. [00 and CIFAR-100 (Krizhevsky & Hinton 2009). We compare the results obtained by CLAW to six different state-of-the-art continual learning algorithms: the VCL algorithm (Nguyen et al. 2018) (original form and one with a coreset), the elastic weight consolidation (EWC) algorithm (Kirkpatrick et al. 2017). the progress and compress (P&C) algorithm (Schwarz et al. 2018), the reinforced continual learning (RCL) algorithm (Xu & Zhu 2018), the one referred to as functional regularisation for continual learning (FRCL) using Gaussian processes (Titsias et al. 2019) and the learn-to-grow
(LTG) algorithm (Li al. 2019b).
4.1 OVERALL CLASSIFICATION ACCURACY
Our main metric is the all- all-important classification accuracy. We consider six continual learning experiments, based on the MNIST, notMNIST, Fashion-MNIST. Omniglot and CIFAR- 100 datasets. The introduced CLAW is compared to VCL versions: VCL with no coreset and VCL with 200-point coreset assembled by the K-center method (Nguyen et al. 2018). EWC, P&C, RCL, FRCL (its TR version) and LTC All the reported classification accuracy values reflect the average classification accuracy over all tasks the learner has trained on so far. More specifically, assume that the continual learner has just finished training on a task then the reported classification accuracy at time is the average accuracy value obtained from testing on equally sized sets each belonging to one of the tasks 1, 2. ..... t. For all the classification experiments, statistics reported are averages of ten repetitions. Statistical significance and standard error of the average classification accuracy obtained after completing the last two tasks of each experiment are displayed in Section E in the Appendix.
As can be seen in Figure D CLAW achieves of-te-of-the-art classification accuracy in all the six experiments. The minibatch size is 128 for Split MNIST and 256 for all the other experiments. More detailed descriptions of the results of every experiment are given next:
Permuted MNIST Using MNIST, Permuted MNIST a standard continual learning benchmark (Goodfellow et al. 2014a| Kirkpatrick et al. 2017] Zenke et al. (017)). For each task t, the correspond- ing dataset is formed by performing a fixed random permutation process on labeled MNIST images. This random permutation is unique per task, i.c. it differs for each task. For the hyperparameter of EWC, which controls the overall contribution from previous data, we experimented with two values, x 1 and À 100. We report the latter since it has always outperformed EWC with > = 1 in this experiment. EWC with > = 100 has also previously produced the best EWC classification results (guuyn al. 2018). In this experiment, fully connected single-head networks with two hidden layers are used. There are 100 hidden units in each layer, with ReLU activations. Adam (Kingma & Bal 2015) is the optimiser used in the 6 experiments with 17 0.001, 31 = 0.9 and 32 = 0.999. Further experimental details are given in Section C in the Appendix. Results of the accumulated classification accuracy, averaged over tasks, on a test set are displayed in Figure Ta After 10 tasks, CLAW achieves significantly (check the Appendix) higher classification results than all the competitors.
Whennvver there is no validation process performed to indicate the hyperparameter values of competitors or characteristics of neural network architectures, this is done for the sake of comparing common ground with the best settings, as specified in the respective papers.

---

Published as a conference paper at ICLR 2020
Split MNIST In this MNIST based experiment, five binary classification tasks are processed in the following sequence: 0/1, 2/3, 4/5, 6/7, and 8/9 (Zenke et al. 2017). The architecture used consists of fully connected multi-head networks with two hidden layers, each consisting of 256 hidden units with ReI.U activations. As can be seen in Figure 1b
CLAW achieves the highest classification accuracy.
Split notMNIST It contains 400,000 training images, and the classes are 10 characters, from A to J. Each image consists of one character, and there are different font styles. The five binary classification tasks are: A/F, B/G, C/H, D/I, and E/J. The networks used here contain four hidden layers, each containing 150 hidden units with ReLU activations. CLAW achieves a clear improvement in classification accuracy over competitors (Figure Ic].
Split Fashion-MNIST Fashion-MNIST is dataset whose size is the same as MNIST but it is based on different (and more challenging) 10 classes. The five binary classification tasks here are: T-shir//rrusse, Pullover/Dress, Coat/Sandals, Shirt/Sneaker, and Bag/Ankle boots. The architecture used is the same as in Split notMNIST. In most of the continual learning tasks (including the more significant, later ones) CLAW achieves a clear classification improvement (Figure [Td].
Omniglot This is sequential learning task of handwritten characters of 50 alphabets (a total of over 1,600 characters with 20 examples each) belonging to the Omniglot dataset (Lake et al. (011). We follow the same way via which this task has been used in continual learning before (Schwarz et al.) 2018: Titsias et al., 2019;; handwritten characters from each alphabet constitute separate task. We thus have 50 tasks, which also allows to evaluate the scalability of the frameworks in comparison. The model used is CNN. To deal with the convolutions in CLAW, we used the idea proposed and referred to as the local reparameterisation trick by Kingma et al.] (2014) 2015). where a single global parameter employed per neuron activation in the variational distribution, rather than employing parameters for every constituent weight elemen Further details about the CNN used are given in The automatically adaptable CLAW achieves better classification accuracy (Figure Ic
Section C
CIFAR-100 This dataset consists of 60,000 colour images of size 32 x 32. It contains 100 classes, with 600 images per class. We use split version CIFAR-100. Similar to Lopez-Paz & Raazato (2017), we perform a 20-task experiment with disjoint subset of five classes per task. CLAW achieves significantly higher classification accuracy (Figure TT) -also higher than the previous state of the art on CIFAR-100 by Kemker& & Kanan (2018) Details of the used CNN are in Section
conclusion that can be taken from Figure [a-f) that CLAN consistently achieves state-of-the-at -frtt-- results (in all the 6 experiments). It can also be seen that CLAW scales well. For instance, the difference between CLAW and the best competitor is more significant with Split notMNIST than is with the first two experiments, which are based on the smaller and less challenging MNIST. Also, CLAW achieves good results with Omniglot and CIFAR-100.
4.2 CATASTROPHIC FORGETTING
To assess catastrophic forgetting, we show how the accuracy the initial task varies over the course of the training procedure on the remaining tasks (Schwarz et al. 2018). Since Omniglot (and CIFAR- 100) contain a larger number of tasks: 50 (20) tasks, i.e. 49 (19) remaining tasks after the initial task, this setting is more relevant for Omniglot and CIFAR-100. We nonetheless display the results for Split MNIST, Split notMNIST, Split Fashion-MNEE -NISTT Omniglot and CIFAR- 1000. As can be seen in Figure 2 CLAW (at times jointly) achieves state-of-the fi-thhe-rt art performance retention degrees. Among the competitors, P&C and LTG also achieve high performance retention degrees.
An empirical conclusion that can be made out of this and the previous experiment, is that CLAW achieves better overall continual learning results, partially thanks to the way addresses catastrophic forgetting. The idea of adapting the architecture by adapting the contributions of neurons of each layer also seems to be working well with datasets like Omniglot and CIFAR-100, giving directions for imminent future work where CLAW can be extended for other application areas based on CNNs.
4.3 POSITIVE FORWARD TRANSFER
The purpose of this experiment is to assess the impact of learning previous tasks on the current task. In other words, we want evaluate whether an algorithm avoids negative transfer, by evaluating the
For more details, see Section 2.3 in (Kingma al. 2015

---

Published as a conference paper at ICLR 2020
Permuled MNIST
100
Split MNIST
183
Split notMNIST
Classificalion accuraty (%)
84
Closiitcaatoon
Tasks
Tasks
Tasks
(a) Permuted MNIST.
(b) Split MNIST.
(c) Split notMNIST.
Split Fashion-MNIST
Omniglot
CIFAR-100
Clacoificatoo accurary (80
Clasifiiatioo accuracy (%6)
Classification accuracy (50 S
14
Tasks
Tasks
Tasks
(d) Split Fashion-MNIST.
Omniglot.
(f) CIFAR-100.
Figure 1: Average test classification accuracy vs. the number of observed tasks in 6 experiments. CLAW achieves significantly higher classification results than the competing continual learning frameworks. Statistical significance values are presented in Section Ein in the Appendix. The value of for EWC is 10,000 (c), and 100 in the other experiments. Best viewed in colour.
relative performance achieved on a unique task after learning a varying number of previous tasks (Schwarz et al. 2018). From Figure we can see that CLAW achieves state-of-the results in 4 out of the experiments (at par in the fifth) in terms of avoiding negative transfer.
Split MNIST
notMNIST
Split Fashion-MNIST
Inbtaccakk perfeemancca (50
Intal back performmnn
Tasks
Tastos
Tackss
(a) Split MNIST.
(b) Split notMNIST.
(c) Split Fashion-MNIST.
Omniglot
CIFAR-100
Intaakaakpee performance (34)
Iniia dask perfoor mance 196)
PACE
Tasica
16
Twske
(d) Omniglot.
(c) CIFAR-100.
Figure 2: Evaluating catastrophic forgetting by measuring performance retention. Classification accuracy of the initial task is monitored along with the progression of tasks. Results are displayed for five datasets. CLAW is the least forgetful algorithm since performance levels achieved on the initial task do not degrade as much as in the other methods after facing new tasks. The legend and À values for EWC are the same as in Figure |
Best viewed in colour.
8

---

Published as a conference paper at ICLR 2020
Split MNIIT
Split notMNIST
Spllt Fashion-MNIST
(a) Split MNIST.
(b) Split notMNIST.
(c) Split Pashion-MNIST
Omniglot
CIFAR-100
04.
14
FACE
Taske
(d) Omniglot.
(e) CIFAR-100.
Figure 3: Evaluating Forward transfer, what extent continual learning framework avoid negative transfer. The impact of learning previous tasks on a specific task (the last task) is inspected and used proxy for evaluating forward transfer. This is performed by evaluating the relative performance achieved on a unique task after learning a varying number of previous tasks. This means that the value at x-axis one task (only itself), the value at 2 refers to the learning accuracy of the last task after having learnt two tasks (an additional previous task), etc. Overall, CLAW achieves state-of-the-an results out of the 5 experiments (at par in the fifth) in terms of avoiding negative transfer. Best viewed in colour.
refers the learning accuracy of the last task after having learnt solely
RELATED WORK
We briefly discuss three related approaches continual learning: (a) regularisation-bast (b) architecture-basee and (c) memory-based. We provide more details of related work in Section A in the Appendix. (a) complementary approach to CLAW the regularisation-base balance adaptability with catastrophic forgetting: a level of stability is kept via protecting parameters
approach to
that greatly influence the prediction against radical changes, while allowing the rest of the parameters to change without restriction (Li & Hoiem, 2016 Lee et al. 2017; Zenke et al.) 2017 Chaudhry
iet al. 2018 Kim et al. 2018 Nguyen et al. 2018: Srivastava et al. 2013) Schwarz et al. 2018 Vuorio al. 2018; Aljundi et al. (019c). The elastic weight consolidation (EWC) algorithm by Kirkpatrick ef al. (2017) is seminal example, where a quadratic penalty is imposed on the difference between parameter values of the old and new tasks. One limitation is the high level of hand tuning required. (b) The architecture- based approach aims to deal with stability and adaptation issues by
fixed division of the architecture into global and local parts (Rusu et al. 2016b
Fernando et al. 2017
Shin et al. 2017 Kaplanis et al., 2018 Xu & Zhu 2018 Yoon et al.
2018 Li et al.) (019b). (c)
The memory- based approach relies on episodic memory to store data (or pseudo-datal from previous
tasks (Ratcliff, 1990; Robins 1993; 1995 Thrun 1996 Schmidhuber 2013 Hattori [0044] Mocanu
al. 2016 Rebufii et al., 2017 Kamra et al.) 2017 Shin al., 2011 Rolnick et al. 2018 van de
Ven & Tolias] 2018 Wu et al. 2018 Titsias et al. 2019). Limitations include overheads for tasks such as data storage, replay, and optimisation to select (or generate) the points. CLAW can as well be seen combination of regularisation-brt based approach (the variational inference mechanism) and a modelling approach which automates the architecture building process in a data-driven manner, avoiding the overhead resulting from either storing or generating data points from previous tasks. CLAW is also orthogonal to (and simple to combine with, needed) memory-batt based methods.
6 CONCLUSION
We introduced a continual learning framework which learns how to adapt its architecture from the tasks and data at hand, based variational inference. Rather than rigidly dividing the architecture

---

Published as a conference paper at ICLR 2020
into shared and task-specific parts, our approach adapts the contributions of each neuron. We achieve that without having to expand the architecture with new layers or new neurons. Results of six different experiments
five datasets demonstrate the strong empirical performance of the introduced framework, in terms of the average overall continual learning accuracy and forward transfer, and also in terms of effectively alleviating catastrophic forgetting.
ACKNOWLEDGMENTS
HZ acknowledges support from the DARPA XAI project, contract(4AA8750112205555 and an Nvidia GPU grant. RT acknowledges support by Google, Amazon, Improbable and EPSRC grants EP/M0269571 and EP/L000776/1.
REFERENCES
A. Achille, T. Eccles, L. Matthey, C. Burgess, N. Watters, A. Lerchner, and Higgins. Life-long disentangled representation learning with cross-domain latent homologies. Advances in neural information processing systems (NIPS), 2018.
Tameem Adel, Han Zhao, and Alexander Wong. Unsupervised domain adaptation with relaxed covariate shift assumption. In Thisty-First AAAI Conference on Artificial Intelligence, 2017.
H. Ahn, D. Lee, S. Cha, and T. Moon. Uncertainty-bast continual learning with adaptive regulariza- tion. arXiv preprint arXiv: 1905. /614, 2019.
R. Aljundi, L. Caccia, E. Belilovsky, M. Caccia, M. Lin, L. Charlin, and T. Tuytelaars. Online continual learning with maximally interfered retrieval. Advances in neural information processing systems (NeurIPS), 2019a.
R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems (NeurIPS), 2019b.
R. Aljundi, M. Rohrbach, and T. Tuytelaars. Selfless sequential learning. International Conference on Learning Representations (ICLR), 2019c.
B. Bakker and Heskes. Task clustering and gating for Bayesian multitask learning. Journal of Machine Learning Research (JMLR), 4:83-99, 99, 2003.
J. Bernardo and A. Smith. Bayesian theory. John Wiley & Sons, 2000.
Y. Butalov. The notMNIST dataset. http://ywanneevvb.ommmmnnohhhhhs 2011.
G. Carpenter and S. Grossberg. A massively parallel architecture for self-organizing neural pattern recognition machine. Computer vision, graphics, and image processing. 37:54-115, 1987.
R. Caruana. Multi-task learning. Machine Learning. 1997.
A. Chaudhry, Dokania, T. Ajanthan, and Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. arXiv preprint arXiv: 2018.
E. Choi, K. Lee, and K. Choi. butoencoder-based incremental class learning without retraining on old data, arXiv preprint arXiv:/907.07888 2019.
N. Rodrigu,,, V. Lomonaco, D. Filliat, and D. Maltoni. Don't forget, there more than forgetting: new metrics for Continual Learning. NIPS Continual Learning Workshop, 2018.
X. Du, G. Charan, F. Liu, and Y. Cao. Single-net continual learning with progressive segmented training (PST), arXiv preprint 1905v: 1550, 2019.
Ebrahimi, M. Elhoseiny, T. Darrell, and M. Rohrbach. Uncertainty-guider continual learning with Bayesian neural networks. arXiv preprint arXiv: 1906. 29t.::299000225.. 2019.
C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. Rusu, A. Pritzel, and D. Wierstra. PathNet: Evolution channels gradient descent super neural networks. arXiv preprint arXiv: 2017.
10

---

Published as a conference paper at ICLR 2020
C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning (ICML), 2017.
R. French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3: 10:1228-3: 1999.
S. Ghosh, J. Yao, and F. Doshi-Velez. Structured variational learning of Bayesian neural networks with horseshoe priors. International Conference on Machine Learning (ICML), 2018.
I. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00164 2016.
I. Goodfellow, D. Fardee, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. Interna- tional Conference on Machine Learning (ICML), 2013.
I. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio. An empirical investigation of catastrophic forgetting in gradient- based neural networks. International Conference on Learning Representations (ICLR), 2014a.
I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems (NIPS), pp. 2672-2680, 2014b.
M. Hattori. A biologically inspired dual-network memory model for reduction of catastrophic forgetting. Neurocomputing, 268, 2014.
X. He, J. Sygnowski, A. Galashov, A. Rusu, Y. Whye Teh, and R. Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv: 1906.0520,, 2019.
T. Heskes. Empirical Bayes for learning to learn. 2000.
W. Hu, Z. Lin, B. Liu, C. Tao, Z. Tao, Ma, D. Zhao, and R. Yan. Overcoming catastrophic forgetting via model adaptation. International Conference on Learning Representations (ICLR), 2019.
D. Isele and A. Cosgun. Selective experience replay for lifelong learning. arXiv preprint arXiv: 10269, 2018.
K. Javed and M. White. Meta-learning representations for continual learning. Advances in neural information processing systems (NeurIPS), 2019.
H. Jung. Ju, M. Jung, and J. Kim. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122. 2016.
N. Kamra, U. Gupta, and Y. Liu. Deep generative dual memory network for continual learning. arXiv preprint arXiv: 1730. 2017.
C. Kaplanis, M. Shanahan, and C. Clopath. Continual reinforcement learning with complex synapses. International Conference on Machine Learning (ICML), 2018.
R. Kemker and C. Kanan. FearNet: Brain-inspired model for incremental learnings. International Conference on Learning Representations (ICLR), 2018.
R. Kemker, M. McClure, A. Abitino, T. Hayes, and C. Kanan. Measuring catastrophic forgetting in neural networks. AAAI Conference on Artificial Intelligence. 32, 2018.
D. Kim, J. Bae, Y. Jo, and J. Choi. Incremental learning with maximum entropy regularization: Rethinking forgetting and intransigence. arXiv preprint arXiv11002.08829, 2019.
H. Kim, S. Kim, and J. Lee. Keep and learn: Continual learning by constraining the latent space for knowledge preservation in neural networks. MICCAI, 2018.
D. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR), 2015.
11

---

Published as a conference paper at ICLR 2020
D. Kingma, D. Rezende, S. Mohamed, and M. Welling. Semi- Sumi-super learning with deep generative models. Advances in neural information processing systems (NIPS), 28:3581-3558 2014.
D. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterization trick. Advances in neural information processing systems (NIPS), pp. 2575-2582 2015.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedinggs of the National Academy of Sciences (PNAS), 2017.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
B. Lake, R. Salakhutdinov. J. Gross, and J. Tenenbaum. One shot learning of simple visual concepts. Proceedings of the Cognitive Science Society, 33, 2011.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, 86(1) 36(11):2278-3324, 1998.
S. Lee, J. Kim, J. Jun, J. Ha, and B. Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems (NIPS), 2017.
H. Li, S. Enshaeifar, F. Ganz, and P. Barnaghi. Continual learning in deep neural network by using a Kalman optimiser. ICML Workshop, 2019a.
X. Li, Y. Zhou, T. Wu, R. Socher, and C. Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting, International Conference on Machine Learning (ICML), 2019b.
Z. Li and D. Hoiem. Learning without forgetting. European Conference on Computer Vision (ECCV), 2016.
M. Lin, J. Fu, and Y. Bengio. Conditional computation for continual learning. NIPS Continual Learning Workshop. 2018.
D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning, Advances in neural information processing systems (NIPS), 2017.
C. Louizos, K. Ullrich, and M. Welling. Bayesian compression for deep learning. Advances in neural information processing systems (NIPS), pp. 3288-3298, 2017.
M. McCloskey and N. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 1989.
D. Mocanu, M. Vega, E. Eaton, P. Stone, and A. Liotta. Online contrastive divergence with generative replay: Experience replay without storing data. arXiv preprint arXhv:1610.05553 2016.
D. Molchanov, A. Ashukha, and D. Vetrov. Variational dropout sparsifies deep neural networks. International Conference Machine Learning (ICML), pp. 2498-25011 2017.
C. Nguyen, Y. Li, T. Bui, and R. Turner. Variational continual learning. International Conference on Learning Representations (ICLR), 2018.
O. Ostapenko, M. Puscas, T. Klein, P. Jahnichen, and M. Nabi. Learning to remember: synaptic plasticity driven framework for continual learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
L. Pape, F. Gomez, M. Ring, and J. Schmidhuber. Modular deep belief networks that do not forget. IEEE International Joint Conference on Neural Networks (IJCNN), 2011.
G. Parisi, R. Kemker, J. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
12

---

Published as a conference paper at ICLR 2020
D. Park, S. Hong, B. Han, and K. Lee. Continual learning by asymmetric loss approximation with single-side overestimation. arXiv preprint 1908v 908.02984, 2019.
B. Pfulb and A. Gepperth. A comprehensive, application orientte study of catastrophic forgetting in DNNs. International Conference on Learning Representations (ICLR), 2019.
J. Rajasegaran, M. Hayat, S. Khan, Khan, and L. Shao. Random path selection for incremental learning. Advances in neural information processing systems (NeurlPSS), 2019.
R. Rateliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 1990.
S. Rehuffi, A. Kolesnikov, G. Sperl, and C. Lampert. iCaRL: Incremental classifier and representation learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
M. Riemer, I. Cases, Ajemian, M. Liu, LRish, Y. Tu, and Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. International Conference Learning Representations (ICLR), 2019.
M. Ring. Continual learning in reinforcement environments. PhD thesis, University of Texas, Austin, 1995.
M. Ring. CHILD: A first step towards continual learning. Machine Learning. 1997.
A. Robins. Catastrophic forgetting in neural networks: The role of rehearsal mechanisms. IEEE Artificial Neural Networks and Expert Systems, pp. 65-68, 1993.
A. Robins. Catastrophic forgetting, rehearsal and pseudorchearsal. Connection Science, 7:123-146, 1995.
D. Rolnick, Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne. Experience replay for continual learning. arXiv preprint 181. 2018.
A. Rusu, N. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.0467, 2016a.
A. Rusu, M. Vecerik, T. Rothoerl, N. Heess, R. Pascanu, and R. Hadsell. Sim-to-Real robot learning from pixels with progressive nets. arXiv preprint arXiv: 1610.0428, 2016b.
J. Schlimmer and D. Fisher. A case study of incremental concept induction. The National Conference on Artificial Intelligence, 1986.
Schmidhuber. Powerplay:
Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology. 4, 2013.
Schmidhuber. One big for everything. arXiv preprint arXiv: 2018.
Schwarz, J. Luketina, W. Czarnecki, A. R. Hadsell. Progress & compress: A scalable framework for continual learning. International Conference on Machine Learning (ICML), 2018.
Y. Whye Teh, R. Pascanu, and
J. Serra, D. Suris, M. Miron, and A. Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. International Conference on Machine Learning (ICML), 2018.
H. Shin, J. Lee, J. Kim, and J. Kim. Continual lcarning with deep generative replay. Advances in neural information processing systems (NIS)), 2017.
N. Srivastava, G. Hinton. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 15: 1929-1958, 2014.
R. Srivastava, J. Masci, Kazerounian, F. Gomez, and Schmidhuber. Compete to compute. Advances in neural information processing systems (NIPS), 2013.
13

---

Published as a conference paper at ICLR 2020
A. Stickland and I. Murray. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. International Conference on Machine Learning (ICML), 2019.
R. Sutton and S. Whitehead, Online learning with random representations. International Conference on Machine Learning (ICML), 1993.
Swietojanski and S. Renals. Learning hidden unit contributions for unsupervised speaker adaptation of neural network acoustic models. IEEE Spoken Language Technology Workshop (SLT), 2014.
D. Teng and S. Dasgupta. Continual learning via online leverage score sampling, arXiv preprint arXiv:1908.00355. 2019.
S. Thrun. Explanation-bann based neural network learning: A lifelong learning approach. Springer Science & Business Media, 357, 1996.
M. Titsias, Schwarz, A. Matthews, R. Pascanu, and Y. Whye Teh. Functional regularisation for continual learning using Gaussian processes. arXiv preprint arXiv: 2019.
G. van de Ven and A. Tolias. Generative replay with feedback connections as general strategy for continual learning, arXiv preprint arXiv: 10635, 2018
R. Vuorio, D. Cho, D. Kim, and J. Kim. Meta continual learning. arXiv preprint arXiv: 2018.
C. Wu, L. Herranz, X. Liu. Y. Wang. J. van de Weijer, and B. Raducanu. Memory replay GANs: Learning to generate new categories without forgetting. Advances in neural information processing systems (NIPS), 2018.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.077 2017.
J. Xu and Z. Zhu. Reinforced continual learning. Advances in neural information processing systems (NIPS), 2018.
J. Xu, J. Ma, and Z. Zhu. Bayesian Optimized Continual Learning with Attention Mechanism. arXiv preprint arviv: 1905 0w0..9950000 0009000 2019.
Yoon, F. Yang. J. Lee, and S. Hwang. Lifelong learning with dynamically expandable networks. International Conference Learning Representations (ICLR), 2018.
J. Yoon, S. Kim, E. Yang, and S. Hwang. ORACLE: Order robust adaptive continual learning. arXiv preprint arXiv:1902.000 99220943,, 2019.
F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. International Conference on Machine Learning (ICML), 2017.
C. Zeno, I. Golan, E. Hoffer, and D. Soudry. Task agnostic continual learning using online variational Bayes. NIPS Bayesian Deep Learning Workshop, 2018.
Han Zhao, Otilia Stretcu, Alex Smola, and Geoff Gordon. Efficient multitask feature and relationship learning. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence. AUAI Press, 2019a.
Han Zhao, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, and Geoffrey Gordon. Learning neural networks with adaptive regularization. In Advances in Neural Information Processing Systems, 2019b.
14

---

Published as a conference paper at ICLR 2020
APPENDIX
We begin by briefly summarising the contents of the Appendix below:
Related works are described in Section A followed by a brief discussion on the potential applicability of CLAW to another continual learning (CL) framework in Section
In Section L we provide the statistical significance and standard error of the average classi- fication accuracy results obtained after completing the last two tasks from each experiment.
Further experimental details are given in Section C
In Section D and Figures 4a- Ae] we display the results of performed ablations which manifest the relevance of each adaptation parameter.
A MORE RELATED WORK
A complementary approach CLAW, which could be combined with it, is the regularisation-basee approach to balance adaptability with catastrophic forgetting: a level of stability is kept via protecting parameters that greatly influence the prediction against radical changes, while allowing the rest of
the parameters to change without restriction (Li & Hoiem, 2016 Vuorio et al. 2018). In (Zenke) ct al. 2017), } the regulariser is based on synapses where an importance measure is locally computed at each synapse during training, based on their respective contributions to the change in the global loss. During task change, the less important synapses are given the freedom to change whereas catastrophic forgetting is avoided by preventing the important synapses from changing (Zenke et al., 2017). The elastic weight consolidation (EWC) algorithm, introduced by Kirkpatrick et al. (2017) 7), is seminal example of this approach where a quadratic penalty is imposed on the difference between parameter values of the old and new tasks. One limitation of EWC, which is rather alleviated by using minibatch or stochastic estimates, appears when the output space is not low-dimensional, since the diagonal of the Fisher information matrix over parameters of the old task must be computed, which requires summation over all possible output labels (Kirkpatrick et al. 2017 7 Zenke et al., 2017, [Schwarz et al., 2018). In addition, the regularisation term involves a sum over all previous tasks with a term from each and a hand-tuned hyperparameter that alters the weight given to it. The accumulation of this leads to lot of hand-tuning. The work in (Chaudhry et al. (018) is based on penalising confident fitting to the uncertain knowledge by a maximum entropy regulariser.
Another seminal algorithm based on regularisation, which can be applied to any model, is variational continual learning (VCL) (Nguyen et al.) 2018) which formulates CL as sequential approximate (variational) inference problem. However, VCL. has only been applied to simple architectures, not involving any automatic model building or adaptation. The framework in (Lee et al., 2017) incrementally matches the moments of the posterior of Bayesian neural network that has been trained on the first and then the second task, and so on. Other algorithms pursue regularisation approaches based in (Aljundi et al. 2019c) encourages sparsity on the neuron activations to alleviate catastrophic forgetting. The l2 distance between the top hidden activations of the old and new tasks is used for regularisation in (Jung et al.] 2016). This approach has achieved good results, but is computationally expensive due the necessity of computing at least a forward pass for every new data point through the network representing the old task (Zenke et al. (2017). Other regularisation-base continual learning algorithms include (Ebrahimi al. 2019 Park al. 2019).
sparsity (Srivastava et al. 2013 Kim et al. 2018). For example, the work
Another approach is the architecture-basee one where the principal aim is to administer both the stability and adaptation issues via dividing the architecture into reusable parts that are less prone to changes, and other parts especially devoted to individual tasks (Rusu et al.] 2016b; Fernando jet al.) 2017: Yoon et al., 2018 Du et al., 2019 He et al.) 2019 Li et al.) 2019a] Xu et al. 2019). To learn a new task in the work by Rusu et al. 2016a). the whole network from the previous task is first copied then augmented with a new part of the architecture. Although this is effective in eradicating catastrophic forgetting, there is clear scalability issue since the architecture growth can be prohibitively high, especially with an increasing number of tasks. The work introduced in (Li let (001bb) bases its continual learning on neural architecture search, whereas the representation in (Javed & White 2019) is optimised such that online updates minimize the error on all samples while limiting forgetting. The framework proposed by Xu & 7hu (2018) interestingly aims at solving
15

---

Published as a conference paper at ICLR 2020
this neural architecture structure learning problem, while balancing the tradeoff between adaptation and stability, via designed reinforcement learning (RL) strategies. When facing a new task, the optimal number of neurons and filters to add to each layer is cast combinatorial optimisation problem solved by an RL strategy whose reward signal is a function of validation accuracy and network complexity. Another RI. based framework is the one presented by Kaplanis et al.] (2018) where catastrophic forgetting is mitigated at multiple time scales via RL agents with a synaptic model inspired by neuroscience. Bottom layers (those near the input) are generally shared among the different tasks, while layers near the output are saec-spe Since the model structure is usually divided a priori and no automatic architecture learning nor adaptation takes place, alteration on the shared layers can still cause performance loss on earlier tasks due to forgetting (Shin et al. 2017).
clipped version of maxout networks (Goodfellow et al., (203)) is developed in (Lin et al. 2018) where parameters are partially shared among examples. The method in (Ostapenko et al. 2019) is
based a dynamic network expansion accomplished by a generative adversarial network.
The memory-based approach, which is the third influential approach to address the adaptation- catastrophic forgetting tradeoff, relies on episodic memory to store data (or pseudodata) from previous tasks (Ratcliff 1990; Robins 1993 1995, Hattori 2014 Rolnick et al. 2018 Teng & Dasgupta 2019). A major limitation of the memory-based approach is that data from previous tasks may not be available in all real-world problems (Shin al. 2017: Choi al.] 2019). Another limitation is the overhead resulting from the memory requirements, e.g. storage, replay, etc. In addition, the optimisation required to select the best observation to replay for future tasks is a source of further overhead (Titsias et al. 2019). In addition to the explicit replay form, some works have been based on generative replay Trun 1996; Schmidhuber 2013 Mocanu et al., 2016 Rebutli et al. 2017 vaa
Kamra et al. 2017 7 Shin et al. 2017]] van de Ven & Tolas 2018 Wu et al. 2018). Notably, Shin et al. (2017) train a deep generative model based on generative adversarial networks (GANs, [Goodfellow eet al. 2014b; Goodfellow 2016) to mimic past data. This mitigates the aforementioned problem, albeit at the added cost of the training of the generative model (Schwarz et al.) 2018) and sharing its parameters. Alleviating catastrophic forgetting via replay mechanisms has also been adopted in reinforcement learning, c.g. (Isele& & Cosgun| 2018; Rolnick et al.) 2018). A similar approach was introduced by Lopez-Paz & Ranzato (2017) where gradients of the previous task (rather than data examples) are stored so that a trust region consisting of gradients of all previous tasks can be formed to reduce forgetting. Other algorithms based replay mechanisms include (Aljundi et al. 2019a b).
Equivalent tradeoffs to the one between adaptation and stability can be found in the literature since the work in (Carpenter & Grossberg 1987), in which a balance was needed resolve the stability- plasticity dilemma, where the latter refers to the ability to rapidly adapt to new tasks. The works introduced in (Chaudhry et al. 2018 Kim et al. 2019) shed light on the tradeoff between adaptation and stability, where they explore measures of intransigence and forgetting. The former refers to the
inability to adapt to new tasks and data, whereas an increase in the latter clearly signifies an instability problem. Other recent works tackling the same tradeoff include (Riemer et al.) 2019) where the transfer-interfernnce (interference is catastrophic forgetting) tradeoff is optimised for the sake of maximising transfer and minimising interference by an algorithm based on experience replay and meta-learning. Other recent algorithms include the ORACLE algorithm by Yoon et al. (2099), which addresses the sensitivity of continual learner to the order of tasks it encounters by establishing an order robust learner that represents the parameters of each task as a sum of task-shared and task-specific parameters. The algorithm in (Titsias et al., 2019) achieves functional regularisation by performing approximate inference over the function (instead of parameter) space. They use a Gaussian process obtained by assuming the weights of the last neural network layer be Gaussian distributed. Our model is also related to the multi-task learning approach (Caruana) 1997 Heskes
2000
Bakker Heskes 2003 Adel et al. 2017 Zhao et al. 2019a Stickland & Murray| 2019 Zhao
et al. 2019b).
A.1 APPLICABILITY OF CLAW TO OTHER CL FRAMEWORKS
As mentioned in the main document, ideas of the proposed CLAW can be applied to continual learning frameworks other than VCL. The latter is more relevant for the inference part of CLAW since both are based on variational inference. As per the modeling ideas, e.g. the binary adaptation parameter depicting whether not to adapt, and the maximum allowed adaptation, these can be integrated within other continual learning frameworks. For example, the algorithm in Xu & Zhu (2018) utilises reinforcement learning adaptively expand the network. The optimal number of nodes and filters to
16

---

Published as a conference paper at ICLR 2020
be added is cast as combinatorial optimisation problem. In CLAW, we do not expand the network. As such, an extension of the work in (Xu & Zhu 2018) can be inspired by CLAW where not only the number of nodes and filters to be added is decided for each task, but also a soft and more general version where an adaptation based on the same network size is performed such that the network expansion needed in (Xu & Zhu 2018) can be further moderated.
B STATISTICAL SIGNIFICANCE AND STANDARD ERROR
In this section, provide information about the statistical significance and standard error of CLAW and the competing continual learning frameworks. In T] we list the average accuracy values (Figure in the main document) obtained after completing the last tasks from each of the six experiments. A bold entry in Table denotes that the classification accuracy of an algorithm is significantly higher than its competitors. Significance results are identified using a paired t-test with = 0.05. Each average accuracy value is followed by the corresponding standard error. Average classification accuracy resulting from CLAW is significantly higher than its competitors on the experiments.
Table 1: Average test classification accuracy of the last two tasks in each of the six experiments: Permuted MNIST, Split MNIST, Split notMNIST, Split Fashion-MNIST. Omniglot and CIFAR- 100, followed by the corresponding standard error. A bold entry denotes that the classification accuracy of algorithm is significantly higher than its competitors. Significance results are identified using a paired t-test with 0.05. Average classification accuracy resulting from CLAR is significantly higher than its competitors on the 6 experiments.
Classification Accuracy
CLAW
VCL
VCL Coreset
EWC
Permuted MNIST (task 9)
99.2 + 0.2 %
93.5 + 0.3 %
95.5 = 0.3 %
92.1 + 0.4 %
Permuted MNIST (task 10)
99.2 + 0.1 %
92.1 +:1::00 0.3 %
95 0.5 %
90.2 + 0.4
Split MNIST (task 4)
9 %
98.6 0.3 %
98.7 0.2 %
94.9 0.4 %
Split MNIST (task 5)
99.1 %
9..0 %
08.4 %
94.2 %
Split notMNIST (task
9.3 %
95.8 %
96.9 %
92.9 %
Split notMNIST (task 5)
98.4 %
92.1 1.1:00 + %
96.0 %
89.4 92.3 %
Split Fashion-MNISS (task 4)
9.2 %
90.0 %
90.7 0.2 %
Split Fashion-MNIST (task 5)
9.2 %
88.0 %
88.5 + 0.4 %
87.6 %
Omniglot (task 49)
8.25 %
81.1 %
81.8 + 0.3 %
78.2
Omniglot (task 50)
84.6 %
80.7 + +.3 %
81.1 + 0.4 %
77.3 1.33000 0.3 %
CIFAR-100 (task 19)
95.6 = 0.3 %
78.7 = 0.4 %
80.8 = 0.3 %
63.1 3.1
CIFAR- 100 (task 20)
95.6 + 0.3 %
77.2 + 0.4 %
79.9 = 0.4 %
62.4 - 0.4 %
Classification Accuracy
P&C
RCL
FRCL
LTG
Permuted MNIST (task 9)
94.4 0.3 %
96.4 0.5 %
98.4 0.4 %
98.7 0.3
Permuted MNIST (task 10)
94.1 %
96.3 %
98.4 0.5 %
98.7 % 98.7
Split MNIST (task 4)
97.3+ 0.5 %
9.78 %
98.2 %
Split MNIST (task
96.4+ 0.4 %
9..5+ %
%
98.3+ %
Split notMNIST (task 4)
9. %
97.7 %
96.1 %
97.8 %
Split notMNIST (task
9.5 %
9.. %
9..2 %
9..4 %
Split Fashion-MNV -NIIT (task 4)
9.3 %
91.1 %
90.4 %
92.5 %
Split Cisshion-MNSS' (task
9.2 %
8..7 %
87.7 %
91.1 %
Omniglot (task 49)
8.2 %
80.1 %
79.9 0.9 %
83.6 %
Omniglot (task 50)
82.7 %
80.2 0.44%
79.8 - 0.5 %
83.5
CIFAR-100 (task 19)
68.3 + 0.6 %
63.7 4.6
77.4 + 0.7 %
2.24049
CIFAR- 100 (task 20)
65.5 + 0.6 %
60.4
76.8 - 0.6 %
8.5
C OTHER EXPERIMENTAL DETAILS
Here additional details about the datasets in use:
17

---

Published as a conference paper at ICLR 2020
The MNIST dataset is used in both the Permuted MNIST and Split MNIST experiments. The MNIST (Mixed National Institute of Standards and Technology) dataset (LeCun et al. 1998) is a handwritten digit dataset. Each MNIST image consists of 28 28 pixels, which is also the pixel size of the notMNIST and Fashion-MNN instances and test set of 10,000 instances.
datasets. The MNIST dataset contains a training set of 60,000
As mentioned the main document, each experiment is repeated ten times. Data is randomly split into three partitions, training, validation and test. A portion of 60% of the data reserved for training, 20% for validation and 20% for testing, Statistics reported are the averages of these ten repetitions.
Number of epochs required per task reach a saturation level for CLAW (and the bulk of the methods in comparison) was 10 epochs for all experiments except for Omniglot and CIFAR-100 (15 epochs). Used values of W1 and W2 are 0.05 and 0.02, respectively.
For Omniglot, we used a network similar to the one used in (Schwarz et al. 2018), which consists of 4 blocks of 3 convolutions with 64 filters, followed by ReLU and a 2 max maxxpooliigg The same CNN is used for CIFAR-100. CLAW achieves clearly higher classification accuracy on both Omniglot and CIFAR-100 (Figures Te and
D ABLATIONS
The plots displayed in this section empirically demonstrate how important the main adaptation parameters are in achieving the classification performance levels reached by CLAW. In each of the Figures 4a the classification performance of CLAW is compared to the following three cases: 1) when the parameter controlling the maximum degree of adaptation is not learnt in multi-task fashion, Le. when the respective general value Si is used instead Si,,tt 2) when adaptation always happens, Le. the binary variable denoting the adaptation decision is always activated. 3) when adaptation never takes place. The differences in classification accuracy between CLAW and each of the other three plots in Figures 4a- empiricllly demonstrate the relevance of each adaptation parameter.
E RUN-TIME
In Table report the wall-clock run time (in seconds) after finishing training in each of the six experiments: Permuted MNIST, Split MNIST, Split notMNIST, Split Fashion-MNIST, Omniglot and CIFAR-100. VCL and CLAX converge (i.e. reach the accuracy levels reported earlier) more quickly than the other methods. CLAW was the fastest in 3 out of the experiments, whereas VCL was the fastest in the other 3, but their training run-time values have always been close to each other. As reported in the earlier sections, there is significant difference in classification accuracy in favour of CLAW. This has been achieved within reasonably acceptable run-time levels thanks the proposed design where the whole data-driven adaptation procedure is kept as part of an amortised variational inference algorithm. RCL the slowest since it is based on reinforcement learning, where large number of trials typically required. This has also been acknowledged and reported therein (Xu & Zhu 2018).
Table 2: Wall-clock run time (in seconds) after finishing training in each of the six experiments: Permuted MNIST, Split MNIST, Split notMNIST, Split Fashion-MNISS Omniglot and CIFAR-100. As mentioned earlier, the statistics reported are averages of ten repetitions.
Training Time (in seconds)
CLAW VCL VCL + Coreset EWC P&C
RCL
FRCL
LTG
Permuted MNIST (after 10 tasks)
667
682
724
1117
1355
32575
919
705
Split MNIST (after tasks)
649
637
708
1054
1312
31110
891
648
Split NotMNIST (after tasks)
722
714
792
1210
1407
34123
898
781
Split
(after tasks)
829
818
901
1284
1498
35086
972
915
Omniglot (after 50 tasks)
1126
1241
1513
1637
1714
37247
1620
1312
CIFAR- 100 (after 20 tasks)
792
810
914
802
1322
6102
1016
896
18

---

Published as a conference paper at ICLR 2020
Permuted MNIST
Split MNIST
Classifiation accuracy (%6)
Classification accuracy (%6
28
CLAY
64
30-
48-
14
Tasks
Tasks
(a) Ablations for Permuted MNIST.
(b) Ablations for Split MNIST.
Split notMNIST
Split Fashion-MNIST
Classification accuracy (%)
scees tead
Classification accuracy
16-
Abay siut i
Ahir cenmets nam.
84-
Heaps staos (1-21
8.
66-
82
50-
64-
89
Tasks
Tasks
(c) Ablations for Split notMNIST.
(d) Ablations for Split MNIST..
Omniglot
CIFAR- 100
Classification accuracy (%)
CLY
Classification accuracy (%)
98
te
adaps
Alrvay genenas.
22
92-
60-
76-
18
Tasks
Tasks
(e) Ablations for Omniglot.
(1) Ablations for CIFAR-100.
Figure 4: Ablation studies on different datasets.
19


Reviews:
[{"review_id": "Sygqf-m0Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces CLAW, a complex but effective approach to continual learning with strong performance in the sequential task learning setting, as demonstrated on a number of standard benchmarks.\n\nI recommend acceptance because:\n- While conceptually similar to VCL, CLAW is convincingly shown to have superior performance across standard benchmarks and measures. The evaluation is thorough across the board, as far as I can tell.\n- Forward transfer is shown to be substantially better compared to other methods. Experiments with long sequences of tasks (Omniglot, CIFAR-100) are particularly telling.\n- Overall, the balance between not forgetting and still learning new tasks seems particularly favourable for the proposed method. This has been an elusive goal of continual learning research, hence the importance of accepting this work.\n\n\nHere are some good reasons why an adversarial reviewer would reject this paper:\n- The sequential task setting for continual learning has very little to add in practice, and to other branches of machine learning: it has little to say for domains where continual learning problems occur naturally, such as reinforcement learning, GAN training, multi-agent learning; all of these domains need continual learning solutions; while progress on standard benchmarks is important, we may be overfitting to these benchmarks.\n- The paper is well written but the method is rather complex and presumably non-trivial to tune. This is actually characteristic of several top competing methods on these benchmarks; getting the last bit of performance seems to require this complexity, but it also makes it that much harder to generalize such methods beyond these benchmarks. For example, exploiting well partitioned datasets into different tasks and known task labels is a good starting point, but once such information is not available all bets are off. Acceptance means encouraging work on these benchmarks; is this really what we should do?\n- It's perfectly tractable to store some small amount of old data for these problems in an 'episodic memory', so one could claim that an entire class of relevant baselines is missing, e.g. A-GEM, iCaRL, etc.\n\n\nLuckily, I am not an adversarial reviewer, but I want to see progress across a more diverse and widely relevant set of continual learning challenges.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HJgp2lKpYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a novel probabilistic continual learning approach which automatically learn an optimal adaptation for arriving tasks while maintaining the performance of the past tasks. CLAW learns element-wise weight masking per task task with respect to the several learnable parameters. \nCLAW fundamentally based on Variational Continual Learning, but it outperforms benchmarks on diverse dataset even without additional coreset. However, the ablation study and analysis on the model is weak and authors only show experimental observations. Also, the experiments are performed on old architectures. Then, it needs to show the model consistently outperform on recent deep network architectures, such as ResNet.\n\n\nI have several questions,\n\n- How about the training time / convergence rate of the CLAW compared to other methods? \n\n- If the model need to divide the sample into two halves, isn't the model vulnerable when there are only a few number of samples with high variance? This situation is quite natural on realistic problem, like Imagenet.\n\n- Why are the VCL variants with CNN not compared?\n\n- There might be used a wrong plots in Figure 1 (e). It doesn't make sense that all methods show equal accuracy on task 46.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xod8Xe9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new continual learning method. The model is based on the probabilistic model and variational inference. To show the effectiveness of the proposed model, the authors do experiments on several benchmarks.\n\nThe model lacks enough technical novelty. The motivation of this paper is a task-specific weight adaptation mechanism, which seems a simple version of [1]. In addition, the design of b^T as s/(1+e^{-a^T})-1 is not well-motivated. It is better to explain more.\n\nIn addition, if the authors compare the performance on both negative transfer (forgetting) and forward transfer (performance on the new task). I suggest the authors compare with [2], which also focuses on the performance of new tasks.\n\nMinor:\n1. The font of figures and its legends are small.\n\n[1] Li, Xilai, et al. \"Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting.\" ICML’19.\n\n[2] C Finn et al. “Online Meta-learning” ICML’19\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new variational-inference-based continual learning algorithm with strong performance.

There was some disagreement in the reviews, with perhaps the one shared concern being the complexity of the proposed method. One reviewer brought up other potentially related work, but this was convincingly rebutted by the authors. Finally, one reviewer had an issue with the simplicity with the networks in the experiments, but the authors rightly pointed out that the architectures were simply designed to match those from the baselines.

Continual learning has been an active area for quite some time and convincingly achieving SOTA in a new way is a strong contribution, and will be of interest to the community. Progress in a field is sometimes made by iteratively simplifying an initially complex solution, and this work lays in a brick in that direction. For these reasons, I recommend acceptance.




Decision: Accept (Poster)