Generalized Variational Continual Learning
Authors: Noel Loo
Siddharth Swaroop
Richard E Turner
Venue: ICLR 2021 Poster (2021)

Abstract: Continual learning deals with training models on new tasks and datasets in an online fashion. One strand of research has used probabilistic regularization for continual learning, with two of the main approaches in this vein being Online Elastic Weight Consolidation (Online EWC) and Variational Continual Learning (VCL). VCL employs variational inference, which in other settings has been improved empirically by applying likelihood-tempering. We show that applying this modification to VCL recovers Online EWC as a limiting case, allowing for interpolation between the two approaches. We term the general algorithm Generalized VCL (GVCL). In order to mitigate the observed overpruning effect of VI, we take inspiration from a common multi-task architecture, neural networks with task-specific FiLM layers, and find that this addition leads to significant performance gains, specifically for variational methods. In the small-data regime, GVCL strongly outperforms existing baselines. In larger datasets, GVCL with FiLM layers outperforms or is competitive with existing baselines in terms of accuracy, whilst also providing significantly better calibration.

Published as a conference paper at ICLR 2021
GENERALIZED VARIATIONAL CONTINUAL LEARNING
Noel Loo, Siddharth Swaroop & Richard E. Turner University of Cambridge
{n1355, s32163, ret26) 8 cam. ac. uk
ABSTRACT
Continual learning deals with training models tasks and datasets in line fashion. One strand of research has used probabilistic regularization for con- tinual learning, with two of the main approaches this vein being Online Elastic Weight Consolidation (Online EWC) and Variational Continual Learning ( (VCL). VCL employs variational inference, which in other settings has been improved empirically by applying likelihood-temperii cation to VCL recovers Online EWC as limiting case, allowing for interpolation between the two approaches. We term the general algorithm Generalized VCI. (GVCL). order mitigate the observed overpruning effect of VI, we take inspi- ration from common multi-task architecture, neural networks with task-specific FiLM layers, and find that this addition leads to significant performance gains, specifically for variational methods. In the small-data regime, GVCL strongly outperforms existing baselines. In larger datasets, GVCL with FiLM layers out- performs or is competitive with existing baselines in terms of accuracy, whilst also providing significantly better calibration.
We show that applying this modifi-
INTRODUCTION
Continual learning methods enable learning when set of tasks changes over time. This topic is of practical interest as many real-world applications require models to be regularly updated as new data is collected or new tasks arise. Standard machine learning models and training procedures fail in these settings (French, 1999), so bespoke architectures and fitting procedures are required.
This paper makes two main contributions to continual learning for neural networks. First, we de- velop new regularization- based approach to continual learning. Regularization approaches adapt parameters to new tasks while keeping them close to settings that are appropriate for old tasks. Two popular approaches of this type are Variational Continual Learning (VCL) (Nguyen et al. 2018) and Online Elastic Weight Consolidation (Online EWC) (Kirkpatrick et al. 2017] Schwarz et al. 2018). The former is based variational approximation of neural network's posterior distribution over weights, while the latter uses Laplace's 's approximation. In this paper, we propose Generalized Vari- ational Continual Learning (GVCL) of which VCL and Online EWC are two special cases. Under this unified framework, we are able to combine the strengths of both approaches. GVCL is closely related to rrelihood-tempeere mance in standard learning settings (Zhang et al. 2018 Osawa et al. 2019). We also see significant performance improvements in continual learning.
Variational Inference (VI), which has been found to improve perfor-
Our second contribution is to introduce an architectural modification to the neural network that combats the deleterious overpruning effect of VI (Trippe & Turner 2018 Turner Sahani 201T). We analyze pruning in VCL. and show how task-specific FiLM layers mitigate it. Combining this architectural change with GVCL results in a hybrid aechiiecural--rgulaaizztttt
based algorithm.
This additional modification results in performance that exceeds or is within statistical error of strong basclines such as HAT (Serra et al. 2018) and PathNet (Fernando et al. (017)..
The paper is organized as follows. Section 2 outlines the derivation of GVCL, shows how it unifies many continual learning algorithms, and describes why it might be expected to perform better than them. [Section 3 introduces FiLM layers, first from the perspective of multi-tase task learning, and then through the lens of variational over- verrpruning, showing how FiLM layers mitigate this pathology of VCL.. Finally, in /Section we test GVCL and GVCL with FiL. layers on many standard bench-

---

Published as a conference paper at ICLR 2021
marks, including ones with few samples, regime that could benefit more from continual learning. We find that GVCL with FiLM layers outperforms existing baselines on a variety of metrics, includ- ing raw accuracy, forwards and backwards transfer, and calibration error. In Seccionn 5.4 we show that FiLM layers provide a disproportionate improvement to variational methods, confirming our hypothesis in Section
GENERALIZED VARIATIONAL CONTINUAL LEARNING
In this section, we introduce Generalized Variational Continual Learning (GVCL) likelihood- tempered version of VCL, with further details in Appendix C We show how GVCL recovers Online EWC. We also discuss further links between GVCL and the Bayesian cold posterior in Appendix
2.1 OKKLIHOOD-TEMPEET
IN VARIATIONAL CONTINUAL LEARNING
Variational Continual Learning (VCL). Bayes' rule calculates posterior distribution over model parameters A based on a prior distribution p()) ) and some dataset Dr {rr. Bayes' rule naturally supports online and continual learning by using the previous posterior p(0) Dr-1) as new prior when seeing new data (Nguyen et al. 2018). Due to the intractability of Bayes' rule in complicated models such as neural networks, approximations are employed, and VCL. (Nguyen al. 2018) uses such approximation, Variational Inference (VI). This approximation is based on approximating the posterior p(0) Dr) with a simpler distribution qr (0), such as a Gaussian. This is achieved by optimizing the ELBO for the optimal (r).
ELBOva. ge [og Dr|0)] - DRL. DKI.(T (o)((rr)-
(1)
where (0) is the approximation to the previous task posterior. Intuitively, this refines a distri- bution over weight samples that balances good predictive performance (the first expected prediction accuracy term) while remaining close to the prior (the second KL-divergence regularization term).
VCL. Optimizing the ELBO will recover the true posterior if the approxi- mating family is sufficiently rich. However, the simple families used in practice typically lead to poor test-set performance. Practitioners have found that performance can be improved by down- weighting the KL-diiergencce regularization term by factor B. with 0 < B 1. Examples of this are seen in ZZang et al. (2018) and Osawa et al. (2019), where the latter uses "data augmentation factor" for down-weighting also been shown to outperform the standard Bayes posterior, where the cold posterior is given by Pr (0] x D)) +, the disentanglement variational autoencoder learned models (Higgins et al. 2017). We down-weight the KL-divergence term in VCL, optimizing the B-ELBC
Likelihood
similar vein, sampling from "cold posteriors" in SG-MCMC has
< 1 (Wenzel et al. 2020). Values of B 1 have also been used to improve
3- B-ELBO (g) [lo - (or
VCL is trivially recovered when 8 1. We will now show that surprisingly 3 - 0, we recover a special case of Online EWC. Then, by modifying the term further as required to recover the full version of Online EWC, we will arrive at our algorithm, Generalized VCL.
2.2 ONLINE EWC A SPECIAL CASE OF GVCL
We analyze the effect of KL-reweighting on VCL in the case where the approximating family is restricted to Gaussian distributions over 0. We will consider training all the tasks with a KL- reweighting factor of 3, and then take the limit / - 0, recovering Online EWC. Let the approxi- mate posteriors at the previous and current tasks be denoted as (T) = Ni(0; 1T- Σr-1) and gr (0; pr, Σr) respectively, where we are learning {UT, Er }. The optimal Er under the 3-ELBO has the form (see Apcendipi C).
Ex
V Ver (o - E
B
'Code is available at https: / httppp/s/giccib...mm//l/////////
We slightly abuse notation by writing the likelihood as p(Dr instead of p(yr Xr))
2

---

Published as a conference paper at ICLR 2021
Now take the limit 3 0. 0. From Equation 2 Ex + 0, so (0) becomes delta function, and
1 1 V vpr (rgp(DDl00 WT) 2T Ilr + -
1
Σ H4 -o t=1
(3)
where Hy is the Tth task Hessiar Although the learnt distribution qu (0) becomes delta function (and not a full Gaussian distribution as in Laplace's 's approximation), we will see that a cancellation of B3 factors in the 3-ELBO will lead to the eventual equivalence between GVCL and Online EWC. Consider the terms in the 3-ELBO that only involve jor:
B (ur PT-1) E Ex (pT PT-1)
3-ELBO Eg (0) [lg [ogg(())()))
2
H₁ + (265)) Σ
= =logp(Dr pT) (UT - - -I-1)²
(WT - PT-1),
(4)
where we have set the form of -r to be in Equation 3
Equation i instance of the objective
function used by a number of continual learning methods, most notably Online EWC (Kirkpatrick
et al. 2017 Schwarz et al. 2018),
Laplace (Ritter al. 2018), and SOLA (Yin
et al. 2020 These algorithms can be recovered by changing the approximate posterior class Q Gaussians with diagonal, block-diagonal Kronecker-factored covariance matrices, and low-rank precision matrices, respectively (see Appendices C.4Jand C.5
Based on this analysis, and continual learning algorithms which use point-wise approximations of curvature as 8 - 0. In Appendix A we explore how 3 controls the scale of the quadratic curvature approximation, verifying with experiments on a toy dataset.. Small B values learn distributions with good local structure, while higher 33 values learn distributions with a more global structure. We explore this in more detail in Appendices A and B where we show the convergence of GVCL to Online-EWC on toy experiment.
that 3 can be seen interpolating between VCL, with B 1.
Inference using GVCL. When performing inference with GVCL at test time, we use samples from the unmodified (0) distribution. This means that when 3 1, we recover the VCL predictive, and 3 + O, the posterior collapses described carlier, meaning that the weight samples effectively deterministic. This is line with the inference procedure given by Online EWC and its variants. In practice, we values of 3 0.05 0.2 in [Section 3| meaning that some uncertainty retained, but not all. We can increase the uncertainty at inference time by using an additional tempering step, which we describe, along with further generalizations in Appendix D
2.3 REINTERPRETING AS COLD POSTERIOR REGULARIZATION
As described above, the 3-ELBO recovers instances of number of existing second-order continual learning algorithms including Online EWC as special cases. However, the correspondence does not recover key hyperparameter À used by these methods that up-weights the quadratic regularization term. Instead, our derivation produces an implicit value of equal sample count. In practice it is found that algorithms such as Online EWC perform best when > 1, typically 10 1000. In this section, we view this hyperparameter as a form of cold posterior regularization.
= 1, i.e. equal weight between tasks
In the previous section, we showed that 3 controls the length-scale over which we approximate the curvature of the posterior. However, the magnitude of the quadratic regularizer stays the same, because the O(3-1) precision matrix and the 3 coefficient in front of the KL-term cancel out. Taking inspiration from cold posteriors (Wenzel et al., 20200), which temper both the likelihood and the prior and improve accuracy with Bayesian neural networks, we suggest tempering the prior in GVCL.
Therefore, rather than measuring the KI. divergence between the posterior and prior, qr and qr-1, respectively, we suggest regularizing towards tempered version of the prior, -- However, this
Thhe actual Hessian may not be positive semidefinite while E, is, here we refer a positive semidefinite approximation the Hessian. 'EWC uses the Fisher information, but our derivation results in the Hessian. The two matrices coincide when the model has near-zero training loss, is often the case (Martens 20201.
3

---

Published as a conference paper at ICLR 2021
form of regularization has problem: in continual learning, over the course of many tasks, old tasks will be increasingly (exponentially) tempered. In order to combat this, we also use the tempered ver- sion of the posterior in the KL divergence, 97. This should allow to gain benefits from tempering the prior while being stable over multiple tasks in continual learning.
As we now show, tempering in this way recovers the À hyperparameter from algorithms such as Online EWC. Note that raising the distributions to the power À is equivalent tempering by X- For Gaussians, tempering a distribution by a temperature 7 = X-1 the same as scaling the covariance by X-1, We can therefore expand our new KL divergence,
DKL (vbllož-1) t(WT - -T-1) AET 1 (HT (T-1) + Tr<AET1A ( log
1E--1AA-3 AA- Er
=1((T -T- X A2T--1 (Her wr-1) ( Er) + log
EEr-il |E
d)
-KLL(erlllrr1))
In the limit of B + O, our À coincides with Online EWC's X, if the tasks have the same number of samples. However, this form of À has a slight problem: it increases the regularization strength of the initial prior Eg on the mean parameter update. We empirically found that this negatively affects performance. We therefore propose a different version of X, which only up-weights the "data- dependent" parts of -1, which can be viewed as likelihood tempering the previous task posterior, as opposed to tempering both the initial prior and likelihood components. This new version still converges Online EWC as 3 + O, since the O(1) prior becomes negligible compared to the O(B-1) Hessian terms. We define,
H. H X(E7 20 + Eg
In practice, it is necessary to clip negative values of Ex² - Epp to keep ΣnT positive definite. This is only required because of errors during optimization. We then use a modified KL-divergence,
= ( - 1, Pr-1) + Tr Er) + log [E
d)
Note that Online EWC, there is another parameter 7, that weights the previous Fisher ma- trices. As shown in Anppendix C we can introduce this hyperparameter by taking the KL divergence priors and posteriors at different temperatures: o² and 77 However, do not find that this approach improves performance. Combining everything, we have our objective for GVCL,
(o)=gr(o))logg(())))) - 3DDI
FILM LAYERS FOR CONTINUAL LEARNING
The Generalized VCL algorithm proposed in Section 2 is applicable to any model. Here we discuss multi-task neural network architecture that especially well-suited to GVCL when the task ID is known at both training and inference time: neural networks with tpek-specific FiLM layers.
3.1 BACKGROUND TO FILM LAYERS
The most common architecture for continual learning is the multi-headed neural network. A shared set of body parameters act as the feature extractor. For every task, features are generated in the same way, before finally being passed separate head networks for each task. This architecture does not allow for task-specific differentiation in the feature extractor, which is limiting (consider, for example, the different tasks of handwritten digit recognition and image recognition). FiLM layers (Perez et al., (2018) address this limitation by linearly modulating features for each specific task so that useful Teatures can be amplified and inappropriate ones ignored. In fully-connected layers, the transformation is applied dlement-wise: for hidden layer with width W and activation values h;, 1 i W, FiLM layers perform the transformation Ky = - + b, before being passed on to the remainder of the network. For convolutional layers, transformations are applied filter-wise. Consider a layer with N filters of size K K, resulting in activations h < S N. 1
م N, W, k < H. where W and H are the dimensions of the resulting feature map. The transformation has the
4

---

Published as a conference paper at ICLR 2021
8 7 6 5
Filer
Fliei
(a) GVCL
(b) GVCL+ FILM
Figure 1: Visualizations of deviation from the prior distribution for filters in the first layer of convolutional networks trained on Hard-CHASY. Lighter colours indicate an active filter for that task. Models are trained either (a) sequentially using GVCL, or (b) sequentially with GVCL + FiLM. FiLM layers increase the number of active units.
form M. = "/ * hi.j.k by. The number of required parameters scales with the number of filters, as opposed the full activation dimension, making them computationally cheap and parameter- efficient. FiLM layers have previously been shown to help with fine-tuning for transfer learning (Rebuffi et al. 2017). multi-task meta-learning (Requeima al. 2019), and few-shot learning (Perez et al.] 2018). 2018)) In Appendix FJ we show how FiLM layer parameters are interpretable, with similarities between Fil layer parameters for similar tasks in a multi-task setup.
3.2 COMBINING GVCL AND FILM LAYERS
is simple to apply GVCL, to models which utilize FiLM layers. Since these layers are specific to each task they do not need distributional treatment regularization continual learning of the shared parameters. Instead, point estimates are found by optimising the GVCL objective function. This has well-defined optimum unlike joint MAP training when FiLM layers are added (see Appendix E for a discussion). We might expect an improved performance for continual learning by introducing task-specific FiLM layers this results more suitable multi-task model. However, when combined with GVCL, there is an additional benefit.
necessary support
When applied multi-head networks, VCL tends to prune out large parts the network (Trippe & Tumeer 2018; Turner & Sahani 2011) and GVCL inherits this behaviour. This occurs in the follow- ing way: First, weights entering a node revert to their prior distribution due to the KL-regularization term in the ELBO. These weights then add noise the network, affecting the likelihood term of the ELBO. To avoid this, the bias concentrates at a negative value so that ReLU activation fectively shuts off the node. In the single task setting, this is often relatively benign and can facilitate compression (Louizos et al. 2017 Molchanov et al. 2017). However, in continual learning the effect pathological: the bias remains negative due to its low variance, meaning that the node effectively shut off from that point forward, preventing the node from re-activating. Ultimately, large sections of the network can be shut off after the first task and cannot be used for future tasks, which wastes network capacity (see Figure lT).
In contrast, when using task-specific FiLM layers, pruning can be achieved by either setting the FiLM layer scale to 0 or the FiLM layer bias to be negative. Since there is no KL-penalty on these parameters, it is optimal to prune this way. Critically, both the incoming weights and the bias of a pruned node can then return to the prior without adding noise to the network, meaning that the node can be re-activated in later tasks. The increase in the number of unpruned units can be seen in Figure Tp. In Appendix G we provide more evidence of this mechanism.
RELATED WORK
Regularization-benn -based continual learning. Many algorithms attempt to regularize network param- eters based metric of importance. Section 2 shows how methods can be seen special cases of GVCL. We now focus on other related methods, [ee et al. (2017) proposed IMM, which is extension EWC which merges posteriors based their Fisher information matrices. Ahn al.
5

---

Published as a conference paper at ICLR 2021
2019), like us, regularizers based the ELBO, but also measure importance per-node basis rather than per-weight one. SI (Zenke et al. 2017) measures importance using "Synaptic Saliency," opposed to methods based on approximate curvature.
Architectural approaches to continual learning. This family of methods modifies the standard neural architecture by adding components to the network. Progressive Neural Networks (Rusu al., 2016) adds a parallel column network for every task, growing the model size over time. PathNet (Fernando et al.) 2017) fixes the model size while optimizing the paths between layer columns. Architectural approaches are often used in tandem with regularization based approaches, such as in HAT (Serra et al., 2018), which uses per-task gating parameters alongside a compression regularizer. Adel et al. (2220) propose CLAW, which also uses variational inference alongside per- task parameters, but requires a more complex meta-learning based training procedure involving multiple splits of the dataset. GVCL with FiLM layers adds to this list of hybrid architectural- regularization based approaches. See bee|Appendix foror a more comprehensive related works section.
EXPERIMENTS
We run experiments in the small-data regime (Easy- CHasY and Hard-CHASY) (eection 5.1), on Split-MNIST (Section 5.I). on the larger Split CIFAR benchmark (Section 5.2). and on much larger Mixed Vision benchmark consisting of different image classification datasets (Section 5.3) In order to compare continual learning performance, transfer (the improvement on the current task as number of past tasks increases (Pan et al. 2020 and backward transfer (the difference in accuracy between when task is first trained and its accu- racy after the final task (Lopez-Paz & Ranzato 2017)). We compare to many baselines, but due to space constraints, only report the best-performing baselines in the main text. We also compare to two offline methods: an upper-bound "joint" version trained on all tasks jointly, and lower-bound "separate" version with each task trained separately (no transfer). Further baseline results A|Ap- pendix J The combination of GVCL on task-specific FiLM layers (GVCL-F) outperforms baselines on the smaller-scale benchmarks and outperforms or performs within statistical error of baselines on the larger Mixed Vision benchmark. We also report calibration curves, showing that GVCL-F is well-calibrated. Full experimental protocol and hyperparameters are reported in Appendix |
compare final average accuracy, forward
5.1 CHASY AND Split-MNIST
10.0
85.0
62.
00.0
[MAP]
just (4AF:
JRST
GICL
DVCLF
GPCL
DVC.
GVCL
75.0
HAT
Number of tasks
(a) CHASY
(b) Hard-CHASY
(c) Split-MNIST
Figure 2: Running average accuracy of Easy-CHASY, Hard-CHASY and Spli--NIST trained con- tinually. GVCL-F and GVCL are compared to the best performing baseline algorithm. GVCL-F and GVCL both significantly outperform HAT on Easy-CHASY. On CHASY GVCL-F still manages to perform well joint MAP training, while GVCL performs well PathNet. In Split- MNIST, GVCL-F narrowly outperforms HAT, with both performing nearly as well as joint training. The CHASY benchmark consists of set of tasks specifically designed for multi-task and continual learning, with detailed explanation in Apppeddix K It is derived from the HASYv2 dataset (Tomma 2017]. which consists of 32x32 handwritten latex characters. Easy-CHASY was designed to max- imize transfer between tasks and consists of similar tasks with 20 classes for the first task, to 11 classes for the last. Hard-CHASY represents scenarios where tasks are very distinct, where tasks range from 18 to 10 classes. Both versions have very few samples per class. Testing our algorithm on these datasets tests two extremes of the continual learning spectrum. For these two datasets we small convolutional network comprising convolutions layers and fully connected layer.

---

Published as a conference paper at ICLR 2021
Riit (9) PIN
(V), RLMO
kint
(V])
Sepurate
[V11
GVCL-F
GVCL
GVCL
PothNot
50
10 Aug
Arg
Tank
Task
(a) Easy-CHAST
(b) Hard-CHASY
Figure 3: Accuracy of Basy-CHASY and Hard-CHASY trained models at the end of learning all 10 tasks continually. Performance of GVCL-R, GVCL and the best performing baselines (HAT and Pathnet) are compared to Joint and Separate training. GVCL--F again strongly outperforms the baselines and performs similar to the upper-bound VI joint training.
For our Split-MNIST experiment, in addition to the standard 5 binary classification tasks for Split- MNIST, we add 5 more binary classification tasks by taking characters from the KMNIST dataset (Clanuwat et al. 2018). For these experiments we used 2-layer fully-connected network, as in common continual learning literature (Nguyen al. 2018 Zenke al. 2017).
Figure 2 shows the raw accuracy results. As the CHASY datasets have very few samples per class (16 per class, resulting in the largest task having training set of 320 samples),
easy to overfit. This few-sample regime is key practical use case for continual learning as it is essential to transfer information between tasks. In this regime, continual learning algorithms based on MAP-inference overfit, resulting in poor performance. As GVCL-F is based as adversely affected by the low sample count, achieving 90.9% accuracy on Easy-CHASY com- pared 82.6% of the best performing MAP-based CI. algorithm, HAT. Hard-CHASY tells similar story, 69.1% % compared to PathNet's 64.8%. Compared to the full joint training baselines, GVCL- achieves nearly the same accuracy (Fiuure 3). The gap between GVCL-F and GVCL is larger for Easy-CHASY than for Hard-CHASY, as the task-specific adaptation that FiLM layers provide more beneficial when tasks require contrasting features, as in Hard-CHASY With SplitMNNIST, GVCL-F also reaches the same performance as joint training, however it is difficult to distinguish approaches on this benchmark as many achieve near maximal accuracy.
Bayesian framework, it is not
GVCL-F
GVCL
HAT
PathNer
VCL
Online EWC
ACC (%)
90.9 0.3
0.6
82.6 22.6:..99
8.94 0.0
78.4
Easy-CHASY
BWT (%)
0.2 0.1
0..
0.6
-1.112
-8.9 2.9
FWT (%)
0.4
0.66
0.4 1.4
-1.5
-7.9
-1.5 U.5
ACC (8))
69.5 0.6
6.
6.4
6.8
15
56.1177
Hard-CHASY
0.1
0.2
0..4
0.0
1.
-7.1
FWT (%)
11.6.....7
-63
-3.7
-2.2
13.
33.4 1.3
SplI-MNIST (10 Tasks)
ACC ACC
98.6 0.1
0.76
0.1
95.2
1.2
94.0 1.4
BWT (%)
0.0
0.7
0.02
0.0
1.11
-3.8 1.4
FWT
00. 0.1
0.0
0.11
1.3
-..
0.88 0.1
ACC
0.5
7.76
0.3
0.87
41.2
T7.1
Spla-CIFAR
BWT((()) (53
0.3 0.2
1.4
-0.1
0.0
12.2 23.9
0.5 0.3
FWT (9))
8.8 + 0.5
13
6.8
-.9
2..
6.9 0.3
Mixed Vision Tasks
ACC (56)
80.0 1.2
1.2
49.0
80.3
7.
26.9
62.8 5.2
BWT (56)
0.9 1.3
13.1
1.6
0.1
0.0
35.0
18.7 5.8
FWT (%)
4.8
1.6
23.5
0..533..4 3.4
5..8 1.0
2.0
23.7 3.8
4.8 0.7
Table 1: Performance metrics of GVCL-F and GVCL compared to baselines (more in Appendix GVCL-F obtains the best accuracy and backwards/forwardk transfer on many datasets/archheecturs..
5.2 Split-CIFAR
The popular Split-CIFAR dataset, introduced in Zenke et al. (017). has CIFAR10 as the first task, and then 5 tasks disjoint 10-way classifications from the first 50 classes of CIFAR100, giving

---

Published as a conference paper at ICLR 2021
total of 6 tasks. We use the same architecture as in other papers (Zenke al. 2017 Pan al. 2020 Like with Easy-CHASY, jointly learning these tasks significantly outperforms networks separately trained on the tasks, indicating potential for forward and backward transfer in continual learning algorithm. Results are in Figure 4 GVCL-F is able to achieve the same final accuracy as joint training with FiLM layers, achieving 80.0++0.5% beating all baseline algorithms by at least 2%. This confirms that our algorithm performs well in larger settings as well as the previous smaller- scale benchmarks, with minimal forgetting. While the backwards transfer metric for many of the best performing continual learning algorithms is near O, GVCL-F has the highest forward transfer, achieving 8.5%.
GVCL consistently outperforms VCL. but unlike in the CHASY experiments, it does not outperform Online EWC. This also occurs in the Mixed Vision tasks considered next. Theoretically this should not happen, but GVCL's hyperparameter search found / 0.2 which is far from Online EWC. We believe this is because optimizing the GVCL cost for small / is more challenging (see/Appendix B). However, since intermediate settings result in more pruning, FiLM layers then bring significant improvement.
Owerage accuracy (96)
Accuracy (86)
(VI. HLM)
Separate
(Vi)
76
GVCL-F
74
Joint IMAP]
GVCL
GVCLF
$ HAT
HAT
Avg
Number tasks
Task
(a) Running average accuracy of SplitCIFAR
(b) Final accuracies Split-CIFAR
Figure 4: Running average accuracy of Splil-CIFAR and final accuracies after continually training on 6 tasks for GVCL-F, GVCL, and HAT. GVCL-F achieves the maximum amount of forwards transfer, and achieves close to the upper-bound joint performance.
5.3 MIXED VISION TASKS
100
GVCL-F
2.5
HAT
Joint
(VI, FILM)
Accuracy
Separate
(VI)
GVCL-F
1.0
HAT
0.0
Oife120
Chara no
MBNS7
SVHN
fauhlenwwiST
Sighti
Recescruu
NatMAUSST
Taslss
(a) Final accuracies after tasks
(b) Relative accuracy after training on task
Figure 5: (a) Average accuracy of mixed vision tasks at the end of training for GVCL-F and HAT. Both algorithms perform nearly equally well in this respect. (b) GVCL-F gracefully forgets, with higher intermediate accuracies, while HAT has lower initial accuracy but does not forget.
We finally test on a set of mixed vision datasets, as in Serra et al. (2018). This benchmark consists of 8 image classification datasets with 10-100 classes and range of dataset sizes, with the order of tasks randomly permuted between different runs. We use the same AlexNet architecture as in Serra al. (2018). Average accuracies of the 8 tasks after continual training shown in Figure 5
8

---

Published as a conference paper at ICLR 2021
GVCL-F's final accuracy matches that of HAT, with similar final performances of 80.0 and 80.3+ for the two methods, respectively. Figure 5b shows the relative accuracy of the model after training intermediate tasks compared to its final accuracy. A positive relative accuracy after t tasks means that the method performs hetter on the tasks seen so far than it does on the same tasks after seeing all tasks (Appendix cooottinn precise definition). HAT achieves its continual learning performance by compressing earlier tasks, hindering their performance in order to reserve capacity for later tasks. In contrast, GVCL-F attempts to maximize the performance for early tasks, but allows performance to gradually decay, as shown by the gradually decreasing relative accuracy in Figure 5b] While both strategies result in good final accuracy, one could argue that pre-compressing network in anticipation of future tasks which may may not arrive is an impractical real-world strategy, as the number of total tasks may be unknown a priori, and therefore one does not know how much to compress the network. The approach taken by GVCL-F is then more desirable, as it ensures good performance after any number of tasks, and frees capacity by "gracefully forgetting".
Exoocted Colloration Erao (No
Expected probsbility
(a) Cifar100 Calibration Curve
(b) Facescrub Calibration Curve
(c) ECE on individual tasks
Figure 6: Calibration curves and Expected Calibration Error for GVCL-F and HAT trained on Mixed Vision Tasks benchmark. GVCL-F achieves much lower Expected Calibration Error, attain- ing a value averaged across all tasks of 0.3% compared to HAT's 1.7%.
Uncertainty calibration. As GVCL-F is based probabilistic framework, expect it to have good uncertainty calibration compared to other baselines. We show this for the Mixed Vision tasks Figure 6) Overall, the average Expected Calibration Error for GVCL-F (averaged over tasks) is 0.32%. compared to HAT's 1.69%, with better ECE on 7 of the 8 tasks. These results demonstrate that GVCL-Fis is generally significantly better calibrated than HAT, which can be extremely important in decision critical problems where networks must know when they are likely to be uncertain.
5.4
RELATIVE GAIN FROM ADDING FILM LAYERS
Algorithm
Easy CHASY
CHASY
MpIS (10 lusks)
Split CIFAR
Mixed Vision Tisks
Average
GVCL.
2.0 4 0.5%
5.1 = 0.4%
4.0 : 0.7%
9.5 = 1.4%
31.0
2.2%%
2.2%
10.3 : 10.6%
VCL
1.5 d: 1.2%
19.2 1.4%
2.4 :: 1.5%
12.0 d: 16.4%
28.6
3.6%
12.8
10.3%
Online FWC
2.6 d: 3.3%
0.3 7.1%
0.1 1.2%
0.1 0.1%
2.1%
2.2 2.9%
Table Relative performance improvement from adding FiLM layers on several benchmarks, for VI and VI based algorithms. VI-based approaches see much more significantly gain over EWC, suggesting that FiLM layers synergize very well with VI and address the pruming issue.
In Section 3)
we suggested that adding FiLM layers to VCL in particular would result in the largest gains, since addresses issues specific VI, and that FiLM parameter values were automatically best allocated based on the prior. In Section 5.4, we compare the relative gain of adding FiLM layers to VI-based approaches and Online EWC. We omitted HAT, since it already has per-task gating mechanisms, so FiLM layers would be redundant. We see that the gains from adding FiLM layers to Online EWC are limited, averaging 2.2% compared to over 10% for both VCL and GVCL. This suggests that the strength of FiLM layers is primarily in how they interact with variational methods for continual learning. As described in Section 3) with VI we do not need any special algorithm to encourage pruning and how to allocate resourees, they done automatically by VI. This contrasts HAT. where specific regularizers and gradient modifications are necessary to encourage the use of FiL. parameters.

---

Published as a conference paper at ICLR 2021
CONCLUSIONS
We have developed a framework, GVCL, that generalizes Online EWC and VCL, and we combined with task-specific FiLM layers mitigate the effects of variational pruning. GVCL with FiLM layers outperforms strong baselines on a number of benchmarks, according to several metrics. Fu- ture research might combine GVCL with memory replay methods, find ways to use FiLM layers when task ID information is unavailable.
10

---

Published as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Char- less Fowlkes, Stefano Soatto, and Pietro Perona. Tesk2Vec: Task Embedding for Meta- Learning. arXiv:1902.03545 [cs, stat), February 2019. URL http: org/ abs/ 1902. 03545
arXiv: 1902.03545.
Alessandro Achille, Giovanni Paolini, and Stefano Soatto. Where is the information a deep neural network?, 2020.
Tameem Adel, Han Zhao, and Richard E. Turner. Continual learning with adaptive weights (claw). In International Conference on Learning Representations, 2020. URL https://
openreview
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-based continual learning with adaptive regularization. In Advances in Neural Information Processing Systems 32, pp. 4392-4402. Curran Associates, Inc., 2019. URL http:: nips. cctpp//.cs...ppps..ppp.....p.pe. 8690 1890-uncee haseellntt---ad--ooto--o inual -ataaddcooooocnaaaaaaaaaaaaaaaaa roouaaaaaaaaaeeeooooaooaaaaaaaee ioguaaitioon........ pdf
Alexander Alemi, Ben Poole, lan Fischer, Joshua Dillon, Rif A. Saurous, and Kevin Murphy, Fixing broken ELBO. volume 80 of Proceedings of Machine Learning Research, pp. 159-168, Stock- holmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL ltttp:/prroceedinng/
mlr. wwwpppresssve.. vww//vv////// http//aemmll.....m html
Tarin Clanuwat, Mikel Bober-Irizar, A. Kitamoto, A. Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. ArXiv, abs/1812 122/0111,,1 2018.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. 01 2017.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4): 125-135, April 1999. ISSN 1364-6613. doi: 10.1016/51364-66139999
01294-2. URL http:// http:://wwww.. www.sciienedirect...... http://tt..cccceeeeee/errrtllaaa
61364661399012942
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. (-VAE: LEARNING BASIC VISUAL CONCEPTS WITH A CONSTRAINED VARIATIONAL FRAMEWORK. pp. 22, 2017.
J. Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, J. Veness, G. Desjardins, Andrei A. Rusu, K. Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwer Barwinskk, Demis Hassabis, C. Clopath, D. Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural net- works. Proceedings of the National Academy of Sciences, 114:3521 3526, 2017.
Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Woo Ha, and Byoung-Tak -Tau Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 30. pp. 4652 4662. Curran Associates, Inc., 2017. URL
nips wwt.:eiwsssssppppeeeeeeeee catpcper/pprrr -oment ma -ac--aaccoocooo cching.
7051-overcome ng- catastrophics wow.ettingggggoeoooooo pdf
incrementa c by- woy-innreee
David Lopez-Paz and Mare Aurelio Ranzato. Gradient episodic memory for continual learning. In Guyon, U. V. Luxburg, Bengio. H. Wallach. Fergus, S. Vish- wanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems 30, pp- 6467-6476. Curran Associates, Inc., 2017. URL htttp://papers. m wttp:r/pppppp.ccppppppppppp 7225 gradient appssodic-meme
for continual-learminn
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learn ing. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems 30, pp-
11

---

Published as a conference paper at ICLR 2021
3288 3298. Curran Associates, Inc., 2017. URL http::/paperr: http:apppwrr.ii.scccc.opppaaaaaa
1921-aayesian--ooaeessooooo for dew.--dee--eaaannnn.g.ee.
James Martens. New insights and perspectives on the natural gradient method. Journal of Ma- chine Learning Research, 21(146):1-76, 2020. URL httt:://jml..rrg/// wwtp:rgggppppp/apapa
17 118.html
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep
neural networks. volume 70 of Proceedings of Machine Learning Research, pp. 2498-2507, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:
//proceedings.. .ww...s....rrr.................. wwwss/eeevv. v700 mc chw.ovlaaoolaaaa html
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner Variational continual
learning. In International Conference Learning Representations, 2018. URL https:
//openreviees tepperrreiee..ee//ooommmmm dotkoorrommmdddokgggggg
Manfred Opper and Cedric Archambeau. The variational gaussian approximation revisited. Neural computation, 21:786-92, 10 2008. doi: 10. 1a.1166//eco.2000 20teeec...00..000 00000808077-52..
Kazuki Osawa, Siddharth Swaroop. Mohammad Emtiyaz E Khan, Anirudh Jain, Runa Es- chenhagen, Richard Turner, and Rio Yokota. Practical deep learning with bayesian principles. In Advances in Neural Information Processing Systems 32, pp. 4287- 4299. Curran Associates, Inc., 2019. URL http://papers... inprrrsnnpiss. nips. cc/paper/
8681 wwacticalll...............a.a.ce deep- Learning with bayesian princippess pdd
Pingho Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard E. Turner, and Mohammad Emtiyaz Khan. Continual Deep Learning by Functional Regularisation of Memo- rable Past. arXiv:2004.1444 14070 [cs, stat], June 2020. URL
14070 arXiv: 2004. 14070.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 506-516. Curran Associates, Inc., 2017. URL http: /ttpt::://ppapss.etee nips cc/paper/ dwwains-wttt-eeeeeee 6654 1es5s-Laannnnnmmtpppl------eeeee http://wtttttiaaaaooooooooaaaaaa seeea pdf
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems 32, pp. 7959-7970. Curran Associates, Inc., 2019.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, Curran Associates, Inc., 2018
pp. 3738-3748
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive Neural Networks. 16Xiv:1606.0 1606.066664 [cs]. September 2016. URL httt:://arxii..rrg/aa/////1// 04671 arXiv: 1606.04671.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. volume 80 of Proceedings of Machine Learning Research, pp. 4528 45377, Stock- holmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http::tpp:/:o/oo htt.httt::/pprccednggg httml
Yee Whye
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. volume 80 of Proceedings of Machine Learning Re- search, pp. Stockholmsmässsan Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceediin www.p//sssss..srsss.mmmmmmm
press v serra html
12

---

Published as a conference paper at ICLR 2021
Alexander Smola, Vishy Vishwanathan, and Eleazar Eskin. Laplace propagation. 01 2003.
Siddharth Swaroop, Cuong V. Nguyen, Thang D. Bui, and Richard E. Turner. Improving and Understanding Variational Continual Learning. arXiv:190S.0220 [cs, stat), May 2019. URL
http://arxixy org/ wwwp////g////ooeee/ooo//eee/eaee abs// 905. 02099
arXiv: 1905.02099.
Martin Thoma. The HASYv2 dataset. arXiv: /701.08800 [cs], January 2017. URL http://
arxiv.org/abaa. /wt.ab//101. 1701. 08380
arXiv: 1701.08380.
Brian Trippe and Richard Turner. Overpruning in Variational Bayesian Neural Networks.
arXiv:1801.066 06230 [stat], January 2018. URL
arXiv: 1801.06230
R. Turner and M. Sahani. Two problems with variational expectation maximisation for time-series models. 2011.
Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? CoRR, abs/2000..02055, 2020. URL https:
//arxiv.org/abs//
02405
Dong Yin, Mehrdad Farajtabar, and Ang Li. SOLA: Continual Learning with Second-Order Loss Approximation. arXiv:2006.100 10974 [cs, stat], June 2020. URL http://aaxiv.rr///a00 2006. 10974
Xrviv: 10974.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. volume 70 Proceedings of Machine Learning Research, pp. 3987-3995, International Conven- tion Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
preas/s70/zenkel html
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse Noisy natural gradi- ent variational inference. volume 80 of Proceedings of Machine Learning Research, pp. 5852-5861, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http: //proceedi ngs.. ml press v80 zhang html
13

---

Published as a conference paper at ICLR 2021
A LOCAL VS GLOBAL CURVATURE IN GVCL
In this section, we look at the effect of on the approximation of local curvature found from opti- mizing the 3-ELBO by analyzing its effect toy dataset. In doing so, aim to provide intuition why different values of 3 might outperform 3 1. We start by looking at the equation of the fixed point of E.
ET = V uu V 1ol lo -
(5)
We consider the T 1 case. We can interpret this as roughly measuring the curvature of log (r)0)) at different samples of 0 drawn from the distribution qr (0). Based on this equation, we know ΣT increases as 3 decreases, so samples from @r()) are more localized, meaning that the curvature measured closer the mean, forming a local approximation of curvature. Conversely, if 3 is larger, 20 broadens and the approximation of curvature is on more global scale. For simplicity, write Ver V ( log p( (Dr (0) Hr.
To test this explanation B, we performed 3-VI VI on a simple toy dataset.
We have true data generative distribution X N(O,, 1), and sample 1000 points forming the dataset, D. Our model is generative model with X ~ (N)( 30), with 0 being the model's only parameter and (() an arbitrary fixed function. With B-VI, aim approximate p(0) D) with (0) = -N((;; u. σ²) with prior p(0) = -N(0; 0, 1). We choose three different equations for f(0):
1. f1(0) = |@]
2. f2(0) = VIO
3. fs(0) = [(10) 0.5) + 0.4
We visualize log ((1() for each of these three functions in Figure 7 likelihoods have very distinct shapes. fi results in a likelihood that is flat locally but curves further away from the origin. f2 is the opposite: there a cusp 0 then flattens out. fa mix, where at very small scale it has high curvature, then flattens, then curves again. Now, we perform 3-VI to get ju and o², for 3 E {0.1, 10). We then have values for a², which acts as >oT in Equation 5 We want extract Hr from these values, so we perform the operation 8²
Here, we see that the data
= -1 which represents
our estimate of the curvature of log D)0) at the mean. This operation also "cancels" the scaling effect of 3. We then plot these approximate log-likelihood functions log p( DI0) Ni p, o??) in Figure
-35
-70
-40
-50
-30
(a) f1(0)
(b) fz(0)
(c) f2())
Figure 7: True data log-likeling loeliikllhoods of generative model of the form p(x(0) N(x;; 1()) Curves are shifted so that they pass through the origin
From these figures, we see a clear trend: small values 3 cause the approximate curvature to be measured locally while larger values cause it to be measured globally, confirming our hypothesis. Most striking is Figure 8c) where the curvature is not strictly increasing or decreasing further from the origin. Here, becomes high again for 3 10. Now imagine in continual learning our posterior for a parameter whose posterior looks like Figure 8a Here, the parameter would be under-regularized with 1,
that the curvature first is high for 8 = 0.1, then flattens out for 3 1 then
14

---

Published as a conference paper at ICLR 2021
10
(a) (0)
(b) f2
(c) fa
Figure 8: Approximate data log-likelihoods found using 3-VI for various values of / for three different generative models. Small values of B cause local approximations of curvature and large values cause global ones.
so the parameter will drift far away, significantly affecting performance. Equally. if the posterior
was like Figure 8b] then values model capacity than in practice could be freed. In practice we found that 3 values of 0.05 0.2 worked the best. We leave finding better ways of quantifying the posterior's variable curvature and ways of selecting appropriate values of future work.
would cause the parameter be over-regularized, limiting
B CONVERGENCE TO ONLINE-EWC ON A TOY EXAMPLE
Task
2
Combined
2-
0-
-1
-2
-2
Figure 9: Visualization of simple 2d logistic regression clustering task. The first task is distin- guishing blue and red, classes and 2 respectively. The second task is distinguishing green (class 1) from yellow (class 2). The combined task is shown on the left
Here, we demonstrate convergence of GVCL to Online-EWCC for small B. In this problem, we deal with 2d logistic regression toy dataset consisting of separated clusters. The clusters shown in Figure 9] The first set of tasks is separating the red/blue clusters, then the second is the yellow/green clusters. Blue and green are the first class and red and yellow are the second. Or model is given by the equation
ply: = 1/w, b, r.) ow x, + b)
Where w; are our datapoints and w and b are our parameters. y/4 = means class (and y; means class 1). x is 2-dimensional we have total of parameters.
Next, we ran GVCL with decreasing values ,9 and compared the resulting values of w and after the second task to solution generated by OnW. For both cases, set À 1. For our prior, we used the unit normal prior on both w and b, our approximating distribution was fully factorized Gaussian. We this experiment for 5 random seeds (of the parameters, not the clusters) and plotted the results.
Figure 10 shows the result. Evidently, the values of the parameters approach those of Online-EWe EWC as we decrease B. in line with our theory. However, it is worth noting that to get this convergent behaviour, had to this experiment for very long. For the lowest 3 value, it took 17 minutes to
15

---

Published as a conference paper at ICLR 2021
12
-0.5
-0.50
10
-1.0
00.5
8-
-1.00
-122
-2.0
1.50
160
1e:1
1e:3
2e/4
EWC
1e-1
1e:2
1e:3
1e:4
EWC
160
1e:1
1e:3
16:4
EWC
B
Figure 10: Convergence GVCL parameter values Online-EWC parameter values for decreasing values of 3 for a toy 2d logistic regression problem
converge compared to 1.7 for 3 1. A small learning rate of lc-4 with 100000 iteration steps was necessary for the smallest 3 =le-4. If the optimization process was for shorter, too large learning rate was used, we would observe convergent behaviour for the first few values of B, but the smallest values of / would result in completely different values.
This shows that while in theory, for small 3, GVCL should approach Online-EWi EWC, it extremely hard achieve in practice. Given that it takes long achieve convergent behaviour model with 3 parameters, is unsurprising that we were not able to achieve the same performance as Online-EWC for our neural networks, and explains why despite GVCL, in theory, encompassing Online-EWC, can sometimes perform worse.
C FURTHER DETAILS ON RECOVERING ONLINE EWC
Here, we show the full derivation to recover Online EWC from GVCL, as - 0. First, we expand the S-ELBO which for Gaussian priors and posteriors has the form:
B-ELBO Ea (0) log (ogp(Dr(()) (Kr(
B
Eg (8) flog Dr 0))
log EEr-1l loog [Er] d
Tr(E-- Σr) + (UT - (1-1) ' (MT 1r-1))
where (0) our approximate distribution with means and covariance pr and Σr, and our prior distribution 1 (0) has mean and covariance pr-1 and Ey 1- Dr refers to the Tth dataset and d the dimension of p. Next, take derivatives wrt En and set 0:
3
B
V B-BBBO = V Mi Eg -ur (0) logp
2
-TT
2
1-1
2T-1
(6)
3
0
V V E8
log (r) +
2 -
2 T-
(7)
Σ
3 Van
V E (0) log Dr + Σ
(8)
We move from Equation to Equation 7 Jsiing Equation 19 in Opper & Archambeau 2008). From
Equation 8) we see that as + O. the precision grows indetinielly, so qr(() approaches a delta function centered its mean. We give precise explanation of this argument in Appendix C.T
16

---

Published as a conference paper at ICLR 2021
We have
V
V po log((Dr0 =r) + -7-1
3 pe
1
ΣT HT + 2T-
where Hy the Hessian of the Tth dataset bog-likelihood. This recursion of Exp gives
ET
11+
t=1
Now, optimizing the 3-ELBO for prt (ignoring terms that do not depend on pr):
B (ur pT-11)
B-ELBO (0)
E pT-11)
(10)
T-1
logr( = WT)
2 1uT (ur -TT-) pT-1))
H. + (50 (UT (T-1). Σ
(11)
Which is the exact optimization problem for Laplace Propagation (Smola et al. 2003). If we note that HT Rs Fr (Martens 2020), where Nr is the number of samples in the 7hh dataset and Fr the Fisher information matrix, we recover Online EWC with = 1 when N1 = N2 = Nr
(with y 1).
C.1
CLARIFICATION OF DELTA-FUNCTION ARGUMENT
In we argued,
Σ7 3 ur Vur -o +E
HT + ST-
for small 3. We argued that for small B, q(0) collapsed its mean and it is safe to treat the expectation as sampling only from the mean. In this section, we show that this argument is justified.
Lemma 1. If q()) has mean and covariance parameters μ and E, and
Σ-1 1 A p + C, C O ). then for small B, Σ =e H + where Hp is the Hessian of f0) evaluated at p. assuming H, = O(1)
Proof. We first assume that f(0) admits Taylor expansion around μ. For notational purposes, define,
Of
Tke Ta..... 10=p ag(ks) (g(k.. d=
For our notation, upper indices brackets indicate vector components (not powers), and lower indices indicate covector components. Note that, H T
0==
Then, Taylor expansion centered at y has the form
f(0) - f() +
Σ Tke
w))ck)) ... .0 0 (0
(kn
n!
d=p
===
II this case, the / in Ha.i. refers the Hessian evaluated μ4, while i, refers the indices
17

---

Published as a conference paper at ICLR 2021
Where Einstein notation,
Th kn 1=p (0 (0-1))(h) 11)(k)) ... (0 [h)
Σ
Tki..... kn 1=
(0 ())(ka) ... ...0 - 1n)
-0-
(12)
With D the dimension 0. denote the central moments of g(0), we define
j (l)) = Eg-g((0) [00 14) (1) ... - n1) (ba)
These moments can computed using Isserlis' theorem. Notably, for a Gaussian, if n odd, ((.......)))
Now, we can compute our expectation as an infinite sum:
je V V. [f(}]] - V. V. Eo)a(o)
f(u) -
The an lamp
(0 - ()) ... (0 4))
n!!
V.VA
f(1) +
T Ter....k
Cl Ree)
n2.
V.VA
f(1)
Σ
TE
oe Cenaen..ku kan)
(odd moments are 0)
2nd
8mp
A for notational simplicity
We can look individual components of A:
① a )yli)
20 Σ
Ar puli)
f(1)
kzn
0=p 1. k2n)
2n!
n=1
= =Ti,
Ts. N...k.......
2n!
a=p
n=1
Now we can insert this into our original equation.
Σ-1 E. (S(0)) C
Σ
+
B
ΣJ B
Aij + Ois
looking at individual indices
EJ
T D=p
f (K
Col.....aal kzn)
+ Cij
=
B
2n!
........ Non
O(1)
0(8)
is too), which means that E must be least ( If I- O()). then Σ 0 From Isserlis' theorem, we know that j k2n.....a)) is composed of the product of n elements of Σ, so ((. = O(3)). Tr..d d= 9, so O(1). Hence, the summation is 0 which for small negligible compared the O(1) term T d=y"
Now we assumed that H,, 0(1) (so Túj
0=
is constant with respect
therefore be ignored. Then, keeping only O (1) terms,
18

---

Published as a conference paper at ICLR 2021
O(3)
0(
O(1)
00
Tis 0mm
2n!
Ligh
Aan
1
kan)
Ci.
((+)
0(1)
0
0(
20
2
Tio A=p
+ B
T Ti Kan.kan
a ah (llnmmnk kan)
+ Cis
2nd
10=p
n=1
+ +Cis
1=
Hiids + C.us
B
Σ 2 H + C
C.2 CORRESPONDING GVCL'S AND ONLINE EWC'S X
We use DKL in place of DKL, with PKLÄ defined as
1 _/2 (IT - (HT--TT1)) - 1,AUT -T- + Tr(ET-1 Ex)
(ktx(orller-))=
2
+ log EE--1 1| - d - log [ET]
with
À T
Σ
=
Σ
Hi
+
2n = A - + -
B
1=1
Now, the fixed point for 2n is still given by Equation 9 but the 3-ELBO for for terms involving pT has the form,
8-ELBO E [ogpp(())) (PT 1T-1)
- HT-1)
T
log = (T) 1 [ - -!T
Σ
He 125 (UT PT-1),,
1=1
which upweights the quadratic terms dependent on the data (and not the prior), similarly to X in Online EWC.
C.3 RECOVERING y FROM TEMPERING
In order to recover X, we used the KL-divergence between tempered priors and posteriors 9 and d Recovering 7 can be done using the same trick, except temper the posterior to d/A²
DKL (1 ((ur pT-1) XE (IT 1TT-1) +Tr(NAT_1 > 11rrr
Er) + log 1 15 11 -d)
[1>>) 12x1 d)
= =]((ur 11) 1 11-1) + +TTT(ETT Er) log (Erl) cons.
=DKLAy(orlQu--)
19

---

Published as a conference paper at ICLR 2021
We can apply the same À to À as before to get D (ex P (1q1-1)) Plugging this into the 3-ELBO and solving yields the recursion for 2i to be
ΣT B
Hr + -
which is exactly that of Online EWC.
C.4 GVCL RECOVERS THE SAME APPROXIMATION OF FT AS ONLINE EWC
The carlier analysis dealt with full rank 27. In practice, however, ET is rarely full rank and deal with approximations of Dr. In this subsection, we consider diagonal Er. like Online EWC, which in practice uses diagonal approximation of Fr. The way Online EWC approximates this diagonal is by matching diagonal entries Fy. There are many ways of producing a diagonal approximation of matrix, for example matching diagonals of the inverse matrix is also valid, depending the metric we use. Here, we aim to show that that the diagonal approximation of Σr that is produced when Q the family of diagonal covariance Gaussians is the same the way Online EWC approximates Fr, that is, diagonals of approx match diagonals of TT, i.c. we match the diagonal precision entries, not the diagonal covariance entries.
Let approxx diag(oi, 12 ..., .2), VI. we are optimizing the forwards KL divergence, i.e. DKL. (Rr.((aapoxx (mme). Therefore, ignoring terms that do not depend on approx.
with d the dimension of the matrix. Because we are performing
1
DKL PKLL(apppou (aupprox (/quee)
Tr(ET apppoxx E Tine
log Epppppo
(constants wrt ETppoo)
2
2
d
T 2T )
log of
2
2
i=1
d
(D k.s) - log of?
2
Optimizing wrt of:
b @KLL DKL (Aapprxx (true)
1 1(
1 0,
0
(Te
lis
do?
2
+ of =
(E
So we have that diagonals of 2 approx match diagonals of T'mee"
C.5 GVCL RECOVERS THE SAME APPROXIMATION OF Hy AS SOLA
SOLA approximates the Hessian with rank-restricted matrix // Yin et al. 2020). We first consider a relaxation of this problem with full rank, then consider the limit when we reduce this relaxation.
concerned with limiting 3 - O, it is sufficient to consider Em H. the true Hessian. Because H symmetric (and assuming is positive-semi-deffiit))) we can also write H as H VDVT Ei= Aparzee with D, and V be the diagonal matrix of eigenvalues and a unitary matrix of eigenvectors, respectively. These eigenvalues and eigenvectors are X; and æ;, respectively, and p the dimension H.
Because
For H. we first consider full-rank matrix which becomes low-rank as 8 + 0:
ñ
A
Σ
8ā.0]
j=
20

---

Published as a conference paper at ICLR 2021
This matrix has A, 1 S i S l as its first i eigenvalues and 6 as its remaining. We also set 2. 2; and म. āj = 0, $.
With KL. minimization, we aim to minimize (up to a constant and scalar factor),
KL = Tr( Tpppoxp Emee - log Epproxx
In our case, this is Equation 13 which we can further expand as,
KL = T
log |
(13)
P
h log(A.)
P +
Tr
H
log8 8
(14)
X
1=1
j=k+1
i=1
j=k+1
k
1
p
1
Tr
2
H
+Tr
8
H
+
log(A
+
Σ
log
(15)
A
x=++1
j=k+1
k
Σ
Hāg +
log(An (2x) +
Σ
log 8
(16)
j=++1
j=k+1
(17)
Taking derivatives wrt dj we have:
0KL
1 2 H +
0 = 0 = _
(18)
X2
+ A = H..
H3,
(19)
Which when put into Equation 16
KL
H
+
Σ
Hā +
Σ log(Ar) +
Σ
log 8
(20)
j=k+1
i=1
j=++1
Hb:
k
Hag
+
(x:( +
log 8
(21)
lb:
b=
1=1
b=
kk
H
+
(x,) +
log 8
(22)
j=++1
j=
P
k Σ
=
Σ
H
+
log(Ai)
(removing constants)
(23)
j=k+1
1=1
p
k
Σ
Hāg
Σ
log(2, Hā;)
(24)
j=k+1
Now we need to consider the constraints *T.; = 1 and a. 3. = 0, * j by adding Lagrange multipliers our KL cost,
P =
Σ iJ
HJ
+
M log(ā, [ Hw;) -
Σ
or. (ā, 2 1) - T
(25)
8
j=
i=1
(),
21

---

Published as a conference paper at ICLR 2021
Taking derivatives wrt øy:
0L
2H
20,,5 - Σ
(26)
0i;
.. H
H
Σ 4
(estp) 8i
(27)
T. H
In Eguation 27 we have ž; expressed as linear combination of žソ.ğ + i, but T; and ã, are orthogonal, zú cannot be expressed such, 0i 0, + and,
Hd; . HJ;
(28)
Meaning ã; are eigenvectors H for S S We can also use the same Lagrange multipliers to show that 3; for k + s i S are also eigenvectors of H.
This means that our cost,
P KL
& Σ
Σ
HJ,
log( H
(29)
j=k+1
p
k
Σ
Ki
log(Fi))
(30)
where the set (Mi, R2,... Rp) a permutation of (A1, A2 .... Ap) and R; X, for sis T.e., A shares k eigenvalues with H, and the rest are 8. It now remains to determine which eigenvalues are shared and which are excluded.
Considering only two eigenvalues, Ao Aj. and let A; > A 2 0. Letr d. excluding X; in the set (i1, R2, ..., Rx) compared to including it is,
%,
The relative cost of
A Aj log A 8
Relative Cost
Xj
1x 1
log
8
If the relative cost is positive, then including X; as of the eigenvalues of A is the optimal choice. Now solving the inequality,
Relative Cost >>
Ax )) -logr
1) logr
X 5(1
Which, for sufficiently small is always true because r Thus, it is always better swap two cigenvalues which included/excluded, if the excluded one is larger. This means that H has the l largest eigenvalues of H, and we already showed that it shares the same cigenvectors. This maximum eigenvalue/eigenveetor pair selection is exactly the procedure used by SOLA.
22

---

Published as a conference paper at ICLR 2021
D COLD POSTERIOR VCL AND FURTHER GENERALIZATIONS
The use of KL-reweighting is closely related related to the idea of "cold-posteriors." in which Pr (0]D) D)+ Finding this cold posterior is equivalent to find optimal q distributions for maximizing the E-ELOO:
--BBO == E0 p DD))) + log - log g(0)
when Q is all possible distributions of 0. This objective is the same the standard ELBO with only the entropy term reweighted, and contrasts the 3-ELBO where both the entropy and prior likelihoods reweighted. Here, 3 acts similarly to T (the temperature, not to be confused with task number). This relationship naturally leads to the transition diagram shown in Figure 11 In this, we can see that we easily transition between posteriors at different temperatures by optimizing either the 3-ELBO, 7-ELBOO or tempering the posterior.
Cold (T < 1)
p p P(O
B-ELBO
D(DD
B.BBO
p p D1.2)
Tempering
r-ELBO
Tempering
r-ELBO
Tempering
Warm (T 1)
p(0)
ELBO
(1)
ELBO
p(0) D1:2)
Figure 11: Transitions between posteriors at different temperatures using tempering and optimizing either the 7-ELBO or 3-ELBO
When Q contains all possible distributions. moving along any path results in the exact same distri- bution, for example optimizing the r-ELBO then tempering the same as directly optimizing the ELBO. However in the case where Q is limited, this transition is not exact, and the resulting poste- rior is path dependent. fact, each possible path represents different valid method for performing continual learning. Standard VCL works by traversing the horizontal arrows, directly optimizing the ELBO, while an alternative scheme of VCL would optimize the T-ELBO to form cold posteriors, then heat the posterior before optimizing the T-ELBO for new task. Inference can be done at either the warm or cold state. Note that for Gaussians, heating the posterior is just a matter of scaling the covariance matrix by a constant factor Taboe
Thefire
While warm posteriors generated through this two-step procedure are not optimal under the ELBO, when is limited, they may perform better for continual learning. Similar to Equation 2] the optimal E when optimizing the T-ELBO is given by
T H
E
E
Where H, is the approximate curvature for a specific value of r for task t, which coincides with the true Hessian for - 0, like with the 3-ELBO. Here, both the prior and data-dependent component are scaled by 1, in contrast to Equation 2) where only the data-dependent component is reweighted. As discussed in [Section 2.2 and further explored in appendix A this leads to different scale of the quadratic approximation, which may lend itself better for continual learning. This also results in a second way to recover y Online EWC by first optimizing the 3-ELBO with 3 7 then tempering by a factor of (i.c. increasing the temperature when 7 < 1).
E MAP DEGENERACY WITH FILM LAYERS
Here we describe how training FiLM layer with MAP training leads to degenerate values for the weights and scales, whereas with VI training, degeneracy occurs. For simplicity, consider only the nodes leading into single node and let there be of them, ie. has dimension d. Because we only have one node, our scale parameter 7 is single variable.
23

---

Published as a conference paper at ICLR 2021
For MAP training, we have the loss function L -p(( 2200, with D the dataset and A the 1.2 regularization hyperparameter. Note that p(D(0,7) p(D]c0, 7)), hence we can scale arbitrarily without affecting the likelihood, so long as y is scaled inversely. If < 1, 300 < ()0)² so increasing e decreases the L. penalty if 0 is inversely scaled by Therefore the optimal setting of the scale parameter is arbitrarily large, while shrinks to O.
At high level, VI-training (with Gaussian posteriors and priors) does not have this issue because the KL-divergence penalizes the variance of the parameters from deviating from the prior addi- tion to the mean parameters, whereas MAP training only penalizes the means. Unlike with MAP training. if we downscale the weights, we also downscale the value of the variances, which increases the KL-divergence. The variances cannot revert the prior either, when they are up-scaled by the FiLM scale parameter, the noise would increase, affecting the log-likelihood component of the ELBO. Therefore, there exists an optimal amount of scaling which balances the mean-squared penalty component of the KL-divergence and the variance terms.
Mathematically we can derive this optimal scale. Consider the scenario with VI training with Gaus- sian variational distribution and prior, where our approximate posterior (0) has mean and variance j and and our prior p(0) has parameters do and Σg. First consider the scenario without FiLM Lay- ers. Now, have our loss function L = - E p( + Dx (()))))()))) For multivariate Gaussians,
1 DKr (10)))))))) (log |Eo| - log [Σ] -d d TT(EET)) + (n- -o0 -o0 (1r - po))-
Now consider another distribution (0), with mean and variance parameters ep and e²E. Now of (0) is paired with FiLM scale parameter ry set at e³ the log-likelihood component is unchanged:
B log p( D10) (D)0) ()) logr(DI0,> == -
with y being our FiLM scale parameter and p p(D 7) representing model with FiLM scale layers. Now consider the DxL (g'( (0))())))))))) (l))))) and optimize with and Σ fixed:
(KL(g(00)lP(()) = 2 (log Σol log -اممم - d Tr(Ep e²Σ) + (cp 220 (cp - Ho)
(log Epl log Σ 2d d Tr
+ (cH mo)² 1 Ho))
0DKL
d
CTTr Σ) + (c' pa) ) 2 /
de c=c
c
-d c
Σ) + 22 / 1b Eo
(Tr(E Σ)
1 1 c¹ 1/ Eo / d
1o En 4 + (p) T p)
Σ)
T p)
2(Tr(E) Σ) + ET p.)
Also note that c = 0 results in an infinitely-large KL-divergence, so there is a barrier at c = 0, i.c. optimized through gradient descent, should never change sign. Furthermore, note that
DKL. d dc2
+T
Σ) + 1 E 0.
So the KL-divergence is concave with respective c, c² is minimizer of DRL. and therefore D (g())((()pp())) 2 Dx (p()))le=cc
which implies the optimal value of the FiLM scale parameter y is 2. While formal data was collected, it was observed that the scale parameters do in fact reach very close to this optimal scale value after training.
24

---

Published as a conference paper at ICLR 2021
F CLUSTERING OF FILM PARAMETERS
Facescrub
Cifar100
Notmnist
SVHN
Trfflesigns
RashionMNiST
(a) Scales
(b) Shifts
(c) Shifts and Scales
Figure 12: T-SNE of FiLM layer parameters of 58 tasks coming from different domains. Shift and scale parameters from the same domain are more similar than those from different ones.
In this section, we test the interpretability of learned FiLM Parameters. Such clustering has been done in the past with FiLM parameters, as well node-wise uncertainty parameters. One would intuitively expect that tasks from similar domains would finds similar features salient, and thus share similar FiLM parameters. To test this hypothesis, took the 8 mixed vision task from (Sention 5.3 and split each task into multi 5-way classification tasks so that there were many tasks from similar domains. For example, CIFAR 100, which originally had 100 classes, became 20 5-way clasification tasks, Trafficsigns became 8 tasks (7 5-way and 8-way), and MNIST (2 5 5-way). Next, we trained the same architecture used Secciion 5.3 except trained all 58 resulting tasks. Joint training was chosen over continual learning to avoid artifacts which would arise from task ordering. Figure 12 shows that the results scale and shift parameters can be clustered and FiLM parameters which arise from the same base task cluster together. Like in Achille et al. (2019), this likely could be used as a means knowing which tasks to learn continually and which tasks to separate (i.e. tasks from the same cluster would likely benefit from joint training, while tasks from different ones should be separately trained), however we did not explore this idea further.
G How FILM LAYERS INTERACT WITH PRUNING
Unprunedi
Uipprundd Node
Pruned Node
(a) Weights, no FiLM layers
(b) Biases, no FiLM layers
(c) Weights, with FiLM layers
(d) Biases, with FiLM layers
Figure 13: Posterior distributions for incoming weights (left) or biases (right) for node in the first layer. Nodes are either unrpruned (left within a column) or pruned (right within a column). Without FiLM Layers (top row), we see that pruned nodes have their bias concentrated at negative value, preventing future tasks from reactivating the node. With FiLM Layers, pruned node prunes using the FiLM parameters rather than the shared ones, allowing the posteriors to revert to the prior distribution, allowing for node reactivation.
we discussed the problem of pruning in variational continual learning and how prevents nodes from becoming reactivated. To reiterate, pruning broadly occurs in three steps:
In Section 3
1. Weights incoming node begin to revert to the prior distribution
2. Noise from these high-variance weights affect the likelihood term in the ELBO
25

---

Published as a conference paper at ICLR 2021
3. To prevent noise, the bias concentrates at negative value to be cut off by the ReLU acti- vation
Later tasks then are initialized with this negative bias with low variance, meaning that the node has a difficult time reactivating the node without incurring a high prior cost. This results in the effect shown in Figure where after the first task, effectively is further exacerbated with larger values of B, where the pruning effect is stronger. Increasing worsens this well, as increasing the quadratic cost further prevents already low-variance negative biases from moving.
more nodes are reactivated. The effect
We verify that this mechanism is indeed the cause of the limited capacity use by visualizing the posteriors for weights and biases entering a node in the first convolutional layer for a network trained on Easy-CHASY (Figure 13). Here, we see that biases in pruned nodes when there are no FiLM Layers do indeed concentrate at negative values. In contrast, biases in models with FiLM layers are able to revert to their prior because the FiLM parameters perform pruning.
H RELATED WORK
Regularization- based continual learning, Many algorithms attempt to regularize network param- eters based on a metric of importance. The most directly comparable algorithms to GVCL are EWC |Kirkpatrck et al. 2017), Online EWC (Schwarz et al. 2018). and VCL (Nguyen et al. 2018). EWC measures importance based on the Fisher information matrix, while VCL uses an approxi- mate posterior covariance matrix an importance measure. Online EWC slightly modifies EWC so that there is only a single regularizer based on the cumulative sum of Fisher information matrices. Lee et al. (2017) proposed IMM, which is extension to EWC which merges posteriors based on their Fisher information matrices. Ritter et al. 2018) and Yin et al. (2020) both aim to approximate the Hessian by using either coonecker-factoree or low-rank forms, using the Laplace approximation to form approximate posteriors of parameters. These methods all use second-orde approximations of the AAh et al. (2019), like us, use regularizers based on the ELBO, but also measure impor- tance on a per-node basis than a per-weight one. SI (Zenee et al. (017) measures importance using "Synaptic Saliency," as opposed to methods based on approximate curvature.
Architectural approaches to continual and meta-learning. This family of methods modifies the standard neural architecture by either adding parallel or series components to the network. Pro- gressive Neural Networks adds parallel column network for every task. Pathnet (Fernando et al.) 2017) can be interpreted as a parallel-network based algorithm, but rather than growing model size over time, the model size remains fixed while paths between layer columns are optimized. FiLM parameters can be interpreted as adding series components to network, and has been a mainstay in the multitask and meta-learning literature. Requeima et al. (2019) use hypernetworks to amortize FiLM parameter learning, and has been shown to be capable of continual learning. Architectural approaches are often used in tandem with regularization based approaches, such in HAT (Serra let al, 2018)) which uses per-task gating parameters alongside a compression- based regularizer, Adel al. 2020 propose CLAW, which also uses variational inference alongside per-task parameters, but requires a more complex meta-learning based training procedure involving multiple splits of the dataset. GVCL with FiLM layers adds to this list of hybrid aechiteturrll-rguularzatioo based approaches.
likelihood-tempering (or KL-reweighting) has been empirically found to improve performance when using variational infer- ence for Bayesian Neural Networks over a wide number of contexts and papers (Osawa al. 2019 Zhang et al. 2018). Cold posteriors are closely related to likelihood tempering, except they temper the full posterior rather than only the likelihood term, and often empirically outperform Bayesian posteriors when using MCMC sampling Wenzel et al. (2020). From an information theorrtic per- spective, KL-reweighted ELBOs have also studied as compression (Achille et al.) (020). |Achille| et al. (2019). like us, considers a limiting case of B, and uses this to measure parameter saliency, but use this information to create a task embedding rather than for continual learning. Outside of the Bayesian Neural Network context, values of 3 > have also been explored (Higgins et al.) 2017). and more generally different values of 3 trace out different points on a rate-distortion curve for VAES (Almi al. 2018).
Cold Posteriors and likelihood -emprorii-teprring.... As mentioned in /ecciion 2
26

---

Published as a conference paper at ICLR 2021
1 EXPERIMENT DETAILS
1.1 REPORTED METRICS
All reported scores and figures present the mean and standard deviation across 5 runs of the algo- rithm with different network initialization. For Easy-CHASY and HarS- train/test splits are also varied across iterations. For the Mixed Vision tasks, task permutation the tasks is also randomized between iterations.
Let the matrix Ra, represent the performance of jth task after the model was trained on the ith task. Furthermore, let Rijd be the mean performance of the jth for network trained only on that task and let the total number of tasks be T', FollowinggLopez-Paee Lopez-Paz & Ranzato (2017) and Pan et al. (2020). we define
I
Average Accuracy (ACC) = T
Σ
RT.
j=1
T
Forward Transfer (FWT)
Σ
Rij
T
j=1
T
Backward Transfer (BWT)
Σ
RT.j
T
j=1
Note that these metrics not exactly the same as those presented in all other works, the FWT and BWT metrics are summed over the indices J S T, whereas Lopez-Paz & Ranzato (2017)
and Pan et al. (2020) sum from 2 S j T and 1 S j < T 1 for FWT and BWT, respectively. For FWT, this definition does not assumes that R1, = Rind and affects algorithms such as HAT and Progressive Neural Networks, which either compress the model, resulting in lower accuracy, use smaller architecture for the first task. The modified BWT transfer is equal to the other BWT metries apart from constant factor T-1
Intuitively, forward transfer equates to how much continual learning has benefited a task when a task is newly learned, while backwards transfer the accuracy drop as the network learns more tasks compared to when task was first learned. Furthermore, in the tables in in[Appendix JJ we also present net performance gain (NET), which quantifies the total gain over separate training, at the end of training continually:
1 T Σ
NET FWT + BWT
R Rind
Note that for computation of Rindd we compare to models trained under the same paradigm, i.e. MAP algorithms (all baselines except for VCL) are compared to a MAP trained model, and VI algorithms (GVCL-F, GVCL and VCL) are compared KL-reweighted VI models. This does not make a difference for most of the benchmarks where Rind Rij However, for Easy and Hard- CHASY, Rind < Rind so we compare VI to VI and MAP to MAP to obtain fair metrics.
In Figure 5b we plot AACC, which we define
AACC Σ 1=1
Rag RT
This metric is useful when the tasks have very different accuracies and their permutation is random- ized, as is the case with the mixed vision tasks. Note that this means that R3 would refer to a different task for each permutation, but we average over the 5 permutations of the runs. Empirically,
27

---

Published as a conference paper at ICLR 2021
if two algorithms have similar final accuracies, this metric measures how much the network forgets about the first tasks from that point to the end, and also measures how high the accuracy would have been if training was terminated after tasks. Plotting this also captures the concept of graceful vs catastrophic forgetring, as graceful forgetting would show up as a smooth downward curve, while catastrophic forgetting would have sudden drops.
1.2 OPTIMIZER AND TRAINING DETAILS
The implementation of all baseline methods was based on the Github repository for HAT (Serra al., 2018)) except the implementions of IMM-Mode and EWC were modified due an error in the computation of the Fisher Information Matrix in the original implementation. Baseline MAP algorithms were trained with SGD with a decaying learning starting at Se-2 with a maximum epochs of 200 per task for the Split-MNIST, Split-CIFAR and the mixed vision benchmarks. The number of maximum epochs for Easy- Easy-CHASY and Hard-CH CHASY was 1000, due to the small dataset size. Early stopping based on the validation set was used. 10% of the training set was used validation for these methods, and for Easy and Hard CHASY, 8 samples per class form the validation set (which disjoint from the training samples test samples).
For VI models, we used Adam optimizer with a learning rate of le-4 for Split-MNIST and Mixture, and 1e-3 for Easy-CHASY, Hard-CHASY and Split- CIFAR. We briefly tested running the baselines algorithms using Adam rather than SGD and performance did not change. Easy-CHASY and Hard- CHASY were run for 1500 epochs per task, Split-MNIST for 100, Split-CIFAR for 60, and 180 for Mixture. The number of epochs was changed so that the number of gradient steps for each task was roughly equal. For Easy-CHASY, Hard-CHASY and Split-CIFAR, this means that later tasks are run for more epochs, since the largest training sets are at the start. For Mixture, we ran 180 equivalents epochs for Facescrub. For how many epochs this equates to the other datasets, we refer the reader to Appendix A in (Serra et al. (218). We did not use early stopping for these VI results. While we understand that in some cases we trained for many more epochs than the baselines, the baselines used early stopping and therefore all stopped long before the 200 epoch limit was reached, so allocating more time would not change their results. [Swaroop et al.] (2019) also finds that allowing VI to converge is crucial for continual learning performance. We leave the discussion of improving this convergence time for future work.
All experiments (both the baselines and VI methods) batch size of 64.
1.3 ARCHITECTURAL DETAILS
Easy and Hard CHASY. We use convolutional architecture with 2 convolutions layers with:
3x3 convolutional layer with filters, padding of 1, ReLU activations
2. 2x2 Max Pooling with stride
3. 3x3 convolutional layer with 32 filters, padding of 1, ReLU activations
4. 2x2 Max Pooling with stride
5. Flattening layer
6. Fully connected layer with 100 units and ReLU activations
7. Task-specific head layers
MPISS-MNIST. We use a standard MLP with:
1. Fully connected layer with 256 units and ReLU activations
2. Fully connected layer with 256 units and ReLU activations
3. Task-specific head layers
Split-CIFAR. We use the same architecture from Zenke et al. (2017):
1. 3x3 convolutional layer with 32 filters, padding of 1, ReLU activations
'Repository at htthth:t////gt hhtp:://////h.....//e///e thub. jomjjo/sj///oj/o/ttttt
28

---

Published as a conference paper at ICLR 2021
2. 3x3 convolutional layer with 32 filters, padding of 1, ReLU activations
3. 2x2 Max Pooling with stride
4. 3x3 convolutional layer with 64 filters, padding of 1, ReLU activations
5. 3x3 convolutional layer with 64 filters, padding of ReLU activations
6. 2x2 Max Pooling with stride
7. Flattening
8. Fully connected layer with 512 units and ReLU activations
9. Tpek-specific head layers
Mixed vision tasks. We the same AlexNet architecture from Serra al. (0018)
4x4 convolutional layer with 64 filters, padding of ReLU activations
2. 2x2 Max Pooling with stride
3. 3x3 convolutional layer with 128 filters, padding of 0, ReLU activations
4. 2x2 Max Pooling with stride
5. 2x2 convolutional layer with 256 filters, padding of O, ReLU activations
6. 2x2 Max Pooling with stride
7. Flattening
8. Fully connected layer with 2048 units and ReLU activations
9. Fully connected layer with 2048 units and ReLU activations
10. Task-specific head layers
For MAP models, dropout layers with probabilities of either 0.2 or 0.5 were added after convolu- tional or fully-connected layers. For GVCL-F, FiLM layers were inserted after convolutional/hidden layers, but before ReLU activations.
1.4 HYPERPARAMETER SELECTION
For all algorithms on Easy-CHAse CHASY, Hard-CHASY, Split-MNIST and Split-CIFAR, hyperparameter selection was done by selecting the combination which produced the best average accuracy on the first 3 tasks. The algorithms were then run on the full number of tasks. For the Mixed Vision tasks, the best hyperparameters for the baselines were taken from the HAT Github repository. For GVCL, we performed hyperparameter selection in the same way as in Serra et al. (2018): we found the best hyperparameters for the average performance that in the mixture tasks, we randomly permute the task order for each iteration (with permutations kept consistent between algorithms), whereas for the other benchmarks, the task order is fixed. Hyperparameter searches were performed using a grid search. The best selected hyperparameters shown in Table 3
the first random permutation of tasks. Note
29

---

Published as a conference paper at ICLR 2021
Algorithm
Hyperparameter
Easy-CHASY
Hard-CHASY
Spli-MNIST
Splii-CIFAR
Mixed Vision
B
0.05
0.05
0.2
GVCL-F
À
10
10
100
100
50
0.2
GVCL
100
100
1000
100
0.025
0.75
HAT
50
50
400
PathNet
# of cvolutions
200
100
20
VCL
None
Online EWC
A
100
500
10000
100
Progressive
None
IMM Mean
A
0.0005
1e.6
Se.4
le44
0
IMM-Mode
À 10-7
0.1
0.1
10-5
LWF
T
2
4
1
Best by perparameteess taken from HAT code
Table 3: Best (selected) hyperparameters for continual learning experiments for various algorithms. We fix Online EWC's
For the Joint and Separate VI baselines, used the same 8. For the mixed vision tasks, we had to used a prior variance of 0.01 (for both VCI, GVCL and GVCL-F), but for all other tasks we did not need tune this.
J FURTHER EXPERIMENTAL RESULTS
In following section present more quantitative results of the various baselines benchmarks. For brevity, in the main text, we only included the best performing baselines and those which are most comparable GVCL, which consisted of HAT, PathNet, Online EWC and VCL.
30

---

Published as a conference paper at ICLR 2021
J.1 EASY-CHASY ADDITIONAL RESULTS
Metric
ACC (%)
BWT (%)
FWT (%)
NET (%)
GVCL-F
90.9 0.3
0.2 0.1
0,4 0.3
0.6 + 0.3
GVCL.
88.9 + 0.6
-088
0.5
-1.4 0.6
НАТ
2.99
0.6
0.4
0.9
PathNet
0.9
0.0
0.9
0.9
VCL
1.0
-..1
+.8
1.1.9
VCL-F
19.
0.91
-.3
1.0
Online EWC
73,4
2.9
0.5
3.4.5
Online EWC-F
1.5
1.6
-.3
1.5
Progressive
8.6
0.0
0.3
0.6
IMM-mean
4.0
0.6
10.
1.0
imm-mode
1.0
-11.2
2.1
1.0
LWF
75.1
-12.9
4.1
-8.8
SGD
75.3 118
-11.1
2.5
1..6
SGD-Frozen
$1.2
0.0
0.8
0.8
Separate (MAP)
88.4 + 0.8
0.0
Separate (B-VI)
90.3 - 0.1
0.0 1.0 0.0
Joint (MAP) Joint (B-VI+ FiLM)
88.6 + 0.7
4.7 + 0.7
91.9 + 0.1
1.6 + 0.1
Table Performance metrics of GVCL-F, GVCL and various baseline algorithms Easy-CHASY. Separate and joint training results for both MAP and 3-VI models are also presented
Easy-CHASY performance on Individual tasks (all approaches)
90
Approaches (Ranked)
GVCL-F
80
GVCL
HAT
Progressive
70
PathNet
SGD Frozen
VCL-F
Accuracy [%)
VCL
Online EWC-F
SGD
LWF
IMM Mode
Online EWC
IMM-Mean
(B-VI, FILM)
Separate
(B-VI)
Joint (MAP)
Separite (MAP)
Task
Figure 14: Mean accuracy of individual tasks after training for all approaches on Easy-CHASY
31

---

Published as a conference paper at ICLR 2021
Easy-CHASY performance individual tasks (lop approaches)
95
Approaches
(Ranked)
90
GVCL-F
GVCL
85
HAT
Progressive
Accuracy
PathNet
Joint
(B-VI, FiLM)
75
Separate
(B-VI)
Joint (MAP)
Separate (MAP)
Task
Figure 15: Mean accuracy of individual tasks after training for the top 5 performing approaches Easy-CHASY
Easy-CHASY running average accuracy (all approaches)
90
Approaches (Ranked)
GVCL-F
80
GVCL
HAT
Average accuracy (%))
Progressive
PathNet
SGD-Frozen
70
VCL-F
VCL
Online EWC-F
SGD
60
LWF
IMM Mode
Online EWC
IMM-Mean
Joint (MAP)
50
Joint (8-VI)
40
Number tasks
Figure 16: Running average accuracy of individual tasks after training for the all approaches on Fasy-CHASY
32

---

Published as a conference paper at ICLR 2021
Easy-CHASY running average accuracy (top approaches)
90
Approaches (Ranked)
Average accuracy (%)
85
GVCL-F
GVCL
80
HAT
Progressive
PathNet
75
Joint (MAP)
Joint (B-VI)
70
2
8
10
Number of tasks
Figure 17: Running average accuracy of individual tasks after training for the top 5 approaches on Easy-CHASY
33

---

Published as a conference paper at ICLR 2021
J.2 HARD-CHASY ADDITIONAL RESULTS
Metric
ACC (%)
BWT (%)
FWT (%)
NET (%)
GVCL-F
69.5 0.6
-0,1 0.1
-16 0.7
1.7 0.6
GVCL.
64.4 + 0.6
-0.6 0.6
-6.3 1.3+00 0.6
668
НАТ
5.
0.
3357
-,
PathNet
0.8
0.0
-.2
0.8
VCL
4.4
-1..9
-13.5
2..4
VCL-F
6.8
-.87
-3.4
-..1
Online EWC
1.74
1.71
1.3
1.7
Online EWC-F
56.7
-.9
-.4
6.
Progressive
1.
0.0
1.6
1.
IMM-mean
3.8
-.0
3..
0.8
imm-mode
4.3
-22.2
1..5
4.3
LWF
2..
23.0
2.+
2.5
SGD
47.1 2.2
2.7
1.2
+.2
SGD-Frozen
61.6 1.4
0.0
1.4
5.
Separate (MAP)
54.1 +
0.0 0.0
Separate (B-VI)
71.2 + 0.5
0.0 0.0
Joint (MAP) Joint (B-VI+ FiLM)
66.4 + 0.6
0.6 + 0.6
70.4 + 0.8
0.8 + 0.8
Table Performance metrics of GVCL-F, GVCL and various baseline algorithms on Hard-CHASY. Separate and joint training results for both MAP and 3- VI models are also presented
Hard-CHASY performance on Individual tasks (all approaches)
Approaches (Ranked)
GVCL-F
Progressive
VCL-F
PathNett
GVCL
HAT
SGD-Frozen
Accuracy 1%6
Online EWC-F
Online EWC
SGD
LWF
80
VCL
IMM-Mode
IMM-Mean
30
(B-VI, FILM)
Separate
(B-VI)
20
Joint (MAP)
Separite (MAP)
10
Task
Figure 18: Mean accuracy of individual tasks after training for all approaches on Hard-CHASY
34

---

Published as a conference paper at ICLR 2021
Hard-CHASY performance individual tasks (top approaches)
85
Approaches (Ranked)
80
GVCL-F
75
Progressive
VCL-F
PathNet
Accuracy
65
Joint
60
(B-VI, FiLM)
Separate
(B-VI)
Joint (MAP)
50
Separate (MAP)
Task
Figure 19: Mean accuracy of individual tasks after training for the top 5 performing approaches
Hard-CHASY
Hard-CHASY running average accuracy (all approaches)
10
Approaches (Ranked)
65
GVCL-F
Progressive
VCL-F
Average accuracy (%))
60
PathNet
GVCL
HAT
55
SGD-Frozen
Online EWC-F
Online EWC
SGD
50
LWF
VCL
IMM-Mode
45
Mean
Joint (MAP)
Joint (8-VI)
40
35
Number tasks
Figure 20: Running average accuracy of individual tasks after training for the all approaches on Hard-CHASY
35

---

Published as a conference paper at ICLR 2021
CHASY running average accuracy approaches)
72.5
Average accuracy (%)
70.0
Approaches (Ranked)
67.5
GVCL-F
Progressive
VCL-F
62.5
PathNet
60.0
GVCL
Joint (MAP)
57.5
Joint (8-VI)
55.0
52.5
Number of tasks
Figure 21: Running average accuracy of individual tasks after training for the top 5 approaches on Hard-CHASY
36

---

Published as a conference paper at ICLR 2021
J.3 Split-MNIST ADDITIONAL RESULTS
Metric
ACC (%)
BWT (%)
FWT (%)
NET (%)
GVCL-F
98.6 + 0.1 9.7
0.0
-0.1
-..0 + 0.1
GVCL
0.7
0.0
4.1
HAT
9.1
0.0
0.1
0.1
PathNet
9.
0.0
1.
1.
VCL
1.2
-5.1111
-08
1.2
VCL-F
94
-39
-..6
-3.9
Online EWC
9.
-3.8
-0.
1.6
Online EWC-F
94.1
-..
0.3
-.440.7
Progressive
9..
0.0
-.2
0.0
IMM-mean
90.
0.5
+.2
1.1
imm-mode
9,2
-.7
-.5
-3.1
LWF
9.2
-1.1
-011
-.2
SGD
1.7
-2.
0.0
-.2.4
SGD-Frozen
91.7
0.0
-.9
0.2
Separate (MAP)
0.0
0.0 0.0
Separate (B-VI)
98.7 0.0
0.0
Joint (MAP)
98.7 + 0.0
0.1
Joint (3-VI+ VI + FiLM)
98.8 + 0.0
0.1 0.0
Table 6: Performance metrics of GVCL-R, GVCL and various baseline algorithms on Split-MNIST. Separate and joint training results for both MAP and 3-VI models are also presented
MNIST performance individual tasks (all approaches)
100
Approaches (Ranked)
90
GVCL-F
Progressive
HAT
LWF
IMM-MMdee
80
PatnNet
VCL-F
Accuracy (%)
GVCL
EWC-F
EWC
70
SGD-Fros
Joint (g-VI, FiLM)
50
Separate
(8-VI)
Joint (MAP)
Separate (MAP)
50
Task
Figure 22: Mean accuracy of individual tasks after training for approaches on Splüt-MNIS
37

---

Published as a conference paper at ICLR 2021
Split- performance on individual tasks (top approaches)
Approaches
(Ranked)
GVCL-F
Progressive
HAT
Accuracy
96
LWF
IMM-Mode
Joint
94
(B-VI, FiLM)
Separate
(B-VI)
92
Joint (MAP)
Separate (MAP)
90
Task
Figure 23: Mean accuracy of individual tasks after training for the top 5 performing approaches
Split-MNIST
Split-MNIST running average accuracy (all approaches)
Approaches (Ranked)
GVCL-F
Progressive
HAT
Average accuracy (%)
LWF
90
IMM-Mode
PathNet
VCL-F
GVCL
Online EWC-F
85
Online EWC
VCL
SGD- CGD-Frozen
IMM-Mean
SGD
80
Joint (MAP)
Joint (B-VI)
75
8
10
Number of tasks
Figure 24: Running average accuracy of individual tasks after training for the all approaches on Split-MNIST
38

---

Published as a conference paper at ICLR 2021
Split-MNISH running average accuracy (top approaches)
100
Average accuracy (%)
99
Approaches (Ranked)
GVCL-F
Progressive
HAT
LWF
97
IMM-Mode
Joint (MAP)
Joint (B-VI)
Number tasks
Figure 25: Running average accuracy individual tasks after training for top 5 approaches on Split-MNIST
39

---

Published as a conference paper at ICLR 2021
J.4 Split- ADDITIONAL RESULTS
Metric
ACC (%)
BWT (%)
FWT (%)
NET (%)
GVCL-F
80.0 + 0.5
-0.3 0.2
8.8 0.5
8.5 + 0.5
GVCL
70.6 + 1.7
-
1.3
-1.0 1.7
HAT
77.3 1.3+00.
0.1
6.2
6.3
PathNet
0.8
0.0
0.9
0.8
VCL
14.2
12.
23.
27.4
VCL-F
56.
332
4.1
2.8
Online EWC
7..1
0.3
6.3
6.2
Online EWC-F
7..1
0.2
6.9
6.5
Progressive
7.8
0.0
0.1
0.1
IMM-mean
6.66
0.3
+.8
+.6
imm-mode
0.3
0.3
1..5
0.3
LWF
7.9
-..
1.2
3.2
SGD
7.
0..
10.6
4.1
SGD-Frozen
70.3
0.0
0.4
0.4
Separate (MAP)
70.6 + 0.6
0.0 0.0
Separate (B-VI)
71.6 0.2
0.0
Joint (MAP) Joint (B-VI+ VI + FiLM)
80.9 + 0.3
10.2 + 0.3
79.8 + 1.0
8.2 + 1.0
Table 7: Performance metrics of GVCL-F, GVCL and various baseline algorithms on Split-CIFAR. Separate and joint training results for both MAP and 3-VI models are also presented
Split-CIFAR performance individual tasks (all approaches)
Approaches (Ranked)
GVCL-F
Online
IMM-Mode
5GD
LWF
Accuracy (%)
Progreasive
RVCI
SGL-Frozen
PathNet
IMM-Mean
VCL-F
50
VCL
Joint
(B-VI, FiLM)
Separate
(B-VI)
40
Joint (MAP)
(MAP)
4
Awg
Task
Figure 26: Mean accuracy of individual tasks after training for all approaches on Split-CIFAR
40

---

Published as a conference paper at ICLR 2021
Split-CIFAR performance individual tasks (top approaches)
Approaches (Ranked)
02.5
GVCL-F
80.0
HAT
Online EWC-F
Accuracy
75.0
Online EWC
IMM Mode
72.5
Joint
(B-VI, FiLM)
70.0
Separate
(B-VI)
Joint (MAP)
65.0
Separate (MAP)
1
2
3
1
5
6
Avg
Task
Figure 27: Mean accuracy of individual tasks after training for the top 5 performing approaches on Split-CIFAR
Split-CIFAR running average accuracy (all approaches)
Approaches (Ranked)
GVCL-F
HAT
Online EWC-F
70
Average accuracy (%))
Online EWC
IMM-Mode
LWF
Progressive
GVCL
SGD-Frozen
PathNet
IMM Mean
VCL
50
Joint (MAP)
Joint (8-VI)
40
Number of tasks
Figure 28: Running average accuracy of individual tasks after training for the all approaches
Split-CIFAR
41

---

Published as a conference paper at ICLR 2021
Split-CIFAR running average accuracy (top 5 approaches)
82
Average accuracy (%)
Approaches (Ranked)
GVCL-F
80
HAT
Online EWC
IMM-Mode
78
Joint (MAP)
Joint (B-VI)
76
2
3
5
Number tasks
Figure 29: Running average accuracy of individual tasks after training for the top 5 approaches on Split-CIFAR
42

---

Published as a conference paper at ICLR 2021
J.5
MIXED VISION TASKS ADDITIONAL RESULTS
Metric
ACC (%)
BWT (%)
FWT (%)
NET (%)
GVCL-F
80.0 + 1.2
-0.9 + 1.3
-4.8 + 1.6
5.6 + 1.2
GVCL
49.0 2.8
13.1 1.6
23.5
+ 3.4
36.7 + 2.8
HAT
80.3 : 1.0
0.1
1.0
1.0
PathNet
0.0
2.0
+ 2.0
VCL
2.1
35.0
23.7
-8.8
VCL-F
2.0
18.
2.1
3.01
Online EWC
5.2
18.7
0.78
5.2
Online EWC-F
4.0
-13.8
0.5
1.5.7
Progressive
0.4
0.0
0.4
-.
IMM-mean
2.0
-.44
-.3
2.0
imm-mode
36.6
-9.1
10.5
18.7
LWF
2.3
55.
0.6
6..4
SGD
3.9
3.9
0.0
3.9
SGD-Frozen
52.9 3.9
0.0
3.9
33.3 = 3.9
Separate (MAP)
86.3 + 0.1
0
Separate (3-VI)
85.7 = 0.1
0.0 = 0.0
Joint (MAP) Joint (8-VI + FiLM)
84.3 + 0.1
2.0 + 0.1
83.8 + 0.2
1.8 + 0.2
Table 8: Performance metrics of GVCL-F, GVCL and various baseline algorithms Mixed Vision tasks. Separate and joint training results for both MAP and 3-VI models are also presented
Mixed vision tasks performance individual tasks (all approaches)
100
Approaches (Ranked)
HAT
GVCL-F
Progressive
PatnNet
Online EWC-F
Online EWC
60
VCL-F
Accuracy (%)
IMM--Mean
5GD-Frozen
GVCL
IMM--Mode
SGD
VCL
IWF
(B-VI, FiLM)
20
Separate
(B-VI)
Joint (MAP)
Separate (MAP)
Gifr10
Clta300
MNIST
SVHN
ReahoonMNi5T
Trafficsigns
Facescrub
NotMIIST
Avg
Task
Figure 30: Mean accuracy of individual tasks after training for all approaches on mixed vision tasks
43

---

Published as a conference paper at ICLR 2021
Mixed vision tasks performance individual tasks (top approaches)
100
Approaches
(Ranked)
HAT
GVCL-F
Accuracy (%)
Progressive
70
PathNet
Online EWC-F
Joint FiLM)
Separate
(B-VI)
Joint (MAP)
Separate (MAP)
40
Glar10o
CHar300
NNIST
SVHN
RashionMRIST
TrafficSigns
Facescrub
NatMiIST
Avg
Task
Figure 31: Mean accuracy of individual tasks after training for the top performing approaches on mixed vision tasks
CIFARIO
CIFARI00
F-MNIST
TrafficSigns
Facescrub
Average
GVCL-F
0.79%
0.01%
0.04%
0.73% 2.55%
0.25% 0.94%
0.10% 0.42%
0.11 5.05%
0.53% 3.88%
0.32%
HAT
0.12%
0.40%
0.13%
1.69%
Table 9: ECE of all 8 mixed vision tasks for model trained continually using GVCL-F HAT. F-MNIST stands for FashionMNIST.
44

---

Published as a conference paper at ICLR 2021
Claner
Cluter2
R
II
Clave
Claser 5
A
x
M
Clewe
2
E
%
8
Clucer
Clasor 10
Claner 11
2
Clanar
E
Custrr
F
v
Clava 15
R
Clacer 16
@ 6
8
Clars
9
Clater
Cluce 20
Jr
Figure 32: Clusters of symbols found by performing K-means clustering with K = 20 based on the embedding layer of model trained with variational inference on 200-way classification task on the 200 most common symbols in the HASYv2 dataset. Easy-CHASY is made by taking the first symbol from each cluster as the first task, then the second, and so on, up 10 tasks. Hard-CHASY is made by taking the clusters with the most classes in order (clusters 1-10).
K CLUSTERED HASYV2 (CHASY)
The HASYv2 dataset is a dataset consisting over 32x32 black/white handwritten Latex characters. There are a total of 369 classes, and over 150 000 total samples (Thoma 2017).
We constructed 10 classification tasks, each with varying number of classes ranging from 20 to 11. To construct these tasks, we first trained mean-field Bayesian neural network on 200-way classification task on the 200 classes with the most total samples. To get an embedding for each class, we use the activations of the second-last layer. Then, we performed K-means clustering with 20 clusters on the means of the embedding generated by each class when the samples of the classes were input into the network. Doing this yielded the classes shown in figure 32 Now, within each cluster are classes which are deemed "similar" by the network. then took classes from each cluster sequentially (in order of the class whose mean closest to the cluster's mean), so that each task contains at most 1 symbol from each cluster. Doing this ensures that tasks are similar to another, since each task consists of classes which different in similar ways. With the classes selected, the training set is made by selecting 16 samples of each classes, and using the remaining the test set. This procedure was used generate the "easy" set of tasks, which should have the maximum amount of similarity between tasks. We also constructed second of tasks, the "hard" set, in which each task is individually difficult. This selecting each task to be classification within each cluster, selecting clusters with the most number of symbols first. This corresponds to clusters 1-10 in figure 322 With the classes for each task selected, 16 samples from each class are used in the training set, and the remainder are used as the test set. Excess samples are discarded so that the test set class distribution also uniform within each task.
make the TO classification tasks,
done by
It was necessary to perform this clustering procedure as we found it difficult to produce sizable transfer gains if we simply constructed tasks by taking the classes with the most samples. While were able to have gains of up to 3% from joint training on 10 20-way classification tasks with the tasks chosen by class sample count, these gains were significantly diminished when performing opposed to MLE estimation, and reduced even further when performing VI. Because one of our benchmark continual learning methods is VCL, showing transfer when trained using VI is necessary.
MAP estimation
45

---

Published as a conference paper at ICLR 2021
MAP training
Number tasite trakied ponttyy
Roleriv accurary gain (W)
0
Number tasis trained joyytty
gain (9))
Average relative performance
(b) Individual task relative performances
Figure 33: Relative test-set accuracy models trained jointly on the easy set of tasks relative individual training for MAP estimation. Figure 33a shows the means aggregated over all tasks while figure the performance differences for individual tasks. Performance increases near monotonically as more tasks are added, achieving an average of around 4.7% gain with 10 tasks
02
8AN, 0.5
38
Savan www.ctaard paid
(a)
(c)
(b)
Figure 34: Relative performance models trained jointly on the easy set of tasks relative to indi- vidual training for variational inference with various KL-reweighting coefficients B. Performance gains reach around 2.0% with 10 tasks in the worst case, which less than with MAP training but still significant
Figures 33a and 34 show the performance gains of joint training over separate training on this new dataset, for both MAP, and KL-reweighted VI, respectively. Figure (33b shows how relative test accuracy varies for each specific task for these training procedures.
46


Reviews:
[{"review_id": "n2JbrFs8a8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work considers online variational Bayesian approaches to continual learning. The authors propose a beta-ELBO objective which they claim interpolates between Gaussian variational inference (beta = 1) and Laplace’s approximation (beta = 0).\nFurthermore, the authors propose task-specific, non-probabilistic (point estimation) FiLM layers that apply an element-wise transformation to the activations.\n\nTheory / Contribution:\nThe two contributions seem quite orthogonal to each other and each of them is rather minor in novelty. \nIt is obvious that using beta=0 leads to MAP estimates from which Laplace’s approximation can be computed. However, I am quite confused what exactly the authors do here and there could be a major mistake:\nFrom the paper, I am not sure if the authors a) compute Laplace’s approximation in the end, at the resulting mean of q, for any beta value? As far as I understand, the authors instead b) only optimise the variance through the beta-ELBO. \nHowever, in this case, the resulting approximation would *not* identical to Laplace’s approximation!\nI need clarification what the authors are doing here.\nConsider the case of beta=0, the covariance will be the dirac distribution as the authors note in Sec. 2.2 or the supplementary material. The authors then go on and write the optimal covariance matrix for which the derivative of the beta-ELBO is zero.\nYou have first postulated that the covariance is zero, in order to be able to pull out the expectation, and then you again allow for a non-zero beta-elbo-minimizing covariance. This would be a contraction. This makes me guess you do compute Laplace’s approximation instead. But then it is not discussed how you deal with beta>0. \n\n\nRelated work:\nThe related work section is rather short mentioning only very few related approaches. More effort is required here.\n\n\nExperiments:\nThe experimental evaluation is thorough and seems promising. Although I am wondering why e.g. Fig. 2 does not include VCL and EWC. Figure 8 in the supplementary material probably has some legends mixed up, or the explanations that small beta values cause locally measured locally are wrong? For Fig. a), the largest beta=10 seems to be a good approximation and also the most local. In case of Fig. B) and C) it is unclear / subjective (from visually inspecting the likelihood function) which is the best approximation. In A), beta=0.1 is the least local approximation, in B) beta=10 and in C) beta=1. I cannot follow the intuition provided here.\n\n\nSummary:\nI am sceptical about the correctness regarding the equivalence between VI and Laplace’s approximation; the exact approach proposed in the paper is unclear and may be based on a contradiction. In case I have a misunderstanding here, I hope the authors will point this out and update the manuscript. \n\n\nUpdate after Rebuttal:\nThe authors provided clarifications and improved the manuscript. \nIn particular, the authors now detail the two special cases (beta=0, beta=1) and how it relates to EWC and VCL. \nI am no longer sceptical that the claims regarding the equivalence to EWC in case of beta=0 is correct. \nBased on this, I changed my evaluation and now suggest acceptance. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "fgM2BnHqLp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Generalized Variational Continual Learning (GVCL). It is shown that Online EWC and VCL are special cases of GVCL, along with other theoretical contributions. Further, GVCL is augmented with FiLM to alleviate weaknesses of VCL and GVCL. GVCL and GVCL-F are applied to a number of continual learning tasks and demonstrate competitive performance. \n\nAlthough GVCL and GVCL-F do not outperform baselines, particularly in hard settings (split-mnist and mixed vision), GVCL is an original and excellent contribution. The paper is clear and well-written, the proposed algorithm is theoretically motivated and analysed, experiments are comprehensive, demonstrating the empirical performance of GVCL. \n\nI have the following comments:\n- It would be interesting to have VCL and Online EWC added to Figures 2 and 3.\n- Why is GVFL significantly worse than baselines for split-mnist (Figure 2c)?\n- Why is split-mnist omitted from Figure 3?\n- The supplementary material contains some analysis on the effect and sensitivity of the value of $\\beta$ on the performance of the algorithm. This should be extended and presented in the main paper.\n\nMinor:\n- \"the node is *effective* shut off\" -> effectively", "extra_scores": {}, "extra_text": {}}, {"review_id": "bHD9wQgOkEX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose Generalized VCL in this paper, which consists of multiple ideas: first, the authors introduce a beta-Elbo, which facilitates downweighting the KL-term of VCL. If beta taken to the limit towards zero, the authors show that the beta-elbo recovers the online EWC learning criterion, which draws an interesting link between VCL and EWC.\nThe authors also discuss reweighting terms to introduce a parameter lambda as in EWC, which they incorporate via a lambda-kl divergence term.\nFinally, furnished with this learning objective that interpolates between VCL and EWC, the authors propose to combine the learning objective with the architectural choice of Film layers, which they show facilitate overcoming the pruning behavior that their method inherits from VCL by offering ways to prune nodes without injecting noise into the network.\n\nExperiments are broad on multiple interesting datasets and quite clearly show that their proposed combined model performs best.\n\nPositives:\nThe paper draws an interesting unification between EWC and VCL, and in fact also other related works, as subtle modifications in a regularizer. This by itself is an interesting contribution. The fact that the authors study the interplay of their learning arlgorithm with architectural biases, i.e. overcoming early pruning via film layers, is also a valuable idea that I find not just interesting in itself, but also stylistically valuable as an approach to studying  deep learning. While the Film layers per se also appear somewhat ad hoc, their empirical benefits -particuarly when paired with the lambda-elbo, are impressive and well put together.\n\nCriticisms:\nWhile I really enjoy the derivation of the beta-elbo in the zero limit, I found the introduction of the reweighting terms in Sec. 2.3 to be ad hoc and not particularly well justified. It feels as if it is reverse engineered to match the desired criterion from EWC. I think the authors should dig deeper here for better justifications for such choices, as they did a good job having a mathematically interesting framework to derive earlier.\n\nAdditionally, the film layers work great, but I maybe missed if they are the main attraction powering performance or if it is the combination with the new ELBO. Would film layers with VCL do equally well? This is empirically confusing, it would be great to get some more help to understand the relative merits of each components here and clarify more how these pieces fit together empirically. I do enjoy the appendix discussing this qualitatively, but I would like to understand it quantitatively better, as theoretically film layers plus VCL (without this paper's innovations) should also benefit similarly.\n\nOne additional criticism is that the title is somewhat misleading, as it does not generalize VCL to broader settings, but rather collapses it towards the limit beta towards zero. The title raised hopes for a richer variational treatment rather than a unification to EWC and an architecture change. The authors might want to consider tweaking the title to sth that is closer to the paper's actual contributions.\n\n\nOverall:\nThis paper takes an interesting approach towards adding to the EWC and VCL literature by unifying them and offering an architectural fix for a key problem in these scenarios. While the contributions are mixed and not consistently derived from clear modeling assumptions, their interplay is well studied and highly relevant to the understanding and improvement of practical continual learning. I also want to again applaud the authors for studying and explaining the interplay of pruning and film layers, I enjoyed reading the supplementary information on this. I wish more papers that discover methods that perform well empirically would study the interplays of algorithm and architecture similarly to expose interesting effects.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tmnu6w2n6QB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a generalized variational continual learning (GVCL) framework using the \\beta - ELBO, and then combined with FiLM layers. The idea is interesting but there is a lack of preciseness. The pros and cons are as follows. \n\nPros: \n1. The proposed GVCL proposed a different and interesting perspective on the online EWC, viewed as a special case of \\beta \\to 0;\n2. FiLM layers are introduced to combine with GVCL, which lead to significant improvement in the performance;\n3. Various experiments are performed, showing some level of advantages. \n\nCons:\n1. The new perspective that online EWC could be viewed as a special case of the GVCL framework is lacking preciseness. First of all, as described in Sec. 2.3, the result of the \\beta-ELBO, even with \\beta \\to 0, does not lead to the key hyper parameter \\lambda in online EWC. To compensate this, the authors introduce a modified KL divergence to make them similar. However, it is not justified, from a unified Bayesian or some other theoretical perspective , why the previous \\beta-ELBO needs to be modified. It is kind of wired to start from the Bayesian  framework and then go back to the non-Bayesian perspective to design a Bayesian algorithm to improve the performance, and then claim that the previous non-Bayesian algorithm is a special case of the unified Bayesian framework. Moreover, as described in Sec. 2.3, the  resultant GVCL when \\beta \\to 0 is actually different from the previous online EWC algorithm. As a result, strictly speaking, it is not approperiate to claim that the online EWC could be recovered as a limiting case. \n\n2. If it is true that the proposed GVCL is a generalization of VCL and Online EWC, which allows interpolation between the two, then it is expected and reasonable that the GVCL alone (without additional FiLM layers) should perform at least the same as VCL and online EWC. Otherwise, the statement is not true and there is no advantage of the proposed GVCL framework . However, as shown in experimental results, e.g., Table 1, GVCL alone performs worse than Online EWC in large datasets, which is really wired. The authors also acknowledged this point and claimed that this is due to the difficulty in optimizing GVCL with small \\beta. It would be better to make such statement more precise because this is really important point for this paper. Otherwise, it implies that the so-called interpolation between VCL and online EWC has no additional advantage. \n\n3. Regarding the results of the GVCL and GVCL-F, it seems that the improvement mainly comes from the FiLM layers, rather than the GVCL framework itself.  To make this more clear and for a more fair comparison, it is highly suggested to compare other methods (online EWC, VCL, HAT, etc) with FiLM layers. Otherwise, the current improvement of the performance is unclear. In addition, the improvement of GVCL-F over the baseline is not consistent. \n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three of four reviewers are in favour of accepting the paper. Some reviewers raised valid criticism regarding the derivations, interpretation of the mathematical analysis and experimental results. So clearly some aspects of the paper could and should be clarified in accordance with the points raised by the reviewers. However, all in all the paper contains enough contributions to warrant publication.   
Decision: Accept (Poster)