Year: 2021

Abstract:
Continual learning deals with training models on new tasks and datasets in an online fashion. One strand of research has used probabilistic regularization for continual learning, with two of the main approaches in this vein being Online Elastic Weight Consolidation (Online EWC) and Variational Continual Learning (VCL). VCL employs variational inference, which in other settings has been improved empirically by applying likelihood-tempering. We show that applying this modification to VCL recovers Online EWC as a limiting case, allowing for interpolation between the two approaches. We term the general algorithm Generalized VCL (GVCL). In order to mitigate the observed overpruning effect of VI, we take inspiration from a common multi-task architecture, neural networks with task-specific FiLM layers, and find that this addition leads to significant performance gains, specifically for variational methods. In the small-data regime, GVCL strongly outperforms existing baselines. In larger datasets, GVCL with FiLM layers outperforms or is competitive with existing baselines in terms of accuracy, whilst also providing significantly better calibration.

heading: 1 INTRODUCTION
text: Continual learning methods enable learning when a set of tasks changes over time. This topic is of practical interest as many real-world applications require models to be regularly updated as new data is collected or new tasks arise. Standard machine learning models and training procedures fail in these settings (French, 1999), so bespoke architectures and fitting procedures are required.
This paper makes two main contributions to continual learning for neural networks. First, we develop a new regularization-based approach to continual learning. Regularization approaches adapt parameters to new tasks while keeping them close to settings that are appropriate for old tasks. Two popular approaches of this type are Variational Continual Learning (VCL) (Nguyen et al., 2018) and Online Elastic Weight Consolidation (Online EWC) (Kirkpatrick et al., 2017; Schwarz et al., 2018). The former is based on a variational approximation of a neural network’s posterior distribution over weights, while the latter uses Laplace’s approximation. In this paper, we propose Generalized Variational Continual Learning (GVCL) of which VCL and Online EWC are two special cases. Under this unified framework, we are able to combine the strengths of both approaches. GVCL is closely related to likelihood-tempered Variational Inference (VI), which has been found to improve performance in standard learning settings (Zhang et al., 2018; Osawa et al., 2019). We also see significant performance improvements in continual learning.
Our second contribution is to introduce an architectural modification to the neural network that combats the deleterious overpruning effect of VI (Trippe & Turner, 2018; Turner & Sahani, 2011). We analyze pruning in VCL and show how task-specific FiLM layers mitigate it. Combining this architectural change with GVCL results in a hybrid architectural-regularization based algorithm. This additional modification results in performance that exceeds or is within statistical error of strong baselines such as HAT (Serra et al., 2018) and PathNet (Fernando et al., 2017).
The paper is organized as follows. Section 2 outlines the derivation of GVCL, shows how it unifies many continual learning algorithms, and describes why it might be expected to perform better than them. Section 3 introduces FiLM layers, first from the perspective of multi-task learning, and then through the lens of variational over-pruning, showing how FiLM layers mitigate this pathology of VCL. Finally, in Section 5 we test GVCL and GVCL with FiLM layers on many standard bench-
marks, including ones with few samples, a regime that could benefit more from continual learning. We find that GVCL with FiLM layers outperforms existing baselines on a variety of metrics, including raw accuracy, forwards and backwards transfer, and calibration error. In Section 5.4 we show that FiLM layers provide a disproportionate improvement to variational methods, confirming our hypothesis in Section 31.

heading: 2 GENERALIZED VARIATIONAL CONTINUAL LEARNING
text: In this section, we introduce Generalized Variational Continual Learning (GVCL) as a likelihoodtempered version of VCL, with further details in Appendix C. We show how GVCL recovers Online EWC. We also discuss further links between GVCL and the Bayesian cold posterior in Appendix D.

heading: 2.1 LIKELIHOOD-TEMPERING IN VARIATIONAL CONTINUAL LEARNING
text: Variational Continual Learning (VCL). Bayes’ rule calculates a posterior distribution over model parameters θ based on a prior distribution p(θ) and some dataset DT = {XT , yT }. Bayes’ rule naturally supports online and continual learning by using the previous posterior p(θ|DT−1) as a new prior when seeing new data (Nguyen et al., 2018). Due to the intractability of Bayes’ rule in complicated models such as neural networks, approximations are employed, and VCL (Nguyen et al., 2018) uses one such approximation, Variational Inference (VI). This approximation is based on approximating the posterior p(θ|DT ) with a simpler distribution qT (θ), such as a Gaussian. This is achieved by optimizing the ELBO for the optimal qT (θ),
ELBOVCL = Eθ∼qT (θ)[log p(DT |θ)]−DKL(qT (θ)‖qT−1(θ)), (1)
where qT−1(θ) is the approximation to the previous task posterior. Intuitively, this refines a distribution over weight samples that balances good predictive performance (the first expected prediction accuracy term) while remaining close to the prior (the second KL-divergence regularization term).
Likelihood-tempered VCL. Optimizing the ELBO will recover the true posterior if the approximating family is sufficiently rich. However, the simple families used in practice typically lead to poor test-set performance. Practitioners have found that performance can be improved by downweighting the KL-divergence regularization term by a factor β, with 0 < β < 1. Examples of this are seen in Zhang et al. (2018) and Osawa et al. (2019), where the latter uses a “data augmentation factor” for down-weighting. In a similar vein, sampling from “cold posteriors” in SG-MCMC has also been shown to outperform the standard Bayes posterior, where the cold posterior is given by pT (θ|D) ∝ p(θ|D) 1 T , T < 1 (Wenzel et al., 2020). Values of β > 1 have also been used to improve the disentanglement variational autoencoder learned models (Higgins et al., 2017). We down-weight the KL-divergence term in VCL, optimizing the β-ELBO2,
β-ELBO = Eθ∼qT (θ)[log p(DT |θ)]− βDKL(qT (θ)‖qT−1(θ)).
VCL is trivially recovered when β = 1. We will now show that surprisingly as β → 0, we recover a special case of Online EWC. Then, by modifying the term further as required to recover the full version of Online EWC, we will arrive at our algorithm, Generalized VCL.

heading: 2.2 ONLINE EWC IS A SPECIAL CASE OF GVCL
text: We analyze the effect of KL-reweighting on VCL in the case where the approximating family is restricted to Gaussian distributions over θ. We will consider training all the tasks with a KLreweighting factor of β, and then take the limit β → 0, recovering Online EWC. Let the approximate posteriors at the previous and current tasks be denoted as qT−1(θ) = N (θ;µT−1,ΣT−1) and qT (θ) = N (θ;µT ,ΣT ) respectively, where we are learning {µT ,ΣT }. The optimal ΣT under the β-ELBO has the form (see Appendix C),
Σ−1T = 1
β ∇µT∇µTEqT (θ)[− log p(DT |θ)] + Σ −1 T−1. (2)
1Code is available at https://github.com/yolky/gvcl 2We slightly abuse notation by writing the likelihood as p(DT |θ) instead of p(yT |θ,XT ).
Now take the limit β → 0. From Equation 2, ΣT → 0, so qT (θ) becomes a delta function, and
Σ−1T = − 1 β ∇µT∇µT log p(DT |θ = µT ) + Σ−1T−1 = 1 β HT + Σ −1 T−1 = 1 β T∑ t=1 Ht + Σ −1 0 , (3)
where HT is the T th task Hessian3. Although the learnt distribution qT (θ) becomes a delta function (and not a full Gaussian distribution as in Laplace’s approximation), we will see that a cancellation of β factors in the β-ELBO will lead to the eventual equivalence between GVCL and Online EWC. Consider the terms in the β-ELBO that only involve µT :
β-ELBO = Eθ∼qT (θ)[log p(DT |θ)]− β 2 (µT − µT−1)>Σ−1T−1(µT − µT−1)
= log p(DT |θ = µT )− 1
2 (µT − µT−1)> ( T−1∑ t=1 Ht + βΣ −1 0 ) (µT − µT−1), (4)
where we have set the form of ΣT−1 to be as in Equation 3. Equation 4 is an instance of the objective function used by a number of continual learning methods, most notably Online EWC4 (Kirkpatrick et al., 2017; Schwarz et al., 2018), Online-Structured Laplace (Ritter et al., 2018), and SOLA (Yin et al., 2020). These algorithms can be recovered by changing the approximate posterior class Q to Gaussians with diagonal, block-diagonal Kronecker-factored covariance matrices, and low-rank precision matrices, respectively (see Appendices C.4 and C.5).
Based on this analysis, we see that β can be seen as interpolating between VCL, with β = 1, and continual learning algorithms which use point-wise approximations of curvature as β → 0. In Appendix A we explore how β controls the scale of the quadratic curvature approximation, verifying with experiments on a toy dataset.. Small β values learn distributions with good local structure, while higher β values learn distributions with a more global structure. We explore this in more detail in Appendices A and B, where we show the convergence of GVCL to Online-EWC on a toy experiment.
Inference using GVCL. When performing inference with GVCL at test time, we use samples from the unmodified q(θ) distribution. This means that when β = 1, we recover the VCL predictive, and as β → 0, the posterior collapses as described earlier, meaning that the weight samples are effectively deterministic. This is in line with the inference procedure given by Online EWC and its variants. In practice, we use values of β = 0.05 − 0.2 in Section 5, meaning that some uncertainty is retained, but not all. We can increase the uncertainty at inference time by using an additional tempering step, which we describe, along with further generalizations in Appendix D.
2.3 REINTERPRETING λ AS COLD POSTERIOR REGULARIZATION
As described above, the β-ELBO recovers instances of a number of existing second-order continual learning algorithms including Online EWC as special cases. However, the correspondence does not recover a key hyperparameter λ used by these methods that up-weights the quadratic regularization term. Instead, our derivation produces an implicit value of λ = 1, i.e. equal weight between tasks of equal sample count. In practice it is found that algorithms such as Online EWC perform best when λ > 1, typically 10 − 1000. In this section, we view this λ hyperparameter as a form of cold posterior regularization.
In the previous section, we showed that β controls the length-scale over which we approximate the curvature of the posterior. However, the magnitude of the quadratic regularizer stays the same, because theO(β−1) precision matrix and the β coefficient in front of the KL-term cancel out. Taking inspiration from cold posteriors (Wenzel et al., 2020), which temper both the likelihood and the prior and improve accuracy with Bayesian neural networks, we suggest tempering the prior in GVCL.
Therefore, rather than measuring the KL divergence between the posterior and prior, qT and qT−1, respectively, we suggest regularizing towards tempered version of the prior, qλT−1. However, this
3The actual Hessian may not be positive semidefinite while Σ is, so here we refer to a positive semidefinite approximation of the Hessian.
4EWC uses the Fisher information, but our derivation results in the Hessian. The two matrices coincide when the model has near-zero training loss, as is often the case (Martens, 2020).
form of regularization has a problem: in continual learning, over the course of many tasks, old tasks will be increasingly (exponentially) tempered. In order to combat this, we also use the tempered version of the posterior in the KL divergence, qλT . This should allow us to gain benefits from tempering the prior while being stable over multiple tasks in continual learning.
As we now show, tempering in this way recovers the λ hyperparameter from algorithms such as Online EWC. Note that raising the distributions to the power λ is equivalent to tempering by τ = λ−1. For Gaussians, tempering a distribution by a temperature τ = λ−1 is the same as scaling the covariance by λ−1. We can therefore expand our new KL divergence,
DKL ( qλT ‖qλT−1 ) = 1
2
( (µT − µT−1)>λΣ−1T−1(µT − µT−1) + Tr(λΣ −1 T−1λ −1ΣT ) + log |ΣT−1|λ−d |ΣT |λ−d − d )
= 1 2 ( (µT − µT−1)>λΣ−1T−1(µT − µT−1) + Tr(Σ −1 T−1ΣT ) + log |ΣT−1| |ΣT | − d ) = DKLλ(qT ‖qT−1).
In the limit of β → 0, our λ coincides with Online EWC’s λ, if the tasks have the same number of samples. However, this form of λ has a slight problem: it increases the regularization strength of the initial prior Σ0 on the mean parameter update. We empirically found that this negatively affects performance. We therefore propose a different version of λ, which only up-weights the “datadependent” parts of ΣT−1, which can be viewed as likelihood tempering the previous task posterior, as opposed to tempering both the initial prior and likelihood components. This new version still converges to Online EWC as β → 0, since the O(1) prior becomes negligible compared to the O(β−1) Hessian terms. We define,
Σ̃−1T,λ := λ
β T∑ t=1 Ht + Σ −1 0 = λ(Σ −1 T − Σ −1 0 ) + Σ −1 0 .
In practice, it is necessary to clip negative values of Σ−1T −Σ −1 0 to keep Σ̃ −1 T,λ positive definite. This is only required because of errors during optimization. We then use a modified KL-divergence,
DKLλ̃(qT ‖qT−1) = 1 2 ( (µT − µT−1)>Σ̃−1T−1,λ(µT − µT−1) + Tr(Σ −1 T−1ΣT ) + log |ΣT−1| |ΣT | − d ) .
Note that in Online EWC, there is another parameter γ, that down-weights the previous Fisher matrices. As shown in Appendix C, we can introduce this hyperparameter by taking the KL divergence priors and posteriors at different temperatures: qλT−1 and q γλ T . However, we do not find that this approach improves performance. Combining everything, we have our objective for GVCL,
Eθ∼qT (θ)[log p(DT |θ)]− βDKLλ̃(qT (θ)‖qT−1(θ)).

heading: 3 FILM LAYERS FOR CONTINUAL LEARNING
text: The Generalized VCL algorithm proposed in Section 2 is applicable to any model. Here we discuss a multi-task neural network architecture that is especially well-suited to GVCL when the task ID is known at both training and inference time: neural networks with task-specific FiLM layers.

heading: 3.1 BACKGROUND TO FILM LAYERS
text: The most common architecture for continual learning is the multi-headed neural network. A shared set of body parameters act as the feature extractor. For every task, features are generated in the same way, before finally being passed to separate head networks for each task. This architecture does not allow for task-specific differentiation in the feature extractor, which is limiting (consider, for example, the different tasks of handwritten digit recognition and image recognition). FiLM layers (Perez et al., 2018) address this limitation by linearly modulating features for each specific task so that useful features can be amplified and inappropriate ones ignored. In fully-connected layers, the transformation is applied element-wise: for a hidden layer with width W and activation values hi, 1 ≤ i ≤ W , FiLM layers perform the transformation h′i = γihi + bi, before being passed on to the remainder of the network. For convolutional layers, transformations are applied filter-wise. Consider a layer withN filters of sizeK×K, resulting in activations hi,j,k, 1 ≤ i ≤ N, 1 ≤ j ≤W, 1 ≤ k ≤ H , where W and H are the dimensions of the resulting feature map. The transformation has the
form h′i,j,k = γi ∗ hi,j,k + bi. The number of required parameters scales with the number of filters, as opposed to the full activation dimension, making them computationally cheap and parameterefficient. FiLM layers have previously been shown to help with fine-tuning for transfer learning (Rebuffi et al., 2017), multi-task meta-learning (Requeima et al., 2019), and few-shot learning (Perez et al., 2018). In Appendix F, we show how FiLM layer parameters are interpretable, with similarities between FiLM layer parameters for similar tasks in a multi-task setup.

heading: 3.2 COMBINING GVCL AND FILM LAYERS
text: It is simple to apply GVCL to models which utilize FiLM layers. Since these layers are specific to each task they do not need a distributional treatment or regularization as was necessary to support continual learning of the shared parameters. Instead, point estimates are found by optimising the GVCL objective function. This has a well-defined optimum unlike joint MAP training when FiLM layers are added (see Appendix E for a discussion). We might expect an improved performance for continual learning by introducing task-specific FiLM layers as this results in a more suitable multi-task model. However, when combined with GVCL, there is an additional benefit.
When applied to multi-head networks, VCL tends to prune out large parts of the network (Trippe & Turner, 2018; Turner & Sahani, 2011) and GVCL inherits this behaviour. This occurs in the following way: First, weights entering a node revert to their prior distribution due to the KL-regularization term in the ELBO. These weights then add noise to the network, affecting the likelihood term of the ELBO. To avoid this, the bias concentrates at a negative value so that the ReLU activation effectively shuts off the node. In the single task setting, this is often relatively benign and can even facilitate compression (Louizos et al., 2017; Molchanov et al., 2017). However, in continual learning the effect is pathological: the bias remains negative due to its low variance, meaning that the node is effectively shut off from that point forward, preventing the node from re-activating. Ultimately, large sections of the network can be shut off after the first task and cannot be used for future tasks, which wastes network capacity (see Figure 1a).
In contrast, when using task-specific FiLM layers, pruning can be achieved by either setting the FiLM layer scale to 0 or the FiLM layer bias to be negative. Since there is no KL-penalty on these parameters, it is optimal to prune in this way. Critically, both the incoming weights and the bias of a pruned node can then return to the prior without adding noise to the network, meaning that the node can be re-activated in later tasks. The increase in the number of unpruned units can be seen in Figure 1b. In Appendix G we provide more evidence of this mechanism.

heading: 4 RELATED WORK
text: Regularization-based continual learning. Many algorithms attempt to regularize network parameters based on a metric of importance. Section 2 shows how some methods can be seen as special cases of GVCL. We now focus on other related methods. Lee et al. (2017) proposed IMM, which is an extension to EWC which merges posteriors based on their Fisher information matrices. Ahn et al.
(2019), like us, use regularizers based on the ELBO, but also measure importance on a per-node basis rather than a per-weight one. SI (Zenke et al., 2017) measures importance using “Synaptic Saliency,” as opposed to methods based on approximate curvature.
Architectural approaches to continual learning. This family of methods modifies the standard neural architecture by adding components to the network. Progressive Neural Networks (Rusu et al., 2016) adds a parallel column network for every task, growing the model size over time. PathNet (Fernando et al., 2017) fixes the model size while optimizing the paths between layer columns. Architectural approaches are often used in tandem with regularization based approaches, such as in HAT (Serra et al., 2018), which uses per-task gating parameters alongside a compression-based regularizer. Adel et al. (2020) propose CLAW, which also uses variational inference alongside pertask parameters, but requires a more complex meta-learning based training procedure involving multiple splits of the dataset. GVCL with FiLM layers adds to this list of hybrid architecturalregularization based approaches. See Appendix H for a more comprehensive related works section.

heading: 5 EXPERIMENTS
text: We run experiments in the small-data regime (Easy-CHASY and Hard-CHASY) (Section 5.1), on Split-MNIST (Section 5.1), on the larger Split CIFAR benchmark (Section 5.2), and on a much larger Mixed Vision benchmark consisting of 8 different image classification datasets (Section 5.3). In order to compare continual learning performance, we compare final average accuracy, forward transfer (the improvement on the current task as number of past tasks increases (Pan et al., 2020)) and backward transfer (the difference in accuracy between when a task is first trained and its accuracy after the final task (Lopez-Paz & Ranzato, 2017)). We compare to many baselines, but due to space constraints, only report the best-performing baselines in the main text. We also compare to two offline methods: an upper-bound “joint” version trained on all tasks jointly, and a lower-bound “separate” version with each task trained separately (no transfer). Further baseline results are in Appendix J. The combination of GVCL on task-specific FiLM layers (GVCL-F) outperforms baselines on the smaller-scale benchmarks and outperforms or performs within statistical error of baselines on the larger Mixed Vision benchmark. We also report calibration curves, showing that GVCL-F is well-calibrated. Full experimental protocol and hyperparameters are reported in Appendix I.
5.1 CHASY AND Split-MNIST
For our Split-MNIST experiment, in addition to the standard 5 binary classification tasks for SplitMNIST, we add 5 more binary classification tasks by taking characters from the KMNIST dataset (Clanuwat et al., 2018). For these experiments we used a 2-layer fully-connected network, as in common in continual learning literature (Nguyen et al., 2018; Zenke et al., 2017).
Figure 2 shows the raw accuracy results. As the CHASY datasets have very few samples per class (16 per class, resulting in the largest task having a training set of 320 samples), it is easy to overfit. This few-sample regime is a key practical use case for continual learning as it is essential to transfer information between tasks. In this regime, continual learning algorithms based on MAP-inference overfit, resulting in poor performance. As GVCL-F is based on a Bayesian framework, it is not as adversely affected by the low sample count, achieving 90.9% accuracy on Easy-CHASY compared to 82.6% of the best performing MAP-based CL algorithm, HAT. Hard-CHASY tells a similar story, 69.1% compared to PathNet’s 64.8%. Compared to the full joint training baselines, GVCLF achieves nearly the same accuracy (Figure 3). The gap between GVCL-F and GVCL is larger for Easy-CHASY than for Hard-CHASY, as the task-specific adaptation that FiLM layers provide is more beneficial when tasks require contrasting features, as in Hard-CHASY. With Split-MNIST, GVCL-F also reaches the same performance as joint training, however it is difficult to distinguish approaches on this benchmark as many achieve near maximal accuracy.
5.2 Split-CIFAR
The popular Split-CIFAR dataset, introduced in Zenke et al. (2017), has CIFAR10 as the first task, and then 5 tasks as disjoint 10-way classifications from the first 50 classes of CIFAR100, giving a
total of 6 tasks. We use the same architecture as in other papers (Zenke et al., 2017; Pan et al., 2020). Like with Easy-CHASY, jointly learning these tasks significantly outperforms networks separately trained on the tasks, indicating potential for forward and backward transfer in a continual learning algorithm. Results are in Figure 4. GVCL-F is able to achieve the same final accuracy as joint training with FiLM layers, achieving 80.0±0.5%, beating all baseline algorithms by at least 2%. This confirms that our algorithm performs well in larger settings as well as the previous smallerscale benchmarks, with minimal forgetting. While the backwards transfer metric for many of the best performing continual learning algorithms is near 0, GVCL-F has the highest forward transfer, achieving 8.5%.
GVCL consistently outperforms VCL, but unlike in the CHASY experiments, it does not outperform Online EWC. This also occurs in the Mixed Vision tasks considered next. Theoretically this should not happen, but GVCL’s hyperparameter search found β = 0.2 which is far from Online EWC. We believe this is because optimizing the GVCL cost for small β is more challenging (see Appendix B). However, since intermediate β settings result in more pruning, FiLM layers then bring significant improvement.

heading: 5.3 MIXED VISION TASKS
text: We finally test on a set of mixed vision datasets, as in Serra et al. (2018). This benchmark consists of 8 image classification datasets with 10-100 classes and a range of dataset sizes, with the order of tasks randomly permuted between different runs. We use the same AlexNet architecture as in Serra et al. (2018). Average accuracies of the 8 tasks after continual training are shown in Figure 5.
GVCL-F’s final accuracy matches that of HAT, with similar final performances of 80.0±1.2% and 80.3±1.0% for the two methods, respectively. Figure 5b shows the relative accuracy of the model after training on intermediate tasks compared to its final accuracy. A positive relative accuracy after t tasks means that the method performs better on the tasks seen so far than it does on the same tasks after seeing all 8 tasks (Appendix I contains a precise definition). HAT achieves its continual learning performance by compressing earlier tasks, hindering their performance in order to reserve capacity for later tasks. In contrast, GVCL-F attempts to maximize the performance for early tasks, but allows performance to gradually decay, as shown by the gradually decreasing relative accuracy in Figure 5b. While both strategies result in good final accuracy, one could argue that pre-compressing a network in anticipation of future tasks which may or may not arrive is an impractical real-world strategy, as the number of total tasks may be unknown a priori, and therefore one does not know how much to compress the network. The approach taken by GVCL-F is then more desirable, as it ensures good performance after any number of tasks, and frees capacity by “gracefully forgetting”.
Uncertainty calibration. As GVCL-F is based on a probabilistic framework, we expect it to have good uncertainty calibration compared to other baselines. We show this for the Mixed Vision tasks in Figure 6. Overall, the average Expected Calibration Error for GVCL-F (averaged over tasks) is 0.32%, compared to HAT’s 1.69%, with a better ECE on 7 of the 8 tasks. These results demonstrate that GVCL-F is generally significantly better calibrated than HAT, which can be extremely important in decision critical problems where networks must know when they are likely to be uncertain.

heading: 5.4 RELATIVE GAIN FROM ADDING FILM LAYERS
text: In Section 3, we suggested that adding FiLM layers to VCL in particular would result in the largest gains, since it addresses issues specific to VI, and that FiLM parameter values were automatically best allocated based on the prior. In Section 5.4, we compare the relative gain of adding FiLM layers to VI-based approaches and Online EWC. We omitted HAT, since it already has per-task gating mechanisms, so FiLM layers would be redundant. We see that the gains from adding FiLM layers to Online EWC are limited, averaging 2.2% compared to over 10% for both VCL and GVCL. This suggests that the strength of FiLM layers is primarily in how they interact with variational methods for continual learning. As described in Section 3, with VI we do not need any special algorithm to encourage pruning and how to allocate resources, as they are done automatically by VI. This contrasts HAT, where specific regularizers and gradient modifications are necessary to encourage the use of FiLM parameters.

heading: 6 CONCLUSIONS
text: We have developed a framework, GVCL, that generalizes Online EWC and VCL, and we combined it with task-specific FiLM layers to mitigate the effects of variational pruning. GVCL with FiLM layers outperforms strong baselines on a number of benchmarks, according to several metrics. Future research might combine GVCL with memory replay methods, or find ways to use FiLM layers when task ID information is unavailable.

heading: A LOCAL VS GLOBAL CURVATURE IN GVCL
text: In this section, we look at the effect of β on the approximation of local curvature found from optimizing the β-ELBO by analyzing its effect on a toy dataset. In doing so, we aim to provide intuition why different values of β might outperform β = 1. We start by looking at the equation of the fixed point of Σ.
Σ−1T = 1
β ∇µT∇µTEqT (θ)[− log p(DT |θ)] + Σ −1 T−1. (5)
We consider the T = 1 case. We can interpret this as roughly measuring the curvature of log p(DT |θ) at different samples of θ drawn from the distribution qT (θ). Based on this equation, we know Σ−1T increases as β decreases, so samples from qT (θ) are more localized, meaning that the curvature is measured closer to the mean, forming a local approximation of curvature. Conversely, if β is larger, Σ−1T broadens and the approximation of curvature is on a more global scale. For simplicity, we write∇µT∇µTEqT (θ)[− log p(DT |θ)] as H̃T . To test this explanation of β, we performed β-VI on a simple toy dataset.
We have a true data generative distribution X ∼ N (0, 1), and we sample 1000 points forming the dataset, D. Our model is a generative model withX ∼ N (f(θ), σ20 = 30), with θ being the model’s only parameter and f(θ) an arbitrary fixed function. With β-VI, we aim to approximate p(θ|D) with q(θ) = N (θ;µ, σ2) with a prior p(θ) = N (θ; 0, 1). We choose three different equations for f(θ):
1. f1(θ) = |θ|1.6 2. f2(θ) = 4 √ |θ|
3. f3(θ) = 3 √ (|θ| − 0.5)3 + 0.4
We visualize log p(D|θ) for each of these three functions in Figure 7. Here, we see that the data likelihoods have very distinct shapes. f1 results in a likelihood that is flat locally but curves further away from the origin. f2 is the opposite: there is a cusp at 0 then flattens out. f3 is a mix, where at a very small scale it has high curvature, then flattens, then curves again. Now, we perform β-VI to get µ and σ2, for β ∈ {0.1, 1, 10}. We then have values for σ2, which acts as Σ−1T in Equation 5. We want to extract H̃T −1 from these values, so we perform the operation σ̃2 = β1
σ2 −1 , which represents
our estimate of the curvature of log p(D|θ) at the mean. This operation also “cancels” the scaling effect of β. We then plot these approximate log-likelihood functions log p̃(D|θ) = N (θ;µ, σ̃2) in Figure 8.
From these figures, we see a clear trend: small values of β cause the approximate curvature to be measured locally while larger values cause it to be measured globally, confirming our hypothesis. Most striking is Figure 8c, where the curvature is not strictly increasing or decreasing further from the origin. Here, we see that the curvature first is high for β = 0.1, then flattens out for β = 1 then becomes high again for β = 10. Now imagine in continual learning our posterior for a parameter whose posterior looks like Figure 8a. Here, the parameter would be under-regularized with β = 1,
so the parameter will drift far away, significantly affecting performance. Equally, if the posterior was like Figure 8b, then values of β = 1 would cause the parameter to be over-regularized, limiting model capacity than in practice could be freed. In practice we found that β values of 0.05 − 0.2 worked the best. We leave finding better ways of quantifying the posterior’s variable curvature and ways of selecting appropriate values of β as future work.

heading: B CONVERGENCE TO ONLINE-EWC ON A TOY EXAMPLE
text: Here, we demonstrate convergence of GVCL to Online-EWC for small β. In this problem, we deal with 2d logistic regression on a toy dataset consisting of separated clusters. The clusters are shown in Figure 9. The first set of tasks is separating the red/blue clusters, then the second is the yellow/green clusters. Blue and green are the first class and red and yellow are the second. Or model is given by the equation
p(yi = 1|w, b, xi) = σ(w>xi + b)
Where xi are our datapoints and w and b are our parameters. yi = 1 means class 2 (and yi = 0 means class 1). x is 2-dimensional so we have a total of 3 parameters.
Next, we ran GVCL with decreasing values of β and compared the resulting values of w and b after the second task to solution generated by Online-EWC. For both cases, we set λ = 1. For our prior, we used the unit normal prior on both w and b, our approximating distribution was a fully factorized Gaussian. We ran this experiment for 5 random seeds (of the parameters, not the clusters) and plotted the results.
Figure 10 shows the result. Evidently, the values of the parameters approach those of Online-EWC as we decrease β, in line with our theory. However, it is worth noting that to get this convergent behaviour, we had to run this experiment for very long. For the lowest β value, it took 17 minutes to
converge compared to 1.7 for β = 1. A small learning rate of 1e-4 with 100000 iteration steps was necessary for the smallest β =1e-4. If the optimization process was run for shorter, or too large a learning rate was used, we would observe convergent behaviour for the first few values of β, but the smallest values of β would result in completely different values.
This shows that while in theory, for small β, GVCL should approach Online-EWC, it is extremely hard to achieve in practice. Given that it takes so long to achieve convergent behaviour on a model with 3 parameters, it is unsurprising that we were not able to achieve the same performance as Online-EWC for our neural networks, and explains why despite GVCL, in theory, encompassing Online-EWC, can sometimes perform worse.

heading: C FURTHER DETAILS ON RECOVERING ONLINE EWC
text: Here, we show the full derivation to recover Online EWC from GVCL, as β → 0. First, we expand the β-ELBO which for Gaussian priors and posteriors has the form:
β-ELBO = Eθ∼qT (θ)log p(DT |θ)− βDKL(qT (θ)||qT−1(θ))
= Eθ∼qT (θ)[log p(DT |θ)]− β
2
( log |ΣT−1| − log |ΣT | − d
+ Tr(Σ−1T−1ΣT ) + (µT − µT−1) >Σ−1T (µT − µT−1)
) ,
where qT (θ) is our approximate distribution with means and covariance µT and ΣT , and our prior distribution qT−1(θ) has mean and covariance µT−1 and ΣT−1. DT refers to the T th dataset and d the dimension of µ. Next, take derivatives wrt ΣT and set to 0:
∇ΣT β-ELBO = ∇ΣTEθ∼qT (θ)[log p(DT |θ)] + β 2 Σ−1T − β 2 Σ−1T−1 (6)
0 = 1
2 ∇µ∇µEqT (θ)[log p(DT |θ)] +
β 2 Σ−1T − β 2 Σ−1T−1 (7)
⇒ Σ−1T = 1
β ∇µT∇µTEqT (θ)[− log p(DT |θ)] + Σ −1 T−1. (8)
We move from Equation 6 to Equation 7 using Equation 19 in Opper & Archambeau (2008). From Equation 8, we see that as β → 0, the precision grows indefinitely, so qT (θ) approaches a delta function centered at its mean. We give a more precise explanation of this argument in Appendix C.1.
We have
Σ−1T = − 1 β ∇µT∇µT log p(DT |θ = µT ) + Σ−1T−1
Σ−1T = 1
β HT + Σ
−1 T−1, (9)
where HT is the Hessian of the T th dataset log-likelihood. This recursion of Σ−1T gives
Σ−1T = 1
β T∑ t=1 Ht + Σ −1 0 .
Now, optimizing the β-ELBO for µT (ignoring terms that do not depend on µT ):
β-ELBO = Eθ∼q(θ)[log p(D|θ)]− β 2 (µT − µT−1)>Σ−1T−1(µT − µT−1) (10)
= log p(D|θ = µT )− 1
2 (µT − µT−1)> ( T−1∑ t=1 Ht + βΣ −1 0 ) (µT − µT−1). (11)
Which is the exact optimization problem for Laplace Propagation (Smola et al., 2003). If we note that HT ≈ NTFT (Martens, 2020), where NT is the number of samples in the T th dataset and FT is the Fisher information matrix, we recover Online EWC with λ = 1 when N1 = N2 = ... = NT (with γ = 1).
C.1 CLARIFICATION OF THE DELTA-FUNCTION ARGUMENT
In C, we argued,
Σ−1T = 1
β ∇µT∇µTEqT (θ)[− log p(DT |θ)] + Σ −1 T−1
≈ 1 β HT + Σ −1 T−1
for small β. We argued that for small β, q(θ) collapsed to its mean and it is safe to treat the expectation as sampling only from the mean. In this section, we show that this argument is justified. Lemma 1. If q(θ) has mean and covariance parameters µ and Σ, and Σ−1 = 1β∇µ∇µEθ∼q(θ)[f(θ)] + C, C = O( 1 β ), then for small β, Σ
−1 ≈ 1βHµ + C, where Hµ is the Hessian of f(θ) evaluated at µ, assuming Hµ = O(1)
Proof. We first assume that f(θ) admits a Taylor expansion around µ. For notational purposes, we define,
Tk1,...,kn ∣∣∣ θ=µ = ∂f ∂θ(k1) . . . ∂θ(kn) ∣∣∣ θ=µ
For our notation, upper indices in brackets indicate vector components (not powers), and lower indices indicate covector components. Note that, Hµ,i,j = Ti,j ∣∣∣ θ=µ . 5
Then, a Taylor expansion centered at µ has the form
f(θ) = f(µ) + ∞∑ n=1 1 n! Tk1,...,kn ∣∣∣ θ=µ (θ − µ)(k1) . . . (θ − µ)(kn)
5In this case, the µ in Hµ,i,j refers to the Hessian evaluated at µ, while i, j refers to the indices
Where we use Einstein notation, so
Tk1,...,kn ∣∣∣ θ=µ (θ − µ)(k1) . . . (θ − µ)(kn) = D∑
k1,...,kn=1
Tk1,...,kn ∣∣∣ θ=µ (θ − µ)(k1) . . . (θ − µ)(kn)
(12)
With D the dimension of θ. To denote the central moments of q(θ), we define µ̃(k1,...,kn) := Eθ∼q(θ) [ (θ − µ)(k1) . . . (θ − µ)(kn) ] These moments can be computed using Isserlis’ theorem. Notably, for a Gaussian, if n is odd, µ̃(k1,...,kn) = 0
Now, we can compute our expectation as an infinite sum:
∇µ∇µEθ∼q(θ)[f(θ)] = ∇µ∇µEθ∼q(θ) [ f(µ) +
∞∑ n=1 1 n! Tk1,...,kn ∣∣∣ θ=µ (θ − µ)(k1) . . . (θ − µ)(kn) ]
= ∇µ∇µ [ f(µ) +
∞∑ n=1 1 n! Tk1,...,kn ∣∣∣ θ=µ µ̃(k1,...,kn) ]
= ∇µ∇µ [ f(µ) +
∞∑ n=1 1 2n! Tk1,...,k2n ∣∣∣ θ=µ µ̃(k1,...,k2n) ] (odd moments are 0)
= A for notational simplicity
We can look at individual components of A:
Ai,j = ∂ ∂µ(i) ∂ ∂µ(j)
[ f(µ) +
∞∑ n=1 1 2n! Tk1,...,k2n ∣∣∣ θ=µ µ̃(k1,...,k2n)
]
= Ti,j ∣∣∣ θ=µ + ∞∑ n=1 1 2n! Ti,j,k1,...,k2n ∣∣∣ θ=µ µ̃(k1,...,k2n)
Now we can insert this into our original equation.
Σ−1 = 1
β ∇µ∇µEθ∼q(θ)[f(θ)] + C
Σ−1 = 1
β A+ C
Σ−1i,j = 1
β Ai,j + Ci,j looking at individual indices
Σ−1i,j︸︷︷︸ O( 1β ) = 1 β
( Ti,j ∣∣∣ θ=µ︸ ︷︷ ︸
O(1)
+ ∞∑ n=1 1 2n! Ti,j,k1,...,k2n ∣∣∣ θ=µ
µ̃(k1,...,k2n)︸ ︷︷ ︸ O(β) ) + Ci,j︸︷︷︸ O( 1β )
Now we assumed thatHµ isO(1) (so Ti,j ∣∣∣ θ=µ is too), which means that Σ−1i,j must be at leastO( 1 β ). If Σ−1 = O( 1β ), then Σ = O(β). From Isserlis’ theorem, we know that µ̃ (k1,...,k2n) is composed of
the product of n elements of Σ, so µ̃(k1,...,k2n) = O(βn). Ti,j,k1,...,k2n ∣∣∣ θ=µ is constant with respect to β, so is O(1). Hence, the summation is O(β), which for small β is negligible compared to the O(1) term Ti,j ∣∣∣ θ=µ , so can therefore be ignored. Then, keeping only O( 1β ) terms,
O( 1β )︷︸︸︷ Σ−1i,j = 1
β
( O(1)︷ ︸︸ ︷ Ti,j ∣∣∣ θ=µ + O(β)︷ ︸︸ ︷ ∞∑ n=1 1 2n! Ti,j,k1,...,k2n ∣∣∣ θ=µ µ̃(k1,...,k2n) ) + O( 1β )︷︸︸︷ Ci,j
O( 1β )︷︸︸︷ Σ−1i,j = O( 1β )︷ ︸︸ ︷ 1
β Ti,j ∣∣∣ θ=µ +
O(1)︷ ︸︸ ︷ 1
β ( ∞∑ n=1 1 2n! Ti,j,k1,...,k2n ∣∣∣ θ=µ µ̃(k1,...,k2n) ) + O( 1β )︷︸︸︷ Ci,j
≈ 1 β Ti,j ∣∣∣ θ=µ + Ci,j
= 1
β Hµ,i,j + Ci,j
Σ−1 ≈ 1 β Hµ + C
C.2 CORRESPONDING GVCL’S λ AND ONLINE EWC’S λ
We use DKLλ̃ in place of DKL, with DKLλ̃ defined as
DKLλ̃(qT ‖qT−1) = 1
2
( (µT − µT−1)>Σ̃−1T−1,λµT − µT−1) + Tr(Σ −1 T−1ΣT )
+ log |ΣT−1| − d− log |ΣT | ) ,
with
Σ̃−1T,λ := λ
β T∑ t=1 Ht + Σ −1 0 = λ(Σ −1 T − Σ −1 0 ) + Σ −1 0 .
Now, the fixed point for ΣT is still given by Equation 9, but the β-ELBO for for terms involving µT has the form,
β-ELBO = Eθ∼q(θ)[log p(D|θ)]− β 2 (µT − µT−1)>Σ̃−1T−1,λ(µT − µT−1)
= log p(D|θ = µT )− 1
2 (µT − µT−1)>
( λ
T∑ t=1 Ht + βΣ −1 0
) (µT − µT−1),
which upweights the quadratic terms dependent on the data (and not the prior), similarly to λ in Online EWC.
C.3 RECOVERING γ FROM TEMPERING
In order to recover λ, we used the KL-divergence between tempered priors and posteriors qλT−1 and qλT . Recovering γ can be done using the same trick, except we temper the posterior to q γλ T :
DKL(q λ T ‖q γλ T−1) = 1 2 ( (µT − µT−1)>λΣ−1T−1(µT − µT−1)
+ Tr(γλΣ−1T−1λ −1ΣT ) + log |λ−1ΣT−1| |(γλ)−1ΣT | − d ) = 12 ( (µT − µT−1)>λΣ−1T−1(µT − µT−1) + γTr(Σ −1 T−1ΣT )− log |ΣT | ) + cons. = DKLλ,γ(qT ‖qT−1)
We can apply the same λ to λ̃ as before to get DKLλ̃,γ(qT ‖qT−1). Plugging this into the β-ELBO and solving yields the recursion for ΣT to be
Σ−1T = 1
β HT + γΣ
−1 T−1,
which is exactly that of Online EWC.
C.4 GVCL RECOVERS THE SAME APPROXIMATION OF FT AS ONLINE EWC
The earlier analysis dealt with full rank ΣT . In practice, however, ΣT is rarely full rank and we deal with approximations of ΣT . In this subsection, we consider diagonal ΣT , like Online EWC, which in practice uses a diagonal approximation of FT . The way Online EWC approximates this diagonal is by matching diagonal entries of FT . There are many ways of producing a diagonal approximation of a matrix, for example matching diagonals of the inverse matrix is also valid, depending on the metric we use. Here, we aim to show that that the diagonal approximation of ΣT that is produced when Q is the family of diagonal covariance Gaussians is the same as the way Online EWC approximates FT , that is, diagonals of Σ−1T,approx match diagonals of Σ −1 T,true, i.e. we match the diagonal precision entries, not the diagonal covariance entries.
Let ΣT,approx = diag(σ21 , σ 2 2 , ..., σ 2 d), with d the dimension of the matrix. Because we are performing VI, we are optimizing the forwards KL divergence, i.e. DKL(qapprox||qtrue). Therefore, ignoring terms that do not depend on ΣT,approx,
DKL(qapprox||qtrue) = 1
2 Tr(ΣT,approxΣ−1T,true)−
1 2 log |ΣT,approx|+ (constants wrt ΣT,approx)
= 1
2 d∑ i=1 (ΣT,approxΣ −1 T,true)i,i − 1 2 d∑ i=1 log σ2i
= 1
2 d∑ i=1 ( σ2i (Σ −1 T,true)i,i)− log σ 2 i ) .
Optimizing wrt σ2i :
∂DKL(qapprox||qtrue) ∂σ2i = 0 = 1 2
( (Σ−1T,true)i,i − 1
σ2i ) ⇒ σ2i = 1
(Σ−1T,true)i,i .
So we have that diagonals of Σ−1T,approx match diagonals of Σ −1 T,true.
C.5 GVCL RECOVERS THE SAME APPROXIMATION OF HT AS SOLA
SOLA approximates the Hessian with a rank-restricted matrix H̃ (Yin et al., 2020). We first consider a relaxation of this problem with full rank, then consider the limit when we reduce this relaxation.
Because we are concerned with limiting β → 0, it is sufficient to consider Σ−1true as H , the true Hessian. Because H is symmetric (and assuming it is positive-semi-definite), we can also write H as H = V DV > = ∑p i=1 λixix > i , with D, and V be the diagonal matrix of eigenvalues and a unitary matrix of eigenvectors, respectively. These eigenvalues and eigenvectors are λi and xi, respectively, and p the dimension of H .
For H̃ , we first consider full-rank matrix which becomes low-rank as δ → 0:
H̃ = k∑ i=1 λ̃ix̃ix̃ > i + p∑ j=k+1 δx̃j x̃ > j
This matrix has λ̃i, 1 ≤ i ≤ k as its first k eigenvalues and δ as its remaining. We also set x̃>i x̃i = 1 and x̃>i x̃j = 0, i 6= j. With KL minimization, we aim to minimize (up to a constant and scalar factor),
KL = Tr(ΣapproxΣ−1true)− log |Σapprox|
In our case, this is Equation 13, which we can further expand as,
KL = Tr(H̃−1H)− log |H̃−1| (13)
= Tr  k∑ i=1 1 λ̃i x̃ix̃ > i + p∑ j=k+1 1 δ x̃j x̃ > j H + k∑ i=1 log(λ̃i) + p∑ j=k+1 log δ (14)
= Tr ( k∑ i=1 1 λi x̃ix̃ > i H ) + Tr  p∑ j=k+1 1 δ x̃j x̃ > j H + k∑ i=1 log(λ̃i) + p∑ j=k+1 log δ (15)
= k∑ i=1 1 λi x̃>i Hx̃i + p∑ j=k+1 1 δ x̃>j Hx̃j + k∑ i=1 log(λ̃i) + p∑ j=k+1 log δ (16)
(17)
Taking derivatives wrt λ̃i, we have:
∂KL ∂λ̃i = 0 = − 1 λ̃2i x̃>i Hx̃i + 1 λ̃i (18)
⇒ λ̃i = x̃>i Hx̃i (19)
Which when put into Equation 16,
KL = k∑ i=1 1 λi x̃>i Hx̃i + p∑ j=k+1 1 δ x̃>j Hx̃j + k∑ i=1 log(λ̃i) + p∑ j=k+1 log δ (20)
= k∑ i=1 x̃>i Hx̃i x̃>i Hx̃i + p∑ j=k+1 1 δ x̃>j Hx̃j + k∑ i=1 log(λ̃i) + p∑ j=k+1 log δ (21)
= k + p∑ j=k+1 1 δ x̃>j Hx̃j + k∑ i=1 log(λ̃i) + p∑ j=k+1 log δ (22)
= 1
δ p∑ j=k+1 x̃>j Hx̃j + k∑ i=1 log(λ̃i) (removing constants) (23)
= 1
δ p∑ j=k+1 x̃>j Hx̃j + k∑ i=1 log(x̃>i Hx̃i) (24)
Now we need to consider the constraints x̃>i x̃i = 1 and x̃ > i x̃j = 0, i 6= j by adding Lagrange multipliers to our KL cost,
L = 1
δ p∑ j=k+1 x̃>j Hx̃j + k∑ i=1 log(x̃>i Hx̃i)− k∑ i=1 φi,i(x̃ > i x̃i − 1)− ∑ i,j,i 6=j φi,j x̃ > i x̃j (25)
Taking derivatives wrt x̃i:
∂L ∂x̃i = 0 = 2Hx̃i x̃>i Hx̃i − 2φi,ix̃i − 2 ∑ i,j 6=i φi,j x̃j (26)
∑ i,j 6=i φi,j x̃j = ( H x̃>i Hx̃i − φi,iIp ) x̃i (27)
In Equation 27, we have x̃i expressed as a linear combination of x̃j , j 6= i, but x̃i and x̃j are orthogonal, so x̃i cannot be expressed as such, so φi,j = 0, i 6= j, and,
Hx̃i x̃>i Hx̃i = φi,ix̃i (28)
Meaning x̃i are eigenvectors of H for 1 ≤ i ≤ k. We can also use the same Lagrange multipliers to show that x̃i for k + 1 ≤ i ≤ p are also eigenvectors of H . This means that our cost,
KL = 1
δ p∑ j=k+1 x̃>j Hx̃j + k∑ i=1 log(x̃>i Hx̃i) (29)
= 1
δ p∑ j=k+1 κ̃j + k∑ i=1 log(κ̃i) (30)
where the set (κ̃1, κ̃2, ..., κ̃p) is a permutation of (λ1, λ2, ..., λp) and κ̃i = λ̃i for 1 ≤ i ≤ k. I.e., H̃ shares k eigenvalues with H , and the rest are δ. It now remains to determine which eigenvalues are shared and which are excluded.
Considering only two eigenvalues, λi, λj , and let λi > λj ≥ 0. Let r = λiλj . The relative cost of excluding λi in the set {κ̃1, κ̃2, ..., κ̃k} compared to including it is,
Relative Cost = λi − λj δ − log λi
λj
= λi(1− 1r )
δ − log r
If the relative cost is positive, then including λi as one of the eigenvalues of H̃ is the more optimal choice. Now solving the inequality,
Relative Cost > 0
λi(1− 1r ) δ − log r > 0
λi > δ(1− 1
r ) log r
Which, for sufficiently small δ is always true because r > 1. Thus, it is always better to swap two eigenvalues which are included/excluded, if the excluded one is larger. This means that H̃ has the k largest eigenvalues of H , and we already showed that it shares the same eigenvectors. This maximum eigenvalue/eigenvector pair selection is exactly the procedure used by SOLA.

heading: D COLD POSTERIOR VCL AND FURTHER GENERALIZATIONS
text: The use of KL-reweighting is closely related related to the idea of “cold-posteriors,” in which pT (θ|D) ∝ p(θ|D) 1 τ . Finding this cold posterior is equivalent to find optimal q distributions for maximizing the τ -ELBO:
τ -ELBO := Eθ∼q(θ)[log p(D|θ) + log p(θ)− τ log q(θ)]
whenQ is all possible distributions of θ. This objective is the same as the standard ELBO with only the entropy term reweighted, and contrasts the β-ELBO where both the entropy and prior likelihoods are reweighted. Here, β acts similarly to T (the temperature, not to be confused with task number). This relationship naturally leads to the transition diagram shown in Figure 11. In this, we can see that we can easily transition between posteriors at different temperatures by optimizing either the β-ELBO, τ -ELBO, or tempering the posterior.
When Q contains all possible distributions, moving along any path results in the exact same distribution, for example optimizing the τ -ELBO then tempering is the same as directly optimizing the ELBO. However in the case where Q is limited, this transition is not exact, and the resulting posterior is path dependent. In fact, each possible path represents a different valid method for performing continual learning. Standard VCL works by traversing the horizontal arrows, directly optimizing the ELBO, while an alternative scheme of VCL would optimize the τ -ELBO to form cold posteriors, then heat the posterior before optimizing the τ -ELBO for a new task. Inference can be done at either the warm or cold state. Note that for Gaussians, heating the posterior is just a matter of scaling the covariance matrix by a constant factor τafterτbefore .
While warm posteriors generated through this two-step procedure are not optimal under the ELBO, whenQ is limited, they may perform better for continual learning. Similar to Equation 2, the optimal Σ when optimizing the τ -ELBO is given by
Σ−1T = 1
τ T∑ t=1 H̃t + 1 τ Σ−10
Where H̃t is the approximate curvature for a specific value of τ for task t, which coincides with the true Hessian for τ → 0, like with the β-ELBO. Here, both the prior and data-dependent component are scaled by 1τ , in contrast to Equation 2, where only the data-dependent component is reweighted. As discussed in Section 2.2 and further explored in appendix A, this leads to a different scale of the quadratic approximation, which may lend itself better for continual learning. This also results in a second way to recover γ in Online EWC by first optimizing the β-ELBO with β = γ, then tempering by a factor of 1γ (i.e. increasing the temperature when γ < 1).

heading: E MAP DEGENERACY WITH FILM LAYERS
text: Here we describe how training FiLM layer with MAP training leads to degenerate values for the weights and scales, whereas with VI training, no degeneracy occurs. For simplicity, consider only the nodes leading into a single node and let there be d of them, i.e. θ has dimension d. Because we only have one node, our scale parameter γ is a single variable.
For MAP training, we have the loss function L = −p(D|θ, γ) + λ2 θ 2, with D the dataset and λ the L2 regularization hyperparameter. Note that p(D|θ, γ) = p(D|cθ, 1cγ), hence we can scale θ arbitrarily without affecting the likelihood, so long as γ is scaled inversely. If c < 1, λ2 θ 2 < λ2 ( 1 cθ)
2, so increasing c decreases the L2 penalty if θ is inversely scaled by c. Therefore the optimal setting of the scale parameter γ is arbitrarily large, while θ shrinks to 0.
At a high level, VI-training (with Gaussian posteriors and priors) does not have this issue because the KL-divergence penalizes the variance of the parameters from deviating from the prior in addition to the mean parameters, whereas MAP training only penalizes the means. Unlike with MAP training, if we downscale the weights, we also downscale the value of the variances, which increases the KL-divergence. The variances cannot revert to the prior either, as when they are up-scaled by the FiLM scale parameter, the noise would increase, affecting the log-likelihood component of the ELBO. Therefore, there exists an optimal amount of scaling which balances the mean-squared penalty component of the KL-divergence and the variance terms.
Mathematically we can derive this optimal scale. Consider the scenario with VI training with Gaussian variational distribution and prior, where our approximate posterior q(θ) has mean and variance µ and Σ and our prior p(θ) has parameters µ0 and Σ0. First consider the scenario without FiLM Layers. Now, have our loss function L = −Eθ∼q(θ) log p(D|θ) + DKL(q(θ)||q0(θ)). For multivariate Gaussians,
DKL(q(θ)||p(θ)) = 1
2 (log |Σ0| − log |Σ| − d+ Tr(Σ−10 Σ) + (µ− µ0)TΣ −1 0 (µ− µ0)).
Now consider another distribution q′(θ), with mean and variance parameters cµ and c2Σ. Now if q′(θ) is paired with FiLM scale parameter γ set at 1c , the log-likelihood component is unchanged:
Eθ∼q(θ) log p(D|θ) = Eθ∼q′(θ) log p(D|θ, γ = 1
c ),
with γ being our FiLM scale parameter and p(D|θ, γ) representing a model with FiLM scale layers. Now consider the DKL(q′(θ)||q0(θ)), and optimize c with µ and Σ fixed:
DKL(q ′(θ)||p(θ)) = 1
2 (log |Σ0| − log |c2Σ| − d+ Tr(Σ−10 c2Σ) + (cµ− µ0)TΣ −1 0 (cµ− µ0))
= 1
2 (log |Σ0| − log |Σ| − 2d log c− d+ c2Tr(Σ−10 Σ)
+ (cµ− µ0)TΣ−10 (cµ− µ0))
∂DKL ∂c |c=c∗ = 0 = − d c∗ + c∗Tr(Σ−10 Σ) + (c ∗µ− µ0)TΣ−10 µ
0 = −d+ c∗2Tr(Σ−10 Σ) + c∗2µTΣ −1 0 µ− c∗µT0 Σ −1 0 µ 0 = c∗2(Tr(Σ−10 Σ) + µ TΣ−10 µ)− c∗µT0 Σ −1 0 µ− d
⇒ c∗ = µT0 Σ −1 0 µ±
√ (µT0 Σ −1 0 µ) 2 + 4d(Tr(Σ−10 Σ) + µ TΣ−10 µ)
2(Tr(Σ−10 Σ) + µ TΣ−10 µ)
.
Also note that c = 0 results in an infinitely-large KL-divergence, so there is a barrier at c = 0, i.e. If optimized through gradient descent, c should never change sign. Furthermore, note that
∂2DKL ∂c2 = d c2 + Tr(Σ−10 Σ) + µ TΣ−10 µ > 0.
So the KL-divergence is concave with respective to c, so c∗ is a minimizer of DKL and therefore DKL(q(θ)||p(θ)) ≥ DKL(q′(θ)||p(θ))|c=c∗, which implies the optimal value of the FiLM scale parameter γ is 1c∗ . While no formal data was collected, it was observed that the scale parameters do in fact reach very close to this optimal scale value after training.

heading: F CLUSTERING OF FILM PARAMETERS
text: In this section, we test the interpretability of learned FiLM Parameters. Such clustering has been done in the past with FiLM parameters, as well as node-wise uncertainty parameters. One would intuitively expect that tasks from similar domains would finds similar features salient, and thus share similar FiLM parameters. To test this hypothesis, we took the 8 mixed vision task from Section 5.3 and split each task into multi 5-way classification tasks so that there were many tasks from similar domains. For example, CIFAR100, which originally had 100 classes, became 20 5-way clasification tasks, Trafficsigns became 8 tasks (7 5-way and 1 8-way), and MNIST 2 (2 5-way). Next, we trained the same architecture used in Section 5.3 except trained all 58 resulting tasks. Joint training was chosen over continual learning to avoid artifacts which would arise from task ordering. Figure 12 shows that the results scale and shift parameters can be clustered and FiLM parameters which arise from the same base task cluster together. Like in Achille et al. (2019), this likely could be used as a means of knowing which tasks to learn continually and which tasks to separate (i.e. tasks from the same cluster would likely benefit from joint training, while tasks from different ones should be separately trained), however we did not explore this idea further.

heading: G HOW FILM LAYERS INTERACT WITH PRUNING
text: In Section 3, we discussed the problem of pruning in variational continual learning and how it prevents nodes from becoming reactivated. To reiterate, pruning broadly occurs in three steps:
1. Weights incoming to a node begin to revert to the prior distribution
2. Noise from these high-variance weights affect the likelihood term in the ELBO
3. To prevent noise, the bias concentrates at a negative value to be cut off by the ReLU activation
Later tasks then are initialized with this negative bias with low variance, meaning that the node has a difficult time reactivating the node without incurring a high prior cost. This results in the effect shown in Figure 1, where after the first task, effectively no more nodes are reactivated. The effect is further exacerbated with larger values of β, where the pruning effect is stronger. Increasing λ worsens this as well, as increasing the quadratic cost further prevents already low-variance negative biases from moving.
We verify that this mechanism is indeed the cause of the limited capacity use by visualizing the posteriors for weights and biases entering a node in the first convolutional layer for a network trained on Easy-CHASY (Figure 13). Here, we see that biases in pruned nodes when there are no FiLM Layers do indeed concentrate at negative values. In contrast, biases in models with FiLM layers are able to revert to their prior because the FiLM parameters perform pruning.

heading: H RELATED WORK
text: Regularization-based continual learning. Many algorithms attempt to regularize network parameters based on a metric of importance. The most directly comparable algorithms to GVCL are EWC (Kirkpatrick et al., 2017), Online EWC (Schwarz et al., 2018), and VCL (Nguyen et al., 2018). EWC measures importance based on the Fisher information matrix, while VCL uses an approximate posterior covariance matrix as an importance measure. Online EWC slightly modifies EWC so that there is only a single regularizer based on the cumulative sum of Fisher information matrices. Lee et al. (2017) proposed IMM, which is an extension to EWC which merges posteriors based on their Fisher information matrices. Ritter et al. (2018) and Yin et al. (2020) both aim to approximate the Hessian by using either Kronecker-factored or low-rank forms, using the Laplace approximation to form approximate posteriors of parameters. These methods all use second-order approximations of the loss.Ahn et al. (2019), like us, use regularizers based on the ELBO, but also measure importance on a per-node basis than a per-weight one. SI (Zenke et al., 2017) measures importance using “Synaptic Saliency,” as opposed to methods based on approximate curvature.
Architectural approaches to continual and meta-learning. This family of methods modifies the standard neural architecture by either adding parallel or series components to the network. Progressive Neural Networks adds a parallel column network for every task. Pathnet (Fernando et al., 2017) can be interpreted as a parallel-network based algorithm, but rather than growing model size over time, the model size remains fixed while paths between layer columns are optimized. FiLM parameters can be interpreted as adding series components to a network, and has been a mainstay in the multitask and meta-learning literature. Requeima et al. (2019) use hypernetworks to amortize FiLM parameter learning, and has been shown to be capable of continual learning. Architectural approaches are often used in tandem with regularization based approaches, such as in HAT (Serra et al., 2018), which uses per-task gating parameters alongside a compression-based regularizer. Adel et al. (2020) propose CLAW, which also uses variational inference alongside per-task parameters, but requires a more complex meta-learning based training procedure involving multiple splits of the dataset. GVCL with FiLM layers adds to this list of hybrid architectural-regularization based approaches.
Cold Posteriors and likelihood-tempering. As mentioned in Section 2, likelihood-tempering (or KL-reweighting) has been empirically found to improve performance when using variational inference for Bayesian Neural Networks over a wide number of contexts and papers (Osawa et al., 2019; Zhang et al., 2018). Cold posteriors are closely related to likelihood tempering, except they temper the full posterior rather than only the likelihood term, and often empirically outperform Bayesian posteriors when using MCMC sampling Wenzel et al. (2020). From an information-theoretic perspective, KL-reweighted ELBOs have also studied as compression (Achille et al., 2020). Achille et al. (2019), like us, considers a limiting case of β, and uses this to measure parameter saliency, but use this information to create a task embedding rather than for continual learning. Outside of the Bayesian Neural Network context, values of β > 1 have also been explored (Higgins et al., 2017), and more generally different values of β trace out different points on a rate-distortion curve for VAEs (Alemi et al., 2018).

heading: I EXPERIMENT DETAILS
text: I.1 REPORTED METRICS
All reported scores and figures present the mean and standard deviation across 5 runs of the algorithm with a different network initialization. For Easy-CHASY and Hard-CHASY, train/test splits are also varied across iterations. For the Mixed Vision tasks, task permutation of the 8 tasks is also randomized between iterations.
Let the matrix Ri,j represent the performance of jth task after the model was trained on the ith task. Furthermore, let Rindj be the mean performance of the jth for a network trained only on that task and let the total number of tasks be T . Following Lopez-Paz & Ranzato (2017) and Pan et al. (2020), we define
Average Accuracy (ACC) = 1
T T∑ j=1 RT,j ,
Forward Transfer (FWT) = 1
T T∑ j=1 Rj,j −Rindj ,
Backward Transfer (BWT) = 1
T T∑ j=1 RT,j −Rj,j .
Note that these metrics are not exactly the same as those presented in all other works, as the FWT and BWT metrics are summed over the indices 1 ≤ j ≤ T , whereas Lopez-Paz & Ranzato (2017) and Pan et al. (2020) sum from 2 ≤ j ≤ T and 1 ≤ j ≤ T − 1 for FWT and BWT, respectively. For FWT, this definition does not assumes that R1,1 = Rind1 , and affects algorithms such as HAT and Progressive Neural Networks, which either compress the model, resulting in lower accuracy, or use a smaller architecture for the first task. The modified BWT transfer is equal to the other BWT metrics apart from a constant factor T−1T .
Intuitively, forward transfer equates to how much continual learning has benefited a task when a task is newly learned, while backwards transfer is the accuracy drop as the network learns more tasks compared to when a task was first learned. Furthermore, in the tables in Appendix J, we also present net performance gain (NET), which quantifies the total gain over separate training, at the end of training continually:
NET = FWT + BWT = 1
T T∑ j=1 RT,j −Rindj .
Note that for computation of Rind, we compare to models trained under the same paradigm, i.e. MAP algorithms (all baselines except for VCL) are compared to a MAP trained model, and VI algorithms (GVCL-F, GVCL and VCL) are compared to KL-reweighted VI models. This does not make a difference for most of the benchmarks where RindMAP ≈ RindVI . However, for Easy and HardCHASY, RindMAP < R ind VI , so we compare VI to VI and MAP to MAP to obtain fair metrics.
In Figure 5b, we plot ∆ACCi, which we define as
∆ACCi = 1
i i∑ j=1 Ri,j −RT,j .
This metric is useful when the tasks have very different accuracies and their permutation is randomized, as is the case with the mixed vision tasks. Note that this means that Ri,j would refer to a different task for each permutation, but we average over the 5 permutations of the runs. Empirically,
if two algorithms have similar final accuracies, this metric measures how much the network forgets about the first i tasks from that point to the end, and also measures how high the accuracy would have been if training was terminated after i tasks. Plotting this also captures the concept of graceful vs catastrophic forgetting, as graceful forgetting would show up as a smooth downward curve, while catastrophic forgetting would have sudden drops.
I.2 OPTIMIZER AND TRAINING DETAILS
The implementation of all baseline methods was based on the Github repository6 for HAT (Serra et al., 2018), except the implementions of IMM-Mode and EWC were modified due to an error in the computation of the Fisher Information Matrix in the original implementation. Baseline MAP algorithms were trained with SGD with a decaying learning starting at 5e-2 with a maximum epochs of 200 per task for the Split-MNIST, Split-CIFAR and the mixed vision benchmarks. The number of maximum epochs for Easy-CHASY and Hard-CHASY was 1000, due to the small dataset size. Early stopping based on the validation set was used. 10% of the training set was used as validation for these methods, and for Easy and Hard CHASY, 8 samples per class form the validation set (which are disjoint from the training samples or test samples).
For VI models, we used Adam optimizer with a learning rate of 1e-4 for Split-MNIST and Mixture, and 1e-3 for Easy-CHASY, Hard-CHASY and Split-CIFAR. We briefly tested running the baselines algorithms using Adam rather than SGD and performance did not change. Easy-CHASY and HardCHASY were run for 1500 epochs per task, Split-MNIST for 100, Split-CIFAR for 60, and 180 for Mixture. The number of epochs was changed so that the number of gradient steps for each task was roughly equal. For Easy-CHASY, Hard-CHASY and Split-CIFAR, this means that later tasks are run for more epochs, since the largest training sets are at the start. For Mixture, we ran 180 equivalents epochs for Facescrub. For how many epochs this equates to in the other datasets, we refer the reader to Appendix A in Serra et al. (2018). We did not use early stopping for these VI results. While we understand that in some cases we trained for many more epochs than the baselines, the baselines used early stopping and therefore all stopped long before the 200 epoch limit was reached, so allocating more time would not change their results. Swaroop et al. (2019) also finds that allowing VI to converge is crucial for continual learning performance. We leave the discussion of improving this convergence time for future work.
All experiments (both the baselines and VI methods) use a batch size of 64.
I.3 ARCHITECTURAL DETAILS
Easy and Hard CHASY. We use a convolutional architecture with 2 convolutions layers with:
1. 3x3 convolutional layer with 16 filters, padding of 1, ReLU activations 2. 2x2 Max Pooling with stride 2 3. 3x3 convolutional layer with 32 filters, padding of 1, ReLU activations 4. 2x2 Max Pooling with stride 2 5. Flattening layer 6. Fully connected layer with 100 units and ReLU activations 7. Task-specific head layers
Split-MNIST. We use a standard MLP with:
1. Fully connected layer with 256 units and ReLU activations 2. Fully connected layer with 256 units and ReLU activations 3. Task-specific head layers
Split-CIFAR. We use the same architecture from Zenke et al. (2017):
1. 3x3 convolutional layer with 32 filters, padding of 1, ReLU activations 6Repository at https://github.com/joansj/hat
2. 3x3 convolutional layer with 32 filters, padding of 1, ReLU activations
3. 2x2 Max Pooling with stride 2
4. 3x3 convolutional layer with 64 filters, padding of 1, ReLU activations
5. 3x3 convolutional layer with 64 filters, padding of 1, ReLU activations
6. 2x2 Max Pooling with stride 2
7. Flattening
8. Fully connected layer with 512 units and ReLU activations
9. Task-specific head layers
Mixed vision tasks. We use the same AlexNet architecture from Serra et al. (2018):
1. 4x4 convolutional layer with 64 filters, padding of 0, ReLU activations
2. 2x2 Max Pooling with stride 2
3. 3x3 convolutional layer with 128 filters, padding of 0, ReLU activations
4. 2x2 Max Pooling with stride 2
5. 2x2 convolutional layer with 256 filters, padding of 0, ReLU activations
6. 2x2 Max Pooling with stride 2
7. Flattening
8. Fully connected layer with 2048 units and ReLU activations
9. Fully connected layer with 2048 units and ReLU activations
10. Task-specific head layers
For MAP models, dropout layers with probabilities of either 0.2 or 0.5 were added after convolutional or fully-connected layers. For GVCL-F, FiLM layers were inserted after convolutional/hidden layers, but before ReLU activations.
I.4 HYPERPARAMETER SELECTION
For all algorithms on Easy-CHASY, Hard-CHASY, Split-MNIST and Split-CIFAR, hyperparameter selection was done by selecting the combination which produced the best average accuracy on the first 3 tasks. The algorithms were then run on the full number of tasks. For the Mixed Vision tasks, the best hyperparameters for the baselines were taken from the HAT Github repository. For GVCL, we performed hyperparameter selection in the same way as in Serra et al. (2018): we found the best hyperparameters for the average performance on the first random permutation of tasks. Note that in the mixture tasks, we randomly permute the task order for each iteration (with permutations kept consistent between algorithms), whereas for the other 4 benchmarks, the task order is fixed. Hyperparameter searches were performed using a grid search. The best selected hyperparameters are shown in Table 3.
For the Joint and Separate VI baselines, we used the same β. For the mixed vision tasks, we had to used a prior variance of 0.01 (for both VCl, GVCL and GVCL-F), but for all other tasks we did not need to tune this.

heading: J FURTHER EXPERIMENTAL RESULTS
text: In following section we present more quantitative results of the various baselines on our benchmarks. For brevity, in the main text, we only included the best performing baselines and those which are most comparable to GVCL, which consisted of HAT, PathNet, Online EWC and VCL.
J.1 EASY-CHASY ADDITIONAL RESULTS
J.2 HARD-CHASY ADDITIONAL RESULTS
J.3 Split-MNIST ADDITIONAL RESULTS
J.4 Split-CIFAR ADDITIONAL RESULTS
J.5 MIXED VISION TASKS ADDITIONAL RESULTS

heading: K CLUSTERED HASYV2 (CHASY)
text: The HASYv2 dataset is a dataset consisting over 32x32 black/white handwritten Latex characters. There are a total of 369 classes, and over 150 000 total samples (Thoma, 2017).
We constructed 10 classification tasks, each with a varying number of classes ranging from 20 to 11. To construct these tasks, we first trained a mean-field Bayesian neural network on a 200-way classification task on the 200 classes with the most total samples. To get an embedding for each class, we use the activations of the second-last layer. Then, we performed K-means clustering with 20 clusters on the means of the embedding generated by each class when the samples of the classes were input into the network. Doing this yielded the classes shown in figure 32. Now, within each cluster are classes which are deemed “similar” by the network. To make the 10 classification tasks, we then took classes from each cluster sequentially (in order of the class whose mean was closest to the cluster’s mean), so that each task contains at most 1 symbol from each cluster. Doing this ensures that tasks are similar to one another, since each task consists of classes which are different in similar ways. With the classes selected, the training set is made by selecting 16 samples of each classes, and using the remaining as the test set. This procedure was used to generate the “easy” set of tasks, which should have the maximum amount of similarity between tasks. We also constructed a second set of tasks, the “hard” set, in which each task is individually difficult. This was done by selecting each task to be classification within each cluster, selecting clusters with the most number of symbols first. This corresponds to clusters 1-10 in figure 32. With the classes for each task selected, 16 samples from each class are used in the training set, and the remainder are used as the test set. Excess samples are discarded so that the test set class distribution is also uniform within each task.
It was necessary to perform this clustering procedure as we found it difficult to produce sizable transfer gains if we simply constructed tasks by taking the classes with the most samples. While we were able to have gains of up to 3% from joint training on 10 20-way classification tasks with the tasks chosen by class sample count, these gains were significantly diminished when performing MAP estimation as opposed to MLE estimation, and reduced even further when performing VI. Because one of our benchmark continual learning methods is VCL, showing transfer when trained using VI is necessary.
Figures 33a and 34 show the performance gains of joint training over separate training on this new dataset, for both MAP, and KL-reweighted VI, respectively. Figure 33b shows how relative test set accuracy varies for each specific task for these training procedures.

Review summaries:
This paper proposes Generalized Variational Continual Learning (GVCL). It is shown that Online EWC and VCL are special cases of GVCL, along with other theoretical contributions. Further, GVCL is augmented with FiLM to alleviate weaknesses of VCL and GVCL. GVCL and GVCL-F are applied to a number of continual learning tasks and demonstrate competitive performance.