Generalized Variational Continual Learning

1 INTRODUCTION . Continual learning methods enable learning when a set of tasks changes over time . This topic is of practical interest as many real-world applications require models to be regularly updated as new data is collected or new tasks arise . Standard machine learning models and training procedures fail in these settings ( French , 1999 ) , so bespoke architectures and fitting procedures are required . This paper makes two main contributions to continual learning for neural networks . First , we develop a new regularization-based approach to continual learning . Regularization approaches adapt parameters to new tasks while keeping them close to settings that are appropriate for old tasks . Two popular approaches of this type are Variational Continual Learning ( VCL ) ( Nguyen et al. , 2018 ) and Online Elastic Weight Consolidation ( Online EWC ) ( Kirkpatrick et al. , 2017 ; Schwarz et al. , 2018 ) . The former is based on a variational approximation of a neural network ’ s posterior distribution over weights , while the latter uses Laplace ’ s approximation . In this paper , we propose Generalized Variational Continual Learning ( GVCL ) of which VCL and Online EWC are two special cases . Under this unified framework , we are able to combine the strengths of both approaches . GVCL is closely related to likelihood-tempered Variational Inference ( VI ) , which has been found to improve performance in standard learning settings ( Zhang et al. , 2018 ; Osawa et al. , 2019 ) . We also see significant performance improvements in continual learning . Our second contribution is to introduce an architectural modification to the neural network that combats the deleterious overpruning effect of VI ( Trippe & Turner , 2018 ; Turner & Sahani , 2011 ) . We analyze pruning in VCL and show how task-specific FiLM layers mitigate it . Combining this architectural change with GVCL results in a hybrid architectural-regularization based algorithm . This additional modification results in performance that exceeds or is within statistical error of strong baselines such as HAT ( Serra et al. , 2018 ) and PathNet ( Fernando et al. , 2017 ) . The paper is organized as follows . Section 2 outlines the derivation of GVCL , shows how it unifies many continual learning algorithms , and describes why it might be expected to perform better than them . Section 3 introduces FiLM layers , first from the perspective of multi-task learning , and then through the lens of variational over-pruning , showing how FiLM layers mitigate this pathology of VCL . Finally , in Section 5 we test GVCL and GVCL with FiLM layers on many standard bench- marks , including ones with few samples , a regime that could benefit more from continual learning . We find that GVCL with FiLM layers outperforms existing baselines on a variety of metrics , including raw accuracy , forwards and backwards transfer , and calibration error . In Section 5.4 we show that FiLM layers provide a disproportionate improvement to variational methods , confirming our hypothesis in Section 31 . 2 GENERALIZED VARIATIONAL CONTINUAL LEARNING . In this section , we introduce Generalized Variational Continual Learning ( GVCL ) as a likelihoodtempered version of VCL , with further details in Appendix C. We show how GVCL recovers Online EWC . We also discuss further links between GVCL and the Bayesian cold posterior in Appendix D . 2.1 LIKELIHOOD-TEMPERING IN VARIATIONAL CONTINUAL LEARNING . Variational Continual Learning ( VCL ) . Bayes ’ rule calculates a posterior distribution over model parameters θ based on a prior distribution p ( θ ) and some dataset DT = { XT , yT } . Bayes ’ rule naturally supports online and continual learning by using the previous posterior p ( θ|DT−1 ) as a new prior when seeing new data ( Nguyen et al. , 2018 ) . Due to the intractability of Bayes ’ rule in complicated models such as neural networks , approximations are employed , and VCL ( Nguyen et al. , 2018 ) uses one such approximation , Variational Inference ( VI ) . This approximation is based on approximating the posterior p ( θ|DT ) with a simpler distribution qT ( θ ) , such as a Gaussian . This is achieved by optimizing the ELBO for the optimal qT ( θ ) , ELBOVCL = Eθ∼qT ( θ ) [ log p ( DT |θ ) ] −DKL ( qT ( θ ) ‖qT−1 ( θ ) ) , ( 1 ) where qT−1 ( θ ) is the approximation to the previous task posterior . Intuitively , this refines a distribution over weight samples that balances good predictive performance ( the first expected prediction accuracy term ) while remaining close to the prior ( the second KL-divergence regularization term ) . Likelihood-tempered VCL . Optimizing the ELBO will recover the true posterior if the approximating family is sufficiently rich . However , the simple families used in practice typically lead to poor test-set performance . Practitioners have found that performance can be improved by downweighting the KL-divergence regularization term by a factor β , with 0 < β < 1 . Examples of this are seen in Zhang et al . ( 2018 ) and Osawa et al . ( 2019 ) , where the latter uses a “ data augmentation factor ” for down-weighting . In a similar vein , sampling from “ cold posteriors ” in SG-MCMC has also been shown to outperform the standard Bayes posterior , where the cold posterior is given by pT ( θ|D ) ∝ p ( θ|D ) 1 T , T < 1 ( Wenzel et al. , 2020 ) . Values of β > 1 have also been used to improve the disentanglement variational autoencoder learned models ( Higgins et al. , 2017 ) . We down-weight the KL-divergence term in VCL , optimizing the β-ELBO2 , β-ELBO = Eθ∼qT ( θ ) [ log p ( DT |θ ) ] − βDKL ( qT ( θ ) ‖qT−1 ( θ ) ) . VCL is trivially recovered when β = 1 . We will now show that surprisingly as β → 0 , we recover a special case of Online EWC . Then , by modifying the term further as required to recover the full version of Online EWC , we will arrive at our algorithm , Generalized VCL . 2.2 ONLINE EWC IS A SPECIAL CASE OF GVCL . We analyze the effect of KL-reweighting on VCL in the case where the approximating family is restricted to Gaussian distributions over θ . We will consider training all the tasks with a KLreweighting factor of β , and then take the limit β → 0 , recovering Online EWC . Let the approximate posteriors at the previous and current tasks be denoted as qT−1 ( θ ) = N ( θ ; µT−1 , ΣT−1 ) and qT ( θ ) = N ( θ ; µT , ΣT ) respectively , where we are learning { µT , ΣT } . The optimal ΣT under the β-ELBO has the form ( see Appendix C ) , Σ−1T = 1 β ∇µT∇µTEqT ( θ ) [ − log p ( DT |θ ) ] + Σ −1 T−1 . ( 2 ) 1Code is available at https : //github.com/yolky/gvcl 2We slightly abuse notation by writing the likelihood as p ( DT |θ ) instead of p ( yT |θ , XT ) . Now take the limit β → 0 . From Equation 2 , ΣT → 0 , so qT ( θ ) becomes a delta function , and Σ−1T = − 1 β ∇µT∇µT log p ( DT |θ = µT ) + Σ−1T−1 = 1 β HT + Σ −1 T−1 = 1 β T∑ t=1 Ht + Σ −1 0 , ( 3 ) where HT is the T th task Hessian3 . Although the learnt distribution qT ( θ ) becomes a delta function ( and not a full Gaussian distribution as in Laplace ’ s approximation ) , we will see that a cancellation of β factors in the β-ELBO will lead to the eventual equivalence between GVCL and Online EWC . Consider the terms in the β-ELBO that only involve µT : β-ELBO = Eθ∼qT ( θ ) [ log p ( DT |θ ) ] − β 2 ( µT − µT−1 ) > Σ−1T−1 ( µT − µT−1 ) = log p ( DT |θ = µT ) − 1 2 ( µT − µT−1 ) > ( T−1∑ t=1 Ht + βΣ −1 0 ) ( µT − µT−1 ) , ( 4 ) where we have set the form of ΣT−1 to be as in Equation 3 . Equation 4 is an instance of the objective function used by a number of continual learning methods , most notably Online EWC4 ( Kirkpatrick et al. , 2017 ; Schwarz et al. , 2018 ) , Online-Structured Laplace ( Ritter et al. , 2018 ) , and SOLA ( Yin et al. , 2020 ) . These algorithms can be recovered by changing the approximate posterior class Q to Gaussians with diagonal , block-diagonal Kronecker-factored covariance matrices , and low-rank precision matrices , respectively ( see Appendices C.4 and C.5 ) . Based on this analysis , we see that β can be seen as interpolating between VCL , with β = 1 , and continual learning algorithms which use point-wise approximations of curvature as β → 0 . In Appendix A we explore how β controls the scale of the quadratic curvature approximation , verifying with experiments on a toy dataset .. Small β values learn distributions with good local structure , while higher β values learn distributions with a more global structure . We explore this in more detail in Appendices A and B , where we show the convergence of GVCL to Online-EWC on a toy experiment . Inference using GVCL . When performing inference with GVCL at test time , we use samples from the unmodified q ( θ ) distribution . This means that when β = 1 , we recover the VCL predictive , and as β → 0 , the posterior collapses as described earlier , meaning that the weight samples are effectively deterministic . This is in line with the inference procedure given by Online EWC and its variants . In practice , we use values of β = 0.05 − 0.2 in Section 5 , meaning that some uncertainty is retained , but not all . We can increase the uncertainty at inference time by using an additional tempering step , which we describe , along with further generalizations in Appendix D. 2.3 REINTERPRETING λ AS COLD POSTERIOR REGULARIZATION As described above , the β-ELBO recovers instances of a number of existing second-order continual learning algorithms including Online EWC as special cases . However , the correspondence does not recover a key hyperparameter λ used by these methods that up-weights the quadratic regularization term . Instead , our derivation produces an implicit value of λ = 1 , i.e . equal weight between tasks of equal sample count . In practice it is found that algorithms such as Online EWC perform best when λ > 1 , typically 10 − 1000 . In this section , we view this λ hyperparameter as a form of cold posterior regularization . In the previous section , we showed that β controls the length-scale over which we approximate the curvature of the posterior . However , the magnitude of the quadratic regularizer stays the same , because theO ( β−1 ) precision matrix and the β coefficient in front of the KL-term cancel out . Taking inspiration from cold posteriors ( Wenzel et al. , 2020 ) , which temper both the likelihood and the prior and improve accuracy with Bayesian neural networks , we suggest tempering the prior in GVCL . Therefore , rather than measuring the KL divergence between the posterior and prior , qT and qT−1 , respectively , we suggest regularizing towards tempered version of the prior , qλT−1 . However , this 3The actual Hessian may not be positive semidefinite while Σ is , so here we refer to a positive semidefinite approximation of the Hessian . 4EWC uses the Fisher information , but our derivation results in the Hessian . The two matrices coincide when the model has near-zero training loss , as is often the case ( Martens , 2020 ) . form of regularization has a problem : in continual learning , over the course of many tasks , old tasks will be increasingly ( exponentially ) tempered . In order to combat this , we also use the tempered version of the posterior in the KL divergence , qλT . This should allow us to gain benefits from tempering the prior while being stable over multiple tasks in continual learning . As we now show , tempering in this way recovers the λ hyperparameter from algorithms such as Online EWC . Note that raising the distributions to the power λ is equivalent to tempering by τ = λ−1 . For Gaussians , tempering a distribution by a temperature τ = λ−1 is the same as scaling the covariance by λ−1 . We can therefore expand our new KL divergence , DKL ( qλT ‖qλT−1 ) = 1 2 ( ( µT − µT−1 ) > λΣ−1T−1 ( µT − µT−1 ) + Tr ( λΣ −1 T−1λ −1ΣT ) + log |ΣT−1|λ−d |ΣT |λ−d − d ) = 1 2 ( ( µT − µT−1 ) > λΣ−1T−1 ( µT − µT−1 ) + Tr ( Σ −1 T−1ΣT ) + log |ΣT−1| |ΣT | − d ) = DKLλ ( qT ‖qT−1 ) . In the limit of β → 0 , our λ coincides with Online EWC ’ s λ , if the tasks have the same number of samples . However , this form of λ has a slight problem : it increases the regularization strength of the initial prior Σ0 on the mean parameter update . We empirically found that this negatively affects performance . We therefore propose a different version of λ , which only up-weights the “ datadependent ” parts of ΣT−1 , which can be viewed as likelihood tempering the previous task posterior , as opposed to tempering both the initial prior and likelihood components . This new version still converges to Online EWC as β → 0 , since the O ( 1 ) prior becomes negligible compared to the O ( β−1 ) Hessian terms . We define , Σ̃−1T , λ : = λ β T∑ t=1 Ht + Σ −1 0 = λ ( Σ −1 T − Σ −1 0 ) + Σ −1 0 . In practice , it is necessary to clip negative values of Σ−1T −Σ −1 0 to keep Σ̃ −1 T , λ positive definite . This is only required because of errors during optimization . We then use a modified KL-divergence , DKLλ̃ ( qT ‖qT−1 ) = 1 2 ( ( µT − µT−1 ) > Σ̃−1T−1 , λ ( µT − µT−1 ) + Tr ( Σ −1 T−1ΣT ) + log |ΣT−1| |ΣT | − d ) . Note that in Online EWC , there is another parameter γ , that down-weights the previous Fisher matrices . As shown in Appendix C , we can introduce this hyperparameter by taking the KL divergence priors and posteriors at different temperatures : qλT−1 and q γλ T . However , we do not find that this approach improves performance . Combining everything , we have our objective for GVCL , Eθ∼qT ( θ ) [ log p ( DT |θ ) ] − βDKLλ̃ ( qT ( θ ) ‖qT−1 ( θ ) ) .

Summary: This paper proposes Generalized Variational Continual Learning (GVCL). It is shown that Online EWC and VCL are special cases of GVCL, along with other theoretical contributions. Further, GVCL is augmented with FiLM to alleviate weaknesses of VCL and GVCL. GVCL and GVCL-F are applied to a number of continual learning tasks and demonstrate competitive performance.