Uncertainty-guided Continual Learning with Bayesian Neural Networks

1 INTRODUCTION . Humans can easily accumulate and maintain knowledge gained from previously observed tasks , and continuously learn to solve new problems or tasks . Artificial learning systems typically forget prior tasks when they can not access all training data at once but are presented with task data in sequence . Overcoming these challenges is the focus of continual learning , sometimes also referred to as lifelong learning or sequential learning . Catastrophic forgetting ( McCloskey & Cohen , 1989 ; McClelland et al. , 1995 ) refers to the significant drop in the performance of a learner when switching from a trained task to a new one . This phenomenon occurs because trained parameters on the initial task change in favor of learning new objectives . Given a network of limited capacity , one way to address this problem is to identify the importance of each parameter and penalize further changes to those parameters that were deemed to be important for the previous tasks ( Kirkpatrick et al. , 2017 ; Aljundi et al. , 2018 ; Zenke et al. , 2017 ) . An alternative is to freeze the most important parameters and allow future tasks to only adapt the remaining parameters to new tasks ( Mallya & Lazebnik , 2018 ) . Such models rely on the explicit parametrization of importance . We propose here implicit uncertainty-guided importance representation . Bayesian approaches to neural networks ( MacKay , 1992b ) can potentially avoid some of the pitfalls of explicit parameterization of importance in regular neural networks . Bayesian techniques , naturally account for uncertainty in parameters estimates . These networks represent each parameter with a distribution defined by a mean and variance over possible values drawn from a shared latent probability distribution ( Blundell et al. , 2015 ) . Variational inference can approximate posterior distributions using Monte Carlo sampling for gradient estimation . These networks act like ensemble methods in that they reduce the prediction variance but only use twice the number of parameters present in a regular neural network . We propose to use the predicted mean and variance of the latent distributions to characterize the importance of each parameter . We perform continual learning with ∗Corresponding author : sayna @ berkeley.edu †Work done while at Facebook AI Research Illustration of evolution of weight distributions through learning two tasks . ( a ) circles represent weight parameters , initialized by distributions with mean and variance values randomly sampled from Ɲ ( 0,0.1 ) . As an example we show five color-coded and plot their distributions . ( b ) Shows posterior distribution after learning Task 1 . While W1 and W2 exhibit lower uncertainties ( more contributions in learning Task 1 ) , W3 , W4 , and W5 appear to have larger uncertainties , with the highest STD in W5 , making them available to learn more tasks . ( c ) Task 2 is learned using higher learning rates for previously uncertain parameters ( W3 and W4 , W5 ) while learning rates for W1 and W2 are moderated according to their predicted low uncertainty after finishing task 1 . 1 3 2 4 5 Bayesian neural networks by controlling the learning rate of each parameter as a function of its uncertainty . Figure 1 illustrates how posterior distributions evolve for certain and uncertain weight distributions while learning two consecutive tasks . Intuitively , the more uncertain a parameter is , the more learnable it can be and therefore , larger gradient steps can be taken for it to learn the current task . As a hard version of this regularization technique , we also show that pruning , i.e. , preventing the most important model parameters from any change and learning new tasks with the remaining parameters , can be also integrated into UCB . We refer to this method as UCB-P . Contributions : We propose to perform continual learning with Bayesian neural networks and develop a new method which exploits the inherent measure of uncertainty therein to adapt the learning rate of individual parameters ( Sec . 4 ) . Second , we introduce a hard-threshold variant of our method that decides which parameters to freeze ( Sec . 4.2 ) . Third , in Sec . 5 , we extensively validate our approach experimentally , comparing it to prior art both on single datasets split into different tasks , as well as for the more difficult scenario of learning a sequence of different datasets . Forth , in contrast to most prior work , our approach does not rely on knowledge about task boundaries at inference time , which humans do not need and might not be always available . We show in Sec . 6 that our approach naturally supports this scenario and does not require task information at test time , sometimes also referred to as a “ single head ” scenario for all tasks . We refer to evaluation metric of a “ single head ” model without task information at test time as “ generalized accuracy ” . Our code is available at https : //github.com/SaynaEbrahimi/UCB . 2 RELATED WORK . Conceptually , approaches to continual learning can be divided into the following categories : dynamic architectural methods , memory-based methods , and regularization methods . Dynamic architectural methods : In this setting , the architecture grows while keeping past knowledge fixed and storing new knowledge in different forms such as additional layers , nodes , or modules . In this approach , the objective function remains fixed whereas the model capacity grows –often exponentially– with the number of tasks . Progressive networks ( Rusu et al. , 2016 ; Schwarz et al. , 2018 ) was one of the earliest works in this direction and was successfully applied to reinforcement learning problems ; the base architecture was duplicated and lateral connections added in response to new tasks . Dynamically Expandable Network ( DEN ) ( Yoon et al. , 2018 ) also expands its network by selecting drifting units and retraining them on new tasks . In contrast to our method , these approaches require the architecture grow with each new task . Memory-based methods : In this regime , previous information is partially stored to be used later as a form of rehearsal ( Robins , 1995 ) . Gradient episodic memory ( GEM ) ( Lopez-Paz et al. , 2017 ) uses this idea to store the data at the end of each episode to be used later to prevent gradient updates from deviating from their previous values . GEM also allows for positive backward knowledge transfer , i.e , an improvement on previously learned tasks , and it was the first method capable of learning using a single training example . Recent approaches in this category have mitigated forgetting by using external data combined with distillation loss and/or confidence-based sampling strategies to select the most representative samples . ( Castro et al. , 2018 ; Wu et al. , 2019 ; Lee et al. , 2019 ) Regularization methods : In these approaches , significant changes to the representation learned for previous tasks are prevented . This can be performed through regularizing the objective function or directly enforced on weight parameters . Typically , this importance measure is engineered to represent the importance of each parameter . Inspired by Bayesian learning , in elastic weight consolidation ( EWC ) method ( Kirkpatrick et al. , 2017 ) important parameters are those to have the highest in terms of the Fisher information matrix . In Synaptic Intelligence ( SI ) ( Zenke et al. , 2017 ) this parameter importance notion is engineered to correlate with the loss function : parameters that contribute more to the loss are more important . Similar to SI , Memory-aware Synapses ( MAS ) ( Aljundi et al. , 2018 ) proposed an online way of computing importance adaptive to the test set using the change in the model outputs w.r.t the inputs . While all the above algorithms are task-dependent , in parallel development to this work , ( Aljundi et al. , 2019 ) has recently investigated task-free continual learning by building upon MAS and using a protocol to update the weights instead of waiting until the tasks are finished . PackNet ( Mallya & Lazebnik , 2018 ) used iterative pruning to fully restrict gradient updates on important weights via binary masks . This method requires knowing which task is being tested to use the appropriate mask . PackNet also ranks the weight importance by their magnitude which is not guaranteed to be a proper importance indicative . HAT ( Serra et al. , 2018 ) identifies important neurons by learning an attention vector to the task embedding to control the gradient propagation . It maintains the information learned on previous tasks using an almost-binary mask per previous tasks . Bayesian approaches : Using Bayesian approach in learning neural networks has been studied for few decades ( MacKay , 1992b ; a ) . Several approaches have been proposed for Bayesian neural networks , based on , e.g. , the Laplace approximation ( MacKay , 1992a ) , Hamiltonian Monte Carlo ( Neal , 2012 ) , variational inference ( Hinton & Van Camp , 1993 ; Graves , 2011 ) , and probabilistic backpropagation ( Hernández-Lobato & Adams , 2015 ) . Variational continual learning ( Nguyen et al. , 2018 ) uses Bayesian inference to perform continual learning where new posterior distribution is simply obtained by multiplying the previous posterior by the likelihood of the dataset belonging to the new task . They also showed that by using a core-set , a small representative set of data from previous tasks , VCL can experience less forgetting . In contrast , we rely on Bayesian neural networks to use their predictive uncertainty to perform continual learning . Moreover , we do not use episodic memory or any other way to access or store previous data in our approach . Natural gradient descent methods : A fast natural gradient descent method for variational inference was introduced in ( Khan & Nielsen , 2018 ) in which , the Fisher Information matrix is approximated using the generalized Gauss-Newton method . In contrast , in our work , we use classic gradient descent . Although second order optimization algorithms are proven to be more accurate than the first order methods , they add considerable computational cost . Tseran et al . ( 2018 ) ; Chen et al . ( 2019 ) both investigate the effect of natural gradient descent methods as an alternative to classic gradient descent used in VCL and EWC methods . GNG ( Chen et al. , 2019 ) uses Gaussian natural gradients in the Adam optimizer ( Kingma & Ba , 2014 ) in the framework of VCL because as opposed to conventional gradient methods which perform in Euclidian space , natural gradients cause a small difference in terms of distributions following the changes in parameters in the Riemannian space . Similar to VCL , they obtained their best performance by adding a coreset of previous examples . Tseran et al . ( 2018 ) introduce two modifications to VCL called Natural-VCL ( N-VCL ) and VCL-Vadam . N-VCL ( Tseran et al. , 2018 ) uses a Gauss-Newton approximation introduced by ( Schraudolph , 2002 ; Graves , 2011 ) to estimate the VCL objective function and used natural gradient method proposed in ( Khan et al. , 2018 ) to exploit the Riemannian geometry of the variational posterior by scaling the gradient with an adaptive learning rate equal to σ−2 obtained by approximating the Fisher Information matrix in an online fashion . VCL-Vadam ( Tseran et al. , 2018 ) is a simpler version of N-VCL to trade-off accuracy for simplicity which uses Vadam ( Khan et al. , 2018 ) to update the gradients by perturbing the weights with a Gaussian noise using a reparameterization trick and scaling by σ−1 instead of its squared . N-VCL/VCL-Vadam both use variational inference to adapt the learning rate within Adam optimizer at every time step , whereas in our method below , gradient decent is used with constant learning rate during each task where learning rate scales with uncertainty only after finishing a task . We show extensive comparison with state-of-the-art results on short and relatively long sequence of vision datasets with Bayesian convolutional neural networks , whereas VCL-Vadam only rely on multi-layer perceptron networks . We also like to highlight that this is the first work which evaluates and shows the working of convolutional Bayesian Neural Networks rather than only fully connected MLP models for continual learning .

Summary: The authors propose a novel method for continual learning with neural networks based on a Bayesian approach. The idea consists in working with Bayesian neural networks, using the Bayes by back-prop approach in which a factorized Gaussian variational distribution is used to approximate the true posterior. To address the continual learning setting, the authors propose to multiply the learning rate of the mean parameters in the posterior approximation by the corresponding standard deviation parameter in the posterior approximation, while the learning rate for the variance parameters in the posterior approximation is not changed. The authors also consider a version of his method which freezes the mean and variance variational parameters when the signal to noise ratio is high. The proposed method is evaluated in exhaustive experiments, showing state-of-the-art results.