Uncertainty-guided Lifelong Learning in Bayesian Networks

Sequentially learning of tasks arriving in a continuous stream is a complex problem and becomes more challenging when the model has a fixed capacity . Lifelong learning aims at learning new tasks without forgetting previously learnt ones as well as freeing up capacity for learning future tasks . We argue that identifying the most influential parameters in a representation learned for one task plays a critical role to decide on what to remember for continual learning . Motivated by the statistically-grounded uncertainty defined in Bayesian neural networks , we propose to formulate a Bayesian lifelong learning framework , BLLL , that addresses two lifelong learning directions : 1 ) completely eliminating catastrophic forgetting using weight pruning , where a hard selection mask freezes the most certain parameters ( BLLL-PRN ) and 2 ) reducing catastrophic forgetting by adaptively regularizing the learning rates using the parameter uncertainty ( BLLL-REG ) . While BLLL-PRN is by definition a zero-forgetting guaranteed method , BLLL-REG , despite exhibiting some small forgetting , is a task-agnostic lifelong learner , which does not require to know when a new task arrives . This feature makes BLLL-REG a more convenient candidate for applications such as robotics or on-line learning in which such information is not available . We evaluate our Bayesian learning approaches extensively on diverse object classification datasets in short and long sequences of tasks and perform superior or marginally better than the existing approaches . 1 INTRODUCTION . Humans can easily accumulate and maintain the knowledge gained from previously observed tasks and continuously learn to solve new problems or tasks . Inspired by this capability of humans , Lifelong Learning ( LLL ) intends to learn sequential tasks without forgetting Unlike the powerful capabilities observed in mammalian brains in acquiring and preserving the knowledge , artificial agents still struggle in continual learning . One of the main challenges for a lifelong learning agent is known as catastrophic forgetting ( McCloskey & Cohen , 1989 ; McClelland et al. , 1995 ) which refers to the significant drop in the performance of a learner when switching from a trained task to a new one . This phenomenon occurs because trained parameters on initial task require to change in favor of learning new objectives . This is the reason that naive finetuning intuitively suffers the most from catastrophic forgetting . Given the limited capacity of a network , one way to see this problem as to deciding which part of the network is best to keep and which part is less important and can thus be used by future tasks without significantly affecting current task performance . In this work , we propose to turn to Bayesian Networks as introduced by Blundell et al . ( 2015 ) . These networks represent each parameter with a distribution defined by a mean and variance which more naturally allows determining which parameters are most important for a task . We integrate this idea with two different LLL paradigms : Hard pruning of part of the network and soft regularization based importance weighting of network parameters . To avoid adding capacity when learning new tasks , one can first prune or compress the network ( Aghasi et al. , 2017 ) and then learn new tasks using free parameters as proposed by ( Mallya & Lazebnik , 2017 ; 2018 ) . Here a part of the network is frozen , while the remaining network is free to adapt for future tasks . While this eliminates forgetting over time , it reduces flexibility when learning large number tasks as the network fills up over time as the percentage which is frozen by previous tasks increases over time . In non-Bayesian lifelong learning approaches , model parameters tend to overconfidently fit training data and there is no internal definition of weight importance . In contrast , Bayesian models , enhanced by their statistically-grounded theoretical framework , provide clear definitions of uncertainty in model weights and/or predictions . Although exact Bayesian inference remains intractable in neural networks , variational approximations have provided a powerful alternative to this problem . In this work we exploit an efficient back-propagation-compatible Bayesian formulation in neural networks , called Bayes-by-Backpop ( BBB ) Blundell et al . ( 2015 ) , to obtain weights ’ uncertainty as a guidance to perform continual learning . The main intuition behind is that Bayesian neural network provide precise uncertainty measurements in weights as a by-product which can be used as an accurate notion of importance to perform lifelong learning . 2 RELATED WORK . Here we list some of the previously introduced approaches taken in lifelong learning that mitigate catastrophic forgetting to different extents . Conceptually , these approaches can be divided into the following categories . We list pros and cons for each and explain why ( not ) our approach is inspired or build upon them . Dynamic architectural methods : In this setting , the architecture will grow such that it carries over the past knowledge as well as storing new gained knowledge in different forms such as additional layers , nodes , or dual-architecture . In this approach , the objective function remains fixed whereas the model capacity grows often exponentially with the number of tasks . Progressive networks Rusu et al . ( 2016 ) ; Schwarz et al . ( 2018 ) was one of the earliest work in this path which was successfully applied on reinforcement learning problems where the base architecture is duplicated and lateral connections are added in response to new tasks . Roy et al . ( 2018 ) also uses hierarchical network , where CNN ’ s at multiple levels , could grow in tree-shape manner to accommodate new tasks . Dynamically Expandable Network ( DEN ) Yoon et al . ( 2018 ) also expands the network by selecting the drifting units and retraining them separately based on the current tasks in hand . Despite the ability of these methods in preventing forgetting , they have large overheads upon arrival of new tasks . We have completely avoided using architectural growth in our method due to this reason . Memory-based methods : In this regime , previous information is partially stored in different fashions to be used later as a form of rehearsal Robins ( 1995 ) . iCaRL Rebuffi et al . uses this idea to perform class incremental learning through nearest-mean-of-exemplars using current data and stored examples of previously seen classes which was shown to scale up to incrementally learning ImageNet . However , this method can not be used in learning multiple sequential tasks from different datasets . Gradient episodic memory ( GEM ) ( Lopez-Paz et al. , 2017 ) was another work for class incremental learning where data is stored at the end of each episode to be used later to restrict gradient updates from deviating away from their previous values . GEM is not only able to mitigate forgetting but also allows for positive backward knowledge transfer which is referred to improvement on previously learned tasks . Another value of this work was that it was the first method capable of learning with seeing each training example only once . Since storing examples can be costly and sometimes unpractical , a generative model was employed to encode/decode pseudo information Shin et al . ( 2017 ) ; Nguyen et al . ( 2017 ) . However , in practice , generative models are difficult to train especially in sequential learning scenarios . Regularization methods : These approaches do not require storing data while retaining the whole network . Catastrophic forgetting is alleviated via preventing significant changes to representation learned for previous tasks . This can be performed through regularizing the objective function or directly applied on weight parameters such that important weights to one task are not allowed to change in the future . Typically , this importance measure is engineered to represent the importance of each parameter . Inspired by Bayesian learning , elastic weight consolidation ( EWC ) method ( Kirkpatrick et al. , 2017 ) important parameters are those to have the highest in terms if the Fisher information metric . In Synaptic Intelligence ( SI ) ( Zenke et al. , 2017 ) this parameter importance notion is engineered to correlate with the loss function , parameters that contribute more to the loss are more important . Similar to SI ( Zenke et al. , 2017 ) , Memory-aware Synapses ( MAS ) Aljundi et al . ( 2017 ) proposed an online way of computing importance adaptive to the test set using the change in the model outputs w.r.t the inputs . PackNet Mallya & Lazebnik ( 2017 ) used iterative pruning to fully restrict gradient updates on important weights via saving binary masks . While applying this binary mask at inference results in zero forgetting of the initial performance , it requires to know which task we are performing to use the appropriate mask . PackNet also ranks the weight importance by their magnitude which is not guaranteed to be a proper importance indicative . HAT ( Serrà et al. , 2018 ) identifies important neuron using an attention vector to the task embedding to control the gradient flow through neurons . 3 BACKGROUND : VARIATIONAL BAYES-BY-BACKPROP . In this section , we review the Bayesian framework we use to be able to learn sequential tasks by remembering task-specific parameters . The Bayesian formulation used here was first introduced by Blundell et al . ( 2015 ) that learns the probability distribution over network parameters . In the original paper , it was shown that this novel back-propagation-compatible algorithm acts as a regularizer and yields comparable performance to dropout on MNIST dataset . In the next section ( Section 4 ) we show how to use the derived weight uncertainty to measure the importance in performing the task in hand and use the rest of the network to learn later tasks with . 3.1 VARIATIONAL INFERENCE IN NEURAL NETWORKS . In Bayesian models , latent variables are drawn from a prior density p ( w ) which are related to the observations through the likelihood p ( x|w ) . During the inference , the posterior distribution p ( w|x ) is computed conditioned on the given input data . However , in practice , this probability distribution is intractable and is often estimated through approximate inference . Markov chain Monte Carlo ( MCMC ) sampling ( Hastings , 1970 ) has been widely used and explored for this purpose ( see Robert & Casella ( 2013 ) for different methods under this category ) . However , MCMC algorithms , despite providing guarantees in finding asymptotically exact samples from the target distribution , they are not suitable for large datasets and/or large models as they are bounded by speed and scalability issues . Alternatively , Variational inference provides a faster solution to the same problem in which the posterior is approximated using optimization rather than being sampled from a chain Peterson ( 1987 ) ; Hinton & Van Camp ( 1993 ) ; Jaakkola & Jordan ( 1996 ; 1997 ) . Similar to any other approximation algorithms , there will be trade-offs between the precision of the results and the efficiency of the optimization . However , variational inference methods always take advantage of fast optimization techniques such as stochastic methods , distributed methods , which allows them to explore data models faster to find the best approximation . See Blei et al . ( 2017 ) for a complete review of the theory . 3.2 BAYES BY BACKPROP . Let x ∈ Rn be a set of observed variables and w be a set of latent variables parametrized by θ sharing a joint density of p ( w , x ) . Given the observations x and by using the set of weight parameters w , a neural network , as a probabilistic model , p ( y|x , w ) can output a probability associated with a possible output y ∈ Y . Variational inference aims to calculate this conditional probability distribution over the latent variables by finding the closest proxy to the exact posterior through solving an optimization problem . We first propose a family D of probability densities over the latent variables ( q ( w|θ ) ∈ D ) . We then find the closest member of this family to the true conditional probability of interest p ( w|x ) by minimizing the Kullback-Leibler ( KL ) divergence between q and p : q∗ ( w|θ ) = argmin q ( w|x ) ∈D KL ( q ( w|θ ) ‖p ( w|x ) ) ( 1 ) Once solved , q ( · ) would be the closest approximation to variational posterior parametrized by θ∗ to construct the parameters of the network . We can unpack Eq . 1 and call it a loss function commonly known as variational free energy or expected lower bound : LBBB ( w , θ ) = KL [ q ( w|θ ) ‖p ( w ) ] − Eq ( w|θ ) [ log ( p ( x|w ) ] ( 2 ) Blundell et al . ( 2015 ) showed that Eq . 2 can be approximated using i Monte Carlo samples from the variational posterior : LBBB ( w , θ ) ≈ n∑ i=1 log q ( w ( i ) |θ ) − logp ( w ( i ) ) − log p ( x|w ( i ) ) ( 3 ) We assume q ( w|θ ) to have a Gaussian pdf with diagonal covariance . θ is parametrized point-wise with a mean µ and standard deviation of σ = log ( 1 + exp ( ρ ) ) . Variational posterior parameters will have a form of µ + log ( 1 + exp ( ρ ) ) ◦ where is a sample drawn from posited q . For the prior , as it was suggested in Blundell et al . ( 2015 ) , a scale mixture of two Gaussian pdfs are chosen which are zero-centered while having difference variances of σ21 and σ 2 2 . The uncertainty obtained for every weight parameter , as opposed to hidden units Rezende et al . ( 2014 ) ; Kingma & Welling ( 2013 ) has been successfully used in model compression Blundell et al . ( 2015 ) , uncertainty-based exploration in reinforcement learning application Blundell et al . ( 2015 ) . In this work we want to use this framework to learn sequential tasks without forgetting using per-weight derived uncertainties . 4 LIFELONG LEARNING USING BAYES BY BACKPROP ( BLLL ) In our method called , Bayesian Lifelong Learning ( BLLL ) , we have considered two common scenarios of lifelong learning : 1 ) sequential learning of multiple datasets ; 2 ) class-incrementally learning of a single or two alternating datasets . Having uncertainty measured precisely for mean and variances of our posterior distributions , one can choose two paths to perform continual learning ; 1 ) catastrophic forgetting elimination and 2 ) catastrophic forgetting mitigations where in the former , called BLLL-PRN , important weights are kept frozen using a binary pruning mask saved per task whereas , in the latter called BLLL-REG , important parameters are regularized using an adaptive learning rate during gradient update in which the a learning rate of a parameter scales with its uncertainty . We will explain BLLL-PRN and BLLL-REG in details in section 4.1 and 4.2 . 4.1 BLLL WITHOUT FORGETTING USING WEIGHT PRUNING ( BLLL-PRN ) Unlike regular deep neural networks , in BBB model , weight parameters are represented by probability distributions parametrized by their mean and standard deviation . In order to take into account both mean and variance , we use the quantity of signal-to-noise ratio ( SNR ) for each parameter defined as SNR = |µ| σ ( 4 ) SNR is a commonly used measure in signal processing to distinguish between “ useful ” information from unwanted noise contained in a signal . In the context of neural models , SNR can be thought as an indicative of parameter importance ; the higher the SNR , the more effective or important the parameter is to the model predictions for a given task . Weight pruning is a long standing solution to reduce inference computation or network compression ( Liu et al. , 2017 ; Molchanov et al. , 2016 ) . Moreover , it has been recently used to perform lifelong learning ( Mallya & Lazebnik , 2017 ) , where the goal is to sequentially learning multiple tasks using a single network ’ s capacity . In ( Mallya & Lazebnik , 2017 ) this has been accomplished by freeing up parameters deemed to be unimportant to the current task according to their magnitude . Forgetting is prevented in pruning by saving a task-specific binary mask of important vs. unimportant parameters . Our work is significantly different from ( Mallya & Lazebnik , 2017 ) in the different criterion we propose to use for importance measurement ; the statistically-grounded uncertainty defined in Bayesian neural networks . BLLL-PRN is performed as follows : for every layer , convolutional or fully-connected , the parameters are ordered by their SNR value and those with the lowest importance are pruned ( set to zero ) . The pruned parameters are “ saved ” using a binary mask so that they can be used later in learning new tasks whereas the important parameters remain fixed throughout training on future tasks . Once a task is learned , an associated binary mask is saved which will be used at inference to recover key parameters to the desired task . Algorithm 1 Bayesian Lifelong Learning with Zero Forgetting ( BLLL-PRN ) or Reduced Forgetting ( BLLL-REG ) Require : hyper parameters for BBB-only : σ , σ1 , σ2 , ρ , µ , π , n Require : hyper parameters for training : learning rate α 1 : ∼ N ( 0 , I ) 2 : σ = log ( 1 + exp ( ρ ) ) . Ensures σ is always positive 3 : w = t ( θ , ) = µ+ σ ◦ . w : = a posterior sample of weights 4 : while not done do 5 : l1 ← ∑n i=1 logN ( wi|µ , σ2 ) . log-posterior 6 : l2 ← ∑n i=1 log ( πN ( wi | 0 , σ21 ) + ( 1− π ) N ( wi | 0 , σ22 ) ) . log prior 7 : l3 ← ∑n i=1 log ( p ( x|w ) ) . log-likelihood of the data 8 : LBBB ← 1M ( l1 − l2 ) − l3 . M : = number of minibatches 9 : if BLLL-PRN then 10 : SNRi ← |µi|σi . signal-to-noise ratio for every weight 11 : for all layers do 12 : for all parameters in layer do 13 : if SNRi ∈ Top p portion of SNR values in layer then 14 : α = 0 15 : µ′ ← µ− α∇LBBBµ . Update parameter using gradient descent 16 : ρ′ ← ρ− α∇LBBBρ . Update parameter using gradient descent 17 : if BLLL-REG then 18 : β ← σ × α 19 : µ′ ← µ− β∇LBBBµ 20 : ρ′ ← ρ− α∇LBBBρ . Update the remaining uniformly using gradient descent Similar to ( Mallya & Lazebnik , 2017 ) the overhead memory per parameter in encoding the mask as well as saving it on the disk is as follows . Assuming we have n tasks to learn using a single network , the total number of required bits to encode an accumulated mask for a parameter is at max log2 n bits assuming a parameter deemed to be important from task 1 and kept being encoded in the mask . Saving the binary mask for our a typical model with n tasks results in a mask size of 1/n2 with respect to the initial size . 4.2 BLLL WITH REDUCED FORGETTING ( BLLL-REG ) Another strategy to perform continual learning is to reduce forgetting by controlling further changes in a representation learned for a task . This can be enforced in a form of regularization in the objective function or through a varying learning rate for weight parameters . In order to prevent drastic changes to the predicted posterior for a learned task , we propose to condition the changes in the mean of each parameter ’ s distribution on its uncertainty i.e. , the more certain the mean of a distribution is , the less it should be allowed to be updated to learn future concepts with . We impose this through scaling the learning rate for µi with its uncertainty represented by σi . The intuition behind this is to develop a gradient regularizer that adjust the gradient updates using the existing uncertainty in the means . In other words , we wish to prevent any further changes on the certain mean by lowering their learning rate at each gradient update while its variance is allowed to change . This results in allowing the model learning more concepts while preserving the critical information obtained in the past . The key benefit of BLLL-Reg variant using learning rate as the regularizer is that it neither requires additional memory , as opposed to pruning technique nor tracking the change in parameters with respect to the previous learned task , as needed in common weight regularization methods . More importantly , this method does not need to be aware of task switching as it only needs to adjust the learning rates of the means in the posterior distribution based on their current standard deviations . Complete algorithm for BLLL with pruning ( BLLL-PRN ) and adaptive learning rate ( BLLL-REG ) . is shown in Algorithm 1 .

Summary: In this paper, a framework for lifelong learning based on Bayesian neural network is proposed. The key idea is to combine iterative pruning for multi-task learning along with the weight regularization. The idea of iterative pruning was first considered by Mallya et al., 2018 and weight regularization was considered for Bayesian neural network by Nguyen et al., 2018.