Overcoming Multi-model Forgetting

We identify a phenomenon , which we refer to as multi-model forgetting , that occurs when sequentially training multiple deep networks with partially-shared parameters ; the performance of previously-trained models degrades as one optimizes a subsequent one , due to the overwriting of shared parameters . To overcome this , we introduce a statistically-justified weight plasticity loss that regularizes the learning of a model ’ s shared parameters according to their importance for the previous models , and demonstrate its effectiveness when training two models sequentially and for neural architecture search . Adding weight plasticity in neural architecture search preserves the best models to the end of the search and yields improved results in both natural language processing and computer vision tasks . 1 INTRODUCTION . Deep neural networks have been very successful for tasks such as visual recognition ( Xie & Yuille , 2017 ) and natural language processing ( Young et al. , 2017 ) , and much recent work has addressed the training of models that can generalize across multiple tasks ( Caruana , 1997 ) . In this context , when the tasks become available sequentially , a major challenge is catastrophic forgetting : when a model initially trained on task A is later trained on task B , its performance on task A can decline calamitously . Several recent articles have addressed this problem ( Kirkpatrick et al. , 2017 ; Rusu et al. , 2016 ; He & Jaeger , 2017 ; Li & Hoiem , 2016 ) . In particular , Kirkpatrick et al . ( 2017 ) show how to overcome catastrophic forgetting by approximating the posterior probability , p ( θ | D1 , D2 ) , with θ the network parameters and D1 , D2 different datasets representing the tasks . In many situations one does not train a single model for multiple tasks but multiple models for a single task . When dealing with many large models , a common strategy to keep training tractable is to share a subset of the weights across the multiple models and to train them sequentially ( Pham et al. , 2018 ; Xie & Yuille , 2017 ; Liu et al. , 2018a ) . This strategy has a major drawback . Figure 1 shows that for two models , A and B , the larger the number of shared weights , the more the accuracy of A drops when training B ; B overwrites some of the weights of A and this damages the performance of A . We call this multi-model forgetting . The benefits of weight-sharing have been emphasized in tasks like neural architecture search , where the associated speed gains have been key in making the process practical ( Pham et al. , 2018 ; Liu et al. , 2018b ) , but its downsides remain virtually unexplored . In this paper we introduce an approach to overcoming multi-model forgetting . Given a dataset D , we first consider two models f1 ( D ; θ1 , θs ) and f2 ( D ; θ2 , θs ) with shared weights θs and private weights θ1 and θ2 . We formulate learning as the maximization of the posterior p ( θ1 , θ2 , θs|D ) . Under mild assumptions we show that this posterior can be approximated and expressed using a loss , dubbed Weight Plasticity Loss ( WPL ) , that minimizes multi-model forgetting . Our framework evaluates the importance of each weight , conditioned on the previously-trained model , and encourages the update of each shared weight to be inversely proportional to its importance . We then show that our approach extends to more than two models by exploiting it for neural architecture search . Our work is the first of which we are aware to propose a solution to multi-model forgetting . We establish the merits of our approach when training two models with partially shared weights and in the context of neural architecture search . For the former , we establish the effectiveness of WPL in the strict convergence case , where each model is trained until convergence , and in the more realistic loose convergence setting , where training is stopped early . WPL can reduce the forgetting effect by 99 % when model A converges fully , and by 52 % in the loose convergence case . For neural architecture search , we implement WPL within the efficient ENAS method of Pham et al . ( 2018 ) , a state-of-the-art technique that relies on parameter sharing and corresponds to the loose convergence setting . We show that , at each iteration , the use of WPL reduces the forgetting effect by 51 % on the most affected model and by 95 % on average over all sampled models . Our final results on the best architecture found by the search confirm that limiting multi-model forgetting yields better results and better convergence for both language modeling ( on the PTB dataset ( Marcus et al. , 1994 ) ) and image classification ( on the CIFAR10 dataset ( Krizhevsky et al. , 2009 ) ) . For language modeling the perplexity decreases from 65.01 for ENAS without WPL to 61.9 with WPL . For image classification WPL yields a drop of top-1 error from 4.87 % to 3.81 % . We also adapt our method to NAO ( Luo et al. , 2018 ) and show , in appendix due to space limitations , that multi-model forgetting is significantly reduced . We will make our code publicly available upon acceptance of this paper . 2 RELATED WORK . Single-model Forgetting . The goal of training a single model to tackle multiple problems is to leverage the structures learned for one task for other tasks . This has been employed in transfer learning ( Pan & Yang , 2010 ) , multi-task learning ( Caruana , 1997 ) and lifelong learning ( Silver et al. , 2013 ) . However , sequential learning of later tasks has visible negative consequences for the initial one . Kirkpatrick et al . ( 2017 ) selectively slow down the learning of the weights that are comparatively important for the first task by defining the importance of an individual weight using its Fisher information ( Rissanen , 1996 ) . He & Jaeger ( 2017 ) project the gradient so that directions relevant to the previous task are unaffected . Other families of methods save the older models separately to create progressive networks ( Rusu et al. , 2016 ) or use regularization to force the parameters to remain close to the values obtained by previous tasks while learning new ones ( Li & Hoiem , 2016 ) . In ( Xu & Zhu , 2018 ) , forgetting is avoided altogether by fixing the parameters of the first model while complementing the second one with additional operations found by an architecture search procedure . This work , however , does not address the multi-model forgetting that occurs during the architecture search . An extreme case of sequential learning is lifelong learning , for which the solution to catastrophic forgetting developed by Aljundi et al . ( 2018 ) is also to prioritize the weight updates , with smaller updates for weights that are important for previously-learned tasks . Parameter Sharing in Neural Architecture Search . In both sequential learning on multiple tasks and lifelong learning , the forgetfulness concerns an individual model . Here we tackle scenarios where one seeks to optimize a population of multiple models that share parts of their internal structure . The use of multiple models to solve a single task dates back to model ensembles ( Dietterich , 2000 ) . Recently , sharing weights between models that are candidate solutions to a problem has shown great promise in the generation of custom neural architectures , known as neural architecture search ( Elsken et al. , 2018 ) . Existing neural architecture search strategies mostly divide into reinforcement learning and evolutionary techniques . For instance , Zoph & Le ( 2017 ) use reinforcement learning to explore a search space of candidate architectures , with each architecture encoded as a string using an RNN trained with REINFORCE ( Williams , 1992 ) and taking validation performance as the reward . MetaQNN ( Baker et al. , 2017 ) uses Q-Learning to design CNN architectures . By contrast , neuro-evolution strategies use evolutionary algorithms ( Bäck , 1996 ) to perform the search . An example is Liu et al . ( 2018a ) , who introduce a hierarchical representation of neural networks and use tournament selection ( Goldberg & Deb , 1991 ) to evolve the architectures . Initial search solutions required hundreds of GPUs due to the huge search space , but recent efforts have made the search more tractable , for example via the use of neural blocks ( Negrinho & Gordon , 2017 ; Bennani-Smires et al. , 2018 ) . Similarly , and directly related to this work , weight sharing between the candidates has allowed researchers to greatly decrease the computational cost of neural architecture search . For neuroevolution methods , sharing is implicit . For example , Real et al . ( 2017 ) define weight inheritance as allowing the children to inherit their parents ’ weights whenever possible . For RL-base techniques , weight sharing is modeled explicitly and has been shown to lead to significant gains . In particular , ENAS ( Pham et al. , 2018 ) , which builds upon NAS ( Zoph & Le , 2017 ) , represents the search space as a single directed acyclic graph ( DAG ) in which each candidate architecture is a subgraph . EAS ( Cai et al. , 2018 ) also uses an RL strategy to grow the network depth or layer width with function-preserving transformations defined by Chen et al . ( 2016 ) where they initialize new models with previous parameters . DARTS ( Liu et al. , 2018b ) uses soft assignment to select paths that implicitly inherit the previous weights . NAO ( Luo et al. , 2018 ) replaces the reinforcement learning portion of ENAS with a gradient-based auto-encoder that directly exploits weight sharing . While weight sharing has proven effective , its downsides have never truly been studied . Bender et al . ( 2018 ) realized that training was unstable and proposed to circumvent this issue by randomly dropping network paths . However , they did not analyze the reasons underlying the training instability . Here , by contrast , we highlight the underlying multi-model forgetting problem and introduce a statistically-justified solution that further improves on path dropout . 3 METHODOLOGY . In this section we study the training of multiple models that share certain parameters . As discussed above , training the multiple models sequentially as in Pham et al . ( 2018 ) , for example , is suboptimal , since multi-model forgetting arises . Below we derive a method to overcome this for two models , and then show how our formalism extends to multiple models in the context of neural architecture search , and in particular within ENAS ( Pham et al. , 2018 ) . 3.1 WEIGHT PLASTICITY LOSS : PREVENTING MULTI-MODEL FORGETTING . Given a dataset D , we seek to train two architectures f1 ( D ; θ1 , θs ) and f2 ( D ; θ2 , θs ) with shared parameters θs and private parameters θ1 and θ2 . We suppose that the models are trained sequentially , which reflects common large-model , large-dataset scenarios and will facilitate generalization . Below , we derive a statistically-motivated framework that prevents multi-model forgetting ; it stops the training of the second model from degrading the performance of the first model . We formulate training as finding the parameters θ = ( θ1 , θ2 , θs ) that maximize the posterior probability p ( θ | D ) , which we approximate to derive our new loss function . Below we discuss the different steps of this approximation , first expressing p ( θ | D ) more conveniently . Lemma 1 . Given a dataset D and two architectures with shared parameters θs and private parameters θ1 and θ2 , and provided that p ( θ1 , θ2 | θs , D ) = p ( θ1 | θs , D ) p ( θ2 | θs , D ) , we have p ( θ1 , θ2 , θs | D ) ∝ p ( D | θ2 , θs ) p ( θ1 , θs ) p ( θ2 , θs ) ∫ p ( D | θ1 , θs ) p ( θ1 , θs ) dθ1 . ( 1 ) Proof . Provided in the appendix . Lemma 1 presupposes that p ( θ1 , θ2 | θs , D ) = p ( θ1 | θs , D ) p ( θ2 | θs , D ) , i.e. , θ1 and θ2 are conditionally independent given θs and the dataset D. While this must be checked in applications , it is suitable for our setting , since we want both networks , f1 ( D ; θ1 , θs ) and f2 ( D ; θ2 , θs ) , to train independently well . To derive our loss we study the components on the right of equation ( 1 ) . We start with the integral in the denominator , for which we seek a closed form . Suppose we have trained the first model and seek to update the parameters of the second one while avoiding forgetting . The following lemma provides an expression for the denominator of equation ( 1 ) . Lemma 2 . Suppose we have the maximum likelihood estimate ( θ̂1 , θ̂s ) for the first model , write Card ( θ1 ) + Card ( θs ) = p1 + ps = p , and let the negative Hessian Hp ( θ̂1 , θ̂s ) of the log posterior probability distribution log p ( θ1 , θs | D ) evaluated at ( θ̂1 , θ̂s ) be partitioned into four blocks corresponding to ( θ1 , θs ) as Hp ( θ̂1 , θ̂s ) = [ H11 H1s Hs1 Hss ] . If the parameters of each model follow Normal distributions , i.e. , ( θ1 , θs ) ∼ Np ( 0 , σ2Ip ) , with Ip the p-dimensional identity matrix , then∫ p ( D | θ1 , θs ) p ( θs , θ1 ) dθ1 = exp { lp ( θ̂1 , θ̂s ) − 1 2 v > Ωv } × ( 2π ) p1/2|det ( H−111 ) |1/2 , ( 2 ) where v = θs − θ̂s and Ω =Hss −H > 1sH−111 H1s . Proof . Provided in the appendix . Lemma 2 requires the maximum likelihood estimate ( θ̂1 , θ̂s ) , which can be hard to obtain with deep networks , since they have non-convex objective functions . In practice , one can train the network to convergence and treat the resulting parameters as maximum likelihood estimates . Our experiments show that the parameters obtained without optimizing to convergence can be used effectively . Moreover Haeffele & Vidal ( 2017 ) showed that networks relying on positively homogeneous functions have critical points that are either global minimizers or saddle points , and that training to convergence yields near-optimal solutions , which correspond to true maximum likelihood estimates . Following Lemmas 1 and 2 , as shown in the appendix , log p ( θ | D ) ∝ log p ( D | θ2 , θs ) + log p ( θ2 , θs ) + log p ( θ1 , θs | D ) + 1 2 v > Ωv . ( 3 ) To derive a loss function that prevents multi-model forgetting , consider equation ( 3 ) . The first term on its right-hand side corresponds to the log likelihood of the second model and can be replaced by the cross-entropy L2 ( θ2 , θs ) , and if we use a Gaussian prior on the parameters , the second term encodes an L2 regularization . Since equation ( 3 ) depends only on the log likelihood of the second model f2 ( D ; θ2 , θs ) , the information learned from the first model f1 ( D ; θ1 , θs ) must reside in the conditional posterior probability log p ( θ1 , θs | D ) , and the final term , 12v > Ωv , must represent the interactions between the models f1 ( D ; θ2 , θs ) and f2 ( D ; θ1 , θs ) . This term will not appear in a standard single-model forgetting scenario . Let us examine these terms more closely . The posterior probability p ( θ1 , θs | D ) is intractable , so we apply a Laplace approximation ( MacKay , 1992 ) ; we approximate the log posterior using a second-order Taylor expansion around the maximum likelihood estimate ( θ̂1 , θ̂s ) . This yields log p ( θ1 , θs | D ) = log p ( θ̂1 , θ̂s | D ) − 1 2 [ ( θ1 , θs ) − ( θ̂1 , θ̂s ) ] > Hp [ ( θ1 , θs ) − ( θ̂1 , θ̂s ) ] , ( 4 ) where Hp ( θ̂1 , θ̂s ) is the negative Hessian of the log posterior evaluated at the maximum likelihood estimate . As the first derivative is evaluated at the maximum likelihood estimate , it equals zero . Equation ( 4 ) yields a Gaussian approximation to the posterior with mean ( θ̂1 , θ̂s ) and covariance matrixH−1p , i.e. , p ( θ1 , θs | D ) ∝ exp { − 1 2 [ ( θ1 , θs ) − ( θ̂1 , θ̂s ) ] > Hp [ ( θ1 , θs ) − ( θ̂1 , θ̂s ) ] } . ( 5 ) Our parameter space is too large to compute the inverse of the negative HessianHp , so we replace it with the diagonal of the Fisher information , diag ( F ) . This approximation falsely presupposes that the parameters ( θ1 , θs ) are independent , but it has already proven effective ( Kirkpatrick et al. , 2017 ; Pascanu & Bengio , 2014 ) . One of its main advantages is that we can compute the Fisher information from the squared gradients , thereby avoiding any need for second derivatives . Using equation ( 5 ) and the Fisher approximation we can express the log posterior as log p ( θ1 , θs | D ) ∝ α 2 ∑ θsi∈θs Fθsi ( θsi − θ̂si ) 2 , ( 6 ) where Fθsi is the diagonal element corresponding to parameter θsi in the diagonal approximation of the Fisher information matrix , which can be obtained from the trained model f1 ( D ; θ1 , θs ) . Now consider the last term in equation ( 3 ) , noting that Ω = Hss −H > 1sH−111 H1s , as defined in Lemma 2 . As our previous approximation relies on the assumption of a diagonal Fisher information matrix , we haveH1s = 0 , leading to Ω =Hss , so 1 2 v > Ωv = 1 2 ∑ θsi∈θs Fθsi ( θsi − θ̂si ) 2 . ( 7 ) The last two terms on the right-hand side of equation ( 3 ) , as expressed in equation ( 6 ) and equation ( 7 ) , can then be grouped . Combining the result with the first two terms , discussed below equation ( 3 ) , yields our Weight Plasticity Loss , LWPL ( θ2 , θs ) = L2 ( θ2 , θs ) + λ 2 ( ‖θs‖2 + ‖θ2‖2 ) + α 2 ∑ θsi∈θs Fθsi ( θsi − θ̂si ) 2 , ( 8 ) where Fθsi is the diagonal element corresponding to parameter θsi in the Fisher information matrix obtained from the trained first model f1 ( D ; θ1 , θs ) . We omit the terms depending on θ1 in equation ( 6 ) because we are optimizing with respect to ( θ2 , θs ) at this stage . The Fisher information in the last term encodes the importance of each shared weight for the first model ’ s performance , so WPL encourages preserving any shared parameters that were important for the first model , while allowing others to undergo larger changes and thus to improve the accuracy of the second model .

Summary: This paper discusses the phenomena of “neural brainwashing”, which refers to that the performance of one model is affected via another model sharing model parameters. To solve the issue, the authors derived a new loss out from maximizing the posterior of the parameters. With the new loss, the neural brainwashing is largely diminished.