On Plasticity, Invariance, and Mutually Frozen Weights in Sequential Task Learning

1 Introduction . Lifelong learning [ 1 , 2 ] holds both the promise of benefitting from past experiences and the challenge of having to continually adapt to new problem settings . The characteristics of this intriguing sequential learning problem make it a balancing act between retaining knowledge and adapting to new experiences referred to as the stability-plasticity dilemma [ 3 ] . Similar to a coach or teacher , we want to understand when a sequence of tasks helps or hinders further learning . Clearly , some pre-training tasks are highly beneficial [ 4–7 ] while others can hurt performance [ 8 , 9 ] . The existing literature on pre-training and curriculum learning sheds some light on the desired characteristics of sequential learning . In some cases , following a sequence of learning tasks can lead to better results than simply training on the target task from scratch [ 10 ] . Work by Achille and Soatto [ 8 ] contrasts this picture by highlighting that neural networks can suffer from pre-training on some tasks . In an extreme case , when learning data is corrupted for a sufficiently long period , even switching back to clean data does not allow the neural network to recover its original performance . Contribution : We formalize the basic ingredients of sequential task learning , introduce mutually frozen weights and establish a connection between plasticity and invariance of deep networks . To understand mutually frozen weights , we investigate their weight update dynamics , show that such frozen weights are different from regular sparse weights that are not connected to other sparse weights and that frozen weights can lead to a decrease in performance when retraining on a new task . Based on this analysis , we show that applying an intervention which “ resets ” frozen weights can improve retraining performance on a number of tasks as long as sufficiently many samples are available for retraining and even achieve state-of-the-art results on FashionMNIST image classification . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 2 Related work . We call the problem setting in this work optimal task tracking which involves training a model on a sequence of tasks to improve performance on a final target task . This strongly resembles the objectives of lifelong learning [ 1 ] , continual learning [ 11 ] and meta-learning [ 12–14 ] and connects to many time-dependent learning aspects in relation to neural networks . Time-dependent learning : We are aware of several time-dependent phenomena in training neural networks such as catastrophic forgetting [ 15 ] , critical learning periods [ 8 ] , and time-sensitivity of regularization and data augmentation [ 16 ] . Achille et al . [ 8 ] show that by dropping out certain frequencies during the beginning of training , networks are unable to recover to full performance , even when those frequencies are reintroduced . Similarly , Liu et al . [ 17 ] show that first optimizing on random labels ruins performance on clean data shown indefinitely to the same network afterwards . Recent information-theoretic analysis [ 18 ] suggests that neural network have distinct learning phases : First a phase where network parameters grow in information content , before , in a second phase , self-regularizing and pruning away irrelevant information . More commonly known time-dependent or sequential heuristics are learning rate schedules [ 19 ] which may even be cyclical [ 20 ] , regularization annealing such as dropout annealing [ 21 ] , student-teacher model transfer [ 22 ] , and pre-training [ 23 ] . Finally , our introduced concept of mutually frozen weights also touches upon the lottery ticket hypothesis [ 24 ] and may help explain some of its underlying gradient dynamics . Curriculum learning and generation : Curriculum learning [ 25 , 10 ] is the technique of training agents by showing them a progression of tasks from easier variants to harder examples to attempt to solve the final target task . Both the utility of transferring between similar or dissimilar tasks is debated , with some results pointing towards similarity [ 26–28 ] and others towards dissimilarity [ 29 , 30 ] . Metalearning [ 13 , 31 ] is similarly related by focusing on speeding up or enhancing performance on a target task by pre-training on other tasks to first learn to learn . Unlike curriculum learning , curriculum generation seeks to find a path through task space , generating an agent that can solve some final , held-out task [ 32–36 ] . Invariance-plasticity-generalization : Empirically , Mehta et al . [ 37 ] showed that ReLU networks with batch normalization trained with gradient descent lead to filter-level sparsity . Evci et al . [ 38 ] demonstrate that training sparse networks usually leads to worse outcomes than without sparsity . On the other hand , Bartoldson et al . [ 39 ] showed that increased sparsity through pruning can lead to better generalization . Bringing sparsity emergence and difficulty of training together , we will demonstrate how certain types of sparsity can lead to problems in retraining a model after pre-training in the context of mutually frozen weights . From a theoretical viewpoint , Neyshabur et al . [ 40 , 41 ] connect the path norm of a network to its generalization via Rademacher bounds , where the path norm is based on the sum across all paths of the product of weights associated to individual paths . 3 Optimal task tracking . We are interested in better understanding what influences the outcome of an optimal task tracking problem , i.e . given a fixed task sequence how do we train a model to optimize the performance on the final target task . To clarify our further writing , we summarize essential elements involved in sequential learning and visualize the overall setting in Fig . 1 . A task ⌧ consists of a probability distribution p ( x , y ) of data points and desired outputs as well as a loss function L : ( p , f ) 7 ! R+ that maps an input-output behavior of a model to a cost score . The learning mapping ! : ( f , ⌧ ) 7 ! fnew , e.g . stochastic gradient descent , uses the information in task ⌧ and initial model finit to train a model flearn optimized for the task until convergence . The tasks and learning mappings ( ⌧i , ! i ) 2 T are applied sequentially starting from an initial model finit to produce the final target model ftarget to be evaluated on the target task ⌧target . 4 Plasticity and frozen weights . With the basic building blocks in place , we ask in which cases , even with an infinite number of data points , previous tasks can hinder learning on new tasks . We are interested in understanding what causes a model to be more difficult to adapt to a new task than another . This is evidently the case when some of the model weights ✓ can not be updated when training on the new task even when this would yield a better model . Based on this reasoning , we introduce the concept of frozen weights , i.e . weights that are unchanged when training on a learning task s.t . plasticity is clearly lacking . Definition 1 ( Frozen weights ) . We define a frozen weight ✓frozen of model f as a parameter that is not changed through learning ! on task ⌧ . ✓frozen ( f ) = ✓frozen ( ! ( ⌧ , f ) ) Associated to frozen weights , plasticity captures how well a model can adapt to a new task . Definition 2 ( Plasticity ) . We define plasticity as the rate of change of the model parameters subject to gradient descent updates P : = r✓L ( f ( ✓ , x ) , y ) . The definitions of frozen weights and plasticity beg the question in which scenarios frozen weights can appear . A promising candidate for such a constellation are mutually frozen weights . Definition 3 ( Mutually frozen weights ) . For a model f ( ✓ ; x ) with parameters ✓ and input x , we consider n non-overlapping sets of parameters { ✓i } as mutually frozen if , for each set , all parameters are zero and if each parameter in each set is exclusively multiplied with all other sets of parameters . We call such parameters n-sparse frozen weights . As the leading example , weights ✓i , ✓j , ✓k 2 R are triple-sparse mutually frozen if , first , the weights ✓i , ✓j , ✓k only appear multiplied together within the model as ✓ijk : = ✓i · ✓j · ✓k s.t . f ( ✓ ; x ) = f ( ✓ , ✓ijk ; x ) , where ✓ refers to all parameters except ✓i , ✓j , ✓k and , secondly if ✓i = ✓j = ✓k = 0 . Given a loss L ( ✓ ; x ) ! R+ we see that the gradient of ✓i is proportional to the remaining weights ✓j · ✓k = 0 : @ L ( ✓ ; x ) @ ✓i = @ L ( ✓ ; x ) @ f ( ✓ ; x ) @ f ( ✓ ; x ) @ ✓ijk @ ✓ijk @ ✓i = @ L ( ✓ ; x ) @ f ( ✓ ; x ) @ f ( ✓ ; x ) @ ✓ijk ✓j✓k Therefore with condition ✓i = ✓j = ✓k = 0 and assuming that the partial gradients L ( ✓ ; x ) @ f ( ✓ ; x ) and @ f ( ✓ ; x ) @ ✓ijk are bounded , the gradient of the mutually frozen weights ✓i , ✓j , ✓k weights are zero . We note that mutually frozen weights describe n-tuples ( n > 1 ) that “ freeze together ” . As depicted in Fig . 2 , we will focus on double-sparse and triple-sparse frozen weights in the remainder of the text . Of course also higher-order freezing involving n-tuples are possible yet do not necessarily contribute further insights . Likewise we deem this less likely to occur in neural network architectures since the influence of individual parameters spreads much further when involving more than three layers . Example 1 ( Linear regression ) . Linear regression does not have “ mutually frozen weights ” . Given f ( ✓ ; x ) = ✓Tx + b , y , b 2 R , x , ✓ 2 Rn , we have @ f ( ✓ ; x ) @ ✓ = x . Hence , unsurprisingly , no weights are multiplied with other weights in a way that could create mutually frozen weights . Thus we would not expect linear regression to incur a loss in plasticity in a sequential task learning setup . Example 2 ( Neural networks ) . A simple example of triple-sparse mutually frozen weights appears for the model f ( ✓i , ✓j , ✓k ; xi ) = ✓i✓j✓kx with x , ✓i , ✓j , ✓k 2 R. In this case , one can readily see that if ✓i = ✓j = ✓k = 0 then the gradients for all weights will also be zero . This can be regarded as a sub-part of a larger piece-wise linear neural network which will not be further trained due to mutually frozen weights . Note that with 3 or more weights multiplied as shown , the zero gradient is locally stable ( see Sec . 4.1 ) . The mutually frozen weights condition is particularly relevant for deep neural networks which consist of many pairwise multiplications of weights . In practice paired weights need not be zero exactly but simply have very small absolute values compared to the remaining weights . Other situations of training problems due to insufficient gradients exist such as the vanishing gradient problem [ 42 ] , wherein the magnitude of the gradient is reduced as it progresses through the network due to small weight norms . This problem however appears to be largely solved by methods such as identity connections such as in residual networks [ 43 ] and normalizations such as batch normalization [ 44 ] . From afar it might appear that the answer to the frozen weights problem is simply to avoid them , i.e . to ensure through training a model to not end up with frozen weights . However as we will document in the following , weight decay , which can cause frozen weights , is often connected to better generalization and thus makes frozen weights non-trivial to avoid . This naturally causes a curious trade-off between how we first train a model on a pre-training and subsequently on a target task .

Summary: The paper studies the "optimal task tracking" problem under the training setting where we use the "weight decay" regularization. More specifically, the paper studies the relationship between plasticity-stability tradeoff and shows that a consequence of training with (large enough) weight decay is the "mutually frozen" weights that can damage the plasticity of the network. Moreover, the paper introduces an intervention to reset the mutually frozen weights and improve the performance.