Sliced Cramer Synaptic Consolidation for Preserving Deeply Learned Representations

Deep neural networks suffer from the inability to preserve the learned data representation ( i.e. , catastrophic forgetting ) in domains where the input data distribution is non-stationary , and it changes during training . Various selective synaptic plasticity approaches have been recently proposed to preserve network parameters , which are crucial for previously learned tasks while learning new tasks . We explore such selective synaptic plasticity approaches through a unifying lens of memory replay and show the close relationship between methods like Elastic Weight Consolidation ( EWC ) and Memory-Aware-Synapses ( MAS ) . We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network ’ s output at an arbitrary layer for previous tasks while learning a new one . We propose the sliced Cramér distance as a suitable choice for such preservation and evaluate our Sliced Cramér Preservation ( SCP ) algorithm through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings . We show that SCP consistently utilizes the learning capacity of the network better than online-EWC and MAS methods on various incremental learning tasks . 1 INTRODUCTION . Incremental learning without catastrophic forgetting is one of the core characteristics of a lifelong learning machine ( L2M ) and has recently gained renewed attention from the machine learning community . In real-world applications , the input distribution of the data ( e.g. , sensory inputs ) is prone to constant changes due to environmental variations ( e.g. , seasonal changes ) , exposure to new situations ( e.g. , change in the surface friction ) , sensory malfunction ( e.g. , water droplets on a camera ) , among others . It is therefore desirable to continue to train the base computational model only on the new data/task and incrementally accumulate knowledge to improve the performance of the system over time , as opposed to retraining the model on the composition of old and new data . The existing computational models , for instance deep convolutional neural networks ( CNNs ) , face two fundamental issues regarding incremental learning , 1 ) catastrophic forgetting ( McCloskey & Cohen , 1989 ) , which refers to the forgetting of previously acquired knowledge when learning new tasks as a result of interference between the old and new tasks , and 2 ) intransigence , which refers to the inability to acquire new knowledge while trying to preserve old knowledge ( e.g. , reducing the learning rate ) Chaudhry et al . ( 2018 ) . Note that we use the term ‘ knowledge ’ here to indicate the input/output behavior of the computational model as in ( Hinton et al. , 2015 ) . A successful incremental learner should be able to overcome both forgetting and intransigence . The commonly used strategies to overcome catastrophic forgetting include : 1. selective synaptic plasticity to preserve learned knowledge ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Aljundi et al. , 2018 ; Chaudhry et al. , 2018 ) , which is rooted in the idea of homeostatic plasticity in neuroscience , 2. additional neural resource allocation to learn new knowledge and preserve old knowledge , ( Rusu et al. , 2016 ; Lee et al. , 2017 ; Li & Hoiem , 2017 ; Rannen et al. , 2017 ; Schwarz et al. , 2018 ; Li et al. , 2019 ) , which is similar to neurogenesis in the hippocampus , 3. memory and experience replay ( Rebuffi et al. , 2017 ; Shin et al. , 2017 ; Wu et al. , 2018 ; Hu et al. , 2019 ; Rostami et al. , 2019 ) , which is based on the well-established theory of complementary learning system ( CLS ) ( McClelland et al. , 1995 ) . Each framework has its advantages and disadvantages , and Parisi et al . ( 2019 ) provide an excellent survey of these methods . We note that the term “ synaptic weights ” refers to the strength of a connection between two nodes . The term ‘ plasticity ’ is used analogously to ‘ neural plasticity ’ in the human brain , which refers to the ability of the neurons to change their synaptic weights . The term “ selective plasticity ” refers to the desired capability of a network to selectively increase or decrease the plasticity of individual synapses throughout the neural architecture . Our focus in this paper is on selective synaptic plasticity . The standard deep neural network architectures are uniformly plastic ; hence , all neurons are prone to changes during training , and this powerful capability is also the demise of these networks and leads to catastrophic forgetting . The idea of selective synaptic plasticity is to partially preserve synapses that are critical for previously learned tasks by rigidifying those synapses ( i.e. , to enforce critical synapses to change less ) . Rigidifying the network over time leads to a loss of learning capability for future tasks , which is known as ‘ intransigence ’ in the literature . Selective synaptic plasticity , by itself , could not fully overcome intransigence . A combination of strategies like efficient memory replay for reconsolidation , neurogenesis , and selective synaptic plasticity could lead to superior methods that defeat both catastrophic forgetting and intransigence . Chaudhry et al . ( 2018 ) , for instance , provide such a combination of memory replay and selective synaptic plasticity . Also , there have been various efforts toward making the idea of neural resource allocation scalable , the progress and compress work Schwarz et al . ( 2018 ) , and the incremental moment matching Lee et al . ( 2017 ) work fall under this category . In this paper , we focus on selective synaptic plasticity to preserve learned representations in a deep neural network . Inspired by ( Chaudhry et al. , 2018 ) , we take a geometric view and devise a new method for selective synaptic plasticity . The proposed method is fundamentally different from the previous approaches like Elastic Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) and Memory Aware Synapses ( MAS ) ( Aljundi et al. , 2018 ) , which we indicate as sample-based approaches . Instead , we focus on identifying synaptic importance parameters that preserve the ‘ distribution ’ of the latent representation of a task . Focusing on preserving the distribution of the latent representation of a neural network at an arbitrary layer , as opposed to the expected change in network ’ s response for individual samples , enforces a less restrict regularization on the network , and enables a better utilization of the network learning capacity . The primary concept of sample-based versus distribution-based regularizations are visualized in Figure 1 . Similar to the MAS framework , our proposed method denoted as Sliced Cramér Preservation ( SCP ) is also able to preserve a task representation in any layer of a neural network , hence , enabling its application to various unsupervised or self-supervised learning settings . Our specific contributions in this paper are : • Introducing a distribution-based regularization using the sliced Cramér distance ( aka , Cramér Wold distance ( Tabor et al. , 2018 ) ) for selective synaptic plasticity that preserves the distributions of the representations of previously seen tasks at an arbitrary layer of a deep neural network while learning a new task . • Providing a geometric interpretation for the MAS algorithm ( Aljundi et al. , 2018 ) that further gives insight into the otherwise heuristic choices made in the approach . • Comparing the proposed method to online-EWC and MAS on the benchmark permuted MNIST dataset , sequential unsupervised learning with auto-encoders , and the more interesting problem of semantic segmentation of driving scenes , and demonstrating significant improvements in overcoming catastrophic forgetting and intransigence over these methods . 2 PROBLEM SET-UP AND PRELIMINARIES . Consider data from a stream of tasks X t = { xti ∼ ptX } nti=1 , where ptX is the probability density function ( PDF ) for task t defined on X ⊂ Rd . We consider both supervised and unsupervised tasks , where in the supervised case the input sample , xti , is accompanied with the corresponding label yti ∈ RK . Let φ ( · ; θ ) : Rd → RK denote a parametric function ( e.g. , a neural network ) that is to be optimized to solve the stream of tasks . In the supervised learning setting , we consider φ to be the mapping to the logits prior to applying the softmax layer . 2.1 A GEOMETRIC VIEW OF ELASTIC WEIGHT CONSOLIDATION . In their seminal work , Kirkpatrick et al . ( 2017 ) considered the problem of overcoming catastrophic forgetting in supervised and also reinforcement learning scenarios where a supervisory signal y exists , whether in the form of labels/annotations , or environmental rewards , respectively . Here we reiterate the geometric interpretation of the EWC framework following the work of ( Chaudhry et al. , 2018 ) , which we will then adapt to define our generic consolidation framework . Let pθ ( y|x ) = softmax ( φ ( x ; θ ) ) , where [ pθ ( y|x ) ] j is the softmax probability of the j-th class . For simplicity , let us consider the case where we want to learn only two tasks consecutively , i.e. , tasks ‘ A ’ and ‘ B. ’ Then , EWC ensures that while learning task ‘ B , ’ the conditional likelihood pθ ( y|xA ) does not drift far from the optimal conditional likelihood pθ∗A ( y|xA ) , where θ∗A are the parameters initially optimized for task A : arg min θ L̃B ( θ ) = arg min θ LB ( θ ) + λEx∼pAX [ DKL ( pθ∗A ( y|x ) || pθ ( y|x ) ) ] = arg min θ LB ( θ ) + λEx∼pAX [ Ey∼pθ∗ A [ log ( pθ∗A ( y|x ) pθ ( y|x ) ) ] ] ( 1 ) where λ is the regularization coefficient . Note that equation 1 is essentially an optimization with a memory replay regularizer . Here , while learning task B , samples from task A ( i.e. , a memory buffer from this task ) are fed through the network and the conditional likelihood is constantly checked against the optimal conditional likelihood for task A , pθ∗A ( y|xA ) , to ensure a minimal deviation from those parameters . The key question answered by the EWC framework is on how to avoid memory replay and yet achieve a similar result , i.e. , not forget the knowledge from old task ( task A ) . The answer lies in the second-order Taylor expansion of the regularizer around the parameters optimized for the old task , θ∗A . It is straightforward to show ( Chaudhry et al. , 2018 ) that the second-order Taylor expansion of the regularizer is of the form : Ex∼pAX [ DKL ( pθ ( y|x ) || pθ+δθ ( y|x ) ) ] ≈ 1 2 δθTFθδθ = ‖δθ‖2Fθ ( 2 ) where Fθ is the Fisher Information Matrix ( FIM ) and is defined as : Fθ = Ex∼pAX [ Ey∼pθ∗ A [ ( ∂log ( pθ ( y|x ) ) ∂θ ) ( ∂log ( pθ ( y|x ) ) ∂θ ) T ] ] ( 3 ) see supplementary material for complete derivations . Therefore , when δθ → 0 the KL-divergence regularizer enforces closeness of θ to θ∗A in a Riemmanian pseudo-manifold induced by the FIM . Given that the number of parameters could easily reach several million in standard deep neural networks , it is practically infeasible to store and use the FIM matrix , Fθ . Therefore , Kirkpatrick et al . ( 2017 ) assume that Fθ is diagonal and further approximate the KL-divergence with : Ex∼pAX [ DKL ( pθ ( y|x ) || pθ+δθ ( y|x ) ) ] ≈ 1 2 M∑ m=1 [ Fθ ] m , m [ δθ ] 2 m ( 4 ) where M is the total number of parameters in the neural network . This leads to the main equation in the EWC framework : arg min θ L̃B ( θ ) = arg min θ LB ( θ ) + λ 2 M∑ m=1 [ Fθ∗A ] m , m [ θ − θ∗A ] 2m ( 5 ) From our point of view , the critical aspect of these derivations is the connection between memory replay and structural plasticity . Note that we started with equation 1 , which uses the idea of a memory replay regularizer . Then by assuming δθ → 0 , using the second-order Taylor expansion of the KL-divergence around θ∗A , and assuming that FIM is a diagonal matrix we arrived at equation 5 , which provides the idea of synaptic importance parameters and is an embodiment of the selective synaptic plasticity framework . Next , we use this critical aspect and develop an analogous regularizer ( i.e. , based on memory replay ) for the MAS algorithm .

Summary: This paper proposes a new method for overcoming catastrophic forgetting in continual learning, based on distribution-based regularization using the sliced Cramer distance, i.e. Sliced Cramer Preservation (SCP). Unlike previous work on catastrophic forgetting, this paper tackles unsupervised learning scenarios as well as supervised learning. They evaluate the proposed SCP on permutated MNIST, sequential learning in autoencoder task, and sequential learning for segmentation.