Title: SLICED CRAM ÉR SYNAPTIC CONSOLIDATION FOR PRESERVING DEEPLY LEARNED REPRESENTATIONS

Abstract: Deep neural networks suffer from the inability to preserve the learned data representation (i.e., catastrophic forgetting) in domains where the input data distribution is non-stationary, and it changes during training. Various selective synaptic plasticity approaches have been recently proposed to preserve network parameters, which are crucial for previously learned tasks while learning new tasks. We explore such selective synaptic plasticity approaches through a unifying lens of memory replay and show the close relationship between methods like Elastic Weight Consolidation (EWC) and Memory-Aware-Synapses (MAS). We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network's output at an arbitrary layer for previous tasks while learning a new one. We propose the sliced Cramér distance as a suitable choice for such preservation and evaluate our Sliced Cramér Preservation (SCP) algorithm through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings. We show that SCP consistently utilizes the learning capacity of the network better than online-EWC and MAS methods on various incremental learning tasks.

1. INTRODUCTION:

Incremental learning without catastrophic forgetting is one of the core characteristics of a lifelong learning machine (L2M) and has recently gained renewed attention from the machine learning community. In real-world applications, the input distribution of the data (e.g., sensory inputs) is prone to constant changes due to environmental variations (e.g., seasonal changes), exposure to new situations (e.g., change in the surface friction), sensory malfunction (e.g., water droplets on a camera), among others. It is therefore desirable to continue to train the base computational model only on the new data/task and incrementally accumulate knowledge to improve the performance of the system over time, as opposed to retraining the model on the composition of old and new data.

The existing computational models, for instance deep convolutional neural networks (CNNs), face two fundamental issues regarding incremental learning, 1) catastrophic forgetting (McCloskey & Cohen, 1989) , which refers to the forgetting of previously acquired knowledge when learning new tasks as a result of interference between the old and new tasks, and 2) intransigence, which refers to the inability to acquire new knowledge while trying to preserve old knowledge (e.g., reducing the learning rate) Chaudhry et al. (2018) . Note that we use the term 'knowledge' here to indicate the input/output behavior of the computational model as in (Hinton et al., 2015) . A successful incremental learner should be able to overcome both forgetting and intransigence. The commonly used strategies to overcome catastrophic forgetting include:

1. selective synaptic plasticity to preserve learned knowledge (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Chaudhry et al., 2018) , which is rooted in the idea of homeostatic plasticity in neuroscience, 2. additional neural resource allocation to learn new knowledge and preserve old knowledge, (Rusu et al., 2016; Lee et al., 2017; Li & Hoiem, 2017; Rannen et al., 2017; Schwarz et al., 2018; Li et al., 2019) , which is similar to neurogenesis in the hippocampus, 3. memory and experience replay (Rebuffi et al., 2017; Shin et al., 2017; Wu et al., 2018; Hu et al., 2019; Rostami et al., 2019) , which is based on the well-established theory of complementary learning system (CLS) (McClelland et al., 1995) .

Each framework has its advantages and disadvantages, and Parisi et al. ( 2019) provide an excellent survey of these methods. We note that the term "synaptic weights" refers to the strength of a connection between two nodes. The term 'plasticity' is used analogously to 'neural plasticity' in the human brain, which refers to the ability of the neurons to change their synaptic weights. The term "selective plasticity" refers to the desired capability of a network to selectively increase or decrease the plasticity of individual synapses throughout the neural architecture.

Our focus in this paper is on selective synaptic plasticity. The standard deep neural network architectures are uniformly plastic; hence, all neurons are prone to changes during training, and this powerful capability is also the demise of these networks and leads to catastrophic forgetting. The idea of selective synaptic plasticity is to partially preserve synapses that are critical for previously learned tasks by rigidifying those synapses (i.e., to enforce critical synapses to change less). Rigidifying the network over time leads to a loss of learning capability for future tasks, which is known as 'intransigence' in the literature. Selective synaptic plasticity, by itself, could not fully overcome intransigence. A combination of strategies like efficient memory replay for reconsolidation, neurogenesis, and selective synaptic plasticity could lead to superior methods that defeat both catastrophic forgetting and intransigence. Chaudhry et al. (2018) , for instance, provide such a combination of memory replay and selective synaptic plasticity. Also, there have been various efforts toward making the idea of neural resource allocation scalable, the progress and compress work Schwarz et al. (2018) , and the incremental moment matching Lee et al. (2017) work fall under this category.

In this paper, we focus on selective synaptic plasticity to preserve learned representations in a deep neural network. Inspired by (Chaudhry et al., 2018) , we take a geometric view and devise a new method for selective synaptic plasticity. The proposed method is fundamentally different from the previous approaches like Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) and Memory Aware Synapses (MAS) (Aljundi et al., 2018) , which we indicate as sample-based approaches. Instead, we focus on identifying synaptic importance parameters that preserve the 'distribution' of the latent representation of a task. Focusing on preserving the distribution of the latent representation of a neural network at an arbitrary layer, as opposed to the expected change in network's response for individual samples, enforces a less restrict regularization on the network, and enables a better utilization of the network learning capacity. The primary concept of sample-based versus distribution-based regularizations are visualized in Figure 1 . Similar to the MAS framework, our proposed method denoted as Sliced Cramér Preservation (SCP) is also able to preserve a task representation in any layer of a neural network, hence, enabling its application to various unsupervised or self-supervised learning settings.

Our specific contributions in this paper are:

• Introducing a distribution-based regularization using the sliced Cramér distance (aka, Cramér Wold distance (Tabor et al., 2018) ) for selective synaptic plasticity that preserves the distributions of the representations of previously seen tasks at an arbitrary layer of a deep neural network while learning a new task.

• Providing a geometric interpretation for the MAS algorithm (Aljundi et al., 2018) that further gives insight into the otherwise heuristic choices made in the approach.

• Comparing the proposed method to online-EWC and MAS on the benchmark permuted MNIST dataset, sequential unsupervised learning with auto-encoders, and the more interesting problem of semantic segmentation of driving scenes, and demonstrating significant improvements in overcoming catastrophic forgetting and intransigence over these methods.

KL-Divergence:



EWC Regularizer MAS Regularizer:

Change in L2-Norm

SCP Regularizer:

Sample-Based Distribution-Based 

E x⇠p A X ⇥ d( (x; ✓), (x; ✓ ⇤ A )) ⇤ d p p A Z (•|✓), p A Z (•|✓ ⇤ A )

2. PROBLEM SET-UP AND PRELIMINARIES:

Consider data from a stream of tasks X t = {x t i ∼ p t X } nt i=1 , where p t X is the probability density function (PDF) for task t defined on X ⊂ R d . We consider both supervised and unsupervised tasks, where in the supervised case the input sample, x t i , is accompanied with the corresponding label

y t i ∈ R K . Let φ(•; θ) : R d → R K denote a parametric function (e.

g., a neural network) that is to be optimized to solve the stream of tasks. In the supervised learning setting, we consider φ to be the mapping to the logits prior to applying the softmax layer.

2.1. A GEOMETRIC VIEW OF ELASTIC WEIGHT CONSOLIDATION:

In their seminal work, Kirkpatrick et al. (2017) considered the problem of overcoming catastrophic forgetting in supervised and also reinforcement learning scenarios where a supervisory signal y exists, whether in the form of labels/annotations, or environmental rewards, respectively. Here we reiterate the geometric interpretation of the EWC framework following the work of (Chaudhry et al., 2018) , which we will then adapt to define our generic consolidation framework. Let p θ (y|x) = softmax(φ(x; θ)), where [p θ (y|x)] j is the softmax probability of the j-th class. For simplicity, let us consider the case where we want to learn only two tasks consecutively, i.e., tasks 'A' and 'B.' Then, EWC ensures that while learning task 'B,' the conditional likelihood p θ (y|x A ) does not drift far from the optimal conditional likelihood p θ * A (y|x A ), where θ * A are the parameters initially optimized for task A:

EQUATION

)

where λ is the regularization coefficient. Note that equation 1 is essentially an optimization with a memory replay regularizer. Here, while learning task B, samples from task A (i.e., a memory buffer from this task) are fed through the network and the conditional likelihood is constantly checked against the optimal conditional likelihood for task A, p θ * A (y|x A ), to ensure a minimal deviation from those parameters.

The key question answered by the EWC framework is on how to avoid memory replay and yet achieve a similar result, i.e., not forget the knowledge from old task (task A). The answer lies in the second-order Taylor expansion of the regularizer around the parameters optimized for the old task, θ * A . It is straightforward to show (Chaudhry et al., 2018) that the second-order Taylor expansion of (5)

From our point of view, the critical aspect of these derivations is the connection between memory replay and structural plasticity. Note that we started with equation 1, which uses the idea of a memory replay regularizer. Then by assuming δθ → 0, using the second-order Taylor expansion of the KL-divergence around θ * A , and assuming that FIM is a diagonal matrix we arrived at equation 5, which provides the idea of synaptic importance parameters and is an embodiment of the selective synaptic plasticity framework. Next, we use this critical aspect and develop an analogous regularizer (i.e., based on memory replay) for the MAS algorithm.

2.2. GENERALIZING TO UNSUPERVISED LEARNING:

The EWC framework as explained in the previous section preserves the softmax probability of samples from Task A while learning Task B. This limits the applicability of the method to networks with outputs living on a K-dimensional simplex, e.g., supervised learning and reinforcement learning where the network outputs a probability over a finite set of actions. More recently, Aljundi et al. (2018) presented their Memory-Aware-Synapses (MAS) framework, which lifts the requirement for EWC outputs to live on a simplex, and enables calculation of the synaptic importance parameters even in unsupervised learning and also during testing. While the method is exciting and practically very useful, there is no geometric motivation behind the algorithm. Here we reverse engineer the importance term used in MAS and show a simple regularizer that leads to the MAS algorithm and more importantly provides a geometric interpretation for the algorithm.

Let the regularizer for Task B, be the expected absolute difference between squared 2 norms of the output of the network for samples from Task A, i.e.:

arg min θ LB (θ) = arg min θ L B (θ) + λE x∼p A X 1 2 ( φ(x; θ) 2 -φ(x; θ * A ) 2 ) 2 (6)

It is straightforward (see Supplementary material) to show that using the second-order Taylor expansion of the above regularizer leads to:

arg min θ LB (θ) = arg min θ L B (θ) + λ M m=1 [Ω] m,m [θ -θ * A ] 2 m (7)

where

[Ω] m,n = E x∼p A X ( ∂ φ(x;θ) 2 ∂[θ]m )( ∂ φ(x;θ) 2 ∂[θ]n

) , which is the importance parameter used by Aljundi et al. (2018) . From a geometric perspective, MAS preserves the norms of the samples from Task A while learning Task B. In other words; MAS enforces closeness of θ to θ * A in a Riemmanian pseudo-manifold induced by matrix Ω.

The general idea of using the expected value of a "suitable" distance/divergence between samples,

E x∼p A X d(φ(x; θ), φ(x; θ * A )

) and leveraging its second-order Taylor expansion to obtain synaptic importance values is crucial here and could lead to various undiscovered algorithms based on new distances/divergences. We refer to these approaches as sample-based regularization methods.

2.3. PRESERVING DISTRIBUTION OF AN ARBITRARY LAYER:

We approach the problem of overcoming catastrophic forgetting from the angle of preserving Task A's distribution at an arbitrary layer of the neural network, when learning Task B. Let

z A i = φ(x A i ; θ) ∈ R K

be the output of the network, for a sample from Task A, at the target layer (e.g., output logits in a NN classifier, or reconstructed image of an autoencoder). The distribution of the random variable z A in the target layer follows from the Random Variable Transform (RVT) theorem (Gillespie, 1983) :

p A Z (z|θ) = X p A X (x)δ(z -φ(x; θ))dx (8)

Then we propose the following general regularization to overcome forgetting when learning task B:

EQUATION

where d(•, •) is a discrepancy measure between the two probability distributions defined in R K . Note that equation 9 could be performed with any discrepancy measure or distance, e.g., the Wasserstein distance (Villani, 2008; Kolouri et al., 2017) , using a memory replay strategy. In what follows, we describe a 'suitable' distance d(•, •) that: 1) respects the underlying geometry of the space, and 2) enables a similar strategy to that of the EWC framework to provide importance parameters.

3 SLICED-CRAM ÉR DISTANCE FOR STRUCTURAL PLASTICITY

3.1. CRAM ÉR DISTANCE:

The p-Cramér distance (Cramér, 1928; Székely & Rizzo, 2013) between two one-dimensional probability density functions p 0 and p 1 is defined as the p -norm between their cumulative distribution functions. Note that here we avoid any measure theoretic notations for simplicity. Let q i (t) = t -∞ p i (τ )dτ denote the cumulative distribution function for p i , then the p-Cramér distance is defined as:

C p (p 0 , p 1 ) = R |q 0 (t) -q 1 (t)| p dt 1 p (10)

for p ≥ 1. Similar to the Wasserstein distance and unlike the KL-divergence and its symmetric form Jensen-Shannon distance (i.e., the square root of the Jensen-Shannon divergence), the Cramér distance respects the underlying geometry of the space. Moreover, the Cramér distance provides unbiased sample gradients (Bellemare et al., 2017) , and for p = 1 , C 1 is equivalent to the 1-Wasserstein distance, W 1 . In addition, and similar to the Wasserstein metric, the dual of the Cramér distance is of the form of an integral probability metric (IPM) (Dedecker & Merlevède, 2007) .

To further demonstrate the favorable characteristics of this distance, consider the following parametric distribution matching in one-dimension, where the target distribution, p, is a box distribution defined on R and p τ (t) = p(t-τ ) is the shifted version of p and the goal is to optimize τ to minimize the distance between p τ and p (τ * = 0). For this simple setting, we calculate the energy landscape (i.e., the distance between p τ and p) as a function of τ for the Jensen-Shannon distance, W p , and C p for p = 1, 2. Figure 2 shows the distributions p and p τ on the left and the energy landscape as a function of τ on the right. It can be clearly seen that the Wasserstein and Cramér distances respect the underlying geometry of the problem, while the Jensen-Shannon (JS) distance fails to do so.

To extend the Cramér distance to higher-dimensional distributions, we utilize the idea of distribution slicing used in various recent publications (Kolouri et al., 2016; 2019; 2018) . We note that the sliced-Cramér distance (also known as the Cramér-Wold distance) was recently used in Tabor et al. (2018) for generative modeling. We briefly describe the idea in the following section. 

3.2. SLICED-CRAM ÉR DISTANCE:

The idea of slicing a higher-dimensional distribution has roots in the Radon transform that is commonly used in computational tomography. The idea is to represent a high-dimensional distribution via the infinite set of its marginal distributions. In short, let p 0 and p 1 be d-dimensional probability density functions defined on X ⊂ R d , then their Radon transform is defined as:

EQUATION

for ∀t ∈ R and ∀ξ ∈ S d-1 where S d-1 denotes the d-dimensional unit sphere. Note that Rp i (•, ξ) is a so called slice of p i , which is a one-dimensional marginal distribution of p i . Let Rq i (•, ξ) be the corresponding cumulative distribution function of Rp i (•, ξ):

EQUATION

then the sliced-Cramér distance between p 0 and p 1 is the expected value of the Cramér distance between their one-dimensional slices, i.e., Rp i (•, ξ) when ξ ∼ U S d-1 for U S d-1 being the uniform distribution on the d-dimensional unit sphere. In other words, the sliced Cramér distance is defined as:

EQUATION

Now we are ready to propose our method for overcoming representation forgetting.

3.3. OVERCOMING REPRESENTATION FORGETTING:

The critical point here is that the sample-based regularizers could over-estimate the importance of synapses, leading to intransigence faster. More importantly, a substantial expected change in the network's output (over individual samples) would not necessarily mean catastrophic forgetting. Changes in the network's output within a mode (e.g., in supervised classification within the distribution of a particular class) are harmless, so long as the representation of the data is not significantly changing. Our proposed regularizer tolerates such changes and therefore has the potential for better utilization of the network's learning capacity. We propose the following regularization for overcoming catastrophic forgetting:

EQUATION

equation 14 requires memory replay from the old task(s) while learning the new one. To transition from memory replay to selective synaptic plasticity, we derive the second-order Taylor expansion of the regularizer SC 2 2 around the optimal parameters for the previous tasks, θ * A . Assuming that θ = θ * A + δθ where δθ → 0, it is straightforward to show that:

EQUATION

)

where Γ θ * A is defined as:

EQUATION

See supplementary materials for the detailed derivations. Note that similar to F θ * A , Γ θ * A is also positive-semi-definite (PSD) and therefore our Sliced-Cramér regularizer enforces closeness of θ and θ * in a Riemannian pseudo-manifold induced by the PSD matrix Γ θ .

While the definition of Γ in equation 16, regardless of its similarity to equation 3, may seem intimidating, it leads to a straightforward empirical algorithm, which we discuss in the next section. Before that, we point out that similar to the FIM, calculating Γ is also practically infeasible, and we approximate Γ with a diagonal matrix that simplifies the regularization into:

arg min θ LB (θ) = arg min θ L B (θ) + λ M m=1 [Γ] m,m [δθ] 2 m (17)

Finally, we emphasize that while equation 17 is similar to equation 5, it enforces a very different constraint on the neural network. equation 5 enforces conditional class likelihoods to be preserved, which is sample-based; however, our proposed formulation in equation 17 preserves the distribution of network's outputs at a particular layer for old tasks, i.e., it preserves the distribution of the previously learned representations.

4. PROPOSED ALGORITHM:

Here we derive the algorithmic steps required to calculate Γ empirically, as shown in Algorithm 1. The empirical distribution at the network's output can be written as,

p A Z (z|θ) ≈ 1 N N n=1 δ(z - φ(x A n ; θ))

, and the slices of this empirical distribution are defined as:

Rp A Z (t, ξ|θ) ≈ 1 N N n=1 δ(t -ξ • φ(x A n ; θ))

Let u(•) denote the step function, which is the cumulative distribution of the Dirac delta function.

Then the cumulative distribution of Rp A Z (t, ξ|θ) can be written as,

Rq A Z (t, ξ|θ) ≈ 1 N N n=1 u(t - ξ • φ(x A n ; θ)

). Therefore we have:

EQUATION

substituting equation 18 into equation 16 and using a Monte-Carlo approximation of the integration of S d-1 , with L samples, leads to:

Γ = 1 L L l=1 d ξ l • z dθ d ξ l • z dθ T (19)

where ξ l s are randomly drawn from S K-1 , and z = 1 N N n=1 φ(x A n ; θ * A ). These derivations give birth to our proposed algorithm shown in Algorithm 1. Given that calculation of matrix Γ is not practically feasible (due to the large number of parameters of a deep neural network), we follow the work of (Kirkpatrick et al., 2017) and approximate Γ to be a diagonal matrix, which simplifies to:

Γ θ * A i,i = 1 L L l=1 1 N N n=1 d ξ l • φ(x A n ; θ * A ) dθ i 2 = 1 L L l=1 d ξ l • φ * A dθ i 2 where φ * A = 1 N N n=1 φ(x A n ; θ * A ).

Algorithm 1: Sliced Cramer Preservation (SCP) 

Input: Data, X A = {x A n ∼ p A X } N n=1 ,

4.1. ONLINE EXTENSION OF THE ALGORITHM:

To extend the framework into sequential learning of multiple tasks and to avoid memorizing taskspecific Γs (or their diagonals), we follow the EWC++ framework proposed by Chaudhry et al. (2018) , which is, in essence, identical to the online-EWC proposed by Schwarz et al. (2018) . The EWC++ (and online-EWC) methods ameliorate the need for predicting task identities and calculating task-specific FIMs and keep the memory requirement of the method constant (EWC requires linear growth of memory as a function of number of tasks). In EWC++, given F (t-1) θ at task (t -1), the accumulated FIM after learning task t is calculated as

F (t) θ = αF θ * t + (1 -α)F (t-1) θ

, where F θ * t is the task specific FIM for task t, and α ∈ [0, 1) is a hyperparameter that indicates the importance of preserving the most recent task over the older ones. Similarly, we use,

Γ (t) θ = αΓ θ * t + (1 -α)Γ (t-1) θ (20)

to obtain the sliced-Cramér regularizer for task (t + 1).

5. NUMERICAL EXPERIMENTS:



5.1. PERMUTED MNIST:

We first test our proposed algorithm on the benchmark permuted MNIST task and compare the performance with online-EWC and our implementation of the online-MAS algorithm. For this experiment, we used a single head model that learns ten tasks, where each task contains a permuted version of the original MNIST dataset. We note that the reported results are a function of the architecture of the underlying network. Meaning that, while all three methods perform well on this task for larger networks, the true competitiveness of the proposed method emerges for smaller networks where "over-estimation" of the synaptic importances significantly hinders learning of the subsequent tasks and leads to intransigence. For this experiment, we used a fully-connected network (i.e., (Ros et al., 2016) . Task 1 (T1) is semantic segmentation in summer, and Task 2 (T2) is semantic segmentation in winter. The first row shows performance after learning T1 on input from T1. Second row, shows performance after learning T2 on input from T2. The third row shows performance on input from T1 after learning T2. The last row, magnifies the last three images in the third row for the ease of comparison.

a multi-layer perceptron) with the following architecture, 784 → 1024 → 512 → 256 → 10 neurons, and for all optimizations we used the ADAM optimizer with learning rate, lr = 1e -4. For our proposed method, SCP, we used L = 100 slices. We repeated each experiment 10 times (with different permutations), and reported the average accuracy over all tasks in Figure 3 (a) . We can see that SCP is capable of utilizing the capacity of the network better, which points to the fact that regularizing the distribution as opposed to the samples provides a less restricted regularization for the network and still enables the network to freely move individual samples so long as the overall latent distribution of the data is consistent.

The regularization coefficients for each algorithm was cross-validated on the following grid, λ ∈ {1e + i | i ∈ [-3, -2, ..., 9]} and the optimal value was used to report the results in Figure 3 (a) . Moreover, in Figure 3 (b) we show the histogram of the logarithm of the product of the regularization coefficients with importance parameters for each method (i.e., λ times the diagonal values of F θ , Ω, and Γ for EWC, MAS, and SCP, respectively). One can see that the optimal values of the regularization coefficients provide, more or less, scale-consistent synaptic importances for all methods and distinguishing factor between the methods is on the difference between these distributions. Another interesting observation is that the distribution of the synaptic importances are more similar for sample-based methods (i.e., EWC and MAS) compared to the proposed distribution-based method.

5.2. SEQUENTIAL LEARNING OF AUTO-ENCODERS:

Next, we consider an experiment consisting of unsupervised/self-supervised sequential learning. To that end, we learn an auto-encoder on single digits of the MNIST dataset sequentially. The model is chosen to be a fully connected auto-encoder, with the following encoder 728 → 1024 → 1024 → 1024 → 256, a mirrored decoder 256 → 1024 → 1024 → 1024 → 784, and Rectified Linear Unit (ReLU) activations. For the loss function, we used cross-entropy plus the 1 -norm of the reconstruction error. Similar to the previous experiment, we used the ADAM optimizer (Kingma & Ba, 2014) for training the network with lr = 1e-4. We perform 50 epochs of learning on each digit, before switching to the next digit. For consolidation, we used L = 100 slices for SCP. Finally, we permute the order of the digits and run our experiments 10 times, and compare 'No Consolidation,' with MAS, and SCP. Due to the unsupervised nature of the experiment, the EWC framework does not apply here.

Figure 4 demonstrate the results of this experiment. Panel (a) shows the reconstruction of a sample digit from one of our runs with the input digit sequence of [2, 7, 4, 0, 9, 1, 6, 5, 8, 3] . It can be seen that both MAS and SCP can successfully retain the learned knowledge while acquiring new knowledge, while without consolidation the auto-encoder suffers from catastrophic forgetting. Panel (b) shows the average 1 -norm of the reconstruction error for all seen tasks over the 10 runs. We can see that SCP and MAS are qualitatively on par, and SCP provides a modest yet statistically significant improvement over MAS for this task.

5.3. SEMANTIC SEGMENTATION OF SYNTHIA DATASET:

Lastly, we go beyond the benchmark yet less practical MNIST dataset and address catastrophic forgetting in a more interesting/critical application of autonomous vehicles. We specifically consider the problem of learning semantic segmentation of road scenes in a sequential manner, where the input distribution of the data changes over time. Semantic segmentation is the task of assigning a class label to every pixel of an input image. To that end, we use two sequences of the SYNTHIA dataset (Ros et al., 2016) , namely 'SYNTHIA-SEQS-01-SUMMER' and 'SYNTHIA-SEQS-01-WINTER' as Task 1 and Task 2, respectively. There are 13 classes in the dataset namely: Miscellaneous, Sky, Building, Road, Sidewalk, Fence, Vegetation, Pole, Car, Sign, Pedestrian, Cyclist, and Lane Marking. We keep the last 100 frames of each sequence as the testing-set and train a deep convolutional U-Net architecture (Ronneberger et al., 2015) on the tasks mentioned above. For the loss function, we used (1 -Dice) (Zou et al., 2004) , and each task was learned over 100 epochs. For the optimizer, we used the ADAM optimizer (Kingma & Ba, 2014) with learning rate, lr = 1e -4. For consolidation, we used L = 100 slices for SCP. We learn the tasks sequentially (summer first and then winter), and a qualitative comparison of online-EWC, MAS, and SCP on a sample test frame is in Figure 5 . The first row shows the performance on T1 after learning T1 (base model), the second row shows performance on T2 after learning on T1 and then T2 (indicator of intransigence), and the third row shows performance on T1 after learning on T1 and then T2 (indicator of the catastrophic forgetting). We can see that catastrophic forgetting happens when no synaptic consolidation is leveraged. Moreover, compared to online-EWC and MAS, SCP suffers less from intransigence as fewer artifacts are present in the second row. Lastly, SCP overcomes catastrophic-forgetting more successfully compared to EWC and MAS as it is apparent from the lack of artifacts in the last row. Finally, we provide a quantitative comparison between the methods in Figure 6 , where we report the Dice score (Zou et al., 2004) , averaged over ten runs, for each task and each method during the sequential training. As can be seen, SCP significantly outperforms online-EWC and MAS on this task both in overcoming catastrophic forgetting (left plot) and overcoming intransigence (right plot).

6. CONCLUSION:

We introduced a new generic approach towards selective synaptic plasticity for preserving the distribution of the network's output, at an arbitrary layer, for previously learned tasks. We started from a memory-replay-based regularization that penalized the change in the distribution of the network's output and showed that a second-order Taylor expansion of such regularization would lead to a selective synaptic plasticity approach that does not require memory of samples from previously seen tasks. Furthermore, we proposed the sliced-Cramér distance as a suitable metric for preserving these distributions, which leads to a straightforward algorithm for selective plasticity. Also, using a similar approach, we reverse-engineered the Memory Aware Synapses (MAS) framework and provided a geometrically meaningful regularization that leads to this algorithm. We then compared the online-EWC, MAS, and SCP methods on a variety of learning tasks, including supervised and unsupervised/self-supervised learning, and consistently showed competitive performance. around θ 0 where we can write θ = θ 0 + δθ. The second-order Taylor expansion is:

7. ACKNOWLEDGMENTS:

D KL (p θ0 ||p θ ) ≈ D KL (p θ0 ||p θ0 ) + δθ T dD KL (p θ0 ||p θ ) dθ | θ0 + 1 2 δθ T d 2 D KL (p θ0 ||p θ ) dθ 2 | θ0 δθ 1.

Where for the first-order term we have:

dD KL (p θ0 ||p θ ) dθ = - X p θ0 (x) d log(p θ (x)) dθ dx = - X p θ0 (x) p θ (x) dp θ (x) dθ dx

Therefore at θ = θ 0 we have:

dD KL (p θ0 ||p θ ) dθ | θ0 = - X dp θ (x) dθ dx = d dθ X p θ0 (x)dx = 0

2. and for the second-order term:

d 2 D KL (p θ0 ||p θ ) dθ 2 = d dθ - X p θ0 (x) p θ (x) dp θ (x) dθ dx = X p θ0 (x) p 2 θ (x) dp θ (x) dθ dp θ (x) dθ T dx - X p θ0 (x) p θ (x) d 2 p θ (x) dθ 2 dx Therefore at θ = θ 0 we have: d 2 D KL (p θ0 ||p θ ) dθ 2 | θ0 = X p θ0 (x) 1 p θ0 (x) dp θ0 (x) dθ 1 p θ0 (x) dp θ0 (x) dθ T dx - X d 2 p θ0 (x) dθ 2 dx = X p θ0 (x) d log(p θ0 (x)) dθ d log(p θ0 (x)) dθ T dx = E x∼p θ 0 d log(p θ0 (x)) dθ d log(p θ0 (x)) dθ T = F

which is the Fisher Information Matrix (FIM).

Finally, putting everything together we have:

D KL (p θ0 ||p θ ) ≈ 1 2 δθ T F δθ

which concludes the derivation.

8.2. TAYLOR EXPANSION OF THE MAS REGULARIZER:

Here we drive the second-order Taylor expansion of the MAS regularizer in equation 6,

MAS reg = E x∼p X 1 2 ( φ(x; θ) 2 -φ(x; θ 0 ) 2 ) 2 = 1 2 X p X (x)( φ(x; θ) 2 -φ(x; θ 0 ) 2 ) 2 dx

around θ 0 . The second-order Taylor expansion is:

MAS reg ≈ δθ T d MAS reg dθ | θ0 + 1 2 δθ T d 2 MAS reg dθ 2 | θ0 δθ 1.

Where for the first-order term we have:

dMAS reg dθ m = X p X (x) d φ(x; θ) 2 d [θ] m ( φ(x; θ) 2 -φ(x; θ 0 ) 2 )

which for θ = θ 0 is zero.

2. For the second-order term we have:

d 2 MAS reg dθ 2 m,n = X p X (x) d φ(x; θ) 2 d [θ] m d φ(x; θ) 2 d [θ] n dx + X p X (x) d 2 φ(x; θ) 2 d θ 2 m,n ( φ(x; θ) 2 -φ(x; θ 0 ) 2 )dx

where evaluated at θ = θ 0 the second term on the right-hand-side vanishes.

Finally, putting everything together we have:

MAS reg ≈ δθ T E x∼p X d φ(x; θ) 2 d θ d φ(x; θ) 2 d θ T Ω δθ

which is the term reported in the paper.

8.3. TAYLOR EXPANSION OF THE SLICED-CRAM ÉR DISTANCE:

Here we derive the second-order Taylor expansion of the squared 2-Sliced-Cramér distance,

SC 2 2 (p θ0 , p θ ) = S d-1 R |Rq θ0 (t, ξ) -Rq θ (t, ξ)| 2 dtdξ

around θ 0 , as reported in equation 16. The second-order Taylor expansion is:

SC 2 2 (p θ0 , p θ ) ≈ SC 2 2 (p θ0 , p θ0 ) + δθ T d SC 2 2 (p θ0 , p θ ) dθ | θ0 + 1 2 δθ T d 2 SC 2 2 (p θ0 , p θ ) dθ 2 | θ0 δθ 1.

Where for the first-order term we have:

d SC 2 2 (p θ0 , p θ ) dθ = 2 S d-1 R d Rq θ (t, ξ) dθ (Rq θ (t, ξ) -Rq θ0 (t, ξ))dtdξ

which for θ = θ 0 is equal to zero.

2. For the second-order term we have:

d 2 SC 2 2 (p θ0 , p θ ) dθ 2 = 2 S d-1 R d 2 Rq θ (t, ξ) dθ 2 (Rq θ (t, ξ) -Rq θ0 (t, ξ))dtdξ + 2 S d-1 R d Rq θ (t, ξ) dθ d Rq θ (t, ξ) dθ T dtdξ which evaluated at θ = θ 0 is: d 2 SC 2 2 (p θ0 , p θ ) dθ 2 | θ0 = 2 S d-1 R d Rq θ0 (t, ξ) dθ d Rq θ0 (t, ξ) dθ T dtdξ = 2Γ

Finally, putting everything together we have: 

SC 2 2 (p θ0 , p θ ) ≈ δθ T Γδθ

8.4. MNIST-TO-SVHN EXPERIMENT:

Given the space constraint of the conference, we include our results on the MNIST-to-SVHN experiment in the supplementary material. In this experiment, Task 1 is learning the MNIST digits, and Task 2 is learning the SVHN digits. We show sample images from these two datasets in Figure 7a .

In this experiment, we used a VGG-like architecture, for which we include the details in 8. As can be seen, all methods can address catastrophic forgetting, while Online-EWC and SCP outperform MAS (However, a one should perform a finer grid search on the importance-parameters for a definitive evaluation). The middle plot shows the testing performance of the model on the SVHN dataset (Task 2). The MAS and SCP methods outperform online-EWC, and SCP performs slightly better than MAS (i.e., overcomes intransigence better). The third plot shows the average testing performance of the methods on both tasks. Similarly, we can see that SCP outperforms MAS and Online-EWC. We repeated the experiments ten times, and the plots show the average performance.

8.5. CIFAR100 EXPERIMENTS:

The CIFAR100 (Krizhevsky et al., 2009) dataset contains images from 20 super-classes, where each super-class contains five sub-classes. For this dataset, we considered an experiment in which the tasks are supervised classification of the super-classes. In short, we split the data into five sequential tasks, where each task contains a sub-class from all the 20 super-classes. We used a Wide Residual Network (Wide-ResNet) Zagoruyko & Komodakis (2016) network as our model and utilized the data augmentation suggested by (DeVries & Taylor, 2017) . Figure 9a shows sample super-classes together with data augmentation. For each method, we used a coarse grid search for the importance parameters {1e + i | i ∈ [2, 3, 4, 5, 6]}. Unfortunately, and due to the lack of time, we were able to only run the experiment for each method once. We report the average classification results in Figure 9b . For the final version of the paper, we will complete this experiment with 1) a finer grid search over the importance parameters and 2) multiple runs of the experiments and reporting the average.

:

m

Figures and Tables:

Figure 1: Sample-based approaches regularize the learning by the expected change, as measured by a dissimilarity measure, of the response of the network for individual samples from Task A, after learning Task A and during learning Task B, E x∼p A X [d(φ(x; θ), φ(x; θ * A ))], where d(•, •) is a dissimilarity measure between two K-dimensional vectors. Therefore, the regularization is an empirical expected change of the response for samples. EWC and MAS fall under the sample-based category. The proposed distribution-based approach, on the other hand, regularizes the change in the overall distribution of the network's output for input samples from Task A, after learning Task A and during learning Task B, d p p A Z (•|θ), p A Z (•|θ * A ) , where p A Z (•|θ) is defined in equation 8 and d p (•, •) is a distance measure between two probability distributions defined on Z ⊆ R K .

Figure 2: The energy landscape of various distances as a function of the translation parameter. It can be seen that both Wasserstein and Cramér distances respect the underlying geometry of the problem while the Jensen-Shannon distance fails to do so.

Figure 3: Comparison among online-EWC, MAS, and SCP on learning ten permuted MNIST tasks (a), and the histogram of Log(λ × •) of the synaptic importances (i.e., the diagonal values of F , Ω, and Γ, for EWC, MAS, and SCP, respectively) for each method (b).

Figure 4: Qualitative and quantitative comparison between MAS and SCP on sequential learning of auto-encoders. Columns in Panel (a) show the reconstruction of data after learning consequent tasks in a random permutation of MNIST sequence. Panel (b) shows the average 1 -reconstruction loss for each method over all tasks and over 10 runs.

Figure 6: Testing Dice score(Zou et al., 2004) of online-EWC, MAS, and SCP on sequential learning for semantic segmentation of the summer images (Task 1) and winter images (Task 2) from SYNTHIA dataset(Ros et al., 2016). The blue and red shadings on the plots indicate the durations in which the models were trained on summer and winter data, respectively

Figure 7: Sample images from MNIST and SVHN dataset (a), and the testing performance of the Online-EWC, MAS, and SCP on sequential learning starting from MNIST (Task 1) and then SVHN (Task 2) (b). The blue and red shadings on the plots indicate the durations in which the models were trained on the MNIST and SVHN datasets, respectively.

Figure 8: The model we used in the MNIST-to-SVHN experiment.

Figure 9: Sample super-classes from the CIFAR100 dataset and visualization of the data augmentation procedure we used in this experiment (a), and the average testing accuracy of the Online-EWC, MAS, and SCP on sequential learning of the tasks (b). We note that we were, unfortunately, only able to run the experiment once, and will update the Figure with the results from multiple runs as soon as they are available.

This material is based upon work supported by the United States Air Force and DARPA under Contract No. FA8750-18-C-0103. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force and DARPA. TAYLOR EXPANSION OF THE KL-DIVERGENCE For the sake of completion, here we derive the second-order Taylor expansion of D KL , D KL (p θ0 ||p θ ) =

Table content: <table><tr><td>8 SUPPLEMENTARY MATERIALS</td><td/><td/><td/></tr><tr><td>8.1 X</td><td>p θ0 (x)log</td><td>p θ0 (x) p θ (x)</td><td>dx</td></tr></table>

Peer review (iclr2020):
I think the paper is written quite well, and the approach makes a lot of sense. I think the idea of replacing generalizing the KL to sliced Cramer distance is quite interested. The authors put in some effort in explaining why they propose this alternative distance between distributions. I think overall this is a great paper, very informative. If I need to nitpick, I think the experimental section relies heavily on MNIST (e.g. permuted MNIST, auto-encoding MNIST), which I think is not a hard enough task. It is though widely used in the continual learning community, though maybe it should not anymore. But I think given the reliance of the field on these datasets it makes sense, plus I think the strength of the paper is not in the empirical evaluation but rather in the derivation of the method. I think while the authors put quite a bit of effort in explaining the difference between KL and Cramer distance, I would have appreciated a even more detailed exposition. I think the difference between these metrics is not well understood by the majority in the community.