SLICED CRAMÉR SYNAPTIC CONSOLIDATION FOR PRESERVING DEEPLY LEARNED REPRESENTATIONS
Year: 2020

Abstract:
Deep neural networks suffer from the inability to preserve the learned data representation (i.e., catastrophic forgetting) in domains where the input data distribution is non-stationary, and it changes during training. Various selective synaptic plasticity approaches have been recently proposed to preserve network parameters, which are crucial for previously learned tasks while learning new tasks. We explore such selective synaptic plasticity approaches through a unifying lens of memory replay and show the close relationship between methods like Elastic Weight Consolidation (EWC) and Memory-Aware-Synapses (MAS). We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network’s output at an arbitrary layer for previous tasks while learning a new one. We propose the sliced Cramér distance as a suitable choice for such preservation and evaluate our Sliced Cramér Preservation (SCP) algorithm through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings. We show that SCP consistently utilizes the learning capacity of the network better than online-EWC and MAS methods on various incremental learning tasks.

text: Deep neural networks suffer from the inability to preserve the learned data representation (i.e., catastrophic forgetting) in domains where the input data distribution is non-stationary, and it changes during training. Various selective synaptic plasticity approaches have been recently proposed to preserve network parameters, which are crucial for previously learned tasks while learning new tasks. We explore such selective synaptic plasticity approaches through a unifying lens of memory replay and show the close relationship between methods like Elastic Weight Consolidation (EWC) and Memory-Aware-Synapses (MAS). We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network’s output at an arbitrary layer for previous tasks while learning a new one. We propose the sliced Cramér distance as a suitable choice for such preservation and evaluate our Sliced Cramér Preservation (SCP) algorithm through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings. We show that SCP consistently utilizes the learning capacity of the network better than online-EWC and MAS methods on various incremental learning tasks.

heading: 1 INTRODUCTION
text: Incremental learning without catastrophic forgetting is one of the core characteristics of a lifelong learning machine (L2M) and has recently gained renewed attention from the machine learning community. In real-world applications, the input distribution of the data (e.g., sensory inputs) is prone to constant changes due to environmental variations (e.g., seasonal changes), exposure to new situations (e.g., change in the surface friction), sensory malfunction (e.g., water droplets on a camera), among others. It is therefore desirable to continue to train the base computational model only on the new data/task and incrementally accumulate knowledge to improve the performance of the system over time, as opposed to retraining the model on the composition of old and new data.
The existing computational models, for instance deep convolutional neural networks (CNNs), face two fundamental issues regarding incremental learning, 1) catastrophic forgetting (McCloskey & Cohen, 1989), which refers to the forgetting of previously acquired knowledge when learning new tasks as a result of interference between the old and new tasks, and 2) intransigence, which refers to the inability to acquire new knowledge while trying to preserve old knowledge (e.g., reducing the learning rate) Chaudhry et al. (2018). Note that we use the term ‘knowledge’ here to indicate the input/output behavior of the computational model as in (Hinton et al., 2015). A successful incremental learner should be able to overcome both forgetting and intransigence. The commonly used strategies to overcome catastrophic forgetting include:
1. selective synaptic plasticity to preserve learned knowledge (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Chaudhry et al., 2018), which is rooted in the idea of homeostatic plasticity in neuroscience,
2. additional neural resource allocation to learn new knowledge and preserve old knowledge, (Rusu et al., 2016; Lee et al., 2017; Li & Hoiem, 2017; Rannen et al., 2017; Schwarz et al., 2018; Li et al., 2019), which is similar to neurogenesis in the hippocampus,
3. memory and experience replay (Rebuffi et al., 2017; Shin et al., 2017; Wu et al., 2018; Hu et al., 2019; Rostami et al., 2019), which is based on the well-established theory of complementary learning system (CLS) (McClelland et al., 1995).
Each framework has its advantages and disadvantages, and Parisi et al. (2019) provide an excellent survey of these methods. We note that the term “synaptic weights” refers to the strength of a connection between two nodes. The term ‘plasticity’ is used analogously to ‘neural plasticity’ in the human brain, which refers to the ability of the neurons to change their synaptic weights. The term “selective plasticity” refers to the desired capability of a network to selectively increase or decrease the plasticity of individual synapses throughout the neural architecture.
Our focus in this paper is on selective synaptic plasticity. The standard deep neural network architectures are uniformly plastic; hence, all neurons are prone to changes during training, and this powerful capability is also the demise of these networks and leads to catastrophic forgetting. The idea of selective synaptic plasticity is to partially preserve synapses that are critical for previously learned tasks by rigidifying those synapses (i.e., to enforce critical synapses to change less). Rigidifying the network over time leads to a loss of learning capability for future tasks, which is known as ‘intransigence’ in the literature. Selective synaptic plasticity, by itself, could not fully overcome intransigence. A combination of strategies like efficient memory replay for reconsolidation, neurogenesis, and selective synaptic plasticity could lead to superior methods that defeat both catastrophic forgetting and intransigence. Chaudhry et al. (2018), for instance, provide such a combination of memory replay and selective synaptic plasticity. Also, there have been various efforts toward making the idea of neural resource allocation scalable, the progress and compress work Schwarz et al. (2018), and the incremental moment matching Lee et al. (2017) work fall under this category.
In this paper, we focus on selective synaptic plasticity to preserve learned representations in a deep neural network. Inspired by (Chaudhry et al., 2018), we take a geometric view and devise a new method for selective synaptic plasticity. The proposed method is fundamentally different from the previous approaches like Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) and Memory Aware Synapses (MAS) (Aljundi et al., 2018), which we indicate as sample-based approaches. Instead, we focus on identifying synaptic importance parameters that preserve the ‘distribution’ of the latent representation of a task. Focusing on preserving the distribution of the latent representation of a neural network at an arbitrary layer, as opposed to the expected change in network’s response for individual samples, enforces a less restrict regularization on the network, and enables a better utilization of the network learning capacity. The primary concept of sample-based versus distribution-based regularizations are visualized in Figure 1. Similar to the MAS framework, our proposed method denoted as Sliced Cramér Preservation (SCP) is also able to preserve a task representation in any layer of a neural network, hence, enabling its application to various unsupervised or self-supervised learning settings.
Our specific contributions in this paper are:
• Introducing a distribution-based regularization using the sliced Cramér distance (aka, Cramér Wold distance (Tabor et al., 2018)) for selective synaptic plasticity that preserves the distributions of the representations of previously seen tasks at an arbitrary layer of a deep neural network while learning a new task.
• Providing a geometric interpretation for the MAS algorithm (Aljundi et al., 2018) that further gives insight into the otherwise heuristic choices made in the approach.
• Comparing the proposed method to online-EWC and MAS on the benchmark permuted MNIST dataset, sequential unsupervised learning with auto-encoders, and the more interesting problem of semantic segmentation of driving scenes, and demonstrating significant improvements in overcoming catastrophic forgetting and intransigence over these methods.

heading: 2 PROBLEM SET-UP AND PRELIMINARIES
text: Consider data from a stream of tasks X t = {xti ∼ ptX}nti=1, where ptX is the probability density function (PDF) for task t defined on X ⊂ Rd. We consider both supervised and unsupervised tasks, where in the supervised case the input sample, xti, is accompanied with the corresponding label yti ∈ RK . Let φ(·;θ) : Rd → RK denote a parametric function (e.g., a neural network) that is to be optimized to solve the stream of tasks. In the supervised learning setting, we consider φ to be the mapping to the logits prior to applying the softmax layer.

heading: 2.1 A GEOMETRIC VIEW OF ELASTIC WEIGHT CONSOLIDATION
text: In their seminal work, Kirkpatrick et al. (2017) considered the problem of overcoming catastrophic forgetting in supervised and also reinforcement learning scenarios where a supervisory signal y exists, whether in the form of labels/annotations, or environmental rewards, respectively. Here we reiterate the geometric interpretation of the EWC framework following the work of (Chaudhry et al., 2018), which we will then adapt to define our generic consolidation framework. Let pθ(y|x) = softmax(φ(x;θ)), where [pθ(y|x)]j is the softmax probability of the j-th class. For simplicity, let us consider the case where we want to learn only two tasks consecutively, i.e., tasks ‘A’ and ‘B.’ Then, EWC ensures that while learning task ‘B,’ the conditional likelihood pθ(y|xA) does not drift far from the optimal conditional likelihood pθ∗A(y|xA), where θ∗A are the parameters initially optimized for task A:
arg min θ L̃B(θ) = arg min θ
LB(θ) + λEx∼pAX [ DKL(pθ∗A(y|x) || pθ(y|x)) ]
= arg min θ
LB(θ) + λEx∼pAX [ Ey∼pθ∗
A
[ log(
pθ∗A(y|x) pθ(y|x) )
]] (1)
where λ is the regularization coefficient. Note that equation 1 is essentially an optimization with a memory replay regularizer. Here, while learning task B, samples from task A (i.e., a memory buffer from this task) are fed through the network and the conditional likelihood is constantly checked against the optimal conditional likelihood for task A, pθ∗A(y|xA), to ensure a minimal deviation from those parameters.
The key question answered by the EWC framework is on how to avoid memory replay and yet achieve a similar result, i.e., not forget the knowledge from old task (task A). The answer lies in the second-order Taylor expansion of the regularizer around the parameters optimized for the old task, θ∗A. It is straightforward to show (Chaudhry et al., 2018) that the second-order Taylor expansion of
the regularizer is of the form:
Ex∼pAX [DKL(pθ(y|x) || pθ+δθ(y|x))] ≈ 1 2 δθTFθδθ = ‖δθ‖2Fθ (2)
where Fθ is the Fisher Information Matrix (FIM) and is defined as:
Fθ = Ex∼pAX [ Ey∼pθ∗
A
[( ∂log(pθ(y|x))
∂θ
)( ∂log(pθ(y|x))
∂θ
)T]] (3)
see supplementary material for complete derivations. Therefore, when δθ → 0 the KL-divergence regularizer enforces closeness of θ to θ∗A in a Riemmanian pseudo-manifold induced by the FIM. Given that the number of parameters could easily reach several million in standard deep neural networks, it is practically infeasible to store and use the FIM matrix, Fθ. Therefore, Kirkpatrick et al. (2017) assume that Fθ is diagonal and further approximate the KL-divergence with:
Ex∼pAX [DKL(pθ(y|x) || pθ+δθ(y|x))] ≈ 1
2
M∑
m=1
[Fθ]m,m [δθ] 2 m (4)
where M is the total number of parameters in the neural network. This leads to the main equation in the EWC framework:
arg min θ L̃B(θ) = arg min θ LB(θ) + λ 2
M∑
m=1
[ Fθ∗A ] m,m [θ − θ∗A]2m (5)
From our point of view, the critical aspect of these derivations is the connection between memory replay and structural plasticity. Note that we started with equation 1, which uses the idea of a memory replay regularizer. Then by assuming δθ → 0, using the second-order Taylor expansion of the KL-divergence around θ∗A, and assuming that FIM is a diagonal matrix we arrived at equation 5, which provides the idea of synaptic importance parameters and is an embodiment of the selective synaptic plasticity framework. Next, we use this critical aspect and develop an analogous regularizer (i.e., based on memory replay) for the MAS algorithm.

heading: 2.2 GENERALIZING TO UNSUPERVISED LEARNING
text: The EWC framework as explained in the previous section preserves the softmax probability of samples from Task A while learning Task B. This limits the applicability of the method to networks with outputs living on a K-dimensional simplex, e.g., supervised learning and reinforcement learning where the network outputs a probability over a finite set of actions. More recently, Aljundi et al. (2018) presented their Memory-Aware-Synapses (MAS) framework, which lifts the requirement for EWC outputs to live on a simplex, and enables calculation of the synaptic importance parameters even in unsupervised learning and also during testing. While the method is exciting and practically very useful, there is no geometric motivation behind the algorithm. Here we reverse engineer the importance term used in MAS and show a simple regularizer that leads to the MAS algorithm and more importantly provides a geometric interpretation for the algorithm.
Let the regularizer for Task B, be the expected absolute difference between squared `2 norms of the output of the network for samples from Task A, i.e.:
arg min θ L̃B(θ) = arg min θ
LB(θ) + λEx∼pAX [ 1 2 (‖φ(x;θ)‖2 − ‖φ(x;θ∗A)‖2)2 ] (6)
It is straightforward (see Supplementary material) to show that using the second-order Taylor expansion of the above regularizer leads to:
arg min θ L̃B(θ) = arg min θ
LB(θ) + λ M∑
m=1
[Ω]m,m [θ − θ∗A]2m (7)
where [Ω]m,n = Ex∼pAX [ (∂‖φ(x;θ)‖ 2 ∂[θ]m )(∂‖φ(x;θ)‖ 2 ∂[θ]n ) ] , which is the importance parameter used by Aljundi et al. (2018). From a geometric perspective, MAS preserves the norms of the samples from
Task A while learning Task B. In other words; MAS enforces closeness of θ to θ∗A in a Riemmanian pseudo-manifold induced by matrix Ω.
The general idea of using the expected value of a “suitable” distance/divergence between samples, Ex∼pAXd(φ(x;θ), φ(x;θ ∗ A)) and leveraging its second-order Taylor expansion to obtain synaptic importance values is crucial here and could lead to various undiscovered algorithms based on new distances/divergences. We refer to these approaches as sample-based regularization methods.

heading: 2.3 PRESERVING DISTRIBUTION OF AN ARBITRARY LAYER
text: We approach the problem of overcoming catastrophic forgetting from the angle of preserving Task A’s distribution at an arbitrary layer of the neural network, when learning Task B. Let zAi = φ(x A i ;θ) ∈ RK be the output of the network, for a sample from Task A, at the target layer (e.g., output logits in a NN classifier, or reconstructed image of an autoencoder). The distribution of the random variable zA in the target layer follows from the Random Variable Transform (RVT) theorem (Gillespie, 1983):
pAZ(z|θ) = ∫
X pAX(x)δ(z − φ(x;θ))dx (8)
Then we propose the following general regularization to overcome forgetting when learning task B:
arg min θ L̃B(θ) = arg min θ LB(θ) + λd(pAZ(·|θ), pAZ(·|θ∗A)) (9)
where d(·, ·) is a discrepancy measure between the two probability distributions defined in RK . Note that equation 9 could be performed with any discrepancy measure or distance, e.g., the Wasserstein distance (Villani, 2008; Kolouri et al., 2017), using a memory replay strategy. In what follows, we describe a ‘suitable’ distance d(·, ·) that: 1) respects the underlying geometry of the space, and 2) enables a similar strategy to that of the EWC framework to provide importance parameters.

heading: 3 SLICED-CRAMÉR DISTANCE FOR STRUCTURAL PLASTICITY

heading: 3.1 CRAMÉR DISTANCE
text: The p-Cramér distance (Cramér, 1928; Székely & Rizzo, 2013) between two one-dimensional probability density functions p0 and p1 is defined as the `p-norm between their cumulative distribution functions. Note that here we avoid any measure theoretic notations for simplicity. Let qi(t) = ∫ t −∞ pi(τ)dτ denote the cumulative distribution function for pi, then the p-Cramér distance is defined as:
Cp(p0, p1) =
(∫
R |q0(t)− q1(t)|pdt
) 1 p
(10)
for p ≥ 1. Similar to the Wasserstein distance and unlike the KL-divergence and its symmetric form Jensen-Shannon distance (i.e., the square root of the Jensen-Shannon divergence), the Cramér distance respects the underlying geometry of the space. Moreover, the Cramér distance provides unbiased sample gradients (Bellemare et al., 2017), and for p = 1 , C1 is equivalent to the 1- Wasserstein distance, W1. In addition, and similar to the Wasserstein metric, the dual of the Cramér distance is of the form of an integral probability metric (IPM) (Dedecker & Merlevède, 2007).
To further demonstrate the favorable characteristics of this distance, consider the following parametric distribution matching in one-dimension, where the target distribution, p, is a box distribution defined on R and pτ (t) = p(t−τ) is the shifted version of p and the goal is to optimize τ to minimize the distance between pτ and p (τ∗ = 0). For this simple setting, we calculate the energy landscape (i.e., the distance between pτ and p) as a function of τ for the Jensen-Shannon distance, Wp, and Cp for p = 1, 2. Figure 2 shows the distributions p and pτ on the left and the energy landscape as a function of τ on the right. It can be clearly seen that the Wasserstein and Cramér distances respect the underlying geometry of the problem, while the Jensen-Shannon (JS) distance fails to do so.
To extend the Cramér distance to higher-dimensional distributions, we utilize the idea of distribution slicing used in various recent publications (Kolouri et al., 2016; 2019; 2018). We note that the slicedCramér distance (also known as the Cramér-Wold distance) was recently used in Tabor et al. (2018) for generative modeling. We briefly describe the idea in the following section.

heading: 3.2 SLICED-CRAMÉR DISTANCE
text: The idea of slicing a higher-dimensional distribution has roots in the Radon transform that is commonly used in computational tomography. The idea is to represent a high-dimensional distribution via the infinite set of its marginal distributions. In short, let p0 and p1 be d-dimensional probability density functions defined on X ⊂ Rd, then their Radon transform is defined as:
Rpi(t, ξ) = ∫
X pi(x)δ(t− x · ξ)dx (11)
for ∀t ∈ R and ∀ξ ∈ Sd−1 where Sd−1 denotes the d-dimensional unit sphere. Note that Rpi(·, ξ) is a so called slice of pi, which is a one-dimensional marginal distribution of pi. LetRqi(·, ξ) be the corresponding cumulative distribution function ofRpi(·, ξ):
Rqi(t, ξ) = ∫ t
−∞ Rpi(τ, ξ)dτ (12)
then the sliced-Cramér distance between p0 and p1 is the expected value of the Cramér distance between their one-dimensional slices, i.e., Rpi(·, ξ) when ξ ∼ USd−1 for USd−1 being the uniform distribution on the d-dimensional unit sphere. In other words, the sliced Cramér distance is defined as:
SCp(p0, p1) =
(∫
Sd−1 Cpp (Rp0(·, ξ),Rp1(·, ξ))dξ
) 1 p
=
(∫
Sd−1
∫
R |Rq0(t, ξ)−Rq1(t, ξ)|pdtdξ
) 1 p
(13)
Now we are ready to propose our method for overcoming representation forgetting.

heading: 3.3 OVERCOMING REPRESENTATION FORGETTING
text: The critical point here is that the sample-based regularizers could over-estimate the importance of synapses, leading to intransigence faster. More importantly, a substantial expected change in the network’s output (over individual samples) would not necessarily mean catastrophic forgetting. Changes in the network’s output within a mode (e.g., in supervised classification within the distribution of a particular class) are harmless, so long as the representation of the data is not significantly changing. Our proposed regularizer tolerates such changes and therefore has the potential for better utilization of the network’s learning capacity. We propose the following regularization for overcoming catastrophic forgetting:
arg min θ L̃B(θ) = arg min θ LB(θ) + λSC22 (pAZ(·|θ∗A), pAZ(·|θ)) (14)
equation 14 requires memory replay from the old task(s) while learning the new one. To transition from memory replay to selective synaptic plasticity, we derive the second-order Taylor expansion of the regularizer SC22 around the optimal parameters for the previous tasks, θ ∗ A. Assuming that θ = θ∗A + δθ where δθ → 0, it is straightforward to show that: SC22 (p
A Z(·|θ∗A), pAZ(·|θ)) ≈ (δθ)TΓθ∗A(δθ) = ‖δθ‖Γθ∗A (15)
where Γθ∗A is defined as:
Γθ∗A :=
∫
Sd−1
∫
R
( dRqAZ (t, ξ|θ∗A)
dθ
)( dRqAZ (t, ξ|θ∗A)
dθ
)T dtdξ (16)
See supplementary materials for the detailed derivations. Note that similar to Fθ∗A , Γθ∗A is also positive-semi-definite (PSD) and therefore our Sliced-Cramér regularizer enforces closeness of θ and θ∗ in a Riemannian pseudo-manifold induced by the PSD matrix Γθ.
While the definition of Γ in equation 16, regardless of its similarity to equation 3, may seem intimidating, it leads to a straightforward empirical algorithm, which we discuss in the next section. Before that, we point out that similar to the FIM, calculating Γ is also practically infeasible, and we approximate Γ with a diagonal matrix that simplifies the regularization into:
arg min θ L̃B(θ) = arg min θ
LB(θ) + λ M∑
m=1
[Γ]m,m[δθ] 2 m (17)
Finally, we emphasize that while equation 17 is similar to equation 5, it enforces a very different constraint on the neural network. equation 5 enforces conditional class likelihoods to be preserved, which is sample-based; however, our proposed formulation in equation 17 preserves the distribution of network’s outputs at a particular layer for old tasks, i.e., it preserves the distribution of the previously learned representations.

heading: 4 PROPOSED ALGORITHM
text: Here we derive the algorithmic steps required to calculate Γ empirically, as shown in Algorithm 1. The empirical distribution at the network’s output can be written as, pAZ(z|θ) ≈ 1N ∑N n=1 δ(z − φ(xAn ;θ)), and the slices of this empirical distribution are defined as:
RpAZ(t, ξ|θ) ≈ 1
N
N∑
n=1
δ(t− ξ · φ(xAn ;θ))
Let u(·) denote the step function, which is the cumulative distribution of the Dirac delta function. Then the cumulative distribution of RpAZ(t, ξ|θ) can be written as, RqAZ (t, ξ|θ) ≈ 1N ∑N n=1 u(t − ξ · φ(xAn ;θ)). Therefore we have:
dRqAZ (t, ξ|θ) dθ ≈ 1 N
N∑
n=1
(−d ξ · φ(xAn ;θ) dθ ) δ(t− ξ · φ(xAn ;θ)) (18)
substituting equation 18 into equation 16 and using a Monte-Carlo approximation of the integration of Sd−1, with L samples, leads to:
Γ = 1
L
L∑
l=1
( d ξl · z̄ dθ )( d ξl · z̄ dθ )T (19)
where ξls are randomly drawn from SK−1, and z̄ = 1N ∑N n=1 φ(x A n ;θ ∗ A). These derivations give birth to our proposed algorithm shown in Algorithm 1. Given that calculation of matrix Γ is not practically feasible (due to the large number of parameters of a deep neural network), we follow the work of (Kirkpatrick et al., 2017) and approximate Γ to be a diagonal matrix, which simplifies to:
[ Γθ∗A ] i,i = 1
L
L∑
l=1
( 1
N
N∑
n=1
d ξl · φ(xAn ;θ∗A) dθi
)2 = 1
L
L∑
l=1
( d ξl · φ̄∗A dθi )2
where φ̄∗A = 1 N ∑N n=1 φ(x A n ;θ ∗ A).
Algorithm 1: Sliced Cramer Preservation (SCP) Input: Data, XA = {xAn ∼ pAX}Nn=1, and the optimized neural network, φ(·;θ∗A), for Task A. Parameters: Number of random projections, L. Output: Synaptic importance matrix Γ

heading: 4.1 ONLINE EXTENSION OF THE ALGORITHM
text: To extend the framework into sequential learning of multiple tasks and to avoid memorizing taskspecific Γs (or their diagonals), we follow the EWC++ framework proposed by Chaudhry et al. (2018), which is, in essence, identical to the online-EWC proposed by Schwarz et al. (2018). The EWC++ (and online-EWC) methods ameliorate the need for predicting task identities and calculating task-specific FIMs and keep the memory requirement of the method constant (EWC requires linear growth of memory as a function of number of tasks). In EWC++, given F (t−1)θ at task (t−1), the accumulated FIM after learning task t is calculated as F (t)θ = αFθ∗t +(1−α)F (t−1) θ , where Fθ∗t is the task specific FIM for task t, and α ∈ [0, 1) is a hyperparameter that indicates the importance of preserving the most recent task over the older ones. Similarly, we use,
Γ (t) θ = αΓθ∗t + (1− α)Γ (t−1) θ (20)
to obtain the sliced-Cramér regularizer for task (t+ 1).

heading: 5 NUMERICAL EXPERIMENTS

heading: 5.1 PERMUTED MNIST
text: We first test our proposed algorithm on the benchmark permuted MNIST task and compare the performance with online-EWC and our implementation of the online-MAS algorithm. For this experiment, we used a single head model that learns ten tasks, where each task contains a permuted version of the original MNIST dataset. We note that the reported results are a function of the architecture of the underlying network. Meaning that, while all three methods perform well on this task for larger networks, the true competitiveness of the proposed method emerges for smaller networks where “over-estimation” of the synaptic importances significantly hinders learning of the subsequent tasks and leads to intransigence. For this experiment, we used a fully-connected network (i.e.,
a multi-layer perceptron) with the following architecture, 784 → 1024 → 512 → 256 → 10 neurons, and for all optimizations we used the ADAM optimizer with learning rate, lr = 1e − 4. For our proposed method, SCP, we used L = 100 slices. We repeated each experiment 10 times (with different permutations), and reported the average accuracy over all tasks in Figure 3 (a). We can see that SCP is capable of utilizing the capacity of the network better, which points to the fact that regularizing the distribution as opposed to the samples provides a less restricted regularization for the network and still enables the network to freely move individual samples so long as the overall latent distribution of the data is consistent.
The regularization coefficients for each algorithm was cross-validated on the following grid, λ ∈ {1e + i | i ∈ [−3,−2, ..., 9]} and the optimal value was used to report the results in Figure 3 (a). Moreover, in Figure 3 (b) we show the histogram of the logarithm of the product of the regularization coefficients with importance parameters for each method (i.e., λ times the diagonal values of Fθ, Ω, and Γ for EWC, MAS, and SCP, respectively). One can see that the optimal values of the regularization coefficients provide, more or less, scale-consistent synaptic importances for all methods and distinguishing factor between the methods is on the difference between these distributions. Another interesting observation is that the distribution of the synaptic importances are more similar for sample-based methods (i.e., EWC and MAS) compared to the proposed distribution-based method.

heading: 5.2 SEQUENTIAL LEARNING OF AUTO-ENCODERS
text: Next, we consider an experiment consisting of unsupervised/self-supervised sequential learning. To that end, we learn an auto-encoder on single digits of the MNIST dataset sequentially. The model is chosen to be a fully connected auto-encoder, with the following encoder 728 → 1024 → 1024 → 1024 → 256, a mirrored decoder 256 → 1024 → 1024 → 1024 → 784, and Rectified Linear Unit (ReLU) activations. For the loss function, we used cross-entropy plus the `1-norm of the reconstruction error. Similar to the previous experiment, we used the ADAM optimizer (Kingma & Ba, 2014) for training the network with lr = 1e−4. We perform 50 epochs of learning on each digit, before switching to the next digit. For consolidation, we used L = 100 slices for SCP. Finally, we permute the order of the digits and run our experiments 10 times, and compare ‘No Consolidation,’ with MAS, and SCP. Due to the unsupervised nature of the experiment, the EWC framework does not apply here.
Figure 4 demonstrate the results of this experiment. Panel (a) shows the reconstruction of a sample digit from one of our runs with the input digit sequence of [2, 7, 4, 0, 9, 1, 6, 5, 8, 3]. It can be seen that both MAS and SCP can successfully retain the learned knowledge while acquiring new knowledge, while without consolidation the auto-encoder suffers from catastrophic forgetting. Panel (b) shows the average `1-norm of the reconstruction error for all seen tasks over the 10 runs. We can see that SCP and MAS are qualitatively on par, and SCP provides a modest yet statistically significant improvement over MAS for this task.

heading: 5.3 SEMANTIC SEGMENTATION OF SYNTHIA DATASET
text: Lastly, we go beyond the benchmark yet less practical MNIST dataset and address catastrophic forgetting in a more interesting/critical application of autonomous vehicles. We specifically consider the problem of learning semantic segmentation of road scenes in a sequential manner, where the input distribution of the data changes over time. Semantic segmentation is the task of assigning a class label to every pixel of an input image. To that end, we use two sequences of the SYNTHIA dataset (Ros et al., 2016), namely ‘SYNTHIA-SEQS-01-SUMMER’ and ‘SYNTHIA-SEQS-01-WINTER’ as Task 1 and Task 2, respectively. There are 13 classes in the dataset namely: Miscellaneous, Sky, Building, Road, Sidewalk, Fence, Vegetation, Pole, Car, Sign, Pedestrian, Cyclist, and Lane Marking. We keep the last 100 frames of each sequence as the testing-set and train a deep convolutional U-Net architecture (Ronneberger et al., 2015) on the tasks mentioned above. For the loss function, we used (1 − Dice) (Zou et al., 2004), and each task was learned over 100 epochs. For the optimizer, we used the ADAM optimizer (Kingma & Ba, 2014) with learning rate, lr = 1e − 4. For consolidation, we used L = 100 slices for SCP.
We learn the tasks sequentially (summer first and then winter), and a qualitative comparison of online-EWC, MAS, and SCP on a sample test frame is in Figure 5. The first row shows the performance on T1 after learning T1 (base model), the second row shows performance on T2 after learning on T1 and then T2 (indicator of intransigence), and the third row shows performance on T1 after learning on T1 and then T2 (indicator of the catastrophic forgetting). We can see that catastrophic forgetting happens when no synaptic consolidation is leveraged. Moreover, compared to online-EWC and MAS, SCP suffers less from intransigence as fewer artifacts are present in the second row. Lastly, SCP overcomes catastrophic-forgetting more successfully compared to EWC and MAS as it is apparent from the lack of artifacts in the last row. Finally, we provide a quantitative comparison between the methods in Figure 6, where we report the Dice score (Zou et al.,
2004), averaged over ten runs, for each task and each method during the sequential training. As
can be seen, SCP significantly outperforms online-EWC and MAS on this task both in overcoming catastrophic forgetting (left plot) and overcoming intransigence (right plot).

heading: 6 CONCLUSION
text: We introduced a new generic approach towards selective synaptic plasticity for preserving the distribution of the network’s output, at an arbitrary layer, for previously learned tasks. We started from a memory-replay-based regularization that penalized the change in the distribution of the network’s output and showed that a second-order Taylor expansion of such regularization would lead to a selective synaptic plasticity approach that does not require memory of samples from previously seen tasks. Furthermore, we proposed the sliced-Cramér distance as a suitable metric for preserving these distributions, which leads to a straightforward algorithm for selective plasticity. Also, using a similar approach, we reverse-engineered the Memory Aware Synapses (MAS) framework and provided a geometrically meaningful regularization that leads to this algorithm. We then compared the online-EWC, MAS, and SCP methods on a variety of learning tasks, including supervised and unsupervised/self-supervised learning, and consistently showed competitive performance.

heading: 7 ACKNOWLEDGMENTS
text: This material is based upon work supported by the United States Air Force and DARPA under Contract No. FA8750-18-C-0103. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force and DARPA.

heading: 8 SUPPLEMENTARY MATERIALS

heading: 8.1 TAYLOR EXPANSION OF THE KL-DIVERGENCE
text: For the sake of completion, here we derive the second-order Taylor expansion of DKL,
DKL(pθ0 ||pθ) = ∫
X
pθ0(x)log
( pθ0(x)
pθ(x)
) dx
around θ0 where we can write θ = θ0 + δθ. The second-order Taylor expansion is: DKL(pθ0 ||pθ) ≈ DKL(pθ0 ||pθ0) + δθT ( dDKL(pθ0 ||pθ) dθ |θ0 ) + 1 2 δθT ( d2DKL(pθ0 ||pθ) dθ2 |θ0 ) δθ
1. Where for the first-order term we have: dDKL(pθ0 ||pθ)
dθ = −
∫
X
pθ0(x) d log(pθ(x))
dθ dx = −
∫
X
pθ0(x)
pθ(x)
dpθ(x)
dθ dx
Therefore at θ = θ0 we have: dDKL(pθ0 ||pθ)
dθ |θ0 = −
∫
X
dpθ(x)
dθ dx =
d
dθ
(∫
X
pθ0(x)dx
) = 0
2. and for the second-order term: d2DKL(pθ0 ||pθ)
dθ2 =
d
dθ
( − ∫
X
pθ0(x)
pθ(x)
dpθ(x)
dθ dx
)
=
∫
X
pθ0(x) p2θ(x)
( dpθ(x)
dθ
)( dpθ(x)
dθ
)T dx−
∫
X
pθ0(x)
pθ(x)
d2 pθ(x)
dθ2 dx
Therefore at θ = θ0 we have:
d2DKL(pθ0 ||pθ) dθ2
|θ0 = ∫
X
pθ0(x)
( 1
pθ0(x)
dpθ0(x)
dθ
)( 1
pθ0(x)
dpθ0(x)
dθ
)T dx−
∫
X
d2 pθ0(x)
dθ2 dx
=
∫
X
pθ0(x)
( d log(pθ0(x))
dθ
)( d log(pθ0(x))
dθ
)T dx
= Ex∼pθ0
[( d log(pθ0(x))
dθ
)( d log(pθ0(x))
dθ
)T] = F
which is the Fisher Information Matrix (FIM).
Finally, putting everything together we have:
DKL(pθ0 ||pθ) ≈ 1
2 δθTFδθ
which concludes the derivation.

heading: 8.2 TAYLOR EXPANSION OF THE MAS REGULARIZER
text: Here we drive the second-order Taylor expansion of the MAS regularizer in equation 6,
MASreg = Ex∼pX [ 1
2 (‖φ(x;θ)‖2 − ‖φ(x;θ0)‖2)2
]
= 1
2
∫
X
pX(x)(‖φ(x;θ)‖2 − ‖φ(x;θ0)‖2)2dx
around θ0. The second-order Taylor expansion is:
MASreg ≈ δθT ( d MASreg dθ |θ0 ) + 1 2 δθT ( d2MASreg dθ2 |θ0 ) δθ
1. Where for the first-order term we have: [ dMASreg dθ ]
m
=
∫
X pX(x) d ‖φ(x;θ)‖2 d [θ]m (‖φ(x;θ)‖2 − ‖φ(x;θ0)‖2)
which for θ = θ0 is zero. 2. For the second-order term we have:
[ d2MASreg dθ2 ]
m,n
=
∫
X pX(x) d ‖φ(x;θ)‖2 d [θ]m d ‖φ(x;θ)‖2 d [θ]n dx+
∫
X
pX(x)
[ d2 ‖φ(x;θ)‖2
d θ2
]
m,n
(‖φ(x;θ)‖2 − ‖φ(x;θ0)‖2)dx
where evaluated at θ = θ0 the second term on the right-hand-side vanishes.
Finally, putting everything together we have:
MASreg ≈ δθT Ex∼pX
[( d ‖φ(x;θ)‖2
d θ
)( d ‖φ(x;θ)‖2
d θ
)T]
︸ ︷︷ ︸ Ω
δθ
which is the term reported in the paper.

heading: 8.3 TAYLOR EXPANSION OF THE SLICED-CRAMÉR DISTANCE
text: Here we derive the second-order Taylor expansion of the squared 2-Sliced-Cramér distance,
SC22 (pθ0 , pθ) =
∫
Sd−1
∫
R |Rqθ0(t, ξ)−Rqθ(t, ξ)|2dtdξ
around θ0, as reported in equation 16. The second-order Taylor expansion is: SC22 (pθ0 , pθ) ≈ SC22 (pθ0 , pθ0) + δθT ( d SC22 (pθ0 , pθ) dθ |θ0 ) + 1 2 δθT ( d2SC22 (pθ0 , pθ) dθ2 |θ0 ) δθ
1. Where for the first-order term we have:
d SC22 (pθ0 , pθ)
dθ = 2
∫
Sd−1
∫
R
dRqθ(t, ξ) dθ (Rqθ(t, ξ)−Rqθ0(t, ξ))dtdξ
which for θ = θ0 is equal to zero. 2. For the second-order term we have:
d2 SC22 (pθ0 , pθ)
dθ2 = 2
∫
Sd−1
∫
R
d2 Rqθ(t, ξ) dθ2 (Rqθ(t, ξ)−Rqθ0(t, ξ))dtdξ +
2
∫
Sd−1
∫
R
( dRqθ(t, ξ)
dθ
)( dRqθ(t, ξ)
dθ
)T dtdξ
which evaluated at θ = θ0 is:
d2 SC22 (pθ0 , pθ)
dθ2 |θ0 = 2
∫
Sd−1
∫
R
( dRqθ0(t, ξ)
dθ
)( dRqθ0(t, ξ)
dθ
)T dtdξ
= 2Γ
Finally, putting everything together we have:
SC22 (pθ0 , pθ) ≈ δθTΓδθ

heading: 8.4 MNIST-TO-SVHN EXPERIMENT
text: Given the space constraint of the conference, we include our results on the MNIST-to-SVHN experiment in the supplementary material. In this experiment, Task 1 is learning the MNIST digits, and Task 2 is learning the SVHN digits. We show sample images from these two datasets in Figure 7a. In this experiment, we used a VGG-like architecture, for which we include the details in 8.
We performed incremental learning with no consolidation, Online-EWC, MAS, and SCP. The importance parameters were cross-validated on a coarse grid (due to the computational and time restrictions) of {1e+ i|i ∈ [2, 3, 4, 5, 6]}, where the optimal parameters for the methods where 1e+ 3 for SCP, 1e+5 for MAS, and 1e+4 for Online-EWC. We chose 20 epochs per task. Figure 7b shows the results of the experiment. Note that the first plot shows the testing performance of the methods on Task1 (i.e., MNIST dataset), where the blue shade indicates the first 20 epochs (training on the MNIST dataset), and the red shade indicates the second 20 epochs (training on the SVHN dataset). As can be seen, all methods can address catastrophic forgetting, while Online-EWC and SCP outperform MAS (However, a one should perform a finer grid search on the importance-parameters for a definitive evaluation). The middle plot shows the testing performance of the model on the SVHN dataset (Task 2). The MAS and SCP methods outperform online-EWC, and SCP performs slightly better than MAS (i.e., overcomes intransigence better). The third plot shows the average testing performance of the methods on both tasks. Similarly, we can see that SCP outperforms MAS and Online-EWC. We repeated the experiments ten times, and the plots show the average performance.

heading: 8.5 CIFAR100 EXPERIMENTS
text: The CIFAR100 (Krizhevsky et al., 2009) dataset contains images from 20 super-classes, where each super-class contains five sub-classes. For this dataset, we considered an experiment in which the tasks are supervised classification of the super-classes. In short, we split the data into five sequential tasks, where each task contains a sub-class from all the 20 super-classes. We used a Wide Residual Network (Wide-ResNet) Zagoruyko & Komodakis (2016) network as our model and utilized the data augmentation suggested by (DeVries & Taylor, 2017). Figure 9a shows sample super-classes together with data augmentation. For each method, we used a coarse grid search for the importance parameters {1e + i | i ∈ [2, 3, 4, 5, 6]}. Unfortunately, and due to the lack of time, we were able to only run the experiment for each method once. We report the average classification results in Figure 9b. For the final version of the paper, we will complete this experiment with 1) a finer grid search over the importance parameters and 2) multiple runs of the experiments and reporting the average.

Review summaries:
This paper proposes a new method for overcoming catastrophic forgetting in continual learning, based on distribution-based regularization using the sliced Cramer distance, i.e. Sliced Cramer Preservation (SCP). Unlike previous work on catastrophic forgetting, this paper tackles unsupervised learning scenarios as well as supervised learning. They evaluate the proposed SCP on permutated MNIST, sequential learning in autoencoder task, and sequential learning for segmentation.