Differentiable Hebbian Consolidation for Continual Learning
Authors: Vithursan Thangarasa
Thomas Miconi
Graham W. Taylor
Venue:  (2020)
Keywords: continual learning
catastrophic forgetting
Hebbian learning
synaptic plasticity
neural networks

Abstract: Continual learning is the problem of sequentially learning new tasks or knowledge while protecting previously acquired knowledge. However, catastrophic forgetting poses a grand challenge for neural networks performing such learning process. Thus, neural networks that are deployed in the real world often struggle in scenarios where the data distribution is non-stationary (concept drift), imbalanced, or not always fully available, i.e., rare edge cases. We propose a Differentiable Hebbian Consolidation model which is composed of a Differentiable Hebbian Plasticity (DHP) Softmax layer that adds a rapid learning plastic component (compressed episodic memory) to the fixed (slow changing) parameters of the softmax output layer; enabling learned representations to be retained for a longer timescale. We demonstrate the flexibility of our method by integrating well-known task-specific synaptic consolidation methods to penalize changes in the slow weights that are important for each target task. We evaluate our approach on the Permuted MNIST, Split MNIST and Vision Datasets Mixture benchmarks, and introduce an imbalanced variant of Permuted MNIST --- a dataset that combines the challenges of class imbalance and concept drift. Our proposed model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting.

Under review as a conference paper at ICLR 2020
DIFFERENTIABLE HEBBIAN CONSOLIDATION FOR CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Continual learning is the problem of sequentially learning new tasks or knowledge while protecting previously acquired knowledge. However, catastrophic forgetting poses a grand challenge for neural networks performing such learning process. Thus, neural networks that are deployed in the real world often struggle scenar- ios where the data distribution is non-stationary (concept drift), imbalanced, or not always fully available, i.c., rare edge cases. We propose a Differentiable Hebbian Consolidation model which is composed of Differentiable Hebbian Plasticity (DHP) Softmax layer that adds a rapid learning plastic component (compressed episodic memory) to the fixed (slow changing) parameters of the softmax output layer, enabling learned representations to be retained for a longer timescale. We demonstrate the flexibility of our method by integrating well-known task-specific synaptic consolidation methods to penalize changes in the slow weights that are important for each target task. We evaluate our approach on the Permuted MNIST. Split MNIST and Vision Datasets Mixture benchmarks, and introduce an imbal- anced variant of Permuted MNIST - dataset that combines the challenges of class imbalance and concept drift. Our proposed model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting.
INTRODUCTION
A key aspect of human intelligence is the ability to continually adapt and learn in dynamic environ- ments, a characteristic which is challenging to embed into artificial intelligence. Recent advances in machine learning (MI.) have shown tremendous improvements to solve one complex task very well, through extensive training on large datasets with millions of training examples or more. However, most of the ML models that are used during deployment in the real-world are exposed to non-stationarity where the distributions of acquired data changes over time. Therefore, after learning is complete, and these models are further trained with new data, re- sponding to distributional changes, performance degrades with respect to the original data. This phe- nomenon known as calastrophic forgetling or catastrophic interference (McCloskey & Cohen 1989] French 1999) presents crucial problem for deep neural networks (DNNs) that tasked with con- tinual learning [Ring] 1994). also called lifelong learning (Thrun & Mitchell 1995 Thrun 1998). In continual learning, the goal is to adapt and learn consecutive tasks without forgetting how to perform well on previously learned tasks, enabling models that are scalable and efficient over long timescales.
various problems, by learning
In most supervised learning methods, DNN architectures require independent and identically dis- tributed (iid) samples from a stationary training distribution. However, for ML systems in real- world applications that require continual learning, the iid assumption is easily violated when: (1) There is concept drift in the training data distribution.
(2) There are imbalanced class distribu- tions and concept drift occuring simultaneously. (3) Data representing all scenarios in which the learner is expected to perform are not initially available. In such situations, learning systems face the caability-plasticiin dilemma" which is a well-known problem for artificial and biological neural networks (Carpenter & Grossberg. 1987 Abbaham & Robins 2005). This presents continual learn- ing challenge for an ML system where the model needs to provide a balance between its plasticity (to integrate new knowledge) and stability (to preserve existing knowledge).
In biological neural networks, synaptic plasticity has been argued to play an important role in learn- ing and memory (Howland & WWang 2008 Takeuchi et al. 2013 Bailey et al. (015) and two major

---

Under review as a conference paper at ICLR 2020
theories have been proposed to explain a human's 's ability to perform continual learning. The first the- ory is inspired by synaptic consolidation in the mammalian neocortex (Benna & Fusi 2016) where subset of synapses are rendered less plastic and therefore preserved for a longer timescale. The general idea for this approach is to consolidate and preserve synaptic parameters that are considered important for the previously learned tasks. This is normally achieved through task-specific updates of synaptic weights in a neural network. The second is the complementary learning system (CLS) theory (McClelland et al. 1995 Kumaran et al. 2016), which suggests that humans extract high- level structural information and store it in different brain areas while retaining episodic memories.
Recent work on differentiable plasticity has shown that neural networks with "fast weights" that leverage Hebbian learning rules (Hebb) 1949) can be trained end-to-end through backpropagation and stochastic gradient descent (SGD) to optimize the standard "slow weights". as well as also the amount of plasticity in each synaptic connection (Miconi] 2016 Miionii et al. 2018). These works use slow weights to refer to the weights normally used to train vanilla neural networks, which are updated slowly and are often associated with long-term memory. The fast weights represent the weights that are superimposed on the slow weights and change quickly from one time step to the next based input representations. These fast weights behave as a form of short-term memory that enable "reactivation" " of long-term memory traces in the slow weights. [Miconi et al. (2018) showed that simple plastic networks with learned plasticity outperform networks with uniform plasticity on various problems. Moreover, there have been several approaches proposed recently for overcoming the catastrophic forgetting problem in fixed-capacity models by dynamically adjusting the plasticity of each synapse based on its importance for retaining past memories (Parisi et al. 2019).
Here, we extend the work on differentiable plasticity to the mask-incremental continual learning setting (van de Ven & Tolias 2019), where tasks arrive in a batch-like fashion, and have clear moodll boundaries. We develop a Difierentiable Hebbian Consolidation model that is capable of adapting quickly to changing environments as well as consolidating previous knowledge by selectively ad- justing the plasticity of synapses. We modify the traditional softmax layer and propose to augment the slow weights in the final fully-connected (FC) layer (softmax output layer) with set of plastic weights implemented using Differentiable Hebbian Plasticity (DHP). Furthermore, we demonstrate the flexibility of our model by combining it with recent task-specific synaptic consolidation based approaches to overcoming catastrophic forgetting such as elastic weight consolidation (Kirkpatrick) et al. 2017 SShwwaz al. 2018), synaptic intelligence (Zenke et al. 2017b) and memory aware synapses (Aljundi et al. 2018).. Our model unifies core concepts from Hebbian plasticity, synaptic consolidation and CLS theory to enable rapid adaptation to new unseen data, while consolidating synapses and leveraging compressed episodic memories in the softmax layer to remember previous knowledge and mitigate catastrophic forgetting. We test our proposed method on established bench- mark problems including the Permuted MNIST (Goodfellow et al., (013)), Split MNIST (Zenke) et al. 2017b) and Vision Datasets Mixture (Ritter et al.) 2018) benchmarks. We also introduce the Imbalanced Permuted MNIST problem and show that plastic networks with task-specific synaptic consolidation methods outperform networks with uniform plasticity.
2 RELEVANT WORK
Neural Networks with Non-Uniform Plasticity: One of the major theories that have been proposed to explain a human's 's ability to learn continually is Hebbian learning (Hebb) 1949), which suggests that learning and memory are attributed to weight plasticity, that is, the modification of the strength of existing synapses according to variants of Hebb's rule (Paulsen & Sejnowski 2000; [Song et al. 2000 Oja 2008). It a form of activity-dependen synaptic plasticity where correlated activation of pre- and post-synaptic neurons leads to the strengthening of the connection between the two neurons. According to the Hebbian learning theory, after learning, the related synaptic strength are enhanced while the degree of plasticity decreases to protect the learned knowledge (Zenke et al. 2017a).
Recent approaches in the meta-learning literature have shown that we can incorporate fast weights into neural network to perform one-shot and few-shot learning (Munkhdalai & Trischler| 2018 Rae et al. 2018). Munkhdalai & Trischler (2018) proposed a model that augments FC layers preceding the softmax with a matrix of fast weights to bind labels representations. Here, the fast weights were implemented with non-trainable Hebbian sarming-bbsce associative memory, |Re et al. (2018)
'Code is available at:
2

---

Under review as a conference paper at ICLR 2020
proposed Hebbian Softmax layer that can improve learning of rare classes by interpolating between Hebbian learning and SGD updates on the output layer using an engineered scheduling scheme.
Miconi et al. (2018) proposed differentiable plasticity, which uses SGD to optimize the plasticity of each synaptic connection, in addition to the standard fixed (slow) weights. Here, each synapse is composed of a slow weight and plastic (fast) weight that automatically increases or decreases based on the activity over time. Although this approach served to be a powerful new method for training neural networks, it was mainly demonstrated on recurrent neural networks (RNNs) for solving pat- tern memorization tasks and maze exploration with reinforcement learning. Also, these approaches were only demonstrated on meta-learning problems and not the continual learning challenge of over- coming catustrophic forgetting. Our work also augments the slow weights in the FC layer with a set of plastic (fast) weights, but implements these using DHP, We only update the parameters of the softmax output layer in order to achieve fast learning and preserve knowledge over time.
This work leverages two strategies to overcome the catas- trophic forgetting problem: 1) Task-specific Synaptic Consolidation - Protecting previously learned knowledge by dynamically adjusting the synaptic strengths to consolidate and retain memories. 2) CLS Theory- - A dual memory system where, the neocortex (neural network) gradually learns to ex- tract structured representations from the data while, the hippocampus (augmented episodic memory) performs rapid learning and individuated storage to memorize new instances or experiences.
Overcoming Catastrophic Forgetting:
There have been several notable works inspired by task-specific synaptic consolidation for over- coming catastrophic forgetting (Kirkpatrick et al., 2017] Zenke et al., 2017b] Aljundi et al., 2018) and they are often categorized as regularization strategies in the continual learning literature Parisi al. 2019). All of these regularization approaches estimate the importance of each parameter or synapse, Sl, where least plastic synapses can retain memories for long timescales and more plastic synapses are considered less important. The parameter importance and network parameters 0₁ are updated in either an online manner or after learning task Tn. Therefore, when learning new task T,+1, a regularizer is added to the original loss function L (0). so that we dynamically adjust the plasticity w.r.t 2g and prevent any changes to important parameters of previously learned tasks:
[0) = (0) + Σ 24 0m-13
k
regularizer
where 66-1 are the learned network parameters after training on the previous 1 tasks and is a hyperparameter for the regularizer to control the amount of forgetting (old versus new memories).
The main difference in these regularization strategies is on the method used to compute the impor- tance of each parameter, S4. In Elastic Weight Consolidation (EWC), Kirkpatrick et al. (2017) used the values given by the diagonal of an approximated Fisher information matrix for 94, and this was computed offline after training a task was completed. An online variant of EWC was proposed by Schwarz et al. (2088) to improve EWC's scalability by ensuring the computational cost of the regularization term does not grow with the number of tasks. Zenke et al. (2017b) proposed on- line method called Synaptic Intelligence (SI) for computing the parameter importance where, Qg is the cumulative change in individual synapses Memory Aware Synapses (MAS) from Aljundi et al. (2018) is an online method that measures 2z by the sensitivity of the learned function the change in parameters to the loss as seen in SI and EWC.
the entire training trajectory
particular task.
perturbation in the parameters, instead of measuring
Our work draws inspiration from CLS theory which is a powerful computational framework for rep- resenting memories with a dual memory system via the neocortex and hippocampus. There have
been numerous approaches based on CLS principles involving pseudo-rehersal (Robins 1995 Ans
et al., 2004 Atkinson et al,, 2018). exact or episodic replay (Lopez-Paz Paz & Ranzato, 2017 Li &
Hoiem| 2018) and generative replay (Shin et al. 2017 Wu et al., 2018). Exact replay methods re- quire storage of the data from previous tasks which are later replayed. Generative replay methods train a separate generative model to generate images to be replayed. iCaRL (Rebuffi et al., 2017) performs rehearsal and regularization, where external memory is used to store exemplar patterns from old task data and rehearse the model via distillation. However, in our work, we are primarily interested in neuroplasticity techniques inspired from CLS theory for alleviating catastrophic forget- ting. Earlier work from Hinton & Plaut (1987); Gardner-Medwin (1989) showed how each synaptic connection be composed of fixed weight where slow learning stores long-term knowledge and
3

---

Under review as a conference paper at ICLR 2020
fast-changing weight for temporary associative memory. This approach involving slow and fast weights is analogous to properties of CLS theory to overcome catastrophic forgetting during contin- ual learning. Recent research in this vein has included replacing soft attention mechanism with fast weights in RNNs (Ba et al.) 2016), the Hebbian Softmax layer (Rae et al. 2018), augmenting slow weights in the FC layer with fast weights matrix (Munkhdalai & 'Trischler] 2018), differentiable plasticity (Miconi 2016 Miconi et al. 2018) and neuromodulated differentiable plasticity (Miconi al. 2019).
We did not evaluate and compare against neuroplasticity-nnpe they were designed for meta-learning problems and would be unfair evaluate their performance on continual learning benchmark problems given some of their limitations. All of these methods were designed for rapid learning on simple tasks or meta-learning over a distribution of tasks or datasets, where a few number of examples from a class are seen by the network when training on different tasks to perform one-shot and few-shot learning. For instance, the Hebbian Softmax layer modifies its parameters by annealing between Hebbian and SGD updates based on an engineered scheduling scheme which achieves fast binding for rarer classes. However, when a large number of examples observed frequently from the same class, the annealing function switches completely to SGD updates. Thus, when evaluating this model in continual learning setups, the effect of the fast weights memory storage becomes non-existent as the network learns from a large number of examples per class on each task. With a focus on continual learning, the goal of our work is to metalearn a local learning rule for the fast weights via the fixed (slow) weights and an SGD optimizer.
CLS methods as baselines because
DIFFERENTIABLE HEBBIAN CONSOLIDATION
In our model, each synaptic connection in the softmax layer has two weights: 1) The slow weights, 0 E mg where is the number of units in the final hidden layer and d the number of outputs of the last layer. 2) A Hebbian plastic component of the same cardinality as the slow weights, composed of the plasticity coefficient, a, and the Hebbian trace, Hebb. The a is a scaling parameter for adjusting the magnitude of the Hebb. The Hebbian traces accumulate the mean hidden activations of the final hidden layer for each target label in the mini-batch {B1.B) } of size B which are denoted by E 1xx (refer to Algorithm D. Given the pre-synaptic activations of neurons i in h, we can formally compute the post-synaptic activations of neurons j using Eq 2q.2aan and obtain the unnormalized log probabilities (softmax pre- rreaactivttions 2. The softmax function is then applied on 2 to obtain the desired predicted probabilities ú thus, ý softmax(2)). 22). The 1) parameter in Eq. 3 is scalar value that dynamically learns how quickly to acquire new experiences into the plastic component, and thus behaves as the learning rate for the plastic connections. The 1) parameter also acts as decay term for the Hebb to prevent instability caused by positive feedback loop in the Hebbian traces.
172
Zj
Ois + ai Hebb: h4
(2)
1=1
slow
plastic (ass))
Hebb.. + (1 Hebbu, + njhaj
(3)
The network parameters 0ggz, n and 0G,, are optimized by gradient descent as the model is trained sequentially on different tasks in the continual learning setup. In standard neural networks the weight connection has only fixed (slow) weights, which is equivalent to setting the plasticity coefficients a = 0 in E
The Hebbian traces are initialized to zero only at the start of learning the first task Ti and during training, the Hebb is automati- cally updated in the forward pass using Algorithm|] [ Specifically, the Hebbian update for a coressponding class in B1.B is computed on line 6. This Heb- bian update Di= h(y0- c] is anal- ogous to another formulaic description of the Hebbian learning update rule
Hebbian Update Rule:
Algorithm 1 Batch update Hebbian traces.
1: Input: A1.B (hidden activations of penultimate layer),
V1.B (target labels). Hebb (Hebbian trace)
2: Output: 21.B (softmax pre-activations)
3: for each target label c E (yI.B) } do
4:
4- -E [uh m c)
MCount total occurences of y:y
5:
if s > then
6:
h h E hly0 c]
MUpdate Hebb for class e.a/
7: Hebb. x (1 1) Hebb + 17h
8: end if
9: end for
10: <- (0 + aHebb)/
CCCmppue softmax pre-activations.

---

Under review as a conference paper at ICLR 2020
wis N Ex= a.a (Hebb 1949), where Wi,j is the change in weight at connection i. and af a² denote the activation levels of neurons and j. respectively, for the h input. Therefore, in our model, w = h the Hebbian weight update, = h the hidden activations of the last hidden layer, a, = y the corresponding target class in f1.B and N = e the number of inputs for the corresponding class in V1.B (see Algorithm D Across the model's lifetime, we only update the Hebbian traces during training as it learns tasks in a continual manner. Therefore, during test time, we maintain and use the most recent Hebb traces to make predictions.
Our model explores an optimization scheme where hidden activations are accumulated directly into the softmax output layer weights when class has been seen by the network. This results in bet- ter initial representations and can also retain these learned deep representations for a much longer timescale. This is because memorized activations for one class not competing for space with tivations from other classes. Fast learning, enabled by a highly plastic weight component, improves test accuracy for given task. Between tasks this plastic component decays to prevent interference, but selective consolidation into a stable component protects old memories, effectively enabling the model to learn to remember by modelling plasticity over a range of timescales to form a learned neural memory (see Section 4. ablation study). In comparison to an external memory, the advan- tage of DHP Softmax is that it is simple to implement, requiring no additional space or computation. This allows it to scale easily with increasing number of tasks.
The plastic component learns rapidly and per- forms sparse parameter updates to quickly store memory traces for each recent experi- ence without interference from other similar recent experiences. Furthermore, the hidden activations corresponding to the same class, G, are accumulated into one vector h, thus forming compressed episodic memory in the Hebbian traces to reflect individual episodic memory traces (similar to the hippocampus in al.
h
(c) xar-
R(
Figure 1: An example of a Hebbian update for the class, c 6 V1.B- Here, are given the hid- den activations of the final hidden layer, h. Multi- ple hidden activations corresponding class c 6 (represented by the pink boxes) are averaged into one vector denoted by h E I_ This Heb- bian update visualization reflects Lines 4-6 in Al- gorithm I and is repeated for each unique class in the target vector y1.B.
biological neural networks (Chadwick 2010 Schapiro et al.) 2017)) As a result, this method improves learning of rare classes and speeds up binding of class labels to deep repre- sentations of the data without introducing any additional hyperparameters. In Appendix [B] we provide a sample implementation of the DHP Softmax using PyTorch.
Hebbian Synaptic Consolidation: Following the existing regularization strategies such as EWC (Kirkpatrick al. 2017), Online EWC (Schwarz et al. (018). SI (Zenke al., (017b) and MAS (Aljundi et al., 2018), we regularize the Toss L(0) as in Eq. and update the synaptic im- portance parameters of the network in an online manner. We rewrite Eq. [ to obtain the updated quadratic loss for Hebbian Synaptic Consolidation in Eq. 4 and show that the network parameters OG, are the weights of the connections between pre- and post-synaptic activities of neurons and j.
as seen in Eq.
+ A I - i.J
(4)
We adapt the existing task-specific consolidation approaches to our model and do not compute the synaptic importance parameters on the plastic component of the network, hence we only regular- ize the slow weights of the network. Furthermore, when training the first task TT=1, the synaptic importance parameter, 24., in Eq. 4 was set to 0 for all of the task-specific consolidation methods that we tested on except for SI. This is because SI is the only method we evaluated that estimates Pr, while training, whereas Online EWC and MAS compute Pr, after learning a task. The plastic component of the softmax layer in our model can alleviate catastrophic forgetting of consolidated classes by allowing gradient descent to optimize how plastic the connections should be (i.e. less plastic to preserve old information or more plastic to quickly learn new information).
5

---

Under review as a conference paper at ICLR 2020
1.00
100
1961
0.95
Awenge Tes: Aouuayy (N)
0.90
Avenge:
0.80
Number Observed Tasks
Number Observed Tanks
Sofhroo
neture:: 70.73
Online EWM 95.24
Firetue: $0.43
DHP Soltmax + St: 95.2
LHEP Softmax: + MAS: 59.53 MAS. BR.53
DHP Sefmmax+ St 50 51 67.77
DHP Softrox MAA 91
5t.04 84.72
MAS:
(a)
(b)
Figure 2: (a) The average test accuracy on a sequence of 10 Permuted MNIST tasks 1: and (b) a sequence of 5 binary classification tasks from the original MNIST dataset T x=515 The average test accuracy over all learned tasks provided in the legend. The addition of DHP in all cases improves the model's 's ability to reduce forgetting. The error bars correspond to the SEM across 10 trials.
EXPERIMENTS
In our experiments, we compare our approach to vanilla neural networks with Online EWC, SI and MAS. Since our approach increases the capacity of the DNN due to the addition of plastic weights, we add an extra set of slow weights to the softmax output layer of the standard neural network to match the capacity. We do this to show that it is not the increased model capacity from the plastic weights that is helping mitigate the forgetting when performing sequential task learning. thus ensuring a fair evaluation. We tested our model on the Permuted MNIST, Split MNIST and Vision Datasets Mixture benchmarks, and also introduce the Imbalanced Permuted MNIST problem.
For all of the benchmarks, we evaluated the model based on the average classification accuracy on all previously learned tasks as a function of n, the number of tasks trained so far. To determine memory retention and flexibility of the model, we are particularly interested in the test performance on the first task and the most recent one. We also measure forgetting using the backward transfer metric, BWT = - ET= RT, R4,, (Lopez-Paz & Ranzato 2017). which indicates how much learning new tasks has influenced the performance on previous tasks. Ry, is the test classification accuracy on task i after sequentially finishing learning the Tht task. While BWT < directly reports catastrophic forgetting, BWT > O indicates that learning new tasks has helped with the preceding tasks. To establish a baseline for comparison of well-known task-specific consolidation methods, trained neural networks with Online EWC, SI and MAS, respectively, manner. The hyperparameters of the consolidation methods (i.e. EWC, SI and MAS) remain the same with and without DHP Sofumax, and the plastic components are not regularized. Descriptions of the hyperparameters and other details for all benchmarks can be found in Appendix A
all tasks in a sequential
4.1 PERMUTED MNIST
In this benchmark, all of the MNIST pixels are permuted differently for each task with a fixed ran- dom permutation. Although the output domain is constant, the input distribution changes between tasks and is mostly independent of each other, thus, there exists a concept drift. In the Permuted MNIST and Imbalanced Permuted MNIST benchmarks we use a multi-layered perceptron (MLP) network with two hidden layers consisting of 400 ReLU nonlinearities, and a cross-entropy loss. The 7) of the plastic component was set to be value of 0.001 and emphasize that spent little to no effort on tuning the initial value of this parameter (see Appendix A.5 for a sensitivity analysis).
We first compare the performance between our network with DHP Softmax and fine-tuned vanilla MLP network we refer to as Finetune in Figure 2a and no task-specific consolidation methods involved. The network with DHP Softmax alone showed improvement in its ability to alleviate catastrophic forgetting across all tasks compared to the baseline network. Then we compared the performance with and without DHP Softmax using the same task-specific consolidation methods.

---

Under review as a conference paper at ICLR 2020
0.030
Reobaar Leariin RafoGacay
0.750
0.023
Task
[HHbbl]]
20
0.700
Task
6ask
0.055
-5
Tesk
0.650
0.010
0.600
0.000
0.5
1.5
1.0
Iterations
1e5
Berations
Te5
Iterations
165
Figure 3: (left) Hebbian learning rate and decay value (middle) Frobenius Norm of the Hebbian memory traces [|Hebb][r, (right) Frobenius Norm of the plasticity coefficients ||]|| while training each task T1:10-
Figure 2a shows the average test accuracy as new tasks are learned for the best hyperparameter com- bination for each task-specific consolidation method. We find our DHP Softmax with consolidation maintains higher test accuracy throughout sequential training of tasks than without DHP Softmax.
Ablation Study: We further examine the structural parameters of the network and Hebb traces to provide further interpretability into the behaviour of our proposed model. The left plot in Figure shows the behaviour of during training 10 tasks in the Permuted MNIST benchmark are learned continually. Initially, in task Tı, increases very quickly from 0.001 to 0.024 suggesting that the synaptic connections become more plastic to quickly acquire new information. Eventually, " de- cays after the 3dd task to reduce the degree of plasticity to prevent interference between the learned representations. We also observe that within each task from T4 to Tio. 7) initially increases then decays. The Frobenius Norm of the Hebb trace (middle plot in Figure 8) suggests that Hebb grows without runaway positive feedback every time new task is learned, maintaining memory which synapses contributed to recent activity. The Frobenius Norm of (right plot in Figure 8 indicates that the plasticity coefficients grow within each task, indicating that the network is leveraging the structure in the plastic component. It is important to note that gradient descent and backpropagation are used as leta-le to tune the structural parameters in the plastic component.
4.2 IMBALANCED PERMUTED MNIST
We introduce the Imbalanced Permuted MNIST problem which is identical the Permuted MNIST benchmark but, now each task is an imbalanced distribution where training samples in each class were artificially removed based on some random probability (see Appendix A.2). This benchmark was motivated by the fact that class imbalance and concept drift can hinder predictive performance, and the problem becomes particularly challenging when they occur simultaneously. Appendix [ Figure [5] shows the average test accuracy for the best hyperparameters of each method. We see that DHP Softmax achieves 80.85% after learning 10 tasks with imbalanced class distributions in sequential manner, thus providing significant 4.41% 1 improvement over the standard neural network baseline of 76.44%. The significance of the compressed episodic memory mechanism in the Hebbian traces is more apparent in this benchmark because the plastic component allows rare classes that are encountered infrequently to be remembered for a longer period of time. We find that DHP Softmax with MAS achieves 0.04 decrease in BWT, resulting in average test accuracy of 88.80% and 1.48% improvement over MAS alone; also outperforming all other methods and across all tasks.
4.3
SPLIT MNIST
We split the original MNIST dataset (LeCun et al., 2001) into a sequence of 5 binary classification tasks: T, {0/1), T. (2/3). T3 = [4/5]. T4 [6/7] and Ts = [8/9). The output spaces are disjoint between tasks, unlike the previous two benchmarks. Similar to the network used by Zenke al. (2017b), we use an MLP network with two hidden layers of 256 ReLU nonlinearities each, and a cross-entropy loss. The initial 17 value was set to 0.001 as seen in previous benchmark experiments. We found that different values of 7 yielded very similar final test performance after learning T3 tasks (see Appendix A.5). We observed that DHP Softmax alone achieves 98.23% thus, provides a 7. 7.80% improvement test performance compared to finetuned ML. network (Figure 26). Also,

---

Under review as a conference paper at ICLR 2020
combining DHP Softmax with task-specific consolidation consistently decreases BWT, leading to higher average test accuracy across all tasks, especially the most recent one, Tg.
4.4 VISION DATASETS MIXTURE
Following previous works (Ritter et al. 2018 Zeno et al. 2018), 8). perform continual learning on sequence of 5 vision datasets: MNIST, notMNIST FashionMNIST (Xiao et al. (017). SVHN (Net- zer et al. 2011) and CIFAR-10 (Krizhevsky 2009) (see Appendix A.4 for dataset details). The MNIST, noLMNIST and FashionMNIST datasets are zero-padded to be of size 32x32 and are repli- cated 3 times to create grayscale images with 3 channels, thus matching the resolution of the SVHN and CIFAR-10 10 images. Here, we use a CNN architecture that is similar to the one used in (Ritter) ict al. 2018 Zeno et al. 2018) (more details in Appendix ().4). The initial r7 parameter value was set to 0.0001. .. We train the network with Mini-batches of size 32 and optimized using plain SGD with fixed learning rate of 0.01 for 50 epochs per task.
We found that DHP Softmax plus MAS decreases BWT by 0.04 resulting in 2.14% improvement in average test accuracy over MAS on its own (see Table [ and Appendix [A.6] Figure 6 Also, SI with DHP Softmax outperforms other competitive methods with an average test performance of 81.75% and BWT of -0.04 after learning all five tasks. In Table we present summary of the final average test performance after learning all tasks in the respective continual learning problems. Here, we summarize the average test accuracy and BWT across ten trials for each of the benchmarks.
Table 1:
The average test accuracy (%. higher is better) and backward transfer (BWT, lower is better) after learning all tasks on each benchmark, respectively. The results are averaged over 10 trials.
Method
Permuted MNIST
Imbalanced Permuted MNIST
SplitMNIST
5- Vision
Finetune
76.73 / -0.19
76.44 / -0.20
90.43/-0.13
60.02 / -0.33
DHP Softmax
78.49 -0.16
80.85 -0.14
98.23/
62.94/ -0.26
SI
84.72 / -0.13
85.92 / -0.06
97.77/-0.04
81.26/ -0.06
DHP Softmax + SI
85,20 -0,09
85,39 / -0.06
99.15 0.00
81.75 -0.04
Online EWC
86.24 / -0.11
87.18/ -0.09
97.65/-0.03
78.61 / -0.07
DHP Softmax + Online EWC
87.30 -0.09
87.43 -0.08
98.96/00.01
79.10 -0.04
MAS DHP Softmax + MAS
88.52 / -0.08
87.32 / -0.09
98.24/-0.02
78.51 / -0.05
89.53 -0.06
88.80 -0.05
98.43/ -0.01
80.66 -0.01
DISCUSSION AND CONCLUSION
We have shown that the problem of catastrophic forgetting in continual learning environments be alleviated by adding compressed episodic memory in the softmax layer through DHP and perform- ing task-specific updates on synaptic parameters based on their individual importance for solving previously learned tasks. The compressed episodic memory allows new information to be learned individual traces without overlapping representations, thus avoiding interference when added to the structured knowledge in the slow changing weights and allowing the model to generalize across experiences. The a parameter in the plastic component automatically learns to scale the magnitude of the plastic connections in the Hebbian traces, effectively choosing when to be less plastic (pro- tect old knowledge) or more plastic (acquire new information quickly). The neural network with DHP Softmax showed noticeable improvement across all benchmarks when compared to neural network with a traditional softmax layer that had an extra set of slow changing weights. The DHP Softmax does not introduce any additional hyperparameters since all of the structural parameters of the plastic part a and are learned, and setting the initial value required very little tuning effort.
at http:///stos.awvb.b.ooooct..... notmnist -taannsstddttsttttt. html and downloaded from notMNIST-CO
Originally
published
8

---

Under review as a conference paper at ICLR 2020
We demonstrated the flexibility of our model where, in addition to DHP Softmax, can perform Hebbian Synaptie Consolidation by regularizing the slow weights using EWC, SI or MAS to im- prove model's ' ability to alleviate catastrophic forgetting after sequentially learning a large number of tasks with limited model capacity. DHP Softmax combined with SI outperforms other consolida- tion methods the Split MNIST and 5-Vision Datasets Mixture. The approach where combine DHP Softmax and MAS consistently leads to overall superior results compared to other baseline methods on the Permuted MNIST and Imbalanced Permuted MNIST benchmarks. This is interest- ing because the local variant of MAS does compute the synaptic importance parameters of the slow weights f., layer by layer based on Hebb's rule, and therefore synaptic connections i. that are highly correlated would be considered more important for the given task than those connections that have less correlation. Furthermore, our model consistently exhibits lower negative BWT across all benchmarks, leading to higher average test accuracy over methods without DHP. This gives a strong indication that Hebbian plasticity enables neural networks to learn continually and remember distant memories, thus reducing catastrophic forgetting when learning from sequential datasets in dynamic environments. Furthermore, continual synaptic plasticity can play a key role in learning from limited labelled data while being able adapt and scale long timescales. We hope that work will open new investigations into gradient descent optimized Hebbian consolidation for learning and memory in DNNs to enable continual learning.
REFERENCES
Wickliffe C. Abraham and Anthony Robins. Memory retention the synaptic stability versus plas- ticity dilemma. Trends in Neurosciences, 28(2):73-78, February 2005.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In The European Conference on Com- puter Vision (ECCV), September 2018.
Bernard Ans, Stéphane Rousset, Robert M. French, and Serban Musca. Self-refreshing memory in artificial neural networks: learning temporal sequences without catastrophic forgetting. Connec- fion Science, 16(2):71-99, June 2004.
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony V. Robins. Pseudo-recursal: Solving the catastrophic forgetting problem in deep neural networks. CoRR, abs/1 802.03875, 2018.
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems 29,
pp. 4331-4339 2016.
Craig H. Bailey, Eric R. Kandel, and Kristen M. Harris. Structural components of synaptic plasticity and memory consolidation. Cold Spring Harbor Perspectives in Biology, 7()):021758, July 2015.
Marcus K Benna and Stefano Fusi. Computational principles of synaptic memory consolidation.
Nature Neuroscience, 19(12)1697-1106, October 2016.
Gail A. Carpenter and Stephen Grossberg. A massively parallel architecture for a oelf-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37(1): 54-115, January 1987.
Martin J. Chadwick, Demis Hassabis, Nikolaus Weiskopf, and Eleanor A. Maguire. Decoding in- dividual episodic memory traces in the human hippocampus. Current Biology, 20(6):544 547, March 2010.
R French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4): 128-135, April 1999.
A. R. Gardner-Medwing Doubly modifiable synapses: A model of short and long term auto- associative memory. Proceedings of the Royal Society B: Biological Sciences, 238 154, November 1989.

---

Under review as a conference paper at ICLR 2020
Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning (ICML), pp. 111-1319-1111----111111111111111 II1-1327, 2013.
D. O. Hebb. The organization of behavior; neuropsychological theory. Wiley, Oxford, England, 1949.
Geoffrey E. Hinton and David C. Plaut. Using fast weights to deblur old memories. In Proceedings of the 9th Annual Conference of the Cognitive Science Society, pp. 177-186, 1987.
John G. Howland and Yu Tian Wang. Chapter 8 synaptic plasticity in learning and memory: Stress effects in the hippocampus. In Essence of Memory, volume 169 of Progress in Brain Research, pp. 145 158. Elsevier, 2008.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins. An- Demis
drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 114 13: 3521 3326,356,,200 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Dharshan Kumaran, Demis Hassabis, and James L. McClelland. What learning systems do intelli- gent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7):512 534, 2016.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. In IEEE Intelligent Signal Processing, pp. 306-351. 2001.
Z. Li and D. Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2947, Dee 2018
David Lopez-Paz. and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30, pp. 6467-6476. 2017.
J. L. McClelland, L. McNaughton, and O'Reily,. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connec- tionist models of learning and memory. Psychological Review, 022:419-557, 1995.
Michael McCloskey and Neil J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. The Psychology of Learning and Motivation, 24:104-169, 1989.
Thomas Miconi. Learning to learn with backpropagation of hebbian plasticity. CoRR, 0tb//609..2228, 2016.
Thomas Miconi, Kenneth O. Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp.
2018.
Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth O. Stanley. Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity. In International Conference Learning Representations (ICLR), 2019.
Tsendsuren Munkhdalai and Adam Trischler. Metalearning with hebbian fast weights. CORR, abs/1807.05072 2018
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Erkki Oja. Oja learning rule. Scholarpedia, 3(3):3612, 2008.
Genman I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 11::54-71, May 2019.
10

---

Under review as a conference paper at ICLR 2020
Ole Paulsen and Terrence Sejnowski. Natural patterns of activity and long-term synaptic plasticity. Current Opinion in Neurobiology, 10(2):172 180, 2000.
Jack W. Rae, Chris Dyer, Peter Dayan, and Timothy P. Lillicrap. Fast parametric learning with acti- vation memorization. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 4225-4234 2018.
Sylvestre-Alvise Rebufñi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In IEEE Conference on Computer Vision and Pattern Recognition CVPR, pp. 5533-5542, 2017.
Mark Bishop Ring. Continual Learning in Reinforcement Environments. PhD thesis, Austin, TX, USA, 1994. UMI Order No. GAX95-06083.
Hippolyt Ritter, Aleksandar Botev, and David Barber, Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems 31, pp. 3738-3748. 2018.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7: 123-146, 1995.
Anna C. Schapiro, Nicholas B. Bark-Browne, Matthew M Botvinick, and Kenneth A. Norman. Complementary learning systems within the hippocampus: neural network modelling approach to reconciling episodic memory with statistical learning. Philosophical transactions of the Royal Society of London. Series Biological sciences, 372 1711, 2017.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barne Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In Proceedings of the 35th International Conference on Machine Learning (ICML),
pp. 4535-4544, 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems 30, pp. 2990-29999999 2017.
Sen Song, Kenneth D. Miller, and L. F. Abbott. Competitive hebbian learning through spike-timing- dependent synaptic plasticity. Nature Neuroscience, 399:99199226, September 2000.
Takeuchi, J. Duszkiewiez, and G. M. Morris. The synaptic plasticity and memory hy- pothesis: encoding, storage and persistence. Philosophical Transactions of the Royal Society B: Biological Sciences, (00203)):221002222211133 December 2013.
Sebastian Thrun. Lifelong Learning Algorithms, pp. 181-209. Springer US, Boston, MA, 1998.
Sebastian Thrun and Tom M. Mitchell. Lifelong robot learning. Robotics and Autonomous Systems, 15(1):25 46, 1995. The Biology and Technology of Intelligent Autonomous Agents. -6,
Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. CoRR, abs/1904.07734, 2019.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint
2019.
Chenshen Wu, Luis Herranz, Xialei Liu, Yaxing Wang. Joost van de Weijer, and Bogdan Raducanu. Memory replay gans: Learning generate categories without forgetting. In Advances in Neural Information Processing Systems 31, pp. 5962-5972. 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. caahion-mnisse a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/ 077707777777..... 2017.
Friedemann Zenke, Wulfram Gerstner, and Surya Ganguli. The temporal paradox of hebbian learn- ing and homeostatic plasticity. Current Opinion in Neurobiology, 43:166 176, 2017a. Neurobi- ology of Learning and Plasticity.
11

---

Under review as a conference paper at ICLR 2020
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 3987- 3995, 2017b.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. CoRR, abs/ 100.11003.1122 2018.
12

---

Under review as a conference paper at ICLR 2020
A DETAILS ON EXPERIMENTAL SETUP AND HYPERPARAMETER SETTINGS
In the continual learning setup, we train a neural network model on a sequence of tasks T Timmaxx where Temax is the maximum number of tasks the model to learn in the respective benchmarks. Unlike the conventional supervised learning setup, continual learning trains a model on data that is fetched in sequential chunks enumerated by tasks. Therefore, in a continual learning sequence, the model receives a sequence of tasks Texmax that is to be learned, each with its associated training data Vn). where Xr the input data and the corresponding label data denoted by Vn. Each task Tn has its own task-specific loss L", that will be combined with a regularizer loss term (refer to Eq. 4 to prevent catastrophic forgetting. After training is complete, the model will have learned an approximated mapping f to the the true underlying function f. The learned f maps new input X to the target outputs VI for all Tim tasks the network has learned so far. Also, it is to be noted that the of classes contained in each task can be different from each other, we have done in the SplitMNIST and Vision Datasets Mixture benchmarks. All experiments were run on either a Nvidia Titan V Nvidia RTX 2080 Ti.
A.1 PERMUTED MNIST
We train the network on a sequence of tasks T==1:11 with mini-batches of size 64 and optimized using plain SGD with a learning rate of 0.01. We train for at least 10 epochs and perform early- stopping once the validation error does not improve for 5 epochs. If the validation error increases for more than 5 epochs, then we terminated the training on the task Tn reset the network weights and Hebbian traces to the values that had the lowest test error, and proceeded to the next task.
Hyperparameters: For the Permuted MNIST experiments shown in Figure 2a the regularization hyperparameter À for each of the task-specific consolidation methods is set
100 for Online EWC (Schwarz et al., 2018). À = 0.1 for SI (Zenke et al. 2017b) and X = 0.1 for MAS (Aljundi et al. 2018). We note that for the SI method, À refers the parameter in the original work (Zenke et al. 2017b) but we use À to keep the notation consistent across other task-specific consolidation 0.1. To find the best hyperparameter combi- nation for each of these synaptic consolidation methods, we performed grid search using task sequence determined by single seed. For Online EWC, we tested values e {10, 20, 50.... 400), -I-X À {0.01, 0.05...., 0.5, 1.0} and MAS - X E {0.01, 0.5. 1.5, 2.0}.
methods. In SI, the damping parameter, was
A.2 IMBALANCED PERMUTED MNIST
For each task in the Imbalanced Permuted MNIST problem, we artificially removed training sam- ples from each class in the original MNIST dataset (LeCun al. 2001) based on some random probability. For each class and each task, we draw a different removal probability from a standard uniform distribution UO, and then remove each sample from that class with that probability. The distribution of classes in each dataset corresponding to tasks T 1:11:10 is given in Table
Table 2: Distribution of classes in each imbalanced dataset for the respective tasks T T2=1:10
Classes
Tasks
2
4
6
9
10
0
4459
3780
1847
3820
5867
122
1013
4608
908
3933
1
1872
3637
1316
6592
1934
1774
5533
2569
831
886
2
2391
4125
2434
4966
5245
4593
4834
4432
3207
3555
3
4433
1907
1682
278
3027
2315
5761
3293
2545
3749
4
186
2728
2002
151
1435
5829
1284
3910
4593
927
5
4292
2472
2924
1369
4094
4858
2265
3289
1134
1413
2339
3403
4771
5569
1414
2851
2921
4074
336
3993
7
4717
3090
4800
2574
4086
1065
3520
4705
5400
3650
8
3295
5493
76
4184
2034
4672
682
196
2409
1709
9
2625
3880
4735
1647
2645
3921
901
4546
4649
2045
Total
30609
34515
26587
31120
31781
32000
287 14
35622
26012
25860
13

---

Under review as a conference paper at ICLR 2020
For the Imbalanced Permuted MNIST experiments shown in Figure the regularization hyperpa- rameter À for each of the task-specific consolidation methods is = 400 for Online EWC (Schwarz et al. 2018). À= 1.0 for SI (Zenke al. 2017b) and À 0.1 for MAS (Aljundi et al. 2018]. In SI, the damping parameter, &. was set to 0.1. Similar to the Permuted MNIST benchmark, to find the best hyperparameter combination for each of these synaptic consolidation methods, we performed grid search using a task sequence determined by a single seed. For Online EWC, we tested values of A E {50, 100....,I I x 10"), SI - E {0.1. 0.5..... 2.5, 3.0} and MAS À E {0.01, 0.05, .... 1.5, 2.0}. Across all experiments, we maintained the the same random probabilities detemined by
single seed to artificially remove training samples from each class.
A.3 SPLIT MNIST
Hyperparameters: For the Split MNIST experiments shown in Figure 2b the regularization hyperparameter À for each of the task-specific consolidation methods is = 400 for Online EWC (Schwarz et al. 2018), À 1.0 for SI (Zenke al. 2017b) and X 1.5 for MAS (Aljundi) et al. 2018). In SI, the damping parameter, E. was set to 0.001. To find the best hyperparameter combination for each of these synaptic consolidation methods, we performed grid search using the
5 task binary classification sequence (0/1, 2/3, 4/5, 6/7, 8/9). For Online EWC. we tested values of E {1. 25, 50, 100, ..... Ix. 10 2x 10 SI E {0.1, 0.5, 1.0, ..... 5.0} and MAS 0.05, 1.0.... 4.5, 5.0}. We train the network on a sequence of 1==1: tasks with
{0.01, of size 64 and optimized using plain SGD with a fixed learning rate of 0.01 for 10 epochs.
A.4 VISION DATASETS MIXTURE
The Vision Datasets Mixture benchmark consists of sequence of tasks where each task is different image classification dataset: MNIST, notMNIST, FashionMNIST, SVHN and CIFAR-10. The notMNIST dataset consists of font glypyhs corresponding to letters 'A' to 'J'. The original dataset has 500,000 and 19,000 grayscale images of size 28 28x28 for training and testing, respectively. However, similar MNIST, we only use 60,000 images for training and 10,000 for testing. FashionMNIST consists of 10 categories of various articles of clothing, and there are 60,000 and 10,000 grayscale images sized 28 x8 for training and testing, respectively. SVHN consists of digits '0' to '9' from Google Street View images and there are 73,257 and 26,032 colour images of size 32x32 for training and testing, respectively. CIFAR-10 10 consists of 50,000 and 10,000 colour images of size 32x32 from 10 different categories for training and testing. respectively.
Dataset Details:
The CNN architecture consists 2 convolutional layers with 20 and 50 channels respectively, and a kernel size of 5. Each convolution layer is followed by LeakyReLU nonlinearities (negative threshold of 0.3) and 2x2 max-pooll max-pooling operations with stride 2. The two convolutional layers are followed by an FC layer of size 500 before the final softmax output layer (refer to Table 3 Similar to (Ritter et al. 2018 Zeno et al. 20188) a multi-headed approach was used because the class definitions are different between datasets.
Architecture:
In the other benchmark problems, we use a single 1) across all connections. In this benchmark, our model has trainable 1) value for each connection in the final output layer thus, 7) E Rx and set the initial y value to be 0.0001. We found that using separate 1) parameters for each connection improved the stability of optimization and convergence optimal test performance. This allows each plastic connection to modulate its own rate of plasticity when learning new experiences. It was observed that using a single 77 value across all connections lead to instability of optimization on the SVHN and CIFAR- 10 tasks.
Hyperparameters: For the V-Vision Datasets Mixture experiments shown Figure the regular- ization byperparameter À for each of the task consolidation methods is X 100 for Online EWC (Schwarz et al.) 2018)) À 0.1 for SI (Zenke al.) 20176) and X 1.0 for MAS (Aljundi) et al. 2018). In SI, the damping parameter, E. was set to O. To find the best hyperparameter combination for each of these synaptic consolidation methods, we performed random search using the same task sequence ordering (MNIST, notMNIST, FNSH, SVHN and CIFAR-10). For Online EWC, we tested values of À E {10, 50, 100.... 500), SI - X E {0.01, 0.05, 0.1...., 1.0] and MAS X {0.01, 0.05, 1.0.... 4.5, 5.0).
14

---

Under review as a conference paper at ICLR 2020
Table 3: Network architecture used for Vision Datasets Mixture benchmark in Section 4.4 For convolutional layers, the output size denotes channel size of output. The negative threshold for all of the LeakyReLU nonlinearities were set 0.2.
output size
kernel
padding
stride
convolution
20
(5
(1,1)
(1,1)
LeakyReLU
MaxPool
(2
convolution
50
()
(I.
(I,
LeakyReLUU
MaxPool
(2.
convolution
50
(5.5)
(1,
(I.
LeakyReLU
MaxPool
(2,2)
fully-connected
500
LeakyReLU
fully-comnected
10
A.5 SENSITIVITY ANALYSIS
We provide a summary of the sensitivity analysis performed on the Hebb decay term and show its effect on the final average test performance after learning a sequence of tasks in the continual learning setup. The plots on the left and center in Figure 4 show the effect of the initial value on the final performance after learning tasks 1:11 in a sequential manner for the Permuted MNIST and Imbalanced Permuted MNIST benchmarks, respectively. We swept through range of values v E {0.1, 0.01, 0.001, 0.0005, 0.0001) and found that setting ? to low values led to the best performance in terms of being able alleviate catastrophic forgetting. Similarly, performed a sensitivity analysis on the parameter for the Split MNIST problem (see the rightmost plot in Figure 4p. Table 4 presents the average test accuracy across 5 trials for the MNIST-variant benchmarks, which corresponds to the sensitivity analysis plots in Figure 4
also
Aicurary
85
Test AuCLraSS (h)
100.0
08.5
90
90
08.0
85
08.5
98.0
10
8
Number af Observed Tasks
Number Observed Tasks
Number ol Observed Taaks
4- 0.1
0.0001
0.1
0.0001
0- 0.0011
9-
n= 0.0005
n
7- 0.003
1- 0.001
Figure 4: A sensitivity analysis on the Hebb decay term in Eq. 3 We show the average test accu- racy for different initial values of after learning all tasks the (left) Permuted MNIST, (center) Imbalanced Permuted MNIST and (right) Split MNIST problems. The shaded regions correspond the standard error of mean (SEM) across trials.
15

---

Under review as a conference paper at ICLR 2020
Table The average test accuracy (%, higher better) for different initial 7) values after learning all tasks on the Permuted MNIST, Imbalanced Permuted MNIST and Split MNIST continual learning benchmarks, respectively. The results are averaged over 5 trials.
Hebbian Plasticity Decay Term (1))
Permuted MNIST
Imbalanced Permuted MNIST
SplitMNIST
7) 0.1
77.43
80.00
98.43
17 0.01
78.60
80.13
98.
1 0.001
78.49
80.85
98.32
0.0001
77.83
80.47
98.43
17 0.0005
78.05
80.40
98.37
A.6 ADDITIONAL FIGURES FOR SPLITNNIST AND VISION DATASETS MIXTURE
1.00
Avernge Test Accuratt (%)
0.85
Number Observed 7
10
DHP Softmax Online Online EWC: 18
Fineture: 76.44
DHP Softmax Sl: 85.39
DHP Softmax + MAS: 68.8
S1: 85.92
MAS: 87.32
Figure 5: The average test accuracy a sequence of on sequence of 10 imbalanced Permuted MNIST tasks T1=1:10- The average test accuracy over all learned tasks is provided in the legend. The shaded regions correspond to the SEM across 10 trials.
100
Average Test Accuracy (%)
95
90
85
80
75
70
65
60
Number Observed Tasks
DHP Softmax: 62.94
DHP Softmmx + Online EWC: 79.1
Finetune: 60.00
Online 78.61
Softmax
St: 81.26
MAS: 76.51
Figure 6: The average test accuracy on a sequence of diffferent vision datasets Ta- The average test accuracy over all learned tasks is provided in the legend. The error bars correspond to the SEM across 10 trials.
16

---

Under review as a conference paper at ICLR 2020
B EXAMPLE PYTORCH IMPLEMENTATION OF DHP SOFTMAX LAYER
class DHP Softmax Layer (nn. Module):
def init (self in features out features, eta rate 00.011:011): self). -
super (DHP_SS Softmax
Applies linear transformation to the hidden activations of the last hidden layer with additional plastic component implemented using Differentiable Hebbian Plasticity (DHP): math: (w + \alpha Hebb)h
Args:
featuree::: size of each input in last hidden layer. fut number of classes. eta.rate: initial learning rate value of plastic connections.
10
12
Returns:
13
2: the softmax pretactivttions (unnormalized log probabilities).
14
hebb: the updated Hebbian traces for the next iteration.
self. fn-tureures ie_fuatures self. out_features out.features self. eta_rate = eta_rate
17
18
19
# Initialize fixed (slow) weights with He initialization.
self. weight Parameter (torch, Tensor (seff iww... in_ self. fetturrr)
22
23
init. kaiming uniform- weight, a=math.sqrie
24
Initialize alpha scaling coefficients for plastic connections.
26
self alpha Parameter ((.01 torch rand self. fnatuurs self self.ontt.eattres)) requires.grad=TTrue)
27
28
Initialize the learning rate of plastic connections.
31
self eta = Parameter (( self rtae torch (n)s (ruuires-rrad-rre)
32
33
def forward h, y. hebb):
35
if self training:
36
e in enumerate (torch, un)que
37
# Get indices of corresponding class, c, in y.
с idx = (y = c). ()
# Count total occurences of corresponding class in y.
40
s torch sum c)
41
42
if >>>>> 0:
# Perform Hebbian update (lines 6-7 in Algorithm 1) h_bar torch div (torch sum -eidxx 0),
45
s. item ())
46
hebb[: = torch add (torch. mul mulltore hebb [c] clone ())
(1, self torch forch.mul(( mathhmmllhhhbbb......... (h.bar. self.eta))
49
50
# Compute softmax pre-activations with plastic (fast) weights. z = torch mm self, weight + torch mul self. alpha, hebb))
51
2, hebb
54
55
def initial zero hebb (self):
return Variable (torch. zeros (self. in.features. self. futtueatu
17

---

Under review as a conference paper at ICLR 2020
requires-grad=Faase
Listing I: PyTorch implementation of the DHP Softmax model which adds a compressed episodic memory to the final output layer a neural network through plastic connections as described in Algorithm frameworks.
We want to emphasize the simplicity of implementation using popular ML
C DIFFERENTIABLE HEBBIAN CONSOLIDATION
Standard Weight Dual-Weight
Output Layer
Layers
Input Layer
Task
Task 2
Task
Task 2
(a) Task-Spocific Synaptic Consolidation
(b) Difiorentiable Hebbian Consolidation
Figure 7: The difference between (a) task spsciiicc consolidation methods (e.g.. EWC, SI and MAS) and (b) Diffcrentiable Hebbian Consolidation a simple two tasks continual learning scenario. The baseline standard weight connections connections are represented by red lines. After learning Task 1, the parts of the network which are deemed important for Task are indicated in orange. Task 2 has different of input patterns and utilizes the parts shown in green. The Differentiable Hebbian Consolidation framework consolidates representations learned by the network and the hidden activations are accumulated directly into the softmax output layer parameters when class has been seen by the network. This allows the network retain these learned deep representations for longer timescale without having sacrifice as much capacity by dynamically learning to adjust the degree of plasticity in the weights.
represented by light gray lines and the dual-weight plastic
18

---

Under review as a conference paper at ICLR 2020
D ADDITIONAL EXPERIMENTS
Split CIFAR-10/100
80
70
Average Test Tost Accuracy (%%
80
50
DHP Softmax: 61 61.28
Finetune: 47.5
40
Sl: 73.33
Scratch: 64.83
30
20
10
CIFAR-10
Task T
Figure 8: This replicates the Split 1IFAR-10/100 experiment of Zenke et al. (017b). First, the network trained on the full CIFAR-I0 dataset (Task 1==1) and sequentially on additional tasks each corresponding to 10 consecutive classes from the CIFAR-100 dataset (Tasks 6x= The test accuracies of CIFAR-10 and the CIFAR-100 splits are reported after having learned the final task in this sequence. The DHP Softmax (purple) alone significantly outperforms Finetune (yellow) on each of the tasks in this class-incremental learning setup. On some tasks, DHP Softmax alone performs as well or better than when training from scratch (light green). The test accuracies of Finetune, when training from scratch and SI (turquoise) were taken from von Oswald et al. (2019).
19


Reviews:
[{"review_id": "r1xvrhmUdr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the continual learning setting, and aims to mitigate catastrophic forgetting, with results on Permuted MNIST, Split MNIST, Vision Datasets Mixture, and their own class-imbalanced version of the Permuted MNIST dataset. The authors propose to augment differentiable plastic weights - a general neural network component - with class-specific updates (similarly to prior work, such as the Hebbian softmax) at the final layer of a neural network, prior to a softmax. While well-motivated in terms of the background and methodology (indeed, this is a simple way to prevent interference in fast weights), and nicely explored experimentally with lots of examinations into the workings of the method, the weak results on the simpler continual learning settings lead me to consider this a weak reject.\n\nThe authors show that fast weights can be applied in the continual learning setting, but alone they do not perform that well on the more challenging datasets, with mixed results on how much better they are as compared to a naive fine-tuning baseline, and they definitely lag behind synaptic consolidation methods. The authors' method combined with synaptic consolidation methods perform the best, but not much beyond the effect of the synaptic consolidation methods themselves. The authors are encouraged to evaluate their method with a large amount of classes, e.g. as done by iCaRL with CIFAR-100 and ILSVRC, with class-incremental training, to show if their method (a) scales (which I would anticipate, given the class-conditional Hebbian update) and (b) can deal with an alternative continual learning setting; a further resource is the work done around CORe50, which I would consider encapsulates more current thinking and practices around continual learning.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1gf5z-pKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce DIFFERENTIABLE HEBBIAN CONSOLIDATION,a new framework for continual learning that can be implemented in the usual differentiable programming setups. This framework is motivated in terms of complementary learning system (CLS) theory which features an episodic memory module. The method is shown to be easily implemented as seen in their pytorch pseudocode (authors also suggest code will be released). Additionally, authors show the method leads to significant  improvements over simple baselines, and can complement other task-specific hebbian-based learning paradigms\n\n\nAs an alien to the continual learning literature, my judgment is mostly positive. First, the paper is well-written, and the method is well-motivated in terms of cognitive principles with a nice commentary about underlying neurobiological substrates. The description of alternative methods seems exhaustive (but see below). The resulting algorithm is simple and doesn't seem to entail a significant computational burden. Importantly, it is differentially programing-friendly. Results show improvements over a baseline. However, I have two significant criticisms regarding the experimental section that I hope the authors will address.\n\n1)Authors comment on two paradigms to hebbian-based continual learning: the task-specific and theirs, based on CLS. Authors show their method complement improvements due to task-specific approaches (which is great), but proper comparisons to other CLS-based approaches seem missing. In other words, I would hope a much better statement than \"\nHowever, all of these methods were focused on rapid learning on simple tasks or metalearning over a distribution of tasks or datasets. Furthermore, they did not examine learning a large\nnumber of new tasks while, alleviating catastrophic forgetting in continual learning\" Perhaps the authors may try find experimental cases where comparison to such approaches are possible, and show the empirical results? I am concerned the finetuned MLP baseline could be a bit weak.\n2)The authors should try more experiments. They showed many MNIST variants and a 5-vision dataset. However, results in other papers in the literature are shown in much more sophisticated contexts (e.g. learning to play games). I understand this could exceed the computational capabilities of authors, but  why, for example, not to try on a different dataset, as CIFAR? (as the Zenke et al paper did).\n\nMInor: On section 3, page 4 authors mention the number of labels d but that number appears undefined", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeRgzjkcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problems of continual learning and catastrophic forgetting in neural networks. It uses methods inspired by Hebbia learning and complementary learning system (CLS), where there are slow and fast weights in the model. \n\nThe paper addresses an important topic. I find the idea of fast/slow weights to be a refreshing and different approach compared to previous work on catastrophic forgetting. \n\nI had trouble following the details of the DHS Softmax, however. This may stem from my inexperience with Hebbian Learning, but here are some questions/suggestions:\n\n- I am a bit confused exactly what is referred to as post/pre-synaptic connection, what is penultimate layer, etc. A figure might be helpful. \n\n- It also might be helpful to write out an equation for the standard Softmax so it can be compared to Eq 2. \n\n- Related to above, I am confused what is indexed by i,j in Eq 4. Compared to Eq 1, where theta only has one index (k), in Eq 4 theta has two indices (i,j). \n\n- In terms of motivation, can you explain why this Hebbian strategy is applied only to the final softmax? \n\nOther questions:\n\n- Does it seem like DHS Softmax is not as strong by itself but works best in conjunction with others, such as EWC? I do not quite follow how they complement each other intuitively. \n\n- Are there any hyperparameters required for DHS Softmax? It seems to be no? \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that this paper tackles an important problem, continual learning, with a method that is well motivated and interesting. The rebuttal was very helpful in terms of relating to other work. However, the empirical evaluation, while good, could be improved. In particular, it is not clear based on the evaluation to what extent more interesting continual learning problems can be tackled. We encourage the authors to continue pursuing this work.
Decision: Reject