Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgetting
Authors: Alex Ororbia
Ankur Mali
C. Lee Giles
Daniel Kifer
Venue: NeurIPS 2022 Accept (2022)
Keywords: Continual Learning
Lifelong Learning
Deep Learning
Credit Assignment
Sparsity
Brain-inspired Learning
Predictive Coding

Abstract: In lifelong learning systems based on artificial neural networks, one of the biggest obstacles is the inability to retain old knowledge as new information is encountered. This phenomenon is known as catastrophic forgetting. In this paper, we propose a new kind of connectionist architecture, the Sequential Neural Coding Network, that is robust to forgetting when learning from streams of data points and, unlike networks of today, does not learn via the popular back-propagation of errors. Grounded in the neurocognitive theory of predictive coding, our model adapts its synapses in a biologically-plausible fashion while another neural system learns to direct and control this cortex-like structure, mimicking some of the task-executive control functionality of the basal ganglia. In our experiments, we demonstrate that our self-organizing system experiences significantly less forgetting compared to standard neural models, outperforming a swath of previously proposed methods, including rehearsal/data buffer-based methods, on both standard (SplitMNIST, Split Fashion MNIST, etc.) and custom benchmarks even though it is trained in a stream-like fashion. Our work offers evidence that emulating mechanisms in real neuronal systems, e.g., local learning, lateral competition, can yield new directions and possibilities for tackling the grand challenge of lifelong machine learning.

Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgetting
Alexander G. Ororbia Rochester Institute of Technology Rochester, NY 14623, USA ago@cs.rite rdt.edu
Ankur Mali University of South Florida Tampa, FL 33620, USA
Makurarjunmal @us edu
C. Lee Giles The Pennsylvania State University State College, PA 16801, USA clg208psu. edu
Daniel Kifer
The Pennsylvania State University State College, PA 16801, USA duk170psu. edu
Abstract
In lifelong learning systems based on artificial neural networks, one of the biggest obstacles is the inability to retain old knowledge as new information is encountered. This phenomenon known as catastrophic forgetting. In this paper, we propose a new kind of connectionist architecture, the Sequential Neural Coding Network, that is robust to forgetting when learning from streams of data points and, unlike networks of today, does not learn via the popular back-propagation of errors. Grounded in the neurocognitive theory predictive coding, our model adapts its synapses in pellloically-plassbbee fashion while another neural system learns direct and control this cortex-like structure, mimicking some of the task-executive control functionality of the basal ganglia. In experiments, we demonstrate that our self-organizing system experiences significantly less forgetting compared standard neural models, outperforming swath of previously proposed methods, including rehearsal/data buffer-based methods, on both standard (SplitMNIST, Split Fashion MNIST, etc.) and custom benchmarks even though it is trained in stream-like fashion. Our work offers evidence that emulating mechanisms in real neuronal systems, e.g., local learning, lateral competition, can yield new directions and possibilities for tackling the grand challenge of lifelong machine learning.
Introduction
Lifelong learning is a part of machine learning and artificial intelligence research with the goal of developing computational agent that can learn time and continually accumulate new knowledge while retaining its previously learned experiences [ 2]. For example, suppose an agent needs to learn to classify digits, then types of clothing, and then sketches of objects. As each task arrives the agent is expected to process the accompanying data and learn the new task but still remember how complete the old tasks, least without significant degradation in performance generalization. Modern-day connectionist systems are typically trained on a fixed pool of data samples, collected controlled environments, in isolation and random order, and then evaluated separate validation data pool. This is a far cry from what we really desire from learning machines.
loss of
When we look to humans other animals, we that they are more than capable of learning in continual manner, making decisions based on sensorimotor input throughout their lifespans [3] This ability incrementally acquire and refine knowledge over long periods of time is driven by cognitive processes that come together to create the experience-drive specialization of motor and perceptual
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

skills 31
Thus, evaluating how neural systems generalize on task sequences, opposed to single, isolated tasks, proves to be a far greater challenge. In order to continually adapt, the brain must retain specific memories of prior tasks. In working towards the challenge of lifelong machine learning. this paper makes the following contributions: 1) we propose the sequential neural coding network, an interactive generative model that jointly reconstructs input and predicts its label, and an algorithm for updating its synapses, 2) we show that memory retention is vastly improved by integrating our model's multi-step nature with lateral competition that is driven by task selection function inspired by the basal ganglia [3], and 3) we compare our model's s performance against state-of-the-art baselines, including both regularization and rehearsal/replay -aased methods, on publicly-availaae benchmarks.
2 Related Work
It is well-known that when artificial neural networks (ANNs) are trained on more than one task sequentially, the new information contained in subsequent tasks leads to catastrophic interference (a.k.a. catastrophic forgetting) with the information acquired in earlier tasks [6 7 g8. This happens in connectionist systems when the new data instances to be learned are significantly different from previously observed ones. This causes the new information to overwrite knowledge currently encoded the system's synaptic weights, due to the sharing of neural representations over tasks [ (this is known as the representational overlap problem). In isolated, task-specific (offline) learning (though it still occurs [0], this type of weight overwriting occurs to a lesser degree because the patterns are presented to the agent pseudo-randomly and multiple times, i.e., via multiple epochs.
Over the decades, there have been many approaches proposed mitigate catastrophic forgetting in neural systems. Some of the earliest attempts proposed memory systems where prior data points were stored and regularly replayed to the network in a process called "rehearsal" which involved
interleaving these data points with samples drawn from new datasets [1] 12 13
14 15).
Though
effective, the main drawback of these approaches that they require explicitly storing old data. Such a mechanism is not known to exist in the brain and, as a practical matter, this leads to exploding working (hardware) memory requirements (inefficiency). In addition, rehearsal- any mechanisms to preserve consolidated knowledge in the face of acquiring new information [4] Other approaches attempt to allocate additional neural "resources", i.e., growing the networks when required 17). motivated by earlier findings (18). However, this leads to dramatically increasing computational requirements over time as the networks grow larger. To compound these issues further, systems with growing capacity cannot know how many resources to allocate at given time since the number of tasks and samples are not known a priori (without imposing strong assumptions on the input distribution). Other approaches try to block old information from being overwritten through regularization [19]. From this vast collection of research, each approach bearing strengths and weaknesses, three suggested remedies have emerged: 1) allocate additional neural resources to accommodate new knowledge, 2) use non-overlapping representations (or semi-dise cini-diitributed ones (20) if resources are fixed, and 3) interleave old patterns with new ones as new information is acquired.
approaches do not offer
In this work, we consider the setting where the space available to the agent grows slowly compared to the rate of new tasks being presented. This means that we cannot just create new, separate network for every task observed in the stream. Furthermore, storing, reshuffling, and re-presenting data in the stream is not feasible in this setting. Concretely, our approach could be classified class incremental learning (Class-IL.) (21] given that we do not use task identifiers at both training and test time. In addition to addressing the above, contribution to lifelong learning is motivated by the premise that developing algorithms [3] 24 25 26] that serve as alternatives to back- propppropngttion will lead to the creation of promising architectures with mechanisms equipped tackle catastrophic interference.
3 Cumulative Learning with Neural Coding
We start by defining key notation.0 indicates a Hadamard product while denotes matrix/vector multiplication. (v)² is the transpose of v. Matrices/vectors are depicted in c.g., matrix M or vector (scalars shown in italic). Finally, |v]lz denotes the Euclidean norm of
Notation:
bold-font,

---

3.1 Sequential Learning and the Data Continuum
This work focuses on adapting a neural system in the context of sequential learning. Starting from an early definition [27] of this form of learning, we assume that there is sequence of tasks 71, T2. Ta.... (with potentially no end) with each task presented to a system in order. When faced with the (N + +1)th task, the system should use the knowledge that it has gained from the previous N tasks to aid in learning and performing the current task. The knowledge of system is stored in a knowledge base (KB), e.g., the synapses of neural model. Each task Ti has its own corresponding dataset D; = X1,1,)1,44) ... (yn.: Xny, {a)} with n, examples. Here Xj E R/ax represents the feature vector of the jth example (Jg is its dimensionality). y, E {0, 1 is the target (label), and t; is the task descriptor that identifies (y,, xj) as being a data point from task i. The task descriptor is one-hot encoded as t E {0, +](N+1)x3 and, when a new task is encountered, the size of the one-hot vector is increased by one thus the network adds an extra randomly initialized input node if it accepts 4; as an extra input. Note that while we present the task data continuum with I, explicitly depicted, we will generalize our models to not depend on t, (making this problem task descriptor-fres free).
Dynamic Output Units:
The output nodes in our setting get re-used for each new task. For example, output node 1 of the network could represent prediction for the digit "1" in task Ti (e.g., digit recognition), while in task T2 (e.g.. clothing recognition) the same node could represent a prediction for "pants". If a new task has more classes than previous tasks, we add output nodes with randomly initialized weights. For example, if prior tasks were binary and the new task has 4 targets, we add more outputs to the network. Note this is a difficult form of cumulative learning (28)
Context Units:
For the model that we develop in this study, when task is encountered, for each layer é in the network, we make use of a task embedding vector gi (this new memory is much smaller than creating a new network for task t, which would require new weight matrices per layer rather than an extra vector). All gi E Rdex1 (Je is the number of units in layer E) are stored in memory matrices M E R(N+I)xJr (M- = {M' .M.M)),,,, where a context can be retrieved using a one-hot encoding of the task descriptor, i.e., M t ( t will be produced by another system see Section 3.3)
3.2 The Interactive Generative Model
The sequential neural coding network (S-NCN) is designed to make flexible conditional predictions e.g.. predicting y given x, predicting both y and x, predicting missing parts of x given y and the observed parts of x, etc. In order to do so, treats inputs/outputs in a non-standard way. The input to the model is the set of task contexts (g.!...,,"? and the output units represent x). To predict y: given x;, we clamp the output nodes responsible for predicting x, forcing their output to be x; [T] During training, outputs are clamped to both y; and x,, forcing the S-NCN to update latent states and synapses. The S-NCN can also operate as a probabilistic generative network by feeding in a random noise vector as input, but we leave this extension to future work (and focus on predicting y given x).
The full computational process of the S-NCN is defined by three key steps: 1) layer-wise hypothesis generation, 2) state error-correction, and 3) synaptic weight evolution. The first two steps iteratively predict and correct the representations of the model for the input and target values of the task. After K iterations, model weights and the current task context memory are updated. In this section, provide details of the above steps and then describe the objective function that our model optimizes.
3.2.1 Inference: Predicting and Correcting States
The architecture of the S-NCN can be viewed as a stack of parallel, stateful neural-based prediction layers Pi,..., Pr, Pr,..., P... where the goal of each predictor is guess the internal state of the predictor in the layer below (i.e., the S-NCN is not a feedforward network). The state of Pt (layer E) is represented by the (zero-initialized) vector zf RJxx1 (Je is the number of units in layer f). Pt makes prediction z about the current state zt of P:-1 (layer & - 1). Furthermore, we let zb and z.) denote clamped outputs. That is, if we want to predict the label y; given the features x,, we set zQ. x; and if we want train, we set both z x, and 2 y,, The
Layer-wise State Prediction.
Siiilarly, in the case of missing data, we can ask the network to predict the y, and the missing parts of x; given the observed parts of x; by clamping outputs only the observed parts of xi. See appendix for details.
3

---

values predicted by layer 1 are denoted by zn.a and z z...- Note that z" = [2]. 2,11. where the two are concatenated (if either is missing, the SNCN completes the required values see Appendix).
SNC
With respect to neural structure, the parallel predictors of the S-NCN are locally connected through (forward) generative weights W Re-xxee+1 and error (feedback) weights E Re which work to transmit error information to relevant regions of neural processing elements, effectively coordinating all of the S-NCN's predictors. Formally, a predictor (which embodies the computation of deep excitatory pyramidal neurons), with state z z that guesses the state of zz, takes on the
following form (given matrix W and activation +b+1);
z = W o (z)) e' (z' zp
(1)
where ef is a block of error units. Error units are paired with each predictor. Their task is to compute the disagreement or mismatch between the predictor's output and the target activity pattern z (embodying the mismatch computation carried out by superficial pyramidal neurons). The error unit vector ex can also be derived from the total discrepancy reduction presented in Equation S (see Appendix). Note that for layer 0, e" |e_. eg] (there are error neurons for x, as well as others for ys, the concatenation of which makes up the bottom-most prediction error signal).
Latent State Correction and Context Updating. Once each layer l has made prediction about the layer below it and error units have been activated, the state of layer l can be adjusted to take into account the local top-down and bottom-up error information. Using its current state and the error nodes (along with its task context embedding g(). layer É in the S-NCN adjusts its state z follows:
z (k) (z (k- - 1) + sd g!). where df = -e + E e
(2)
where B is the state correction rate and k marks one step of the S-NCN's s K K-step inference process. Note that d' is the perturbation that adjusts the values of the hidden states it combines a top-down expectation of layer l with bottom-up pressure from layer c e 1. The error feedback weights E are parameters that play a crucial role in this calculation, as they are responsible for transmitting the error at é back up to layer + 1. Notably, part of the state-correction requires competition among the individual units in a given layer through the function f'(z', gf) There are various ways in which this function can be implemented and, in this study, implemented it as follows:
fz(z", gf) = OIO V²) z where, V BKWTA K) BKWTA((g))) .K)
where BK BKWTA(Y, K) is the binarized K winners-take-al function, yielding a binary vector with 1 at the index of each of the K winning units, or formally:
BKWTA(v, K)- = {I E {K largest elements of v} and @(v) = 0 otherwise
In the Appendix, we study other forms of the competition function (we found that the above performed best, so we report this in the main paper). Note that this lateral inhibition is function of evolving context g triggered by task pointer L; (produced by task selector model, which we define later).
In real neural systems, intra-layer competition between units is thought facilitate contextual processing (29), where only some neuronal signals are strengthened while the activity of others is suppressed. Moreover, lateral competition, often classically modeled with anti- neti-Hebbian learning [30), encourages the formation of sparse codes [3] 32). Since the S-NCN is an interactive model, incorporating lateral competition is natural and computed online, highlighting its flexibility compared to ANNs Crucially, this lateral activity is an inductive bias that leads the S-NCN to acquire task- dependent representations that encode information for multiple, disjoint tasks. sense, this task specialization that we build into the neural dynamics is similar in spirit to activation sharpening [33
3.2.2 Updating Synaptic Parameters
Given that the S-NCN is an interactive network 34], inferring its states requires running Equations ] and K times, where the model alternates between making predictions and then correcting states once eтror units have been computed. Once latent states have been inferred, the S-NCN then adjusts its synaptic values. The synaptic updates take the form of Hebbian- like rules:
A e = e (00 (z +
Sty and, AE a (d (ef)r) о St.
(3)
RRecurentt weights could model lateral activity in ANNs but this would require using backprop through time.

---

Algorithm/ State inference procedure.
Algorithm 2 Weight update computation.
1: Input: sample (y. x, t), 8, M, & e
M, 1: Input: A, A, M, & @
2: function INFERSTATES x, t),
2:
function UPDATEWEIGHTSAA, @)
(g'..... 8: ...., 8²)
gelConteatts(t.MM)
3:
// Calenlate weight displacements
Set z z 0
4:
1 to do
et 0. 2 x, x =
5:
AW (ef-1
(0 (z"))) S
6: for k: == 1 to K do
6:
AE = (
7:
for { = to
do
b Run layerwise predictors
7:
AW: 2 = (el (6+ (z)))) S
if > then
b Latent prediction layer
9:
z w (z')
8:
AE = (a" (el))) OS B.
10:
e-
(x--1 2
9:
AW; , (e) (ot (z (²) 1 S
else
b Sensory prediction layer
10:
AE, = =n(d" (ef))) OSR.
12:
<-1
W (xxx)
11:
13:
(z)-1
12:
// Update current weights
14:
W (x
13:
14:
for f I. to do
W W + AAW!
15:
ey
8-1
(xy
15:
E' E' + AAE
16:
for
= L to do
b Correct internal states 16:
WT W. + !
17:
if f then
17:
E E. AAE!
18:
d' ef
+ E
F
e
18:
W W + XAW!
19:
else
19:
E E + AAE
20:
x' d o + .e--1
20:
Update (g',". gt M. via Equ.
21:
(8 (xf) sad, 8²)
21:
Return new weights
22:
é (W W
W W4
22:
A = (z) of ".,,""
zt, d' d'..... d' d'.,
23:
23:
E E
E4
24: Return A
24:
Return θ
where a is a scaling factor, usually set to < 1.0, that makes the error feedback weights change at slower rate than the forward weights (this improves convergence (35)). S E R and S E R. XJ are modulation factors that provide stability to the weight updates (see Appendix).
An important property of the above weight update rules is that they are local - to compute changes in the synapses, all we require is the information immediately available to the neuron(s) of interest (making these rules function similarly to classical Hebbian updates [6). although there are important differences to them, as discussed in (35). Since a neuron is able to immediately generate hypothesis given its own internal state, without requiring the active generation of other predictors, and its error can be readily computed after prediction by comparing to the current state of the target neuron state, the weight updates of any predictor layer may be computed in parallel to others. This would allow us to allocate dedicated computing cores to particular predictors, or "pieces", of the S-NCN. Observe that the S-NCN does not require activation derivatives in any of its computations (this is neurobiologically more realistic and favorable for specialized hardware implementations).
Furthermore, during the learning process, each context vector gi and its corresponding memory matrix is adjusted according to the following simple contrastive rule:
gi+1 = gi dd (g
Σ (g))), and, AM -gL- (t2)T 5=1
(4)
where 7) modulates long-term memory update using the current perturbation to be applied to layer "/ controls the repulsion term, which "pushes" context codes away from each other (for diversity). These adapted codes, which influence inter-neuronal competition in task-sensitive manner, could be viewed as a simplification of distributed temporal context (37) where contiguity, i.c., recall/generation of one item is influenced by the presence of another, is introduced into S-NCN distributed processing.
The pseudocode illustrating how the elements described so far are combined in an S-NCN system is presented in Algorithms I aadd 2 The transmission of bottom-ur up and top-down errors in the S-NCN
4 and classical work on interactive networks [3 45 46), where models undergo settling process to process input stimuli more than once (see Appendix for intuition). Though this requires extra computation, the process endows the network with desirable properties, e.g., the ability to conduct constraint satisfaction 47 48). By using multi-step, competiivv processing scheme, the S-NCN is able to "select" subnetworks (portions of neurons) for specific tasks, reducing representational overlap and,
motivated by the theory of predictive processing [38 39 40
5

---

Context T(x)
Context T(x)
Generative
Generative Model
11
Podiation
[Ks, v1)
(6) y1
=|xy]
(-7())
Task Stream/ / Data Continuum
Figure I: (Left) The sequential neural coding network is shown processing an input (y: x) over one step, given a task context (produced by the task selection model T(x)), overriding the current z after error-correction and application of lateral inhibition matrices {V), V2j. Inside the red box is shown one possible emergent subnetwork. Green diamonds indicate error units. (Right) The full complementary neural system shown processing patterns from data continuum.
ultimately, forgetting. This selection is driven by the task pointer t. produced by the task selector, T(x), the final piece of the S-NCN system, which we describe in Section 3.3
3.2.3 Objective Function
During training, when presented with stimulus (y,, t.), the S-NCN adjusts its internal states and synapses so that the output z" of layer 1 (Pi) is as close as possible to (y,, ). It does this by minimizing total discrepancy (ToD) [49 measure of its total internal disorder, is the of all mismatches between layerwise guesses and actual states. Formally, the ToD for an S-NCN is:
(()) Σ C'((") where L (²(
1 (112² 2f.11223
(5)
2
! contains all of the synaptic parameters, i.e., θ (W
W4, E+} and O = (W",
The above loss decomposes the problem of credit assignment in the S-NCN into sub sub-problems with each sub- uub-problem focusing on the comparison between the prediction z made by layer E + and the actual state value z' of layer f. The resulting updates each state z as well as relevant synapses, will then depend on a bottom-up transmitted error signal and the top-down influence of the mismatch with the expectation of the predictor immediately above 44 35), While have motivated aspects of our model from a neuro-cognitive perspective, the error units and weight updates can be derived from the total discrepancy function above (35) (and cast approximately minimizing free energy (50)
3.3 The Neural Task Selection Model
Earlier, we described the S-NCN as taking in a task signal t that drives the system's 's task context memory g for layer (i.e., the routine gelConterlsst, M) in Algorithm 1]. To create this task pointer, we design second neural circuit that we call the task selector T T(x) this will allow the S-NCN to automatically decide when new task has been encountered and to determine, at test time, what task observed data points belong to. The motivation behind T comes from the neuroscientific idea that the basal ganglia plays the role of information routing (among other roles), which serves as form of task selection [51] (2)). In other words, it selects/enables various cognitive programs stored in other cortical regions (5). As a result, we develop a type of complementary learning system (CLS) interaction between the hippocampus and neocortex), which pairs our cortex-like like model with a basal ganglia-like model. We refer to our task selector T(x) the "functional neural basal ganglia" (FNBG) in order to emphasize that the actual basal ganglia in the human brain is more complex and does more than what our computational model does. Our FNBG model, built with competitive learning. has two roles: 1) task shift detection deciding whether data from an input stream indicates the presence of new task, and 2) task recognition deciding whether incoming data requires switching to existent task context or creating a new one.
(different than that of (53), which models
Task Shift Detection: In order to detect the occurrence of a new task while processing data from the pattern stream, Txx) utilizes the output error neurons of the generative model described earlier to detect spikes in their values which might indicate distributional drift. Specifically, T(x) maintains an

---

exponentially weighted running mean иe and variance of of the norm of the cortex model's label error neurons ey. The necessary statistics are calculated as follows:
A ||o] - pellt -
(6)
Mc(() pc(t - 1) + +
(7)
(1) = (1 - 1) -1) 71))²
(8)
with 7 = 0.1 (determined by preliminary experiments). Using the above dynamic statistics, a shift is detected by determining if the following inequality evaluates to true (repeatedly for series of
5 consecutive batches): (tc(t)= Pc(t 1) 2 2//2((1 1). Upon detection of boundary, suppress the check until 1000 samples have been seen after the last detected shift (creating a refractory
period to allow the competitive learning model, described next, to acquire enough data to learn).
Task Recognition through Competitive Learning: To conduct task recognition, T(x) first randomly projects the imput x to a low-dimensional space ("key") k R x (R is initialized from a Gaussian distribution). We then update a rolling average estimate of the streaming imput using this generated key follows: k, (1 r)k,, + 7 (with the FNBG maps this rolling representation k, to a decision as to which task context the S-NCN generative cortex is to utilize, i.e., i = BK AKWWAAQ (k,/ (///1lk)]l2.. K - The task pointer is then used to retrieve contexts {g) g/) where g' = M f (implementing getContezts(t M,)).
0.65). Finally, with the matrix Q.
To update the FNBG weights, while also avoiding catastrophic interference in T(x) itself, we propose learning rule based on competitive learning. Specifically, we develop what we call "guided competitive learning,, since during the act of processing stream of certain samples from the task that we know that we are operating on, we also know which neuron out of a set of T 1 task output neurons should be selected. This leads to the following update rule formally defined as:
a chologically-inspiee
AQ = -pt)) (k,//1kulll)
where p is the competitive weight adjustment factor (a value we found works well in the range of [0.5, 1)). Note that = f recovers an unsupervised classical competitive Hebbian learning. However, we force the model a specific task pointer value by using t, the one-hot encoding of the dynamic integer variable t maintained by the FNBG itself, initialized as t = O. Every time a task shift is this dynamic variable is incremented by one, i.e., 1 < t+ 1.
detected according to Equations 6 8
For both task recognition and the FNBG's synaptic update, note that the rolling representation k, is normalized by its Euclidean norm so that may utilize a vectorizable form of competitive learning based on dot products (taking advantage of GPU-based matrix multiplication). In essence, we take the (normalized) rolling average representation of the input stream for a given task, compute its dot-product with all surrently-availabe
task weight vectors, and choose the dot product with maximal value as the winner. Finally, we re-normalize the matrix by its column Euclidean norms after each of its updates, i.e., Q = (Q): i] + AQI: ()l: a)//IQQQ | al]la where Q[: i] indicates the extraction of all values in column from Q (this normalization similar to that of adaptive resonance theory (54)
3.4 Putting It All Together: A Complementary System
At a high level, given the above, the full S-NCN complementary system, depicted in Figure (Right), consists of: 1) a task selection model (inspired by the information routing/cxecutive control behavior of the basal ganglia (5)) which creates the task contexts that laterally inhibit/gate the activities of the generative S-NCN, and 2) a generative model that learns to predict inputs/labels given task context. In essence, the FNBG Txx) takes in x, to produce i (a one-hot representation of task pointer (;) which is then fed into the generative S-NCN (along with x and y) to compute predictions.
4 Experiments
Simulation Setup: In our experiments, we train models with three hidden layers, whether they be multilayer perceptrons (MLPs) or S-NCNs and compare against baselines from the literature. All models were restricted to contain (a maximum of) 500 units per layer. For the S-NCN, weights were initialized from Gaussian distribution scaled by each layer's 's fan-in and were optimized using
3 Pleass the Appendix where we summarize symbols, notation, and abbreviation used in this work.
7

---

stochastic gradient descent with learning rate of À 0.01. Baseline models were trained on each task for 40 epochs while the S-NCN only made a single pass. The output layer for each MLP was a maximum entropy classifier and the objective was to minimize Categorical cross entropy in the S-NCN, this was encoded in its label error neurons e.. (See Appendix for experimental details, computing infrastructure,
details, and code details.)
Evaluation Metries: To measure model generalization over the sequence of tasks, we make use of the resulting task matrix R (as in ), an N x N matrix of task accuracy scores (normalized to [0, 1)), where in this study N = '. We measure average accuracy (ACC) (mean performance across tasks) and backward transfer (BWT). BWT measures the influence that learning a task T, has on the performance of task Ts- < Tg. A positive BWT indicates that a learning task Tj increases performance on preceding task Tg- As such, higher BWT is better and a strongly negative BWT means there is stronger (more catastrophic) forgetting. Mean and standard deviation (10 trials) are reported for ACC and mean (10 trials) for BWT (see Appendix for its standard deviation). The formulas for ACC, BWT, and new set of metrics we created to analyze memory retention, are provided in the Appendix.
4.1 The Multi-Dataset Task Stream
To start, we tested the S-NCN model on a complex task sequence composed of several learning benchmarks [ We create task sequences by breaking apart MNIST (M), Fashion MNIST (FM), and Google Draw (GD) each into two "sub-tasks" (e.g., for MNIST, MI and M2). or portions of data with a particular subset of the original dataset's classes. See the Appendix for details on tab-tasks/task sequence creation. In Table 1 we present two task orderings (Ordering #1 is "High Color Sim." and Ordering #2 "Low Color Sim.") each under two conditions: sub-tasks that have 1) an equal number of classes (5 each), and 2) an unequal number of classes (number of classes was chosen randomly, omitting 5 as an option). The number of classes was sampled once and held constant for all trials.
We evaluate our proposed S-NCN system (as well as four variations of it in the Appendix)- - hyperpa- rameters were 3 = 0.05, K 10, 1/g 0.9, 7e = 0.01, trained only with backprop (Backprop). Elastic Weight Consolidation (EWC) 19), the Mode-IMM method [56]. the Md-IMM method combined with either Drop Transfer (DT+Md-IN) (MM)) or both DT-Mode-IMMM [56). and the competitive model, hard attention to task (HAT) [57] For each baseline, we tuned hyper-parameters based on their accuracy on each task's 's development set. See Appendix for additional baseline results.
0.98). The baselines include an MLP
L2- 22-Transfer and Drop DropTransfer
Discussion: Results are reported in Table T (see Appendix for more results). Each simulation was run 10 times (each trial used unique seed).As observed in our results, outperforms all baselines consistently, in terms of ACC and BWT, exhibiting improved memory retention over modern, backprop-centric baselines (and, in the Appendix, our expanded results show that the FNBG-driven lateral inhibition is key to improving memory retention the most). This result robust across both task sequences and equal/unequal class settings. S-NCNs were only tweaked minorly, with the same values used in all scenarios. The observation that lateral inhibition improves the neural computation (in our model) also corroborates the result of [48
that the S-NCN
parammeerrs for the
4.2 Continual Learning Benchmarks
To connect our model current learning results, we experimented with wide swath of approaches on three benchmarks Split MNIST, Split NotMNIST, and Split Fashion MNIST (FMNIST). Further- more, we compare multi-head (below dashed line in Table 2 and single-head models (above dashed line). We compare the S-NCN to both replay/rehearsal and non-replay methods: naïve rehearsal with memory (NR+M), EWC, synaptic intelligence (SI) (58), MAS [59]. Lwf [60], GEM [61]. ICarl (62) Lucir 633, and Mnemonics (64] (additional basclines can be found in the Appendix).
Discussion: In Table we report ACC and BWT, averaged over 10 trials, offering a comprehensive comparison of methods and demonstrating that, for all three benchmarks, the proposed S-NCN outperforms all of them in terms of ACC, and nearly all in terms of BWT (and on par with GDumb and GEM, the difference in BWT being negligible) demonstrating the power afforded by designing
S N-NCN is single head model opposed to HAT ,GEM and others that use multiple-head
8

---

Table 1: Generalization metrics (10 trials) for sequence orderings # & #2 (higher values are better).
Ordering #1: {MI, M2, GD1, FM2, GD2] (High Color Sim.)
Equal
Unequal
ACC
BWT
ACC
BWT
Backprop
0.241 0.050
-0.759 0.030
0.185 0.048
-0799 0.048
EWC
0.280 + 0.023
-0.714 0.030
0.185 + 0.046
-0.726 0.039
Md-IMM
0.521 + 0.027
-0.392 0.023
0.480 + 0.039
-0.240
DT+Md-IMM
0.530 0.021
-0.387 0.021
0.551 + 0.042
-0.220
L2+DT+Md-IMM
0.532 0.025
-0.237 0.027
0.520 0.040
-0.240
HAT
0.550 0.019
-0.211 0.020
0.492 + 0.031
-0.231 0.036
S-NCN (ours)
0.716 + 0.013
0.031 + 0.017
0.713 + 0.011
0.041 + 0.012
Ordering #2:
{GD2, M1, FM2, M2, GD1, FM1; (Low Color Sim.)
Backprop
0.303 0.030
-0.644 1: 0.037
0.287 + 0.043
-0.671 0.043
EWC
0.303 0.031
-0.643 0.033
0.291 1: 0.039
-0.663 1: 0.047
Md-IMM
0.584 +: 0.027
-0.091 + 0.030
0.533 1: 0.034
-0.230 :: 0.036 0.036
DT+Md-IMM
0.591 0.020
-0.088 0.032
0.528 1 0.036
-0.211 0.039
L2+DT+Md-IMM
0.630 0.029
-0.076 0.030
0.551 + 0.037
-0.201 0.041
HAT
0.596 0.026
0.0229
0.563 + 0.081
-0,210 0.044
S-NCN (ours)
0.721 + 0.014
-0.042 0.013
0.667 + 0.011
0.097 + 0.013
Table 2: Generalization metrics (10 trials) for Split MNIST, Split Fashion MNIST (FMNIST) and
Not-MNIST benchmarks. Above dashed line are multi-head models below are single-head models.
MNIST
Fashion MNIST
NotMNIST
ACC
BWT
ACC
BWT
ACC
BWT
EWC
0.760 + 0.030
0.210
0.739 = 0.020
-02201
0.790 I 0.020
00.16
VCL
0.980 0,210
-0.033
0.980 0.20
-0.002
0.953 + 0.003
-0.004
IMM
0.951 0.018
0..07
19500
-0.005
0.925 =
-0.006
HAT
0.972 0.011
-0.000
0.968 0.011
-0.004
0.942 0.009
0.005
GEM
0.110
+0.001
0.930 0.12
+0.001
0.919 0.021
0.003
A-GEM
0.900
+0.001
0.955 0.11
+0.001
0.925 0.020
0.002
ER
0.938
0.002
0.945 1 0.10
00.04
0.927 0.016
0.004
EWC
0..00
0.357
0.00
0.350
0.186 1666:-00 0.020
0.36T
NR+M
0.950 +
-0.100
0.948
+ 0.380
0.090
0.880 0.028
0.103
SI
0.197
-03677
0.198 0.100
-03770
0.161 0.030
0.370
MAS
0.195 1 0.290
0.340
0.180 1 0.250
0.340
0.178 0.060
0.341
Lwf
0.846 0.340
0.120
0.875 1 0.300
0.130
0.626 1 0.091
0.130
ICarl
0.940 0.410
0.100
0.960 0.400
0.110
0.887 1 0.102
0.109
Lucir
0.940 0.310
-0.103
0.950 - 0.340
-0.110
0.935 = 0.093
00.101
GDumb
0.978 0.09
0.005
0.973 0.09
-0.006
0.940 0.080
-0.004
Mnem
0.960 0.320
0.091
0.968 + 0.300
+0.007
0.950 0.071
00.011
S-NCN
0.981 + 0.300
0.005
0.982 = 0.400
-0.003
0.957 = 0.400
-0.004
models with stronger grounding in neurobiology. Furthermore, is promising to see that the S-NCN outperforms/matee performance with not only the single-head models but also with multi-head models (except GEM), which enjoy an easier version the problem, i.e., they can utilize a different classifier per task. Finally, note that the S-NCN, due to the FNBG-driven competition, learns to compose task contexts in a data-dependent manner. Desirably, the S-NCN a single-head model, meaning that it does not grow out separate softmax classifier per task (as in multi-headed models, e.g.. HAT, IMM, GEM), which means that it tackles the harder form the forgetting problem, learning representations that preserve knowledge across disjoint tasks. Furthermore, our model is online, whereas models such as IMM or SI require multiple passes per dataset, and does not require validation data in order to expensive neural architecture search (NAS) in 65
4.3 Discussion: On Model Limitations
Although our results with respect to catastrophic forgetting are promising, the S-NCN does have several limitations that are important to discuss. First and foremost, the S-NCN attempts to learn generative model the joint distribution over inputs and labels, i.e.. p(y, x), which has (classically) been shown to be much more difficult problem compared to directly learning a conditional mapping between inputs and labels (66). Second, to learn p(y, x), the S-NCN adapts to data through an expectation maximization process where, given input, it must iteratively infer maximum

---

posteriori (MAP) estimate of its latent neural activities over K step stimulus window. Despite the fortunate fact that, for the benchmarks studied in this work, values for K were fairly low (only 10 to 20 steps were needed per sample/mini-batch, for more complex data types, such as natural images with multiple objects and background scenery, the value of K will quite likely need to be much higher, increasing the S-NCN's 8 computation time when conducting its online inference. We remark that this drawback could be mitigated by integrating mechanisms to support amortized inference, e.g., as in predictive sparse decomposition [67]. and by designing custom software/hardware implementations that exploit the S-NCN's ' layer-wise parallelism (which would also work in asynchronous settings) in both its inference and weight updating cycles. Future work should explore adapting the S-NCN to only learning a conditional mapping as opposed to a full joint distribution as well as develop mechanisms for pre- rre-training the generative side of the system (which would allow freezing of the generative synapses and only require updating discriminative ones - this could reduce the value of K for more complex sensory inputs). While the S-NCN's objective of minimizing ToD is important for breaking free of backprop and its limitations, offering benefits such as resolving weight transport, eschewing the need for derivatives, facilitating local learning without the need for global feedback pathway, inferring richer stateful representations, it also creates a more challenging optimization problem, i.e., the model must match the values imposed by input data as well as ensure that its internal activities and local predictions are aligned. While the full complementary system largely mitigates catastrophic interference, this primarily benefits measurements of BWT and could potentially damage the model's 's per-task performance on more complex datasets, i.e., the main diagonal of task matrix R. Since we do not impose any distributional assumptions over the latent activities (such as a clean Gaussian prior as in variational autoencoders), if the S-NCN's estimated value of the latent activities is far from their true posterior distribution, then the S-NCN might experience sub-optimal performance in complex setups. Even though all continual learning systems suffer from this issue (especially most modern-day ANN-based ones), our model's ss dual optimization nature could experience this problem more frequently. We believe that integrating memory-awan retrieval driven by (a brain-like form of) replay, which would exploit the S-NCN's generative nature, could be a plausible solution moving forward (helping the system come closer to its true posterior and avoiding bad local optima).
4.4 On Negative Societal Impacts
The potential negative societal impact of the proposed S-NCN complementary system is indirect while the model and algorithmic framework we develop is foundational in nature, it could potentially affect the myriad of application//systen the future. As result, best, the same negative consequences that result from using backprop- ANNs are still present when using the S-NCN training process instead. At worst, given that we have presented promising results across several lifelong learning benchmarks, the S-NCN could facilitate the development of better-perte setter-performing robotic agents that might be used in military applications that might result the loss of life, such as drones. Despite the benefits that S-NCN offers to the statistical learning and cognitive neuroscience communities, one should consider the drawbacks of building powerful neural systems drive applications. To safeguard against its potential negative impact, we emphasize that developing an ethical framework to guide the design and training of general intelligent systems will be important for ensuring safe integration into human society.
currently in development today or those to be developed in
Conclusion
In this paper, we proposed the sequential neural coding network (S-NCN), an interactive generative model, and its local learning procedure for lifelong machine learning. As demonstrated on several benchmarks and setups, this model retains the knowledge it acquires from prior tasks when learning new ones in the face of task data streams, primarily when lateral inhibition, driven by a self-organizing task selection model, sharpens neural activities within its layers. As result, this work marks an important step towards developing wrain-inspired agents that are capable of robustly combating catastrophic forgetting, especially ones that do not require task descriptors to guide the learning process, offering promising pathway towards achieving the ultimate goal of designing agents that act and adapt ways more similar to animals and humans.
A Acknowledgements
We would like to thank Alexander Ororbia (Sr.) for useful discussions related to cognitive types, a concept that served as key motivation for the task contexts that drove lateral inhibition in this work.
10

---

References
[1] S. Thrun and T. M. Mitchell, "Lifelong robot learning," Robotics and autonomous systems, vol. 15, no. 1-2, pp. 25-46, 1995.
[2] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, "Continual lifelong learning with neural networks: A review," arXiv preprint arXiv:1802.07556 2018.
[3] J. D. Power and B. L. Schlaggar, "Neural plasticity across the lifespan," " Wiley Interdisciplinary Reviews: Developmental Biology, vol. 6, no. 1, 2017.
[4] F. Zenke and W. Gerstner, "Hebbian plasticity requires compensatory processes on multiple timescales," Phil. Trans. R. Soc. B, vol. 372, no. 1715, p. 20160259, 2017.
[5] G. Leisman, O. Braun-Benjamin, and R. Melillo, "Cognitive-motor interactions of the basal ganglia in development," Frontiers in systems neuroscience, vol. 8, p. 16, 2014.
[6] M. McCloskey and N. J. Cohen, "Catastrophic interference in connectionist networks: The sequential learning problem," The psychology of learning and motivation, vol. 24, no. 109, p. 92, 1989.
[7] R. Ratcliff, "Connectionist models of recognition memory: constraints imposed by learning and forgetting functions." " Psychological review, vol. 97, no. 2, p. 285, 1990.
[8] S. Lewandowsky. "On the relation between catastrophic interference and generalization in connectionist networks," " Journal of Biological Systems, vol. 2, no. 03, pp. 307-333, 1994.
[9] R. M. French, 'Catastrophic forgetting in connectionist networks," Trends in cognitive sciences, vol. 3, no. 4. pp. 128-135, 1999.
[10] M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y. Bengio, and G. J. Gordon, "An empirical study of example forgetting during deep neural network learning," " arXiv preprint 1rXiv: 2018.
[11] A. Robins, "Catastrophic forgetting in neural networks: the role of rehearsal mechanisms," in Artificial Neural Networks and Expert Systems, 1993. Proceedings. First New Zealand International Two-Stream Conference on. IEEE, 1993, pp. 65-68.
[12] R. Anthony, "Catastrophic forgetting, rehearsal and pseudorehearsal." Connection Science, vol. 7, no. 2, pp. 123-146, 1995.
[13] A. Robins, "Consolidation in neural networks and in the sleeping brain," Connection Science, vol. 8, no. 2. pp. 279-276, 1996.
[14] A. Gepperth and C. Karaoguz, "A bio-inspired incremental learning architecture for applied perceptual problems." Cognitive Compulation, vol. 8, no. 5, pp. 924-934, 2016.
[15] S.-A. Rebuffi, A. Kolesnikov, and C. H. Lampert, "icarl: Incremental classifier and representa- tion learning," " in Proc. CVPR, 2017.
[16] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, "Progressive neural networks," arXiv preprint arXiv:1606.044: 04671, 2016.
[17] G. I. Parisi and S. Wermter, "Lifelong learning of action representations with deep neural self- organization," in The AAAI 2017 Spring Symposium on Science of Intelligence: Computational Principles of Natural and Artificial Intelligence, Standford, US, 2017, pp. 608-612.
[18] E. Meier, L. Hertz, and A. Schousboe, "Neurotransmitters as developmental signals," " Neuro- chemistry international, vol. 19, no. 1-2, pp. 1-15, 1991.
[19] Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska- Grabska-Barwinske et al., "Overcoming catastrophic forgetting in neural networks," Proceedings of the National Academy of Sciences, vol. 114, no. 13, pp. 3521-3526, 2017.
[20] R. M. French, "Semi-distributed representations and catastrophic forgetting in connectionist. networks," Connection Science, vol. 4, no. 3-4, pp. 365-377, 1992.
[21] G. M. Van de Ven and A. S. Tolias, "Three scenarios for continual learning," arXiv preprint 1xX1v:1904.07734.0 2019.
11

---

[22] J. R. Movellan, "Contrastive hebbian learning in the continuous hopfield model," in Connec- tionist Models. Elsevier, 1991, pp. 10-17.
[23] B. Scellier and Y. Bengio. "Equilibrium propagation: Bridging the gap between energy- based models and backpropagation." Frontiers in computational neuroscience, vol. 11, 24, 2017.
[24] D.-H. Lee. S. Zhang, A. Fischer, and Y. Bengio. "Difference target propagation." in Joint European Conference on Machine Learning and Knowledge Discovery Databases. Springer, 2015, pp. 498-515.
[25] A. G. Ororbia and A. Mali, "Biologically motivated algorithms for propagating local target representations," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 4651-46555
[26] S. Bartunov, A. Santoro, B. Richards, L. Marris, G. E. Hinton, and T. Lillicrap. "Assessing the scalability of biologically-motivteee deep learning algorithms and architectures," in Advances in Neural Information Processing Systems, 2018, pp. 9390-94000
[27] S. Thrun, "Is learning the n-th thing any easier than learning the first?" in Advances in neural information processing systems, 1996. pp. 640-646.
[28] G. Fei, S. Wang, and B. Liu, "Learning cumulatively become more knowledgeable," in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016, pp. 1565-1574.
[29] H. Adesnik and M. Scanziani, "Lateral competition for cortical space by layer-specific horizontal circuits," Nature, vol. 464, no. 7292. p. 1155, 2010.
[30] P. Földiak, "Forming sparse representations by local anti-hebbian learning," Biological cyber- neties, vol. 64, no. 2. pp. 165-170, 1990.
[31] B. A. Olshausen and D. J. Field, "Sparse coding with overcomplete basis set: A strategy employed by v1?" Vision research, vol. 37, no. 23, pp. 331 133325,,, 1997.
[32] A. D. Szlam, K. Gregor, and Y. L. Cun, "Structured sparse coding via lateral inhibition." in Advances in Neural Information Processing Systems, 2011, pp. 11 1126-1124.
[33] R. M. French, "Using semi-distributte representations overcome catastrophic forgetting in connectionist networks," in Proceedings of the 13th annual cognitive science society conference. Erlbaum, 1991, pp. 173-178.
[34] J. I.. McClelland and D. E. Rumelhart, "An interactive activation model of context effects in letter perception: I. an account of basic findings." Psychological review, vol. 88, no. 5, p. 375, 1981.
[35] A. Ororbia, A. Mali, C. L. Giles, and D. Kifer, "Continual learning of recurrent neural networks by locally aligning distributed representations" IEEE Transactions on Neural Networks and Learning Systems, 2020.
[36] D. O. Hebb et al., "The organization of behavior," 1949.
[37] M. W. Howard and M. Kahana, "A distributed representation of temporal context," " Journal of Mathematical Psychology. vol. 46, no. 3, pp. 269-269-626666666 269-299,22 2002.
[38] R. P. Rao and D. H. Ballard, "Dynamic model of visual recognition predicts neural response properties in the visual cortex," Neural computation, vol. 9, no. 4, pp. 721-763, 1997.
[39] R. R. PN and B. D. H, "Predictive coding in the visual cortex: functional interpretation of some extra- classical receptive-fied field effects." Nature neuroscience, vol. 2, no. 1, 1999.
[40] Y. Huang and R. P. Rao, "Predictive coding," Wiley Interdisciplinary Reviews: Cognitive Science, vol. 2, no. 5, pp. 580-593, 2011.
[41] C. Wacongne, J.-P. Changeux, and S. Dehaene, "A neuronal model of predictive coding ac- counting for the mismatch negativity," Journal of Neuroscience, vol. 32, no. 11, pp. 3665-3678, 2012.
[42] R. Chalasani and J. C. Principe, "Deep predictive coding networks," arXiv preprint arXiv:1301.3541 2013.
[43] A. Clark, Surfing uncertainty: Prediction, action, and the embodied mind. Oxford University Press, 2015.
12

---

[44] E. Santana, M. S. Emigh, P. Zegers, and J. C. Principe, "Exploiting apatio-tempont structure with recurrent winner-take-al networks," IEEE Transactions on Neural Networks and Learning Systems, 2017.
[45] J. L. McClelland, processing," Attention and Performance (Volc. XIV): Synergies in Experimental Psychology, Artificial Intelligence, and Cognitive Neuroscience. 1993.
"The grain model: A framework for modeling the dynamics of information
[46] R. C. O' O'Reilly. "Biologically plausible error-driven learning using local activation differences: The generalized recirculation algorithm," Neural computation, vol. 8, no. 5, pp. 895-938, 1996.
[47] O. R. C. "Six principles for biologically based computational models of cortical cognition." Trends in cognitive sciences, vol. 2, no. 11, pp. 455-462, 1998.
[48] R. C. O'Reilly, "Generalization in interactive networks: The benefits of inhibitory competition and hebbian learning," Neural Computation, vol. 13, no. 6, pp. 1199-1241, 2001.
[49] A. G. Ororbia II, P. Haffner, D. Reitter, and C. L. Giles, "Learning to adapt by minimizing discrepancy," " arXiv preprint arXiv: vrxiv:/711.11 1711. 2017.
[50] K. Friston, "The free-energy principle: rough guide to the brain?" Trends in cognitive sciences, vol. 13, no. 7, pp. 293-301, 2009.
[51] E. Yehene, N. Meiran, and N. Soroker, "Basal ganglia play a unique role in task switching within the fromtal-subcortical circuits: evidence from patients with focal lesions," Journal of Cognitive Neuroscience, vol. 20, no. 6, pp. 1079-1093, 2008.
[52] J. Buschman and E. K. Miller, "Goal-direction and top-down control," Philosophical Trans- actions of the Royal Society B: Biological Sciences, vol. 369, no. 1655, p. 201 20130471, 2014.
[53] J. L. McClelland, B. L.. McNaughton, and R. C. O'reilly, "Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory." Psychological review, vol. 102, no. 3, p. 419, 1995.
[54] Grossberg, "Adaptive resonance theory: How a brain learns to consciously attend, learn, and recognize changing world," Neural networks, vol. 37, pp. 1-47, 2013.
[55] D. Lopez-Pare Pazezz-azeeee et al., "Gradient episodic memory for continual learning," in Advances in Neural Information Processing Systems, 2017, pp. 6470-6479.
[56] S. Lee, J. Kim, J. Ha. and B. Zhang, "Overcoming catastrophic forgetting by incremental moment matching." CoRR, vol. abs/1703.08475, 2017. [Online]. Available:
http://arxiv.org/ab/// 703.08847
[57] Serrà, D. Surís, M. Miron, and A. Karatzoglou, "Overcoming catastrophic forgetting with hard attention to the task," CoRR, vol. abs/1801.01423, 2018. [Online]. Available: http://arxiv.oe hwtpauwiiiiogg/b///////e wtrg/abs/1801.00 1811 881.011433
[58) F. Zenke, B. Poole, and S. Ganguli, "Continual learning through synaptic intelligence," in International Conference on Machine Learning. PMLR, 2017, pp. 3987-3995.
[59] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, "Memory aware synapses: Learning what (not) to forget," in Proceedings of the European Conference on Computer Vision (ECCV), September 2018
[60] 7. Li and D. Hoiem, "Learning without forgetting," IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 12, pp. 2935-2947, 2017.
[61] D. Lopez-Paz and M. Ranzato, "Gradient episodic memory for continual learning," arXiv preprint arXiv: 08840, 2017.
[62] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, "icarl: Incremental classifier and representation learning," in Proceedings of the IEEE conference Computer Vision and Pattern Recognition, 2017, pp. 2001-2010.
[63] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, "Learning unified classifier incrementally via rebalancing." " in Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2019, pp. 831-839.
[64] Y. Liu, Y. Su, A.-A Liu, B. Schiele, and Q. Sun, "Mnemonics training: Multi-class incremental learning without forgetting." " in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12245-12254
13

---

[65] X. Li, Y. Zhou, T. Wu, R. Socher, and C. Xiong, "Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting," arXiv preprint
903.00030, 2019.
[66] A. Y. Ng and M. I. Jordan, "On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes." in Advances in neural information processing systems, 2002, pp. 841-848.
[67] K. Kavukeuoglu, M. Ranzato, and LeCun, "Fast inference in sparse coding algorithms with applications to object recognition," arXiv preprint arXiv:1010.367,, 2010.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Limitations are discussed in the
Appendix/Supleemnntary Material.
(c) Did you discuss any potential negative socictal impacts of your work? [Yes] Potential negative societal impacts of the work are discussed in the conclusion and in the Appendix/Supplementay Material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] The code and the data are proprietary. We will provide complete codebase to support this paper upon acceptance. We do provide detailed pesudocode of model, describing overall flow which can be used to re-create our model.
(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report one standard deviation of uncertainty (10 trials).
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Yes in appendix we provide details about the compute used to run all these experiments.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes, we do cite all the existing assests in our paper as well as in our codebase.
(b) Did you mention the license of the assets? [Yes] We do mention licensing in our
codebase.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] In the appendix, we provide details for creating the special challenge benchmark of this paper.
(d) Did you discuss whether and how consent obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
14

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "lhj6RTho65N", "reviewer": "Reviewer_WKzA", "summary": "The authors propose a biologically inspired/plausible neural network for continual learning. The proposed method leverages a context mechanism to learn multiple tasks while minimising forgetting; and compare favourably against baselines across standard benchmark tasks.", "questions": "1 - It seems to me that the main difference between the proposed approach and \"Learning to Adapt by Minimizing Discrepancy\" is the addition of contexts. Could the authors discuss the similarities and the differences?\n\n2 - Why do all the baselines use same learning rate (\\lambda = 0.01)? This seems quite high for conventional neural networks used in conjunction with conventional preprocessing.\n\n3 - The proposed model has two independent learning components: \"predictive\" weights and contexts. Such models, that aren't trained end-to-end, can be challenging to train due to hyperparameter sensitivity. Can the authors comment on this?", "limitations": "Limitations are adequately addressed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The formalism based on predictive coding is attractive because of its biological plausibility as well as its flexibility. At face value, the model demonstrates very impressive empirical results by outperforming backprop-based neural networks -- which is not an easy feat. However, the experiments section is missing some important analysis in my view. \n\nRegarding the experiments, I understand that no meta-parameter optimisation is done for the baseline methods, which makes the comparison a bit unfair, but it is not the end of the world. However, I do not understand why the original learning rates are altered and set to 0.01 for all baselines (see Question 2). \n\nMany continual learning papers display learning curves across multiple tasks (eg, the EWC paper). This is very helpful for identifying the degree of learning speed and forgetting. In addition, it is helpful to compare the learning curves across papers, which gives some form of reassurance that the implementations or hyperparameter choices are correct/decent.\n\nIn terms of novelty, I would like to learn more about the differences between the proposed approach,  \"Learning to Adapt by Minimizing Discrepancy\", and \"Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations\" (Question 1). In fact, this should ideally be discussed in \"Related Work\".\n\nThe paper is written clearly and is easy to understand overall. It would be helpful to show how the update equations can be derived from Equation 6. Because, it seems to me that just differentiating the loss wrt weights result in backprop without zeroing/stopping some gradients. Also, the line 20 in Algorithm 2 should refer to Equation 4 I believe.\n\nI am open to bumping up my rating if my concerns are adequately addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JNx9ClBAQRE", "reviewer": "Reviewer_36Ec", "summary": "The authors in the paper propose a complementary system trained using local learning rules and bio-inspired mechanisms inspired from basal ganglia, such as lateral inhibition and gating based on context, task selection using context memory. The authors propose a decomposed learning mechanism which eliminates the need for weight transport mechanisms. The methods proposed in the paper combine multiple mechanisms to demonstrate continual learning capabilities on image recognition datasets.", "questions": "1. Based on the algorithm, the initial prediction states are being computed using a top-down approach. So, how is $z^l$ being computed for the initial state, is it a random initialization or is it directly the one hot encoded label? If the z values for the intermediate labels are initialized to 0, how would we compute the $z^l_u$ values for the corresponding layers based on equation 1.\n2. \"Concretely, our approach could be classified as task incremental learning (Domain-IL) [21] with the exception that no task descriptors are used at both training and test time (an aspect of Class-IL [21])\" - This sentence needs to be clarified and rephrased. Task IL is not Domain IL.\n3. Isn’t there an overhead for storing the task-specific context memory. And would that not also increase/grow with the number of tasks? there are other replay-based approaches which also operate with very limited and small buffer sizes or there are other generative based models like brain inspired replay, so the point regarding the proposed model being more memory efficient needs to be discussed in the paper.\n4. The learning rule decomposing the weight update towards local layers without transmitting the error backwards has been already proposed by other surrogate gradient-based mechanisms. So, however using local knowledge only for learning is novel. How would you compare the proposed approach to the other surrogate gradient-based approaches?\n5. The proposed weight update rule uses modulation factors as an optimizer factor for updating the weights. The modulation factors are similar to some of the previously proposed approaches\n\n    * Beaulieu, S., Frati, L., Miconi, T., Lehman, J., Stanley, K. O., Clune, J., & Cheney, N. (2020). Learning to continually learn. arXiv preprint arXiv:2002.09571.\n\n    * Tsuda, B., Tye, K. M., Siegelmann, H. T., & Sejnowski, T. J. (2020). A modeling framework for adaptive lifelong learning with transfer and savings through gating in the prefrontal cortex. Proceedings of the National Academy of Sciences, 117(47), 29872-29882.\n\n    * Imam, N., & Cleland, T. A. (2020). Rapid online learning and robust recall in a neuromorphic olfactory circuit. Nature Machine Intelligence, 2(3), 181-191.\n\n    * Madireddy, S., Yanguas-Gil, A., & Balaprakash, P. (2020). Neuromodulated neural architectures with local error signals for memory-constrained online continual learning. arXiv preprint arXiv:2007.08159.\n\nWhat is the role of the modulation factors? Is it just for inducing weight stability or are these factors necessary for continual learning? An ablation analysis would enable the reader to understand the role of the modulation factors? Generally modulation is considered necessary for continual learning and is helpful for selective attention in the network. Are the modulation factors performing a similar role?\n\n6. The experimental setup in Section 4 is not clear. In Table 2, the accuracies for multi-head models looks inaccurate? Do you mean task incremental learning scenario for multihead or class Incremental learning? Class-IL based models are generally not categorized as multi-headed models? Please clarify the description and check the correctness in the accuracies for compared works. Additionally, newer works which demonstrate superior performance need to be compared. You can find some of the works in\n    * Mai, Z., Li, R., Jeong, J., Quispe, D., Kim, H., & Sanner, S. (2022). Online continual learning in image classification: An empirical survey. Neurocomputing, 469, 28-51.\n", "limitations": "limitations have not been discussed in detail but potential negative societal impacts are briefly touched upon ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The back propagation-alternative approaches are closer to the biological neural networks and provide an avenue for efficient learning\n* The treatment of local knowledge propagation among the hidden layers is novel.\n\n**Weaknesses**\n* The writing/presentation of ideas need to be improved for clarity\n* The experimental setup is not clear/reproducible\n* Comparison with other local learning-based continual learning approaches is missing", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JJ2wqtTUl6j", "reviewer": "Reviewer_qc3j", "summary": "This work uses a generative predictive coding model to perform continuous learning. The goal of such models is ti perform learning and inference on multiple tasks by using the same network, with the hope that (1) training on a new dataset won't make the network forget previously learned information, and (2) that the network will actually use previously learned information to perform better inference on the new task. To this end, the authors propose a generative PC model, conceptually similar to the one proposed by Rao and Ballard, with multiple differences:\n\n1) Feedforward and feedback weights are different;\n\n2) Multiple regularizers are used to improve the performance (modulation factors);\n\n3) To make continual learning possible, learnable task embedding vectors, that depend on specific tasks and are separated using a contrastive loss, are added. \n\nThe method is then tested on multiple benchmarks, with good results. As a disclaimer, I am not an expert on continual learning, hence I'm judging the experiments based on the reported results. After a couple of google searches, it seems to me that the authors have well picked the baselines and datasets, but I may be wrong here. ", "questions": "Above, I have highlighted the three differences between generative PCNs and your proposed method above: Feedback connections, task embedding vectors, and regularisers. Of course, task embedding vectors are essential for continual learning, and are probably the main reason your model works well.  However, why have you decided to implement the other two differences? so, more in detail:\n\n1) How would a model with symmetric connections behave?\n\n2) How important is the regularization in the final performance? \n\nI'm asking this, as I have the feeling that removing one of the two, would harm performance, but I'm having troubles understanding why. Hence, I would like to read a discussion that better tackles this. Even better, I would like to see experiments on one task made by using a network where  (1) and/or (2) are removed, and how the performance compare with the final model. \n\n\nMinor:\n\nEquation 5 is not that clear to me, what is M?\n\nWTA seems to only depend on the value and not on the absolute value, right? Why this choice?\n\nFeedback weights are called \"recurrent weights\". As a reader, I often associate \"recurrent\" with \"lateral connections\", and not feedback ones. Maybe changing the name would make this clear.\n\nEquation 4: shouldn't it be g^l_{t+1} instead of g_l_t.\n\n", "limitations": "Well and largely discussed in the supplementary material.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\nThe idea of the method is nice, and the experimental results confirm the intuition of the authors. It is original, as I'm not aware of any application of generative predictive coding networks (PCNs) to continual learning. The paper reads ok: it is well written, but sometimes the large number of details makes the text flow heavy. To conclude, this work could be significant to two different communities: computational neuroscience, more in detail the branch that tries to tackle machine learning problems this way, and continual learning. \n\nCons: \n\nWhile being a generative PCN, the literature about these models is not discussed properly: although some of the relevant work is actually cited In block in line 197, a proper discussion feels needed. Particularly, a proper comparison with the models that inspired your architecture would raise multiple interesting questions that would be nice to discuss in the paper, that I will state in the next paragraph.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept