Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks
Authors: Mingqing Xiao
Qingyan Meng
Zongpeng Zhang
Di He
Zhouchen Lin
Venue: ICLR 2024 poster (2024)
Keywords: Hebbian learning
neuromorphic computing
continual learning
spiking neural networks
Primary area: general machine learning (i.e., none of the above)

Abstract: Neuromorphic computing with spiking neural networks is promising for energy-efficient artificial intelligence (AI) applications. However, different from humans who continually learn different tasks in a lifetime, neural network models suffer from catastrophic forgetting. How could neuronal operations solve this problem is an important question for AI and neuroscience. Many previous studies draw inspiration from observed neuroscience phenomena and propose episodic replay or synaptic metaplasticity, but they are not guaranteed to explicitly preserve knowledge for neuron populations. Other works focus on machine learning methods with more mathematical grounding, e.g., orthogonal projection on high dimensional spaces, but there is no neural correspondence for neuromorphic computing. In this work, we develop a new method with neuronal operations based on lateral connections and Hebbian learning, which can protect knowledge by projecting activity traces of neurons into an orthogonal subspace so that synaptic weight update will not interfere with old tasks. We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities and enable orthogonal projection. This provides new insights into how neural circuits and Hebbian learning can help continual learning, and also how the concept of orthogonal projection can be realized in neuronal systems. Our method is also flexible to utilize arbitrary training methods based on presynaptic activities/traces. Experiments show that our method consistently solves forgetting for spiking neural networks with nearly zero forgetting under various supervised training methods with different error propagation approaches, and outperforms previous approaches under various settings. Our method can pave a solid path for building continual neuromorphic computing systems. The code is available at https://github.com/pkuxmq/HLOP-SNN.

Published as a conference paper at ICLR 2024
HEBBIAN LEARNING BASED ORTHOGONAL PROJEC- TION FOR CONTINUAL LEARNING OF SPIKING NEU- RAL NETWORKS
Mingqing Xiao*, Qingyan Meng2.., Zongpeng Zhang", Di Hc Zhouchen Lin
1 National Key Lab of General AI, School of Intelligence Science and Technology, Peking University The Chinese University of Hong Kong, Shenzhen
3 "Shenzhen Research Institute Big Data
*Department of Biostatistics, School of Public Health, Peking University Instittte for Artificial Intelligence, Peking University Peng Cheng Laboratory
ABSTRACT
Neuromorphic computing with spiking neural networks is promising for energy- efficient artificial intelligence (Al) applications. However, different from humans who continually learn different tasks in a lifetime, neural network models suffer from catastrophic forgetting. How could neuronal operations solve this problem is important question for AI and neuroscience. Many previous studies draw in- spiration from observed neuroscience phenomena and propose episodic replay or synaptic metaplasticity, but they are not guaranteed to explicitly preserve knowl- edge for neuron populations. Other works focus on machine learning methods with more mathematical grounding, e.g., orthogonal projection on high dimen- sional spaces, but there is no neural correspondence for neuromorphic computing. In this work, develop new method with neuronal operations based lateral connections and Hebbian learning, which can protect knowledge by projecting ac- tivity traces of neurons into an orthogonal subspace so that synaptic weight update will not interfere with old tasks. We show that Hebbian and anti-Hebbian learn- ing on recurrent lateral connections can effectively extract the principal subspace of neural activities and enable orthogonal projection. This provides new insights into how neural circuits and Hebbian learning can help continual learning, and also how the concept of orthogonal projection can be realized in neuronal systems. Our method is also flexible to utilize arbitrary training methods based on presynaptic activities/traces. Experiments show that our method consistently solves forgetting for spiking neural networks with nearly zero forgetting under various supervised training methods with different error propagation approaches, and outperforms previous approaches under various settings. Our method can pave a solid path for building continual neuromorphic computing systems. The code is available https:/ http:/gtttubb.cco wtttoo//pppumq///gmme HLOP- SNN
INTRODUCTION
Brain-inspired neuromorphic computing. c.g., biologically plausible spiking neural networks (SNNs), has attracted tremendous attention (Roy et al. 2019) and achieved promising results with deep architectures and gradient- traseent-based methods recently (Shrestha & Orchard 2018 Zheng et al.) 2021 Fang et al., 2021; Yin et al., [202]] Meng et al.) (022). The neuromorphic computing systems imitate biological neurons with parallel in-memory and event-driven neuronal computation and can be efficiently deployed on neuromorphic hardware for energy-efficies applications (Akopyan al.,
2015 Davies et al. 2018 Pei et al. 2019; Wožniak et al. 2020; Rao et al., (022). SNNs are also believed as one of important approaches toward artificial general intelligence (Pei al. 2019).
However, neural network models typically suffer from catastrophic forgetting of old tasks when learning new ones (French 1999. Goodiellow al. 2013 Kirkpatrick et al. 2017). which sig-
ccoreesponding author

---

Published as a conference paper at ICLR 2024
nificantly different from the biological systems of human brains that can build up knowledge in the whole lifetime. This is because the weights in the network are updated for new tasks and are usually not guaranteed to preserve the knowledge of old tasks during learning. To establish human- like life- long learning in dynamically changing environments, developing neuromorphic computing systems of SNNs with continual learning (CI.) ability is important for artificial intelligence.
To deal with this problem, biological underpinnings are investigated (Kudithipudi et al. 2022), and various machine learning methods are proposed. Most works can be categorized into two classes.
The first category draws inspiration from observed neuroscience phenomena and proposes methods such replay (Rebuffi al. 2017: Shin al. 2017; Rolnick al., 2019 van de Ven et al. 2020) or regularization (Kirkpatrick et al. 2021]. For example, replay methods are inspired by the episodic replay phenomenon during sleep or rest, and regularization is based on the metaplasticity of synapses, i.e., the ease with which a synapse can be modified depends on the history of synaptic modifications and neural activities (Kudithipudi eet al. 22022). However, replay methods only implicitly protect knowledge by retraining and require storing a lot of old input samples for frequent replay, and metaplasticity methods only focus on reg- ularization for each individual synapse, which is not guaranteed to preserve knowledge considering the high dimensional transformation of interactive neuron populations. It is also unclear how other biological rules such as Hebbian learning can systematically support continual learning.
et al. 2017 Aljundi et al. 2018 Laborieux et al.,
The second category focuses on developing pure machine learning methods with complex com- putation, e.g., pursuing better knowledge preservation considering projection high dimensional spaces (He & Jaeger, 2018) Zeng et al.] 2019) Saha et al.) [202]) but they require operations such as calculating the inverse of matrices or singular value decomposition (SVD), which may not be implemented by neuronal operations. Considering neuromorphic computing systems of SNNs, it is important to investigate how neural computation can deal with the problem.
In this work, we develop for task- and inneamnnnncremmntall continual learning of neuromorphic computing systems, which is based on lateral connections as well as Hebbian and anti-Hebbian learning. The core idea is to use lateral neural computation with Hebbian learning to extract principle subspaces of neuronal activ- ities from streaming data and project activity traces for synaptic weight update into an orthogonal subspace so that the learned abilities are not influenced much when learning new knowledge. Such a way of learning is flexible to utilize arbitrary training methods based on presynaptic activity traces.
new method, Hebbian learning based orthogonal projection (HLOP),
Our method can explicitly preserve old knowledge by neural computation without frequent replay, and impose a more systematic constraint for weight update considering neuron populations instead of individual synapses, compared with replay and metaplasticity methods. Meanwhile, our method focuses knowledge protection during the instantaneous learning of new tasks and is compatible with post- post-processing methods such as episodic replay. Compared with previous projection methods, method enables a better, unbiased construction of the projection space based on online learn- ing from streaming large data instead of a small batch of data (Saha et al., 2021), which leads to better performance, and our work is also the first to show how the concept of projection with more mathematical guarantees can be realized with pure neuronal operations.
Experiments on continual learning of spiking neural networks under various settings and training methods with different error propagation approaches consistently show the superior performance of HLOP with nearly zero forgetting, outperforming previous methods. Our results indicate that lateral circuits can play a substantial role in continual learning, providing new insights into how neural circuits and Hebbian learning can support the advanced abilities of neural systems. With purely neuronal operations, HI.OP can also pave paths for continual neuromorphic computing systems.
2 RELATED WORK
Many previous works explore continual learning of neural networks. Most methods focus on arti- ficial neural networks (ANNs) and only few replay or metaplasticity/local learning works (Tadros)
et al., 2022; Soures et al.) 2021 Panda et al.) 2017: Wu et al.) 2022) consider SNNs. Methods networks can be mainly categorized as replay. regularization-be based, and param- eter isolation (De Lange al. 2021). Replay methods will replay stored samples (Rebuffi et al. 2017 Rolnick et al. 2019)) or generated pseudo-samples (Shin et al. 2017: van de Ven et al.| 2020) during after learning new task serve rehearsals restrictions (Lopez-Paz & Ranzato,
for fixed-capacio
2

---

Published as a conference paper at ICLR 2024
2017
Chaudhry al.) 2019). which are inspired by the episodic replay during sleep or rest in the
brain (Kudithipudi et al.) 2022). Regularization based methods are mostly inspired by the meta-
plasticity of synapses in neuroscience, and regularize the update of each synaptic weight based on
the importance estimated by various methods (Kirkpatrick et al., 2017] Zenke et al.) 2017 Aljundi
et al. 2018 Laborieux et al. 2021 [Soures et al. 2021) knowledge distillation (Li & Hoiem
2017
Parameter isolation methods separate parameters for different tasks, c.g., with pruning and
mask (Mallya & Lazebnik
2018) or hard attention (Serra et al. 2018). There are also expansion
methods (Rusu et al. [016 Yoon et al. 2018) to enlarge network capacity for every new task.
Orthogonal projection methods take a further step of regularization to consider the projection of gradient so that the update of weight is constrained to an orthogonal direction considering the high- dimensional linear transformation. Farajtabar et al. (Farajtabar et al., 2020) store previous gradients and project gradients into the direction orthogonal to old ones. |e & Jaeger (018)) Zeng et al. (2019) and [Saha et al. (202T) project gradients into the orthogonal subspace of the input space for more guarantee and better performance. Lin et al. (2022) improve (Sha et al. (202I) with trust region projection considering similarity between tasks. However, these methods only estimate the subspace with small batch of data due to the large costs of SVD, and there is no neural correspondence. We generalize the thought of projection to neuronal operations under streaming data.
Mcanwhile, continual learning be classified into three fundamental types: task-incremente innraininccemmental, and class-incremental (van de Ven et al., 2022). Task-CL. requires task ID) at training and testing, domain-CL does not need task ID, while class-CL. requires comparing new tasks with old tasks and inferring context without task ID. Following previous gradient projection methods, we mainly consider task- and domain-incremmental settings. As for class-CL, it inevitably requires some kind of replay of old experiences (via samples or other techniques) for good perfor- mance since expects explicit comparison between new classes and old ones (van de Ven et al.] 2020). Our method may be combined with replay methods to first update with projection and then learn with replay, or with context-dependent processing modules similar to biological systems (Zeng et al. 2019, Kudithipudi et al. 2022), e.g., task classifiers. This is not the focus of this paper.
3
PRELIMINARIES
3.1
SPIKING NEURAL NETWORK
We consider SNNs with the commonly used leaky integrate and fire (LIF) neuron model as in previ- ous works (Wu et al. 2018 Xiao al. 2022). Each spiking neuron maintains membrane potential u to integrate input spike trains and generates a spike once u reaches a threshold. The dynamics of (u ureat) + R I(t) for < Vin. where I is the input
du the membrane potential are Tm di current, ViA is the threshold, and R and Tm are resistance and time constant, respectively. When reaches Vo, at time t¹, a spike is generated and u is reset to the resting potential u = urest- The out- put spike train is defined using the Dirac delta function: s(t) = Ens 8(t - t)). Consider the simple current model (tt) = E, Wiys,(t (t) + b, where represents the i-th neuron, wij is the weight from neuron j to neuron i and 6; is a bias, then the discrete computational form described as:
[t 1] ((u,(t] st) + 2 Wijs((t +b,
(1)
[[t++] Hu, (t ++11 Vi.),
where (xx) is the Heaviside step function, s is the spike train of neuron at discrete time step t, and À < 1 is a leaky term based on Tm and the discrete step size. The constant R, Ti. and time step size are absorbed into the weights and bias. The reset operation is implemented by subtraction.
Different from ANNs, the supervised learning of SNNs is hard due to the non non-differentiabiliy of spiking neurons, and different methods are proposed to tackle the problem. One kind of training approach is based on the explicit encoding of spike trains, such as (weighted) firing rate (Wu et al.) [021Xiaa et al. 202T Meng et al. 2022) or spiking time (Mostafa 2017). and calculates gradients through analytical transtormations between the spike representation. Another kind is spike- based approaches by backpropagation through time (BPTT) and surrogate gradients (SG) without the ex- plicit coding scheme (Shrestha & Orchard 2018 Bellec al., 2018 Wu et al. 2018 Zheng al.) 2021 Li et al. 2021 Fang et al., [201]] Yin et al. (021). Some works further improve BPIT for temporally online learning that more consistent with biological learning and friendly for on-chip training (Bellec et al. 2020 Xiao et al. [2022 Meng et al. (202)). We will consider these three kinds
of training methods and more details can be found in Appendix A
3

---

Published as a conference paper at ICLR 2024
3.2 ORTHOGONAL GRADIENT PROJECTION
Orthogonal gradient projection methods protect knowledge for each neural network layer. For feed- forward synaptic weights W E IRxx between two layers, suppose that the presynaptic input vec- tors x IR. previous tasks span a subspace X, the goal is to make the updates A AWP orthogo- AWPxold x
0. This ensures that the weight update does not interfere with old tasks, since (W + AWW "xx " = Wx d we calculate projection matrix P to the subspace orthegonal to the principal subspace of X, then gradients can be projected as AWP AWPT. Previous works leverage different methods to calculate P. For example, Zeng et al. 2019) estimate it as P = A(A A + aI A (where A represents all previous inputs) with the recursive least square algorithm. Saha et al. 2022 instead calculate P = I- I-M M, where M E R&*x denotes the matrix of top k principal components of X calculated by SVD with small batch of data and M M the projection matrix to the principal subspace of X. However, these methods cannot be implemented by neuronal operations for neuromorphic computing, and they only mainly estimate the projection matrix with a small batch of data, which can be biased.
nal to the subspace X so that 7x E X, A AWP
(a) Overview
Activity trace
x
Hebbian oxtracts principle subspprce
Orthogonal projection activity
Darameter update
liebbian learning expands principle
Consolidation
Learning the first task
Learning successive
after learning
(b) Lateral circuits
(c) Hebbian learning in lateral circuits
(d) Projection with lateral circuits
trublien etage juebiiis
Hubbian
dil
Eroo
Forward
New subspace
AW-WW
W+
HT
Subspace
Hebbian
neurons
Consolidated
Notatioms:
umeal attiattes
eutqpore
After consolication
-ith cone-ngmmetrc
Yut' H..H"" ("" druats tonaess interviill resporess projected activty trose
priginal synaptic weight uodate
synaptic weight update projection
Figure 1: Illustration of the proposed Hebbian learning based orthogonal projection (HLOP). (a) Overview of HLOP for continual learning. Hebbian learning extracts the principal subspace of neuronal activities support orthogonal projection. For successive tasks, orthogonal projection is based on the consolidated subspace, and a new subspace is constructed by Hebbian learning, which merged for consolidation after learning. (b) Illustration of lateral circuits with memettc connection weights. (c) Hebbian learning in lateral circuits for construction of new subspaces for new tasks. (d) Orthogonal projection based on recurrent lateral connections. The presynaptic activity traces whose definition depends on training algorithms, are modified by signals from lateral connections. Synaptic weights W are updated based on the projected traces.
METHODS
4.1
HEBBIAN LEARNING BASED ORTHOGONAL. PROJECTION
We propose Hebbian learning based orthogonal projection (HLOP) to achieve orthogonal projection with lateral neural connections and Hebbian learning, and the sketch is presented in Fig. [ The major conception to leverage Hebbian learning in a lateral circuit to extract the principal sub- space of neural activities in current tasks, which enables the lateral circuit to project activity traces (traces recording activation information used for synaptic update) into the orthogonal subspace, as mod-
illustrated in Fig. aa). To this end, different from the common feedforward networks, els consider recurrent lateral connections to a set of "subspace neurons" for each layer (Fig. [bb). Lateral connections will not influence feedforward propagation and are only for weight updates.
4

---

Published as a conference paper at ICLR 2024
Recall that the thought of orthogonal projection is to project gradients with P = M M (Sec-
tion 3.2). Since existing supervised learning methods calculate the weight update as A AW= = ōx where o is the eπror signal and x is the activity trace of neurons whose definition depends on training
algorithms, we can correspondingly obtain the projected update by AWP = 8 (x - M (MMx) The above calculation only requires linear modification on presynaptic activity traces x which can be implemented by recurrent lateral connections with skew-symmetrie synaptic weights, as shown in Fig. [b..d.. Specifically, the lateral circuit first propagates x to "subspace neurons" with con- nections H as y = Hx, and then recurrently propagates y with connections -H for the response x = = y. The activity trace will be updated as & = x x = x H Hx. So as long as the lateral connections H equal or behave similarly to the principal component matrix M, the neural circuits can implement orthogonal projection.
Actually, H can be updated by Hebbian learning to fulfill the requirements, as shown in Fig. Tc). Hebbian-type learning has long been regarded as the basic learning method in neural systems Hebb 2005) and has shown the ability to extract principal components of streaming inputs (Oja 1982 1989). Particularly, Oja's rule (Ojal 1982) was first proposed to update weights h of input neu- rons x to onc-unit output neuron y as he h + 7)t3 (x, 1) so that h, will converge to the principal component of streaming inputs, which can achieve the state-of-thh-an conver- gence rate in the streaming principal component analysis problem (Chou & Wang. 2020). Sev- eral methods extend original Oja's rule to top k principal components (Ojal 1989] Diamantaras & Kung 1996; Pehlevan al.) 2015), and we focus on the subspace algorithm that updates weights as
n (yx yy H). which enables weights to converge to a dominant principal sub- space Yan et al. 1994). We propose again leverage the recurrent lateral connections for Hebbian learning. The neurons first propagate signals to "subspace neurons" as y = Hx, and "subspace neurons" transmit information through the recurrent connections as x = - y. The connection weights H are updated by AH = yx +yx with two-stage Hebbian rules considering presynap- tic activities and postsynaptic responses. The skew-symmetrie weights share symmetric but opposite update directions, which corresponds to the Hebbian and anti-Hebbian learning, respectively. The above condition is the construction of the first subspace neurons. If we further consider learning new subspace neurons with connections H' for new tasks with existing consolidated subspace neu- rons, we can only update the connections to new neurons by the similar Hebbian-type learning with presynaptic activities and integrated postsynaptic responses X x x- from both new and consolidated subspace neurons, i.e., AH' = y'xT + y'x as shown in Fig. [c). This enables the extraction of new principal subspaces not included in the existing subspace neurons unbiasedly from streaming large data. More details are in Appendix C
HLOP is neuromorphhi--riendly as only requires neuronal operations with neural circuits. At the same time, HLOP may better leverage the asynchronous parallel ad advantage of neuromorphic com- puting since the projection and Hebbian learning can be parallel with the forward and error propaga- tion of the feedforward network. The proposed HI.OP also has wide adaptability to the combination with training methods based on presynaptic activity traces and other post-processing methods. As HIOP mainly focuses on the constraint of weight update direction during the instantaneous learning of new tasks, it can also be combined with post-proct pest-proeessing methods such as episodic replay after the learning to improve knowledge transfer between different tasks.
4.2 COMBINATION WITH SNN TRAINING
The basic thought of HLOP is to modify presynaptic traces, and thus HLOP can be flexibly plugged into arbitrary training methods that are based on the traces, such as various SNN training methods.
As introduced in Section 3.1 activity traces varies for diverse training methods, and HI.OP is adapted differently. For methods with explicit coding schemes, activity traces correspond to specified spike representations of spike trains (c.g., weighted firing rates). HLOP acts the presynaptic spike representation. For BPTT with SG methods, activity traces correspond to spikes at each time step. HLOP can recursively act on presynaptic spike signals at all time steps. For online methods with eligibility traces, activity traces correspond to eligibility traces. HLOP can act on traces at all time steps, which can be implemented by modifying eligibility traces with neuronal responses to lateral circuits.
we will consider different SNN training methods. The definition of
We illustrate the combination between HLOP and eligibility trace based methods in Fig. 2 (more details about other training methods can be found in Appendix A). Such temporarily online methods
5

---

Published as a conference paper at ICLR 2024
(a)
Forward of SNNs
(b) Standard training with eligibility traces
8
[4]
Forward
W
AW [4]
s'(:)
s'[c]
Time
Time
(c) HLOP training by modifying eligibility traces with lateral circuits
(d) HLOP with lateral spiking neurons
[C]
High-frequency
Wlllo
s|c
Rate y
Responses
[c]
Figure 2: Illustration of combining HLOP with SNN training. (a) Illustration of spiking neural networks with temporal spike trains. (b) SNN training methods based eligibility traces and online calculation through time (Bellec et al. 2020 Xiao et al. 2022). (c) Applying HL.P by modifying traces with lateral circuits. The eligibility traces correspond
activity traces in the illustration of HLOP and are modified for synaptic weight update. Neuronal traces will consider both spike responses for feedforward inputs and responses for recurrent lateral circuits. (d) Illustration of HLOP bursts.
with lateral spiking neurons based on the rate coding of
may better suit on-chip training of SNNs, and HLOP can be integrated with online modification of traces by taking additional neuronal response to lateral circuits into account. In experiments, we will consider DSR (Meng et al. 2022). BPTT with SG (Wu et al. 2018 Shrestha & Orchard 2018), and OTTT (Xiao et al. 2022) as the representative of the three kinds of methods.
The original HLOP leverages linear neurons in lateral circuits. Considering neuromorphic com- puting systems, this may be supported by some hybrid hardware (Pei et al.) (019). For more general systems, it is important investigate if HLOP can be implemented by the same spiking neurons as well. To this end, we also propose HLOP with lateral spiking neurons based on the rate coding of high-frequency bursts. As shown Fig. d), spiking neurons in lateral circuits will generate high-frequency bursts, and the responses are based on spiking rates. We simulate this by quantizing the output of subspace neurons in experiments, i.c., the output y is quantized as slamp sxc.eii,,,,, x7i
9 scale X spiking neurons as specified in experiments (see Appendix D for more details).
scule
where scale is taken as 20 and Ti is the time steps of lateral
5 EXPERIMENTS
We conduct experiments for comprehensive evaluation under different training settings, input do- main settings, datasets, and network architectures. We consider two metrics to evaluate continual learning: Accuracy and Backward Transfer (BWT). Accuracy measures the performance on one task, while BWT indicates the influence of learning new tasks on the old ones, which is defined as BWT Accuracy
Accuracy,, i. where Accuracy,, is the accuracy on the i-th task after learning & tasks (k: 2 i). The average accuracy after task & is the average of these 1 tasks, while the average BWT is the average of previous k tasks. Experimental details are Appendix
5.1 RESULTS OF DIFFERENT SNN TRAINING METHODS
We first evaluate the effectiveness of HLOP for different SNN training methods on the PMNIST dataset under the online and domain-incremente setting. As shown in Table HLOP significantly improves the results by overcoming catastrophic forgetting under all training methods. The vanilla baseline (i.e., direct training for sequential tasks) suffers from significant forgetting (> 20%) that leads to poor accuracy, while HLOP can achieve almost no forgetting and behave similarly for different training methods. In the following, we leverage DSR as the representative for more settings.

---

Published as a conference paper at ICLR 2024
Table I: Results of average accuracy (ACC) and average backward transfer (BWT) for SNNs on PMNIST under different training methods.
SNN Training Method
Method
ACC (%)
BWT (%)
DSR DSR
Baseline
70.61
28.88
HL.OP
95.15
-1.30
BPTT with SG
Baseline
71.17
-28.23
BPTT with SG
HLOP
95.08
-1.54
OTТТ
Baseline
74.30
-24.86
OTTT
HLOP
94.76
-1.56
100
Average accuracy (16)
Multitakk (8P)
95
Multitak (SS)
Multitak (FA)
Average BWT
Baseline
-10
Baseline (55)
(BP)
-35
Baseline (FA)
(55)
-20
NLOP (aP)
(FA)
-25
NLOP (55)
(BP)
-30
HLOP (1A)
Task number
HLOP (FA)
Task number
(a) PMNIST
Average accuracy (1K)
Multitak (sP)
Average BWT (%)
Multtask (55)
Baseline (BP)
Baseline (BP)
Banaline (5S)
Baseline (55)
MR (SP)
MR (BP)
MR (55)
MR (55)
HLOP (BP)
HLDP (AP)
-10
(55)
10
HLOP (55)
10
Task number
Task number
(b) CIFAR-100
Multitak
Average accuracy
85
Multtask (55)
Baseline (BP)
Average BWT
-.t. Baseline (55)
65
Baseline
MR (BP)
MR (SP)
MR (55)
(55)
a5
HEOPP
(55)
HLOP (55)
(BP)
Task number
HLOP-MR (55)
Task number
(c) 5-Datasets
Figure 3: Continual learning results under different settings. "Multitask" does not adhere to contin- ual learning and can be viewed as an upper bound. "BP" (backpropagation), "FA" (feedback align- ment), "SS" (sign symmetric) denote different enror propagation methods. "MR" denotes memory replay. (a) Average accuracy and backward transfer (BWT) results on PMNIST under the online setting. (b) Average accuracy and BWT results 10-split CIFAR-100. (c) Average accuracy and BWT results on 5-Datasets (CIFAR-10 10, MNIST, SVHN, FashionMNIST, and notMNIST).
5.2 RESULTS UNDER DIFFERENT DATASETS AND SETTINGS
Then we consider three different settings similar to Saha et al. 2021 (1) continual learning on PMNIST with fully connected networks and single-head classifier (i.c.. the online setup (i.c., each sample only appears once); (2) continual learning on 10-split CIFAR- 100 with convolutional network and multi-head classifiers (i.e., :ss-incrementtl);; (3) continual learning on 5-Datasets (i.e., the sequential learning CIFAR-10 10, MNIST, SVHN, FashionMNIST, and notMNIST) with deeper ResNet- 18 networks and multi-head classifiers (i.e., mask-incremental).
under
These settings cover comprehensive evaluation with similar (CIFAR-1O4 or distinct (PMNIST, 5- Datasets) input domains, task- and domain-incremental settings, online multi-epochs training.

---

Published as a conference paper at ICLR 2024
simple to complex datasets, and shallow fully connected to deeper convolutional architectures. We also consider different error propagation methods including backpropagation (BP), feedback align- ment (FA) (Lillicrap et al.) 2016), and sign symmetric (SS) (Xiao al., 2018). Since BP is often regarded as biologically implausible due to the "weight transport" problem (Lillicrap et al. 2016 I
FA and SS can handle the problem more biologically plausible alternatives by relaxing feedback
weights as random weights or only sharing the sign of forward weights (see Appendix B for more details). They differ from BP in the way to propagate error signals while still calculating gradients based on presynaptic activity traces and errors, so they are naturally compatible with HLOP.. Since FA does not work well for large convolutional networks (Xiao al. 2018)) we only consider FA in the PMNIST task with fully connected layers and include SS in all settings.
As shown in Fig. 3 HI.P consistently solves catastrophic forgetting of SNNs under different set- tings as well as different error propagation methods. While in the baseline setting the forgetting differs for different error propagation methods, is similar when HLOP is applied, demonstrating the consistent ability of HLOP to overcome forgetting. Compared with memory replay (MR) meth- ods, HLOP achieves higher accuracy and less forgetting, and HLOP can also be combined with memory replay to further improve performance.
300
Nostra BINT (96
60
Moltte Randlin WHIOP
Bassline
20
HLOP
H4.0P (paldig
-30
tipking
tiphing,
topling 230)
40
Average Acc Average BWTT
Task number
Task number
(a) PMNIST
Moonne SUUT 19
Mulfite Roodee
16.0P
20
480)
HEOP Upliing. 4M0)
batknng 1130)
15.77 jusking 1190)
WHLOP ryplding
HROP tppiing, 20001
10
HLOP juilinng 4000)
10
HLDP burking 4890)
Avorage Acc
Average BWT
Task number
Task number
(b) CIFAR-100
Figure 4: Continual learning results of HLOP with lateral spiking neurons. "Multitask" does not
adhere to continual learning and can be viewed as an upper bound. "(spiking, TiY denotes lateral spiking neurons with Ti time steps for discrete simulation of bursts. (a) Average accuracy and backward transfer (BWT) results on PMNIST under different time steps for HI.OP (spiking). (b) Average accuracy and BWT results on 10-split CIFAR-100 under different time steps for HLOP (spiking).
5.3 RESULTS OF HLOP WITH LATERAL SPIKING NEURONS
We verify the effectiveness of HLOP with lateral spiking neurons and compare the results with the original linear HLOP in Fig. 4 As shown in the results, HLOP (spiking) also effectively deals with forgetting. And on PMNIST, HLOP (spiking) can achieve similar performance with small number of time steps (e.g., 40). shows that we can effectively leverage spiking neurons to implement the whole neural network model and achieve good performance for continual learning.
5.4 COMPARISON WITH OTHER CONTINUAL LEARNING METHODS
We compare the performance with other representative continual learning methods. We compare HLOP to the baseline, the small memory replay (MR) method, the regularization-based method EWC (Kirkpatrick et al., 2017), the parameter isolation method HAT (Serra et al., 2018)) and the gradient projection-based method GPM (Saha et al.) (021). We consider Multitask as an upper bound, which all tasks are available and learned together. Experiments on task-incremental 20- split minilmageNet are also compared following Saha et al. (2021). We also provide results for the domain-CU Caainn-C&sse setting on 5 5-Datasets and comparison results between ANNs and SNNs in Appendix G
'That means the inverse connection weights between neurons should be exactly symmetric to the forward ones, which is hard realize for unidirectional synapses in biological systems and neuromorphic hardware.
8

---

Published as a conference paper at ICLR 2024
Table 2: Comparison results with other continual learning methods under different settings.
PMNIST
10-split CIFAR-100
minilmageNet
5-Dattaets
Method
ACC (%) BWT (%)
ACC (%)
BWT (%)
ACC (%) BWT (%)
ACC (%) BWT (%)
Multitask
96.15
/
79.70
/
79.05
89.67
/
Baseline
70.61
28.88
75.13
-4.33
40.56
-32.42
45.12
60.12
MR
92.53
-4.73
77.67
-1.01
58.15
-8.71
79.66
-14.61
EWC
91,45
33.20
73.75
-4.89
47.29
-26.77
57.06
-44.55
HAT
N.A.
N.A.
73.67
-0.13
50.11
-7.63
72.72
-22.90
GPM
94.80
- -1662
77.48
-1.37
63.07
-2.57
79.70
-15.52
HLOP (ours) 95.15
-133
78.58
-0.26
63.40
-0.48
88.65
-3.71
Experiments are under online setting, i.e., each sample only appears once (except for the Memory Replay method), and the comain-incrrmennte setting.
Multitask does not adhere to the continual learning setting and can be viewed as an upper bound.
Accuracy (%)
80
90
Accuracy (%)
90
80
Accuracy (%)
70
70
60
Accuracy
80
EWC
30
38
20
20
HLOP
10
10
HLOP-MR
2
1
2
1
Task number
Task number
Task number
Task number
Task 1
Task 2
Task
Task 4
Figure 5: Continual learning results of different methods on 5-Datasets for each task after learning successive ones.
As shown in Table 2 HLOP consistently outperforms previous continual learning approaches under various settings considering both accuracy and backward transfer. The comparison results of accu- racy for each task after learning successive ones by different methods 5-Datasets is illustrated in Fig. 5 showing that HLOP successfully solves forgetting for each task and outperforms previous approaches. Compared with replay and regularization methods, HLOP enables less forgetting and higher accuracy due to the better constraint of weight update direction. Compared with the param- eter isolation method, HLOP achieves higher accuracy due to the flexibility of network constraints with nigh-dimensional subspaces rather than explicit neuronal separation. Compared with the previ- ous gradient projection-based based method, HI.PP achieves better performance since HLOP has better construction of projection space from streaming data, especially on the 5-Datasets with distinct input distributions. Moreover, HLOP is based on purely neuronal operations friendly for neuromorphic hardware. These results show the superior performance of HLOP and indicate the potential for building high-performance continual neuromorphic computing systems.
CONCLUSION
In this work, we propose new method based on lateral connections and Hebbian learning for continual learning of SNNs. Our study demonstrates that lateral neural circuits and Hebbian learning can systematically provide strong continual learning ability by extracting principle subspaces of neural activities and modifying presynaptic activity traces for projection. This sheds light on how neural circuits and biological learning rules can support the advanced abilities of neuromorphic computing systems, which is rarely considered by popular neural network models. We are also the first to show how the popular thought of orthogonal projection can be realized in pure neuronal systems. Since our HLOP method is fully composed of neural circuits with neuronal operations and effective for SNNs as well as more biologically plausible error propagation methods, it may be applied to neuromorphic computing systems. And the possible parallelism of the lateral connections and Hebbian learning with the forward and error propagation may better leverage its advantages. Although mainly focus gradient-based supervised learning for SNNs in this paper, HLOP may also be extended to other learning methods based on presynaptic activity traces. We expect that approach pave solid paths for building continual neuromorphic computing systems.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Z. Lin was supported by National Key R&D Program of China (2022200160300) the NSF China (No. 62276004), and the major key project of PCL, China (No. PCL2021A12). D. He was supported by National Science Foundation of China (NSFC62376007).
REFERENCES
Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, et al. TrueNorth: Design and tool flow of a 65 mw million neuron programmable neurosynaptic chip. IEEE Transactions on
Computer-Aided Design of Integrated Circuits and Systems, 34(10):1537-1557 2015.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Confer- ence on Computer Vision (ECCV), 2018.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass, Long short-term memory and learning-to-leann in networks of spiking neurons. In Advances Neural Information Processing Systems, 2018. 8.
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajck, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neu- rons. Nature Communications, 11(1):1-15, 2020.
Thomas Bohnstingl, Stanislaw Wožniak, Angeliki Pantazi, and Evangelos Eleftheriou. Online spatio-temporal learning in deep neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Natalia Caporale, Yang Dan, et al. Spike lming-dependeet plasticity: a hebbian learning rule. Annual Review of Neuroscience, (1(1):25-66 2008.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2019.
Chi-Ning Chou and Mien Brabeeba Wang. ODF-inspired analysis for the biological version of Oja's rule in solving streaming PCA. In Conference on Learning Theory, 2020.
Julia M Comsa, Krzysztof Potempa, Luca Versari, Thomas Fischbacher, Andrea Gesmundo, and Jyrki Alakuijala. Temporal coding in spiking neural networks with alpha synaptic function. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). 2020.
Mike Davies, Narayan Srinivasa, Tsung- Han Lin, Gautham Chinya, Yongqiang Cao. Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-9)) 99, 2018.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(7):3366-333 3385, 2021.
Konstantinos Diamantaras and Sun Yuan Kung. Principal component neural networks: theory and applications. John Wiley & Sons, Inc., 1996.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Advances in Neural Information Processing Systems, 2021.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In International Conference on Artificial Intelligence and Statistics, 2020.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128-135, 1999.
10

---

Published as a conference paper at ICLR 2024
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-basee neural networks. arXiv preprint 1rXin:1312..2211,000 2013.
Xu He and Herbert Jaeger. Overcoming catastrophic interference using conceptor-aided backprop- agation. In International Conference Learning Representations, 2018.
Donald Olding Hebb. The organization of behavior: neuropsychological theory. Psychology Press, 2005.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554-255 2558, 1982.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka baabska-Barwinska al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3522 3526, 2017.
Dhireesha Kudithipudi, Mario Aguilar-Simon Jonathan Babb, Maxim Bazhenov, Douglas Black- iston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196- 210, 2022.
Axel Laborieux, Maxence Ernoult, Tifenn Hirtzlin, and Damien Querlioz. Synaptic metaplasticity in binarized neural networks. Nature Communications, 12(1):1-12, 2021.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10:508, 2016.
Yuhang Li, Yufei Guo, Shanghang Zhang. Shikuang Deng, Yongqing Hai, and Shi Gu. Differen- tiable spike: Rethinking gradient-descent for training spiking neural networks. In Advances in Neural Information Processing Systems, 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935 2900)9979999997,, 2017.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, (1):1-10, 2016.
Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. In International Conference on Learning Representations, 2022.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, 2018.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi Quan Luo. Training high-pen gigh-performance low- low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Towards memory-and time-efficient backpropagation for training spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Hesham Mostafa. Supervised learning based on temporal coding in spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 29(7):327-3235, 2017.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In Advances in Neural Information Processing Systems, 2016.

---

Published as a conference paper at ICLR 2024
Erkki Oja. Simplified neuron model principal component analyzer. Journal of Mathematical Biology, 15(3):267-273, 1982.
Erkki Oja. Neural networks, principal components, and subspaces. International Journal of Neural Systems, 1(01):61-68, 1989.
Priyadarshini Panda, Jason M Allred, Shriram Ramanathan, and Kaushik Roy. Asp: Learning to forget with adaptive synaptic plasticity in spiking neural networks. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 8(1):51-64, 2017. 2017.
Cengiz Pehlevan, Tao Hu, and Dmitri B Chklovskii. A hebbian/anti--ebbiin neural network for linear subspace learning: A derivation from multidimensional scaling of streaming data. Neural Computation, 27(7):1461-1499 2015.
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao. Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid Tiajjic chip architecture. Nature, 572(7767):06-111 2019.
Arjun Rao, Philipp Plank, Andreas Wild, and Wolfgang Maass. A long short-term memory for ai applications in spike-based neuromorphic hardware. Nature Machine Intelligence, 4(5):467-479, 2022.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, 2019.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- based machine intelligence with neuromorphic computing. Nature, 575(7784)607--677 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv:1606.044 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference Learning Representations, 2021.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference Machine Learning, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, 2017.
Sumit Bam Shrestha and Garrick Orchard. Slayer: spike layer error reassignment in time, In Advances in Neural Information Processing Systems, 2018.
Nicholas Soures, Peter Helfer, Anurag Daram, Tej Pandit, and Dhireesha Kudithipudi. Tacos: task agnostic continual learning in spiking neural networks. In Theory and Foundation of Continual Learning Workshop at ICML. 2021, 2021.
Timothy Tadros, Giri P Krishnan, Ramyaa Ramyaa, and Maxim Bazhenov. Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks. Nature Communications, 13 (1):7742, 2022.
Gido M van de Ven, Hava Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11(1):1-14, 2020.
Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185-1197 2022.
12

---

Published as a conference paper at ICLR 2024
Stanistaw Wožniak, Angeliki Pantazi, Thomas Bohnstingl, and Evangelos Eleftheriou. Deep learn- ing incorporating biologically inspired neural dynamics and in-memory computing. Nature Ma- chine Intelligence, (6)):25-3366 2020.
Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learning rule for effective training and rapid inference of deep spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. training high-performance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
backpropagation for
Yujie Wu, Rong Zhao, Jun Zhu, Feng Chen, Mingkun Xu, Guoqi Li, Sen Song, Lei Deng, Guanrui Wang, Hao Zheng, et al. Brain-inspired global-local learning incorporated with neuromorphic computing. Nature Communications, 13(1 (11):65,000) 2022.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang. and Zhouchen Lin. Training feed- back spiking neural networks by implicit differentiation on the equilibrium state. In Advances in Neural Information Processing Systems, 2021.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, and Zhouchen Lin. Online training through time for spiking neural networks. In Advances in Neural Information Processing Systems, 2022.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang, and Zhouchen Lin. SPIDE: A purely spike-based method for training feedback spiking neural networks. Neural Networks, 161: 9-24, 2023.
Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. satogicclly-plausiiiee learning al- gorithms can scale to large datasets. In International Conference on Learning Representations, 2018.
Wei- Yong Yan, Uwe Helmke, and John B Moore. Global analysis of oja's flow for neural networks. IEEE Transactions on Neural Networks, 5(5) 683,,44433361 1994.
Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient time-domain classi- fication with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10): 905-913, 2021.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependen pro- cessing in neural networks. Nature Machine Intelligence, 1(8):364 372, 2019.
Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural Computation, 154 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning. 2017.
Wenrui Zhang and Peng Li. Spike-train level backpropagation for training deep recurrent spiking neural networks. In Advances in Neural Information Processing Systems, 2019.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-traines larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Shibo Zhou, Xiaohua Li, Ying Chen, Sanjeev Chandrasekaran, and Arindam Sanyal. Temporal- coded deep spiking neural network with easy training and robust performance. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
13

---

Published as a conference paper at ICLR 2024
A DETAILS OF SNN TRAINING METHODS AND COMBINATION WITH HLOP
In this section, we provide a detailed introduction to SNN training methods used in our work and the combination of the proposed HLOP with them.
A.1 DSR
DSR (Meng et al. 2022) is a training method based on the explicit encoding of spike trains and the analytical transtormations between spike representations. For the LIF neuron model, DSR considers the discrete update equations as:
u(t] = e-= At vlt 10-1 +1 -1- Tlt)
+[1] = Hu([1] - Ven):
v|{] = vl!! - Vt].
where u(t] is the membrane potential before spiking, s(t] is the output spike, v[t] is membrane potential after spiking and resetting. [[t] is the input current, t is the time step index, is the Heaviside step function, Ve, is the threshold, At is the discrete step, and r is the time constant.
DSR defines the spike representation of the spike train with time steps the weighted firing rate:
Veh
T
X [
a [
(3)
where À - For multi-layer feedforward neural networks, let a' [T] denote the spike represen- tations neurons at layer 1, DSR derives that a' [T] approximates sub-differeentiable mappings z' with the analytical closed-form transformation as:
VL
z [T] = clamp ( (T),0,
(4)
At
where W the weight matrix between layer | and layer Then a [T] can be viewed as following such transformation well.
With the explicit encoding a' [T] of spike trains and the approximate closed-form transforma- tions between a' (T). DSR backpropagates errors and calculates gradients based a (T] and their transformations, shown in Fig. 6()). The gradients for weights are calculated by at AL -y Pa ([T] da Ba [T) where 81+1 is the backpropagated error signal. For static images, the inputs to SNNs are set as real-valued pixel values at each time step, which can be viewed as the input currents, and the loss is calculated based on the encoding aÑ at the last layer. To reduce the representation error, DSR additionally proposes to train the spike threshold and introduce a hyperparameter a adjust the threshold and reset potential. We maintain these techniques following the default setting in the released code. Hyperparameters are set as the default value (Meng et al. 2022): T' 20, ViA- 0.3, T 1.0, At 0.05, a 0.3, bounnd 0.0005, where /hound hyperpa- rameter to bound the trainable threshold.
37 aw. and can be rewritten in the form V 8 a
When applying our proposed HLOP the DSR method, a [T] can be viewed the activity traces and we act on them, as shown in Fig. 6(b). Note that a' a'[T] is calculated based spikes at each time step with different coefficients, so is equivalent to applying HLOP to all spike signals: the calculation of the projection for weight update is the same. Considering Hebbian learning of lateral weights, there could be two choices to update weights: using the weighted firing rate (spike rep- resentation) for calculation using all spike trains at each time step for multiple calculations (i.e.. Hebbian learning is carried out at each time step for multiple times). We verified both methods and their results are similar, shown in Table So leverage the weighted firing rate by default for efficiency.
The thought to train SNNs with analytical transformations for the explicit encoding of spike trains is also shared by many works (Lee et al. 2016 Zhang & Li 2019 Wu et al.) 202T] Xiao et al. 2021
2023 Mostafa 2017 Comsa al. 2020 Zhou al. 2021 These works leverage (weighted) firing
14

---

Published as a conference paper at ICLR 2024
(a) Forward
(b)
[c]
Rep(s!+1)
Forward
HLOP
JoJ.
st
W1
At
WilL+
Rep(s!)
6 Saup(e))
Time
Time
(c) BPTT with SG
Forward
Error
Wi
Time
Reversed Time
Forward computational graph
Backpropagation through
Surrogate gradient
(d) OTTT
s'(E]
Forward
W!
W
W
HLOP
Time
Forward eligibility traces
Online through
Madify eligibility traces
Figure 6: Illustration of applying HLOP to different supervised SNN training methods. (a) Illustra- tion of spiking neural networks with temporal spike trains. (b) Training methods based explicit coding schemes, e.g., DSR (Meng et al. 2022). Gradients are calculated by analytical transforma- tions between spike representation (Rep(s s') the figure), such as weighted firing rate. Activity traces correspond to the spike representation and HLOP can act on the presynaptic spike represen- tation. (c) Training methods with backpropagation through time and surrogate gradients. Gradients are recursively calculated for all previous time steps. Activity traces correspond to spikes at each time step and HLOP can act on all presynaptic spike signals.
(d) Training methods with online calculation through time and eligibility traces (Bellec al.) 2020 Xiao et al. (022).. Gradients are calculated with tracked traces and instantaneous errors. Activity traces correspond the tracked traces and HLOP act all the traces. This only requires online modifying traces by lateral signals, i.c., neuronal traces will consider both spike responses for feedforward inputs and responses for lateral circuits.
Table 3: Results of average accuracy (ACC) and average backward transfer (BWT) with different Hebbian learning settings for the DSR SNN training method.
PMNIST
10-split CIFAR-100
Hebbian Learning Setting
ACC (%)
BWT (%)
ACC (%)
BWT (%)
weighted firing rate
95.15
-1.30
78.58
-0.26
spike trains
95.23
-1.34
78.42
-0.41
rate / accumulated response for transformations (Lee et al. 2021 Wu et al.] 2021 Xiao et al.] 2021]] 2023) or consider the first spiking time as the encoding
2016 Zhang & Li 2019 Zhou al.
and derive the analytical transformations (Mostafa. 2017 Comsa al. 2020 Zhou et al. 2021
15

---

Published as a conference paper at ICLR 2024
By treating the explicit encoding activity traces, our proposed HLOP can be applied all these methods similarly to DSR. HLOP provides a general way to support continual learning for training methods based on presynaptic traces, which is orthogonal to specific SNN training approaches.
A.2 BPTT WITH SG
Backpropagation through time (BPTT) with surrogate gradients (SG) is a popular way to train SNNs without specific coding schemes (Shrestha & Orchard 2018 Bellec et al. 2018 Wu et al. 2018 Zheng et al., 202T; Li et al., 2021 Fang et al.) |021] Yin et al. 2021). BPTT unfolds the discrete computational graph of SNNs through time and treats it similarly to recurrent neural networks, as shown in Fig. (().. The non-differentiable problem of the spiking function is tackled by leveraging the derivative of a smoothed function as a surrogate. Specifically, for the computation through time:
uu [++ A(u [t] Vins w' +bbbbb
(5)
s s'[t+1] (u' [t + 1] Vin),
the gradients are calculated as:
OL
ÖL 0s [[]]
@du+1 [t]
1-T II
Ou 1 i+ Oult i]
+
Ow
0s Ou'+ [t]
aw!
T<t i=1 Oul+"(7)
(6)
0u 0u'-1 - i 1]
Os
|t i]
Ou
Os 0s|+1(1 i i] au +ul+1[
[
ow
ar is the error backpropagated to layer + at time I, and the non-differentiable terms where a
0s|[ Bul
will be replaced by surrogate derivatives, e.g., derivatives of rectangular sigmoid func-
a= tions (Wu et al. 2018;: au sign (Iu - Vaa < 2 or a= du a2 (1+e(Vu-w)/eay) (a1 and 02 are hyperparameters). We leverage the sigmoid-like functions in experiments and set 02 0.25. The gradients can be rewritten in the form as VwL - Σi= 8 where [1+ is the error for each time step recursively backpropagated through time. For static images, the inputs to SNNs are also set as real-valued pixel values each time step as the input currents, and the loss is calculated based on the firing rate at the last layer.
When applying our proposed HLOP to the BPTT with SG method, the activity traces are all spikes [t]] at each time step, and HLOP acts on all spike signals, as shown in Fig. ((c). Both projection and Hebbian learning are based on spikes, and the traces of spikes are all projected to formulate the projected gradient of L.
A.3 OTTT
OTTT (iaoo et al. 2022) is training method to improve BPTT for temporally online learning. BPTT requires storing the computational graph unfolded over time and backpropagating through previous time steps, which is inconsistent with biological online learning and learning rules on neu- romorphic hardware. Some works introduce eligibility traces to improve BPTT for online training through time (Zenke & Ganguli 2018: Bellec et al. 2020 Bohnstingl et al. [022 Xiao et al. 2022), in which OTTT's is an efficient and scalable approach with more theoretical guarantees. Specifically, OTTT analyzes the gradients of BPTT and proposes to decouple the temporal dependency by track- ing the presynaptic activities and leveraging instantancous gradients as:
a a'(t)
Σ X (T),
a'lt + 1] Aa' [t]
s' + 1].
TSA
[
N-1-2 os N
as [t]
T
Butti [t]
(8)
Os [t)
N t
0u
t=0
V
Vw
Bn--1 [t]]((t]
(9)
[=1
16

---

Published as a conference paper at ICLR 2024
L is the upper bound of the commonly used firing-rate- based loss when L is convex function such as cross- entropy. With such calculation. OTTT does not require backpropagation through time and enables forward-in-time learning, as shown in Fig. d).
where Lit] - (t)(t,)) is the instantaneous loss and the total loss L= -
When applying our proposed HLOP to these eligibility traces-based methods, the activity traces correspond to the tracked traces at each time step, and HLOP acts on them, as shown in Fig. d.d. Note that the tracked traces are caleulated based on spike trains, the projection of traces only requires modifying eligibility traces by lateral signals, i.e., the neuronal traces consider both spike responses for feedforward inputs and responses for lateral circuits. Specifically, the traces are orig- inally accumulated by a'(t + 1] = Aá [t] + s'(t + 1] for each neuronal spike, but will change to [t+1] when HLOP is applied, considering the response s' [t 1] for lateral circuits with the input signal s'[t 1]. c.g., s' + 1] -H, H,s"(t + 1]. In this way, both projection and Hebbian learning are based on spikes well. The online training methods are more friendly for neuromorphic online learning, and the easy application of our HLOP with purely neuronal operations to these methods can pave solid path for continual on-chip training of neuromorphic computing systems.
a'[ [s
B DETAILS OF BACKPROPAGATION, FEEDBACK ALIGNMENT, AND SIGN SYMMETRIC
In this section, we provide a detailed introduction to backpropagation (BP), feedback alignment (FA), and sign symmetric (SS) used our work.
Forward
Error
Forward
Forward
Error
W
WT
W
W
(W)
(a) Backpropagation (BP)
(b) Feedback Alignment (FA)
(c) Sign Symmetric (SS)
Figure 7: Illustration of different error propagation methods.
As shown Fig. BP, FA, and SS mainly differ in the way they back-propagate error signals across layers. BP follows the chain rule of gradient calculation to propagate errors. Specifically, for the feedforward calculation y Wx, the error will be backpropagated by VxL W V. This requires symmetric weights for forward and backward connections between neurons.
FA (Lillicrap al. 2016) relaxes the symmetric weight requirement and proposes leverage ran- dom feedback weights F to propagate error. They observe that training with random feedback weights can lead the alignment of forward weights and feedback weights, enabling effective learning. FA replaces the calculation of V = w V for each layer by V FV There are other variants of FA such as direct feedback alignment (DFA) (Nøkland, (016)) etc., and we con- sider FA in experiments. The feedback weights F are initialized following the same distribution as the forward weights, e.g., the Kaiming uniform distribution.
SS (Xiao al. 2018) also relaxes the strict symmetric weight requirement but does not fully random feedback weights. They consider that it is possible to pass a little information about the sign direction between forward and feedback weights, and this enables SS to scale to large-scale models and tasks. SS replaces the calculation of VxL= = w Vy for each layer by V = s x sign W V.L. where is a scale factor to prevent gradient explosion. We follow the same setting in Xiao al. (2018).
17

---

Published as a conference paper at ICLR 2024
C MORE DETAILS OF HLOP
Our HLOP method adds recurrent lateral connections for each layer of neural network models. For the 1-th layer of SNNs, the output spike trains s' st[t] of feedforward neurons will also be passed to the lateral subspace neurons with skew-symmetric connection weights H2 and -H," For DSR, the spike trains are mainly treated as the specified spike representation for the calculation of HI.OP, and for BPTT with SG and OTTT, spikes or eligibility traces at each time step are considered for HLOP. In the following, we first denote activity traces x; for simplicity.
When learning the first task, there is no consolidated subspace but only newly expanded subspace neurons with weights Hi. So there will be projection for the feedforward synaptic update and is still based on the activity trace X The lateral synaptic weight is updated by Hebbian learn- ing as AH;
y/x; + y/ where y' = x & -H; y/- Since the Hebbian learn- ing can be viewed as stochastic gradient descent for minimizing an explicit objective function minn; Ex, X H H/x; (Pehlevan al. 2015), inspired by the momentum technique com- monly used in gradient- based methods, we also introduce momentum update in practice. We use momentum to track the synaptic update and modify weights based on the momentum. The momen- tum is taken as 0.9 in practice. After learning each task, the newly expanded subspace neurons will be consolidated.
When learning successive tasks, consolidated subspace neurons with weights H2 will project the ac- tivity traces, and new subspace neurons with weights H/, will be constructed with Hebbian learning. Let Y: H x -H, yı, v/ H and x, - y/ denote signals of recurrent lateral connections of consolidated and new subspaces, respectively. & = Xi + + is the modified activity trace and the feedforward synaptic update is calculated based on the projected activity trace & in- stead of the original one, as well as the error signal. x; = x; + x is the integrated postsynaptic response to the recurrent connections for Hebbian update. Only the new lateral synaptic weight H; is updated by Hebbian learning as AH; y' y/&,, which can be viewed as minimizing the objective function minH; Ex, x H. H2x: - H(x: to extract the new principal subspace not included in the existing subspace neurons. The momentum technique is also applied. And in practice, for each mini-batch data, we update lateral synaptic weights by K times to accelerate con- vergence. We take K and the learning rate as 0.01 for Hebbian learning in experiments. After learning the new task, the new subspace neurons will be consolidated and the connection weights H are concatenated into H,.
For convolutional operations, the shared kernel will linearly act on multiple patches of feature maps. We view the dimension of presynaptic inputs of feedforward neural networks as the dimension of these patches and view them as different input samples. Hebbian learning is similarly applied to the shared lateral connections between these patches and subspace neurons.
D MORE DETAILS OF HLOP WITH LATERAL SPIKING NEURONS
In this section, we provide more details about HLOP with lateral spiking neurons. We consider that the subspace neurons in the lateral circuits can generate high-frequency bursts in a short time and then the neuron model is approximated by the non-leaky integrate and fire (IF) model. The computation of the IF neuron model with input current is described as:
it + 1] (u[t] Vea s(t) + I, + = H(u [t + 1] - ViA),
(10)
is easy to prove that the scaled output spiking rate of the IF model, i.c., r[T] = Ypp ΣT= s'tt)
is quantization of the input when I is positive, i.e., r[T] = Yiك V with a clipping to the range [o, V.a. If we initialize the membrane potential as 0 and modify the spiking threshold and reset po- tential Vis 2 and Vis respectively, we obtain a more precise quantization r r[7] T V,
Via T7
By considering the rate of figh-frequency bursts and the response to the burst, can obtain HLOP with lateral spiking neurons by taking the output of subspace neurons as the quantization of Hx and the (scaled) response to burst spikes x .H' y, shown in Fig. 8 a).
18

---

Published as a conference paper at ICLR 2024
(a)
(b)
High-freeuunccy Surst
s[c]
H
Rate
Will
Responses [[t]
HTT
(c)
Input
(d)
HT
spike $1
spike S2
+ (-52)
output
Figure 8: Illustration HLOP with lateral spiking neurons. (a) HLOP with rate coding of high- frequency bursts of spiking subspace neurons. (b) Illustration of high-frequency bursts with differ- ent time scale than feedforward spiking neurons. Each feedforward spike can correspond to burst of lateral neurons. (c) Conceptual illustration of ternary neuron couples with common neuron mod- els for ternary outputs. (d) Illustration of realizing ternary neuron couples for HLOP with opposite connection weights.
One problem is that the common neuron models only spike for positive inputs, limiting the ability to deal with negative signals. We require ternary spiking in order to imitate the original linear neurons of HLOP, i.e., the spiking function should be:
1,
Va ult
al + #1+1=777 Tul( 1), Voa)
0, lu + 1]] S Vin
(11)
-1,
ult + < Vah
One possible approach to enable such ternary spiking on hardware. Another way is to realize the ternary output by neuron couples with the common positively spiking neuron model, intro- duced in (Xiao et al.] 2023). The conceptual illustration of such ternary neuron couples is shown in Fig. 8 c), where two neurons receive opposite inputs and share small recurrent connection to synchronize resetting. The operation of taking negative can be realized by reconnecting neurons with opposite weights, i.c., the two coupled neurons are oppositely connected to other neurons. Fig. d) illustrates such realization for HLOP modules. This may require weight sharing between connections to neuron couples.
With such a design, we can leverage spiking neurons to imitate linear neurons except for some quantization. We simulate our spiking HIOP in experiments by quantizing the output subspace (sumply -xcate xT]
neurons in experiments. Specifically, the output is quantized as ý = scale x
seate
T: Both projection and Hebbian learning are based on ÿ. Since it is expected to be realized by high- frequency bursts short time, the spiking HLOP can have different time scale than spiking neurons in the feedforward network shown in Fig. ((b). and projection and Hebbian learning with rate coding burst spikes can carried out for spikes/eliiibility traces in the feedforward network at each discrete time step which has larger temporal interval.
E EXPERIMENTAL DETAILS
Our experiments are conducted on PMNIST, 10-split CIFAR- 100, 5 S-Datasets, and 20-split minilm- ageNet. The PMNIST dataset is variant of the MNIST dataset and each task is different random
19

---

Published as a conference paper at ICLR 2024
permutation of the original pixels. The 10 sequential tasks share the same classifier with 10 classes' output. This single-head classifier setting will require orthogonal projection for the update of the classifier. The 10-split CIFAR-100 dataset splits 100 classes of the CIFAR- 100 dataset into 10 tasks with 10 classes per task. 5-Datasets is a sequential of CIFAR-1O, 10, MNIST, SVHN, Fashion MNIST, and notMNIST, where each task has 10 classes. The 20-split minilmageNc dataset splits 100 classes of the minilmageNet dataset into 20 tasks with 5 classes per task. These three settings use multi- head classifiers for different tasks and do not need projection for classifiers. We normalize the inputs based on the dataset statistics and do not use any data augmentation. We do not split the training and validation set and take the model at the last training epoch as the final model for testing because it would be vague to determine the best model considering both the accuracy and fitness of Hebbian learning.
As for the network structures, for experiments on PMNIST, we consider the fully connected net- works with two hidden layers 184-800-800-10, and all tasks share the same classifier. For experi- ments on 10-split 100, we consider three convolutional hidden layers whose kernel size is 3 and whose channels are 64, 128, and 256, and different tasks have different final classifiers. A batch normalization (BN) is added after each convolutional operation, which follows the practice in DSR (Meng et al. 2022) and can be absorbed into linear layers after training. Average pooling is added after each spiking layer. For experiments on 5-Datasets, we consider reduced ResNet- network following previous work (Saha et al. (02T), whose base channel is set as 20, and different tasks have different final classifiers. For experiments on 20-split minilmageNet, we also consider the reduced ResNet-18 network, and the first convolution has stride 2 and kernel size 5, similar to Saha et al. (20TT) For SNN models, the input and output encodings follow the practice in differ- ent training methods (Meng et al. 2022 Xiao et al. 2022) when we use them (BPTT with SG and OTTT follow the same setting as in Xiao et al. (2022) and BPTT with SG uses the rate-based loss), and other details and hyperparameters also follow them, e.g., we take the default 20 time steps for DSR while the default 6 time steps for BPTT with SG and OTTT.
For the continual learning setting, we only track the statistics of BN for the first task and will fix them for the successive tasks following the practice in Mallya & Lazebnik (2018), in order to avoid interference with old tasks. The learnable threshold in the DSR method is also learned in the first task and fixed in the following tasks. We follow the common practice to track statistics during training and use stored statistics during testing.
Our comparison methods include baseline, small memory replay (MR), EWC (Kirkpatrick et al. 2017). HAT (Serra et al., 2018), and GPM (Saha al. 2021). The baseline method simply trains models for each task sequentially. The MR method randomly stores 50 input samples for each class in current tasks and replays all stored samples for learning with additional 20 epochs after each new task (for minilmageNet, the stored sample size is 5 for each class, corresponding to one percent of data similar other settings). EWC and HAT are implemented by adapting the official code in (nfrrrra eet al. (2018), and GPM is implemented by adapting the official implementation. We also consider the Multitask method as an upper bound that simultaneously learns all tasks at once. All methods share the same training setting.
For PMNIST, we consider the online setting and train models for epoch by SGD with a batch size of 64 and a learning rate of 0. The Multitask setting for PMNIST is slightly different the batch size of each task is set as to enable a similar number of weight updates under the online setting and the learning rate is taken as 0.01. The hyperparameters for HAT, EWC, and GPM are taken as the default in the code. For HLOP, both hidden layers and the classifier require orthogonal projection, and the number of subspace neurons for the three layers is set as [80, 200, 100] (from bottom to top) at the first task. For each later task, the number of newly expanded subspace neurons is initially [70, 70, 70] for each layer and is reduced by 20 every 3 tasks.
For 10-split CIFAR-100, we train all models for 200 epochs by SGD with momentum 0.9. The batch size is 64, and the learning rate is 0.01 with the cosine annealing scheduler 0 well as linear warm-up in the first epochs. The hyperparameters for EWC, HAT, and GPM are taken as the default in the code. For HLOP, the number of subspace neurons is set as [6. 100, 200] for each layer at the first task (around 1/5 as the dimension of presynaptic inputs), and the number of newly expanded subspace neurons for successive tasks is set as [2. 20, 40]. For the combination between HLOP and MR, we do not update lateral connections during replay.
20

---

Published as a conference paper at ICLR 2024
For 5-Datasets, we train all models for 100 epochs by SGD with momentum 0.9. The batch size is 64, and the initial learning rate is 1. for the first task and 0.01 for successive tasks, with the cosine annealing scheduler to O. The hyperparameters for EWC, HAT, and GPM are taken as the default in the code. For HIOP,, the number of subspace neurons is set as |6, [40, 40], [40, 40), [40, 100, 6]. [100, 100], [100, 200, 8), [200, 200], [200, 200, 16], [200, 200]] for each presynaptic layer in ResNet- 18 at the first task (around 1/5 as the dimension of presynaptic inputs) and the number of newly expanded neurons is the same.
For 20-split minilmageNNt, we train all models for 100 epochs by SGD with momentum 0.9. The batch size is 64, and the initial learning rate 0.1 for the first task and 0.01 for successive tasks, with the cosine annealing scheduler to 0. The learning rate for multitask is 0.01. The hyperparameters for FWC, HAT, and GPM are taken as the default in the code/paper. For HI.P,, the number of subspace neurons is set as [24, [90, 90], [90, 90], [90, 180, 10]. [180, 180], [180, 360, 20], [360, 360], [360, 720, 40), [720, 720]] for each presynaptic layer in R8sNet-18 at the first task (around 1/2 as the dimension of presynaptic inputs) and the number of newly expanded neurons is [2, [6. 6]. [6. 6]. [6, 12, 1J. [12, 12). [12. 24, 2). (24, 24), [24, 48, 4], [48, 48]] (around 1/30) for the first five successive tasks and [0, [2. 21. [2, 21, 12, 4, 0]. [4, 4), [4, 8, 0], [8, 8). [8. 16, 0], [16, 16]] (around 1/90) for others. The number is decreasing because there are many shared subspaces between different tasks (minilmageNNt has similar input domains) and the considered network capacity is not large enough.
Currently, the number of subspace neurons is hyperparameter for HIOP, which is similar to the threshold hyperparameter in GPM (Saha al.. 2021). It may be determined with some prior knowl- edge about the task difficulty, e.g., first train the network for a short time to decide the roughly required number (for example, the ratio of the dimension of presynaptic inputs) and then formally perform training with Hebbian learning. It can be interesting future work to study how to automati- cally and adaptively determine the number with neuronal mechanisms.
All code implementations are based on the PyTorch framework and experiments are carried out on one NVIDIA GeForee RTX 3090 GPU, All experiments and dataset split are under the same random seed 2022.
F MORE DISCUSSION
F.1 DISCUSSION ON HEBBIAN LEARNING
One major component of our method is Hebbian learning. Hebbian learning is often related to memory modeling, e.g., Hopfield network (Hopfeld 1982) with Hebbian-type learning can serve as associative memory systems. In this work, we provide new perspective on how Hebbian learning can support advanced learning abilities, which is not in an associative approach. Extraction of the principal subspace of streaming data is to some extent similar to memory but is not in an associative approach, and can well fit into our projection- focuses Hebbian learning based on neuronal spiking activities, which can be shown to optimize synaptic weights for principal component analysis (Ojal 1982 1989). In biological systems, spike- timing- -eeenddd plasticity (STDP) is the often observed Hebbian learning considering the time of spikes (Caporale et al.] 2008). However, STDP has not been shown to optimize an explicit objec- tive. It is interesting future work to study if the biological STDP rule can be similarly leveraged to systematically promote advanced abilities such as continual learning.
continual learning method. This work mainly
A basic focus of our work is neuromorphic computing. Neuromorphic computing aims at the com- putation inspired by the structure and function of the human brain. At the hardware level, neu- romorphic chips are designed to imitate biological neurons (Akopyan et al. 2015; Davies et al.) 2018 Pei et al. 2019). such as their spiking property and synaptic connections with local storage of weights for in- m--memory computation, for highly energy-efficient and parallel event-driven computa- tion with avoidance of frequent memory transpose (its computation architecture is expected to be different from the commonly used hardware with von Neumann architecture such as CPU or GPU). At the algorithm level, we are interested in developing methods compatible with some properties and operations of neurons so that they are possible for deployment on the hardware. Also, since neuro- morphic hardware is under development considering software-harde are simulated on common hardware (c.g.. GPU) while considering neuromorphic properties. From the perspective of neuromorphic computing, Hebbian learning can be mapped to the computation of
co-design, most algorithms
21

---

Published as a conference paper at ICLR 2024
neurons and synapses and implemented on neuromorphic hardware, so it is more suitable for SNNs than other PCA approximation algorithms.
F.2 DISCUSSION OF LMITATIONS/CIALLENGHH
As for method, HLOP introduces additional computational costs for learning the lateral weights of each layer during training, which will increase training costs in our current implementation codes. While this process can theoretically be parallel the normal forward-backward propagation of net- work training as discussed in Section 4.1 such parallelization may not be easily realized in estab- lished deep learning libraries such as PyTorch. So for our implementation of GPU training, the training time would be slightly longer and the evaluation of parallelization is lacking. It can be fu- ture work on low-level code optimization or consideration of asynchronously parallel neuromorphic computing hardware. Additionally, HLOP currently requires a manual specification for the num- ber of subspace neurons, which may be improved to automatically and adaptively determine the allocation. For example, the Generalized Hebbian Algorithm can perform Crram-Shmmidt orthonor- malization on the rows of the weight matrix, which may help to sweep out unnecessary neurons and connections, but it requires some non-local information and may not be directly suitable to method. It can be interesting for future work to study improvements.
As for the evaluation, similar to other gradient projection methods, this work mainly focuses task- incremental and domain-incremental continual learning settings. There another setting which, discussed in Section _ inevitably requires some kind of replay of old experiences for good performance since it expects explicit comparison between new classes and old ones. So following previous works, our evaluation mainly focuses on task- and domain-incremennal settings. can be future work to study if HLOP can be combined with some replay methods or context- dependent processing modules similar biological systems (e.g., with a task classifier) for better class- iaaseinncremetal tasks.
G ADDITIONAL RESULTS
G.1 DOMAIN-CL ON 5-DATASETS
To further show the scalability of our method for the Comain-incremmen setting larger datasets and networks apart from PMNIST, we supplement the domain results on 5-Datasets. Compared with the task-incremental settings, it shares the final classifier for all tasks, i.e., considering single- head classifier. All models are trained for 10 epochs and MR also performs 10 epochs for additional replay training. As shown in Table 4 (where HAT requires task ID and is not feasible), HLOP significantly outperforms other methods, indicating the superiority of our method.
Table 4: Comparison results on the Domain- CL 5-Datasets.
Method
ACC (%)
BWT (%)
Multitask
82.81
/
Baseline
32.46
-74.86
MR
70.26
-24.12
EWC
35.80
-65.18
HAT
N.A.
N.A.
GPM
45.62
-56.28
HLOP (ours)
79.59
-9.94
G.2 COMPARISON BETWEEN ANNS AND SNNS
To further study whether the improvement of method others mainly comes from the bet- ter combination with SNNs or the continual learning ability, we supplement all the corresponding results of ANNs that replace the spiking neuron by the ReLU activation.
As shown in Table 5 the performance of some methods can differ considering ANNs and SNNs. Particularly, 5-Datasets, HLOP has more improvements for SNNs. It may imply that our method
22

---

Published as a conference paper at ICLR 2024
Table 5: Comparison results between ANNs and SNNs.
PMNIST
10-split
CIFAR-100
minilmageNet
5-Datasets
Network
Method
ACC
BWT
ACC
BWT
ACC
BWT
ACC
BWT
Multitask
96.15
79.70
79.05
89.67
Baseline
70.61
-28.88
75.13
-4.33
40.56
-32.42
45.12
-60.12
MR
92.53
-4.73
77.67
-1.01
58.15
-8.71
79.66
-14.61
SNN
EWC
91.45
-3.20
73.75
-4.89
47.29
-26.77
57.06
-44.55
HAT
N.A.
N.A.
73.67
-0.13
50.11
-7.63
72.72
-22.90
GPM
94.80
-1.62
77.48
-1.37
63.07
-2.57
79.70
-15.52
HTOP (ours)
95.15
-1.30
78.58
0.26
63.40
-0.48
88.65
-3.71
Multitask2
96.81
80.33
82.05
88.65
Baseline
73.55
25.67
71.47
-7.82
38.87
-31.58
68.54
33032
MR
92.44
55.00
75.59
-266
55.15
-6.79
83.98
-9.29
ANN
EWC
90.16
-3.46
72.49
6601
47.41
-21.46
68.60
29.90
HAT
N.A.
N.A.
71.31
0.00
56.98
-1.64
88.40
-3.63
GPM
94.92
1.56
77.56
-1.42
65.13
-096
84.88
-9.17
HLOP (ours)
95.25
-1.28
78.98
-0.34
66.80
1.93
88.68
-2.7
Experiments under the online setting, i.c., each sample only appears once (except for the Memory Replay method), and the -inrminnnncrmmentlll setting.
Multitask does not adhere the continual learning setting and can viewed as an upper bound.
can be better combined with spikes, for example probably because subspaces expanded by spike signals are harder learn and therefore our Hebbian learning performing streaming PCA has more advantages over GPM that only performs SVD/PCA on a small batch of data which can be biased. Also, our method is promising for ANNs well. As there is no systematic study on the difference between the continual learning performance of ANNs and SNNs, it can be interesting for future work to study it.
23


Reviews:
[{"review_id": "T5H5HwOROl", "reviewer": "Reviewer_oRoY", "summary": "Unlike biological intelligence, current deep learning suffers from catastrophic forgetting— upon learning new tasks, networks often lose the ability to solve previously learned tasks. \nOne set of methods to solve catastrophic forgetting is orthogonal gradient projection, in which the learning gradient for new tasks are projected to a subspace that is approximately orthogonal to the subspace of the gradient of the network w.r.t. the weights for old tasks.\nHowever, they are not applicable to Spiking Neural Networks (SNNs), which is the predominant architecture for neuromorphic learning.\nThis paper proposes Hebbian Learning based Orthogonal Projection, or HLOP, an orthogonal gradient projection method that extracts principal subspaces of neuronal activities using Hebbian learning.\nHLOP is compatible with SNNs, and outperforms other continuous learning methods on several computer vision datasets.\nThus, it may be a promising new direction for continual neuromorphic learning.", "questions": "- At the end of Section 3.2, you mention that previous methods “cannot be implemented by neuronal operations for neuromorphic computing,” but do not provide further explanation. \nTo me, this is important because it provides critical motivation for your method; it highlights why your novel approach using Hebbian learning is required.\nI understand that these methods cannot be directly implemented on SNNs out-of-the-box, but can you elaborate on why it is difficult or infeasible to adapt them for SNNs?\n\n- The performance of HLOP is close to the upper bound (specified by Multitask performance) for each dataset except miniImageNet, where there is a large gap. Can you provide reasoning or intuition for why this is the case?\n\n- Your results show that HLOP outperforms several other continual learning methods. Can I assume that the results achieved with HLOP are state-of-the-art on these datasets? Or are there other methods that you did not compare against?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The authors introduce a novel combination of Hebbian learning with orthogonal projection to solve catastrophic forgetting for SNNs. While using Hebbian learning to find principal subspaces is not a new technique, its application to this problem is both novel and elegant.\n- HLOP achieves strong empirical performance, surpassing all other continual learning methods that the authors benchmarked against.", "weaknesses": "- The final paragraph of the intro that discusses your contributions is a bit dense. I suggest breaking it up, and allocating more intro space to discuss your contributions, as it’s the most important part of your intro. In particular, I think you should have a separate paragraph to discuss the experimental setup and results, and include some performance numbers to quantify the strength of your method; currently its strength is hard to judge from the intro.\n\n- The background on SNNs provided in Section 3.1 is a word-for-word replica of Section 3.1 in Xiao et al. (2022).\nIt’s fine to reuse definitions from previous work, but a direct replica like this should be explicitly attributed to avoid plagiarism concerns (even if it’s your own work).\n\n- There is almost no discussion on the current limitations/challenges of HLOP and the experimental design, making it hard to judge the tradeoffs between using HLOP versus other approaches.\nI’d like to see the authors also include a discussion on the potential downsides of HLOP and limitations of the evaluation process.\n\n- Minor issue, but the sentence in Section 5.2 explaining the weight transport problem of backprop is very long and hard to parse.\nI suggest either rewriting it and breaking it down to smaller chunks, or removing it entirely as it’s not central to your work; just stating that FA and SS are more biologically-plausible and more amenable to neuromorphic hardware is enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eQaHaUMLGM", "reviewer": "Reviewer_braR", "summary": "This work proposes a task-incremental continual learning (CL) method for spiking neural networks.\nIt can be categorized as a CL approach based on orthogonal gradient projection.\nIn orthogonal gradient projection approaches, the update $\\Delta \\mathbf W^P$ of each layer is projected to a subspace to minimize changes to the outputs for previous tasks.\nThis is achieved by performing PCA on the input data for each layer and projecting the gradient to the subspace orthogonal to the principal subspace of the input data.\n\nThe main technical novelty of this work is the application of a Hebbian learning rule to perform PCA.\nThis rule is claimed to be more suitable for neuromorphic hardware.", "questions": "- Is the Hebbian rule a better fit for spiking neural networks compared to other approximate PCA approaches?\n- If it is, what makes the Hebbian rule more suitable, and why aren't the other approaches as effective?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n- The combination of spiking neural networks and continual learning seems interesting, although this is not the first work to address such problems.\n- Code is provided in the supplementary material.", "weaknesses": "### Not Comparing Other Approximations of PCA\n\nThe essence of the proposed Hebbian approach is to perform PCA.\nWhile there is a huge literature on more efficient approximate PCA with various forms of tradeoffs, the Hebbian approach is just one of such variants of PCA.\n\nI think the authors need to justify why their Hebbian approach is particularly suitable for spiking neural networks.\nThey vaguely argue that the Hebbian rule only requires \"neuronal operations,\" but the neuroscience-backed algorithm eventually boils down to some matrix-vector arithmetic, just like many other approximate PCA algorithms.\nCurrently, I do not see any reason that the Hebbian rule should be more suitable for spiking neural networks while others are not.\n\n### Task-Incremental Settings\n\nThis paper exclusively focuses on the task-incremental settings.\nTask-incremental CL is often considered the most naive and easiest form of CL.\nEspecially, providing task IDs even at test time is far from realistic and significantly reduces its practical utility.\nI believe that relying solely on task-incremental experiments is insufficient to establish meaningful results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NJ7EslKRUJ", "reviewer": "Reviewer_Txkh", "summary": "This paper proposes Hebbian learning based orthogonal projection (HLOP) as a novel method for implementing orthogonal projection using neuronal operations. Building upon the method of calculating projection matrices through SVD proposed by Saha et al. (2021), HLOP combines the properties of Hebbian learning to approximate orthogonal projection using learned weight matrices. This is the first approach that fully utilizes neuronal operations for implementing orthogonal projection. Furthermore, HLOP outperforms several baseline methods on multiple continual learning datasets, demonstrating the reliability of the proposed method through experimental results.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper introduces for the first time a method that fully utilizes neuronal operations to approximate orthogonal projection matrices, and achieves the best performance surpassing multiple baseline methods on various datasets in continual learning task.\n2.The implementation based on Hebbian learning can seamlessly integrate with different training methods and aligns well with the parallel local learning approach of neuromorphic chips.\n3.HLOP can be directly applied to SNNs, providing a new approach for continual learning in SNNs.", "weaknesses": "1.The method lacks sufficient detail in the methodology section. Providing complete formulas or a specific illustrative example would enhance understanding. \n2.The baselines compared in the study include EWC (Kirkpatrick et al., 2017), HAT (Serra et al., 2018), and GPM (Saha et al., 2021). It would be valuable to investigate if there are recent works that achieve better results than these methods.\n3.Although the continual learning approach in this study relies solely on neuronal operations without directly utilizing past data, the increasing number of subspace neurons with each new task learned implies a form of data compression and storage to some extent.\n4. Although the paper uses Hebbian learning to achieve a pure neuronal operation method, it seems that the newly added subspace neurons do not participate in the model's forward process. If this is the case, then despite being a pure neuronal operation approach, it is essentially just an estimation method for orthogonal projection matrices.\n5. While the paper demonstrates the effectiveness of HLOP, it does not provide an explanation or analysis of why the weight matrices obtained through Hebbian learning can serve as a substitute for the orthogonal projection matrix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ONAde7cRvk", "reviewer": "Reviewer_Me3Q", "summary": "The paper applies Oja's rule to implement gradient projection for continual learning. The method uses lateral circuits to avoid storing gradient explicitly, and works with spiking networks.", "questions": "The proposed architecture looks like a model of memory, since neurons only update their weights if they haven't seen a specific input before. Can all task-specific lateral circuits be combined into a single associative memory module, like a Hopfield net, with each new memory being a new $y$ for the task?\n\nBefore Sec. 2:\n>  Our results indicate that lateral circuits, which are long ignored by popular feedforward neural network\n\nI’d say that lateral circuits are often present implicitly through normalization layers.\n\nTab. 2: boldface should be used for the best performing method in a column, not the author's method.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "To the best of my knowledge, the method is novel. Orthogonalized gradients are known to perform well for continual learning, and Oja's rule is known to do PCA, but (as far as I know) they've never been combined like in this paper. The resulting method doesn't need to explicitly store gradients, and computes the projection matrix on the go with bio plausible operations (and not recursive least squares). This is an advantage over other projection-based methods. \n\nThe method only uses Hebbian and anti-Hebbian plasticity, which makes it suitable for neuromorphic hardware and also biologically plausible. There's a caveat for biological networks though: the lateral circuits are only active during the backward pass, and don't interfere with the forward one. However, an exact biological implementation seems out of scope for this work.\n\nPerformance: on all (standard) benchmarks the method performs very well and usually outperforms other algorithms.", "weaknesses": "Some (not very critical) weaknesses:\n\n1. The method needs to create new subspaces for each task, and then coordinate activity in a new subspace with the old ones. It's not clear if that can scale to many tasks (e.g. due to noise in PCA through Oja's rule) \n2. The lateral connectivity doesn't influence forward propagation (and it shouldn't due to projections), which might make it hard to implement with real neurons (not so sure about neuromorphic chips).\n3. There are no evaluations on non-spiking networks, so it's not clear if the performance improvement over other methods is due to those being poorly suited for spikes or this method being better at continual learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)