Mitigating Catastrophic Forgetting in Spiking Neural Networks through Threshold Modulation
Authors: Ilyass Hammouamri
Timothée Masquelier
Dennis George Wilson
Venue: Accepted by TMLR ()

Abstract: Artificial Neural Networks (ANNs) trained with Backpropagation and Stochastic Gradient Descent (SGD) suffer from the problem of Catastrophic Forgetting; when learning tasks sequentially, the ANN tends to abruptly forget previous knowledge upon being trained on a new task. On the other hand, biological neural networks do not suffer from this problem. Spiking Neural Networks (SNNs) are a class of Neural Networks that are closer to biological networks than ANNs and their intrinsic properties inspired from biology could alleviate the problem of Catastrophic Forgetting. In this paper, we investigate if the firing threshold mechanism of SNNs can be used to gate the activity of the network in order to reduce catastrophic forgetting. To this end, we evolve a Neuromodulatory Network that adapts the thresholds of an SNN depending on the spiking activity of the previous layer. Our experiments on different datasets show that the neurmodulated SNN can mitigate forgetting significantly with respect to a fixed threshold SNN. We also show that the evolved Neuromodulatory Network can generalize to multiple new scenarios and analyze its behavior.

Published in Transactions on Machine Learning Research (11/2022)
Mitigating Catastrophic Forgetting in Spiking Neural Net- works through Threshold Modulation
Tlass Hammouamri
syosshammmoummriecnn..
CerCo CNRS UMR 5549, Université Toulouse III, Toulouse, France
Timothée Masquelier
timotheee.aasqueeerrrmen
CerCo CNRS UMR 5549, Université Toulouse III, Toulouse, France
Dennis Wilson ESpperr,,, Université de Toulouse, Toulouse, France
dennis, wilson @isae.
Reviewed on OpenReview: https: // openreview. net/ forum? id= 15SoThZmtU
Abstract
Artificial Neural Networks (ANNs) trained with Backpropagation and Stochastic Gradient Descent (SGD) suffer from the problem of Catastrophic Forgetting; when learning tasks sequentially, the ANN tends to abruptly forget previous knowledge upon being trained on a new task. On the other hand, biological neural networks do not suffer from this problem. Spiking Neural Networks (SNNs) are class of Neural Networks that are closer to biological networks than ANNs and their intrinsic properties inspired from biology could alleviate the problem of Catastrophic Forgetting. In this paper, we investigate if the firing threshold mechanism of SNNs can be used to gate the activity of the network in order to reduce calastrophic forgetting. To this end, we evolve a Neuromodulatorr Network that adapts the thresholds of an SNN depending on the spiking activity the previous layer. Our experiments on different datasets show that the neurmodulated SNN can mitigate lorgetting significantly with respect to a fixed threshold SNN. We also show that the evolved Neuromodulator, Network can generalize to multiple new scenarios and analyze its behavior.
1 Introduction
Agents in nature have the ability to learn novel tasks while still retaining previously acquired knowledge, it is a fundamental quality of biological neural networks. In contrast, Artificial Neural Networks (ANNs) suffer from the infamous problem of Catastrophic Forgetting (CF) (McCloskey & Cohen 1989 French 1999), where learning a new task causes the ANN to completely and abruptly forget previously learned tasks. Usually, this problem is not encountered when the the data is shuffled aka independent and identically distributed (i.i.d)). Most, if not all, of deep learning's (DL) achievements are in the i.i.d. case. However, a large portion of problems in the real world requires agents to learn sequentially from new encountered data.
Spiking Neural Networks (SNNs) are class of neural networks which are more biologically inspired than standard ANNs (Maass 1997). In SNNs, neurons are stateful and emit spike only when their potential exceeds a certain threshold. Recently, it has been made possible to train deep SNNs using backpropagation and gradient descent through surrogate gradients (Neftci et al., 2019 Zenke et al. 2021) to achieve good performance on classification tasks. Furthermore, SNNs make an interesting candidate for many real world applications due to their properties such as energy efficiency and fast inference when implemented on neuro- morphic hardware. However, the field of SNNs is still in its early phases, and their properties are still under study.
When learning sequentially using an ANN, training on a new task with backpropagation and SGD will continue to modify the weights of the network, including weights that were important for previous tasks,

---

Published in Transactions on Machine Learning Research (11/2022)
and push them in the direction of the new task. This is not the case in SNNs, weight update only occurs when the corresponding pre-synaptic neuron's membrane potential reaches a threshold and emits a spike, unlike ANNs where the weights of a neuron with a very small activation value still gets updated. This mechanism could potentially serve as a task-dependent gating, where weights that are important for previous tasks are preserved if the corresponding neurons do not fire during the learning of new tasks. Previous works have explored artificially reproducing this mechanism in ANNs: Masse al. (2018) used a gating signal which informs the network explicitly of the task it is being trained or tested on and [Beaulieu et al. (2020) used neuromodulatory network that produces an element-wise wise multiplicative mask to gate the activations.
In this paper, we investigate if the intrinsic mechanisms of SNNs can be leveraged to gate neural activity in a way that mitigates catastrophic forgetting. To achieve this, we use neuromodulatory network that adapts the firing threshold of neurons. This neurmodulatory network only has access to the local information of the previous layer's spiking activity, and not the network input data as in Beaulieu et al. (2020). Instead of introducing an artificial gating mechanism, we use the inherent gating of SNNs and take direct inspiration from neuromodulatory mechanisms in the brain by acting on the threshold of spiking neurons.
2 Background
2.1
Neuron Models
The neuron models we use for our SNNs are The Leaky Intograte-and-Fire (LIF) model (Gerstner et al. 2014) for dynamic datasets and the Integrate-and-Firee (IF) model (without leak) for static datasets. The LIF model is widely used in computational neuroscience and machine learning for its simplicity and performance.
The sub-threshold dynamics of the 4t LIF/IE neuron are described by the following differential equation:
AV* dt = (((((),)(()
(1)
where f function that depends on the neuron's model, for the LIF and IF, f is described by Eq and 3 respectively.
((V((),T())) ((V(()) Vrest) (1() f(V'(t) = (t)
(2)
(3)
In the equations above, V is the nouron's membrane potential, I the input current received by the neuron, T is the membrane time constant and Vreat the neuron's potent ial at rest.
For numerical simulation, following Fang et al. (2021), we use discrete-time equations that are an approxi- mation of the differential equations above:
H' f(V 11,1 + V 1] v'[ = [d[ -t + S1
(4)
(5)
1'[ [ Via)
(6)
where Ht] and Vi[t] are the membrane potential of the ith neuron at time-step t, after the integration of the input current r'[t] (Eq. 4 and after the spiking dynamics (Eq. and 6), respectively. θ is the Heaviside function; S*(4] equals if the membrane potential is above the firing threshold Vtn (neuron fires at time-step t) and otherwise. If a spike occurs, the membrane potential is instantly reset to Vreat, which corresponds a so-called "hard reset", otherwise follows the sub-threshold dynamics (Eq As suggested by Zenke & Vogels (2021), we ignore the neuronal reset when computing gradients by detaching them from the computational graph.
2

---

Published in Transactions on Machine Learning Research (11/2022)
The imput current [d] is defined
1'1 = Σ wiy St
where wi denotes the synaptic weight between neuron and the afferent neuron
If a neuron i is subject to neuromodulation, its firing threshold Vi V it is a function of time and the network's ' spiking activity:
at time is not constant, but instead [4]
Vil. S(t)
(8)
2.2 Surrogate Gradient Learning
Recently, it has been made possible train deep SNNs a supervised manner using backpropagation and gradient descent in ANNs through surrogate gradient learning (SGL) (Neftci et al. 2019 Zenke et all 2021 One of the key challenges in this method is the non-differeniaablliy of the spiking function. The derivative of i required by backpropagation, however, the Heaviside function's derivative is zero almost everywhere and not defined at zero, which makes LIF neurons unsuitable for gradient based optimization.
To overcome this issue, Neftci et al. 2019 proposed using a surrogate gradient as a smooth continuous relaxation the spiking function. During the forward pass, the spiking function is unchanged, whereas the backward pass the derivative of the Heaviside function is approximated by the derivative of smooth continuous function. In our work, we use the sigmoid function for static datasets.
O'(x) ( (())=(())== =
(9)
(1 + e my
and atan (are tangent) function for the dynamic datasct
1
O (x) R aarctannn2 1 1 x²
(10)
2.3 Continual Learning
Continual Learning concerns the problem in which a single neural network must learn the tasks T2, Tw) in sequential manner and under the condition that only the data of the current task is available for training, without catastrophically forgetting the previously learned tasks. This differs from standard learning in ANNs, where data from multiple tasks T are presented in mixed mini-batches sampled independently and which are meant to be identically distributed (i.id.).
However, due to differences in experimental protocols, the difficulty of the problem can change drastically. A categorization scheme proposed in van de Ven & Tolias (2019), where different continual learning scenarios are defined based on the availability of the task-ID the network during training. Task-IL, domain-IL and Class IL (in order increasing difficulty, where IL stands for Incremental Learning) denote the scenarios where the task-ID) is provided, task-ID is not provided and task-ID needs to be inferred, respectively. Another categorization scheme proposed by Farquhar & Gal 2018) is based on the architecture of the network to distinguish between multi-headed and single-headed architectures. A multi-headed architecture has separate output layer parameters for each task (and requires the task-ID to switch between heads), while in the single-headed architecture, all the parameters are shared between tasks.
Multiple solutions have been proposed to alleviate the problem of catastrophic forgetting, such replaying certain data, adding new parameters, adjusting learning through meta-learning, and finally selective learning of subset of network parameters.
Replay methods (Rolnick et al. 2019) rely on storing previously encountered data or a representation of it in order to combined with new data which leads to an approximation of an i..d. sampling over all of the soquentially encountered data. In these methods, the learning system is retrained over previous data many times. While replay methods can greatly alleviate catastrophic forgetting, they do not scale well with the number of sequential tasks.
3

---

Published in Transactions on Machine Learning Research (11/2022)
Similarly, other solutions such as Rusu et al. 2016) avoids interference between tasks by adding new separate parameters for each new task. These new parameters are trained exclusively on the new task, inheriting in- formation from layers in previous tasks with frozen weights. While this approach allows for high performance in the different task sub-networks, it similarly suffors from poor scaling with the number of tasks.
Several works use meta-learning (Finn et al. 2017 to adjust the learning process for continual learning (Javed & White, 2019 Xu et al. 2020: Harrison et al. (020). Meta-learning aims to optimize the learning of a model over distribution of learning tasks such that learning generalizes to other tasks, usually through few-shot learning test set of tasks. Such approaches can therefore naturally be expanded to continuous learning, as in Javed & White (2019) which learns high-level representations which aim to minimize catastrophic interference between tasks.
The approaches which are closest to this work use selective plasticity to control the modification parameters during weight update. In Kirkpatrick et al. (2017) and Zenke et al. (2017) the modification of paramcters during training controlled by regularization term that penalizes changes to important parameters. In Masse et al. (0018) and Beaulieu et al. 22200) gating signal is fed to the network in order to block some neurons from being activated mask certain parameters to prevent them from being updated during new tasks. Our method uses threshold modulation to control which parts of the network can spike more or less easily which then influences weight. update.
The problem of catastrophic forgetting specifically in SNNs has also been studied. Vaila et al. (2020)
proposes a regularization term that penalizes the update of certain parameters termed the cost per synapse as metric of the importance of every parameter in respect to previous tasks. In Vaila et al. (2020), a part of pretrained using STDP; this spiking part is
the feature extraction section of the network is spiking and then frozen and only the non-spiking classification layers are subject to regularization. Allred & Roy (2020) proposes a single-layer SNN trained in a unsupervised manner using STDP for continual learning on MNIST using dopaminergic excitatory neurons that stimulate other neurons to fire. Our work differs from these works in the use of convolutional SNN trained with backpropagation regulated through spiking threshold modulation.
3 Methods
3.1
Task
The sconario of Continual Learning we adopt in this study is single-hcaded Class Incremental-Learming (van de Ven Tolias 2019 Farquhar & Gal 2018). The model is neither informed of the task ID nor of the occurrence of a change in tasks and all of the parameters of the model are shared between tasks, This is considered the most difficult Continual Learning scenario in van de Ven & Tolias (2019)
In the experiments presented in this paper, a task T. consists of learning to correctly classily single class C from a dataset D. More precisely, in step i the model is trained using a small number & of instances of class C.. Seeing that all the parameters (including the output layer) are shared, the challenge is to learn to classify the current class C, correctly with few examples without forgetting the previously learned classes (C))sei
To achieve this, we modulate the spiking thresholds of the final fully connected (FC) layer of our model using external network that we call the Neuromodulatoor Network (NmN), that has been evolved to mitigate catastrophic forgetting. SNNs equipped with an NmN are referred to as Neuromodulated-SNN (Nm-SNN) and standard SNNs without neuromodulation are referred to as SNN.
Each dataset D is divided into three partitions with different classes: classes in Dpre are used for pre-training classes D Devo are used in the evolutionary optimization of the NmN and classes in Diest are used to test the generalization of our method. Moreover, each dataset D is divided into 80% training instances and 20% testing instances.
4

---

Published in Transactions on Machine Learning Research (11/2022)
Nm
Dense Layer
Spiking Node
Dense Layer
Tanh
Threshold Modulation
SNN
Spiking Block
Spiking Node
Dense Layer
Spiking Node
Dense Layer
Spiking Node
Conv2D
Maxpool
Input
FonneCCdd Layer Neuromodulated Spking Naurons
depth
Figure 1: The Nm-SNN architecture is composed of two spiking networks: a SNN and the NmN. The convolutional SNN performs classification; the first convolutional blocks and the subsequent linear weights are pre-trained on Drre using SGT. and frozen during the CT. scenario. The fully connected spiking network (NmN) takes as imput the spiking activity of the final SNN convolutional layer and outputs the threshold adaplation value for each spiking neuron in the final hidden layer of the SNN during each time-step.
3.2 Neuromodulated SNN
We present the Nm-SNN's architecture FFiuure 1,, which consists of two separate networks, convolutional SNN that lcarns the classification task and the Neuromodulator Network (NmN) which is responsible for adapting the spiking thresholds of the final fully connected (FC) layer of the SNN. We chose to only modulate the last layer as the first layers have been shown have only small contribution the observed forgetting in neural networks (Ramasesh al. 2021
The convolutional SNN, as shown in Figure is composed of spiking convolutional block with a depth that depends on the complexity of the task T which performs a two dimensional convolution operation on the input with an activation using a LIF model for dynamic datasets or IF model for static datasets. This is followed by a Max Pooling layer and a fully connected layer (FC) nife LIF (or IF) neurons with adaptive thresholds. Finally, the output layer has Nelasses spiking neurons.
We use direct input encoding; the first layer neuron's input currents correspond the pixel intensities of the current frame z[t] (for static datasets, x[t] prediction of the network is chosen based on the spike count over the whole simulation time T in the output layer where each output neuron corresponds to a certain predefined class.
is constant throughout the simulation time T). The
The SNN's convolutional block and its subsequent fully connected layer are pre-trained using SGL on classes from Prre and are frozen during the CL scenario. We found that this pro-training step can improve perfor- mance significantly, seeing that we use a very small number of examples for each class. The classes in D Prre neither used in the evolutionary optimization in the testing phase.
5

---

Published in Transactions on Machine Learning Research (11/2022)
The SNN is equipped with threshold Neuromodulator Network (NmN) which is a simple Fully connected Spiking Neural Network with two hidden FC layers of LIF neurons. The NmNN interacts with the SNN by modulating the spiking thresholds of its final FC, both in the training and inference phases (see Figure 1 More precisely, at each time-step t, the NmN takes as input Scone [t] the spikes emitted from the convolutional block at Lime-step 1 and outpuls a vector AV.. [4] that has the same size as the SNN's FC layer, with values in [-1,1] that corresponds the change to apply to the threshold of each neuron in the FC layer. We don't 't allow threshold values to be zero or negative (for biological plausibility purposes), thus the final threshold adaptation is mar(0, Vialt] + [V) +e. After T time-steps (the total simulation time for an imput instance), all the spiking thresholds are reset to base value Vnho (see Algorithm
I
During each task of the CL scenario, the SNN's trainable parameters are optimized using SGL, while the NmN's parameters are frozen and are optimized after the end of the CL scenario using an Evolutionary Strategy detailed in subsection 3.3
Algorithm Neuromodulated training step
Require: NNN, T; Total timesteps, x: Input, y Label, h: Predictions
Reset SNN and NmNN tresholds for == L, 2, ..., T do
Scone [t] < SNN.NcNNn WWW..oowbllll (om_bbock(=(t))
b Get the cony block's spiking activity
AV [1] f NmN (Sconol))
b The adaptation value from the NmN
[t] N...N........ [v
> Adapt the SNN's FC thresholds
h[t] + S.N.SNNN... SNN.FC((coo
end for
loss + calculate loss(h,y) v)
Update SNN's weights using loss
3.3 Neuromodulator Optimization
In order to optimize the NmN, we use a canonical (n/A) Evolutionary Strategy ((n/A) ES) Chrabaszez lt al.) 2018), where and X are the parent population size and the offspring population size, respectively. The population is composed of vectors that correspond to the parameters of NmN. Individuals are generated by adding random noise from N(0 o²) to a distribution center and then updating the center based on fitness information from the individuals.
Another approach to optimize the NmN would be to run CL scenario (i.c. learn sequential tasks containing k examples, with the Nm-SNN) and then backpropagate the error using the surrogate gradient through a computational graph constructed over all the sequential tasks. This is similar to the approach taken in Beaulieu et al. (2020) on ANNs However, we found this approach to have significant memory requirements and could only use it for very small n and k as we calculated gradients over the entire CL scenario. An evolutionary method allows for optimization of the NmN parameters without. the gradient signal at the cost of longer total computation time (cach generation takes a considerable amount of time to compute) and with the critical choice of the fitness function that we optimize for.
We chose a fitness function (Algorithm 2 that encompasses the entirety of the CL scenario, by taking the weighted mean of the classification accuracies acc; over all learned tasks (including the current onc) at every step of the CL scenario. accij denotes the classification accuracy of task j at the CL step (where i 2 j ), thus we take into consideration the accuracy of remembering previous tasks j < i as well as the accuracy of lcarning the current task i at every step of the CL scenario. We evaluate the fitness of each NmN in the population by equipping it to the SNN and carrying out the CL scenario for different task permutations. In Algorithm 2 Tip denotes the 7th task in permutation p, acCijp denotes the classification accuracy of task j evaluated at step of the CL scenario with permutation p.
"odde available

---

Published in Transactions on Machine Learning Research (11/2022)
Algorithm 2 Fitness function
Require: NMN,
T: Tasks, P: Set of permutations
a: Positional coefficients
SNN = reinitialize SNNO
P reinitialize neuron's potentials and thresholds > equip current NmN to SNN iterate through all task ordering permutations
SN N.nmn NmN for p in P do
for i
2,,,,.... for x in Tip do SNN t train Nm, x)
1,2, ..., ni do
is the number of tasks in T
> Algorithm
end for
for j = 1.2,.... do
for x in Tip do
f evaluate(SNNN, Nm, x)
evaluate on previous tasks (Algorithm
acCijp t aijf end for end for end for end for return Mean(acc)
We observed that the order at which the sequential tasks are learned by the network has a significant effect on the severity of catastrophic forgetting; it possible find a task ordering where the interference between different tasks is minimal and thus allows for an easier retention of information. Furthermore, the single specific task ordering does not necessarily generalize to
performance of an Nm-SNN optimized different orderings. For this reason, our fitness function evaluates an individual on multiple permutations of the task ordering: for n tasks, we use permutations obtained by translation to the right.
Finally, we use positional coefficients a to calculate the weighted mean of accuracies to compensate for the differences in the difficulties ween lasks. For example, at CL step i, remembering the immediate previous task significantly easier than remembering the first learned task 1.
4 Results
Our goal was to compare the performance standard SNN and a Nm-SNN, where both networks have the same architecture, same parameter size, same pre-trained frozen weights and the same trainable parameter initializatinn. The only difference lies the spiking thresholds of the final FC layer. For the standard SNN, the spiking thresholds are constant with Va for all neurons, while in the case of the Nm-SNN, the spiking thresholds are adaptive and are modulated by the NmN.
To this end, we use two different type of datasets: neuromorphic dataset DVS128 Gesture Amir et al. 2017) composed of hand gestures captured with an event-based camera, and static image datasets EMNIST -Extended-MNIST letters (Cohen et al. 2017) composed of images of handwritten uppercase and lowercase letters and MNIST (Deng) 2012). For all experiments, the tasks (as defined in subsection 3.1 consist. of learning correctly classify single class. We used SpikingJelly (Fang et al. 2020 open-source deep learning framework for SNNs.
which is Py PyTorch-basod
We also compare with an SNN trained using EWC (Kirkpatrick et al. 2017 state-of-the-art method for continual learning, which we term EWC-SNN. The experimental protocol for using EWC differs from the training method used for the neuromodulated and standard SNNs. When switching between tasks, EWC recomputes gradient information from the previous task to inform weight updates on the next task, requiring knowledge of task change. The neuromodulated and standard SNNs are trained over multiple tasks completely online without any information transfer between tasks or about task change. EWC-SNN also has the same architecture, parameter size and pre-trained frozen weights as the other models, on each task
7

---

Published in Transactions on Machine Learning Research (11/2022)
update calculate the approximation of the fisher information term using all data from the previous task, and for each of the settings presented below, we use simple search method to find the best À value, the coefficient for the EWC regularization term.
We validated our method by three tests of generalization. First, we test the generalization to new instances that were not used during the evolutionary algorithm from the tasks Devo- Second, we test the generaliza- tion to new tasks Dtest, unseen during both the evolution and pre-training. Third, we test the generalization to bigger number of sequential tasks than the used for the evolutionary optimization. For each of these settings, also test the generalization to larger number of gradient steps than the one used during evolution.
For each setting, we also lest on n different permutation of the tasks ordering (as defined in subsection 3.3) The SNN reinitialized for every CL scenario and is optimized using SGL and the neurmodulator is the evolved NmN with the best fitness during evolution (with frozen weights).
4.1 DVS128 Gesture
For DVS128 Gesture, the evolution configuration is as follows: we use the classes from Devo with n 3 sequential tasks, each class is learned through 20 SGD updates where we have k 40 instances of each class and batch size of 2, each instance consists of T 16 frames (after pre-processing of the event data) which is also our simulation time. The evolution of the NmN lasted approximately 600 generations until convergence.
As stated before, for the Devo sctting we use new instances of the classes that were not used during the evolutionary strategy, except for the 50 gradient step setting DVS128 Gesture, where part of the EA instances were reused since the total number of examples of each class in the dataset limited.
For the third test, due to having only 11 different classes, mix Devo and Diest (we alternate tasks from each dataset instead of concatenation for more difficulty) to obtain n = 6 sequential tasks, we note this dataset as Do. The concatenation was done as follows. If Deco fa, b. c} and Drest fe. then De = {a, e, b, f,c,g).
Table 1: Mcan accuracies for DVS128 Gesture
Setting D Devo
SNN (20)
EWC-SNN (20)
Nm-SNN (20)
SNN (50)
EWC-SNN (50)
Nm-SNN (50)
66.89%
83.56%
98.09%
61.98%
72.69%
85.19%
Dtest
75.35%
85.88%
99.02%
64.87%
82.18%
91.72%
Do
45.32%
51.06%
64.65%
39.89%
44.41%
56.37%
Table I shows that the Nm-SNN significantly outperforms the standard SNN in terms of mean accuracy and successfully generalizes to new settings. Nm-SNN also outperforms EWC-SNN both scttings and across datasets. The mean is taken over all accuracies at every stage of the sequential learning (previous classes accuracies are re-caleulated whenever new class is learned, and not just at the end) and all n permutations and different random seeds. We note for the hardest setting of De (50), with n = sequential classes and 50 SGD updates per class, there a sharp drop in mean accuracy due to forgetting, however the performance improvement between the Nm-SNN and standard SNN still similar to the easier settings. Finally, the performances on Dtest are better than Devo for Nm-SNN, even though it has been optimized for Devo. This might be due to the fact that the classes of D have less interference between them as seen in the imporvement of the standard SNN well.
Figure 2 allows for more thorough comparison of the Nm-SNN and SNN accuracy distributions. The figure shows that the mean accuracy for paired samples in both settings is siguificantly higher for Nm-SNN, the paired test value for Drest (20) 13.751 (with p = 2.79 x 10 10 10 ) and 7.12 (with a p 2.43 x 10-06 for Drest (50). remarkable difference between the (20) and (50) distributions that the interquartile range is much larger in the (50). We presume that this is due to the fact that some permutations become significantly harder the number of SGD updates grows
8

---

Published in Transactions on Machine Learning Research (11/2022)
D test (20)
D t.test (50)
100
95
90
90
Accuracy
Accuracy
80
85
80
70
75
70
50
NM-SNN
SNN
Nm-5NN
SNN
(a)
(b)
Figure 2: Distribution of accuracies of Nm-SNN (in blue) and SNN (in orange), for the settings of Drest (20)
Figure 2a and Drest (50) Figure 2b for DVS128 Gesture Dataset
As we previously mentioned in subsection 3.3 the order in which the classes are learned crucial to the performance of the network in terms of retaining previous information; some permutations are favorable to continual learning in such as learning the new task naturally does not cause interference, while some completely overwrite retained information. Table 2a and Table 2b show the difference in accuracy during sequential learning (standard SNN without Nm) for two different permutations. We note that the second permutation Table 2b) is harder than the first one, as the first two tasks were completely forgotten at step 3.
Table Accuracies of two different permutations
(a) Permutation (1, 3)
(b) Permutation (3, 1, 2)
Steps
Task 1
Task 2
Task
Steps
Task 3
Task
Task 2
100%
100%
95.83%
100%
33.33%
100%
12.50%
33.33%
100%
4.17%
0%
100%
To assess this, we count. the total number of individual accuracies that dropped below a cerlain threshold throughout all permutations, stages of sequential learning, and random seeds, to capture the cases where catastrophic forgetting occurs (sce Figure 3), since the mean accuracy only reflects the general performance and does not account for singular cases where the accuracy dropped sharply. In the setting of Prest(20) Figure 3a we see that for Nm-SNN, no individual accuracy is below the threshold of 90%, and only 5% of accuracies are below 95%. On the other hand, the standard SNN clearly suffers from catastrophic forgetting; some classes are completely forgotten, 4% of accuracies are 0% and approximately 22% are below 40%. In the more difficult setting of Dreec(50) Figure 3b we observe heightened forgetting the SNN; 15% of accuracies are 0%. While we see docrease performance for Nm-SNN compared learning with SGD updates, calastrophic forgetting is still mitigated as only 5% below 50% and lask completely forgotten.
Figure 4 shows the evolution of the accuracy of the first learned class with respect to every new SGD update newer classes. At first, both the Nm-SNN and SNN accuracics are at 100%, and as soon as the 50 SGD updates of learning the second task have been completed (al the vertical dotted line) we can see that the SNN falls sharply to 40% while the Nm-SNN is at 90%. An interesting phenomena that we observe at the beginning of the third and final step of the sequential learning the sudden inerease in accuracy of the SNN on the first task even though it is learning the third task. An analysis of the spiking activity of the output layer of the SNN shows that during the first few batches, the output neuron corresponding to the current

---

Published in Transactions on Machine Learning Research (11/2022)
60
SGD updates
SGD
50
No NM
50
No NM
Percanlage
40
Percenlage
30
30
20
10
Accuracy threshold
Accunacy threshold
(a)
(b)
Figure 3: Percentage of individual accuracies (y-axis) that are below a certain threshold (æ-axis) in DVS128 Gesture (lower better). For example, figure (b), the 4th bar from the left indicates that from the total number of accuracies that we evaluated during every permutation and step of CL, 30% of them dropped below 15% for the SNN and only 1% dropped below 15% for the Nm-SNN). The total number of accuracies is calculated for the Diest (20) Figure 3a and Drest (50) Figure 3b throughout all stages of sequential learning, all permutations, and multiple random seeds.
100
80
Accuracy
60
40
Task (SNN)
Task (Nm-SNN)
20
Task (SNN)
Task 3(Nm-SNN)
0
Task start
20
40
60
80
100
batches
Figure 4: Evolution of the accuracy of the first learned class as new batches (SGD updates) of new classes are learned in the Drest (50) setting. The vertical dotted line represents the end of the second sequential task. Finally, the green and red lines represent the accuracy of the third and final learned task.
class does not spike at all and the accuracy 0% at this stage (green dotted line). While learning task 3, the SGD updates tend to increase the synaptic weights of the SNN to encourage spiking. Due to the fact that of these weights are also used for the first task, spike in the output neuron corresponding the first task occurs before, and we can see that once the accuracy of the third task increases (corresponding to a spike in the oulput neuron of the third task), the SNN starts forgetting the first task. However, even though the same phenomenon occurs for Nm-SNN, is mitigated; we only observe a small increase in the first task and we also notice that learning the current task takes much longer (rod dotted linc). This is due to the fact that shared weights are being protected from the update by inhibiting the spiking of their pre-synaptic neurons through threshold modulation.
10

---

Published in Transactions on Machine Learning Research (11/2022)
80
SGD
80
SGD
NM
No NM
No
Perceslagg
50
50
40
40
30
30
10
10
Accuracy threshold
Accunccy threshold
(a)
(b)
Figure 5: Percentage of individual accuracies (y-axis) that are below a. certain threshold (æ-axis) in EMNIST letters, the total number of accuracies is calculated for the Dteat (20) Figure 5a and Drest (50) Figure 5b throughout all stages of sequential learning, all permutations and multiple random seeds.
4.2 EMNIST / MNIST
EMNIST letters is challenging test for method since cach class is composed of both uppercase and lowercase instances of letters, 80 the individual tasks are more difficult to learn and to retain. We also use more sequential tasks n 5 for Devo and Diest since we are not constrained with the total mumber of classes (26 letters) as in DVS128 Gesture. For the pre-training the convolutional layers, we use the first 10 letters as classes noted by Dpre. We ran the evolution for approximately 1000 generations using the next letters. Each class learned through SGD updates with a batch size of and 80 instances.
For the third test, we use completely new dataset, MNIST, and double the number of sequential tasks 10. We note this setting by mmnist,10
Table 3: Mean accuracies for EMNIST/MINIST
Settings
SNN (20)
EWC-SNN (20)
Nm-SNN (20)
SNN (50)
EWC-SNN (50)
Nm-SNN (50)
D Devo
51.63%
61.65%
66.17%
48.92%
52.46%
60.25%
Drest
37.04%
49.53%
52.67%
34.31%
49.12%
41.23%
Dmnisst1
35.66%
54.60%
44.23%
32.13%
42.95%
39.95%
Even though the gap between SNN and Nm-SNN smaller in EMNIST compared to DVS10 Gesture (see Table 3) especially in the hardest setting of Dmmist,10 which tests the generalization to a completely new dataset, our method can effectively slow down forgetting all settings terms of mean accuracy. We note that EWC-SNN outperforms Nm-SNN Dmnist, in both regimes, and on Drest with 50 gradient steps. We interpret from this that EWC is able to adapt to the different task of digit analysis possibly through larger changes in the post-convolutional layer than was possible through surrogate gradient leading in the SNNs. We note that, as EWC is based on gradient updates and not activity modulation, advantages of the two methods could be combined.
Figure 3 also shows the improvement in retaining previous information between the two networks. However, we see that catastrophic forgetting still occurs in both cases (0% accuracics), a closer inspection shows that this occurs only in some permutations for Nm-SNN, while all permulations suffer from this problem for the case of SNN. It is possible that better neuromodulator optimization protocol could solve this issue.
The comparison of the two distributions in Figure 6
indicates the improvement between paired accuracy samples, the I. test value for Drest (20) is 15.545 (with a p value of 2.66 x 10-15 11 and 14.98 (with a value of 2.43 x 10 for Drest (50).
11

---

Published in Transactions on Machine Learning Research (11/2022)
D test (20)
D test (50)
55
42
Accuracy
Accuracy
40
45
38
40
36
35
NM-SNN
SNN
Nm-5NN
SNN
aa)
(b)
Figure 6: Distribution of accuracies of Nm-SNN (in blue) and SNN (in orange), for the settings of Diest (20)
Figure 6a and Drest (50) Figure 6b for EMNIST letters
Nm-SNN
Number of neurons with the spike count
500000
10
420020
300030
200020
10
100030
20
1.00
0.5 Neuromodulltorry Network's output
-0.5
0.0
1.0
500
1000
1500
2000
Total spike count
(a)
(b)
Figure 7: (a) represents the distribution of NmN outputs during a whole CL scenario; the output values are mainly grouped near -1 and 1. (b) represents the distribution of the total spike count of individual neurons; each bar represents the number of neurons that have a total spike count the correspouding bin, for the Nm-SNN (blue upright) and SNN (orange upside down); for example, the first bar shows that there 20 neurons in the Nm-SNN and 14 for the SNN that have a spike count between 0 and 30.
4.3 Analyzing Neuromodulatory Behavior
The motivation behind our work was that the NmN could potentially evolve adapt the firing thresholds of neurons in such way that enables task-specific gating, that the important neurons for previous tasks would not spike for new tasks. However, we did not use any constraints to enforce this behavior during the evolutionary optimization; the only criteria was the performance on the continual learning scenario. In this section, we analyze the behavior of an evolved NmN on the DVS128 Gesture setting (similar results were found on EMNIST/MNIST).
An evolved NmN's output converges to approximately binary threshold adaptation values of f-1, shown in Figure 7a During complete CL scenario, about 63% of the NmN output values are positive,
12

---

Published in Transactions on Machine Learning Research (11/2022)
10
potential threshold
253
500
150
1003
1250
15:0
1750
2000
Nm putput
spilees
259
500
750
1003
1250
1500
1750
2000
0 250 500 750
1003
1250 15:0 1750 2000
Timesteps
Figure 8: The characteristies of an individual neuron in the FC layer for every time-step of full CL scenario, the vertical black lines separates between the three different sequential tasks. The first row represents the membrane potential and the firing threshold value, the second row represents the spikes (blue vertical bar) and the NmN output for the neuron's 's threshold adaption, and the third bar represents the spikes of the same neuron while not being neuromodulated.
which indicates that most outputs tend to restrain neurons from spiking which is consistent with the gating we expected. However, about 37% of outputs are negative, meaning that at some times neurons are encouraged to spike. Overall, if we compare the Nm-SNN to a standard SNN that's been through the same CL scenario, we find that the Nm-SNN has about 54% less spikes than the SNN. Thus, on the network level, the behavior of the NmN is more inhibitory.
To assess how the network's spiking activity is distributed between neurons of the neuromodulated FC layer, we can look at the distribution of the total spike count of individual neurons. Figure 7b]shows this distribution for both Nm-SNN (in blue) and standard SNN (in orange), and demonstrates that neuromodulation allows for more regular distribution of spiking activity. The Nm-SNN's distribution shows that most neurons are catcentrated on the lower total spike count region (less than 1000 spikes), while only few neurons spike more than 1000 times during the CL scenario. In comparison, the standard SNN's distribution doesn't 't show this regularity, with an important difference at the high total spike count regions. Without neuromodulation. there is a higher number of neurons that spike more than 1000 times. This diffcrence could be explained by the out puts of the NmN, as the positive values restrain the most active neurons.
On the network level, the neuromodulator's behavior tends to reduce the overall spiking activity while evenly distributing it over the FC layer's neurons, these properties suitable to mitigate catastrophic forgetting. If the spiking activity is concentrated in a few neurons, their corresponding weights would be updated each time they spike; a sparse and evenly distributed spiking activity could potentially allow for a separation of network parameters that are used for each task.
To analyze the NmN on neuron level, we pick the same individual neuron and compare its characteristics while nouromodulated and while having a constant threshold. Figure & represents the membrane potential and threshold values of the neuron (Ist row), its spikes while neuromodulated and its correspondent NmN output value with which its threshold was adapted during that time-step (2nd row) and finally the spikes of the neuron while not being neuromodulated (3rd row). The vertical lines in black separates between the three different sequential tasks in this CL scenario.
In the first row, the periodicity of the threshold value due to the fact that after each input instance (16 timc-steps) the threshold reset to its base value of The second row shows that the NmN outputs (red dots) are consistent with the previous network level analysis as they are mostly concentrated near the values 1-1,1} with some exceptions that fall in between. Moreover, the figure shows that for this neuron, the NmN
13

---

Published in Transactions on Machine Learning Research (11/2022)
exhibits task-specific behavior; we see that for the first task, most of the NmN outputs are negative thus encouraging the neuron to spike; while for the remaining two, most outpuis are positive and restrain the neuron from spiking. This can also be seen by comparing the spike trains in the second and third row; we have 19% more spikes with neuromodulation for the first task and 23% and 88% less on the second and third tasks respectively. The behavior of the NmN on the final two lasks is what we were expecting: gating the activity can protect weights from being updated. This can be seen in particular for the third fask where the extensive spiking activity of the neuron without neuromodulation leads to lot of updates of its correspondent weights and ultimately to catastrophic forgetting. Furthermore, the NmN also encourages spiking in the first task. This shows that this neuron is mostly important for the first task and is restrained for the others. However, the behavior of the NmN is different for other neurons, where some are restrained on the first lask and encouraged on others, while some are completely restrained on all tasks and so on.
While the overall behavior is task-specifie, the NmN outputs in Figure 8 for the first and second task aren't 't exclusively negative or positive respectively: the NmN behaves differently for cach input frame in every time-step. This could be explained by the fact that even though the input tasks are different, individual frames from the different tasks (for hand gestures for example) could have resemblance, and thus the NmN treats them differently. This frame-specific behavior shows that using only the CL scenario performance as metric during the evolutionary optimization can lead to richer and more flexible behaviors compared to imposing constraints or using handerafted gating mechanisms.
5 Discussion
The study of SNNs and their properties could potentially lead to the development of faster and energy efficient neural networks that are less prone to the fundamental problems that plague standard ANNs. In this work, we demonstrated that threshold modulation could leverage an intrinsic property of SNNs to mitigate catastrophic forgetting in a continual learning setting where all network weights are shared and no lask information is available, and where our neuromodulator has access only to local input, which is the previous convolutional layer's spiking activity. We used an evolutionary strategy to optimize our NmN where the criteria of selection is the performance on the continual learning scenario. This has lead to an NmN with rich task-specific and frame-specific behavior; which is also generalizable to new tasks not seen during pre-training and evolutionary optimization.
This preliminary study on threshold modulation in SNNs has limitations which can be addressed through future work. The optimization of the NmN was performed with an evolutionary process, which could be replaced by more samplo-efficient methods such as gradient descent. Further study of the optimized NmNs could also be used to develop generic threshold modulation policies which do not require further optimization.
Although method was tested on simple networks and small number of sequential taks, it opens the door for improved methods that leverage the intrinsic propertics of SNNs, thus enabling the development of fast and energy efficient SNNs deployed on neuromorphic hardware that could learn continually in real world scenarios.
References
Jason M. Allred and Kaushik Roy. Controlled forgetting: 'arrgted stimulation and dopaminergic plasticity modulation for unsupervised lifelong learning in spiking neural networks. Fronliers in Neuroscience, 14, 2020. ISSN 1662-453X. doi: 10.3389/fuinns 202in.2020.00007. URL htttp::/www..ronttie www./oooii.e...........oo
10. 00.3389/fnins.. wtt.:fiiinnn.......mmm2mmm 2020-00007
Arnon Amir, Brian Taba, David Berg, Timothy Melano, JefTrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Essor, Tobi Delbruck, Myron Flickner, and Dharmendra Modha. A low power, fully event-based gesture recognition system. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7388-7397, 7397, 2017. doi:
14

---

Published in Transactions on Machine Learning Research (11/2022)
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. In ECAT, pp. 992-1001, 2020. URL https://doi org/10.330 3233/ FAIA200198
Patryk Chrabaszez, Llya. Loshchilov, and Frank Hutter. Back to basics: Benchmarking canonical evolution strategies for playing atari. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pp. 1419 1426. International Joint Conferences on Artificial Intelligence Organization, 2018.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: extension of MNIST to handwritten letters. CoRR,
2017. URL http://arxiv. wxg/abb/1702..5373
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2944)114 2012.
Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors. Spikingjelly.
spips//j//iiinnilii///./ 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21056 21069. Curran Associates, Inc., 2021. URL https://proceeding..nt heeripss.ccppper/20011 .df
S. Farquhar and Y. Gal. Differentially private continual learning. In Privacy in Machine Learning and Artificial Intelligence workshop, ICML, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126 1135. PMLR, 2017.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3 URL https:
(4):128-133) 135, 1999. ISSN 1364-6613. doi: http:///daaooo///oooo0////666660
Wulfram Gerstner, Werner M. Kistler, Richard Naud, and Liam Paninski. Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition. Cambridge University Press, 2014. doi: 10.1017/ C1009781107476155
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. Advances in neural information processing systems, 33:17571-17581 2020.
Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwingt Demis Hassabis, Clandia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the Nalional Academy of Sciences, 114(13):3521 3526, 2017.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):1659 -6711 1997.
Nicolas Y. Masse, Gregory D. Grant, and David J. Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Procerdings of the National Academy of Sciences, 115(44):E10467 E10475, 2018. doi: 10. 10.1073/pmas 1803839118
URL httpp://wwet pnas org/doi/abs/
Michael McCloskey and Ncal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pp. 109-165. Academic Press, 1989.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-bsse optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 63, 2019. doi: 10.1109/MSP 201....9....1..915555
Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Representations, 2021. URL
https:://pennreview mtt:/foruurrtc---ayydddo0SC
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buu, E. Fox, and R. Garneu (ods.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaune Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/ 1606.0461, 2016. URL
Ruthvik Vaila, John Chiasson, and Vishal Saxena. Continuous learning in a single-incrementalltass scenario with spike features. In International Conference on Neuromorphic Systems 2020, ICONS 2020, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 0781450388511. 3407213. URL httpp::///i..rrg//// http//1..11553401111 0101011111///0777733071111
doi: 10.1145/3407197.
Gido M. van de Ven and Andreas S. Tolias. Three continual learning scenarios and a case for generative replay, 2019. URL https::/ htt.:///w.....t..oooooooooooanna
Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual learning of control primitives: Skill discovery via reset-games. Advances in Neural Information Processing Systems, 33:4999-5010,, 2020.
Friodemann Zenke and Tim P Vogels. The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks. Neural Compulation, pp. 1 1-27, jan 2021. ISSN 0899-7667. doi: 10.1162 10.1162/neco 01367. URL {eco(_)a(_J013e)
1162/
Friodemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Pro- ceediings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3987-3995 3995. PMLR, 06-11 Aug 2017.
Friedemann Zenke, Sander M. Bohté, Claudia Clopath, Tulia M. Comsa, Julian Gölız, Wolfgang Maass, Timothée Masquelier, Richard Naud, Emre O. Neftei, Mihai A. Petrovici, Franz Scherr, and Dan F.M. Goodman. Visualizing joint future of neuroscience and neuromorphic engineering. Newron, 109(4):571 575, feb 2021. ISSN 08966273. doi: 10.1016/j nttp:/jj.nuro...0000 2021. 01.009. URL http:: cam//eerceeee/////09967222100990
elsevier.
16


Reviews:
[{"review_id": "8n5WbdwSQ5", "reviewer": "Reviewer_yN44", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. Gating the weight update via controlling the voltage threshold is natural for the SNN. The proposed method doesn't require any additional changes to the spiking neuron model and has the potential to be deployed on neuromorphic processors.\n\n2. The proposed method doesn't require any class-specific information when changing the class to be trained. This makes it has the potential to be used in real-world tasks when the task or class transition is not as transparent as the artificial dataset used in the paper.\n\n3. Experiments in the paper demonstrate the effectiveness of the proposed method in mitigating the catastrophic forgetting problem during continual learning. The Nm-SNN achieved much higher accuracy than the regular SNN in both DVS-Gesture and EMNIST datasets. In addition, Figure 4 of the paper shows the Nm-SNN forgets much slower than the regular SNN.\n\n4. The analysis of the neuromodulatory behavior within the Nm-SNN (Section 4.3) shows changing the voltage threshold during learning can effectively prevent the classification overfit to a specific feature. By solving this overfitting problem in the output layer, the proposed method can prevent aggressive weight updates when learning new classes. From analyzing the results in Figure 8, the reviewer thinks this is what truly happens inside the Nm-SNN and how it can mitigate the catastrophic forgetting problem.\n\nWeakness\n\n1. The evolutionary strategy used to optimize the NmN is very costly. The algorithm needs to iterate all tasks in the evolution dataset for every individual group of randomly generated parameters. This is not generalizable for large datasets and large NmN. First, increasing the number of tasks and data for optimization will directly increase the iteration times. Second, increasing the size of the NmN for more complex tasks will exponentially increase the search space for the evolution algorithm.\n\n2. The backprop-based approach to optimize the NmN may not be as memory costly as the paper claims. Although the training needs to iterate all tasks and data in the dataset, intermediate gradient summations can be used to reduce the memory cost. Unlike the evolution algorithm, the gradient-based method may give a more reliable optimization direction requiring fewer update epochs.\n\n3. The proposed method only solved the catastrophic forgetting problem in a linear classifier. Since all parameters except the output layer are fixed during continual learning, what the experiment truly trained is a linear classifier using the features generated by the pre-trained SNN. However, learning on a linear classifier has limited ability on complex tasks. This may explain why the SNN has poor performance on the EMNIST.\n\n4. Building upon the previous point, the proposed method may not be able to work for training parameters in a multi-layered network. In the current setting, the parameters in the output layer for different classes have no interference. During training, forgetting may happen when the data of two classes share similar features. For example, the feature represented by the neuron in Figure 8 of the paper is shared for all three classes. Thus, overstimulation in one class may trigger forgetting in other learned classes. However, a multi-layered network will have a different mechanism for forgetting during continual learning. There will be much more interference in the parameters for different classes. Thus, good results on a linear classifier cannot guarantee the method works on a multi-layered network.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a voltage threshold modulation method to mitigate the catastrophic forgetting problem in SNN during continual learning. The reviewer thinks the paper has three major contributions:\n\n1. The authors observed the gating effect of the voltage threshold on weight update when training SNN using backprop. By changing the threshold, spiking neurons can have different activation behaviors. This directly influences the weight update magnitude for synapses to the post-synaptic neurons.\n\n2. The paper introduced a separate spiking network (NmN) trained by an evolutionary algorithm to control the voltage threshold of neurons in the last fully-connected layer of the SNN. By modulating the thresholds, the network regulates the update of output synaptic weights in the continual learning scenario.\n\n3. The authors performed experiments on evet-based and static datasets. The proposed method generates better performance than a regular SNN and prevents accuracy drop on old classes when a new class is learned.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Important Changes:\n\n1. The authors need to include clear descriptions of partitions used in the dataset. From the manuscript, we know D_evo is used to perform evolutionary optimization (EA). However, the results in Table 1 and Table 3 also included D_evo for continual learning (CL). Is D_evo divided into EA and CL datasets? If so, how much data is used for evolutionary optimization and generating the results in the tables? Since the regular SNN doesn't need evolutionary optimization, is the SNN use the same group of data in D_evo for CL?\n\n2. In addition, how does the experiment divide the data of each class when performing continual learning? How much data is used to train the new class and test the trained model? Since the D_evo has much fewer data per class for continual learning than D_test, how does the experiment deal with this difference when combining them for D_6 in Table 1?\n\n3. The authors need to clarify experiment details related to training and testing the Nm-SNN during continual learning. Is the number of neurons in the output layer fixed and equal to the number of classes? Or new output neurons are added when new classes are introduced in the continual learning? Is the NmN used only during training the SNN? Or both in training and testing?\n\n4. Are the hyperparameters for the regular SNN optimized for best performance? We know hyperparameters like voltage thresholds can significantly influence the performance of SNN. The authors need to show they have optimized the performance of the SNN in the tasks.\n\n5. Since a large amount of the NmN output is -1, as shown in Figure 7(a), will the network prevent the voltage threshold go below zero? If not, how will the neuron behave when the voltage threshold is negative?\n\n6. The authors give an overclaim that the proposed method performs better for dynamic datasets in the last paragraph of Section 4.3. The accuracy numbers between the DVS-Gesture and the EMNIST datasets are not comparable. These two datasets have different sizes and numbers of classes. The difference in accuracies doesn't give enough evidence to support the claim in the paper.\n\nOther Changes:\n\n1. The authors need to add detailed explanations on the relationship between voltage threshold and weight update, preferably using math expressions. This information is needed to complete the background of the proposed method and for readers who are not that familiar with the concept.\n\n2. Figures 3 and 5 show the accuracy differences between Nm-SNN and the regular SNN. However, it's unclear from the figure how the accuracy is evolved in the time dimension when new tasks are learned. Adding an additional time dimension in both figures will be better to show how the accuracies change after learning new tasks.\n\n3. The quality of Figure 7 is low. First, the word \"count\" is not complete in Figure 7(a). Second, the number of neurons for SNN should be positive on the y-axis of Figure 7(b).", "broader_impact_concerns": "There is no concern about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S5wDsD5nqn", "reviewer": "Reviewer_NavC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## General opinion\n\nThe paper is well written and the training technique is quite clearly defined although there are a few inconsistencies.\n\nThe experiments are clearly described and the results seem to make sense but I find two major issues which makes the results incomparable with existing literature and hard to judge in terms of performance.\n\n## Two major issues\n\n(1) I do not understand what is the rationale for using spiking networks in combination with continual learning techniques.\n\nMy first problem is that there is no clear definition of what is a spiking neuron for the authors. If the motivation is to derive algorithm for neurmorphic hardware then it is typically assumed that all communications should be binary. However the tanh modulation requires communication from NmN to SNN which is not binary. If the choice of using spiking neurons is motivated by biology it is fine, but more explicit parallels with biological data would be needed to a more convincing model of biology.\n\nIt is not clear to me what is motivation to combine spiking neural networks with continual learning techniques. Since the authors rely on straight through estimators (called surrogate gradients in the context of spiking networks), the discontinuity are ignored and the spiking networks can be optimized as regular artificial neural networks. However I do not see why spiking networks would be more or less affected by catastrophic forgetting in comparison with non-spiking artificial neural networks. In this sense, I think that having spikes in this paper does not make a crucial technical difference and the method could have been compared to other methods which are not necessarily applied to spiking networks.\n\n(2) This lead me to the second major issue: there are no comparison with other continual learning techniques on the presented benchmarks.\n\nUnfortunately, the method is not compared to other continual learning techniques despite the existing literature which is well reviewed and cited at the beginning of the paper. Since the task is custom for this paper I cannot judge whether the performance is very good although it looks like it was done seriously.\n\nAlso comparisons with other control algorithm using the same implementation could have been performed to provide more data points to judge the quality of the algorithm. I provide some suggestions in the following section of the review.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper describes a method for continual learning. In an SNN architecture with two hidden layers, a separate neuromodulatory network is trained via evolutionary strategies to modulate the threshold of the second layer given the output of the first layer. The fitness function for the algorithm is the mean accuracy over a continual learning benchmark with permuted task order. In simulation on a DVS gesture and (E)MNIST datasets, this strategy reduces the impact of catastrophic forgetting.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "## Controls and baselines\n\n- To give a better idea on the success of the method it would be great to compute the performance of control models. One simple control model would be to optimize the NmN network on the same loss function without evolutionary strategy but with gradient descent as for the rest of the SNN. This control makes sense to verify that it is not the architecture of the network which makes all the difference in performance.\n\n- It would be great to implement at least one classical continual learning algorithm like EWC [ref. Kirkpatrick et al. 2017] or any other appropriate algorithm as a comparison.\n\n- One thing that surprises me, fitness function requires many training step and seems rather slow. This is rather surprising because training a neural network with evolutionary strategy benefits from fast fitness evaluation. So I wonder whether there are any simpler version of a similar algorithm which would work better with faster fitness computation. \n\n## Clarification\n\n- Equation (9) and (10) $f'$ is used as if it was approximately equal to $\\theta'$ but $f$ had a different definition which was given in equation (2) and (3). There seems to be some inconsistency here.\n\n- Equation (9) and (10), I am surprised to see a pseudo derivative which is equal to the sigmoid function. Usually one has: $\\theta'(x) \\approx \\sigma '(x) = \\sigma(x) (1 - \\sigma(x))$\n\n- Unfortunately I could not read the performance in Figure 3 and 5. The sentence describing how to read these plots was a bit cryptic to me.", "broader_impact_concerns": "No specific concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m4QU5jCNEV", "reviewer": "Reviewer_nxFu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: I like the idea of using a new network to dynamically control the threshold. It seems to be another way to apply inhibition. The idea of using the evolutionary method rather than BP to optimize the neuromodulator network is also interesting. Overall, the methodology is good and the results are promising.\n\nWeakness: This paper does not introduce the existing method for the catastrophic forgetting problems on SNN. It is hard to judge how good the proposed method is.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a neuromodulator network to dynamic control the threshold of spike neurons or another SNN for the catastrophic forgetting problem. The results show the neuromodulator network can significantly improve performance compared to SNN when learning the multiple tasks.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Suggestions:\n1. In the background introduction, it would be better to briefly review existing methods of catastrophic forgetting problems on SNNs.\n2. Insteading of using EMNIST, I think it's better to use the neuromorphic version of these image datasets like spike-based EMNIST, FMNIST, or N-MNIST.\n3. As mentioned above, can the authors compare performance with existing papers?\n\nQuestions:\n1. In the NM of figure 1, the outputs of neurons are passed to tanh function. Can the authors explain the motivation for adding this function?\n2. What is the preprocessing step of DVS-128? Are methods like reducing dimension and shrinking time window used?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Given the evaluations of the reviewers, I can propose acceptance of the manuscript, given that some minor revisions are made.
The reviewers raised the following point:
- The proposed method to mitigate catastrophic forgetting was restricted to the final layer, potentially limiting the performance on some data sets and the scalability of the method. The authors extended the analysis to hidden layers in the revision, but the results did not significantly improve. Hence, the claim that the approach could be scaled to larger networks and more complex problems is not well supported by the shown evidence.
The authors should discuss this point in the Discussion and adjust their claim on scalability made there accordingly.
Decision: Accept (poster)