Training Spiking Neural Networks with Local Tandem Learning
Authors: Qu Yang
Jibin Wu
Malu Zhang
Yansong Chua
Xinchao Wang
Haizhou Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Spiking Neural Network (SNN)
Teacher-Student Learning
On-chip Learning
Local Learning
Neuromorphic Computing

Abstract: Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient over their predecessors. However, there is a lack of an efficient and generalized training method for deep SNNs, especially for deployment on analog computing substrates. In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL). The LTL rule follows the teacher-student learning approach by mimicking the intermediate feature representations of a pre-trained ANN. By decoupling the learning of network layers and leveraging highly informative supervisor signals, we demonstrate rapid network convergence within five training epochs on the CIFAR-10 dataset while having low computational complexity. Our experimental results have also shown that the SNNs thus trained can achieve comparable accuracies to their teacher ANNs on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. Moreover, the proposed LTL rule is hardware friendly. It can be easily implemented on-chip to perform fast parameter calibration and provide robustness against the notorious device non-ideality issues. It, therefore, opens up a myriad of opportunities for training and deployment of SNN on ultra-low-power mixed-signal neuromorphic computing chips.

Training Spiking Neural Networks with Local Tandem Learning
Qu Yang', Jibin W Malu Zhang?, Yansong Chua*, Xinchao Wang', Haizhou L
National University of Singapore
2The Hong Kong Polytechnic University
3 University of Flectronic Science and Technology of China
CChina Nanhu Academy of Electronics and Information Technology Thhe Chinese University of Hong Kong, Shenzhen, China 6 Kriston AI, Xiamen, China
Abstract
Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient over their predecessors. However, there is lack of an efficient and generalized training method for deep SNNs, especially for deployment on analog computing substrates. In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL). The LTL rule follows the teacher- student learning approach by mimicking the intermediate feature representations pre-trained ANN. By decoupling the learning of network layers and leveraging highly informative supervisor signals, we demonstrate rapid network convergence within five training epochs the CIFAR-10 10 dataset while having low computa- tional complexity. Our experimental results have also shown that the SNNs thus trained can achieve comparable accuracies to their teacher ANNs CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. Moreover, the proposed LTL rule is hardware friendly. It can be easily implemented on-chip perform fast parameter calibration and provide robustness against the notorious device non-ideality issues. It, therefore, opens up myriad of opportunities for training and deployment of SNN on ultra-low-power mixed-signal neuromorphic computing chips.
Introduction
Over the last decade, artificial neural networks (ANNs) have improved the perceptual and cognitive capabilities of machines by leaps and bounds, and have become the de-facto standard for many pattern recognition tasks including computer vision [30, 52, 53, 62, 35], speech processing [60, 39], language understanding [3], and robotics [51]. Despite their superior performance, ANNs are computationally expensive to deployed on ubiquitous mobile and edge computing devices due to high memory and computation requirements.
Spiking neural networks (SNNs), the third-generation artificial neural networks, have gained growing research attention due to their greater biological plausibility and potential to realize ultra-low- power computation observed in biological neural networks. Leveraging the sparse, spike-driven computation and fine-grain parallelism, the fully digital neuromorphic computing (NC) chips like TrueNorth [2], Loihi [11], and Tianjic [42]. that support the efficient inference of SNNs, have indeed demonstrated orders of magnitude improved power efficiency over GPU-based AI solutions.
Moreover, the emerging situ mixed- signal NC chips [54, 47], enabled by nascent non-volatile technologies, can further boost the hardware efficiency by large margin over the aforementioned
digital chips.
"Corresponding Author: jibin. blbin.wu@@ppr @oittwuw@olyyy....hh
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Despite remarkable progress in neuromorphic hardware development, how to efficiently and effec- tively train the core computational model, spiking neural network, remains a challenging research topic. It, therefore, impedes the development of efficient neuromorphic training chips as well as the wide adoption of neuromorphic solutions in mainstream AI applications. The existing train- ing algorithms for deep SNNs can be grouped into two categories: ANN-to-SNN conversion and gradient-based direct training.
For ANN-10-SNN conversion methods, they propose to reuse network weights from more easily trainable ANNs. This can be viewed as a specific example of Teacher-Student (T-S) learning that transfers the knowledge from teacher ANN to student SNN in the form of network weights. By properly determining the neuronal firing threshold and initial membrane potentials for SNNs, recent studies show that the activation values of ANNs can be well approximated with the firing rate of spiking neurons, achieving near-lossless network conversion on number of challenging Al benchmarks [7, 49, 14, 22, 21, 46, 34, 12, 4, 5, 63]. Nevertheless, these network conversion methods are developed solely based on the non-leaky integrate-and and-fire (IF) neuron model and typically require large time window as to reach reliable firing rate approximation. It is, therefore, not straightforward and efficient to deploy these converted SNNs onto the existing neuromorphic chips.
In another vein of research, the gradient-based direct training methods explicitly model each spiking neuron as a self-recurrent neural network and leverage the canonical Backpropagation Through Time (BPTT) algorithm to optimize the network parameters. The non non-differentiable spiking activation function is typically circumvented with continuous surrogate gradient (SG) functions during error backpropagation [50, 58, 59, 38, 8, 15, 45, 61]. Despite their compatibility with event-based inputs and different spiking neuron models, they are computational and memory inefficient to operate in practice. Moreover, the gradient approximation error introduced by these SG functions tends to accumulate over layers, causing significant performance degradation in the face of the deep network structure and short time window [57].
In general, SNN learning algorithms can be categorized into off-chip learning [16, 64] and on- chip learning [9, 41, 43]. Almost all of the direct SNN training methods discussed above belong to the off-chip learning category. Due to the lack of effective ways exploit the high level of sparsity in spiking activities and the requirement to store non-local information for credit assignment, these off-chip methods exhibit very low training efficiency. Moreover, due to notorious device non-ideality problems [6], the actual network dynamics will deviate from the off-chip simulated ones, causing the accuracy of off-chip trained SNNs degrades significantly when deployed onto the analog computing substrates [44, 24, 37, 1]. To address these problems, recent work proposes on-chip learning algorithms in the form of local Hebbian learning [11, 1, 41, 28] and approximation of gradient-based learning [32, 40, 10, 19], while the effectiveness of these algorithms had only been demonstrated on simple benchmarks, such as MNIST and N-MNIST datasets.
To address the aforementioned problems in SNNs training and hardware deployment, we put forward a generalized SNN learning rule this paper, which we referred to as the Local Tandem Learning (LTL.) rule. The L.T. rule takes the best of both ANN-to-SNN conversion and gradient-based training methods. On one hand, it makes good use of highly effective intermediate feature representations of ANNs to supervise the training of SNNs. By doing so, we show that it can achieve rapid network convergence within five training epochs on the CIFAR-10 dataset with low computational complexity. On the other hand, the LTL rule adopts the gradient-based approach to perform knowledge transfer, which can support different neuron models and achieve rapid pattern recognition. By propagating gradient information locally within layer, it can also alleviate the compounding gradient approximation errors of the SG method and lead to near-lossless knowledge transfer on CIFAR- 10, CIFAR-100, and Tiny ImageNet datasets. Moreover, the LTL rule is designed to be hardware friendly, which can perform efficient on-chip learning using only local information. Under this on on-chip setting, demonstrate that the LTL rule is capable of addressing the notorious device non-ideality issues of analog computing substrates, including device mismatch, quantization noise, thermal noise, and neuron silencing.
2

---

(student)
(teacher)
Layer
Layar
Layer
Lowe
Layer
(2)
2
9a
Layer
Layer
8
8
g²
[0]
Layer
Layer
S 10]
[
[3]
S [0]
| [1]
5 [2]
Data
T=2
T=0
T=1
T=2
(a)
Host Computer
NC Chips
(8* (,'"))
Data
Main Processor
Memory
Cache
S
S
Synapse
Spike
Layer
Borcarij
Reset
S
Modules
Control Signal
Layer
V.
g
Layer
g
Emrr
Acoumulatio
Spkke
(d)
Figure 1: Illustration of the proposed LTL rule and its on-chip implementation. (a) The LTL rule follows the teacher-student learning approach, whereby the SNN tries mimic the feature representation of pre-trained ANN through local loss functions. (b) Computational graph of the offline LTL rule. (c) Computational graph of the online LTL rule. (d) Functional block diagram the proposed on-chip implementation, where the host computer transfers the control signal and training data (input spike train, layerwise targets) NC chips. The proposed SNN on-chip learning circuit consists of two parts: spiking neuron (green) and learning circuits (red).
2 Methods
2.1 Spiking Neuron Model
To demonstrate the proposed LTL rule is compatible with different spiking neuron models, we base study both non-leaky integrate-ane and-fire (IF) [48] and leaky integrate- tinggate-and--ae and-fire (LIF) neuron models [18], whose neuronal dynamics can described by the following discrete-ting formulation:
Utiti oU. 1] [
US 1]
with
-t== = D
6
where U][[] and [[[ refer to the subthreshold membrane potential of and input current neuron i at the membrane potential decaying constant, wherein Fm is
layer respectively. a = exp(-dt/ dd//mm)))))) Fm. the membrane time constant and dt is the simulation time step. For IF neuron, takes value of 1. 0 denotes the neuronal firing threshold. /-1 represents the connection weight from neuron, j of the preceding layer and denotes the constant injecting current to neuron i. S - 1] indicates the occurrence of an output spike from neuron i at time step 1, which is determined according to the spiking activation function as per
f1,
if x x otherwise
$/Ms = (U-14 8) with +(x)
3

---

2.2 Local Tandem Learning
As illustrated in Figure 1(a), the LTL rule follows the T-S learning approach [20], whereby the intermediate feature representation of a pre-trained ANN is transferred to SNN through layer-wise local loss functions. In contrast the ANN-to-SNN conversion methods, we establish the feature representation equivalence at the neuron level rather than at the synapse level, which provides the flexibility for choosing any neuron model to be used in the SNN. On the other hand, with the proposed spatially local loss function, we simplify the spatial-temporal credit assignment required in the end- to-end direct training methods, which can dramatically improve the network convergence speed and meanwhile reduce the computational complexity. In the following, we introduce two versions of the LTL rule, offline and online, depending on whether the temporal locality constraint is imposed.
Offline Learning Following the T-S learning approach, we consider the intermediate feature repre- sentation of a pre-trained ANN as the knowledge and train an SNN to reproduce an equivalent feature representation via layer-wise loss functions. In particular, we establish an equivalence between the normalized activation values of an ANN and the global average firing rates of an SNN. To reduce the discrepancy between these two quantities, we adopt the mean square error (MSE) loss function and apply it separately for each layer. Thus, for any layer 1, the local loss function C is defined as follows
C "V.V'' (wl) -
g'
[T Te
(4)
Inorm
where ÿ' is the output of ANN layer L. Inorm is normalization constant that takes the value of 99th or 99.9th percentile across all g This can alleviate the effect of outliers compared to using the maximum activation value [48]. Tw is the time window size. ['[Tw] = S is total spike count.
As the computational graph shown in Figure 1(b), we adopt the BPTT algorithm to resolve the temporal credit assignment problem, and the weight gradients can be derived
ac
To oc ou!t
ac' ou! or
T
ac
-
-
(5)
Owl
ail owly
11
ouly
au!
with
3c
+
1
8:10 sU!
asid
if < Tw
00 |+1
(6)
[
os [Tol
if Tw
20 T
where
8S [++] 00 + [[T]]
ac
if < Tw
lise
- - S(
if t= Tw
T
Ynorm
To resolve the problem of the non-differentiable spiking activation function, we apply the surrogate gradient method, i.e., O'(x) x 8' ((xx). Specifically, we adopt the boxcar function for 8' (x) that supports convenient and efficient on-chip implementation.
[t aui
o (c - () = sign (1u(t) 01 ()
P)
(8)
where p controls the permissible range of membrane potentials that allow gradients to pass through, and we tune this hyperparameter separately for each dataset. By substituting Eq. (8) into Eqs. (6) and (7), we can yield the ultimate form of weight gradients, and we can update the weights according to the stochastic gradient descent method or its adaptive variants. See Supplementary Materials Section A.1 for a more detailed derivation of the gradients to weight and bias terms.
Online Learning The offline LTL rule requires storing intermediate synaptic and membrane state variables so as to be used during error backpropagation, which prohibited for on-chip learning where memory resources are limited. To address this problem, we introduce an online LTL rule, whose loss function is designed to be both spatially and temporally local. To achieve the temporal locality, we use the moving average firing rate, which can be calculated at each time step, to replace the global firing rate used in Eq. (4). hence yields the following local loss function
C'
Unorm
2

---

Compared to the offline version, the gradient update is much simpler now:
ac (t] ac as [t] ohlid = c (1
aulit) 5 '[t
1]
(10)
audt | [uf auly
where [' can be directly computed from Eq. (9):
ac'(t) [t] d| = [ = t (
2
| t E
(11)
Ynorm
The computational graph of the online LTL rule is provided in Figure 1(c). It is worth noting that the first few time steps of the firing rate calculation are relatively noisy. Nevertheless, this issue can be easily addressed by treating the first few steps as the warm-up period, during which the parameter updates are not allowed (see Supplementary Materials Section C for a study on the effect of the warm-up period), By doing so, it can also reduce the overall training cost. As will be discussed in Sections 3.1 and 3.4, this online version can significantly reduce the computational complexity, while achieving a comparable test accuracy to that of the offline version.
On-chip Implementation To allow convenient and efficient on-chip implementation of the proposed online LTL rule, we carefully designed the learning circuits as illustrated in Figure (d). The output spike count C' is updated at the spike accumulator, and it is compared to the local target g' following the layer-wise loss function defined in Eq. (9). This error term is further feedback to the neuron to update the synaptic parameters. Note that the synaptic updates are gated by sign(-) and bozzar(-) functions, which can significantly reduce the overall number of parameter updates.
We would like to highlight that the proposed LTL learning rule is more hardware-friendle than the recently introduced hardware-in-the-logg (HIL)) training approach [10, 19]. The HIL training approaches require two-way information communication, that is, (1) reading intermediate neuronal states from the NC chip the host computer to perform off-chip training and (2) writing the updated weights from the host computer to the NC chip. Given the sequential nature of these two processes and the high implementation cost for reading neuronal states (e.g.. requiring implement costly mmalog-to-dighin converters for analog spiking neurons), HIL training approaches are expensive to deploy in practice.
In contrast, the LTL training rule can be implemented efficiently on-chip by simultaneously extracting the layerwise targets from ANNs, running on the host computer, for data batch i + 1 and performing on-chip SNN training for data batch i. This is similar to conventional ANN training. where the data preprocessing of the next data batch is performed on the CPU and meanwhile the current data batch is used for ANN training on the GPU. The only difference is that the input data is preprocessed by the pre-trained ANN to extract the targets for intermediate layers for the proposed LTL rule. Given the inference of ANN can be performed in parallel on the host computer, the overall training time bottlenecked by the NC chip that operates sequential mode, where only one sample is been processed at time. Therefore, our method has much lower hardware and time complexity.
3 Experiments
In this section, we evaluate the effectiveness of the proposed LTL rule on the image classification task with CIFAR- 10 [29], CIFAR- [29], and Tiny ImageNet [55] datasets. We perform a comprehensive study to demonstrate its superiority in: 1. accurate, rapid, and efficient pattern recognition; 2. rapid network convergence with low computational complexity; 3. provide robustness against hardware- related noises. More details about the experimental datasets and implementation details are provided in the Supplementary Materials Section B, and the source code can be found at?
3.1 Accurate and Scalable Image Classification
Here, we report the classification results of LTL trained SNNs on CIFAR-IO, CIFAR- and Tiny ImageNet datasets against other SNN learning rules, including ANN-10-SNN conversion [57, 49, 46, 34, 22, 21, 12, 4, 31, 17] and direct SNN training [45, 13] methods. Given the network architectures and data preparation processes vary slightly across different work, therefore, we focus our discussions on the conversion or transfer errors between the ANNs and SNNs whenever the data is available.
Phttps:/ htt.ccub..oommrree/////eee tandemm Leaeddm___errningru..
5

---

100
2,00
SynOps Ratio Ratio (SNN/ANN)
Accuracy (%
1.25
0.87
0.75
0.5
0.50
STBP
0.2:0.25
Ailine LTL
quartioed ANN
QCES
32
Time Window Quantization Level
Time Window
(a)
(h)
Figure 2: (a) Comparison of the network performance under different time window sizes (SNN) and quantization levels (ANN). The results are obtained on the CIFAR-10 dataset using the VGG- architecture. (b) The ratio of total synaptic operations between SNN and ANN as function of the time window size.
On the CIFAR-10 dataset, reported in Table 1, LTT. offline trained spiking VGG-11 has surpassed the accuracy of teacher ANN at Tw = 16, this result suggests high effectiveness of the proposed T-S learning approach in transferring the knowledge through local objectives. Moreover, the LTL rule can take advantage of a larger time window that offers greater representation power, and further improves the performance of SNNs T. 32. It is also worth mentioning that by transferring the knowledge in learnable way, the proposed LTL rule is compatible with different neuron models, manifested through high accuracies achieved by both IF and LIF neuron models.
Although slightly degrades from the offiine version, the online trained SNNs can still achieve comparable results their teacher models with accuracies dropping by less than 1%, which indicates that removing temporal dependency between time steps has little impact on the training of SNNs (see Section D for detailed study on why the online LTL rule can perform as effectively as the offline version).
For the VGG-16 model, we notice modest accuracy drop that is more than 1% from the teacher model, which can be explained by the compounding layerwise representation errors. Fortunately, such errors be well addressed with larger time window. Specifically, T... 32, the accuracy gap to the teacher ANN narrows down to only 0. 13% and 0.33% respectively for LIF and IF neuron models under the offline training setting. When compared against other SOTA learning algorithms, the proposed LTL rule can achieve comparable performance in terms of both the final SNN model accuracies and the accuracy drop from ANN models. The same conclusions the results of CIFAR-100 and Tiny ImageNet datasets, hence suggests the proposed LTL rule is generalized, scalable, and highly effective method for constructing perforrrrerm SNNs.
also be drawn from
Rapid and Efficient Pattern Recognition
To realize the goal as a power-efficient alternative to the mainstream ANNs in solving pattern recognition tasks, SNNs should make accurate and rapid predictions with low firing rate. Here, we study whether the proposed LTL rule able to train SNNs that can cope with short time window. Specifically, design experiments that progressively increase the time window Tep from 4 to 32 during training, and we compare the offline LTL rule against the STBP [58] learning rule on the CIFAR-10 dataset. To ensure fair comparison, experimental settings, except that the LTL rule uses local objectives and the STBP rule uses a global objective. We note that the discrete neural representation of SNNs is equivalent to ANNs with quantized activation values [57]. Hence, we also compare our results to quantized ANNs obtained through quantization-awarr training [25].
use the VGG-11 architecture with the same
As shown in Figure 2(a), we observe that the accuracy of the quantized ANN degrades rapidly when the total quantization level is lower than 16, while both LTL and STBP trained SNNs can

---

Table 1: Summary of classification accuracies CIFAR-10. 10, CIFAR-100 and Tiny ImageNet datasets. Acc.- = SNN Acc. - ANN Acc.; Tw: time window size.
Method
Ncuron
ANN (%) NNN
(8)
SNN
[57]
VGG-I1
0.65
VGG-11"
(1,-60
(Osime)
VGG
19.20/92111
(0.78)
95.29 (92.75)
(.14)
Offine (Oeline) LTT
VGG--11
22.80
08.08,99..5
0.19 (-0.6)
93.25 (92.31)
(-0.68)
SPIKF-NORM [oo]
V6GG-1
(7.--200))
Hybrid Train [46]
91.33 17.-100)
Calibration [34]
VGG- 16
96.72
93.71
-2.01
RMP 1221
[
VGGG-1 16
93.83
95.3 (040)
[SC[2]] [2]]
IP
VGG 16
93.03
95.63 (24))
Opt. [12]
IP
VGB--1
02.34
02.29
-0.05
92.20
-1.05
Ogt, mon.. [4]
IP
VGGG-1
94.57
93.35
-1.19
94.20
-1.37
CHFAR-DD
OCFS [S]
IF
VGG-16
95,52
95,40
-0.12
95.54
0.02
Diet SNN [45]
LIF
VGG 16
92.70 (5
TET [13] Offiine (Ouline) LTT.
LIP
VGG 16
93.89 (6,
LIF
VGG-16
94.05
98.25 (92.8)
40,82 (-1.20)
93.92 (93 52)
20.33 (-0.3))
Offline (Onling) LTD.
VGG-16
94,05
93.04 (92
-1,01 (-1.3)
93.72 (93.17)
0.33 (-0.88)
SPIKE-NORM (40]
IF
RosNet-00
89,10
87.46 25000))
Hlybnd Train [16]
IF
ResNet-20
93.15
92.22
(T,-250)
Calibration
ResNet-20
94.78
RMP [22]
ResNet
91.47
91.36 2200)))))))
TSC [23)
RcsNa1-0
(74--
[32]
ResNa-20
-1.20
95.30
mom.
ReNa-22
QCES [S]
92.24
(45)
(T.
Offinne (Oaline) LTTL
RosNat20
15.76(98.15)
(-2.21)
95.25 (94.95)
Offine (Oaline) LTT.
ReoNe-20
95.36
(4.82(91.33)
00.54 (-4/00)
95.28 (93.84)
-0.08 (21
Ilyhnd Train [16]
VGG-1 -I
71.21
67.87 (7.-115)
Offine (Online) LTT.
LI
VGG-11
71.59
(70,3
1:04 6.0.9)
73.08 (72.45)
1499 86)
Otfine (Oslnee) LTL
1
VGG-11
71.99
(70.8))
1.15 (0.78)
72.71 (22.29)
1.12 (0.70)
NORK [oo]
11
VGGG 16
11.22
70.77
(50))
Calibration [34]
1P
VGG 16
77.89
73.55
-..4
RMP [:2]
IP
VG6-16
71.23
70.91 (74)2
TSC [2]]
IP
VGG 16
71.22
70.97 (7.)2048)
Opt. [12]
IF
VGG-16
70.49
65 65.94
-1.55
69.50
-0.89
Opt. mem. [-]
VGG-16
76.31
70.72
-5.59
74,82
-1.49
100
QCFS [s]
IP
VGG 16
76.28
76.24
0.04
77.01
0.79
Dict-SNN [45]
LIF
VGG-16
69.67 (T,-5)
TET (13)
LIP
VGG-16"
72/45 (T.-6)
Offline (Online) LTT,
LIP
VGG-16
74,42
14.19
-0.23 (-3.33
74.90 (72.97)
0.50 (-1.5)
Offline (Oaline) LTL
VGG
73.67 (69.55)
(4485)
74.56
0.14 (-3.88)
SPIKE
ResNot20
(50))
Calibration
ResNat-2
77.16
76:32
RMP [22]
RcsNa-300
(T,-2
TSC
ResNet20
(T.-2048)
[12]
-6.77
mem.
ResNet-20
-18.09
DieiSNNN [45]
RosNat-00
64.77 (T.-5)
Offine (Onine) LTT
LIF
ReoNa-20
76.36
75.62 (73.22)
(-3.44)3)4
76.12 (75.02)
0024 (-134)
Offline (Online) L.TL
ResNat20
76.36
75.22
-1.14 (-5.00
76.08 (73.93)
-0.28 (-2.4)
DCT [17]
VGG-13
36,90
32 (T.=-15)
Otfinne (Oaline) LTL
LIF
VGG-13
56.16
$5.37 (54.2))
0079 .079 G1.34)
55.5 (55.91)
4031 (-025)
Offline (Oeline) LTT1.
IF
VGG-11
56.16
(5.27(544200
(-1.88)1.88)
55.85 (55.61)
-0.31 (-0-55)
Thy AeaggNNe
Hybrid [31]
11
VGGG-6
36.56
51.92 (1.
Calibration [34]
IF
VGG-16*
60.95
44.30
-16.56
$3.06
-6.99
OCTS [S]
IP
VGG 16*
57.82
44.53
-13.29
53.54
-.28
Offine (Onliee) LTL
LIP
VGG-16
57.39
56 65.8500 (96.87)
(-0.5))
57.73 (57.45)
0.34 (0.06)
Otfline (Oslane) LTT
18
VGG-16
$7.39
50.59 (50.39)
(-.000
57.49
(57.000
0.10 (-0.99)
Calibration [34]
1F
RosNes-20"
58.29
44.89
-13,40
52.79
-5.5
QCFS [5]
1P
ResNet-20"
55.81
$2.32
.3.49
55.06
-1.75
Offfine (Oalinc) LTT.
LJF
ResNei-20
56.68
56 24 (52.52)
-0.44 (-4.16)
57.42
0. (-1.36)
Offline (Online) LTL
IF
ResNet
(51.92
0.900
56.48 (53.92)
-0.0 (-776)
OOu repeoduced results using publicly available codes.
maintain high accuracy even at T., = 4. It suggests the surrogate gradient approach adopted in SNN training can outperform the naive straight-through estimator [25] in addressing the discontinuity in the activation function. Moreover, the offline L.T.. rule performs better than the STBP rule, which can be explained by the fact that the gradient approximation errors of the surrogate gradient function will not compound across layers as in the STBP training [57]. Similar results have also been observed in the VGG-16 architecture (See Supplementary Materials Section E).
To shed light on the energy efficiency of LTL trained SNN models, we follow the common practice by computing the total synaptic operations (SynOps) ratio of SNN to ANN under different T. [56]. In general, the total SynOps (i.e., FLOPs) required by an ANN is a constant number that depends on the specific network architecture in use, while the total computations typically increase linearly with T, for rate-based SNNs. As shown in Figure 2(b), to achieve comparable accuracy to the ANN counterpart, our SNN only requires 0.43x total SynOps with Tu 8. It worth noting that the accluulea (MAC) operation required by ANNs is 5x more expensive than the accumulate
7

---

90
(%)
Accurscy (%6
60-
offline LTL
anline L.T.
STBP
TET
75
ANN (tencher)
Epoch
Epech (b)
Figure 3: Comparison of the learning curves of offline LTL, online LTL, STBP, and TET learning rules. The experiments are performed on the CIFAR-10 dataset with the VGG-11 architecture.
(AC) operations used in SNNs under 45 nm CMOS technology [23]. As such, our SNNs can yield order of magnitude energy saving the mainstream ANNs. Notably, this value be further boosted when deploying our SNNs onto the emerging mixed-signal NC chips. Moreover, when comparing with QCFS [5] which is recently introduced ANN-to-SNN conversion method, the proposed LTL rule requires fewer SynOps probably due to a higher percentage of neurons remaining silent our network. This can be explained by the round-down operation that has been taken when calculating the target spike count in our method.
3.3 Rapid Learning Convergence
By decoupling the learning of each layer through local objectives, the proposed LTL rule cleverly circumvents the spatial credit assignment and reduces the credit assignment to only the temporal domain. Moreover, the gradient approximation errors of the surrogate gradient learning method alleviated as the effective gradient transportation distance reduced. Taking benefits from these, the network convergence speed can be improved significantly compared to those trained with the STBP and TET [13] rule. As the learning curves are shown in Figure 3, both the offline and online LTL rules can converge rapidly within five epochs on the CIFAR-10 dataset. In contrast, the accuracy is still increasing at Epoch 50 for both the STBP and TET learning rules. Note that we follow the same settings for all these experiments, except for the learning objectives. Similar results have also been observed in the VGG-16 16 architecture (see Supplementary Materials Section F).
3.4 Low Memory and Time Complexity
In this section, we study the memory and time complexity of the proposed LTL rule and compare it against other popular learn- ing rules in Table 2. The STBP [58] rule requires storing the intermediate state of each neuron for all time steps and layers so to be used during backpropagation, resulting in O L) memory complex- ity, where M the number of neurons layer, T. is the time window size, and L. the total number of layers. The offline LTL rule has the same memory complexity as the STBP rule if all layers are trained in parallel. However, will be discussed soon, we can reduce the memory complexity by training all layers in sequence. Notably, by updating the network parameters using only local information, the online LTL and DECOLLE [26] learning rules do not require storing any intermediate state, resulting in (1) memory complexity.
Table 2: Comparison of the memory and time complexity of different learning rules. N: number of input neurons to a layer, M: number of neurons in a layer, N,: number of readout neurons in layer, L: number of layers, T. time window size
Method
Memory
Time
STBP [58]
O
O(N O + MN,) O(N O(NM)
DECOLLE [26]
O(1)
Offline LTL
O(MT.L.)
Online LTL
O(1)
In terms of time complexity, each parameter update of the STBP rule requires NMT, multiplications, where N is the number of input neurons to a layer. Due layer-wiss interlocking, all layers need to be updated in sequence from top to bottom, requiring O MT,L) time complexity update

---

all network parameters once. For offline LTL, by decoupling the dependency between layers and training all layers in parallel, the time complexity can be reduced to (NNMT, As mentioned earlier, we can trade off the memory complexity with the time complexity by training the network layers progressively one after the other. In this way, the memory usage of the offline L.TL. rule will be reduced to O ((MTw)): while the time consumption will be increased to ON T.). To validate the effectiveness of this training strategy, we perform an experiment on the CIFAR-IO dataset using VGG-11 architecture and we obtain a comparable result (92.98%) to that obtained when all layers are trained concurrently (93.20%). In this work, to achieve the shortest training time, we use the parallel training strategy the default for all the other reported experiments. Other users can flexibly select the training strategy according to their available computing resources.
For online LTL, by leveraging the temporal local errors, the time complexity can be further reduced to MN) Note that the DECOLLE requires an additional readout layer to calculate local errors, thereby requiring more time than online L.TI.. It is also worth mentioning that by gating the parameter updates with the proposed efficient on-chip implementation, we can further reduce the time complexity for online LTL. To provide a more concrete analysis of the memory and time complexity, we have measured the actual GPU memory usage and the training time on the CIFAR-10 dataset, which aligns well with the earlier theoretical analysis (see Supplementary Materials Section G). In sum, by coupling the low computational complexity with the rapid learning convergence discussed in Section 3.3, the proposed LTL rule can dramatically improve the training efficiency of SNNs.
3.5 Robust to Hardware-related Noises
Device non-ideality issues of analog computing substrates remain a major hurdle for deploying SNNs ultra-low-powe low-power mixed-signal NC chips. The proposed LTL rule can, however, address this challenge by enabling on-chip training in a noise-aware manner. Here, we examine the effectiveness of the on-chip LTL rule in addressing hardware-related noises and focus on four typical noise sources: device mismatch, quantization noise, thermal noise, and neuron silencing [6]. In particular, we use the noise models introduced by Julian al. [6], which were measured on DYNAP-SE [36] mixed-signal neuromorphic processor. Therefore, it can faithfully reflect the actual hardware noises that one can expect in the real scenario. To understand the effect of these noises, we first take an LTL pre-trained SNN and apply noise at different levels to it. Then, we use the online LTL rule to calibrate the pre-trained SNN model under the given noise.
Device Mismatch Device mismatch causes neuronal and synaptic parameters to vary from the desired ones. To simulate the parameter mismatch, add Gaussian noise to both the initial network parameters and their parameter updates following 8' ~ ~N, o0). In this model, a controls the level of mismatch, and we vary it from 0.05 0.4 to simulate different levels of noise. As shown in Figure 4(a), the accuracy of the pre-trained SNN degrades by more than 20% under the mismatch level 0.4. Notably, the proposed LTL rule can perform fast calibration and restore the original accuracy with only one training epoch. This will be advantageous for those devices that have poor endurance.
Quantization Noise The non-volatile memory devices typically have a limited number of analog states, thereby limiting the usable number of bits per network parameter. To simulate such quantization noise, we perform wtw--rrainnnguaattiinn in Figure 4(b), the SNN performs robustly to the quantization noise for bit bit-precision above 3. After which, the accuracy drops sharply by more than 30%. Following the quantization aware training [25], the proposed LTL rule can alleviate the effect of this type of noise and successfully recover the accuracy by around 17% Nevertheless, due to the reduced representation power, the resulting 3-bit precision model is still lagging behind the full precision one by about 15%.
to the bit-width of 7, 6,5, 5, 4, 3. As shown
Thermal Noise Thermal noise is intrinsic to the neuromorphic devices, which can be modeled as Gaussian noise on the input currents I' _N((,, Similar to the device mismatch noise, we simulate different levels of thermal noise by varying the a from 0.01 to 0.2. As illustrated in Figure 4(c), the model is highly sensitive to thermal noise and the accuracy drops steadily with a growing level of noise. We notice that the LTL rule can address such noise and quickly recover the accuracy in large part with only one training epoch.
Neuron Silencing Neuron silencing noise corresponds to the situation when a portion of spiking neurons fails to respond, and hence disturbs the network dynamics. We simulate neuron silencing
9

---

Device Mismatch
Quontization Noise
Thermal Noise
Neuron Silencing
90
90-
Accuracy (%)
40
80
80
80
60
60
60
50
w/o fine-tune
50
50
1st sptch
40
spuch
40
40
30
30
30
30
0.05 0.1 0.2 0.3 0.4
0.01 0.05 0.1 0.15 0.2
0.4 0.5
Mismatch level
Bit-precison
Noise lovel 0
Silencing probability (d)
(a)
(b)
(c)
Figure 4: The classification accuracy is function of the level of device noises for (a) device mismatch, (b) quantization noise, (c) thermal noise, and (d) neuron silencing. w/o fine-tune: without any noise- aware fine-tuning, 1st epoch: the result after training with the online LTT. rule for 1 epoch, best epoch: the best result obtained when training with the online LTL. rule for 50 epochs. The experiments are performed on the CIFAR-10 dataset with the VGG-11 architecture.
noise by randomly masking the neuron outputs with a failure rate varying from 10% to 50%. As shown in Figure 4(d), the accuracy drops steadily with a growing amount of failed neurons. Similar to other noise types, the accuracies are quickly restored with the on-chip I.TL training and maintained above 80% even under the challenging case when 50% of the neurons failed.
Comparison with the HIL Training Approach The proposed LTL rule exhibits better noise robustness and scalability than the HIL training approaches. To demonstrate this, we compare the online LTL rule with the HIL training approach [10] in addressing the aforementioned device-related noises (see Supplementary Material Section H for the details of experimental set-up and results). The LTL rule achieves comparable accuracy with the HIL approach for small and moderate levels of noise (see Table 7), but it appears to be more robust when facing stronger noise levels (see Table 8). Additionally, the LTL rule is more robust to the choice of the learning rate (see Table 9). In summary, the proposed LTI. learning rule is both effective and efficient for addressing the device non-idealities of mixed-signal NC chips.
4 Conclusion
In this paper, we propose the Local Tandem Learning (LTL.) rule for SNNs. Taking inspiration from the teacher-student learning approach, we leverage the highly effective feature representations of easily trainable ANN to guide the training of SNNs. This simplifies the credit assignment problem and leads to rapid network convergence with significantly reduced computational complexity. We also demonstrate the required temporal credit assignment can be well approximated with locally available information, and we provide an efficient on-chip implementation of the proposed LTL rule. Under this on-chip setting, we demonstrate that the LTL rule can provide robustness against various device non non-ideality issues. It, therefore, opens up myriad opportunities for rapid and efficient training and deployment of SNNs on ultra-low-pout learning approach on the recurrent neural networks and transformer
mix mix-signal NC platforms. For future work, we will explore
the proposed teacher-studee architectures so as to handle those temporally rich signals, such as speech, video, and text.
Acknowledgments and Disclosure of Funding
This research work is supported by IAF, A SOITEC, NXP, National University of Singapore under FD-fAbrICS: Joint Lab for FD-SOI Always-on Intelligent & Connected Systems (Award 12001E0053), and National Research Foundation, Singapore under its Medium Sized Centre for Advanced Robotics Technology Innovation (WBS: X-0009428-00--00 This research is also supported by National Science Foundation of China (Grant Number: 62106038), Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen, China (Grant No. B101202101 17-KK02), Shenzhen Research Institute of Big Data, the National Key Research and Development Program of China (Grant No. 2021ZD0200000), CCF-Hikvision Open Fund, and The Hong Kong Polytechnic University under Grant No. PO043563.
10

---

References
[1] Syed Ahmed Aamir, Yannik Stradmann, Paul Müller, Christian Pehle, Andreas Hartel, Andreas Grübl, Johannes Schemmel, and Karlheinz Meier. An accelerated lif neuronal network array for a large-scale mixed-signal neuromorphic architecture. IEEE Transactions on Circuits and Systems I: Regular Papers, 65(12)4429-4312, 2018.
[2] Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, et al. Truenorth: Design and tool flow of 65 mw I million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 34(10):1537-1555 2015.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint orXiv:/409.0473. 2014.
[4] Tong Bu, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Optimized potential initialization for low-latency spiking neural networks. arXiv preprint arXiv:2202.014442 2022.
[5] Tong Bu, Wei Fang. Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- snn conversion for high-accuracy and ultra-low-latenn loaenyy spiking neural networks. In International Conference Learning Representations, 2021.
[6] Julian Büchel, Dmitrii Zendrikov, Sergio Solinas, Giacomo Indiveri, and Dylan Muir. Super- vised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processors. Scientific reports, 11(1):1-12, 2021.
[7] Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1)554-66 2015.
[8] Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spiking neural networks through gradient rewiring. arXiv preprint 10.iv:2105.04999 2021.
[9] Erika Covi, Stefano Brivio, Alexander Serb, Themis Prodromakis, Marco Fanciulli, and Sabina Spiga. Analog memristive synapse in spiking networks implementing unsupervised learning. Frontiers in neuroscience, 10:482, 2016.
[10] Benjamin Cramer, Sebastian Billaudelle, Simeon Kanya, Aron Leibfried, Andreas Grübl, Vitali Karasenko, Christian Pehle, Korbinian Schreiber, Yannik Stradmann, Johannes Weis, et al. Surrogate gradients for analog neuromorphic computing. Proceedings of the National Academy of Sciences, (19(4))2209194111 2022.
[11] Mike Davies, Narayan Srinivasa, Tsung-Han Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1):82-99, 2018.
[12] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference Learning Representations, 2021.
[13] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re- re-weighting. arXiv preprint arXiv:2202. 1946, 2022.
[14] Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. clas-claasi,ying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference on neural networks (LJCNN), pages 1-8. iece, 2015.
[15] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[16] Simon Friedmann, Johannes Schemmel, Andreas Grübl, Andreas Hartel, Matthias Hock, and Karlheinz Meier. Demonstrating hybrid learning flexible neuromorphic hardware system. IEEE transactions biomedical circuits and systems, 11(1):128-11) 142, 2016.
11

---

[17] Isha Garg, Sayeed Shafayet Chowdhury, and Kaushik Roy. Det-snn: Using det to distribute spatial information over time for learning low-latency spiking neural networks. arXiv preprint arXiv:2010.01795, 2020.
[18] Wulfram Gerstner and Werner M Kistler. Spiking neuron models: Single newrons, populations. plasticity. Cambridge university press, 2002.
[19] Julian Göltz, Laura Kriener, Andreas Baumbach, Sebastian Billaudelle, Oliver Breitwieser, Benjamin Cramer, Dominik Dold, Akos Ferene Kungl, Walter Senn, Johannes Schemmel, et al. Fast and energy-efficient neuromorphic deep learning with first-spike times. Nature machine intelligence, 3(9):833-833)) 2021.
[20] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: survey. International Journal of Computer Vision, 129(6):188--181 2021.
[21] Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European Conference on Computer Vision, pages 388-404, Springer, 2020.
[22] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13558-13567, 2020.
[23] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[24] Giacomo Indiveri and Yulia Sandamirskaya. The importance of space and time for signal processing in neuromorphic agents: the challenge of developing low-power, autonomous agents that interact with the environment. JEEE Signal Processing Magazine, 36(6):16 28, 2019.
[25] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient antegerarithhneticonnf inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018.
[26] Jacques Kaiser, Hesham Mostafa, and Emre Neftci. Synaptic plasticity dynamics for deep continuous local learning (decolle). Frontiers in Neuroscience, 14:424, 2020.
[27] P. Kanerva. Hyperdimensional computing: An introduction to computing in distributed rep- resentation with high digh-dimensional random vectors. Cognitive computation, 1(2):139-159) 2009.
[28] Chul-Heung Kim, Soochang Lee, Sung Yun Woo, Won-Mook Kang, Suhwan Lim, Jong-Ho Ho Bae, Jaeha Kim, and Jong-Ho Ho Lee. Demonstration of unsupervised learning with tikee-timing- dependent plasticity using a tft-type nor flash memory array. IEEE Transactions on Electron Devices, 65(5):1774-1780. 2018
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of 2009.
features from tiny images.
[30] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[31] Souvik Kundu, Gourav Datta, Massoud Pedram, and Peter A Beerel. Spike- thrif- Towards energy-efficient deep spiking neural networks by limiting spiking activity via attention- guided compression. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3953-3962, 2021.
[32] Dongseok Kwon, Suhwan Lim, Jong-Ho Bae, Sung-Tae Lee, Hyeongsu Kim, Young-Tak Tak Seo, Seongbin Oh, Jangsaeng Kim, Kyuho Yeom, Byung-Gook Park, et al. On-chip training spiking neural networks using approximated backpropagation with analog synaptic devices. Frontiers in neuroscience, page 423, 2020.
12

---

[33] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))227823333 1998.
[34] Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In Internationall Conference on Machine Learning, pages 63 6316-6325. PMLR, 2021.
[35] Songhua Liu, Jingwen Ye, Sucheng Ren, and Xinchao Wang. Dynast: Dynamic sparse trans- former for caamllar-gudde image generation. In European Conference on Computer Vision, 2022.
[36] Saber Moradi, Ning Qiao, Fabio Stefanini, and Giacomo Indiveri. A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dynaps). IEEE transactions on biomedical circuits and systems, 12(1) 10:6:1212 2017.
[37] Emre Neftci and Giacomo Indiveri. A device mismatch compensation method for vlsi neural networks. In 2010 Biomedical Circuits and Systems Conference (BioCAS), pages 262-265. IEEE, 2010.
[38] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 13: 2019.
[39] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609, 03499, 2016.
[40] Melika Payvand, Mohammed E Fouda, Fadi Kurdahi, Ahmed M Eltawil, and Emre O Neftci. On-chip error- trigg-triggeed learning of multi-layer memristive spiking neural networks. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 10(4):522-535, 2020.
[41] G Pedretti, V Milo, S Ambrogio, R Carboni, S Bianchi, A Calderoni, N Ramaswamy, AS Spinelli, and D lelmini. Memristive neural network for on-line learning and tracking with brain-inspired spike timing dependent plasticity. Scientific reports, 701):1-10,200 2017.
[42] Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip 2019.
architecture. Nature, 572(7767):006-11
[43] M Prezioso, MR Mahmoodi, F Merrikh Bayat, H Nili, H Kim, A Vincent, and DB Strukov, Spike-timing-dependeen plasticity learning of coincidence detection with passively integrated memristive circuits. Nature communications, 9-1):188, 2018.
[44] Ning Qiao and Giacomo Indiveri. Scaling mixed-signal neuromorphic processors to 28 nm fd-soi technologies. In 2016 IEEE Biomedical Circuits and Systems Conference (BioCAS), pages 552-555. IEEE, 2016.
[45] Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transuctions on Neural Networks and Learning Systems, pages 1-9, 2021.
[46] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.01807, 2020.
[47] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784),607-617, 2019.
[48] Bodo Rueckauer, Iulia-Alexandr Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous- valued deep networks efficient event-driven networks for image classification, Frontiers in neuroscience. 11:682, 2017.
13

---

[49] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures, Frontiers in neuroscience, 13:95, 2019.
[50] Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time, Advances in neural information processing systems, 31, 2018.
[51] David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-339 2017.
[52] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1406, 2014.
[53] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9, 2015.
[54] Stanislaw Woćniak, Angeliki Pantazi, Thomas Bohnstingl, and Evangelos Eleftheriou. Deep learning incorporating biologically inspired neural dynamics and in-memory computing. Nature Machine Intelligence, 26):325-3366 2020.
[55] Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report, 2017.
[56] Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learning rule for effective training and rapid inference of deep spiking neural networks. JEEE Transactions Neural Networks and Learning Systems, 2021.
[57] Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li, and Kay Chen Tan. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[58] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-tempone backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
[59] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1311-1318, 2019.
[60] Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Michael L. Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. Toward human parity in conversational speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(12):440--2423, 2017.
[61] Qi Xu, Jiangrong Shen, Xuming Ran, Huajin Tang, Gang Pan, and Jian K Liu, Robust transcoding sensory information with neural spikes. IEEE Transactions on Neural Networks and Learning Systems, 3335):195511946, 2021.
[62] Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. In European Conference on Computer Vision, 2022.
[63] Yiding Yang. Zunlei Feng, Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. In Neural Information Processing Systems, 2020.
[64] Friedemann Zenke and Emre O Neftci. Brain- insp--idsired learning on neuromorphic substrates. Proceedings of the IEEE, 109(5))935-9)))) -500, 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4
(c) Did you discuss any potential negative societal impacts of your work? [No] Not
applicable to our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Supplementary Materials Section
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] See Supple- mentary Materials Section B
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Supplementary Materials Section B
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] We report the error bar for MNIST experiments. Due to time constraint, we not run for other datasets and we may update in future.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Supplementary Materials Section B
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [N/A] We use public datasets.
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "c7GmH_kyLu", "reviewer": "Reviewer_EHpf", "summary": "This paper proposes a Local Tandem Learning (LTL) method to train spiking neural networks by distilling knowledge from a pre-trained ANN in a layer-wise way. Two versions of LTL, offline learning and online learning, are introduced and the on-chip implementation is discussed. Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet show competitive performance and robustness to hardware-related noises.", "questions": "See Weakness above.", "limitations": "There is no potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed LTL method considers both offline and online learning, and the on-chip implementation is discussed. It is novel to transfer knowledge of ANNs by distilling intermediate feature representations instead of reusing network weights.\n2. Experiments from many perspectives are conducted, including classification accuracy, synaptic operations, convergence of learning, and robustness to hardware-related noises.\n\nWeakness:\n1. This paper emphasizes the on-chip implementation throughout the paper, but no real on-chip implementation is demonstrated, and more importantly, it is not explained clearly how conceptually the learning from a pre-trained ANN (rather than direct SNN training) can be implemented on chips. Figure 1(a) shows that the inference of both SNN and ANN is required in order to calculate the loss for each layer. But how can the inference of ANNs be implemented on neuromorphic chips without conversion to SNNs? Or is the inference of ANNs realized on common hardware and the signal is passed to chips? If so, is it only partly on-chip learning, and what is the advantage over off-chip learning since it still requires much off-chip computation (the inference of ANNs)? If the motivation is to address the real device mismatch problem, it is better to show some preliminary real on-chip results, otherwise the simulation of noises can also be leveraged by off-chip methods for finetuning. Or some comparison with finetuning by other off-chip methods in this simulation scenario is ok. There is some related work to address the real device mismatch problem [1]. They combine the on-chip and off-chip computation to correct real device mismatch by surrogate gradient learning. What is the comparison of the proposed method to their method? There should be some discussion and clarification to the above questions.\n\n2. The state-of-the-art ANN-to-SNN conversion method outperforms the results (accuracy and latency) in this paper [3]. Since LTL also requires knowledge from pre-trained ANNs and involves more complex training procedure than conversion methods (conversion methods only need to train ANNs, LTL should first train ANNs and then train SNNs with knowledge from the pretrained ANNs), what is the advantage of LTL if ANN-to-SNN can achieve better results?\n\n3. Experiments do not include the large-scale ImageNet dataset. Typical ANN-to-SNN conversion methods include results on large-scale datasets [2,3] and recently even direct training methods [4,5,6,7] have demonstrated competitive performance on ImageNet. Since this paper also transfer knowledge from ANNs, it would be better to include ImageNet results to show the scalability of the method.\n\n4. Is the proposed method applicable to dynamic DVS datasets, such as CIFAR10-DVS? SNNs can be more powerful to process DVS inputs and have demonstrated strong results [4,5,6,7]. Since the proposed method relies on ANNs, is it limited to static inputs? How is the flexibility compared with direct training methods?\n\n[1] Cramer B, Billaudelle S, Kanya S, et al. Surrogate gradients for analog neuromorphic computing. Proceedings of the National Academy of Sciences, 2022, 119(4): e2109194119.\n\n[2] Li Y, Deng S, Dong X, et al. A free lunch from ANN: Towards efficient, accurate spiking neural networks calibration. International Conference on Machine Learning, 2021.\n\n[3] Bu T, Fang W, Ding J, et al. Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks. International Conference on Learning Representations, 2022.\n\n[4] Zheng H, Wu Y, Deng L, et al. Going deeper with directly-trained larger spiking neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.\n\n[5] Li Y, Guo Y, Zhang S, et al. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 2021.\n\n[6] Meng Q, Xiao M, Yan S, et al. Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.\n\n[7] Deng S, Li Y, Zhang S, et al. Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. International Conference on Learning Representations, 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vl8acFyGHWN", "reviewer": "Reviewer_6gNN", "summary": "This paper proposes a novel method of training spiking neural networks (SNNs) by matching the intermediate feature representations of SNNs with pretrained ANNs. The method is on-chip and local, allowing SNNs to be learned directly on neuromorphic hardware. Experiments on image classification benchmarks demonstrate that the proposed LTL rule performs comparably to other ANN-to-SNN conversion methods. Moreover, LTL is able to train SNNs with high accuracy even under small time windows, which is important to limit the number of synaptic operations of SNNs. LTL trained SNNs can also be trained with a small number of training iterations relative to STBP. Theoretically, STBP has lower or comparable memory and time complexity to baselines. Finally, LTL is robust to a variety of noises.", "questions": "How does the number of synaptic operations of LTL compare with other SNN-to-ANN conversion methods?\n\nDoes the asymptotic analysis presented in section 3.4 match empirical observations?\n\nHow does the noise robustness of LTL compare with other SNN-to-ANN conversion methods?", "limitations": "The authors touch on some limitations of the work in the conclusion: they do not study architectures for sequential data such as RNNs and transformers. It would also be good to note any practical difficulties that might arise when implementing LTL on neuromorphic hardware.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**\nThe proposed method appears novel. To my knowledge, this method is the first on-chip ANN-to-SNN conversion method that uses layerwise feature-based transfer of information from an ANN to an SNN. \n\n**Quality**\nThe method is not theoretically motivated. However, this is not strictly necessary if there are good empirical results.\n\nExperiments are extensive. The authors demonstrate that LTL achieves comparable performance to many baseline methods (see Table 1). However, given that one of the main advantages of LTL is its efficiency and robustness, the experiments in section 3.2 - 3.5 could be significantly expanded to highlight this. In particular, the authors may want to consider additional SNN training baselines on top of STBP. Moreover, in section 3.4 the authors study the asymptotic time and memory complexity of LTL. It would be valuable to empirically validate these asymptotic trends. Finally, the experiments in section 3.5 are interesting, but unfortunately difficult to contextualize without a baseline method to compare with.\n\nIf the authors can adequately address the concerns on the experiments, then I think the overall quality of the paper can be dramatically increased.\n\n**Clarity**\nThe paper is generally well written. The mathematical notation is clear and figures are well illustrated.\n\n**Significance**\nThe paper is potentially quite significant. It appears to be a more efficient and robust method of training SNNs on-chip than previous method, while maintaining high accuracy. However, further empirical experiments are needed to better justify these claims. Given that the experiments are improved, the paper would be an important contribution to the field of SNN training, and may potentially be used on actual neuromorphic chips.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KFssSzpRGP", "reviewer": "Reviewer_6iuj", "summary": "This paper proposes a generalized learning rule called Local Tandem Learning (LTL) which the SNN mimics the feature representation of a pre-trained ANN through local loss functions. The intermediate feature representation of a pre-trained ANN is transferred to SNN through layer-wise local loss functions. This method has not only high accuracy but also fast convergence It has two versions: offline and online and both versions have high performance. In the experiments, the authors verify the high performance of this method on CIFAR-10, CIFAR-100, and ImageNet dataset. Besides, this method is hardware friendly and can be easily implemented on-chip which is robust to hardware-related noises.", "questions": "(1)\tThe comparison of accuracy between the Local Tandem Learning (LTL) method and only use the surrogate gradient method to training SNN can be added to verify that this method learns the knowledge in ANN.\n(2)\tMore architectures such as ResNet can be used to verify the high performance of this algorithm rather than just VGG network architecture in order to better demonstrate the advantages of this approach in different network structures.\n", "limitations": "The authors put forward a generalized learning rule, termed Local Tandem Learning (LTL) which the SNN mimics the feature representation of a pre-trained ANN through local loss functions. It uses the intermediate feature represents of the pre-trained ANN to supervise the training of SNNs. This method considers the problem that the SNN is difficult to train because of the discrete spikes. It improves the accuracy and computational complexity of SNN. The research of this article is very useful and valuable.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1)\tThe method proposes a local loss function trains the SNN by learning the intermediate feature representation of the middle layers of the pre-trained ANN. It combines the advantages of ANN-to-SNN conversion and gradient-based training methods. This work is very valuable.\n(2)\tThe authors have done sufficient experiments. This method reduces the epochs required for training to convergence. It can achieve rapid network convergence within five training epochs on the CIFAR-10 dataset.\n(3)\tThe accuracy of SNN trained by this method gap to the teacher ANN is almost zero. Compared with other SOTA learning algorithms, the proposed methods also can achieve comparable performance.\nWeaknesses:\n(1)\tBackground knowledge of the Teacher-Student (T-S) learning approach is not introduced and some introductions about this aspect can be added.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4EZnLd7biE", "reviewer": "Reviewer_XCfF", "summary": "Summary:\nThis paper proposes a spiking neural network (SNN) learning rule, namely Local Tandem Learning (LTL). It combines the teacher-student learning framework and a novel layer-wise local learning method for ANN-to-SNN conversions. Experiments demonstrate near lossless knowledge transfer can be achieved on Cifar-10, Cifar-100 and Tiny-ImageNet datasets with fast convergence and low computational complexity. Notably, authors also show that the LTL rule is hardware friendly. It can address the hardware-related noises through on-chip learning, which remains a grand challenge for deploying SNNs on the ultra-low-power mixed-signal neuromorphic computing chips.\n\nOverall, this paper is well-written and easy to understand. In terms of technical quality, the proposed local tandem learning rule is novel, significant, and well-motivated. It provides an appreciable advance in the field of neuromorphic computing. \n\n\n\n", "questions": "1. Please summarize the key differences from previous ANN-to-SNN work[1]\n2. Please fix the typo of one of the works in Table 1: OCFS  \n\nRef. \nWu J, Chua Y, Zhang M, et al. A tandem learning rule for effective training and rapid inference of deep spiking neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021.", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nThe topic of this paper is of sufficient interests to the neuromorphic computing community. As clearly summarized in the paper, the LTL learning rule has several attractive feats:\n1.\tThe proposed Teacher-Student learning framework is a generalized knowledge transfer approach. It can be applied for different neuron models, which is out of the reach of the existing ANN-to-SNN conversion methods.\n2.\tThe proposed local learning approach is a valuable exploration beyond the mainstream e2e SNN training algorithms. By decoupling the learning process for each layer, the proposed method reduces the required spatial-temporal credit assignment to only the temporal domain, which greatly improve the network convergence speed and can also alleviate the compounding gradient approximation errors of the surrogate gradient method.\n3.\tThis paper also addresses an under-explored direction of algorithm-hardware co-development. This paper shows the proposed LTL rule can be easily implemented on-chip using only locally available information.  \n\nCons:\n1.\tFor experimental results in Table 1, the comparisons are mainly focus on the ANN-to-SNN conversion methods, while authors only compared with one direct training method. I suggest authors to include other more recent direct training algorithms to show the superiority. \n2.\tFor online LTL learning, as the first few time-steps are generally unstable, with inaccurate firing rate estimation, hence result in noisy training. I am wondering how this affects the learning performance.\n3.\tIn the hardware-related noise experiments, authors using the Gaussian noise to simulate the parameter mismatch and thermal noise. However, in the real hardware implementation, these noises may not follow the Gaussian distribution.  I am curious why authors use the Gaussian Noise in this experiment.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept