A Progressive Training Framework for Spiking Neural Networks with Learnable Multi-hierarchical Model
Authors: Zecheng Hao
Xinyu Shi
Zihan Huang
Tong Bu
Zhaofei Yu
Tiejun Huang
Venue: ICLR 2024 poster (2024)
Keywords: Spiking Neural Networks
Learnable Multi-hierarchical Model
Spatio-Temporal Back-propagation
Primary area: applications to neuroscience & cognitive science
TL;DR: We propose Learnable Multi-hierarchical (LM-H) neuron, which is an advanced model that can dynamically regulate the extraction ratio between historical and current representation.

Abstract: Spiking Neural Networks (SNNs) have garnered considerable attention due to their energy efficiency and unique biological characteristics. However, the widely adopted Leaky Integrate-and-Fire (LIF) model, as the mainstream neuron model in current SNN research, has been revealed to exhibit significant deficiencies in deep-layer gradient calculation and capturing global information on the time dimension. In this paper, we propose the Learnable Multi-hierarchical (LM-H) model to address these issues by dynamically regulating its membrane-related factors. We point out that the LM-H model fully encompasses the information representation range of the LIF model while offering the flexibility to adjust the extraction ratio between historical and current information. Additionally, we theoretically demonstrate the effectiveness of the LM-H model and the functionality of its internal parameters, and propose a progressive training algorithm tailored specifically for the LM-H model. Furthermore, we devise an efficient training framework for our novel advanced model, encompassing hybrid training and time-slicing online training. Through extensive experiments on various datasets, we validate the remarkable superiority of our model and training algorithm compared to previous state-of-the-art approaches. Code is available at [https://github.com/hzc1208/STBP_LMH](https://github.com/hzc1208/STBP_LMH).

Published as a conference paper at ICLR 2024
A PROGRESSIVE TRAINING FRAMEWORK FOR SPIK- ING NEURAL NETWORKS WITH LEARNABLE MULTI- HIERARCHICAL MODEL
Zecheng Hao', Xinyu Shil., Zihan Huang', Tong Bu Zhaofei Yu! & Tiejun Huang!2
School of Computer Science, Peking University
2 Institute for Artificial Intelligence, Peking University
ABSTRACT
Spiking Neural Networks (SNNs) have garnered considerable attention due their energy efficiency and unique biological characteristics. However, the widely adopted Leaky Integrate-and-Fire (LIF) model, the mainstream neuron model in current SNN research, has been revealed to exhibit significant deficiencies in deep- layer gradient calculation and capturing global information on the time dimension. In this paper, we propose the Learnable Multi-hierarche (LM-H) model to address factors. We point out
these issues by dynamically regulating its membrane- that the LM-H model fully encompasses the information representation range of the LIF model while offering the flexibility to adjust the extraction ratio between historical and current information. Additionally, we theoretically demonstrate the effectiveness of the LM-H model and the functionality of its internal parameters, and propose progressive training algorithm tailored specifically for the LM-H model. Furthermore, we devise an efficient training framework for our novel advanced model, encompassing hybrid training and time-slicing online training. Through extensive experiments on various datasets, we validate the remarkable superiority of our model and training algorithm compared to previous state-of- the-art approaches. Code is available at https: : / giigth...... com com/hzcl200 STBP LMH.
INTRODUCTION
Spiking Neural Networks (SNNs) are considered as the third generation of artificial neural net- works (Maass, 1997), which represents fascinating paradigm with superior biological plausibility and high energy efficiency in the field of computational neuroscience, Unlike traditional Analog Neural Networks (ANNs), which rely on continuous-van vaalued outputs, SNNs draw inspiration from the structure of the human brain and emulate the behavior of real biological neurons by utilizing binary and sparse spiking signals to transmit information. In SNNs, neurons only generate spikes and transmit them to the postsynaptic layer when their membrane potential exceeds certain firing threshold. This event-driven characteristic of SNNs provides significant advantages in terms of computational efficiency and power consumption reduction (Roy et al., 2019), especially when deployed on neuromorphic chips (Merolla et al., 2014; Davies et al., 2018; DeBole et al., 2019). ()... In recent years. SNNs have garnered increasing attention due their potential address certain limitations of ANNs. They have shown promise in applications such as image recognition (Fang et al., 20) object detection (Kim al., 2020b), natural language processing (Lv al., 2023), cue combination (Yu et al., 2020), and network robustness (Bu et al., 2023), etc.
The Leaky Integrate- and and-Fire (LIF) model (Gerstner Kistler, 2002) widely used spiking neuron model that consists of three main processes: charging. membrane potential leakage, and firing of spikes. However, the non-differentiable nature of the spike firing process poses challenges for gradient calculations. To address this issue, researchers have developed the current mainstream Temporl Back-pre propagation (STBP) algorithm (Wu et al., 2018), which introduces surrogate gradients to enable gradient-based based learning in LIF-based SNNs. Nevertheless, the LIF model still
Corresponding author; yuzf1 2 2erf12@pkuu 2@pku.edu.cn

---

Published as a conference paper at ICLR 2024
encounters gradient calculation failures when applied to deep residual networks (Fang et al., 202 la). Additionally, the LIF model struggles to effectively utilize historical information, such as the input current and membrane potential previous time-steps, which can be crucial for capturing temporal dependencies and improving network performance.
To address these issues, propose novel Learnable model, the scaling factors from the dendrite layer regulate the proportion of historical information extracted by the model, while the factors from the soma layer determine the degree of potential leakage and the intensity of the input current at present. By effectively leveraging these factors, the LM-H model enables precise gradient calculation and optimal learning performance. It is worth noting that our model is inspired by the TC-LIF model (Zhang et al., 2023). However, the TI- model suffers from the issue of setting certain membrane-related parameters to fixed or contradictory values. Moreover, it fails to accurately recognize the relationship between the LIF and advanced neurons, resulting in significant impact on the stability of learning gradients. In this work, we investigate the relationship between the LIF and LM-H models, and achieve more precise information extraction and gradient propagation by designing appropriate optimization intervals and calculation modes for weammbann--ellttee parameters. Our main contributions can be summarized as follows:
(LM-H) model. In the LM-H
We identify the limitations of the vanilla LIF model in terms of its representation capabilities and propose the LM-H model with a wider calculation scope. We mathematically demon- strate that our proposed model can effectively extracting global information along the time dimension and propagate gradients in deep networks.
We systematically analyze the specific roles of parameters on the dendrite and soma layers, and further develop a progressive STBP training algorithm for the LM-H model, which can dynamically optimize the membrane-relater parameters during the learning process. To enhance the energy efficiency of SNN learning, we propose an efficient training frame- work specifically designed for the LM-H model, which includes hybrid training and time- slicing online training.
Experimental results validate the significant advantages of the LM-H model in the field of SNN supervised learning. Our proposed method achieves state-of-the-art performance on multiple datasets with various scales and data types. For example, we achieve 80. 80.31% % top-1 accuracy with 4 time-steps on CIFAR-100 using the ResNet-19 architecture.
2
RELATED WORK
Supervised Learning for SNNs. STBP algorithm has emerged as the most mainstream supervised learning approach in the SNN community, which draws inspiration from the Back-Propagation Through Time (BPTT) gradient computation framework used in Recurrent Neural Networks (RNNs). Pioneered by researchers such as Wu al. (2018) and Neftci et al. (2019), STBP addresses the non-differentiable nature of firing spikes during the back-propagation process by employing surrogate gradient functions. Building upon the foundation laid by STBP, subsequent studies have further enhanced the accuracy and stability of SNN training. For instance, Li et al. (2021) and Wang et al. (2023) proposed families of differentiable functions that can be adaptively evolved during the learning process, leading to improved training outcomes. Deng et al. (2022) and Guo et al. (2022a) introduced novel loss functions to regulate the distribution of spikes and membrane potentials along the temporal dimension, resulting in more precise alignment of learning gradients. Furthermore, researchers have developed various batch normalization structures tailored for SNN learning (Wu et al., 2019; Zheng al., 2021; Duan al., 2022). These structures facilitate better network performance and training convergence. Additionally, Fang et al. (2021a) significantly advanced the effective learning of SNNS on deep networks with more than 100 layers by improving the connection patterns of residual blocks. It is worth noting that supervised learning methods based on spike firing timing (Mostafa, 2017; Kim et al., 2020a; Zhou et al., 2021) have garnered significant attention in recent years. However, these approaches are currently limited to shallow networks and small-scale datasets, and further research is needed to extend their applicability to more complex scenarios (Zhu et al., 2022).
Efficient Training for SNNs. The ANN-SNN Conversion (Cao et al., 2015; Han et al., 2020; Bu et al., 2022) is another widely used algorithm family of SNN learning, which converts pre-trained ANNs into SNNs based on the approximate linear relationship between the average spike firing rate of adjacent layers. While the precision of ANN-SNN Conversion may be compromised under few time-steps (Hao al., 2023a;b), it offers significant time and memory savings compared to
2

---

Published as a conference paper at ICLR 2024
m
A3
8
5.8
mm
x
W
W
W
J-1
(a) LIF Neuron
(b) L Learnable Multi-hierarchiaal Neuron
Time-step
Figure 1: The structural description for the vanilla LIF and LM-H models.
STBP. Furthermore, efficient training frameworks such online learning (Xiao al., 2022; Yang et al., 2022; Meng et al., 2023) have been proposed. These frameworks perform gradient updates at each time-step, maintaining constant memory consumption instead of linearly increasing it. Hybrid training (Rathi & Roy, 2023; Wang et al., 2022) is another popular training method, which typically employs STBP to optimize and fine-tune parameters based on models obtained from ANN-SNN Conversion. Researchers have discovered that applying STBP for a limited number of epochs (s 30) can significantly enhance the performance of SNNs under low time-latency.
Structural optimization for spiking neurons. In early research, the relevant model parameters of spiking neurons were often viewed as super-parameters, which hindered the ability of SNNs to effectively capture and extract information during the training process. Rathi & Roy (2023) and Fang et al. (2021b) set the membrane leaky constant and firing threshold as learnable variables, thereby enhancing the biological similarity of spiking neurons. Stöckl & Maass (2021) further optimized membrane-related parameters by aiming to reduce spike firing rate. Another advanced model, GLIF (Yao al., 2022), selectively regulates the input current, decay factor, and reset mechanism, thereby improving the dynamic learning ability of SNNs. Additionally, Wang et al. (2023) proposed novel spiking neuron with mixed output alleviate the impact of the oon-ifferrntiable problem on gradient precision. TC-LIF (Zhang et al., 2023) aimed to enhance the capability of SNNs to tackle long-term sequences, which also provided inspiration for this work. However, the problem of gradient divergence still exists for TC-LIF within a certain parameter range.
3 PRELIMINARIES
3.1 LEAKY INTEGRATE-AND-FEE (LIF) NEURON MODEL
The LIF model widely used spiking neuron model in the neuroscience community. As shown in Fig. (a). the LIF model emulates the functioning of biological neurons by incorporating three key processes: charging, leakage, and firing. The following equations describe the iterative version of the LIF model in discrete form.
m [t] = X' X - 1]+ ['.. of e'() = m [t] s' [t]o'.
f1,
11,
m 2 6¹
['t]] w i 1-][t
s' Hmm 8¹) =
(1)
o,
otherwise
In the equations, m m and e' represent the membrane potential before and after emitting a spike at the f-th time-step, respectively. ['( denotes the presynaptic input current and W is the weight matrix of the 1-th layer. The membrane decay constant X controls the extent which neurons retain previous information. When X' = 1. the LIF model degenerates into the Integrate-and and-Fire (IF) model. The binary variable { determines whether a spike is fired or not. We choose the soft-reset mechanism in Eq.(1), where the membrane potential o'[[] is decreased by threshold magnitude when it exceeds the firing threshold 0'. The relationship between s' [t] and vn' [t] mathematically represented by the Hcaviside step function (.).
3.2
BACK-PROPAGATION (STBP) WITH SURROGATE GRADIENT
Drawing inspiration from the calculation mode of vanilla BPTT, we can transfer it to the learning of SNNs. As shown in (2),)), it can be observed that the gradient propagation in SNNs unfolds from
3

---

To overcome the limitations of the LIF model discussed earlier, we propose a brand-new LM-H neuron model. As shown in Fig. (b)) the LM-H model incorporates separate dendrite and soma layers. The input current flows through the dendrite layer before reaching the soma layer, where the Published as a conference paper at ICLR 2024
s/
0
LM-H Neuron
LMH NEuront
137-1
3.00
14-1
W
$1
$
H5 15.
2n/4 /
phlobe
Figure 2: The specific extraction process of the historical and current information in the LM-H model.
back front in both spatial and temporal dimensions. Here represent the chosen loss function. Note that the Heaviside function is non-differentiable, which introduces an intractable node 0& [4]] [1]/3m' in the gradient calculation graph. To address it, previous works have used a surrogate function H' (-) that closely resembles the Hcaviside function but is differentiable for back prok-propagation. In Eq.(3), we employ the well-known triangle function, where 7 controls the width of the non-zero interval.
0C ac
0s
Oc
0ml [t. 1]
(2)
0m'l [t] as
0ml
Om' + 1] dml
ds' [1] H' ('mm'1 [t) 8*) 72
1
max (0. 7 |ml [t] 01)
(3)
0m
[t]
4 METHODOLOGY
4.1 REPRESENTATION DEFECTS OF THE LIF MODEL
The gradient vanishing & exploding problem in deep residual architectures. For the ResNet family, researchers have identified a potential issue with gradient calculation in the LIF model when comes the identity mapping path (Fang et al., 2021a). Considering residual network with L blocks, if we assume vl E [1, L), o' it = 0 and choose an identity mapping path for each residual
block, at the t-th time-step, we obtain the overall nack-propagation chain I
dz'(d] az'(d]
ar' 0e'-11
H' (s' [t] 8¹). Since s' s'[t] is binary variable, the value of H' (t'( 6¹) can be less or greater than 1. Consequently, as the number of blocks increases, there possibility that I1 (1 0¹) + 0 or II (1'(s'-(((] - 0') + +oc, leading to the ineffectiveness of identity mapping and gradient calculation.
The inability to differentiate the current response through extracting past information. Con- sidering two LIF neurons and in the I-th layer, if etit 1 0/ii 1 1] and I [t. both neurons and j will exhibit identical spike response and membrane potential renewal at the t-th step. However, during the preceding
steps, neurons i and may actually have different spike firing sequences [s][1),-_ zt - 1]} and [.. st
1]}. which corresponds to different latent information. Unfortunately, the LIF model lacks the capability differentiate between neurons and
j at the i-th time step based on their previous spike firing patterns.
4.2 LEARNABLE MULTI-HIERARCHICCN (LM-H) NEURON MODEL
decision generate spikes is determined by the specific membrane potential of the soma layer. We
4

---

Published as a conference paper at ICLR 2024
a'o(()+b
Bound
Optimization Interval
m5
Initial Value
19
Lower Hound
Learning Function for Membrane Parameter
(b) From Initial State (IF/IIF Neuron) Optimal State (L. -H Neuron)
Figure 3: Progressive STBP Training framework for the LM-H Neuron.
provide dynamic description of the LM-H model follows.
o [t] p v 1]
[t]. - s s( mls( = N/so'st - 1| X xt).
s [t] = m
I't [ w "'j s s(t) (m'slt) -0)) {b.
if m 2 0¹ otherwise
(4)
0.
Here ob is the dendrite membrane potential at the f-th step. m's [t] and v's1 denote the soma potential before and after firing a spike, respectively, uD uls, Nb. A/s are the relevant scaling factors of the LM-H model.
Interpreting the variables and parameters in the LM-H model. In order to facilitate a comprehen- sive analysis of this multi-layer neuron, we can transform the LM-H model into single-layer form by applying E4) iteratively along the temporal dimension. This allows us to express all variables in terms of equations involving v'slt and [t].]. Here we set v to simplify the expression.
b [t]
ulsoss(t - 1] r' [t] + 1 1]
(u/so//lt - 1] r -tt 2) 1)) (2b)))
(ubb)* (ul (kk))
(5)
k=1
m/ Nsu'stt - 1] + Xb
t-1
Xbbub)) - (u'sossk +]+
1'))) + ((tss + \br's 1] xt))
(6)
representation
historical representation
In (5), can find that vi [4] retains information about the membrane potential and input current at each previous time step, which implies that the dendrite layer plays a role in long-tern term memory. Furthermore, (6) reveals that the LM-H neuron extracts two types of information, as depicted in Fig.2. One type is historical information obtained from previous time-steps, while the other type corresponds the current information, which aligns with the behavior of vanilla LIF neuron. Among them, ADD determines the overall proportion of historical information extraction, while p's describes the ratio of previous potential information in the historical representation. A's affects the extent of current potential leakage, and X scales the input current. Notably, X also plays a crucial role in regulating gradient calculation in deep residual structure. Regarding the possible gradient failure on the identity path mentioned above, the corresponding calculated gradient chain for the LM-H model becomes I ADH (X''s' (f 0'), which means that appropriately chosen values for Xb, can ensure proper gradient caleulations in deep networks.
Furthermore, from (6), we note that the current LM-H model can only obtain historical in- formation through the form of exponential decay by regulating d'p. which cannot flexibly deter- mines the proportion of information extraction at each time-step. Therefore, as shown in Fig.2, we also consider more radical version where membrane-related parameters each time-step
5

---

Published as a conference paper at ICLR 2024
slicen n
slicent
Sina
0
Pbz
Rg
9t-1
(a) Hybrid Training
(b) Time-slicing Online Training
Figure 4: Efficient Training framework for the LM-H Neuron.
(b.. Nss.. Nb,)) ) to learnable, resulting in (6) becoming the following form:
4 Σ
m's(]]
II k=1 j=k+1
(7)
4.3 PROGRESSIVE STBP TRAINING FRAMEWORK FOR LM-H MODEL.
From (6), observe that when 1DD p's = 0 and X) 1, the LM-H model degenerates into vanilla LIF model with decay constant equal to Xz.. This indicates that the LIF model is special case of the LM-H model. Inspired by this, we propose to progressively optimize the model parameters (e.g., 1dD, uls. Xb, Xz)) during the training process, starting with the degradation of the LM-H model to the LIF model.
As illustrated in Fig.3, we set p uds. Xb in each layer learnable parameters. To optimize these parameters stably and smoothly. we propose a scalable-shiftable sigmoid learning function defined as h(x) while b represents the initial position where the parameters can be chosen, and (xx) denotes the sigmoid function. In our training process, we μ' := d'
(xx) b, where controls the length of the parameter optimization interval,
dn,D D o (2)),)) + by by. where s/u, the actual learnable tensor. To maintain consistency with the initial condition of our progressive training, we set a' d a' u.S = A' ax 1, b b b' bh.s -0.5, bb.D byy 0.5 and the initial values of x D. D. - u... Th x.s to O.
Eqs.(8)-()) outline the back- proppproppg chain of our progressive training framework. As shown in Fig.3, our gradient calculation involves spatial and temporal dimensions. For spatial dimension, further involves two distinct levels: the dendrite-level and soma-level. During the training process. 1u
0u set 0miIt
1 rather than Tm'Il 1 0¹ Tm This choice allows the connection strength of
0a'(t]
various propagation chains to be entirely regulated by the parameters u'D M's. Xbr Ns.
OC
OC (s'(t)
OC
3mly
[t +1
[ml)(()
as [t] 0m|s(
3m [1. +
3m's
ac
ac 0ml'(t)
Oc
@ublt +1]
(8)
[t]
Öm'glt) 0t
Dublt 1| 0ub
0mlI (t + 11
@ml'lt + 00% 0m'zlt
du [t+
N/s
Xouls.
[m
00's1t] 1mly
dmlyt
dub [t + +1] dvb + dvb [t 11 @ml'(
u'n + XDuss
(9)
0ul 0 @mlglt
4.4 OPTIMIZING THE LM-H MODEL THROUGH EFFICIENT TRAINING
Based previous discussions, have established that the LIF model is special case of the LM-H model. In this section, we aim to extend the hybrid training method proposed in (Rathi & Roy, 2023) and the online training method proposed in (Xiao et al., 2022) to the LM-H model.

---

Published as a conference paper at ICLR 2024
Table Comparison with previous SOTA works. * denotes improved network structure.
Dataset
Method
Architecture
Time-steps
Accuracy( %-)
STBP-1dBN (Zheng et al., 2021)
ResNet-19 19
4
92.92
Dspike (Li et al., 2021)
ResNct-18
4
93.66
TET (Deng et al., 2022)
ResNet-19
4
94.44
CIFAR-I0
ResNet-18
4,
94.67, 94.88
GLIF (Yao al., 2022)
ResNet-19
4,
94.85, 95.03
ResNet-11
95.62
Ours
Re
4
96.36
Dspike (Li et al,, 2021)
ResNet-18
73.35
TET (Deng al., 2022)
ResNet-19
4
74.47
ResNet-18
4.
76.42, 77.28
GLIF (Yao et al., 2022)
ResNet-19
4,
77.05, 77.35
CIFAR-100
TEBN (Duan al., 2022)
ResNet-19
4.
78.71, 78.76
ResNet-18
4
78.58
Ours
ResNet-19
4
80.31
ResNet-19
4
81.65
DCT (Garg al., 2020)
VGG-13
125
$6.90
Online-LTL (Yang et al., 2022)
VGG-13
16
54.82
ImageNet-200
Offline-LTL. (Yang et al., 2022)
VGG-13
16
55.37
ASGL (Wang et al., 2023)
VGG-13
4.8
56.57, 56.81
Ours
VGG-13
4
59.93
STBP-dBNN (Zheng cl al., 2021)
ResNet-19
10
67.80
RecDis -NNN (Guo et al., 2022a)
ResNet-19 19
10
72,42
DVS-CIFARIO
MPBN (Guo al., 2023)
ResNet-19
10
74.40
Ours
ResNet-19
10
79.10
Hybrid Training. Similar to the LIF model, the average spike firing rate of the LM-H model also follows approximate linear relationship between layers. By combining the dynamic equations for v's(() and [ from Eq.(4), and then performing sum and average operations along the time dimension, arrive at the following equation:
ΣT= s' it
ET
w's [t)
EL 8h
T
8'stt)
-1=1
T
T
T
T
5 -1-
mbbbl -1],
1],
8/5(1 0's1 (X's rls)os - 1]. (10)
Here T denotes the total time period. 1/ and 0's(1] the conversion errors generated by the dendrite and soma layer. When μ³D X 1, 1, X/s 1. the impact of conversion errors is further reduced. By defining ['[ Σt= 8'11 /T, we obtain the following simplified results.
rd [[T] = - ( (1- Xb (T] mbobl0
D's [T) - v'/10]
(11)
T
One can find that (1I( further simplifies to r'[T] W' -' when X
T
1, p = O. In this case, the relationship between the spike firing rates of adjacent layers in the LM-H model aligns completely with that of the LIF model. To leverage this similarity, as depicted in Fig,4(a), we initially utilize the conventional ANN-SNN conversion framework (Bu et al., 2022) to obtain a specialized LM-H model with Xt, and p'n
We then proceed to optimize the model parameters through the progressive leamning approach mentioned earlier. Notably, the progressive learning stage requires only a small number epochs (typically < 30) to significantly enhance the performance of the LM-H model. Considering that the time and memory costs of ANN-SNN conversion are (11) and the corresponding costs of STBP are O(T), our proposed hybrid training method will significantly save energy consumption.
Online Training. For the LIF model, previous works achieved gradient renewal at each step by can- celing the dependencies between propagation chains at different time-steps (e.g. 8m 0m [t]/0m 1)), ensuring that memory consumption does not increase linearly with the growth of time-steps. However, for the LM-H model, it is not possible to cancel the gradient calculation for the learnable parameters connecting propagation chains from different time steps (e.g. uDD u.g. Xg). To address it, we propose online training framework called time-slicing training. As shown Fig. 4(b), we group consecutive
7

---

Published as a conference paper at ICLR 2024
Table 2: Performance of hybrid training for LM-H model.
Dataset
Method
Architecture
Time-steps
Accuracy( %)
ResNet-18
4
93.66
QCFS (Bu et al., 2022)
ResNet-20
83.75
CIFAR-10
R8
4
94.02
Ours
ResNet-20
87.56
VGG-16
69.62
QCFS (Bu et al., 2022)
ResNet-20
34.14
CIFAR-100
VGG-16
73.11
Ours
ResNet-20
57.12
QCFS (Bu et al., 2022)
VGG-13
45.15
ImageNet-200
Ours
VGG-13
49.09
Table 3: Performance of time-slicing online training on ResNet- 18.
Dataset
Method
Time-steps
Accuracy( %)
SLIT (Meng et al., 2023)
6
94.44
CIFAR-10
3 time-steps per slice, 2 slices
4, 6, 8
95.05, 95.42, 95.49
SLTT (Meng al., 2023)
6
74.38
2 time-steps per slice, slices
4, 6, 8
76.27, 77.10, 77.56
2 time-steps per slice, slices
4, 6, 8
75.99, 77.35, 77.81
CIFAR-100
2 time-steps per slice, 4 slices
6. 8
74.81, 76.28, 77.01
time-steps per slice, 2 slices
4, 6,
77.28, 78.21, 78.66
time-steps per slice, slices
6, 8
77.23, 78.30, 78.59
k time steps into single slice, effectively dividing the total time period into [T/k] slices. Within each slice, we cancel the gradient calculation between different slices. After each time slice ends, gradient refresh will be performed to ensure that the total memory overhead does not exceed (l)).
5
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
To validate the effectiveness of our proposed model and training frameworks, we conduct experimen- tation various datasets, including CIFAR Family (Krizhevsky et al., 2009), ImageNet-200 (Deng et al., 2009), and DVS-CIFAR10 10 (Li et al., 2017) datasets. We specifically select mainstream network architectures that consistent with previous works, including VGG (Simonyan & Zisserman, 2014) 4) and ResNet (He et al., 2016). More additional experiments and details are provided in the Appendix.
5.2 COMPARISON WITH PREVIOUS STATR-OF-THE-ARY WORKS
CIFAR Family (standard dataset). We have selected previous state-of-the-art works in various areas of SNN learning benchmarks for comparing
proposed method. These benchmarks include optimization of learning function (Deng et al., 2022), surrogate gradient (Li et al., 2021), batchnorm layers (Zheng al., 2021; Duan al., 2022), and membrane-related parameters (Yao et al., 2022). As shown in Tab. the LM-H model demonstrates significant advantages over other methods in terms of performance. For example, the CIFAR-100 dataset, the LM-H model achieves top-1 accuracy of 78.58% and 80.31 % on ResNet-18 and ResNet-19 19 with 4 time-steps, respectively, which 2. 16% and 3.26% higher than GLIF on the corresponding network architecture. Furthermore, for ResNet- with the improved structure, our method outperforms TEBN with 2.94% accuracy improvement under 4 time-steps. is worth noting that the performance of our method at steesttess significantly surpasses that of other works at 6 time-steps for the CIEAR- 10 and CIFAR-100 100 datasets.
ImageNet-200 (large-seale dataset). From Tab. we can note that the performance of our progressive training is 3.36% higher than ASGL, current advanced neuron model with adaptive smoothing gradient. Additionally, our model exhibits superior performance at 4 time-steps compared to other methods (Garg al., 2020; Yang al., 2022: Wang al., 2023) at higher time-step ranges of 8-1. DVS-CIFARIE (neuromorphic dataset). We also evaluate the representation ability of the LM- H model on the neuromorphic dataset. Compared to STBP-ddN (Zheng et al., 2021), RecDis-
8

---

Published as a conference paper at ICLR 2024
78
77/22
78.33
X2
79.39
82
Accurscy
Accuracy
Accuracy
78.77
Accuracy
74
73.95
74
76
7617
24
78
70
74
12
Time-steps
Time-steps
Time-stops
Time-stops
(a) ResNet-18. 18, T=2
(b) ResNet--8 18, T=3
(c) ReRNet--1,, 18, T=6
(d) ResNet-19, T-4
Figure 5: Performance of LM-H neuron (which is obtained after T-steps training) at different time- steps on the CIFAR-100 dataset.
SNN (Guo et al., 2022a) and MPBN (Guo al., 2023), our model achieves accuracy improvements of 11 11.30% 6.68% and 4.70%, which demonstrate the effectiveness of our proposed method.
5.3 PERFORMANCE ANALYSIS OF EFFICIENT TRAINING FRAMEWORK
Hybrid Training, We utilize the QCFS (Bu et al., 2022) as our ANN-SNN conversion framework during the first stage to obtain an LM-H model under special conditions (u'p = 0, X 1. + +/3- 1). For the progressive training stage, since we will merely optimize the LM-H model with very limited iterations (30 epochs), we can consider this stage as a fine-tuning process. For the CIFAR Family and ImageNet-200 datasets, we set the initial learning rate as 2 x 10 and 1 x 10-4 respectively. As shown in Tab.2, we note that the second stage still plays a crucial role, even with a finite number of optimization iterations. This is particularly evident for QCFS conversion models that exhibit poor performance under low time-latency conditions. For example, on the CIFAR- 100 dataset, achieve accuracy improvements of 3.49% for VGG-16 and 22.98% for ResNet-20 during further optimization within 30 epochs.
Online Training. We have chosen the current state-of-the-ar work of online training (Meng et al., 2023) as the target for our comparison. As shown in Tab.3, our proposed scheme (3 time-steps per slice, slices) outperforms SLTT by 3.83% on the CIFAR- 100 dataset with 6 time-steps. addition, we investigate the impact of slice length and slice quantity on network performance. It is evident that using a very short slice length (e.g. 2 time-steps per slice) may lead slight decrease in training precision. On the other hand, increasing the number of slices excessively (e.g. 4 slices) does not yield further improvements network performance.
5.4 LONG-RANGE PERFORMANCE REWARD FOR LM-H MODEL
We also discover an interesting phenomenon regarding the LM-H model: despite using specific training steps during the learning process, the model's precision can still be further improved as the number of time-step increases during the inference stage. As shown in Fig.5, we use pink dotted lines to indicate the accuracy achieved under specific training steps, while purple numbers represent the peak performance of our model during the inference process. Across various training parameter configurations, observe that the models exhibit improvement range of approximately 0.6% to 3.3% over an extended period of time. We named this phenomenon as "Long-Rane Range Performance Reward", which indicates that the membrane-relater parameters we have learned during the training stage (e.g. p'D u's, X/s, XD) can still effectively extract information over a prolonged time range.
CONCLUSIONS
In this paper, we propose the LM-H neuron that can comprehensively utilize historical and current information. We demonstrate that the LM-H model the generalization of LIF neurons and addresses the calculation deficiencies associated with LIF neurons. Moreover, we design progressive STBP training framework to dynamically regulate historical and current representation, and propose efficient training schemes significantly reduce the computational overhead. Considering that our method has outperformed previous state-of-the-as works in multiple experiments, we believe that the proposal of the LM-H model will further promote relevant research in the field of advanced spiking neurons.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (62176003, 62088102) and by Beijing Nova Program (20230484362).
REFERENCES
Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pp- 421-436. Springer, 2012.
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ANN-SNN conversion for high-accuracy and lltra-low-latency spiking neural networks. In International Conference on Learning Representations, 2022.
Tong Bu, Jianhao Ding, Zecheng Hao, and Zhaofei Yu. Rate gradient approximation attack threats deep spiking neural networks. In IEEE Conference on Computer Vision and Pattern Recognition. 2023.
Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1):54-66 66, 2015.
Yanqi Chen, Zhaofei Yu, Wei Fang, Zhengyu Ma, Tiejun Huang, and Yonghong Tian. State tran- sition of dendritic spines improves learning of sparse spiking neural networks. In International Conference on Machine Learning, 2022.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In JEEE Conference on Computer Vision and Pattern Recognition, 2019.
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018.
Michael V DeBole, Brian Taba, Arnon Amir, Filipp Akopyan, Alexander Andreopoulos, William Risk, Jeff Kusnitz, Carlos Ortega Otero, Tapan K Nayak, Rathinakumar Appuswamy, et al. TrueNorth: Accelerating from zero 64 million neurons in 10 years. Computer, 52(5):20-29 2019.
Jia Deng, Richard Socher, Lijia Li, Kai Li, and Feifei Li. Imagenet: A large-scale hierarchical image database, In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. International Conference Learning Representations, 2022.
Terrance Dries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint /708 04552, 2017.
Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Advances in Neural Information Processing Systems, 2022.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Advances in Neural Information Processing Systems, 2021a.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothee Masquelier. Tiejun Huang. and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021b.
Isha Garg, Sayeed Shafayet Chowdhury, and Kaushik Roy. DCT-SNN: Using det to distribute spatial information over time for learning low-latency spiking neural networks. arXiv preprint arXiv:2010.01795. 2020.
10

---

Published as a conference paper at ICLR 2024
Wulfram Gerstner and Werner M Kistler. Spiking Neuron Models: Single Neurons, Populations, Plasticity. Cambridge university press, 2002.
Yufei Guo, Xinyi Tong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Zhe Ma, and Xuhui Huang. RecDis-SNN: Rectifying membrane potential distribution for directly training spiking neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2022a.
Yufei Guo, Liwen Zhang, Yuanpei Chen, Xinyi Tong, Xiaode Liu, Yinglei Wang. Xuhui Huang, and Zhe Ma. Real spike: Learning real-valued spikes for spiking neural networks. In European Conference on Computer Vision, 2022b.
Yufei Guo, Yuhan Zhang, Yuanpei Chen, Weihang Peng, Xiaode Liu, Liwen Zhang, Xuhui Huang, and The Ma. Membrane potential batch normalization for spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. RMP-SNN: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In IEEE Conference on Computer Vision and Puttern Recognition, 2020.
Zecheng Hao, Tong Bu, Jianhao Ding, Tiejun Huang, and Zhaofei Yu. Reducing ANN-SNN con- version error through residual membrane potential. In AAAI Conference on Artificial Intelligence, 2023a.
Zecheng Hao, Jianhao Ding, Tong Bu, Tiejun Huang, and Zhaofei Yu. Bridging the gap between ANNs and SNNs by calibrating offset spikes. In International Conference on Learning Represen- tations, 2023b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Yifan Hu, Lei Deng. Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks towards deep residual learning. arXiv preprint arXiv:21 1rXiv:2112.08954 2021.
Jinseok Kim, Kyungsu Kim, and Jae-Joon Kim. Unifying activation- and timing-based learning rules for spiking neural networks. In Advances in Neural Information Processing Systems, 2020a.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-YOLOO spiking neural network for energy-efficient object detection. In AAAI Conference on Artificial Intelligence, 2020b.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: an event-stream dataset for object classification. Frontiers in Neuroscience, 11, 2017.
Yuhang Li. Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongging Hai, and Shi Gu. Differentiable spike: Rethinking gradient-desceen for training spiking neural networks. In Advances in Neural Information Processing Systems, 2021.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text
classification. In International Conference on Learning Representations, 2023.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models.
Neural Networks, 10(9):1659-1671, 1997.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhiquan Luo. Towards memory and time-efficient backpropagation for training spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Paul A Merolla, John Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with a scalable communication network and interface. Science, 345 (6197):668 673, 2014.
11

---

Published as a conference paper at ICLR 2024
Hesham Mostafa.
Supervised learning based on temporal coding in spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 29(7)3227-33233 2017.
Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-basee optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-66 2019.
Nitin Rathi and Kaushik Roy. DIET-SNN: A low low-natency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 34:3174-3183 174171822 2023.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607-617, 2019.
Jiangrong Shen, Qi Xu, Jian K. Liu, Yueming Wang, Gang Pan, and Huajin Tang. ESL-SNNs: An evolutionary structure learning strategy for spiking neural networks. In AAAI Conference on Artificial Intelligence, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 14X.v:1409.1556 2014.
Christoph Stöckl and Wolfgang Maass. Optimized spiking neurons can classify images with high accuracy through temporal coding with two spikes. Nature Machine Intelligence, 3(3):230-238, 2021.
Ziming Wang, Shuang Lian, Yuhao Zhang, Xiaoxin Cui, Rui Yan, and Huajin Tang. Towards lossless ANN-SNN conversion under ultra-low latency with dual-phase optimization. arXiv preprint arXiv:2205.07477. 2022.
Ziming Wang, Runhao Jiang, Shuang Lian, Rui Yan, and Huajin Tang. Adaptive smoothing gradient learning for spiking neural networks. In International Conference on Machine Learning, 2023.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training nigh-perffrmance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In AAAI Conference on Artificial Intelligence, 2019.
Mingging Xiao, Qingyan Meng, Zongpeng Zhang, Di He, and Zhouchen Lin. Online training through time for spiking neural networks. In Advances in Neural Information Processing Systems, 2022.
Qu Yang, Jibin Wu, Malu Zhang, Yansong Chua, Xinchao Wang, and Haizhou Li. Training spiking neural networks with local tandem learning. In Advances in Neural Information Processing Systems, 2022.
Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng. GLIF:
A unified gated leaky integrate-and-fire neuron for spiking neural networks. In Advances in Neural Information Processing Systems, 2022.
Zhaofei Yu, Shangqi Guo, Fei Deng, Qi Yan, Keke Huang, Jian K Liu, and Feng Chen. Emergent inference of hidden Markov models in spiking neural networks through Winner-take-all. IEEE Transactions on Cybernetics, 50(3):1347-1354, 2020.
Shimin Zhang, Qu Yang, Chenxiang Ma, Jibin Wu, Haizhou Li, and Kay Chen Tan. TC-LIF: A two-compartment spiking neuron model for long-term sequential modelling. arXiv preprint MrXiv:2308.13250. 2023.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with firectly-trained larger spiking neural networks. In AAAI Conference on Artificial Intelligence, 2021.
Shibo Zhou, Xiaohua Li, Ying Chen, Sanjeev T Chandrasekaran, and Arindam Sanyal. Temporal- coded deep spiking neural network with easy training and robust performance. In AAAI Conference on Artificial Intelligence, 2021.
Yaoyu Zhu, Zhaofei Yu, Wei Fang, Xiaodong Xie, Tiejun Huang, and Timothée Masquelier. Training spiking neural networks with event-driven backpropagation. In Advances in Neural Information Processing Systems, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 EXPERIMENTAL IMPLEMENTATION DETAILS
In this paper, we choose Stochastic Gradient Descent (SGD) optimizer (Bottou, 2012) and Cosine Annealing scheduler (Loshchilov & Hutter, 2017) for all experimental cases. Following the approach of previous works, we incorporate aata-augmentation techniques to enhance the performance of SNN models. including Cutout VDeVs & Taylor, 2017) and Auto-Augment (Cubuk et al., 2019). For the DVS-CIFAR 10 dataset, we resize the neuromorphic data to 48 x 48 pixels and integrate each image into 10 frames. In the progressive STBP training framework, we consider two optimization configurations: TET function (Deng et al., 2022) with vanilla batchnorm layers and Cross Enooss- function based on firing rate with tdBN layers (Zheng et al., 2021). We select the configuration that achieves optimal performance on the CIFAR and ImageNet-200 datasets. addition, for ImageNet- Ik dataset, choose the combination of Cross-Entropy function, tdBN layers and improved version of ResNet (Hu et al., 2021). For the hybrid training framework, our further training is based on the pre-trained QCFS models (Bu al., 2022) with L 4 or L where L denotes the quantization level of the QCFS function. For time-slicing online training, we adopt the TET function to optimize gradients for each time slice. The detailed I.
settings can be found in
Table S1: Experimental hyper-parameter configuration.
Framework
Dataset
Architecture
Optimizer (Ir, wd)
Batchsize
Epochs
ResNet-18
CIFAR-10
ResNet-19
ResNet-18
SGD (0.025, 10-4)
64
CIFAR-100
Re
300
Progressive Training
ResNet- 19"
ImageNet-200
VGG-13
DVS C
Re
SGD (0.1, x 10-*)
32 64 x 4
ImageNet-1k 1k
ResNet-34
SGD SGD
320
ResNet-18
CIFAR-10
ResNet-20
SGD (0.002, 5 10 -0
Hybrid Training
VGG-16
CIFAR-100
ResNet-20
ImageNet-200
VGG-13
SGD (0.001, 5 x 10-*)
CIFAR-10
time-steps per slice
SGD (0.0125, 5 x 10-4)
CIFAR-100
CIFAR-10
time-steps per slice
ResNet-18
SGD (0.01875, 5 x 10-4)
64
300
CIFAR- 100
CIFAR-10
time-steps per slice
SGD (0.025, 5 10-4)
CIFAR-I00
A.2 PERFORMANCE OF THE LM-H MODEL UNDER SPARSE LEARNING
We further explore the representation ability of the LM-H model under sparse weights by employing the STDS pruning strategy (Chen et al., 2022) to sparsify the weights of SNN models. The specific equations for weight sparsification described as follows:
Wi sparse sign( W max W d,0),
(S1)
d'
sin
π
+ 1
D.
(S2)
Here d, d' represents the clipping width of the current weights, while denotes the final target width. Ta is the total number of updates to d during the training process. Throughout the sparse training procedure, d continuously increases according to Eq.S2, with set to 0.05.
Tab.S2 demonstrates the superior capability of the LM-H model in sparse learning. Even when the sparsity exceeds 80%, we still outperform ESL-SNN (Shen et al., 2023) by 4.94% and 5.83% on the CIFAR-10 and CIFAR-100 datasets, respectively. Notably, the performance of the LM-H model under
13

---

Published as a conference paper at ICLR 2024
0.4
0.3
parameter value
0.2
0.0
-0.1-
-02
Hb
-0.3
-0.4-
3
4
5
6
9
10
11
12
13
14
15
16
17
18
Layer#
1.4
1.3
X
1.2
A
parameter value
1.0
0.9
0.8
0.7
0.6
0.5
2
3
4
5
6
10
11
12
13
14
15
16
17
18
Layer#
Figure $1: The learning situation for LM-H model on DVS-CIFAR10, ResNet-19.
sparse weights is significantly better than that of previous works under normal learning. In addition, we observe that despite trimming more than 80% the weights, the LM-H model's performance does not exhibit significant degradation (with accuracy loss generally below 1%).
Table S2: Comparison of sparse STBP training with previous works.
Dataset
Method
Architecture
Sparsity %)
Accuracy %)
STBP-1dBN (Zheng et al., 2021)
92.92
TET (Deng et al., 2022)
94.44
ResNet-19
GLIF (Yao al., 2022)
94.85
CIFAR-10
ESL-SNN (Shen et al., 2023)
50.00
91.09
89.59
96.03
Ours
ResNet-19
96.36
TET (Deng et al., 2022)
74.47
GLIF (Yao al., 2022)
ResNet-19
77.05
CIFAR-100
ESL-SNN (Shen et al., 2023)
50.00
73.48
81.60
79.31
Ours
ResNet-19
80.31
A.3 LEARNING SITUATION OF MEMBRANE-RELLATEE PARAMETERS
We visualize the learning results of the relevant membrane-relateee parameters the LM-H model. As shown in Fig.SI. HDD U's, X/. XD dynamically learn suitable values for different layers, achieving effective global information extraction layer by layer.
14

---

Published as a conference paper at ICLR 2024
A.4 DETAILED EXPLANATION ABOUT EFFICIENT TRAINING FOR THE LM-H MODEL
Hybrid Training. The first stage of the hybrid training framework is an ANN-SNN Conversion with Cunniizaiio-Clii---ii Shift (QCFS) function, which can be described follows:
0! a' = clip (1 a a''-1 0' (1=='
...L)
(S3)
L
Here a' represents the activation output of the QCFS ANN model in the 1-th layer and L denotes the quantization level. Bu et al. (2022) has proved that the discrepancy between ('( and a' will be optimized minimum value for IF model under the condition of adopting QCFS function. As the LM-H model will degenerate to a vanilla IF model when μtD = O, p's + X = 1, = 1, we can obtain an LM-H model under special conditions through QCFS ANN-SNN Conversion.
Once the ANN-SNN Conversion procedure is completed, we will copy the weights and membrane- related parameters from the QCFS ANN model to the corresponding positions in the LM-H model. Due to the fact that the dendrite layer is actually hidden when PDD = O, the initial value of the membrane potential 0' /2 and the learnable firing threshold 6' are directly copied to the soma layer.
During the progressive training stage, we introduce learnable parameters for the membrane-related factors and weights to facilitate more stable fine-tuning, However, the firing thresholds are treated as scalars. As the QCFS function simulates the average firing rate of an LM-H neuron over L steps, we typically set the training step T" at this stage to L. The overall algorithm description can be found in Algorithm
Algorithm Hybrid training framework for the LM-H model.
Require: Pretrained QCFS ANN model J'NN (W 0) with L layers; Dataset D; Number of time- steps used for training T: Loss function C: Cross- Eatss-E function LCg-
Ensure: SNN model fsNN W 8, UD. Us, 8, JDD us, Ap).
1: # Convert ANN to SNN
2: for I 1 to T. do
3:
fSNN. W = fANN. W
4:
fsNN. O fANN.0'
5:
fsNN 0',11' 0, SENN.Ug/00 = SSNN. O'
6: JSNN-J1p 0, JSNN- + SSNN- = JSNN X1p =
7: end for
8: # Progressive training for LM-H model
9: # Set μ.D. Mss, Às, Ap as learnable parameters and as scalars
10: for (Image,Laae Label) in D do
11: # LM-H model performs forward propagation based on (4.
for = to L do
13: if Use tdBN layer then
[fsNN. [1]...... [ell.....ssss fSNN ((T)s = fsNN(1.ssNN W! fsNy. ([1], [.], fSNN. W s'
15:
end if
if Use vanilla batchnorm layer then for /o to 7 do
17:
18:
fSNN. s' [t] = (SNN(SNNN. W s'
19:
end for
20:
end if
21:
end for
22:
if Use
TET loss function then L= E Lan Label)
23:
24:
end if
if Use vanilla loss function then
26:
C= CcE(# Et=1 fsNN.s [tt], Label)
27:
end if'
28: # LM-H model performs back pock-propagation based on (8)
end for
30:
return SSNN( W, 0, UD, U5. uss An, As, An)
15

---

Published as a conference paper at ICLR 2024
Time-slicing Online Training. Algorithm 2 has depicted the overall procedure of time-slicing training. Assume two adjacent & time-steps slices: slice n and slice n + 1. Within the slice, we perform normal progressive STBP training for the LM-H model. For the calculation chains between Oml [mx de beb[ox
we separate them from the entire gradient calculation graph, which means that forward propagation can calculate normally on these chains but back-po prcp-prpagattinn cannot. During back propppropagttion the slices are completely detached from each other, which achieves gradient updates slice by slice and effectively saves computational memory overhead.
slices (e.g. 0 (nxk
[ax
Algorithm 2 Time-slicing online training framework for the I.MHH model.
Require: Dataset D; Time-slice containing time-steps; Number of time-steps used for training T;
Loss function C; Cross-Entropy function Log.
Ensure: SNN model fSNN W, 0, UD Us, 8, Ms, Äs, XD).
Set p.D. ,s,, As Ap as learnable parameters and scalars
2: for Immge,Label in D do
3: for slice = 1 to [T/k]] do
4: for / to L do
5:
LM-H model performs forward propagation based on Eq.(-)
6:
if Use tdBN layer then
7:
{fSNN s (slice- - 1) + .l...., JSNN, of x slice]} = faNN(1fsNN W of
(slice 1) 1),,,,.,,,,, ..., JSNN. W W' fsNN. I slicel])
8:
end
9:
if Use vanilla batchnorm layer then
10:
for t= to k do
fsNN. (slice 1) + = Bo W' s' (slice 1) + f)) end for end
12:
14:
# Detach fsNN.bbll x slice), fssN. slice] from the gradient computational graph
end for
16:
# LM-H model performs gradient updates for each time-slice
17:
Use TET loss function then
18:
L=
E k kxslice
(CE(NNNNss [t], Label)
19:
end if
20:
Use vanilla loss function then
21:
L LCE 2t= 1)+1 JSNN-s' s [t], Label)
22:
end if
23:
# LM-H model performs back proppaaapantionn based on (8)
end for
25:
end for
return SSNN W, 0, UD, vs, 8, (,, AD)
Table S3: Experimental results about the radical version on multiple datasets.
Dataset
Method
Architecture
Time-steps
Accuracy(%)
Ours
ResNet-18
95.62
CIFAR-10
Ours (radical version)
ResNet-18
4
95.82
Ours
ResNet-18
78,58
CIFAR-100
Ours (radical version)
ResNet-18
4
78.90
Ours
VGG-13
4
59.93
ImageNet-2(0)
Ours (radical version)
VGG-13
4
60.37
STEP-1dBN (Zheng et al., 2021)
ResNet-34
6
63.72
TET (Deng et al., 2022)
ResNet-34
6
64.79
RecDis-SNN (Guo et al., 2022a)
ResNet-34
67.33
ImageNet-Ik
MBPN (Guo et al., 2023)
ResNet-34
4
64.71
SEW ResNet (Fang et al., 2021a)
ResNet-34
67.04
GLIF (Yao et al., 2022)
ResNet-34
67.52
Ours (radical version)
ResNet-34
4
69.73
16

---

Published as a conference paper at ICLR 2024
A.5 COMPARISON BETWEEN THE VANILLA AND RADICAL VERSIONS THE LM-H MODEL
As shown in (6), the overall proportion term for historical information is represented as (=b) indicating that the extracted proportions different time-steps follow exponential distribution. In contrast, in Eq. the corresponding term changes to I Hb.5- allowing for more freedom and randomness in the extracted proportions. In addition, in the radical version, relevant factors that regulate the membrane potential (u5,) X(s,) and input current (Ab,)) are independently assigned at each sime-step, making the information representation range of the radical version wider than that of the vanilla version.
Tab.S3 shows the relevant performance of the radical version on CIFAR and ImageNet datasets, which demonstrates that the radical version does have the advantage of learning membrane-relate parameters more precisely. addition, during the experimental procedure, noticed that the convergence speed of the radical version of the training accuracy was also ahead of the vanilla version, especially on large-scale datasets.
A.6 ADDITIONAL EXPERIMENTS ON VGG STRUCTURE
We have also conducted additional experiments using the VGG architecture the CIFAR-10 and CIFAR-100 datasets. As shown in Tab.S4, our results demonstrate superior performance compared to previous SOTA methods, even when employing smaller network structures and fewer time-steps.
Table $4: Comparison with previous works the CIFAR datasets.
Dataset
Method
Architecture
Time-steps
Accuracy(%
Diet-SNN (Rathi & Roy, 2023)
VGG-16
5
92.70
Real Spike (Guo et al., 2022b)
VGG-16
92.90, 93.58
CIFAR-10
Ours
VGG-13
4
93.98
Ours (radical version)
VGG-13
94.80
Diet-SNN (Rathi & Roy, 2023)
VGG-16
5
69.67
RecDis-SNN (Guo et al., 2022a)
VGG-16
69.88
CIFAR-100
Real Spike (Guo et al., 2022b)
VGG-16
10
70.62, 71.17
Ours
VGG-13
4
73.99
Ours (radical version)
VGG-13
74.79 4
17


Reviews:
[{"review_id": "B7SjTtX2Ep", "reviewer": "Reviewer_SiwN", "summary": "The paper extends the LIF model for SNN to a more generalized version called LM-H, enhances its flexibility by making certain parameters learnable, and designs a progressive learning procedure to effectively train the network. Some experiments on relatively small datasets were presented to show that the proposed approach is superior to relevant prior methods.", "questions": "Why only small datasets were experimented upon, was it because the proposed approach has scalability issues on larger and more practical datasets?\n\nWhy the performance of the radical version was only presented for ImageNet while missing for other datasets, was it because that it didn't show better accuracy?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The LM-H model and the learning algorithm were respectively inspired by and similar to existing works, however put together the paper still proposed a novel and practical framework for SNN learning.", "weaknesses": "The experiments only covered several relatively small datasets. The ImageNet 200 dataset was the largest one in the paper is actually a tiny subset of ImageNet. Datasets like ImageNet 1k/22k would be more convincing to validate the practical value of the proposed approach.\n\nPerformance comparison in the experiment section is somewhat inconsistent, e.g. only ImageNet 200 results included the radical version showing better performance.\n\nGrammar errors scatter through the paper, further proof-reading is suggested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VAdUBqpS0S", "reviewer": "Reviewer_uzHG", "summary": "The authors of this paper address the limitations of the widely used LIF model in SNN by proposing a novel LM-H model. The LM-H model overcomes issues related to gradient calculation in deep networks and capturing global information along the time dimension. The authors also develop a progressive training algorithm specifically for the LM-H model. Experiments on various datasets demonstrate the superior performance of the LM-H model and the training algorithm compared to previous state-of-the-art SNN approaches.", "questions": "1. LIF neurons have a gradient vanishing or exploding problem in deep residue architecture, how did LH-M solved this problem?\n\n2. ImageNet and CIFAR are all static image classification datasets, but LH-M involves historical information and performed better. Can you give an explaination?\n\n3. How did you perform ANN2SNN conversion? ANN do not include temporal information, but in LH-M historical information is an important property.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1 Did a lot baselines and achieved SOTA performance\n2 A novel neuron model LH-M that is a extension of LIF model", "weaknesses": "1. In the context of deep residual architectures, LIF neurons are known to exhibit issues to either vanishing or exploding gradients. How has the LH-M model effectively addressed and mitigated these challenges?\n\n2. Both ImageNet and CIFAR datasets focus on static image classification, whereas the LH-M model incorporates historical data and demonstrates superior performance. Could you elucidate the underlying reasons for this enhanced performance in the context of LH-M's utilization of historical information?\n\n3. How was the conversion from Artificial Neural Networks (ANN) to Spiking Neural Networks (SNN) executed, given that ANNs do not inherently incorporate temporal information? Considering the significance of historical data within the LH-M model, how was this temporal aspect effectively integrated during the conversion process?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VB0XDsitqz", "reviewer": "Reviewer_T4SQ", "summary": "This work proposed a new neuron model, Learnable Multi-hierarchical (LM-H) model, to effectively extracting global information along the time dimension and propagate gradients in deep networks. In the LM-H model, the scaling factors from the dendrite layer regulate the proportion of historical information extracted by the model, while the factors from the soma layer determine the degree of potential leakage and the intensity of the input current at present.", "questions": "Does the author address the gradient vanishing & exploding problem in deep residual architectures with the new LMU model? \nHow? \nThe authors listed results of cifar data set on resnet, can you provide results on vgg as well?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed LMH model, along with GLIF, TC-LIF, enriches the family of the spiking neuron models. This approach is unique and original, as it combines existing ideas in a new way to solve a problem in Spiking Neural Networks. The proposed model and training algorithm address the deficiencies of the widely adopted LIF model and offer a new approach to solving problems in this field.", "weaknesses": "While the proposed model and training algorithm are unique and original, the paper could benefit from a more detailed discussion of why they propose the new model, how they differ from existing methods, and what specific contributions they make to the field of Spiking Neural Networks. \nWhat are the significant deficiencies in deep layer gradient calculation and capturing global information on the time dimension, as mentioned in this paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "urywRzBM3L", "reviewer": "Reviewer_LLZy", "summary": "This paper analyzes the limitations of the vanilla LIF neuron model, including the problems of gradient vanishing and exploding, as well as the inability to differentiate the current response by extracting past information. Based on these analyses, the authors propose a novel learnable multi-hierarchical model that has a wider calculation scope along the time dimension by incorporating the functions of dendrite and soma. Additionaly, they further design a progressive STBP training framework for the LM-H model.", "questions": "1. Please provide a more comprehensive explanation and analysis of the radical version of the LM-H model, and discuss its advantages compared to the vanilla LM-H model.\n\n2. Regarding the hybrid training framework of the LM-H model, the author used the conversion framework based on the IF model during the ANN-SNN conversion phase. How about LIF neurons? Does it fit into this framework?\n\n3. The authors have illustrated the distinction between their work and that of TC-LIF in the related works section. I suggest adding it into the introduction as well.\n\n\n-------------------------------------------\nThank you for the clarification and for incorporating my suggestions in the revised version! Having read all the reviews, I concur with the opinions of the other reviewers that the authors have made a substantial contribution to developing novel neuron models for SNNs. This work will be of sufficient significance to advance the field of neuromorphic computing.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.The proposed model is neural-inspired.  \n\n2.The work is solid. The authors provide rigorous theoretical analysis of the LM-H model and demonstrate that the LIF model is merely a subset of the LM-H model.\n\n3.The progressive training method efficiently solves the multi-parameter learning problem of the LM-H model.\n\n4.Experimental results demonstrate the significant advantages of the LM-H model across multiple datasets.", "weaknesses": "1.The authors illustrate the issues of gradient vanishing and exploding faced by the LIF neuron. However, they have not demonstrated how the proposed LM-H model addresses these problems.\n\n2.The figures displayed in this paper solely depict the vanilla LM-H model. It would be good if the authors can also integrate the radical version of the LM-H model into these figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)